Lesson 10 of 13 · about 18 minutes · A perfect score on training can still fail the test.
⚡ 0 XP
Step 1 of 2
The student who mugged up
Two students prepare for maths. One memorises every answer in the guide. The other understands the method. In the guide’s questions, both score 100%. In the real exam, with new numbers, the memoriser is lost.
Models do this too. A model that fits the training data too closely — every wiggle and every bit of noise — is overfitting. It looks perfect in training and fails on new data. A model that captures the general pattern generalises.
Try it above: raise the wiggliness slider. The training error keeps dropping, but watch the test error — after a point it shoots up.
▶ Try it Overfitting: memorising vs understanding
● Training error0.026
◆ Test error0.071
The model learns only from ● (training) points. It never sees ◆ (test) points. Raise the wiggliness: training error keeps falling, but after a point the test error climbs.
🎯 Challenge: A model that only memorises +15 XP
Build the laziest “model” possible — a lookup table:
memorise(xs, ys) returns an object where key xs[i] stores ys[i]
lookupPredict(model, x) returns the stored value if x is in the model, otherwise 0 (it has no idea!)
Try twice to unlock the solution (0/2)
script.jsCtrl ⏎
Console
Output from console.log appears here.
Checklist
Press Run to check your code.
○
It remembers every training answer
○
Anything new gets 0
○
A stored 0 is still remembered
Lesson notes (text version)
1. The student who mugged up
Two students prepare for maths. One memorises every answer in the guide. The other understands the method. In the guide’s questions, both score 100%. In the real exam, with new numbers, the memoriser is lost.
Models do this too. A model that fits the training data too closely — every wiggle and every bit of noise — is overfitting. It looks perfect in training and fails on new data. A model that captures the general pattern generalises.
Try it above: raise the wiggliness slider. The training error keeps dropping, but watch the test error — after a point it shoots up.
Challenge: A model that only memorises
Build the laziest “model” possible — a lookup table:
memorise(xs, ys) returns an object where key xs[i] stores ys[i]
lookupPredict(model, x) returns the stored value if x is in the model, otherwise 0 (it has no idea!)
2. Training error vs test error
Now compare the memoriser with a simple line, using the test set as the judge.
training error: how well it fits data it has seen
test error: how well it does on data it has never seen
The lookup table gets training error 0 — perfect! — and a terrible test error. The line is a little off on training, but close on the test. Only the test error tells you how the model will do in the real world. When training error is low but test error is high, suspect overfitting.
Challenge: Who wins on the test?
Write mseOf(predictFn, xs, ys): the average of (predictFn(xs[i]) - ys[i]) ** 2.
Compute lookupTrain, lookupTest, lineTrain and lineTest with mseOf, using the lookup and line functions given.
Set winner to "line" if lineTest is smaller than lookupTest, else "lookup".