What you are looking at
The filled dots are 14 training points and the hollow ones are 14 validation points. Both come from the same smooth curve, the dashed line, y = sin(2.8x + 0.4), plus random noise with a standard deviation of 0.25. The data is made up for this lab; New data draws a fresh set from the same curve.
The indigo curve is a polynomial, ŷ = w0 + w1x + w2x2 + … + wdxd, fitted to the training points only: its weights are the ones that make the squared error on those points as small as possible. The validation points take no part in the fit. They stand in for the new data a model meets once it is in use.
The chart refits every degree from 0 to 12 and plots two errors for each, as RMSE (root mean squared error, the typical miss in the units of y): one on the training points, one on the validation points. The verdict in the panel compares the current fit with the one that has the lowest validation error. Within a quarter of it, the fit is good. Worse than that with a lower training error, it overfits; worse with a higher training error, it underfits.
Too simple, too flexible
Set the degree to 1. A straight line can't follow the bend in the data, so it misses even the training points (error 0.40), and the validation points too (0.57). That is underfitting: the model is too simple for the pattern.
Now drag the degree to 12. That is 13 weights for 14 points, enough for the curve to pass close to nearly every training point: the training error falls to 0.03, far below the noise level. The validation error climbs to 5.18. The model has learnt the noise in those 14 points, which no new point shares. That is overfitting. In between, from degree 3 to 6, the validation error stays near its lowest, 0.24. The chart shows the pattern that gives overfitting away: training error only falls as the degree rises, while validation error falls, flattens, then turns up.
The expected squared error on new data splits into three parts. Bias is error from a model too simple for the pattern: a straight line misses the bend however many points it gets. Variance is error from a model too sensitive to its particular training points: press New data a few times at degree 12 and the curve changes completely, while at degree 1 it barely moves. The third part is the noise itself, which no model can remove: the dashed line on the chart. Raising the degree trades bias for variance, and the best degree is where their sum is smallest.
Regularisation
Ridge regression, or L2 regularisation, adds a penalty to the squared error: λ(w12 + … + wd2). Large weights now cost something, so the fit settles for smaller ones, and smaller weights make a smoother curve: the wild swings of a degree-12 polynomial come from huge weights that almost cancel each other out. The fit solves (XᵀX + λI)w = Xᵀy, where each row of X is one point's powers of x.
Try it at degree 12. With λ at 0 the largest weight is about 2,500 and the validation error is 5.18. Move λ to 0.01: the largest weight drops to 2.1, the wiggles go, and the validation error falls to 0.26, close to the best of any degree. The chart changes too: at λ = 0.01 every degree from 3 to 12 does about as well. Keep going to 10 and the curve flattens out, and the panel says it underfits again. So λ, like the degree, has a sweet spot.
The intercept w₀, also called the bias term, is left out of the penalty: the first 1 on the diagonal of I becomes 0. The intercept only moves the curve up or down, and penalising it would drag the curve towards y = 0 rather than smooth it. scikit-learn's Ridge leaves the intercept out too. L1 regularisation, or the lasso, penalises the sum of the absolute weights instead; it pushes some weights to exactly zero, so it also picks which features to keep.
More data, and choosing the settings
More data is the other cure. Set the training points to 140 and go back to degree 12 with λ at 0. With more than ten points per weight, the curve can't bend to reach each one, and the validation error stays at 0.24. At 42 points the top degrees still overfit, though far less than at 14.
The training error can't choose the degree or λ for you: it always prefers the most flexible model. This lab holds out a validation set and you choose by that. With little data, one split is a noisy guide: press New data a few times and the bottom of the U moves. K-fold cross-validation is steadier. Split the training data into k parts (5 or 10 is usual), train k times, each time scoring on the part left out, and average the k scores. Pick the degree and λ with the best average, refit on all the training data, and report the final score on a test set that played no part in choosing.
Early stopping is the same idea in neural networks. Training error keeps falling with every epoch, while validation error falls, then rises once the network starts to memorise: the same U, with training time in place of the degree. Stop at the lowest validation error and keep those weights. Weight decay, which shrinks every weight a little at each step, does the work of the L2 penalty there; with plain stochastic gradient descent the two are the same thing.
In an interview
Questions this lab prepares you for:
- What is overfitting, and how do you spot it? (Training error well below validation error, and validation error that rises as the model gets more complex.)
- Explain the bias-variance trade-off.
- How does L2 regularisation reduce overfitting, and how does L1 differ? (L1 pushes some weights to exactly zero.)
- How would you choose a polynomial's degree, or λ? (Cross-validation, never the training error.)
- What is the difference between a validation set and a test set?
- Your model scores 98% on its training data and 75% on validation. What do you try? (More data, a simpler model, stronger regularisation, early stopping; for a neural network, dropout and data augmentation too.)
- Why is the intercept, the bias term, usually left out of the penalty?
The code
The same fit in NumPy: the matrix of powers, λ added down the diagonal except for the intercept (the zero in penalty[0, 0]), and one solve. np.linalg.solve uses an LU factorisation with partial pivoting, the elimination the lab runs. At this size the normal equations are fine. For larger or badly scaled problems, libraries avoid forming XᵀX, which squares the condition number, and use a QR or SVD factorisation instead.
import numpy as np def ridge_fit(x, y, degree, lam): X = np.vander(x, degree + 1, increasing=True) penalty = lam * np.eye(degree + 1) penalty[0, 0] = 0 return np.linalg.solve(X.T @ X + penalty, X.T @ y) def rmse(w, x, y): X = np.vander(x, len(w), increasing=True) return np.sqrt(np.mean((X @ w - y) ** 2)) for degree in range(13): w = ridge_fit(x_train, y_train, degree, lam=0.01) print(degree, rmse(w, x_train, y_train), rmse(w, x_val, y_val))