Overfitting and regularisation

Machine learning workflow · applies to: Model capacity, training diagnostics · last updated 2016-06-04

A model that memorises its training set looks excellent right up to the moment it meets new data.

What the curves look like

Plot the loss on the training split and on a held-out split against training progress. Early on both fall together. If capacity exceeds what the data supports, the training curve keeps falling while the held-out curve flattens and then turns upward. The gap between them is the memorisation.

Two curves that are both high and flat mean the opposite problem: the model has too little capacity, or the columns simply do not carry the signal being asked for.

The usual remedies

More data is the remedy that always works and is usually unavailable. After that: reduce capacity, penalise large parameters, stop training at the point the held-out curve turned, or randomly disable parts of the network during training so no single path can carry the whole answer.

A penalty on parameter size pushes the model towards simpler explanations. Applying it as a sum of squares shrinks everything gently; applying it as a sum of absolute values drives some parameters to exactly zero and quietly performs feature selection.

Keeping the held-out split honest

The held-out split stops being a fair estimate the moment decisions start being made from it. Choosing a stopping point, a capacity or a penalty strength by looking at it means it has become part of training. Anything reported as a final number needs a third split that no decision has touched.

Diagnosing a gap

1. Plot training and held-out loss on the same axes against the same progress unit.
2. Find the point where the held-out curve stops improving.
3. If the training curve is still falling past that point, the model has spare capacity to memorise.
4. Add a penalty or reduce capacity, retrain, and compare the gap rather than the absolute numbers.
5. Report the final figure from a split that took part in no decision.

sum of squares:          cost = loss + lam * sum(w ** 2)
sum of absolute values:  cost = loss + lam * sum(abs(w))

The two penalty shapes

Return to Machine learning workflow