12.3.3 — Overfitting and Generalisation

Cartwise wants a better delivery-time model. Short trips go by van, and the time rises with distance. Very long trips, beyond about a thousand kilometres, go by air and actually come back faster. So the true relationship is a curve: it rises, peaks, and falls. Ten past deliveries are available to learn from, and nine more, kept aside, will be used to test.

An engineer fits three models of increasing flexibility:

Model	Error on the 10 training deliveries	Error on the 9 test deliveries
Straight line	0.331	0.261
Gentle curve (degree 2)	0.065	0.048
Very bendy curve (degree 9)	0.000	0.697

(The errors are mean squared error in days squared, from 12.2.4.)

The degree-9 curve is perfect on the training data: zero error. It is also by far the worst on the test data, fourteen times worse than the gentle curve. For one 1,800 km test parcel that really took 1.9 days, it predicts minus 0.28 days. The model has not learned anything about deliveries. It has memorised ten specific points, including their random noise, and it wiggles wildly everywhere in between.

That is overfitting: fitting the training data so closely that you capture its accidents, and then failing on new data. Its opposite, a model too simple to capture the real pattern, is underfitting, and the straight line is doing that. The thing you actually want is generalisation, from 12.1.1: doing well on data the model has never seen. This page is about why the gap between training and test opens up, how to measure it honestly, and the tools that close it.

1. Seeing the three fits

The same ten training deliveries (blue dots), distance increasing to the right and days upwards. The straight line cannot bend, the degree-2 curve follows the real rise-and-fall, and the degree-9 curve threads every dot exactly by wiggling, then dives below zero where there was no training dot to hold it.

Look at the right panel near its right edge. Between the last two training dots there was nothing to hold the degree-9 curve in place, so it swung down to below zero, an impossible answer, and the real test parcel (green square) sits far above it. An overfit model is confident and precise exactly where it has no business being either. The middle curve is not perfect on the training dots, and that is the point: it refused to chase the noise.

"Degree" here is just how many powers of distance the model is allowed to use. Degree 2 means w_0 + w_1 x + w_2 x^2. Degree 9 goes up to x^9. These are still linear regressions from 12.3.1, with extra columns for x^2, x^3, \ldots, so the higher the degree, the more freedom the model has to bend.

2. Bias, variance and the noise you cannot remove

There is a precise way to describe what went wrong on each side, and it gives you the vocabulary every ML engineer uses.

Bias is error that comes from the model being too rigid to represent the truth. The straight line has high bias: no matter which ten deliveries you train it on, it will always draw a straight line, and it will always miss the curve. Its mistakes are systematic.

Variance is error that comes from the model being too sensitive to the particular training examples it happened to get. The degree-9 curve has high variance: give it ten slightly different deliveries and it would wiggle in a completely different way. Its mistakes come from chasing randomness.

And there is a third part: irreducible error, the randomness in the world itself. Two identical parcels going the same distance will not take exactly the same time, because of traffic, weather and a courier's day off. No model can predict that part. In our data the test noise has a variance of about 0.045, which is why the best model's test error, 0.048, is very close to that number and cannot go much lower. When your test error is close to the noise level, you are done; no cleverer model will help.

For squared error, these three add up exactly. If f(x) is the true average delivery time at distance x, \hat{f}(x) is the model's prediction, and \sigma^2 is the variance of the noise, the expected error on a new delivery is:

E\big[(y - \hat{f}(x))^2\big] = \text{Bias}\big[\hat{f}(x)\big]^2 + \text{Var}\big[\hat{f}(x)\big] + \sigma^2

Read it as "expected squared error equals bias squared, plus variance, plus noise". The expectation here is over all the different training sets you might have been given. Bias is how far the average model (averaged over those training sets) is from the truth. Variance is how much individual models scatter around that average. The noise term is fixed by the world.

Training error (blue) always falls as the model gets more flexible. Validation error (red) falls while the model is learning real structure, then rises once it starts memorising noise. The best model sits at the bottom of the red U, just above the noise floor.

This picture is the most useful diagnostic in all of classical ML, because it tells you which way to move. If training error and validation error are both high and close together, you are on the left: underfitting. The fix is more flexibility: more features, a more expressive model, less regularisation. If training error is low but validation error is much higher, you are on the right: overfitting. The fix is less flexibility or more data: regularisation, a simpler model, more training examples. Our three models sit exactly there: the straight line has both errors high (0.33 and 0.26), and the degree-9 curve has training at zero and validation at 0.70.

One honest footnote. Very large neural networks sometimes break this picture: as they grow far past the point where they can memorise the training data, validation error can come back down again, a pattern researchers call double descent. The full explanation is still being worked out. For classical models and ordinary amounts of data, the U-shape above is what you will see.

3. Keeping the measurement honest: train, validation, test

The test-set numbers in the opening table are only meaningful because those nine deliveries were never used in any decision. That rule is easy to break without noticing, so it needs a structure.

Split your data into three parts, each with one job:

1. Training set. The model learns its weights from this. Usually the largest part, say 70 percent.
2. Validation set. You use this to make choices: which degree, how much regularisation, which features, when to stop training. Say 15 percent.
3. Test set. You use this exactly once, at the very end, to report how good the final model is. Say 15 percent. Until then it stays locked away.

Why three and not two? Because every choice you make by looking at a set of data teaches you something about that data. If you try twenty different degrees and pick the one that scores best on the test set, you have used the test set to train a twenty-first parameter: the degree itself. The winning score is now optimistic, because you picked the best of twenty attempts on those very examples. Some of that "best" was luck. The validation set absorbs that luck so the test set can stay clean. Letting the test set influence any decision is a form of the leakage from 12.1.3, and it is the most common way honest teams fool themselves.

The settings you choose using the validation set, as opposed to the weights the model learns, are called hyperparameters. The degree of the polynomial, the strength of regularisation, the depth of a tree and the learning rate are all hyperparameters. The prefix "hyper" just means "above": they sit above the ordinary parameters and control how those are learned.

Cross-validation: when data is scarce

With a small dataset, setting aside 15 percent for validation wastes precious examples and gives a noisy estimate. k-fold cross-validation solves this. Split the non-test data into k equal parts, called folds, typically 5. Train five times. Each time, hold out a different fold for validation and train on the other four. Average the five validation scores.

Left: 5-fold cross-validation, where every example is used for validation exactly once and for training four times. Right: for data that changes over time, such as fraud, split by date instead, so that the model is always tested on a period after the one it learned from.

Every example gets used for validation exactly once and for training four times, so nothing is wasted, and averaging five scores gives a steadier estimate than one. The cost is training five times.

The split that matters most at Cartwise: by time

Random splitting has a hidden flaw for data that changes over time, and fraud data changes constantly. If you shuffle two years of orders and split randomly, the training set contains orders from after some test orders. The model learns about fraud tricks that, at the moment the test orders were placed, had not been invented yet. Its test score is better than anything it could achieve in real life, where it always faces tomorrow's fraud with only yesterday's examples.

So for anything time-ordered, split by date: train on January to September, validate on October, test on November, as the right panel shows. This simulates exactly what will happen in production. The score is usually lower than the random-split score, and that lower number is the true one. If the drop is large, that is itself useful news: the pattern you are modelling is changing quickly, which is the concept drift of 12.1.4.

Two related traps are worth checking every time. Duplicates across the split: if the same customer's near-identical orders land in both training and test, the model can recognise the customer rather than the fraud. Split by customer, so all of one customer's orders go to the same side. Preprocessing on all the data: computing the averages for standardisation, from 12.2.4, on the full dataset before splitting lets test-set information shape the training inputs. Fit every transformation on the training part only.

4. Regularisation: paying the model to stay simple

The degree-9 curve overfits because its weights are free to become enormous. When its weights are fitted with no restraint, the largest one comes out at about 7,400. Huge weights on high powers of x are exactly what produce the wild swings between the dots: a tiny change in distance gets multiplied by thousands.

Regularisation adds a second term to the loss that charges the model for large weights:

J(\mathbf{w}) = \underbrace{\frac{1}{n}\sum_i (y_i - \hat{y}_i)^2}_{\text{fit the data}} + \lambda \underbrace{\sum_j w_j^2}_{\text{keep weights small}}

Read it as "the total cost is the usual error, plus lambda times the sum of the squared weights". The model now has to balance two goals: fit the data, and keep its weights small. The Greek letter \lambda (lambda) sets the exchange rate between them, and it is a hyperparameter you choose with the validation set. With \lambda = 0 there is no restraint. A huge \lambda crushes every weight towards zero and the model underfits.

Here is what that does to the degree-9 model:

λ (strength)	Largest weight	Train error	Test error
0	about 7,400	0.000	0.697
0.001	1.5	0.053	0.051
0.1	1.7	0.060	0.064
1	2.0	0.095	0.100

With just a tiny penalty, \lambda = 0.001, the largest weight collapses from 7,400 to 1.5, the wiggles disappear, and the test error drops from 0.697 to 0.051, almost as good as the degree-2 model. The model kept all ten of its powers of x but was forbidden from using them wildly. Push \lambda to 1 and the weights are held back so hard that the curve can no longer follow the real shape, and both errors rise: we have slid back to the left side of the U.

import numpy as np
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression, Ridge

x_tr = np.array([0.1, 0.3, 0.5, 0.7, 0.9, 1.1, 1.3, 1.5, 1.7, 1.9]).reshape(-1, 1)
y_tr = np.array([1.68, 1.82, 2.75, 2.47, 3.08, 3.38, 2.52, 2.70, 1.87, 1.68])
x_te = np.array([0.2, 0.4, 0.6, 0.8, 1.0, 1.2, 1.4, 1.6, 1.8]).reshape(-1, 1)
y_te = np.array([1.62, 2.48, 2.83, 2.67, 3.30, 2.72, 2.78, 1.98, 1.92])

def report(model):
    model.fit(x_tr, y_tr)                                              # (1)
    mse = lambda x, y: np.mean((model.predict(x) - y) ** 2)
    return round(mse(x_tr, y_tr), 3), round(mse(x_te, y_te), 3)

print(report(make_pipeline(PolynomialFeatures(2, include_bias=False), LinearRegression())))
# → (0.065, 0.048)
print(report(make_pipeline(PolynomialFeatures(9, include_bias=False), LinearRegression())))
# → (0.0, 0.697)                                                      # (2)
print(report(make_pipeline(PolynomialFeatures(9, include_bias=False), Ridge(alpha=0.001))))
# → (0.053, 0.051)                                                    # (3)

Line (1) fits whatever model it is given on the training deliveries and returns the error on both sets. Each model is a pipeline: first PolynomialFeatures turns the single distance column into columns for x, x^2, \ldots, then a regression fits them. Line (2) is the unrestrained degree-9 model: perfect on training, poor on test. Line (3) is the same degree-9 model with a ridge penalty. scikit-learn calls the strength alpha rather than \lambda, because lambda is a reserved word in Python. (In real work you would also standardise the columns before a ridge penalty, so that every power is penalised on a fair scale; the numbers here skip that to stay comparable with the table.)

L2 versus L1: shrink everything, or throw things away

The penalty above, the sum of squared weights, is called L2 regularisation, and a linear regression with it is called ridge regression. There is a second common penalty, the sum of the absolute values of the weights, \lambda \sum_j |w_j|, called L1 regularisation, and a linear regression with it is called the lasso. They behave very differently, and the difference is easiest to remember with a packing analogy.

You are packing for a trek with a small backpack and a huge pile of gear you might need. L2 is the counsellor who says "bring everything, but only the travel-sized version": every item shrinks, nothing is left behind, and nothing dominates the bag. L1 is the minimalist who says "if it is not essential, leave it at home": most items are dropped entirely, and only the important few come along at full size. In model terms, L2 makes all weights small but rarely exactly zero, and L1 sets many weights to exactly zero, which removes those features from the model completely. That makes L1 a built-in feature selector: give it 200 candidate features and it may keep 15.

Why L1 produces exact zeros. The penalty limits the weights to a region: a circle for L2, a diamond for L1. The best regularised weights are where the error's oval contours first touch that region. The diamond has sharp corners on the axes, and the contours usually touch a corner first, where one weight is exactly zero.

The figure shows the geometry behind the analogy. You can think of the penalty as fencing the weights into a region around zero. For L2 the fence is a circle, because the points with the same value of w_1^2 + w_2^2 form a circle. For L1 it is a diamond, because the points with the same value of |w_1| + |w_2| form a diamond with corners on the axes. The red ovals are lines of equal error around the unpenalised best fit. Expand them outward until they first touch the fenced region, and that touching point is the regularised answer. A smooth circle is usually touched somewhere on its curve, with both weights non-zero. A diamond is usually touched at one of its sharp corners, and every corner sits on an axis, where one weight is exactly zero.

Regularisation is not only for polynomials. It is the default in almost every real model. The LogisticRegression in 12.3.1 uses an L2 penalty unless you switch it off, and it also calms down the unstable weights caused by nearly duplicate features. Neural networks use L2 under the name weight decay, and they add two more tools of their own: early stopping, which halts training at the bottom of the validation U before memorisation sets in, and dropout, which randomly switches off parts of the network during training so it cannot depend on any single part. Both are explained in Chapter 12.5.4.

The other cure: more data

Regularisation restrains the model. The other way to close the gap is to give it more to learn from. With ten deliveries, the degree-9 curve can thread every point. With ten thousand, it cannot: there is no way to pass through all of them, so the curve is forced towards the real average shape, and the noise in individual points cancels out. More data reduces variance; it does nothing for bias. That is also how to read a team's request for more data: if the model is underfitting, more data will not help, and you need a more flexible model instead.

5. When the model learns the wrong thing: shortcuts

Overfitting to noise is one way to fail to generalise. There is a sneakier way: the model finds a real pattern in the training data that has nothing to do with the task, and that pattern happens not to hold anywhere else.

A well-documented example comes from medical imaging. In a 2018 study, John Zech and colleagues trained networks to detect pneumonia in chest X-rays collected from several hospitals. The models did well when tested on new images from the same hospitals and noticeably worse on images from a hospital they had not trained on. Investigating why, the researchers found that the networks could identify which hospital an X-ray came from, from small details such as the metal markers technicians place on the film and the characteristics of the scanners. Because pneumonia was much more common at some hospitals than others, "which hospital is this" was a useful clue for "is this pneumonia" inside the training data, and the network learned to lean on it. Nobody had told it to look at the lungs rather than the labels.

This is called shortcut learning, and it has an exact Cartwise equivalent. Suppose the fraud team marks suspected fraud in the order system with an internal flag, and that flag accidentally ends up as a feature. The model will learn "flagged orders are fraud", score superbly, and be useless for orders nobody has looked at yet. The defence is not a regularisation setting. It is understanding what each feature means and when it becomes known, and checking which features the model actually relies on, which is the next section.

6. Looking inside: which features did the model use?

Once a model generalises well, you still need to know why it made its decisions. That builds trust with the people affected, it is often a legal requirement, and above all it catches shortcuts.

For linear and logistic regression, the weights already tell you, as 12.3.1 showed. For more complex models, three tools are standard.

Permutation importance asks: how much worse does the model get if I scramble one feature? Take the validation set, randomly shuffle the values in the "account age" column so it no longer matches its rows, and measure the drop in performance. A big drop means the model relied on that feature heavily. No drop means it ignored it. It is simple, works on any model, and is the first thing to run. If an innocent-sounding internal column causes the biggest drop, you have probably found a shortcut.

SHAP explains a single prediction: why was this order flagged? It is based on an idea from game theory called the Shapley value, introduced by the economist Lloyd Shapley in 1953 to split a team's winnings fairly among its players. Here the "players" are the features and the "winnings" are how far this prediction sits above the average prediction. Each feature gets the average of its extra contribution over every order in which the features could have been added.

A tiny worked example shows the idea. Suppose the model's average fraud probability is 0.05. For one order, it gives 0.20 if it only knows the account is new, 0.15 if it only knows the amount is large, and 0.60 if it knows both. How much of the 0.55 rise (from 0.05 to 0.60) belongs to each feature? Consider the two orders in which the features could arrive:

• New account first: it adds 0.20 - 0.05 = 0.15; then the amount adds 0.60 - 0.20 = 0.40.
• Amount first: it adds 0.15 - 0.05 = 0.10; then the new account adds 0.60 - 0.15 = 0.45.

Average each feature's contribution over the two orders: the new account gets (0.15 + 0.45)/2 = 0.30, and the amount gets (0.40 + 0.10)/2 = 0.25. They add to exactly 0.55, so the explanation accounts for the whole prediction. The order was flagged mostly because of the new account, and a little less because of the amount. Real SHAP libraries compute this efficiently for hundreds of features. The name stands for SHapley Additive exPlanations.

LIME (Local Interpretable Model-agnostic Explanations) takes a different route to the same kind of answer. It makes many small changes to one input, watches how the model's output moves, and fits a simple linear model to those reactions, which is then easy to read. It is faster than SHAP for some models but less stable: run it twice and the explanation can change.

One warning applies to all three. They tell you what the model relies on, not what causes fraud in the world. If account age and fraud are linked only because both depend on something the model never saw, the explanation will still point at account age.

What the interviewer will push on

"How do you know if a model is overfitting?" They are checking that you compare training and validation error rather than looking at one number. The good answer describes the gap: low training error with much higher validation error. Mentioning the learning-curve U and what each side means is the tell. The common wrong answer is "the accuracy is too high".

"Explain the bias–variance trade-off." They want the plain definitions (rigid and systematically wrong, versus sensitive and erratic), the decomposition into bias squared plus variance plus noise, and what moves each one. A strong answer adds that more data reduces variance but not bias.

"L1 or L2?" They are checking that you know L1 produces exact zeros and why: the diamond's corners. A good answer says L1 when you suspect most features are useless and want selection, L2 when many features each contribute a little or they are correlated. Mentioning that the penalty strength is a hyperparameter chosen on validation data is expected.

"How would you split the data for a fraud model?" This is the question that separates people who have shipped models. The answer they want is a time-based split, plus grouping by customer to avoid the same customer on both sides, plus fitting preprocessing on training data only. Random shuffling is the wrong answer.

"Your model is 99 percent accurate in testing and fails in production. What happened?" They want a list of suspects in order: leakage of future or label-derived features, a random split on time-ordered data, a shortcut feature, and distribution shift between test data and production (Chapter 12.12.5). The tell is checking feature importances for something suspicious.

Volunteer this: a test set you have looked at many times is no longer a test set. Teams that compare dozens of models on the same test set slowly overfit to it, and the honest fix is a fresh holdout, usually the most recent weeks of data, used once.

Recall

• Overfitting fits the training data's noise and fails on new data; underfitting is too rigid to capture the real pattern. The goal is generalisation.
• Expected error = bias squared + variance + irreducible error. Training error always falls with flexibility; validation error traces a U. Both high means underfit; a big gap means overfit. Test error near the noise level means you are done.
• Use training, validation and test sets with separate jobs; choices made on the test set are leakage. Hyperparameters are chosen on validation data, or with k-fold cross-validation when data is scarce. Split time-ordered data by date and grouped data by customer.
• Regularisation charges for large weights: L2 regularisation shrinks them all, L1 regularisation zeroes many (the diamond's corners). A tiny ridge penalty cut the degree-9 test error from 0.697 to 0.051. More data cuts variance, not bias.
• Shortcut learning exploits a pattern unrelated to the task. Permutation importance, SHAP (from the Shapley value) and LIME show what a model relies on, not what causes the outcome.

Self-test: A model has training error 0.40 and validation error 0.42. Which side of the U is it on, and would more data help? Why does choosing the degree using the test set make the test score optimistic? Recompute the Shapley values in the worked example if the "both features" prediction were 0.40 instead of 0.60.

Next: 12.3.4 Trees, Forests and Boosting — linear models can only cut the data with straight lines; trees cut it into boxes, and when hundreds of trees are combined, they become the strongest models there are for data in rows and columns.