Appearance
12.3 — Classical Machine Learning
A fraud model reports 99.7% accuracy. The team celebrates.
Fraud is 0.3% of transactions. A model that predicts "not fraud" for everything scores 99.7% and catches nothing. The number was never wrong; it was the wrong number.
Most of what goes wrong in applied machine learning is like this — not a modelling failure but a measurement failure. This chapter covers the algorithms that solve the majority of real business problems, and spends as much space on how to know whether they work.
1. Regression: predicting a number
Linear regression fits a straight line: \hat{y} = w_1x_1 + w_2x_2 + \dots + b. Training finds the weights minimising squared error.
It is worth taking seriously rather than skipping. The weights are readable — "each extra bedroom adds £24,000" — which no deep model gives you, it trains in milliseconds, and it is a real baseline. If a complex model cannot beat linear regression by a margin that justifies its cost, ship the line.
Regularisation is the important refinement. Adding a penalty on large weights stops the model contorting itself to fit noise:
- Ridge (L2) penalises squared weights — shrinks them all, keeps them all.
- Lasso (L1) penalises absolute weights — drives some to exactly zero, which performs feature selection for you.
Logistic regression predicts a probability, not a number. It computes the same weighted sum, then squashes it through a sigmoid into (0,1). Despite the name it is a classifier, and it remains the default for binary classification in regulated settings because the coefficients can be explained to an auditor.
2. Trees, and why boosting still wins
A decision tree asks a series of yes/no questions, choosing at each step the split that best separates the classes. It is completely interpretable — you can print it and read it — and a single deep tree overfits badly, memorising the training data.
Two ways to fix that, and the difference between them is the useful part.
Random forest trains many trees, each on a random sample of rows and considering a random subset of features at each split, then averages them. The randomness is the point: it makes the trees' errors different, and averaging cancels uncorrelated errors. This is bagging — training in parallel and combining.
Gradient boosting trains trees sequentially, each one fitted to the errors the ensemble has made so far. Each tree corrects its predecessors. This is usually more accurate and more prone to overfitting, so it needs a learning rate and early stopping.
XGBoost, LightGBM and CatBoost are the practical implementations, and here is the fact that surprises people:
On tabular data — rows and columns, the shape of most business data — gradient boosted trees still generally beat deep learning. They handle mixed types and missing values, need far less tuning, train in minutes on a laptop, and produce feature importances. Reach for a boosted tree first on any table, and only consider a neural network when you have a very large dataset or an unusual structure.
3. The rest of the toolkit
k-nearest neighbours — classify a point by the majority vote of its k closest training points. No training at all; all the cost is at prediction, and it needs the whole dataset in memory. Useful as a baseline and as the intuition behind vector search (Chapter 12.6.2).
Naive Bayes — applies Bayes' theorem (Chapter 12.2) assuming features are independent. The assumption is obviously false and it works anyway, especially for text. Extremely fast, a strong baseline for classification, and the original spam filter.
Support vector machines — find the boundary with the widest margin between classes, and use the kernel trick to handle non-linear boundaries without ever computing the higher-dimensional coordinates. Dominant in the 2000s, now largely displaced by boosted trees on tabular data and neural networks elsewhere.
k-means clustering — unsupervised: pick k centres, assign each point to the nearest, move each centre to the mean of its points, repeat. You must choose k, results depend on the random start, and it assumes roughly spherical clusters of similar size. Useful, and its output is a hypothesis rather than a finding.
Principal component analysis — find the directions of greatest variance and project onto the first few. Compresses while keeping structure, and unlike t-SNE (Chapter 12.2) the transformation is linear and reversible, so it is safe to feed into another model.
4. Features are where the accuracy is
In classical machine learning, feature engineering matters more than algorithm choice. A boosted tree on good features beats a neural network on raw ones, and the ranking of algorithms rarely changes the outcome as much as one well-constructed feature.
The routine transformations:
Categorical variables. One-hot encoding creates a column per value — fine for a handful, catastrophic for postcodes. Target encoding replaces a category with the mean outcome for it, which is powerful and leaks badly unless computed within cross-validation folds only.
Numeric scaling. Distance-based methods (k-nearest neighbours, k-means, SVM) and gradient-based ones need features on comparable scales, or a feature measured in thousands dominates one measured in units. Trees do not care, because they only compare within a feature.
Missing values. Trees can handle them natively. Otherwise impute — and add a "was missing" indicator column, because missingness is frequently informative: a blank income field may predict the outcome better than any imputed value.
Dates. Never feed a raw timestamp. Extract day of week, hour, month, whether it is a holiday, and time since a previous event.
Interactions and ratios. Price per square metre, transactions per day, ratio of this purchase to the customer's average. These are usually the features that lift a model, because they encode a relationship the model would otherwise have to discover.
Data leakage is the single most damaging mistake in the field. It means a feature contains information that would not exist at prediction time, and the symptom is unmistakable: an implausibly good validation score that collapses in production.
The classic forms: a column derived from the outcome (refund_amount when predicting refunds), scaling or imputing using statistics computed over the whole dataset before splitting, target encoding computed on all rows, and any time-ordered problem split randomly instead of by time.
The discipline: split first, then fit every transformation on the training set only, and apply it to the others. A pipeline object that fits and transforms together is not a style preference — it is the mechanism that makes leakage structurally hard.
5. Measuring it properly
Three splits, not two. Training fits the model, validation chooses hyperparameters, test is touched once at the very end. If you tune against the test set, it has become a validation set and your reported number is optimistic.
Cross-validation splits the training data into k folds, trains k times holding out each fold, and averages. It gives a more reliable estimate on small datasets at k times the cost.
For anything time-ordered, split by time and never randomly. A random split lets the model learn from the future and predict the past, which cannot happen in production. Use forward-chaining: train on months 1–6, validate on 7; then 1–7, validate on 8.
The metrics, and when each lies
The confusion matrix is the source of all of them:
| Predicted positive | Predicted negative | |
|---|---|---|
| Actually positive | True positive | False negative |
| Actually negative | False positive | True negative |
Accuracy — proportion correct. Useless on imbalanced data, as the opening showed.
Precision — of those predicted positive, how many were. When it says yes, is it right?
Recall (sensitivity) — of the actual positives, how many were found. Does it catch them?
They trade off, and the trade is a product decision, not a technical one. A cancer screen wants high recall — a missed case is catastrophic, a false alarm is a follow-up test. A spam filter wants high precision — a missed spam is an annoyance, a real email in the spam folder is a lost customer.
F1 — the harmonic mean of precision and recall. A single number, useful for comparing models and unhelpful for choosing an operating point, because it presumes precision and recall matter equally, which they almost never do.
ROC-AUC — how well the model ranks positives above negatives, across all thresholds. It is misleading on heavily imbalanced data, because a large number of true negatives keeps the false-positive rate low no matter what.
Precision-recall AUC — use this instead when positives are rare. It ignores true negatives entirely, which is exactly right when they are 99.7% of the data.
The threshold is a separate decision from the model. A classifier outputs a probability; turning it into a yes or no requires a cut-off, and 0.5 is a default, not an answer. Choose it from the cost of each error type — if a false negative costs £500 and a false positive costs £5, the threshold should be far below 0.5.
Calibration is the property people forget. A calibrated model that says 70% is right about 70% of the time. Boosted trees are typically poorly calibrated, and it matters enormously when the probability feeds a downstream decision — expected loss, a pricing formula, a risk score. Check with a reliability plot and correct with Platt scaling or isotonic regression.
Class imbalance is handled by class weights, by resampling, or by threshold choice. Try class weights and threshold tuning first; synthetic oversampling techniques are widely used and frequently do not help, and they can leak if applied before the split.
6. Sentiment analysis, worked
A concrete example of a classical pipeline, and of how the numbers people quote are produced.
Polarity is how positive or negative a text is, usually in [-1, +1]. Subjectivity is how much it expresses opinion rather than fact, in [0, 1].
The lexicon approach — the simplest — assigns each word a prior polarity and subjectivity from a hand-built dictionary, then combines them: a rough average of the words that appear, adjusted by modifiers. "not" flips the sign; "very" amplifies; "slightly" damps. That is genuinely how classic library implementations compute the two numbers, which is worth knowing before quoting them as a measurement.
Its weaknesses are structural: sarcasm ("brilliant, another delay"), domain drift ("this thriller was predictable" versus "the delivery was predictable"), negation beyond a small window, and comparatives ("better than their last attempt", which is not praise).
The classical machine learning approach — bag-of-words or TF-IDF (Chapter 7.7) features into logistic regression or a linear SVM — beats the lexicon when you have labelled data, because it learns the vocabulary of your domain.
A fine-tuned transformer beats that, and a large language model beats it with no training data at all — which is exactly the shift Chapter 12.1 described. The reason to know the classical version is cost: classifying a hundred million reviews with a linear model costs almost nothing and runs in minutes, and a large model costs real money and time. Match the tool to the volume and the required nuance.
7. Interpretability
Some models explain themselves: linear coefficients, a printed decision tree, feature importances from a forest.
For anything else there are two standard tools. SHAP assigns each feature a contribution to an individual prediction, based on a game-theoretic argument about fair attribution; it is the current default and is slow on large models. LIME fits a simple model locally around one prediction to approximate what mattered there.
Two cautions worth carrying. Feature importance shows what the model used, not what causes the outcome — a correlated proxy scores highly and is not a lever. And an explanation for a single prediction is not a description of the model; a plausible local story can be produced for a model that is broadly wrong.
Where explanation is a requirement rather than a nicety — credit, employment, insurance, anything with a legal right to an explanation (Chapter 8.7) — choose an inherently interpretable model. A slightly less accurate model you can defend beats a slightly better one you cannot.
8. Choosing
| Situation | Start with |
|---|---|
| Tabular data, any size | Gradient boosted trees |
| Need to explain every decision | Logistic regression, or a shallow tree |
| Tiny dataset | Logistic regression, naive Bayes |
| Text classification with labels | TF-IDF + linear model |
| Text classification without labels | An LLM (Chapter 12.6) |
| Images, audio, sequences | Deep learning (Chapter 12.4) |
| Grouping with no labels | k-means, then look at it critically |
And the meta-rule: build the dumbest possible baseline first. Predict the majority class. Predict yesterday's value. Use three hand-written rules. That number is what every later model must beat, and it is astonishing how often it is not beaten — which is information worth having in week one rather than month six.
Recall
- Accuracy lies on imbalanced data — 99.7% by predicting "no fraud" every time. Use precision, recall, and precision-recall AUC rather than ROC-AUC when positives are rare.
- Linear and logistic regression are readable baselines. Ridge shrinks weights, Lasso drives some to exactly zero and performs feature selection.
- Random forest is parallel and averages uncorrelated errors; gradient boosting is sequential and fits the residual errors. On tabular data, boosted trees still generally beat deep learning — reach for them first.
- Feature engineering beats algorithm choice. Scale for distance- and gradient-based methods (trees do not care), add a "was missing" indicator, explode dates, and build ratios and interactions.
- Data leakage is the most damaging mistake: a feature that would not exist at prediction time, statistics fitted before the split, or a time-ordered problem split randomly. Split first, fit transformations on train only.
- Three splits: train, validation, test touched once. Tuning against the test set turns it into a validation set.
- Precision versus recall is a product decision — cancer screening wants recall, spam filtering wants precision. The threshold is separate from the model and follows from the cost of each error type.
- Calibration matters whenever a probability feeds a decision, and boosted trees are typically poorly calibrated. Always build the dumbest baseline first — majority class, yesterday's value, three rules — because it is often not beaten.
Self-test: Why is 99.7% accuracy a failing fraud model? · What makes random forest's randomness essential? · Name three ways leakage enters a pipeline · When is ROC-AUC misleading? · Why is 0.5 not a threshold? · Why does a lexicon sentiment score fail on "brilliant, another delay"?