12.3.1 — Linear and Logistic Regression

Two models open every serious ML project at Cartwise, and they open most projects everywhere else too. One predicts a number: how many days will this parcel take? The other predicts a probability: how likely is this order to be fraud? They are called linear regression and logistic regression, they are more than a century old in their mathematical form, and for data that arrives as rows and columns (an orders table, a customers table, a deliveries table) they are still very hard to beat by much.

There are three reasons to learn them properly before anything fancier. First, they are the honest baseline that every complex model has to justify itself against, which is the habit from 12.1.3. Second, their weights can be read and explained: you can tell an auditor or a customer exactly why a decision came out the way it did, which is why banks and hospitals still use them by default. Third, and most useful for the rest of this Part, a neural network is a stack of these exact models. Every piece on this page, the weighted sum, the sigmoid, the cross-entropy loss, the p - y gradient, reappears unchanged in Chapter 12.5. Understand them here, on four rows of data, and deep learning becomes an extension rather than a new subject.

Both models are built entirely from the tools of Chapter 12.2. This page puts those tools together.

1. Linear regression: a weighted sum that predicts a number

Here is the idea in one sentence: every input gets a weight saying how much it pushes the answer up or down, and the prediction is the sum of all those pushes plus a starting value. Each feature makes its own separate contribution, and the contributions simply add up. That additive structure is what "linear" means here.

For Cartwise's delivery problem, with two features, distance x_1 in hundreds of kilometres and parcel weight x_2 in kilograms, the model is:

\hat{y} = w_1 x_1 + w_2 x_2 + b

Read it as "predicted days equals w-one times distance, plus w-two times weight, plus b". As in 12.2.4, \hat{y} ("y-hat") is the model's guess, the $w$s are the weights, and b is the bias, the prediction you would get if every feature were zero. With many features this is exactly the dot product from 12.2.1: \hat{y} = \mathbf{w} \cdot \mathbf{x} + b.

What "best fit" means

In 12.2.4 we fitted a line through four deliveries using distance alone, and gradient descent found w = 1.4, b = 0.5. That line did not pass through every point, so "best" needs a definition. Here is what the fit looks like, with each error drawn in:

The best straight line through four deliveries. Each red segment is a residual: how far the true value (amber dot) sits above or below the line. Least squares picks the line that makes the average of the squared residuals as small as possible.

Each red segment is the gap between what really happened and what the line predicted. That gap is called a residual: what is left over after the model has had its say. For the first delivery, the truth was 2 days and the line predicts 1.4 \times 1 + 0.5 = 1.9, so the residual is +0.1. The four residuals are +0.1, -0.3, +0.3 and -0.1.

"Best fit" means the line that makes the mean squared error smallest: square each residual, average them. Here that is (0.01 + 0.09 + 0.09 + 0.01)/4 = 0.05. Any other line gives a bigger number. Finding the line this way is called least squares, a method published by Adrien-Marie Legendre in 1805 and used, by his own account, by Carl Friedrich Gauss for years before that to predict where asteroids would reappear after passing behind the Sun.

Why squared errors, rather than plain distances? Two reasons. Squaring makes every error count as positive, so errors above and below the line cannot cancel each other out. And it turns the loss into a smooth bowl with a single bottom, the convex shape from 12.2.4, so there is always exactly one best answer and gradient descent is guaranteed to find it. (There is also a deeper reason: if the errors in the world follow the normal distribution from 12.2.5, least squares is exactly the maximum likelihood answer, the same principle that gave us cross-entropy in 12.2.6.)

Fitting with two features, in code

Linear regression is special because the bottom of its bowl can be found directly, without walking downhill at all. Setting every partial derivative of the mean squared error to zero gives a small system of equations, the normal equations, and solving them gives the exact best weights in one step. Libraries do this for you. Here are six past deliveries, now with weight included:

import numpy as np
from sklearn.linear_model import LinearRegression

#              distance (×100 km), weight (kg)
X = np.array([[1,  1],
              [2,  5],
              [3,  2],
              [4,  8],
              [5,  3],
              [6, 10]])
y = np.array([2.0, 3.5, 4.5, 6.5, 6.5, 9.0])           # days taken

model = LinearRegression().fit(X, y)                   # (1)
print(model.coef_.round(4), round(model.intercept_, 4))  # (2) → [1.0958 0.1699] 0.6767

new_parcel = np.array([[4.5, 6]])                      # 450 km, 6 kg
print(model.predict(new_parcel).round(2))              # (3) → [6.63]

pred = model.predict(X)
print(np.mean((y - pred) ** 2).round(4))               # (4) → 0.0209

Line (1) creates a linear regression and fits it. fit solves the normal equations behind the scenes, so no learning rate or step count is involved. This is scikit-learn, the standard Python library for classical machine learning: every model in it has the same two methods, fit to learn from examples and predict to make guesses, so once you know one you know the pattern for all of them.

Line (2) prints the learned weights and the bias. Read them as sentences, because that is what makes linear models valuable:

• 1.0958 on distance: each extra hundred kilometres adds about 1.1 days.
• 0.1699 on weight: each extra kilogram adds about 0.17 days, perhaps because heavy parcels go by slower freight.
• 0.6767 for the bias: about two-thirds of a day of fixed handling, whatever the parcel.

Line (3) predicts a new parcel of 450 km and 6 kg: 1.0958 \times 4.5 + 0.1699 \times 6 + 0.6767 \approx 4.93 + 1.02 + 0.68 = 6.63 days. Line (4) prints the mean squared error on the training rows, 0.0209, which is small, because these six rows happen to be close to a flat plane in three dimensions.

Notice that the distance weight here, 1.0958, is a little smaller than the 1.4 we found with distance alone. That is not an inconsistency. In this data, longer trips also tended to be heavier parcels, so when distance was the only feature, it had to soak up the effect of weight too. Once weight has its own column, distance only gets credit for its own effect. A weight in a linear model always means "the effect of this feature with all the other features held fixed", so adding or removing a feature can change every other weight.

Where linear regression goes wrong

The model is simple, and its failures are predictable. Knowing them is how you tell when a linear baseline is trustworthy and when it is not.

The world is not a straight line. Delivery time might rise steeply for the first few hundred kilometres (an extra overnight hub) and then flatten out. A single straight line cannot bend. The fix is to give the model bent versions of the inputs to work with: add a column for distance squared, or the logarithm of distance, or a yes-or-no column for "over 500 km". The model is still linear in its weights, which is all "linear" requires, but it can now draw curves in the original input. Creating such columns is part of feature engineering, covered in Chapter 12.4.2.

Features interact. Perhaps weight matters a lot for long trips and hardly at all for short ones. A plain sum cannot express "it depends", because each feature adds its own amount regardless of the others. The fix is an interaction column: distance multiplied by weight. The tree models of Chapter 12.3.4 find interactions like this on their own, which is a large part of why they usually beat linear models on tabular data.

Outliers drag the line. Squaring the residuals means one parcel that took 40 days because it was lost produces a squared error of over a thousand, and the line tilts noticeably just to reduce that one enormous number. Look at your largest residuals before trusting the weights, and decide whether they belong in the data at all.

Nearly duplicate features make the weights unstable. If you include both "distance in kilometres" and "estimated driving hours", which move almost perfectly together, the model cannot tell which one deserves the credit. It might give a large positive weight to one and a large negative weight to the other, and a tiny change in the data can swap them. The predictions stay fine, but the weights stop meaning anything. This is called multicollinearity. The regularisation in Chapter 12.3.3 calms it down.

It will confidently extrapolate. Ask about a 5,000 km parcel and the line will happily answer 55 days, because a straight line goes on forever. It has never seen anything near that distance and has no way to know that the answer is actually "it goes by air and takes 3".

2. Logistic regression: a weighted sum that predicts a probability

Now the fraud question. The answer is yes or no, so what we want is a probability between 0 and 1. A weighted sum can produce any number at all, from very negative to very positive. So we keep the weighted sum and pass it through the sigmoid from 12.2.6, which squashes any number into the range 0 to 1:

p = \sigma(\mathbf{w} \cdot \mathbf{x} + b) = \frac{1}{1 + e^{-(\mathbf{w} \cdot \mathbf{x} + b)}}

That is the entire model. It is called logistic regression, and despite the word "regression" in its name, it is a classification model. The name is historical: the S-shaped curve was called the logistic curve by the Belgian mathematician Pierre-François Verhulst in the 1840s, who used it to describe how a population grows quickly and then levels off, and fitting it to data was regression in the statistical sense of fitting a curve. The name stuck even though today it is used almost entirely to classify.

Training it: cross-entropy and the p − y gradient

Linear regression's bowl can be solved in one step. Logistic regression's cannot, so it is trained with gradient descent. The loss is the binary cross-entropy from 12.2.6, which punishes confident wrong answers hard. And 12.2.6 showed that the gradient of that loss with respect to the raw score is simply p - y, the predicted probability minus the truth. Push that through the weighted sum with the chain rule, and the gradient for each weight is that same error multiplied by the feature value, averaged over the examples. So the whole training loop fits in a few lines.

Here are fourteen past orders: amount in units of ₹10,000, whether the account was new (1) or not (0), and whether it turned out to be fraud.

import numpy as np

amount = np.array([0.5, 1, 2, 2.5, 3, 4, 5, 6,  0.5, 1, 1.5, 2, 3, 4])
new    = np.array([0,   0, 0, 0,   0, 0, 0, 0,  1,   1, 1,   1, 1, 1])
fraud  = np.array([0,   0, 0, 1,   0, 0, 1, 1,  0,   1, 0,   1, 0, 1])
X = np.column_stack([amount, new])                     # (1) shape (14, 2)

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

w, b, lr = np.zeros(2), 0.0, 0.5
for step in range(20_000):
    p = sigmoid(X @ w + b)                             # (2) a probability for each order
    err = p - fraud                                    # (3) the p − y from 12.2.6
    w -= lr * (X.T @ err) / len(fraud)                 # (4) each weight: error × its feature
    b -= lr * err.mean()

print(w.round(3), round(b, 3))                         # → [0.84  1.588] -3.235
print(np.exp(w).round(2))                              # (5) → [2.32 4.89]
print(sigmoid(np.array([3.5, 1]) @ w + b).round(3))    # (6) → 0.785
print(sigmoid(np.array([3.5, 0]) @ w + b).round(3))    #     → 0.427

Line (1) stacks the two feature columns side by side into one matrix with a row per order. Line (2) computes every order's probability in one go: the matrix-times-vector product from 12.2.2 gives each order's weighted sum, and the sigmoid squashes each one.

Line (3) is the error for each order: how far its probability is from the true 0 or 1. Line (4) is the gradient step. X.T @ err multiplies each feature column by the errors and adds them up, which is "error times feature, summed over orders", exactly the gradient described above. Dividing by the number of orders turns the sum into an average.

At the very start, with all weights at zero, every order gets a probability of exactly 0.5, and the average loss is -\ln 0.5 \approx 0.693. After training, the loss is 0.528, and the weights have settled at 0.84 per ₹10,000 of amount, 1.588 for a new account, and a bias of -3.235.

Reading the weights: odds multipliers

Line (5) is what makes logistic regression so valuable in regulated work. In 12.2.6 we saw that the raw score is the log of the odds, so raising e to the power of each weight tells you how much that feature multiplies the odds of fraud:

• e^{0.84} \approx 2.32: every extra ₹10,000 multiplies the odds of fraud by about 2.3.
• e^{1.588} \approx 4.89: a new account multiplies the odds of fraud by almost 5.

That is a sentence you can put in front of a compliance officer, a regulator, or a customer who wants to know why their order was held. No deep network gives you anything as direct.

Lines (6) show the model at work on two ₹35,000 orders. From a new account, the probability is 0.785. From an established account, 0.427. Same amount, very different risk, and the difference is entirely the new-account weight.

The decision boundary

Logistic regression gives a probability, and a threshold turns it into a decision. With the plain threshold of 0.5, where does the model switch from "approve" to "flag"? The sigmoid equals 0.5 exactly when its input is 0, so the boundary is wherever \mathbf{w} \cdot \mathbf{x} + b = 0. For established accounts, that is 0.84 \times \text{amount} - 3.235 = 0, so amount \approx 3.85, or about ₹38,500. For new accounts, add the 1.588: 0.84 \times \text{amount} + 1.588 - 3.235 = 0, so amount \approx 1.96, about ₹19,600.

The trained model as two probability curves, one for established accounts (blue) and one for new accounts (amber). Each crosses the dashed 0.5 line at its decision boundary. The dots are the fourteen training orders, fraud along the top and genuine along the bottom, coloured by account type.

Two things in this figure are worth taking in. First, the new-account curve is the established-account curve slid to the left. That is all a positive weight on a yes-or-no feature can do: shift the whole curve. Second, look at the dots. Some fraud sits below the boundary (the ₹25,000 fraud from an established account) and some genuine orders sit above it. No setting of the weights can get every dot right, because the classes overlap. A model can only be as good as the information in its features allows, and adding a more informative feature, such as how many different cards the account has tried this week, does more than any amount of tuning.

With more features, the boundary \mathbf{w} \cdot \mathbf{x} + b = 0 is a straight line in two dimensions, a flat plane in three, and a flat "hyperplane" in more, which is the general word for a flat surface in any number of dimensions. Logistic regression can only ever draw a straight cut through the feature space. If fraud lives in a pocket, say "small orders at night or very large orders in the day", one straight cut cannot isolate it. That limit is exactly what trees and neural networks remove, and it is the same limit the perceptron hit in 1969 in the story from 12.1.1.

The same thing in scikit-learn

In practice you would not write the loop yourself:

from sklearn.linear_model import LogisticRegression

clf = LogisticRegression(penalty=None).fit(X, fraud)   # (1)
print(clf.coef_.round(3), clf.intercept_.round(3))     # ≈ [[0.84 1.588]] [-3.235]
print(clf.predict_proba([[3.5, 1]]).round(3))          # (2) ≈ [[0.215 0.785]]

Line (1) fits the same model. penalty=None matters: by default, scikit-learn adds a regularisation penalty that pulls weights towards zero, which is usually a good idea and is explained in Chapter 12.3.3, but here we switch it off so the result matches our hand-written loop. Line (2) shows a quirk of the library: predict_proba returns one probability per class, so the second column is the fraud probability and the first is its complement. Mixing those columns up is a classic bug.

3. Multi-class: one weighted sum per class

Cartwise's support router from 12.2.6 needs four classes, not two. The extension is direct. Give each class its own weight vector and its own bias, which is exactly a weight matrix with one row per class as in 12.2.2, compute the four scores, and pass them through softmax instead of the sigmoid. Train with cross-entropy. This is called multinomial logistic regression or softmax regression, and scikit-learn's LogisticRegression does it automatically when the labels have more than two values.

It is worth pausing on what that means. A softmax regression over the words in an email is a one-layer neural network. An LLM's final layer, which turns its internal representation into a probability for every token in the vocabulary, is a softmax regression with a hundred thousand classes. The model on this page is not a toy that real systems leave behind. It is the last layer of nearly all of them.

Recall

• Linear regression predicts a number as a weighted sum plus a bias; each weight is the effect of its feature with the others held fixed. Fitting by least squares minimises the mean squared error of the residuals and has an exact one-step solution.
• Its weights read as sentences (about 1.1 days per 100 km). It fails on curves, interactions, outliers, nearly duplicate features and extrapolation.
• Logistic regression passes the weighted sum through a sigmoid to get a probability. Despite its name it classifies. It is trained by gradient descent on cross-entropy, with gradient (p − y) × feature.
• e^{w} is an odds multiplier: here a new account multiplied fraud odds by about 4.9. That explainability is why regulated industries default to it.
• The decision boundary \mathbf{w}\cdot\mathbf{x} + b = 0 is always a straight cut, so it cannot isolate fraud that lives in a pocket. Softmax regression handles many classes and is the final layer of most neural networks and LLMs.

Self-test: Why did the distance weight drop from 1.4 to about 1.1 when weight was added as a feature? Using the trained fraud model, what is the probability for a ₹10,000 order from a new account? Why can no setting of the weights classify all fourteen training orders correctly?

Next: 12.3.2 Measuring a Classifier — the fraud model gives probabilities and the threshold makes decisions; now we need the numbers that tell you whether those decisions are any good, and why the most obvious one, accuracy, is the most misleading.