12.2.4 — Gradient Descent

Imagine you are standing on a hillside in thick fog. You cannot see the valley, but you want to get down. What do you do? You feel the slope of the ground under your feet, take a step in the direction that goes most steeply downhill, and then feel the slope again. Repeat until the ground is flat.

That is the whole algorithm, and it trains nearly every model in this Part: linear regression, the fraud classifier, a hundred-layer image network, and every LLM ever built. The fog is real, too. A model with a billion weights lives in a landscape with a billion directions that nobody can see. All you can ever do is feel the slope where you stand, which is exactly what the gradient from 12.2.3 tells you.

1. The update rule

Gradient descent repeats one line, over and over:

w \leftarrow w - \eta \, \frac{dL}{dw}

Read it as "w becomes w minus eta times the derivative of the loss with respect to w". The arrow \leftarrow means "is replaced by", like w = w - lr * grad in code. The Greek letter \eta (eta) is the learning rate: how big a step to take. The minus sign is the "go against the slope" rule from the last page. With many weights, the same line runs for every weight at once, using the whole gradient vector:

\mathbf{w} \leftarrow \mathbf{w} - \eta \, \nabla L(\mathbf{w})

Let us walk it by hand on the loss from the last page, L(w) = (w - 3)^2, whose derivative is 2(w - 3). Start at w = 5 with learning rate \eta = 0.1:

Step	w before	slope 2(w − 3)	step = 0.1 × slope	w after	loss after
1	5.000	4.000	0.400	4.600	2.560
2	4.600	3.200	0.320	4.280	1.638
3	4.280	2.560	0.256	4.024	1.049
4	4.024	2.048	0.205	3.819	0.671
5	3.819	1.638	0.164	3.655	0.429

Look at the "step" column. The steps shrink on their own, from 0.4 to 0.164, even though the learning rate never changed. That happens because the slope shrinks as you approach the bottom, so a fixed fraction of a shrinking slope is a shrinking step. Gradient descent slows down naturally as it arrives. You do not have to tell it when to be careful.

There is a neat pattern hiding in the table. Each step multiplies the distance from the answer, w - 3, by the same factor. The update is w - 0.1 \times 2(w - 3), which means the new distance is (w - 3) - 0.2(w - 3) = 0.8\,(w - 3). The distance starts at 2, then becomes 1.6, then 1.28, and so on. After 30 steps it is 2 \times 0.8^{30} \approx 0.0025. That factor, 1 - 2\eta for this loss, is the key to everything in the next section.

2. The learning rate decides everything

The factor 1 - 2\eta tells you exactly what each learning rate will do on this loss, and the same three behaviours show up in every real model:

The same loss and the same starting point, w = 5, with three learning rates. Too small wastes time, about right walks steadily to the bottom, and too large jumps across the valley to a higher point each time.

Too small (\eta = 0.01, factor 0.98). Each step removes only 2 percent of the remaining distance. The left panel shows the dots barely moving: after 100 steps you are still about 0.27 away from the answer. Training like this is correct but slow, and on a real model "slow" can mean weeks of graphics card time you are paying for.

About right (\eta = 0.1, factor 0.8). The middle panel: a steady walk downhill, exactly the table above.

Too large (\eta = 1.1, factor -1.2). Now the factor is negative and bigger than 1 in size. A negative factor means each step jumps over the bottom to the other side. Bigger than 1 means it lands further from the bottom than it started. The right panel shows it: from w = 5 to w = 0.6 to w = 5.88, each time higher up the opposite wall. This is called divergence, and on a real model it looks like the loss suddenly shooting up and then printing nan, which stands for "not a number": the values got so large that the arithmetic broke.

There is even a "perfect" setting for this toy loss: \eta = 0.5 makes the factor zero, so one step lands exactly on w = 3. Real losses are never that tidy, but it shows the general shape of the trade-off. The ideal learning rate depends on how curved the loss is, and the most curved direction sets the limit, because a step that is fine for a gentle slope will overshoot on a steep one.

How do you choose it in practice? You try a few values spaced by factors of about three to ten (0.1, 0.03, 0.01, 0.003, 0.001), run each for a short while, and keep the largest one that does not make the loss jump around. Modern training also changes the learning rate during the run, usually starting small, rising, and then slowly decaying, which is called a learning rate schedule and is covered in Chapter 12.5.3.

3. A real fit: delivery days from distance

Now something that looks like a real model. Cartwise has four past deliveries: distance in hundreds of kilometres, and how many days each took.

Distance (×100 km)	1	2	3	4
Days	2	3	5	6

We want the straight line \hat{y} = w x + b that fits these best. The little hat on \hat{y} (read "y-hat") is the standard way to mark "the model's guess" as opposed to the true value y. For a loss, we use the mean squared error: for each delivery, take the guess minus the truth, square it, and average over all four:

L(w, b) = \frac{1}{n}\sum_{i=1}^{n} (w x_i + b - y_i)^2

Squaring does two jobs: it makes every error positive, so a guess that is too high cannot cancel one that is too low, and it punishes big errors much more than small ones. Chapter 12.3.1 explains why this particular loss is the natural one for regression.

To run gradient descent we need the two partial derivatives. Use the chain rule from 12.2.3 on each squared term: the outer function is "square it", with slope 2 \times (\text{inside}), and the inside w x_i + b - y_i has slope x_i with respect to w and slope 1 with respect to b. Writing e_i = w x_i + b - y_i for each error:

\frac{\partial L}{\partial w} = \frac{2}{n}\sum_{i} e_i\, x_i, \qquad \frac{\partial L}{\partial b} = \frac{2}{n}\sum_{i} e_i

In words: the slope for w is the average error weighted by each input, and the slope for b is just the average error. Here it is in NumPy:

import numpy as np

x = np.array([1.0, 2.0, 3.0, 4.0])    # distance, hundreds of km
y = np.array([2.0, 3.0, 5.0, 6.0])    # days taken

w, b, lr = 0.0, 0.0, 0.05             # (1) start from nothing
for step in range(1, 2001):
    err = w * x + b - y               # (2) guess minus truth, for all four at once
    grad_w = 2 * np.mean(err * x)     # (3)
    grad_b = 2 * np.mean(err)         # (4)
    w -= lr * grad_w                  # (5) step against the slope
    b -= lr * grad_b
    if step in (1, 2, 10, 100, 2000):
        loss = np.mean((w * x + b - y) ** 2)
        print(step, round(w, 4), round(b, 4), round(loss, 4))

# → 1    1.175  0.4    0.5522
# → 2    1.3687 0.4662 0.0637
# → 10   1.4062 0.4816 0.0501
# → 100  1.4016 0.4953 0.05
# → 2000 1.4    0.5    0.05

Line (1) starts both weights at zero and sets the learning rate to 0.05. At this starting point the model guesses 0 days for everything, and the loss is (2^2 + 3^2 + 5^2 + 6^2)/4 = 74/4 = 18.5.

Line (2) computes all four errors in one go, because NumPy applies arithmetic to every entry of an array at once. Lines (3) and (4) are the two formulas above. Line (5) is the update rule.

Let us check the first step by hand. At w = b = 0, every error is just -y_i, so the errors are (-2, -3, -5, -6). Then \partial L/\partial w = \tfrac{2}{4}(-2 \times 1 - 3 \times 2 - 5 \times 3 - 6 \times 4) = 0.5 \times (-47) = -23.5, and \partial L / \partial b = 0.5 \times (-16) = -8. The update gives w = 0 - 0.05 \times (-23.5) = 1.175 and b = 0 - 0.05 \times (-8) = 0.4, exactly the first printed line. The loss has dropped from 18.5 to 0.55 in one step.

By step 2000 the weights have settled at w = 1.4 and b = 0.5, which means "about 1.4 extra days per hundred kilometres, plus half a day of fixed handling". The loss stops at 0.05, not zero, because no straight line passes through all four points exactly. For this simple problem there is also a direct formula that gives the best line without any stepping, called least squares, and it gives exactly w = 1.4, b = 0.5, which confirms that gradient descent found the true best fit.

4. Stochastic and mini-batch gradient descent

The loop above used all four deliveries to compute every step. The fraud model has two million orders. Computing the exact gradient over all of them for every single step would make each step painfully slow, and you need thousands of steps.

The fix is to estimate the gradient from a small random handful of examples, called a mini-batch, typically 32 to a few thousand. The estimate is noisy, because a random handful will not represent all two million orders perfectly, but it is cheap, and it points roughly downhill. You can take a hundred noisy steps in the time one exact step would take, and a hundred roughly-right steps beat one perfect step by a wide margin. This is stochastic gradient descent, or SGD. "Stochastic" is a Greek-derived word meaning "involving randomness". Strictly, SGD meant using one example per step, but today people use the name for the mini-batch version too.

Three words describe the bookkeeping, and you will see them in every training log:

• Batch size: how many examples go into each gradient estimate, for example 256.
• Step (or iteration): one update of the weights, using one mini-batch.
• Epoch: one full pass through the whole training set. With two million orders and a batch size of 256, one epoch is 2{,}000{,}000 / 256 \approx 7{,}813 steps.

The noise turns out to be useful, not just tolerable. A perfectly accurate gradient can settle into a narrow, sharp dip in the landscape that happens to fit the training data very precisely but does not hold up on new data. The jitter from small batches tends to shake the model out of such narrow dips and into wider valleys, and models found in wide valleys generally do better on data they have not seen. That is one reason very large batch sizes often need extra tricks to match the results of smaller ones.

5. When the landscape is not a simple bowl

The losses on this page are shaped like bowls, with one lowest point and every downhill path leading to it. A bowl-shaped function like this is called convex. Linear regression and logistic regression, the first two models in Chapter 12.3, have convex losses, so gradient descent is guaranteed to find their best possible weights, given a sensible learning rate.

Neural networks do not have convex losses. Their landscapes have valleys, ridges, flat plateaus and saddle points. A saddle point is a spot that is a low point in one direction and a high point in another, like the middle of a horse's saddle, which dips between front and back but rises from side to side. The gradient there is zero even though it is not a minimum. A local minimum is a dip that is lower than everything around it but not the lowest point overall.

For years this was the main argument against neural networks: surely gradient descent would get stuck in some poor local minimum. In practice, for large networks, it mostly does not. One reason is the sheer number of dimensions. For a point to be a true trap, the ground has to rise in every one of millions of directions at once, and that is very unlikely. Much more often, the ground still slopes downhill in at least one direction, and the noise from mini-batches is enough to find it. Researchers still argue about the full explanation, but the practical fact is that gradient descent on very large networks finds good solutions far more reliably than anybody in the 1990s expected.

6. A practical trap: features on different scales

Here is a problem you will hit on your first real dataset. Suppose the fraud model uses two features: order amount in rupees, which ranges up to about 200,000, and account age in years, which ranges from 0 to about 10. The weight on the amount gets multiplied by enormous numbers, so the loss is extremely sensitive to it. The weight on account age gets multiplied by small numbers, so the loss barely notices it.

In landscape terms, the bowl becomes an extremely long, thin valley, like the squashed ellipses of 12.2.3 stretched to an absurd degree. A learning rate small enough not to overshoot in the steep "amount" direction is far too small to make progress in the gentle "age" direction, so training zigzags back and forth across the valley and crawls along it.

The fix is cheap and you should always do it: put every feature on a similar scale before training. The usual recipe is standardisation: subtract the feature's average and divide by its spread (its standard deviation, from Chapter 12.2.5), so that every feature ends up centred on zero with a typical size of about 1.

amount = np.array([1200., 54000., 800., 199000.])
z = (amount - amount.mean()) / amount.std()          # (1)
print(amount.mean(), amount.std().round())          # → 63750.0 81029.0
print(z.round(2))                                    # → [-0.77 -0.12 -0.78  1.67]

Line (1) standardises the amounts. The average is ₹63,750 and the spread is about ₹81,029, so the ₹199,000 order becomes (199000 - 63750) / 81029 \approx 1.67. After this, a large order is "about 1.7 spreads above average" instead of "199,000", and every feature speaks the same language as far as gradient descent is concerned. One rule matters here and is easy to break: the average and spread must be computed on the training data only and then reused unchanged on test data and in production. Computing them on everything, including the test set, quietly leaks information about the test set into training, which is the leakage problem from 12.1.3 in a subtler form.

Recall

• Gradient descent repeats \mathbf{w} \leftarrow \mathbf{w} - \eta\,\nabla L: feel the slope, step against it. Steps shrink by themselves near the bottom because the slope shrinks.
• The learning rate \eta decides everything: too small crawls, too large overshoots into divergence (the loss explodes to nan). Try values a factor of three to ten apart; a learning rate schedule changes it during training.
• With mean squared error on a line fit, gradient descent from zero reached w = 1.4, b = 0.5, the same answer as the exact least-squares formula.
• Stochastic gradient descent estimates the gradient from a random mini-batch. Noisy but far cheaper, and the noise helps find wide valleys. An epoch is one full pass over the data.
• Convex losses (linear and logistic regression) have one bottom. Neural network losses have saddle points and plateaus, but large networks train well anyway. Always apply standardisation to features, using statistics from the training data only.

Self-test: On L(w) = (w-3)^2, what does a learning rate of 0.25 do to the distance from the answer each step? With 500,000 examples and a batch size of 500, how many steps make one epoch? Why must the average used for standardisation come from the training set alone?

Next: 12.2.5 Probability and Bayes — a fraud model does not say "fraud"; it says "83 percent likely to be fraud", and to understand what that number means and how to reason with it, we need probability.