12.2.3 — Derivatives, Gradients and the Chain Rule

Here is the situation training puts you in. Cartwise's fraud model has a weight vector, and for any choice of weights you can compute one number that says how wrong the model is across all the past orders. That number is called the loss: zero would mean every past order was scored perfectly, and bigger means worse. Training means finding the weights that make the loss as small as possible.

A real model has millions of weights, so you cannot try every combination. You need to stand at the current weights and ask a sharper question: "If I nudge this one weight up a tiny bit, does the loss go up or down, and how fast?" If nudging it up makes the loss go down, nudge it up. If it makes the loss go up, nudge it down. Do that for every weight at once, repeat a few thousand times, and you have trained a model.

The number that answers "how fast does the output change when I nudge the input" is called a derivative. This page builds it from nothing, extends it to many weights (the gradient), and then explains the chain rule, which is the trick that lets you compute the derivative for every weight in a hundred-layer network in one pass. That trick is called backpropagation, and it is how every neural network in existence is trained.

1. A derivative is a slope

Start with the smallest possible example: a "model" with one weight w, and a loss that happens to be

L(w) = (w - 3)^2

Read it as "L of w equals w minus three, squared". This loss is smallest, exactly zero, when w = 3. You can see that just by looking at it, but pretend you cannot, because with a million weights you will not be able to.

The loss L(w) = (w − 3)². At w = 5 the red tangent line shows the slope is 4: the loss is rising as w increases, so the way down is to the left. At the bottom the slope is zero, which is how you know you have arrived.

Suppose the current weight is w = 5. The loss there is (5 - 3)^2 = 4. Now nudge w up by a tiny amount, say 0.001, so w = 5.001. The loss becomes (2.001)^2 = 4.004001. It went up by 0.004001 when w went up by 0.001. Divide one by the other:

\frac{\text{change in loss}}{\text{change in } w} = \frac{0.004001}{0.001} = 4.001

That ratio is the steepness of the curve at w = 5: right there, the loss rises about 4 units for every 1 unit of w. If you made the nudge smaller still, 0.000001, the ratio would come out even closer to exactly 4. The exact value you are heading towards as the nudge shrinks to nothing is the derivative. It is the slope of the red line in the figure, a straight line that just touches the curve at that point, called the tangent line.

The derivative is written in two common ways, and you will see both:

• L'(w), read "L prime of w". This short notation comes from the mathematician Joseph-Louis Lagrange.
• \dfrac{dL}{dw}, read "dee L by dee w". This comes from Gottfried Wilhelm Leibniz, one of the two inventors of calculus in the late 1600s (Isaac Newton was the other). The d stands for "a tiny difference in", so the whole thing reads naturally as "a tiny change in L divided by the tiny change in w that caused it". That is exactly the ratio we just computed.

Formally, the derivative is the limit of that ratio as the nudge h shrinks to zero:

\frac{dL}{dw} = \lim_{h \to 0} \frac{L(w + h) - L(w)}{h}

Read it as "the limit, as h goes to zero, of L of w plus h minus L of w, all over h". "Limit" means "the value this gets closer and closer to". (Volume II builds limits properly in its calculus Part, if you want the full treatment.)

Getting the exact formula

Instead of nudging numerically, we can work out the slope at every point at once. Expand L(w + h) = (w + h - 3)^2. Write a = w - 3 to keep it short, so it becomes (a + h)^2 = a^2 + 2ah + h^2. Subtract L(w) = a^2 and you are left with 2ah + h^2. Divide by h and you get 2a + h. Now let h shrink to zero, and the h vanishes, leaving 2a:

\frac{dL}{dw} = 2(w - 3)

Check it at w = 5: 2 \times (5 - 3) = 4. That matches the nudging experiment. At w = 3, the bottom of the curve, it gives 2 \times 0 = 0: the curve is flat there, which is the green line in the figure.

Now read the derivative as an instruction, which is the whole reason we want it. The amber box in the figure says it plainly:

• Positive derivative (to the right of the bottom): the loss goes up as w goes up, so decrease w.
• Negative derivative (to the left of the bottom): the loss goes down as w goes up, so increase w.
• Zero derivative: you are at the bottom, or at least on flat ground. Stop.

So the rule is always the same: move the weight in the opposite direction to the derivative. That single sentence is gradient descent, and Chapter 12.2.4 turns it into an algorithm.

The handful of rules you actually need

You will almost never compute a derivative by hand in real work, because libraries do it for you (section 5). But a few rules come up constantly when reading papers and documentation, and each one comes from the same nudging argument above:

Function	Its derivative	In words
c (a constant)	0	a flat line has no slope
w^n	n\,w^{n-1}	bring the power down, reduce it by one
c \cdot f(w)	c \cdot f'(w)	constants just come along
f(w) + g(w)	f'(w) + g'(w)	slopes of a sum add
e^w	e^w	the exponential is its own slope
\ln w	1/w	the logarithm's slope shrinks as w grows

The exponential e^w and the natural logarithm \ln w are explained properly in Chapter 12.2.6, where they turn up in the loss used by every classifier. For now, just note that they are on the list.

2. Many weights: partial derivatives and the gradient

Real losses depend on many weights. Take a loss with two weights:

L(w_1, w_2) = (w_1 - 3)^2 + 2\,(w_2 + 1)^2

This loss is smallest at w_1 = 3 and w_2 = -1. To find out which way to move, we ask the derivative question about each weight separately, holding all the others fixed. That is called a partial derivative, and it is written with a curly d, \partial, read "partial". So \dfrac{\partial L}{\partial w_1} is "partial L by partial w-one".

To take the partial with respect to w_1, treat w_2 as if it were a fixed number. The second term then does not change at all when w_1 changes, so its slope is 0, and only the first term matters:

\frac{\partial L}{\partial w_1} = 2(w_1 - 3), \qquad \frac{\partial L}{\partial w_2} = 4(w_2 + 1)

The second one comes from the same rule: the slope of (w_2 + 1)^2 is 2(w_2 + 1), and the 2 in front of it multiplies that to give 4(w_2 + 1).

Now collect all the partial derivatives into one vector. That vector is called the gradient, written \nabla L and read "grad L" or "nabla L" (the upside-down triangle is called nabla, after an ancient harp it resembles):

\nabla L = \left(\frac{\partial L}{\partial w_1},\ \frac{\partial L}{\partial w_2}\right) = \big(2(w_1 - 3),\ 4(w_2 + 1)\big)

At the point w_1 = 5, w_2 = 1, the gradient is (2 \times 2,\ 4 \times 2) = (4, 8).

The loss over two weights, seen from above. The rings are lines of equal loss. The gradient (red) at your position points in the direction the loss rises fastest; its negative (green) points the way down.

Here is why the gradient is the right object. Think of the loss as a landscape, where your position is the current weights and the height is the loss. The rings in the figure are like the contour lines on a hiking map, each one joining points of equal height. The gradient points in the direction in which the ground rises most steeply, and its length says how steep it is there. So the negative of the gradient, the green arrow, points in the direction of steepest descent. Gradient descent simply means: take a small step along the green arrow, recompute the gradient, and repeat.

Notice that the red arrow leans more towards w_2 (8) than w_1 (4), even though you are the same distance, 2 units, from the minimum in both directions. That is because the loss is twice as sensitive to w_2, thanks to the 2 in front of its term, which is also why the rings are squashed into ellipses rather than circles. It also means the green arrow does not point straight at the minimum: it points down the steepest slope, which is not quite the same thing. This unevenness between weights is a real problem in training, because one step size that suits a steep direction is far too timid for a gentle one, and Chapter 12.5.3 shows how the Adam optimiser handles it.

A model with a million weights has a million-entry gradient. You cannot draw that landscape, but the meaning is exactly the same: it is a million-number vector that says, for each weight, which way to nudge it and how much it matters.

3. The chain rule: slopes of functions inside functions

Models are not single formulas. They are functions fed into other functions fed into other functions. A neural network computes a dot product, passes it through a squashing function, feeds that into another dot product, and so on, layer after layer, before a loss is computed at the very end. To train it, we need the derivative of that final loss with respect to a weight buried right at the beginning.

The chain rule tells us how. Here is the idea with gears. Suppose gear A turns gear B, and B turns 3 times for every turn of A. Gear B turns gear C, and C turns 2 times for every turn of B. How many times does C turn for each turn of A? Obviously 3 \times 2 = 6. When things are connected in a chain, their rates of change multiply. That is the whole chain rule.

Here is a small concrete chain. Start with weight w, compute u = 3w + 1, then compute y = u^2:

The chain rule on a two-step chain. Going forward (blue) computes the values. Going backward (red), each box contributes its own local slope, and the slopes multiply to give the slope of the end with respect to the start.

Walk the figure in two passes. The forward pass, along the blue arrows, computes the values: with w = 1, u = 3 \times 1 + 1 = 4, and y = 4^2 = 16.

The backward pass, along the red arrow, computes the slope. Each box knows only its own small, local slope. The first box, u = 3w + 1, has slope \dfrac{du}{dw} = 3, because u goes up 3 for each unit of w. The second box, y = u^2, has slope \dfrac{dy}{du} = 2u by the power rule, which at u = 4 is 8. The chain rule says to multiply the local slopes:

\frac{dy}{dw} = \frac{dy}{du} \cdot \frac{du}{dw} = 8 \times 3 = 24

Let us check it by nudging, so it is not just a claim. Make w = 1.001. Then u = 4.003, and y = 4.003^2 = 16.024009. The output rose by 0.024009 when w rose by 0.001, a ratio of 24.009, which heads to exactly 24 as the nudge shrinks. The chain rule was right.

Notice how naturally Leibniz's notation reads here: the du on the bottom of one fraction and the top of the other look as if they cancel, leaving dy/dw. They are not literally fractions, but the notation was designed so that this "cancelling" always gives the correct answer, which is a large part of why it won over Newton's notation.

Why this is the engine of all deep learning

Now picture a network with fifty layers and a loss at the end. To nudge a weight in layer 1, you need \dfrac{\partial L}{\partial w} for that weight, and the loss is fifty functions away from it. The chain rule says: multiply the fifty local slopes along the way.

The clever part is the order. You compute the slopes starting from the loss and working backwards, and every layer reuses the product already computed for the layers after it. The slope of the loss with respect to layer 50's output is computed once, then multiplied by layer 50's local slope to get the slope with respect to layer 49's output, which is then reused for layer 48, and so on down to layer 1. One backward sweep gives the derivative for every weight in the network, at a cost similar to a single forward pass. Running the chain rule backwards like this is backpropagation, the method from the 1986 paper mentioned in 12.1.1. Chapter 12.5.2 traces it through a small network with real numbers.

4. The computer does this for you: automatic differentiation

In real work you never write derivative formulas. You write the forward calculation, and the library records every operation as you go, then runs the chain rule backwards automatically. This is called automatic differentiation, often shortened to autodiff or autograd. Here is the chain from section 3 in PyTorch, the library most deep learning is written in:

import torch

w = torch.tensor(1.0, requires_grad=True)   # (1)
u = 3 * w + 1                               # (2)
y = u ** 2                                  # (3)
y.backward()                                # (4)

print(y.item())                             # → 16.0
print(w.grad)                               # (5) → tensor(24.)

Line (1) creates the weight as a PyTorch tensor (a number, here, that PyTorch can track) and the flag requires_grad=True tells PyTorch to remember every operation that uses it. Lines (2) and (3) are the ordinary forward calculation, exactly as in the figure. Behind the scenes PyTorch has recorded "multiply by 3, add 1, square" as a little graph of steps.

Line (4) is where the chain rule happens. backward() walks that recorded graph from y back to w, multiplying the local slopes, and stores the result on w. Line (5) prints it: 24, the same answer as the hand calculation and the nudging check.

And here is the same answer found by nudging, which is a useful way to check any gradient you are unsure about:

def f(w):
    return (3 * w + 1) ** 2

h = 1e-6                                    # (1) a very small nudge
print((f(1 + h) - f(1 - h)) / (2 * h))      # (2) → 24.000000...

Line (1) picks a tiny nudge, one millionth. Line (2) nudges in both directions and divides by the total distance, which is a slightly more accurate version of the one-sided nudge used earlier. Checking an automatic gradient against a nudged one like this is called a gradient check, and it is how people find bugs when they write a new kind of layer by hand.

Why not just train by nudging and skip the chain rule? Because nudging needs two full runs of the model for every single weight. With a billion weights, that is two billion runs per training step. Backpropagation gets every derivative from one forward run and one backward run. That difference is the reason training large networks is possible at all.

Recall

• The loss is one number measuring how wrong the model is. Training means making it small.
• A derivative is the slope of a function at a point: how fast the output changes per unit nudge of the input, the slope of the tangent line. For (w-3)^2 it is 2(w-3). Move a weight against its derivative to reduce the loss.
• A partial derivative \partial L/\partial w_i nudges one weight with the rest held fixed. The gradient \nabla L is the vector of all of them; it points steepest uphill, so its negative points downhill.
• The chain rule: along a chain of functions, local slopes multiply (dy/dw = dy/du \cdot du/dw). Running it backwards from the loss, reusing products, gives every weight's derivative in one sweep: backpropagation.
• Automatic differentiation (PyTorch's backward()) records the forward steps and applies the chain rule for you. Nudging both ways is a gradient check, but far too slow to train with.

Self-test: For L(w) = (w-3)^2, what is the derivative at w = 1, and which way should w move? Compute the gradient of L(w_1, w_2) = (w_1-3)^2 + 2(w_2+1)^2 at (3, 0) and say what it means that the first entry is zero. For y = (2w - 5)^2, use the chain rule to find dy/dw at w = 4.

Next: 12.2.4 Gradient Descent — we know which way is downhill; now we walk, step by step, with real numbers, and find out what happens when the steps are too small, too large, or noisy.