12.5.2 — Backpropagation by Hand

The XOR network in 12.5.1 worked because someone chose its seven numbers by hand. Cartwise's photo checker will have millions of weights, starting from random values, and the only information available is this: after a photo goes through the network, the output was wrong by some amount. The question every neural network must answer, millions of times during training, is this one:

The output was wrong. How much is each individual weight to blame, and in which direction should it move?

For the last layer, that is not hard: its weights touch the output directly. The difficulty is the hidden layers. A weight in the first layer affects the output only indirectly, through every layer after it. Backpropagation is the method that answers the blame question for every weight at once, in one backward sweep. It is the chain rule from 12.2.3, applied systematically, from the loss back towards the input. This page traces it by hand twice: once on the smallest possible chain, and once on a real two-layer network with ReLU, sigmoid and cross-entropy, and then checks every number with PyTorch.

1. The smallest chain

One input, one hidden value, one output, no activation functions yet. That is small enough to see every number:

x ──(× w1)──▶ h ──(× w2)──▶ ŷ        h = w1 · x       ŷ = w2 · h       L = ½ (ŷ − y)²

Take the input x = 2, the starting weights w_1 = 3 and w_2 = 4, and a target of y = 30. The loss here is half the squared error. The half is there only to cancel the 2 that appears when squaring is differentiated, which keeps the numbers tidy.

Forward pass. Compute the values from left to right:

h = 3 \times 2 = 6, \qquad \hat{y} = 4 \times 6 = 24, \qquad L = \tfrac{1}{2}(24 - 30)^2 = 18

The prediction is 24 and the target is 30, so the output is too low.

Backward pass. Now compute slopes from right to left. At each step we ask: how does the loss change if this value changes? Each answer is built from the one before it.

Step 1, the loss with respect to the output. The slope of \tfrac{1}{2}(\hat{y} - y)^2 with respect to \hat{y} is \hat{y} - y:

\frac{\partial L}{\partial \hat{y}} = 24 - 30 = -6

Read it as: raising the prediction by a little would lower the loss at a rate of 6 per unit. Negative means "the prediction should go up".

Step 2, the loss with respect to w_2. The output is \hat{y} = w_2 h, so changing w_2 changes \hat{y} at a rate of h = 6. The chain rule multiplies the two slopes:

\frac{\partial L}{\partial w_2} = \frac{\partial L}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial w_2} = -6 \times 6 = -36

Step 3, the loss with respect to the hidden value h. Changing h changes \hat{y} at a rate of w_2 = 4:

\frac{\partial L}{\partial h} = \frac{\partial L}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial h} = -6 \times 4 = -24

This is the step that gives the method its name. The error at the output has been carried backwards through the weight w_2 and has become an error signal on the hidden value. The hidden value now "knows" how it should change, even though nobody ever gave it a target of its own.

Step 4, the loss with respect to w_1. The hidden value is h = w_1 x, so changing w_1 changes h at a rate of x = 2:

\frac{\partial L}{\partial w_1} = \frac{\partial L}{\partial h} \cdot \frac{\partial h}{\partial w_1} = -24 \times 2 = -48

Update. With a learning rate of 0.01, gradient descent from 12.2.4 moves each weight against its slope:

w_1 \leftarrow 3 - 0.01 \times (-48) = 3.48, \qquad w_2 \leftarrow 4 - 0.01 \times (-36) = 4.36

Check that it helped. Run forward again: h = 3.48 \times 2 = 6.96, \hat{y} = 4.36 \times 6.96 \approx 30.35, and L = \tfrac{1}{2}(0.35)^2 \approx 0.06. The loss fell from 18 to 0.06 in one step. (A step that good is luck of the numbers; on real problems each step improves things a little.)

Three facts that fall out of this trace

Every gradient is a product of local slopes along the path. No step needed to know anything about the whole network. Each one only needed its own small operation (multiply by w_2, multiply by x) and the gradient arriving from the step after it. That locality is what makes backpropagation possible to build into a library: each kind of operation only has to know its own slope.

The forward values are needed again on the way back. Step 2 needed h = 6, and Step 4 needed x = 2. So during training, every intermediate value computed on the way forward must be kept in memory until the backward pass uses it. This is why training a network takes far more memory than running it, and section 5 returns to the consequences.

The work is shared. Step 4 reused the -24 computed in Step 3 rather than starting again from the loss. In a network with a hundred layers, each layer's gradient is computed once and reused by the layer before it. That reuse is why one backward pass costs only about as much as one or two forward passes, instead of one forward pass per weight.

2. A real two-layer network

Now the same procedure on a network with everything a real one has: two inputs, a hidden layer of two neurons with ReLU, an output neuron with a sigmoid, and the binary cross-entropy loss from 12.2.6. Think of the inputs as two photo features and the output as "probability the photo is bad". One training example: \mathbf{x} = (1, 2), and it really is a bad photo, y = 1.

The starting weights:

W_1 = \begin{bmatrix} 0.5 & -0.3 \\ 0.2 & 0.4 \end{bmatrix},\quad \mathbf{b}_1 = \begin{bmatrix} 0 \\ 0.1 \end{bmatrix},\qquad \mathbf{w}_2 = (0.7,\ -0.6),\quad b_2 = 0

The two-layer network traced in both directions. The blue row computes and stores every intermediate value. The red row carries the error back, step by step, multiplying by each step's local slope. Notice the first hidden neuron: its sum was negative, so ReLU output zero, and on the way back ReLU blocks its gradient completely.

Forward

Follow the blue row of the figure.

1. Hidden sums. z_{1,1} = 0.5 \times 1 + (-0.3) \times 2 + 0 = -0.1, and z_{1,2} = 0.2 \times 1 + 0.4 \times 2 + 0.1 = 1.1.
2. ReLU. h_1 = \max(0, -0.1) = 0, and h_2 = \max(0, 1.1) = 1.1. The first hidden neuron is switched off for this photo.
3. Output sum. z_2 = 0.7 \times 0 + (-0.6) \times 1.1 + 0 = -0.66.
4. Sigmoid. p = \sigma(-0.66) \approx 0.341. The network gives this bad photo only a 34 percent chance of being bad.
5. Loss. Since y = 1, the cross-entropy is -\ln p = -\ln 0.341 \approx 1.077.

Backward

Now the red row, right to left.

Step 1, the output. 12.2.6 showed that for a sigmoid followed by cross-entropy, the slope of the loss with respect to the raw score is simply the probability minus the truth:

\delta_2 = \frac{\partial L}{\partial z_2} = p - y = 0.341 - 1 = -0.659

The Greek letter \delta (delta) is the usual name for "the error signal at this point". Negative means the score z_2 should go up.

Step 2, the output weights. Since z_2 = w_{2,1} h_1 + w_{2,2} h_2 + b_2, each output weight's slope is \delta_2 times the hidden value it multiplies:

\frac{\partial L}{\partial w_{2,1}} = -0.659 \times 0 = 0, \qquad \frac{\partial L}{\partial w_{2,2}} = -0.659 \times 1.1 \approx -0.725, \qquad \frac{\partial L}{\partial b_2} = -0.659

The first is zero because h_1 was zero: a weight on an input that was off had no effect on this mistake, so it gets no blame.

Step 3, back to the hidden values. Each hidden value influenced z_2 through its output weight, so its error signal is \delta_2 times that weight:

\frac{\partial L}{\partial h_1} = -0.659 \times 0.7 \approx -0.461, \qquad \frac{\partial L}{\partial h_2} = -0.659 \times (-0.6) \approx +0.396

Read the signs. Hidden neuron 1 pushes the output towards "bad photo" (its weight is +0.7), so the network would like it to be larger. Hidden neuron 2 pushes against "bad photo" (its weight is −0.6), so the network would like it to be smaller: a positive slope means "decrease me".

Step 4, through the ReLU. The ReLU's slope is 1 where its input was positive and 0 where it was negative. Hidden neuron 1's input was -0.1, so its slope is 0, and its error signal is blocked: \delta_{1,1} = -0.461 \times 0 = 0. Hidden neuron 2's input was 1.1, so its slope is 1, and the signal passes through unchanged: \delta_{1,2} = 0.396.

This is exactly the zero-gradient behaviour of ReLU from 12.5.1, in arithmetic. Neuron 1 would like to be larger, but because it was switched off for this example, it receives no learning signal from it. If it were switched off for every example, it would never learn anything again: a dying ReLU.

Step 5, the first-layer weights. Each first-layer weight connects one input to one hidden neuron, so its slope is that hidden neuron's error signal times that input:

\frac{\partial L}{\partial W_1} = \begin{bmatrix} 0 \times 1 & 0 \times 2 \\ 0.396 \times 1 & 0.396 \times 2 \end{bmatrix} = \begin{bmatrix} 0 & 0 \\ 0.396 & 0.791 \end{bmatrix}, \qquad \frac{\partial L}{\partial \mathbf{b}_1} = \begin{bmatrix} 0 \\ 0.396 \end{bmatrix}

Update and check

With a learning rate of 0.1, every weight moves against its slope. The second row of W_1 becomes (0.2 - 0.0396,\ 0.4 - 0.0791) \approx (0.160, 0.321), its bias becomes 0.1 - 0.0396 \approx 0.060, the second output weight becomes -0.6 + 0.0725 \approx -0.527, and b_2 becomes 0.066. The first row of W_1 does not move at all.

Run forward again: z_{1,2} = 0.160 + 0.642 + 0.060 \approx 0.863, so h_2 \approx 0.863; z_2 = -0.527 \times 0.863 + 0.066 \approx -0.389; p = \sigma(-0.389) \approx 0.404. The loss is now -\ln 0.404 \approx 0.907, down from 1.077. The network now gives the bad photo a 40 percent chance instead of 34. It got there in two ways at once: it weakened the hidden neuron that argued against "bad", and it made that neuron's vote count less.

The same thing, as matrices

Written for whole layers instead of single numbers, the backward pass is three lines, and every deep learning library implements exactly these:

\boldsymbol{\delta}_2 = \mathbf{p} - \mathbf{y}

\frac{\partial L}{\partial W_2} = \boldsymbol{\delta}_2\, \mathbf{h}^\top, \qquad \boldsymbol{\delta}_1 = \big(W_2^\top \boldsymbol{\delta}_2\big) \odot f'(\mathbf{z}_1)

\frac{\partial L}{\partial W_1} = \boldsymbol{\delta}_1\, \mathbf{x}^\top

Read them aloud. The output error is prediction minus truth. A layer's weight gradient is its error signal times the transpose of its input, which is "error times input" for every pair at once. To pass the error back one layer, multiply by the transpose of that layer's weight matrix, which sends each output's error back along the same connections it came forward on, and then multiply entry by entry (the \odot symbol) by the activation's slope at the values computed on the way forward. For deeper networks, the middle line simply repeats once per layer. The transpose from 12.2.2 appears here for a reason: the forward pass sends signals through W, so the backward pass sends blame through W^\top.

3. Checking every number with PyTorch

You will never write the backward pass yourself. PyTorch's automatic differentiation from 12.2.3 records the forward operations and runs this exact procedure. Here it is on the same network, to confirm the hand calculation:

import torch
import torch.nn.functional as F

x  = torch.tensor([1.0, 2.0])
y  = torch.tensor(1.0)
W1 = torch.tensor([[0.5, -0.3], [0.2, 0.4]], requires_grad=True)   # (1)
b1 = torch.tensor([0.0, 0.1], requires_grad=True)
w2 = torch.tensor([0.7, -0.6], requires_grad=True)
b2 = torch.tensor(0.0, requires_grad=True)

h = torch.relu(W1 @ x + b1)                                          # (2) forward
p = torch.sigmoid(w2 @ h + b2)
loss = F.binary_cross_entropy(p, y)
print(round(loss.item(), 3))                                         # → 1.077

loss.backward()                                                      # (3) backward
print(w2.grad)   # → tensor([-0.0000, -0.7252])   (−0 is just zero)
print(W1.grad)   # → tensor([[0.0000, 0.0000], [0.3956, 0.7911]])
print(b1.grad)   # → tensor([0.0000, 0.3956])
print(b2.grad)   # → tensor(-0.6593)

Line (1) creates each weight with requires_grad=True, which tells PyTorch to track every operation that uses it. Line (2) is the forward pass, written exactly as the maths: matrix times vector plus bias, ReLU, dot product plus bias, sigmoid, cross-entropy. Line (3) runs backpropagation through the recorded steps and stores each gradient in that tensor's .grad. Every printed number matches the hand trace, including the zeros in the first row of W1.grad where the ReLU blocked the signal.

4. Why backpropagation was a breakthrough

The idea of applying the chain rule backwards through a computation was worked out more than once: Seppo Linnainmaa described the general method for computer programs in his 1970 master's thesis, and Paul Werbos proposed using it for neural networks in 1974. It became widely known through a 1986 paper in Nature by David Rumelhart, Geoffrey Hinton and Ronald Williams, which showed that networks trained this way learned useful internal features on their own, exactly the hidden-layer behaviour from 12.5.1.

Its importance comes down to cost. The naive alternative, nudging each weight separately and re-running the network to see what happens, needs a full forward pass for every single weight. For a network with a hundred million weights, that is a hundred million forward passes for one training step. Backpropagation gets all hundred million gradients from one forward pass and one backward pass, at a cost of roughly two to three forward passes in total. Without that, training networks of any useful size would be out of reach, however fast the hardware.

5. What the backward pass costs in memory

The second fact from section 1, that forward values are needed again on the way back, has large practical consequences.

During training, every intermediate result of the forward pass (each layer's sums and activations, for every example in the batch) must be kept until the backward pass has used it. For a big network and a big batch, these stored values, called the activations, take far more memory than the weights themselves. That is why:

• Training needs much more memory than inference. At inference, each layer's values can be thrown away as soon as the next layer has used them.
• Batch size is usually limited by memory. Doubling the batch doubles the stored activations.
• Gradient checkpointing exists. You deliberately throw away some stored activations during the forward pass and recompute them during the backward pass when they are needed, trading roughly a third more computation for a large saving in memory. Chapter 12.5.7 covers it with the other tricks that make big models fit.

It also explains a line you will see constantly in PyTorch code, with torch.no_grad():. Inside it, PyTorch does not record operations or keep values for a backward pass, which saves memory and time. You use it whenever you are only running the model, as in evaluation or serving, and that is what the weight-setting block in 12.5.1 used it for.

6. When the backward signal goes wrong

The backward pass multiplies many local slopes together, one per layer. That product can misbehave in two opposite ways.

If the slopes are mostly below 1, as with sigmoids, the product shrinks towards zero as it travels back, and early layers stop learning: the vanishing gradient from 12.5.1. If the slopes or the weights are mostly above 1, the product grows layer after layer until the numbers overflow and the loss becomes nan: the exploding gradient. Both are about the same thing, a long product of numbers that are not close to 1, and the cures in Chapter 12.5.4, careful starting weights, normalisation, residual connections and gradient clipping, all work by keeping that product close to 1.

Recall

• Backpropagation answers "how much is each weight to blame?" by applying the chain rule backwards from the loss, one step at a time. Each step needs only its own local slope and the error signal arriving from the step after it.
• On the smallest chain, the error -6 at the output became -24 at the hidden value and -48 on w_1; one update cut the loss from 18 to 0.06.
• On a real layer: \delta_2 = p - y; weight gradient = error × input; pass error back through W^\top; multiply by the activation's slope. A ReLU whose input was negative blocks the gradient completely.
• Forward values must be stored for the backward pass, so training needs far more memory than inference; gradient checkpointing trades recomputation for memory, and torch.no_grad() skips the bookkeeping at inference.
• One backward pass gives every gradient for the cost of about two or three forward passes, which is what made large networks trainable. Long products of slopes can vanish or explode.

Self-test: In the smallest chain, what would \partial L / \partial w_1 be if x were 1 instead of 2? In the two-layer trace, why did the first row of W_1 not change at all? If the true label had been y = 0, what would \delta_2 have been, and which way would w_{2,2} have moved?

Next: 12.5.3 The Training Loop in PyTorch — one example and one update is the core; real training repeats it over batches and epochs with smarter optimisers, and the next page builds that loop line by line.