12.4.1 — From a Neuron to Backpropagation

Give a linear model this problem:

x_1	x_2	output
0	0	0
0	1	1
1	0	1
1	1	0

That is XOR, and no straight line separates the ones from the zeros. Draw the four points and try: the two 1s are on opposite corners. A single-layer model computes a weighted sum and a threshold, which is a straight line, so it cannot do this — the result that ended the first wave of neural network research (Chapter 12.1).

The fix is to stack layers with something non-linear between them. Doing that raises one hard question — how do you know which of the inner weights to blame for an error? — and backpropagation is the answer.

1. The neuron

One neuron does three things: multiply, sum, and bend. The bending is what makes depth worth anything.

z = w_1x_1 + w_2x_2 + \dots + b \qquad a = f(z)

The weighted sum is the dot product from Chapter 12.2. The bias b shifts the threshold — without it, the neuron must fire based purely on the inputs' balance and cannot represent "output 1 unless strongly pushed down".

The activation f is the only non-linear part, and it is essential. Two stacked linear layers are W_2(W_1x) = (W_2W_1)x, which is one linear layer with a different matrix. Without a non-linearity, a hundred layers have exactly the expressive power of one — they can only ever draw a straight line, and XOR stays unsolvable.

2. The activation functions

Sigmoid — \sigma(z) = \frac{1}{1+e^{-z}}, squashing to (0,1). Historically standard, now used only at the output of a binary classifier. Its problem is the derivative: \sigma'(z) = \sigma(z)(1-\sigma(z)), which peaks at 0.25 and approaches zero for large inputs in either direction. Chain ten of those together (Chapter 12.2) and the gradient is at most 0.25^{10} \approx 10^{-6}. That is the vanishing gradient problem, and it is why deep networks were untrainable for years.

Tanh — the same shape mapped to (-1,1). Zero-centred, which helps, and it still saturates.

ReLU — \max(0, z). This is the change that made deep learning practical, and the reasons are almost embarrassingly simple:

• The derivative is exactly 1 for positive inputs. No shrinking, so gradients survive many layers.
• It is a comparison and a select — far cheaper than an exponential.
• It produces exact zeros, so many neurons are inactive for any given input, which is efficient and acts as a mild regulariser.

Its failure is the dying ReLU: a neuron whose input is always negative outputs zero, has a zero gradient, and can never recover — it is permanently dead. Usually caused by too large a learning rate.

Leaky ReLU — \max(0.01z, z) — keeps a small slope for negatives so nothing dies.

GELU and SiLU/Swish are the smooth modern versions, and GELU is what transformers use (Chapter 12.5.1). They behave like ReLU for large values and curve gently near zero, which empirically trains slightly better.

At the output, the activation is decided by the task: none for regression, sigmoid for binary classification, softmax for multi-class (Chapter 12.2).

3. A forward pass with real numbers

A network with two inputs, two hidden neurons and one output — enough to solve XOR.

Weights (chosen here, not trained, so the arithmetic is checkable):

Hidden neuron 1:  w = [ 1,  1],  b = -0.5    ← fires if either input is on   (OR)
Hidden neuron 2:  w = [ 1,  1],  b = -1.5    ← fires only if both are on    (AND)
Output neuron:    w = [ 1, -2],  b = -0.5    ← "OR but not AND"             (XOR)

Input [1, 0], with ReLU on the hidden layer and sigmoid on the output:

h1 = ReLU(1×1 + 1×0 − 0.5) = ReLU(0.5)  = 0.5
h2 = ReLU(1×1 + 1×0 − 1.5) = ReLU(−0.5) = 0
z  = 1×0.5 + (−2)×0 − 0.5  = 0
a  = σ(0) = 0.5

Input [1, 1]:

h1 = ReLU(2 − 0.5) = 1.5
h2 = ReLU(2 − 1.5) = 0.5
z  = 1×1.5 + (−2)×0.5 − 0.5 = −0.5
a  = σ(−0.5) = 0.38

Read what the hidden layer did. Neuron 1 learned OR, neuron 2 learned AND, and the output computed OR minus twice AND — which is XOR. The hidden layer built new features that made the problem linearly separable, and that sentence is what a hidden layer is for. Nobody told it to compute OR and AND; with training, that is the kind of decomposition it finds.

In matrix form the whole layer is one operation:

\mathbf{h} = \text{ReLU}(W_1\mathbf{x} + \mathbf{b}_1), \qquad \hat{y} = \sigma(W_2\mathbf{h} + b_2)

which is why a GPU runs it well (Chapter 12.2).

4. Backpropagation, traced by hand

The question backpropagation answers: the output was wrong by some amount — how much is each individual weight to blame?

Take a minimal network so every number is visible: one input, one hidden neuron, one output, no activations for now.

x ──(w1)──▶ h ──(w2)──▶ ŷ          h = w1·x        ŷ = w2·h

With x = 2, w_1 = 3, w_2 = 4, and a target y = 30:

Forward:

h  = 3 × 2 = 6
ŷ  = 4 × 6 = 24
L  = ½(ŷ − y)² = ½(24 − 30)² = 18

Backward. Work from the loss toward the inputs, applying the chain rule at each step.

Step 1 — the loss with respect to the output:

\frac{\partial L}{\partial \hat{y}} = \hat{y} - y = 24 - 30 = -6

Read it as: increasing the prediction by 1 changes the loss by −6 — so the prediction is 6 too low.

Step 2 — the loss with respect to w_2. Since \hat{y} = w_2 h, we have \frac{\partial \hat{y}}{\partial w_2} = h = 6. Chain them:

\frac{\partial L}{\partial w_2} = \frac{\partial L}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial w_2} = -6 \times 6 = -36

Step 3 — the loss with respect to h. Since \frac{\partial \hat{y}}{\partial h} = w_2 = 4:

\frac{\partial L}{\partial h} = -6 \times 4 = -24

This is the step that gives the method its name: the error signal has been carried backwards through w_2 to become an error signal on the hidden value.

Step 4 — the loss with respect to w_1. Since h = w_1x, \frac{\partial h}{\partial w_1} = x = 2:

\frac{\partial L}{\partial w_1} = -24 \times 2 = -48

Update, with learning rate \eta = 0.01:

w1 ← 3 − 0.01×(−48) = 3.48
w2 ← 4 − 0.01×(−36) = 4.36

Check that it helped. New forward pass: h = 3.48 \times 2 = 6.96, \hat{y} = 4.36 \times 6.96 = 30.35, and L = \frac{1}{2}(0.35)^2 = 0.06. From 18 to 0.06 in one step.

Three general points fall out of that trace.

Every gradient is a product of local derivatives along the path. Nothing global is computed; each node only needs to know its own operation and the gradient arriving from above.

The forward values are needed for the backward pass. \frac{\partial L}{\partial w_2} required h = 6. This is why training uses far more memory than inference — every intermediate activation must be kept until its gradient is computed. It is also why gradient checkpointing exists: discard some activations and recompute them, trading time for memory.

With an activation function there is one extra factor per layer: \frac{\partial a}{\partial z} = f'(z). With sigmoid that factor is at most 0.25, and multiplying many of them is exactly the vanishing gradient. With ReLU it is 1 or 0, so nothing shrinks — which is section 2's claim, now visible in the arithmetic.

5. Gradient descent in practice

Batch gradient descent computes the gradient over the entire dataset before one update. Accurate, and impossibly slow for large data.

Stochastic gradient descent updates after every single example. Fast and noisy — and the noise is partly useful, because it can knock the model out of a poor local region.

Mini-batch gradient descent uses 32 to 512 examples per update, and is what everyone actually does. It balances the noise, and — the practical reason — a batch is a matrix multiply, which is what the hardware wants.

Beyond plain descent, three refinements that are standard:

Momentum accumulates a running average of past gradients, so consistent directions build speed and oscillations cancel. The physical analogy is a ball rolling downhill rather than teleporting.

Adaptive learning rates (RMSProp, Adam) keep a per-parameter step size, so rarely-updated parameters take larger steps. Adam combines momentum and adaptivity and is the default optimiser; AdamW, which fixes how weight decay interacts with it, is what large models use.

Learning rate schedules start with a short warmup — a few hundred steps ramping up from near zero, which prevents early instability — then decay, often as a cosine curve. This is not decoration; large models fail to train without warmup.

6. What goes wrong

Vanishing gradients — early layers stop learning. Fixed by ReLU-family activations, careful initialisation, normalisation layers, and residual connections (Chapter 12.4.2), which give the gradient a direct path backwards.

Exploding gradients — the product grows instead of shrinking, weights become NaN. Fixed by gradient clipping: rescale the gradient if its magnitude exceeds a threshold. One line, and it saves training runs.

Dead ReLUs — usually too high a learning rate; use leaky ReLU or lower the rate.

Bad initialisation. Never initialise all weights to zero — every neuron in a layer then computes the same thing and receives the same gradient, so they stay identical forever and the layer has the capacity of one neuron. Use a scheme sized to the layer (He initialisation for ReLU, Xavier for tanh) so the signal neither shrinks nor explodes as it passes through.

Loss not decreasing at all — in order: check the learning rate, check the data pipeline (labels aligned with inputs?), and try to overfit a single batch deliberately. A model that cannot drive the loss to near zero on ten examples has a bug, not a tuning problem. That is the single most useful debugging technique in deep learning.

Recall

• A single layer draws a straight line, so XOR is unsolvable without depth — and depth is worthless without a non-linear activation, because stacked linear layers collapse into one.
• ReLU made deep learning practical: derivative exactly 1 for positive inputs, so gradients survive many layers, and it is cheap. Sigmoid's derivative peaks at 0.25, and chaining ten gives 10^{-6} — the vanishing gradient. Dying ReLU is the cost; leaky ReLU and GELU are the refinements.
• A hidden layer builds new features that make the problem separable — in the worked XOR, one neuron becomes OR, the other AND, and the output computes OR minus twice AND.
• Backpropagation is the chain rule applied backwards: each node needs only its own operation and the gradient arriving from above. The trace moved a loss of 18 to 0.06 in one step.
• Forward activations must be kept for the backward pass, which is why training uses far more memory than inference — and why gradient checkpointing trades recomputation for memory.
• Mini-batches (32–512) are standard because a batch is a matrix multiply. Adam/AdamW is the default optimiser, and large models need a learning-rate warmup or they fail outright.
• Gradient clipping prevents explosion in one line. Never initialise weights to zero — every neuron in the layer stays identical forever.
• If the loss will not move, try to overfit ten examples. Failure there means a bug, not a hyperparameter.

Self-test: Why does stacking linear layers gain nothing? · What exactly does the 0.25 in sigmoid's derivative cause? · In the XOR walk-through, what did each hidden neuron represent? · Why does training need more memory than inference? · What breaks if all weights start at zero? · What is the first thing to try when the loss will not decrease?