Appearance
12.5.1 — The Neuron and Activation Functions
Cartwise lets sellers upload their own product photos, and about one photo in twelve is a problem: blurry, the wrong product, a screenshot of somebody else's listing, or a photo with a phone number painted across it. The catalogue team wants a model that flags bad photos before they go live.
None of the models in Chapter 12.3 can do this well, and the reason is worth stating precisely. Those models work on features: meaningful columns that somebody computed, like order amount or account age. A photo has no such columns. It is a grid of about 150,000 numbers, the red, green and blue brightness of every pixel in a 224 by 224 image. No single pixel means anything. "Is this photo blurry?" depends on patterns spread across thousands of pixels at once, and nobody knows how to write those patterns down as features by hand. People tried for decades.
Deep learning's answer is to let the model build its own features, layer by layer, out of a very simple repeated unit. This page builds that unit, the artificial neuron, then shows the one ingredient that makes stacking neurons worthwhile, the activation function, using the smallest problem that needs both.
1. The neuron: multiply, add, bend
An artificial neuron does three things, and you already know the first two.
z = w_1 x_1 + w_2 x_2 + \cdots + w_n x_n + b = \mathbf{w} \cdot \mathbf{x} + b, \qquad a = f(z)
Read it as "z equals the dot product of the weights with the inputs, plus the bias; a equals f of z". Steps ① and ② are exactly the weighted sum of logistic regression from 12.3.1, and the bias b shifts the result up or down so the neuron can respond even when its inputs are small. Step ③ passes z through a fixed function f, called the activation function, and its output a is called the neuron's activation. If f is the sigmoid, this single neuron is logistic regression.
The name comes from biology. In 1943, the neuroscientist Warren McCulloch and the logician Walter Pitts proposed a simplified mathematical model of a brain cell: it collects signals from other cells, and if the total crosses a threshold, it "fires". Real neurons are far more complicated than this, and modern artificial neurons owe little to biology beyond the name and the rough idea. It is best to think of one as what it is: a weighted sum followed by a bend.
A layer is many neurons; a network is many layers
Put many neurons side by side, all looking at the same inputs, each with its own weights, and you have a layer. As 12.2.2 showed, a whole layer is one matrix multiplication: stack each neuron's weights as a row of a matrix W, and the layer computes \mathbf{a} = f(W\mathbf{x} + \mathbf{b}), where f is applied to each entry separately.
Feed one layer's outputs into the next, and you have a neural network. The most basic kind, where every neuron in one layer connects to every neuron in the next, is called a multilayer perceptron, or MLP, and its layers are called fully connected or dense layers. "Perceptron" is the name of Frank Rosenblatt's 1958 single-layer learning machine from 12.1.1. The layers between the input and the output are called hidden layers, because nobody tells them what to compute: their job is decided entirely by training.
2. Why one layer is not enough: exclusive-or
Here is the smallest problem that a single neuron cannot solve. It is called "exclusive or", or XOR: the output is 1 when exactly one of two inputs is 1.
| x_1 | x_2 | output |
|---|---|---|
| 0 | 0 | 0 |
| 0 | 1 | 1 |
| 1 | 0 | 1 |
| 1 | 1 | 0 |
This is not a toy. It is the shape of the fraud pocket from 12.3.4: "a small order is suspicious at night, a large order is suspicious in the day". There, an order is risky exactly when "small" and "night" are both true or both false, which is XOR turned upside down, and just as impossible for one straight line. Plot the four points and try to draw one straight line with the two 1s on one side and the two 0s on the other. You cannot: the 1s sit on opposite corners. A single neuron computes a weighted sum and then thresholds it, and a weighted sum compared with a threshold is always a straight-line boundary, so no choice of weights works. That was the result in Marvin Minsky and Seymour Papert's 1969 book that deflated the first wave of neural network research.
Two layers solve it
Add one hidden layer with two neurons, and use the simplest useful activation, the ReLU, which keeps positive numbers and turns negative ones into zero: \text{ReLU}(z) = \max(0, z). Here are weights that solve XOR. They are chosen by hand so every number can be checked; training would find something similar.
text
hidden neuron 1: w = [1, 1], b = 0 → h1 = ReLU(x1 + x2) "how many inputs are on"
hidden neuron 2: w = [1, 1], b = −1 → h2 = ReLU(x1 + x2 − 1) "on only if both are on"
output neuron: w = [2, −4], b = −1 → z = 2·h1 − 4·h2 − 1, output = sigmoid(z)Run all four inputs through it:
| Input | h_1 | h_2 | z = 2h_1 - 4h_2 - 1 | sigmoid(z) | Target |
|---|---|---|---|---|---|
| (0, 0) | 0 | 0 | −1 | 0.27 | 0 |
| (0, 1) | 1 | 0 | +1 | 0.73 | 1 |
| (1, 0) | 1 | 0 | +1 | 0.73 | 1 |
| (1, 1) | 2 | 1 | −1 | 0.27 | 0 |
Every input lands on the right side of 0.5. Now look at what the hidden layer actually did, because it is the central idea of deep learning:
On the left, the problem is impossible for a straight line. On the right, the hidden layer has moved the four points to new positions: both red points land on exactly the same spot, (1, 0), and the blue ones end up on either side of them. Now the output neuron's straight-line cut works. The hidden layer built new features that turned an impossible problem into an easy one. Neuron 1 learned something like "count the inputs that are on", neuron 2 learned "are both on", and the output computed "two times the count, minus four times both, minus one", which is high only for exactly one.
Nobody programmed those features. With random starting weights and training (the subject of 12.5.2), a network finds intermediate features like these by itself, because they reduce the loss. Scale the same idea up: in a photo network, the first layers end up detecting edges, the next layers combine edges into textures and corners, and later layers combine those into shapes like a phone's outline or a printed phone number. That is the "learns its own features" promise from 12.1.1, seen in four rows. It is also the matrix geometry of 12.2.2: each layer moves the data to new positions until a straight cut can separate it.
Here is the same network in PyTorch, the library most deep learning is written in:
python
import torch
import torch.nn as nn
net = nn.Sequential(
nn.Linear(2, 2), # (1) hidden layer: 2 inputs → 2 neurons
nn.ReLU(), # (2) the bend
nn.Linear(2, 1), # (3) output neuron: 2 hidden values → 1 score
nn.Sigmoid(),
)
with torch.no_grad(): # (4)
net[0].weight.copy_(torch.tensor([[1., 1.], [1., 1.]]))
net[0].bias.copy_(torch.tensor([0., -1.]))
net[2].weight.copy_(torch.tensor([[2., -4.]]))
net[2].bias.copy_(torch.tensor([-1.]))
X = torch.tensor([[0., 0.], [0., 1.], [1., 0.], [1., 1.]])
print(net(X).round(decimals=4)) # (5) → tensor([[0.2689], [0.7311], [0.7311], [0.2689]])Line (1) creates a fully connected layer: nn.Linear(2, 2) holds a 2 by 2 weight matrix and 2 biases, and computes W\mathbf{x} + \mathbf{b}. Line (2) applies ReLU to each of its outputs. Line (3) is the output neuron, and nn.Sigmoid() squashes its score into a probability. nn.Sequential simply chains the pieces in order. Line (4) overwrites the random starting weights with the hand-chosen ones; torch.no_grad() tells PyTorch not to record these assignments for training, which section 12.5.2 explains. Line (5) runs all four inputs as one batch and prints the same probabilities as the table.
3. Why the bend is everything
Here is the most important fact about activation functions. Suppose you left them out, and stacked two plain linear layers: the first computes W_1 \mathbf{x} and the second computes W_2(W_1 \mathbf{x}). Matrix multiplication can be regrouped, so that is the same as (W_2 W_1)\mathbf{x}, and W_2 W_1 is just another single matrix. Two linear layers are exactly one linear layer in disguise. So are a hundred. Without a bend between them, depth adds nothing: the network can still only draw straight lines, and XOR stays unsolvable.
The activation function is the bend that stops the layers from collapsing into one. Because it is not a straight line, the composition of layers is not a straight line either, and each new layer can bend the space further. In fact, a network with a single hidden layer and enough neurons can approximate any reasonable function as closely as you like, a result called the universal approximation theorem, proved for the sigmoid by George Cybenko in 1989. In practice, deep networks with many moderate layers reach good approximations with far fewer neurons than one enormous shallow layer, because each layer can reuse the features built by the one before.
4. The activation functions you will meet
Sigmoid, \sigma(z) = 1/(1 + e^{-z}), from 12.2.6. It was the standard activation for decades because it looks like a smooth version of "fire or don't fire". Today it is used only at the very output of a yes-or-no classifier, and the right panel shows why. Its slope is \sigma(z)(1 - \sigma(z)), which is largest at z = 0, where it equals 0.5 \times 0.5 = 0.25, and drops towards zero for large inputs in either direction. A flat region where the slope is nearly zero is called saturation.
Here is why that matters. When a network learns, the error signal travels backwards through every layer, and as the chain rule from 12.2.3 says, it gets multiplied by each layer's slope on the way. With sigmoids, each multiplication is by at most 0.25. After ten layers, the signal is at most 0.25^{10} \approx 0.000001 of its original size. The early layers receive almost no information about how to improve and effectively stop learning. This is the vanishing gradient problem, and it is a large part of why deep networks were considered impractical through the 1990s and early 2000s.
Tanh, the hyperbolic tangent, has the same S-shape but runs from −1 to +1 and is centred on zero, which helps training somewhat. Its slope peaks at 1 instead of 0.25, but it still saturates on both sides.
ReLU, the rectified linear unit, \max(0, z). ("Rectified" is borrowed from electronics, where a rectifier lets current through in one direction only.) This almost embarrassingly simple function is one of the changes that made deep learning practical, popularised for deep networks by Vinod Nair and Geoffrey Hinton in 2010 and by the AlexNet model in 2012. The reasons:
- Its slope is exactly 1 for every positive input. The error signal passes back through active neurons without shrinking at all, so gradients survive many layers.
- It is cheap. A comparison with zero, instead of computing an exponential.
- It produces exact zeros. For any given input, many neurons output nothing, which makes the network's representation sparse and efficient.
ReLU has one failure of its own. A neuron whose weighted sum is negative for every input in the data always outputs zero, so its slope is always zero, so it never receives any learning signal, so it can never change. It is permanently switched off. This is called a dying ReLU, usually caused by a learning rate so high that one big update pushes a neuron's weights far into the negative. Leaky ReLU, \max(0.01z, z), keeps a small slope for negative inputs so no neuron can die completely.
GELU (Gaussian error linear unit) and SiLU (also called Swish) are smooth versions of ReLU: nearly zero for very negative inputs, nearly the input itself for positive ones, with a gentle curve and a slight dip just below zero in between. The dashed amber curve on the left is GELU. They train slightly better than ReLU in large models, and GELU and SiLU are what transformers, the networks behind every LLM, use in their hidden layers, as Chapter 12.7.4 shows.
The output layer is different
Hidden layers use ReLU or one of its smooth cousins. The output layer's activation is chosen by the task, because it must produce the right kind of answer:
| Task | Output activation | Loss |
|---|---|---|
| Regression (delivery days) | none | mean squared error |
| Yes or no (bad photo?) | sigmoid | binary cross-entropy |
| One of many (which team?) | softmax | cross-entropy |
| Several tags at once | a sigmoid per tag | binary cross-entropy per tag |
These are exactly the pairings from Chapters 12.2.6 and 12.3.1. A neural network is a stack of learned feature builders with one of the familiar classical models sitting on top.
5. Sizing a network: how many layers and neurons?
The honest answer, which surprises people, is that nobody calculates it. There is no formula that turns a dataset into "three layers of 128 neurons". The practice is empirical, and it follows a pattern:
- Start from an architecture known to work for your kind of data. For photos, take a standard published image network, as Chapter 12.5.5 does. For text, take a pre-trained transformer. For tables, start with gradient-boosted trees from 12.3.4, and if you do use a network, one to three hidden layers is typical.
- Make it big enough to overfit. If training loss stays high, the network lacks capacity: widen it (more neurons per layer) or deepen it (more layers).
- Then regularise it back with the tools of Chapter 12.5.4 until validation loss is as low as it goes.
Widths are usually powers of two, such as 64, 128, 256 or 512, not for any deep reason but because hardware processes data in blocks of those sizes, so they waste less of it.
Recall
- A neuron computes a = f(\mathbf{w}\cdot\mathbf{x} + b): multiply, add (with a bias), bend. A layer is many neurons, one matrix multiply; a multilayer perceptron stacks fully connected layers, with hidden layers in between.
- One neuron draws one straight line, so XOR is impossible. With one hidden layer, both positive cases land on the same point and a single line separates them: hidden layers build new features.
- Without an activation function, stacked linear layers collapse into one. The bend is what makes depth useful; one wide hidden layer can approximate any function in principle.
- Sigmoid's slope is at most 0.25, so ten layers shrink gradients to about a millionth: the vanishing gradient. ReLU's slope is exactly 1 for positive inputs, which made deep networks trainable; its failure is the dying ReLU. GELU and SiLU are the smooth versions used in transformers.
- The output activation matches the task: none, sigmoid or softmax. Network size is found empirically: start from a known architecture, grow until it overfits, then regularise.
Self-test: Why are two linear layers with no activation equivalent to one? Using the XOR network's weights, what does input (1, 1) give if the second hidden neuron's bias were 0 instead of −1? Why does a sigmoid's 0.25 maximum slope make deep sigmoid networks hard to train?
Next: 12.5.2 Backpropagation by Hand — the XOR weights here were chosen by hand; next we see how a network finds such weights by itself, working out the blame for an error one layer at a time.