Appearance
12.5.5 — Convolutional Networks
Try to build Cartwise's photo checker out of the fully connected layers of 12.5.1, and the arithmetic stops you. A 224 by 224 colour photo is 224 \times 224 \times 3 = 150{,}528 numbers. A first hidden layer of 1,000 neurons, each connected to every input, needs 150{,}528 \times 1{,}000 \approx 150 million weights. That is one layer, before the network has learned anything, and a model that size would need tens of millions of labelled photos to avoid memorising them.
The waste comes from the network ignoring two facts that are true of every photo:
- Nearby pixels belong together. An edge, a letter, the corner of a box: all are made of pixels next to each other. A pixel in the top-left corner has almost nothing to do with one in the bottom-right.
- A pattern means the same thing wherever it appears. A phone number painted across a photo is a phone number whether it is at the top or the bottom. A detector for it learned in one place should work everywhere.
A fully connected layer knows neither fact and has to learn both from data, separately, for every position. A convolutional neural network, or CNN, builds both facts into its structure, and that is what made computers able to see.
1. A convolution: one small pattern detector, slid everywhere
A convolution takes a small grid of weights, called a kernel or filter, typically 3 by 3, and slides it across the image. At each position it computes the dot product from 12.2.1 between the kernel and the 3 by 3 patch of pixels underneath it: multiply each pair, add them all up. That one number goes into the output at the matching position. Slide one pixel along and repeat.
Here is a kernel that detects vertical edges, applied to one patch of a greyscale photo where the left side is dark and the right side is bright (values are brightness, 0 for black to 255 for white):
text
kernel image patch dot product
[ 1 0 -1 ] [ 10 10 40 ]
[ 1 0 -1 ] · [ 12 11 45 ] = (10 + 12 + 9) − (40 + 45 + 42) = 31 − 127 = −96
[ 1 0 -1 ] [ 9 10 42 ]The kernel adds up the left column, ignores the middle column, and subtracts the right column. Where the image is the same brightness on both sides, the two cancel and the output is near zero. Where the left is dark and the right is bright, as here, the output is strongly negative, -96. Where the left is bright and the right dark, it would be strongly positive. So the output is large, in one direction or the other, exactly where there is a vertical edge.
The grid of outputs is called a feature map: a map of where in the image the kernel's pattern appears. A 6 by 6 image and a 3 by 3 kernel give a 4 by 4 map, because the kernel fits in 4 positions across and 4 down.
Now the crucial point. In a CNN, nobody designs the kernels. The nine numbers in each kernel are weights, learned by backpropagation exactly like any other weight in 12.5.2. Training discovers whatever small patterns are useful for the task. A convolutional layer has many kernels, typically 32 to 512, each producing its own feature map, so one layer detects dozens of different patterns at every position at once.
The two ideas that save millions of weights
Look at how the two facts from the opening are built in.
Locality. Each output depends only on a small neighbourhood of pixels, never on the whole image. That is the first fact.
Weight sharing. The same nine weights are used at every position in the image. That is the second fact: an edge detector learned in one corner works everywhere, for free.
The saving is enormous. A 3 by 3 kernel looking at all 3 colour channels has 3 \times 3 \times 3 = 27 weights plus one bias. A layer of 64 such kernels has 64 \times 28 = 1{,}792 weights in total, against 150 million for the fully connected layer, and it is applied to the whole photo. Built-in assumptions like these, which narrow down what a model has to learn, are called inductive bias. "Bias" here does not mean unfairness; it means a built-in preference. CNNs are biased towards local, position-independent patterns, which happens to be exactly true of images, and that is why they need so much less data than a fully connected network.
Channels, stride and padding
Three practical details control a convolutional layer's shape.
Channels. A colour photo has three channels, red, green and blue, so a kernel is really 3 by 3 by 3: it looks at a 3 by 3 patch in all three colours at once and adds everything up. The output of a layer with 64 kernels has 64 channels, one feature map per kernel, and the next layer's kernels are then 3 by 3 by 64. Each channel in a deep layer is one kind of detected pattern.
Stride. How far the kernel moves each time. A stride of 1 visits every position; a stride of 2 skips every other one and halves the size of the output.
Padding. A border of zeros added around the image, so the kernel can be centred on the edge pixels too. With a 3 by 3 kernel, padding of 1 keeps the output the same size as the input.
For an input of width n, kernel size k, padding p and stride s, the output width is
\left\lfloor \frac{n - k + 2p}{s} \right\rfloor + 1
where \lfloor\ \rfloor means "round down". Our example: (6 - 3 + 0)/1 + 1 = 4. A 224-wide photo with a 3 by 3 kernel, padding 1, stride 2: \lfloor(224 - 3 + 2)/2\rfloor + 1 = 111 + 1 = 112, half the width. Checking this formula by hand is how you debug the shape errors from 12.2.2 in a CNN.
Pooling
A pooling layer shrinks a feature map by summarising small blocks of it. Max pooling with a 2 by 2 window keeps the largest value in each block, so a 112 by 112 map becomes 56 by 56. It throws away exactly where inside each 2 by 2 block the pattern was found while keeping whether it was found, which makes the network a little tolerant of small shifts, and it cuts the computation for every later layer. Many modern networks use convolutions with stride 2 instead of pooling, which does the same shrinking with learned weights.
2. Depth builds a hierarchy
Here is why CNNs are stacked many layers deep, and it is the "learns its own features" idea from 12.5.1 in its most visible form.
A first-layer kernel sees a 3 by 3 patch of pixels. A second-layer kernel sees a 3 by 3 patch of first-layer outputs, each of which saw 3 by 3 pixels, so it effectively sees a 5 by 5 area of the photo. With each layer, and especially after each halving by stride or pooling, the area of the original photo that a neuron depends on grows. That area is called its receptive field. Deep in the network, a single neuron's receptive field covers most of the photo.
When researchers visualised what trained networks had learned, most famously Matthew Zeiler and Rob Fergus in 2013, the picture was remarkably consistent. First-layer kernels detect edges at various angles and patches of colour. Middle layers respond to textures and simple shapes built from those edges: grids, curves, corners. Later layers respond to parts: wheels, faces, text. Nobody designed this hierarchy. It emerged from training, because each layer can only build from the layer below, and building complicated things from simpler ones is the efficient way to recognise the world. For the photo checker, later layers end up containing detectors for printed text and phone numbers, for screenshot borders, and for blur, because those are what separate good photos from bad ones.
A typical CNN repeats one pattern: a few convolutions with ReLU, then a halving of the width and height, with more channels at each stage. The photo shrinks in size and grows in depth, from 224 by 224 by 3 to something like 7 by 7 by 512, and a final layer averages each of the 512 maps into one number and feeds the resulting 512-number vector to a classifier. That 512-number vector is a compressed description of the photo, and it is exactly what the stand-in features in 12.5.3 represented.
3. The photo checker, built properly
Cartwise has about 20,000 labelled seller photos. That is far too few to train a deep CNN from random weights; it would memorise them. But it does not need to. Networks trained on millions of general photos, such as the ImageNet collection from 12.1.1, have already learned the edges, textures and parts that every photo is made of. Only the last decision, "bad photo or not", is specific to Cartwise. So the standard approach is transfer learning: take a pre-trained network, replace its final layer, and train on your own photos.
python
import torch, torch.nn as nn
from torchvision import models, transforms
weights = models.ResNet18_Weights.IMAGENET1K_V1
net = models.resnet18(weights=weights) # (1) pre-trained on ImageNet
print(sum(p.numel() for p in net.parameters())) # → 11689512 (about 11.7 million)
for p in net.parameters(): # (2) freeze what it already knows
p.requires_grad = False
net.fc = nn.Linear(net.fc.in_features, 1) # (3) new head: 512 → 1, trainable
train_tf = transforms.Compose([ # (4) augmentation for training
transforms.RandomResizedCrop(224, scale=(0.8, 1.0)),
transforms.RandomHorizontalFlip(),
transforms.ColorJitter(brightness=0.2, contrast=0.2),
transforms.ToTensor(),
transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]), # (5)
])
opt = torch.optim.AdamW(net.fc.parameters(), lr=1e-3) # (6) only the head learnsLine (1) downloads ResNet-18, an 18-layer residual network from 12.5.4, already trained on ImageNet's 1,000 categories. torchvision is PyTorch's companion library for images, holding standard architectures, their pre-trained weights, and image transformations. The printed parameter count is about 11.7 million, nearly all of it in convolutional kernels, which is small for what it does because of weight sharing.
Line (2) freezes every existing weight: requires_grad = False means backpropagation will not compute gradients for them, so they stay exactly as ImageNet training left them. Line (3) replaces the final layer, fc ("fully connected"), which originally mapped the 512-number photo description to 1,000 ImageNet categories, with a new one mapping it to a single "bad photo" logit. The new layer starts random and is trainable.
Line (4) sets up the data augmentation from 12.4.3: every time a photo is used, it is randomly cropped by up to 20 percent, possibly flipped left to right, and nudged in brightness and contrast. A blurry photo stays blurry, and a photo with a phone number painted on it keeps the number, so these changes preserve the label. (A vertical flip is left out deliberately: product photos are never upside down, and a flipped photo of text looks unlike anything the checker will see.) Line (5) standardises each colour channel using the averages and spreads of the ImageNet photos, because that is what the pre-trained network saw during its own training; using different statistics would hand it inputs on a different scale from the ones it learned on. Line (6) gives the optimiser only the new layer's weights.
Train this with the loop from 12.5.3. With frozen layers it trains in minutes, even on an ordinary processor. If it plateaus, the next step is fine-tuning: unfreeze the last block or two of the network and continue training everything unfrozen at a learning rate 10 to 100 times smaller, so the pre-trained features adapt to seller photos without being destroyed. Chapter 12.5.7 returns to why the smaller rate matters.
4. CNNs today
CNNs dominated computer vision from AlexNet in 2012 through the late 2010s, and residual networks like ResNet are still excellent, practical defaults when you have thousands to hundreds of thousands of images. In 2020, researchers at Google showed that the transformer architecture behind LLMs, covered in Chapter 12.7, could also handle images by cutting a photo into small square patches and treating each patch like a word. This vision transformer has weaker built-in assumptions than a CNN: it does not know in advance that nearby pixels belong together, and has to learn that from data. With modest data, that makes it worse than a CNN. With enormous data, the weaker assumptions let it learn patterns a CNN's structure would not allow, and it pulls ahead. That trade between built-in assumptions and data is one of the deepest ideas in the field, and it reappears with sequences on the next page. The image models inside today's multimodal LLMs, which read screenshots and photos, are mostly vision transformers, as Chapter 12.13.1 explains.
Recall
- A fully connected layer on a 224 × 224 photo needs about 150 million weights and ignores what images are like. A convolutional neural network builds in two facts: nearby pixels belong together, and a pattern means the same thing anywhere.
- A convolution slides a small learned kernel across the image, taking a dot product at each position; the result is a feature map of where the pattern appears. The vertical-edge kernel gave −96 on a dark-to-bright patch.
- Locality and weight sharing are inductive bias: 64 kernels of 3 × 3 × 3 cost 1,792 weights. Channels hold one map per kernel; output width is \lfloor (n - k + 2p)/s \rfloor + 1; pooling or stride 2 halves the size.
- Depth grows the receptive field and builds a learned hierarchy: edges, textures, parts, objects.
- With limited data, use transfer learning: a pre-trained ResNet with a new final layer, frozen at first, ImageNet normalisation, label-preserving augmentation, then fine-tuning at a much lower learning rate. Vision transformers win only with very large data.
Self-test: How many weights does a layer of 32 kernels of size 3 × 3 have if its input has 64 channels (ignore biases)? What output width does a 56-wide input give with a 3 × 3 kernel, padding 1, stride 2? Why is a vertical flip a poor augmentation for the photo checker?
Next: 12.5.6 Sequences and the Road to Attention — photos have a fixed shape, but sentences, support emails and speech come as sequences of any length; the networks built for them, and the reason they were replaced, lead straight to the transformer.