Appearance
4.2 — Matrices as Transformations
A matrix looks like a table of numbers, and every course teaches you to multiply them with a rule about rows and columns that seems arbitrary. Do that and the subject stays a chore forever.
Here is the idea that makes it click, and it should come first: a matrix is a function that moves space. Feed it a vector, get back a different vector. Rotations, stretches, shears, reflections, projections — every one of them is a matrix, and matrix multiplication is nothing more than doing one transformation after another.
1. What a matrix is
A matrix is a rectangular grid of numbers. Its dimensions are rows by columns, always in that order:
A = \begin{bmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \end{bmatrix}
is a 2 \times 3 matrix — two rows, three columns. The entry in row i and column j is written a_{ij}, so a_{23} = 6.
Adding matrices is unsurprising: same size, add entry by entry. Multiplying by a scalar multiplies every entry. Neither is interesting.
Multiplication is where the content is.
2. Matrix times vector: the operation everything else follows from
\begin{bmatrix} 2 & 1 \\ 0 & 3 \end{bmatrix}\begin{bmatrix} 4 \\ 5 \end{bmatrix} = \begin{bmatrix} 2(4) + 1(5) \\ 0(4) + 3(5) \end{bmatrix} = \begin{bmatrix} 13 \\ 15 \end{bmatrix}
Mechanically: each entry of the answer is the dot product of one row of the matrix with the vector. That is the rule, and it is worth being able to execute — but the rule is not the idea.
The idea is this: look at what the matrix does to the basis vectors.
Take \mathbf{e}_1 = \begin{bmatrix}1\\0\end{bmatrix} and feed it in:
\begin{bmatrix} 2 & 1 \\ 0 & 3\end{bmatrix}\begin{bmatrix}1\\0\end{bmatrix} = \begin{bmatrix}2\\0\end{bmatrix}
That is the first column of the matrix. Now feed in \mathbf{e}_2 = \begin{bmatrix}0\\1\end{bmatrix}:
\begin{bmatrix} 2 & 1 \\ 0 & 3\end{bmatrix}\begin{bmatrix}0\\1\end{bmatrix} = \begin{bmatrix}1\\3\end{bmatrix}
The second column.
\textbf{The columns of a matrix are where the basis vectors land.}
That single sentence is the whole of this chapter. To know what a matrix does, look at its columns and ask where they send the axes.
And once the basis vectors' destinations are known, everything else follows, because any vector is a combination of the basis vectors and the transformation preserves combinations:
\begin{bmatrix}4\\5\end{bmatrix} = 4\mathbf{e}_1 + 5\mathbf{e}_2 \;\longrightarrow\; 4\begin{bmatrix}2\\0\end{bmatrix} + 5\begin{bmatrix}1\\3\end{bmatrix} = \begin{bmatrix}13\\15\end{bmatrix}
The same answer as the mechanical rule, arrived at by understanding rather than by drill.
Preserving combinations is the definition of "linear". A transformation T is linear when
T(\mathbf{a}+\mathbf{b}) = T(\mathbf{a}) + T(\mathbf{b}) \qquad\text{and}\qquad T(c\mathbf{a}) = c\,T(\mathbf{a})
Geometrically this means grid lines stay straight, stay parallel and stay evenly spaced, and the origin does not move. Every such transformation is a matrix, and every matrix is such a transformation. That correspondence is exact, and it is why matrices are worth studying at all.
3. The transformations you should recognise on sight
Identity — changes nothing. Ones on the diagonal, zeros elsewhere.
I = \begin{bmatrix} 1 & 0 \\ 0 & 1\end{bmatrix}
The basis vectors go where they already were. This is the matrix equivalent of the number 1.
Scaling — stretches each axis.
\begin{bmatrix} 2 & 0 \\ 0 & 3 \end{bmatrix}
doubles horizontally, triples vertically. If both numbers are equal it is a uniform zoom.
Rotation by angle \theta, anticlockwise:
R_\theta = \begin{bmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta\end{bmatrix}
Derive it rather than memorise it. Where does \mathbf{e}_1 = (1,0) go under a rotation by \theta? Onto the unit circle at angle \theta, which is (\cos\theta, \sin\theta) by Chapter 3.5 — and that is the first column. Where does \mathbf{e}_2 = (0,1) go? It starts at 90° and ends at \theta + 90°, which is (-\sin\theta, \cos\theta) — the second column. The matrix writes itself.
Check \theta = 90°: \begin{bmatrix} 0 & -1 \\ 1 & 0\end{bmatrix}. It sends (1,0) to (0,1) and (0,1) to (-1,0) — a quarter turn, exactly as multiplying by i did in Chapter 2.5.
Reflection in the horizontal axis: \begin{bmatrix} 1 & 0 \\ 0 & -1\end{bmatrix}, which leaves x alone and flips y.
Shear — slides one axis along the other:
\begin{bmatrix} 1 & 1 \\ 0 & 1 \end{bmatrix}
leaves \mathbf{e}_1 fixed and sends \mathbf{e}_2 to (1,1). A square becomes a leaning parallelogram. This is what italic type does to letterforms.
Projection onto the horizontal axis: \begin{bmatrix} 1 & 0 \\ 0 & 0\end{bmatrix}, which flattens everything onto a line. Note it destroys information — every point with the same x lands in the same place, and there is no way back. Section 6 and Chapter 4.4 make that precise.
Before combining two matrices, it is worth handling one. Move the four numbers below and watch what happens to the whole plane.
try
A = [1 0; 0 1]det A = 1trace = 2λ₁ = 1λ₂ = 1Av = (1, 0.5)
Try the presets in order. Rotating by 45° keeps every length and angle, and the determinant stays at 1. Stretching x doubles the determinant because it doubles every area. Reflecting in the x-axis gives −1: the same area, mirrored. And 'squash flat' gives zero, which is the case section 6 shows can never be undone.
4. Matrix times matrix is "do one, then the other"
Now the rule that seems arbitrary makes sense.
If A transforms space and B transforms space, then AB means apply B first, then A. The product is the single matrix that does both in one step, and it is called composition.
The order is right to left, which looks backwards and is not: AB\mathbf{v} means A(B\mathbf{v}), so B touches the vector first. It is the same convention as f(g(x)) in ordinary functions.
To find the product, ask the usual question: where do the basis vectors go? B sends them to B's columns, then A transforms those. So each column of AB is A times the corresponding column of B — which unpacks into the familiar row-times-column rule.
\begin{bmatrix} 1 & 2 \\ 3 & 4\end{bmatrix}\begin{bmatrix}5 & 6\\7 & 8\end{bmatrix} = \begin{bmatrix} 1(5)+2(7) & 1(6)+2(8) \\ 3(5)+4(7) & 3(6)+4(8)\end{bmatrix} = \begin{bmatrix} 19 & 22 \\ 43 & 50\end{bmatrix}
Dimensions must match. An (m \times n) matrix times an (n \times p) matrix gives (m \times p). The inner numbers must agree, and they vanish; the outer numbers survive. Read it as "the middle must match", and most shape errors in numerical code fix themselves.
Matrix multiplication is not commutative
AB \neq BA in general, and this is not a technicality — it is a fact about the world.
Take a book. Rotate it 90° about the vertical axis, then 90° about the horizontal axis. Note where the cover faces. Now start again and do the two rotations in the other order. The book ends up somewhere different. Try it; the result is genuinely surprising the first time.
Matrix multiplication is non-commutative because composing transformations is non-commutative. Putting on socks then shoes is not the same as shoes then socks.
It is associative, though: (AB)C = A(BC). Grouping does not matter, order does. That is why a graphics engine can pre-multiply a chain of transformations into one matrix and apply it once — a genuinely large saving when there are a million vertices.
5. Special matrices and the transpose
The transpose A^\mathsf{T} flips a matrix over its diagonal: rows become columns.
\begin{bmatrix} 1 & 2 & 3\\ 4& 5& 6\end{bmatrix}^\mathsf{T} = \begin{bmatrix} 1 & 4 \\ 2 & 5\\ 3& 6\end{bmatrix}
A useful rule, and the reversal in it is not a typo: (AB)^\mathsf{T} = B^\mathsf{T}A^\mathsf{T}.
Names worth knowing:
- Square: same number of rows and columns. Only square matrices can have inverses or eigenvalues.
- Diagonal: nonzero only on the diagonal. Pure scaling, one factor per axis, and trivially easy to multiply or invert.
- Symmetric: A = A^\mathsf{T}. These have exceptionally good properties, covered in Chapter 4.5.
- Orthogonal: A^\mathsf{T}A = I, meaning the transpose is the inverse. Geometrically these are exactly the rotations and reflections — the transformations that preserve all lengths and angles. Because inverting is free, they are the transformations a computer likes best, and it is why 3D graphics keeps its rotations orthogonal.
- Sparse: mostly zeros. Stored and multiplied by special methods, since writing out a million-by-million grid of mostly zeros is absurd. Almost every large real-world matrix — a road network, a social graph, a document-term table — is sparse.
6. What a matrix can destroy
Consider
A = \begin{bmatrix} 1 & 2 \\ 2 & 4 \end{bmatrix}
Its columns are (1,2) and (2,4), and the second is twice the first. They are linearly dependent, in the sense of Chapter 4.1. So both basis vectors land on the same line, and the entire plane is squashed onto a single line.
This transformation cannot be undone. Infinitely many starting points land on the same destination, so given a result there is no way to say where it came from. Such a matrix is called singular, and Chapter 4.4 gives the test for it — the determinant is zero.
Three important quantities describe this:
The column space is the span of the columns — everything the matrix can output. For A above it is a line, not the whole plane.
The rank is the dimension of that column space, meaning the number of genuinely independent columns. Here the rank is 1, not 2. A square matrix with rank equal to its size is full rank and is invertible; anything less loses information.
The null space (or kernel) is everything that gets sent to the origin. For A, the vector (2,-1) gives \begin{bmatrix}1(2)+2(-1)\\2(2)+4(-1)\end{bmatrix} = \begin{bmatrix}0\\0\end{bmatrix}. A nonzero null space is exactly what "information was destroyed" means, and it is why a system of equations can have infinitely many solutions — Chapter 4.3.
Where this bites in practice. A dataset whose columns are dependent (weight in kilograms and weight in pounds, or three percentages that always sum to 100) produces a rank-deficient matrix, and any procedure that needs to invert it will either fail or produce garbage that looks plausible. Statisticians call it multicollinearity, and it is a real and common problem in Chapter 7.8's regression.
7. Matrices in three dimensions and in graphics
Everything above extends. A 3\times3 matrix transforms 3D space, and its three columns are where the three axes land.
There is one problem: translation — sliding everything sideways — is not linear, because it moves the origin. A 3\times3 matrix cannot express "shift right by 5".
The trick used by every graphics system in existence is to work in four dimensions. Represent the 3D point (x,y,z) as the 4D vector (x,y,z,1) — these are homogeneous coordinates — and use 4\times4 matrices:
\begin{bmatrix} 1 & 0 & 0 & t_x \\ 0 & 1 & 0 & t_y \\ 0 & 0 & 1 & t_z \\ 0 & 0 & 0 & 1\end{bmatrix}\begin{bmatrix}x\\y\\z\\1\end{bmatrix} = \begin{bmatrix}x+t_x\\y+t_y\\z+t_z\\1\end{bmatrix}
The extra dimension turns a translation into a shear, which is linear. Now rotation, scaling, translation and perspective projection are all 4\times4 matrices, they all compose by multiplication, and a whole chain collapses into one matrix applied once per vertex.
This is what a GPU does. It multiplies 4\times4 matrices by 4-vectors, millions of times per frame, and it has thousands of cores because every one of those multiplications is independent of the others. Volume I, 12.9 explains why the same hardware turned out to be exactly what neural networks needed — a neural network layer is also a matrix times a vector.
Every formula above, built from scratch
None of the results in this chapter are worth memorising, because each one can be rebuilt in under a minute from something simpler. What follows is that rebuilding, one result at a time, so the formula and the reason for it sit on the same page as the explanation that needed them.
Matrices
(A+B)_{ij} = a_{ij}+b_{ij}, \qquad (cA)_{ij} = c\,a_{ij}
(AB)_{ij} = \sum_{k} a_{ik}b_{kj}
Read it aloud. "Entry i-j of the product is the sum over k of row i of A times column j of B" — that is, the dot product of A's i-th row with B's j-th column.
Why multiplication is defined in that strange way. Because it has to be, if a matrix is going to mean "a transformation". Apply B first, then A. Starting from \mathbf{x}, you get B\mathbf{x}, then A(B\mathbf{x}). For the single matrix AB to do the same job in one step, its entries are forced to be exactly that sum. The definition is not a convention; it is the only definition that makes AB mean "do B, then A".
The consequences.
AB \ne BA \text{ in general}, \qquad (AB)C = A(BC), \qquad (AB)^T = B^TA^T, \qquad (AB)^{-1} = B^{-1}A^{-1}
The reversal in the last two is not a quirk. Putting on socks then shoes is undone by removing shoes then socks. Order reverses when you undo.
The columns are where the basis lands
For a 2\times2 matrix A = \begin{pmatrix}a & b\\ c& d\end{pmatrix}:
A\begin{pmatrix}1\\0\end{pmatrix} = \begin{pmatrix}a\\c\end{pmatrix}, \qquad A\begin{pmatrix}0\\1\end{pmatrix} = \begin{pmatrix}b\\d\end{pmatrix}
The first column is where the arrow \hat{\imath} ends up; the second is where \hat{\jmath} ends up. Every other point follows, because
A\begin{pmatrix}x\\y\end{pmatrix} = x\begin{pmatrix}a\\c\end{pmatrix} + y\begin{pmatrix}b\\d\end{pmatrix}
A matrix does not have to be read as a grid of numbers. It can be read as a picture, and the picture is more useful.
try
A = [1 0; 0 1]det A = 1trace = 2λ₁ = 1λ₂ = 1Av = (1, 0.5)
The standard transformations
\text{rotate by }\theta: \begin{pmatrix}\cos\theta & -\sin\theta\\ \sin\theta & \cos\theta\end{pmatrix}, \qquad \text{scale}: \begin{pmatrix}s_x & 0\\0&s_y\end{pmatrix}, \qquad \text{shear}: \begin{pmatrix}1&k\\0&1\end{pmatrix}
\text{reflect in the } x\text{-axis}: \begin{pmatrix}1&0\\0&-1\end{pmatrix}, \qquad \text{project onto the } x\text{-axis}: \begin{pmatrix}1&0\\0&0\end{pmatrix}
Where the rotation matrix comes from. Build it by asking only where the two basis arrows go. Rotating \hat{\imath} = (1,0) by \theta lands it on the unit circle at angle \theta, which is (\cos\theta, \sin\theta) — that is the first column. Rotating \hat{\jmath} = (0,1), which starts at angle 90°, lands it at angle 90° + \theta, which is (\cos(90°+\theta), \sin(90°+\theta)) = (-\sin\theta, \cos\theta) — the second column. Write those two columns side by side and you have the matrix. No memorisation, and the minus sign is in the right place for a reason you can see.
8. Where this shows up in your life
Every image filter. Blur, sharpen and edge detection apply a small matrix to each neighbourhood of pixels; Chapter 9.4 explains this as convolution.
Every 3D scene you have ever looked at. Model, view and projection matrices, composed and applied per vertex.
Every neural network. A layer is output = activation(W @ input + b) — a matrix multiply plus a vector, then a non-linear function. Everything else is detail.
Every spreadsheet of data. Rows are records, columns are features, and it is a matrix whether or not anyone calls it one.
Every rotation your phone makes when you turn it. The accelerometer readings are a vector; deciding the orientation is a matrix operation.
Every economic input-output model, every population projection, every PageRank computation. All are "multiply this vector by that matrix, repeatedly", and Chapter 4.5 explains what happens when you do it many times.
We can now describe transformations. The oldest use of matrices, though, is not transformation but solving equations — many equations, many unknowns, mechanically. That is next.