Appearance
4.1 — Vectors
A plane's instruments report a speed of 900 km/h. That is not enough to fly it. You also need to know which way it is going, and you need to combine it with a 60 km/h wind coming from the northwest to work out where the aircraft actually ends up.
Speed is a single number. Velocity is a number with a direction, and adding two of them is not ordinary addition. The object that carries both is a vector, and once you have it, a very large amount of mathematics becomes bookkeeping.
1. Three ways to see the same thing
A vector can be understood in three ways, and holding all three at once is the whole skill.
As an arrow. A quantity with size and direction, drawn as an arrow. Its length is the size, its heading is the direction, and — importantly — where you draw it does not matter. Two arrows of the same length and direction are the same vector. A vector is a displacement, not a position.
As a list of numbers. In coordinates, the arrow from the origin to (3, 4) is written
\mathbf{v} = \begin{bmatrix} 3 \\ 4 \end{bmatrix} \quad\text{or}\quad (3, 4)
The numbers are the components: how far along each axis. This is the form a computer stores, and it is just an array.
As a point in a space. A list of n numbers is a point in n-dimensional space, and every operation below works the same for n = 2 or n = 768. That you cannot picture 768 dimensions does not matter in the slightest, because the algebra never needed the picture.
The third view is the one that makes linear algebra powerful. A customer is a list of numbers (age, purchases, days since last visit). A pixel is a list of three numbers. A sentence, after the machinery of Volume I, 12.5, is a list of several hundred numbers. All of them are vectors, and everything in this Part applies to all of them equally.
The notation varies: bold \mathbf{v} in print, \vec{v} with an arrow when handwritten. The size, or magnitude, is written \|\mathbf{v}\| or |\mathbf{v}|.
2. Adding vectors
Add component by component:
\begin{bmatrix} 3 \\ 4 \end{bmatrix} + \begin{bmatrix} 1 \\ 2 \end{bmatrix} = \begin{bmatrix} 4 \\ 6 \end{bmatrix}
Geometrically, this is "walk the first, then walk the second from where you landed" — put the tail of the second arrow at the head of the first, and the sum is the arrow from the original start to the final end. This is the triangle rule. Equivalently, draw both from a common start and complete the parallelogram; the diagonal is the sum.
The aircraft problem. Heading east at 900 km/h is \begin{bmatrix} 900 \\ 0\end{bmatrix}. A wind from the northwest blowing towards the southeast at 60 km/h is about \begin{bmatrix} 42 \\ -42 \end{bmatrix}. The actual motion over the ground is the sum, \begin{bmatrix} 942 \\ -42\end{bmatrix} — a ground speed of \sqrt{942^2 + 42^2} = 943 km/h, drifting 2.6° south of the intended heading. Over four hours that is 110 km off target, which is why pilots correct for it.
Scalar multiplication stretches a vector without turning it:
3\begin{bmatrix} 2 \\ 1\end{bmatrix} = \begin{bmatrix} 6 \\ 3 \end{bmatrix}
A scalar is just an ordinary number, so called because it scales. Multiplying by a negative flips the direction as well: -1 times a vector points the opposite way, same length.
Subtraction is adding the negative, exactly as in Chapter 1.2. And \mathbf{b} - \mathbf{a} has a useful reading: it is the arrow from \mathbf{a} to \mathbf{b}, which is how you get a direction between two known points.
3. Length, and the unit vector
By Pythagoras from Chapter 3.4:
\|\mathbf{v}\| = \sqrt{v_1^2 + v_2^2 + \cdots + v_n^2}
The formula is written once and works in every dimension. In three dimensions it is the same with one more term; in 768 dimensions it is the same with 766 more.
A unit vector has length 1. To make one from any vector, divide by its own length:
\hat{\mathbf{v}} = \frac{\mathbf{v}}{\|\mathbf{v}\|}
This is called normalising, and it strips out the size to leave pure direction. For \begin{bmatrix}3\\4\end{bmatrix}, whose length is 5, the unit vector is \begin{bmatrix}0.6\\0.8\end{bmatrix}.
This is done constantly in practice. A game needs the direction from the player to an enemy but not the distance, so it normalises. A search engine compares documents by direction rather than by length, so that a long document is not automatically more similar to everything. Whenever you want to compare orientation without magnitude, you normalise first.
4. The dot product
Two vectors can be multiplied in a way that produces a single number:
\mathbf{a} \cdot \mathbf{b} = a_1b_1 + a_2b_2 + \cdots + a_nb_n
Multiply matching components, add them all up. For (3,1,4) and (2,0,1): 6 + 0 + 4 = 10.
That is the arithmetic. Here is the meaning, and it is the reason the dot product matters more than any other operation in this Part:
\mathbf{a} \cdot \mathbf{b} = \|\mathbf{a}\|\,\|\mathbf{b}\|\cos\theta
where \theta is the angle between them. So the dot product measures how much two vectors point the same way, scaled by their lengths.
Read off the three cases:
- Positive — the angle is under 90°, they broadly agree.
- Zero — they are at right angles. \cos 90° = 0.
- Negative — the angle is over 90°, they broadly oppose.
The zero case is the useful one. Testing whether two vectors are perpendicular becomes a single sum with no trigonometry at all. (3,4)\cdot(-4,3) = -12 + 12 = 0: perpendicular, confirmed by arithmetic. Perpendicular vectors are called orthogonal, and orthogonality is one of the central ideas of the whole subject — Chapter 9.1 shows that Fourier's decomposition of a signal works precisely because the sine waves are orthogonal to each other.
Cosine similarity rearranges the formula to isolate the angle:
\cos\theta = \frac{\mathbf{a}\cdot\mathbf{b}}{\|\mathbf{a}\|\,\|\mathbf{b}\|}
This runs from -1 (opposite) through 0 (unrelated) to 1 (identical direction). It is how a recommendation system decides two users have similar taste, how a search engine ranks documents against a query, and how a language model finds related meanings. Volume I, 12.6.2 covers vector search, and this formula is the whole of it.
Also, note that \mathbf{v}\cdot\mathbf{v} = \|\mathbf{v}\|^2 — a vector dotted with itself gives its length squared, since the angle is zero and the cosine is one. Length is a special case of the dot product.
Physical meaning: work. In physics, work done is force times distance in the direction of motion. Push a box at an angle and only the component along the floor does anything. That is exactly W = \mathbf{F}\cdot\mathbf{d}, and the dot product's projection behaviour is precisely what physics needed.
5. The cross product, in three dimensions only
There is a second product that returns a vector rather than a number, and it exists only in three dimensions.
\mathbf{a} \times \mathbf{b} = \begin{bmatrix} a_2b_3 - a_3b_2 \\ a_3b_1 - a_1b_3 \\ a_1b_2 - a_2b_1 \end{bmatrix}
The formula is unmemorable and the properties are what matter:
- The result is perpendicular to both inputs.
- Its length is \|\mathbf{a}\|\|\mathbf{b}\|\sin\theta, which is the area of the parallelogram the two vectors span.
- It is anti-commutative: \mathbf{a}\times\mathbf{b} = -(\mathbf{b}\times\mathbf{a}). Swapping the order flips the result. Which of the two perpendicular directions you get is fixed by the right-hand rule — point your fingers along \mathbf{a}, curl them towards \mathbf{b}, and your thumb points along the product.
- It is zero when the vectors are parallel, since \sin 0 = 0.
Where it is used. Every 3D graphics engine computes surface normals — the direction a triangle faces — with a cross product of two of its edges, and then dots that normal with the light direction to decide brightness. Physics uses it for torque (\boldsymbol\tau = \mathbf{r}\times\mathbf{F}, which is why a longer spanner turns a bolt more easily) and for the magnetic force on a moving charge.
A quick contrast worth holding: the dot product measures alignment and returns a number; the cross product measures spread and returns a perpendicular vector.
6. Linear combinations, span and independence
Three ideas that sound abstract and are the actual content of the subject.
A linear combination of vectors is any sum of scaled copies:
c_1\mathbf{v}_1 + c_2\mathbf{v}_2 + \cdots + c_k\mathbf{v}_k
That is all you are allowed to do to vectors — scale them and add them — which is why the subject is called linear algebra.
The span of a set of vectors is everything you can reach by linear combination. In two dimensions:
- One nonzero vector spans a line through the origin.
- Two vectors pointing in different directions span the whole plane — any point is reachable.
- Two vectors pointing along the same line span only that line, because scaling one gives you nothing the other did not already give.
A set is linearly independent when no vector in it is a combination of the others — nothing is redundant. Formally: the only way to combine them and land back at the origin is to take every coefficient zero.
(1,0) \text{ and } (0,1): \text{ independent}
(1,2) \text{ and } (2,4): \text{ dependent, since the second is twice the first}
A basis is an independent set that spans the whole space. It is a coordinate system: a minimal set of directions from which everything can be built, in exactly one way. The standard basis of the plane is (1,0) and (0,1), but infinitely many other bases exist, and choosing a good one is often the whole solution to a problem — which is what Chapter 4.6 is about.
The dimension of a space is the number of vectors in any basis for it, and it is the same number whichever basis you pick. That is a theorem, and it is what makes "dimension" a meaningful word.
Why this matters in practice. If your dataset has 50 columns but they are linearly dependent — say, height in centimetres and height in inches, or three columns that always sum to 100 — then the real dimension is lower than 50. The redundant columns carry no new information, and they make everything downstream unstable. Chapter 4.6's principal component analysis finds the real dimension automatically.
Every formula above, built from scratch
None of the results in this chapter are worth memorising, because each one can be rebuilt in under a minute from something simpler. What follows is that rebuilding, one result at a time, so the formula and the reason for it sit on the same page as the explanation that needed them.
Vectors
\mathbf{v} = \begin{pmatrix}v_1\\v_2\\\vdots\\v_n\end{pmatrix}, \qquad \mathbf{u}+\mathbf{v} = \begin{pmatrix}u_1+v_1\\ \vdots \\ u_n+v_n\end{pmatrix}, \qquad c\mathbf{v} = \begin{pmatrix}cv_1\\ \vdots \\ cv_n\end{pmatrix}
Addition works coordinate by coordinate because moving 3 east then 2 east puts you 5 east, and the east-west bookkeeping never interacts with the north-south bookkeeping.
Length
\|\mathbf{v}\| = \sqrt{v_1^2 + v_2^2 + \cdots + v_n^2}
Where it comes from. In two dimensions this is Pythagoras with legs v_1 and v_2. In three, apply Pythagoras twice: first to get the length of the shadow on the floor, \sqrt{v_1^2+v_2^2}, then to combine that shadow with the height v_3, giving \sqrt{\left(\sqrt{v_1^2+v_2^2}\right)^2 + v_3^2} = \sqrt{v_1^2+v_2^2+v_3^2}. In n dimensions the same argument runs n-1 times. There is no new idea after the second step, which is why higher dimensions are less frightening than they sound.
The dot product
\mathbf{u}\cdot\mathbf{v} = u_1v_1 + u_2v_2 + \cdots + u_nv_n = \|\mathbf{u}\|\,\|\mathbf{v}\|\cos\theta
Read it aloud. "u dot v is the sum of the matching coordinates multiplied together, and it also equals the length of u, times the length of v, times the cosine of the angle between them."
Why those two expressions are the same thing. Put the two vectors tail to tail and join their tips. The joining vector is \mathbf{u}-\mathbf{v}, and the three vectors form a triangle. Apply the law of cosines from 3.2 — triangles and pythagoras to it:
\|\mathbf{u}-\mathbf{v}\|^2 = \|\mathbf{u}\|^2 + \|\mathbf{v}\|^2 - 2\|\mathbf{u}\|\|\mathbf{v}\|\cos\theta
Now expand the left-hand side using coordinates. In two dimensions:
\|\mathbf{u}-\mathbf{v}\|^2 = (u_1-v_1)^2 + (u_2-v_2)^2 = u_1^2 - 2u_1v_1 + v_1^2 + u_2^2 - 2u_2v_2+v_2^2
Group the squares: u_1^2+u_2^2 = \|\mathbf{u}\|^2 and v_1^2+v_2^2 = \|\mathbf{v}\|^2. So
\|\mathbf{u}-\mathbf{v}\|^2 = \|\mathbf{u}\|^2+\|\mathbf{v}\|^2 - 2(u_1v_1+u_2v_2)
Compare the two expressions for the same quantity. The first two terms match, so the last terms must match too:
u_1v_1+u_2v_2 = \|\mathbf{u}\|\|\mathbf{v}\|\cos\theta
That is the whole derivation, and it explains why a formula about multiplying coordinates knows anything about angles: the law of cosines is the bridge.
The consequences, all immediate.
\cos\theta = \frac{\mathbf{u}\cdot\mathbf{v}}{\|\mathbf{u}\|\|\mathbf{v}\|}, \qquad \mathbf{u}\perp\mathbf{v} \iff \mathbf{u}\cdot\mathbf{v} = 0, \qquad \mathbf{v}\cdot\mathbf{v} = \|\mathbf{v}\|^2
Perpendicular means \theta = 90°, and \cos 90° = 0, so the product is zero. This is the test that runs everywhere: zero dot product means at right angles, in two dimensions or in two thousand.
Projection
\text{proj}_{\mathbf{u}}\mathbf{v} = \frac{\mathbf{u}\cdot\mathbf{v}}{\|\mathbf{u}\|^2}\,\mathbf{u}
Read it aloud. "The projection of v onto u is u, scaled by the dot product of u and v divided by u's length squared."
Where it comes from. The shadow of \mathbf{v} on the line through \mathbf{u} must point along \mathbf{u}, so it is c\mathbf{u} for some number c. What must c be? The leftover piece \mathbf{v} - c\mathbf{u} has to be perpendicular to \mathbf{u} — that is what "shadow" means. So its dot product with \mathbf{u} is zero:
(\mathbf{v}-c\mathbf{u})\cdot\mathbf{u} = 0 \quad \Rightarrow \quad \mathbf{v}\cdot\mathbf{u} - c(\mathbf{u}\cdot\mathbf{u}) = 0 \quad \Rightarrow \quad c = \frac{\mathbf{u}\cdot\mathbf{v}}{\|\mathbf{u}\|^2}
This single formula is the engine of least-squares fitting, of Gram–Schmidt, and of every recommendation system that measures how much of your taste points along someone else's.
The cross product (three dimensions only)
\mathbf{u}\times\mathbf{v} = \begin{pmatrix}u_2v_3-u_3v_2\\ u_3v_1-u_1v_3\\ u_1v_2-u_2v_1\end{pmatrix}, \qquad \|\mathbf{u}\times\mathbf{v}\| = \|\mathbf{u}\|\|\mathbf{v}\|\sin\theta
The result is a vector perpendicular to both inputs, whose length is the area of the parallelogram they span. The easy way to remember the components is as a symbolic determinant:
\mathbf{u}\times\mathbf{v} = \begin{vmatrix} \mathbf{i} & \mathbf{j} & \mathbf{k} \\ u_1 & u_2 & u_3 \\ v_1 & v_2 & v_3\end{vmatrix}
Why the length is an area. The parallelogram with sides \mathbf{u} and \mathbf{v} has base \|\mathbf{u}\| and height \|\mathbf{v}\|\sin\theta, since the height is the part of \mathbf{v} perpendicular to \mathbf{u}. Multiply.
\mathbf{u}\times\mathbf{v} = -\,\mathbf{v}\times\mathbf{u}
Order matters, and swapping the inputs reverses the direction. This is why the cross product needs a hand rule and the dot product does not.
7. Where this shows up in your life
Every game and every animation. Positions, velocities, directions to targets, surface normals, camera orientation.
Every physics calculation. Force, velocity, acceleration, momentum and field strength are all vectors, and the whole of Volume IV, Part 1 runs on this chapter.
Every recommendation and search result. You are a vector of preferences; a film is a vector of attributes; the recommendation is a dot product.
Every AI model. Text and images become vectors — hundreds or thousands of numbers — and similarity is cosine similarity. Volume I, 12.5 and 12.6 assume exactly this chapter.
Every navigation correction. The aircraft-and-wind problem of Section 2 is done by every autopilot continuously, and by every sailor since long before vectors were named.
Every colour on your screen. A colour is a three-component vector, and blending two colours is a linear combination.
We can now describe points and directions. What we cannot yet do is describe a transformation — a rotation, a stretch, a projection — as an object we can compute with. That object is the matrix, and it is the heart of the Part.