12.2.1 — Vectors and the Dot Product

Cartwise wants to show each customer "people like you also bought…". To do that, it first needs to answer a simpler question: how similar are two customers?

Start with something tiny. Describe every customer by just two numbers: how many electronics orders they placed last year, and how many grocery orders. Here are three customers:

Customer	Electronics orders	Grocery orders
Asha	4	3
Bilal	8	6
Chen	1	7

Who is most like Asha? Bilal places twice as many orders of each kind, so in raw numbers he looks quite different. But his taste is identical to Asha's: for every four electronics orders, three grocery orders. Chen is a grocery shopper who rarely buys electronics, so his taste is different even though his total number of orders (8) is closer to Asha's (7) than Bilal's (14) is.

We want a calculation that says "Asha and Bilal are alike, Chen is different", and we want it to work just as well when each customer is described by 300 numbers instead of 2. That calculation is the dot product, and it is the single most used operation in all of machine learning. The fraud score from 12.1.1 is a dot product. Every neuron in a neural network computes one. Search engines rank documents with them. The attention mechanism inside every LLM is billions of them. Learn this one properly and a large part of the rest of this Part becomes familiar.

1. A vector is a list of numbers, and also an arrow

A vector is an ordered list of numbers. Asha is the vector (4, 3). "Ordered" matters: (4, 3) means 4 electronics and 3 groceries, while (3, 4) would be a different customer. Each position in the list is called a dimension, and the number of positions is the vector's dimension count. Asha's vector is 2-dimensional. A customer described by 300 behaviours would be a 300-dimensional vector.

We write vectors in bold, such as \mathbf{a}, and the individual numbers inside with a small number underneath, called a subscript: \mathbf{a} = (a_1, a_2) is read aloud as "vector a equals a-one, a-two". For Asha, a_1 = 4 and a_2 = 3.

That is the list view. The second view is the one that gives you intuition. Draw two axes on paper, electronics going right and groceries going up. Then each customer is a point, and the arrow from the corner where the axes meet (the origin) to that point is the same vector drawn as an arrow.

Three customers as arrows. The direction of each arrow describes the mix of what they buy, and the length describes how much they buy in total. Similar taste means similar direction, whatever the length.

The picture gives the answer to our question immediately. Asha's blue arrow and Bilal's amber arrow lie on top of each other and point in exactly the same direction. Bilal's is simply twice as long. Chen's green arrow points somewhere else, steeply upwards, because nearly all his orders are groceries. So direction captures taste and length captures volume. If what you care about is taste, you want to compare directions and ignore lengths.

This picture is drawn in two dimensions because paper is flat. Nothing about the idea changes in 300 dimensions. You cannot draw it, but the arithmetic is identical, and the same two things still hold: each vector has a direction and a length, and similar things point in similar directions.

2. Adding and scaling vectors

Two operations come up constantly, and both work position by position.

Adding two vectors means adding matching positions. If Asha's household is Asha (4, 3) plus her brother (2, 5), the household vector is (4 + 2,\; 3 + 5) = (6, 8). As arrows, you place the second arrow's tail at the first arrow's tip, and the sum is the arrow from the start to the final tip.

Scaling means multiplying every position by the same number. Bilal is Asha scaled by 2: 2 \times (4, 3) = (8, 6). As an arrow, scaling stretches or shrinks it without changing its direction, which is exactly why Bilal points the same way as Asha. A plain single number like that 2 is called a scalar, because its job is to scale.

3. The length of a vector

How long is Asha's arrow? It goes 4 units right and 3 units up. Those two moves and the arrow itself form a right-angled triangle, so Pythagoras' theorem gives the answer: the long side squared equals the sum of the other two sides squared. That is 4^2 + 3^2 = 16 + 9 = 25, and the square root of 25 is 5. Asha's arrow has length 5.

The length of a vector is called its norm and is written with double bars, \|\mathbf{a}\|, read aloud as "the norm of a". For any number of dimensions, you do the same thing: square every entry, add them all up, take the square root.

\|\mathbf{a}\| = \sqrt{a_1^2 + a_2^2 + \cdots + a_n^2} = \sqrt{\sum_{i=1}^{n} a_i^2}

Read the right-hand side aloud as "the square root of the sum, for i from 1 to n, of a-i squared". The large symbol \sum is the Greek capital letter sigma, standing for "sum". It is simply shorthand for a loop: let a counter i run from 1 to n, compute what is to its right for each value of i, and add the results together. In Python it is sum(a_i**2 for a_i in a).

For our three customers:

• Asha: \sqrt{4^2 + 3^2} = \sqrt{25} = 5
• Bilal: \sqrt{8^2 + 6^2} = \sqrt{100} = 10, exactly twice Asha's, as the picture showed
• Chen: \sqrt{1^2 + 7^2} = \sqrt{50} \approx 7.07

A vector whose length is exactly 1 is called a unit vector. You can turn any vector into a unit vector pointing the same way by dividing every entry by its length. Asha becomes (4/5,\; 3/5) = (0.8, 0.6), and Bilal becomes (8/10,\; 6/10) = (0.8, 0.6), the very same thing. This step is called normalisation, and it is how you throw away the "how much" information and keep only the "what kind" information.

4. The dot product

Here is the operation itself. The dot product of two vectors multiplies matching positions and adds up the results:

\mathbf{a} \cdot \mathbf{b} = a_1 b_1 + a_2 b_2 + \cdots + a_n b_n = \sum_{i=1}^{n} a_i b_i

Read it as "a dot b equals the sum of a-i times b-i". The answer is a single number, not a vector. For our customers:

• Asha · Bilal = 4 \times 8 + 3 \times 6 = 32 + 18 = 50
• Asha · Chen = 4 \times 1 + 3 \times 7 = 4 + 21 = 25
• Bilal · Chen = 8 \times 1 + 6 \times 7 = 8 + 42 = 50

Look at those results and something is off. Bilal · Chen is 50, exactly the same as Asha · Bilal, even though we know Asha and Bilal have identical taste while Bilal and Chen do not. The raw dot product is being inflated by Bilal's sheer volume of orders. To see why, and how to fix it, we need the geometric meaning of the dot product.

What the dot product measures: the angle

The dot product has a second formula, and the two formulas always give the same number:

\mathbf{a} \cdot \mathbf{b} = \|\mathbf{a}\|\;\|\mathbf{b}\|\;\cos\theta

Here \theta (the Greek letter theta) is the angle between the two arrows, and \cos\theta is the cosine of that angle. In words: the dot product equals the length of a, times the length of b, times the cosine of the angle between them.

If cosine is hazy, here is all you need. The cosine of an angle is a number between −1 and 1 that measures how much two directions agree. When two arrows point the same way, the angle is 0° and the cosine is 1. When they are at right angles, the angle is 90° and the cosine is 0: they have nothing in common. When they point in opposite directions, the angle is 180° and the cosine is −1. Between those, it slides smoothly. At 45°, the cosine is about 0.707.

Why are the two formulas equal? It is worth seeing, because then you will never wonder whether the geometric meaning is just a convenient story. Draw arrows \mathbf{a} and \mathbf{b} from the same starting point. The arrow from the tip of \mathbf{b} to the tip of \mathbf{a} is \mathbf{a} - \mathbf{b}, and the three arrows form a triangle. School geometry has a rule for any triangle, called the law of cosines, which is Pythagoras' theorem with a correction for when the angle is not a right angle:

\|\mathbf{a} - \mathbf{b}\|^2 = \|\mathbf{a}\|^2 + \|\mathbf{b}\|^2 - 2\,\|\mathbf{a}\|\,\|\mathbf{b}\|\cos\theta

Now compute the same left-hand side a second way, using only arithmetic. The squared length of any vector is that vector dotted with itself, because \mathbf{v} \cdot \mathbf{v} = v_1^2 + v_2^2 + \cdots, which is exactly the sum of squares inside the norm. So:

\|\mathbf{a} - \mathbf{b}\|^2 = (\mathbf{a} - \mathbf{b}) \cdot (\mathbf{a} - \mathbf{b}) = \mathbf{a}\cdot\mathbf{a} - 2\,\mathbf{a}\cdot\mathbf{b} + \mathbf{b}\cdot\mathbf{b} = \|\mathbf{a}\|^2 - 2\,\mathbf{a}\cdot\mathbf{b} + \|\mathbf{b}\|^2

The expansion in the middle works exactly like multiplying out (x - y)^2 = x^2 - 2xy + y^2 from school algebra. Now set the two right-hand sides equal. The \|\mathbf{a}\|^2 and \|\mathbf{b}\|^2 appear on both sides and cancel, leaving -2\,\mathbf{a}\cdot\mathbf{b} = -2\,\|\mathbf{a}\|\,\|\mathbf{b}\|\cos\theta. Divide both sides by -2 and you have the formula. The arithmetic recipe and the geometric meaning are the same thing. (Volume II's linear algebra chapter goes further into this if you want it.)

So the dot product mixes two things together: how long the arrows are, and how closely they point the same way. That is exactly why Bilal · Chen came out large. Bilal's arrow is long, and the length inflated the score.

The dot product as a shadow

There is one more way to see it, and it is the picture to keep in your head when we get to neural networks.

The dot product as a projection. The amber shadow is how much of b points along a's direction. Multiply that shadow's length by a's length and you have the dot product.

Imagine a light shining straight down onto arrow \mathbf{a}. Arrow \mathbf{b} casts a shadow along \mathbf{a}, shown in amber. The length of that shadow is \|\mathbf{b}\| \cos\theta, which is the part of \mathbf{b} that goes in \mathbf{a}'s direction. Multiply it by \|\mathbf{a}\| and you have the dot product. When \mathbf{b} leans further away from \mathbf{a}, the shadow shortens. When \mathbf{b} stands at a right angle, the shadow vanishes and the dot product is zero. When \mathbf{b} points backwards, the shadow falls behind the start and the dot product goes negative.

The useful way to say it: the dot product measures how much of one vector points along another. When one of the two is a fixed set of weights, as it will be in a neuron, the dot product measures how strongly the input matches the pattern those weights describe.

5. Cosine similarity: comparing direction only

To compare taste without volume, divide the dot product by both lengths. What remains is the cosine of the angle alone:

\text{cosine similarity}(\mathbf{a}, \mathbf{b}) = \frac{\mathbf{a} \cdot \mathbf{b}}{\|\mathbf{a}\|\;\|\mathbf{b}\|} = \cos\theta

This is cosine similarity, and it always lies between −1 and 1. Let us compute it for the three customers using the numbers already worked out above:

• Asha and Bilal: \dfrac{50}{5 \times 10} = 1.0. Identical taste, exactly as the picture showed.
• Asha and Chen: \dfrac{25}{5 \times 7.07} \approx 0.707, which is the cosine of 45°.
• Bilal and Chen: \dfrac{50}{10 \times 7.07} \approx 0.707. The same as Asha and Chen, which is right, because Bilal and Asha have the same taste.

The volume inflation has disappeared, and the answer matches intuition: Asha's nearest neighbour is Bilal. Here is the same thing in NumPy, the Python library that does vector arithmetic quickly:

import numpy as np

asha  = np.array([4, 3])
bilal = np.array([8, 6])
chen  = np.array([1, 7])

print(asha @ bilal, asha @ chen)                 # (1) → 50 25
print(np.linalg.norm(asha), np.linalg.norm(chen))# (2) → 5.0 7.0710678...

def cosine(a, b):
    return (a @ b) / (np.linalg.norm(a) * np.linalg.norm(b))   # (3)

print(cosine(asha, bilal), cosine(asha, chen))   # (4) → 1.0 0.7071067...

unit = lambda v: v / np.linalg.norm(v)           # (5)
print(unit(asha), unit(bilal))                   # → [0.8 0.6] [0.8 0.6]
print(unit(asha) @ unit(chen))                   # (6) → 0.7071067...

Line (1) uses the @ operator, which in NumPy means "dot product" when both sides are plain vectors. It prints the same 50 and 25 we computed by hand. Line (2) computes the norms. np.linalg is NumPy's linear algebra toolbox, and norm is the square root of the sum of squares from section 3. Line (3) defines cosine similarity exactly as the formula above. Line (4) prints 1.0 for Asha and Bilal and 0.707 for Asha and Chen.

Line (5) normalises a vector by dividing it by its own length, and the printout shows that Asha and Bilal become the same unit vector, [0.8 0.6]. Line (6) shows the trick that real systems use: once vectors have been normalised, a plain dot product is cosine similarity, because both lengths are 1 and dividing by 1 changes nothing. Search systems normalise every stored vector once, ahead of time, and then a single dot product per comparison is all they need. You will see this in Chapter 12.6.2.

6. Why machine learning is made of dot products

Now look back at the fraud score from 12.1.1. It multiplied each fact about the order by a weight and added everything up. That is a dot product. Put the facts into one vector and the weights into another:

\mathbf{x} = (\text{amount in thousands},\ \text{new account},\ \text{countries differ}) = (12,\ 1,\ 0)

\mathbf{w} = (0.4,\ 3.0,\ 0.2), \qquad b = -6

Then the score is \mathbf{w} \cdot \mathbf{x} + b = 0.4 \times 12 + 3.0 \times 1 + 0.2 \times 0 - 6 = 4.8 + 3.0 + 0 - 6 = 1.8. The extra number b is called the bias. It is the "starting offset" from 12.1.1, and it shifts the score up or down regardless of the inputs.

Read the shadow picture with this in mind. The weight vector \mathbf{w} describes a direction in the space of all possible orders, the "fraud direction". The dot product measures how far a given order points along that direction. Orders that point along it strongly get a high score. Training, which Chapter 12.2.4 builds, is the process of turning that weight vector until it points in the most useful direction.

This pattern repeats across the whole field:

• A neuron in a neural network computes a dot product of its inputs with its weights, adds a bias, and passes the result through a simple function. Chapter 12.5.1.
• Semantic search turns every product description and every query into a vector, and ranks products by the dot product of the query with each one. Chapter 12.6.
• Attention, the core of every LLM, decides how much each word should pay attention to each other word by computing a dot product between them. Chapter 12.7.3.

A modern LLM performs hundreds of billions of dot products to produce one word of output. The reason graphics chips run these models well is that they can do thousands of multiply-and-add operations at the same moment. The dot product is not a piece of the maths of ML. It is most of it.

Recall

• A vector is an ordered list of numbers and also an arrow. Each position is a dimension. The arrow's direction says what kind and its length says how much.
• The norm \|\mathbf{a}\| = \sqrt{\sum a_i^2} is the length, from Pythagoras. Dividing a vector by its norm gives a unit vector pointing the same way; that step is normalisation.
• The dot product \mathbf{a}\cdot\mathbf{b} = \sum a_i b_i = \|\mathbf{a}\|\|\mathbf{b}\|\cos\theta. It measures how much one vector points along another, like a shadow, and it is inflated by length.
• Cosine similarity divides out both lengths and keeps only the angle, from −1 (opposite) through 0 (unrelated) to 1 (same direction). For unit vectors it equals the plain dot product.
• A linear model's score is \mathbf{w}\cdot\mathbf{x} + b, with weights \mathbf{w} and a bias b. Neurons, search and attention are all built from dot products.

Self-test: Compute the cosine similarity of (2, 0) and (0, 5) and say what it means. Why did Bilal · Chen equal Asha · Bilal even though their tastes differ? If every stored product vector has been normalised, why can a search engine skip the division in the cosine formula?

Next: 12.2.2 Matrices as Machines — one dot product scores one order against one pattern; a matrix does thousands of them at once, and that is exactly what a layer of a neural network is.