M.A.T.R.I.X

Appearance

12.2 — The Math You Actually Need ​

Four ideas carry almost all of machine learning: a list of numbers, a way to multiply lists of numbers, a slope, and a probability. Everything in the next eight chapters is built from those, and none of them requires more than school algebra to understand.

This page defines each one from zero, because Chapter 12.4 will use them without pausing.

1. Vectors: a list of numbers with a direction ​

A vector is an ordered list of numbers, written \mathbf{v} = [3, 1, 4]. That is all it is in code — an array.

The reason it feels like more is that a list of numbers can be read as a point in space. [3, 1] is a point three units right and one unit up. [3, 1, 4] is a point in three dimensions. A list of 768 numbers is a point in 768-dimensional space, which nobody can picture — and every operation below works identically regardless of the number of dimensions, which is why the inability to visualise it does not matter.

Adding vectors adds element by element: [3,1] + [1,2] = [4,3].

Multiplying by a number scales it: 2 \times [3,1] = [6,2].

The dot product multiplies matching elements and sums them:

[3,1,4] \cdot [2,0,1] = 3{\times}2 + 1{\times}0 + 4{\times}1 = 10

The dot product is the single most important operation in this Part, because of what it measures. Geometrically:

\mathbf{a} \cdot \mathbf{b} = \|\mathbf{a}\| \, \|\mathbf{b}\| \cos\theta

where \|\mathbf{a}\| is the vector's length and \theta the angle between them. So:

  • Large positive — pointing the same way.
  • Zero — at right angles, unrelated.
  • Negative — pointing opposite ways.

Cosine similarity is the dot product with the lengths divided out, leaving only \cos\theta, in the range -1 to 1:

\text{similarity}(\mathbf{a},\mathbf{b}) = \frac{\mathbf{a}\cdot\mathbf{b}}{\|\mathbf{a}\|\|\mathbf{b}\|}

This is how vector search works (Chapter 12.6.2). Text becomes a vector; similar meanings point in similar directions; finding related documents is finding the vectors with the largest cosine similarity. Length is divided out because a longer document should not automatically be more similar.

2. Matrices: applying the same operation to many vectors ​

A matrix is a grid of numbers — a list of vectors. A 3 \times 2 matrix has 3 rows and 2 columns.

Matrix multiplication is dot products, arranged. Element (i,j) of the result is the dot product of row i of the first matrix with column j of the second.

\begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix} \begin{bmatrix} 5 \\ 6 \end{bmatrix} = \begin{bmatrix} 1{\times}5 + 2{\times}6 \\ 3{\times}5 + 4{\times}6 \end{bmatrix} = \begin{bmatrix} 17 \\ 39 \end{bmatrix}

Two facts follow, and they explain the shape of the whole field.

The inner dimensions must match. An (m \times n) matrix times an (n \times p) matrix gives (m \times p). A shape mismatch is the most common error in machine learning code, and reading it as "the middle numbers must agree" fixes most of them.

Every dot product in the multiplication is independent, so they can all be computed simultaneously. That is why GPUs won (Chapter 12.9): a graphics processor has thousands of small cores, and a matrix multiply is thousands of independent multiply-and-add operations. A neural network is mostly matrix multiplication, so it maps onto that hardware almost perfectly.

A matrix as a transformation. Multiplying a vector by a matrix moves it — rotating, stretching, projecting into fewer or more dimensions. A neural network layer is exactly this: take a vector, multiply by a weight matrix, add a bias vector, apply a non-linear function. Stack those and you have a deep network.

3. Derivatives: which way is downhill ​

Training means adjusting numbers to reduce error. To adjust them you must know which direction reduces the error, and that is what a derivative gives you.

A derivative is the slope of a function at a point — how much the output changes for a tiny change in the input, written \frac{df}{dx}.

  • Positive slope: increasing x increases f. To decrease f, decrease x.
  • Negative slope: the reverse.
  • Zero: flat — a minimum, a maximum, or a plateau.

A gradient is the derivative when there are many inputs: one slope per input, collected into a vector, written \nabla f. It points in the direction of steepest increase, so the opposite direction is steepest decrease.

That gives the entire training algorithm:

w_{\text{new}} = w_{\text{old}} - \eta \, \frac{\partial L}{\partial w}

Read it plainly: each weight moves a small step in the direction that reduces the loss. L is the loss (how wrong we are), \frac{\partial L}{\partial w} is how much this weight contributes to it, and \eta — the learning rate — is the step size.

The learning rate is the hyperparameter that most often ruins a training run. Too large and the updates overshoot, bouncing around or diverging to infinity. Too small and training takes forever or stalls in a poor spot. Chapter 12.4.2 covers the schedules used to manage it.

The chain rule is what makes deep networks trainable. If y depends on u and u depends on x, then:

\frac{dy}{dx} = \frac{dy}{du} \cdot \frac{du}{dx}

Slopes multiply along a chain. A ten-layer network is a chain of ten functions, so the gradient of the loss with respect to a first-layer weight is a product of ten local derivatives. That is backpropagation (Chapter 12.4.1) — and it also explains why very deep networks were once untrainable: multiply many numbers below one and the result vanishes to nothing, so early layers stopped learning.

4. Probability ​

A probability distribution assigns a number between 0 and 1 to each possible outcome, summing to 1. A language model's output is exactly this: a probability for every token in its vocabulary.

Expectation is the average outcome weighted by probability. Roll a fair die: \frac{1}{6}(1+2+3+4+5+6) = 3.5.

Conditional probability is P(A \mid B) — the probability of A given that B happened. This is the whole shape of a language model: the probability of the next token given everything before it.

Bayes' theorem reverses the condition:

P(A \mid B) = \frac{P(B \mid A) \, P(A)}{P(B)}

The intuition matters more than the formula: a test result updates a prior belief; it does not replace it. A 99%-accurate test for a disease affecting 1 in 10,000 people still produces mostly false positives, because there are so many more healthy people to be wrong about. Most people get this wrong, and it is the reason a highly accurate classifier for a rare event can be nearly useless in production.

5. Why logarithms appear everywhere ​

Two reasons, both practical.

Multiplying many small probabilities underflows. The probability of a 1,000-token sequence is a product of 1,000 numbers below 1 — far smaller than a floating-point number can represent (Chapter 1.4). Taking logs turns the product into a sum, which is numerically safe:

\log(a \times b) = \log a + \log b

And logs make the loss function well-behaved. The standard loss for classification is cross-entropy:

L = -\sum_i y_i \log(\hat{y}_i)

where y_i is 1 for the correct class and 0 otherwise, and \hat{y}_i is the predicted probability. For the correct class this reduces to -\log(\hat{y}), which is small when the prediction is confident and correct, and grows without bound as the prediction approaches zero.

That unbounded growth is the point. A model that is confidently wrong is punished enormously, so training pushes hard away from confident errors. This is the same quantity as Shannon's entropy from Chapter 1.8: cross-entropy measures how many bits you waste encoding the truth using the model's beliefs.

Softmax turns arbitrary numbers into a probability distribution, and it is the last step of nearly every classifier and every language model:

\text{softmax}(z_i) = \frac{e^{z_i}}{\sum_j e^{z_j}}

Exponentiate each raw score (a logit), then divide by the total so they sum to 1. The exponential is what makes it "soft max": it amplifies differences, so a slightly larger logit becomes a substantially larger probability. Chapter 12.5.3 shows how dividing the logits by a temperature before softmax controls how sharp the distribution is, which is the knob behind creative versus deterministic generation.

6. Dimensions, and why high ones are strange ​

Models work in hundreds or thousands of dimensions, and intuition from two and three dimensions fails there in ways worth knowing.

Distances concentrate. In very high dimensions, the distance between the nearest and furthest points in a random set becomes almost the same. "Nearest neighbour" loses meaning if the representation is poor — which is why embedding quality matters more than the search algorithm in a retrieval system.

Volume moves to the edges. Almost all the volume of a high-dimensional ball is near its surface, so random points are almost always far from the centre and roughly at right angles to each other.

That last property is useful rather than a problem. Because random directions in high-dimensional space are nearly at right angles, a space of a few hundred dimensions can hold an enormous number of nearly independent concepts. That is why a 768-number vector can meaningfully represent a sentence.

Dimensionality reduction compresses while keeping structure — principal component analysis for a linear view, t-SNE and UMAP for visualisation. Use the visual ones only to look, never to feed another model: they distort distances deliberately to make a readable picture, so clusters in a t-SNE plot are suggestive rather than measured.

7. What this buys you ​

You can now read the rest of this Part:

  • An embedding is a vector. Similar meanings, similar directions, compared by cosine similarity.
  • A layer is a matrix multiply plus a bias plus a non-linear function.
  • Training is gradient descent: measure the loss, compute the gradient by the chain rule, step every weight downhill.
  • A model's output is a probability distribution, produced by softmax over logits.
  • Attention (Chapter 12.5.1) is dot products deciding how much each token should attend to each other token.

That last line is worth holding onto. The mechanism behind the entire current generation of models is the operation defined in section 1 of this page.

Recall

  • A vector is a list of numbers read as a point in space; the operations are identical in 2 or 768 dimensions.
  • The dot product is the key operation: it measures alignment, because \mathbf{a}\cdot\mathbf{b} = \|\mathbf{a}\|\|\mathbf{b}\|\cos\theta. Cosine similarity divides out length, which is why a longer document is not automatically more similar.
  • Matrix multiplication is arranged dot products; inner dimensions must match, and every entry is independent — which is exactly why GPUs win. A network layer is a matrix multiply, a bias, and a non-linear function.
  • A gradient is one slope per input and points at steepest increase, so training steps the other way: w \leftarrow w - \eta \frac{\partial L}{\partial w}. The learning rate is the hyperparameter that most often ruins a run.
  • The chain rule multiplies slopes along a chain — that is backpropagation, and it is also why very deep networks once failed: many factors below one vanish.
  • Bayes updates a prior, it does not replace it. A 99%-accurate test for a 1-in-10,000 condition still yields mostly false positives — the reason accurate classifiers for rare events can be useless.
  • Logs turn products into sums (no underflow) and give cross-entropy, which punishes confident errors without bound. Softmax turns logits into a distribution, and its exponential amplifies differences — the basis of the temperature knob.
  • In high dimensions, distances concentrate and random directions are nearly at right angles — which is what lets a few hundred numbers hold many independent concepts. Use t-SNE and UMAP only to look at data, never to feed a model.

Self-test: What does a dot product of zero mean? · Why must the inner dimensions match, and why does that make GPUs the right hardware? · Why does the chain rule explain both backpropagation and vanishing gradients? · Why does cross-entropy grow without bound? · What does dividing logits by a temperature change? · Why can a 768-dimension vector represent so many distinct concepts?