12.2.6 — Logs, Softmax and Cross-Entropy

Cartwise's support router reads an email and has to choose one of four teams: refunds, delivery, account or payment. At the end of the model, after all the dot products and matrices of the earlier pages, out come four raw numbers, one per team:

refund   2.0
delivery 1.0
account  0.1
payment -1.0

These raw scores are called logits. They are useful for ranking, because refund clearly scores highest. But they are not probabilities: they do not lie between 0 and 1, one of them is negative, and they do not add up to 1. So two questions arise, and every classifier and every LLM answers them in exactly the same way:

1. How do you turn raw scores into probabilities? The answer is softmax (for several classes) or the sigmoid (for yes-or-no).
2. How do you punish the model when its probabilities are wrong? The answer is cross-entropy.

Both are built from two old functions, the exponential and the logarithm, so we start there.

1. The exponential function

The number e \approx 2.71828 is one of the most important constants in mathematics. It was first found in a question about money. In 1683 the Swiss mathematician Jacob Bernoulli asked: if a bank pays 100 percent interest per year, and you let it add the interest more and more often, how much do you end up with? Paid once at the end of the year, ₹1 becomes ₹2. Paid twice, as 50 percent every six months, it becomes 1.5^2 = 2.25. Paid monthly, (1 + 1/12)^{12} \approx 2.613. Paid every instant, the amount does not grow forever. It settles at about 2.71828. Leonhard Euler later named that number e and showed how it underlies all smooth growth.

The exponential function is e^x, read "e to the x". Three facts about it matter for ML:

• It is always positive. e^{-5} \approx 0.0067, which is small but still above zero. No input can make it negative or zero.
• It grows very fast. e^1 \approx 2.7, e^2 \approx 7.4, e^5 \approx 148, e^{10} \approx 22{,}026. Small differences in the input become large ratios in the output: adding 1 to x always multiplies e^x by about 2.718.
• It turns addition into multiplication: e^{a + b} = e^a \times e^b.

2. The logarithm

The logarithm undoes the exponential. If e^x = y, then \ln y = x, read "the natural log of y is x". It answers the question "what power do I need to raise e to, to get y?" So \ln e^2 = 2, \ln 1 = 0 (because e^0 = 1), and \ln 0.5 \approx -0.69.

The word was invented by the Scottish mathematician John Napier, who published the first tables of logarithms in 1614. He built it from two Greek words, logos, here meaning "ratio", and arithmos, "number". Before calculators, logarithms were a time-saving device for astronomers and navigators, because of the property that is still the reason ML uses them today:

\ln(a \times b) = \ln a + \ln b

A logarithm turns multiplication into addition. Two more facts follow. The log of any number between 0 and 1 is negative, and the closer the number is to 0, the more negative its log, without limit: \ln 0.5 \approx -0.69, \ln 0.01 \approx -4.6, \ln 0.0001 \approx -9.2. And the log grows very slowly for big numbers, which is why it is also used to squash quantities that range over many sizes, like order amounts from ₹100 to ₹200,000.

Why ML adds up logs instead of multiplying probabilities

Here is a problem you will meet for real. The probability a model assigns to a whole dataset is the product of the probabilities it assigns to each example, because each example is treated as independent. A language model scoring a 500-word document might give each word a probability of around 0.01. The probability of the whole document is then 0.01^{500} = 10^{-1000}.

A computer cannot store that number. The standard 64-bit floating-point format from Chapter 1.4 bottoms out at around 10^{-308}, and anything smaller is rounded to exactly zero. That rounding is called underflow, and once a probability has underflowed to zero, every later calculation with it is ruined.

import numpy as np

p = np.full(500, 0.01)        # 500 words, each given probability 0.01
print(np.prod(p))             # (1) → 0.0
print(np.sum(np.log(p)))      # (2) → -2302.585...

Line (1) multiplies the 500 probabilities directly and gets exactly 0.0: underflow has destroyed the answer. Line (2) adds up the logs instead and gets -2302.6, which is 500 \times \ln 0.01, a perfectly ordinary number. Because the log turns the product into a sum, working with log-probabilities gives the same information without ever creating an impossibly small number. Every serious ML library works with log-probabilities internally for exactly this reason.

3. The sigmoid: one score into one probability

For a yes-or-no question like fraud, the model produces one raw score z, and we need to turn it into a probability between 0 and 1. The function that does this is the sigmoid:

\sigma(z) = \frac{1}{1 + e^{-z}}

Read it as "sigma of z equals one over one plus e to the minus z". The name means "S-shaped", from the Greek letter sigma in its old curved form, because that is what its graph looks like:

The sigmoid squeezes any score into the range 0 to 1. Zero maps to exactly one half. The fraud score of 1.8 from Chapter 12.2.1 becomes a probability of about 0.86.

Walk it with real numbers. A score of z = 0 gives 1/(1 + e^0) = 1/(1 + 1) = 0.5: no evidence either way. A score of z = 2 gives 1/(1 + e^{-2}) = 1/(1 + 0.135) \approx 0.881. A score of z = -2 gives 1/(1 + e^{2}) = 1/(1 + 7.389) \approx 0.119, and notice that 0.881 + 0.119 = 1: the sigmoid is perfectly balanced around 0. The fraud score of 1.8 from 12.2.1 becomes 1/(1 + e^{-1.8}) \approx 0.858.

Why this formula and not some other S-shaped one? Run it backwards. If p = \sigma(z), then a little algebra gives

z = \ln\frac{p}{1 - p}

The fraction p / (1 - p) is called the odds: the chance of yes divided by the chance of no. A probability of 0.8 is odds of 0.8/0.2 = 4, often said as "4 to 1". So the raw score z is the log of the odds. That is where the word logit comes from: it is short for "log-odds unit". Check it with the fraud order: \ln(0.858 / 0.142) \approx 1.80, the original score. This is not just a curiosity. It means each weight in a model like this has a plain meaning: adding w to the score multiplies the odds of fraud by e^w. A weight of 3.0 on "new account" means a new account multiplies the odds of fraud by e^3 \approx 20. Chapter 12.3.1 builds logistic regression on exactly this idea.

4. Softmax: many scores into many probabilities

Back to the four support teams. We need four probabilities that are all positive and add up to 1, and we want a higher score to give a higher probability. Softmax does it in two steps: take e to the power of every score, then divide each result by the total.

\text{softmax}(z)_i = \frac{e^{z_i}}{\sum_{j} e^{z_j}}

Read it as "the softmax of z, for class i, equals e to the z-i, divided by the sum over all classes j of e to the z-j".

Softmax in two moves. The exponential makes every score positive and stretches the gaps between them; dividing by the total makes them add up to 1. The order never changes: the highest score always gets the highest probability.

Follow the figure with the actual arithmetic:

1. Exponentiate. e^{2.0} \approx 7.389, e^{1.0} \approx 2.718, e^{0.1} \approx 1.105, e^{-1.0} \approx 0.368. All four are now positive, including the payment score that started negative. That is the first job of the exponential.
2. Add them up. 7.389 + 2.718 + 1.105 + 0.368 = 11.580.
3. Divide each by the total. Refund: 7.389 / 11.580 \approx 0.638. Delivery: 0.235. Account: 0.095. Payment: 0.032. These add up to 1, so they are valid probabilities.

Why use the exponential, rather than something simpler like shifting the scores to be positive and dividing by the total? Because of the "stretches gaps" fact from section 1. A difference of 1 between two logits always becomes a ratio of about 2.718 between their probabilities, wherever the logits sit. Refund's logit is 1 higher than delivery's, so refund is e^1 \approx 2.7 times as likely (0.638 / 0.235 \approx 2.7). That means only the differences between logits matter, never their absolute size. Add 100 to every logit and the probabilities do not change at all.

Libraries use that last fact for safety. With logits like 1,000, e^{1000} overflows to infinity. So softmax is always computed by first subtracting the largest logit from all of them, which changes nothing about the answer but keeps the biggest exponential at exactly e^0 = 1.

The name "softmax" describes what it does. A hard maximum would pick refund and give it probability 1 and everything else 0. Softmax is a soft version: the winner gets most of the probability, but the others keep a share in proportion to how close they came.

Temperature: making softmax sharper or flatter

One small change to softmax shows up everywhere in LLMs. Before exponentiating, divide every logit by a number T called the temperature:

\text{softmax}(z / T)_i = \frac{e^{z_i / T}}{\sum_j e^{z_j / T}}

Dividing by a small T makes the gaps between logits bigger, so the winner takes even more. Dividing by a large T shrinks the gaps, so the probabilities spread out more evenly. With our four logits:

Temperature	refund	delivery	account	payment
0.5 (sharper)	0.862	0.117	0.019	0.002
1.0 (plain)	0.638	0.235	0.095	0.032
2.0 (flatter)	0.451	0.274	0.174	0.101

The name comes from physics. In the statistical mechanics of the 1870s, Ludwig Boltzmann showed that the share of particles in each energy state follows exactly this formula, and at high temperature particles spread across many states while at low temperature they crowd into the lowest one. When an LLM chooses its next word from a softmax over its whole vocabulary, a low temperature makes it pick the likeliest word almost every time, and a high temperature makes it take more chances. That is the "temperature" setting you see on every LLM API, explained fully in Chapter 12.7.7.

5. Cross-entropy: punishing wrong probabilities

Now suppose the email was really about a payment. The model gave payment only 0.032. How should training punish it? And if the email was really about a refund, where the model gave 0.638, the punishment should be smaller. The loss used for this, in every classifier and every LLM, is the cross-entropy loss:

L = -\ln p_{\text{correct}}

Read it as "the loss is minus the natural log of the probability the model gave to the correct answer". That is the whole formula. Take the probability the model assigned to the right class, take its log, and flip the sign so the result is positive.

Cross-entropy as a function of the probability given to the correct answer. Being right and confident costs almost nothing. Being confidently wrong costs a great deal, and the cost keeps rising as the probability heads to zero.

With our numbers: if the right answer is refund, the loss is -\ln 0.638 \approx 0.45. If the right answer is payment, the loss is -\ln 0.032 \approx 3.45, almost eight times as much. A model that gave the right answer probability 0.99 would pay only -\ln 0.99 \approx 0.01. A model that hedged at 0.5 pays -\ln 0.5 \approx 0.69. And a model that gave the right answer 0.01, confidently wrong, pays 4.6. As the probability heads towards zero, the loss rises without any ceiling.

That shape is exactly what we want from a teacher. Being right and confident is nearly free. Being unsure costs a little. Being confidently wrong costs a lot. A model trained this way learns to be confident only when it has reason to be.

Notice a neat detail in the numbers: the payment loss (3.45) is exactly 3 more than the refund loss (0.45), and the payment logit is exactly 3 lower than the refund logit. That is not a coincidence. With softmax followed by a log, the log undoes the exponential, so the loss for each class is just "how far its logit is below the others", measured on the original scale. This is one more reason the exponential and the logarithm are used together.

For a yes-or-no problem with a sigmoid, the same idea is written to cover both cases in one line, where y is 1 for fraud and 0 for genuine, and p is the model's probability of fraud:

L = -\big[\,y \ln p + (1 - y) \ln(1 - p)\,\big]

When y = 1, the second term vanishes and this is -\ln p. When y = 0, the first term vanishes and it is -\ln(1 - p), minus the log of the probability given to "genuine". It is the same rule: minus the log of the probability given to whatever was actually true. This is called binary cross-entropy or log loss.

Why this loss and not squared error?

You could, in principle, train a classifier with the mean squared error from 12.2.4, treating the right answer as 1 and the probability as the guess. Two reasons make cross-entropy the right choice.

It comes straight from probability. The best model is the one that makes the observed data most probable. The probability of the whole training set is the product of p_{\text{correct}} over every example. Maximising a product is the same as maximising its log, which is the sum of the \ln p_{\text{correct}} terms, and maximising that is the same as minimising its negative. That negative sum is the total cross-entropy. So minimising cross-entropy means "find the weights under which what actually happened was most likely". This principle is called maximum likelihood, and most loss functions in ML turn out to be maximum likelihood for some assumption about the data.

Its gradient is beautifully simple. For a sigmoid followed by binary cross-entropy, applying the chain rule from 12.2.3 through both functions, almost everything cancels and you are left with:

\frac{\partial L}{\partial z} = p - y

The slope with respect to the raw score is just "the predicted probability minus the truth". For the fraud order with score 1.8, if it really was fraud (y = 1), the slope is 0.858 - 1 = -0.142: push the score up a little. If it was genuine (y = 0), the slope is 0.858: push the score down hard. The size of the push is exactly the size of the mistake. Softmax with cross-entropy gives the same result for every class, p_i - y_i. With squared error instead, the sigmoid's flat ends would shrink the gradient whenever the model is badly and confidently wrong, which is exactly when it most needs a strong push, and training would crawl.

Where the name comes from

"Entropy" in this sense comes from Claude Shannon's 1948 paper that founded information theory, covered in Chapter 1.8. Shannon measured the surprise of an event with probability p as -\log p: a certain event carries no surprise, and a rare one carries a lot. Cross-entropy is the average surprise you experience when reality follows one distribution (the true labels) but you are predicting with another (the model's probabilities). Training a classifier means making the model as unsurprised by reality as possible.

6. The same pieces inside an LLM

Everything on this page appears, unchanged, inside every large language model. At each step, the model produces one logit for every token in its vocabulary, which for current models is somewhere between about 32,000 and 250,000 entries. Softmax turns those logits into a probability for every possible next token. Temperature reshapes that distribution before a token is chosen. And training is nothing more than minimising the cross-entropy of the actual next token, averaged over trillions of examples. When a training report says "loss 1.9", that is the average of -\ln p_{\text{correct}} over the next tokens.

A related number you will see in LLM papers is perplexity, which is just e^{\text{loss}}. A loss of 1.9 is a perplexity of about 6.7, which you can read as "the model is, on average, as unsure as if it were choosing evenly among 6.7 words". Lower is better.

Here are all the pieces of this page in code:

import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

def softmax(z, T=1.0):
    z = np.asarray(z) / T
    e = np.exp(z - z.max())                   # (1) subtract the max: same answer, no overflow
    return e / e.sum()                        # (2)

logits = [2.0, 1.0, 0.1, -1.0]                # refund, delivery, account, payment
p = softmax(logits)
print(p.round(3))                             # → [0.638 0.235 0.095 0.032]
print(softmax(logits, T=0.5).round(3))        # → [0.862 0.117 0.019 0.002]

print(-np.log(p[0]), -np.log(p[3]))           # (3) → 0.449... 3.449...
print(sigmoid(1.8).round(3))                  # → 0.858
print(sigmoid(1.8) - 1)                       # (4) → -0.141...  the gradient if it was fraud

Line (1) is the safety trick from section 4: subtracting the largest logit leaves the probabilities unchanged but keeps every exponential at or below 1. Line (2) divides by the total. Line (3) computes the cross-entropy for the two cases in the figure. Line (4) is the p - y gradient for the fraud score when the order really was fraud.

Recall

• Logits are a model's raw scores. The exponential function e^x is always positive and turns differences into ratios; the logarithm undoes it and turns products into sums.
• Multiplying many probabilities causes underflow to zero, so libraries add log-probabilities instead.
• The sigmoid \sigma(z) = 1/(1+e^{-z}) maps one score to a probability; the score is the log of the odds. Softmax maps many scores to probabilities that sum to 1; only differences between logits matter. Dividing logits by a temperature sharpens (T below 1) or flattens (T above 1) the result.
• The cross-entropy loss is -\ln p_{\text{correct}}: nearly free when right and sure, very expensive when wrong and sure. It is maximum likelihood, and with a sigmoid or softmax its gradient is simply p - y.
• LLMs use exactly these pieces: softmax over the vocabulary, temperature when choosing, cross-entropy as the training loss. Perplexity is e^{\text{loss}}.

Self-test: Compute the softmax of (1, 1, 1) without a calculator. What is the cross-entropy if the model gave the right answer probability 1.0? What happens to the probabilities if you add 5 to every logit, and why? A model's loss falls from 2.3 to 1.6; what happens to its perplexity?

Next: 12.3.1 Linear Regression — the maths toolkit is complete; now we build the first real model from it, predicting a number, and see every piece of this chapter at work.