12.2.5 — Probability and Bayes

Cartwise's new fraud model was tested on last year's orders, and the report looked excellent. It catches 90 percent of fraudulent orders. It wrongly flags only 2 percent of genuine ones. The team switched it on, sending every flagged order to a human reviewer.

On the first day, the reviewers complained. Out of every five orders they were sent, roughly four were perfectly genuine customers. They asked, reasonably, how a model that is "wrong only 2 percent of the time" could be wrong 80 percent of the time on the orders they actually saw.

Nobody had lied and nothing was broken. The answer is one of the most important ideas in all of applied machine learning, and you need only a little probability to see it. This page builds that probability from the beginning, and then uses it to explain the reviewers' complaint, what a model's "0.83" really means, and how to turn probabilities into decisions.

1. What a probability is

A probability is a number between 0 and 1 that says how likely something is. 0 means it cannot happen, 1 means it is certain, and 0.5 means it is as likely as not. We write P(\text{fraud}) = 0.005 and read it as "the probability of fraud is 0.005", which is half a percent.

There are two ways to think about what that number means, and both are useful:

• As a long-run frequency. Out of a very large number of orders, about 0.5 percent turn out to be fraud. Out of 10,000 orders, expect about 50.
• As a degree of belief. For one particular order you know nothing else about, 0.005 is how confident you should be that it is fraud.

The first view is easier to check. The second is what a model's output means when it scores a single order. The two agree whenever the model is honest about its own uncertainty, which section 6 comes back to.

A few rules follow directly from the meaning. The probability that something does not happen is one minus the probability that it does: P(\text{genuine}) = 1 - 0.005 = 0.995. And if two outcomes cannot both happen, the probability of one or the other is the sum of the two.

2. Conditional probability: "given that…"

The reviewers' problem is about a different kind of probability. They are not asking "how likely is any order to be fraud?" They are asking "how likely is an order to be fraud given that the model flagged it?" That is a conditional probability, written with a vertical bar:

P(\text{fraud} \mid \text{flagged})

Read the bar as "given". The whole thing is "the probability of fraud, given flagged". The condition on the right of the bar shrinks the world you are looking at: you are no longer considering all orders, only the flagged ones, and asking what fraction of those are fraud.

The order matters enormously, and confusing the two orders is the most common reasoning mistake in this whole field. The test report gave you:

• P(\text{flagged} \mid \text{fraud}) = 0.90: out of fraudulent orders, 90 percent get flagged.
• P(\text{flagged} \mid \text{genuine}) = 0.02: out of genuine orders, 2 percent get flagged.

Neither of those is the number the reviewers care about. They want P(\text{fraud} \mid \text{flagged}), which is the same two words in the opposite order and can be a completely different number.

Working it out with counts

The clearest way to compute it is to imagine a concrete number of orders and simply count. Take 10,000 orders, and use the fact from section 1 that fraud is 0.5 percent of them.

Ten thousand orders, counted. Fraud is rare, so even a small 2 percent false-flag rate on the huge genuine pile produces far more flagged orders (199, amber) than the 45 real frauds caught (red). The purple box is what the reviewers see.

Walk the tree from left to right:

1. Of 10,000 orders, 0.5 percent are fraud: 50 fraud and 9,950 genuine.
2. The model flags 90 percent of the fraud: 45 flagged, 5 missed.
3. The model also flags 2 percent of the genuine orders: 2 percent of 9,950 is 199 flagged, and 9,751 pass through.
4. The reviewers see every flagged order: 45 + 199 = 244. Of those, only 45 are fraud.

P(\text{fraud} \mid \text{flagged}) = \frac{45}{244} \approx 0.18

So only about 18 percent of the flagged orders are fraud, and the other 82 percent are genuine customers. The reviewers were exactly right.

The reason is in the figure. Fraud is rare, so the red branch is small. Genuine orders are common, so the green branch is enormous, and even a small error rate on an enormous pile produces a lot of mistakes. 2 percent of 9,950 is four times bigger than 90 percent of 50. When the thing you are looking for is rare, most of your alarms will be false, even with a good detector. This is called the base rate effect, because the result depends heavily on the base rate, the underlying 0.5 percent, which the impressive test numbers did not mention at all. Ignoring it is called the base rate fallacy, and it fools doctors reading test results, juries hearing forensic evidence, and engineers reading model reports.

The practical consequences for Cartwise are concrete. If the reviewer team can handle 300 orders a day and the store has 40,000 orders a day, this model would send them about 976 flagged orders a day. To reduce the load, you would raise the model's threshold so it flags fewer genuine orders, accepting that it will also catch less fraud. Chapter 12.3.2 gives the vocabulary for this trade: the 45/244 is called precision, and the 90 percent is called recall.

The general definition

Counting gives the general formula. The conditional probability of A given B is the fraction of B-cases in which A also happens:

P(A \mid B) = \frac{P(A \text{ and } B)}{P(B)}

In the tree, "fraud and flagged" is 45 out of 10,000, and "flagged" is 244 out of 10,000. Dividing one by the other, the 10,000s cancel and you get 45/244, the same answer.

3. Bayes' rule

The calculation we just did, turning "flagged given fraud" around into "fraud given flagged", is so common that it has a name. Bayes' rule says:

P(A \mid B) = \frac{P(B \mid A)\; P(A)}{P(B)}

Read it as "the probability of A given B equals the probability of B given A, times the probability of A, divided by the probability of B". With A as fraud and B as flagged, the pieces are:

• P(A) = 0.005 is the prior: what you believed before seeing the flag. "Prior" means "before".
• P(B \mid A) = 0.90 is the likelihood: how likely the evidence is if A were true.
• P(B) is the total probability of seeing the evidence at all. Getting flagged can happen two ways, through fraud or through genuine, so P(B) = 0.90 \times 0.005 + 0.02 \times 0.995 = 0.0045 + 0.0199 = 0.0244.
• P(A \mid B) is the posterior: what you should believe after seeing the evidence. "Posterior" means "after".

Plugging in: \dfrac{0.90 \times 0.005}{0.0244} = \dfrac{0.0045}{0.0244} \approx 0.18, matching the count. The formula and the tree are the same calculation; the tree just makes it impossible to forget the big green branch.

The rule is named after Thomas Bayes, an English minister and amateur mathematician who worked it out in the 1740s or 1750s. He never published it. After his death in 1761, his friend Richard Price found the manuscript among his papers, edited it, and had it read to the Royal Society in 1763 under the title "An Essay towards solving a Problem in the Doctrine of Chances". The French mathematician Pierre-Simon Laplace independently developed the same idea further a few years later, and did more than Bayes to make it a working tool.

The idea is worth stating in plain words, because it is how careful reasoning works everywhere: start with how common something is, then update by how much more likely the evidence is when it is true than when it is false. A flag is 45 times more likely on a fraudulent order than a genuine one (0.90 against 0.02), which is strong evidence. But fraud starts out 199 times rarer than genuine (0.005 against 0.995), and that is stronger still. Bayes' rule is the bookkeeping that weighs the two against each other.

Chapter 12.3.5 builds a complete classifier, naive Bayes, directly out of this rule.

4. Random variables and distributions

So far we have talked about events: fraud or not, flagged or not. Often the thing you care about is a number that varies, like how many days a delivery takes or how much a customer spends. A quantity whose value depends on chance is called a random variable, and the full description of which values it can take, and how likely each is, is its probability distribution.

There are two kinds, and the difference matters for how you write a model's output.

Discrete variables take separate values you can list: fraud or not (two values), which of four support teams (four values), how many items in a basket (0, 1, 2, …). Their distribution is a list of probabilities, one per value, adding up to 1. A yes-or-no variable is called a Bernoulli variable, after the Swiss mathematician Jacob Bernoulli, and a fraud classifier's output is exactly the probability of "yes" for a Bernoulli variable. A choice among several categories is called categorical, and that is what a multi-class classifier outputs.

Continuous variables can take any value in a range: 3.0 days, 3.07 days, 3.0712 days. You cannot list a probability for each exact value, because there are infinitely many. Instead the distribution is a curve, and the probability of landing in a range is the area under the curve over that range.

The normal distribution

The most famous continuous distribution is the normal distribution, also called the Gaussian after the German mathematician Carl Friedrich Gauss, or simply the bell curve because of its shape.

A normal distribution of delivery times with mean 3 days and standard deviation half a day. About 68 percent of deliveries fall within one standard deviation of the mean (green), and about 95 percent within two (blue).

The normal distribution is described completely by two numbers: where its centre is (the mean) and how spread out it is (the standard deviation), both of which are defined in the next section. Its shape has a famous rule of thumb: about 68 percent of values lie within one standard deviation of the mean, about 95 percent within two, and about 99.7 percent within three. In the figure, with a mean of 3 days and a standard deviation of half a day, about 68 percent of deliveries take between 2.5 and 3.5 days, and about 95 percent between 2 and 4.

Why does this one shape turn up so often, in heights, measurement errors and delivery times alike? Because of a remarkable result called the central limit theorem: whenever a quantity is the sum of many small, independent random effects, its distribution comes out close to normal, whatever the shapes of those small effects. A delivery time is the sum of a packing delay, a sorting delay, a few transport legs and a final-mile delay, so it ends up roughly bell-shaped. Volume II's chapter on the Gaussian shows why this happens.

A warning that matters in practice: not everything is normal. The amounts customers spend are usually skewed: most orders are small and a few are huge, so the curve has a long tail to the right. Treating such data as normal makes extreme values look far rarer than they are, and in fraud work, the extreme values are exactly the ones you care about.

5. Expectation and variance

Expectation: the probability-weighted average

The expected value of a random variable is its average, with each possible value weighted by how likely it is. It is written E[X] and read "the expectation of X". For a discrete variable:

E[X] = \sum_{\text{each value } x} x \cdot P(X = x)

Here is why expectation is the bridge from a model's probability to a business decision. Suppose an order of ₹20,000 comes in, and the model gives it a 10 percent chance of being fraud. If Cartwise approves it, two things can happen. With probability 0.1 it is fraud, and Cartwise loses the ₹20,000 of goods. With probability 0.9 it is genuine, and Cartwise earns its profit margin, say ₹1,500. The expected value of approving is:

E[\text{approve}] = 0.1 \times (-20{,}000) + 0.9 \times 1{,}500 = -2{,}000 + 1{,}350 = -650

On average, approving orders like this one loses ₹650 each. Blocking it has an expected value of roughly zero (no sale, no loss, ignoring the customer's annoyance). So the model's 0.1 should lead to a block here, even though 0.1 sounds low. For a ₹2,000 order with the same 0.1, approving gives 0.1 \times (-2{,}000) + 0.9 \times 150 = -65, much closer to break-even. The right threshold depends on the money at stake, and expectation is how you compute it. This is why a single fixed threshold of 0.5 is rarely correct.

Variance and standard deviation: how spread out

Two couriers both average 3 days. FastShip always takes between 2.8 and 3.2. SlowPost sometimes takes 1 day and sometimes 6. Their expectations are equal, but you would much rather promise a date with FastShip. The variance measures that spread: the average squared distance from the mean.

\text{Var}(X) = E\big[(X - E[X])^2\big]

The standard deviation, written \sigma (the Greek letter sigma), is the square root of the variance. Taking the square root puts it back in the original units, days instead of days squared, so it can be read directly as "a typical distance from the average".

A worked example. SlowPost's last four deliveries took 1, 2, 4 and 5 days. The mean is (1 + 2 + 4 + 5)/4 = 3. The distances from the mean are -2, -1, 1, 2, their squares are 4, 1, 1, 4, and the average of those is 10/4 = 2.5. That is the variance. The standard deviation is \sqrt{2.5} \approx 1.58 days. For FastShip with 2.8, 2.9, 3.1, 3.2, the distances are -0.2, -0.1, 0.1, 0.2, the variance is (0.04 + 0.01 + 0.01 + 0.04)/4 = 0.025, and the standard deviation is about 0.16 days, ten times smaller.

import numpy as np

slow = np.array([1.0, 2.0, 4.0, 5.0])
fast = np.array([2.8, 2.9, 3.1, 3.2])
print(slow.mean(), slow.var(), slow.std())   # (1) → 3.0 2.5 1.5811...
print(fast.mean(), fast.var(), fast.std())   # → 3.0 0.025 0.1581...  (tiny float noise possible)

Line (1) prints the mean, the variance and the standard deviation, which match the hand calculation. This standard deviation is exactly the "spread" that was divided out during standardisation in 12.2.4.

6. What a model's probability really means

When the fraud model outputs 0.83, what is it claiming? It is claiming that among orders it scores around 0.83, about 83 percent are fraud. A model whose scores mean what they say in this way is called calibrated.

Many models are not. A model can be excellent at ranking orders, putting the fraud above the genuine, while its numbers are systematically too confident or not confident enough. Perhaps of all the orders it scores 0.83, only 40 percent are fraud. If you only use the score to sort orders for review, that does not matter. But if you plug the score into the expected-value calculation from section 5, a badly calibrated 0.83 gives you the wrong decision. Chapter 12.3.2 shows how to check calibration with a simple chart and how to fix it.

Notice that the base rate is part of what a calibrated probability means. A model trained on data where fraud was 0.5 percent of orders learns that base rate. If you later train it on a sample where you deliberately included more fraud to make learning easier (a common trick, covered in Chapter 12.4.3), its probabilities will be too high in production, because the prior it learned is wrong. Bayes' rule is also how you correct for that.

7. Estimates from samples have error

One last idea, because it decides how much you can trust any number in a model report. The test said the model catches 90 percent of fraud. How sure is that number?

The test set had 50 fraud cases, and the model caught 45. If you had drawn a different 50 fraud cases, you might have caught 43, or 47. Any number computed from a sample is an estimate of the true value, and it has its own uncertainty. For a percentage measured on n cases, a rough measure of that uncertainty, called the standard error, is:

\text{SE} = \sqrt{\frac{p\,(1 - p)}{n}}

With p = 0.9 and n = 50, that is \sqrt{0.9 \times 0.1 / 50} = \sqrt{0.0018} \approx 0.042. By the 95-percent rule from section 4, the true catch rate is plausibly anywhere within two standard errors of the estimate: from about 82 to 98 percent. That is a wide range.

This has a direct consequence for how you compare models. If a new model catches 92 percent on the same 50 cases, you cannot claim it is better: the difference of 2 points is well inside the noise. To tell models apart reliably on rare events, you need test sets with many more examples of the rare thing, which usually means a longer stretch of history. Many "improvements" reported in model reviews are just luck in a small test set, and asking "how many positive examples was this measured on?" is one of the most useful questions an engineer can ask.

Recall

• A probability is a number from 0 to 1, read as a long-run frequency or a degree of belief. A conditional probability P(A \mid B) is the fraction of B-cases in which A happens, and P(A\mid B) and P(B\mid A) can be very different.
• The base rate effect: when the target is rare, most alarms are false even from a good detector. With 0.5 percent fraud, 90 percent caught and 2 percent of genuine flagged, only 45 of 244 flags (18 percent) are fraud.
• Bayes' rule: P(A\mid B) = P(B\mid A)P(A)/P(B), turning a prior and a likelihood into a posterior.
• A random variable has a probability distribution, discrete or continuous. The normal distribution holds about 68 / 95 / 99.7 percent within 1 / 2 / 3 standard deviations; sums of many small effects come out normal.
• The expected value weights outcomes by probability and turns a model's score into a decision; the right threshold depends on the money at stake. Variance and standard deviation measure spread. A calibrated model's 0.83 really means 83 percent. Estimates from small samples carry a standard error.

Self-test: Redo the tree with fraud at 2 percent of orders, keeping 90 percent caught and 2 percent of genuine flagged. What fraction of flagged orders is now fraud? An order of ₹5,000 has a 5 percent fraud score and a ₹400 margin; should it be approved on expected value? Why can a 2-point recall improvement measured on 50 fraud cases not be trusted?

Next: 12.2.6 Logs, Softmax and Cross-Entropy — a model needs to produce probabilities and be punished when they are wrong; the exponential, the logarithm, softmax and cross-entropy are the four tools that do it, and every classifier and every LLM is trained with them.