Appearance
7.2 — Probability
Probability began, unglamorously, with gambling. In 1654 a French nobleman, the Chevalier de Méré, asked Blaise Pascal about a dice problem he was losing money on. Pascal wrote to Pierre de Fermat, and their correspondence that summer created the subject.
The specific problem is worth seeing because it shows how sharp the reasoning has to be. De Méré had made money for years betting that he could roll at least one six in four throws of a die. He then reasoned that since a double six is 6 times rarer, 24 throws of two dice should give the same odds. It did not, and he lost.
P(\text{at least one six in 4}) = 1 - \left(\frac56\right)^4 = 0.5177
P(\text{at least one double six in 24}) = 1 - \left(\frac{35}{36}\right)^{24} = 0.4914
The first is a winning bet and the second is a losing one, separated by less than three percentage points. Intuition about probability is unreliable at exactly this level of precision, which is why the subject needed a formal treatment.
1. The vocabulary
An experiment is any process with an uncertain outcome. The sample space S is the set of all possible outcomes. An event is any subset of the sample space.
Rolling a die: S = \{1,2,3,4,5,6\}, and "even" is the event \{2,4,6\}.
When all outcomes are equally likely:
P(A) = \frac{\text{number of outcomes in } A}{\text{number of outcomes in } S}
which is why Chapter 7.1 came first. P(\text{even}) = \frac36 = \frac12.
The equally-likely assumption must be earned. It holds for a fair die and fails for almost everything real. The chance of rain tomorrow is not "rain or no rain, so one half".
2. The axioms
Kolmogorov put probability on a rigorous footing in 1933 with three rules, and everything else follows from them.
- 0 \le P(A) \le 1 for every event.
- P(S) = 1 — something happens.
- If A and B cannot both occur, P(A\text{ or }B) = P(A)+P(B).
Immediate consequences:
The complement rule: P(\text{not }A) = 1 - P(A).
This is the single most useful computational trick in probability. "At least one" problems are almost always easier as "one minus the probability of none" — which is exactly how de Méré's calculation above was done.
The addition rule, when events can overlap:
P(A\cup B) = P(A) + P(B) - P(A\cap B)
Inclusion–exclusion from Chapter 7.1, in probability form. Drawing one card: P(\text{heart or king}) = \frac{13}{52}+\frac{4}{52}-\frac{1}{52} = \frac{16}{52}. Subtract the king of hearts, counted twice.
3. Independence, and what it really requires
Two events are independent when one happening tells you nothing about the other:
P(A\cap B) = P(A)\times P(B)
Two coin flips are independent. Two cards drawn without replacement are not — taking an ace changes what remains.
Two mistakes that cost real money
The gambler's fallacy. A roulette wheel has come up red eight times. Surely black is due?
No. The wheel has no memory. Each spin is independent, and the chance of black is exactly what it always was. In 1913 at the Monte Carlo casino, black came up 26 times in a row, and gamblers lost millions betting ever larger sums on red, reasoning that the streak must break.
The prosecutor's fallacy — assuming independence that is not there. In 1999 Sally Clark was convicted in England of murdering her two infant sons, largely on expert testimony that the chance of two cot deaths in one family was 1 in 73 million. That figure came from squaring the rate for a single death.
Squaring assumes independence, and cot deaths within a family are not independent — shared genetics, shared environment, shared sleeping arrangements all correlate them. The true figure was very much smaller. She spent three years in prison, the conviction was overturned in 2003, the Royal Statistical Society issued a public statement about the misuse, and she died in 2007. Assuming independence without checking it is not a technicality.
4. Conditional probability
P(A\mid B) = \frac{P(A\cap B)}{P(B)}
Read as "the probability of A given B". You have learned that B happened, so the sample space shrinks to B, and you ask what fraction of that smaller world also has A.
Worked example. A die shows an even number. What is the chance it is greater than 3?
Given evens \{2,4,6\}, the ones above 3 are \{4,6\}. So \frac23. Without the information it would have been \frac12.
Rearranged, this gives the multiplication rule:
P(A\cap B) = P(B)\,P(A\mid B)
Worked example. Two cards from a deck, both aces:
\frac{4}{52}\times\frac{3}{51} = \frac{1}{221}
The second fraction is conditional — one ace is gone and so is one card.
And independence is now definable properly: A and B are independent exactly when P(A\mid B) = P(A), meaning knowing B changes nothing.
5. The law of total probability
Break a problem into exhaustive, non-overlapping cases and add the weighted pieces:
P(A) = \sum_i P(A\mid B_i)\,P(B_i)
Worked example. A factory has two machines. Machine A makes 60% of items with a 2% defect rate; machine B makes 40% with a 5% rate. What fraction of output is defective?
P(D) = 0.02(0.6) + 0.05(0.4) = 0.012+0.020 = 0.032
3.2%. The technique is: condition on which case you are in, solve the easy conditional problem, and average with the case probabilities as weights. It is the standard way to break any messy probability question into manageable pieces, and Chapter 7.3 runs it backwards to get Bayes' theorem.
6. The Monty Hall problem
Three doors. A car behind one, goats behind two. You pick door 1. The host, who knows where the car is, opens door 3 to reveal a goat and offers you the switch to door 2.
Should you switch?
Yes. Switching wins two thirds of the time.
This problem generated thousands of angry letters to Marilyn vos Savant when she published the correct answer in 1990, including from mathematicians with doctorates. Paul Erdős, one of the most prolific mathematicians of the century, reportedly refused to accept it until shown a simulation.
The clearest way to see it. Your initial pick has a \frac13 chance of being right. So there is a \frac23 chance the car is behind one of the other two doors. The host then removes the wrong one from that pair, using his knowledge. The whole \frac23 therefore concentrates on the single remaining door.
Case by case, and this is worth reading slowly:
- Car behind door 1 (chance \frac13): you picked right. Switching loses.
- Car behind door 2 (chance \frac13): the host must open door 3. Switching wins.
- Car behind door 3 (chance \frac13): the host must open door 2. Switching wins.
Two of three cases win by switching.
The key detail everyone glosses over: the host's knowledge. If the host opened a door at random and it happened to show a goat, the probabilities really would be 50-50. The host's information leaks into the game through which door he chooses to open, and that leak is the entire effect. Change the host's rule and you change the answer, which is why the problem must be stated precisely.
Scale it up if it still resists. A hundred doors. You pick one. The host opens 98 others, all goats. Do you switch to the single remaining door? Obviously — your first pick had a 1% chance, and the host has just concentrated the other 99% onto one door.
7. Odds, and how to read them
Gamblers use odds rather than probability. Odds of a to b in favour means the probability is \frac{a}{a+b}.
Odds of 3:1 against means winning with probability \frac14.
Converting a probability to odds: \frac{p}{1-p}. A probability of 0.75 is odds of 3:1 on.
Why this matters beyond betting. The odds ratio is the standard measure in medical statistics, and the log odds is what logistic regression predicts — the model in Volume I, 12.3 that classifies emails as spam or estimates a probability of default outputs log odds, then converts with the logistic function of Chapter 6.1.
Bookmakers' odds are not probabilities. If a bookmaker offers odds implying probabilities that sum to 110%, the extra 10% is their margin — the overround. Always convert quoted odds to implied probabilities and add them up; the excess over 100% is what the house takes.
Every formula above, built from scratch
None of the results in this chapter are worth memorising, because each one can be rebuilt in under a minute from something simpler. What follows is that rebuilding, one result at a time, so the formula and the reason for it sit on the same page as the explanation that needed them.
Probability
P(A) = \frac{\text{outcomes in } A}{\text{total outcomes}} \quad \text{when all outcomes are equally likely}
0 \le P(A)\le 1, \qquad P(\text{everything}) = 1, \qquad P(A') = 1 - P(A)
P(A\cup B) = P(A)+P(B)-P(A\cap B)
Why the subtraction. Adding P(A) and P(B) counts the overlap twice, once in each. Removing it once fixes the double count — the same argument as inclusion–exclusion above.
P(A\cap B) = P(A)P(B|A) = P(B)P(A|B)
P(A|B) = \frac{P(A\cap B)}{P(B)}
Read the conditional aloud. "The probability of A given B." What it does: it throws away every outcome where B did not happen and rescales what remains so the probabilities add to 1 again. The division by P(B) is that rescaling.
\text{Independent} \iff P(A\cap B) = P(A)P(B) \iff P(A|B) = P(A)
Independence means learning B tells you nothing about A. It is a property to be checked, not assumed, and assuming it wrongly is the single most expensive error in applied probability — the subject of the Sally Clark case in 7.3.
Total probability and Bayes
P(B) = \sum_i P(B|A_i)P(A_i)
Split the world into cases that cover everything and do not overlap, work out the probability within each case, and weight by how likely each case is.
P(A|B) = \frac{P(B|A)P(A)}{P(B)} = \frac{P(B|A)P(A)}{P(B|A)P(A)+P(B|A')P(A')}
Where Bayes' theorem comes from — two lines. The intersection can be written two ways:
P(A\cap B) = P(A|B)P(B) \quad\text{and}\quad P(A\cap B) = P(B|A)P(A)
Set them equal and divide by P(B). That is the whole proof. Everything remarkable about the theorem is in what it means, not in its derivation.
What it means. It reverses a conditional. You know how often a test is positive when the disease is present; the theorem tells you how often the disease is present when the test is positive. Those two numbers can differ enormously, and the difference is the base rate P(A). A test that is 99% accurate for a disease affecting 1 in 10,000 people gives a positive result that is wrong about 99% of the time — worked out in full in 7.P.
8. Where this shows up in your life
Every insurance premium. Expected payout times probability, plus a margin.
Every medical test result, which is Chapter 7.3 and is more counterintuitive than anything here.
Every reliability figure in engineering. A system with 100 components each 99.9% reliable is 0.999^{100} = 90.5\% reliable overall, which is why redundancy exists. Volume I, 10.9.
Every weather forecast percentage. The ensemble fraction from Chapter 6.3.
Every risk assessment and every game you play.
Every quality-control decision. The factory example of Section 5 is exactly how defect rates are budgeted.
Everything so far has computed forwards: given the causes, find the chance of the effect. The question that actually matters in medicine, law and machine learning runs the other way — given the effect, what were the causes? That is Bayes.