Appearance
7.5 — The Distribution Zoo
There are infinitely many possible probability distributions and about eight that describe most of what you will ever meet. Each one arises from a specific mechanism, and the skill worth having is recognising the mechanism, because that tells you the distribution without any fitting.
This chapter is organised around those mechanisms rather than around formulas.
1. Uniform: no reason to prefer any outcome
Discrete uniform. Every one of n outcomes equally likely. A fair die, a fair coin, a randomly chosen index.
P(X=k) = \frac1n, \qquad E[X] = \frac{n+1}{2}
Continuous uniform on [a,b]. Constant density \frac{1}{b-a}.
E[X] = \frac{a+b}{2}, \qquad \operatorname{Var}(X) = \frac{(b-a)^2}{12}
Where it appears. Rounding errors are uniform over half a unit either way. Every random number generator produces uniform values first, and every other distribution is generated by transforming them. A general technique: if U is uniform on [0,1] and F is any cumulative distribution function, then F^{-1}(U) has that distribution. This is inverse transform sampling, and it is how a simulation produces samples from anything.
2. Bernoulli and binomial: counting successes
Bernoulli is a single yes/no trial with success probability p. Mean p, variance p(1-p).
Note that the variance is largest at p = 0.5 and shrinks towards zero at either extreme. A fair coin is the most unpredictable coin, which is obvious in hindsight and is the seed of Shannon's entropy in Volume I, 1.8.
Binomial counts successes in n independent Bernoulli trials, all with the same p:
P(X=k) = \binom{n}{k}p^k(1-p)^{n-k}
Read the three factors: p^k for the successes, (1-p)^{n-k} for the failures, and \binom nk from Chapter 7.1 for the number of orders in which they could occur.
E[X] = np, \qquad \operatorname{Var}(X) = np(1-p)
The four conditions, and all four must hold: a fixed number of trials, exactly two outcomes each, constant probability, and independence.
Worked example. A component has a 2% failure rate. In a batch of 50, what is the chance of at least one failure?
Use the complement rule from Chapter 7.2:
P(\ge 1) = 1 - P(0) = 1 - (0.98)^{50} = 1 - 0.364 = 0.636
A 64% chance of at least one failure, from a 2% part. This is why component reliability must be so high when systems have many parts, and Volume I, 10.9 works through the consequences.
Where it appears. Defect counts, conversion rates in A/B tests, poll results, the number of packets lost, how many patients respond to a treatment.
3. Poisson: rare events in a continuous stream
Events happen at a steady average rate, independently, and you count how many occur in a fixed interval:
P(X=k) = \frac{\lambda^k e^{-\lambda}}{k!}
where \lambda is the average count per interval.
E[X] = \lambda, \qquad \operatorname{Var}(X) = \lambda
The mean equals the variance, which is a distinctive signature. If your count data has variance much larger than its mean, it is not Poisson, and the usual reason is that events are clustering rather than arriving independently.
Where it comes from. It is the binomial with a huge number of trials and a tiny probability each, with np = \lambda held fixed. Any situation with many opportunities and a small chance each produces it.
Worked example. A website gets 3 requests per second on average. What is the chance of 6 or more in a given second?
P(X \le 5) = \sum_{k=0}^{5}\frac{3^ke^{-3}}{k!} = 0.916
so P(\ge6) = 0.084, about 8%.
This is why capacity planning cannot use averages. A server sized for 3 requests per second will be overloaded 8% of the time by bursts. Volume I, 10.12 covers the estimation practice, and the arithmetic here is why headroom is not optional.
Where it appears. Arrivals at a queue, calls to a call centre, radioactive decay counts, typing errors per page, goals in a football match, mutations per genome, and the original 1898 study by Bortkiewicz of Prussian cavalrymen killed by horse kicks — which fits beautifully and gave the distribution its early fame.
4. Geometric and exponential: waiting for the first event
Geometric counts trials until the first success:
P(X=k) = (1-p)^{k-1}p, \qquad E[X] = \frac1p
Expect 6 rolls to get a six, 2 flips to get a head.
Exponential is the continuous version — the waiting time until the next Poisson event:
f(t) = \lambda e^{-\lambda t}, \qquad E[T] = \frac1\lambda
Both are memoryless, which is their defining and most counterintuitive property:
P(T \gt s+t \mid T \gt s) = P(T \gt t)
Having already waited does not change how much longer you will wait. A component with exponential lifetime that has survived 5 years has exactly the same remaining life expectancy as a new one. It does not age.
This is true for some things and false for most. Radioactive nuclei genuinely do not age — a caesium atom's decay chance is the same today as a million years ago. Electronic components in their useful life approximate it. Mechanical parts and living things do not — they wear out, so their failure rate rises with age, and their lifetimes follow a Weibull distribution instead, which has an extra shape parameter precisely to capture wear-out.
Assuming exponential lifetimes for something that wears out will badly underestimate late-life failures, and it is a real and expensive modelling error.
5. Normal: the sum of many small effects
f(x) = \frac{1}{\sigma\sqrt{2\pi}}e^{-\frac{(x-\mu)^2}{2\sigma^2}}
The bell curve. It gets its own chapter next, because it deserves it — and because \pi appearing in a statistics formula needs the explanation from Chapter 5.7.
6. Power laws: when the average is meaningless
P(X \gt x) \propto x^{-\alpha}
A very different shape. The normal distribution has thin tails — values far from the mean are essentially impossible. Power laws have fat tails, where extreme values are rare but not negligible, and they change everything.
Where they appear: city populations, wealth, book and record sales, website traffic, earthquake magnitudes, word frequencies (Zipf's law), file sizes, degrees in a social network, and the size of losses in finance.
Why they behave so differently. Human height is normal. The tallest person ever was 2.72 m, about 1.5 times the average — and no matter how many people you sample, nobody is 10 m tall. Wealth follows a power law, and the richest person is millions of times the average.
With a fat-tailed distribution, one sample can dominate the entire dataset. Take a hundred people at random and the average height barely moves if you add one more. Add the richest person in the world to a hundred random people and the average wealth changes by orders of magnitude.
Practical consequences. The mean is a poor summary and the median is better. Sample averages converge slowly or not at all — for \alpha \le 2 the variance is infinite, and for \alpha \le 1 even the mean does not exist. Risk models built on normal assumptions catastrophically underestimate extreme events, which is a significant part of what went wrong in 2008: the models treated once-in-ten-thousand-year moves as impossible, and they happened repeatedly.
The 80/20 rule is a power law. It shows up as 20% of customers producing 80% of revenue, 20% of code containing 80% of bugs, 20% of pages getting 80% of traffic — Volume I, 14.5 uses it to argue for profiling before optimising.
7. Choosing the right one
| Situation | Distribution |
|---|---|
| Equally likely outcomes | uniform |
| One yes/no trial | Bernoulli |
| Successes in n trials | binomial |
| Count of rare events per interval | Poisson |
| Trials until first success | geometric |
| Time until next event | exponential |
| Sum of many small effects | normal |
| Winner-takes-most, extreme values matter | power law |
Ask the mechanism, not the shape of the histogram. "How was this number generated?" identifies the distribution far more reliably than eyeballing a plot, and it tells you when the assumptions break.
Every formula above, built from scratch
None of the results in this chapter are worth memorising, because each one can be rebuilt in under a minute from something simpler. What follows is that rebuilding, one result at a time, so the formula and the reason for it sit on the same page as the explanation that needed them.
The discrete distributions
Bernoulli — one trial
P(X=1) = p, \quad P(X=0)=1-p, \qquad E[X]=p, \qquad \operatorname{Var}(X)=p(1-p)
Derivation of the variance. E[X^2] = 1^2\cdot p + 0^2(1-p) = p, so \operatorname{Var} = p - p^2 = p(1-p).
Note that p(1-p) is largest at p = 0.5. Maximum uncertainty is at 50/50, and a near-certain event has almost no variance.
Binomial — n independent trials
P(X=k) = \binom nk p^k(1-p)^{n-k}, \qquad E[X]=np, \qquad \operatorname{Var}(X)=np(1-p)
Where the formula comes from, piece by piece. One specific sequence with k successes and n-k failures has probability p^k(1-p)^{n-k}, by independence. There are \binom nk such sequences, since the successes could fall in any k of the n positions. Multiply.
Where the mean comes from without any algebra. A binomial variable is the sum of n Bernoulli variables. Expectations add, so E[X] = np. The variables are independent, so variances add too, giving np(1-p). Building a distribution out of simpler pieces is almost always easier than summing the series directly.
Geometric — waiting for the first success
P(X=k) = (1-p)^{k-1}p, \qquad E[X] = \frac1p, \qquad \operatorname{Var}(X) = \frac{1-p}{p^2}
Where the mean comes from, by a trick worth knowing. Let \mu = E[X]. Consider the first trial. With probability p it succeeds and we are done after 1 trial. With probability 1-p it fails, and we are back where we started having used one trial. So
\mu = p(1) + (1-p)(1+\mu)
Expand: \mu = p + 1 - p + \mu - p\mu = 1 + \mu - p\mu. Cancel \mu from both sides:
0 = 1 - p\mu \quad \Rightarrow \quad \mu = \frac1p
No infinite series needed. This is called conditioning on the first step, and it works on many waiting-time problems.
The memoryless property. P(X>m+n\mid X>m) = P(X>n). A coin that has failed to come up heads twenty times is no more likely to do so on the next throw. The distribution has no memory of how long you have waited, and the geometric is the only discrete distribution with that property.
Poisson — rare events in a fixed interval
P(X=k) = \frac{\lambda^ke^{-\lambda}}{k!}, \qquad E[X]=\operatorname{Var}(X)=\lambda
Where it comes from. Take a binomial with a huge n and a tiny p, holding np = \lambda fixed — many opportunities, each very unlikely. Substituting p = \frac\lambda n into the binomial formula and letting n\to\infty gives, term by term, exactly \frac{\lambda^ke^{-\lambda}}{k!}. The e^{-\lambda} arrives from \left(1-\frac\lambda n\right)^n \to e^{-\lambda}, which is the limit that defined e in 1.4 — powers roots logarithms.
The signature. The mean and the variance are equal. If your count data has variance far exceeding its mean, the events are not independent and a Poisson model is the wrong one — that is called overdispersion, and noticing it is what stops a bad model shipping.
Hypergeometric — drawing without replacement
P(X=k) = \frac{\binom Kk\binom{N-K}{n-k}}{\binom Nn}
Choose k of the K successes and n-k of the N-K failures, out of all \binom Nn possible draws. Use it when the population is small enough that removing an item changes the odds — a card deck, a quality-control batch. When N is large the answer converges to the binomial.
The continuous distributions
For continuous variables, probability is area:
P(a\le X\le b) = \int_a^b f(x)\,dx, \qquad \int_{-\infty}^\infty f(x)dx = 1, \qquad P(X = a) = 0
The last one is not a paradox. A single point has zero width, so zero area. Ask instead for a range.
F(x) = P(X\le x) = \int_{-\infty}^x f(t)\,dt, \qquad f(x) = F'(x)
Uniform
f(x) = \frac{1}{b-a} \text{ on } [a,b], \qquad E[X]=\frac{a+b}{2}, \qquad \operatorname{Var}(X)=\frac{(b-a)^2}{12}
Where the variance comes from. E[X^2] = \int_a^b\frac{x^2}{b-a}dx = \frac{b^3-a^3}{3(b-a)} = \frac{a^2+ab+b^2}{3}, using the difference-of-cubes factorisation from 2.2 — polynomials. Subtract \left(\frac{a+b}{2}\right)^2 and simplify to get \frac{(b-a)^2}{12}.
Exponential — waiting time for a Poisson event
f(x) = \lambda e^{-\lambda x} \ (x\ge0), \qquad E[X]=\frac1\lambda, \qquad \operatorname{Var}(X)=\frac{1}{\lambda^2}, \qquad P(X>t) = e^{-\lambda t}
Where P(X>t) = e^{-\lambda t} comes from, without integrating. "No event before time t" is the Poisson probability of zero events in an interval of length t, whose mean count is \lambda t:
P(0 \text{ events}) = \frac{(\lambda t)^0e^{-\lambda t}}{0!} = e^{-\lambda t}
The exponential is the continuous cousin of the geometric, and it is memoryless for the same reason: a component with a constant failure rate is no more likely to fail tomorrow because it has already lasted a year.
The normal distribution
f(x) = \frac{1}{\sigma\sqrt{2\pi}}\exp\left(-\frac{(x-\mu)^2}{2\sigma^2}\right)
Read the formula in pieces. The (x-\mu)^2 says only the distance from the centre matters, and symmetrically. The minus sign in the exponent makes the height fall as you move away. Dividing by \sigma^2 sets the width. The \frac{1}{\sigma\sqrt{2\pi}} out front is exactly what is needed to make the total area 1, and the \sqrt{2\pi} appearing there is one of the genuine surprises of mathematics.
Z = \frac{X-\mu}{\sigma} \quad \Rightarrow \quad Z \sim N(0,1)
Why standardising works. Subtracting \mu slides the centre to zero; dividing by \sigma rescales the spread to 1. By the rules of §3, E[Z] = 0 and \operatorname{Var}(Z) = \frac{\sigma^2}{\sigma^2} = 1. One table then serves every normal distribution there is.
P(\mu-\sigma<X<\mu+\sigma) = 68.3\%, \qquad \pm2\sigma: 95.4\%, \qquad \pm3\sigma: 99.7\%
The central limit theorem
\bar X \approx N\left(\mu, \frac{\sigma^2}{n}\right) \quad \text{for large } n
What it says. Take any distribution with a finite mean and variance — skewed, lumpy, discrete, it does not matter. Average n independent draws from it. As n grows, the distribution of that average becomes normal, centred on the true mean, with standard deviation
\text{SE} = \frac{\sigma}{\sqrt n}
Why the \sqrt n. The variance of a sum of n independent copies is n\sigma^2. Dividing the sum by n to get the average divides the variance by n^2, giving \frac{n\sigma^2}{n^2} = \frac{\sigma^2}{n}. Take the square root for the standard deviation.
The consequence everybody should know. To halve your uncertainty you need four times the data. That single fact governs the cost of every poll, every clinical trial and every A/B test ever run.
8. Where this shows up in your life
Every queue you stand in. Poisson arrivals and exponential service times are the foundation of queueing theory, which sizes call centres, checkout lanes, hospital wards and server pools.
Every reliability specification. Exponential during useful life, Weibull for wear-out.
Every A/B test. Binomial conversions.
Every insurance and finance risk model — and the disasters that come from assuming normal when the truth is a power law.
Every capacity plan. Section 3's arithmetic, and the reason the answer is never "average load".
Every discussion about inequality. Wealth is power-law distributed, which is why the mean and median diverge so dramatically and why quoting the average income tells you almost nothing.
One distribution appears more than all the others combined, and there is a theorem explaining exactly why. It is arguably the most important result in probability.