Appearance
7.4 — Random Variables, Expectation and Variance
"The probability of heads is one half" is a statement about an event. "The average number of heads in 100 flips is 50, and it is usually within 10 of that" is a statement about a number that varies, and it is far more useful.
A random variable is a number whose value depends on a random outcome. Height of a randomly chosen person, number of defective items in a batch, time until a server fails, tomorrow's closing price. The convention is a capital letter for the variable, X, and a small letter for a particular value it might take.
1. Discrete and continuous
Discrete random variables take separate, countable values — usually whole numbers. Number of heads, number of customers, number of defects.
Each value has a probability, and the list of them is the probability mass function:
P(X = x)
For one die: P(X=k) = \frac16 for k=1 to 6. The probabilities must sum to 1.
Continuous random variables take any value in a range. Height, time, temperature, weight.
Here something changes, and it catches people out: the probability of any exact value is zero. The probability that someone is exactly 1.75000... metres tall, to infinite precision, is zero. There are uncountably many possible heights (Chapter 1.7) and they cannot each have positive probability.
So instead of a probability per value, there is a probability density function f(x), and probability is the area under it:
P(a \le X \le b) = \int_a^b f(x)\,dx
with the total area equal to 1. This is Chapter 5.5's integral doing real work, and it is why "probability of an interval" is the only meaningful question for a continuous variable.
Probability zero does not mean impossible
Pick a real number uniformly at random from [0,1]. Every specific number has probability zero, and yet one of them occurs.
The resolution: for continuous variables, "probability zero" means "contributes no area", not "cannot happen". Chapter 1.7 showed that the rationals, though infinite and dense, are a vanishingly thin scattering in the reals — and correspondingly, picking a random real gives a rational with probability zero, though rationals are certainly there to be picked.
This is not a paradox once you accept that measure and possibility are different things.
The cumulative distribution function works for both kinds:
F(x) = P(X \le x)
It always runs from 0 up to 1 and never decreases. For continuous variables it is the integral of the density, so by the Fundamental Theorem of Calculus the density is its derivative.
2. Expectation
The expected value is the long-run average:
E[X] = \sum_i x_i P(X=x_i) \qquad\text{or}\qquad E[X] = \int_{-\infty}^{\infty}xf(x)\,dx
It is a weighted average — each value weighted by how likely it is.
One die:
E[X] = \frac{1+2+3+4+5+6}{6} = 3.5
Note that 3.5 is not a possible outcome. The expected value need not be a value the variable can take, which is why "expected" is a slightly misleading name. It is the balance point of the distribution, in the literal sense: cut the probability mass function out of card and it balances on a pin at the mean.
Two properties do enormous work.
E[aX+b] = aE[X]+b
E[X+Y] = E[X]+E[Y] \quad\textbf{always, even if } X \text{ and } Y \text{ are dependent}
That second one is called linearity of expectation, and its unconditional truth is startling and extremely useful.
Worked example. A hundred people leave their hats at a party, and the hats are returned at random. How many people get their own hat back, on average?
The joint distribution here is horrible — whether one person gets their hat affects everyone else. But define X_i = 1 if person i gets their hat and 0 otherwise. Then E[X_i] = \frac{1}{100}, since each person is equally likely to receive any hat.
E[\text{total}] = \sum_{i=1}^{100}E[X_i] = 100\times\frac{1}{100} = 1
Exactly one person, on average, regardless of the number of guests. No messy dependency analysis was needed, because linearity does not care.
Expectation decides whether a bet is worth taking
European roulette. 37 pockets, betting on a single number pays 35 to 1.
E = 35\times\frac{1}{37} + (-1)\times\frac{36}{37} = \frac{35-36}{37} = -\frac{1}{37} = -0.027
You lose 2.7 paise per rupee staked, on average, forever. No system changes this, because expectation is linear — a sequence of bets each with negative expectation has negative expectation whatever their sizes or order.
American roulette adds a double zero, making 38 pockets and an expectation of -0.053 — twice as bad, for a game that looks identical.
Insurance has negative expectation too, and buying it is still rational. If it did not, the insurer would go bankrupt. You buy it because the utility of losing a small certain amount is much better than the utility of a small chance of catastrophic loss. Expected money and expected value-to-you are different quantities, and Section 5 says more.
3. Variance and standard deviation
Two distributions can share a mean and be completely different. Always winning ₹100 and a coin flip between ₹0 and ₹200 both average ₹100.
Variance measures spread — the average squared distance from the mean:
\operatorname{Var}(X) = E[(X-\mu)^2] = E[X^2] - (E[X])^2
The second form is usually easier to compute.
Why squared? Because plain deviations average to zero — the positives and negatives cancel exactly, by the definition of the mean. Squaring makes everything positive. Absolute values would also work and give the mean absolute deviation, which is used sometimes; squares win because they are differentiable everywhere (Chapter 5.2 noted that |x| is not) and because they make the algebra come out clean, especially the addition rule below.
Standard deviation is the square root of the variance, written \sigma. It is preferred for reporting because it is in the same units as the data: if X is in rupees, the variance is in rupees-squared and the standard deviation is in rupees.
One die: E[X^2] = \frac{1+4+9+16+25+36}{6} = \frac{91}{6} = 15.17, so \operatorname{Var} = 15.17 - 12.25 = 2.92 and \sigma = 1.71.
The rules, and note the difference from expectation:
\operatorname{Var}(aX+b) = a^2\operatorname{Var}(X)
Adding a constant shifts the distribution without spreading it, so b vanishes. Scaling by a scales the spread by a and therefore the variance by a^2.
\operatorname{Var}(X+Y) = \operatorname{Var}(X)+\operatorname{Var}(Y) \quad\textbf{only if independent}
This is the crucial asymmetry with expectation. Means always add; variances add only for independent variables. Chapter 7.2's Sally Clark case was a failure of exactly this condition, and every risk model that assumes independent failures is making the same bet.
The square-root law
Add n independent copies of a variable with mean \mu and standard deviation \sigma:
- The mean of the sum is n\mu — grows linearly.
- The variance of the sum is n\sigma^2, so the standard deviation is \sigma\sqrt n — grows only like the square root.
So the sum grows faster than its own spread, and relative variability shrinks as \frac{1}{\sqrt n}.
This single fact explains an enormous amount:
- Why casinos always win. One night's takings are highly variable; a year's are almost exactly the expected value.
- Why insurance works. Individual claims are unpredictable; a portfolio of a million policies is not.
- Why bigger samples give better estimates, and why halving your error needs four times the data. This is the most important practical consequence in all of statistics, and Chapter 7.7 develops it.
- Why diversification reduces risk. Twenty independent investments have twenty times the expected return and only \sqrt{20} \approx 4.5 times the standard deviation of one.
4. Covariance and correlation
For two variables together, covariance measures whether they move together:
\operatorname{Cov}(X,Y) = E[(X-\mu_X)(Y-\mu_Y)]
Positive means they tend to rise together, negative means one rises as the other falls, zero means no linear relationship.
Its size depends on the units, which makes it hard to interpret. Dividing it out gives the correlation coefficient:
\rho = \frac{\operatorname{Cov}(X,Y)}{\sigma_X\sigma_Y}
always between -1 and +1. This is exactly Chapter 4.1's cosine similarity applied to the centred data, which is a genuinely useful thing to know: correlation is the cosine of the angle between two centred data vectors.
Two things correlation does not tell you
Correlation is not causation. Ice cream sales correlate with drownings; both are driven by hot weather. This is the most repeated caution in statistics and it is still routinely ignored in reporting.
Zero correlation is not independence. Correlation only detects linear relationships. Take X uniform on [-1,1] and Y = X^2. They are perfectly dependent — Y is a function of X — and their correlation is exactly zero, because the relationship is symmetric and curved. Independence implies zero correlation; the reverse does not hold.
5. Expectation is not always what you should maximise
Two situations where blindly maximising expected value gives the wrong answer.
The St Petersburg paradox. Flip a coin until it lands heads. If it takes n flips, you win 2^n rupees. What is a fair price for this game?
E = \sum_{n=1}^{\infty}2^n\cdot\frac{1}{2^n} = 1+1+1+\cdots = \infty
Infinite expected value. Yet nobody would pay ₹1,000 to play, and they are right not to: the median payout is ₹2, and there is a 99.9% chance of winning less than ₹1,024.
Daniel Bernoulli's 1738 resolution was utility: what matters is not money but what the money is worth to you, and the value of an extra rupee falls as you get richer. With logarithmic utility the expected utility is finite and small. This idea founded modern economics, and it is why insurance, diversification and risk aversion are rational rather than timid.
Ruin. A bet with positive expected value can still be a bad idea if losing wipes you out, because you cannot play the long run from bankruptcy. Expectation assumes you get to repeat the game, and if a single outcome ends your participation, that assumption fails. This is why professional gamblers and traders size their bets by the Kelly criterion rather than by expectation, and why a company with a positive-expectation strategy can still be destroyed by one bad quarter.
Every formula above, built from scratch
None of the results in this chapter are worth memorising, because each one can be rebuilt in under a minute from something simpler. What follows is that rebuilding, one result at a time, so the formula and the reason for it sit on the same page as the explanation that needed them.
Random variables
E[X] = \sum_i x_iP(x_i) \qquad\text{or}\qquad E[X] = \int_{-\infty}^{\infty}xf(x)\,dx
The expected value is the long-run average: each value weighted by how often it happens.
\operatorname{Var}(X) = E\left[(X-\mu)^2\right] = E[X^2]-\left(E[X]\right)^2, \qquad \sigma = \sqrt{\operatorname{Var}(X)}
Where the second form comes from, and it is the one you actually compute with. Expand the square:
E\left[(X-\mu)^2\right] = E\left[X^2 - 2\mu X + \mu^2\right]
Expectation is linear, so split it up. Note \mu is a fixed number, not random:
= E[X^2] - 2\mu E[X] + \mu^2 = E[X^2]-2\mu^2+\mu^2 = E[X^2]-\mu^2
The rules for combining.
E[aX+b] = aE[X]+b, \qquad \operatorname{Var}(aX+b) = a^2\operatorname{Var}(X)
Why b vanishes from the variance. Adding a constant slides the whole distribution along without changing its spread. And the a^2 is there because variance is measured in squared units — doubling every value quadruples the variance but only doubles the standard deviation.
E[X+Y] = E[X]+E[Y] \qquad \text{always, even if } X \text{ and } Y \text{ are related}
\operatorname{Var}(X+Y) = \operatorname{Var}(X)+\operatorname{Var}(Y) \qquad \textbf{only if independent}
That asymmetry is worth pausing on. Expectations always add. Variances only add when the variables are independent; otherwise you need
\operatorname{Var}(X+Y) = \operatorname{Var}(X)+\operatorname{Var}(Y)+2\operatorname{Cov}(X,Y)
\operatorname{Cov}(X,Y) = E[XY]-E[X]E[Y], \qquad \rho = \frac{\operatorname{Cov}(X,Y)}{\sigma_X\sigma_Y}
The correlation \rho is the covariance rescaled to lie between -1 and 1, so it does not depend on the units. It measures straight-line association only: a perfect parabola has correlation zero, and that is not a flaw in the formula but a limit of what a single number can say.
6. Where this shows up in your life
Every insurance premium and every casino game. Expectation, on the other side of the table from you.
Every measurement error quoted as \pm something. Standard deviation.
Every portfolio's risk figure. Variance, covariance and the diversification effect of Section 3.
Every A/B test's sample size calculation. The \frac{1}{\sqrt n} law tells you how much data you need.
Every quality-control tolerance. Six Sigma is literally a statement about how many standard deviations fit between the process mean and the specification limit.
Every service level agreement about latency. Volume I, 10.9 explains why the mean is nearly useless and percentiles are what matter — the mean hides the tail, and the tail is what your users experience.
Random variables in the abstract are one thing. In practice, a small number of specific distributions describe almost everything, and knowing which one applies to a situation is most of the skill.