Appearance
7.7 — Describing Data and Estimating What You Cannot Measure
Probability runs forwards: given the mechanism, what data would we see? Statistics runs backwards: given the data, what was the mechanism?
The backwards direction is harder, because many mechanisms could produce the same data, and because the answer is never certain — only more or less well supported. This chapter covers describing what you have and estimating what you do not.
1. Populations and samples
The population is everyone or everything you care about. The sample is the part you actually measured.
You want to know the average income in India. You cannot ask 1.4 billion people, so you ask 5,000 and generalise. The entire discipline of statistics is about how much you are entitled to conclude from that leap.
The vocabulary distinguishes them carefully:
| Quantity | Population (unknown truth) | Sample (what you computed) |
|---|---|---|
| Mean | \mu | \bar x |
| Standard deviation | \sigma | s |
| Proportion | p | \hat p |
Greek for the truth, Latin for the estimate. The sample values are known and vary from sample to sample; the population values are fixed and unknown. Keeping this straight prevents most confusion about what a confidence interval means.
Sampling bias beats sample size, every time
In 1936 the Literary Digest polled 2.4 million people and predicted Alf Landon would beat Franklin Roosevelt in a landslide. Roosevelt won 46 of 48 states.
Their sample came from telephone directories, club memberships and car registrations — which in 1936 meant the wealthy. The sample was enormous and systematically wrong.
George Gallup polled 50,000 with attention to representativeness and got it right. The magazine folded within two years.
A biased sample of a million is worse than a random sample of a thousand, because the large sample gives you false confidence in a wrong answer. Every internet poll, every "customers who responded to our survey", every dataset of "users who opted in" carries this risk. Ask who is missing before you ask how many responded.
2. Describing a dataset
Centre.
The mean is the arithmetic average. Sensitive to extreme values.
The median is the middle value when sorted. Insensitive to extremes.
The mode is the most common value. The only one that works for categories.
Which to use. For symmetric data they agree and the mean is fine. For skewed data they diverge sharply and the median is almost always the honest summary. Average UK household income is considerably higher than median household income, because a small number of very high incomes pull the mean up. Anyone quoting the mean for income, house prices or wealth is either being careless or making a point.
A useful rule: if the mean and median differ substantially, the distribution is skewed and you should say so rather than picking whichever number suits you.
Spread.
The range is max minus min — simple and dominated by outliers.
The interquartile range is the middle 50%: the third quartile minus the first. Robust.
The standard deviation is Chapter 7.4's, and it is the standard for symmetric data.
Why the sample standard deviation divides by n−1
s^2 = \frac{1}{n-1}\sum(x_i - \bar x)^2
That n-1 looks like a mistake. It is called Bessel's correction and it fixes a real bias.
The deviations are measured from the sample mean, and the sample mean is itself the point that minimises the sum of squared deviations for this sample. So the deviations from it are systematically a little too small — smaller than the deviations from the true population mean would have been.
Dividing by n-1 instead of n inflates the result by exactly the right amount to compensate. The technical phrasing is that you used up one degree of freedom estimating the mean, leaving n-1 independent pieces of information about spread.
Concretely: with n=2, the two deviations from their own mean are equal and opposite, so knowing one determines the other. There is only one independent piece of information, and dividing by 1 rather than 2 acknowledges that.
Shape.
Skewness measures asymmetry. Positive skew means a long right tail (income). Negative means a long left tail (age at death in a developed country).
Kurtosis measures tail heaviness. High kurtosis means more extreme values than a normal distribution would give — the fat tails of Chapter 7.5.
The five-number summary — minimum, first quartile, median, third quartile, maximum — is what a box plot draws, and it conveys centre, spread and skew at a glance without assuming anything about the distribution.
Always plot your data
Anscombe's quartet is four small datasets with identical means, variances, correlations and regression lines. Plotted, they are completely different: one is a clean linear relationship, one is a curve, one is a perfect line with a single outlier, and one is a vertical stack plus one distant point.
Summary statistics are a compression, and compression loses information. Look at the picture before you trust the numbers.
3. Estimation
A point estimate is a single best guess: \bar x for \mu, \hat p for p.
An estimator is unbiased if it is right on average across many samples. The sample mean is unbiased for the population mean; the sample variance with n-1 is unbiased for the population variance, which is exactly what Bessel's correction achieves.
But a point estimate alone is close to useless, because it says nothing about how much to trust it. "The average is 52%" from a sample of 10 and from a sample of 10,000 are very different claims.
4. Confidence intervals
The standard error, from Chapter 7.6's central limit theorem:
SE = \frac{s}{\sqrt n}
This measures how much the sample mean would bounce around if you repeated the whole study. A 95% confidence interval is roughly
\bar x \pm 1.96\times SE
The 1.96 comes from the normal distribution: 95% of the area lies within 1.96 standard deviations of the centre.
Worked example. 400 people are surveyed and 220 support a policy, so \hat p = 0.55. For a proportion, SE = \sqrt{\frac{p(1-p)}{n}} = \sqrt{\frac{0.55\times0.45}{400}} = 0.0249.
0.55 \pm 1.96(0.0249) = 0.55\pm0.049
Between 50.1% and 59.9% — and this is where the newspaper phrase "margin of error of about 5 percentage points" comes from.
What a confidence interval does NOT mean
It does not mean "there is a 95% probability the true value is in this interval". Under the frequentist view of Chapter 7.3, the true value is a fixed number — it is either in your interval or it is not, with no probability about it.
What it means: if you repeated the entire study many times, 95% of the intervals you constructed this way would contain the true value. The 95% describes the procedure, not this particular interval.
If you want the sentence everyone actually wants to say, you need the Bayesian version, which is called a credible interval and does license it.
This distinction is subtle and is misstated in most media coverage. It is worth knowing so that you understand what you are being told.
Sample size. Since the error goes as \frac{1}{\sqrt n}, halving the margin of error requires four times the sample. To reach \pm1\% on a proportion you need about 10,000 respondents. This is why national polls cluster around 1,000 people — that gives roughly \pm3\%, and going further gets expensive fast for diminishing returns.
A striking consequence that people find hard to believe: the required sample size barely depends on the population size. A sample of 1,000 gives about the same precision for a city of 100,000 as for a country of a billion, because n appears in the formula and the population size does not. What matters is that the sample is representative, not what fraction of the population it is.
5. Maximum likelihood
The general principle for constructing an estimator: choose the parameter value that makes the observed data most probable.
Worked example. A coin lands heads 7 times in 10. What is the most likely value of p?
L(p) = \binom{10}{7}p^7(1-p)^3
Maximise it. Taking the logarithm first makes the calculus easier, and works because the logarithm is increasing so it does not move the maximum:
\frac{d}{dp}\left[7\ln p + 3\ln(1-p)\right] = \frac7p - \frac{3}{1-p} = 0
7(1-p) = 3p \quad\Longrightarrow\quad p = 0.7
The answer is the obvious one, which is reassuring. The point is that the same method works when the answer is not obvious — and it is the standard route to estimators throughout statistics and machine learning. Training a model by minimising cross-entropy loss (Volume I, 12.4) is maximum likelihood estimation, and the equivalence is exact.
6. Where statistics goes wrong
Simpson's paradox. A trend that appears in every subgroup can reverse when the groups are combined.
The 1973 Berkeley graduate admissions case is the standard example. Overall, men were admitted at a higher rate than women, and it looked like clear discrimination. Department by department, most departments admitted women at a higher rate.
Both were true. Women applied disproportionately to departments with low admission rates across the board, and men to departments that admitted most applicants. The aggregate comparison mixed together the effect of department choice with the effect of gender.
The lesson: an aggregate can lie about every one of its parts. Whenever you compare two groups, ask what else differs between them.
Survivorship bias. In the Second World War, Abraham Wald was asked where to add armour to bombers, based on the damage pattern on returning aircraft. The obvious answer was to armour the places with the most bullet holes.
Wald said the opposite: armour the places with no holes. The data came only from planes that made it back. Hits to the engines and cockpit were absent from the data not because they did not happen but because those aircraft did not return.
The same reasoning applies to every study of successful companies, successful people or successful investments. The failures are not in your dataset, and they are the comparison you need.
Regression to the mean, from Chapter 7.6, which manufactures apparent treatment effects out of nothing.
Correlation and causation. Chapter 7.4 covered it, and it remains the most-repeated and least-heeded warning in the subject.
Every formula above, built from scratch
None of the results in this chapter are worth memorising, because each one can be rebuilt in under a minute from something simpler. What follows is that rebuilding, one result at a time, so the formula and the reason for it sit on the same page as the explanation that needed them.
Statistics: estimating from a sample
\bar x = \frac{1}{n}\sum x_i, \qquad s^2 = \frac{1}{n-1}\sum(x_i-\bar x)^2
Why n-1 and not n. The deviations are measured from \bar x, which was itself computed from the same data. The sample mean sits, by construction, in the middle of the data, so the deviations from it are slightly smaller than the deviations from the true mean would have been. Dividing by n would therefore underestimate the spread every time. Dividing by n-1 corrects it exactly. The intuition: with one data point you have no information about spread at all, and n-1 = 0 correctly refuses to give you a number.
\text{Confidence interval for a mean}: \quad \bar x \pm t^*\frac{s}{\sqrt n}
Read it. The estimate, plus or minus a multiple of the standard error. The multiplier t^* comes from the t distribution with n-1 degrees of freedom, and it is about 1.96 for 95% confidence with a large sample.
What a 95% confidence interval actually means. If you repeated the whole study many times, 95% of the intervals you built this way would contain the true value. It does not mean there is a 95% chance the true value is in this particular interval — the true value is not random, the interval is.
\text{For a proportion}: \quad \hat p\pm z^*\sqrt{\frac{\hat p(1-\hat p)}{n}}
Maximum likelihood estimation. Write the probability of the observed data as a function of the unknown parameter, then choose the parameter making that as large as possible. Usually one takes the logarithm first — turning the product into a sum, which is much easier to differentiate — and the answer is the same, since \ln is increasing.
7. Where this shows up in your life
Every poll and survey you read. Margin of error, and the question of who was sampled.
Every "average salary" figure. Mean or median, and the answer matters enormously.
Every quality report and dashboard. Percentiles rather than means, for the reasons in Chapter 7.4.
Every medical study. Sample size, confidence intervals, control groups, and regression to the mean.
Every claim about a successful strategy. Survivorship bias is the first thing to check.
Every machine learning model's training. Maximum likelihood, under a different name.
Estimation says what a quantity probably is. The other half of statistics asks a sharper question: is this effect real, or is it noise? That question is answered by hypothesis testing, and it has been getting answered badly for a century.