Appearance
10.1 — Floating Point and Error
On 4 June 1996 the Ariane 5 rocket exploded 37 seconds after launch, destroying about $370 million of payload. The cause was a 64-bit floating-point number converted to a 16-bit integer. The value did not fit, the conversion failed, the backup system had already failed identically a moment earlier, and the flight computer commanded a course correction that tore the rocket apart.
On 25 February 1991 a Patriot missile battery in Dhahran failed to intercept a Scud, and 28 soldiers were killed. The system tracked time in tenths of a second, stored in a 24-bit fixed-point register. One tenth cannot be represented exactly in binary — Chapter 1.3 explained why, because 10 has a factor of 5. After 100 hours of continuous operation the accumulated error was 0.34 seconds, which at Scud speed is about 600 metres. The missile looked in the wrong place.
Every formula in Parts 1 to 9 quietly assumes two things a computer does not have: exact numbers and infinite patience. A real number can need infinitely many digits and a machine has a fixed number of them; a limit takes infinitely many steps and a machine has to stop somewhere. This chapter is about that gap — how large the error is, how fast a method closes in on the answer, and when a perfectly correct algorithm returns nonsense. It is short and it prevents a specific and expensive class of mistake.
1. How a computer stores a real number
There are uncountably many reals (Chapter 1.7) and finitely many bit patterns, so almost every real number cannot be stored. What is stored is a nearby one.
The IEEE 754 standard, used by essentially every processor, stores a number as
(-1)^{\text{sign}}\times 1.\text{mantissa}\times2^{\text{exponent}}
A double uses 64 bits: 1 for sign, 11 for exponent, 52 for mantissa. That gives about 15 to 17 significant decimal digits and a range up to about 10^{308}.
It is scientific notation in base two, and it makes the same trade: a fixed number of significant digits, with the exponent providing range. The spacing between representable numbers is therefore proportional to their size — close together near 1, enormously far apart near 10^{300}.
Volume I, 1.4 covers the bit layout in detail. What matters here are the consequences.
2. Why 0.1 + 0.2 is not 0.3
js
0.1 + 0.2 === 0.3 // false
0.1 + 0.2 // 0.30000000000000004Not a bug in JavaScript — it happens in Python, Java, C, and every language using IEEE doubles.
In base two, \frac{1}{10} is a repeating fraction, exactly as \frac13 repeats in base ten. Chapter 1.3's rule: a fraction terminates only when its denominator's prime factors divide the base. Base two has only the prime 2, so only denominators that are powers of 2 terminate. Ten has a factor of 5, so one tenth repeats forever and must be cut off.
The stored value of 0.1 is very slightly more than one tenth. Add two such approximations and the errors accumulate into the visible range.
The rule that follows: never test floating-point numbers for exact equality. Compare against a tolerance:
js
Math.abs(a - b) < 1e-9And never store money as a floating-point number. Use integer paise or cents, or a decimal type. Accumulating fractions of a paisa across millions of transactions produces discrepancies that auditors will find and that you will not be able to explain.
3. Catastrophic cancellation
The most dangerous error, because it destroys accuracy silently.
Subtracting two nearly equal numbers annihilates the significant digits.
Suppose you know two values to 8 significant digits:
a = 1.2345678, \qquad b = 1.2345677
a - b = 0.0000001
The inputs had 8 good digits; the answer has one. The leading digits, which carried all the reliable information, cancelled and left only the least reliable part.
Worked example: the quadratic formula. For x^2 + 10^8x + 1 = 0, the formula from Chapter 2.3 gives
x = \frac{-10^8 \pm\sqrt{10^{16}-4}}{2}
The square root is very close to 10^8. The root using the + sign subtracts two nearly equal numbers, and in double precision it comes out badly wrong — often exactly zero.
The fix uses Vieta's relation from Chapter 2.3, that the roots multiply to c/a. Compute the well-conditioned root normally, then get the other by division:
x_1 = \frac{-b-\operatorname{sign}(b)\sqrt{b^2-4ac}}{2a}, \qquad x_2 = \frac{c}{a\,x_1}
No subtraction of near-equal quantities anywhere. Every numerical library implements it this way, and the schoolbook formula is not what runs.
Where else cancellation appears: computing a variance as E[X^2]-(E[X])^2 when the mean is large (use the two-pass or Welford method instead), computing 1-\cos x for small x (use the identity 2\sin^2(x/2)), and any difference of large similar quantities in a physics simulation.
4. Condition number
Some problems are hard to solve accurately no matter how careful the algorithm.
\text{condition number} = \frac{\text{relative change in output}}{\text{relative change in input}}
A large condition number means small input errors become large output errors. The problem is ill-conditioned, and this is a property of the problem, not the method.
Chapter 4.3's example was two nearly parallel lines whose crossing point slides enormously with a small change. Chapter 4.6 gave the computable version: the ratio of largest to smallest singular value.
The practical rule. In double precision you start with about 16 digits. A condition number of 10^k costs you about k of them. A condition number of 10^{16} leaves you nothing, and the answer your computer prints is entirely rounding noise, formatted to look authoritative.
Two distinct concepts, and confusing them is common. An algorithm is stable if it does not add much error beyond what the problem forces. A problem is well-conditioned if it does not amplify input error. A stable algorithm on an ill-conditioned problem still gives a bad answer — there is nothing to be done except reformulate the problem or get better inputs.
5. Kinds of error, and where they come from
Rounding error — every operation rounds to the nearest representable value. Individually about 10^{-16} relative; accumulated over billions of operations, potentially much more.
Truncation error — using a finite approximation to an infinite process. Stopping a Taylor series (Chapter 5.8) after four terms, or approximating a derivative with a difference quotient over a small but nonzero step.
These two pull in opposite directions, which produces one of the most useful facts in numerical analysis.
Approximate a derivative by \frac{f(x+h)-f(x)}{h}. Truncation error shrinks as h shrinks. Rounding error grows as h shrinks, because the numerator is a difference of two nearly equal numbers — Section 3's cancellation.
So there is an optimal step size, and going smaller makes things worse. For a simple forward difference in double precision it is around h \approx 10^{-8}, and at h = 10^{-16} the answer is pure noise. Anyone who has tried to improve a numerical derivative by shrinking the step has met this, and it surprises people every time.
Accumulated error. Summing a million numbers naively accumulates a million rounding errors. Kahan summation keeps a running correction term for the lost low-order bits and recovers most of the accuracy for a few extra operations — a genuinely free improvement that most naive summation code does not use.
6. Special values
IEEE 754 defines values for the cases where a number does not exist.
Infinity. 1/0 gives Infinity rather than crashing, and arithmetic continues. This is a pragmatic engineering choice, not a claim that Chapter 1.2 was wrong about division by zero.
NaN, not a number. Produced by 0/0, \infty-\infty, \sqrt{-1}.
NaN propagates through every operation, which is deliberate: a single bad value contaminates everything downstream so the corruption is visible rather than silently plausible.
And NaN !== NaN. It is the only value in IEEE 754 not equal to itself, which is why x !== x is a valid NaN test and why sorting an array containing NaN can produce nonsense — the comparator is inconsistent, breaking the total order that Chapter 8.4 said sorting requires.
Two zeros. +0 and -0 are distinct bit patterns that compare equal. The sign matters for things like 1/(-0) = -\infty, and it preserves the direction of approach in limits.
7. Practical rules
Never compare floats for equality. Use a tolerance appropriate to the magnitudes involved.
Never use floats for money. Integer minor units or a decimal type.
Never subtract nearly equal numbers if you can restructure to avoid it.
Watch the order of operations. Adding a tiny number to a huge one loses it entirely; adding many small numbers together first preserves them. (1e20 + 1) - 1e20 gives 0.
Floating point addition is not associative. (a+b)+c and a+(b+c) can differ. This is why parallel reductions can give different answers on different runs — the summation order depends on thread scheduling — and it is a real source of non-reproducible results in scientific computing and machine learning.
Prefer library implementations. hypot(x,y) avoids overflow that naive \sqrt{x^2+y^2} suffers. log1p(x) computes \ln(1+x) accurately for tiny x where the naive version cancels. expm1 likewise. These exist because someone was bitten.
Check condition numbers before trusting a linear solve.
Every formula above, built from scratch
None of the results in this chapter are worth memorising, because each one can be rebuilt in under a minute from something simpler. What follows is that rebuilding, one result at a time, so the formula and the reason for it sit on the same page as the explanation that needed them.
Floating point and error
x = \pm\, m\times 2^{e}, \qquad 1\le m<2
A double-precision number uses 1 sign bit, 11 exponent bits and 52 bits for the fractional part of m.
\varepsilon_{\text{machine}} = 2^{-52} \approx 2.22\times10^{-16}
What machine epsilon means. It is the gap between 1 and the next representable number. Every stored value carries a relative error of at most \frac{\varepsilon}{2}, so doubles give about 16 significant decimal digits — no more, ever, regardless of how the number is printed.
\text{fl}(x) = x(1+\delta), \qquad |\delta|\le\frac{\varepsilon}{2}
Absolute and relative error.
E_{\text{abs}} = |x_{\text{true}}-x_{\text{approx}}|, \qquad E_{\text{rel}} = \frac{|x_{\text{true}}-x_{\text{approx}}|}{|x_{\text{true}}|}
Relative error is almost always the one that matters. Being 1 metre out is fine when measuring a country and catastrophic when machining a bearing.
Why 0.1 is not 0.1
0.1_{10} = 0.0001100110011\overline{0011}_2
One tenth is a repeating fraction in binary, exactly as one third is in decimal. It cannot be stored exactly in any number of bits. So
0.1+0.2 = 0.30000000000000004
is not a bug; it is two rounding errors that failed to cancel. The rule that follows: never test floating-point numbers for equality. Test whether the difference is smaller than a tolerance you choose deliberately.
The two ways error gets amplified
Catastrophic cancellation happens when you subtract two nearly equal numbers. The leading digits cancel and the rounding noise, which was in the last digits, is promoted to the front. Computing \sqrt{x+1}-\sqrt x directly for large x loses most of its digits; rewriting it as
\sqrt{x+1}-\sqrt{x} = \frac{1}{\sqrt{x+1}+\sqrt x}
using the conjugate trick from 2.1 — what algebra is removes the subtraction entirely and the accuracy comes back.
The condition number measures how much a problem amplifies error, before any algorithm is chosen:
\kappa = \left|\frac{x f'(x)}{f(x)}\right| \qquad \text{for evaluating } f, \qquad \kappa(A) = \|A\|\,\|A^{-1}\| \quad\text{for solving } A\mathbf{x}=\mathbf{b}
How to read it. A condition number of 10^k means you can lose k significant digits no matter how careful the algorithm is. With \kappa = 10^{10} and 16 digits of precision, you have 6 left. An ill-conditioned problem is not a bad algorithm; it is a bad question, and the fix is to reformulate rather than to compute harder.
The quadratic formula, done properly
x = \frac{-b\pm\sqrt{b^2-4ac}}{2a}
When b^2\gg4ac, the root using the sign that subtracts suffers catastrophic cancellation, because \sqrt{b^2-4ac}\approx|b|. The numerically stable version computes the safe root first and gets the other from Vieta's product rule:
q = -\tfrac12\left(b+\operatorname{sign}(b)\sqrt{b^2-4ac}\right), \qquad x_1 = \frac qa, \quad x_2 = \frac cq
With a=1, b=10^8, c=1: the naive formula gives x_2 = 0 (completely wrong); the stable one gives -10^{-8} (correct).
8. Where this shows up in your life
Every price displayed by every application. Rounding decisions, and why totals sometimes disagree by one paisa.
Every game's physics. Accumulated error is why objects drift, jitter, or occasionally fly off through a wall.
Every machine learning training run. Different GPU counts give slightly different results, because the summation order changed.
Every scientific result. Reproducibility requires specifying not just the algorithm but the arithmetic.
Every long-running control system. The Patriot failure was accumulated drift, and the fix in such systems is to periodically resynchronise rather than integrate forever.
With the limits of machine arithmetic understood, the rest of this Part is about what a computer does when no formula exists: find roots, integrate, step through differential equations, and search for the best answer.