Appearance
5.8 — Infinite Series and Taylor Expansion
Your processor has circuits for addition and multiplication. It has none for sine. Yet Math.sin(1.2) returns an answer in nanoseconds.
What it actually computes is a polynomial:
\sin x \approx x - \frac{x^3}{6} + \frac{x^5}{120} - \frac{x^7}{5040}
At x = 1.2 that gives 0.93204, and the true value is 0.93204. Four terms, agreeing to five decimal places, using nothing but multiplication and addition.
Where those coefficients come from is the subject of this chapter, and it is one of the most practically useful ideas in the volume.
1. Infinite sums that add to something finite
Chapter 5.1 met Zeno's series:
\frac{1}{2}+\frac14+\frac18+\cdots = 1
An infinite sum is defined as the limit of its partial sums. Add the first term, then the first two, then the first three; if that sequence of running totals approaches a number, the series converges to it. If it grows without bound or oscillates, it diverges.
Geometric series are the case you can always evaluate. Each term is a fixed ratio r times the previous one:
a + ar + ar^2 + \cdots = \frac{a}{1-r} \quad\text{provided } |r| \lt 1
The derivation is worth seeing because it is three lines. Call the sum S. Then rS is the same series shifted along by one term, so S - rS = a, giving S(1-r) = a and S = \frac{a}{1-r}. This is the identical trick used on repeating decimals in Chapter 1.3.
If |r| \ge 1 the terms do not shrink and the sum diverges. The condition is not a technicality — at r = 1 you are adding a forever.
Terms shrinking to zero is not enough
The harmonic series
1 + \frac12+\frac13+\frac14+\cdots
has terms going to zero, and it still diverges to infinity.
The proof, by Nicole Oresme around 1350, is charming. Group the terms:
1 + \frac12 + \underbrace{\left(\frac13+\frac14\right)}_{\gt\; 1/2} + \underbrace{\left(\frac15+\cdots+\frac18\right)}_{\gt\; 1/2} + \underbrace{\left(\frac19 + \cdots + \frac{1}{16}\right)}_{\gt\; 1/2} + \cdots
Each bracket exceeds \frac12, because every term in it is at least as big as the last one, and there are enough of them. So the sum exceeds 1 + \frac12 + \frac12 + \frac12 + \cdots, which grows without bound.
It grows agonisingly slowly — you need about 10^{43} terms to pass 100 — but it passes every number eventually. Slow growth is not bounded growth, and confusing the two is a genuine error in estimating algorithms and physical processes.
Change the exponent slightly and everything changes: \sum\frac{1}{n^2} converges (to \frac{\pi^2}{6}, a result of Euler's that Chapter 11.1 tells the story of). The boundary sits exactly at exponent 1, the same boundary as the improper integrals of Chapter 5.5.
2. Taylor series: rebuilding a function from its derivatives at one point
Here is the question. Suppose you know everything about a function at a single point a — its value, its slope, its curvature, and every higher derivative. Can you reconstruct the function everywhere?
For a very large class of functions, yes.
Build a polynomial that matches f at a in value, then in first derivative, then in second, and so on. Working out the coefficients that achieve this gives:
f(x) = f(a) + f'(a)(x-a) + \frac{f''(a)}{2!}(x-a)^2 + \frac{f'''(a)}{3!}(x-a)^3 + \cdots
= \sum_{n=0}^{\infty}\frac{f^{(n)}(a)}{n!}(x-a)^n
This is the Taylor series. When a = 0 it is called a Maclaurin series, and that is the common case.
Where the factorials come from. The notation n! means n\times(n-1)\times\cdots\times1, so 4! = 24. They appear because differentiating x^n repeatedly brings down n, then n-1, then n-2 — after n differentiations you have accumulated a factor of n!, and dividing by it cancels that out so each coefficient is exactly the derivative you wanted.
The first two terms are the tangent-line approximation of Chapter 5.4. Taylor series is that idea continued: a line, then a parabola, then a cubic, each hugging the curve more closely.
3. The series worth knowing
e^x = 1 + x + \frac{x^2}{2!} + \frac{x^3}{3!} + \cdots \qquad\text{(all } x)
Every derivative of e^x is e^x, which is 1 at zero, so every coefficient is \frac{1}{n!}. This is the simplest Taylor series there is, and it is why e^x is the most tractable function in mathematics.
\sin x = x - \frac{x^3}{3!}+\frac{x^5}{5!}-\cdots \qquad\text{(all } x)
Odd powers only, because sine is an odd function (Chapter 3.5) — its graph has rotational symmetry about the origin, and only odd powers do.
\cos x = 1 - \frac{x^2}{2!}+\frac{x^4}{4!}-\cdots \qquad\text{(all } x)
Even powers only, for the mirror reason.
\ln(1+x) = x - \frac{x^2}{2}+\frac{x^3}{3}-\cdots \qquad (-1 \lt x \le 1)
\frac{1}{1-x} = 1 + x + x^2 + x^3 + \cdots \qquad (|x| \lt 1)
The last is just the geometric series, which is reassuring — it says the two ideas are consistent.
Now look at the first three together and read down the terms. The e^x series contains every power. The \cos series contains exactly the even ones and the \sin series exactly the odd ones, with alternating signs. It looks as though e^x has been split into two halves and had signs sprinkled on.
Substitute x = i\theta and watch. The powers of i cycle i, -1, -i, 1 (Chapter 2.5), so the even-power terms become real with alternating signs, and the odd-power terms become imaginary with alternating signs:
e^{i\theta} = \underbrace{\left(1 - \frac{\theta^2}{2!}+\frac{\theta^4}{4!}-\cdots\right)}_{\cos\theta} + i\underbrace{\left(\theta - \frac{\theta^3}{3!}+\cdots\right)}_{\sin\theta}
e^{i\theta} = \cos\theta + i\sin\theta
Euler's formula, derived. Chapter 2.5 stated it and promised this proof. The alternating signs that looked arbitrary in the sine and cosine series turn out to be the powers of i, and the exponential's terms sort themselves perfectly into the two halves. Nothing was arranged; it fell out.
That claim about polynomials creeping outwards to match a function is worth watching term by term.
The agreement spreads outwards from zero rather than improving everywhere at once, and that is exactly what the radius of convergence measures.
4. Radius of convergence, and a genuine mystery
A Taylor series does not always work everywhere. The radius of convergence is how far from a it remains valid.
For e^x, \sin x and \cos x it is infinite — they work for every input.
For \frac{1}{1-x} it is 1, which makes sense: the function blows up at x=1, so no polynomial can follow it past there.
Now the puzzle. Consider
f(x) = \frac{1}{1+x^2}
This function is perfectly smooth for every real number. It never blows up, never has a corner, and looks like a gentle bump. Yet its Taylor series
1 - x^2 + x^4 - x^6 + \cdots
diverges for |x| \gt 1. Why should a series stop working at x = 1 when nothing whatsoever happens to the function there?
The answer is invisible from the real line. In the complex plane (Chapter 2.5), \frac{1}{1+x^2} blows up at x = \pm i — those are the roots of 1+x^2 = 0. Those points are at distance 1 from the origin, and the radius of convergence is the distance to the nearest singularity in the complex plane, whether or not it lies on the real axis.
This is one of the most persuasive arguments that complex numbers are not a computational convenience but a description of what is really going on. A fact about real numbers has no explanation among the real numbers, and a complete one among the complex.
5. How a computer actually evaluates a function
Not by summing a Taylor series naively, which converges slowly far from the centre. The real recipe has three stages.
Range reduction. Use identities to bring the input into a small interval. For \sin, subtract multiples of 2\pi and use symmetry to get into [0, \frac{\pi}{4}], where the series converges very fast.
Polynomial evaluation. Use a polynomial whose coefficients are not quite Taylor's — the minimax polynomial, chosen to minimise the worst error across the interval rather than the error at one point. Taylor is optimal at the centre and drifts at the edges; minimax spreads the error evenly and does better overall for the same number of terms.
Horner's method (Chapter 2.2) to evaluate it in the fewest multiplications.
Result: a sin accurate to the last bit of a 64-bit float in a few dozen nanoseconds, from a handful of multiplications and additions.
The same idea underlies fixed-point arithmetic in embedded systems, GPU shader functions, and every lookup-table-plus-interpolation scheme in graphics. When something must be fast, it is usually a polynomial pretending to be something else.
6. Series in physics and engineering
Small-angle approximations. Keep one term: \sin\theta \approx \theta and \cos\theta\approx 1-\frac{\theta^2}{2}. The pendulum's simple harmonic motion in Chapter 6.2 depends entirely on this, which is why a pendulum clock keeps good time only for small swings.
The binomial approximation. (1+x)^n \approx 1 + nx for small x. A 3% rise followed by a 2% rise is 1.03\times1.02 = 1.0506, and the approximation gives 1.05 — close enough for mental arithmetic, and this is why small percentage changes nearly add (Chapter 1.3's warning was about large ones).
Relativity reducing to Newton. Einstein's kinetic energy is
E = \frac{mc^2}{\sqrt{1-v^2/c^2}} - mc^2
Expand as a series in \frac{v}{c} and the leading term is \frac{1}{2}mv^2 — Newton's kinetic energy, recovered as the first term of the relativistic one. The corrections are of order \frac{v^2}{c^2}, which is 10^{-14} at aircraft speeds, which is exactly why nobody noticed for two hundred years. Volume IV, Chapter 6.
Perturbation theory. When a problem is a small modification of a solvable one, expand the answer in powers of the small parameter. This is how planetary orbits are computed with the pull of other planets included, and how much of quantum mechanics is done.
Numerical methods. Every finite-difference formula and every step of Chapter 10.3's differential equation solvers is a truncated Taylor series, and the discarded terms are precisely the method's error.
Every formula above, built from scratch
None of the results in this chapter are worth memorising, because each one can be rebuilt in under a minute from something simpler. What follows is that rebuilding, one result at a time, so the formula and the reason for it sit on the same page as the explanation that needed them.
Series and Taylor expansion
Convergence tests
| Test | Converges when |
|---|---|
| Geometric \sum ar^n | \lvert r\rvert<1 |
| p-series \sum \frac{1}{n^p} | p>1 |
| Ratio | \lim\left\lvert\frac{a_{n+1}}{a_n}\right\rvert<1 |
| Comparison | smaller than a convergent series |
| Alternating | terms shrink to zero |
The harmonic series \sum\frac1n is the boundary case: its terms shrink to zero and it still diverges. Group them as 1 + \frac12 + \left(\frac13+\frac14\right) + \left(\frac15+\cdots+\frac18\right)+\cdots; each bracket sums to at least \frac12, and there are infinitely many brackets, so the total grows without limit. Slowly — it takes over 10^{43} terms to reach 100 — but without limit.
The Taylor series
f(x) = \sum_{n=0}^{\infty}\frac{f^{(n)}(a)}{n!}(x-a)^n = f(a) + f'(a)(x-a) + \frac{f''(a)}{2!}(x-a)^2+\cdots
Where the coefficients come from, and why the factorials appear. Suppose f can be written as a power series around a:
f(x) = c_0 + c_1(x-a) + c_2(x-a)^2 + c_3(x-a)^3+\cdots
Put x = a: every term with a bracket vanishes, leaving f(a) = c_0.
Differentiate once, then put x=a: the derivative is c_1 + 2c_2(x-a)+3c_3(x-a)^2+\cdots, so f'(a) = c_1.
Differentiate again: 2c_2 + 6c_3(x-a)+\cdots, so f''(a) = 2c_2, giving c_2 = \frac{f''(a)}{2}.
Once more: 6c_3 + \cdots, so c_3 = \frac{f'''(a)}{6} = \frac{f'''(a)}{3!}.
The pattern is now clear: differentiating (x-a)^n exactly n times produces n!, so dividing by n! is what recovers the coefficient. The factorial is not decoration; it is the accumulated product from repeated differentiation.
The series worth knowing by heart
e^x = 1 + x + \frac{x^2}{2!}+\frac{x^3}{3!}+\cdots \qquad \text{(all } x)
\sin x = x - \frac{x^3}{3!}+\frac{x^5}{5!}-\cdots \qquad \text{(all } x)
\cos x = 1 - \frac{x^2}{2!}+\frac{x^4}{4!}-\cdots \qquad \text{(all } x)
\frac{1}{1-x} = 1+x+x^2+x^3+\cdots \qquad (\lvert x\rvert<1)
\ln(1+x) = x - \frac{x^2}{2}+\frac{x^3}{3}-\cdots \qquad (-1<x\le1)
(1+x)^k = 1 + kx + \frac{k(k-1)}{2!}x^2+\cdots \qquad (\lvert x\rvert<1)
Sine has only odd powers and cosine only even ones. That is forced by symmetry: sine is odd, so its series can contain nothing that would survive x\to -x unchanged.
Euler's formula, derived in three lines. Put x = i\theta into the exponential series:
e^{i\theta} = 1 + i\theta + \frac{(i\theta)^2}{2!} + \frac{(i\theta)^3}{3!}+\frac{(i\theta)^4}{4!}+\cdots
The powers of i cycle i, -1, -i, +1:
= 1 + i\theta - \frac{\theta^2}{2!} - i\frac{\theta^3}{3!} + \frac{\theta^4}{4!}+\cdots
Separate the terms without an i from the terms with one:
= \underbrace{\left(1 - \frac{\theta^2}{2!}+\frac{\theta^4}{4!}-\cdots\right)}_{\cos\theta} + i\underbrace{\left(\theta - \frac{\theta^3}{3!}+\frac{\theta^5}{5!}-\cdots\right)}_{\sin\theta}
e^{i\theta} = \cos\theta + i\sin\theta
The two most famous series in mathematics were sitting inside the third one all along, sorted by whether their term carried a factor of i.
7. Where this shows up in your life
Every sin, cos, exp, log and sqrt your computer or phone computes.
Every GPS position. The relativistic corrections are series expansions, and without them positions would drift by about 10 km per day.
Every physics approximation in an engineering calculation. "For small oscillations", "to first order", "neglecting terms in x^2" — all series truncation.
Every financial duration and convexity calculation. Bond price sensitivity to interest rates is a Taylor expansion: duration is the first derivative term, convexity the second.
Every animation easing curve and every antialiasing filter. Polynomial approximations to smooth functions.
Part 5 is finished. We can measure change, accumulate it, optimise it and approximate it. The next Part uses all of it for the thing calculus was invented to do: describe how a system evolves when you know only the rule governing its rate of change.
More places these turn up
Every speedometer computes a derivative and every odometer an integral, of the same journey. The chain rule is what trains every neural network, one partial derivative per weight. Newton's method is inside the square-root button on your phone. Taylor series are how that phone computes \sin at all — there is no sine circuit, only a polynomial with the first few terms of the series above. The area between two curves is how a hospital measures the drug in a patient's bloodstream over time, and Lagrange multipliers are how a delivery route or a portfolio is optimised subject to what you can actually afford.
Next: 5.P — Worked Problems works through limits, derivatives, optimisation, every integration technique, and a Taylor approximation with its error, step by step.