9.1 — Functions as Sums: The Idea of a Basis

Play a chord on a piano and one pressure wave reaches your ear. Yet you hear three distinct notes. Your ear has taken a single complicated signal and pulled it apart into pure tones.

That decomposition is the idea this entire Part is built on, and it is worth stating before any mathematics: a complicated thing can be written as a sum of simple standard pieces, and once it is, questions that were hard become easy.

1. The idea you already know

Chapter 4.1 said that any vector in the plane is a combination of two basis vectors:

\begin{bmatrix}3\\4\end{bmatrix} = 3\begin{bmatrix}1\\0\end{bmatrix}+4\begin{bmatrix}0\\1\end{bmatrix}

The numbers 3 and 4 are the coordinates — how much of each basis direction is present.

Now make the leap: a function is a vector too.

Think of a function as a list of its values, one for each input. A vector in 3 dimensions has 3 components; a function on the interval [0,1] has one component for every point in that interval — infinitely many, uncountably many, but the idea is identical. Adding two functions adds them point by point; scaling a function scales every value. Those are exactly the two operations Chapter 4.1 said define a vector space.

So functions live in a vector space, and the question that follows is the natural one: what is a good basis for it?

2. The polynomial basis, and why it disappoints

An obvious candidate: 1, x, x^2, x^3, \ldots Every polynomial is a combination of these, and Chapter 5.8's Taylor series says a great many other functions are too.

e^x = 1 + x + \frac{x^2}{2}+\frac{x^3}{6}+\cdots

The Taylor coefficients are the coordinates in this basis.

Two things go wrong.

Taylor series describe a function near one point. All the information comes from derivatives at a single place, so the approximation is excellent there and can be poor far away. Chapter 5.8's example of \frac{1}{1+x^2} failing beyond |x|=1 is exactly this.

The basis is badly conditioned. The functions x^5 and x^6 look extremely similar over [0,1] — nearly parallel, in the vector sense. Fitting with them numerically is like solving the ill-conditioned system of Chapter 4.3, and it goes wrong fast.

What is wanted is a basis whose members are perpendicular to each other. Chapter 4.1 explained why: with perpendicular basis vectors, finding a coordinate is one dot product, and each coordinate is independent of the others. With near-parallel ones, everything is tangled.

3. What perpendicular means for functions

The dot product of two vectors multiplies matching components and adds. For functions, matching components means matching input values, and adding infinitely many things means integrating (Chapter 5.5):

\langle f, g\rangle = \int_a^b f(x)g(x)\,dx

This is the inner product of two functions, and everything from Chapter 4.1 carries over unchanged.

Two functions are orthogonal when their inner product is zero. The name is the same word as perpendicular, and the meaning is the same: they contain nothing of each other.

And here is the fact the whole Part rests on. Over one full period, sines and cosines of different whole-number frequencies are orthogonal:

\int_0^{2\pi}\sin(mx)\sin(nx)\,dx = 0 \quad\text{whenever } m\neq n

\int_0^{2\pi}\sin(mx)\cos(nx)\,dx = 0 \quad\text{always}

\int_0^{2\pi}\cos(mx)\cos(nx)\,dx = 0 \quad\text{whenever } m\neq n

When m = n the integral is \pi, not zero — a function is not orthogonal to itself, exactly as \mathbf{v}\cdot\mathbf{v} = \|\mathbf v\|^2 in Chapter 4.1.

Why it is true, in words. Two waves of different whole-number frequencies spend exactly as much time reinforcing each other as cancelling, over a full period. The product is positive half the time and negative half the time, and the areas cancel exactly. Draw \sin x times \sin 2x and you can see it.

That orthogonality is what makes Fourier analysis possible, and it is why the next chapter can extract each frequency's amplitude independently of all the others.

4. Extracting a coordinate

With an orthogonal basis, finding how much of a basis function is present is a single integral.

Suppose f(x) = \sum_n c_n\phi_n(x) for some orthogonal family \phi_n. Take the inner product of both sides with one particular \phi_m:

\langle f,\phi_m\rangle = \sum_n c_n\langle\phi_n,\phi_m\rangle

Every term on the right vanishes except the one with n=m, because everything else is orthogonal. So:

c_m = \frac{\langle f,\phi_m\rangle}{\langle\phi_m,\phi_m\rangle}

One integral gives one coefficient, with no reference to any of the others. That independence is exactly why an orthogonal basis is worth having, and it is the same reason Chapter 4.6's SVD produces orthogonal directions.

The mechanical description: to find out how much of a pure tone is in a signal, multiply the signal by that tone and integrate. If the tone is present, the product is systematically positive and the integral is large. If it is absent, the product oscillates around zero and the integral vanishes. The integral is a detector, and this is precisely what a radio receiver does with a carrier frequency.

5. Why the change of basis is worth doing

Chapter 4.5 made the general point: a hard operation in one coordinate system can be trivial in another. Diagonalisation was one instance. This is another, and the payoff is larger.

Differentiation becomes multiplication.

\frac{d}{dx}\sin(nx) = n\cos(nx)

Each basis function differentiates into a multiple of another basis function at the same frequency. So a differential equation becomes an algebraic equation, frequency by frequency. That is what makes Chapter 6.4's separation of variables work, and it is why Fourier invented the method while solving the heat equation.

Convolution becomes multiplication. Chapter 9.4 covers this, and it converts an expensive smearing operation into simple pointwise products.

Compression becomes obvious. In the frequency basis, most real signals concentrate their energy into a few components. Keep those, discard the rest, and you have compressed with little visible or audible loss. That is the whole principle behind JPEG, MP3 and every video codec — Chapter 9.3 gives the mechanism.

Filtering becomes selection. Removing hum from a recording is impossibly fiddly in the time domain and trivial in the frequency domain: find the 50 Hz component, set it to zero, transform back.

6. Other useful bases

Sines and cosines are not the only orthogonal family, and each alternative is designed for a different kind of signal.

Legendre polynomials — orthogonal on [-1,1], used for numerical integration and in physics for spherical problems.

Chebyshev polynomials — the basis behind the minimax approximations of Chapter 5.8, which is why your processor's sin uses them rather than Taylor's.

Spherical harmonics — the natural basis on a sphere. Used for the cosmic microwave background, for the shapes of electron orbitals in chemistry, and for lighting in 3D graphics.

Wavelets — the important modern alternative. Sines and cosines extend forever, so they describe which frequencies are present but not when. A wavelet is localised in both time and frequency, so it can say "a high-frequency burst happened here". This suits signals with sudden events — an image with sharp edges, an earthquake trace, a heartbeat. The JPEG 2000 standard and the FBI's fingerprint database both use wavelets.

Why a lower resolution image looks blurry rather than blocky

Most compression works by transforming into a frequency-like basis, keeping the large coefficients and discarding the small ones.

Fine detail — sharp edges, texture — lives in the high-frequency coefficients, and those are small in most natural images, so they are the first to go. The result is that detail disappears before shape does, which is why a heavily compressed photograph looks soft rather than fragmented.

When compression is pushed too far you see the artefacts of the specific basis being used: JPEG's 8×8 blocks appear because JPEG transforms each 8×8 block separately. Chapter 9.3 explains why that choice was made.

Every formula above, built from scratch

None of the results in this chapter are worth memorising, because each one can be rebuilt in under a minute from something simpler. What follows is that rebuilding, one result at a time, so the formula and the reason for it sit on the same page as the explanation that needed them.

Orthogonality — the fact everything rests on

\int_{-\pi}^{\pi}\sin(mx)\sin(nx)\,dx = \begin{cases}0 & m\ne n\\ \pi & m=n\end{cases}

\int_{-\pi}^{\pi}\cos(mx)\cos(nx)\,dx = \begin{cases}0 & m\ne n\\ \pi & m = n\end{cases}

\int_{-\pi}^{\pi}\sin(mx)\cos(nx)\,dx = 0 \qquad \text{for all } m, n

Where the first one comes from. Use the product-to-sum identity from 3.5 — trigonometry:

\sin A\sin B = \tfrac12\left[\cos(A-B)-\cos(A+B)\right]

So

\int_{-\pi}^\pi\sin mx\sin nx\,dx = \frac12\int_{-\pi}^{\pi}\cos\left((m-n)x\right)dx - \frac12\int_{-\pi}^\pi\cos\left((m+n)x\right)dx

Now, for any non-zero whole number k:

\int_{-\pi}^\pi\cos(kx)\,dx = \left[\frac{\sin kx}{k}\right]_{-\pi}^{\pi} = \frac{\sin k\pi - \sin(-k\pi)}{k} = 0

because the sine of any whole multiple of \pi is zero. A whole number of complete waves integrates to exactly nothing: the positive humps cancel the negative ones.

When m\ne n, both m-n and m+n are non-zero, so both integrals vanish and the answer is 0. When m=n, the first integrand becomes \cos 0 = 1, whose integral over the interval is 2\pi, giving \frac12(2\pi) - 0 = \pi.

Why this matters so much. It means each frequency can be extracted from a mixture without disturbing the others. Multiply a signal by \sin(3x) and integrate, and every component except the third vanishes. That is a sieve for frequencies, and it is the entire mechanism of the next section.

7. Where this shows up in your life

Every piece of digital audio and video you consume. Transform, discard, reconstruct.

Every equaliser, every noise filter, every guitar tuner. Operations in the frequency basis.

Every AI embedding. Volume I, 12.5 turns a word into a vector of several hundred numbers, and that vector's components are coordinates in a learned basis. The basis is not sines and cosines and the idea is exactly this chapter's — represent a complicated object as coordinates in a well-chosen space, then do arithmetic on the coordinates.

Every principal component analysis. Chapter 4.6 finds a basis adapted to your particular data rather than a fixed one.

Every solution of a partial differential equation. Chapter 6.4's separation of variables is a basis expansion.

---

The general idea is in place. The next chapter makes it concrete with the specific basis that changed the world, and tells the story of the man who was told he was wrong.