Appearance
5.7 — Calculus with Several Variables
The temperature in a room depends on where you are — three coordinates — and on the time. A company's profit depends on price, marketing spend, production volume and a dozen other things. A neural network's error depends on billions of weights, all at once.
Single-variable calculus cannot touch any of this. The extension is easier than you might expect, because the central move is: vary one thing and hold the rest still.
1. Functions of several variables
f(x,y) = x^2 + y^2
takes two numbers and returns one. Its graph lives in three dimensions — a surface, in this case a bowl with its lowest point at the origin.
For more than two inputs the picture is unavailable, and that is fine. Everything below is written so that it works for any number of variables, and none of it depends on being able to draw it.
A contour map is the practical way to see a two-input function on flat paper: draw the curves along which f is constant. That is exactly what a walker's map does with height, and what a weather chart does with pressure. Contours close together mean a steep slope; far apart means gentle.
2. Partial derivatives
A partial derivative asks how f changes when you nudge one input and freeze the others. The symbol is a rounded d:
\frac{\partial f}{\partial x} = \lim_{h\to0}\frac{f(x+h,\,y) - f(x,\,y)}{h}
Notice the y is unchanged in both terms. Computationally, treat every other variable as a constant and differentiate normally. There is nothing new to learn.
Worked example. f(x,y) = x^2 y^3 + 4x - 7y.
\frac{\partial f}{\partial x} = 2xy^3 + 4 \qquad\text{(treating } y^3 \text{ as a constant multiplier, and } -7y \text{ as a constant)}
\frac{\partial f}{\partial y} = 3x^2y^2 - 7
Physical reading. On a hillside, \frac{\partial h}{\partial x} is the slope if you walk due east, and \frac{\partial h}{\partial y} the slope if you walk due north. Two different numbers at the same spot, because slope depends on which way you face.
Second partials come in four varieties for two variables, and there is a pleasant surprise:
\frac{\partial^2 f}{\partial x\,\partial y} = \frac{\partial^2 f}{\partial y\,\partial x}
The order does not matter, for any function whose second partials are continuous. This is Clairaut's theorem, and it is not obvious. In our example both mixed partials come to 6xy^2 — check it either way round.
3. The gradient
Collect all the partial derivatives into one vector:
\nabla f = \left[\frac{\partial f}{\partial x},\; \frac{\partial f}{\partial y},\; \ldots\right]
The symbol \nabla is called "nabla" or "del", after an ancient harp whose shape it resembles.
The gradient is the single most important object in applied mathematics after the derivative itself, and it has three properties that carry everything.
It points in the direction of steepest increase. Standing on a hillside, the gradient points straight uphill — the direction where you gain height fastest.
Its length is how steep that steepest direction is. A long gradient means a cliff; a short one means nearly flat.
It is perpendicular to the contour lines. The steepest way up is always at right angles to the direction of constant height, which is why a zigzag path up a mountain is the shallow route.
The directional derivative answers "how fast does f change if I walk in direction \mathbf{u}?", where \mathbf{u} is a unit vector:
D_{\mathbf{u}}f = \nabla f\cdot\mathbf{u}
A dot product, from Chapter 4.1 — and that immediately explains the three properties above. The dot product is largest when the two vectors align (so the steepest direction is along the gradient), and zero when they are perpendicular (so walking along a contour changes nothing).
Gradient descent, which is how machines learn
Training a model means minimising an error function of millions of parameters. You cannot solve \nabla f = 0 algebraically at that scale, so instead:
\mathbf{w}_{\text{new}} = \mathbf{w}_{\text{old}} - \eta\,\nabla f(\mathbf{w}_{\text{old}})
Take a small step in the direction opposite to the gradient — downhill — and repeat. The step size \eta is the learning rate.
That is the entire training algorithm for every neural network in existence. The chain rule of Chapter 5.3 computes the gradient (backpropagation); this equation uses it. Chapter 10.4 covers the refinements — momentum, adaptive rates — that make it work in practice.
4. Optimisation with several variables
A critical point is where every partial derivative is zero, that is, where \nabla f = \mathbf{0}. All directions are locally flat.
But there is a new possibility that does not exist in one dimension: the saddle point. Imagine a mountain pass, or a horse's saddle. It is a minimum along the ridge direction and a maximum across it. The gradient is zero, and it is neither a peak nor a trough.
f(x,y) = x^2 - y^2
has \nabla f = (2x, -2y), zero at the origin. Along the x axis it is a minimum; along the y axis a maximum. A saddle.
To classify a critical point you look at the second derivatives, collected into a matrix called the Hessian:
H = \begin{bmatrix} f_{xx} & f_{xy} \\ f_{yx} & f_{yy}\end{bmatrix}
It is symmetric, by Clairaut. Its eigenvalues (Chapter 4.5) decide the type: all positive means a minimum (the surface curves up in every direction), all negative means a maximum, and mixed signs mean a saddle.
This matters far beyond textbooks. In high-dimensional optimisation — training a large model — saddle points vastly outnumber true minima, because a critical point is a minimum only if all of a million eigenvalues happen to be positive, which is overwhelmingly unlikely. For years people believed neural networks got stuck in bad local minima; the modern understanding is that they mostly get slowed down by saddles, and that the fixes are different.
Lagrange multipliers
Optimise subject to a constraint. Maximise f(x,y) while staying on the curve g(x,y) = c.
The insight, and it is a good one. Walk along the constraint curve. As long as the gradient of f has any component along your path, you can improve by moving that way. So at the best point, the gradient of f must have no component along the curve — it must be perpendicular to the curve. And the gradient of g is also perpendicular to the curve, since the curve is a contour of g.
Two vectors perpendicular to the same curve must be parallel:
\nabla f = \lambda\nabla g
The number \lambda is the Lagrange multiplier. Solve this together with the constraint.
Worked example. Maximise f = xy subject to x + y = 10 — the largest rectangle for a given perimeter.
\nabla f = (y, x), \qquad \nabla g = (1,1)
So y = \lambda and x = \lambda, giving x = y. With the constraint, x = y = 5 and the maximum area is 25. A square, confirming the arithmetic-geometric mean result of Chapter 2.4 from a completely different direction.
The multiplier itself has a meaning that economists use constantly: \lambda is the rate at which the optimum improves if you relax the constraint by one unit. It is the shadow price — what one more unit of the scarce resource is worth.
5. Multiple integrals
Integrating over a region rather than an interval:
\iint_R f(x,y)\,dA
sums f over every little patch of area in the region R. If f is a density, this is total mass; if f is 1, it is the region's area; if f is height, it is the volume underneath.
Compute it as an iterated integral — integrate in one variable at a time, treating the other as constant:
\int_0^1\int_0^2 xy\,dx\,dy = \int_0^1\left[\frac{x^2y}{2}\right]_0^2 dy = \int_0^1 2y\,dy = 1
Inner integral first, then outer. Fubini's theorem says the order can be swapped for well-behaved functions, which matters practically because one order is often far easier than the other.
Changing coordinates, and the factor everyone forgets. For a circular region, polar coordinates from Chapter 3.4 are far better. But when you change coordinates you must multiply by the Jacobian — the determinant from Chapter 4.4 that measures how much the coordinate change stretches area. For polar coordinates it is r:
dA = dx\,dy = r\,dr\,d\theta
Why there is an extra r: a small patch in polar coordinates spans dr radially and r\,d\theta around, because arc length is radius times angle (Chapter 3.3). Patches far from the origin are wider for the same angle, and the r accounts for it.
The famous payoff. Chapter 5.6 said \int e^{-x^2}dx has no elementary antiderivative. Yet the definite integral over the whole line can be computed exactly, by a trick that only works in two dimensions. Call the answer I and consider I^2:
I^2 = \int_{-\infty}^{\infty}e^{-x^2}dx\int_{-\infty}^{\infty}e^{-y^2}dy = \iint e^{-(x^2+y^2)}dA
In polar coordinates x^2+y^2 = r^2, and the extra r from the Jacobian is exactly what makes it integrable by substitution:
I^2 = \int_0^{2\pi}\int_0^\infty e^{-r^2}r\,dr\,d\theta = 2\pi\cdot\frac{1}{2} = \pi
\int_{-\infty}^{\infty}e^{-x^2}dx = \sqrt\pi
The impossible integral became easy by squaring it and moving to polar coordinates — the same dimension-raising trick as Monge's theorem in Chapter 3.6. And this is why \pi appears in the normal distribution's formula in Chapter 7.6, which otherwise looks like an inexplicable intrusion of circles into statistics.
6. Vector calculus, in outline
When the output is also a vector, three operators appear. You will meet them in physics and it is worth knowing what each one asks.
Divergence \nabla\cdot\mathbf{F} — a number at each point, measuring how much the field spreads out from there. Positive means a source, negative a sink. For water flow, a positive divergence means water is appearing.
Curl \nabla\times\mathbf{F} — a vector at each point, measuring rotation. Drop a tiny paddle wheel into the flow; the curl says how fast it spins.
Laplacian \nabla^2 f — the divergence of the gradient, measuring how much a value at a point differs from the average of its neighbours. It is the operator in the heat equation and the wave equation of Chapter 6.4, and it is what an image-sharpening filter computes.
Three big theorems (Green's, Stokes', and the divergence theorem) all say the same thing in different dimensions: what happens inside a region is determined by what crosses its boundary. The Fundamental Theorem of Calculus from Chapter 5.5 is the one-dimensional case, where the "boundary" is just the two endpoints.
Maxwell's four equations, which describe all of electricity, magnetism and light, are written in exactly this language. Volume III, Part 7 and Volume IV, Chapter 4 develop them.
Every formula above, built from scratch
None of the results in this chapter are worth memorising, because each one can be rebuilt in under a minute from something simpler. What follows is that rebuilding, one result at a time, so the formula and the reason for it sit on the same page as the explanation that needed them.
Multivariable calculus
\frac{\partial f}{\partial x} = \text{differentiate with respect to } x, \text{ holding every other variable still}
The gradient, the vector of all the partial derivatives:
\nabla f = \left(\frac{\partial f}{\partial x}, \frac{\partial f}{\partial y}, \ldots\right)
What it means. The gradient points in the direction of steepest increase, and its length is how steep that is. Standing on a hillside, it points straight uphill. This is the single fact that machine learning runs on: gradient descent steps in the direction -\nabla f, which is straight downhill, and repeats.
The directional derivative, the slope in an arbitrary direction \mathbf{u} (a unit vector):
D_{\mathbf{u}}f = \nabla f\cdot\mathbf{u} = \|\nabla f\|\cos\theta
The \cos\theta shows why the gradient direction is steepest: the dot product is largest when \theta = 0, that is, when you walk the way the gradient points.
The multivariable chain rule.
\frac{df}{dt} = \frac{\partial f}{\partial x}\frac{dx}{dt} + \frac{\partial f}{\partial y}\frac{dy}{dt}
Each variable contributes its own path from the change in t to the change in f, and the contributions add. This is exactly the backpropagation rule used to train a neural network: the error's sensitivity to a weight is the sum of its sensitivities along every path through the network.
Second derivatives and the Hessian.
H = \begin{pmatrix}f_{xx} & f_{xy}\\ f_{yx}&f_{yy}\end{pmatrix}, \qquad f_{xy} = f_{yx} \text{ for smooth functions}
At a point where \nabla f = \mathbf{0}, the classification uses D = f_{xx}f_{yy}-f_{xy}^2:
| D | f_{xx} | Type |
|---|---|---|
| >0 | >0 | minimum |
| >0 | <0 | maximum |
| <0 | any | saddle |
| =0 | any | undecided |
D is the determinant of the Hessian, so this test is the eigenvalue story of 4.5 — eigenvalues in disguise: both eigenvalues positive is a bowl, both negative is a dome, one of each is a saddle.
Lagrange multipliers, for optimising subject to a constraint:
\nabla f = \lambda\nabla g \quad \text{together with} \quad g(x,y) = c
Why. At the best point on the constraint curve, you cannot improve f by sliding along the curve. That means f's uphill direction has no component along the curve, so \nabla f must point straight across it — parallel to \nabla g, which is always perpendicular to the constraint curve. Two parallel vectors differ by a scalar, and that scalar is \lambda.
Double and triple integrals.
\iint_R f\,dA, \qquad \iiint_V f\,dV
computed as repeated single integrals, innermost first. In polar coordinates the area element picks up an extra factor:
dA = r\,dr\,d\theta
Where the r comes from. A small polar rectangle has sides dr and r\,d\theta — the second because arc length is radius times angle. Multiply them. Forgetting the r is the most common error in the whole topic.
Vector calculus, in one table:
\operatorname{div}\mathbf{F} = \nabla\cdot\mathbf{F} = \frac{\partial F_1}{\partial x}+\frac{\partial F_2}{\partial y}+\frac{\partial F_3}{\partial z} \qquad (\text{how much is flowing outwards})
\operatorname{curl}\mathbf{F} = \nabla\times\mathbf{F} \qquad (\text{how much it is swirling})
\oint_C \mathbf{F}\cdot d\mathbf{r} = \iint_R\left(\frac{\partial Q}{\partial x}-\frac{\partial P}{\partial y}\right)dA \qquad \text{(Green)}
\iint_S\mathbf{F}\cdot d\mathbf{S} = \iiint_V(\nabla\cdot\mathbf{F})\,dV \qquad \text{(divergence theorem)}
All of these say one thing: what happens on the boundary is determined by what happens inside. The fundamental theorem of calculus is the one-dimensional case — the total change over an interval equals the accumulated rate inside it.
7. Where this shows up in your life
Every machine learning model. Gradient descent, Section 3.
Every weather forecast. The atmosphere is a vector field; the equations are partial differential equations built from these operators.
Every economics or operations model with constraints. Lagrange multipliers, and the shadow prices they produce.
Every 3D rendering. Surface normals are gradients; lighting is a dot product with them.
Every medical scan. CT reconstruction integrates over lines through the body and inverts the result.
Every contour on a map, every isobar on a weather chart, every heat map in a dashboard. Level sets of a function of two variables, with the gradient perpendicular to them.
One tool remains before Part 5 is complete: the ability to replace a complicated function with a polynomial that behaves almost identically nearby. It is how every calculator and every processor actually computes a sine.