Appearance
5.3 — The Rules of Differentiation
Computing every derivative from the limit definition would make calculus unusable. A handful of rules replaces the whole business, and they compose: any function you can write down as a combination of standard pieces can be differentiated mechanically, step by step, with no cleverness at all.
That mechanical quality is not an accident. It is why a computer can differentiate — and automatic differentiation, the technique that trains every neural network, is exactly these rules applied by a program.
1. The basic rules
Constant: \dfrac{d}{dx}(c) = 0. Nothing changes, so the rate of change is zero.
Power rule: \dfrac{d}{dx}(x^n) = nx^{n-1} — bring the exponent down in front, and reduce it by one.
It holds for every n: positive, negative, fractional. Check against Chapter 5.2's hand-computed cases: x^2 gives 2x^1 = 2x ✓, and \frac{1}{x} = x^{-1} gives -1x^{-2} = -\frac{1}{x^2} ✓. And \sqrt{x} = x^{1/2} gives \frac{1}{2}x^{-1/2} = \frac{1}{2\sqrt x}.
Constant multiple: \dfrac{d}{dx}(cf) = c f'. Scaling a function scales its rate of change.
Sum: \dfrac{d}{dx}(f + g) = f' + g'. Differentiate term by term.
Those four already handle every polynomial:
\frac{d}{dx}(3x^4 - 5x^2 + 7x - 2) = 12x^3 - 10x + 7
The constant -2 vanished, and the 7x became a plain 7, since 7x^1 gives 7x^0 = 7.
2. The product rule
The tempting guess is wrong: the derivative of a product is not the product of the derivatives.
\frac{d}{dx}(fg) = f'g + fg'
Why, with a picture. Think of f and g as the sides of a rectangle, so fg is its area. Increase x a little; both sides grow. The new area gains a strip along the top (f \cdot \Delta g), a strip along the side (\Delta f \cdot g), and a tiny corner square (\Delta f \cdot \Delta g).
Divide by \Delta x and take the limit. The two strips give fg' + f'g. The corner has two small factors, so relative to \Delta x it vanishes — it is second-order small. The rule is the two strips, and the missing corner is the reason the naive guess fails.
Example. \frac{d}{dx}(x^2\sin x) = 2x\sin x + x^2\cos x.
3. The quotient rule
\frac{d}{dx}\left(\frac{f}{g}\right) = \frac{f'g - fg'}{g^2}
The order in the numerator matters, since subtraction is not commutative. A rhythm that helps: "low d-high minus high d-low, over the square of what's below."
Example. \frac{d}{dx}\tan x = \frac{d}{dx}\frac{\sin x}{\cos x} = \frac{\cos x\cos x - \sin x(-\sin x)}{\cos^2 x} = \frac{\cos^2 x + \sin^2 x}{\cos^2 x} = \frac{1}{\cos^2 x} = \sec^2 x
using the Pythagorean identity from Chapter 3.5 in the last step.
4. The chain rule, which is the important one
For a function inside a function:
\frac{d}{dx}f(g(x)) = f'(g(x)) \cdot g'(x)
In Leibniz notation, where it looks obvious:
\frac{dy}{dx} = \frac{dy}{du}\cdot\frac{du}{dx}
The du appears to cancel. It is not really a fraction and the cancellation is not really cancellation, but Leibniz chose the notation so that this would work, and it does.
What it means. If a bicycle wheel turns twice as fast as the pedals, and the pedals turn three times as fast as your foot circles, then the wheel turns six times as fast as your foot. Rates multiply along a chain.
Worked example. y = (3x^2+1)^5.
Outer function: something to the fifth. Inner: 3x^2+1.
\frac{dy}{dx} = 5(3x^2+1)^4 \cdot 6x = 30x(3x^2+1)^4
Differentiate the outside leaving the inside alone, then multiply by the derivative of the inside.
Nested further. y = \sin(e^{x^2}). Work outward in:
\frac{dy}{dx} = \cos(e^{x^2}) \cdot e^{x^2} \cdot 2x
Three layers, three factors.
The chain rule is what trains AI
A neural network is a chain of functions — each layer takes the previous layer's output and transforms it. To improve the network you need to know how the final error changes when you nudge a weight in the first layer, and that is a derivative through the whole chain.
Backpropagation is the chain rule, applied backwards through the layers so that each intermediate derivative is computed once and reused rather than recomputed for every weight. That reuse is the only reason training a network with billions of parameters is feasible at all.
It also explains a famous failure. Chapter 1.4 noted that multiplying many numbers below one drives the product to nothing. In a deep chain the gradient is a product of many layer derivatives, so if each is small the gradient reaching the early layers vanishes and those layers stop learning. That is the vanishing gradient problem, and the ReLU activation and residual connections of Volume I, 12.4 exist to fix it.
5. The exponential and logarithm
\frac{d}{dx}e^x = e^x
The function that is its own derivative. At every point its height equals its steepness. That property is unique to e^x up to a constant multiple, and it is the real reason e is the natural base — Chapter 1.4 promised this explanation and here it is.
For another base, the chain rule gives a leftover factor:
\frac{d}{dx}a^x = a^x\ln a
which is precisely why base e is cleaner: \ln e = 1, so the factor disappears.
\frac{d}{dx}\ln x = \frac{1}{x}
Two things are worth noticing about this. First, it explains where logarithms come from in integration — Chapter 5.5 will need the antiderivative of \frac{1}{x}, and this is it. Second, it says the logarithm's steepness falls off as \frac{1}{x}, which is the precise version of "logarithms grow ever more slowly" from Chapter 1.4.
For a general base: \frac{d}{dx}\log_a x = \frac{1}{x\ln a}.
6. The trigonometric functions
\frac{d}{dx}\sin x = \cos x, \qquad \frac{d}{dx}\cos x = -\sin x
These hold only in radians. In degrees each would carry a factor of \frac{\pi}{180}, and it would propagate through every subsequent formula. This is the payoff Chapter 3.3 promised for using radians.
The derivation needs the limit \lim_{\theta\to0}\frac{\sin\theta}{\theta} = 1 from Chapter 5.1, together with the angle-addition formula from Chapter 3.5 — expand \sin(x+h), and the two limits fall out.
Differentiate four times and you return to the start:
\sin x \to \cos x \to -\sin x \to -\cos x \to \sin x
A cycle of length four, exactly like the powers of i in Chapter 2.5. That is not a coincidence — Euler's formula says e^{ix} = \cos x + i\sin x, and differentiating e^{ix} multiplies it by i, which is a quarter turn. Differentiating a sine wave rotates its phase by 90°, which is why in an electrical circuit the current through a capacitor leads the voltage by exactly a quarter cycle. Volume III, Part 1 uses this constantly.
The rest follow from the quotient rule:
\frac{d}{dx}\tan x = \sec^2 x, \qquad \frac{d}{dx}\arctan x = \frac{1}{1+x^2}, \qquad \frac{d}{dx}\arcsin x = \frac{1}{\sqrt{1-x^2}}
The last two are worth noting because they are how algebraic-looking integrals in Chapter 5.6 end up producing trigonometric answers.
7. Implicit differentiation
Sometimes y is not written explicitly in terms of x. A circle:
x^2 + y^2 = 25
You could solve for y = \pm\sqrt{25-x^2} and handle two cases. Better: differentiate both sides as they stand, treating y as a function of x and applying the chain rule whenever y appears.
2x + 2y\frac{dy}{dx} = 0 \quad\Longrightarrow\quad \frac{dy}{dx} = -\frac{x}{y}
The \frac{dy}{dx} appears because differentiating y^2 with respect to x needs the chain rule: the outer function gives 2y, and the inner derivative is \frac{dy}{dx}.
Check it against geometry. At the point (3,4) the gradient is -\frac{3}{4}. The radius to that point has gradient \frac{4}{3}. Their product is -1, so they are perpendicular — which is exactly Chapter 3.3's theorem that a tangent is perpendicular to the radius. Two chapters, two methods, same answer.
8. Related rates: the classic application
A ladder 5 m long leans against a wall. Its foot slides away at 0.5 m/s. How fast is the top sliding down when the foot is 3 m from the wall?
Let x be the distance from the wall to the foot and y the height of the top. Pythagoras holds at every instant:
x^2 + y^2 = 25
Differentiate with respect to time, since both x and y are changing as time passes:
2x\frac{dx}{dt} + 2y\frac{dy}{dt} = 0
At the moment in question, x = 3, so y = 4, and \frac{dx}{dt} = 0.5:
2(3)(0.5) + 2(4)\frac{dy}{dt} = 0 \quad\Longrightarrow\quad \frac{dy}{dt} = -\frac{3}{8} = -0.375\ \text{m/s}
Negative, meaning downward, as expected. The relationship between the quantities becomes a relationship between their rates, by differentiating it.
That move — differentiate a constraint to get a relation between rates — is the standard technique for anything where several changing quantities are tied together, and it is used constantly in physics and engineering.
Every formula above, built from scratch
None of the results in this chapter are worth memorising, because each one can be rebuilt in under a minute from something simpler. What follows is that rebuilding, one result at a time, so the formula and the reason for it sit on the same page as the explanation that needed them.
The rules of differentiation
\left(f\pm g\right)' = f'\pm g', \qquad (cf)' = cf'
Product rule.
(fg)' = f'g + fg'
Where it comes from. Think of f and g as the two sides of a rectangle whose area is fg. Grow both sides a little. The new area is
(f+\Delta f)(g+\Delta g) = fg + f\Delta g + g\Delta f + \Delta f\Delta g
The increase is f\Delta g + g\Delta f + \Delta f\Delta g: a strip along the top, a strip along the side, and a tiny corner square. Divide by \Delta x and let it shrink. The two strips give fg' + gf'. The corner has two small factors, so it shrinks much faster than the others and disappears entirely in the limit.
Quotient rule.
\left(\frac fg\right)' = \frac{f'g - fg'}{g^2}
Where it comes from. Write \frac fg = f\cdot g^{-1} and use the product rule together with the chain rule on g^{-1}, whose derivative is -g^{-2}g':
= f'g^{-1} + f\left(-g^{-2}g'\right) = \frac{f'}{g} - \frac{fg'}{g^2} = \frac{f'g - fg'}{g^2}
The order in the numerator matters, unlike in the product rule, and the reliable check is a case you know: \frac{d}{dx}\frac1x should be -\frac{1}{x^2}, and the formula gives \frac{0\cdot x - 1\cdot 1}{x^2} = -\frac1{x^2} ✓
Chain rule.
\frac{d}{dx}f(g(x)) = f'(g(x))\cdot g'(x), \qquad \text{or} \qquad \frac{dy}{dx} = \frac{dy}{du}\cdot\frac{du}{dx}
Where it comes from. If u changes 3 times as fast as x, and y changes 5 times as fast as u, then y changes 15 times as fast as x. Rates multiply along a chain. Written with the fractions, the du appears to cancel — which is not a proof, but it is why Leibniz's notation is the one everybody uses.
Implicit differentiation. When y is tangled up with x, differentiate every term with respect to x and attach \frac{dy}{dx} each time you differentiate a y. On the circle x^2+y^2 = 25:
2x + 2y\frac{dy}{dx} = 0 \quad \Rightarrow \quad \frac{dy}{dx} = -\frac xy
Logarithmic differentiation, for things like y = x^x where neither the power rule nor the exponential rule applies. Take logs first, which turns the exponent into a multiplier:
\ln y = x\ln x
Differentiate both sides, using the chain rule on the left and the product rule on the right:
\frac1y\frac{dy}{dx} = \ln x + 1 \quad \Rightarrow \quad \frac{dy}{dx} = x^x\left(\ln x + 1\right)
The complete derivative table
| f(x) | f'(x) | Where it comes from |
|---|---|---|
| c | 0 | a flat line has no slope |
| x^n | nx^{n-1} | binomial expansion |
| e^x | e^x | definition of e |
| a^x | a^x\ln a | write a^x = e^{x\ln a}, chain rule |
| \ln x | 1/x | inverse of e^x |
| \log_a x | \frac{1}{x\ln a} | change of base |
| \sin x | \cos x | addition formula + squeeze |
| \cos x | -\sin x | same |
| \tan x | \sec^2 x | quotient rule on \sin/\cos |
| \cot x | -\csc^2x | quotient rule |
| \sec x | \sec x\tan x | chain rule on (\cos x)^{-1} |
| \csc x | -\csc x\cot x | same |
| \arcsin x | \frac{1}{\sqrt{1-x^2}} | inverse function rule |
| \arccos x | -\frac{1}{\sqrt{1-x^2}} | same |
| \arctan x | \frac{1}{1+x^2} | same |
| \sinh x | \cosh x | differentiate the definition |
| \cosh x | \sinh x | note: no minus sign |
| \tanh x | \operatorname{sech}^2x | quotient rule |
The derivations for the four that are not obvious.
\tan x: apply the quotient rule to \frac{\sin x}{\cos x}:
\frac{\cos x\cos x - \sin x(-\sin x)}{\cos^2x} = \frac{\cos^2x+\sin^2x}{\cos^2x} = \frac{1}{\cos^2 x} = \sec^2x
a^x: rewrite the base in terms of e, since a = e^{\ln a}:
a^x = \left(e^{\ln a}\right)^x = e^{x\ln a}
Now the chain rule, with the inside function x\ln a whose derivative is the constant \ln a:
\frac{d}{dx}e^{x\ln a} = e^{x\ln a}\cdot\ln a = a^x\ln a
\arcsin x: let y = \arcsin x, so \sin y = x. Differentiate both sides with respect to x:
\cos y\frac{dy}{dx} = 1 \quad \Rightarrow \quad \frac{dy}{dx} = \frac{1}{\cos y}
Now convert \cos y into something in x. From \sin^2y+\cos^2y = 1 we get \cos y = \sqrt{1-\sin^2y} = \sqrt{1-x^2}, taking the positive root because arcsine's output range is where cosine is positive.
\frac{d}{dx}\arcsin x = \frac{1}{\sqrt{1-x^2}}
\cosh x: differentiate \frac{e^x+e^{-x}}{2} directly. The second term's derivative is -e^{-x} by the chain rule, so
\frac{e^x - e^{-x}}{2} = \sinh x
No minus sign appears, unlike with \cos. That single difference propagates through everything hyperbolic.
9. Where this shows up in your life
Every neural network trained anywhere. Chain rule, run backwards.
Every physics engine. Positions from velocities from accelerations, with constraints differentiated exactly as in Section 8.
Every financial risk model. The "greeks" of options trading are derivatives of the option price: delta with respect to the share price, gamma the second derivative, theta with respect to time, vega with respect to volatility. Traders live in this chapter's language.
Every control system. A PID controller uses the error, its integral and its derivative — the D term looks at how fast the error is changing and damps the response before it overshoots. Volume III, 6.6.
Every curve-drawing tool. Smoothly joining two curves means matching not just position but first and often second derivatives, which is what makes a font look right rather than kinked.
We can now compute how fast anything changes. The next chapter turns that into the thing derivatives were really invented for: finding the best possible value of something.