Appearance
14.5 — Performance Engineering
A team spends three weeks optimising a JSON serialiser. It gets 40% faster. The endpoint's p99 does not move.
The serialiser was 3% of the request. The other 97% was waiting on a database query that ran once per item in a loop.
Amdahl's law says the whole story: if a part accounts for fraction p of the time, speeding it up infinitely gives at most \frac{1}{1-p}. At p = 0.03, the ceiling is a 3% improvement, and they achieved 1.2% of it after three weeks.
Everything in performance work is a consequence of measuring before changing.
1. Define what you are optimising
"Make it faster" is not a target. Three questions first.
Which metric? Latency (how long one request takes) and throughput (how many per second) are different and often opposed — batching improves throughput and worsens individual latency. Decide which one the user experiences.
Which percentile? The mean is nearly useless: an endpoint averaging 100 ms might serve most requests in 40 ms and 1% in 3 seconds, and that 1% is what people complain about.
And the tail matters more than it appears, because of tail latency amplification: if a page makes 20 backend calls and each has a 1% chance of being slow, the probability that at least one is slow is 1 - 0.99^{20} \approx 18\%. A rare backend event becomes a common user experience, which is why fan-out architectures care about p99 rather than the median.
For whom? An average over all users hides the segment on a slow network or an old device (Chapter 6.7).
Then set a target with a number: "p95 under 200 ms for the checkout endpoint at 500 requests per second". Without one, optimisation has no stopping condition — and section 9 is about stopping.
2. Find the bottleneck before touching anything
Two frameworks that make this systematic.
USE for resources — for each of CPU, memory, disk and network, check Utilisation (percentage busy), Saturation (queue depth or waiting) and Errors. Saturation is the one people skip and the one that matters: a CPU at 70% utilisation with a run queue of 12 is a saturated system, and utilisation alone did not show it.
RED for services — Rate, Errors, Duration per endpoint. Chapter 10.10 covers the observability side.
Then the sequence:
- Reproduce it with a realistic load, on realistic data. Performance on a 1,000-row table tells you nothing about 10 million (Chapter 7.3.1).
- Profile to find where time actually goes.
- Change one thing.
- Re-measure. If it did not help, revert it — an unmeasured "optimisation" is complexity with no benefit.
3. Profiling
A sampling profiler interrupts the program many times a second and records the stack. Low overhead, so it can run in production, and it is statistical — rare things are missed. This is what you want by default.
An instrumenting profiler records every function entry and exit. Exact, and the overhead distorts what it measures — small functions look disproportionately expensive because the instrumentation dominates them.
The distinction that resolves the most confusion is CPU time versus wall-clock time.
A CPU profile shows where the processor was busy. If your service spends 90% of its time waiting on a database, the CPU profile shows almost nothing — and people conclude the application is fine.
A wall-clock (or off-CPU) profile shows where time passed, including blocking. For a typical I/O-bound service this is the profile you need, and reaching for a CPU profiler first is the most common mistake in this whole chapter.
Other profile types worth knowing: allocation profiles (what is creating garbage, which drives garbage-collection pauses, Chapter 3.4), heap profiles (what is retained — for leaks), and lock or contention profiles (which lock threads wait on, Chapter 9.5.2).
Tools, briefly: perf and eBPF on Linux for anything including the kernel; async-profiler for the JVM (it avoids the safepoint bias that makes older Java profilers misleading); py-spy for Python, which attaches to a running process without modifying it; Node's built-in profiler and clinic; and the browser's performance panel (Chapter 6.7). For databases, the profiler is EXPLAIN ANALYZE (Chapter 7.2.3).
4. Reading a flamegraph
The standard visualisation, and it is routinely misread.
Each box is a stack frame. Width is the proportion of samples that frame appeared in. The vertical axis is stack depth.
The x-axis is not time. Frames are sorted alphabetically so that identical stacks merge. Reading it left to right as a timeline is the most common error, and it leads to conclusions that are simply not in the data.
What to look for:
Wide boxes near the top. A wide frame with nothing above it is doing the work itself — that is where the time is.
A wide box with many narrow children is a dispatcher; the cost is spread, and the fix is usually to call it less often rather than to make it faster.
A wide tower is deep recursion or a long call chain.
Repeated identical subtrees at the same width is the visual signature of something in a loop — often an N+1 query (Chapter 7.3.2).
A differential flamegraph compares two profiles, colouring what grew and what shrank. This is the fastest way to find a regression between two releases, and it is under-used.
An icicle graph is the same thing drawn downward, which reads more naturally for some people; it carries the same information.
5. Benchmarking, and how it lies
Microbenchmarks are wrong more often than they are useful, and the failure modes are specific enough to list.
Dead code elimination. The compiler notices the result is unused and removes the work. Your benchmark measures an empty loop, and the number is spectacular. Consume the result or use a benchmark framework that handles it.
Warm-up. A JIT-compiled runtime is interpreting at first and optimising after thousands of iterations (Chapter 3.2). The first runs measure the interpreter. Discard the warm-up, and report steady state.
Unrealistic data. A cache with one key has a 100% hit rate. A sort on already-sorted data measures the best case. Real data has skew, and skew is where things break.
Measurement overhead larger than what is measured, for anything in the microsecond range.
Environment noise. A shared cloud instance has noisy neighbours; a laptop throttles thermally; another process runs. Variance between runs is frequently larger than the difference you are testing.
Statistics done wrong. One run of each, comparing means. Run many, interleave them (A, B, A, B — not all of A then all of B, which confounds the comparison with drift), and report median plus percentiles. If the difference is inside the run-to-run variance, there is no difference.
And the honest limit: a microbenchmark tells you about a function in isolation. Cache behaviour, memory pressure, contention and allocation patterns all differ in a real system. Macro-benchmarks — the real workload against the real system — are what decide.
6. The optimisation hierarchy
In descending order of value:
1. Do not do it. The fastest work is work not done. Cache the result (Chapter 10.14), remove the feature, compute it once instead of per request, or return less data.
2. Do it less often. Batch, debounce, coalesce, deduplicate. Turning 100 calls into 1 beats making each 30% faster — and this is where nearly all real wins are.
3. Do it later or elsewhere. Move it off the request path into a queue; return 202 Accepted (Chapter 5.8) and do the work asynchronously.
4. Do it in parallel, if the work is genuinely independent.
5. Do it faster. A better algorithm first (Chapter 4.1) — an O(n^2) to O(n \log n) change beats any constant-factor tuning at scale. Then data layout and cache behaviour (Chapter 1.6). Micro-optimisation last, and only with a profile pointing at it.
The wins that keep recurring in real systems, roughly in order of how often they are the answer:
- N+1 queries (Chapter 7.3.2) — latency, not query time.
- A missing index (Chapter 7.2.3).
- Chatty network calls in a loop — the same shape as N+1, between services.
- Serialisation and parsing of payloads far larger than needed.
- Allocation churn creating garbage-collection pressure.
- Lock contention serialising work that looks parallel.
- A synchronous call on an async path, blocking an event loop (Chapter 3.8.2).
- A cold cache after every deploy, which looks like a performance regression and is a warm-up problem.
7. Load testing
Two models, and the difference matters more than the tool.
A closed model has N virtual users, each waiting for a response before sending the next request. When the system slows down, the load drops — which hides the failure mode you are looking for.
An open model sends requests at a fixed arrival rate regardless of responses, which is how real traffic behaves. This is what reveals queue growth and collapse.
Coordinated omission is the related measurement bug and it is worth understanding, because it makes almost every naive load test optimistic. If your generator sends a request, waits 5 seconds for a slow response, and then sends the next — the requests that should have been sent during those 5 seconds were never sent, and their latency was never recorded. The worst latencies are systematically missing. Use a tool that corrects for it, or send at a fixed rate.
Test shapes: a ramp to find the breaking point, a soak at moderate load for hours to find leaks and slow degradation, a spike to test autoscaling response, and a stress test past the limit to check that it degrades rather than collapses.
Test with production-like data volumes, or you are measuring a different system.
8. Keeping it fast
A performance budget in CI — a benchmark or a synthetic transaction with a threshold — is what stops slow drift. Without it, performance work is undone within a year by a hundred small regressions nobody noticed (Chapter 6.7 makes the same point for the front end).
Continuous profiling in production — a low-overhead sampling profiler always running — turns "it was slow last Tuesday" from unanswerable into a lookup.
Watch percentiles over time, per endpoint, and alert on the change rather than only on an absolute threshold.
9. When to stop
When you hit the target you wrote down in section 1.
When the next improvement costs more than it returns. Three weeks of engineering for 5% on an endpoint nobody complains about is a bad trade.
When the complexity is not worth it. A hand-optimised routine that nobody else can safely modify has a maintenance cost that lasts as long as the code does. Mark it with a comment explaining the constraint and the measurement, or the next person will "simplify" it back.
And the one to notice: when the bottleneck has moved. Fix the database and the bottleneck becomes serialisation; fix that and it becomes the network. There is always a bottleneck — the question is whether it is now below the level anyone experiences. That is what the target was for.
Recall
- Amdahl's law: optimising a part that is 3% of the time gains at most 3%. Measure before changing — and define the metric (latency or throughput), the percentile, and for whom.
- The mean hides the tail, and tail latency amplification means 20 backend calls with a 1% slow chance each give an 18% chance of a slow page. Fan-out systems live or die on p99.
- USE (utilisation, saturation, errors) for resources; RED (rate, errors, duration) for services. Saturation is the one people skip — 70% CPU with a run queue of 12 is saturated.
- Sampling profilers are the default (production-safe, statistical); instrumenting ones distort small functions. A CPU profile shows nothing for an I/O-bound service — use a wall-clock or off-CPU profile.
- In a flamegraph, width is proportion of samples and the x-axis is NOT time. Wide boxes at the top do the work; repeated identical subtrees mean a loop. Differential flamegraphs find regressions fastest.
- Microbenchmarks lie through dead code elimination, missing warm-up, unrealistic data, measurement overhead, environment noise, and one-run-each statistics. Interleave runs, report median and percentiles, and trust macro-benchmarks.
- The hierarchy: do not do it → do it less often → do it later or elsewhere → in parallel → faster (algorithm first, micro last). Recurring wins: N+1, missing index, chatty calls, oversized payloads, allocation churn, lock contention, sync on an async path, cold caches.
- Load test with an open model, and beware coordinated omission — a generator that waits for a slow response never sends (or records) the requests that should have happened. A CI performance budget is what stops a year of small regressions.
Self-test: Why did a 40% faster serialiser change nothing? · Why does a fan-out architecture care about p99? · Which profile type do you need for an I/O-bound service? · What does the x-axis of a flamegraph represent? · Name three ways a microbenchmark produces a false result · What does coordinated omission hide?
Next: 14.6 covers the skill that separates engineers more than any other — debugging as a method rather than as guessing, and the systematic approach that works when the obvious explanations are wrong.