14.6 — Debugging as a Discipline

Two engineers get the same bug. One finds it in twenty minutes; the other is still on it two days later.

The difference is usually not knowledge of the system. It is that one of them formed a hypothesis, designed an experiment that could disprove it, and wrote down the result — and the other read the code repeatedly hoping to spot something, then changed things to see what happened.

Debugging is a method. It is teachable, most engineers were never taught it, and it separates people more than almost any other skill.

1. The method

Observe → hypothesise → predict → test → refine.

Observe exactly what happens. Not "it is broken" — the precise input, the precise output, the timing, the frequency, who and what is affected.

Hypothesise a specific cause. "The cache is returning a stale value", not "something is wrong with caching".

Predict something that must be true if the hypothesis holds — and, more usefully, something that must be false. A hypothesis that explains everything predicts nothing.

Test the prediction with the smallest experiment that distinguishes it.

Refine and repeat.

Write it down as you go. A three-line running note — tried X, expected Y, got Z — prevents the two failure modes of a long debugging session: re-testing something you already ruled out, and losing a clue you noticed an hour ago and did not understand yet. On anything lasting more than an hour, the note is what makes progress cumulative rather than circular.

2. Reproduce it first

A bug you can reproduce reliably is most of the way to being fixed. A bug you cannot reproduce is a research project.

So spend the effort here first:

Find the exact conditions. Which user, which data, which sequence, which environment, which time of day.

Then minimise. Remove everything that does not stop it reproducing. A twelve-step reproduction that becomes three steps has usually revealed the cause on the way, because each removal is an experiment.

If it only happens in production, work toward a smaller reproduction anyway — a specific request replayed, a specific record, a specific concurrency level. And if you truly cannot, section 7 covers debugging live.

If it is intermittent, quantify it. "One in fifty" and "one in ten thousand" point at different causes, and a loop that runs the operation a thousand times turns "sometimes" into a test.

3. The rules that do the work

Understand the system before you change it. Not all of it — the part you are in. Reading the code you are about to modify is faster than guessing at it, and this is where most of the two-day version of the story goes wrong.

Quit thinking and look. The single most valuable rule. Engineers reason about what the code should do instead of observing what it does. Print the value. Attach the debugger. Read the actual request. The assumption you did not check is where the bug lives, by definition — if you had checked it, you would have found it.

Divide and conquer. Section 4.

Change one thing at a time. Two changes and a fix leaves you not knowing which mattered — and one of them may have introduced the next bug.

Check the obvious. Is it running? Is it the version you think? Is it the environment you think? Is the configuration what you believe? Is it plugged in? An embarrassing proportion of long debugging sessions end here, and checking takes thirty seconds.

Get a fresh pair of eyes. Explaining the problem out loud is the cheapest debugging technique that exists — the rubber duck works because articulating it forces you to state assumptions you had been skipping. Roughly a third of the time you find it mid-sentence.

If you did not fix it, it is not fixed. A bug that "went away" after an unrelated change is still there. Confirm the fix by making the bug come back: revert the fix, see the failure, reapply it, see it gone. That is the only proof.

4. Bisection, in four dimensions

Halving the search space is the highest-leverage move available, and it applies to more than history.

In time — git bisect run (Chapter 14.1.2) finds the introducing commit in about \log_2 n tests, unattended.

In space — is the failure in the client, the network, the load balancer, the application or the database? Test at the boundary: curl the service directly, bypassing the balancer; run the query in a client, bypassing the application. Each test eliminates half the system.

In data — does it fail for every record or one? Halve the dataset and re-run. A bug that reproduces on one row out of a million is nearly solved, because you can then look at that row.

In configuration — diff the working environment against the broken one, then halve the differences. "Works locally, fails in production" is always a difference, and the list of candidates is finite: data volume, configuration, permissions, versions, timezone, locale, filesystem case sensitivity, network policy, resource limits.

5. Reading what you are given

Read the whole stack trace, not the top line. The innermost frame is where the exception was raised, which is often not where the bug is — a null appearing three frames down was created five frames up. Look for the first frame in your own code, and start there.

Read the cause chain to the bottom. Caused by: sections are usually more informative than the wrapper.

In a cascade, the first error is the real one. A hundred timeouts following one connection-pool exhaustion are all symptoms. Sort by time and read the earliest, which is the opposite of what a dashboard showing "most frequent error" encourages.

Read the error literally. permission denied means permission, not "probably a network issue". connection refused means nothing was listening — a different thing from a timeout, which means something accepted and did not answer (Chapter 5.4.2). These distinctions each eliminate half the possibilities, and skipping past them is how a five-minute problem becomes an afternoon.

6. Instrumentation and tools

Logging. Log the inputs to the function that misbehaves, not just the output — you usually find the wrong value arrived rather than being computed. Structured logs with a correlation id (Chapter 10.10) are what make a distributed failure traceable at all.

Temporary high-detail logging behind a flag is legitimate: a debug mode that logs everything for one user id, enabled in production for ten minutes. Far better than shipping print statements, and it can stay for next time.

The debugger, properly (Chapter 14.2 makes the case). Three features that are underused:

• Conditional breakpoints — stop when orderId == "ord_991". Replaces a dozen print-and-rerun cycles.
• Watchpoints — break when a variable changes, rather than at a line. This is the tool for "who is setting this to null", which is otherwise painful.
• Post-mortem debugging — load a core dump or a crash snapshot and inspect the state at the moment of failure, for something that cannot be reproduced live.

Time-travel debugging (rr and similar) records an execution and lets you step backwards. For a hard non-deterministic bug it is transformative, because you can run from the failure back to the cause instead of guessing where to breakpoint.

Sanitizers and dynamic analysis — address and thread sanitizers, race detectors, valgrind — find the class of bug where the symptom appears far from the cause.

Distributed tracing for anything crossing services: one trace shows which hop consumed the time and where the error originated.

7. Heisenbugs and production

A bug that disappears when observed usually means timing: adding a log statement changed the interleaving of a race (Chapter 9.5.1), or a debugger's pause let a slow operation finish.

Strategies that do not perturb timing:

• A ring buffer in memory, dumped on failure — records history with almost no cost.
• Sampling rather than logging every event.
• Stress it — more threads, more load, more iterations. A race that appears once a day appears in a minute under contention.
• Reduce nondeterminism deliberately: fixed seeds, injected clocks, a single-threaded mode for reproduction (Chapter 9.7.28 uses exactly this).

In production you cannot attach a debugger, so the toolkit is different:

• Metrics, logs and traces — which is why observability is built before you need it.
• Continuous profiling (Chapter 14.5).
• Dynamic instrumentation — eBPF can observe a running process without restarting or modifying it.
• Core dumps on crash, analysed offline.
• Canary comparison — one instance on the new version, compared directly against the old.

And the mindset: every production experiment must be safe and reversible. A feature flag, one instance, a read-only query, a shadow request. "Let me just try restarting it" destroys the evidence, and the bug returns tomorrow with nothing learned.

8. The usual suspects

When the obvious explanations are exhausted, this list is worth walking, because these recur:

Caching — yours, the framework's, the CDN's, the browser's, the DNS resolver's. "I deployed the fix and it is still broken" is a cache more often than not.

Configuration differing between environments — the single largest category of "works locally".

Clock skew — expiring tokens, Kerberos, TLS, ordering by timestamp (Chapters 8.4.8, 10.3).

Timezones and daylight saving — a job that runs at 00:30 breaks twice a year, and a date that is one day off is almost always a UTC conversion.

Encoding — UTF-8 versus a legacy code page, a byte-order mark at the start of a file, a filename that is not valid UTF-8.

Case sensitivity — a filename or a database collation that differs between a developer's machine and a Linux server.

Someone else's change — a deploy, a config change, a certificate rotation, a dependency's new version, a provider incident. Check the deployment log and the provider's status page early, because it costs a minute.

A limit you did not know existed — connections, file descriptors, disk, memory, request size, rate limit, quota. These fail suddenly and completely at a threshold, which is why the symptom is "it was fine yesterday".

Retries hiding a failure, so the error rate looks normal and latency does not.

9. Three worked cases

Intermittent 502s, roughly 0.1% of requests, no pattern. The load balancer's idle timeout was 60 seconds and the application server's keep-alive timeout was 5. The server closed an idle connection at the moment the balancer reused it, producing a race with a tiny window. The fix is one line — the application's keep-alive must be longer than the balancer's idle timeout — and the diagnosis came from noticing the failures clustered at exactly the reuse boundary.

Works locally, fails in production, "no differences". The team compared code and configuration and found nothing. The difference was the filesystem: a developer's machine was case-insensitive, and an import './Utils' of a file named utils.ts worked locally and failed on Linux. Environment differences are rarely in the list people check.

Slow only for one customer. The endpoint was fast for everyone except one account, whose data had 40,000 items where the median was 12. An N+1 query (Chapter 7.3.2) invisible at 12 items is 40,000 round trips at that size. The general lesson: test with the shape of your largest customer's data, not the average.

10. Afterwards

A good bug report contains the exact reproduction steps, what you expected, what happened, the environment and version, and the relevant logs or trace id. "It does not work" costs someone else an hour of the work you already did.

Blameless postmortems ask what made the failure possible, not who caused it. The reason is practical rather than kind: in a blaming culture people stop reporting, and the information you need to prevent the next one stops arriving.

The five whys is a starting technique with a real limitation — it produces a single chain of causes, and real incidents have several contributing factors. Ask "what else had to be true" alongside "why", or you will find one cause and fix a system that had four.

Every postmortem produces action items with owners and dates, or it produced nothing. And the most valuable action is usually not the fix — it is the detection: what would have told us sooner, and can we alert on that now.

11. Knowing when to stop

Set a time box. Ninety minutes with no progress means the approach is wrong, not that you need more of it.

Then change something structural: explain it to someone, take a break (the walk-away effect is real and well documented), read the code path end to end instead of poking at it, or go back and question an assumption you have been treating as fact.

Ask for help earlier than feels comfortable. The cost of ten minutes of a colleague's time against another day of yours is not close. And bring the running note from section 1 — "here is what I have ruled out and how" makes you fast to help rather than a fresh start for someone else.

Recall

• Observe → hypothesise → predict → test → refine, and write it down. A hypothesis must predict something that would be false if it is wrong. The note is what makes a long session cumulative rather than circular.
• Reproduce first, then minimise. A twelve-step reproduction that becomes three has usually revealed the cause. Quantify "intermittent" — one in fifty and one in ten thousand point at different causes.
• Quit thinking and look. The bug is in the assumption you did not check, by definition. Change one thing at a time, check the obvious, explain it out loud, and prove the fix by making the bug come back.
• Bisect in four dimensions: time (git bisect run), space (test at each boundary), data (halve the dataset), configuration (diff the environments).
• Read the whole stack and the cause chain; in a cascade the earliest error is the real one; and read errors literally — connection refused and a timeout are different failures.
• Log the inputs to the failing function, use correlation ids, and prefer a flagged debug mode over shipped print statements. Underused debugger features: conditional breakpoints, watchpoints ("who set this to null"), post-mortem dumps, and time-travel replay.
• Heisenbugs are timing: use a ring buffer, stress the system, and remove nondeterminism with fixed seeds and injected clocks. In production every experiment must be safe and reversible — restarting destroys the evidence.
• Usual suspects: caching, environment configuration, clock skew, timezones, encoding, case sensitivity, someone else's deploy, an unknown limit, retries hiding failures. Afterwards: blameless postmortems (because blame stops reports), "what else had to be true" alongside the five whys, and an action item about detection.

Self-test: What must a good hypothesis predict? · Which of the four bisection dimensions applies to "works locally, fails in production"? · Why is the earliest error the important one in a cascade? · What is a watchpoint for? · Why must a production experiment be reversible? · What is the most valuable postmortem action item usually about?

Next: 14.7 covers the other half of engineering effectiveness — design documents, code review that changes outcomes, and writing that gets read.