10.9 — Reliability: Timeouts, Retries, Breakers, Bulkheads & Error Budgets

Distributed systems fail partially and constantly (10.1); reliability engineering is the discipline of failing well — degrading instead of collapsing, and containing damage instead of amplifying it. This page covers the mechanisms in the order they compose (timeout → retry → circuit breaker → bulkhead → fallback), the amplification traps that turn small failures into outages, and the SLI/SLO/error-budget framework that decides how much reliability to buy.

1. Timeouts: the foundation

Every network call must have a timeout, and defaults are almost always wrong (many HTTP clients default to no timeout, meaning a hung dependency holds your resources forever — the connection-pool exhaustion incident that starts most cascade stories).

Set them from data, not intuition: a timeout should sit slightly above the dependency's p99.9 latency for that operation — too tight and you fail healthy requests (and, if you retry, double the load on a struggling service); too loose and you hold threads, connections, and memory while users wait for a response nobody wants. Three refinements that separate careful systems: separate connect and read timeouts (a refused connection should fail in milliseconds; a slow response may legitimately take seconds); per-operation values (a search and a report generation do not share a budget); and — the one most teams miss — deadline propagation: pass the remaining budget down the call chain (gRPC deadlines, a X-Request-Deadline header, AbortSignal — 3.8.5) so that a service with 50 ms left doesn't start a 2-second operation whose result the caller has already abandoned. Without propagation, work continues for callers who have given up — pure waste at exactly the moment capacity is scarce.

2. Retries — and the amplification trap

Retries convert transient failures into successes, and un-tuned retries convert small failures into outages. The rules: ^{⚑Retry strategies, exponential backoff and jitter. [EQ-495b]}

• Only retry what's safe to repeat — idempotent operations, or non-idempotent ones carrying an idempotency key (10.4). Retrying a bare POST is how double charges happen.
• Only retry what can succeed — a 400 or 422 will fail identically forever; retry 429, 503, timeouts, and connection errors. Classify errors before retrying (9.9.3's operational-vs-programming split).
• Exponential backoff with jitter — doubling delays gives a struggling dependency room; jitter (randomizing the delay) prevents the thundering herd where every client retries at the same instant, hammering the recovering service in synchronized waves. Full jitter (random(0, base × 2^attempt)) is the well-documented default.
• Cap attempts and total time — bounded by the caller's deadline (section 1), because a retry that outlives its caller's patience is pure load.
• Retry budgets — the fleet-level control: allow retries to be at most a small fraction (say 10%) of total requests; beyond that, fail fast. This is what actually prevents retry amplification: with three services each retrying 3×, a single failure at the bottom becomes 27 requests — a self-inflicted DDoS at the worst possible moment.

3. Circuit breakers

When a dependency is down rather than flaky, retries are worse than useless — they waste your resources, add latency to every request, and delay the dependency's recovery. A circuit breaker is a state machine (9.4.14) that stops calls to a failing dependency: ^{⚑What is a circuit breaker and how does it work? [EQ-493b]}

• Closed — calls pass through; failures are counted (rate or consecutive count within a window).
• Open — the threshold was crossed: calls fail immediately without touching the dependency (fast failure, no resource consumption, no added load on the sick service). After a cooldown, move to half-open.
• Half-open — allow a small number of trial calls: success closes the circuit, failure re-opens it (with a longer cooldown). This is the hysteresis that prevents flapping (9.7.6's health machine, same logic).

Design details that matter: the breaker's granularity should match the failure domain (per-dependency, and often per-endpoint — a slow /reports shouldn't open the circuit for /health); the open state should have a fallback (section 5) rather than merely erroring; and the breaker's state transitions must be observable (an open circuit is one of the highest-signal alerts a system can emit — it means "we have stopped calling X").

Figure 1 — The circuit breaker. Failing fast in the open state protects both sides: the caller stops burning threads and time, and the failing dependency stops receiving load while it recovers. Half-open trials with hysteresis prevent oscillation.

4. Bulkheads and load shedding

Bulkheads (named for a ship's watertight compartments) isolate resources so one failure can't consume everything: separate connection pools per dependency (so a slow partner API can't exhaust the pool your database needs), separate thread/worker pools per workload (9.5.6), per-tenant quotas (one customer's spike doesn't starve the rest), and separate deployments for critical vs non-critical paths (checkout and reporting on different fleets). The failure they prevent is the one every incident review recognizes: "a slow third-party recommendation service consumed all our worker threads, and the whole site went down."

Load shedding is the deliberate refusal of work when overloaded — rejecting requests fast with 429/503 rather than accepting everything and collapsing. It's counterintuitive but essential: a queue of 10,000 requests each timing out serves nobody, while shedding 40% keeps 60% healthy. Refinements: prioritize (shed background and low-value traffic first — health checks and paying customers last), shed at the edge (cheaper than after auth and database work), and use adaptive thresholds keyed to a real saturation signal (queue depth, event-loop lag — 3.8.2) rather than static limits.

5. Graceful degradation and fallbacks

The design question every dependency deserves: what do we serve when this is down? Options, in descending quality: cached/stale data (a slightly old recommendation list is fine); a reduced experience (hide the personalization module, show the page); a default (standard shipping estimate instead of the calculated one); queue for later (accept the write, process when the dependency returns — 10.8.1); and only then an error. This classification — deciding per dependency whether it is critical (the feature genuinely cannot work) or enhancing (the feature is better with it) — is the highest-value reliability exercise a team can do, because most outages are enhancing dependencies taking down critical paths through shared resources or absent fallbacks.

6. SLIs, SLOs, and error budgets

How much reliability should you buy? The SRE framework answers it with numbers: ^{⚑SLI, SLO, SLA and error budgets. [EQ-1108b]}

• SLI (indicator) — a measured quality signal: request success rate, p99 latency, freshness. Measure it where the user is (at the edge or in the client), not deep in a service where a working component can report success while users see errors.
• SLO (objective) — the target: "99.9% of requests succeed over 30 days," "p99 < 300 ms." Internal, chosen deliberately.
• SLA — the contractual version with financial penalties; always looser than the SLO (you want to breach the internal target long before the customer-facing one).
• Error budget — the arithmetic that makes it actionable: 99.9% over 30 days allows ~43 minutes of failure. That budget is a currency: spend it on risky deploys and experiments; when it's exhausted, the team's priority shifts from features to reliability by prior agreement. This converts "how much reliability?" from an argument into a policy, and it explains why chasing 100% is wrong — each nine costs roughly ten times more, and beyond a point users cannot perceive the difference while your velocity does.

The related discipline: measure percentiles, not averages (10.12 does the math). An average hides the tail entirely; p50/p95/p99/p99.9 describe the experience of real users, and in fan-out systems the tail dominates — if a page makes 20 backend calls, a p99 on each means most pages hit at least one slow call (10.6's tail amplification).

7. The expert lens

Reliability mechanisms compose in a fixed order, and skipping one breaks the rest. Timeout (bound the wait) → retry with backoff and jitter (recover from transients) → circuit breaker (stop when it's not transient) → bulkhead (contain the damage) → fallback (serve something useful). A retry without a timeout is unbounded; a breaker without a fallback merely fails faster; a bulkhead without shedding just fills up more slowly. Reviewing a service's dependency configuration against this ladder finds real gaps in minutes.

Most large outages are amplification, not failure. The initial trigger is usually small — one slow shard, one bad deploy, a brief network blip — and the outage is manufactured by the system's response: synchronized retries, unbounded queues, cache stampedes ([7.6]), health checks failing en masse and restarting a healthy fleet (9.9.7). Design reviews should ask "what does our system do when this dependency slows by 10×?" — the answer, not the failure itself, determines whether you have an incident or an outage.

Error budgets end the reliability-versus-features argument by making it arithmetic. Without them, reliability is negotiated by whoever argues loudest after the last incident; with them, the budget's state dictates the tradeoff automatically and impersonally. It also legitimizes spending the budget: a team with budget remaining should be shipping riskier changes, because unused reliability is unrealized velocity.

Next: 10.10 — you cannot operate what you cannot see: logs, metrics, traces, and the alerting discipline that makes them actionable.

Recall

• Timeouts on every call, set from p99.9 data, with separate connect/read values, per-operation budgets, and deadline propagation down the chain (gRPC deadlines, AbortSignal) so nobody works for a caller who has given up.
• Retries: only what's safe (idempotent or keyed — 10.4) and only what can succeed (retry 429/503/timeouts, never 4xx logic errors); exponential backoff + jitter (prevents thundering herd); cap attempts and total time within the deadline; enforce a fleet-level retry budget — otherwise 3 services × 3 retries = 27× amplification.
• Circuit breaker states: closed (pass, count) → open (fail fast, no load on the sick dependency) → half-open (trial calls; success closes, failure re-opens with longer cooldown). Per-dependency/per-endpoint granularity; pair with a fallback; alert on state transitions.
• Bulkheads isolate resources (per-dependency pools, per-workload workers, per-tenant quotas, separate fleets) so one slow dependency can't consume everything. Load shedding refuses work fast when saturated — prioritized, at the edge, keyed to a real saturation signal.
• Degradation ladder: stale cache → reduced experience → default value → queue for later → error. Classify every dependency as critical vs enhancing — most outages are enhancing dependencies taking down critical paths.
• SLI (measured at the user) → SLO (internal target) → SLA (contractual, looser) → error budget (99.9% ≈ 43 min/30 days) as a currency: spend on risk, and when exhausted, reliability work preempts features. Percentiles, not averages — and tails dominate under fan-out.

Self-test: Why must deadlines propagate? Give the four retry rules and compute 3×3-service amplification. Walk the breaker's three states and say what half-open prevents. Name four bulkhead forms. What does an error budget convert an argument into?

Quiz Bank

FoundationalHow do you set timeouts correctly, and what is deadline propagation?

Set them from measured latency, not intuition: slightly above the dependency's p99.9 for that operation, so healthy-but-slow requests aren't killed while genuinely hung ones are released quickly. Too tight causes false failures (and, with retries, doubles load on a struggling service); too loose holds threads, connections, and memory — the resource exhaustion that begins most cascades. Refinements:

separate connect and read timeouts (connection refusal should fail in milliseconds; a legitimate slow response may take seconds); per-operation budgets (a report and a lookup shouldn't share one number); and never rely on client defaults, many of which are infinite.

Deadline propagation is the piece most teams miss: the caller's remaining time budget travels with the request (gRPC deadlines, a deadline header, an AbortSignal threaded through — 3.8.5), so a downstream service with 40 ms left refuses to begin a 2-second query, and every layer cancels work whose result nobody will read. Without it, a user who timed out ten seconds ago still has five services burning CPU on their behalf — waste that peaks exactly when the system is already saturated, which is why propagation is a reliability mechanism and not just tidiness.

FoundationalState the rules for safe retries and explain retry amplification.

(1) Only retry what is safe to repeat: idempotent operations, or non-idempotent ones carrying an idempotency key that the callee honors (10.4) — retrying a bare payment POST is how double charges occur. (2) Only retry what can succeed: 429, 503, timeouts, and connection errors are transient; 400/422 will fail identically forever and retrying them wastes capacity and hides bugs (9.9.3's error classification).

(3) Exponential backoff with jitter: doubling delays gives the dependency room to recover, and randomizing them prevents the thundering herd — thousands of clients whose retries synchronize into waves that re-kill a recovering service (full jitter, random(0, base × 2^attempt), is the standard).

(4) Bound attempts and total time within the caller's deadline, and enforce a retry budget at the fleet level (retries ≤ ~10% of requests; beyond that, fail fast). Amplification: each layer multiplies. Service A retries 3× into B, which retries 3× into C: one user request becomes up to 9 calls at C, and with a third layer, 27 — so a small hiccup at the bottom is met with an order-of-magnitude increase in load precisely when it's least able to cope. This is why most large outages are amplification rather than failure, and why budgets and jitter are not optional polish.

AppliedExplain the circuit breaker's states and what each one protects.

Closed — requests flow normally while the breaker counts failures (by rate within a rolling window, or consecutive failures). Protects nothing yet; it's the observation state. Open — the failure threshold was crossed, so calls fail immediately without contacting the dependency. This protects both sides: the caller stops consuming threads, connections, and latency budget on calls that will fail anyway (preventing resource exhaustion and cascade), and the failing dependency stops receiving traffic while it recovers (a service being hammered by retries cannot restart cleanly — the breaker is often what allows recovery).

Half-open — after a cooldown, a limited number of trial requests are allowed: success closes the circuit; failure re-opens it, typically with a longer cooldown. This is hysteresis, and it prevents flapping — without it, the breaker would close on the first success and immediately re-open under real load, oscillating traffic (9.7.6's health-machine lesson). Design notes: scope breakers per dependency and often per endpoint (a failing report endpoint shouldn't stop health checks); always pair the open state with a fallback (stale cache, default, queue — section 5) or you've only made failure faster; and emit metrics/alerts on state transitions, because "we have stopped calling X" is among the highest-signal events a system produces.

InterviewWhat are SLIs, SLOs, SLAs and error budgets, and why is 100% the wrong target?

SLI — a measured indicator of user-visible quality: success rate, p99 latency, freshness, availability. Crucially, measure it where the user experiences it (edge or client), because internal components can report success while users see failures. SLO — the internal objective for that indicator ("99.9% success over 30 days", "p99 < 300 ms"), chosen deliberately per service and user journey.

SLA — the contractual commitment with penalties, deliberately looser than the SLO so you breach internal targets long before customer-facing ones. Error budget — the arithmetic complement of the SLO: 99.9% over 30 days permits ≈43 minutes of failure; that budget is spendable currency. With budget remaining, ship risky changes and run experiments; when exhausted, a pre-agreed policy shifts priority from features to reliability — which replaces the post-incident argument with a rule everyone accepted in advance.

Why not 100%: each additional nine costs roughly an order of magnitude more (redundancy, testing, operational rigor), while users' perception saturates — their ISP, device, and network already impose more failure than your marginal nine removes; and a 100% target forbids any change, since deploys are the largest source of risk. The mature stance: pick the lowest reliability target users can't distinguish from perfect, then deliberately spend the remainder on velocity.

StaffPost-incident: a 30-second slowdown in a recommendation service (an optional homepage module) caused a 20-minute full-site outage. Reconstruct the amplification chain and prescribe the fixes.

The chain, reconstructed: (1) the recommendation service slowed to ~5 s per call; (2) the homepage handler called it without a timeout (or with a 30 s default), so request threads/connections piled up waiting; (3) those requests held the shared connection pool, so unrelated database calls started queueing — the failure crossed from an optional feature into critical paths through a shared resource; (4) health checks began timing out because the process was saturated, so the orchestrator restarted healthy instances (9.9.7), reducing capacity and re-cold-starting caches; (5) clients and the CDN retried failed page loads, multiplying inbound load; (6) recovery was prevented because every restarted instance immediately re-saturated. Note that steps 2–6 are all our system's behavior — the trigger lasted 30 seconds, the outage 20 minutes (section 7).

Fixes by layer, in priority order: (a) Timeout on the recommendation call at ~p99.9 (a few hundred ms) with deadline propagation so it can never exceed the page's budget. (b) Fallback and classification: recommendations are an enhancing dependency — the homepage must render without them (cached list, or hide the module); this single decision would have prevented the outage regardless of everything else. (c) Bulkhead: separate connection/thread pools per dependency so a slow optional call cannot consume the resources the critical path needs — the mechanism that stops feature failures becoming site failures.

(d) Circuit breaker on the recommendation client with a stale-cache fallback, so a sustained slowdown stops being attempted at all. (e) Health-check hygiene: liveness must be process-local (9.9.7) so dependency slowness never triggers mass restarts; readiness may shed traffic without killing instances. (f) Retry discipline: client and CDN retry budgets with jitter, plus load shedding at the edge so saturation produces fast 503s rather than infinite queueing.

Process changes: add "what happens if this dependency slows 10×?" to design review; classify every dependency critical vs enhancing in the service catalog; and run a game day injecting latency into a dependency to verify the fallback path actually works — because a fallback nobody has exercised is a hypothesis, not a control.

Flashcards

FlashThe reliability ladder

Timeout → retry (backoff + jitter) → circuit breaker → bulkhead → fallback. Skipping one breaks the next: retries without timeouts are unbounded; breakers without fallbacks just fail faster.

FlashRetry rules

Safe to repeat (idempotent/keyed) · can succeed (429/503/timeouts, not 4xx) · exponential backoff + jitter · bounded by deadline · fleet retry budget (~10%). 3 layers × 3 retries = 27×.

FlashDeadline propagation

Pass remaining budget downstream so nobody starts work the caller has abandoned. Waste peaks exactly when capacity is scarce.

FlashBreaker states

Closed (pass, count) → open (fail fast, protects both sides) → half-open (trials; success closes, failure re-opens longer). Hysteresis prevents flapping. Alert on transitions.

FlashBulkhead + shedding

Isolate pools per dependency/workload/tenant; shed fast at the edge by priority when saturated. A queue of timeouts serves nobody; shedding 40% keeps 60% healthy.

FlashError budget

SLO 99.9%/30d ≈ 43 min. Budget = currency: spend on risky changes; exhausted ⇒ reliability preempts features. Each nine costs ~10×; 100% forbids change.

Scenario Drill

DrillDesign the reliability configuration for a checkout flow that calls: inventory (internal, critical), pricing (internal, critical), payments (external PSP, critical), fraud scoring (external, enhancing), recommendations (internal, enhancing), and email (internal, async). Specify timeouts, retries, breakers, bulkheads, fallbacks, and the SLO — then state what happens when each dependency is down.

Classification first, because it determines everything: critical = inventory, pricing, payments; enhancing = fraud scoring, recommendations; async = email. Per-dependency configuration. Inventory (p99 ~20 ms): timeout 150 ms, 1 retry with jitter (the conditional decrement is idempotent by design — 9.5.4), breaker per-endpoint, no fallback — if inventory is down, checkout genuinely cannot proceed; fail fast with a clear message and a retry-later affordance.

Pricing (p99 ~30 ms): timeout 200 ms, 1 retry, breaker with a stale-price cache fallback bounded by a freshness rule (serve cached prices up to N minutes old; beyond that, fail) — a business decision, not an engineering one, and worth having explicitly. Payments/PSP (p99 ~800 ms, external): timeout 3 s (external variance is real), no blind retries — retry only with the idempotency key the PSP honors (10.4/9.6.3), maximum 2 attempts, breaker with generous thresholds (PSPs have brief blips; opening too eagerly rejects revenue), fallback = queue the authorization attempt and inform the user honestly rather than silently accepting.

Fraud scoring (external, enhancing): timeout 200 ms — aggressive, zero retries, breaker with a fast trip, fallback = default risk decision by policy (usually "approve with monitoring" for low amounts, "hold for manual review" above a threshold) — the enhancing dependency must never block checkout, and the fallback is a risk choice signed off by the fraud team.

Recommendations (enhancing): timeout 100 ms, no retries, breaker, fallback = omit the module. Email (async): not in the request path at all — an outbox event and a queue consumer (10.8.4), so its availability is irrelevant to checkout.

Bulkheads: separate connection pools per dependency, with the two enhancing dependencies sharing a small pool that cannot starve the critical ones — the section 7 lesson pre-applied; plus a separate worker fleet for report/admin traffic so checkout capacity is never consumed by internal tooling.

Load shedding: at the edge, prioritized — shed anonymous browsing before authenticated checkout; keep health checks always. SLO: checkout success rate 99.95% and p99 latency < 2.5 s over 30 days (≈22 minutes of budget), measured at the client (a completed purchase, not a 200 from a service); error budget policy agreed in writing.

What happens when each is down: inventory → checkout unavailable (honest error, retry affordance, incident); pricing → cached prices within freshness window, else unavailable; PSP → attempts queued, user told "we're confirming your payment", saga compensations ready (10.8.4); fraud → policy default applied, flagged for review, business notified; recommendations → module hidden, nobody notices; email → messages queue and flush when it returns.

The artifact this drill produces — a table of dependency, classification, timeout, retry policy, breaker config, fallback, and blast radius — is the single most useful reliability document a service can have, and it's reviewable in ten minutes by anyone joining the team.