10.9 — Reliability: Timeouts, Retries, Breakers, Bulkheads & Error Budgets

Distributed systems fail partially and constantly (10.1); reliability engineering is the discipline of failing well — degrading instead of collapsing, and containing damage instead of amplifying it. This page covers the mechanisms in the order they compose (timeout → retry → circuit breaker → bulkhead → fallback), the amplification traps that turn small failures into outages, and the SLI/SLO/error-budget framework that decides how much reliability to buy.

1. Timeouts: the foundation

Every network call must have a timeout, and defaults are almost always wrong (many HTTP clients default to no timeout, meaning a hung dependency holds your resources forever — the connection-pool exhaustion incident that starts most cascade stories).

Set them from data, not gut feeling. A timeout should sit just above that dependency's p99.9 response time for that operation. Too tight, and you fail healthy requests. If you then retry them, you double the load on a service that is already struggling. Too loose, and you tie up threads, connections and memory while users wait for an answer nobody wants any more.

Three refinements mark a careful system:

• Separate connect and read timeouts. A refused connection should fail in milliseconds. A slow response may honestly need seconds.
• A value per operation. A search and a report do not share the same budget.
• Pass the deadline down, the one most teams miss. Send the time remaining along the chain of calls, using gRPC deadlines, an X-Request-Deadline header or an AbortSignal (3.8.5). Then a service with 50 ms left does not start a 2-second job whose result the caller has already given up on.

Without propagation, work continues for callers who have given up — pure waste at exactly the moment capacity is scarce.

2. Retries — and the amplification trap

Retries convert transient failures into successes, and un-tuned retries convert small failures into outages. The rules: ^{⚑Retry strategies, exponential backoff and jitter. [EQ-495b]}

• Only retry what's safe to repeat — idempotent operations, or non-idempotent ones carrying an idempotency key (10.4). Retrying a bare POST is how double charges happen.
• Only retry what can succeed — a 400 or 422 will fail identically forever; retry 429, 503, timeouts, and connection errors. Classify errors before retrying (9.9.3's operational-vs-programming split).
• Exponential backoff with jitter — doubling delays gives a struggling dependency room; jitter (randomizing the delay) prevents the thundering herd where every client retries at the same instant, hammering the recovering service in synchronized waves. Full jitter (random(0, base × 2^attempt)) is the well-documented default.
• Cap attempts and total time — bounded by the caller's deadline (section 1), because a retry that outlives its caller's patience is pure load.
• Retry budgets — the fleet-level control: allow retries to be at most a small fraction (say 10%) of total requests; beyond that, fail fast. This is what actually prevents retry amplification: with three services each retrying 3×, a single failure at the bottom becomes 27 requests — a self-inflicted DDoS at the worst possible moment.

3. Circuit breakers

When a dependency is down rather than flaky, retries are worse than useless — they waste your resources, add latency to every request, and delay the dependency's recovery. A circuit breaker is a state machine (9.4.14) that stops calls to a failing dependency: ^{⚑What is a circuit breaker and how does it work? [EQ-493b]}

• Closed — calls pass through; failures are counted (rate or consecutive count within a window).
• Open — the threshold was crossed: calls fail immediately without touching the dependency (fast failure, no resource consumption, no added load on the sick service). After a cooldown, move to half-open.
• Half-open — allow a small number of trial calls: success closes the circuit, failure re-opens it (with a longer cooldown). This is the hysteresis that prevents flapping (9.7.6's health machine, same logic).

Design details that matter:

• One breaker per thing that can fail. Use one per dependency, and often one per endpoint. A slow /reports should not open the circuit for /health.
• A fallback when it is open (section 5), instead of just an error.
• Visible changes of state. An open circuit is one of the most useful alerts a system can send. It means "we have stopped calling X".

Figure 1 — The circuit breaker. Failing fast in the open state protects both sides: the caller stops burning threads and time, and the failing dependency stops receiving load while it recovers. Half-open trials with hysteresis prevent oscillation.

Building one

The checkout service calls a payment provider. When the provider starts returning errors, this wrapper stops the calls after three consecutive failures, fails fast for ten seconds, and then lets exactly one trial call through to see whether the provider has recovered.

type State = "closed" | "open" | "half-open";

class CircuitOpenError extends Error {}

class CircuitBreaker {
  private state: State = "closed";
  private failures = 0;
  private openedAt = 0;
  private trialInFlight = false;

  constructor(
    private readonly failureThreshold = 5,       // (1) consecutive failures that open the circuit
    private readonly cooldownMs = 10_000,
    private readonly now = () => performance.now(),   // (2) monotonic clock
  ) {}

  async call<T>(action: () => Promise<T>): Promise<T> {
    if (this.state === "open") {
      if (this.now() - this.openedAt < this.cooldownMs) {
        throw new CircuitOpenError("circuit open: failing fast");     // (3)
      }
      this.state = "half-open";
    }
    if (this.state === "half-open") {
      if (this.trialInFlight) throw new CircuitOpenError("trial call in progress");   // (4)
      this.trialInFlight = true;
    }
    try {
      const result = await action();
      this.onSuccess();
      return result;
    } catch (err) {
      this.onFailure();
      throw err;
    } finally {
      this.trialInFlight = false;
    }
  }

  private onSuccess(): void {
    this.state = "closed";                         // (5)
    this.failures = 0;
  }

  private onFailure(): void {
    this.failures++;
    if (this.state === "half-open" || this.failures >= this.failureThreshold) {   // (6)
      this.state = "open";
      this.openedAt = this.now();
    }
  }

  get current(): State { return this.state; }
}

1. The breaker counts consecutive failures. Production breakers often use a failure rate over a sliding window instead, so that one success among many failures does not reset the count; the state machine is the same.
2. Time is read from a monotonic clock, which never jumps when the wall clock is corrected (10.3). It is passed in as a function so the behaviour can be checked without waiting ten real seconds.
3. While open, calls fail immediately, without touching the provider. That is the entire benefit: no threads tied up waiting, and no extra load on a service that is already struggling.
4. Half-open allows one trial, not a flood. A tempting version simply switches to half-open and lets every waiting request through; at that moment hundreds of requests hit a provider that may have only just come back, which can knock it straight over again. Here the first caller becomes the trial and everyone else keeps failing fast until it finishes.
5. Any success closes the circuit and clears the count.
6. A failure while half-open re-opens the circuit immediately and restarts the cooldown; otherwise the circuit opens when the threshold is reached.

Run against a fake payment provider with a threshold of 3: five calls to a failing provider leave the circuit open after only 3 of them reached it, the last two failing fast. After the cooldown, with the provider healthy again, four calls arrive at the same moment: one becomes the trial and succeeds, the other three are refused with trial call in progress, and the circuit closes. When the provider fails again, three failures re-open it, and a failed trial after the next cooldown keeps it open.

4. Bulkheads and load shedding

Bulkheads are named after the watertight walls inside a ship. They keep resources apart, so one failure cannot use up everything:

• A separate connection pool for each dependency, so a slow partner API cannot use up the connections your database needs.
• A separate pool of threads or workers for each kind of work (9.5.6).
• A quota per customer, so one customer's spike does not starve the rest.
• Separate deployments for critical and non-critical paths, for example checkout and reporting on different servers.

The failure they prevent is the one every incident review recognizes: "a slow third-party recommendation service consumed all our worker threads, and the whole site went down."

Load shedding is the deliberate refusal of work when overloaded — rejecting requests fast with 429/503 rather than accepting everything and collapsing. It's counterintuitive but essential: a queue of 10,000 requests each timing out serves nobody, while shedding 40% keeps 60% healthy. Refinements: prioritize (shed background and low-value traffic first — health checks and paying customers last), shed at the edge (cheaper than after auth and database work), and use adaptive thresholds keyed to a real saturation signal (queue depth, event-loop lag — 3.8.2) rather than static limits.

5. Graceful degradation and fallbacks

The design question every dependency deserves: what do we serve when this is down? The options, best first:

1. Cached or slightly old data. A recommendation list that is a bit old is fine.
2. A smaller experience. Hide the personalised section, and still show the page.
3. A default. Show the standard delivery estimate instead of the calculated one.
4. Queue it for later. Accept the write, and process it when the dependency comes back (10.8.1).
5. Only then, an error.

The most valuable reliability exercise a team can do is to sort every dependency into one of two kinds. Critical means the feature really cannot work without it. Enhancing means the feature is better with it. Most outages are an enhancing dependency taking down a critical path, either through a shared resource or because there was no fallback.

6. SLIs, SLOs, and error budgets

How much reliability should you buy? The SRE framework answers it with numbers: ^{⚑SLI, SLO, SLA and error budgets. [EQ-1108b]}

• SLI (indicator) — a measured quality signal: request success rate, p99 latency, freshness. Measure it where the user is (at the edge or in the client), not deep in a service where a working component can report success while users see errors.
• SLO (objective) — the target: "99.9% of requests succeed over 30 days," "p99 < 300 ms." Internal, chosen deliberately.
• SLA — the contractual version with financial penalties; always looser than the SLO (you want to breach the internal target long before the customer-facing one).
• Error budget — the arithmetic that makes it actionable: 99.9% over 30 days allows ~43 minutes of failure. That budget is a currency: spend it on risky deploys and experiments; when it's exhausted, the team's priority shifts from features to reliability by prior agreement. This converts "how much reliability?" from an argument into a policy, and it explains why chasing 100% is wrong — each nine costs roughly ten times more, and beyond a point users cannot perceive the difference while your velocity does.

The related discipline: measure percentiles, not averages (10.12 does the math). An average hides the tail entirely; p50/p95/p99/p99.9 describe the experience of real users, and in fan-out systems the tail dominates — if a page makes 20 backend calls, a p99 on each means most pages hit at least one slow call (10.6's tail amplification).

7. The expert lens

Reliability mechanisms compose in a fixed order, and skipping one breaks the rest. Timeout (bound the wait) → retry with backoff and jitter (recover from transients) → circuit breaker (stop when it's not transient) → bulkhead (contain the damage) → fallback (serve something useful). A retry without a timeout is unbounded; a breaker without a fallback merely fails faster; a bulkhead without shedding just fills up more slowly. Reviewing a service's dependency configuration against this ladder finds real gaps in minutes.

Most large outages are amplification, not failure. The initial trigger is usually small — one slow shard, one bad deploy, a brief network blip — and the outage is manufactured by the system's response: synchronized retries, unbounded queues, cache stampedes ([7.6]), health checks failing en masse and restarting a healthy fleet (9.9.7). Design reviews should ask "what does our system do when this dependency slows by 10×?" — the answer, not the failure itself, determines whether you have an incident or an outage.

Error budgets end the reliability-versus-features argument by making it arithmetic. Without them, reliability is negotiated by whoever argues loudest after the last incident; with them, the budget's state dictates the tradeoff automatically and impersonally. It also legitimizes spending the budget: a team with budget remaining should be shipping riskier changes, because unused reliability is unrealized velocity.

Next: 10.10 — you cannot operate what you cannot see: logs, metrics, traces, and the alerting discipline that makes them actionable.

Recall

• Timeouts on every call, set from p99.9 data, with separate connect/read values, per-operation budgets, and deadline propagation down the chain (gRPC deadlines, AbortSignal) so nobody works for a caller who has given up.
• Retries:
  • Retry only what is safe to repeat, meaning idempotent or protected by a key (10.4).
  • Retry only what might succeed next time: 429, 503 and timeouts. Never retry a 4xx caused by bad input.
  • Use exponential backoff with jitter, so clients do not all retry at the same instant (the thundering herd).
  • Limit both the number of attempts and the total time, inside the deadline.
  • Set a retry budget for the whole fleet. Otherwise 3 layers of services, each retrying 3 times, multiply the load by 27.
• Circuit breaker states: closed (pass, count) → open (fail fast, no load on the sick dependency) → half-open (trial calls; success closes, failure re-opens with longer cooldown). Per-dependency/per-endpoint granularity; pair with a fallback; alert on state transitions.
• Bulkheads isolate resources (per-dependency pools, per-workload workers, per-tenant quotas, separate fleets) so one slow dependency can't consume everything. Load shedding refuses work fast when saturated — prioritized, at the edge, keyed to a real saturation signal.
• Degradation ladder: stale cache → reduced experience → default value → queue for later → error. Classify every dependency as critical vs enhancing — most outages are enhancing dependencies taking down critical paths.
• SLI (measured at the user) → SLO (internal target) → SLA (contractual, looser) → error budget (99.9% ≈ 43 min/30 days) as a currency: spend on risk, and when exhausted, reliability work preempts features. Percentiles, not averages — and tails dominate under fan-out.

Self-test: Why must deadlines propagate? Give the four retry rules and compute 3×3-service amplification. Walk the breaker's three states and say what half-open prevents. Name four bulkhead forms. What does an error budget convert an argument into?

Quiz Bank

FoundationalHow do you set timeouts correctly, and what is deadline propagation?

Set them from measured latency, not intuition: slightly above the dependency's p99.9 for that operation, so healthy-but-slow requests aren't killed while genuinely hung ones are released quickly. Too tight causes false failures (and, with retries, doubles load on a struggling service); too loose holds threads, connections, and memory — the resource exhaustion that begins most cascades. Refinements:

separate connect and read timeouts (connection refusal should fail in milliseconds; a legitimate slow response may take seconds); per-operation budgets (a report and a lookup shouldn't share one number); and never rely on client defaults, many of which are infinite.

Deadline propagation is the piece most teams miss: the caller's remaining time budget travels with the request (gRPC deadlines, a deadline header, an AbortSignal threaded through — 3.8.5), so a downstream service with 40 ms left refuses to begin a 2-second query, and every layer cancels work whose result nobody will read. Without it, a user who timed out ten seconds ago still has five services burning CPU on their behalf — waste that peaks exactly when the system is already saturated, which is why propagation is a reliability mechanism and not just tidiness.

FoundationalState the rules for safe retries and explain retry amplification.

(1) Only retry what is safe to repeat: idempotent operations, or non-idempotent ones carrying an idempotency key that the callee honors (10.4) — retrying a bare payment POST is how double charges occur. (2) Only retry what can succeed: 429, 503, timeouts, and connection errors are transient; 400/422 will fail identically forever and retrying them wastes capacity and hides bugs (9.9.3's error classification).

(3) Exponential backoff with jitter: doubling delays gives the dependency room to recover, and randomizing them prevents the thundering herd — thousands of clients whose retries synchronize into waves that re-kill a recovering service (full jitter, random(0, base × 2^attempt), is the standard).

(4) Bound attempts and total time within the caller's deadline, and enforce a retry budget at the fleet level (retries ≤ ~10% of requests; beyond that, fail fast). Amplification: each layer multiplies. Service A retries 3× into B, which retries 3× into C: one user request becomes up to 9 calls at C, and with a third layer, 27 — so a small hiccup at the bottom is met with an order-of-magnitude increase in load precisely when it's least able to cope. This is why most large outages are amplification rather than failure, and why budgets and jitter are not optional polish.

AppliedExplain the circuit breaker's states and what each one protects.

Closed — requests flow normally while the breaker counts failures (by rate within a rolling window, or consecutive failures). Protects nothing yet; it's the observation state. Open — the failure threshold was crossed, so calls fail immediately without contacting the dependency. This protects both sides: the caller stops consuming threads, connections, and latency budget on calls that will fail anyway (preventing resource exhaustion and cascade), and the failing dependency stops receiving traffic while it recovers (a service being hammered by retries cannot restart cleanly — the breaker is often what allows recovery).

Half-open — after a cooldown, a limited number of trial requests are allowed: success closes the circuit; failure re-opens it, typically with a longer cooldown. Waiting like this before changing state is called hysteresis, and it stops the breaker flipping back and forth. Without it, the breaker would close on the first success, then open again at once under real load, and traffic would swing on and off (9.7.6).

Design notes:

• Have a breaker per dependency, and often per endpoint. A failing report endpoint should not stop health checks.
• Always pair the open state with a fallback: a cached copy, a default or a queue (section 5). Otherwise you have only made failure faster.
• Record and alert on every change of state. "We have stopped calling X" is one of the most useful signals a system can produce.

InterviewWhat are SLIs, SLOs, SLAs and error budgets, and why is 100% the wrong target?

SLI — a measured indicator of user-visible quality: success rate, p99 latency, freshness, availability. Crucially, measure it where the user experiences it (edge or client), because internal components can report success while users see failures. SLO — the internal objective for that indicator ("99.9% success over 30 days", "p99 < 300 ms"), chosen deliberately per service and user journey.

SLA — the contractual commitment with penalties, deliberately looser than the SLO so you breach internal targets long before customer-facing ones. Error budget — the arithmetic complement of the SLO: 99.9% over 30 days permits ≈43 minutes of failure; that budget is spendable currency. With budget remaining, ship risky changes and run experiments; when exhausted, a pre-agreed policy shifts priority from features to reliability — which replaces the post-incident argument with a rule everyone accepted in advance.

Why not 100%? Each extra nine costs roughly ten times more, in spare capacity, checks and careful operations. Meanwhile users stop noticing. Their internet provider, their phone and their home network already fail more often than your extra nine would fix. And a 100% target forbids any change, because deploys are the biggest source of risk. The mature stance: pick the lowest reliability target users can't distinguish from perfect, then deliberately spend the remainder on velocity.

StaffPost-incident: a 30-second slowdown in a recommendation service (an optional homepage module) caused a 20-minute full-site outage. Reconstruct the amplification chain and prescribe the fixes.

The chain of events, rebuilt:

1. The recommendation service slowed down to about 5 seconds per call.
2. The homepage handler called it with no timeout, or with a 30-second default. So request threads and connections piled up, waiting.
3. Those waiting requests were holding connections from the shared connection pool. Unrelated database calls started to queue. The failure had jumped from an optional feature into critical paths, through a resource they shared.
4. Health checks started timing out, because the process was overloaded. So the orchestrator restarted healthy servers (9.9.7). That cut capacity and emptied the caches.
5. Browsers and the CDN retried the failed page loads, multiplying the incoming load.
6. Recovery never came, because every restarted server was overloaded again at once.

Note that steps 2–6 are all our system's behavior — the trigger lasted 30 seconds, the outage 20 minutes (section 7).

Fixes by layer, in priority order: (a) Timeout on the recommendation call at ~p99.9 (a few hundred ms) with deadline propagation so it can never exceed the page's budget. (b) Fallback and classification: recommendations are an enhancing dependency — the homepage must render without them (cached list, or hide the module); this single decision would have prevented the outage regardless of everything else. (c) Bulkhead: separate connection/thread pools per dependency so a slow optional call cannot consume the resources the critical path needs — the mechanism that stops feature failures becoming site failures.

(d) Circuit breaker on the recommendation client with a stale-cache fallback, so a sustained slowdown stops being attempted at all. (e) Health-check hygiene: liveness must be process-local (9.9.7) so dependency slowness never triggers mass restarts; readiness may shed traffic without killing instances. (f) Retry discipline: client and CDN retry budgets with jitter, plus load shedding at the edge so saturation produces fast 503s rather than infinite queueing.

Process changes: add "what happens if this dependency slows 10×?" to design review; classify every dependency critical vs enhancing in the service catalog; and run a game day injecting latency into a dependency to verify the fallback path actually works — because a fallback nobody has exercised is a hypothesis, not a control.

Flashcards

FlashThe reliability ladder

Timeout → retry (backoff + jitter) → circuit breaker → bulkhead → fallback. Skipping one breaks the next: retries without timeouts are unbounded; breakers without fallbacks just fail faster.

FlashRetry rules

Safe to repeat (idempotent/keyed) · can succeed (429/503/timeouts, not 4xx) · exponential backoff + jitter · bounded by deadline · fleet retry budget (~10%). 3 layers × 3 retries = 27×.

FlashDeadline propagation

Pass remaining budget downstream so nobody starts work the caller has abandoned. Waste peaks exactly when capacity is scarce.

FlashBreaker states

Closed (pass, count) → open (fail fast, protects both sides) → half-open (trials; success closes, failure re-opens longer). Hysteresis prevents flapping. Alert on transitions.

FlashBulkhead + shedding

Isolate pools per dependency/workload/tenant; shed fast at the edge by priority when saturated. A queue of timeouts serves nobody; shedding 40% keeps 60% healthy.

FlashError budget

SLO 99.9%/30d ≈ 43 min. Budget = currency: spend on risky changes; exhausted ⇒ reliability preempts features. Each nine costs ~10×; 100% forbids change.

Scenario Drill

DrillDesign the reliability configuration for a checkout flow that calls: inventory (internal, critical), pricing (internal, critical), payments (external PSP, critical), fraud scoring (external, enhancing), recommendations (internal, enhancing), and email (internal, async). Specify timeouts, retries, breakers, bulkheads, fallbacks, and the SLO — then state what happens when each dependency is down.

Sort the dependencies first, because that decides everything:

• Critical: inventory, pricing and payments.
• Enhancing: fraud scoring and recommendations.
• Can wait: email.

Settings for each dependency. Inventory (p99 about 20 ms): a 150 ms timeout, and 1 retry with jitter. The retry is safe, because the conditional stock reduction can be repeated by design (9.5.4). A breaker per endpoint. And no fallback. If inventory is down, checkout really cannot go ahead. Fail fast, with a clear message and a way to try again later.

Pricing (p99 about 30 ms): a 200 ms timeout and 1 retry. A circuit breaker whose fallback is cached prices, with a freshness rule: serve cached prices up to N minutes old, and after that, fail. How old is too old is a business decision, not an engineering one, and it is worth writing down.

Payments, through the payment provider (p99 about 800 ms, outside our control): a 3-second timeout, because outside services really do vary. Never retry blindly. Retry only with the idempotency key the provider honours (10.4, 9.6.3), and at most 2 attempts. Give the breaker generous thresholds, because providers have brief hiccups, and opening too eagerly turns away paying customers. The fallback is to queue the payment attempt and tell the user honestly, rather than quietly pretending it worked.

Fraud scoring (an outside service, enhancing): a tight 200 ms timeout, no retries, and a breaker that trips quickly. The fallback is a default decision set by policy. Usually that is "approve and keep watching" for small amounts, and "hold for a person to review" above a limit. An enhancing dependency must never block checkout. And the fallback is a decision about risk, so the fraud team must approve it.

Recommendations (enhancing): timeout 100 ms, no retries, breaker, fallback = omit the module. Email (async): not in the request path at all — an outbox event and a queue consumer (10.8.4), so its availability is irrelevant to checkout.

Bulkheads: separate connection pools per dependency, with the two enhancing dependencies sharing a small pool that cannot starve the critical ones — the section 7 lesson pre-applied; plus a separate worker fleet for report/admin traffic so checkout capacity is never consumed by internal tooling.

Load shedding: at the edge, prioritized — shed anonymous browsing before authenticated checkout; keep health checks always. SLO: checkout success rate 99.95% and p99 latency < 2.5 s over 30 days (≈22 minutes of budget), measured at the client (a completed purchase, not a 200 from a service); error budget policy agreed in writing.

What happens when each one is down:

• Inventory: checkout is unavailable. Show an honest error with a way to try again, and raise an incident.
• Pricing: use cached prices while they are fresh enough. After that, unavailable.
• Payment provider: queue the attempts, and tell the user "we're confirming your payment". The saga's undo steps stand ready (10.8.4).
• Fraud check: apply the default policy, flag the order for review, and tell the business.
• Recommendations: hide the section. Nobody notices.
• Email: messages wait in a queue and go out when it comes back.

The artifact this drill produces — a table of dependency, classification, timeout, retry policy, breaker config, fallback, and blast radius — is the single most useful reliability document a service can have, and it's reviewable in ten minutes by anyone joining the team.