Appearance
9.7.6 — Load Balancer
"Design a load balancer: requests come in, servers are behind it, decide which server gets each request."
At 14:02 one of four servers starts failing one request in five. It returns a 500 in about eight milliseconds, because failing is fast. Its /health endpoint keeps answering 200, because the process is alive and the event loop is responsive, and that is all /health was ever checking.
The balancer is using least-connections, so it looks at how many requests each server currently has open and picks the one with the fewest. The broken server finishes its failures in eight milliseconds while the healthy servers take two hundred milliseconds to do real work, so the broken one always has the fewest open requests. The balancer sends it more and more traffic, and by 14:09 it is receiving nearly half of everything and failing nearly half of that.
Nothing in that story is a bug in the code. Every component did exactly what it was written to do. The design was wrong in one specific way: a policy is only as good as the facts it reads, and both facts here — the health probe and the connection count — stopped describing reality at the moment it mattered.
A load balancer answers two separate questions, and almost every follow-up in this interview belongs to one of them. Which server should get this request? is the routing algorithm. Should this server be getting requests at all? is the health machine. They are different mechanisms and conflating them is what produced the incident above.
1. The routing algorithms, and what each one is blind to
typescript
interface RoutingStrategy { // (1)
pick(healthy: Server[], req: RequestInfo): Server;
}(1) The strategy only ever sees servers that are already considered healthy. That separation is deliberate: the routing algorithm should not be deciding whether a server is broken, and the health machine should not be deciding who gets the next request. Section 3 is what fills that array.
Round robin. Rotate through the pool.
typescript
class RoundRobin implements RoutingStrategy {
#i = 0;
pick(healthy: Server[]): Server {
return healthy[this.#i++ % healthy.length]; // (1)
}
}(1) There is a subtlety here worth saying before it is pointed out. The pool changes between calls, because servers go in and out of the healthy set. An index modulo a list whose length keeps changing does not distribute perfectly evenly; when the list shrinks from four to three, the rotation jumps. It is fair enough in practice and it is not exact, and knowing the difference is the point.
What round robin is blind to is how much work each request is. If one endpoint takes two seconds and another takes five milliseconds, an even split of requests is a wildly uneven split of load, and one server ends up holding all the slow ones by chance.
Weighted round robin fixes the case where servers differ. A machine with twice the capacity is given twice the weight and appears twice as often in the rotation. Same blindness to request cost, now applied to unequal machines.
Least connections watches what is actually happening rather than assuming.
typescript
class LeastConnections implements RoutingStrategy {
pick(healthy: Server[]): Server {
return healthy.reduce((best, s) => s.inFlight < best.inFlight ? s : best);
}
}
class Server {
#inFlight = 0;
get inFlight() { return this.#inFlight; }
acquire() { this.#inFlight++; }
release() { this.#inFlight--; }
}A server that is busy with slow requests has a high count and stops being chosen, which happens automatically without anybody telling the balancer which endpoints are slow. That adaptation is the reason it is popular.
Its bookkeeping is where it breaks, and in exactly one way:
typescript
server.acquire();
try {
return await proxyTo(server, req);
} finally {
server.release();
}If release is not in a finally, then every request that throws leaves the count permanently one higher. After an hour of intermittent errors, a perfectly healthy server has a phantom count of forty, so the balancer believes it is the busiest machine in the fleet and stops sending it anything. The server is fine and the balancer's opinion of it is wrong forever, and nothing recovers it except a restart. This is the same release-on-every-path rule that governs locks and pooled connections in 9.5.2, and it is the most common real bug in a hand-written balancer.
Least connections is also what caused the opening incident, which is the deeper lesson. The count measures how many requests are open, and it is being used as a proxy for how loaded a server is. Those two are the same thing right up until a server starts failing fast, at which point the count says "idle" about a machine that is broken. The fact stopped meaning what the policy assumed it meant, precisely under the conditions the policy existed for.
Latency-aware routing replaces the count with a moving average of recent response times, so a server that is slow but up gets less traffic. It costs an average to maintain per server and a decision about how quickly it forgets. Weighted towards recent samples, it reacts fast and gets jumpy; weighted towards older ones, it is stable and slow to notice. There is no correct setting, only a stated one.
And it has the same failure as least connections unless you are careful: if the average includes failed responses, a server returning instant 500s looks like the fastest server you own. Averaging only successful responses is a one-word change that removes the entire class of problem.
Consistent hashing sends the same key to the same server every time, so each server builds a useful local cache for its share of the keys, and a session stays where its data is. What it costs is that servers no longer get equal traffic: if one key is far busier than the rest, one server carries it. The ring mathematics and the hot-key repairs are in 10.6; what matters here is that it is a routing strategy like the others and slots into the same interface.
Two random choices is the one worth knowing because it sounds too simple to work. Pick two servers at random and send the request to whichever of the two has fewer connections.
It is better than picking the least-loaded server overall, and the reason is specific. When several balancers each independently pick the globally least-loaded server, they all pick the same one at the same instant, and it is buried by the combined traffic of every balancer before any of them observes it getting busy. Their view of "least loaded" is a few hundred milliseconds stale and they are all acting on the same stale view. Choosing between two random servers means each balancer looks at a different pair, so the herd cannot form, and the load still ends up remarkably even. It costs two lookups and no coordination, and it is what several large fleets actually run. The fuller argument is in 10.15.
2. Health is a state machine, not a boolean
The tempting model is a flag: isUp. Here is what happens with a flag, and it is not hypothetical.
A server has a two-second garbage-collection pause. One health probe times out and the flag flips to down. All of its traffic moves to the other three servers. The pause ends, the next probe passes, the flag flips to up, and all of its traffic arrives at once on a process that has just restarted a cold connection pool with an empty cache. It falls over, the flag flips down, and the cycle repeats every ten seconds for as long as the load lasts. The balancer is not responding to instability. It is generating it.
typescript
type Health =
| { state: "healthy"; recentFailures: number } // (1)
| { state: "unhealthy"; since: number }
| { state: "probation"; successes: number }; // (2)
function nextHealth(h: Health, ok: boolean, now: number): Health {
switch (h.state) {
case "healthy":
if (ok) return { state: "healthy", recentFailures: 0 }; // (3)
return h.recentFailures + 1 >= FAIL_THRESHOLD
? { state: "unhealthy", since: now }
: { state: "healthy", recentFailures: h.recentFailures + 1 };
case "unhealthy":
return ok ? { state: "probation", successes: 0 } : h; // (4)
case "probation":
if (!ok) return { state: "unhealthy", since: now }; // (5)
return h.successes + 1 >= RECOVER_THRESHOLD
? { state: "healthy", recentFailures: 0 }
: { state: "probation", successes: h.successes + 1 };
}
}(1) Healthy carries a count of consecutive failures, so a single blip is remembered without being acted on.
(2) Probation is the state that does the real work. A server here is not trusted and is not ignored either; it receives a small share of real traffic and has to earn its way back.
(3) One success resets the counter to zero, which is what "consecutive" means. Without the reset, a server that fails once an hour eventually accumulates enough failures to be removed for no reason.
(4) Coming back out of unhealthy takes one passing probe, and it only gets you as far as probation. Cheap to leave the penalty box, expensive to get all the way back.
(5) A single failure during probation sends it straight back. This is the asymmetry that makes the whole thing work: three failures to be removed, five consecutive successes to be restored, and one failure to be removed again. A genuinely broken server cannot flap in and out, because the path back is long enough that it will fail somewhere along it.
Why probation exists rather than going straight back to healthy. A /health probe is a tiny request that touches almost nothing. Passing it proves the process is running; it does not prove the server can handle real work with real database queries and a cold cache. A trickle of actual traffic proves that, and if the server cannot cope, only a few real users are affected instead of a quarter of them. This is the same idea as releasing a change to 1% of traffic first, applied to a machine rather than a version.
The threshold numbers are a choice with a cost on each side, and it is worth stating rather than defaulting. Removing a server after one failure reacts fastest and produces flapping. Removing after ten is stable and means ten failed user requests before anything happens. Three failures against a probe every two seconds means roughly six seconds to notice a dead server, which is a reasonable place to sit, and the right answer depends on how expensive a failed request is in your product.
3. The probe that lies, and the fix
The opening incident happened because the only input to health was an endpoint that was answering honestly about the wrong thing. /health said "this process is running and can serve an HTTP request", which was true. What the balancer needed to know was "this server can do the work customers are asking for", which was false.
The repair is that real request outcomes feed the same state machine.
typescript
async forward(req: Request): Promise<Response> {
const healthy = this.pool.healthy();
if (healthy.length === 0) throw new NoUpstreamError(); // (1)
const server = this.strategy.pick(healthy, req.info);
server.acquire();
try {
const res = await proxyTo(server, req, { timeoutMs: this.timeoutMs });
server.record(res.status < 500); // (2)
return res;
} catch (e) {
server.record(false); // (3)
throw e;
} finally {
server.release(); // (4)
}
}(1) What to do when nothing is healthy is a decision, not a default, and section 5 covers it.
(2) Every real response is a health signal. A 500 counts against the server; a 404 does not, because a missing resource is the client asking for something that is not there rather than the server being broken. That distinction matters: counting 4xx responses as failures means one client requesting nonexistent pages can remove your entire fleet.
(3) A timeout or a connection error counts too.
(4) The release is in finally, for the reason section 1 gave.
Real traffic is the better probe because it is the actual question. Probes stay useful for one specific job that live traffic cannot do: a server receiving no traffic produces no signal, so probing is how an unhealthy server is discovered to have recovered. That division is clean and worth saying out loud. Passive signals decide when a server goes out; active probes decide when it may start coming back.
One more rule that only appears once you have both inputs. Rate matters more than count on the passive side. A server handling a thousand requests a second will produce a few failures a minute even when perfectly healthy, so three consecutive failures is the wrong test there. The passive signal should be an error rate over a recent window — say, more than 10% of the last hundred requests — while probe failures stay a consecutive count. Same state machine, two different triggers into it.
4. Retrying on another server, and the precondition nobody volunteers
A request times out. The obvious thing is to send it to a different server, and the obvious thing is sometimes seriously wrong.
A timeout does not mean the request did not happen. It means you stopped waiting. The server may have completed the work perfectly and the response was lost, or the response was still in flight when you gave up. Retry a POST /payments on another server and you may have charged the customer twice.
So the rule has a precondition:
Retry freely when the request is safe to run twice. GET, PUT and DELETE are defined to be safe to repeat, and anything carrying an idempotency key that the server behind you honours is safe by construction. The definitions and the machinery are in 9.6.3.
Never retry automatically a bare POST that changes something. The balancer cannot know whether it executed, so the failure goes back to the client, who knows what the request meant and can retry it with a key.
And distinguish the two failures, because they are not equally ambiguous. A connection refused means nothing was sent, so nothing ran, so anything is safe to retry, including a payment. A timeout after the request was sent is the ambiguous one, and it is the only case the rule above is about. Collapsing them means refusing to retry requests that are provably safe.
Then three limits, each with a concrete reason.
Cap the number of retries. Two attempts, not "keep going". A request that has failed twice is unlikely to succeed on the third and is now costing latency.
Do not retry onto the same server. It just failed. This sounds obvious and is easy to get wrong when the pool is small.
Keep a retry budget. This is the one people miss. Retries multiply load at exactly the moment capacity is short: a fleet at 80% capacity that starts failing 20% of requests and retries each one is now at 96%, which pushes the failure rate up, which produces more retries. The fleet finishes itself off. A budget caps retries at, say, 10% of total requests, so a widespread failure degrades instead of collapsing. Volunteering this is a strong signal, because it only occurs to people who have watched retries turn a partial outage into a total one.
5. Draining, cold starts, and having nothing left
Draining is how a server leaves without dropping anything. A deploy is about to restart it, and it currently has forty requests in flight.
The sequence is: mark it as not selectable so the strategy stops choosing it, wait for the in-flight count to reach zero, then let the restart happen. Because release runs in a finally, that count is trustworthy, which is the second time that one line has paid for itself.
The wait needs a limit. A request stuck for ten minutes cannot hold up a deploy forever, so after a grace period the connections are closed anyway and those users see an error. Choosing thirty seconds means almost every request finishes and a stuck one costs half a minute. This is the same shutdown sequence a Node process runs in 9.9.7, viewed from the balancer's side.
Coming back is the mirror image and it is the half people forget. A freshly restarted server has an empty cache, an unopened connection pool, and a runtime that has not yet optimised its hot code paths. Sending it a full quarter of the traffic in the first second makes it slow, which under least-connections makes it look busy and gets it left alone, or under round robin makes it fail and get removed. Either way the deploy looks like an incident.
Slow start fixes it by ramping the server's share up over thirty seconds or so, from a trickle to its full weight. It is a small feature and it removes a whole category of "the deploy caused a spike" investigations. Notice that it is the same mechanism as probation, used for a different reason: probation ramps traffic to prove a server is well, slow start ramps traffic to let it get well.
And the case with no good answer: every server is unhealthy. Something must happen, and each option is bad in a different way.
Reject everything immediately. Honest, fast, and every user sees an error.
Send traffic to unhealthy servers anyway. Sometimes right. If all four servers are marked unhealthy because a shared database is slow, they are all still capable of serving what they can, and refusing to try guarantees an outage where there might have been degraded service.
Queue and wait. Almost always wrong, because the queue fills, memory grows, and the eventual failure is worse and later.
The choice depends on what "unhealthy" was measuring, which is the real point: a fleet marked down by a dependency failure is a different situation from a fleet that has genuinely crashed, and a balancer that cannot tell them apart cannot choose well. Having decided this in advance is what matters, because the alternative is discovering your balancer's default during an incident.
6. What the interviewer will push on
"How do you decide which server gets a request?" They want a menu with trade-offs, not a favourite. Round robin is blind to request cost; least connections adapts to slow requests and needs a count that is released in finally; latency-aware needs a decision about how fast it forgets; consistent hashing buys cache locality and pays with uneven load. The strongest addition is two random choices, with the reason: independent balancers all picking the globally least-loaded server pick the same one at the same moment and bury it, and choosing between two random servers stops the herd forming without any coordination.
"Your least-connections balancer is sending most of its traffic to the broken server. Why?" This is the opening incident, and it is the best question on the topic. Failures return in eight milliseconds while real work takes two hundred, so the broken server always has the fewest open connections and looks idle. The fact stopped meaning what the policy assumed it meant. The fix is not a better picker; it is removing the server from the pool via passive health signals, and then hardening the picker so it never averages failed responses into a latency score.
"Why is health three states and not a boolean?" Because a flag makes the balancer generate instability. A garbage-collection pause flips it down, the recovery flips it up, all the traffic returns at once to a cold process, and it falls over again on a ten-second cycle. Three failures out and five consecutive successes back, with probation carrying a trickle of real traffic in between, means a genuinely broken server cannot flap. The tell is being able to say why probation gets real traffic: a health probe proves the process is running, not that it can do work.
"Upstream timed out. Retry elsewhere?" They are checking whether idempotency comes up unprompted. A timeout is ambiguous, so the request may have executed and been lost, so a bare POST must not be retried automatically. GET, PUT, DELETE and anything with an idempotency key are safe. Then the two refinements that complete it: connection-refused is unambiguous and safe to retry for anything, and a retry budget is needed because retries multiply load exactly when capacity is short and can finish off a struggling fleet.
"How do you take a server out for a deploy?" Stop selecting it, wait for in-flight to reach zero, restart, with a grace period after which stuck connections are closed. The half that is usually missing is coming back: a cold server given its full share immediately is slow or fails, so ramp it up over about thirty seconds. Without slow start, every deploy looks like a small incident.
"All servers are unhealthy. What happens?" They want to see you treat this as a decision rather than a default. Reject fast, send to unhealthy servers anyway, or queue — and the right choice depends on whether "unhealthy" meant crashed or meant a shared dependency is slow. In the second case, refusing to try guarantees an outage where degraded service was available.
The thing to volunteer that nobody asks for: the in-flight counter that is not released on the error path. A server that throws forty times over an hour accumulates a phantom count of forty, so the balancer decides it is the busiest machine in the fleet and stops sending it anything. The server is healthy, the balancer's view of it is permanently wrong, and nothing recovers it but a restart. One finally prevents it, and it is the most common real bug in a hand-written balancer.
Recall
- Two separate questions: which server gets this request (routing), and should this server get requests at all (health). Conflating them causes the classic outage.
- Round robin is blind to request cost, and an index over a pool whose length changes is fair enough rather than exact.
- Least connections adapts to slow requests. Its count must be released in
finally, or errors leave a phantom count and the balancer permanently ignores a healthy server. - Least connections is fooled by fast failures. A server returning 500s in 8 ms always has the fewest connections and gets more traffic. A fact stopped meaning what the policy assumed.
- Latency-aware routing must average only successful responses, or instant errors look like the fastest server you own.
- Consistent hashing buys cache locality and session stickiness, and pays with uneven load when one key is hot.
- Two random choices beats picking the global minimum, because independent balancers all pick the same "least loaded" server at once and bury it. Two random servers stops the herd, with no coordination.
- Health is three states. Three failures out; one passing probe reaches probation; five consecutive successes to return; one failure sends it back. The asymmetry is what stops flapping.
- Probation carries a trickle of real traffic, because a probe proves the process is running, not that it can do work.
- Passive signals decide when a server goes out; active probes decide when it may come back. A server with no traffic produces no passive signal.
- Passive health uses an error rate over a window, not a consecutive count, because a busy server always has some failures.
- Do not count 4xx as server failures, or one client requesting missing pages removes the fleet.
- Retry elsewhere only when the request is safe to repeat. A timeout is ambiguous; connection-refused is not. Cap retries, never retry the same server, and keep a retry budget, because retries multiply load exactly when capacity is short.
- Draining: stop selecting, wait for in-flight to hit zero, then a grace period before closing anyway.
- Slow start: ramp a restarted server's share over about thirty seconds, or a cold process with an empty cache fails and the deploy looks like an incident.
- All-down is a decision: fail fast, send anyway, or queue — and it depends on whether unhealthy meant crashed or meant a shared dependency is slow.
Self-test: Why did least connections send more traffic to the broken server? What exactly does an unreleased in-flight count do? Why five successes back but only three failures out? Why does probation get real traffic? Which failures are safe to retry, and what is a retry budget for? Name both halves of a deploy.
Quiz Bank
FoundationalPresent the routing algorithms with what each adapts to, what it costs, and what it is blind to.
Round robin. Rotate an index through the healthy pool. No per-request state, no measurement, and it is fair when every request costs about the same.
It is blind to request cost. One endpoint taking two seconds and another taking five milliseconds means an even split of requests is a wildly uneven split of work, and one server ends up with a run of slow ones by luck. There is also a small subtlety worth raising before it is pointed out: the pool changes as servers enter and leave the healthy set, so an index modulo a shifting length is approximately fair rather than exactly fair.
Weighted round robin. Same, with a machine of twice the capacity appearing twice as often. It fixes unequal servers and keeps the blindness to unequal requests.
Least connections. Pick whichever server has the fewest requests currently open. It adapts on its own: a server holding several slow requests has a high count and stops being chosen, without anybody telling the balancer which endpoints are slow.
It costs bookkeeping, and that bookkeeping has exactly one dangerous failure:
typescript
server.acquire();
try { return await proxyTo(server, req); }
finally { server.release(); }Without the finally, every request that throws leaves the count one higher permanently. After an hour of intermittent errors a healthy server carries a phantom count of forty, so the balancer believes it is the busiest machine it has and sends it nothing, forever, until a restart.
And it has a second blindness that caused a real incident: fast failures look like idleness. A server returning 500s in eight milliseconds while healthy servers take two hundred always has the fewest connections, so least connections routes more traffic to the broken one. The count measures open requests and was being used as a proxy for load, and those two stop agreeing exactly when a server breaks.
Latency-aware. Keep a moving average of recent response times and prefer the fastest. This catches the case least connections misses: a server that is up but degraded.
It costs an average per server and a decision about how quickly it forgets old samples. Weighted towards recent data it reacts fast and gets jumpy; weighted towards older data it is stable and slow. There is no correct setting, only a stated one. And it has a trap: average only successful responses, or a server returning instant errors is measured as the fastest thing you own.
Consistent hashing. The same key always goes to the same server, so each one builds a useful cache for its share and a session stays where its data is.
It costs even distribution. If one key is far busier than the rest, one server carries it, and the repairs for that are in 10.6.
Two random choices. Pick two servers at random, send to whichever of the two has fewer connections.
This sounds worse than picking the minimum and is better, for a reason worth stating precisely. When several balancers each pick the globally least-loaded server, they all pick the same server at the same instant, because their views are equally stale, and it is buried by their combined traffic before any of them sees it get busy. Sampling two at random means each balancer looks at a different pair, so the herd cannot form, and the resulting distribution is still very even. It costs two lookups and no coordination at all. 10.15 has the full argument.
The framing that ties it together: every one of these is a policy reading a fact. Round robin reads nothing, least connections reads a count, latency-aware reads an average, consistent hashing reads the key. The failures on this page are all the same shape, which is a fact that stopped describing reality under precisely the conditions the policy was built for.
AppliedDesign the health machine. Why three states, why asymmetric thresholds, and where do the inputs come from?
Start with why a boolean fails, because it fails actively rather than passively.
A server has a two-second garbage-collection pause. One probe times out, the flag goes to down, and all its traffic moves to the others. The pause ends, the next probe passes, the flag goes to up, and all of its traffic returns at once to a process with a cold connection pool and an empty cache. It cannot cope, it fails, the flag goes down, and the cycle repeats every ten seconds. The balancer is not reacting to instability. It is creating it.
Three states:
typescript
type Health =
| { state: "healthy"; recentFailures: number }
| { state: "unhealthy"; since: number }
| { state: "probation"; successes: number };Healthy carries a count of consecutive failures, so one blip is remembered without acting on it, and any success resets the count to zero. Without the reset, a server that fails once an hour eventually accumulates enough to be removed for no reason.
Unhealthy receives nothing. One passing probe moves it to probation, which is cheap, because getting out of the penalty box should be easy while getting all the way back should not.
Probation receives a trickle of real traffic and must produce five consecutive successes to be restored. A single failure sends it straight back to unhealthy.
The asymmetry is the entire mechanism. Three failures to leave, five successes to return, one failure to leave again. A genuinely broken server cannot oscillate, because the path back is long enough that it will fail somewhere along it. A server that had a momentary blip walks that path easily. This is the same principle as a thermostat that turns on at 18° and off at 21° rather than switching at a single temperature: the gap between the thresholds is what stops it chattering.
Why probation gets real traffic rather than more probes. A /health endpoint is a tiny request that touches nothing. Passing it proves the process is alive and the event loop is responsive. It does not prove the server can run a real query against a cold pool. Only real traffic proves that, and sending a trickle means a server that cannot cope affects a handful of users rather than a quarter of them.
Where the inputs come from, which is the half that caused the incident.
Passive signals — every real response. A 500 or a timeout counts against the server; a 404 does not, because a missing resource is the client asking for something absent rather than the server being broken. Counting 4xx as failures means one client requesting nonexistent pages can remove your whole fleet.
Active probes — periodic requests to /health.
The clean division: passive signals decide when a server goes out, active probes decide when it may come back. Live traffic is the truest test of whether a server works, and a server that is receiving no traffic produces no signal at all, so probing is the only way to discover that an ejected server has recovered.
One refinement that only appears once both inputs exist. The passive side should use an error rate over a recent window rather than a consecutive count, because a server handling a thousand requests a second produces a few failures a minute while perfectly healthy. More than 10% of the last hundred requests failing is a signal; three failures in a row is noise at that volume. Probe failures stay a consecutive count, because probes are infrequent and a failure means something. Same state machine, two differently shaped triggers.
And the thresholds are a choice with a cost on both sides. Ejecting after one failure reacts fastest and flaps. After ten it is stable and costs ten failed user requests before anything happens. Three failures against a two-second probe is roughly six seconds to notice a dead server, which is a reasonable default, and the right number depends on what a failed request costs your product.
InterviewA request to an upstream server times out. Do you retry it somewhere else?
It depends on the request, and the dependency is idempotency.
A timeout is ambiguous and that is the whole problem. It does not mean the request failed. It means you stopped waiting. The server may have executed it perfectly and the response was lost, or the response was still travelling when you gave up. From the balancer's position those are indistinguishable.
So retrying a POST /payments on a second server may charge the customer twice, and it will do so silently, and the customer will find out before you do.
The rule:
Retry freely for requests that are safe to run more than once. GET, PUT and DELETE are defined that way, and anything carrying an idempotency key that the upstream honours is safe by construction, because the upstream recognises the repeat and returns the original result instead of doing the work again. The mechanism is in 9.6.3.
Never retry automatically a bare POST that changes something. The balancer cannot know whether it ran, so the failure goes to the client, who knows what the request meant and can retry it with a key.
Then the refinement that completes the answer, because not all failures are equally ambiguous.
A connection refused means the request was never accepted, so nothing ran, so anything is safe to retry — including a payment. Treating this the same as a timeout means refusing to retry requests that are provably safe, which costs availability for no benefit.
A timeout after the request was sent is the ambiguous one, and the rule above is entirely about this case.
A server that closes the connection before sending any response bytes is also usually safe, and this is where you should be a little careful and say so, because "before sending any bytes" is a statement about the network and the proxy's own buffering rather than about what the server did.
Three limits, each with a reason.
Cap the attempts. Two, not "keep trying". A request that has failed twice is unlikely to succeed on the third and is now spending the user's patience.
Never retry onto the same server. It just failed, and with a small pool a naive picker will choose it again.
Keep a retry budget, and this is the one worth volunteering. Retries multiply load at precisely the moment capacity is short. A fleet at 80% utilisation that begins failing 20% of requests and retries each one is now at 96%, which raises the failure rate, which produces more retries. The fleet finishes itself off, and the retries did it. Capping retries at something like 10% of total request volume means a widespread failure degrades rather than collapses.
Either way, the failure is recorded against the server's health machine. A retry that succeeds elsewhere still means the first server failed, and forgetting to record it means the balancer keeps sending requests to a server that only ever works on the second attempt.
What this question is really testing is whether HTTP semantics, ambiguous failures and load dynamics are connected in your head, or filed as three separate topics. The strongest answer moves through all three without being led.
StaffPost-mortem the opening incident properly: name every design flaw, its fix, and the general lesson.
The incident. At 14:02, server B begins failing 20% of requests with 500s that return in about eight milliseconds. Its /health probe keeps passing. Least-connections routing sends it progressively more traffic until it is receiving nearly half of everything by 14:09.
Flaw one: health had a single input, and that input tested the wrong thing.
/health answered "the process is running and the event loop is responsive", which was true. The balancer needed "this server can do the work customers are asking for", which was false. The probe and the work had diverged, and the balancer had no way to see it.
Fix: real request outcomes feed the same health state machine, using an error rate over a recent window rather than a consecutive count, since a busy server always has some failures. Twenty percent of the last hundred requests failing would have ejected server B within seconds. Probes keep one job, which is detecting that an ejected server has recovered, because a server receiving no traffic produces no passive signal.
Flaw two: the routing policy read a fact that stopped meaning what it assumed.
In-flight count is a proxy for how loaded a server is, and the proxy holds until a server starts failing fast. B's eight-millisecond failures meant it always had the fewest open requests, so the picker read "least loaded" and amplified the damage. The policy was not wrong in general; it was wrong under exactly the conditions it existed to handle.
Fix, in two parts. First, the health fix removes B from the pool entirely, so the picker never sees it, and that is the real repair. Second, harden the picker so it cannot be fooled this way again: score on successful-response latency rather than raw connection count, or penalise a server's score by its recent error rate. Two random choices also limits the blast radius here, because no single balancer can concentrate everything on one server.
Flaw three, which is a design question rather than a bug: nothing capped what one server could receive.
Even with a perfectly sensible picker, "send to the least loaded" has no ceiling, so a single server can end up receiving an arbitrary share. Capping any one server at some multiple of its fair share turns a policy failure into a partial one. It is a small guard and it converts an outage into a degradation.
The general lesson, and it is the sentence to say in the room. All three flaws are the same shape: a policy trusted a fact that stopped describing reality under precisely the conditions the policy existed for. The probe measured process liveness and was read as service health. The connection count measured open requests and was read as load. Neither was lying; both were being asked a question they did not answer.
The practice that follows from it is to ask, for every policy in a system, under which failure does this fact stop meaning what I am assuming? Asked at design time, all three of these are an afternoon's work. Asked at post-mortem time, they are this document.
What I would add to the balancer afterwards, in priority order. Passive health first, because it is the fix that would have prevented the incident. A cap on any single server's share second, because it bounds the damage from whatever the next policy mistake turns out to be. Then error-rate penalties in the picker. And finally the observability that would have made this obvious in two minutes rather than seven: per-server error rate and per-server share of traffic on one chart, because the incident's signature is one line going up while the other goes up with it, and no dashboard in the world shows that if the two numbers live on different screens.
Flashcards
FlashWhy least connections fed the broken server
Failures return in 8 ms while real work takes 200 ms, so the broken server always had the fewest open connections and looked idle. The count measured open requests and was read as load, and those stopped agreeing exactly when the server broke.
FlashThe unreleased in-flight count
Without release in a finally, every thrown request leaves the count one higher forever. A healthy server accumulates a phantom count, the balancer decides it is the busiest machine it has, and it never gets traffic again until a restart.
FlashWhy three health states
A boolean flips down on a GC pause and back up into a full traffic load on a cold process, oscillating every ten seconds. Three failures out, five consecutive successes back, one failure to leave again — the asymmetry is what stops flapping.
FlashProbes versus real traffic
Passive signals decide when a server goes out; active probes decide when it may come back, because a server with no traffic produces no passive signal. Passive uses an error rate over a window; probes use a consecutive count.
FlashWhen to retry elsewhere
Only when the request is safe to repeat: GET, PUT, DELETE, or anything with an idempotency key. A timeout is ambiguous; connection-refused is not, so that one is safe for anything. Cap attempts, avoid the same server, and keep a retry budget.
FlashBoth halves of a deploy
Out: stop selecting, wait for in-flight to reach zero, grace period, then close anyway. In: ramp the restarted server's share up over about thirty seconds, because a cold process given its full share is slow or fails and the deploy looks like an incident.
Next: 9.7.31 — the logging framework, the component that looks trivial and is the one most likely to take down the service it was installed to explain.