Appearance
10.15 — Load Balancing and Traffic Management
One server handles 2,000 requests per second and you are getting 3,000. You add a second server. Now something has to decide, for each incoming request, which server gets it.
That decision looks trivial and it is not. Choosing badly gives you two servers where one is at 95 percent and the other at 30 percent. Choosing badly in a different way gives you a system where one sick server absorbs every request and fails all of them, because it is failing so fast that it always looks least busy.
1. Layer 4 or layer 7: what the balancer is allowed to see
The first decision is how deep into the traffic the balancer looks, and it changes what it can do.
A layer 4 balancer works with TCP connections. It sees addresses and ports and nothing else. A connection arrives, it picks a server, and every packet on that connection goes to the same place for its whole life. It cannot read the URL, the headers, or the cookies, because it never assembles them.
A layer 7 balancer terminates the connection, reads the HTTP request, and makes a decision per request. It sees the path, the method, the headers and the body.
| Layer 4 | Layer 7 | |
|---|---|---|
| Sees | Addresses, ports | URL, headers, cookies, body |
| Decides per | Connection | Request |
| Can route by path | No | Yes |
| Can retry a failed request | No | Yes |
| Handles TLS | Passes it through | Usually terminates it |
| Throughput | Very high | Lower, and enough for nearly everyone |
What layer 7 buys you, concretely. Routing /api/* to one pool and /images/* to another. Retrying a request that failed on one server against a different one, which layer 4 cannot do because by the time the failure is visible the connection is already committed. Rewriting headers, enforcing rate limits, adding a request id, and terminating TLS in one place so your application servers never handle certificates.
What layer 4 buys you. Raw speed and protocol independence. It balances anything over TCP — a database protocol, a message broker, a game server — because it does not care what is inside.
The practical answer, and the one to give unless there is a reason otherwise: layer 7 for HTTP traffic, layer 4 for everything else. The features you get from reading the request are worth far more than the throughput you give up, and modern layer 7 balancers handle enormous traffic comfortably.
One thing that changes when you terminate TLS. Your application no longer sees the client's address, because the connection now comes from the balancer. The balancer adds it as a header — conventionally X-Forwarded-For — and you must configure which balancer you trust, because a client can send that header too. Trusting it blindly means anybody can claim any address, which defeats rate limiting, audit logs and geographic rules in one step.
2. The algorithms
Round robin. Rotate through the servers in order. Zero state, perfectly even counts, and correct when every request costs roughly the same and every server is identical. It is wrong the moment requests vary in cost, because four expensive requests and four cheap ones are not the same load.
Weighted round robin. Same, but a server with twice the capacity gets twice the share. Use it when the fleet is not uniform, which happens constantly during a hardware migration.
Least connections. Send the request to whichever server currently has the fewest in flight. This adapts automatically to variable request cost, because a server stuck on slow work accumulates open connections and stops being chosen. It is the right default for most HTTP services and the reason is exactly Figure 1.
Least response time. Combine in-flight count with recently observed latency. Slightly better than least connections, and it has a failure mode worth knowing: a server that is broken and failing instantly looks like the fastest server in the fleet, so it attracts all the traffic and fails all of it. This is sometimes called the black hole problem, and the cure is that health checking must catch the failure rather than relying on the load metric to notice.
Hash by client address. The same client always reaches the same server. It gives you stickiness with no state, and it distributes badly whenever many clients share an address, which is true of corporate networks, mobile carriers and anyone behind a shared gateway.
Consistent hashing. Route by a key — a user id, a cache key, a tenant — so requests for the same key land on the same server, and adding or removing a server moves only a fraction of the keys (10.6). This is how you get a per-server cache with a decent hit rate, and it is the algorithm to reach for when the backends hold state derived from the key.
Random. Pick one at random. Surprisingly close to round robin at scale, and the basis for the next one.
Two random choices, which deserves attention because it is the most useful underrated result here. Pick two servers at random and send the request to whichever of the two has fewer connections. That is it. The improvement over pure random is dramatic — the worst-case imbalance drops from growing with the fleet size to growing with the logarithm of it — and unlike least connections it does not require the balancer to track the state of every server, so it works when there are many balancers that cannot see each other.
That last property is why it matters in practice. With ten balancers each independently choosing "least connections", they all agree on which server is least loaded and all send to it simultaneously, producing an oscillation where the emptiest server is repeatedly overwhelmed. Two random choices has no such herd behaviour, because no two balancers are looking at the same pair.
3. Health checks, and the ways they go wrong
A balancer must know which servers are usable. It sounds simple and there are three ways to get it badly wrong.
Active checking means the balancer asks each server periodically — an HTTP request to /health, say. Simple and predictable, and it adds constant traffic and only notices a failure at the next check.
Passive checking means watching real traffic and marking a server unhealthy after consecutive failures. It notices instantly and costs nothing extra, and it needs real requests to fail first, so those users have already had a bad experience.
Use both. Passive to react quickly to a server that has just broken, active to notice when it has recovered.
Mistake one: checking that the port is open. A process can accept connections while being completely unable to serve — out of memory, deadlocked, unable to reach its database. A check that only proves the socket answers keeps sending traffic to a dead server.
Mistake two: checking too much. The opposite failure, and it is worse. If /health queries the database, then a database blip marks every server unhealthy at the same instant, the balancer has nowhere to send traffic, and a partial degradation becomes a total outage. The rule that avoids this is: a health check should report whether this server can do its job, not whether the whole system is working. Dependencies belong in a separate deep check used by dashboards and humans, never by the balancer.
Mistake three: no hysteresis. A server on the edge of health flips between in and out every few seconds, and each flip disrupts connections. The fix is asymmetric thresholds — take a server out after two consecutive failures, put it back only after five consecutive successes — so removal is quick and return is cautious.
And the failure that catches everyone eventually: all servers unhealthy at once. If a bad deploy or a shared dependency fails every check, the balancer has zero healthy servers. Sending traffic nowhere guarantees a total outage; sending it to servers that might be broken at least gives partial success. Most balancers implement this as a panic threshold: if fewer than some percentage of servers are healthy, ignore health status entirely and spread traffic across everything. It is an ugly rule and it is the right one, because a system that is half working beats one that is off.
4. Sticky sessions, and why to avoid them
Session affinity means a user's requests always reach the same server. Two ways to do it: hash the client address, or have the balancer set a cookie naming the chosen server.
It exists because somebody stored session state in a server's memory. And it causes four problems, which is why the modern answer is to remove the need for it.
Deploys drop sessions. Restarting a server loses everything held in its memory, so every user pinned to it is logged out. This makes zero-downtime deploys impossible in the one way users actually notice.
Load goes uneven. Servers accumulate long-lived sessions unevenly, and a server that happens to collect the heavy users stays hot while others idle. Adding a server helps nothing, because existing sessions do not move to it.
Scaling in is destructive. Removing a server means abandoning its sessions, so autoscaling either causes logouts or cannot scale in.
A hot tenant cannot be spread. All of one large customer's traffic is pinned to one server by construction.
The fix is to stop keeping state in the process. Put the session in a shared store, or in a signed cookie the client carries. Then any server can serve any request, deploys are invisible, autoscaling works, and load is even. This is the same rule as 9.5.6: anything in one process's memory stops being a guarantee the moment there are six processes.
Where affinity is still legitimate. Long-lived connections — WebSockets and streams — are inherently bound to one server for their lifetime, and that is fine because it is the connection that is sticky rather than the user's identity. And cache affinity via consistent hashing is a different thing: it improves hit rate, and correctness never depends on it, so losing a server costs performance rather than sessions.
5. The balancer itself must not be the single point of failure
You added a balancer so one server's death would not matter. Now the balancer's death matters instead.
A pair with a floating address. Two balancers, one active. They watch each other, and if the active one dies the standby takes over the shared address. Simple, well understood, and it wastes half the hardware.
Several balancers behind anycast. The same address announced from several places, with the network routing each packet to the nearest healthy announcement. Failover is automatic and needs no coordination, because a dead balancer stops announcing and traffic moves.
Several balancers behind DNS. Return several addresses and let clients choose. Cheap and slow to fail over, because clients and resolvers cache DNS answers for as long as the record's lifetime says — and some ignore it. This is fine as a coarse layer and inadequate as your only failover mechanism.
Layers, which is what large systems actually run. DNS or anycast spreads traffic across regions, a layer 4 balancer spreads it across layer 7 balancers within a region, and the layer 7 balancers spread it across application servers. Each layer solves a different problem, and each one can lose a member without the layer above noticing.
6. Deploys without dropping requests
Taking a server out of rotation abruptly kills the requests it is currently serving. Four mechanisms make a deploy invisible, and they need to happen in order.
Connection draining. Stop sending new requests to a server, but let the in-flight ones finish before shutting it down. Every balancer supports this and it needs a timeout, because one request that never completes must not block the deploy forever.
Fail the health check before shutting down. The server should start reporting itself unhealthy, then wait for the balancer's check interval to pass, and only then stop accepting connections. Skipping the wait means requests are still arriving when the process exits. This ordering is the detail teams get wrong, and it is the difference between a clean deploy and a small burst of errors on every release.
Slow start. A newly added server should receive gradually increasing traffic rather than its full share immediately. A cold process has an empty cache, an unwarmed connection pool, and code that has not been optimised by the runtime yet (3.2). Hitting it with a full share means slow responses and possibly a failed health check, which removes it, which sends its traffic elsewhere, which is a restart loop.
Roll one at a time. Replacing the fleet at once means every request lands on a cold server, and any bug reaches every user simultaneously. Rolling gives you both warm-up and a chance to stop.
7. Retries, and the trap in them
A layer 7 balancer can retry a request that failed, which is genuinely useful: one server hiccups and the user never notices.
It is also how a small problem becomes an outage. If your service is struggling and every request is retried twice, you have tripled the load on a system that is already failing. This is retry amplification, and it is worse in layers — a client retry multiplied by a balancer retry multiplied by a service-to-service retry produces eight attempts from one user action.
Four rules keep retries safe.
Only retry what is safe to repeat. A GET is fine. A POST that charges a card is not, unless it carries an idempotency key (9.6.3).
Only retry connection-level failures, where the request demonstrably never reached the application. Retrying a 500 means running the work twice, which may be exactly what you must not do.
Retry on a different server. Retrying against the one that just failed is a wasted attempt.
Budget the retries. Cap total retries as a fraction of total requests — a few percent — so that when everything is failing, retries cannot multiply the load. This is the single most important of the four and the one most often missing.
8. Sending traffic to the right region
Once you have more than one region, something has to choose between them.
By geography. Send European users to Europe. Simple, and it uses a crude map of where addresses are, which is often wrong.
By latency. Send users to whichever region actually responds fastest for them, measured rather than assumed. Better, because network topology does not follow geography — a user physically nearer one data centre may have a faster path to another.
By anycast. Let the internet's routing choose. The least configuration and the least control, and the failover is free.
By weight. Send five percent of traffic to a new region and watch. This is how you test a region before trusting it, and it is worth naming because it is how the previous three get rolled out safely.
The thing that makes all of them harder than they look is that routing traffic to a region is easy and routing data is not. A user sent to a nearby region whose database replica is behind may write to one region and read stale data from another (10.7.1). Geographic routing is a data placement decision wearing a networking costume, and treating it as purely a networking question is how teams end up with users who cannot see their own writes.
9. What the interviewer will push on
"Round robin or least connections?" They want to hear that round robin is fair in request count and least connections is fair in work, and that the difference only matters when request costs vary — which they always do. Bonus for naming two-random-choices and explaining why it beats least connections when there are several independent balancers: no herd, because no two balancers are comparing the same pair.
"What does your health check check?" The trap is checking the database. If /health touches a shared dependency, one blip marks every server unhealthy simultaneously and turns a degradation into an outage. The right answer is that the check reports whether this server can do its job, dependencies get a separate deep check for humans, and the balancer needs a panic threshold for the case where everything looks unhealthy at once.
"Would you use sticky sessions?" Almost always no, and the four reasons are deploys logging users out, uneven load, autoscaling that cannot scale in, and hot tenants that cannot be spread. The real answer is to move session state out of process memory. Then note the legitimate exceptions — long-lived connections, and cache affinity where losing a server costs hit rate rather than correctness.
"Your balancer dies." They are checking whether you noticed you moved the single point of failure rather than removing it. Pair with a floating address, anycast, or layers — and be able to say why DNS alone is not enough, which is that clients cache records for longer than they should and some ignore the lifetime entirely.
"You deploy and users see errors for ten seconds." The ordering. Fail the health check first, wait for the balancer to notice, then drain in-flight requests, then exit. Teams that shut down first and rely on draining alone get a burst of errors on every release.
"When are retries dangerous?" Retry amplification. A struggling service plus retries at three layers is eight attempts per user action arriving at a system already failing. The answer is a retry budget capping retries as a fraction of traffic, plus retrying only connection failures on a different server, plus idempotency keys for anything that writes.
The thing to volunteer that nobody asks for: slow start. A newly added server has a cold cache, an empty connection pool and unoptimised code, so giving it a full share immediately produces slow responses and can fail its health check, which removes it, which loops. Ramping its traffic over thirty seconds costs nothing and removes an entire class of deploy-time incident.
Next: 10.16 — how the server pushes data back to the user, once the request has found its way in.
Recall
- Layer 4 sees addresses and ports and balances any TCP protocol. Layer 7 reads the request, so it can route by path, retry on another server, rewrite headers and terminate TLS. Layer 7 for HTTP, layer 4 for everything else.
- Terminating TLS hides the client's address; the balancer adds it as a header, and you must configure which balancer you trust or anyone can forge it.
- Round robin is fair in request count; least connections is fair in work, which is what matters. Least response time has the black-hole failure — a server failing instantly looks fastest.
- Two random choices — pick two at random, send to the less loaded — is nearly as good as least connections and has no herd behaviour when several balancers cannot see each other.
- Consistent hashing routes by key so per-server caches work and adding a server moves only a fraction of keys.
- Health checks: use active and passive together. Do not check only the port. Do not check shared dependencies, or one blip marks everything unhealthy at once. Use asymmetric thresholds, and keep a panic threshold so zero-healthy means spread everywhere rather than serve nobody.
- Sticky sessions cost you deploys, even load, scale-in and hot-tenant spreading. Move session state out of process memory instead. Legitimate only for long-lived connections and for cache affinity.
- Deploy order: fail the health check, wait for the balancer to notice, drain in-flight requests, then exit. Add slow start for new servers.
- Retries amplify: cap them with a budget, retry only connection failures, on a different server, and only for idempotent work.
Self-test: When does round robin give uneven load? Why does two-random-choices beat least connections across many balancers? What goes wrong when /health queries the database? Give the four costs of sticky sessions. In what order do you take a server out for a deploy? Why is slow start needed?
Quiz Bank
FoundationalCompare layer 4 and layer 7 load balancing, and say which you would choose for an HTTP API.
Layer 4 operates on TCP connections. It sees source and destination addresses and ports, picks a server when the connection opens, and forwards every packet on that connection to the same place. It never assembles the HTTP request, so it does not know the URL, the method, or any header.
Layer 7 terminates the connection and reads the request. It sees the path, the headers, the cookies and the body, and it decides per request rather than per connection.
What layer 7 gains you, and each of these is a real capability rather than a nicety. Routing by path, so /api and /images go to different pools that scale independently. Retrying a failed request against a different server — impossible at layer 4, because by the time the failure is visible the connection is already committed to one backend. Terminating TLS in one place, so certificates live at the edge instead of on every application server. Adding a request id, enforcing rate limits, and rewriting headers.
What layer 4 gains you. Much higher throughput, because it forwards packets rather than parsing protocols, and complete protocol independence — it will balance a database protocol, a message broker or a game server equally well, because it does not care what is inside.
For an HTTP API I would choose layer 7, and the reasoning is that the features are worth more than the throughput. Per-request retries alone remove a class of user-visible errors that layer 4 cannot touch. Path routing lets one hostname front several services. And centralised TLS termination removes certificate management from every application server, which is an operational saving that compounds.
The one thing to configure carefully as a consequence. Once the balancer terminates the connection, your application sees the balancer's address rather than the client's. The real address arrives in a header, and you must tell your application which proxies to trust, because a client can send that header too. Trusting it unconditionally means anyone can claim any address, which silently defeats rate limiting, audit trails and geographic rules at the same time.
And the arrangement large systems actually use: layer 4 in front of layer 7. A fast layer 4 tier spreads connections across a fleet of layer 7 balancers, which then make the intelligent per-request decisions. You get both properties, and each tier can lose a member without the other noticing.
AppliedYour fleet of ten servers shows one at 95 percent CPU and the rest around 40 percent, with round robin balancing. Diagnose the possible causes.
Round robin gives every server the same number of requests, so an imbalance in load means the requests are not equivalent, or the servers are not equivalent, or something is pinning traffic. Four candidates, in the order I would check them.
Requests vary in cost. The most likely cause. Round robin cannot know that one request is a cached lookup and the next is a report over a year of data. If the expensive requests happen to distribute unevenly — and over short windows they always do — one server ends up with several concurrent heavy requests while the others handle cheap ones. Least connections fixes this directly, because a server working on slow requests accumulates in-flight connections and stops being chosen.
Long-lived connections. If clients use keep-alive, the balancer's rotation applies to connections rather than requests. A client that opens one connection and sends ten thousand requests down it sends all of them to one server. This is extremely common and invisible unless you look for it, and the tell is that connection counts are even while request counts are not. The fix is per-request balancing at layer 7, or capping the number of requests per connection so connections are periodically redistributed.
The servers are not identical. A mixed fleet after a hardware migration, a noisy neighbour on shared infrastructure, or one instance that got a slower host. Weighted round robin handles the known case; the unknown case is another argument for least connections, which adapts without being told.
Something is pinned to that server. A scheduled job, a leader-elected task, a monitoring agent, or a single very large customer whose connection landed there. Check what that server is doing that the others are not, because if the answer is "the nightly aggregation" then the balancing is fine and the problem is co-location.
How to tell them apart quickly. Compare requests per second per server against CPU per server. Even requests with uneven CPU means cost variation or unequal hardware. Uneven requests with even connections means keep-alive is pinning traffic. Uneven both means something is pinned or the rotation is broken.
What I would change regardless of the answer. Move to least connections, because it is strictly better than round robin whenever request costs vary and it costs nothing to adopt. Add per-server request rate and latency to the dashboard, because an imbalance you cannot see is one you will diagnose from CPU graphs and guesswork. And if there are several balancers making independent decisions, prefer two-random-choices over least connections, because ten balancers all independently identifying the same "least loaded" server will send to it simultaneously and produce an oscillation.
InterviewWhat should a health check endpoint check, and what happens if you get it wrong in each direction?
The principle: a health check answers whether this server can do its job — not whether the system as a whole is working.
Too shallow, and traffic goes to dead servers. A check that only proves the port is open, or that returns a hard-coded 200, will keep a server in rotation while it is out of memory, deadlocked, or has a corrupted internal state. The process is accepting connections and failing every one of them, and the balancer cannot tell. A useful shallow check verifies that the application can actually run code and reach its own critical in-process resources.
Too deep, and one dependency takes down everything. This is the worse failure and the one that catches experienced teams. If /health queries the database, then a database blip fails the check on every server simultaneously. The balancer sees zero healthy servers and has nowhere to send traffic. What should have been a partial degradation — some queries failing, cached reads still working, static pages still serving — becomes a complete outage, and the outage was caused by the health check rather than by the database.
The rule is that shared dependencies belong in a separate deep check used by dashboards, alerting and humans, and never by the load balancer. Two endpoints, two audiences.
A third failure: flapping. A server hovering at the threshold flips in and out every few seconds, and each removal disrupts its in-flight connections. The fix is asymmetric thresholds — out after two consecutive failures, back only after five consecutive successes — so removal is fast and return is cautious.
And the safety net for the case where you get it wrong anyway: a panic threshold. If fewer than some percentage of servers report healthy, the balancer should ignore health status entirely and spread traffic across the whole fleet. The reasoning is blunt: with zero healthy servers, sending traffic nowhere guarantees a hundred percent failure, while sending it to possibly-broken servers gives some chance of success. It is an inelegant rule and it converts a whole class of self-inflicted outages into partial degradation.
What I would actually implement: a shallow /health for the balancer that checks only in-process readiness, a separate /health/deep that checks dependencies and is scraped by monitoring, passive health checking so a genuinely broken server is ejected on real failures rather than at the next poll, asymmetric thresholds, and a panic threshold around fifty percent.
StaffDesign the traffic layer for a service running in three regions, including how a deploy happens without a single failed request.
Four layers, each solving one problem, and the discipline is that no layer tries to solve another layer's problem.
Layer one, choosing a region. Latency-based routing rather than geographic, because network paths do not follow maps and a user physically closer to one region often has a faster route to another. Health is part of the decision: a region failing its checks stops receiving traffic. And weights are the important operational feature — being able to send five percent of traffic to a region is how you validate it before trusting it, and how you drain it gracefully during an incident.
The hard part of this layer is not the routing, it is the data, and saying so is the difference between a networking answer and a systems answer. Sending a user to a nearby region whose replica lags means they may not see their own writes (10.7.1). Either writes go to a primary region and reads are served locally with the staleness accepted and bounded, or users are pinned to a home region for consistency and only truly global read-only content is served from anywhere. That choice must be made explicitly, because the failure mode — a user updating their profile and then seeing the old version — is confusing rather than obviously broken, so it survives testing and reaches customers.
Layer two, spreading within a region. A layer 4 tier behind anycast, distributing connections across a fleet of layer 7 balancers. Anycast because failover requires no coordination and no waiting for DNS to expire.
Layer three, the layer 7 balancers. Per-request decisions using two random choices rather than least connections, because there are several balancers and they cannot see each other — ten balancers independently agreeing on the least loaded server will all send to it at once and oscillate. Path routing to separate pools, TLS termination, a request id added here so it flows through everything, retry budgets, and rate limiting.
Layer four, the application servers. Stateless, so any server can serve any request, which is what makes everything above it simple. No sticky sessions; session state lives in a shared store or a signed cookie.
Now the deploy, which is a sequence and the order is the whole answer.
Roll one instance at a time, or in small batches, never the whole fleet — partly for warm-up and partly so a bad build reaches a fraction of users before you stop.
Take an instance out in this order. The instance starts failing its health check while still serving requests normally. You then wait for at least one full health check interval plus a margin, so the balancer has definitely noticed and stopped sending new work. Only then do you begin draining. Skipping that wait is the single most common cause of errors during a deploy: the process shuts down while requests are still arriving, and they fail with a connection reset.
Drain with a timeout. Let in-flight requests finish, but cap the wait so one stuck request cannot block the deploy indefinitely. Anything still running at the cap is terminated, and it should be a small enough number to be within your error budget.
Bring the new instance in slowly. It has an empty cache, an unwarmed connection pool and code the runtime has not yet optimised (3.2). Giving it a full share immediately produces slow responses, which can fail its health check, which removes it, which loops. Ramp it over about thirty seconds.
Watch, then continue. Error rate and latency for the new instance specifically, compared against the fleet. If it is worse, stop the rollout rather than continuing and hoping.
What makes zero failed requests actually achievable, and it is worth saying plainly: the deploy sequence handles requests already in flight, and the retry layer handles the small residue. Neither alone is sufficient — draining without retries loses the requests that arrive in the notice-the-health-check gap, and retries without draining means retrying a large number of failures rather than a handful. Together they cover each other, which is why both belong in the answer.
Flashcards
FlashLayer 4 versus layer 7
L4: addresses and ports, per connection, any TCP protocol, very fast. L7: reads the request, per request, path routing, retries, TLS termination. L7 for HTTP; L4 in front of L7 at scale.
FlashRound robin versus least connections
Round robin is fair in request count. Least connections is fair in work, and adapts when request costs vary. Least response time has the black-hole failure: instant failures look fastest.
FlashTwo random choices
Pick two servers at random, send to the less loaded. Nearly as good as least connections, and no herd effect when many balancers cannot see each other.
FlashHealth check rule
Report whether this server can do its job. Never check shared dependencies — one blip marks everything unhealthy at once. Asymmetric thresholds, plus a panic threshold so zero-healthy means spread everywhere.
FlashDeploy order
Fail the health check → wait a full check interval → drain in flight with a timeout → exit. New instances get slow start. Roll one at a time.
FlashRetry budget
Cap retries as a percentage of total requests. Retry only connection failures, on a different server, only for idempotent work. Three layers of retries is eight attempts per user action.