10.2 — Scaling Fundamentals

Every system-design conversation assumes you know four ideas:

• Vertical versus horizontal scaling, including the honest case for the unfashionable one, the bigger machine.
• Stateless versus stateful, the property that makes adding more servers possible at all.
• Load balancing across a whole fleet. 9.7.6 built the algorithms. Here they meet many servers, health checks and shifting traffic.
• CDNs.

You also need the arithmetic that tells you which lever to pull, because "add more servers" only fixes one kind of bottleneck.

1. Vertical, horizontal, and the ceiling

Vertical scaling (scale up) buys a bigger machine: more cores, more RAM, faster disks. Horizontal scaling (scale out) buys more machines. The reflex answer is "horizontal, obviously" — the honest answer is a trade with a crossover point:

Vertical is genuinely underrated. Modern cloud instances reach ~200 cores and multiple terabytes of RAM; a single Postgres on such a box handles workloads most teams will never approach. It requires no application changes, preserves single-machine simplicity (transactions, joins, no distributed anything), and is often cheaper in total once you count engineering time. Its limits are real, but specific:

• There is a hard ceiling. You cannot rent an infinitely big machine.
• There is a price cliff. The biggest machines cost far more than their size suggests.
• And the one that decides it: it does nothing for availability. One machine is one thing that can fail, and when it restarts, the whole system goes with it.

Horizontal is the only path past the ceiling and the only path to redundancy: N machines mean the loss of one costs 1/N of capacity rather than everything, and capacity grows by adding boxes. Its price is everything in Part 10: state must live somewhere shared, requests must be routed, failures become partial, and the deployment story gets an orchestration layer.

The practical sequencing most successful systems follow: scale vertically until it's uncomfortable, horizontally when you must, and make the stateless tier horizontal first — because that's where the cost is lowest and the benefit is immediate.

2. Stateless is the magic word

A service is stateless when any instance can serve any request, because everything needed lives in the request itself or in shared storage. It's not an aesthetic preference — it's the precondition that makes horizontal scaling cheap: ^{⚑Stateless vs stateful services and horizontal scaling. [EQ-219b]}

• Add or remove instances at any moment (autoscaling works, spot instances become viable).
• Any instance can die without losing user-visible state (redundancy is real).
• No sticky routing, so the load balancer's job stays simple and deploys can drain cleanly (9.9.7).

Where the state actually goes. It never disappears. It moves:

• Sessions move to Redis, or into signed tokens (Part 8.4.2).
• Uploaded files move to object storage.
• Caches move to a shared cache layer. Or you accept that each server has its own copy.
• Scheduled jobs move to an outside scheduler that elects one leader to run them (10.7.2).
• Sending WebSocket messages to many users moves to a shared publish-subscribe system.

This list is 9.9.7's multi-process breakage list, one level up: the same law (processes share nothing) applied to machines.

Stateful services still exist and are harder: databases, caches with real data, stateful stream processors, game servers holding a match. They scale by replication (10.5) and partitioning (10.6) rather than by cloning, and they need identity-aware routing (this shard, this leader, this match). The design principle that follows: push statefulness down into a small number of systems built for it, and keep the many application instances above them stateless. A system with one stateful tier and fifty stateless services is operable; fifty stateful services is a career.

3. Load balancing at system scale

9.7.6 built the routing algorithms and health machine as objects. At system scale, three additions matter: ^{⚑How does load balancing work at the system level? [EQ-220b]}

Layer 4 vs Layer 7. An L4 balancer routes by IP/port — it forwards TCP connections without reading them: extremely fast, protocol-agnostic, no TLS termination, and no content-based decisions. An L7 balancer parses HTTP: it can route by path or header, terminate TLS, retry idempotent requests, inject headers, and do per-route rate limiting — at higher cost per request. Rule of thumb: L7 at the edge (where routing intelligence lives), L4 where raw throughput matters (internal fan-out, non-HTTP protocols).

Shifting traffic: the real answer to "when does it send requests elsewhere?" 9.7.6 answered it for one request. Here it is for the whole fleet. There are three mechanisms:

• Health. The balancer stops sending to a server when its health state machine says unhealthy. It checks with its own probes, and it also watches real requests failing. It waits for several results in a row before changing its mind, so one blip does not flip it.
• Weights. The balancer sends less traffic when weights change. A canary gets 1%, blue-green switches 100%, and deploys shift traffic gradually (10.11).
• Draining. Before a server shuts down, it reports "not ready", finishes the requests in progress, and then closes (9.9.7).

Health, weights and draining are how every deploy without downtime, and every automatic failover, actually works.

Above the balancer: DNS and anycast. One load balancer is itself something that can fail. So a group of load balancers sits behind one of two things:

• DNS, with several A records and health checks. The catch is that DNS caching makes failover slow, because clients respect cache expiry times unevenly.
• Anycast. The same IP address is announced from many places, and the network sends each user to the nearest. That is how CDNs and public DNS servers are present all over the world.

Nearly every large web system has this layered shape: anycast or DNS, then edge L7 balancers, then internal L4 balancers, then the servers.

Figure 1 — The standard traffic path. Each layer removes load from the next: the CDN answers what it can without touching origin, the L7 balancer spreads what remains across interchangeable stateless instances, and all durable state concentrates in a small number of systems built to hold it.

4. CDNs: scaling by not serving

The cheapest request is the one your origin never sees. A CDN is a globally distributed cache in front of your system: static assets, images, videos, and increasingly cacheable API responses are served from an edge location near the user. Two wins simultaneously — latency (a Sydney user gets bytes from Sydney, not Virginia) and origin load shedding (a 95% hit ratio means your origin handles 5% of requests).

The mechanics that matter for design:

• Cache keys. The key is the URL plus whatever you list in Vary. A Vary: Authorization header makes every user's copy separate, which in practice switches caching off. That is why public and personalised content should live on different paths.
• Expiry versus checking. Cache-Control: max-age says how long a copy stays fresh. After that, an ETag lets the CDN cheaply ask "has it changed?" (9.6.3).
• Clearing the cache. Purge APIs are slow and limited. The reliable approach is versioned URLs. /assets/app.9f2c1d.js is cached forever, and each deploy changes the file name, so nothing ever needs clearing.
• The edge can now run code. Edge functions can check logins, split traffic for A/B tests and personalise pages without a trip to your servers (Part 13.4).

The scaling insight generalizes beyond CDNs: layered caching is how large systems survive — browser → CDN → API gateway/response cache → application cache → database cache ([7.6]'s patterns). Each layer's job is to make the next layer's traffic smaller, and each layer's staleness budget is a product decision, not a technical default.

5. The expert lens

Scaling is bottleneck-chasing, and the bottleneck is never where you assume. Adding instances helps only if the constraint is CPU/memory in the stateless tier; if the constraint is the database's write throughput, more instances make it worse (more connections, more contention — a genuinely common way to turn a slowdown into an outage). The discipline is measure-then-scale: identify the saturating resource (CPU, memory, connections, IOPS, a partner's rate limit — including the invisible pools of 3.8.2), then choose the lever that relieves that resource. 10.12's arithmetic is how you find it before the incident.

Read-heavy and write-heavy scale differently — classify first. Most consumer systems are read-dominated by 100:1 or more, and reads scale beautifully: caching, replicas, CDNs, denormalization all multiply read capacity cheaply. Writes are the hard direction — they need partitioning (10.6), and they're where consistency questions bite. The first question in any scaling discussion should be the ratio, because it determines which half of Part 10 you're about to use.

Statelessness is bought, not found. Every "we'll just keep it in memory" — sessions, in-progress uploads, rate-limit counters, WebSocket registries, computed caches — is a decision to be stateful, usually made by accident and discovered during the first horizontal scale-out (9.9.7's breakage list). Design reviews should ask the question explicitly: what does this instance know that another instance doesn't? The answer is your migration work, and finding it early is worth a quarter of incident time later.

Next: 10.3 — the property distribution takes away most quietly: a shared "now", and what replaces it.

Recall

• Vertical (bigger machine) is underrated: no app changes, single-machine simplicity, often cheapest in total — but a hard ceiling, a price cliff, and zero availability benefit. Horizontal is the only path past both. Sequence: scale up until uncomfortable, out when you must, stateless tier first.
• Stateless = any instance serves any request ⇒ autoscaling, redundancy, clean drains. State doesn't vanish, it moves: sessions → Redis/tokens, files → object storage, caches → shared tier, jobs → external scheduler + leader, sockets → pub/sub backplane. Stateful systems (DBs, matches, stream state) scale via replication + partitioning, and should be few.
• Load balancing at scale: L4 (fast, protocol-agnostic, no content decisions) vs L7 (TLS, path/header routing, retries, per-route limits). Traffic shifts via three mechanisms — health (probes + passive signals + hysteresis), weights (canary/blue-green), draining (readiness → in-flight → close). Above it: DNS (slow failover due to caching) or anycast.
• CDN = scale by not serving: latency + origin shedding. Design levers: cache keys and Vary (personalization kills caching — split paths), TTL + ETag validation, and versioned URLs instead of purges. Generalizes to layered caching: browser → CDN → gateway → app → DB.
• Lens: scaling is bottleneck-chasing (more instances can worsen a DB-bound system); classify read-heavy vs write-heavy first (reads scale cheaply; writes need partitioning); statelessness is bought — ask what this instance knows that others don't.

Self-test: Give vertical scaling's three genuine advantages and its decisive weakness. Where does each kind of "just keep it in memory" state have to move? Name the three mechanisms behind every traffic shift. Why can adding instances make a system slower? What replaces cache purging, and why?

Quiz Bank

FoundationalCompare vertical and horizontal scaling honestly — including when vertical is the right answer.

Vertical (bigger machine) requires no application changes, preserves single-machine semantics (real transactions, joins, no distributed reasoning), and is frequently cheaper once engineering time is counted; modern instances (hundreds of cores, terabytes of RAM) exceed what most workloads ever need. It is the right answer when three things are true: the bottleneck is one resource on one component, usually the database; the team is small; and traffic is within ten times what you handle now. But its limits are decisive:

• There is a hard ceiling.
• The biggest machines cost far more than their size suggests.
• Resizing usually means downtime.
• And the one that ends the debate for anything users touch: it does nothing for availability. One machine is one thing that can fail.

Horizontal (more machines) is the only route past the ceiling and the only route to redundancy (losing one of N costs 1/N), but it buys the entire Part 10 cost list: shared state, routing, partial failure, orchestration. The pragmatic sequence: scale vertically while it's cheap, scale the stateless tier horizontally first (lowest cost, immediate availability benefit), and treat horizontal scaling of stateful components (10.5/10.6) as the serious engineering project it is.

FoundationalWhat makes a service stateless, why does it matter, and where does the state go?

Stateless means any server can handle any request. Everything needed is either in the request itself, such as tokens and IDs, or in shared storage. No server holds anything a user would miss that the other servers lack.

It matters because it is what makes adding servers cheap:

• Servers can be added or removed at any moment, for autoscaling or cheap spare capacity.
• When a server dies, no user data dies with it.
• Users do not have to keep going back to the same server.
• Deploys can drain servers cleanly (9.9.7).

The state does not disappear. It moves:

• Sessions go to Redis, or into signed tokens (Part 8.4.2).
• Uploaded files go to object storage.
• Rate-limit counters go to a shared store (9.9.5).
• In-memory caches either stay as a copy on each server, which you accept, or move to a shared cache.
• Scheduled jobs go to an outside scheduler, or to one server elected as leader (10.7.2).
• Pushing WebSocket or SSE messages to many users goes through a shared publish-subscribe system.

That relocation list is exactly 9.9.7's multi-process breakage list one level up — the same law (nothing is shared) applied to machines instead of processes. The design guidance: concentrate statefulness in a small number of purpose-built systems and keep the many application instances above them stateless.

AppliedExplain L4 vs L7 load balancing and the three mechanisms that shift traffic away from an instance.

L4 operates on IP/port: it forwards TCP connections without inspecting payloads — very fast, protocol-agnostic (works for anything, not just HTTP), no TLS termination, and no content-aware decisions; ideal for internal, high-throughput, non-HTTP traffic. L7 parses HTTP: routing by path/header/cookie, TLS termination, per-route rate limiting, header injection, request retries, and response manipulation — richer and more expensive per request; ideal at the edge where routing intelligence and TLS belong (9.9.7's reverse-proxy roles, at fleet scale).

Three shift mechanisms: (1) health — the per-instance state machine (9.7.6) fed by active probes and passive request outcomes, with hysteresis so flapping doesn't oscillate traffic; failing health removes the instance from rotation.

(2) Weights — deliberate proportional shifts: 1% canary, gradual ramp, blue-green cutover, zone-preference (10.11). (3) Draining — planned removal: readiness goes false, the balancer stops sending new requests, in-flight requests finish, then the process exits (9.9.7). Every zero-downtime deploy and every automatic failover is some combination of these three — which is why "how does the balancer decide?" has a per-request answer (the algorithm)

and a fleet answer (these mechanisms).

InterviewHow does a CDN scale a system, and what are the four design levers you control?

A CDN caches your content at edge locations worldwide, so requests are answered near the user and never reach your origin — winning latency (Sydney bytes from Sydney) and origin load shedding (a 95% hit ratio leaves 5% of traffic for your servers) simultaneously.

The four levers: (1) Cache key — URL plus whatever you Vary on; personalization is the enemy of caching (a Vary: Authorization makes every user's copy unique, i.e. no caching), so public content and personalized content belong on different paths or use edge personalization.

(2) Freshness policy — Cache-Control: max-age for how long the edge may serve without asking, plus stale-while-revalidate to serve slightly stale content while refreshing (a big availability win); ETags for cheap revalidation after expiry (9.6.3).

(3) Invalidation strategy — purge APIs are slow, eventually consistent, and rate-limited; the robust pattern is content-addressed URLs (app.9f2c1d.js, immutable, cached for a year) so a deploy changes the URL and invalidation never happens. (4) What you push to the edge — beyond static assets: cacheable API responses (product catalogs, menus, public profiles) and, increasingly, edge compute for auth checks and routing (Part 13.4). The generalization worth stating: this is one layer of a caching hierarchy (browser → CDN → gateway → app → DB), where each layer exists to shrink the next layer's traffic.

StaffTraffic tripled; the team added 20 more API instances and latency got worse. Diagnose the likely causes and give the correct scaling response.

Adding stateless instances only helps when the stateless tier is the bottleneck; if it isn't, more instances amplify pressure on the real constraint. Likely causes, in order: (1) The database runs out of connections. Each server holds its own pool of connections. So 20 more servers, multiplied by the pool size, can go over the database's max_connections. Requests then wait, or are refused. The fix is a connection pooler such as PgBouncer, or smaller pools, not more app servers (7.2).

(2) Database CPU/IO saturation — the write path or an unindexed query is the true limit; more instances mean more concurrent queries competing, longer lock waits, and worse tail latency; the fix is query/index work, read replicas for the read share, and caching (10.5/[7.6]).

(3) A downstream partner or shared service with its own rate limit — the fleet now exceeds it, so 429s and retries add latency and load (10.9's retry amplification). (4) A shared cache being thrashed — more instances, more cold local caches, higher miss rate, more origin load (the per-instance cache duplication problem, section 2).

(5) Coordination overhead — leader-elected or lock-based work now contends more (10.7.2). The right response:

1. Stop adding servers, and measure which resource is full. Look at database CPU and disk, connection counts, time spent waiting for a pooled connection, how often partners return 429, the cache hit rate, and event-loop lag on each server (3.8.2).
2. Pull the lever that relieves that resource. For the database, use connection pooling and better queries. For too many reads, add read copies and caching. For too many writes, partition the data (10.6). For partners, negotiate a bigger quota, or combine duplicate requests.
3. Cut the number of servers back to what the real bottleneck can serve. Servers you cannot feed only add delay.
4. Write down the arithmetic (10.12), so the next time you scale you start from a model instead of a reflex.

Flashcards

FlashVertical vs horizontal

Vertical: no app changes, simple semantics, often cheapest — but ceiling, price cliff, ZERO availability gain. Horizontal: past the ceiling + redundancy, at Part 10's full cost.

FlashStateless payoff

Any instance serves any request ⇒ autoscaling, redundancy, clean drains. State moves: sessions→Redis/tokens, files→object storage, jobs→scheduler+leader, sockets→pub/sub.

FlashL4 vs L7

L4: IP/port, fast, protocol-agnostic, no content decisions. L7: HTTP-aware — TLS, path/header routing, retries, per-route limits. L7 at the edge, L4 for internal throughput.

FlashThree traffic-shift mechanisms

Health (probes + passive + hysteresis) · weights (canary/blue-green) · draining (readiness → in-flight → close). Every failover and zero-downtime deploy is these three.

FlashCDN levers

Cache key + Vary (personalization kills caching) · TTL + stale-while-revalidate + ETags · versioned immutable URLs instead of purges · edge compute.

FlashScaling's first question

Read-heavy or write-heavy? Reads scale cheaply (cache, replicas, CDN); writes need partitioning. And: adding instances can worsen a DB-bound system.

Scenario Drill

DrillA news site is going from 50k to 5M daily readers before an election. Current setup: one Node app on a big VM, one Postgres, images served by the app, personalized homepage per logged-in user, live comment counts on every article. Produce the scaling plan in priority order with the reasoning for each step, and identify the one requirement that fights caching hardest.

Establish the shape first: read-heavy by roughly 1000:1 (millions read, thousands comment) — so nearly all capacity work is on the read path, and the write path (comments) needs only modest attention plus isolation. Priority order. (1) Get images off the app — a CDN in front of object storage with content-addressed URLs: this alone removes the majority of bytes and a large share of requests, costs nothing architecturally, and fixes the 9.9.1 express.static problem (3.8.2's thread pool stops serving files).

(2) CDN the article pages themselves — articles are the definition of cacheable: written once, read millions of times, identical for everyone. Serve them as anonymous HTML with a short TTL plus stale-while-revalidate (so an origin blip doesn't take the site down), and hydrate personalization client-side. This is the single highest-leverage change: a 95%+ hit ratio means origin traffic barely moves as readership goes 100×.

(3) Make the app tier stateless and horizontal — sessions to Redis or tokens, no local files, then autoscale behind an L7 balancer with health/drain configured (9.9.7); this is the redundancy the single VM lacks entirely.

(4) Read replicas for Postgres — article reads that miss the CDN go to replicas; writes (comments) stay on the primary (10.5); accept replica lag for reads that tolerate it, and route "just posted my comment" reads to the primary (read-your-writes — 10.5).

(5) Live comment counts — the requirement that fights everything: it makes every article page unique-per-second, which would defeat CDN caching entirely if embedded in the HTML. Fix by separation: the article page is cached and count-free; the count arrives via a separate, tiny, aggressively cached endpoint (1–5 s TTL, or a WebSocket/SSE stream on high-traffic articles), so a number that changes constantly never invalidates a document that doesn't.

(6) Protect the write path — comment posting gets its own rate limits and, under election-night load, a queue so bursts don't touch the primary synchronously (10.8.1). The hardest-fighting requirement, named explicitly: the personalized homepage. Personalization makes the cache key user-specific, which is the CDN's kryptonite (section 4). The resolutions, in preference order: render a cacheable shell and personalize client-side after load; or move personalization to the edge (edge functions assembling from cached fragments); or accept an uncached homepage but ensure it's the only uncached page, sized accordingly. The plan's shape is the chapter's thesis: most of the answer was making requests never reach the origin, and the one requirement that resisted caching got redesigned rather than scaled.