Appearance
10.2 — Scaling Fundamentals
The first arsenal entry (10.1), and the vocabulary every system-design conversation assumes: vertical vs horizontal (and the honest case for the unfashionable one), stateless vs stateful (the property that makes horizontal scaling possible at all), load balancing at the system level (9.7.6 built the algorithms; here they meet fleets, health, and traffic shifts), and CDNs — plus the arithmetic that tells you which lever to pull, because "add more servers" is the answer to exactly one class of bottleneck.
1. Vertical, horizontal, and the ceiling
Vertical scaling (scale up) buys a bigger machine: more cores, more RAM, faster disks. Horizontal scaling (scale out) buys more machines. The reflex answer is "horizontal, obviously" — the honest answer is a trade with a crossover point:
Vertical is genuinely underrated. Modern cloud instances reach ~200 cores and multiple terabytes of RAM; a single Postgres on such a box handles workloads most teams will never approach. It requires no application changes, preserves single-machine simplicity (transactions, joins, no distributed anything), and is often cheaper in total once you count engineering time. Its limits are real but specific: a hard ceiling (you cannot rent an infinite machine), a price cliff (top-end instances cost superlinearly), and — decisively — it does nothing for availability: one machine is one failure domain, and the restart takes the whole system with it.
Horizontal is the only path past the ceiling and the only path to redundancy: N machines mean the loss of one costs 1/N of capacity rather than everything, and capacity grows by adding boxes. Its price is everything in Part 10: state must live somewhere shared, requests must be routed, failures become partial, and the deployment story gets an orchestration layer.
The practical sequencing most successful systems follow: scale vertically until it's uncomfortable, horizontally when you must, and make the stateless tier horizontal first — because that's where the cost is lowest and the benefit is immediate.
2. Stateless is the magic word
A service is stateless when any instance can serve any request, because everything needed lives in the request itself or in shared storage. It's not an aesthetic preference — it's the precondition that makes horizontal scaling cheap: ⚑Stateless vs stateful services and horizontal scaling. [EQ-219b]
- Add or remove instances at any moment (autoscaling works, spot instances become viable).
- Any instance can die without losing user-visible state (redundancy is real).
- No sticky routing, so the load balancer's job stays simple and deploys can drain cleanly (9.9.7).
Where the state actually goes — because it never disappears, it moves: sessions to Redis or into signed tokens (Part 8.4.2); uploaded files to object storage; caches to a shared tier (or accepted as per-instance duplicates); scheduled jobs to an external scheduler with leader election (10.7.2); WebSocket fan-out to a pub/sub backplane. This list is 9.9.7's multi-process breakage list, one level up: the same law (processes share nothing) applied to machines.
Stateful services still exist and are harder: databases, caches with real data, stateful stream processors, game servers holding a match. They scale by replication (10.5) and partitioning (10.6) rather than by cloning, and they need identity-aware routing (this shard, this leader, this match). The design principle that follows: push statefulness down into a small number of systems built for it, and keep the many application instances above them stateless. A system with one stateful tier and fifty stateless services is operable; fifty stateful services is a career.
3. Load balancing at system scale
9.7.6 built the routing algorithms and health machine as objects. At system scale, three additions matter: ⚑How does load balancing work at the system level? [EQ-220b]
Layer 4 vs Layer 7. An L4 balancer routes by IP/port — it forwards TCP connections without reading them: extremely fast, protocol-agnostic, no TLS termination, and no content-based decisions. An L7 balancer parses HTTP: it can route by path or header, terminate TLS, retry idempotent requests, inject headers, and do per-route rate limiting — at higher cost per request. Rule of thumb: L7 at the edge (where routing intelligence lives), L4 where raw throughput matters (internal fan-out, non-HTTP protocols).
Traffic shifts — the actual question "when does it redirect?" (9.7.6 answered it per-request; here it's fleet-level): the balancer stops sending to an instance when its health state machine says unhealthy (active probes plus passive real-request failures, with hysteresis); it reduces traffic when weights change (canary at 1%, blue-green at 100%, gradual shifts during deploys — 10.11); and it drains an instance before shutdown (readiness false → in-flight finish → close — 9.9.7). The three mechanisms — health, weights, draining — are how every zero-downtime deploy and every automatic failover actually works.
Above the balancer: DNS and anycast. A single balancer is itself a failure domain, so the fleet of balancers is fronted by DNS (multiple A records, health-checked, with the caveat that DNS caching makes failover slow — TTLs are honored unevenly by clients) or by anycast (the same IP announced from many locations; the network routes to the nearest — how CDNs and public DNS resolvers get their global presence). This layering — anycast/DNS → edge L7 → internal L4 → instances — is the shape of essentially every large web system.
4. CDNs: scaling by not serving
The cheapest request is the one your origin never sees. A CDN is a globally distributed cache in front of your system: static assets, images, videos, and increasingly cacheable API responses are served from an edge location near the user. Two wins simultaneously — latency (a Sydney user gets bytes from Sydney, not Virginia) and origin load shedding (a 95% hit ratio means your origin handles 5% of requests).
The mechanics that matter for design: cache keys (URL plus whatever you Vary on — a Vary: Authorization header effectively disables caching, which is why public and personalized content should live on different paths); TTLs vs validation (Cache-Control: max-age for how long it's fresh, ETags for cheap revalidation afterward — 9.6.3); invalidation (purge APIs are slow and rate-limited — the reliable pattern is versioned URLs: /assets/app.9f2c1d.js cached forever, with deploys changing the filename, so invalidation is never needed); and the edge is programmable now (edge functions do auth checks, A/B routing, and personalization without an origin round trip — Part 13.4).
The scaling insight generalizes beyond CDNs: layered caching is how large systems survive — browser → CDN → API gateway/response cache → application cache → database cache ([7.6]'s patterns). Each layer's job is to make the next layer's traffic smaller, and each layer's staleness budget is a product decision, not a technical default.
5. The expert lens
Scaling is bottleneck-chasing, and the bottleneck is never where you assume. Adding instances helps only if the constraint is CPU/memory in the stateless tier; if the constraint is the database's write throughput, more instances make it worse (more connections, more contention — a genuinely common way to turn a slowdown into an outage). The discipline is measure-then-scale: identify the saturating resource (CPU, memory, connections, IOPS, a partner's rate limit — including the invisible pools of 3.8.2), then choose the lever that relieves that resource. 10.12's arithmetic is how you find it before the incident.
Read-heavy and write-heavy scale differently — classify first. Most consumer systems are read-dominated by 100:1 or more, and reads scale beautifully: caching, replicas, CDNs, denormalization all multiply read capacity cheaply. Writes are the hard direction — they need partitioning (10.6), and they're where consistency questions bite. The first question in any scaling discussion should be the ratio, because it determines which half of Part 10 you're about to use.
Statelessness is bought, not found. Every "we'll just keep it in memory" — sessions, in-progress uploads, rate-limit counters, WebSocket registries, computed caches — is a decision to be stateful, usually made by accident and discovered during the first horizontal scale-out (9.9.7's breakage list). Design reviews should ask the question explicitly: what does this instance know that another instance doesn't? The answer is your migration work, and finding it early is worth a quarter of incident time later.
Next: 10.3 — the property distribution takes away most quietly: a shared "now", and what replaces it.
Recall
- Vertical (bigger machine) is underrated: no app changes, single-machine simplicity, often cheapest in total — but a hard ceiling, a price cliff, and zero availability benefit. Horizontal is the only path past both. Sequence: scale up until uncomfortable, out when you must, stateless tier first.
- Stateless = any instance serves any request ⇒ autoscaling, redundancy, clean drains. State doesn't vanish, it moves: sessions → Redis/tokens, files → object storage, caches → shared tier, jobs → external scheduler + leader, sockets → pub/sub backplane. Stateful systems (DBs, matches, stream state) scale via replication + partitioning, and should be few.
- Load balancing at scale: L4 (fast, protocol-agnostic, no content decisions) vs L7 (TLS, path/header routing, retries, per-route limits). Traffic shifts via three mechanisms — health (probes + passive signals + hysteresis), weights (canary/blue-green), draining (readiness → in-flight → close). Above it: DNS (slow failover due to caching) or anycast.
- CDN = scale by not serving: latency + origin shedding. Design levers: cache keys and
Vary(personalization kills caching — split paths), TTL + ETag validation, and versioned URLs instead of purges. Generalizes to layered caching: browser → CDN → gateway → app → DB. - Lens: scaling is bottleneck-chasing (more instances can worsen a DB-bound system); classify read-heavy vs write-heavy first (reads scale cheaply; writes need partitioning); statelessness is bought — ask what this instance knows that others don't.
Self-test: Give vertical scaling's three genuine advantages and its decisive weakness. Where does each kind of "just keep it in memory" state have to move? Name the three mechanisms behind every traffic shift. Why can adding instances make a system slower? What replaces cache purging, and why?
Quiz Bank
FoundationalCompare vertical and horizontal scaling honestly — including when vertical is the right answer.
Vertical (bigger machine) requires no application changes, preserves single-machine semantics (real transactions, joins, no distributed reasoning), and is frequently cheaper once engineering time is counted; modern instances (hundreds of cores, terabytes of RAM) exceed what most workloads ever need. It's the right answer when the bottleneck is a single resource on a single component (typically the database), the team is small, and the traffic is within an order of magnitude of current capacity. Its limits are decisive though: a hard ceiling; superlinear pricing at the top end; downtime to resize (usually); and — the one that ends the debate for user-facing systems — no availability benefit: one machine is one failure domain.
Horizontal (more machines) is the only route past the ceiling and the only route to redundancy (losing one of N costs 1/N), but it buys the entire Part 10 cost list: shared state, routing, partial failure, orchestration. The pragmatic sequence: scale vertically while it's cheap, scale the stateless tier horizontally first (lowest cost, immediate availability benefit), and treat horizontal scaling of stateful components (10.5/10.6) as the serious engineering project it is.
FoundationalWhat makes a service stateless, why does it matter, and where does the state go?
Stateless means any instance can serve any request — everything needed is in the request (tokens, IDs) or in shared storage; no instance holds user-visible state that others lack. It matters because it's the precondition for cheap horizontal scaling: instances can be added or destroyed at any moment (autoscaling, spot capacity), any instance's death costs no user data, no sticky routing is required, and deploys drain cleanly (9.9.7). State doesn't disappear — it relocates: sessions to Redis or into signed tokens (Part 8.4.2); uploaded files to object storage; rate-limit counters to a shared store (9.9.5); in-memory caches either accepted as per-instance duplicates or moved to a shared tier; scheduled jobs to an external scheduler or leader-elected owner (10.7.2); WebSocket/SSE fan-out to a pub/sub backplane.
That relocation list is exactly 9.9.7's multi-process breakage list one level up — the same law (nothing is shared) applied to machines instead of processes. The design guidance: concentrate statefulness in a small number of purpose-built systems and keep the many application instances above them stateless.
AppliedExplain L4 vs L7 load balancing and the three mechanisms that shift traffic away from an instance.
L4 operates on IP/port: it forwards TCP connections without inspecting payloads — very fast, protocol-agnostic (works for anything, not just HTTP), no TLS termination, and no content-aware decisions; ideal for internal, high-throughput, non-HTTP traffic. L7 parses HTTP: routing by path/header/cookie, TLS termination, per-route rate limiting, header injection, request retries, and response manipulation — richer and more expensive per request; ideal at the edge where routing intelligence and TLS belong (9.9.7's reverse-proxy roles, at fleet scale).
Three shift mechanisms: (1) health — the per-instance state machine (9.7.6) fed by active probes and passive request outcomes, with hysteresis so flapping doesn't oscillate traffic; failing health removes the instance from rotation.
(2) Weights — deliberate proportional shifts: 1% canary, gradual ramp, blue-green cutover, zone-preference (10.11). (3) Draining — planned removal: readiness goes false, the balancer stops sending new requests, in-flight requests finish, then the process exits (9.9.7). Every zero-downtime deploy and every automatic failover is some combination of these three — which is why "how does the balancer decide?" has a per-request answer (the algorithm)
and a fleet answer (these mechanisms).
InterviewHow does a CDN scale a system, and what are the four design levers you control?
A CDN caches your content at edge locations worldwide, so requests are answered near the user and never reach your origin — winning latency (Sydney bytes from Sydney) and origin load shedding (a 95% hit ratio leaves 5% of traffic for your servers) simultaneously.
The four levers: (1) Cache key — URL plus whatever you Vary on; personalization is the enemy of caching (a Vary: Authorization makes every user's copy unique, i.e. no caching), so public content and personalized content belong on different paths or use edge personalization.
(2) Freshness policy — Cache-Control: max-age for how long the edge may serve without asking, plus stale-while-revalidate to serve slightly stale content while refreshing (a big availability win); ETags for cheap revalidation after expiry (9.6.3).
(3) Invalidation strategy — purge APIs are slow, eventually consistent, and rate-limited; the robust pattern is content-addressed URLs (app.9f2c1d.js, immutable, cached for a year) so a deploy changes the URL and invalidation never happens. (4) What you push to the edge — beyond static assets: cacheable API responses (product catalogs, menus, public profiles) and, increasingly, edge compute for auth checks and routing (Part 13.4). The generalization worth stating: this is one layer of a caching hierarchy (browser → CDN → gateway → app → DB), where each layer exists to shrink the next layer's traffic.
StaffTraffic tripled; the team added 20 more API instances and latency got worse. Diagnose the likely causes and give the correct scaling response.
Adding stateless instances only helps when the stateless tier is the bottleneck; if it isn't, more instances amplify pressure on the real constraint. Likely causes, in order: (1) Database connection exhaustion — each instance holds a pool, so 20 more instances × pool size can exceed the database's max_connections, causing queueing or refusals; the fix is a connection pooler (PgBouncer) or right-sized pools, not more app servers ([7.2]-territory).
(2) Database CPU/IO saturation — the write path or an unindexed query is the true limit; more instances mean more concurrent queries competing, longer lock waits, and worse tail latency; the fix is query/index work, read replicas for the read share, and caching (10.5/[7.6]).
(3) A downstream partner or shared service with its own rate limit — the fleet now exceeds it, so 429s and retries add latency and load (10.9's retry amplification). (4) A shared cache being thrashed — more instances, more cold local caches, higher miss rate, more origin load (the per-instance cache duplication problem, section 2).
(5) Coordination overhead — leader-elected or lock-based work now contends more (10.7.2). Correct response: stop scaling and measure the saturating resource — DB CPU/IO, connection counts, pool wait times, downstream 429 rates, cache hit ratio, event-loop lag per instance (3.8.2); then apply the lever that relieves that resource: pooling and query work for the database, read replicas and caching for read pressure, partitioning for write pressure (10.6), quota negotiation or request coalescing for partners. Then reduce the instance count back to what the real bottleneck can serve — capacity you can't feed is latency you've added — and record the arithmetic (10.12) so the next scale-up starts from a model rather than a reflex.
Flashcards
FlashVertical vs horizontal
Vertical: no app changes, simple semantics, often cheapest — but ceiling, price cliff, ZERO availability gain. Horizontal: past the ceiling + redundancy, at Part 10's full cost.
FlashStateless payoff
Any instance serves any request ⇒ autoscaling, redundancy, clean drains. State moves: sessions→Redis/tokens, files→object storage, jobs→scheduler+leader, sockets→pub/sub.
FlashL4 vs L7
L4: IP/port, fast, protocol-agnostic, no content decisions. L7: HTTP-aware — TLS, path/header routing, retries, per-route limits. L7 at the edge, L4 for internal throughput.
FlashThree traffic-shift mechanisms
Health (probes + passive + hysteresis) · weights (canary/blue-green) · draining (readiness → in-flight → close). Every failover and zero-downtime deploy is these three.
FlashCDN levers
Cache key + Vary (personalization kills caching) · TTL + stale-while-revalidate + ETags · versioned immutable URLs instead of purges · edge compute.
FlashScaling's first question
Read-heavy or write-heavy? Reads scale cheaply (cache, replicas, CDN); writes need partitioning. And: adding instances can worsen a DB-bound system.
Scenario Drill
DrillA news site is going from 50k to 5M daily readers before an election. Current setup: one Node app on a big VM, one Postgres, images served by the app, personalized homepage per logged-in user, live comment counts on every article. Produce the scaling plan in priority order with the reasoning for each step, and identify the one requirement that fights caching hardest.
Establish the shape first: read-heavy by roughly 1000:1 (millions read, thousands comment) — so nearly all capacity work is on the read path, and the write path (comments) needs only modest attention plus isolation. Priority order. (1) Get images off the app — a CDN in front of object storage with content-addressed URLs: this alone removes the majority of bytes and a large share of requests, costs nothing architecturally, and fixes the 9.9.1 express.static problem (3.8.2's thread pool stops serving files).
(2) CDN the article pages themselves — articles are the definition of cacheable: written once, read millions of times, identical for everyone. Serve them as anonymous HTML with a short TTL plus stale-while-revalidate (so an origin blip doesn't take the site down), and hydrate personalization client-side. This is the single highest-leverage change: a 95%+ hit ratio means origin traffic barely moves as readership goes 100×.
(3) Make the app tier stateless and horizontal — sessions to Redis or tokens, no local files, then autoscale behind an L7 balancer with health/drain configured (9.9.7); this is the redundancy the single VM lacks entirely.
(4) Read replicas for Postgres — article reads that miss the CDN go to replicas; writes (comments) stay on the primary (10.5); accept replica lag for reads that tolerate it, and route "just posted my comment" reads to the primary (read-your-writes — 10.5).
(5) Live comment counts — the requirement that fights everything: it makes every article page unique-per-second, which would defeat CDN caching entirely if embedded in the HTML. Fix by separation: the article page is cached and count-free; the count arrives via a separate, tiny, aggressively cached endpoint (1–5 s TTL, or a WebSocket/SSE stream on high-traffic articles), so a number that changes constantly never invalidates a document that doesn't.
(6) Protect the write path — comment posting gets its own rate limits and, under election-night load, a queue so bursts don't touch the primary synchronously (10.8.1). The hardest-fighting requirement, named explicitly: the personalized homepage. Personalization makes the cache key user-specific, which is the CDN's kryptonite (section 4). The resolutions, in preference order: render a cacheable shell and personalize client-side after load; or move personalization to the edge (edge functions assembling from cached fragments); or accept an uncached homepage but ensure it's the only uncached page, sized accordingly. The plan's shape is the chapter's thesis: most of the answer was making requests never reach the origin, and the one requirement that resisted caching got redesigned rather than scaled.