Appearance
10.1 — Why Distributed, and the Senior Smell Test
Part 9 designed systems that live inside one process, where a function call either returns or throws, memory is shared, and "now" is unambiguous. Part 10 removes all three guarantees. The moment work spans machines, calls can hang forever, partial failure becomes the normal case, clocks disagree, messages duplicate and reorder, and the same operation may execute twice with nobody able to prove it. This page frames why anyone accepts that trade, catalogs the arsenal of concerns the rest of the Part develops, and — most valuable for interviews and for real design reviews — teaches the smell test: how seniors recognize which distributed concerns a problem carries, in seconds, instead of reciting a checklist. ⚑Senior engineer system design arsenal. [EQ-235b]
1. Three reasons to distribute — and one bad one
Distribution is a cost you pay to buy something specific. The legitimate purchases:
- Scale beyond one machine — traffic, data, or compute exceeding the biggest box you can rent (and vertical scaling has a real ceiling and a price cliff — 10.2).
- Availability through redundancy — one machine dies, others serve; one region burns, another region answers. Single machines have single-digit-nines availability whatever you spend.
- Latency by locality — users in Sydney shouldn't wait for Virginia; data near its consumers is the CDN's entire business (Part 13.4).
The bad reason, stated plainly because it drives most unnecessary microservice migrations: organizational aspiration. "We should be microservices" without a scaling, availability, or team-autonomy problem buys you every cost on this page — network failure modes, distributed debugging, eventual consistency, deployment coordination — in exchange for a diagram that looks modern. The honest test: name the specific limit you're hitting. If you can't, the distributed version is strictly worse than the monolith you have, and Part 11's case studies will show you that even at genuine scale, the number of moving pieces is a cost the best designs actively minimize.
The costs, upfront and unavoidable: the network is unreliable, slow, and partitionable (the fallacies of distributed computing: the network is not reliable, latency is not zero, bandwidth is not infinite, topology does change, and there is no single administrator — Deutsch and Gosling's list, 1994, still describing every incident review); partial failure replaces success/failure (component B is down for A but up for C); no global now (10.3); coordination costs latency (every consistency guarantee is a round trip you must budget); and debugging becomes archaeology unless you build for it (10.10).
2. The arsenal: the fourteen concerns
Every distributed design decision falls into one of these categories. This is the map of Part 10 and the checklist a design review runs — but section 3 explains why you should not recite it linearly:
| # | Concern | The question it asks | Where it's built |
|---|---|---|---|
| 1 | Communication pattern | sync request/response, async messaging, or streaming? | 5.8, 10.8.1 |
| 2 | Execution model | inline, background job, scheduled, or event-driven? | 10.8.1 |
| 3 | State ownership | who is the source of truth for this fact? | 10.4 |
| 4 | Consistency guarantee | strong, causal, read-your-writes, or eventual? | 10.7.1 |
| 5 | Delivery semantics | at-most-once, at-least-once, effectively-once? | 10.4 |
| 6 | Ordering | global, per-key, or none? | 10.3, 10.6 |
| 7 | Failure handling | timeout, retry, circuit break, degrade, compensate? | 10.9 |
| 8 | Coordination | leader election, locks, consensus — or avoidable? | 10.7.2 |
| 9 | Scaling axis | more replicas, more partitions, or more caching? | 10.2, 10.6 |
| 10 | Observability | how will we see this fail? | 10.10 |
| 11 | API contract | versioning, compatibility, idempotency keys | 9.6, 10.11 |
| 12 | Storage pattern | replicated, partitioned, cached, denormalized? | 10.5, 10.6 |
| 13 | Security boundary | service-to-service auth, trust zones | Part 8.6 |
| 14 | Evolution & cost | rollout, migration, and the bill | 10.11, 10.12 |
3. The smell test: how seniors actually work
Juniors walk the checklist. Seniors smell the shape of the problem and jump to the three or four concerns that matter, because certain requirement phrasings imply their machinery. The recognition table — memorize the triggers, not the list:
| When you hear… | Smell | Reach for |
|---|---|---|
| "must not lose it" / money / legal | durability + delivery semantics | durable queue, outbox, idempotency keys (10.4, 10.8.4) |
| "should be instant for the user" | async + eventual consistency | accept work, return 202, do it in background (10.8.1) |
| "two users can do X at once" | contention | per-entity serialization, conditional writes (9.5.4, 10.7.1) |
| "the other team's service" | partial failure | timeout + retry + circuit breaker + fallback (10.9) |
| "it retried and charged twice" | duplicate delivery | idempotency keys, dedupe windows (10.4) |
| "we need it to be fast globally" | locality | CDN, read replicas, regional writes (10.2, 10.5) |
| "the table is too big" | partitioning | shard key choice, hot-spot analysis (10.6) |
| "one customer's spike broke everyone" | isolation | bulkheads, quotas, per-tenant limits (10.9) |
| "in the right order" | ordering | per-key partitioning, sequence numbers (10.3, 10.6) |
| "eventually both systems agree" | reconciliation | outbox, idempotent sync, drift detection (10.8.4) |
The senior instinct compressed into one sentence — the thing interviewers listen for: "async + at-least-once + idempotent + observable" covers a startling fraction of real designs, because most requirements decompose into do it in the background, accept that messages may repeat, make repeats harmless, and be able to see it. When you hear a candidate say that unprompted, you're hearing someone who has operated systems, not read about them.
4. The expert lens
Distribution is a liability you take on for a named asset. Every service boundary adds a network hop (latency, failure mode), a deployment coordination point, a data-consistency question, and a debugging seam. The best distributed systems are the ones that distribute as little as possible: a well-factored monolith with a queue for background work handles more traffic than most teams will ever see, and Part 11's designs earn each split by pointing at a specific limit. When a design review can't name what a boundary buys, the boundary is a cost with no purchase.
Everything in Part 10 is Part 9's ideas plus partial failure. The queue is 9.5.4's producer-consumer with durability; the saga is 9.4.15's command-with-compensation across machines; the circuit breaker is a state machine (9.4.14); idempotency keys are 9.6.3's HTTP mechanism generalized. That continuity is deliberate and it's why this book taught LLD first: you already own the shapes; Part 10 teaches what breaks when the arrows cross machines.
Design reviews are won by naming trade-offs, not by naming technologies. "We'll use Kafka" is not a design; "these events must survive consumer downtime, be replayable for a new consumer, and preserve per-customer order — which is why a partitioned log rather than a queue" is. Every chapter ahead ends in a decision ledger style — alternatives, choice, cost — because that's what distinguishes an architect from a catalog.
Next: 10.2 — the first arsenal entry in depth: how systems actually scale, why stateless is the magic word, load balancing at the system level, and where CDNs fit.
Recall
- Distribute to buy scale beyond one machine, availability through redundancy, or latency by locality — never for organizational fashion. If you can't name the limit you're hitting, the monolith wins.
- The costs are structural: the fallacies of distributed computing (unreliable network, non-zero latency, finite bandwidth, changing topology, no single admin), partial failure, no global "now", coordination-as-latency, and archaeology-grade debugging.
- The arsenal (14 concerns): communication pattern · execution model · state ownership · consistency · delivery semantics · ordering · failure handling · coordination · scaling axis · observability · API contract · storage pattern · security boundary · evolution & cost.
- The smell test: requirement phrasings map directly to machinery — "must not lose it" → durable queue + outbox + idempotency; "instant for the user" →
202+ background; "two at once" → per-entity serialization + conditional writes; "flaky partner" → timeout/retry/breaker/fallback; "too big a table" → partitioning; "one tenant broke everyone" → bulkheads. Compressed instinct: async + at-least-once + idempotent + observable. - Continuity: Part 10 = Part 9's shapes (producer-consumer, command+compensation, state machines, idempotency) plus partial failure. Reviews are won by naming trade-offs, not technologies.
Self-test: Give the three legitimate reasons to distribute and the honest test for the bad one. Recite five fallacies. Name the machinery for: "we can't lose payments", "search must feel instant", "two admins edit one record". What four words compress the senior instinct?
Quiz Bank
FoundationalWhen should a system be distributed, and what does it cost?
Distribute to buy one of three things: scale past a single machine's ceiling (traffic, data volume, or compute — and note vertical scaling is real and often cheaper up to a point, 10.2); availability through redundancy (any single machine has limited nines regardless of spend); or latency through locality (serving users from near them). The bad reason is organizational fashion — adopting microservices without a named limit — which buys every cost and no benefit. Costs, all unavoidable: the fallacies of distributed computing (the network is unreliable, latency non-zero, bandwidth finite, topology changing, administration plural); partial failure — the defining difference, where a component is simultaneously up for one caller and down for another, and a timeout leaves you unable to distinguish "not done" from "done, reply lost"; no global clock (10.3); coordination priced in round trips; and observability that must be engineered rather than assumed (10.10). The test to apply in any review: name the specific limit being hit — if none exists, the simpler topology is strictly better.
AppliedA product manager says: 'When a customer places an order, send the confirmation email, update inventory, notify the warehouse, and refresh their loyalty points — and the order page must respond instantly.' Run the smell test.
Four smells in one sentence. "Must respond instantly" + four downstream effects → async execution model: accept the order, persist it, return 202/201 immediately, and do the rest in the background (10.8.1) — putting four network calls in the request path makes latency the sum and availability the product of all four.
"Inventory" and "loyalty points" → state ownership + delivery semantics: these are must-happen effects on other systems' state, so they need durable messaging with at-least-once delivery and idempotent consumers (a retried inventory decrement must not double-decrement — 10.4); the write of the order and the publishing of the event must not diverge, which is the transactional outbox (10.8.4).
"Email" → loss-tolerable-ish but user-visible: same queue, lower priority, with retry and a dead-letter path (10.8.1). "Warehouse" → a third-party or separate-team system → failure handling: timeout, retry with backoff, circuit breaker, and a degradation story (queue and alert rather than fail the order — 10.9). Then the consistency conversation the PM didn't have: inventory becomes eventually consistent with the order, so oversell is possible in the window — either accept it with reconciliation, or reserve inventory synchronously before accepting the order (a deliberate latency-vs-correctness trade to put in front of the business). That last move — surfacing the trade rather than silently choosing — is the senior behavior the exercise trains.
InterviewWhat is partial failure, and why does it change design more than any other distributed property?
In a single process, a function call has two outcomes: it returns or it throws, and either way you know. Across a network there is a third: it times out, and you cannot distinguish "the request never arrived" from "it executed and the reply was lost" (2.7's lost-reply problem). That ambiguity propagates into every layer of design:
retries become dangerous unless operations are idempotent (hence idempotency keys — 10.4); failure is no longer global — service B may be reachable from C and not from A (a network partition, not a crash), so "is it down?" has no single answer and health must be per-observer (9.7.6's per-server health machine, at system scale); cascading failure becomes the dominant outage mode (A's retries amplify B's overload — 10.9); and consistency requires coordination, which costs latency and availability during partitions (CAP — 10.7.1). It changes design more than scale or latency because it invalidates the reasoning model — you can no longer assume a call's effect matches its result — and every mechanism in Part 10 (timeouts, retries, idempotency, sagas, consensus, reconciliation) exists to restore some usable form of certainty on top of that ambiguity.
StaffA director proposes splitting your 200k-line monolith into 30 microservices to 'move faster'. Deployments currently take 40 minutes and three teams contend for releases. Give the analysis and the counter-proposal.
Separate the symptom from the diagnosis. The stated pain is delivery coupling — three teams serializing on one release train — not scale, availability, or latency. Microservices can address delivery coupling (independent deploys), but they do it by adding network failure modes, distributed data consistency, and operational surface: paying a distributed-systems tax to buy an organizational property. Before accepting that trade, exhaust the cheaper purchases of the same property:
(1) modular monolith — enforce module boundaries in-process (clear ownership, no cross-module database access, contracts as interfaces — 9.9.6), which delivers most of the autonomy with none of the network; (2) deployment pipeline work — 40-minute deploys are usually test-suite and build problems, and halving them halves the contention without touching architecture; (3) trunk-based development with feature flags — decouples release from deploy, which is what "move faster" usually means; (4) extract the two or three genuinely independent components that have different scaling profiles or change cadences (the classic candidates: async job processing, a public API with different SLOs, a compute-heavy pipeline) — a strangler-fig extraction with a named limit each (10.11).
Counter-proposal: a staged plan whose first quarter is (1)+(2)+(3), measured by deploy frequency, lead time, and change-failure rate; and whose second quarter extracts services only where a metric says so — with each extraction carrying a written justification naming which of the three legitimate reasons it buys, plus the operational prerequisites (distributed tracing, per-service SLOs, an on-call rotation that can debug across the seam — 10.10).
State the risk plainly: 30 services with 3 teams is a staffing mismatch — the well-known failure where the distributed system's coordination cost lands on the same humans who were already the bottleneck, and delivery gets slower, not faster.
Flashcards
FlashThree reasons to distribute
Scale past one machine · availability via redundancy · latency via locality. Not "we should be microservices" — name the limit or don't split.
FlashFallacies of distributed computing
Network is reliable · latency is zero · bandwidth infinite · topology static · one administrator (+ transport cost zero, network homogeneous). Every incident review revisits one.
FlashPartial failure
Timeout ≠ failure: can't distinguish never-arrived from done-reply-lost. Invalidates the call/return reasoning model — hence idempotency, retries, sagas, reconciliation.
FlashThe senior instinct in four words
Async + at-least-once + idempotent + observable — covers a startling fraction of real designs.
FlashSmell test samples
"Must not lose" → durable queue + outbox + idempotency · "instant" → 202 + background · "two at once" → per-entity serialization + CAS · "flaky partner" → timeout/retry/breaker.
Scenario Drill
DrillInterview opener: 'Design a system that lets restaurants update their menus, and shows updated menus to millions of customers within a minute.' Before drawing anything, run the full smell test aloud and produce the concern list that will structure your design — then state the two questions you'd ask the interviewer.
Smell test, phrase by phrase. "Restaurants update menus" — a write path with a clear state owner (the restaurant's menu; concern 3): low volume, high correctness, needs validation and an audit trail. "Millions of customers" + "shows menus" — a read path four to six orders of magnitude larger than the write path: this is the design's central asymmetry, and it implies caching and replication (concerns 9, 12) — read replicas or, better, a CDN-fronted, versioned menu document, because a menu is a document that changes rarely and is read constantly (10.2).
"Within a minute" — an explicit consistency budget (concern 4): not strong consistency, not "eventually" — a bounded staleness SLO, which is a gift: it licenses aggressive caching with a ≤60 s TTL, or event-driven invalidation with a TTL backstop (10.5).
"Updates propagate" — delivery semantics (concern 5): publish menu-changed events, at-least-once, with consumers idempotent on (menuId, version) so duplicates are harmless (10.4); the write and the publish must not diverge → outbox (10.8.4).
Implied but unstated — ordering (concern 6): two rapid edits must not land out of order, so events carry a monotonic version and consumers ignore older ones (last-writer-wins by version, not by arrival — a cheap, correct choice here); failure handling (concern 7): if propagation lags, customers see a stale menu — a degradation, not an outage, and the design should make staleness visible (a lastUpdated timestamp) rather than pretend; observability (concern 10): propagation-lag as a first-class metric, because "within a minute" is an SLO you must be able to prove.
The two questions to ask: (1) "What happens if a customer orders an item that was just removed — is that an acceptable failure at order time, or must the menu be authoritative at purchase?" — this decides whether the read path may be eventually consistent (cache-friendly) or whether the order path needs a synchronous validation against the source of truth (a completely different, and more expensive, design); (2)
"Is the one-minute bound a hard SLO or a comfort statement?" — because a hard bound requires monitoring, alerting, and a fallback path, while a soft one licenses simple TTL caching and nothing more. Note what the smell test produced before any boxes were drawn: a read/write asymmetry, a consistency budget, an outbox, a versioning rule, a staleness-visibility decision, and the two requirement ambiguities whose answers would change the architecture — which is exactly the opening a strong candidate delivers while a weaker one is already drawing load balancers.