10.1 — Why Distributed, and the Senior Smell Test

Part 9 designed systems that live inside one process, where a function call either returns or throws, memory is shared, and "now" is unambiguous. Part 10 removes all three guarantees. The moment work spans machines, calls can hang forever, partial failure becomes the normal case, clocks disagree, messages duplicate and reorder, and the same operation may execute twice with nobody able to prove it. This page frames why anyone accepts that trade, catalogs the arsenal of concerns the rest of the Part develops, and — most valuable for interviews and for real design reviews — teaches the smell test: how seniors recognize which distributed concerns a problem carries, in seconds, instead of reciting a checklist. ^{⚑Senior engineer system design arsenal. [EQ-235b]}

1. Three reasons to distribute — and one bad one

Distribution is a cost you pay to buy something specific. The legitimate purchases:

• Scale beyond one machine — traffic, data, or compute exceeding the biggest box you can rent (and vertical scaling has a real ceiling and a price cliff — 10.2).
• Availability through redundancy — one machine dies, others serve; one region burns, another region answers. Single machines have single-digit-nines availability whatever you spend.
• Latency by locality — users in Sydney shouldn't wait for Virginia; data near its consumers is the CDN's entire business (Part 13.4).

The bad reason, stated plainly because it drives most unnecessary microservice migrations: organizational aspiration. "We should be microservices" without a scaling, availability, or team-autonomy problem buys you every cost on this page — network failure modes, distributed debugging, eventual consistency, deployment coordination — in exchange for a diagram that looks modern. The honest test: name the specific limit you're hitting. If you can't, the distributed version is strictly worse than the monolith you have, and Part 11's case studies will show you that even at genuine scale, the number of moving pieces is a cost the best designs actively minimize.

The costs, which you pay from day one and cannot avoid:

• The network is unreliable and slow, and it can split in two. Peter Deutsch and James Gosling wrote down the false beliefs about networks in 1994, and they still describe every incident review. They are called the fallacies of distributed computing. The network is not reliable. Latency is not zero. Bandwidth is not infinite. The layout of the network does change. And there is no single person in charge of all of it.
• Partial failure. Things no longer simply work or fail. Component B can be down for A while it is up for C.
• There is no single "now" that all machines agree on (10.3).
• Coordination costs time. Every promise of consistency is a network round trip you must pay for.
• Debugging turns into detective work, unless you build for it from the start (10.10).

2. The arsenal: the fourteen concerns

Every distributed design decision falls into one of these categories. This is the map of Part 10 and the checklist a design review runs — but section 3 explains why you should not recite it linearly:

#	Concern	The question it asks	Where it's built
1	Communication pattern	sync request/response, async messaging, or streaming?	5.8, 10.8.1
2	Execution model	inline, background job, scheduled, or event-driven?	10.8.1
3	State ownership	who is the source of truth for this fact?	10.4
4	Consistency guarantee	strong, causal, read-your-writes, or eventual?	10.7.1
5	Delivery semantics	at-most-once, at-least-once, effectively-once?	10.4
6	Ordering	global, per-key, or none?	10.3, 10.6
7	Failure handling	timeout, retry, circuit break, degrade, compensate?	10.9
8	Coordination	leader election, locks, consensus — or avoidable?	10.7.2
9	Scaling axis	more replicas, more partitions, or more caching?	10.2, 10.6
10	Observability	how will we see this fail?	10.10
11	API contract	versioning, compatibility, idempotency keys	9.6, 10.11
12	Storage pattern	replicated, partitioned, cached, denormalized?	10.5, 10.6
13	Security boundary	service-to-service auth, trust zones	Part 8.6
14	Evolution & cost	rollout, migration, and the bill	10.11, 10.12

3. The smell test: how seniors actually work

Juniors walk the checklist. Seniors smell the shape of the problem and jump to the three or four concerns that matter, because certain requirement phrasings imply their machinery. The recognition table — memorize the triggers, not the list:

When you hear…	Smell	Reach for
"must not lose it" / money / legal	durability + delivery semantics	durable queue, outbox, idempotency keys (10.4, 10.8.4)
"should be instant for the user"	async + eventual consistency	accept work, return 202, do it in background (10.8.1)
"two users can do X at once"	contention	per-entity serialization, conditional writes (9.5.4, 10.7.1)
"the other team's service"	partial failure	timeout + retry + circuit breaker + fallback (10.9)
"it retried and charged twice"	duplicate delivery	idempotency keys, dedupe windows (10.4)
"we need it to be fast globally"	locality	CDN, read replicas, regional writes (10.2, 10.5)
"the table is too big"	partitioning	shard key choice, hot-spot analysis (10.6)
"one customer's spike broke everyone"	isolation	bulkheads, quotas, per-tenant limits (10.9)
"in the right order"	ordering	per-key partitioning, sequence numbers (10.3, 10.6)
"eventually both systems agree"	reconciliation	outbox, idempotent sync, drift detection (10.8.4)

The senior instinct fits in four words, and interviewers listen for it: "async, at-least-once, idempotent, observable". It covers a surprising share of real designs, because most requirements break down into four things:

• do it in the background;
• accept that messages may arrive more than once;
• make repeats harmless;
• make sure you can see what is happening.

A candidate who says this without being asked has run real systems, not just read about them.

Figure 1 — The smell test. Senior designers don't walk the fourteen-concern checklist linearly; they hear a requirement phrasing and jump directly to the machinery it implies, then verify with the checklist afterward.

4. The expert lens

Distribution is a liability you take on for a named asset. Every service boundary adds a network hop (latency, failure mode), a deployment coordination point, a data-consistency question, and a debugging seam. The best distributed systems are the ones that distribute as little as possible: a well-factored monolith with a queue for background work handles more traffic than most teams will ever see, and Part 11's designs earn each split by pointing at a specific limit. When a design review can't name what a boundary buys, the boundary is a cost with no purchase.

Everything in Part 10 is Part 9's ideas plus partial failure. The queue is 9.5.4's producer-consumer with durability; the saga is 9.4.15's command-with-compensation across machines; the circuit breaker is a state machine (9.4.14); idempotency keys are 9.6.3's HTTP mechanism generalized. That continuity is deliberate and it's why this book taught LLD first: you already own the shapes; Part 10 teaches what breaks when the arrows cross machines.

Design reviews are won by naming trade-offs, not by naming technologies. "We'll use Kafka" is not a design; "these events must survive consumer downtime, be replayable for a new consumer, and preserve per-customer order — which is why a partitioned log rather than a queue" is. Every chapter ahead ends in a decision ledger style — alternatives, choice, cost — because that's what distinguishes an architect from a catalog.

Next: 10.2 — the first arsenal entry in depth: how systems actually scale, why stateless is the magic word, load balancing at the system level, and where CDNs fit.

Recall

• Distribute to buy scale beyond one machine, availability through redundancy, or latency by locality — never for organizational fashion. If you can't name the limit you're hitting, the monolith wins.
• The costs are structural: the fallacies of distributed computing (unreliable network, non-zero latency, finite bandwidth, changing topology, no single admin), partial failure, no global "now", coordination-as-latency, and archaeology-grade debugging.
• The arsenal (14 concerns): communication pattern · execution model · state ownership · consistency · delivery semantics · ordering · failure handling · coordination · scaling axis · observability · API contract · storage pattern · security boundary · evolution & cost.
• The smell test. The wording of a requirement points straight at the machinery:
  • "Must not lose it": a durable queue, an outbox and idempotency.
  • "Instant for the user": return 202 and do the work in the background.
  • "Two at once": handle one entity at a time, with conditional writes.
  • "Flaky partner": timeout, retry, circuit breaker and fallback.
  • "Too big a table": partitioning.
  • "One tenant broke everyone": bulkheads.
  • The instinct in four words: async, at-least-once, idempotent, observable.
• Continuity: Part 10 = Part 9's shapes (producer-consumer, command+compensation, state machines, idempotency) plus partial failure. Reviews are won by naming trade-offs, not technologies.

Self-test: Give the three legitimate reasons to distribute and the honest test for the bad one. Recite five fallacies. Name the machinery for: "we can't lose payments", "search must feel instant", "two admins edit one record". What four words compress the senior instinct?

Quiz Bank

FoundationalWhen should a system be distributed, and what does it cost?

Spread a system across machines only to buy one of three things:

1. Scale beyond one machine's limit, in traffic, data or computing power. Note that a bigger machine is a real option, and often cheaper up to a point (10.2).
2. Availability through spare copies. Any single machine can only be so reliable, however much you spend on it.
3. Lower latency by being close to users.

The bad reason is fashion: adopting microservices without naming the limit you hit. That buys every cost and no benefit.

The costs, all unavoidable:

• The fallacies of distributed computing: the network is unreliable, latency is never zero, bandwidth is limited, the network's layout changes, and many people run parts of it.
• Partial failure, the defining difference. A component can be up for one caller and down for another at the same moment. And after a timeout you cannot tell "not done" from "done, but the reply was lost".
• No shared clock (10.3).
• Coordination, paid for in round trips.
• Visibility into the system, which you have to build, not assume (10.10).

The test in any design review: name the specific limit you are hitting. If there is none, the simpler layout is always better.

AppliedA product manager says: 'When a customer places an order, send the confirmation email, update inventory, notify the warehouse, and refresh their loyalty points — and the order page must respond instantly.' Run the smell test.

Four smells in one sentence. "Must respond instantly" + four downstream effects → async execution model: accept the order, persist it, return 202/201 immediately, and do the rest in the background (10.8.1) — putting four network calls in the request path makes latency the sum and availability the product of all four.

"Inventory" and "loyalty points" point at who owns the data, and how messages are delivered. These are changes to other systems' data that must happen. So they need durable messaging, delivered at least once, and consumers that are safe to run twice. A retried stock reduction must not reduce stock twice (10.4). Also, saving the order and publishing the event must never get out of step. That is the transactional outbox (10.8.4).

"Email" → loss-tolerable-ish but user-visible: same queue, lower priority, with retry and a dead-letter path (10.8.1). "Warehouse" → a third-party or separate-team system → failure handling: timeout, retry with backoff, circuit breaker, and a degradation story (queue and alert rather than fail the order — 10.9). Then the consistency conversation the PM didn't have: inventory becomes eventually consistent with the order, so oversell is possible in the window — either accept it with reconciliation, or reserve inventory synchronously before accepting the order (a deliberate latency-vs-correctness trade to put in front of the business). That last move — surfacing the trade rather than silently choosing — is the senior behavior the exercise trains.

InterviewWhat is partial failure, and why does it change design more than any other distributed property?

In a single process, a function call has two outcomes: it returns or it throws, and either way you know. Across a network there is a third: it times out, and you cannot distinguish "the request never arrived" from "it executed and the reply was lost" (2.7's lost-reply problem). That ambiguity propagates into every layer of design:

• Retries become dangerous unless operations are safe to repeat, which is why idempotency keys exist (10.4).
• Failure is no longer the same for everyone. Service B may be reachable from C but not from A. That is a network split, not a crash. So "is it down?" has no single answer, and each caller must track health for itself (9.7.6).
• Cascading failure becomes the most common kind of outage. A's retries pile extra load onto B while B is already overloaded (10.9).
• Consistency needs coordination, and during a network split that costs time and availability (CAP, 10.7.1).

It changes design more than scale or latency do, because it breaks the way you reason. You can no longer assume that what a call did matches what it returned. Every mechanism in Part 10, from timeouts and retries to idempotency, sagas, consensus and reconciliation, exists to win back some useful certainty on top of that doubt.

StaffA director proposes splitting your 200k-line monolith into 30 microservices to 'move faster'. Deployments currently take 40 minutes and three teams contend for releases. Give the analysis and the counter-proposal.

Separate the symptom from the diagnosis. The stated pain is delivery coupling — three teams serializing on one release train — not scale, availability, or latency. Microservices can address delivery coupling (independent deploys), but they do it by adding network failure modes, distributed data consistency, and operational surface: paying a distributed-systems tax to buy an organizational property. Before accepting that trade, exhaust the cheaper purchases of the same property:

1. A modular monolith. Enforce module boundaries inside one process: clear owners, no module reading another module's tables, and interfaces as the contracts between them (9.9.6). That gives most of the independence, with none of the network.
2. Fix the deploy pipeline. A 40-minute deploy is usually a slow build, not an architecture problem. Halve it, and teams wait on each other half as much, without changing the architecture.
3. Trunk-based development with feature flags. This separates releasing a feature from deploying code, which is usually what "move faster" really means.
4. Pull out the two or three parts that are truly independent, because they scale differently or change at a different pace. Typical candidates are background job processing, a public API with different service targets, and a heavy computing pipeline. Pull out each one gradually, and name the limit that justifies it (10.11).

The counter-proposal is a plan in stages. In the first quarter, do (1), (2) and (3), and measure how often you deploy, how long a change takes to reach production, and how often a change breaks something. In the second quarter, split out services only where a metric says so. Each split needs a written reason, naming which of the three good reasons it buys. It also needs the operational basics in place first: tracing across services, a service target for each service, and an on-call rotation that can debug across the new boundary (10.10).

State the risk plainly: 30 services with 3 teams is a staffing mismatch — the well-known failure where the distributed system's coordination cost lands on the same humans who were already the bottleneck, and delivery gets slower, not faster.

Flashcards

FlashThree reasons to distribute

Scale past one machine · availability via redundancy · latency via locality. Not "we should be microservices" — name the limit or don't split.

FlashFallacies of distributed computing

Network is reliable · latency is zero · bandwidth infinite · topology static · one administrator (+ transport cost zero, network homogeneous). Every incident review revisits one.

FlashPartial failure

Timeout ≠ failure: can't distinguish never-arrived from done-reply-lost. Invalidates the call/return reasoning model — hence idempotency, retries, sagas, reconciliation.

FlashThe senior instinct in four words

Async + at-least-once + idempotent + observable — covers a startling fraction of real designs.

FlashSmell test samples

"Must not lose" → durable queue + outbox + idempotency · "instant" → 202 + background · "two at once" → per-entity serialization + CAS · "flaky partner" → timeout/retry/breaker.

Scenario Drill

DrillInterview opener: 'Design a system that lets restaurants update their menus, and shows updated menus to millions of customers within a minute.' Before drawing anything, run the full smell test aloud and produce the concern list that will structure your design — then state the two questions you'd ask the interviewer.

Smell test, phrase by phrase. "Restaurants update menus" — a write path with a clear state owner (the restaurant's menu; concern 3): low volume, high correctness, needs validation and an audit trail. "Millions of customers" plus "shows menus" means a read path ten thousand to a million times bigger than the write path. That imbalance is the heart of the design. It points to caching and copies (concerns 9 and 12): read replicas, or better, a menu document with a version number, served through a CDN. A menu is a document that rarely changes and is read all the time (10.2).

"Within a minute" — an explicit consistency budget (concern 4): not strong consistency, not "eventually" — a bounded staleness SLO, which is a gift: it licenses aggressive caching with a ≤60 s TTL, or event-driven invalidation with a TTL backstop (10.5).

"Updates propagate" — delivery semantics (concern 5): publish menu-changed events, at-least-once, with consumers idempotent on (menuId, version) so duplicates are harmless (10.4); the write and the publish must not diverge → outbox (10.8.4).

Implied, but not said out loud:

• Ordering (concern 6). Two quick edits must not arrive in the wrong order. So each event carries a version number that only goes up, and consumers ignore anything older than what they have. The newest version wins, not the newest arrival. That is cheap, and correct here.
• Failure handling (concern 7). If updates are slow to spread, customers see an old menu. That is a degraded service, not an outage. The design should show that the data is old, with a lastUpdated time, instead of pretending it is fresh.
• Observability (concern 10). Track how long updates take to spread as a main metric, because "within a minute" is a target you must be able to prove.

The two questions to ask:

(1) "What happens if a customer orders an item that was just removed? Is it fine to reject it when they place the order, or must the menu be exactly right at the moment they buy?" The answer decides the design. If rejecting at order time is fine, the menu can be read from caches that are slightly behind. If not, every order needs a live check against the main database, which is a completely different and more expensive design.

(2)

"Is the one-minute bound a hard SLO or a comfort statement?" — because a hard bound requires monitoring, alerting, and a fallback path, while a soft one licenses simple TTL caching and nothing more. Note what the smell test produced before any boxes were drawn: a read/write asymmetry, a consistency budget, an outbox, a versioning rule, a staleness-visibility decision, and the two requirement ambiguities whose answers would change the architecture — which is exactly the opening a strong candidate delivers while a weaker one is already drawing load balancers.