Appearance
9.5.4 — Concurrency Patterns
Almost every concurrent system ever built is made of seven shapes. 9.5.2 gave you the primitives; this page gives you the arrangements those primitives are actually used in, in the order you are most likely to need them.
A useful thing to hold in mind as you read: every one of these is a queue wearing a different costume. A pool is a queue of threads waiting for a slot. An actor is a queue with a state owner attached. A pipeline is queues between stages. Once you see that, three questions apply to every shape here, and you should ask them of every arrow in every architecture diagram you ever draw:
What bounds this queue? What happens when it is full? Who can see how deep it is?
Most production concurrency incidents are one of those three left unanswered.
1. Signalling: waiting for a moment to arrive
The smallest pattern. One thread needs to wait until something has happened elsewhere. Three variants, distinguished by how many participants and how many times.
A latch opens once and stays open. Every waiter blocks until the count reaches zero, and then all of them proceed forever after. Use it for "wait until startup is complete" or "wait until all ten workers have finished".
typescript
class Latch {
#remaining: number;
#waiters: Array<() => void> = [];
constructor(count: number) { this.#remaining = count; }
countDown(): void {
if (--this.#remaining > 0) return; // (1) not there yet
for (const w of this.#waiters) w(); // (2) release EVERYONE, permanently
this.#waiters = [];
}
async wait(): Promise<void> {
if (this.#remaining <= 0) return; // (3) already open: never block
await new Promise<void>(r => this.#waiters.push(r));
}
}
const warm = new Latch(3); // cache, config and DB pool must all be ready
Promise.all([loadCache(), loadConfig(), openPool()]).then(() => { /* each calls warm.countDown() */ });
app.get("/health/ready", async (req, res) => {
await warm.wait(); // (4) requests park until the service is warm
res.send("ok");
});(1) Each completed piece of startup counts down. (2) When the last one lands, all waiters are released at once and the latch is spent — a latch never re-closes. (3) A waiter arriving after the gate opened must not block, which is what makes a latch safe to call from anywhere at any time. (4) The readiness endpoint becomes one line, and requests that arrive during warm-up wait rather than failing.
A barrier is a latch that resets. All N participants must arrive before any may continue, and then it rearms for the next round. Use it for work that proceeds in phases, where phase two cannot start until every worker has finished phase one.
A semaphore starting at zero is the general signal, as 9.5.2 described: whoever completes the work releases a permit, whoever needs it acquires one.
In JavaScript you will mostly not write these, because a promise already is a latch: it settles once, everyone awaiting it proceeds, and awaiting an already-settled promise returns immediately. Recognising that a promise is exactly a one-shot latch is worth more than memorising the class above.
2. Producer–consumer: the buffer between two rhythms
The problem. Work arrives at one speed and gets processed at another. Web requests burst; the workers that handle them grind along steadily. Neither side should have to wait for, or even know about, the other.
The shape. Put a queue between them. Producers append, consumers drain, and the queue absorbs the mismatch.
typescript
class BoundedQueue<T> {
#items: T[] = [];
#waitingConsumers: Array<(item: T) => void> = [];
#waitingProducers: Array<() => void> = [];
constructor(private readonly capacity: number) {}
async put(item: T): Promise<void> {
const starving = this.#waitingConsumers.shift();
if (starving) { starving(item); return; } // (1) hand straight over
while (this.#items.length >= this.capacity) { // (2) FULL: producer waits
await new Promise<void>(r => this.#waitingProducers.push(r));
}
this.#items.push(item);
}
async take(): Promise<T> {
if (this.#items.length > 0) {
const item = this.#items.shift()!;
this.#waitingProducers.shift()?.(); // (3) space freed: wake one
return item;
}
return new Promise<T>(r => this.#waitingConsumers.push(r)); // (4) EMPTY: consumer waits
}
get depth(): number { return this.#items.length; } // (5)
}(1) If a consumer is already waiting, skip the buffer entirely and hand the item over directly. This is not just an optimisation; it keeps the queue empty in the common case where consumers keep up, so depth stays a meaningful signal.
(2) The important line. When the queue is full, the producer waits. Note the while rather than an if, for the same reason as a condition variable in 9.5.2: by the time you are woken, another producer may already have taken the space.
(3) Removing an item frees space, so wake one waiting producer.
(4) An empty queue parks the consumer rather than spinning on it.
(5) Expose the depth. This is the single most useful health metric any queue-based system has, and a queue that does not expose it is a queue you cannot operate.
The design decision the pattern forces on you is what happens when the queue is full. There are exactly three answers and you must pick one deliberately:
| Policy | Right when | Cost |
|---|---|---|
| Block the producer | Producers can wait — internal pipelines | Backpressure travels upstream |
| Reject | At a trust boundary — HTTP 429 | Client must retry |
| Drop | Loss-tolerable data — metrics, logs | Silent data loss, so count it |
Blocking is what 3.8.4 calls backpressure: the slowness propagates back to whoever is producing too fast, which is exactly right when that producer is your own code reading a file. Rejecting is right when the producer is a stranger, because making a stranger wait ties up your resources on their behalf. Dropping is right only for data whose loss you can tolerate, and even then you must increment a counter, because silent loss is indistinguishable from working correctly.
Where you already use this: Node's event loop is a queue between the platform producing events and your callbacks consuming them (3.8.1). A stream's highWaterMark is exactly the capacity above. And when the queue needs to survive a process restart, it becomes a durable queue, at which point you are in 10.8.1 and the vocabulary carries over unchanged.
3. The worker pool: bounded concurrency as an object
The problem. You have five thousand tasks. Running all five thousand at once destroys something — the database's connection limit, a partner's rate limit, your own memory.
The shape. A counting semaphore with a nicer interface. Tasks queue for a slot; a fixed number run at a time.
typescript
class WorkerPool {
#active = 0;
#waiting: Array<() => void> = [];
constructor(private readonly size: number) {}
async run<T>(task: () => Promise<T>): Promise<T> {
if (this.#active >= this.size) { // (1) no slot free
await new Promise<void>(r => this.#waiting.push(r));
}
this.#active++;
try {
return await task();
} finally {
this.#active--;
this.#waiting.shift()?.(); // (2) pass the slot on
}
}
get stats() { return { active: this.#active, waiting: this.#waiting.length }; } // (3)
}
const enrichPool = new WorkerPool(10); // matches the DB pool size
const results = await Promise.all(
userIds.map(id => enrichPool.run(() => enrichUser(id))), // (4) 5,000 queued, 10 running
);(1) The queue-for-a-slot path, identical in structure to the semaphore.
(2) The finally is the line that matters most on this page. If a task throws and the slot is not returned, the pool permanently loses capacity. Lose all of them and the pool wedges shut with zero active tasks and an infinite queue, which is a genuinely baffling incident to debug because every metric says the service is idle.
(3) Expose active and waiting. Waiting climbing while active sits at the maximum is the signature of a pool that is too small or a task that is too slow, and you cannot see either without this.
(4) The call site is the payoff: map starts all five thousand promises, but only ten are ever inside task() at once. The rest are parked at line (1).
How to size it. Match the bounded resource that is actually scarce, and be able to name that resource. For database work, the connection pool size. For CPU-bound work, roughly the core count, since more threads than cores adds switching cost without adding capacity (9.5.1 section 3). For a partner API, whatever their documentation says. A pool sized by guesswork is a number nobody can defend when it needs changing.
And the trap worth repeating: this bounds concurrency, not rate. Ten slots with 5 ms tasks is two thousand calls per second. If the constraint is "50 requests per second", you need a token bucket alongside the pool, not a smaller pool.
4. Futures: coordinating results that do not exist yet
A future, spelled Promise in JavaScript, is a placeholder for a result that is still in flight. The mechanics are in 3.6.8. The pattern content is two rules.
Rule one: start before you await. Concurrency comes from launching the work, not from awaiting it. These two look almost identical and differ by a factor of the number of items:
typescript
// SEQUENTIAL — each await finishes before the next call starts
for (const id of ids) { results.push(await fetchUser(id)); }
// CONCURRENT — all calls launched, then all awaited together
const results = await Promise.all(ids.map(id => fetchUser(id))); The accidental version is one of the most common performance bugs in async code, and it is invisible in review unless you are looking for it.
Rule two: the combinator you choose is a statement about failure. This is the part people skip.
| Combinator | Settles when | Use for |
|---|---|---|
all | All succeed, or one fails | A batch that is meaningless if any part fails |
allSettled | Every one finishes | A batch where failures are reported, not fatal |
race | The first one settles | Timeouts |
any | The first one succeeds | Redundant sources, mirrors |
Picking all when the requirement was "failures should not stop the batch" is not a style preference; it is the requirement implemented incorrectly.
And the rule that ties this section to the last one: futures coordinate, they do not protect. Promise.all over five thousand tasks starts five thousand tasks. It provides no limit whatsoever. Coordination composes with bounding — ids.map(id => pool.run(() => work(id))) — it does not replace it.
5. The pipeline: stages joined by queues
The problem. Work goes through several steps, and the steps have different speeds and different resource needs. Parsing an upload is CPU work, validating it hits the database, and delivering the result is network work.
The shape. Give each stage its own worker pool, sized for what that stage needs, and join the stages with bounded queues.
Two properties make this worth the extra machinery.
Each stage gets the limit that fits it. Four parallel parses because you have four cores; a hundred concurrent deliveries because network calls are mostly waiting. A single pool for the whole job would have to be sized for the tightest constraint and would leave the other stages idle.
The bottleneck announces itself. Whichever inter-stage queue is full is the stage that cannot keep up. That is a diagnosis you get for free from a metric you were already exposing, and it beats profiling by a wide margin.
The cost is real and should be stated: more moving parts, more places for an item to be lost or stuck, and harder end-to-end tracing since one item's journey now spans several stages. Do not build a pipeline for a two-step job. Build it when the steps genuinely have different resource profiles, which is the condition that makes the extra structure pay.
6. One owner per entity: the actor
The problem. Shared mutable state plus concurrency is the whole horror show of 9.5.1. Locks manage it; they do not remove it.
The shape. Give each piece of state exactly one owner, and have everybody else send that owner a message. The owner processes messages one at a time. Now nothing is shared, so nothing can race — not because you were careful, but because there is no second writer to be careful about. This is the actor model.
typescript
type AccountMessage =
| { kind: "deposit"; amount: Money }
| { kind: "withdraw"; amount: Money; reply: (ok: boolean) => void }
| { kind: "balance"; reply: (b: Money) => void };
class AccountActor {
#balance = Money.zero(); // (1) private to this object
#mailbox = new BoundedQueue<AccountMessage>(100); // (2) bounded, like everything else
constructor() { void this.#loop(); } // (3) start draining immediately
send(msg: AccountMessage): Promise<void> {
return this.#mailbox.put(msg); // (4) the only way in
}
async #loop(): Promise<void> {
while (true) {
const msg = await this.#mailbox.take(); // (5) one at a time, forever
switch (msg.kind) {
case "deposit":
this.#balance = this.#balance.add(msg.amount);
break;
case "withdraw": {
const ok = this.#balance.gte(msg.amount); // (6) check and act, safely
if (ok) this.#balance = this.#balance.sub(msg.amount);
msg.reply(ok);
break;
}
case "balance":
msg.reply(this.#balance);
break;
}
}
}
}(1) The balance is private and there is no getter. No code outside this class can reach it.
(2) The mailbox is a bounded queue from section 2, which is the point: an actor is built from producer-consumer plus encapsulation, not a new idea.
(3) The drain loop starts with the object and runs for its lifetime.
(4) send is the entire public surface. Everything is a message.
(5) One message is fully handled before the next is taken. This is the guarantee that makes the rest work.
(6) Look closely at the withdraw case. It is a check-then-act, the exact shape that caused every bug in 9.5.1 — and here it is completely safe, with no lock anywhere, because there is no possible interleaving. Nothing else can touch #balance between the check and the subtraction. That is what the pattern buys you: the dangerous shape becomes safe by construction.
Where this already runs in your stack. Node's event loop is one big actor — a single thread taking one event at a time — and that is precisely why you have never needed a mutex for a plain JavaScript object (3.8.1). A worker_thread communicating by postMessage is a literal actor, with an isolated heap and message passing (3.8.6). Erlang and Elixir built entire languages around the model.
When to reach for an explicit one: when a single entity receives concurrent commands and the order matters per entity. A seat during a flash sale. A game room. A device session. An actor gives you per-entity serialisation without a global lock, so unrelated entities never wait for each other.
Two ways to get it wrong. Actorising something stateless, which is pure ceremony around a function call. And leaving the mailbox unbounded, which reintroduces the deferred out-of-memory crash the bound was there to prevent.
7. An order that moves through statuses
DrillCuriosity #7 (verbatim): What is the low level design pattern for pipeline kind of system, sequence order status driven events
There is no single Gang of Four pattern for this, and the honest answer is that it is a combination of three, which is why it feels hard to name. Anything shaped like an order goes placed → paid → packed → shipped → delivered, and something has to happen at each step — a loan application, an identity check, a provisioning workflow — is built the same way.
One: a state machine as the backbone. List the statuses. Attach the data each status requires. Write one transition(current, event) function that is the only thing allowed to move an order forward, and have it reject illegal moves loudly (9.4.14). A discriminated union with an exhaustive switch is usually the right spelling, because then the compiler tells you what you forgot.
Two: something that drives the machine and writes things down. On each event: check the transition is legal, persist the new status and the event together, then run whatever that status requires. Because the status is stored rather than held on a call stack, the process can restart, and the order simply resumes from where it was. A middleware chain or a call stack cannot do that, which is the reason this shape exists at all.
Three: split the per-status actions by what they are owed. Actions that must happen — charge the card, reserve the stock — run in the transaction or through a durable queue with retries. Actions that are merely nice — an email, an analytics event — fan out afterwards and are allowed to be lost. And each must-happen action pairs with a way to undo it, so when packing fails after payment succeeded, the order can walk backwards lawfully by refunding rather than being stuck. That backwards walk is the same idea as the Saga in 10.8.4, in one process.
The figure first, then the code that implements it.
typescript
type OrderPhase =
| { status: "placed"; items: Line[] }
| { status: "paid"; items: Line[]; paymentId: PaymentId }
| { status: "packed"; items: Line[]; paymentId: PaymentId; parcelId: ParcelId }
| { status: "shipped"; trackingId: TrackingId }
| { status: "cancelled"; reason: string; refunded: boolean };
class OrderWorkflow {
constructor(
private readonly store: OrderStore, // stored status + event log
private readonly actions: StatusActions, // the must-happen work
private readonly events: TypedEmitter<OrderEvents>,
) {}
async handle(orderId: OrderId, event: OrderEvent): Promise<OrderPhase> {
return this.store.withLock(orderId, async current => { // (1)
const next = transition(current, event); // (2)
await this.actions.run(current, next); // (3)
await this.store.commit(orderId, next, event); // (4)
this.events.emit(next.status, { orderId, phase: next });// (5)
return next;
});
}
}(1) One writer per order. Two pay events arriving together must not both charge the card. This is the actor idea from section 6 applied per entity, and in a real system it is a database row lock or a per-key queue rather than an in-memory mutex, for the replica reason from 9.5.2.
(2) Legality is decided in one pure function. transition reads the current phase and the event and returns the next phase or throws. It touches nothing else, so it is trivial to reason about and every illegal move is rejected in exactly one place.
(3) Must-happen actions run before anything is saved. If the charge fails, the exception propagates and nothing is committed, so the order stays placed and can be retried. It never becomes falsely paid. Ordering these two lines the other way round is a genuine money bug.
(4) The status and the event are written together, atomically. This is what makes the audit log trustworthy: it cannot disagree with the current status, because one write produced both. It is also what lets the workflow park for three days between paid and packed and pick up exactly where it left off.
(5) The optional reactions fire only after the truth is durable. A lost email is annoying; an email announcing a payment that was never committed is a support incident. And if something in this group turns out to be mandatory after all, it moves up to line (3) or onto a durable queue — the classification is the design decision, and the code position follows from it.
What to do when a later step fails. Packing fails after payment succeeded. The pack action's failure path triggers the undo for the paid step — refund through the payment port — and moves the order to cancelled { refunded: true }. Note that walking backwards goes through the same gate as walking forwards, so it is validated and logged like everything else, rather than being a special path that nobody tests.
Recognise this shape, build it, and defend those five numbered lines, and you have covered order systems, payment flows, document approvals, provisioning, and most of what an interviewer means by "design the order lifecycle".
8. Choosing
| Situation | Shape |
|---|---|
| Rates differ between two sides | Bounded queue |
| Too many tasks for a scarce resource | Worker pool |
| Several results needed together | Futures + combinator |
| Steps with different resource needs | Pipeline of stages |
| One entity, concurrent commands, order matters | One owner per entity |
| Something must finish before something starts | Latch or barrier |
| Long-running process with statuses | State machine + one gate + actions by guarantee |
And the classification that produces the right row every time. For each piece of work, ask: does it need to be serialised per entity? (one owner) Must it definitely happen? (transaction or durable queue, plus an undo) Is losing it acceptable? (fire and forget) Does it need bounding? (pool). Teams that skip this classification end up at one of two symmetric failures — everything transactional, which is slow and tangled, or everything fire-and-forget, which is fast and silently wrong.
Next: 9.5.5 puts every shape on this page to work on the concurrency problems interviewers actually set.
Recall
- Every shape here is a queue in costume. Ask of each one: what bounds it, what happens when it is full, who can see its depth.
- Producer–consumer: a bounded queue between two rhythms. Full-queue policy is a real decision — block (backpressure), reject (
429), or drop (and count it). Unbounded means a crash you have deferred, not avoided. - Worker pool: a semaphore with a nicer interface. Release the slot in
finallyor the pool silently loses capacity. Size it to the scarce resource and be able to name that resource. Bounds concurrency, not rate. - Futures: start before you await, or you have written a sequential loop. The combinator you pick is your failure policy —
allfails fast,allSettledreports everything,racetimes out,anytakes the first success. Futures coordinate; pools protect. - Pipeline: a pool per stage, sized for that stage's resource, joined by bounded queues. The full queue names your bottleneck for free.
- Actor: private state + a bounded mailbox + one message at a time. Check-then-act becomes safe with no lock, because no interleaving exists. Node's loop and
worker_threadsalready are actors. - Order-through-statuses: a state machine, one gate that validates and persists status plus event atomically, and actions split by what they are owed — must-happen before the save, optional after, with an undo for each must-happen.
Self-test: Name the three full-queue policies and when each is right. What does the pool's finally prevent, and what does the failure look like? Why is check-then-act safe inside an actor? Give the five numbered guarantees of the order workflow. Which combinator means "failures must not stop the batch"?
Quiz Bank
FoundationalBuild producer-consumer from first principles, and explain why the bound is the important part.
The parts. Producers that append work, consumers that drain it, and a queue between them that absorbs the difference in their speeds. Plus two waiting lists: consumers parked when the queue is empty, and producers parked when it is full. That is exactly the two-semaphore construction from 2.4 — one counting empty slots, one counting filled ones — spelled in Node with parked promise resolvers instead, because a single thread makes each individual operation atomic without needing a lock.
Why the bound is the design. An unbounded queue looks simpler and is actually the same design with one question left unanswered: what should happen when work arrives faster than it can be processed? The unbounded version answers "accumulate", which means memory grows until the process dies. Worse, it grows silently — every metric looks fine, latency creeps up slowly as the queue lengthens, and the failure arrives all at once with no warning.
Putting a bound on it forces you to answer the question at design time, and there are only three possible answers.
Block the producer. Correct when the producer is your own code and can safely wait — reading a file, processing a batch. The slowness travels back upstream to whoever is going too fast, which is the definition of backpressure (3.8.4).
Reject. Correct at a trust boundary, where the producer is somebody else. Return 429 with a Retry-After header. Making a stranger wait means holding your own connection and memory on their behalf, which is how one abusive client takes down a service for everyone.
Drop. Correct only when losing the data is genuinely acceptable — sampled metrics, debug logs. And you must count the drops, because silent loss looks exactly like healthy operation on every dashboard.
The corollary worth stating: queue depth is the leading health indicator of any producer-consumer system, and its rate of change is even better. Depth tells you the current state; a rising slope tells you what the state will be in five minutes, which is when you would still have time to do something about it.
FoundationalWhat is the actor model, why are data races impossible inside one, and where is it already running in your stack?
An actor is an object that owns private state, receives messages through a mailbox, and processes them strictly one at a time. Its state is touched only by itself, only between messages.
Why races become impossible. A race requires two writers reaching the same state concurrently. Inside an actor there is only ever one writer, and it is never interrupted mid-message. So the interleaving that a race depends on cannot be expressed. Concurrency still exists in the system — many actors drain their mailboxes at the same time — but it exists between actors, never inside one.
The clearest demonstration is that check-then-act, which is the shape behind nearly every concurrency bug, becomes completely safe inside an actor with no lock anywhere. if (balance >= amount) balance -= amount is a bug in shared-memory code and correct in an actor, and the difference is not carefulness but structure.
Where it already runs. Node's event loop is one large actor: a single thread pulling one event at a time off a queue. That is the actual reason you have never written a mutex for a plain JavaScript object (3.8.1). A worker_thread talking over postMessage is a textbook actor, with an isolated heap and copied messages (3.8.6). Erlang and Elixir built whole languages on the model, and their reliability reputation comes largely from it.
When to build one explicitly. When a single entity receives concurrent commands and per-entity ordering matters: a seat during a flash sale, a game room, a connected device. You get serialisation for that one entity without a global lock, so unrelated entities never wait for each other. That property — fine-grained serialisation with no lock ordering to get wrong — is the actor's real selling point over a mutex.
Two disciplines. Bound the mailbox, or you have rebuilt the unbounded-queue crash. And do not actorise stateless things; an actor around a pure function is ceremony with a queue attached.
AppliedA batch job must call a partner API for 10,000 items. The partner allows 20 concurrent requests. Failures must not stop the batch, and you need a report of what succeeded and what failed. Compose the patterns from this page.
Three requirements, three tools, and the interesting part is how they compose rather than any one of them.
Bounding: a worker pool of 20. Wrap each call so that ten thousand tasks exist but exactly twenty are in flight. Without this, Promise.all over ten thousand items opens ten thousand sockets at once, and you will exhaust file descriptors, memory or the partner's patience — usually all three.
typescript
const partnerPool = new WorkerPool(20);
const settled = await Promise.allSettled(
items.map(item => partnerPool.run(() => callPartner(item))),
);Failure policy: allSettled, not all. This is the requirement "failures must not stop the batch", written as a combinator. Promise.all rejects the moment any single call fails, abandoning the other nine thousand, which is the opposite of what was asked for. Choosing the combinator is choosing the failure semantics, and getting it wrong here is not a style issue but an unimplemented requirement.
The report: fold the settled results. Partition into fulfilled and rejected, and keep the item alongside each rejection so the failure report doubles as the input to a retry pass:
typescript
const failed = settled.flatMap((r, i) =>
r.status === "rejected" ? [{ item: items[i], reason: r.reason }] : []);Three sharp edges worth naming before the interviewer does.
Start before you await. The map must launch the calls and collect promises. A for loop with an await inside it silently serialises everything and the pool becomes irrelevant.
The pool's finally. If a throwing task fails to release its slot, the pool loses capacity permanently, and after twenty failures the batch stops with no error and no activity — the worst kind of incident to diagnose.
Concurrency is not rate. Twenty concurrent calls at 50 ms each is four hundred requests per second. If the partner's actual limit is "100 per second", the pool alone will get you throttled or banned, and you need a token bucket in front of it (9.7.5). Systems commonly need both limits, expressed separately.
One more thing a strong answer adds: retries belong inside the pooled task, not around the whole batch, so a retry consumes a slot like any other work and cannot cause a burst above twenty. And they need jittered backoff (9.5.3) so ten thousand items do not synchronise their second attempt.
InterviewDesign the low-level structure for an order moving placed to paid to packed to shipped, where the card must never be double-charged, emails are best effort, and a packing failure must refund. Name every guarantee and the mechanism that provides it.
The shape is section 7's, and the answer is best given guarantee by guarantee, because that is how the design was derived.
The backbone. OrderPhase as a discriminated union, one variant per status, each carrying only the data that status actually has — a shipped order has a tracking number, a placed one does not, and the type makes that impossible to get wrong. One pure transition(current, event) function decides legality, so ship before paid is rejected in exactly one place, and an exhaustive switch means the compiler lists what you forgot when a status is added.
No double charge, guarantee one: one writer per order. Every event for a given order goes through a single serialisation point — a database row lock, or a per-key queue, which is the actor idea applied per entity. Two pay events arriving in the same second are then processed one after the other, and the second one finds the order already paid and is rejected by the transition function rather than charging again.
No double charge, guarantee two: an idempotency key on the charge itself. Serialisation alone is not enough, because of the crash window: if the process dies between the successful charge and the commit, a retry after restart sees placed and charges again. The charge therefore carries a key derived from the order id and the transition, so the payment provider recognises the repeat and returns the original result instead of taking more money. Naming this window unprompted is the difference between a good answer and an excellent one (10.4 scales the same idea).
Ordering of effects. Must-happen actions run before the commit. A failed charge therefore leaves the order placed and retryable, and never produces a record saying paid for money that was never taken. The status and the event are then written as one atomic operation, so the audit log physically cannot disagree with the current status.
Best-effort email: fired after the commit, and allowed to fail. The reason email is safely lossy and charging is not comes down to consequences — a lost email costs a support ticket, a lost charge costs money and trust. Saying why the classification differs, rather than just stating it, is the core of this answer. If the business later decides the email is mandatory, it moves onto a durable queue with retries, and note that the code change follows the classification change rather than the other way round.
Packing failure must refund. The pack action's failure path invokes the undo for the paid step — a refund through the payment port — and transitions the order to cancelled { refunded: true }. Two details make it robust: the refund is itself idempotent, keyed the same way as the charge, so a retry of the compensation cannot refund twice; and the backwards move goes through the same gate as every forwards move, so it is validated and logged identically instead of being a side path nobody exercises.
Close by naming the composite: a state machine, one coordinator that owns the writes, and actions classified by what they are owed. The distributed version of exactly this is the Saga (10.8.4), and the only thing that changes is that the participants are now separate services and the coordinator has to survive their partial failures.
StaffA ticketing system oversells during flash sales. Two hundred requests hit buy-the-last-seat in the same second and the check-then-decrement races. The team proposes a global mutex. Give the full design conversation.
First, why the race exists at all in single-threaded Node, because the team may not believe it does. The check and the decrement are separated by awaits: read the seat count, compare it, write the new count. Node interleaves other requests' continuations into those gaps, so two hundred handlers can all read seats: 1 before any of them writes 0 (9.5.1). Single-threaded guarantees no two lines run simultaneously; it guarantees nothing about the world holding still across an await. No threads are required for this bug.
Second, why the global mutex is wrong — and it is wrong in three separate ways.
It is wrong in scope. One mutex for all purchases means a hot seat for one concert throttles ticket sales for every other event in the system. A single popular show would cap the entire platform's throughput.
It is wrong in reach. The mutex lives in one process's memory. Run the service on four instances, which any system under flash-sale load certainly does, and there are four mutexes, so four requests can be inside the critical section at once. The oversell returns exactly when traffic is highest, which is the worst possible failure mode: correct in staging, broken in production.
It is wrong in layer. It protects only the code paths that remember to take it. An admin tool, a bulk import, or a new endpoint written next quarter will walk straight past it, and the invariant everyone believes is enforced turns out to be enforced only by convention.
Third, the layered answer, from the inside out.
The invariant lives with the state. UPDATE seats SET remaining = remaining - 1 WHERE event_id = ? AND remaining > 0. The check is the WHERE clause, the act is the UPDATE, and the database performs both indivisibly. Zero rows affected means you lost, and that is your "sold out" response. Overselling is now unrepresentable regardless of who races, how many instances exist, or which tool is doing the writing. This one change fixes correctness completely, and everything after it is about experience and load rather than about being right.
Per-entity serialisation, for fairness and for user experience. An owner per hot event — an in-process per-key queue on one node, a single-partition consumer or a per-key distributed lock at scale — turns a stampede into an ordered line. That lets you show a queue position and give each buyer a definite yes or no, instead of two hundred people simultaneously receiving an optimistic failure. Note that this is layered on top of a correctness guarantee that already holds without it, which is what makes it optional and tunable rather than something the design depends on.
Bounding at the edge. A limiter and a waiting-room page mean the two hundred simultaneous database round trips never happen at all. This is capacity management, and it is what actually keeps the site up during a sale; the full funnel is built in the flash-sale study, Chapter 11.16.
The sentence that reframes the team's mental model, and the one to end on: concurrency control belongs at the state, scoped to the entity, and sized at the edge. A global mutex makes all three of those decisions, and gets all three wrong at once.
Scenario Drill
DrillDesign the ingestion side of a webhook platform. Partners POST events at up to 50,000 per minute in bursts. Each partner's events must be delivered in order. Delivery calls customer endpoints that are slow and flaky, with timeouts up to 30 seconds. No event may be lost, and the operations team needs to see the backlog per partner. Compose this page end to end, and say what you are deferring to Part 10.
Map each requirement sentence onto a shape before writing any code. That mapping is the design, and doing it out loud is what an interviewer is grading.
Bursty arrival against slow processing: producer–consumer. The HTTP handler does the smallest trustworthy thing and no more — authenticate, validate the shape, persist the event, return 202 Accepted. Everything else happens behind a queue. Doing delivery inline would tie a partner's POST to a customer's 30-second timeout, and 50,000 of those per minute is an immediate collapse.
Because nothing may be lost, the queue must be durable rather than the in-memory one from section 2. A database table or a Redis stream now; a real broker later (10.8.1). This is the one place where the in-process pattern genuinely cannot satisfy the requirement, and saying so explicitly is better than silently upgrading it.
The full-queue policy belongs at the edge: once a partner's backlog crosses a threshold, return 429 with Retry-After for that partner. Partners retry — that is what their client libraries are built to do — whereas unbounded intake gives you a backlog that can never drain and eventually an out-of-memory crash. Note this is per partner, so one partner flooding you does not cause rejections for everyone else.
Per-partner ordering: one owner per key. Exactly one logical consumer per partner, so that partner's events are processed strictly in sequence. Within one node that is a per-key queue; across nodes it is partitioning by partner id so that one partition has one ordered consumer, which is the actor idea graduating into infrastructure.
Say the other half out loud too: ordering between partners is deliberately not promised, and this should be written in the partner documentation. Unrequested global ordering is the classic self-inflicted bottleneck, because it forces every event in the system through a single serialisation point.
Slow and flaky delivery: a bounded pool, plus per-destination isolation. A delivery pool bounds total in-flight customer calls, because 30-second timeouts multiplied by unbounded concurrency exhausts sockets and memory. But a single shared pool has a failure mode worth naming: one customer whose endpoint hangs will slowly occupy every slot, and deliveries to healthy customers stop. So the pool needs per-destination sub-limits — a small concurrency budget per customer — so a dead endpoint can only consume its own share. That is a bulkhead (10.9), and adding a circuit breaker per destination on top means you stop calling an endpoint you already know is failing, which both protects your pool and gives the customer room to recover.
Retries use exponential backoff with jitter (9.5.3), and after N attempts the event moves to a parked state per partner rather than being deleted — the no-loss requirement again — with replay as an ordinary, logged operation rather than a database script somebody runs by hand.
Each event is a small state machine. received → queued → delivering(attempt n) → delivered | parked, one writer per event, with the status and each attempt written atomically. This is section 7 in miniature, and it pays for itself immediately: "did customer X receive event Y, and when?" becomes a query instead of a log search, replay becomes a legal transition instead of a hack, and the audit trail exists without anyone building one.
Operational visibility: the three questions, turned into metrics. Backlog depth per partner. Delivery pool active and waiting. Age of the oldest undelivered event, which is the metric that actually correlates with customer pain. Park rate per destination. And alert on the slope of the backlog rather than its level, because level tells you where you are and slope tells you where you will be in ten minutes, which is while you can still act.
Deferred to Part 10, explicitly. Exactly-once delivery is not on offer: customers will see occasional duplicates, so every delivery carries a stable event id and the documentation tells customers to deduplicate on it — the honest contract is at-least-once (10.4). Rebalancing partitions when a node dies, so a partner's ordered consumer moves without two consumers briefly overlapping, is a broker concern (10.8.2). And the persist-then-enqueue step is only atomic here because both live in one database transaction; when the queue becomes a separate system, that atomicity needs the outbox pattern (10.8.4).
The lesson to end on: every requirement sentence named one shape. Bursty means queue. Ordered means one owner per key. Flaky means pool plus breaker plus a parking area. No loss means durable and parked rather than dropped. Visibility means depth and age. The low-level design is that mapping, written down together with the guarantee each piece provides.