9.7.19 — A Publish-Subscribe Broker

"Design a publish-subscribe system. Publishers send messages to topics, subscribers receive them, and a message must not be lost if a subscriber is slow or restarts."

There is exactly one decision in this problem, and everything else follows from it: who remembers how far each subscriber has got.

Answer "the broker does, by holding a queue per subscriber" and you get one design. Answer "the subscriber does, by holding a position in a shared log" and you get a different one, with different memory behaviour, different failure behaviour, and replay for free. Candidates who start with classes get lost. Candidates who start with that question have the design in ninety seconds.

1. The two shapes

Figure 1 — The whole design decision. Copying messages into a queue per subscriber, or keeping one log and letting each subscriber remember a position. The second stores each message once, makes replay free, and turns a slow subscriber into an integer that is behind.

Shape A — a queue per subscriber. When a message is published, the broker copies it into a queue for each subscriber and deletes each copy once that subscriber acknowledges it.

What it is good at: independence. A slow subscriber grows only its own queue.

What it costs: memory is messages × subscribers, and a delivered message is gone, so replaying yesterday is not possible at any price. Adding a new subscriber gives it nothing from the past.

Shape B — one log per topic, one offset per subscriber. The message is appended once. Each subscriber records a number saying how far it has read. Delivery is "give me what comes after 2,341".

What it is good at: one copy of the data, replay by moving a number backwards, a new subscriber can start from the beginning, and a slow subscriber costs nothing but an integer being small.

What it costs: messages must be retained on a policy — by time or size — rather than being removed when consumed, and a subscriber that falls further behind than the retention window loses data. That is a real cost and naming it is part of the answer.

Choose B and say why. It is the design behind every modern messaging system, and the reason is the property that sounds least important: the log does not change when it is read. Reading is a pure function of a position, so nothing about consumption mutates shared state, which removes most of the concurrency problems before they start.

2. The model

interface Message {
  offset: bigint;                     // (1) assigned by the log, never by the publisher
  key: string | null;                 // (2) decides ordering, see section 4
  payload: Uint8Array;                // (3)
  headers: ReadonlyMap<string, string>;
  publishedAt: Instant;
}

class TopicLog {
  #entries: Message[] = [];           // (4)
  #base = 0n;                          // (5) offset of entries[0] after trimming

  append(m: Omit<Message, "offset">): bigint {
    const offset = this.#base + BigInt(this.#entries.length);
    this.#entries.push({ ...m, offset });
    return offset;                    // (6)
  }

  read(from: bigint, max: number): readonly Message[] {   // (7)
    const start = Number(from - this.#base);
    if (start < 0) throw new OffsetTooOld(from, this.#base);   // (8)
    return this.#entries.slice(Math.max(0, start), Math.max(0, start) + max);
  }
}

(1) The broker assigns the offset, because it is the only thing that knows the order. A publisher-supplied identifier cannot be a position.

(2) The key is what routes a message and what guarantees ordering between related messages. Section 4 is about it.

(3) Bytes, not a parsed object. The broker never looks inside the payload, and that single restraint is what lets one broker carry every kind of message in a company without knowing about any of them.

(4) An array standing in for an append-only file. The real thing is a segmented file with an index; the shape of the operations is identical.

(5) After old messages are trimmed, entry zero is no longer offset zero. Keeping a base offset is what lets offsets stay stable and monotonic forever while storage is finite — and forgetting it is the bug where every subscriber silently rewinds after a trim.

(6) Append returns the offset, so a publisher that needs confirmation gets a position it can refer to.

(7) Read is a pure function of a position and a limit. It mutates nothing, so any number of subscribers can read concurrently without coordination.

(8) Asking for something already trimmed is an error rather than a silent jump forward, because silently skipping data is the failure a subscriber most needs to be told about.

Subscriptions hold the position:

interface Subscription {
  group: GroupId;                     // (1)
  topic: TopicId;
  committed: bigint;                  // (2) everything before this is done
  inFlight: Map<bigint, Instant>;     // (3) delivered, not yet acknowledged
}

(1) A group rather than a single consumer, because several processes usually share the work of one logical subscriber. Two different groups on the same topic each get every message; two members of the same group split them.

(2) One number: everything before it has been handled. Committing a position rather than acknowledging individual messages is what makes recovery simple after a crash — you restart from the number.

(3) Messages handed out but not yet acknowledged, with the time they were handed out. Section 3 is about what happens when that time gets old.

3. Delivery: the three guarantees and where they actually come from

At-most-once. Deliver, and forget. If the subscriber dies mid-processing, the message is gone. Fast, simple, and acceptable only for data that does not matter individually — metrics samples, presence pings.

At-least-once. Deliver, wait for an acknowledgement, and redeliver if none arrives. Nothing is lost, and duplicates happen whenever a subscriber processes a message and dies before acknowledging. This is what a broker can actually provide, and it is the right default.

Exactly-once is the one to be careful about in an interview, because the honest answer is more impressive than the confident one. A broker cannot deliver a message exactly once to a process that might crash at any instant — the acknowledgement and the work are two separate actions and no ordering of them survives a crash between the two. What is achievable is exactly-once effect: at-least-once delivery plus a consumer whose processing is idempotent, usually by recording the message identifier it has handled and ignoring repeats (10.4).

The mechanism for at-least-once is a visibility timeout, and it is worth writing because the details are where people slip:

function poll(sub: Subscription, log: TopicLog, max: number, now: Instant) {
  const expired = [...sub.inFlight]                       // (1)
    .filter(([, sentAt]) => now.minus(sentAt) > VISIBILITY_TIMEOUT)
    .map(([offset]) => offset);

  const fresh = log.read(sub.committed + BigInt(sub.inFlight.size), max);  // (2)
  for (const m of fresh) sub.inFlight.set(m.offset, now);                  // (3)
  return { redeliver: expired, fresh };
}

(1) Anything handed out longer ago than the timeout is assumed lost and becomes eligible for redelivery. The subscriber is not asked whether it is alive; silence is the signal, which is the same principle as the traffic controller's heartbeat in 9.7.15 — a component that must report its own failure cannot report the failure of being hung.

(2) New messages are read from beyond both the committed position and everything currently in flight.

(3) Handing a message out records when, so the clock starts.

The timeout is a real trade-off to state. Too short and a subscriber doing slow work gets its message redelivered while it is still processing it, so the work happens twice — and if that work is not idempotent, twice is a bug. Too long and a genuinely dead subscriber's messages sit unprocessed for minutes. The usual escape is to let a subscriber extend the timeout while it works, which turns a guess into a heartbeat.

Committing the offset has an ordering with a right answer. Commit after processing, never before. Committing first turns a crash into lost data — at-most-once with extra steps. Committing after turns a crash into a duplicate, which idempotent processing absorbs. When you must choose between losing and repeating, choose repeating, because a duplicate can be defended against and a loss cannot be detected.

After several failures, a message goes to a dead-letter topic. Without that, one message that always throws is redelivered forever and blocks everything behind it in its partition. The dead-letter topic keeps the poison message, keeps its error, and lets the rest flow. Volunteering this is a reliable way to show you have operated one of these rather than read about it.

4. Ordering, keys and parallelism

A single log has a total order and one consumer, because handing message 5 to one process and message 6 to another abandons the order immediately. So there is a direct conflict: order needs one reader, throughput needs many.

The standard resolution is partitions. A topic is several independent logs. A message's key decides which one it goes to, usually by hashing the key.

function partitionFor(key: string | null, count: number): number {
  return key === null
    ? Math.floor(Math.random() * count)    // (1) no key, no ordering promise
    : hash(key) % count;                   // (2) same key, same partition, always
}

(1) No key means no ordering requirement, so spread it for balance.

(2) The same key always lands in the same partition, so all events for one order, one user, one account are in one log and therefore in order relative to each other. Events with different keys have no ordering promise at all, and that is the deal.

Three consequences worth stating without being asked, because together they show you understand what partitioning really costs:

Ordering is per key, never global. If the requirement is a total order across everything, there is exactly one partition and therefore one consumer, and the throughput ceiling is a single process. Say that plainly rather than promising both.

Parallelism is capped by partition count. Twenty consumers on eight partitions means twelve idle. Partition count is a capacity decision made in advance, and increasing it later re-maps keys to partitions, which breaks ordering across the change — one of the few genuinely painful operations in this design.

A hot key is a hot partition. One customer generating half the traffic puts half the traffic on one log, and no number of consumers helps. The fix is a better key, and that is a modelling decision rather than an infrastructure one.

Assigning partitions to group members needs one coordinator and one rule. Each partition is owned by exactly one member of a group at a time. When a member joins or dies, the assignment is recomputed and the group briefly pauses — a rebalance. The rule that keeps it correct is that a member stops consuming a partition before another may take it, because two owners of one partition means duplicated processing and interleaved commits.

5. Topic matching, when subscribers want patterns

Subscribers often want orders.*.created rather than one exact topic. The naive version tests every subscription against every published topic, which is fine for ten subscriptions and quadratic for ten thousand.

Store the subscriptions as a tree over the segments of the topic name. orders.eu.created becomes the path orders → eu → created. Matching walks the tree, following both the literal child and the wildcard child at each level.

class TopicTrie {
  match(segments: readonly string[], node = this.root, i = 0): Set<SubscriberId> {
    if (i === segments.length) return node.subscribers;
    const out = new Set<SubscriberId>();
    const exact = node.children.get(segments[i]);          // (1)
    if (exact) for (const s of this.match(segments, exact, i + 1)) out.add(s);
    const single = node.children.get("*");                 // (2)
    if (single) for (const s of this.match(segments, single, i + 1)) out.add(s);
    const multi = node.children.get("#");                  // (3)
    if (multi) for (const s of multi.subscribers) out.add(s);
    return out;
  }
}

(1) Follow the exact segment if a subscription used it.

(2) * matches exactly one segment, so it consumes this segment and continues.

(3) # matches the rest of the name, so it terminates immediately with everything below it.

The cost changes from "every subscription" to "the depth of the name times the branching the wildcards create", which for real topic names is a handful of steps. This is the same trade as any index: build a structure once so that lookups stop scanning.

6. The slow subscriber, and what the broker owes it

A subscriber that reads slower than publishers write is not an error and must not be treated as one.

In the log design, a slow subscriber costs almost nothing — its offset is just further back, and the data it has not read is the same data everyone else already read. That is the property that makes this design win. The subscriber only breaks when it falls behind the retention window and the messages it has not read are trimmed.

So retention is the real control, and it has two settings that mean different things:

By time — keep a week. Predictable for people, unpredictable for disk.

By size — keep 100 GB. Predictable for disk, unpredictable for people. A traffic spike silently shortens how far back a subscriber may fall.

Real systems set both and trim on whichever is hit first, and the alert that matters is not disk usage — it is consumer lag, the distance between a subscriber's offset and the end of the log. Lag rising steadily means a subscriber that will run out of retention at a calculable time, which turns a future outage into a scheduled piece of work.

Push versus pull is the related question, and pull is right here for one reason: the subscriber controls the rate. In a push design the broker must guess how fast each subscriber can go, and getting it wrong overwhelms them — which is backpressure, arriving as a design flaw (9.5.4). Pull has one weakness, which is that a poll returning nothing wastes a round trip, and the standard fix is a long poll: the broker holds the request open for a second or two and answers the moment something arrives.

If you do build the queue-per-subscriber shape, then bounded buffers are mandatory and there are exactly three policies when one fills — block the publisher, drop the oldest, or drop the newest. Every one of them is a product decision. Unbounded is not a fourth option; it is the decision to fail later, in memory, all at once.

7. Concurrency inside one broker

Appending is the only true contention point. One writer per partition, taking a short lock or using an atomic increment to claim the next offset. It is short because appending is copying bytes and incrementing a number.

Reading takes no lock at all, which is the payoff for making the log append-only. A reader reading up to a published end position cannot see a partially written entry, because entries become visible only after they are complete. The one rule that must hold is that the visible end position is published after the entry is written, never before — a store-release, in the vocabulary of 9.5.1.

Offsets are per group and are written rarely, so a small lock per group is ample. The race that matters is two members of one group believing they own the same partition, and that is prevented by the ownership rule in section 4 rather than by a lock — which is worth saying, because it is an example of a concurrency problem solved by assignment rather than by synchronisation.

Everything about a broker's speed comes from batching. Publishers batch messages into one append; consumers fetch many messages in one read; offsets are committed periodically rather than per message. Each of these trades a bounded amount of duplication or delay for a large multiple of throughput, and knowing which knob costs what is the difference between tuning and guessing.

8. What the interviewer will push on

"Queue per subscriber or one log with offsets?" One log with offsets, and the reasons are memory (one copy rather than one per subscriber), replay (move a number backwards), new subscribers (can start from the beginning), and concurrency (reading is a pure function of a position and mutates nothing). Then name the cost honestly: retention becomes a policy, and a subscriber that falls behind the window loses data.

"Can you guarantee exactly-once?" No, and the honest answer is the strong one. A broker cannot deliver exactly once to a process that can crash between doing the work and acknowledging it. What is achievable is at-least-once delivery plus an idempotent consumer that records which message identifiers it has handled — exactly-once effect, which is what everyone actually means.

"When do you commit the offset?" After processing, never before. Committing first turns a crash into lost data; committing after turns it into a duplicate that idempotent processing absorbs. The general rule: when the choice is between losing and repeating, choose repeating, because a duplicate can be defended against and a loss cannot even be detected.

"How do you keep order and still go fast?" You do not, globally. Order is per partition, and a message's key chooses its partition, so events sharing a key are ordered relative to each other and nothing else is. Then state the three consequences: parallelism is capped by partition count, a global total order means one partition and one consumer, and a hot key is a hot partition that no amount of consumers can help.

"A subscriber is down for two hours." Nothing happens — its offset stops moving and the log does not care. It becomes a problem only when it falls behind retention, so the metric to watch is consumer lag rather than disk usage, because rising lag turns a future outage into a date you can calculate.

"One message always fails." Redelivery would retry it forever and block everything behind it in its partition. After a bounded number of attempts it moves to a dead-letter topic with its error attached, and someone looks at it later while the rest flows.

The thing to volunteer that nobody asks for: the broker never parses the payload. It carries bytes plus headers and knows nothing about the contents. That single restraint is why one broker can serve every team in a company without becoming a dependency on anybody's schema — and it is the discipline that breaks first when someone asks for routing "just on this one field".

Recall

• The one decision: who remembers how far each subscriber got. A queue per subscriber, or one log plus one offset per subscriber. Choose the log.
• The log wins on one copy of the data, replay by moving a number back, new subscribers can start anywhere, and reads mutate nothing. The cost is that retention becomes a policy and falling behind the window loses data.
• The broker assigns offsets. Keep a base offset after trimming or every subscriber silently rewinds.
• The broker carries bytes, never parsed payloads. That restraint is why it can serve everyone.
• At-least-once is what a broker provides. Exactly-once delivery is impossible; exactly-once effect is at-least-once plus an idempotent consumer.
• Redelivery runs on a visibility timeout: too short repeats live work, too long leaves dead work stuck. Let subscribers extend it.
• Commit after processing, never before. Choose repeating over losing — a duplicate is defensible, a loss is undetectable.
• Order is per partition, and the key chooses the partition. Parallelism is capped by partition count, a global order means one consumer, and a hot key is a hot partition.
• A group shares partitions, one owner per partition, and a member must stop before another may take over.
• Watch consumer lag, not disk. Poison messages go to a dead-letter topic after bounded attempts.
• Pull with a long poll, so the subscriber sets the rate. Speed comes from batching at every level.

Self-test: What is the single decision this design turns on? What does the base offset protect against? Why is exactly-once delivery impossible, and what replaces it? Which side of the commit does processing go on? What caps parallelism? What metric predicts a subscriber's outage?

Quiz Bank

FoundationalDesign the core of a publish-subscribe broker. State the central decision and justify your answer.

The central decision is who remembers how far each subscriber has got, and there are two answers.

A queue per subscriber. Publishing copies the message into one queue per subscriber, and each copy is deleted when that subscriber acknowledges it. Memory is messages × subscribers. A delivered message is gone, so replay is impossible at any price, and a new subscriber gets nothing from before it existed.

One log per topic, one offset per subscriber. The message is appended once. Each subscriber holds a number saying how far it has read, and delivery means "give me what comes after this number".

Choose the log, for four reasons that are worth listing separately because each one answers a different follow-up.

Storage. One copy of each message regardless of how many subscribers exist.

Replay. Reprocessing a day is moving a number backwards. In the queue design it is not a slow operation, it is an impossible one.

New subscribers. They can start at the beginning, the end, or a timestamp, because the data is still there.

Concurrency. Reading is a pure function of a position and mutates nothing, so any number of subscribers read without coordinating. This is the quiet reason the design is simple, and it is the one candidates rarely say.

The model is small:

interface Message {
  offset: bigint;          // assigned by the log
  key: string | null;      // decides partition and ordering
  payload: Uint8Array;     // bytes — the broker never parses it
  publishedAt: Instant;
}

interface Subscription {
  group: GroupId;
  topic: TopicId;
  committed: bigint;               // everything before this is done
  inFlight: Map<bigint, Instant>;  // handed out, not yet acknowledged
}

Three details that separate a working design from a sketch.

The broker assigns the offset, because it is the only component that knows the order. A publisher-supplied identifier is an identifier, not a position.

Keep a base offset once old messages are trimmed, so entry zero in storage is not offset zero. Without it, every subscriber silently rewinds after the first trim, which is a data-corruption bug wearing the costume of a performance optimisation.

The payload is bytes. The broker knowing nothing about the contents is what lets one broker serve every team without becoming a dependency on anyone's schema.

And the cost of the log design, stated honestly: messages are removed on a retention policy rather than when they are consumed, so a subscriber that falls further behind than the retention window loses data permanently. That makes consumer lag the metric that matters, and it is covered properly in the Applied answer.

AppliedGuarantee that no message is lost when a subscriber crashes mid-processing. Describe the mechanism precisely and name what it costs.

Start by naming the guarantee you can actually provide: at-least-once. At-most-once loses data on a crash. Exactly-once delivery is not achievable at all, for a reason worth stating plainly — the work and the acknowledgement are two separate actions, and no ordering of them survives a crash between the two. Acknowledge first and a crash loses the message; work first and a crash repeats it. There is no third arrangement.

The mechanism is a visibility timeout with redelivery.

When a message is handed to a subscriber it is recorded as in flight with the time it was sent. If an acknowledgement arrives, it leaves that set and the committed position eventually advances past it. If no acknowledgement arrives within the timeout, the message becomes eligible for delivery again.

const expired = [...sub.inFlight]
  .filter(([, sentAt]) => now.minus(sentAt) > VISIBILITY_TIMEOUT)
  .map(([offset]) => offset);

The subscriber is never asked whether it is alive. Silence is the signal, because a hung process cannot report that it is hung — the same reasoning as a heartbeat everywhere else in this book.

The commit ordering is the part with a right answer. Commit the offset after processing, never before. Committing first converts a crash into lost data, which is at-most-once with more machinery. Committing after converts it into a duplicate, and duplicates are absorbable. The general rule: when the choice is between losing and repeating, choose repeating, because a duplicate can be defended against and a loss cannot even be detected.

What it costs, in three parts.

Duplicates are now normal, not exceptional. Every consumer must be idempotent, usually by recording the identifiers it has processed and ignoring repeats. That is the price of the guarantee, and it is paid by consumer authors rather than by the broker.

The timeout is a genuine trade-off. Too short and a subscriber doing slow work has its message redelivered while it is still working, so the work happens twice concurrently. Too long and a dead subscriber's messages sit idle for minutes. The escape is to let a subscriber extend the lease while it works, which turns a guess into a heartbeat.

A poison message blocks its partition. A message that always fails is redelivered forever and everything behind it waits. So attempts are counted, and after a bounded number the message moves to a dead-letter topic with its error attached. Without this, one bad message stops a partition indefinitely, which is one of the most common real outages in systems like this.

One refinement worth volunteering. Committing a single number means everything before it is done, which is simple and fast but interacts awkwardly with processing messages out of order inside a batch. If message 10 finishes before message 9, the committed position cannot advance past 8 until 9 completes. Either process strictly in order within a partition — usually the right answer, since ordering was the reason for partitions — or track the completed set explicitly and accept the extra bookkeeping. Naming that tension shows you have thought past the happy path.

InterviewA subscriber needs messages for one customer strictly in order, but the system must also handle a million messages a second. How?

Say the conflict out loud first, because it cannot be dissolved. A single log has a total order and can therefore be read by exactly one consumer at a time — handing message 5 to one process and message 6 to another abandons the order at the first opportunity. Ordering wants one reader; throughput wants many.

The resolution is to narrow what "in order" means. The requirement is not a global order over everything; it is an order per customer. Those are enormously different promises, and the design exists to serve the second one cheaply.

So a topic is several independent logs, and a key chooses one:

function partitionFor(key: string | null, count: number): number {
  return key === null ? randomPartition(count) : hash(key) % count;
}

The customer identifier is the key, so every message for one customer lands in the same partition, is appended in one order, and is read by one consumer in that order. Messages for different customers have no ordering promise relative to each other, which nobody asked for and nobody will notice.

Parallelism then comes from partitions. A thousand partitions can be read by up to a thousand consumers at once, and the total order within each is untouched. That is how both requirements are met without either being weakened.

Three consequences to state before being asked, because they are what the interviewer is checking:

Parallelism is capped by partition count. Twenty consumers on eight partitions leaves twelve idle. Partition count is a capacity decision made in advance, so it is chosen for the traffic you expect to have, not the traffic you have.

Changing partition count re-maps keys. A customer that used to hash to partition 3 now hashes to partition 7, and messages for that customer exist in both. Ordering across the change is broken, so the operation is either done during a quiet window with consumers drained, or avoided by over-provisioning partitions from the start. This is one of the few genuinely painful operations in the design and it is worth knowing that in advance.

A hot key is a hot partition. One customer generating half the traffic puts half the traffic in one log, and no number of consumers helps because that partition has one owner. The fix is a better key — customer plus something else, if the ordering requirement really is per customer per something. That is a modelling decision, and noticing that the fix lives in the model rather than in the infrastructure is the point.

One more piece completes the answer: within a group, each partition has exactly one owner. When a consumer joins or dies, ownership is recomputed and the group briefly pauses. The rule that keeps it correct is that a member must stop consuming a partition before another may start, because two owners means duplicated work and interleaved commits — and interleaved commits can move the committed position backwards, which loses data rather than merely repeating it.

StaffMake this broker durable and survivable: it must not lose acknowledged messages when the machine dies, and it must keep working when one node is lost.

Start with what "acknowledged" must mean, because everything else is a consequence. A publisher that received a success has been promised the message will not be lost. If the broker acknowledged after writing only to memory, a power cut breaks that promise. So the acknowledgement point is a design decision with a directly visible cost, and there are three defensible settings:

Acknowledge on memory write. Fastest, and a single machine failure loses recent messages.

Acknowledge after the write reaches disk. Survives a process crash. Survives a power cut only if the write was genuinely flushed, not merely handed to the operating system — the distinction that makes fsync the expensive call it is (2.6).

Acknowledge after the message is on several machines. Survives losing a machine. This is the only setting that meets "must not lose acknowledged messages when the machine dies", so it is the answer to this question, and the others are worth naming so the trade is visible.

Replication follows the log's own shape, which is the pleasant part. A partition has a leader and followers; the leader appends and followers copy the same append-only sequence. Because the log is append-only and offsets are dense, a follower is described completely by one number — how far it has copied. That makes catching up a range fetch rather than a reconciliation, and it is why a log replicates far more simply than mutable state does (10.5).

Then the acknowledgement rule that makes it a guarantee: a message is committed when a defined number of replicas have it, and only committed messages are visible to consumers. Two properties follow. A consumer never sees a message that could be lost, which prevents the nastiest possible failure — a consumer acting on data that then un-exists after a failover. And the publisher's acknowledgement can be tied to the same point, so "acknowledged" and "will survive" mean the same thing.

Leader failure is where the design earns its keep. A follower that has all committed messages is promoted. Uncommitted messages that existed only on the old leader are discarded, which is correct because they were never acknowledged and never visible. The important design consequence is that a publisher may have timed out on a message that was in fact written — so publishers must be able to retry safely, which means a producer identifier and a sequence number so the broker can recognise and drop a retried duplicate. Without that, "no loss" is achieved by allowing duplicates at the publishing end, which is a poor trade when it can be avoided.

Consumer offsets need the same durability as the data. Storing them in memory on the broker means a failover resets every subscriber to an old position, which is a mass redelivery event at the worst possible moment. Store them in the log itself — a topic whose messages are offset commits — and they inherit replication, retention and recovery from the machinery that already exists. That reuse is worth calling out as a design decision rather than a trick.

Two things I would insist on, both about failure being visible.

Under-replicated partitions must be alarming, not informational. A partition running with fewer replicas than required still works perfectly and has silently lost its guarantee. That is exactly the state that must never be discovered during the next failure.

Failover must be measured, not assumed. The number that matters is how long consumers are stalled while leadership moves, and it is only knowable by doing it deliberately and often.

And the boundary worth ending on. All of this is durability and availability. None of it changed the delivery guarantee: it is still at-least-once, consumers still need to be idempotent, and no amount of replication makes exactly-once delivery possible. Replication protects against losing what was acknowledged; it does not change the fact that a consumer can crash between doing work and saying so.

Flashcards

FlashThe one decision

Who remembers how far each subscriber got. A queue per subscriber copies every message N times and cannot replay. One log with one offset per subscriber stores it once, replays by moving a number, and mutates nothing on read.

FlashExactly-once

Impossible as a delivery guarantee — work and acknowledgement are two actions and a crash can land between them. Achievable as an effect: at-least-once plus an idempotent consumer.

FlashCommit ordering

After processing, never before. Committing first turns a crash into loss; committing after turns it into a duplicate. Choose repeating over losing — a duplicate is defensible, a loss is undetectable.

FlashOrder versus throughput

Order is per partition; the key picks the partition. Parallelism is capped by partition count, a global total order means one consumer, and a hot key is a hot partition no consumer count can fix.

FlashThe metric that matters

Consumer lag — the distance from a subscriber's offset to the end of the log. Rising lag turns a future outage into a calculable date. Disk usage tells you nothing about who is about to lose data.

FlashPoison message

Redelivery would retry it forever and block everything behind it in its partition. Count attempts and move it to a dead-letter topic with its error, then keep flowing.

Next: 9.7.20 — the online auction, where the last three seconds contain most of the design.