10.8.1 — Queues, Streams & Delivery Machinery

Most production distributed systems are, structurally, a few services and a lot of messaging. The messaging layer is built from a handful of ideas:

• Queue or stream. This one decision shapes everything after it.
• Consumer groups and offsets.
• Acknowledging messages, and delivering them again.
• Dead-letter queues and retry layouts.
• Backpressure.
• Fan-out, one message reaching many consumers.
• And the honest choice between Kafka and RabbitMQ.

9.5.4 built these shapes inside one process. Here they survive crashes and run across machines, and they bring the failures that come with both.

1. Queue vs stream: the defining difference

Both move messages from producers to consumers. The difference is what happens after consumption: ^{⚑Difference between message queues and event streams. [EQ-494b]}

Message queue (RabbitMQ, SQS, ActiveMQ) — a message is delivered to one consumer of the queue and removed on acknowledgment. The queue is a work list; consumers compete for tasks (the competing consumers pattern); adding consumers increases throughput; the message is gone once done. Mental model: work distribution.

Event stream / log (Kafka, Pulsar, Kinesis, Redpanda) — messages are appended to a durable, ordered log and retained for a configured period regardless of consumption. Each consumer group tracks its own offset (position); many independent groups read the same events at their own pace; a new consumer can start from the beginning and replay history. Mental model: a shared, replayable record of what happened.

The consequences that decide your architecture:

	Queue	Stream
After consumption	deleted	retained (time/size based)
Consumers	compete for messages	independent groups, each with offsets
New consumer joins	sees only future messages	can replay from any offset
Ordering	per-queue, weak under concurrency	strict per partition
Scaling reads	more consumers = faster drain	more consumers ≤ partition count
Natural fit	jobs, commands, RPC-ish work	events, analytics, multi-consumer facts

The design tell: if a message is a command to do work once ("resize this image"), you want a queue. If it's a fact that happened ("order placed") that several unrelated systems may care about now or later, you want a stream. Getting this backwards is expensive: commands on a stream can be re-executed by replay; facts on a queue are consumed by one system and lost to the others forever.

2. Acknowledgment, redelivery, and the visibility window

The mechanism that makes at-least-once real (10.4): a consumer receives a message, processes it, and acknowledges. Until the ack arrives, the broker keeps the message; if the consumer dies, the message is redelivered.

You will meet two ways of doing it:

• Explicit ack and nack, as in RabbitMQ. A message that was never acknowledged is delivered again when the connection closes. nack with requeue retries it right away.
• A visibility timeout, as in SQS. A message you receive becomes invisible for N seconds. If you do not delete it in that time, it reappears.

Both lead to the same three rules in practice:

1. Acknowledge only after the work is safely saved. If you acknowledge on arrival, your queue delivers at most once, and a crash loses the message.
2. Long tasks must extend their lease, with a heartbeat or ChangeMessageVisibility. Otherwise the message reappears while you are still working, and a second consumer starts on it too. That is the lease problem from 10.7.2.
3. Duplicates are normal, so consumers must be safe to run twice (10.4).

3. Failure topology: retries, DLQs, and poison messages

A consumer fails. What happens next is a designed topology, not a default: ^{⚑What are dead letter queues and retry strategies? [EQ-496b]}

• Immediate requeue — the naive choice, and usually wrong: a message that fails due to a downed dependency will fail again in milliseconds, spinning a hot loop that burns CPU and floods logs.
• Retry with backoff — exponential delays with jitter (10.9), implemented either by the broker (delayed exchanges, SQS delay queues) or by retry queues per delay tier (a common Kafka/Rabbit pattern: retry-5s, retry-1m, retry-10m topics, each with a consumer that waits then republishes).
• Dead letter queue (DLQ) — after N attempts, the message moves to a separate queue where it stops blocking progress and waits for a human. This is the release valve that keeps a single poison message (malformed payload, a bug on one record) from halting a partition or consuming your retry budget forever.

Three rules separate teams that use dead-letter queues well from teams with a dead-letter queue nobody looks at:

1. Alert on how many messages are in it. A message landing there is a sign of an incident, not just a log line.
2. Keep the context of the failure: the original message, how many attempts were made, the last error and the trace ID (10.10). Then nobody has to reproduce the problem to understand it.
3. Build the way back. Have a tool, already tried out, that replays the messages after the bug is fixed. Otherwise the queue becomes a graveyard for data, and everyone learns to ignore it.

Ordering caveat: with strict per-partition ordering (Kafka), a failing message blocks its partition — retrying in place stalls everything behind it. The standard resolutions: move the failure to a retry topic (preserving progress, sacrificing strict order for that key), or accept head-of-line blocking deliberately when order genuinely matters more than throughput. Naming this trade is a senior signal.

4. Backpressure, fan-out, and consumer lag

Consumer lag is the gap between the newest message and where the consumer has got to. It is the health number of a messaging system, for three reasons:

• It warns early. It grows before anything breaks.
• It explains itself. Rising lag means consumers are slower than producers.
• It tells you what to do: add consumers, add partitions, or fix the slow step.

Alert on lag and on how fast it changes. The lag says how far behind you are. Its growth rate says whether you are catching up.

Backpressure — the queue's bound and its overflow policy (9.5.4's trilemma at network scale). A durable broker's "buffer" is disk, so the failure is slower but real: unbounded lag means unbounded retention cost and eventually broker limits. Policies: block/throttle producers (correct for internal pipelines), reject (429 at the API edge — protect the queue by protecting the entrance), drop (loss-tolerant telemetry, with a counter), or shed selectively (drop low-priority topics first). And the structural fix: decouple the fast path — accept work at the edge, persist minimally, and let workers drain at their own rate.

Fan-out — one event, many consumers — is the pattern behind most event-driven architectures. Streams do it natively (independent consumer groups). Queues need an exchange or topic to copy messages into per-consumer queues (RabbitMQ's fanout/topic exchanges; SNS→SQS on AWS). The design difference matters at scale: with per-consumer queues, a slow consumer's backlog is isolated to its own queue; with a shared stream, a slow consumer group merely lags while others proceed — in both cases the key property to preserve is that one slow consumer must not slow the others.

Figure 1 — Queue versus stream. A queue distributes work: one message, one worker, deleted on ack. A stream retains an ordered record that any number of independent groups consume at their own offsets — including a group created next year that replays from the beginning.

5. Kafka vs RabbitMQ: the honest decision

The question every design review asks, answered by workload rather than fashion: ^{⚑Kafka vs RabbitMQ — when to use which? [EQ-499b]}

Choose RabbitMQ, or SQS, when:

• Each message is a task with its own life: acknowledged, rejected, requeued, given a priority, expired or delayed.
• Routing is complicated, and exchanges must send work to the right consumers based on topics or headers.
• You have many queues, and they come and go.
• You need actions on single messages, such as "retry this one in 10 minutes".

It's a broker — it manages message state, which is exactly the feature and exactly the cost (state per message limits throughput).

Choose Kafka, or Pulsar or Kinesis, when you need any of these:

• Keeping and replaying history. A new consumer must be able to read old messages.
• High throughput, hundreds of MB/s. Appending to a log in order is cheap.
• Processing in order, per key.
• Several separate consumer groups reading the same data.
• Stream processing: joins, totals and views kept up to date (10.8.4).

It's a log — it stores an ordered record and tracks only offsets, which is why it's fast and why per-message operations are awkward.

The tell in one line: do you need to remember what happened, or just to get work done? Many mature systems run both — Kafka as the event backbone, a task queue for jobs — and that's a legitimate answer, not indecision. What isn't legitimate is choosing Kafka for a job queue and then rebuilding per-message retry, priority, and delay on top of it (a well-known path to sadness), or choosing RabbitMQ for an event backbone and discovering that the analytics team cannot replay last month.

6. The expert lens

Asynchrony is a contract change, not an implementation detail. Moving work behind a queue converts a synchronous failure ("your order failed") into a deferred one ("your order was accepted... and silently failed 20 seconds later"). Every async design therefore owes the user a status model — a way to observe outcome (9.6.1's 202 + status resource) — and owes the operator a failure path (DLQ + alert + redrive). Teams that add queues without those two artifacts trade visible failures for invisible ones.

The queue is not the design — the consumer's idempotency is. Every guarantee discussed here (at-least-once, redelivery, retries, DLQ replay, offset resets) multiplies messages. A consumer that is idempotent makes all of that safe; one that isn't makes each mechanism a new way to double-charge (10.4). Review consumers, not brokers.

Lag is the metric that predicts incidents. CPU and error rates are lagging indicators; consumer lag rises before the user notices, and its shape names the cause: steady growth = under-provisioned consumers; a step change = a slow dependency or a poison message; sawtooth = batch producers with insufficient drain capacity. Dashboards with lag-by-consumer-group and its derivative, plus DLQ depth, cover most of a messaging system's operability (10.10).

Next: 10.8.2 — inside the log: partitions, replication, offsets, consumer-group rebalancing, exactly-once semantics, and how Kafka actually achieves its throughput.

Recall

• Queue = work distribution: one message → one competing consumer, deleted on ack. Stream/log = replayable record: appended, retained, many independent groups with their own offsets, new consumers can replay. Tell: commands → queue; facts → stream.
• Ack machinery: explicit ack/nack or visibility timeout; ack after durable work (acking on receipt = at-most-once); long tasks must extend the lease or the message reappears mid-work; duplicates are normal ⇒ idempotent consumers (10.4).
• Failure topology (designed, not default): immediate requeue = hot loop; retry with backoff + jitter (broker delays or per-tier retry queues); DLQ after N attempts so a poison message can't block progress. Operational rules: alert on DLQ depth, preserve failure context, build and test the redrive path. Kafka caveat: a failing message blocks its partition — move it to a retry topic or accept head-of-line blocking deliberately.
• Consumer lag is the leading health metric (alert on value and growth rate); backpressure policies = throttle producers / reject at the edge / drop with a counter / shed by priority; fan-out via consumer groups (streams) or exchange→per-consumer queues (brokers) — one slow consumer must never slow the others.
• Kafka vs RabbitMQ: broker (per-message state: ack/nack/priority/TTL/delay, rich routing, many short queues) vs log (retention, replay, throughput, per-key order, multiple groups, stream processing). Do you need to remember what happened, or just get work done? Running both is a legitimate answer.
• Lens: async changes the user contract (owe a status model and a failure path); the consumer's idempotency is the real design; lag predicts incidents and its shape names the cause.

Self-test: Give the after-consumption difference and three consequences. Why must ack follow durable work, and what breaks with long tasks? Design a retry topology with a DLQ and name the two operational rules teams forget. When does a failing message block a whole partition? Choose Kafka or RabbitMQ for: image resizing jobs, order events consumed by four teams.

Quiz Bank

FoundationalWhat is the fundamental difference between a message queue and an event stream, and how does it change design?

After consumption: a queue deletes an acknowledged message; a log retains it for a configured window. Everything else follows. In a queue, consumers compete — each message goes to exactly one worker, so adding consumers drains faster and the message is gone once handled; the mental model is work distribution. In a stream, messages are appended to an ordered, durable log and each consumer group tracks its own offset, so many unrelated systems read the same events independently, at their own pace, and a group created next year can replay from the beginning; the mental model is a shared record of what happened. Design consequences: ordering is per-partition and strict in a log versus weak-under-concurrency in a queue; read scaling is bounded by partition count in a log versus by consumer count in a queue; and adding a new consumer is free in a log but requires routing changes (and loses history) with a queue.

The selection tell: a command to be executed once ("resize this image") belongs on a queue; a fact that several systems may care about now or later ("order placed") belongs on a stream — and reversing this is expensive, because replayed commands re-execute and consumed facts are lost to everyone else.

FoundationalExplain acknowledgment, visibility timeouts, and the three rules they impose on consumers.

A broker holds a message as in flight until the consumer acknowledges it; if the consumer dies, the message becomes available again — this is what makes at-least-once delivery real. Two mechanisms: explicit ack/nack (RabbitMQ — unacked messages are redelivered when the channel closes; nack can requeue immediately or route onward) and visibility timeout (SQS — a received message is hidden for N seconds and reappears unless deleted).

Rule 1: acknowledge only after the work is durable. Acking on receipt turns the system into at-most-once — a crash between ack and completion loses the message with no trace. Rule 2: long tasks must extend the lease. If processing outlives the visibility window without heartbeating (ChangeMessageVisibility, consumer heartbeats), the message reappears and a second consumer starts the same work — the 10.7.2 lease-expiry problem, arriving as duplicate execution.

Rule 3: duplicates are normal, so consumers must be idempotent — redelivery happens on crashes, rebalances, timeouts, and network hiccups, none of which are exceptional (10.4's claim-and-apply in one transaction). Together these are why "we use a queue" is not a design: the design is what the consumer does with duplicates and how the ack boundary is placed relative to durability.

AppliedDesign the retry and dead-letter topology for a payment-notification consumer that calls a flaky partner API.

Layered, with each layer's failure class: (1) In-process retry with exponential backoff and jitter for transient errors (connection resets, 503s, timeouts) — 3 attempts over a few seconds, bounded so the consumer doesn't hold the message for minutes (10.9). (2)

Requeue with delay for longer outages — instead of hot-looping, republish to a delay tier (retry-1m, retry-10m, retry-1h queues/topics, each consumed by a worker that waits then republishes to the main queue), carrying an incremented attempt count in the message header. This preserves the main queue's throughput while a dependency is down, and the tiers make the retry schedule inspectable. (3)

Circuit breaker around the partner call so that during a sustained outage the consumer stops attempting immediately and messages flow to the delay tier without burning latency and connections (10.9). (4) DLQ after N total attempts (say 6 across tiers), preserving the original message, attempt history, last error, and trace id — because triage should not require reproducing the failure.

The rules that make it work in practice:

• Alert as soon as anything lands in the dead-letter queue. Each message there is a customer whose notification failed. That is a sign of an incident, not just a log line.
• Alert when the delay queues keep growing. That means a dependency is down right now.
• Build a tool that replays dead-letter messages after the bug is fixed, and try it before you need it. Consumers that are safe to run twice make the replay safe (10.4).

Poison-message discipline: messages that fail deterministically (malformed payload, unsupported currency) should be routed to the DLQ on the first failure rather than consuming six attempts — distinguish transient from permanent by error classification (9.9.3's operational-vs-programming split, applied to consumers).

InterviewKafka or RabbitMQ? Give the decision framework and two worked examples.

Framework — ask what the message is. RabbitMQ/SQS are brokers: they track per-message state, enabling ack/nack, priority, TTL, delayed delivery, and rich routing (topic/header exchanges) — ideal when each message is a task with a lifecycle, when queues are numerous or short-lived, and when you need per-message operations. The cost of that state is throughput. Kafka/Pulsar/Kinesis are logs: append-only, retained, partitioned, tracking only consumer offsets — ideal when you need retention and replay, very high throughput (sequential writes), strict per-key ordering, multiple independent consumers of the same data, or stream processing. Per-message operations (retry this one later, prioritize that one) are awkward by design.

Example 1 — image resizing jobs: RabbitMQ/SQS. Each job is a command executed once; you want priority (paid users first), per-message retry with delay, and visibility timeouts; nobody replays yesterday's resize requests, and throughput is modest. Example 2 — order events consumed by billing, search, analytics, and notifications: Kafka. It's a fact with four independent consumers whose speeds differ, analytics must replay history when the model changes, per-customer ordering matters, and adding a fifth consumer next quarter must not touch producers.

The honest closing note: mature systems often run both — the event backbone in Kafka, job queues in Rabbit/SQS — and the anti-patterns to avoid are rebuilding per-message retry/priority on Kafka, or discovering that your event history is unreplayable because facts went through a queue.

StaffA team moved order processing to async queues to improve API latency. Latency improved; now customers complain about orders 'disappearing', and support has no way to answer 'what happened to my order?'. Diagnose and design the fix.

Diagnosis: the team changed the contract without building the artifacts asynchrony requires (section 6). Synchronously, a failure was visible at the moment of ordering; asynchronously, the API returns success and any downstream failure is invisible — messages that exhausted retries are presumably sitting in a DLQ nobody watches (or worse, were dropped by an unbounded requeue loop that eventually hit a limit), and there is no per-order status anyone can query. The customer's "disappeared" is precisely accurate: the order exists nowhere the customer or support can see.

The fix — three artifacts, all mandatory for async work. (1) A status record. The API saves the order before replying, with a status (9.5.4). It returns 202 with an address where the status can be read (9.6.1). Every change of stage is written to that record: accepted, paid, reserved, then confirmed or failed. So "what happened to my order?" is a quick lookup, not an investigation, and the customer's screen can show real progress. The queue now carries work about an order that is already saved. It is never the only copy of the order. Letting the queue hold the only copy was the real reason orders "disappeared". (2)

A failure path with an owner: DLQ per stage with alerting on depth, failure context preserved, a redrive tool, and — critically — a customer-visible failure state plus a compensating action (refund the authorization, notify the customer, 10.8.4's saga discipline). Silent DLQ accumulation is the same defect as silent drops. (3)

Being able to see it:

• Pass trace IDs from the API through every consumer, so one ID links the HTTP request, the messages and the logs (10.10).
• Dashboards for consumer lag, dead-letter queue depth, and how long each stage change takes.
• An alert on orders stuck in an unfinished state for more than X minutes. That one check catches every kind of failure, including the ones you never thought of.

The principle to record in the postmortem: asynchrony moves failures from the request path to the background, so it must move failure visibility there too — an async system without a status model, a DLQ policy, and stuck-state alerting hasn't improved latency, it has hidden its defects behind it.

Flashcards

FlashQueue vs stream

Queue: one consumer, deleted on ack — work distribution (commands). Stream: retained log, per-group offsets, replayable, per-partition order — record of facts (events).

FlashAck rules

Ack after durable work (else at-most-once) · extend leases for long tasks (or it reappears mid-work) · duplicates are normal ⇒ idempotent consumers.

FlashRetry topology

Backoff + jitter (not immediate requeue — hot loop) → delay tiers → DLQ after N. Alert on DLQ depth, preserve failure context, build a tested redrive.

FlashHead-of-line blocking

With strict per-partition order, one failing message stalls its partition. Move it to a retry topic (lose strict order for that key) or accept the block deliberately.

FlashConsumer lag

The leading health metric. Steady growth = under-provisioned; step change = slow dependency/poison message; sawtooth = batch producers. Alert on value AND derivative.

FlashAsync's contract debt

Async owes users a status model (202 + status resource) and operators a failure path (DLQ + alert + redrive). Without them you traded visible failures for invisible ones.

Scenario Drill

DrillDesign the messaging architecture for a food-delivery platform: order events consumed by restaurant dispatch, courier assignment, analytics, and notifications; courier location updates at 50k/s; payment captures that must never be lost; and a promotions team that wants to replay last month's orders to test a new pricing model. Choose queue vs stream per flow and justify.

Order events → stream (Kafka), partitioned by order_id. Here is why:

• Four separate consumers, each with its own speed and its own ways of failing, is the textbook case for fan-out.
• Partitioning by order ID keeps each order's events in order, so a cancelled event can never be handled before its placed event. It also spreads the load.
• The promotions requirement settles it. Replaying last month's orders is only possible if messages are kept. A queue would have deleted each message the moment dispatch consumed it.

Keep messages long enough to cover the replay, say 30 to 90 days. The promotions team then runs as a new consumer group starting from an old offset. It reads production events at its own pace, without touching the producers or affecting the other three consumers. That single capability is what makes the stream choice correct rather than fashionable.

Courier locations at 50k/s → stream, but a separate topic with short retention and aggressive settings. The volume decides the technology. Appending to a log in order can handle it. A broker tracking the state of every single message could not.

How much each message is worth decides the settings. A courier's location is only useful for a moment (10.7.1). So keep messages for minutes, acks=1 is acceptable, and a consumer that falls behind should jump to the newest offset instead of catching up. Replaying old positions is worse than useless.

Keeping it off the order topic is essential. If you mix a 50,000-per-second flood with business-critical order events, they end up sharing how long messages are kept, how they are partitioned and how they fail.

Payment captures → queue (or a stream with strict guarantees), and the real answer is the outbox. "Must never be lost" is a durability requirement that no messaging choice alone satisfies, because the classic loss happens before the broker: the service writes the payment record and then crashes before publishing. So the capture is written with a transactional outbox (10.8.4) and relayed to a durable queue with acks=all/persistent delivery; the consumer is idempotent on capture id and calls the PSP with a derived idempotency key (10.4). Use a queue here rather than the shared event stream because captures are commands with per-message lifecycle needs (delayed retry against a flaky PSP, priority, DLQ with human triage) — the section 5 tell applied precisely.

Notifications → consumer group on the order stream, with its own retry/DLQ topics, because notification failures must not block dispatch or analytics (independent groups give that isolation for free) and because a failed SMS deserves backoff and eventual DLQ rather than infinite retry.

Decisions that apply everywhere, which you should state:

• A dashboard of consumer lag for each group, with alerts when lag grows quickly.
• Alerts on dead-letter queue depth, with a named owner for each consumer.
• Trace IDs passed from the ordering API through every consumer (10.10).
• An explicit rule that replaying for the promotions team runs in a separate consumer group, writing somewhere read-only. Then replaying a month of orders cannot send out deliveries, notifications or payments again. That is the most common and most embarrassing replay accident, and here it is prevented by the design, not by a checklist.