10.8.1 — Queues, Streams & Delivery Machinery

Most production distributed systems are, structurally, a few services and a lot of messaging. This page builds that layer: the queue vs stream distinction (the decision that shapes everything downstream), consumer groups and offsets, acknowledgment and redelivery, DLQs and retry topologies, backpressure, fan-out, and the honest Kafka vs RabbitMQ decision. 9.5.4 built these shapes in-process; here they gain durability, distribution, and the failure modes that come with both.

1. Queue vs stream: the defining difference

Both move messages from producers to consumers. The difference is what happens after consumption: ^{⚑Difference between message queues and event streams. [EQ-494b]}

Message queue (RabbitMQ, SQS, ActiveMQ) — a message is delivered to one consumer of the queue and removed on acknowledgment. The queue is a work list; consumers compete for tasks (the competing consumers pattern); adding consumers increases throughput; the message is gone once done. Mental model: work distribution.

Event stream / log (Kafka, Pulsar, Kinesis, Redpanda) — messages are appended to a durable, ordered log and retained for a configured period regardless of consumption. Each consumer group tracks its own offset (position); many independent groups read the same events at their own pace; a new consumer can start from the beginning and replay history. Mental model: a shared, replayable record of what happened.

The consequences that decide your architecture:

	Queue	Stream
After consumption	deleted	retained (time/size based)
Consumers	compete for messages	independent groups, each with offsets
New consumer joins	sees only future messages	can replay from any offset
Ordering	per-queue, weak under concurrency	strict per partition
Scaling reads	more consumers = faster drain	more consumers ≤ partition count
Natural fit	jobs, commands, RPC-ish work	events, analytics, multi-consumer facts

The design tell: if a message is a command to do work once ("resize this image"), you want a queue. If it's a fact that happened ("order placed") that several unrelated systems may care about now or later, you want a stream. Getting this backwards is expensive: commands on a stream can be re-executed by replay; facts on a queue are consumed by one system and lost to the others forever.

2. Acknowledgment, redelivery, and the visibility window

The mechanism that makes at-least-once real (10.4): a consumer receives a message, processes it, and acknowledges. Until the ack arrives, the broker keeps the message; if the consumer dies, the message is redelivered.

Two implementations you'll meet: explicit ack/nack (RabbitMQ: unacked messages are redelivered when the channel closes; nack with requeue for immediate retry) and visibility timeout (SQS: a received message becomes invisible for N seconds; if not deleted in that window, it reappears). Both create the same three operational realities: ack after the work is durable (acking on receipt converts your queue into at-most-once and loses messages on crash); long tasks must extend the lease (heartbeat/ChangeMessageVisibility, or the message reappears while you're still working and a second consumer starts it — 10.7.2's lease problem); and duplicates are normal, so consumers must be idempotent (10.4).

3. Failure topology: retries, DLQs, and poison messages

A consumer fails. What happens next is a designed topology, not a default: ^{⚑What are dead letter queues and retry strategies? [EQ-496b]}

• Immediate requeue — the naive choice, and usually wrong: a message that fails due to a downed dependency will fail again in milliseconds, spinning a hot loop that burns CPU and floods logs.
• Retry with backoff — exponential delays with jitter (10.9), implemented either by the broker (delayed exchanges, SQS delay queues) or by retry queues per delay tier (a common Kafka/Rabbit pattern: retry-5s, retry-1m, retry-10m topics, each with a consumer that waits then republishes).
• Dead letter queue (DLQ) — after N attempts, the message moves to a separate queue where it stops blocking progress and waits for a human. This is the release valve that keeps a single poison message (malformed payload, a bug on one record) from halting a partition or consuming your retry budget forever.

The operational rules that separate teams who use DLQs well from teams who have a DLQ nobody looks at: alert on DLQ depth (a message arriving in a DLQ is an incident signal, not a log line); preserve the failure context (original message, attempt count, last error, trace id — 10.10) so triage doesn't require reproduction; and build the redrive path (a tested tool to replay DLQ messages after a fix — otherwise the DLQ is a data graveyard, and everyone learns to ignore it).

Ordering caveat: with strict per-partition ordering (Kafka), a failing message blocks its partition — retrying in place stalls everything behind it. The standard resolutions: move the failure to a retry topic (preserving progress, sacrificing strict order for that key), or accept head-of-line blocking deliberately when order genuinely matters more than throughput. Naming this trade is a senior signal.

4. Backpressure, fan-out, and consumer lag

Consumer lag — the gap between the newest message and the consumer's position — is the health metric of a messaging system: it's leading (it grows before anything breaks), it's diagnosable (rising lag means consumers are slower than producers), and it drives the decision (add consumers, add partitions, fix the slow step). Alert on lag and its derivative: absolute lag says how far behind, growth rate says whether you're catching up.

Backpressure — the queue's bound and its overflow policy (9.5.4's trilemma at network scale). A durable broker's "buffer" is disk, so the failure is slower but real: unbounded lag means unbounded retention cost and eventually broker limits. Policies: block/throttle producers (correct for internal pipelines), reject (429 at the API edge — protect the queue by protecting the entrance), drop (loss-tolerant telemetry, with a counter), or shed selectively (drop low-priority topics first). And the structural fix: decouple the fast path — accept work at the edge, persist minimally, and let workers drain at their own rate.

Fan-out — one event, many consumers — is the pattern behind most event-driven architectures. Streams do it natively (independent consumer groups). Queues need an exchange or topic to copy messages into per-consumer queues (RabbitMQ's fanout/topic exchanges; SNS→SQS on AWS). The design difference matters at scale: with per-consumer queues, a slow consumer's backlog is isolated to its own queue; with a shared stream, a slow consumer group merely lags while others proceed — in both cases the key property to preserve is that one slow consumer must not slow the others.

Figure 1 — Queue versus stream. A queue distributes work: one message, one worker, deleted on ack. A stream retains an ordered record that any number of independent groups consume at their own offsets — including a group created next year that replays from the beginning.

5. Kafka vs RabbitMQ: the honest decision

The question every design review asks, answered by workload rather than fashion: ^{⚑Kafka vs RabbitMQ — when to use which? [EQ-499b]}

Choose RabbitMQ (or SQS) when: the unit of work is a task with per-message lifecycle (ack, nack, requeue, priority, TTL, delayed delivery); routing is complex (topic/header exchanges fan work to the right consumers); queues are many and short-lived; you need per-message operations like "retry this one in 10 minutes." It's a broker — it manages message state, which is exactly the feature and exactly the cost (state per message limits throughput).

Choose Kafka (or Pulsar/Kinesis) when: you need retention and replay (a new consumer must read history), high throughput (hundreds of MB/s — sequential log appends are cheap), ordered per-key processing, multiple independent consumer groups on the same data, or stream processing (joins, aggregations, materialized views — 10.8.4). It's a log — it stores an ordered record and tracks only offsets, which is why it's fast and why per-message operations are awkward.

The tell in one line: do you need to remember what happened, or just to get work done? Many mature systems run both — Kafka as the event backbone, a task queue for jobs — and that's a legitimate answer, not indecision. What isn't legitimate is choosing Kafka for a job queue and then rebuilding per-message retry, priority, and delay on top of it (a well-known path to sadness), or choosing RabbitMQ for an event backbone and discovering that the analytics team cannot replay last month.

6. The expert lens

Asynchrony is a contract change, not an implementation detail. Moving work behind a queue converts a synchronous failure ("your order failed") into a deferred one ("your order was accepted... and silently failed 20 seconds later"). Every async design therefore owes the user a status model — a way to observe outcome (9.6.1's 202 + status resource) — and owes the operator a failure path (DLQ + alert + redrive). Teams that add queues without those two artifacts trade visible failures for invisible ones.

The queue is not the design — the consumer's idempotency is. Every guarantee discussed here (at-least-once, redelivery, retries, DLQ replay, offset resets) multiplies messages. A consumer that is idempotent makes all of that safe; one that isn't makes each mechanism a new way to double-charge (10.4). Review consumers, not brokers.

Lag is the metric that predicts incidents. CPU and error rates are lagging indicators; consumer lag rises before the user notices, and its shape names the cause: steady growth = under-provisioned consumers; a step change = a slow dependency or a poison message; sawtooth = batch producers with insufficient drain capacity. Dashboards with lag-by-consumer-group and its derivative, plus DLQ depth, cover most of a messaging system's operability (10.10).

Next: 10.8.2 — inside the log: partitions, replication, offsets, consumer-group rebalancing, exactly-once semantics, and how Kafka actually achieves its throughput.

Recall

• Queue = work distribution: one message → one competing consumer, deleted on ack. Stream/log = replayable record: appended, retained, many independent groups with their own offsets, new consumers can replay. Tell: commands → queue; facts → stream.
• Ack machinery: explicit ack/nack or visibility timeout; ack after durable work (acking on receipt = at-most-once); long tasks must extend the lease or the message reappears mid-work; duplicates are normal ⇒ idempotent consumers (10.4).
• Failure topology (designed, not default): immediate requeue = hot loop; retry with backoff + jitter (broker delays or per-tier retry queues); DLQ after N attempts so a poison message can't block progress. Operational rules: alert on DLQ depth, preserve failure context, build and test the redrive path. Kafka caveat: a failing message blocks its partition — move it to a retry topic or accept head-of-line blocking deliberately.
• Consumer lag is the leading health metric (alert on value and growth rate); backpressure policies = throttle producers / reject at the edge / drop with a counter / shed by priority; fan-out via consumer groups (streams) or exchange→per-consumer queues (brokers) — one slow consumer must never slow the others.
• Kafka vs RabbitMQ: broker (per-message state: ack/nack/priority/TTL/delay, rich routing, many short queues) vs log (retention, replay, throughput, per-key order, multiple groups, stream processing). Do you need to remember what happened, or just get work done? Running both is a legitimate answer.
• Lens: async changes the user contract (owe a status model and a failure path); the consumer's idempotency is the real design; lag predicts incidents and its shape names the cause.

Self-test: Give the after-consumption difference and three consequences. Why must ack follow durable work, and what breaks with long tasks? Design a retry topology with a DLQ and name the two operational rules teams forget. When does a failing message block a whole partition? Choose Kafka or RabbitMQ for: image resizing jobs, order events consumed by four teams.

Quiz Bank

FoundationalWhat is the fundamental difference between a message queue and an event stream, and how does it change design?

After consumption: a queue deletes an acknowledged message; a log retains it for a configured window. Everything else follows. In a queue, consumers compete — each message goes to exactly one worker, so adding consumers drains faster and the message is gone once handled; the mental model is work distribution. In a stream, messages are appended to an ordered, durable log and each consumer group tracks its own offset, so many unrelated systems read the same events independently, at their own pace, and a group created next year can replay from the beginning; the mental model is a shared record of what happened. Design consequences: ordering is per-partition and strict in a log versus weak-under-concurrency in a queue; read scaling is bounded by partition count in a log versus by consumer count in a queue; and adding a new consumer is free in a log but requires routing changes (and loses history) with a queue.

The selection tell: a command to be executed once ("resize this image") belongs on a queue; a fact that several systems may care about now or later ("order placed") belongs on a stream — and reversing this is expensive, because replayed commands re-execute and consumed facts are lost to everyone else.

FoundationalExplain acknowledgment, visibility timeouts, and the three rules they impose on consumers.

A broker holds a message as in flight until the consumer acknowledges it; if the consumer dies, the message becomes available again — this is what makes at-least-once delivery real. Two mechanisms: explicit ack/nack (RabbitMQ — unacked messages are redelivered when the channel closes; nack can requeue immediately or route onward) and visibility timeout (SQS — a received message is hidden for N seconds and reappears unless deleted).

Rule 1: acknowledge only after the work is durable. Acking on receipt turns the system into at-most-once — a crash between ack and completion loses the message with no trace. Rule 2: long tasks must extend the lease. If processing outlives the visibility window without heartbeating (ChangeMessageVisibility, consumer heartbeats), the message reappears and a second consumer starts the same work — the 10.7.2 lease-expiry problem, arriving as duplicate execution.

Rule 3: duplicates are normal, so consumers must be idempotent — redelivery happens on crashes, rebalances, timeouts, and network hiccups, none of which are exceptional (10.4's claim-and-apply in one transaction). Together these are why "we use a queue" is not a design: the design is what the consumer does with duplicates and how the ack boundary is placed relative to durability.

AppliedDesign the retry and dead-letter topology for a payment-notification consumer that calls a flaky partner API.

Layered, with each layer's failure class: (1) In-process retry with exponential backoff and jitter for transient errors (connection resets, 503s, timeouts) — 3 attempts over a few seconds, bounded so the consumer doesn't hold the message for minutes (10.9). (2)

Requeue with delay for longer outages — instead of hot-looping, republish to a delay tier (retry-1m, retry-10m, retry-1h queues/topics, each consumed by a worker that waits then republishes to the main queue), carrying an incremented attempt count in the message header. This preserves the main queue's throughput while a dependency is down, and the tiers make the retry schedule inspectable. (3)

Circuit breaker around the partner call so that during a sustained outage the consumer stops attempting immediately and messages flow to the delay tier without burning latency and connections (10.9). (4) DLQ after N total attempts (say 6 across tiers), preserving the original message, attempt history, last error, and trace id — because triage should not require reproducing the failure.

Operational rules that make it real: alert on DLQ depth > 0 (each message is a customer whose notification failed — an incident signal, not a log line); alert on delay-tier depth growth (a dependency outage in progress); and build a tested redrive tool that replays DLQ messages after a fix, with idempotent consumers making replay safe (10.4).

Poison-message discipline: messages that fail deterministically (malformed payload, unsupported currency) should be routed to the DLQ on the first failure rather than consuming six attempts — distinguish transient from permanent by error classification (9.9.3's operational-vs-programming split, applied to consumers).

InterviewKafka or RabbitMQ? Give the decision framework and two worked examples.

Framework — ask what the message is. RabbitMQ/SQS are brokers: they track per-message state, enabling ack/nack, priority, TTL, delayed delivery, and rich routing (topic/header exchanges) — ideal when each message is a task with a lifecycle, when queues are numerous or short-lived, and when you need per-message operations. The cost of that state is throughput. Kafka/Pulsar/Kinesis are logs: append-only, retained, partitioned, tracking only consumer offsets — ideal when you need retention and replay, very high throughput (sequential writes), strict per-key ordering, multiple independent consumers of the same data, or stream processing. Per-message operations (retry this one later, prioritize that one) are awkward by design.

Example 1 — image resizing jobs: RabbitMQ/SQS. Each job is a command executed once; you want priority (paid users first), per-message retry with delay, and visibility timeouts; nobody replays yesterday's resize requests, and throughput is modest. Example 2 — order events consumed by billing, search, analytics, and notifications: Kafka. It's a fact with four independent consumers whose speeds differ, analytics must replay history when the model changes, per-customer ordering matters, and adding a fifth consumer next quarter must not touch producers.

The honest closing note: mature systems often run both — the event backbone in Kafka, job queues in Rabbit/SQS — and the anti-patterns to avoid are rebuilding per-message retry/priority on Kafka, or discovering that your event history is unreplayable because facts went through a queue.

StaffA team moved order processing to async queues to improve API latency. Latency improved; now customers complain about orders 'disappearing', and support has no way to answer 'what happened to my order?'. Diagnose and design the fix.

Diagnosis: the team changed the contract without building the artifacts asynchrony requires (section 6). Synchronously, a failure was visible at the moment of ordering; asynchronously, the API returns success and any downstream failure is invisible — messages that exhausted retries are presumably sitting in a DLQ nobody watches (or worse, were dropped by an unbounded requeue loop that eventually hit a limit), and there is no per-order status anyone can query. The customer's "disappeared" is precisely accurate: the order exists nowhere the customer or support can see.

The fix — three artifacts, all mandatory for async work. (1) A status model: the order is persisted synchronously at the API with a status (9.5.4's pipeline), and the API returns 202 with a status resource (9.6.1); every stage transition (accepted → paid → reserved → confirmed | failed) is written to that record, so "what happened to my order?" is a lookup, not an investigation, and the customer UI can show real progress. The queue then carries work about a persisted entity, never the only copy of the fact — which is the structural error at the root of "disappearing." (2)

A failure path with an owner: DLQ per stage with alerting on depth, failure context preserved, a redrive tool, and — critically — a customer-visible failure state plus a compensating action (refund the authorization, notify the customer, 10.8.4's saga discipline). Silent DLQ accumulation is the same defect as silent drops. (3)

Observability: trace ids propagated from the API through every consumer so one identifier joins the HTTP request, the messages, and the logs (10.10); dashboards for consumer lag, DLQ depth, and stage-transition latency; an alert on orders stuck in a non-terminal state beyond X minutes — the check that catches every failure mode including ones you didn't anticipate.

The principle to record in the postmortem: asynchrony moves failures from the request path to the background, so it must move failure visibility there too — an async system without a status model, a DLQ policy, and stuck-state alerting hasn't improved latency, it has hidden its defects behind it.

Flashcards

FlashQueue vs stream

Queue: one consumer, deleted on ack — work distribution (commands). Stream: retained log, per-group offsets, replayable, per-partition order — record of facts (events).

FlashAck rules

Ack after durable work (else at-most-once) · extend leases for long tasks (or it reappears mid-work) · duplicates are normal ⇒ idempotent consumers.

FlashRetry topology

Backoff + jitter (not immediate requeue — hot loop) → delay tiers → DLQ after N. Alert on DLQ depth, preserve failure context, build a tested redrive.

FlashHead-of-line blocking

With strict per-partition order, one failing message stalls its partition. Move it to a retry topic (lose strict order for that key) or accept the block deliberately.

FlashConsumer lag

The leading health metric. Steady growth = under-provisioned; step change = slow dependency/poison message; sawtooth = batch producers. Alert on value AND derivative.

FlashAsync's contract debt

Async owes users a status model (202 + status resource) and operators a failure path (DLQ + alert + redrive). Without them you traded visible failures for invisible ones.

Scenario Drill

DrillDesign the messaging architecture for a food-delivery platform: order events consumed by restaurant dispatch, courier assignment, analytics, and notifications; courier location updates at 50k/s; payment captures that must never be lost; and a promotions team that wants to replay last month's orders to test a new pricing model. Choose queue vs stream per flow and justify.

Order events → stream (Kafka), partitioned by order_id. Four independent consumers with different speeds and failure profiles is the textbook fan-out case; partitioning by order id gives per-order ordering (a cancelled event can never be processed before its placed) while spreading load; and the promotions requirement is decisive — replay of last month's orders is only possible with retention, and a queue would have deleted those messages the moment dispatch consumed them. Retention set to cover the replay window (say 30–90 days), with the promotions team running as a new consumer group from an old offset, reading production events at their own pace without touching producers or affecting the other three consumers. That single capability is what makes the stream choice correct rather than fashionable.

Courier locations at 50k/s → stream, but a separate topic with short retention and aggressive settings. Volume dictates the technology (sequential appends handle this; a broker tracking per-message state would not), while the value profile dictates the settings: locations are ephemeral (10.7.1's eventual/lossy classification), so retention is minutes, acks=1 is acceptable, and consumers that fall behind should skip to the latest offset rather than catch up — replaying stale positions is worse than useless. Keeping it off the order topic is essential: mixing a 50k/s firehose with business-critical order events couples their retention, partitioning, and failure behavior.

Payment captures → queue (or a stream with strict guarantees), and the real answer is the outbox. "Must never be lost" is a durability requirement that no messaging choice alone satisfies, because the classic loss happens before the broker: the service writes the payment record and then crashes before publishing. So the capture is written with a transactional outbox (10.8.4) and relayed to a durable queue with acks=all/persistent delivery; the consumer is idempotent on capture id and calls the PSP with a derived idempotency key (10.4). Use a queue here rather than the shared event stream because captures are commands with per-message lifecycle needs (delayed retry against a flaky PSP, priority, DLQ with human triage) — the section 5 tell applied precisely.

Notifications → consumer group on the order stream, with its own retry/DLQ topics, because notification failures must not block dispatch or analytics (independent groups give that isolation for free) and because a failed SMS deserves backoff and eventual DLQ rather than infinite retry.

Cross-cutting decisions to state: consumer lag dashboards per group with alerts on growth rate; DLQ depth alerts with named owners per consumer; trace ids propagated from the ordering API through every consumer (10.10); and an explicit note that the promotions replay must run against a separate consumer group and a read-only sink, so a replay of a month of orders cannot re-trigger dispatch, notifications, or payments — the most common and most embarrassing replay accident, prevented by design rather than by a runbook.