10.8.2 — Kafka Internals

Kafka is the default event backbone of the industry, and it's worth understanding mechanically rather than as a black box — because nearly every operational surprise (rebalance storms, lag that won't drain, ordering violations, "exactly-once" that isn't) follows directly from four design decisions: the partitioned log, the pull-based consumer group, replication with in-sync replicas, and offsets as the only consumer state. This page opens all four, plus the throughput tricks that make it fast and the transactional semantics that make "exactly-once" a bounded claim.

1. The log, partitions, and ordering

A topic is a named stream, split into partitions — each an append-only, ordered, immutable log stored as segment files on disk. A message's position in its partition is its offset (a monotonically increasing integer). Two consequences define everything:

• Ordering is per partition, never per topic. Messages within a partition are strictly ordered; across partitions there is no order at all. So the partition key decides your ordering guarantee: key = order_id means all events for an order are ordered; the topic as a whole is not.
• Partitions are the unit of parallelism. Each partition is consumed by at most one consumer in a group, so a group's maximum useful parallelism equals the partition count. Ten partitions cap you at ten consumers; adding an eleventh gives it nothing to do.

Choosing partition count is therefore a capacity and ordering decision made early and painfully changed later: increasing partitions changes the key→partition mapping (hash(key) mod partitions), so existing keys move and per-key ordering breaks across the boundary — old events for a key sit in the old partition while new ones land elsewhere. Practical guidance: over-provision moderately (more partitions than consumers you expect, within reason — each partition costs file handles, memory, and rebalance time), and treat repartitioning as a migration, not a config change (10.6's fixed-partition lesson).

2. Consumer groups, offsets, and rebalancing

A consumer group is a set of consumers sharing a group.id; Kafka assigns each partition to exactly one member, and the group's progress is stored as committed offsets in an internal topic (__consumer_offsets). Two independent groups on the same topic read everything independently — the fan-out property of 10.8.1.

Offset commit semantics decide your delivery guarantee, and this is the single most consequential consumer setting:

• Commit before processing → at-most-once (crash loses the message).
• Commit after processing → at-least-once (crash re-delivers — the correct default, paired with idempotent consumers, 10.4).
• Auto-commit on a timer (enable.auto.commit=true, the default!) → neither, reliably: offsets advance on a schedule regardless of your processing, so a crash can lose messages (committed but unprocessed) or duplicate them. Turn it off for anything that matters and commit explicitly after work is durable.

Rebalancing is what happens when group membership changes (a consumer joins, leaves, or is deemed dead): partitions are reassigned. Classically this is a stop-the-world event — every consumer pauses, revokes its partitions, and waits for the new assignment — so a rolling deploy of ten consumers can trigger ten rebalances, each stalling consumption ("rebalance storm"). The mitigations to know: static group membership (group.instance.id — a restarting consumer reclaims its partitions without triggering a full reassignment), cooperative/incremental rebalancing (only affected partitions move, the rest keep consuming), and tuning session.timeout.ms/max.poll.interval.ms so a slow-but-alive consumer isn't declared dead mid-batch — the classic cause of a consumer that "keeps restarting and never drains," where the fix is smaller max.poll.records or a longer poll interval, not more consumers.

Consumers pull. Kafka doesn't push to consumers; each consumer polls, which is why backpressure is automatic (a slow consumer simply polls less — 10.8.1), batching is natural, and consumer lag is measurable as a simple subtraction (log end offset − committed offset).

3. Replication, ISR, and durability

Each partition has a leader and followers on other brokers. Producers write to the leader; followers fetch and replicate. The set of replicas that are sufficiently caught up is the ISR (in-sync replicas), and it's the hinge of Kafka's durability model: ^{⚑How does Kafka replication and acks work? [EQ-1097b]}

• acks=0 — fire and forget; fastest, loses data freely.
• acks=1 — the leader has it; loses data if the leader dies before followers replicate.
• acks=all — every in-sync replica has it. Combined with min.insync.replicas=2 (and replication factor 3), this is the durable configuration: a write is acknowledged only when at least two replicas hold it, so one broker failure loses nothing.

The subtlety that produces real data-loss incidents: acks=all alone is not enough — if min.insync.replicas=1 and followers have fallen out of the ISR, "all in-sync replicas" can mean just the leader, and its death loses the data. The two settings must be configured together. The mirror-image control is unclean.leader.election: allowing an out-of-sync replica to become leader trades data loss for availability — default it to false for anything you care about, and know that flipping it during an incident is a decision to discard records.

Figure 1 — Partitions, groups, and durability. Each partition has one leader and a set of in-sync replicas; each partition is consumed by at most one member of a group, so a fourth consumer on a three-partition topic is idle. Durable writes require acks=all and a minimum ISR size.

4. Why it's fast

Kafka's throughput comes from mechanical sympathy with the operating system (2.5/2.7) rather than clever data structures:

• Sequential disk I/O — appends to a log file are sequential writes, which even spinning disks do fast and SSDs do very fast; there is no random-access index to maintain per message.
• Page cache, not application cache — Kafka writes to the OS page cache and lets the kernel flush; recent messages are served from RAM without Kafka managing a cache (2.5), which also means restarting a broker doesn't cold-start its cache.
• Zero-copy (sendfile) — sending a batch to a consumer moves bytes from page cache to socket inside the kernel, never through Kafka's heap (9.9.7's Nginx argument, at broker scale).
• Batching and compression — producers batch by size (batch.size) and time (linger.ms), and compress whole batches (lz4/zstd), which is where most of the network and disk savings come from; the batch stays compressed on disk and is decompressed by the consumer.
• No per-message broker state — the broker doesn't track acknowledgments per message (only offsets per group), which is exactly why it scales where a per-message broker cannot (10.8.1's log-vs-broker distinction).

Retention completes the picture: messages are deleted by time (retention.ms) or size, or the topic is compacted (cleanup.policy=compact — keep only the latest message per key, turning the log into a durable, replayable snapshot of current state; the mechanism behind changelog topics and Kafka-backed materialized views — 10.8.4).

5. Exactly-once semantics, bounded honestly

Kafka's exactly-once semantics (EOS) are real within a well-defined boundary and misunderstood outside it (10.4): ^{⚑What are Kafka transactions and exactly-once semantics? [EQ-498b]}

• Idempotent producer (enable.idempotence=true) — each producer gets a producer id and per-partition sequence numbers, so a retry after a network failure doesn't create a duplicate record. This solves producer-side duplicates and should essentially always be on.
• Transactions — a producer can write to multiple partitions and commit consumer offsets in one atomic transaction, so a consume-transform-produce pipeline either fully happens or doesn't. Consumers reading with isolation.level=read_committed never see uncommitted records.

The boundary: this gives exactly-once processing within Kafka. The moment a side effect leaves that boundary — a payment API call, an email, a write to Postgres — the transaction cannot include it, and you are back to at-least-once plus idempotent effects (10.4). The correct sentence in a design review: "we get exactly-once for Kafka-to-Kafka pipelines; everything touching an external system is at-least-once with idempotency keys."

6. The expert lens

Almost every Kafka operational problem is a partition/consumer arithmetic problem. Lag that won't drain with more consumers → you're at the partition ceiling. Uneven lag across consumers → key skew putting a hot key on one partition (10.6). Consumers restarting forever → max.poll.interval.ms exceeded by slow processing, so the group keeps evicting a live member. Ordering violations → keys not set, or partitions increased. Learning to reach for the partition/consumer/key model first resolves these in minutes rather than escalations.

ZooKeeper is gone (mostly) — know the current shape. Modern Kafka uses KRaft (a built-in Raft quorum — 10.7.2) for metadata instead of ZooKeeper, which simplifies operations and speeds failover. If you learned Kafka from older material, the mental model to update is: metadata now lives in a self-managed Raft log rather than an external coordination service; the data-plane concepts on this page are unchanged.

Kafka is a database you don't query. Retention plus compaction plus replication means Kafka holds your data with real durability guarantees — which is why compacted topics can serve as source-of-truth changelogs and why event sourcing on Kafka is viable (10.8.4). But it has no queries, no indexes, no updates, and no random access: the only reads are sequential scans from an offset. Treating it as a database (storing state you need to look up) leads to painful designs; treating it as a durable, replayable transport and history is what it's for.

Next: 10.8.3 — the architectural layer messaging enables: what a microservice genuinely is, service boundaries, discovery, gateways, BFFs, and sidecars.

Recall

• Topic = partitions; each partition is an ordered append-only log; position = offset. Ordering is per partition (the key decides your guarantee) and partitions are the parallelism unit (a group's consumers ≤ partitions; extra consumers idle). Increasing partitions changes key→partition mapping — a migration, not a setting.
• Consumer groups track committed offsets (in __consumer_offsets); independent groups read the same data. Commit after processing = at-least-once (correct default with idempotent consumers); auto-commit is neither guarantee — disable it. Rebalancing is stop-the-world classically (rebalance storms on rolling deploys) — mitigate with static membership, cooperative rebalancing, and correct max.poll.interval.ms/max.poll.records. Consumers pull ⇒ natural backpressure and simple lag math.
• Durability: leader + followers, ISR; acks=all with min.insync.replicas=2 (RF=3) is the durable pair — acks=all alone can mean "just the leader"; unclean.leader.election=false or you're choosing data loss for availability.
• Speed: sequential I/O, page cache (not app cache), zero-copy sendfile, batching + compression (batch.size/linger.ms), and no per-message broker state. Retention by time/size, or log compaction (latest value per key — durable snapshots, changelogs).
• EOS: idempotent producer (dedupes retries) + transactions (atomic multi-partition writes and offset commits; read_committed consumers) = exactly-once within Kafka. External side effects remain at-least-once + idempotency keys.
• Lens: most operational problems are partition/consumer/key arithmetic; metadata now lives in KRaft, not ZooKeeper; Kafka is a durable replayable log, not a queryable database.

Self-test: Why can't 12 consumers speed up a 6-partition topic? What breaks when you increase partition count? Which two settings make writes durable, and why is one alone insufficient? Name four reasons Kafka is fast. State exactly what Kafka's exactly-once covers and where it stops.

Quiz Bank

FoundationalExplain partitions, keys, ordering, and the parallelism ceiling.

A topic is divided into partitions, each an append-only ordered log; a record's position is its offset. A producer chooses a partition by key (hash(key) mod partitions) or round-robins if no key is set. Two facts follow. Ordering is per partition only — records with the same key land on the same partition and are strictly ordered relative to each other; records on different partitions have no defined order. So "keep events for an order in sequence" is achieved by keying on order_id, and there is no such thing as topic-wide ordering (achievable only with a single partition, which caps throughput to one consumer).

Parallelism is capped by partition count — within a consumer group, each partition is assigned to at most one consumer, so a 6-partition topic supports at most 6 useful consumers; a 7th sits idle. This makes partition count a capacity decision made at creation and painful later:

increasing partitions changes the key→partition mapping, so a key's history is split across old and new partitions and per-key ordering breaks across the change. Practical rule: over-provision partitions moderately (each costs file handles, memory, and rebalance time), and treat repartitioning as a data migration with a key-rehash plan (10.6).

FoundationalHow do consumer offsets and commit strategy determine delivery semantics?

A consumer group's progress is a committed offset per partition, stored in Kafka's internal __consumer_offsets topic — the only consumer state the broker keeps, which is why brokers scale. On restart or rebalance, consumption resumes from the committed offset, so where you commit relative to your work defines the guarantee. Commit before processing → at-most-once: a crash after commit but before completion silently loses the record. Commit after processing → at-least-once: a crash after work but before commit re-delivers, so consumers must be idempotent (10.4) — this is the correct default. Auto-commit (enable.auto.commit=true, the client default) commits on a timer independent of your processing, giving neither guarantee reliably: it can commit records you haven't finished (loss on crash) or re-deliver ones you completed (duplicates). Disable it for anything meaningful and commit explicitly after the work is durable. Advanced variant: with transactions, offsets can be committed inside the same transaction as the produced output, giving exactly-once for Kafka-to-Kafka pipelines (section 5).

AppliedYour consumer group keeps rebalancing and never drains its lag. Diagnose systematically.

The signature — repeated rebalances plus non-draining lag — almost always means consumers are being evicted while alive. Diagnosis path: (1) max.poll.interval.ms exceeded — the consumer fetched a batch (max.poll.records, default 500) and its processing took longer than the allowed interval, so the group coordinator declared it dead and rebalanced; the consumer then finishes, tries to commit, discovers it's been kicked, and rejoins — triggering another rebalance, forever. Fix: reduce max.poll.records, increase max.poll.interval.ms to exceed worst-case batch processing, or move slow work off the poll thread.

(2) Session timeout vs heartbeats — session.timeout.ms too low relative to GC pauses or network jitter causes spurious evictions (3.6.9). (3) Rolling deploys with eager rebalancing — each pod restart triggers a full stop-the-world reassignment; ten pods means ten stalls. Fix:

static group membership (group.instance.id) so restarts reclaim partitions, and cooperative/incremental rebalancing so unaffected partitions keep consuming. (4) Genuine under-provisioning — check whether consumer count already equals partition count; if so, more consumers cannot help and you need more partitions (a migration) or faster processing.

(5) Key skew — if lag is concentrated on one or two partitions, it's a hot key (10.6), not a consumer-count problem. Order of investigation matters: check rebalance logs and poll timings before scaling consumers, because adding consumers to a rebalance-storm makes it worse.

InterviewWhat exactly does Kafka's exactly-once give you, and where does it stop?

Two mechanisms. Idempotent producer (enable.idempotence=true): the producer is assigned a producer id, and each record carries a per-partition sequence number, so a retried send after a network failure is recognized and discarded by the broker — this removes producer-side duplicates and should be on by default.

Transactions: a producer can atomically write to multiple partitions and commit its consumer offsets in one transaction; consumers configured with isolation.level=read_committed never see records from aborted or in-flight transactions. Together these give exactly-once processing for consume-transform-produce pipelines whose inputs and outputs are both Kafka — which is genuinely valuable (Kafka Streams builds on it).

Where it stops: any side effect outside Kafka. A transaction cannot include a payment API call, an email, or a row in Postgres, so if your consumer writes to an external system the guarantee degrades to at-least-once and you need idempotency keys in that system (10.4) — or the transactional-outbox pattern in reverse (10.8.4). The precise sentence for a design review: "Kafka-to-Kafka is exactly-once; anything touching an external system is at-least-once with idempotent effects." Also worth noting: transactions add latency and complexity, so use them where the pipeline's correctness demands it, not by default.

StaffPost-incident: a broker failure lost 40 minutes of events, though the topic had replication factor 3. Explain the likely configuration causes and the correct settings, plus what else you'd change.

RF=3 provides capacity for durability but does not deliver it alone — three configuration failures could each produce this. (1) acks=1 (or acks=0) on producers: the leader acknowledged writes before followers replicated them, so the leader's death discarded its unreplicated tail — the most common cause, and it produces exactly this shape (recent minutes lost, older data intact). Fix: acks=all.

(2) acks=all with min.insync.replicas=1: "all in-sync replicas" can mean only the leader if followers had fallen out of the ISR (slow disks, network, or a rolling restart); the write is acknowledged with one copy and dies with it. Fix: min.insync.replicas=2 with RF=3 — the pair must be configured together, and the trade must be understood (with 2 required, losing 2 brokers makes the partition unavailable for writes rather than silently lossy — usually the correct choice, and one the business should confirm).

(3) unclean.leader.election.enable=true: an out-of-sync replica was promoted, truncating everything it hadn't replicated — availability purchased with data loss. Fix: false for topics with real data. What else to change beyond the settings: producer-side enable.idempotence=true and bounded retries so failover doesn't create duplicates or silently drop on retry exhaustion; ISR shrink alerting (an under-replicated partition is a pre-incident signal, and this outage almost certainly had hours of warning in that metric); broker distribution across failure domains (RF=3 in one rack or AZ is one power event from total loss); a game-day that kills a leader under load and verifies zero loss — because the configuration you believe you have and the one running are different until tested; and — because durability is end-to-end — verifying that critical producers are publishing via the transactional outbox (10.8.4) rather than fire-and-forget after a database commit, since the other classic "lost events" cause is that they were never published at all.

Flashcards

FlashPartitions

Ordered append-only logs; ordering per partition only (key decides); parallelism ≤ partition count; increasing partitions rehashes keys (migration, not a setting).

FlashOffsets & commits

Commit before work = at-most-once; after work = at-least-once (the default, with idempotent consumers); auto-commit = neither reliably — disable it.

FlashRebalance troubles

Live consumers evicted via max.poll.interval.ms; storms on rolling deploys. Fix: static membership, cooperative rebalancing, smaller max.poll.records.

FlashDurable write pair

acks=all AND min.insync.replicas=2 with RF=3; unclean.leader.election=false. acks=all alone can mean "just the leader."

FlashWhy Kafka is fast

Sequential I/O · OS page cache · zero-copy sendfile · batching + compression · no per-message broker state (offsets only).

FlashEOS boundary

Idempotent producer + transactions (multi-partition writes + offset commit) = exactly-once INSIDE Kafka. External side effects: at-least-once + idempotency keys.

Scenario Drill

DrillDesign the Kafka topology for a payments platform: authorization events (must never be lost, strict per-account ordering), fraud scoring (needs replay for model retraining), settlement batches (hourly, huge), and a real-time dashboard. Specify topics, keys, partitions, retention, producer/consumer settings, and the two places you'd deliberately not use Kafka.

Topic and key design. payments.authorizations — keyed by account_id (not payment_id): per-account ordering is the requirement (a reversal must follow its authorization), and account-level keys give good spread while keeping causally related events together (10.3/10.6). Partitions sized for peak throughput and future consumer parallelism — say 48, over-provisioned because raising it later breaks key mapping (section 1). Retention long (90 days) since fraud retraining replays history; consider compaction on a separate payments.account-state topic for current-state snapshots. fraud.scores — keyed by account_id (so scores stay ordered with their inputs and stream-joins are co-partitioned). settlement.batches — hourly, huge payloads: keep the event in Kafka and the payload in object storage (the claim-check pattern — a Kafka record carries a pointer, not a 2 GB batch; broker messages should stay small, and this is the standard resolution). dashboard.metrics — short retention (hours), lossy-tolerant.

Producer settings for authorizations: acks=all, min.insync.replicas=2 with RF=3 across three AZs, enable.idempotence=true, bounded retries with max.in.flight.requests.per.connection=5 (safe with idempotence), and linger.ms small (single-digit) because authorization latency is user-facing — the batching/latency trade made explicitly.

Consumer settings: auto-commit off, commit after durable processing, idempotent consumers keyed by payment_id (10.4); isolation.level=read_committed if producers use transactions; static group membership and cooperative rebalancing so deploys don't stall the payment pipeline (section 2).

Fraud replay runs as a separate consumer group from an old offset into a sandboxed sink — never the production scoring output — the replay-safety rule from 10.8.1's drill. The two places to deliberately not use Kafka. (1)

The synchronous authorization decision itself — the card network needs an answer in ~hundreds of milliseconds; that's a request/response call with timeouts and a circuit breaker (10.9), not a message round trip. Kafka carries the record of what was decided, not the decision path — conflating them adds latency and an availability dependency to the most critical user-facing operation. (2)

Querying a payment's current state — Kafka has no random access (section 6); "show me payment X" belongs to the database (populated from the log), and building a lookup by scanning topics is the classic Kafka-as-database mistake. Adjacent third: settlement's huge payloads stay in object storage with claim-check pointers, as above.

Operational package: ISR-shrink and under-replicated-partition alerts, per-group lag with growth-rate alerting, DLQ topics with owners for each consumer, and a documented rule that changing partition counts on payments.* is a migration requiring a key-rehash plan. The design-doc sentence: Kafka is the durable, replayable record of what happened, partitioned by account for ordering and by AZ for durability — while decisions stay synchronous, lookups stay in the database, and payloads stay in object storage.