10.4 — State Ownership, Delivery Semantics & Idempotency

Three questions always come up together in a distributed system (10.1):

• Who owns each fact? Answering this prevents most data-integrity disasters.
• How many times does a message arrive? And why "exactly once" is mostly a marketing claim.
• How do you make repeats harmless? That is idempotency, and a family of related techniques.

9.6.3 covered these at the HTTP level. Here they are across a whole system.

1. State ownership: one source of truth per fact

The single most useful discipline in distributed design: for every fact in your system, name exactly one component that owns it. Everything else holds a copy, and copies are explicitly labeled as such. Three categories: ^{⚑State ownership and source of truth in distributed systems. [EQ-244b]}

• Source of truth — the authoritative store for a fact. The orders service owns order state; the identity provider owns who a user is; the ledger owns balances.
• Derived state — computed from a source: search indexes, materialized views, aggregates, caches, read models (10.8.4). Derived state may be rebuilt from scratch, which is its defining property and its safety net.
• Ephemeral state — valid only for a moment and reconstructible: session presence, in-flight request context, connection registries.

The failure mode this prevents is dual ownership: two services both writing the same fact (an order's status updated by both the orders service and the fulfilment service; inventory decremented by both the warehouse system and the storefront). Two owners guarantee that the copies drift apart. Not "might", guarantee. No protocol keeps two independent writers in agreement without them coordinating. And if you build that coordination, you have just rebuilt a single owner the long way round.

The cures, best first:

1. Have one owner. One service owns the fact. The others call it, or listen to its events.
2. Split the fact. Each service owns its own field, or its own range of keys.
3. If two writers really cannot be avoided, detect conflicts and resolve them explicitly, with vector clocks or CRDTs (10.3). Never let "the last write wins" decide silently.

The practical review question, worth asking on every design: "If these two stores disagree tomorrow, which one is right, and what process fixes the other?" An answer of "they can't disagree" is always wrong (10.1's partial failure); an answer of "we'd look at both and decide" means you have no owner. And the second question follows it: "What rebuilds the derived copy?" — because a derived store that cannot be rebuilt is a source of truth wearing a cache costume.

2. Delivery semantics: three promises, one honest option

When a message crosses a network, three guarantees are conceivable: ^{⚑At-least-once vs at-most-once vs exactly-once delivery. [EQ-497b]}

• At-most-once — send it, don't retry. Fast, simple, loses messages on any failure. Legitimate for lossy telemetry (metrics samples, non-critical analytics) where a gap is cheaper than a duplicate.
• At-least-once — retry until acknowledged. Never loses, may deliver duplicates (the ack was lost, the consumer was slow, the broker rebalanced). This is what real queues give you, and what you should design for.
• Exactly-once — delivered precisely once. Not achievable as a pure delivery guarantee across an unreliable network with independent failures (the two-generals intuition: the sender can never be certain the receiver got it without an ack, which itself may be lost, forever).

Systems that advertise "exactly once" really give you one of two things. Knowing the difference is the point of the interview question.

(a) Exactly-once processing inside a closed system. Kafka's transactional producer saves the consumer's position in the same transaction as its output. So reading, processing and writing happen as one step inside Kafka (10.8.2). But the moment an effect leaves Kafka, such as a call to a payment API or an email, the guarantee stops.

(b) Effectively-once. Messages are delivered at least once, and consumers are safe to run twice, so a duplicate makes no visible difference. That's the design pattern to internalize, and the reason section 3 exists: you do not prevent duplicates; you make them harmless.

3. Idempotency and its siblings

DrillCuriosity #22 (verbatim): What is Idempotency key? What are other similar and must know concepts like this?

An idempotency key is a client-supplied unique identifier for a logical operation — Idempotency-Key: 3f2a… on a payment request — that lets the server recognize a retry of the same operation and return the original outcome rather than executing again (9.6.3 built the mechanics: atomic claim, stored response replay, in-flight conflict, key/body-hash mismatch). It exists because a network timeout is ambiguous — the client cannot tell "never arrived" from "done, reply lost" — so retrying must be safe. The sibling family, which is what the question is really asking for:

• Deduplication — the consumer-side twin: keep a window of seen message IDs and drop repeats. Differs from idempotency keys in who supplies the identity (producer/message vs client/operation) and in scope (a time window vs a stored outcome). Practically: exactly the same defense, applied at the queue boundary (10.8.1).
• Fencing tokens — a monotonically increasing number issued with a lock or lease; the resource rejects operations bearing a token older than the newest it has seen. This is the fix for the zombie-writer problem: a process that paused (GC, VM freeze) past its lease expiry wakes up and writes with a stale token, which the storage layer refuses (10.7.2). Locks without fencing are advisory hopes; with fencing they're enforceable.
• Compare-and-set / optimistic concurrency — write only if the version matches what you read (UPDATE … WHERE version = 42, HTTP If-Match → 412). Prevents lost updates and makes conflicts visible (9.6.3, [7.4]).
• Idempotent operations by design — the cheapest option: prefer set quantity to 3 over increment by 1, mark as shipped over advance status, absolute over relative. A large share of duplicate-protection work disappears if the operation is naturally repeatable (9.6.3's first defense).
• Natural/business keys — dedupe on a domain-unique value (an invoice number, an external transaction reference) via a unique constraint; the database becomes the dedup mechanism, atomically and durably.
• Sequence numbers / monotonic versions — per-writer counters that let receivers detect gaps (loss) and reorderings, and reject stale writes (10.3).
• Outbox pattern — makes "wrote to the database" and "published the event" atomic, eliminating the other duplicate/loss source: the dual-write problem (10.8.4).
• Sagas with compensation — when an operation can't be retried harmlessly, provide an inverse and undo forward (10.8.4).

The unifying idea worth carrying: every one of these gives the system a way to recognize "I have seen this before" or "this is stale" — identity plus a decision rule. Duplicate protection is always identity (key, version, token, sequence) plus a policy (replay, reject, ignore, merge).

4. Making it work in practice

The consumer-side recipe, which most production message handlers should follow:

async function handleMessage(msg: Message): Promise<void> {
  const key = msg.idempotencyKey ?? msg.id;             // identity: from producer or broker
  await db.transaction(async (tx) => {                  // ONE transaction
    const claimed = await tx.insertIfAbsent("processed_messages", {
      key, processedAt: now(),                          // (1) dedup claim — unique index
    });
    if (!claimed) return;                               // (2) already handled: no-op, ack
    await applyBusinessEffect(tx, msg);                 // (3) the effect — SAME transaction
  });                                                   // (4) commit ⇒ effect and dedup
}                                                       //     record are atomic

The critical property is (4): the dedup record and the business effect commit together. If they're in separate transactions, a crash between them re-opens the duplicate window — the same class of bug as the dual-write problem (10.8.4). Practical details:

• Put a limit on the dedup table. Delete entries after a while, based on when they were processed. A table that grows forever becomes an outage.
• Choose the ID carefully. An ID made by the broker catches redeliveries. It does not catch a producer that published the same thing twice. For that, the producer must supply its own key for the action.
• Make side effects outside the database safe to repeat too, such as an email or a charge at another company. Give them their own keys, passed along to the next service (9.6.3).

Figure 1 — Effectively-once, assembled. The broker promises only at-least-once; duplicates are expected traffic. The consumer's atomic claim-and-apply converts them into no-ops, so the observable state changes exactly once — the achievable version of "exactly-once."

5. The expert lens

"Exactly-once delivery" is a red flag in a design doc; "effectively-once processing" is the credible claim. When a vendor or a colleague promises the former, the useful follow-up is "across which boundary?" — the answer is always a closed system (one broker, one database), and your side effects usually leave it. Designing for at-least-once and investing in idempotent consumers is strictly more robust than hunting for a delivery guarantee that evaporates at the first external call.

Idempotency is a property of the operation, not of the transport. Teams routinely try to buy it with infrastructure (dedup in the broker, a "smart" gateway) and then discover the gap: the broker dedupes redeliveries but not a client's double-click, the gateway dedupes HTTP retries but not a queue consumer's rebalance. Because the ambiguity is end-to-end, the defense must be end-to-end: identity generated where the logical operation originates, carried through every hop, and honored by whoever applies the effect (9.6.3's chain composition).

Ownership questions surface disasters early and cheaply. Most severe data-integrity incidents trace back to a fact with two writers or a derived store nobody could rebuild. Both are design-review-detectable in one minute with two questions — "who owns this fact?" and "what rebuilds this copy?" — which makes them among the highest-value questions an architect can ask habitually. The corollary discipline: derived stores get a rebuild job that is exercised, not just written, because an untested rebuild is a rebuild that fails during the incident when you need it.

Next: 10.5 — the first mechanism that makes state survivable: copies of data, the lag they carry, and the anomalies that lag produces.

Recall

• State ownership: every fact has exactly one source of truth; everything else is derived (rebuildable — its defining property) or ephemeral. Dual ownership guarantees divergence. Review questions: who owns this fact? and what rebuilds this copy? A derived store that can't be rebuilt is a source of truth in disguise.
• Delivery: at-most-once (may lose — fine for telemetry) · at-least-once (may duplicate — what real systems give) · exactly-once (not achievable as pure delivery). What vendors mean: exactly-once processing inside a closed boundary, or effectively-once = at-least-once + idempotent consumers.
• The sibling family (Curiosity #22): idempotency keys · consumer-side dedup windows · fencing tokens (reject stale lease holders — the zombie-writer fix) · CAS/optimistic concurrency · naturally idempotent operations (set, not increment) · natural/business keys with unique constraints · sequence numbers · outbox · sagas with compensation. Unifying idea: identity + a decision rule (replay / reject / ignore / merge).
• Consumer recipe: claim the key and apply the effect in one transaction (separate transactions re-open the duplicate window); bound the dedup table with a TTL; choose identity deliberately (broker message ID ≠ logical operation key); make external side effects idempotent too by forwarding keys downstream.
• Lens: "exactly-once delivery" is a red flag — ask across which boundary; idempotency belongs to the operation, not the transport (end-to-end ambiguity needs end-to-end identity); ownership questions catch disasters in one minute.

Self-test: Give the three state categories and the test that distinguishes derived from source. Why is exactly-once delivery impossible, and what do vendors actually sell? Name six siblings of the idempotency key and what each recognizes. Why must the dedup claim and the effect share a transaction?

Quiz Bank

FoundationalWhat is state ownership, why does dual ownership fail, and what are the cures?

Every fact should have exactly one source of truth — the authoritative writer — with all other stores holding explicitly labeled derived copies (search indexes, caches, read models — defined by being rebuildable) or ephemeral state (presence, in-flight context).

Dual ownership — two services independently writing the same fact — doesn't risk divergence, it guarantees it: without coordination there is no protocol keeping two writers consistent, and once you add that coordination you've rebuilt a single owner with extra latency and failure modes. Typical instances: order status written by both orders and fulfilment; inventory decremented by both warehouse and storefront; a user's email updated in both the identity provider and the CRM.

Cures, best first:

1. Have one owner. Everyone else reads through its API, or listens to its events.
2. Split the fact. Divide it by field or by key range, so each writer owns its own separate slice.
3. If several writers really are needed, detect and resolve conflicts explicitly, with version vectors or CRDTs (10.3). Never let "the last write wins" silently decide.

The two questions that surface these in a design review:

"if these disagree tomorrow, which is right and what fixes the other?" and "what rebuilds the derived copy?"

FoundationalWhy is exactly-once delivery impossible, and what does effectively-once mean?

Because acknowledgment is itself a message that can be lost. A sender that gets no ack cannot distinguish "the receiver never got it" from "the receiver got it and the ack died" — so it must choose: don't retry (at-most-once, may lose) or retry (at-least-once, may duplicate). No amount of protocol removes the ambiguity, because the same problem recurs one level up (an ack of an ack can be lost — the two-generals problem). Systems advertising "exactly once" mean one of two things:

• Exactly-once processing inside a closed boundary. For example, Kafka's transactional producer saves the consumer's position in the same transaction as the output, so consuming, transforming and producing happen as one step inside Kafka (10.8.2). It stops applying the instant an effect leaves that boundary: a charge, an email, or a call to another service.
• Effectively-once. Messages are delivered at least once, and consumers are safe to run twice, so a duplicate changes nothing you can see.

Effectively-once is the design you can actually build, anywhere. Treat duplicates as normal traffic. Give every real action an ID. And make repeats do nothing.

AppliedWrite the idempotent consumer pattern and explain the transactional requirement.

await db.transaction(async (tx) => {
  const claimed = await tx.insertIfAbsent("processed", { key, at: now() });  // unique index
  if (!claimed) return;                    // duplicate: no-op, then ack
  await applyEffect(tx, message);          // the business change — same tx
});                                        // commit makes both atomic

Why it must be one transaction. Suppose the dedup record and the effect are saved in separate transactions, and the process crashes between them. There are two ways that goes wrong:

• The key is marked as processed, but the effect never happened. The message is lost forever, because the retry will be skipped as a duplicate.
• The effect happened, but the key is not marked. The retry does it again.

Saving them together turns "this message's effect exists" and "this message is marked as processed" into one fact that is either all true or all false.

Supporting habits:

• The dedup table needs a rule for deleting old entries. If it grows forever, it becomes a slow outage.
• Which ID you use matters. An ID assigned by the broker protects against redelivery, but not against a producer sending the same action twice. That needs a key from the producer that names the business action.
• Effects outside the database, such as a payment API or an email, cannot join the transaction. So they need their own idempotency keys, passed along (9.6.3). Protection has to be added at every hop, or it does not work at all.

InterviewWhat is a fencing token, and which failure does it solve that locks alone cannot?

A fencing token is a monotonically increasing number handed out with a lock or lease; every write to the protected resource carries its token, and the resource rejects any token lower than the highest it has already seen. It solves the zombie writer problem, which plain locks cannot. Here is how that problem happens, step by step:

1. Process A gets a lease, then freezes. It might be a long garbage-collection pause (3.6.9), a frozen virtual machine or a network split.
2. The freeze lasts long enough for the lease to run out.
3. The lock service cannot tell "frozen" from "dead" (10.3), so it gives the lease to B.
4. A wakes up, still believing it holds the lock, and writes.

Now there are two writers, and the lock has failed at its only job. No lock design can stop this, because the damage happens after the lock was checked.

With fencing, A's write carries token 33 while the storage has already seen B's token 34, so A's write is refused by the resource — the enforcement moves to where the effect lands. This is why "we use Redis for distributed locks" is an incomplete answer: the lock provides mutual exclusion most of the time, and the fencing token is what makes correctness independent of timing assumptions.

StaffA payments platform reports occasional double charges. Architecture: mobile client → API gateway → payments service → queue → PSP-caller worker → external PSP. Retries exist at the client, gateway, and queue. Design the end-to-end idempotency strategy and the investigation.

Investigation first, because the fix depends on which hop duplicates. Give every hop a correlation ID. For each charge attempt, record the key the client sent, how many times the gateway retried, how many times the queue delivered it, and the transaction ID the payment provider returned. Then match the provider's transactions against your own charge records. That gives you the real list of duplicates, and where each one came from. Expect one of three patterns:

• The app submitted twice. You will see two different keys. That is a problem in the app's screens.
• The gateway or the queue retried. You will see the same key, delivered several times. The consumer is not safe to run twice.
• The provider retried after a timeout on its own side. You will see the same key, but two transactions at the provider. The key was never passed on to the provider.

End-to-end strategy — identity is generated once, at the logical origin, and honored at every hop: 1. The mobile app creates the idempotency key when the user taps pay. It is one key per payment, not one per HTTP attempt, so a retry reuses it. The app saves the key on the phone, so if the app restarts in the middle of paying, it carries on with the same key. Greying out the button helps users, but it is not a safety control. 2. The payments API claims the key atomically, and sends back the saved response for repeats (9.6.3). So gateway retries cost nothing. 3. Queuing the work and saving the charge record happen in one transaction, through the outbox (10.8.4). Otherwise a crash between saving and publishing either loses a charge or publishes it twice. 4. The worker is safe to run twice. It claims and applies each charge in one atomic step, keyed by the charge ID (section 4). 5. The call to the payment provider carries an idempotency key built from the charge ID, always the same for the same charge. So even if the worker runs twice, the provider spots the duplicate. This is the last link in the chain, and the one teams most often leave out, because "our queue handles it". 6. Reconciliation is the safety net. A daily job compares the provider's transactions with your own charge records, alerts on any mismatch, and automatically refunds confirmed duplicates (9.4.15). In money systems, the right stance is prevention plus detection, never prevention alone.

The principle to write down: protection against duplicates covers every step, or it does not exist. Every step that can retry must either carry the original ID or be safe to repeat. And the last step, the one you do not control, needs its own key.

Flashcards

FlashThree state categories

Source of truth (one owner per fact) · derived (rebuildable — indexes, caches, read models) · ephemeral. Dual ownership guarantees divergence.

FlashDelivery semantics

At-most-once (may lose) · at-least-once (may duplicate — what you get) · exactly-once (impossible as delivery). Real goal: effectively-once = at-least-once + idempotent consumer.

FlashIdempotency siblings

Dedup windows · fencing tokens (zombie writers) · CAS/optimistic locking · naturally idempotent ops (set not increment) · business keys + unique constraints · sequence numbers · outbox · saga compensation.

FlashConsumer recipe

Claim key + apply effect in ONE transaction; TTL the dedup table; broker message ID ≠ logical operation key; forward keys to external side effects.

FlashFencing token

Monotonic number with the lease; resource rejects lower tokens ⇒ a GC-paused zombie holder's write is refused. Locks alone can't fix post-check pauses.

FlashThe two ownership questions

"If these disagree tomorrow, which is right and what fixes the other?" · "What rebuilds this derived copy?" — one minute, catches most integrity disasters.

Scenario Drill

DrillDesign the state-ownership and delivery model for an e-commerce platform where: orders live in an Orders service, inventory in a Warehouse system (legacy, batch-updated nightly), search in Elasticsearch, the storefront caches product data at the CDN, and analytics consumes everything. Assign ownership for 'available quantity', specify the delivery guarantees per flow, and name the reconciliation processes.

Ownership assignment, fact by fact. Available quantity is the one both sides fight over. The honest answer is to split it into two facts, each with its own owner:

• The warehouse system owns physical stock, meaning what is actually on the shelves. Its own processes update it every night. We cannot write to it, and we should not.
• The orders service owns reservations, meaning what has been promised to customers.

"Available to sell" then becomes a value worked out from the two: physical stock from the last warehouse sync, minus our active reservations. We calculate it, we own it, and we can rebuild it from the two sources at any time. That avoids the classic trap of two systems both subtracting from one number, and the old system does not have to change at all.

Order state is owned solely by Orders. Product catalog data is owned by the PIM/catalog service; the CDN and the storefront hold derived copies with a staleness budget (10.2). Search index is derived from catalog + availability — rebuildable, and that rebuild must be exercised.

Analytics is derived from everything and authoritative for nothing. Delivery guarantees per flow. Order placed → reservation created: synchronous, in-transaction with the order write (same service, same database — no distributed problem, and this is why reservations belong to Orders). Order events → search, analytics, notifications:

at-least-once via the outbox (10.8.4), with idempotent consumers keyed by event id; search indexing is naturally idempotent (upsert by document id — set, not increment: section 3's cheapest defense). Nightly warehouse sync: a batch import, idempotent by construction (it replaces a snapshot rather than applying deltas — the single most valuable property to demand from a legacy integration), with the previous snapshot retained for diffing. Reservation expiry: a scheduled job, must be idempotent and leader-elected so N workers don't double-release (10.7.2, 9.9.7).

Reconciliation processes — three, because derived state drifts and legacy systems lie. (1) Availability reconciliation: after each nightly sync, compare our computed availability against warehouse physical stock plus open reservations; alert on discrepancies beyond a threshold and quarantine affected SKUs from overselling (the drift is expected — the process is what makes it safe). (2) Search index reconciliation: a periodic full rebuild (and a continuous sampling diff) proving the index can be reconstructed from the catalog — the rebuild job runs in staging weekly so it's exercised, not theoretical. (3) Order-payment reconciliation: internal charge records vs PSP transactions daily (sectionStaff's safety net).

The design in short:

• No fact has two writers.
• Every store that is worked out from others says how to rebuild it.
• Every background flow expects duplicates, and is safe to repeat.
• Wherever the truth comes from a system we do not control, a regular job compares the two and alerts on differences.

Joining up with an old system is not really a data problem. It is a question of who owns what, plus a scheduled job that catches and fixes the differences.