10.4 — State Ownership, Delivery Semantics & Idempotency

Three arsenal concerns that always travel together (10.1): who owns a fact (state ownership — the question that prevents most data-integrity disasters), how many times a message arrives (delivery semantics — and why "exactly-once" is mostly a marketing claim), and how to make repeats harmless (idempotency and its family of sibling mechanisms). 9.6.3 gave the HTTP surface; this page gives the distributed-systems substance.

1. State ownership: one source of truth per fact

The single most useful discipline in distributed design: for every fact in your system, name exactly one component that owns it. Everything else holds a copy, and copies are explicitly labeled as such. Three categories: ^{⚑State ownership and source of truth in distributed systems. [EQ-244b]}

• Source of truth — the authoritative store for a fact. The orders service owns order state; the identity provider owns who a user is; the ledger owns balances.
• Derived state — computed from a source: search indexes, materialized views, aggregates, caches, read models (10.8.4). Derived state may be rebuilt from scratch, which is its defining property and its safety net.
• Ephemeral state — valid only for a moment and reconstructible: session presence, in-flight request context, connection registries.

The failure mode this prevents is dual ownership: two services both writing the same fact (an order's status updated by both the orders service and the fulfilment service; inventory decremented by both the warehouse system and the storefront). Dual ownership guarantees divergence — not "might", guarantees, because there is no protocol that keeps two independent writers consistent without coordination, and if you build that coordination you have re-created a single owner with extra steps. The cures, in preference order: collapse (one service owns it, others call or subscribe); partition the fact (each owns a distinct field or key range); or, if genuinely unavoidable, explicit conflict resolution with detection (10.3's vector clocks/CRDTs) — never silent last-write-wins.

The practical review question, worth asking on every design: "If these two stores disagree tomorrow, which one is right, and what process fixes the other?" An answer of "they can't disagree" is always wrong (10.1's partial failure); an answer of "we'd look at both and decide" means you have no owner. And the second question follows it: "What rebuilds the derived copy?" — because a derived store that cannot be rebuilt is a source of truth wearing a cache costume.

2. Delivery semantics: three promises, one honest option

When a message crosses a network, three guarantees are conceivable: ^{⚑At-least-once vs at-most-once vs exactly-once delivery. [EQ-497b]}

• At-most-once — send it, don't retry. Fast, simple, loses messages on any failure. Legitimate for lossy telemetry (metrics samples, non-critical analytics) where a gap is cheaper than a duplicate.
• At-least-once — retry until acknowledged. Never loses, may deliver duplicates (the ack was lost, the consumer was slow, the broker rebalanced). This is what real queues give you, and what you should design for.
• Exactly-once — delivered precisely once. Not achievable as a pure delivery guarantee across an unreliable network with independent failures (the two-generals intuition: the sender can never be certain the receiver got it without an ack, which itself may be lost, forever).

What systems that advertise "exactly-once" actually provide is one of two things, and knowing the difference is the interview's point: (a) exactly-once processing within a closed system — Kafka's transactional producer plus consumer offsets committed in the same transaction gives atomic read-process-write within Kafka (10.8.2); the moment a side effect leaves that boundary (a payment API call, an email), the guarantee doesn't extend to it. (b) effectively-once — at-least-once delivery plus idempotent consumers, so duplicates produce no observable difference. That's the design pattern to internalize, and the reason section 3 exists: you do not prevent duplicates; you make them harmless.

3. Idempotency and its siblings

DrillCuriosity #22 (verbatim): What is Idempotency key? What are other similar and must know concepts like this?

An idempotency key is a client-supplied unique identifier for a logical operation — Idempotency-Key: 3f2a… on a payment request — that lets the server recognize a retry of the same operation and return the original outcome rather than executing again (9.6.3 built the mechanics: atomic claim, stored response replay, in-flight conflict, key/body-hash mismatch). It exists because a network timeout is ambiguous — the client cannot tell "never arrived" from "done, reply lost" — so retrying must be safe. The sibling family, which is what the question is really asking for:

• Deduplication — the consumer-side twin: keep a window of seen message IDs and drop repeats. Differs from idempotency keys in who supplies the identity (producer/message vs client/operation) and in scope (a time window vs a stored outcome). Practically: exactly the same defense, applied at the queue boundary (10.8.1).
• Fencing tokens — a monotonically increasing number issued with a lock or lease; the resource rejects operations bearing a token older than the newest it has seen. This is the fix for the zombie-writer problem: a process that paused (GC, VM freeze) past its lease expiry wakes up and writes with a stale token, which the storage layer refuses (10.7.2). Locks without fencing are advisory hopes; with fencing they're enforceable.
• Compare-and-set / optimistic concurrency — write only if the version matches what you read (UPDATE … WHERE version = 42, HTTP If-Match → 412). Prevents lost updates and makes conflicts visible (9.6.3, [7.4]).
• Idempotent operations by design — the cheapest option: prefer set quantity to 3 over increment by 1, mark as shipped over advance status, absolute over relative. A large share of duplicate-protection work disappears if the operation is naturally repeatable (9.6.3's first defense).
• Natural/business keys — dedupe on a domain-unique value (an invoice number, an external transaction reference) via a unique constraint; the database becomes the dedup mechanism, atomically and durably.
• Sequence numbers / monotonic versions — per-writer counters that let receivers detect gaps (loss) and reorderings, and reject stale writes (10.3).
• Outbox pattern — makes "wrote to the database" and "published the event" atomic, eliminating the other duplicate/loss source: the dual-write problem (10.8.4).
• Sagas with compensation — when an operation can't be retried harmlessly, provide an inverse and undo forward (10.8.4).

The unifying idea worth carrying: every one of these gives the system a way to recognize "I have seen this before" or "this is stale" — identity plus a decision rule. Duplicate protection is always identity (key, version, token, sequence) plus a policy (replay, reject, ignore, merge).

4. Making it work in practice

The consumer-side recipe, which most production message handlers should follow:

async function handleMessage(msg: Message): Promise<void> {
  const key = msg.idempotencyKey ?? msg.id;             // identity: from producer or broker
  await db.transaction(async (tx) => {                  // ONE transaction
    const claimed = await tx.insertIfAbsent("processed_messages", {
      key, processedAt: now(),                          // (1) dedup claim — unique index
    });
    if (!claimed) return;                               // (2) already handled: no-op, ack
    await applyBusinessEffect(tx, msg);                 // (3) the effect — SAME transaction
  });                                                   // (4) commit ⇒ effect and dedup
}                                                       //     record are atomic

The critical property is (4): the dedup record and the business effect commit together. If they're in separate transactions, a crash between them re-opens the duplicate window — the same class of bug as the dual-write problem (10.8.4). Practical details: bound the dedup table (TTL by processed date — a table that grows forever becomes an outage), choose the identity carefully (a broker-generated message ID dedupes redeliveries but not a producer that published twice — for that you need a producer-supplied logical key), and handle side effects outside the database (an email, an external charge) by making them idempotent too, with their own keys forwarded downstream (9.6.3's chain composition).

Figure 1 — Effectively-once, assembled. The broker promises only at-least-once; duplicates are expected traffic. The consumer's atomic claim-and-apply converts them into no-ops, so the observable state changes exactly once — the achievable version of "exactly-once."

5. The expert lens

"Exactly-once delivery" is a red flag in a design doc; "effectively-once processing" is the credible claim. When a vendor or a colleague promises the former, the useful follow-up is "across which boundary?" — the answer is always a closed system (one broker, one database), and your side effects usually leave it. Designing for at-least-once and investing in idempotent consumers is strictly more robust than hunting for a delivery guarantee that evaporates at the first external call.

Idempotency is a property of the operation, not of the transport. Teams routinely try to buy it with infrastructure (dedup in the broker, a "smart" gateway) and then discover the gap: the broker dedupes redeliveries but not a client's double-click, the gateway dedupes HTTP retries but not a queue consumer's rebalance. Because the ambiguity is end-to-end, the defense must be end-to-end: identity generated where the logical operation originates, carried through every hop, and honored by whoever applies the effect (9.6.3's chain composition).

Ownership questions surface disasters early and cheaply. Most severe data-integrity incidents trace back to a fact with two writers or a derived store nobody could rebuild. Both are design-review-detectable in one minute with two questions — "who owns this fact?" and "what rebuilds this copy?" — which makes them among the highest-value questions an architect can ask habitually. The corollary discipline: derived stores get a rebuild job that is exercised, not just written, because an untested rebuild is a rebuild that fails during the incident when you need it.

Next: 10.5 — the first mechanism that makes state survivable: copies of data, the lag they carry, and the anomalies that lag produces.

Recall

• State ownership: every fact has exactly one source of truth; everything else is derived (rebuildable — its defining property) or ephemeral. Dual ownership guarantees divergence. Review questions: who owns this fact? and what rebuilds this copy? A derived store that can't be rebuilt is a source of truth in disguise.
• Delivery: at-most-once (may lose — fine for telemetry) · at-least-once (may duplicate — what real systems give) · exactly-once (not achievable as pure delivery). What vendors mean: exactly-once processing inside a closed boundary, or effectively-once = at-least-once + idempotent consumers.
• The sibling family (Curiosity #22): idempotency keys · consumer-side dedup windows · fencing tokens (reject stale lease holders — the zombie-writer fix) · CAS/optimistic concurrency · naturally idempotent operations (set, not increment) · natural/business keys with unique constraints · sequence numbers · outbox · sagas with compensation. Unifying idea: identity + a decision rule (replay / reject / ignore / merge).
• Consumer recipe: claim the key and apply the effect in one transaction (separate transactions re-open the duplicate window); bound the dedup table with a TTL; choose identity deliberately (broker message ID ≠ logical operation key); make external side effects idempotent too by forwarding keys downstream.
• Lens: "exactly-once delivery" is a red flag — ask across which boundary; idempotency belongs to the operation, not the transport (end-to-end ambiguity needs end-to-end identity); ownership questions catch disasters in one minute.

Self-test: Give the three state categories and the test that distinguishes derived from source. Why is exactly-once delivery impossible, and what do vendors actually sell? Name six siblings of the idempotency key and what each recognizes. Why must the dedup claim and the effect share a transaction?

Quiz Bank

FoundationalWhat is state ownership, why does dual ownership fail, and what are the cures?

Every fact should have exactly one source of truth — the authoritative writer — with all other stores holding explicitly labeled derived copies (search indexes, caches, read models — defined by being rebuildable) or ephemeral state (presence, in-flight context).

Dual ownership — two services independently writing the same fact — doesn't risk divergence, it guarantees it: without coordination there is no protocol keeping two writers consistent, and once you add that coordination you've rebuilt a single owner with extra latency and failure modes. Typical instances: order status written by both orders and fulfilment; inventory decremented by both warehouse and storefront; a user's email updated in both the identity provider and the CRM.

Cures in preference order: (1) collapse — one owner, others read via API or subscribe to its events; (2) partition the fact — split by field or key range so each writer owns a disjoint slice; (3) explicit conflict detection and resolution (10.3's version vectors/CRDTs) if multi-writer is genuinely required — never silent LWW. The two questions that surface these in a design review:

"if these disagree tomorrow, which is right and what fixes the other?" and "what rebuilds the derived copy?"

FoundationalWhy is exactly-once delivery impossible, and what does effectively-once mean?

Because acknowledgment is itself a message that can be lost. A sender that gets no ack cannot distinguish "the receiver never got it" from "the receiver got it and the ack died" — so it must choose: don't retry (at-most-once, may lose) or retry (at-least-once, may duplicate). No amount of protocol removes the ambiguity, because the same problem recurs one level up (an ack of an ack can be lost — the two-generals problem). Systems advertising exactly-once mean one of two things: exactly-once processing within a closed boundary — e.g. Kafka's transactional producer plus offset commits in the same transaction, giving atomic consume-transform-produce inside Kafka (10.8.2) — which stops applying the instant a side effect leaves that boundary (a charge, an email, a call to another service); or effectively-once: at-least-once delivery combined with idempotent consumers, so duplicates produce no observable state change. Effectively-once is the achievable, portable design: assume duplicates are normal traffic, give every logical operation an identity, and make repeats no-ops.

AppliedWrite the idempotent consumer pattern and explain the transactional requirement.

await db.transaction(async (tx) => {
  const claimed = await tx.insertIfAbsent("processed", { key, at: now() });  // unique index
  if (!claimed) return;                    // duplicate: no-op, then ack
  await applyEffect(tx, message);          // the business change — same tx
});                                        // commit makes both atomic

Why one transaction: if the dedup record commits in a separate transaction from the effect, a crash between them leaves an inconsistent pair — either the key is marked processed while the effect never happened (message lost forever, because the retry will be deduped away) or the effect happened while the key is unmarked (the retry re-applies it). Committing them together makes "this message's effect exists" and "this message is recorded as processed" a single fact. Supporting practices: the dedup table needs a TTL/partition-drop policy (unbounded growth is a slow outage); the identity choice matters — a broker-assigned message ID protects against redelivery but not against a producer publishing the same logical operation twice (that needs a producer-supplied business key); and effects outside the database (payment API, email) can't join the transaction, so they need their own idempotency keys forwarded downstream (9.6.3) — protection composes hop by hop or not at all.

InterviewWhat is a fencing token, and which failure does it solve that locks alone cannot?

A fencing token is a monotonically increasing number handed out with a lock or lease; every write to the protected resource carries its token, and the resource rejects any token lower than the highest it has already seen. It solves the zombie-writer problem, which plain locks cannot: process A acquires a lease, then pauses — a long GC pause (3.6.9), a VM freeze, a network partition — long enough for the lease to expire; the lock service, unable to distinguish "paused" from "dead" (10.3's FLP), grants the lease to B; A then wakes up believing it still holds the lock and writes. Two writers, mutual exclusion violated, and no lock implementation can prevent it because the violation happens after the lock check.

With fencing, A's write carries token 33 while the storage has already seen B's token 34, so A's write is refused by the resource — the enforcement moves to where the effect lands. This is why "we use Redis for distributed locks" is an incomplete answer: the lock provides mutual exclusion most of the time, and the fencing token is what makes correctness independent of timing assumptions.

StaffA payments platform reports occasional double charges. Architecture: mobile client → API gateway → payments service → queue → PSP-caller worker → external PSP. Retries exist at the client, gateway, and queue. Design the end-to-end idempotency strategy and the investigation.

Investigation first, because the fix depends on which hop duplicates. Instrument each hop with a correlation id and record, per charge attempt, the client-supplied key, the gateway retry count, the queue delivery count, and the PSP's returned transaction id; then reconcile PSP transactions against internal charge records to build the actual duplicate set and their hop signatures. Expect one of three patterns: client double-submit (two different keys — a UI problem), gateway/queue retry (same key, multiple deliveries — a consumer idempotency gap), or PSP-side retry after a timeout on their ack (same key, two PSP transactions — the missing downstream key).

End-to-end strategy — identity is generated once, at the logical origin, and honored at every hop: (1) the mobile client generates the idempotency key when the user taps pay (not per HTTP attempt — a retry must reuse it) and persists it locally so an app restart mid-payment resumes with the same key; a disabled button is UX, not a control. (2) The API/payments service claims the key atomically and replays the stored response on repeats (9.6.3), so gateway retries are free. (3) The enqueue and the charge record are written in one transaction via the outbox (10.8.4) — otherwise a crash between DB write and publish creates either a lost charge or a duplicate publish. (4) The worker is an idempotent consumer (section 4's atomic claim-and-apply keyed by the charge id). (5) The PSP call carries an idempotency key derived deterministically from the charge id, so even if the worker runs twice, the PSP dedupes — the chain's last link, and the one teams most often omit because "our queue handles it." (6)

Reconciliation as the safety net: a daily job compares PSP transactions to internal charges, alerts on mismatches, and auto-refunds confirmed duplicates (9.4.15's compensation) — because in money systems the correct posture is defense plus detection, never defense alone.

The principle to write down: duplicate protection is end-to-end or absent — every hop that can retry must either carry the original identity or be provably idempotent, and the last hop out of your control needs its own key.

Flashcards

FlashThree state categories

Source of truth (one owner per fact) · derived (rebuildable — indexes, caches, read models) · ephemeral. Dual ownership guarantees divergence.

FlashDelivery semantics

At-most-once (may lose) · at-least-once (may duplicate — what you get) · exactly-once (impossible as delivery). Real goal: effectively-once = at-least-once + idempotent consumer.

FlashIdempotency siblings

Dedup windows · fencing tokens (zombie writers) · CAS/optimistic locking · naturally idempotent ops (set not increment) · business keys + unique constraints · sequence numbers · outbox · saga compensation.

FlashConsumer recipe

Claim key + apply effect in ONE transaction; TTL the dedup table; broker message ID ≠ logical operation key; forward keys to external side effects.

FlashFencing token

Monotonic number with the lease; resource rejects lower tokens ⇒ a GC-paused zombie holder's write is refused. Locks alone can't fix post-check pauses.

FlashThe two ownership questions

"If these disagree tomorrow, which is right and what fixes the other?" · "What rebuilds this derived copy?" — one minute, catches most integrity disasters.

Scenario Drill

DrillDesign the state-ownership and delivery model for an e-commerce platform where: orders live in an Orders service, inventory in a Warehouse system (legacy, batch-updated nightly), search in Elasticsearch, the storefront caches product data at the CDN, and analytics consumes everything. Assign ownership for 'available quantity', specify the delivery guarantees per flow, and name the reconciliation processes.

Ownership assignment, fact by fact. Available quantity is the contested one, and the honest answer is split it into two facts with different owners: the Warehouse system owns physical stock (what exists on shelves — updated nightly by its own processes; we cannot and should not write to it), while the Orders service owns reservations (what has been promised to customers). "Available to sell" is then a derived value — physical (from the last warehouse sync) − active reservations (ours) — computed by us, owned by us, and rebuildable from the two sources. This resolves the classic dual-ownership trap (both systems decrementing one number) without asking the legacy system to change.

Order state is owned solely by Orders. Product catalog data is owned by the PIM/catalog service; the CDN and the storefront hold derived copies with a staleness budget (10.2). Search index is derived from catalog + availability — rebuildable, and that rebuild must be exercised.

Analytics is derived from everything and authoritative for nothing. Delivery guarantees per flow. Order placed → reservation created: synchronous, in-transaction with the order write (same service, same database — no distributed problem, and this is why reservations belong to Orders). Order events → search, analytics, notifications:

at-least-once via the outbox (10.8.4), with idempotent consumers keyed by event id; search indexing is naturally idempotent (upsert by document id — set, not increment: section 3's cheapest defense). Nightly warehouse sync: a batch import, idempotent by construction (it replaces a snapshot rather than applying deltas — the single most valuable property to demand from a legacy integration), with the previous snapshot retained for diffing. Reservation expiry: a scheduled job, must be idempotent and leader-elected so N workers don't double-release (10.7.2, 9.9.7).

Reconciliation processes — three, because derived state drifts and legacy systems lie. (1) Availability reconciliation: after each nightly sync, compare our computed availability against warehouse physical stock plus open reservations; alert on discrepancies beyond a threshold and quarantine affected SKUs from overselling (the drift is expected — the process is what makes it safe). (2) Search index reconciliation: a periodic full rebuild (and a continuous sampling diff) proving the index can be reconstructed from the catalog — the rebuild job runs in staging weekly so it's exercised, not theoretical. (3) Order-payment reconciliation: internal charge records vs PSP transactions daily (sectionStaff's safety net).

The design-doc sentence: no fact has two writers; every derived store names its rebuild path; every asynchronous flow assumes duplicates and is idempotent; and every place where truth is inherited from a system we don't control has a reconciliation job with an alert — because integration with legacy is not a data problem, it's an ownership problem with a scheduled apology.