10.8.4 — Data Patterns: Saga, Outbox, CQRS, Event Sourcing

Once each service owns its own database (10.8.3), the transactional guarantees you took for granted disappear: no cross-service BEGIN…COMMIT, no joins, no single place holding the truth of a multi-step business process. Four patterns fill that gap, and the Curiosity bank asks for exactly this set. Each below: the tension, the mechanism, the honest cost, and when not to use it — because these are the patterns most frequently adopted for prestige rather than need.

DrillCuriosity #150 (verbatim): SAGA > CQRS > SIDE CAR > API GATEWAY > TRANSACTIONAL OUTBOX

The five named patterns, in one sentence each before the depth below: Saga — a business transaction spanning services, executed as a sequence of local transactions each with a compensating action, because distributed ACID (2PC — 10.7.2) is impractical across services.

CQRS — separating the write model from one or more read models, so each is optimized for its job instead of one schema serving both badly. Sidecar — a per-instance helper process taking over network concerns (mTLS, retries, tracing) so application code stays clean across languages (10.8.3 covered it).

API gateway — a single entry point owning cross-cutting concerns (TLS, authn, rate limiting, routing) for many services (10.8.3). Transactional outbox — writing "the state change" and "the event to publish" in one local transaction, then relaying the event asynchronously, so a database commit and a message publication can never diverge. This page develops the three data-oriented ones plus event sourcing, the pattern CQRS is most often (and most mistakenly) assumed to require.

1. The dual-write problem, and the outbox

Start with the failure that motivates half of this page. A service must update its database and publish an event:

await db.orders.insert(order);        // (1) succeeds
await broker.publish("order.placed", event);   // (2) crashes here → event NEVER sent

Two systems, no shared transaction: a crash between them leaves the database updated and the world uninformed (or, if you publish first, an event about an order that doesn't exist). Retrying doesn't fix it — you can't retry a process that died. This is the dual-write problem, and it silently produces the "missing events" incidents every event-driven system eventually investigates.

Transactional outbox: write the event into your own database, in the same transaction as the state change, then relay it: ^{⚑What is the transactional outbox pattern? [EQ-486b]}

BEGIN;
  INSERT INTO orders (...) VALUES (...);
  INSERT INTO outbox (id, topic, payload, created_at)          -- same transaction
       VALUES (gen_random_uuid(), 'order.placed', '{...}', now());
COMMIT;                                    -- both or neither. Atomic by construction.

A separate relay reads unpublished outbox rows and publishes them, marking them sent (or deleting them). Two relay implementations: polling (simple: SELECT … WHERE published_at IS NULL ORDER BY id LIMIT n, with an index and a bounded batch) and change data capture (CDC — Debezium tailing the database's replication log, which is lower latency and doesn't poll, at the cost of operating CDC infrastructure). Either way the relay is at-least-once (it may publish and crash before marking), so consumers must be idempotent (10.4) — which they must be anyway.

The pattern's cost is modest (one table, one relay, some retention management) and its benefit is categorical: it converts "we might lose events" from a probability into an impossibility. Use it for every event whose loss matters — which, in practice, is most of them. The inbox mirror-image handles the consumer side: record processed message ids in the same transaction as their effects (10.4's claim-and-apply).

2. Saga: distributed transactions without 2PC

A booking spans three services: reserve inventory, charge payment, confirm order. Any step can fail; there is no distributed transaction to roll back. A saga executes the business transaction as a sequence of local transactions, each with a compensating transaction that semantically undoes it: ^{⚑What is the saga pattern and how does compensation work? [EQ-484b]}

Step	Forward action	Compensation
1	reserve seat	release seat
2	charge card	refund charge
3	issue ticket	void ticket + notify

If step 3 fails, the saga runs step 2's and step 1's compensations, in reverse order. Two crucial properties: compensation is semantic, not literal — you cannot un-charge a card, you refund it, which is a new fact appearing in statements (9.4.15's command-undo lesson at system scale); and the intermediate states are visible — for a while, the seat is reserved but unpaid, which means the business must accept (and often design UI for) states that a single ACID transaction would have hidden.

Two coordination styles:

• Orchestration — a coordinator service holds the saga's state machine, calls each participant, and triggers compensations on failure. Advantages: the workflow is explicit and inspectable in one place (9.5.4's status-driven pipeline, distributed); easy to add steps, easy to answer "where is order 4417?" Cost: the orchestrator is a component to build and operate (and a coupling point if it grows business rules for every domain).
• Choreography — no coordinator: each service listens for events and emits its own (OrderPlaced → InventoryReserved → PaymentCharged). Advantages: no central component, maximum decoupling. Costs: the workflow exists nowhere explicitly — understanding it requires reading N services (and changing it requires touching several), cycles are easy to create accidentally, and debugging "why did this stop at step 2?" is genuinely hard.

The mature default: orchestration for anything with more than ~3 steps or with money/compliance involved, choreography for simple event chains. And regardless of style, a saga needs the 9.5.4 machinery: persisted state, idempotent steps (10.4), timeouts per step, retries with backoff, and a terminal "needs human attention" state — because compensation itself can fail, and the honest design says what happens then.

Figure 1 — A saga. Each step is a local transaction that commits immediately; a later failure triggers compensations in reverse order. Compensation is semantic (a refund, not an un-charge), and the temporarily-inconsistent middle states are a design input, not an implementation detail.

3. CQRS: separating reads from writes

CQRS (Command Query Responsibility Segregation) separates the model that changes state from the model(s) that read it. The tension it resolves is real and common: a normalized schema optimized for correct writes serves complex reads badly (joins across many tables, N+1s, expensive aggregations), while a denormalized read model would corrupt easily under writes. ^{⚑What is CQRS and when is it justified? [EQ-483b]}

The spectrum matters more than the label, because "CQRS" spans a huge range of commitment:

1. Separate read/write methods — different code paths, one database. Nearly free, often just good design (9.9.6's read module vs repositories).
2. Separate read/write models — write to normalized tables, read from denormalized views/materialized views in the same database. Modest cost, big query wins.
3. Separate read/write stores — writes to Postgres, reads from Elasticsearch/Redis/a document store, synchronized by events. Now you have eventual consistency between them, a synchronization pipeline, and a rebuild story (10.4's derived state).
4. CQRS + event sourcing — the full version (section 4), and a much larger commitment.

The costs at levels 3–4, stated plainly: eventual consistency becomes user-visible (a user updates a record and the search result still shows the old value — you must design for it: return the write's result directly, or use read-your-writes routing — 10.5); the sync pipeline is a system to operate (lag, failures, replay); and the read model must be rebuildable or it becomes an un-fixable source of truth. Use it when the read and write workloads genuinely differ in shape or scale (search over orders, analytics dashboards, feeds); don't use it because the architecture diagram looks sophisticated — level 1 or 2 covers most applications.

4. Event sourcing

Event sourcing stores state as an append-only sequence of events rather than as current values: instead of an orders row updated in place, you store OrderPlaced, ItemAdded, PaymentReceived, OrderShipped, and derive current state by folding them. 9.7.29's ledger and 9.4.15's command log were this pattern in miniature.

What it genuinely buys: a complete, immutable audit trail (not a side effect — the data model is the audit); temporal queries ("what did this look like on 3 March?" — replay to that point); debugging by replay (reproduce a bug by replaying the exact event sequence); and new read models for free (a new projection replays history rather than backfilling). For domains where how you got here is as important as where you are — finance, insurance, healthcare, compliance-heavy anything — it's the natural fit.

What it genuinely costs: schema evolution is hard (old events are immutable, so version them and upcast on read — forever); queries require projections (you cannot SELECT … WHERE over an event log — every query shape needs a maintained projection, which is why CQRS almost always accompanies it); snapshots become necessary (folding 100k events per read is untenable — snapshot every N events, which adds machinery); eventual consistency between the log and projections; and a steep team learning curve with a small talent pool. The honest verdict: event sourcing is powerful and specific — adopt it for a bounded context where auditability or temporal reasoning is a core requirement, not as an application-wide default. (Note that Kafka's log-compacted topics and 10.8.2's durability make it a plausible event store, though purpose-built stores like EventStoreDB handle per-aggregate streams more naturally.)

5. The expert lens

Outbox is nearly free and nearly always right; the others are trades. If you take one pattern from this page into every event-driven design, take the outbox: it removes an entire class of silent data-loss incidents for the cost of one table and a relay. Saga, CQRS, and event sourcing each impose visible complexity on the team and should be justified per bounded context, with the deferral triggers written down (9.9.6's discipline).

These patterns exist because service boundaries removed the database's guarantees. Sagas replace cross-service transactions; CQRS replaces the joins you can no longer do; outbox replaces the atomicity of "update and notify." Seeing them that way has a design consequence: if a business process needs all three, the boundary may be wrong (10.8.3's transaction test) — a saga with five steps spanning five services is sometimes a signal that two of those services should be one, and re-drawing the boundary can delete more complexity than any pattern adds.

Every one of these makes state derived, so make the rebuild real. Read models, projections, search indexes, and caches must be reconstructible from their source, and that rebuild must be exercised — run it in staging on a schedule, and time it, because "we can always rebuild" is only true if someone has done it recently and knows it takes four hours rather than four days (10.4).

Next: 10.9 — keeping all of this up: timeouts, retries, circuit breakers, bulkheads, and the SLO framework that decides how hard to try.

Recall

• Dual-write problem: DB write + broker publish are two systems with no shared transaction ⇒ a crash between them loses events silently. Transactional outbox: insert the event into an outbox table in the same transaction, relay it asynchronously (polling or CDC/Debezium); relay is at-least-once ⇒ idempotent consumers (10.4). The inbox is its consumer-side mirror. Cheap, categorical benefit — use it for every event whose loss matters.
• Saga: a business transaction as local transactions each with a semantic compensation (refund, not un-charge), run in reverse on failure; intermediate states are visible and must be designed for. Orchestration (explicit state machine, inspectable, default for >3 steps or money) vs choreography (event chains, no coordinator, workflow exists nowhere explicitly). Needs persisted state, idempotency, per-step timeouts, and a human-attention terminal state (compensations fail too).
• CQRS spectrum: separate methods → separate models (views) → separate stores (eventual consistency, sync pipeline, rebuild story) → CQRS + event sourcing. Justify by genuinely different read/write shape or scale; levels 1–2 cover most apps.
• Event sourcing: state as an append-only event log, current state folded from it. Buys audit as the data model, temporal queries, replay debugging, free new projections. Costs: event versioning/upcasting forever, projections required for every query shape, snapshots, eventual consistency, steep learning curve. Adopt per bounded context where history is a core requirement.
• Lens: outbox first (nearly free); the rest are per-context trades. These patterns exist because boundaries removed database guarantees — needing all three may mean the boundary is wrong. All of them make state derived ⇒ exercise the rebuild.

Self-test: Show the dual-write failure and how the outbox makes it impossible. Why is compensation semantic rather than literal, and what does that imply for the UI? Contrast orchestration and choreography with a selection rule. Name the four CQRS levels and the two costs that appear at level 3. Give two things event sourcing buys and three it costs.

Quiz Bank

FoundationalExplain the dual-write problem and how the transactional outbox eliminates it.

The problem: a service must both persist a state change and publish an event about it, but the database and the broker are separate systems with no shared transaction. Whatever order you choose, a crash in between produces divergence: commit-then-publish can lose the event (the world never learns the order exists); publish-then-commit can announce something that never happened. Retries don't help — the process that would retry is the one that died. The symptom in production is "missing events" that nobody can reproduce, appearing only during deploys and crashes.

The outbox: insert the event as a row in your own database, inside the same transaction as the state change — so both commit or neither does, atomically, by construction. A separate relay then publishes unpublished rows and marks them sent, either by polling (SELECT … WHERE published_at IS NULL, indexed, batched) or by CDC (Debezium tailing the replication log — lower latency, more infrastructure). The relay may publish and crash before marking, so delivery is at-least-once and consumers must be idempotent (10.4) — a requirement they already have. Costs: one table, one relay process, and a retention policy for published rows. Benefit: an entire class of silent data loss becomes structurally impossible, which is why this is the one pattern on this page worth adopting by default.

FoundationalWhat is a saga, why is compensation semantic, and what are the two coordination styles?

A saga implements a business transaction spanning services as a sequence of local transactions, each committing immediately, with a compensating transaction defined for each step; if a later step fails, compensations run in reverse order. It exists because cross-service ACID (2PC — 10.7.2) is blocking, availability-multiplying, and unsupported by most modern stores and brokers.

Compensation is semantic, not literal: you cannot un-charge a card or un-send an email — you issue a refund, which is a new fact appearing on the customer's statement, or you send a correction. Two consequences follow: some actions are irreversible (an email, a physical shipment), so the saga's design must place them last or accept that compensation is partial and human-mediated; and intermediate states are visible — a seat reserved but unpaid, an order paid but unconfirmed — which the product must acknowledge in its UI and its policies rather than pretend away.

Styles: orchestration centralizes the workflow in a coordinator holding the state machine — explicit, inspectable, easy to modify and to answer "where is order 4417?", at the cost of a component to build and operate; choreography has each service react to events and emit its own — no central component, maximum decoupling, but the workflow is written nowhere, changing it touches several services, and debugging a stalled saga means reading N codebases. Rule of thumb: orchestrate anything beyond ~3 steps or involving money/compliance; choreograph simple chains.

AppliedPresent the CQRS spectrum and say what changes at each level.

Level 1 — separate read and write methods/paths over one schema: commands go through domain objects with invariants, queries go through a straightforward read module (avoiding the repository method-explosion of 9.9.6). Cost ≈ zero; this is just good layering, and most applications should do it. Level 2 — separate read models in the same database: writes to normalized tables, reads from denormalized or materialized views. Cost: view maintenance and refresh strategy; benefit: complex read queries stop dragging the write schema out of shape. Still transactionally consistent if views are updated in the same transaction. Level 3 — separate read stores (writes in Postgres, reads in Elasticsearch/Redis/a document store), synchronized by events: now three new things appear — eventual consistency becomes user-visible (a just-updated record still reads stale, so you must return write results directly or route reads for the writer, 10.5), the sync pipeline is a system to operate (lag metrics, failure handling, replay), and the read store must be rebuildable from the source or it has quietly become a source of truth (10.4). Level 4 — CQRS + event sourcing: the write model is an event log and read models are projections (section 4), adding versioning, snapshots, and a substantial learning curve. Selection rule: climb only when read and write workloads differ genuinely in shape or scale (search, dashboards, feeds, per-tenant analytics) — and record the trigger for the next level rather than adopting it preemptively.

InterviewWhen is event sourcing the right choice, and what are its real costs?

Right when history is part of the product: regulated finance and insurance (every state change must be explainable years later), healthcare records, anything with audit obligations, and domains needing temporal queries ("what did the policy look like on the day of the accident?") or replay debugging (reproduce a defect by re-running the exact event sequence). It also shines when new read models appear frequently — a new projection replays history instead of requiring a backfill migration.

Real costs, each of which surprises teams: event schema evolution is permanent work — events are immutable, so changes require versioning and upcasting old events on read, forever; you cannot query the log — every query shape needs a maintained projection, which is why CQRS is effectively mandatory alongside it; snapshots are needed once aggregates accumulate thousands of events (fold from the last snapshot, not from the beginning), adding machinery and its own correctness questions; eventual consistency between log and projections, with the read-your-writes handling that implies; operational surface (event store, projection rebuilds, replay tooling); and a team learning curve with a limited hiring pool — the most underestimated cost.

The calibrated verdict: adopt it per bounded context where auditability or temporal reasoning is core (an accounting ledger, a claims history), and keep the rest of the system on conventional state — a hybrid that most successful adopters converge on after trying it everywhere.

StaffA team proposes CQRS + event sourcing + choreographed sagas for a new B2B SaaS product with 5 engineers and no launched customers. Give your assessment and a counter-plan.

Assessment: three high-commitment patterns adopted before any of their tensions exist. Event sourcing pays for auditability and temporal reasoning — requirements a pre-launch product hasn't validated; its costs (event versioning forever, projections for every query, snapshots, learning curve) land immediately and permanently on five engineers who should be finding product-market fit. CQRS at level 3–4 imposes eventual consistency and a sync pipeline before there's a read/write workload asymmetry to justify it. Choreographed sagas are the riskiest of the three for this team: with no central workflow definition, the business process exists only as emergent behavior across services — debugging and changing it will consume the capacity that should go to features, and it's the style even experienced teams regret at small scale (section 2). Compounding all of it: with 5 engineers, the service count that would motivate any of this shouldn't exist yet (10.8.3's staffing lesson).

Counter-plan, staged with triggers. Now: a modular monolith with clear internal boundaries (9.9.6), one database, real transactions — plus the two cheap patterns that cost nothing and preserve options:

the transactional outbox for every externally-meaningful state change (so an event stream exists from day one, giving replay and future consumers for the price of a table), and CQRS level 1–2 (separate read paths and views). Trigger for sagas: the first genuinely cross-boundary process with an external system (payments) — implement it orchestrated, as a persisted state machine inside the monolith (9.5.4's pipeline), which is the same design that survives a later service extraction.

Trigger for CQRS level 3: a read workload that measurably harms the write store (search, analytics) — then add the dedicated read store with a rebuildable projection. Trigger for event sourcing: a specific bounded context where a customer or regulator requires history as a first-class artifact — adopt it there only.

The pitch to the team: none of this plan forecloses their ambitions — the outbox gives them the event log, the orchestrated state machine gives them the saga shape, and the module boundaries give them the extraction seams — while spending their five-engineer year on customers instead of on infrastructure whose justifying problems they have not yet met.

Flashcards

FlashDual write → outbox

DB write + publish = two systems, no shared transaction ⇒ silent event loss. Outbox: event row in the SAME transaction, relayed by polling or CDC (at-least-once ⇒ idempotent consumers).

FlashSaga

Local transactions + semantic compensations, reversed on failure. Intermediate states are visible. Compensation ≠ rollback (refund, not un-charge); some actions are irreversible.

FlashOrchestration vs choreography

Orchestrator: explicit state machine, inspectable, easy to change — default for >3 steps or money. Choreography: event chains, no coordinator, workflow written nowhere.

FlashCQRS levels

1: separate methods · 2: separate models/views · 3: separate stores (eventual consistency + sync pipeline + rebuild) · 4: + event sourcing. Climb on demonstrated asymmetry.

FlashEvent sourcing ledger

Buys: audit as the data model, temporal queries, replay debugging, free new projections. Costs: versioning/upcasting forever, projections for all queries, snapshots, learning curve.

FlashBoundary smell

A 5-step saga across 5 services often means two of them should be one. These patterns replace what boundaries removed — sometimes the cheaper fix is redrawing the boundary.

Scenario Drill

DrillDesign the data architecture for an online travel booking system: a booking reserves a flight seat (external airline API), a hotel room (partner API), and charges the customer; bookings must be auditable for 7 years; agents need a search view across all bookings; and partial failures are common (airline confirms, hotel rejects). Specify the patterns, the failure handling, and what you deliberately keep simple.

The core process is a saga, orchestrated — three steps across two external systems and a payment provider, with partial failure as the normal case, not the exception. An orchestrator owns the booking's state machine (initiated → flight_held → hotel_held → paid → confirmed, plus compensating branches) as a persisted 9.5.4 pipeline: explicit, queryable ("where is booking 88123?"), and changeable in one place — choreography would scatter this seven-state process across services and make the common failure paths invisible.

Step order is a design decision, not an accident: hold the reversible things first (airline hold, hotel hold — both typically support cancellation windows) and take the irreversible or costly-to-reverse action last (the charge), so the majority of failures compensate cleanly without touching money. Each step carries an idempotency key derived from (bookingId, step) and forwarded to the partner API (10.4) — because these APIs time out routinely, and a retried hold that double-books is a customer-facing disaster.

Compensations, with their honesty: release flight hold, release hotel hold, refund payment — all semantic, all logged as new events; and because partner cancellation APIs themselves fail, the saga has a terminal needs_manual_resolution state that alerts an operations queue with full context. Saying that out loud is the mark of a real design: compensation can fail, and pretending otherwise produces bookings that are silently half-made.

The outbox is mandatory here: every state transition writes its event in the same transaction as the booking update, so no confirmation, cancellation, or charge can exist without its corresponding event (section 1) — this is also what feeds everything downstream.

Auditability for 7 years → the event log is the audit (section 4-flavored, without full event sourcing): the ordered, immutable transition log per booking, archived to object storage with retention/immutability policies, answers "what happened and when" for any booking, seven years later — and satisfies the requirement without restructuring the write model into event sourcing (the deliberate simplification, below).

Agent search view → CQRS level 3: a dedicated search store (Elasticsearch) projected from the event stream, since agents need cross-booking queries by traveler, date, partner, and status that would punish the operational database. It is explicitly derived and rebuildable, with the rebuild exercised on a schedule (section 5) and lag monitored — and agents are shown a "last updated" indicator so eventual consistency is visible rather than mysterious.

What is deliberately kept simple: (1) no full event sourcing — the append-only transition log gives the audit and replay value the requirement actually names, without imposing event versioning, projection-only queries, and snapshots on every read path; the trigger to revisit is a requirement for arbitrary temporal reconstruction of all entity state, not just booking history. (2)

No choreography and no service mesh — one orchestrator, HTTP to partners with timeouts/retries/circuit breakers (10.9), and the ordinary observability stack. (3) One database for bookings — the search store is the only derived store, added because a measured workload demanded it. The design-doc sentence: the saga is the product's real complexity, so it gets an explicit, persisted, inspectable state machine; the outbox makes its history unloseable; the search projection is the one derived store, rebuildable and monitored; and the patterns we skipped are listed with the triggers that would bring them back.