10.8.4 — Data Patterns: Saga, Outbox, CQRS, Event Sourcing

Once each service owns its own database (10.8.3), the transactional guarantees you took for granted disappear: no cross-service BEGIN…COMMIT, no joins, no single place holding the truth of a multi-step business process. Four patterns fill that gap, and the Curiosity bank asks for exactly this set. Each below: the tension, the mechanism, the honest cost, and when not to use it — because these are the patterns most frequently adopted for prestige rather than need.

DrillCuriosity #150 (verbatim): SAGA > CQRS > SIDE CAR > API GATEWAY > TRANSACTIONAL OUTBOX

The five named patterns, in one sentence each before the depth below: Saga — a business transaction spanning services, executed as a sequence of local transactions each with a compensating action, because distributed ACID (2PC — 10.7.2) is impractical across services.

CQRS — separating the write model from one or more read models, so each is optimized for its job instead of one schema serving both badly. Sidecar — a per-instance helper process taking over network concerns (mTLS, retries, tracing) so application code stays clean across languages (10.8.3 covered it).

API gateway — a single entry point owning cross-cutting concerns (TLS, authn, rate limiting, routing) for many services (10.8.3). Transactional outbox — writing "the state change" and "the event to publish" in one local transaction, then relaying the event asynchronously, so a database commit and a message publication can never diverge. This page develops the three data-oriented ones plus event sourcing, the pattern CQRS is most often (and most mistakenly) assumed to require.

1. The dual-write problem, and the outbox

Start with the failure that motivates half of this page. A service must update its database and publish an event:

await db.orders.insert(order);        // (1) succeeds
await broker.publish("order.placed", event);   // (2) crashes here → event NEVER sent

Two systems, no shared transaction: a crash between them leaves the database updated and the world uninformed (or, if you publish first, an event about an order that doesn't exist). Retrying doesn't fix it — you can't retry a process that died. This is the dual-write problem, and it silently produces the "missing events" incidents every event-driven system eventually investigates.

Transactional outbox: write the event into your own database, in the same transaction as the state change, then relay it: ^{⚑What is the transactional outbox pattern? [EQ-486b]}

BEGIN;
  INSERT INTO orders (...) VALUES (...);
  INSERT INTO outbox (id, topic, payload, created_at)          -- same transaction
       VALUES (gen_random_uuid(), 'order.placed', '{...}', now());
COMMIT;                                    -- both or neither. Atomic by construction.

A separate relay reads unpublished outbox rows and publishes them, marking them sent (or deleting them). Two relay implementations: polling (simple: SELECT … WHERE published_at IS NULL ORDER BY id LIMIT n, with an index and a bounded batch) and change data capture (CDC — Debezium tailing the database's replication log, which is lower latency and doesn't poll, at the cost of operating CDC infrastructure). Either way the relay is at-least-once (it may publish and crash before marking), so consumers must be idempotent (10.4) — which they must be anyway.

The pattern's cost is modest (one table, one relay, some retention management) and its benefit is categorical: it converts "we might lose events" from a probability into an impossibility. Use it for every event whose loss matters — which, in practice, is most of them. The inbox mirror-image handles the consumer side: record processed message ids in the same transaction as their effects (10.4's claim-and-apply).

2. Saga: distributed transactions without 2PC

A booking spans three services: reserve inventory, charge payment, confirm order. Any step can fail; there is no distributed transaction to roll back. A saga executes the business transaction as a sequence of local transactions, each with a compensating transaction that semantically undoes it: ^{⚑What is the saga pattern and how does compensation work? [EQ-484b]}

Step	Forward action	Compensation
1	reserve seat	release seat
2	charge card	refund charge
3	issue ticket	void ticket + notify

If step 3 fails, the saga runs step 2's and step 1's compensations, in reverse order. Two properties are crucial. First, compensation undoes the meaning, not the action itself. You cannot un-charge a card. You refund it, and that refund is a new line on the customer's statement. This is the undo lesson from 9.4.15, at the scale of a whole system. Second, the in-between states are visible. For a while, the seat is reserved but not paid for. So the business must accept those in-between states, and often design screens for them, where a single ACID transaction would have hidden them.

Two coordination styles:

• Orchestration — a coordinator service holds the saga's state machine, calls each participant, and triggers compensations on failure. Advantages: the workflow is explicit and inspectable in one place (9.5.4's status-driven pipeline, distributed); easy to add steps, easy to answer "where is order 4417?" Cost: the orchestrator is a component to build and operate (and a coupling point if it grows business rules for every domain).
• Choreography — no coordinator: each service listens for events and emits its own (OrderPlaced → InventoryReserved → PaymentCharged). Advantages: no central component, maximum decoupling. Costs: the workflow exists nowhere explicitly — understanding it requires reading N services (and changing it requires touching several), cycles are easy to create accidentally, and debugging "why did this stop at step 2?" is genuinely hard.

The mature default: orchestration for anything with more than ~3 steps or with money/compliance involved, choreography for simple event chains. And regardless of style, a saga needs the 9.5.4 machinery: persisted state, idempotent steps (10.4), timeouts per step, retries with backoff, and a terminal "needs human attention" state — because compensation itself can fail, and the honest design says what happens then.

Figure 1 — A saga. Each step is a local transaction that commits immediately; a later failure triggers compensations in reverse order. Compensation is semantic (a refund, not an un-charge), and the temporarily-inconsistent middle states are a design input, not an implementation detail.

3. CQRS: separating reads from writes

CQRS (Command Query Responsibility Segregation) separates the model that changes state from the model(s) that read it. The tension it resolves is real and common: a normalized schema optimized for correct writes serves complex reads badly (joins across many tables, N+1s, expensive aggregations), while a denormalized read model would corrupt easily under writes. ^{⚑What is CQRS and when is it justified? [EQ-483b]}

The spectrum matters more than the label, because "CQRS" spans a huge range of commitment:

1. Separate read/write methods — different code paths, one database. Nearly free, often just good design (9.9.6's read module vs repositories).
2. Separate read/write models — write to normalized tables, read from denormalized views/materialized views in the same database. Modest cost, big query wins.
3. Separate read/write stores — writes to Postgres, reads from Elasticsearch/Redis/a document store, synchronized by events. Now you have eventual consistency between them, a synchronization pipeline, and a rebuild story (10.4's derived state).
4. CQRS + event sourcing — the full version (section 4), and a much larger commitment.

The costs at levels 3 and 4, said plainly:

• Users can see the delay. A user updates a record, and the search still shows the old value. You have to design for that. Either return the result of the write directly, or send that user's reads to the copy that is already up to date (10.5).
• The sync pipeline is one more system to run, with lag, failures and replays.
• You must be able to rebuild the read model from the source. Otherwise it quietly becomes a second source of truth that you cannot fix.

Use it when reads and writes really differ in shape or size, such as searching orders, analytics dashboards or feeds. Do not use it because the diagram looks impressive. Level 1 or 2 is enough for most applications.

4. Event sourcing

Event sourcing stores state as an append-only sequence of events rather than as current values: instead of an orders row updated in place, you store OrderPlaced, ItemAdded, PaymentReceived, OrderShipped, and derive current state by folding them. 9.7.29's ledger and 9.4.15's command log were this pattern in miniature.

What it really buys:

• A complete audit trail that cannot be changed. It is not an extra feature. The data itself is the audit.
• Questions about the past, such as "what did this look like on 3 March?". Replay the events up to that date.
• Debugging by replay. Reproduce a bug by replaying exactly the same events.
• New read models for free. A new view replays the whole history, instead of needing a one-off data migration.

For domains where how you got here is as important as where you are — finance, insurance, healthcare, compliance-heavy anything — it's the natural fit.

What it really costs:

• Changing the event format is hard. Old events can never be changed, so you version them and convert old ones each time you read them, forever.
• Every query needs a projection. You cannot run SELECT … WHERE over a log of events. Each kind of query needs its own view, kept up to date. That is why CQRS almost always comes with it.
• Snapshots become necessary. Replaying 100,000 events on every read is far too slow. So you save a snapshot every N events, which is more machinery.
• The views lag behind the log.
• The team has a lot to learn, and few people for hire already know it.

The honest verdict: event sourcing is powerful, but only for specific cases. Use it for the one part of the business where an audit trail or reasoning about the past is a core need. Do not make it the default for the whole application. Kafka's compacted topics and its durability (10.8.2) make it a possible event store. Stores built for the job, such as EventStoreDB, handle one stream per entity more naturally.

5. The expert lens

Outbox is nearly free and nearly always right; the others are trades. If you take one pattern from this page into every event-driven design, take the outbox: it removes an entire class of silent data-loss incidents for the cost of one table and a relay. Saga, CQRS, and event sourcing each impose visible complexity on the team and should be justified per bounded context, with the deferral triggers written down (9.9.6's discipline).

These patterns exist because service boundaries removed the database's guarantees. Sagas replace cross-service transactions; CQRS replaces the joins you can no longer do; outbox replaces the atomicity of "update and notify." Seeing them this way changes how you design. If one business process needs all three, the boundary may be in the wrong place (10.8.3). A saga with five steps across five services sometimes means that two of those services should really be one. Moving the boundary can remove more complexity than any pattern adds.

Every one of these makes state derived, so make the rebuild real. Read models, projections, search indexes, and caches must be reconstructible from their source, and that rebuild must be exercised — run it in staging on a schedule, and time it, because "we can always rebuild" is only true if someone has done it recently and knows it takes four hours rather than four days (10.4).

Next: 10.9 — keeping all of this up: timeouts, retries, circuit breakers, bulkheads, and the SLO framework that decides how hard to try.

Recall

• Dual-write problem: DB write + broker publish are two systems with no shared transaction ⇒ a crash between them loses events silently. Transactional outbox: insert the event into an outbox table in the same transaction, relay it asynchronously (polling or CDC/Debezium); relay is at-least-once ⇒ idempotent consumers (10.4). The inbox is its consumer-side mirror. Cheap, categorical benefit — use it for every event whose loss matters.
• Saga: a business transaction as local transactions each with a semantic compensation (refund, not un-charge), run in reverse on failure; intermediate states are visible and must be designed for. Orchestration (explicit state machine, inspectable, default for >3 steps or money) vs choreography (event chains, no coordinator, workflow exists nowhere explicitly). Needs persisted state, idempotency, per-step timeouts, and a human-attention terminal state (compensations fail too).
• CQRS spectrum: separate methods → separate models (views) → separate stores (eventual consistency, sync pipeline, rebuild story) → CQRS + event sourcing. Justify by genuinely different read/write shape or scale; levels 1–2 cover most apps.
• Event sourcing: state as an append-only event log, current state folded from it. Buys audit as the data model, temporal queries, replay debugging, free new projections. Costs: event versioning/upcasting forever, projections required for every query shape, snapshots, eventual consistency, steep learning curve. Adopt per bounded context where history is a core requirement.
• Lens: outbox first (nearly free); the rest are per-context trades. These patterns exist because boundaries removed database guarantees — needing all three may mean the boundary is wrong. All of them make state derived ⇒ exercise the rebuild.

Self-test: Show the dual-write failure and how the outbox makes it impossible. Why is compensation semantic rather than literal, and what does that imply for the UI? Contrast orchestration and choreography with a selection rule. Name the four CQRS levels and the two costs that appear at level 3. Give two things event sourcing buys and three it costs.

Quiz Bank

FoundationalExplain the dual-write problem and how the transactional outbox eliminates it.

The problem: a service must both persist a state change and publish an event about it, but the database and the broker are separate systems with no shared transaction. Whatever order you choose, a crash in between produces divergence: commit-then-publish can lose the event (the world never learns the order exists); publish-then-commit can announce something that never happened. Retries don't help — the process that would retry is the one that died. The symptom in production is "missing events" that nobody can reproduce, appearing only during deploys and crashes.

The outbox: insert the event as a row in your own database, inside the same transaction as the state change — so both commit or neither does, atomically, by construction. A separate relay then publishes unpublished rows and marks them sent, either by polling (SELECT … WHERE published_at IS NULL, indexed, batched) or by CDC (Debezium tailing the replication log — lower latency, more infrastructure). The relay may publish and crash before marking, so delivery is at-least-once and consumers must be idempotent (10.4) — a requirement they already have. Costs: one table, one relay process, and a retention policy for published rows. Benefit: an entire class of silent data loss becomes structurally impossible, which is why this is the one pattern on this page worth adopting by default.

FoundationalWhat is a saga, why is compensation semantic, and what are the two coordination styles?

A saga implements a business transaction spanning services as a sequence of local transactions, each committing immediately, with a compensating transaction defined for each step; if a later step fails, compensations run in reverse order. It exists because cross-service ACID (2PC — 10.7.2) is blocking, availability-multiplying, and unsupported by most modern stores and brokers.

Compensation is semantic, not literal: you cannot un-charge a card or un-send an email — you issue a refund, which is a new fact appearing on the customer's statement, or you send a correction. Two things follow:

• Some actions cannot be undone, such as an email or a parcel that has shipped. So the saga must put them last. Otherwise you must accept that undoing them is partial, and needs a person.
• The in-between states are visible: a seat reserved but not paid, or an order paid but not confirmed. The product must admit them, in its screens and its policies, instead of pretending they do not exist.

Two styles:

• Orchestration puts the whole workflow in one coordinator, which holds the state machine. The flow is written down in one place, easy to inspect and easy to change. It is easy to answer "where is order 4417?". The cost is one more component to build and run.
• Choreography has each service react to events and send out its own. There is no central component, and services are as independent as they can be. But the workflow is written down nowhere. Changing it means changing several services, and debugging a stuck saga means reading several codebases.

Rule of thumb: orchestrate anything longer than about 3 steps, or anything involving money or regulations. Use choreography for simple chains.

AppliedPresent the CQRS spectrum and say what changes at each level.

Level 1: separate read and write paths, over one schema. Writes go through domain objects that enforce the rules. Reads go through a simple read module, which keeps repositories from filling up with query methods (9.9.6). It costs almost nothing. This is just good layering, and most applications should do it.

Level 2: separate read models in the same database. Writes go to normalised tables. Reads come from flattened tables or materialised views. The cost is keeping those views up to date. The benefit is that complicated reads stop bending the write tables out of shape. If the views are updated in the same transaction, everything is still consistent.

Level 3: separate stores for reading, for example writes in Postgres and reads in Elasticsearch, Redis or a document store, kept in sync by events. Now three new problems appear:

• Users can see the delay. A record they just updated still shows the old value. So you must return the result of the write directly, or send that user's reads to an up-to-date copy (10.5).
• The sync pipeline is one more system to run, with lag metrics, failure handling and replays.
• The read store must be rebuildable from the source. Otherwise it has quietly become a second source of truth (10.4).

Level 4: CQRS plus event sourcing. The write side is a log of events, and the read models are views built from it (section 4). That adds versioning, snapshots and a lot to learn.

How to choose: move up a level only when reads and writes truly differ in shape or size, such as search, dashboards, feeds or per-customer analytics. Write down what would trigger the next level, instead of adopting it early.

InterviewWhen is event sourcing the right choice, and what are its real costs?

Right when history is part of the product: regulated finance and insurance (every state change must be explainable years later), healthcare records, anything with audit obligations, and domains needing temporal queries ("what did the policy look like on the day of the accident?") or replay debugging (reproduce a defect by re-running the exact event sequence). It also shines when new read models appear frequently — a new projection replays history instead of requiring a backfill migration.

The real costs, each of which surprises teams:

• Changing the event format is work that never ends. Events can never be changed, so every change means a new version, and old events converted (upcast) every time they are read, forever.
• You cannot query the log. Every kind of query needs its own projection, kept up to date. That is why CQRS is effectively required alongside it.
• Snapshots become necessary once an entity has thousands of events. You start from the last snapshot instead of the beginning. That is more machinery, with its own ways to go wrong.
• The projections lag behind the log, and users who just made a change need special handling to see it.
• More to operate: the event store, rebuilding projections, and replay tools.
• The team has a lot to learn, and few people for hire already know it. This is the most underestimated cost.

The calibrated verdict: adopt it per bounded context where auditability or temporal reasoning is core (an accounting ledger, a claims history), and keep the rest of the system on conventional state — a hybrid that most successful adopters converge on after trying it everywhere.

StaffA team proposes CQRS + event sourcing + choreographed sagas for a new B2B SaaS product with 5 engineers and no launched customers. Give your assessment and a counter-plan.

Assessment: three high-commitment patterns adopted before any of their tensions exist. Event sourcing pays for auditability and temporal reasoning — requirements a pre-launch product hasn't validated; its costs (event versioning forever, projections for every query, snapshots, learning curve) land immediately and permanently on five engineers who should be finding product-market fit. CQRS at level 3–4 imposes eventual consistency and a sync pipeline before there's a read/write workload asymmetry to justify it. Choreographed sagas are the riskiest of the three for this team: with no central workflow definition, the business process exists only as emergent behavior across services — debugging and changing it will consume the capacity that should go to features, and it's the style even experienced teams regret at small scale (section 2). Compounding all of it: with 5 engineers, the service count that would motivate any of this shouldn't exist yet (10.8.3's staffing lesson).

Counter-plan, staged with triggers. Now: a modular monolith with clear internal boundaries (9.9.6), one database, real transactions — plus the two cheap patterns that cost nothing and preserve options:

the transactional outbox for every externally-meaningful state change (so an event stream exists from day one, giving replay and future consumers for the price of a table), and CQRS level 1–2 (separate read paths and views). Trigger for sagas: the first genuinely cross-boundary process with an external system (payments) — implement it orchestrated, as a persisted state machine inside the monolith (9.5.4's pipeline), which is the same design that survives a later service extraction.

Trigger for CQRS level 3: a read workload that measurably harms the write store (search, analytics) — then add the dedicated read store with a rebuildable projection. Trigger for event sourcing: a specific bounded context where a customer or regulator requires history as a first-class artifact — adopt it there only.

The pitch to the team: none of this plan forecloses their ambitions — the outbox gives them the event log, the orchestrated state machine gives them the saga shape, and the module boundaries give them the extraction seams — while spending their five-engineer year on customers instead of on infrastructure whose justifying problems they have not yet met.

Flashcards

FlashDual write → outbox

DB write + publish = two systems, no shared transaction ⇒ silent event loss. Outbox: event row in the SAME transaction, relayed by polling or CDC (at-least-once ⇒ idempotent consumers).

FlashSaga

Local transactions + semantic compensations, reversed on failure. Intermediate states are visible. Compensation ≠ rollback (refund, not un-charge); some actions are irreversible.

FlashOrchestration vs choreography

Orchestrator: explicit state machine, inspectable, easy to change — default for >3 steps or money. Choreography: event chains, no coordinator, workflow written nowhere.

FlashCQRS levels

1: separate methods · 2: separate models/views · 3: separate stores (eventual consistency + sync pipeline + rebuild) · 4: + event sourcing. Climb on demonstrated asymmetry.

FlashEvent sourcing ledger

Buys: audit as the data model, temporal queries, replay debugging, free new projections. Costs: versioning/upcasting forever, projections for all queries, snapshots, learning curve.

FlashBoundary smell

A 5-step saga across 5 services often means two of them should be one. These patterns replace what boundaries removed — sometimes the cheaper fix is redrawing the boundary.

Scenario Drill

DrillDesign the data architecture for an online travel booking system: a booking reserves a flight seat (external airline API), a hotel room (partner API), and charges the customer; bookings must be auditable for 7 years; agents need a search view across all bookings; and partial failures are common (airline confirms, hotel rejects). Specify the patterns, the failure handling, and what you deliberately keep simple.

The core process is a saga, orchestrated — three steps across two external systems and a payment provider, with partial failure as the normal case, not the exception. An orchestrator owns the booking's state machine (initiated → flight_held → hotel_held → paid → confirmed, plus compensating branches) as a persisted 9.5.4 pipeline: explicit, queryable ("where is booking 88123?"), and changeable in one place — choreography would scatter this seven-state process across services and make the common failure paths invisible.

Step order is a design decision, not an accident: hold the reversible things first (airline hold, hotel hold — both typically support cancellation windows) and take the irreversible or costly-to-reverse action last (the charge), so the majority of failures compensate cleanly without touching money. Each step carries an idempotency key derived from (bookingId, step) and forwarded to the partner API (10.4) — because these APIs time out routinely, and a retried hold that double-books is a customer-facing disaster.

Compensations, with their honesty: release flight hold, release hotel hold, refund payment — all semantic, all logged as new events; and because partner cancellation APIs themselves fail, the saga has a terminal needs_manual_resolution state that alerts an operations queue with full context. Saying that out loud is the mark of a real design: compensation can fail, and pretending otherwise produces bookings that are silently half-made.

The outbox is mandatory here: every state transition writes its event in the same transaction as the booking update, so no confirmation, cancellation, or charge can exist without its corresponding event (section 1) — this is also what feeds everything downstream.

Keeping an audit trail for 7 years: the event log is the audit. It borrows from section 4, without full event sourcing. Each booking has an ordered log of its state changes, which can never be edited. It is archived to object storage with rules that stop it being changed or deleted early. It answers "what happened, and when?" for any booking, seven years later. And it meets the requirement without rebuilding the write side as event sourcing, which is the deliberate simplification below.

Agent search view → CQRS level 3: a dedicated search store (Elasticsearch) projected from the event stream, since agents need cross-booking queries by traveler, date, partner, and status that would punish the operational database. It is explicitly derived and rebuildable, with the rebuild exercised on a schedule (section 5) and lag monitored — and agents are shown a "last updated" indicator so eventual consistency is visible rather than mysterious.

What is kept simple, on purpose:

(1) No full event sourcing. The append-only log of state changes gives the audit and replay the requirement actually asks for. It does not force event versions, query-only views and snapshots onto every read. Revisit this only if someone needs to rebuild every entity's state at any past moment, not just booking history. (2)

No choreography and no service mesh — one orchestrator, HTTP to partners with timeouts/retries/circuit breakers (10.9), and the ordinary observability stack. (3) One database for bookings — the search store is the only derived store, added because a measured workload demanded it. The design in short:

• The saga is where the product's real complexity lives, so it gets an explicit state machine that is saved and can be inspected.
• The outbox means its history can never be lost.
• The search view is the only store worked out from others, and it can be rebuilt and is monitored.
• The patterns we skipped are listed, each with what would make us add it.