10.24 — Distributed Transactions: 2PC, Sagas, Outbox & Inbox

A local transaction relies on a single database's ACID properties. When you split a monolith into microservices or shard a database, a single business workflow—like placing an e-commerce order—now requires updating multiple independent data stores. You can no longer wrap the operation in a simple BEGIN ... COMMIT block. If the Payment service succeeds but the Inventory service crashes, your system is in an inconsistent state.

The tools below split into two families. Two-phase commit makes several systems commit together, at the price of waiting. Sagas, the outbox and the inbox give up "together" and instead make every step safe to retry and possible to undo.

Two-Phase Commit (2PC) and Three-Phase Commit (3PC)

Two-Phase Commit (2PC) is a distributed algorithm that ensures all nodes participating in a transaction either commit or abort simultaneously. It relies on a central Coordinator.

2PC Mechanism

1. Prepare Phase: The coordinator asks all nodes, "Are you ready to commit?" The nodes acquire local locks and reply "Yes" or "No".
2. Commit/Abort Phase: If all nodes reply "Yes", the coordinator broadcasts "Commit". If any node replies "No" (or times out), it broadcasts "Abort".

Sequence diagram showing a Two-Phase Commit

Why 2PC fails at scale:

• Blocking: Nodes must hold database locks from the Prepare phase until the Commit phase. This destroys throughput.
• Single Point of Failure (SPOF): If the coordinator crashes after nodes vote "Yes" but before sending "Commit", the nodes are stuck indefinitely holding locks (a state called blocking).

Three-Phase Commit (3PC) tries to remove the blocking by adding a "pre-commit" round, so that a participant which times out can decide on its own. The catch is in the fine print: that decision is only safe if the network never splits. If a partition separates participants, one side can time out and commit while the other times out and aborts, which is exactly the inconsistency 2PC existed to prevent. Real networks do split, so 3PC is taught but almost never deployed.

Where 2PC really lives today. It is wrong to say 2PC has been abandoned. What has been abandoned is 2PC across independent services, where one slow or crashed coordinator freezes everybody. Inside distributed databases it is everywhere: Google Spanner, CockroachDB and YugabyteDB commit multi-shard transactions with 2PC, but each participant and the coordinator are themselves Raft or Paxos groups (10.7.2), so no single machine failure can leave a transaction stuck. The rule of thumb: inside one database product, let it run 2PC for you; across services you own separately, use the saga below.

The Saga Pattern

A Saga breaks a distributed transaction into a sequence of smaller, local ACID transactions. Each local transaction updates the database and triggers the next step. If a step fails, the Saga executes Compensating Actions to undo the previous steps.

There are two ways to implement Sagas: Choreography (events trigger each other like a domino effect) and Orchestration (a central brain commands the steps). Choreography is fine for two or three steps. Beyond that, nobody can read the whole business flow in one place, and "why is this order stuck?" needs a search across several services' logs, so this page uses orchestration. 10.8.4 compares the two in depth.

Saga Orchestration Example

Let's build a central OrderOrchestrator that manages a checkout flow: Order Creation -> Payment -> Inventory.

Sequence diagram of Saga Orchestration checkout flow with failure handling

TypeScript: Saga Orchestrator Implementation

Two facts drive every line of this code. The orchestrator can crash at any moment, so its progress must live in a database, not in memory. And a remote call that fails with a timeout may still have worked: the card can be charged even though we never saw the reply. A saga that only compensates "the steps it saw succeed" will leave that customer charged for an order that was cancelled.

type Step =
  | "STARTED" | "CHARGING" | "CHARGED" | "RESERVED" | "COMPLETED"
  | "COMPENSATING" | "ABORTED";

interface CheckoutSagaRecord {
  sagaId: string;
  step: Step;
  userId: string;
  itemId: string;
  amountPaise: number;
}

class OutOfStock extends Error {}                         // a definite business "no"

class CheckoutSaga {
  constructor(
    private readonly log: SagaLog,                        // (1)
    private readonly orders: OrderService,
    private readonly payments: PaymentService,
    private readonly inventory: InventoryService,
  ) {}

  async run(saga: CheckoutSagaRecord): Promise<void> {    // (2)
    try {
      if (saga.step === "STARTED") {
        await this.orders.createPending(saga.sagaId, saga.itemId);
        await this.moveTo(saga, "CHARGING");              // (3)
      }
      if (saga.step === "CHARGING") {
        await this.payments.charge(saga.userId, saga.amountPaise, `${saga.sagaId}:charge`);   // (4)
        await this.moveTo(saga, "CHARGED");
      }
      if (saga.step === "CHARGED") {
        await this.inventory.reserve(saga.itemId, `${saga.sagaId}:reserve`);
        await this.moveTo(saga, "RESERVED");
      }
      if (saga.step === "RESERVED") {
        await this.orders.markComplete(saga.sagaId);
        await this.moveTo(saga, "COMPLETED");
      }
    } catch (err) {
      if (!(err instanceof OutOfStock)) throw err;        // (5)
      await this.moveTo(saga, "COMPENSATING");
    }
    if (saga.step === "COMPENSATING") {                   // (6)
      await this.payments.refundIfCharged(`${saga.sagaId}:charge`);
      await this.orders.cancel(saga.sagaId);
      await this.moveTo(saga, "ABORTED");
    }
  }

  private async moveTo(saga: CheckoutSagaRecord, step: Step): Promise<void> {
    saga.step = step;
    await this.log.save(saga);                            // (7)
  }
}

1. The saga log is a table with one row per checkout. It is the orchestrator's memory, and the only state that survives a crash.
2. run is written so it can be called again at any time with whatever the log says. A small background job calls it for every saga that is not COMPLETED or ABORTED, which is how a crashed checkout resumes.
3. Record the intent before the side effect. The saga moves to CHARGING before calling the payment service. If we crash during the call, the log says "a charge may have happened", which is the honest state.
4. Every side effect carries a key derived from the saga id. When a resumed saga calls charge again after a timeout, the payment service recognises the key and returns the original charge instead of taking the money twice. This is the idempotency-key idea from 10.21, applied to each step.
5. Only a definite business rejection (the item is out of stock) starts compensation. Any other error, such as a timeout or a refused connection, is ambiguous, so the saga stops and lets the background job retry later. Compensating on a timeout would cancel orders that were about to succeed.
6. Compensation runs every undo step, and each one is safe to run whether or not the forward step happened. refundIfCharged looks up the charge by its key and does nothing if there is none, and refunding twice by key refunds once. So a crash half-way through compensation is harmless too: the saga resumes in COMPENSATING and runs the same steps again.
7. Every state change is written to the log before the next step begins.

Run against fake services, the four cases behave like this:

Case	Final order	Cards charged	Refunds
Everything succeeds	complete	1	0
Item out of stock after charging	cancelled	1	1
Payment times out after charging, saga resumed	complete	1	0
Crash in the middle of compensating, saga resumed	cancelled	1	1

The third row is the one the naive version gets wrong. Without the CHARGING state and the per-step key, the timeout looks like "payment failed", the order is cancelled without a refund, and the customer has paid for nothing.

If compensation itself keeps failing, for example because the payment provider rejects the refund, retrying forever will not help. The saga needs a human: park it in a dead-letter table with an alert, and give support a screen to finish it by hand.

Compensating Actions must be semantically equivalent to an undo. You cannot physically "undo" an email sent to a user, so the compensating action is sending an apology email.

The Transactional Outbox Pattern

In microservices, you often need to save state to your database AND publish an event to a message broker (like Kafka) so other services can react. This is the Dual-Write Problem:

1. Save to DB.
2. Publish to Kafka. If step 2 fails after step 1 succeeds, your system is forever inconsistent.

The Outbox Pattern solves this by writing the event to an outbox table in the same local database transaction as the business entity.

Flowchart illustrating the Transactional Outbox pattern

TypeScript: Outbox Implementation

async function placeOrder(orderData: OrderPayload) {
  // We use the database's local transaction mechanism
  return await db.transaction(async (tx) => {
    
    // 1. Write the business data
    const order = await tx.orders.insert({
      userId: orderData.userId,
      total: orderData.total,
      status: 'PENDING'
    });

    // 2. Write the event to the outbox table IN THE SAME TX
    await tx.outbox.insert({
      eventId: generateUuid(),
      aggregateType: 'ORDER',
      aggregateId: order.id,
      payload: JSON.stringify(order),
      createdAt: new Date(),
      status: 'UNPUBLISHED' // Relay will poll for this status
    });

    return order;
  });
  // If the database crashes here, both inserts roll back. No dual-write problem.
}

A separate background process (or CDC tool like Debezium) tails the outbox table, reads UNPUBLISHED rows, publishes them to Kafka, and marks them PUBLISHED.

The Transactional Inbox Pattern

While the Outbox guarantees at-least-once delivery of messages, the Inbox Pattern handles the receiving side to guarantee exactly-once processing.

When a service receives a message, it writes the eventId to an inbox table within the exact same database transaction that processes the message.

1. Begin a database transaction.
2. Insert the eventId into the inbox table, which has a unique key on eventId, using INSERT ... ON CONFLICT DO NOTHING.
3. If the insert added no row, this event was already processed. Commit, acknowledge the message, and stop.
4. Otherwise run the business logic (for example, deduct inventory) inside the same transaction.
5. Commit. The inbox row and the business change are saved together or not at all.

Inserting first, rather than "check whether it exists, then insert at the end", matters when two copies of the same message are processed at the same moment. With check-first, both copies see "not processed" and both deduct the stock. With insert-first, the unique key lets exactly one copy through. The same reasoning, worked in full, is in 10.4.

Idempotency Keys

Every pattern on this page leans on one more idea: a request that is retried must not take effect twice. An idempotency key is a unique id the client sends with a request so the server can recognise a retry and return the first result instead of acting again. Getting it right has three traps: the claim must be atomic, the key must be passed on to any downstream system that has side effects, and a key must never be deleted after an ambiguous failure. The worked, tested implementation is in 10.21, and the full design, including request-body checks and key expiry, is 9.6.3.

---

What the interviewer will push on

1. "What happens if your Saga's compensating action fails?"Answer: Compensations must be retried automatically. Therefore, compensating endpoints must be idempotent. If a compensation fails indefinitely (e.g., the 3rd party API is permanently down or returns a 400 Bad Request), the Saga enters a stuck state. You must log a critical alert and push the failed context to a Dead Letter Queue (DLQ) for manual operator intervention.

2. "Why use CDC (Change Data Capture) for the Outbox pattern instead of a polling worker?"Answer: A polling worker SELECT * FROM outbox WHERE status = 'UNPUBLISHED' puts heavy read pressure on the primary database, especially if run every 500ms. CDC tools like Debezium read directly from the database's Write-Ahead Log (WAL), having near-zero performance impact on the database while delivering changes in milliseconds.

3. "Is 2PC ever the right choice?"Answer: Yes, in two places. Inside a distributed database, where the product runs 2PC over replicated (Raft or Paxos) groups so no coordinator crash can block it; that is how Spanner and CockroachDB give you multi-shard transactions. And between two systems you control that both support the XA protocol, at low volume, where a short lock wait is acceptable. Across independently deployed services it is the wrong tool: every service must be up for any transaction to finish, and locks are held across network round trips. The tell of a strong answer is saying where 2PC runs rather than declaring it dead.

---

Recall

• 2PC: all-or-nothing across systems, but participants hold locks while waiting and a crashed coordinator leaves them stuck. Wrong across independent services; right inside distributed databases that run it over Raft or Paxos groups. 3PC is unsafe when the network splits, so it is not used.
• Saga Pattern: Breaks distributed transactions into local ACID transactions chained together.
• Compensating Actions: The undo mechanisms for failed Sagas.
• Outbox Pattern: Solves the dual-write problem by saving the domain state and the outgoing event in a single local database transaction.
• Inbox Pattern: Deduplicates incoming events using local database transactions to achieve exactly-once processing.
• Idempotency: Ensuring that repeating an operation (due to network retries) does not result in duplicate side effects.

Quiz

1. Why does 2PC destroy database throughput in high-traffic systems?
2. In Saga Orchestration, what happens if Step 2 succeeds but Step 3 fails?
3. How does the Transactional Outbox pattern circumvent the dual-write problem?
4. What is the primary purpose of an idempotency key in a payment API?
5. Why is it critical that a Saga's Compensating Actions are idempotent?

Next: 10.25: every call on this page crosses a network, and every one of them has to prove who is calling.