10.5 — Replication: Copies, Lag, and the Anomalies They Cause

Replication keeps copies of the same data on multiple machines, and it buys three things at once: availability (a replica survives a failure), read scalability (many copies serve many readers), and locality (a copy near the user). What it costs is the subject of this page — because the moment there is more than one copy, "the data" has a version per copy, and the gap between them (replication lag) produces user-visible anomalies with specific names, specific cures, and specific interview questions.

1. The three topologies

Single-leader (primary/replica) — all writes go to one leader; followers replicate its log and serve reads. This is Postgres/MySQL/MongoDB's default and the model most systems should start with. Its virtues: no write conflicts (one writer per dataset means ordering is free — 10.3's single-writer sequence), simple reasoning, and mature tooling. Its costs: the leader is a write bottleneck and a failure point (failover takes seconds and risks lost writes — section 3), and followers serve stale reads.

Multi-leader — several nodes accept writes and replicate to each other. Used for multi-datacenter writes, offline-capable clients (each device is a leader), and collaborative editing. It buys write availability and locality; it costs conflicts, which are now guaranteed to occur and must be detected and resolved (10.3's vector clocks; section 4 below). Most teams underestimate this cost, which is why multi-leader is the topology you should adopt deliberately, never accidentally.

Leaderless (Dynamo-style) — clients (or a coordinator) write to several replicas and read from several, using quorums. Cassandra, Riak, DynamoDB's internals. Availability is excellent. There is no failover, because any node accepts writes. And the consistency setting is out in the open. With N replicas, you write to W of them and read from R of them. If W + R > N, every read overlaps every write, so the latest value is among what you read. You still have to decide which returned value is the latest, using versions or vector clocks, not timestamps.

Two repair mechanisms complete it:

• Read repair fixes out-of-date replicas when a read finds them.
• Anti-entropy compares replicas in the background, often using Merkle trees.

2. Synchronous vs asynchronous, and the durability trade

The second axis, independent of topology, and the one that decides what you lose in a failure:

• Asynchronous replication — the leader acknowledges the write immediately and ships it to followers afterward. Fast, available (a slow follower doesn't block writes), and lossy on failover: writes acknowledged but not yet replicated are gone when the leader dies. This is the default nearly everywhere, and it means "committed" is a per-node word unless you say otherwise.
• Synchronous replication — the leader waits for follower acknowledgment before confirming. No loss on failover, at the price of latency (a network round trip inside every write) and availability (a stuck follower stalls writes).
• Semi-synchronous — the practical compromise: wait for one follower (or a quorum), not all. Bounded loss, bounded latency; this is what most serious deployments run, and it's the right default answer in an interview.

The framing that makes the choice concrete: how many acknowledged writes are you willing to lose when the leader dies? Zero is purchasable, at latency; "a few hundred milliseconds' worth" is what async gives you; and the business — not the database — should answer.

3. Failover: where data goes missing

When the leader fails, someone must promote a follower. The steps are simple; the failure modes are famous: ^{⚑How does leader failover work and what can go wrong? [EQ-992b]}

1. Detect — usually a timeout, which cannot distinguish crashed from slow (10.3's FLP).
2. Elect — pick the most up-to-date follower (consensus or a controller — 10.7.2).
3. Reconfigure — clients and remaining followers must learn the new leader.

The three classic disasters:

• Lost writes. With asynchronous replication, a follower can be a little behind. If that follower is promoted to leader, the writes it had not received yet are thrown away, even though the old leader had already said "saved". That is a disaster if other systems already saw those writes, for example an order number already sent to a customer.
• Split brain. The old leader did not crash. It was only cut off from the network. Now two leaders accept writes, and merging them afterwards means fixing data by hand. The cure is fencing: forcibly shutting the old leader off, leases with fencing tokens, or requiring a majority so that a leader on the smaller side of the split cannot act (10.4).
• Failover storms. Timeouts that are too eager promote a new leader during a brief blip, and all that switching is worse than the blip was. The cure is to wait for several failed checks in a row before acting (9.7.6).

4. Replication lag: three anomalies with three cures

Followers are behind the leader by some amount; under load or long transactions, "some" can become seconds or minutes. Each anomaly below is a real user complaint with a standard fix: ^{⚑Read-your-writes, monotonic reads, and replication lag anomalies. [EQ-994b]}

Read-your-writes violation — a user posts a comment, the write goes to the leader, the subsequent read hits a lagging follower, and their own comment is missing. They refresh, panic, post again. Cures: route reads for recently-written keys to the leader (track "this user wrote within the last N seconds"); read from a replica only if it has caught up past the write's position (log sequence number/LSN comparison); or serve the client's own write from its local state until the read catches up.

Monotonic reads violation — a user reads a value from an up-to-date replica, refreshes, hits a laggier replica, and time appears to go backward (their comment vanishes, a count decreases). Cure: make each user always read from the same replica (hash the user id to a replica), so their view can be stale but never regresses.

Consistent prefix violation — with partitioned data replicating independently, an observer sees an answer before the question, a reply before the message. Cure: ensure causally-related writes share a partition (per-conversation keys — 10.6) or track causality explicitly (10.3).

Figure 1 — Read-your-writes, violated and cured. The write lands on the leader; the follow-up read reaches a lagging follower and the user's own action seems to have vanished. Routing that user's reads to the leader (or to a replica proven caught up past their write position) restores the guarantee without giving up replica reads for everyone else.

5. Multi-leader conflicts

If two leaders accept writes to the same record, conflicts are certain. The ways to resolve it, from worst to best:

1. The latest timestamp wins. Simple, but it silently loses data, as 10.3 warned.
2. The node with the highest ID wins. Predictable, but just as arbitrary.
3. The application merges them. For example, combine both shopping carts, or keep both edits with a rule per field. This is usually the right answer.
4. CRDTs (conflict-free replicated data types). These are data structures whose merge is mathematically guaranteed to end in the same result everywhere, such as counters, sets and sequences. Collaborative editors are built on them (Chapter 11.13).
5. Show the conflict to a person, the way git does.

The design guidance: pick the resolution before deploying multi-leader, because the alternative is discovering your policy is "silently lose data" during an incident.

A CRDT, worked: the like counter that cannot lose a like

CRDTs sound like magic until you build the smallest one, so here it is. A post's like count is stored in two regions, Mumbai and Dublin, and each region accepts likes locally so nobody waits for an ocean crossing. Three people like the post in Mumbai while two people like it in Dublin at the same moment. The true answer is 5.

The naive merge loses data. If each region stores a single number, Mumbai holds 3 and Dublin holds 2. When they sync, what should the merged value be? Taking the larger number gives 3, which loses Dublin's two likes. Adding them gives 5 the first time, but syncing is retried constantly, and the second sync adds again and reports 7. A single number cannot tell "a like I have not seen yet" from "a like I already counted".

The fix is to stop storing one number. Each region keeps its own slot, and only ever increases its own slot. The total is the sum of the slots. Merging takes the larger value per slot, which is safe to repeat because taking the larger of two numbers a second time changes nothing. This structure is called a grow-only counter (G-Counter).

Here is the state after each step, with the slots written as {mumbai, dublin}:

Step	Mumbai's copy	Dublin's copy	Totals
3 likes in Mumbai, 2 in Dublin	{3, 0}	{0, 2}	3 and 2
Mumbai merges Dublin's copy	{3, 2}	{0, 2}	5 and 2
Dublin merges Mumbai's copy	{3, 2}	{3, 2}	5 and 5
The same sync is delivered again	{3, 2}	{3, 2}	5 and 5

The last row is the whole point. A duplicate or reordered sync message, which 10.4 told you to expect, cannot corrupt the count.

type ReplicaId = string;

class LikeCounter {
  private counts = new Map<ReplicaId, number>();              // (1)

  constructor(private readonly self: ReplicaId) {}

  like(): void {
    this.counts.set(this.self, (this.counts.get(this.self) ?? 0) + 1);   // (2)
  }

  total(): number {
    let sum = 0;
    for (const n of this.counts.values()) sum += n;           // (3)
    return sum;
  }

  merge(other: LikeCounter): void {
    for (const [id, n] of other.counts) {
      this.counts.set(id, Math.max(this.counts.get(id) ?? 0, n));   // (4)
    }
  }
}

const mumbai = new LikeCounter("mumbai");
const dublin = new LikeCounter("dublin");

mumbai.like(); mumbai.like(); mumbai.like();   // 3 likes land in Mumbai
dublin.like(); dublin.like();                  // 2 likes land in Dublin, at the same time

console.log(mumbai.total(), dublin.total());   // → 3 2

mumbai.merge(dublin);
dublin.merge(mumbai);
dublin.merge(mumbai);                          // a duplicate sync changes nothing
console.log(mumbai.total(), dublin.total());   // → 5 5

1. One slot per region. The region's own id is the key, so a region never writes into another region's slot.
2. A like only ever increases this region's slot. That single rule is what makes the merge safe.
3. The visible count is the sum of every slot this copy has heard about.
4. The merge takes the larger value per slot. Each slot only grows, so the larger value is always the more recent one, and taking the maximum twice gives the same answer as taking it once. In maths terms the merge is commutative (order of syncs does not matter), associative (grouping does not matter) and idempotent (repeats do not matter). Those three properties are the definition of a state-based CRDT, and they are why replicas converge without any coordination.

The program prints 3 2 before syncing and 5 5 after, including after the duplicate sync.

Where the clean idea meets reality. An unlike cannot be a decrement, because a slot that shrinks breaks rule (2). The standard answer is a PN-counter: two G-Counters, one for likes and one for unlikes, with the total being their difference. Sets, text and lists need cleverer structures (observed-remove sets, sequence CRDTs), which is what collaborative editors use (Chapter 11.13). The metadata also grows with the number of replicas: one slot per writer is cheap for three regions and expensive if every phone is a writer. And a CRDT guarantees only that copies converge. It does not enforce a business rule such as "stock never goes below zero", because enforcing that needs coordination, which is exactly what a CRDT avoids.

6. The expert lens

Replication is availability and read scale; it is not write scale. Every replica takes every write, so a write-bound system gets no relief from adding followers — that requires partitioning (10.6). This distinction resolves a large fraction of scaling confusion: replicas multiply read capacity and shrink failure impact; shards divide write load. Systems at scale need both, and knowing which lever you're pulling is the whole skill.

Lag is a product decision that leaks into the UI. "Is a few seconds of staleness acceptable?" cannot be answered by engineering alone, and it varies per read: a public article can be minutes stale; a user's own profile after editing cannot be stale at all; a balance display after a transfer had better not be. The mature pattern is per-read consistency levels — most reads go to replicas; a small, explicitly-marked set demands the leader — which makes staleness a deliberate, auditable choice rather than an accident of routing.

Failover is an availability and durability event. The two questions to ask any datastore, and the answers most teams cannot give: how many writes can we lose if the leader dies right now? (async lag window) and what prevents two leaders from accepting writes during a partition? (fencing/quorum). Rehearse it — a failover you have never triggered on purpose is an untested code path in the most dangerous position in your architecture.

Next: 10.6 — the other axis: splitting data so writes scale, and the hot-spot problems that decide whether your shard key was any good.

Recall

• Three layouts:
  • Single-leader: no write conflicts, and simple. But all writes go through one node, and followers can be behind. It is the right default.
  • Multi-leader: writes stay close to users and keep working during a split. Conflicts are guaranteed, so choose it on purpose.
  • Leaderless, with quorums: W + R > N makes reads overlap writes. Read repair and anti-entropy fix stale copies.
• Sync axis: async (fast, lossy on failover — "committed" is per-node), synchronous (no loss, pays latency and stalls on a slow follower), semi-sync (one follower or a quorum — the practical default). Frame the choice as how many acknowledged writes may we lose?
• Failover = detect (timeout — can't tell slow from dead) → elect (most up-to-date follower) → reconfigure. Disasters: lost writes (unreplicated tail discarded), split brain (partitioned old leader still writing — cure with fencing/quorum), failover storms (aggressive timeouts; use hysteresis).
• Lag anomalies and cures: read-your-writes (route the writer's reads to leader or a caught-up replica by LSN) · monotonic reads (pin a user to one replica — stale but never regressing) · consistent prefix (keep causally related writes in one partition).
• Multi-leader conflict menu: LWW (silent loss) → node-id → application merge → CRDTs → surface to user. Choose before deploying.
• Lens: replication buys availability + read scale, never write scale (that's partitioning); lag is a per-read product decision; failover is a durability event — rehearse it.

Self-test: Which topology has no write conflicts and why? What exactly is lost on async failover? Name the three lag anomalies with their cures. What does W + R > N guarantee — and what does it not? Why doesn't adding replicas help a write-bound system?

Quiz Bank

FoundationalCompare single-leader, multi-leader, and leaderless replication.

Single-leader: one node accepts writes, followers replicate and serve reads. No write conflicts ever (a single writer gives free ordering — 10.3), simple failure reasoning, best tooling; costs are a write bottleneck, a failover event with potential data loss, and stale follower reads (section 4's anomalies). Default choice for most systems.

Multi-leader: multiple nodes accept writes and replicate to each other — used for multi-region write locality, offline-first clients (each device is a leader), and collaborative apps. Buys write availability during partitions and low write latency everywhere; costs guaranteed conflicts requiring a resolution policy (merge, CRDTs, user prompt — never silent LWW).

Leaderless (Dynamo-style): writes go to several replicas, reads read several, with quorums — W + R > N ensures a read set intersects the latest write set. No failover step (any node accepts writes), excellent availability, and consistency tuned per operation; costs are conflict handling on read (which value is newest — versions, not timestamps), plus repair machinery (read repair, anti-entropy with Merkle trees). Selection rule: start single-leader; move to leaderless for always-writable high-availability stores; adopt multi-leader only when geography or offline clients force it, with the conflict policy decided up front.

FoundationalExplain the three replication-lag anomalies and their standard cures.

Read-your-writes: a user writes (leader) then reads (lagging follower) and doesn't see their own change — the most user-visible and trust-destroying anomaly. Cures: route reads to the leader for a window after that user writes; or compare the replica's applied log position (LSN) against the write's position and only use replicas that have caught up; or have the client hold its own write optimistically until confirmed.

Monotonic reads: successive reads hit replicas with different lag, so the user sees data go backward (a comment appears then vanishes, a counter decreases). Cure: sticky replica routing per user (hash the user id) — the view may be stale but never regresses.

Consistent prefix: with independently-replicating partitions, causally ordered writes can be observed out of order (an answer before its question). Cure: keep causally related writes in the same partition (10.6 — e.g. partition by conversation id) or propagate causality metadata (10.3). All three share a diagnosis pattern: they appear only under load (when lag grows), affect a minority of requests, and are reported as "the app is buggy" rather than as staleness — which is why lag must be a monitored, alertable metric, not an assumption.

AppliedWhat does W + R > N guarantee in a quorum system, and what does it not?

With N replicas, writing to W and reading from R, W + R > N guarantees the read set and the write set overlap in at least one node — so any successful read touches at least one replica holding the latest acknowledged write. Common configurations: N=3, W=2, R=2 (balanced); W=N, R=1 (fast reads, slow/fragile writes); W=1, R=N (fast writes, expensive reads).

What it does not guarantee:

1. It does not tell you which value you got back is newest. You still need versions or vector clocks to choose. Timestamps are not safe (10.3).
2. It is not linearizability. Operations running at the same time can still mix in surprising ways. And a write that failed but reached some replicas leaves them holding a value that may later "win".
3. It says nothing about failures that hit several replicas at once, such as three replicas in the same rack.
4. Sloppy quorums break the guarantee on purpose. They accept writes on any W nodes they can reach and pass them on later, to stay available. That is a fair trade, but you must know you are making it.

The complete answer names the accompanying machinery:

read repair (update stale replicas discovered during reads) and anti-entropy (background Merkle-tree comparison) are what actually converge the system over time.

InterviewYour leader dies. Walk through what happens and what can go wrong.

Detect: a monitor or the followers notice missed heartbeats past a timeout — which cannot distinguish a crashed leader from a slow/partitioned one (10.3's FLP), so this step is a guess with consequences. Elect: promote the most up-to-date follower — via consensus (10.7.2) or an external controller; "most up-to-date" is measured by replication position, and choosing wrong means discarding more writes.

Reconfigure: clients, remaining followers, and any dependent systems must learn the new leader (connection strings, service discovery, DNS with its caching lag — 10.2). What goes wrong: lost writes — under async replication, writes acknowledged to clients but not yet shipped are simply gone; catastrophic when those writes escaped elsewhere (an order id given to a customer, an event published downstream) because now systems disagree about what happened.

Split brain: the old leader was only cut off, not dead. It keeps accepting writes. Afterwards, joining the two different histories back together is painful work by hand. The cures are:

• elections that need a majority, so the smaller side cannot elect a leader;
• leases plus fencing tokens, so the storage rejects the old leader's writes (10.4);
• or forcibly powering off the old leader, known as STONITH.

Failover storms — aggressive timeouts promote during a transient blip, and the churn (cold caches, reconnect storms, another timeout) is worse than the blip; cure with hysteresis and conservative thresholds. Split-second dual writes to derived systems — downstream consumers may see the same events twice from both leaders, which is why consumers should be idempotent (10.4).

StaffA social product's users complain: 'my post disappears after I publish it, then comes back a minute later.' Reads are served from three replicas behind a round-robin balancer, writes go to the leader, replication is async. Diagnose, fix, and design the guarantee model.

Diagnosis: two lag anomalies compounding. The immediate one is read-your-writes — the publish goes to the leader; the redirect/refresh read is balanced round-robin to a follower that hasn't applied it yet, so the author's own post is missing. The "comes back, then vanishes again on refresh" pattern is monotonic reads — successive reads land on replicas with different lag, so the timeline oscillates. Confirm with data rather than theory: expose per-replica lag as a metric (seconds behind leader), sample it during peak, and correlate complaint timestamps with lag spikes — expect the complaints to cluster exactly where lag exceeds the user's think-time.

Fix it in layers.

(1) Read your own writes. After any write, note in the user's session where that write sits in the log, and start a short window, say 10 seconds. During that window, send that user's reads either to the leader, or to a replica that has already applied the log at least that far. The second option is the precise one. It keeps most reads on the replicas. The simple fix, sending everyone to the leader, just makes the leader the new bottleneck. (2)

Monotonic reads — sticky per-user replica selection (hash the user id) so a user's view never moves backward, with fallback on replica failure. (3) Lag control — alert on lag SLO breach, and investigate the causes (long transactions, batch jobs on the leader, network saturation, single-threaded apply on the replica).

Write down a consistency level for every read. This is what will still matter long after this incident. There are three levels:

• Strong, from the leader: for confirming a write, for balances, and for anything the user just changed.
• Slightly behind, within a limit: from a replica whose lag is under X, otherwise from the leader. Good for feeds and profiles.
• Best effort: from any replica, and the CDN may cache it. Good for public content (10.2).

Encode it in the data-access layer as an argument (read(query, { consistency: "strong" })) so the choice is visible in code review, not implied by which client someone imported.

The lens for the write-up: replication lag isn't a bug to eliminate — it's the price of read scale, and the engineering work is deciding, per read, how much staleness is acceptable and enforcing that decision in routing rather than hoping.

Flashcards

FlashThree topologies

Single-leader (no conflicts, write bottleneck) · multi-leader (locality/offline; conflicts guaranteed) · leaderless quorum (W+R>N overlap; read repair + anti-entropy).

FlashSync trade

Async = fast, loses the unreplicated tail on failover. Sync = no loss, pays RTT and stalls on slow followers. Semi-sync (one/quorum) = the practical default.

FlashFailover disasters

Lost writes (async tail) · split brain (partitioned old leader — fence/quorum) · failover storms (aggressive timeouts; add hysteresis).

FlashLag anomalies

Read-your-writes (route writer to leader/caught-up replica) · monotonic reads (pin user to one replica) · consistent prefix (co-partition causal writes).

FlashW + R > N

Read set overlaps write set ⇒ latest value is present. Doesn't say which value is newest (need versions), isn't linearizability, and sloppy quorums break it deliberately.

FlashReplication ≠ write scale

Every replica takes every write. Replicas = availability + read scale; write scale = partitioning (10.6).

Scenario Drill

DrillDesign the replication strategy for a global SaaS with EU data-residency requirements: customers are pinned to a region, users travel, dashboards are read-heavy, audit logs must never be lost, and the EU tenant's data may not be stored outside the EU. Specify topology, sync mode, read routing, and the two hardest constraints this combination creates.

Topology — partition by tenant, single-leader per region. Data residency makes this straightforward and turns a hard problem (multi-region consistency) into an easy one: each tenant's data lives entirely in one region with a single-leader cluster there plus in-region followers. There is no cross-region replication of tenant data — which satisfies residency by construction rather than by policy, and simultaneously avoids multi-leader conflicts entirely (section 1's "adopt deliberately" advice, sidestepped). What is global is a small, non-personal control plane (tenant → region routing, feature flags, billing metadata), replicated read-only worldwide with generous staleness.

Sync mode — differentiated by data class. Audit logs: synchronous or semi-synchronous replication to at least one in-region follower before acknowledging, because "must never be lost" is a durability requirement that async cannot honor — the acknowledged-but-unreplicated tail is exactly what a leader crash discards (section 2). Everything else:

semi-synchronous (one follower ack) as the balanced default, with the loss window documented. Additionally, audit records go to append-only storage with object-lock semantics — replication protects against node loss, immutability protects against the other failure mode (a bad actor or bad migration deleting them).

Where reads go: three levels, enforced in the data layer, using the model above:

• Strong, from the leader, for confirming a write and for anything the user just changed. Only those reads go to the leader, so the leader is not flooded.
• Slightly behind, from a replica, for dashboards. They are read a lot and can be a few seconds old. Each user sticks to one replica, so their view never jumps backwards in time.
• Best effort, or cached, for static data and totals across all customers.

The two hardest constraints this combination creates. (1) Travelling users vs pinned data: a user physically in Singapore whose tenant lives in Frankfurt will experience ~150–200 ms round trips no matter what caching you do for writes and strong reads. You cannot replicate their data closer (residency forbids it), so the honest engineering is latency masking: an edge tier that terminates TLS near the user and keeps a warm connection to the home region (10.2), aggressive client-side optimism for writes with server confirmation, and a UI that doesn't block on strong reads. Say this to the business explicitly — it's a physics-and-law constraint, not a tuning opportunity. (2)

Cross-tenant analytics: any global dashboard aggregating all tenants pulls EU personal data out of the EU the moment it's computed centrally. The resolution is compute in-region, export only aggregates — each region produces anonymized/aggregated rollups that leave the region, with the pipeline's data classification reviewed by legal ([Part 8.7]); a naive central data warehouse would be a compliance incident wearing a BI dashboard.

Failover posture: in-region only (a Frankfurt outage must not fail over to Virginia — residency again), so regional redundancy must be genuinely multi-AZ, failover must be rehearsed, and the disaster-recovery story is "restore in-region from in-region backups," with RTO/RPO stated and tested. The design-doc sentence: residency converted our replication problem into a partitioning problem, which is the good trade — the remaining costs are travel latency and analytics topology, and both are business decisions we surfaced rather than engineering problems we hid.