10.5 — Replication: Copies, Lag, and the Anomalies They Cause

Replication keeps copies of the same data on multiple machines, and it buys three things at once: availability (a replica survives a failure), read scalability (many copies serve many readers), and locality (a copy near the user). What it costs is the subject of this page — because the moment there is more than one copy, "the data" has a version per copy, and the gap between them (replication lag) produces user-visible anomalies with specific names, specific cures, and specific interview questions.

1. The three topologies

Single-leader (primary/replica) — all writes go to one leader; followers replicate its log and serve reads. This is Postgres/MySQL/MongoDB's default and the model most systems should start with. Its virtues: no write conflicts (one writer per dataset means ordering is free — 10.3's single-writer sequence), simple reasoning, and mature tooling. Its costs: the leader is a write bottleneck and a failure point (failover takes seconds and risks lost writes — section 3), and followers serve stale reads.

Multi-leader — several nodes accept writes and replicate to each other. Used for multi-datacenter writes, offline-capable clients (each device is a leader), and collaborative editing. It buys write availability and locality; it costs conflicts, which are now guaranteed to occur and must be detected and resolved (10.3's vector clocks; section 4 below). Most teams underestimate this cost, which is why multi-leader is the topology you should adopt deliberately, never accidentally.

Leaderless (Dynamo-style) — clients (or a coordinator) write to several replicas and read from several, using quorums. Cassandra, Riak, DynamoDB's internals. Availability is excellent (no failover — any node takes writes), and the consistency knob is explicit: with N replicas, writing to W and reading from R, W + R > N guarantees a read overlaps a write, so the latest value is seen (though you must still resolve which returned value is latest — versions or vector clocks, not timestamps). Repair mechanisms complete it: read repair (fix stale replicas on read) and anti-entropy (background comparison, often via Merkle trees).

2. Synchronous vs asynchronous, and the durability trade

The second axis, independent of topology, and the one that decides what you lose in a failure:

• Asynchronous replication — the leader acknowledges the write immediately and ships it to followers afterward. Fast, available (a slow follower doesn't block writes), and lossy on failover: writes acknowledged but not yet replicated are gone when the leader dies. This is the default nearly everywhere, and it means "committed" is a per-node word unless you say otherwise.
• Synchronous replication — the leader waits for follower acknowledgment before confirming. No loss on failover, at the price of latency (a network round trip inside every write) and availability (a stuck follower stalls writes).
• Semi-synchronous — the practical compromise: wait for one follower (or a quorum), not all. Bounded loss, bounded latency; this is what most serious deployments run, and it's the right default answer in an interview.

The framing that makes the choice concrete: how many acknowledged writes are you willing to lose when the leader dies? Zero is purchasable, at latency; "a few hundred milliseconds' worth" is what async gives you; and the business — not the database — should answer.

3. Failover: where data goes missing

When the leader fails, someone must promote a follower. The steps are simple; the failure modes are famous: ^{⚑How does leader failover work and what can go wrong? [EQ-992b]}

1. Detect — usually a timeout, which cannot distinguish crashed from slow (10.3's FLP).
2. Elect — pick the most up-to-date follower (consensus or a controller — 10.7.2).
3. Reconfigure — clients and remaining followers must learn the new leader.

The three classic disasters: lost writes (async replication + promotion of a lagging follower — the acknowledged-but-unreplicated tail is discarded, which is catastrophic if other systems already saw those writes, e.g. an order id handed to a customer); split brain (the old leader didn't crash — it was partitioned; now two leaders accept writes, and the merge afterward is manual data surgery — the cure is fencing: STONITH, leases with fencing tokens, or a quorum requirement that makes a minority leader impossible — 10.4); and failover storms (too-aggressive timeouts promote during a transient blip, and the resulting churn is worse than the blip — hysteresis again, 9.7.6).

4. Replication lag: three anomalies with three cures

Followers are behind the leader by some amount; under load or long transactions, "some" can become seconds or minutes. Each anomaly below is a real user complaint with a standard fix: ^{⚑Read-your-writes, monotonic reads, and replication lag anomalies. [EQ-994b]}

Read-your-writes violation — a user posts a comment, the write goes to the leader, the subsequent read hits a lagging follower, and their own comment is missing. They refresh, panic, post again. Cures: route reads for recently-written keys to the leader (track "this user wrote within the last N seconds"); read from a replica only if it has caught up past the write's position (log sequence number/LSN comparison); or serve the client's own write from its local state until the read catches up.

Monotonic reads violation — a user reads a value from an up-to-date replica, refreshes, hits a laggier replica, and time appears to go backward (their comment vanishes, a count decreases). Cure: make each user always read from the same replica (hash the user id to a replica), so their view can be stale but never regresses.

Consistent prefix violation — with partitioned data replicating independently, an observer sees an answer before the question, a reply before the message. Cure: ensure causally-related writes share a partition (per-conversation keys — 10.6) or track causality explicitly (10.3).

Figure 1 — Read-your-writes, violated and cured. The write lands on the leader; the follow-up read reaches a lagging follower and the user's own action seems to have vanished. Routing that user's reads to the leader (or to a replica proven caught up past their write position) restores the guarantee without giving up replica reads for everyone else.

5. Multi-leader conflicts

If two leaders accept writes to the same record, conflicts are certain. The resolution menu, in ascending order of quality: last-write-wins by timestamp (simple, silently loses data — 10.3's warning); highest-node-id wins (deterministic and equally arbitrary); application-defined merge (union the shopping carts, keep both edits with per-field rules — usually the right answer); CRDTs (conflict-free replicated data types — structures whose merge is mathematically guaranteed to converge: counters, sets, sequences; the basis of collaborative editors — Chapter 11.13); or surface the conflict to a user (git's model). The design guidance: pick the resolution before deploying multi-leader, because the alternative is discovering your policy is "silently lose data" during an incident.

6. The expert lens

Replication is availability and read scale; it is not write scale. Every replica takes every write, so a write-bound system gets no relief from adding followers — that requires partitioning (10.6). This distinction resolves a large fraction of scaling confusion: replicas multiply read capacity and shrink failure impact; shards divide write load. Systems at scale need both, and knowing which lever you're pulling is the whole skill.

Lag is a product decision that leaks into the UI. "Is a few seconds of staleness acceptable?" cannot be answered by engineering alone, and it varies per read: a public article can be minutes stale; a user's own profile after editing cannot be stale at all; a balance display after a transfer had better not be. The mature pattern is per-read consistency levels — most reads go to replicas; a small, explicitly-marked set demands the leader — which makes staleness a deliberate, auditable choice rather than an accident of routing.

Failover is an availability and durability event. The two questions to ask any datastore, and the answers most teams cannot give: how many writes can we lose if the leader dies right now? (async lag window) and what prevents two leaders from accepting writes during a partition? (fencing/quorum). Rehearse it — a failover you have never triggered on purpose is an untested code path in the most dangerous position in your architecture.

Next: 10.6 — the other axis: splitting data so writes scale, and the hot-spot problems that decide whether your shard key was any good.

Recall

• Three topologies: single-leader (no write conflicts, simple, but write bottleneck + stale follower reads — the right default), multi-leader (write locality/availability; conflicts guaranteed — adopt deliberately), leaderless/quorum (W + R > N overlaps reads with writes; read repair + anti-entropy).
• Sync axis: async (fast, lossy on failover — "committed" is per-node), synchronous (no loss, pays latency and stalls on a slow follower), semi-sync (one follower or a quorum — the practical default). Frame the choice as how many acknowledged writes may we lose?
• Failover = detect (timeout — can't tell slow from dead) → elect (most up-to-date follower) → reconfigure. Disasters: lost writes (unreplicated tail discarded), split brain (partitioned old leader still writing — cure with fencing/quorum), failover storms (aggressive timeouts; use hysteresis).
• Lag anomalies and cures: read-your-writes (route the writer's reads to leader or a caught-up replica by LSN) · monotonic reads (pin a user to one replica — stale but never regressing) · consistent prefix (keep causally related writes in one partition).
• Multi-leader conflict menu: LWW (silent loss) → node-id → application merge → CRDTs → surface to user. Choose before deploying.
• Lens: replication buys availability + read scale, never write scale (that's partitioning); lag is a per-read product decision; failover is a durability event — rehearse it.

Self-test: Which topology has no write conflicts and why? What exactly is lost on async failover? Name the three lag anomalies with their cures. What does W + R > N guarantee — and what does it not? Why doesn't adding replicas help a write-bound system?

Quiz Bank

FoundationalCompare single-leader, multi-leader, and leaderless replication.

Single-leader: one node accepts writes, followers replicate and serve reads. No write conflicts ever (a single writer gives free ordering — 10.3), simple failure reasoning, best tooling; costs are a write bottleneck, a failover event with potential data loss, and stale follower reads (section 4's anomalies). Default choice for most systems.

Multi-leader: multiple nodes accept writes and replicate to each other — used for multi-region write locality, offline-first clients (each device is a leader), and collaborative apps. Buys write availability during partitions and low write latency everywhere; costs guaranteed conflicts requiring a resolution policy (merge, CRDTs, user prompt — never silent LWW).

Leaderless (Dynamo-style): writes go to several replicas, reads read several, with quorums — W + R > N ensures a read set intersects the latest write set. No failover step (any node accepts writes), excellent availability, and consistency tuned per operation; costs are conflict handling on read (which value is newest — versions, not timestamps), plus repair machinery (read repair, anti-entropy with Merkle trees). Selection rule: start single-leader; move to leaderless for always-writable high-availability stores; adopt multi-leader only when geography or offline clients force it, with the conflict policy decided up front.

FoundationalExplain the three replication-lag anomalies and their standard cures.

Read-your-writes: a user writes (leader) then reads (lagging follower) and doesn't see their own change — the most user-visible and trust-destroying anomaly. Cures: route reads to the leader for a window after that user writes; or compare the replica's applied log position (LSN) against the write's position and only use replicas that have caught up; or have the client hold its own write optimistically until confirmed.

Monotonic reads: successive reads hit replicas with different lag, so the user sees data go backward (a comment appears then vanishes, a counter decreases). Cure: sticky replica routing per user (hash the user id) — the view may be stale but never regresses.

Consistent prefix: with independently-replicating partitions, causally ordered writes can be observed out of order (an answer before its question). Cure: keep causally related writes in the same partition (10.6 — e.g. partition by conversation id) or propagate causality metadata (10.3). All three share a diagnosis pattern: they appear only under load (when lag grows), affect a minority of requests, and are reported as "the app is buggy" rather than as staleness — which is why lag must be a monitored, alertable metric, not an assumption.

AppliedWhat does W + R > N guarantee in a quorum system, and what does it not?

With N replicas, writing to W and reading from R, W + R > N guarantees the read set and the write set overlap in at least one node — so any successful read touches at least one replica holding the latest acknowledged write. Common configurations: N=3, W=2, R=2 (balanced); W=N, R=1 (fast reads, slow/fragile writes); W=1, R=N (fast writes, expensive reads).

What it does not guarantee: (1) it doesn't tell you which returned value is newest — you still need versions/vector clocks to pick, and timestamps are unsafe (10.3); (2) it isn't linearizability — concurrent operations can still interleave surprisingly, and a failed write that reached some replicas leaves them holding a value that may later "win"; (3) it says nothing about durability across correlated failures (three replicas in one rack); (4) sloppy quorums (accepting writes on any W reachable nodes, with hinted handoff) explicitly break the overlap guarantee in exchange for availability — a legitimate trade you must know you're making. The complete answer names the accompanying machinery:

read repair (update stale replicas discovered during reads) and anti-entropy (background Merkle-tree comparison) are what actually converge the system over time.

InterviewYour leader dies. Walk through what happens and what can go wrong.

Detect: a monitor or the followers notice missed heartbeats past a timeout — which cannot distinguish a crashed leader from a slow/partitioned one (10.3's FLP), so this step is a guess with consequences. Elect: promote the most up-to-date follower — via consensus (10.7.2) or an external controller; "most up-to-date" is measured by replication position, and choosing wrong means discarding more writes.

Reconfigure: clients, remaining followers, and any dependent systems must learn the new leader (connection strings, service discovery, DNS with its caching lag — 10.2). What goes wrong: lost writes — under async replication, writes acknowledged to clients but not yet shipped are simply gone; catastrophic when those writes escaped elsewhere (an order id given to a customer, an event published downstream) because now systems disagree about what happened.

Split brain — the old leader was partitioned, not dead; it keeps accepting writes, and reconciling two divergent write histories afterward is manual surgery; cures are quorum-based election (a minority can't elect), leases plus fencing tokens so the storage rejects the old leader's writes (10.4), or STONITH.

Failover storms — aggressive timeouts promote during a transient blip, and the churn (cold caches, reconnect storms, another timeout) is worse than the blip; cure with hysteresis and conservative thresholds. Split-second dual writes to derived systems — downstream consumers may see the same events twice from both leaders, which is why consumers should be idempotent (10.4).

StaffA social product's users complain: 'my post disappears after I publish it, then comes back a minute later.' Reads are served from three replicas behind a round-robin balancer, writes go to the leader, replication is async. Diagnose, fix, and design the guarantee model.

Diagnosis: two lag anomalies compounding. The immediate one is read-your-writes — the publish goes to the leader; the redirect/refresh read is balanced round-robin to a follower that hasn't applied it yet, so the author's own post is missing. The "comes back, then vanishes again on refresh" pattern is monotonic reads — successive reads land on replicas with different lag, so the timeline oscillates. Confirm with data rather than theory: expose per-replica lag as a metric (seconds behind leader), sample it during peak, and correlate complaint timestamps with lag spikes — expect the complaints to cluster exactly where lag exceeds the user's think-time.

Fix, layered: (1) Read-your-writes — after any write, mark the session with the write's log position and a short window (e.g. 10 s); during that window, route that user's reads to the leader or to a replica whose applied position ≥ the recorded one (the precise version — it keeps most traffic on replicas instead of pushing everyone to the leader, which is the naive fix that then makes the leader the bottleneck). (2)

Monotonic reads — sticky per-user replica selection (hash the user id) so a user's view never moves backward, with fallback on replica failure. (3) Lag control — alert on lag SLO breach, and investigate the causes (long transactions, batch jobs on the leader, network saturation, single-threaded apply on the replica).

Guarantee model to write down — the deliverable that outlives this incident: classify every read path with an explicit consistency level: strong (leader) for post-write confirmation, balances, and anything a user just changed; bounded-staleness (replica with lag < X, else leader) for feeds and profiles; best-effort (any replica, CDN-cacheable) for public content (10.2). Encode it in the data-access layer as an argument (read(query, { consistency: "strong" })) so the choice is visible in code review, not implied by which client someone imported.

The lens for the write-up: replication lag isn't a bug to eliminate — it's the price of read scale, and the engineering work is deciding, per read, how much staleness is acceptable and enforcing that decision in routing rather than hoping.

Flashcards

FlashThree topologies

Single-leader (no conflicts, write bottleneck) · multi-leader (locality/offline; conflicts guaranteed) · leaderless quorum (W+R>N overlap; read repair + anti-entropy).

FlashSync trade

Async = fast, loses the unreplicated tail on failover. Sync = no loss, pays RTT and stalls on slow followers. Semi-sync (one/quorum) = the practical default.

FlashFailover disasters

Lost writes (async tail) · split brain (partitioned old leader — fence/quorum) · failover storms (aggressive timeouts; add hysteresis).

FlashLag anomalies

Read-your-writes (route writer to leader/caught-up replica) · monotonic reads (pin user to one replica) · consistent prefix (co-partition causal writes).

FlashW + R > N

Read set overlaps write set ⇒ latest value is present. Doesn't say which value is newest (need versions), isn't linearizability, and sloppy quorums break it deliberately.

FlashReplication ≠ write scale

Every replica takes every write. Replicas = availability + read scale; write scale = partitioning (10.6).

Scenario Drill

DrillDesign the replication strategy for a global SaaS with EU data-residency requirements: customers are pinned to a region, users travel, dashboards are read-heavy, audit logs must never be lost, and the EU tenant's data may not be stored outside the EU. Specify topology, sync mode, read routing, and the two hardest constraints this combination creates.

Topology — partition by tenant, single-leader per region. Data residency makes this straightforward and turns a hard problem (multi-region consistency) into an easy one: each tenant's data lives entirely in one region with a single-leader cluster there plus in-region followers. There is no cross-region replication of tenant data — which satisfies residency by construction rather than by policy, and simultaneously avoids multi-leader conflicts entirely (section 1's "adopt deliberately" advice, sidestepped). What is global is a small, non-personal control plane (tenant → region routing, feature flags, billing metadata), replicated read-only worldwide with generous staleness.

Sync mode — differentiated by data class. Audit logs: synchronous or semi-synchronous replication to at least one in-region follower before acknowledging, because "must never be lost" is a durability requirement that async cannot honor — the acknowledged-but-unreplicated tail is exactly what a leader crash discards (section 2). Everything else:

semi-synchronous (one follower ack) as the balanced default, with the loss window documented. Additionally, audit records go to append-only storage with object-lock semantics — replication protects against node loss, immutability protects against the other failure mode (a bad actor or bad migration deleting them).

Read routing — three levels, enforced in the data layer ([sectionStaff]'s model): strong/leader reads for post-write confirmation and anything the user just changed (read-your-writes without pushing all traffic to the leader); bounded-staleness replica reads for dashboards (heavily read, tolerant of seconds — with per-user sticky replica selection for monotonic reads); best-effort/cached for static and cross-tenant aggregates.

The two hardest constraints this combination creates. (1) Travelling users vs pinned data: a user physically in Singapore whose tenant lives in Frankfurt will experience ~150–200 ms round trips no matter what caching you do for writes and strong reads. You cannot replicate their data closer (residency forbids it), so the honest engineering is latency masking: an edge tier that terminates TLS near the user and keeps a warm connection to the home region (10.2), aggressive client-side optimism for writes with server confirmation, and a UI that doesn't block on strong reads. Say this to the business explicitly — it's a physics-and-law constraint, not a tuning opportunity. (2)

Cross-tenant analytics: any global dashboard aggregating all tenants pulls EU personal data out of the EU the moment it's computed centrally. The resolution is compute in-region, export only aggregates — each region produces anonymized/aggregated rollups that leave the region, with the pipeline's data classification reviewed by legal ([Part 8.7]); a naive central data warehouse would be a compliance incident wearing a BI dashboard.

Failover posture: in-region only (a Frankfurt outage must not fail over to Virginia — residency again), so regional redundancy must be genuinely multi-AZ, failover must be rehearsed, and the disaster-recovery story is "restore in-region from in-region backups," with RTO/RPO stated and tested. The design-doc sentence: residency converted our replication problem into a partitioning problem, which is the good trade — the remaining costs are travel latency and analytics topology, and both are business decisions we surfaced rather than engineering problems we hid.