10.11 — Evolution & Cost: Deploying, Migrating, and Paying for It

Systems that survive are systems that can change safely. Changing safely takes four kinds of tool:

• ways to deploy: blue-green, canary, rolling and feature flags;
• changing the database schema without downtime;
• staying backward compatible, as a habit;
• and the strangler fig, for replacing an old system piece by piece.

After those come the levers that drive real bills: N+1 queries, fetching too much or too little, hot and cold paths, and the trade between speed and volume.

1. Deployment strategies

Each buys a different property; the right answer depends on what failure you fear most: ^{⚑Blue-green, canary and rolling deployments. [EQ-1091b]}

• Rolling — replace instances in batches. Default in Kubernetes: no extra capacity beyond a surge, gradual, but two versions run simultaneously (which is fine only if your API and schema are backward compatible — section 3) and rollback is another rolling operation (slow).
• Blue-green — stand up a complete new environment (green), test it, then switch traffic wholesale. Instant rollback (switch back), clean separation, one version serving at a time — at the cost of double infrastructure during the switch, and a hard problem with shared state: the database is usually not duplicated, so schema compatibility across both versions is still required.
• Canary — route a small slice (1%, then 5%, 25%…) to the new version while watching metrics, promoting or aborting based on data. The best risk-per-deploy profile because problems are detected with 1% of users affected. Needs traffic-splitting infrastructure and, decisively, automated analysis (compare error rate and latency between canary and baseline, roll back automatically) — a canary a human forgets to watch is just a slow rollout.
• Feature flags — decouple deploy from release: ship the code dark, enable per user/segment/percentage, and kill instantly without a deploy. This is the most powerful of the four for product risk, and its cost is flag debt (stale flags multiply code paths and testing combinatorics — every flag needs an owner and an expiry).

The mature combination in practice: rolling or blue-green for the deploy mechanism + feature flags for the release decision + canary analysis for automated safety — three different concerns that teams often conflate into one argument.

2. Schema migrations without downtime

The rule that governs everything: during a rolling deploy, old and new code run simultaneously against the same database, so any migration must be compatible with both. Hence the expand–migrate–contract pattern (also called parallel change): ^{⚑How do you do zero-downtime schema migrations? [EQ-993b]}

Renaming a column, correctly, in four deploys:

1. Expand — add the new column, nullable; deploy. (Old code ignores it; new code doesn't exist yet.)
2. Dual-write — deploy code that writes both columns and reads the old one. Backfill existing rows in batches (never one giant UPDATE: it locks, blows up replication lag, and can fill the WAL — 10.5).
3. Flip reads — deploy code that reads the new column (still writing both), verify with metrics/comparison for a soak period.
4. Contract — stop writing the old column, deploy, then drop it later (after you're certain no rollback will need it — dropping a column is the one irreversible step).

The same shape covers table splits, type changes, and adding constraints: never change and use a schema element in the same deploy. More rules learned the hard way:

• On some databases, adding a NOT NULL column with a default rewrites the whole table. Add it as nullable, fill in the values, and then add the constraint. Check how your database version behaves.
• Create indexes CONCURRENTLY, or online, or they block writes.
• Long migrations must be able to resume, by working in chunks and recording progress, because they will be interrupted.

The rollback question deserves explicit thought: code rolls back in seconds, data does not. So every migration should be forward-compatible (old code tolerates the new schema) for at least one deploy cycle — which is precisely what expand-migrate-contract guarantees, and why it's worth the extra deploys.

3. Backward compatibility as a discipline

Compatibility isn't only an API concern (9.6.3) — in a distributed system it applies to every contract crossing a version boundary:

• API: additive changes only within a version; tolerant readers; deprecate with headers and telemetry.
• Events: this is the one teams forget. Events are read by consumers you don't control and by replays of history (10.8.4) — so event schemas need versioning, optional-with-default fields, and a compatibility policy enforced by a schema registry (Avro/Protobuf with backward/forward compatibility checks in CI). An event schema change that breaks replay is a data-model change disguised as a deploy.
• Database: section 2's expand-contract.
• Messages in flight: during a deploy, queued messages produced by the old version are consumed by the new one (and vice versa) — so consumers must handle both shapes for at least one release.

The strangler fig is the same discipline applied to replacing a whole system: route traffic through a facade, implement new functionality in the new system while the old one still serves the rest, migrate capability by capability, and delete the old system when nothing routes to it. It's incremental, reversible at each step, and the only responsible way to replace a system with users — as opposed to the big-bang rewrite, whose failure rate is legendary because it must reproduce years of accumulated behavior before delivering any value.

Figure 1 — Expand, migrate, contract. Four deploys instead of one, because at every moment both the previous and the current version of the application must work against the schema as it exists. Only the final drop is irreversible, and it happens after confidence is earned.

4. Performance and cost levers

The recurring inefficiencies that dominate both latency and bills:

• N+1 queries — fetch a list, then query per item: 1 + N round trips where 2 would do. The classic ORM lazy-load trap (9.4.10's proxy dark side); cures are eager loading (include), batching (DataLoader), or a join. It reappears at service scale as chatty inter-service calls — the distributed N+1, which is far more expensive because each hop is milliseconds rather than microseconds.
• Over-fetching and under-fetching — returning 40 fields when 4 are needed (bandwidth, serialization, cache pressure) versus forcing clients into multiple round trips (9.6.2's projection question; BFFs and GraphQL are two answers — 10.8.3).
• Hot vs cold paths — optimize what runs constantly, leave the rest simple. The 90/10 rule holds hard in practice: caching the hot path (10.2) usually beats optimizing everything, and knowing which path is hot requires measurement (10.10), not intuition.
• Latency and throughput trade against each other.
  • Batching raises throughput, but adds delay, as with Kafka's linger.ms (10.8.2).
  • More work at once raises throughput, until queueing makes delay shoot up. Past about 70 to 80% busy, delay grows much faster than load. That is why capacity planning aims for spare room, not for running everything flat out (10.12).
• The cost levers that actually move cloud bills: egress bandwidth (cross-region and internet — often the biggest surprise), always-on over-provisioned instances, storage tiers and retention (logs and backups grow silently — 10.10), and idle non-production environments. The engineering-adjacent truth: the cheapest request is the one you don't serve (caching, CDN), and the second cheapest is the one you serve from the cheapest tier.

5. The expert lens

Deploy frequency is a reliability metric, not a velocity vanity metric. Teams that deploy rarely accumulate large, risky changes; teams that deploy continuously ship small, reversible ones — and the research consensus (DORA) is that high deploy frequency correlates with lower change-failure rates and faster recovery. Every mechanism on this page — canary, flags, expand-contract, strangler — exists to make small changes safe, which is what makes frequent deployment possible, which is what makes changes small. It's a virtuous cycle you enter by investing in the mechanisms.

Migrations are where confidence is bought, not assumed. Every step in expand-migrate-contract exists so that the next step is reversible. The corresponding discipline: verify with data at each stage (row-count and checksum comparisons, dual-read comparison in production with mismatch metrics), and keep the old path warm until the new one has soaked. Teams that skip verification discover their migration's flaw during the contract step — the one point where rollback is no longer available.

Cost is a design property, and it is usually decided by architecture rather than by tuning. A chatty inter-service design pays network cost on every request forever; a system that stores raw events for a year pays storage forever; a cross-region read path pays egress forever. Reviewing designs for recurring cost — "what does this cost per request, per stored record, per month at 10×?" — catches decisions that no amount of later optimization can undo, and it's a question that should appear in design review alongside latency and availability.

Next: 10.12 — the arithmetic that makes these judgments concrete: latency numbers, capacity math, and back-of-envelope estimation under interview conditions.

Recall

• Deployments:
  • Rolling is the default. Two versions run side by side, and rolling back is slow.
  • Blue-green rolls back instantly, but needs double the servers, and the shared database must still work with both versions.
  • Canary has the lowest risk, because only 1% of users see the new version first. It needs traffic splitting and automatic comparison of metrics.
  • Feature flags separate deploying code from releasing a feature, and let you switch a feature off instantly. The cost is old flags piling up, so each needs an owner and an expiry date.
  • The mature combination: rolling or blue-green, plus flags, plus automatic canary analysis.
• Zero-downtime migrations = expand–migrate–contract, because old and new code run simultaneously: add nullable → dual-write + batched backfill → flip reads (soak, verify) → stop writing → drop later (the only irreversible step). Never change and use a schema element in one deploy; index creation must be online; long migrations must be resumable.
• Backward compatibility applies to four contracts: API (additive, tolerant readers, telemetry-driven deprecation), events (versioned schemas + registry with CI compatibility checks — replay reads old events forever), database (expand-contract), and messages in flight during deploys. Strangler fig applies the same discipline to replacing whole systems — incremental, reversible, no big-bang rewrite.
• Levers for speed and cost:
  • N+1 queries, and the distributed version: services making many small calls to each other.
  • Fetching too much or too little. Fix it with projections, a BFF or GraphQL.
  • Hot and cold paths. Optimise the 10% of code that runs all the time, measured, not guessed.
  • Latency versus throughput. Batching trades one for the other. Latency shoots up once a resource is more than about 70 to 80% busy.
  • What really drives the bill: data sent out (egress), servers bigger than needed, keeping data too long, and environments sitting idle.
• Lens: deploy frequency is a reliability metric (small reversible changes); migrations buy confidence step by step (verify with dual-read comparisons before the irreversible contract); cost is architectural — ask what a design costs per request/record/month at 10×.

Self-test: Why do rolling deploys constrain schema changes? Walk the four migration steps and name the only irreversible one. Which four contracts need backward compatibility, and which is most often forgotten? What happens to latency past 80% utilization, and why does that shape capacity planning?

Quiz Bank

FoundationalCompare rolling, blue-green, canary and feature flags — what does each actually buy?

Rolling replaces instances in batches: no extra capacity needed, gradual exposure, and the Kubernetes default — but two versions run simultaneously, so every API and schema change must be compatible with both (section 2), and rollback is another slow rolling pass.

Blue-green runs a full parallel environment and switches traffic at once: instant rollback (flip back), a single version serving at any moment, and a clean smoke-test window — at the cost of double infrastructure during the cutover and the fact that the database is usually shared, so schema compatibility is still required (the most common misunderstanding of blue-green).

Canary exposes a small percentage to the new version while comparing metrics against the baseline: the best risk-per-deploy profile, because a bad release harms 1% of users for minutes rather than 100% — but it requires traffic-splitting infrastructure and, critically, automated analysis with automatic rollback; a canary that a human is supposed to watch is just a slower rolling deploy.

Feature flags decouple deployment from release: code ships dark and is enabled per segment/percentage, with instant disable and no deploy — the strongest tool for product risk and A/B testing; its cost is flag debt, since each live flag doubles code paths and testing combinations, so flags need owners and expiry dates. They compose rather than compete: a rolling or blue-green deploy mechanism, feature flags for the release decision, and canary analysis as the automated safety net.

FoundationalWhy can't you just rename a column, and what is the correct sequence?

Because during any rolling deploy — and in blue-green with a shared database — old and new application code run at the same time against the same schema. A rename breaks the old code instantly (its queries reference a column that no longer exists), so the deploy fails halfway and rollback is impossible without a second migration. The correct sequence is expand–migrate–contract:

(1) Expand — add the new column, nullable, deploying nothing that uses it; both versions still work. (2) Dual-write — deploy code that writes both columns and reads the old one; backfill existing rows in batches (a single large UPDATE locks rows, spikes replication lag, and can exhaust WAL/undo space — 10.5), with progress tracking so it's resumable.

(3) Flip reads — deploy code that reads the new column while still writing both; soak, and ideally run a dual-read comparison that reports mismatches as a metric before trusting it. (4) Contract — deploy code that no longer writes the old column, then drop it later, once you're confident no rollback will need it — this is the only irreversible step, and it's deliberately last. The same shape handles type changes, table splits, and constraint additions; the general rule is never change and use a schema element in the same deploy, because code rolls back in seconds and data does not.

AppliedWhich contracts need backward compatibility in a distributed system, and which one do teams most often forget?

Four. (1) APIs — additive-only within a version, tolerant readers, deprecation via headers plus per-consumer telemetry (9.6.3). (2) Database schemas — expand-migrate-contract, because two code versions share one database (section 2).

(3) Messages in flight — during any deploy, the queue contains messages produced by the old version being consumed by the new one, and vice versa on rollback; consumers must accept both shapes for at least one release, which is easy to overlook because it only manifests during deploys.

(4) Event schemas — the forgotten one. Events are consumed by services you don't control and replayed from history (10.8.4): a schema change that breaks old events breaks replay forever, turning your event log — the thing you built for auditability and rebuildability — into partially-unreadable data. The controls: a schema registry (Avro/Protobuf) with backward/forward compatibility checks enforced in CI, additive fields with defaults, explicit versioning, and upcasting logic for readers (10.8.4's event-sourcing cost). The rule that unifies all four: any contract that crosses a version boundary — in space (another service) or in time (a replay, a queued message) — must tolerate both sides being different versions.

InterviewExplain the latency/throughput trade and why capacity planning targets headroom.

They are different quantities that trade against each other. Throughput is work per unit time; latency is time per unit of work. Batching improves throughput (fewer round trips, better amortization, compression wins) while increasing latency (a message waits for its batch — Kafka's linger.ms, 10.8.2). Concurrency raises throughput until contention and queueing dominate, after which latency degrades sharply for no throughput gain.

Why spare room? Queueing theory says waiting time grows much faster than how busy a server is. Roughly:

\text{latency} \approx \frac{\text{service time}}{1 - \text{utilisation}}

Read it as: "the delay is the time the work takes, divided by the fraction of time the server is idle". So at 50% busy, the total delay is about twice the work time. At 80% busy it is five times. At 95% busy it is twenty times.

That sharp bend is why systems planned to run "efficiently" at 90% have terrible slow requests, and fall over when traffic rises a little. There is no room to soak up bumps, and bumps always come: retries, garbage-collection pauses, deploys and traffic spikes.

What to do about it:

• Aim for about 50 to 70% busy in normal running, for services where speed matters.
• Measure the slowest requests (p99, p99.9), not averages. The slow tail is what adds up when one request calls many others (10.6).
• Autoscaling does not remove the need for spare room. It only adds capacity automatically, after a delay, and a spike can arrive faster than that.

StaffYou must replace a 12-year-old monolithic billing system that processes €50M/month, with no downtime, while continuing to ship features. Design the program.

Strategy: strangler fig, capability by capability, never a big-bang rewrite — because reproducing 12 years of accumulated rules before delivering any value is the failure mode with the worst track record in our industry, and with €50M/month flowing through, "we cut over and it mostly worked" is not survivable.

Step 1 — build the facade. Route all billing traffic through a routing layer (API gateway or an in-app dispatcher) that today forwards 100% to the legacy system. This adds no behavior but creates the seam through which every subsequent migration flows, and it's where per-capability routing rules will live.

Step 2 — characterize before touching. Capture the legacy system's behavior for the capabilities you'll move: golden-master tests over real (anonymized) inputs, because the specification is the current behavior, quirks included, and discovering that only after cutover is how billing incidents happen.

Step 3 — pick the first capability by risk-adjusted value: something with clear boundaries, meaningful pain, and low blast radius (often invoice PDF generation or dunning notifications — not the core charge path). Implement it in the new system, then run it in shadow mode: the facade sends the request to both, serves the legacy response, and compares — mismatches become a metric and a work queue. This is the single most valuable technique in the program: it buys confidence with production traffic at zero user risk.

Step 4 — cut over that capability behind a flag, ramped by percentage and by customer segment (internal accounts first, then small customers, then large), with instant rollback and reconciliation checks comparing money movements between systems daily (10.4).

Step 5 — repeat, sequencing toward the core, and delete legacy code as soon as nothing routes to it (leaving dead code is how the "replacement" becomes a second system to maintain — the lava-flow anti-pattern, 9.4.24).

Data: the new system owns new data with expand-migrate-contract migrations (section 2); shared reference data gets a single owner and events for the rest (10.4); money records are append-only in both systems so reconciliation is always possible.

Shipping features meanwhile: new functionality is built in the new system behind the facade — which is what makes the program politically survivable, because it delivers value continuously instead of asking for a two-year feature freeze. Governance: a monthly reconciliation report (legacy vs new totals), a published capability-migration board, and an explicit rule that any capability whose shadow-mode mismatch rate doesn't reach zero does not cut over.

The framing for the steering committee: we are not rewriting a system; we are moving capabilities one at a time behind a facade, with every step verifiable, reversible, and delivering value — the only approach with an acceptable failure mode when €50M a month is in flight.

Flashcards

FlashDeployment strategies

Rolling (default, two versions coexist) · blue-green (instant rollback, double infra, shared DB) · canary (1% exposure + automated analysis) · flags (deploy ≠ release; cost = flag debt).

FlashExpand-migrate-contract

Add nullable → dual-write + batched backfill → flip reads (soak/verify) → stop writing → drop later (only irreversible step). Never change and use in one deploy.

FlashFour compatibility contracts

API · database · messages in flight during deploys · EVENT schemas (the forgotten one — replay reads old events forever; use a schema registry with CI checks).

FlashStrangler fig

Facade in front → migrate capability by capability (shadow mode, then flagged ramp) → delete legacy when nothing routes to it. The only responsible replacement of a live system.

FlashUtilization knee

Latency ≈ service/(1−utilization): 50% ⇒ ~2×, 80% ⇒ ~5×, 95% ⇒ ~20×. Plan 50–70% steady state; measure tails, not averages.

FlashCost is architectural

Egress, over-provisioning, retention, idle envs dominate bills. Chatty designs and long retention pay forever — ask cost per request/record/month at 10× in design review.

Scenario Drill

DrillYour team must split a heavily-used 'users' table across two services (identity and profile), with 40M rows, 3k writes/sec, and zero downtime. Design the migration, the verification at each step, and the rollback plan for every stage.

Frame it as expand-migrate-contract at the service level, with the database split last — the sequencing that keeps every step reversible. Stage 0 — establish the seam (no data movement). Introduce a ProfileService module inside the current application with its own interface, and route all profile reads/writes through it while it still reads and writes the same tables (9.9.6's modular-monolith step). Verification: no behavior change — errors and latency flat. Rollback: revert one deploy. This stage does the hard part (finding every access path) with zero risk, and it's the stage teams skip and regret.

Stage 1 — expand the schema. Create the new profile tables in the target store; add nothing to the read path. Verification: schema exists, migrations resumable, no production impact. Rollback: drop the unused tables. Stage 2 — dual-write with backfill. ProfileService writes to both old and new stores in the same request path (accepting the latency cost temporarily), while a batched, resumable backfill copies 40M rows in chunks sized to keep replication lag within SLO (10.5) — at 3k writes/sec, the backfill must handle rows changing under it, so use a watermark plus a second pass over rows modified during the run. Verification: a continuous comparison job sampling both stores and reporting mismatch rate as a metric (target: zero, and investigate every non-zero — mismatches here are the bugs that would otherwise appear after cutover); backfill progress dashboard; replication lag and write-latency alarms. Rollback: stop dual-writing (new store becomes stale but harmless).

Stage 3 — flip reads behind a flag, ramped. Read from the new store for 1% → 10% → 50% → 100% of traffic, comparing responses in shadow for a soak period at each step. Verification: response-comparison mismatch rate, p99 latency of profile reads, error rate by segment. Rollback: flip the flag — seconds, no deploy (this is why the flag exists rather than a deploy-based switch).

Stage 4 — extract the service. Move ProfileService out of process, behind an API; the interface established in stage 0 means callers change a client, not their logic. Verification: latency budget for the new network hop, circuit breaker and fallback configured (10.9), tracing spans present (10.10). Rollback: route back to the in-process implementation (keep it deployable for at least one release).

Stage 5 — contract. Stop writing profile columns in the old table; after a soak, drop them. Verification: zero writes observed in metrics for the full soak; a final comparison run. Rollback: none for the drop — hence the soak, hence doing it last, and hence taking a verified backup immediately before.

Decisions that apply throughout, which you should state:

• Identity stays the only owner of the login fields, and profile owns the rest. No field is owned by both (10.4).
• Any query that used to join identity and profile must be replaced. Either the BFF or the caller makes two calls. Or, on a busy path, a copy of the needed fields is kept up to date (10.8.4).
• Plan for losing that join before stage 4. Discovering it afterwards is exactly what makes teams undo an extraction.

The plan's shape is the lesson: every stage is independently deployable, independently verifiable, and independently reversible — except the last one, which happens only after the evidence is overwhelming.