10.11 — Evolution & Cost: Deploying, Migrating, and Paying for It

Systems that survive are systems that change safely. This page covers the mechanics of change — deployment strategies (blue-green, canary, rolling, feature flags), schema migrations without downtime, backward compatibility as a discipline, and the strangler fig for replacing systems — then the performance and cost levers that dominate real bills: N+1s, over/under-fetching, hot vs cold paths, and latency versus throughput.

1. Deployment strategies

Each buys a different property; the right answer depends on what failure you fear most: ^{⚑Blue-green, canary and rolling deployments. [EQ-1091b]}

• Rolling — replace instances in batches. Default in Kubernetes: no extra capacity beyond a surge, gradual, but two versions run simultaneously (which is fine only if your API and schema are backward compatible — section 3) and rollback is another rolling operation (slow).
• Blue-green — stand up a complete new environment (green), test it, then switch traffic wholesale. Instant rollback (switch back), clean separation, one version serving at a time — at the cost of double infrastructure during the switch, and a hard problem with shared state: the database is usually not duplicated, so schema compatibility across both versions is still required.
• Canary — route a small slice (1%, then 5%, 25%…) to the new version while watching metrics, promoting or aborting based on data. The best risk-per-deploy profile because problems are detected with 1% of users affected. Needs traffic-splitting infrastructure and, decisively, automated analysis (compare error rate and latency between canary and baseline, roll back automatically) — a canary a human forgets to watch is just a slow rollout.
• Feature flags — decouple deploy from release: ship the code dark, enable per user/segment/percentage, and kill instantly without a deploy. This is the most powerful of the four for product risk, and its cost is flag debt (stale flags multiply code paths and testing combinatorics — every flag needs an owner and an expiry).

The mature combination in practice: rolling or blue-green for the deploy mechanism + feature flags for the release decision + canary analysis for automated safety — three different concerns that teams often conflate into one argument.

2. Schema migrations without downtime

The rule that governs everything: during a rolling deploy, old and new code run simultaneously against the same database, so any migration must be compatible with both. Hence the expand–migrate–contract pattern (also called parallel change): ^{⚑How do you do zero-downtime schema migrations? [EQ-993b]}

Renaming a column, correctly, in four deploys:

1. Expand — add the new column, nullable; deploy. (Old code ignores it; new code doesn't exist yet.)
2. Dual-write — deploy code that writes both columns and reads the old one. Backfill existing rows in batches (never one giant UPDATE: it locks, blows up replication lag, and can fill the WAL — 10.5).
3. Flip reads — deploy code that reads the new column (still writing both), verify with metrics/comparison for a soak period.
4. Contract — stop writing the old column, deploy, then drop it later (after you're certain no rollback will need it — dropping a column is the one irreversible step).

The same shape covers table splits, type changes, and adding constraints: never change and use a schema element in the same deploy. Additional hard-won rules: adding a NOT NULL column with a default rewrites the table on some engines (add nullable, backfill, then add the constraint — verify per database version); creating indexes must be CONCURRENTLY/online or it locks writes; and long-running migrations need to be resumable (chunked with progress tracking) because they will be interrupted.

The rollback question deserves explicit thought: code rolls back in seconds, data does not. So every migration should be forward-compatible (old code tolerates the new schema) for at least one deploy cycle — which is precisely what expand-migrate-contract guarantees, and why it's worth the extra deploys.

3. Backward compatibility as a discipline

Compatibility isn't only an API concern (9.6.3) — in a distributed system it applies to every contract crossing a version boundary:

• API: additive changes only within a version; tolerant readers; deprecate with headers and telemetry.
• Events: this is the one teams forget. Events are read by consumers you don't control and by replays of history (10.8.4) — so event schemas need versioning, optional-with-default fields, and a compatibility policy enforced by a schema registry (Avro/Protobuf with backward/forward compatibility checks in CI). An event schema change that breaks replay is a data-model change disguised as a deploy.
• Database: section 2's expand-contract.
• Messages in flight: during a deploy, queued messages produced by the old version are consumed by the new one (and vice versa) — so consumers must handle both shapes for at least one release.

The strangler fig is the same discipline applied to replacing a whole system: route traffic through a facade, implement new functionality in the new system while the old one still serves the rest, migrate capability by capability, and delete the old system when nothing routes to it. It's incremental, reversible at each step, and the only responsible way to replace a system with users — as opposed to the big-bang rewrite, whose failure rate is legendary because it must reproduce years of accumulated behavior before delivering any value.

Figure 1 — Expand, migrate, contract. Four deploys instead of one, because at every moment both the previous and the current version of the application must work against the schema as it exists. Only the final drop is irreversible, and it happens after confidence is earned.

4. Performance and cost levers

The recurring inefficiencies that dominate both latency and bills:

• N+1 queries — fetch a list, then query per item: 1 + N round trips where 2 would do. The classic ORM lazy-load trap (9.4.10's proxy dark side); cures are eager loading (include), batching (DataLoader), or a join. It reappears at service scale as chatty inter-service calls — the distributed N+1, which is far more expensive because each hop is milliseconds rather than microseconds.
• Over-fetching and under-fetching — returning 40 fields when 4 are needed (bandwidth, serialization, cache pressure) versus forcing clients into multiple round trips (9.6.2's projection question; BFFs and GraphQL are two answers — 10.8.3).
• Hot vs cold paths — optimize what runs constantly, leave the rest simple. The 90/10 rule holds hard in practice: caching the hot path (10.2) usually beats optimizing everything, and knowing which path is hot requires measurement (10.10), not intuition.
• Latency vs throughput — they trade: batching increases throughput and adds latency (Kafka's linger.ms — 10.8.2); more concurrency raises throughput until queueing raises latency sharply (the knee of the utilization curve — beyond ~70–80% utilization, latency rises non-linearly, which is why capacity planning targets headroom rather than efficiency — 10.12).
• The cost levers that actually move cloud bills: egress bandwidth (cross-region and internet — often the biggest surprise), always-on over-provisioned instances, storage tiers and retention (logs and backups grow silently — 10.10), and idle non-production environments. The engineering-adjacent truth: the cheapest request is the one you don't serve (caching, CDN), and the second cheapest is the one you serve from the cheapest tier.

5. The expert lens

Deploy frequency is a reliability metric, not a velocity vanity metric. Teams that deploy rarely accumulate large, risky changes; teams that deploy continuously ship small, reversible ones — and the research consensus (DORA) is that high deploy frequency correlates with lower change-failure rates and faster recovery. Every mechanism on this page — canary, flags, expand-contract, strangler — exists to make small changes safe, which is what makes frequent deployment possible, which is what makes changes small. It's a virtuous cycle you enter by investing in the mechanisms.

Migrations are where confidence is bought, not assumed. Every step in expand-migrate-contract exists so that the next step is reversible. The corresponding discipline: verify with data at each stage (row-count and checksum comparisons, dual-read comparison in production with mismatch metrics), and keep the old path warm until the new one has soaked. Teams that skip verification discover their migration's flaw during the contract step — the one point where rollback is no longer available.

Cost is a design property, and it is usually decided by architecture rather than by tuning. A chatty inter-service design pays network cost on every request forever; a system that stores raw events for a year pays storage forever; a cross-region read path pays egress forever. Reviewing designs for recurring cost — "what does this cost per request, per stored record, per month at 10×?" — catches decisions that no amount of later optimization can undo, and it's a question that should appear in design review alongside latency and availability.

Next: 10.12 — the arithmetic that makes these judgments concrete: latency numbers, capacity math, and back-of-envelope estimation under interview conditions.

Recall

• Deployments: rolling (default, two versions coexist, slow rollback) · blue-green (instant rollback, double infra, shared DB still needs compatibility) · canary (best risk profile — 1% exposure; requires traffic splitting and automated metric comparison) · feature flags (decouple deploy from release, instant kill, cost = flag debt with owners and expiry). Mature combo: rolling/blue-green + flags + automated canary analysis.
• Zero-downtime migrations = expand–migrate–contract, because old and new code run simultaneously: add nullable → dual-write + batched backfill → flip reads (soak, verify) → stop writing → drop later (the only irreversible step). Never change and use a schema element in one deploy; index creation must be online; long migrations must be resumable.
• Backward compatibility applies to four contracts: API (additive, tolerant readers, telemetry-driven deprecation), events (versioned schemas + registry with CI compatibility checks — replay reads old events forever), database (expand-contract), and messages in flight during deploys. Strangler fig applies the same discipline to replacing whole systems — incremental, reversible, no big-bang rewrite.
• Performance/cost levers: N+1 (and its distributed form: chatty service calls), over/under-fetching (projections, BFF, GraphQL), hot vs cold paths (optimize the 10% that runs constantly — measured, not guessed), latency vs throughput (batching trades one for the other; latency rises non-linearly past ~70–80% utilization), and the bill's real drivers — egress, over-provisioning, retention, idle environments.
• Lens: deploy frequency is a reliability metric (small reversible changes); migrations buy confidence step by step (verify with dual-read comparisons before the irreversible contract); cost is architectural — ask what a design costs per request/record/month at 10×.

Self-test: Why do rolling deploys constrain schema changes? Walk the four migration steps and name the only irreversible one. Which four contracts need backward compatibility, and which is most often forgotten? What happens to latency past 80% utilization, and why does that shape capacity planning?

Quiz Bank

FoundationalCompare rolling, blue-green, canary and feature flags — what does each actually buy?

Rolling replaces instances in batches: no extra capacity needed, gradual exposure, and the Kubernetes default — but two versions run simultaneously, so every API and schema change must be compatible with both (section 2), and rollback is another slow rolling pass.

Blue-green runs a full parallel environment and switches traffic at once: instant rollback (flip back), a single version serving at any moment, and a clean smoke-test window — at the cost of double infrastructure during the cutover and the fact that the database is usually shared, so schema compatibility is still required (the most common misunderstanding of blue-green).

Canary exposes a small percentage to the new version while comparing metrics against the baseline: the best risk-per-deploy profile, because a bad release harms 1% of users for minutes rather than 100% — but it requires traffic-splitting infrastructure and, critically, automated analysis with automatic rollback; a canary that a human is supposed to watch is just a slower rolling deploy.

Feature flags decouple deployment from release: code ships dark and is enabled per segment/percentage, with instant disable and no deploy — the strongest tool for product risk and A/B testing; its cost is flag debt, since each live flag doubles code paths and testing combinations, so flags need owners and expiry dates. They compose rather than compete: a rolling or blue-green deploy mechanism, feature flags for the release decision, and canary analysis as the automated safety net.

FoundationalWhy can't you just rename a column, and what is the correct sequence?

Because during any rolling deploy — and in blue-green with a shared database — old and new application code run at the same time against the same schema. A rename breaks the old code instantly (its queries reference a column that no longer exists), so the deploy fails halfway and rollback is impossible without a second migration. The correct sequence is expand–migrate–contract:

(1) Expand — add the new column, nullable, deploying nothing that uses it; both versions still work. (2) Dual-write — deploy code that writes both columns and reads the old one; backfill existing rows in batches (a single large UPDATE locks rows, spikes replication lag, and can exhaust WAL/undo space — 10.5), with progress tracking so it's resumable.

(3) Flip reads — deploy code that reads the new column while still writing both; soak, and ideally run a dual-read comparison that reports mismatches as a metric before trusting it. (4) Contract — deploy code that no longer writes the old column, then drop it later, once you're confident no rollback will need it — this is the only irreversible step, and it's deliberately last. The same shape handles type changes, table splits, and constraint additions; the general rule is never change and use a schema element in the same deploy, because code rolls back in seconds and data does not.

AppliedWhich contracts need backward compatibility in a distributed system, and which one do teams most often forget?

Four. (1) APIs — additive-only within a version, tolerant readers, deprecation via headers plus per-consumer telemetry (9.6.3). (2) Database schemas — expand-migrate-contract, because two code versions share one database (section 2).

(3) Messages in flight — during any deploy, the queue contains messages produced by the old version being consumed by the new one, and vice versa on rollback; consumers must accept both shapes for at least one release, which is easy to overlook because it only manifests during deploys.

(4) Event schemas — the forgotten one. Events are consumed by services you don't control and replayed from history (10.8.4): a schema change that breaks old events breaks replay forever, turning your event log — the thing you built for auditability and rebuildability — into partially-unreadable data. The controls: a schema registry (Avro/Protobuf) with backward/forward compatibility checks enforced in CI, additive fields with defaults, explicit versioning, and upcasting logic for readers (10.8.4's event-sourcing cost). The rule that unifies all four: any contract that crosses a version boundary — in space (another service) or in time (a replay, a queued message) — must tolerate both sides being different versions.

InterviewExplain the latency/throughput trade and why capacity planning targets headroom.

They are different quantities that trade against each other. Throughput is work per unit time; latency is time per unit of work. Batching improves throughput (fewer round trips, better amortization, compression wins) while increasing latency (a message waits for its batch — Kafka's linger.ms, 10.8.2). Concurrency raises throughput until contention and queueing dominate, after which latency degrades sharply for no throughput gain.

Why headroom: queueing theory says waiting time grows non-linearly with utilization — informally, latency ≈ service_time / (1 − utilization), so at 50% utilization you wait about as long as you're served, at 80% about four times, and at 95% roughly twenty times. That knee is why systems planned to run at 90% "efficient" utilization exhibit terrible tail latency and collapse under small traffic increases: you have no room to absorb variance, and variance is guaranteed (retries, GC pauses, deploys, traffic spikes). Practical consequences: target ~50–70% steady-state utilization for latency-sensitive services; measure the tail (p99, p99.9), not averages, because tail latency is what compounds through fan-out (10.6); and remember that autoscaling doesn't remove the need for headroom — it merely automates adding it, with a delay that the spike may outrun.

StaffYou must replace a 12-year-old monolithic billing system that processes €50M/month, with no downtime, while continuing to ship features. Design the program.

Strategy: strangler fig, capability by capability, never a big-bang rewrite — because reproducing 12 years of accumulated rules before delivering any value is the failure mode with the worst track record in our industry, and with €50M/month flowing through, "we cut over and it mostly worked" is not survivable.

Step 1 — build the facade. Route all billing traffic through a routing layer (API gateway or an in-app dispatcher) that today forwards 100% to the legacy system. This adds no behavior but creates the seam through which every subsequent migration flows, and it's where per-capability routing rules will live.

Step 2 — characterize before touching. Capture the legacy system's behavior for the capabilities you'll move: golden-master tests over real (anonymized) inputs, because the specification is the current behavior, quirks included, and discovering that only after cutover is how billing incidents happen.

Step 3 — pick the first capability by risk-adjusted value: something with clear boundaries, meaningful pain, and low blast radius (often invoice PDF generation or dunning notifications — not the core charge path). Implement it in the new system, then run it in shadow mode: the facade sends the request to both, serves the legacy response, and compares — mismatches become a metric and a work queue. This is the single most valuable technique in the program: it buys confidence with production traffic at zero user risk.

Step 4 — cut over that capability behind a flag, ramped by percentage and by customer segment (internal accounts first, then small customers, then large), with instant rollback and reconciliation checks comparing money movements between systems daily (10.4).

Step 5 — repeat, sequencing toward the core, and delete legacy code as soon as nothing routes to it (leaving dead code is how the "replacement" becomes a second system to maintain — the lava-flow anti-pattern, 9.4.24).

Data: the new system owns new data with expand-migrate-contract migrations (section 2); shared reference data gets a single owner and events for the rest (10.4); money records are append-only in both systems so reconciliation is always possible.

Shipping features meanwhile: new functionality is built in the new system behind the facade — which is what makes the program politically survivable, because it delivers value continuously instead of asking for a two-year feature freeze. Governance: a monthly reconciliation report (legacy vs new totals), a published capability-migration board, and an explicit rule that any capability whose shadow-mode mismatch rate doesn't reach zero does not cut over.

The framing for the steering committee: we are not rewriting a system; we are moving capabilities one at a time behind a facade, with every step verifiable, reversible, and delivering value — the only approach with an acceptable failure mode when €50M a month is in flight.

Flashcards

FlashDeployment strategies

Rolling (default, two versions coexist) · blue-green (instant rollback, double infra, shared DB) · canary (1% exposure + automated analysis) · flags (deploy ≠ release; cost = flag debt).

FlashExpand-migrate-contract

Add nullable → dual-write + batched backfill → flip reads (soak/verify) → stop writing → drop later (only irreversible step). Never change and use in one deploy.

FlashFour compatibility contracts

API · database · messages in flight during deploys · EVENT schemas (the forgotten one — replay reads old events forever; use a schema registry with CI checks).

FlashStrangler fig

Facade in front → migrate capability by capability (shadow mode, then flagged ramp) → delete legacy when nothing routes to it. The only responsible replacement of a live system.

FlashUtilization knee

Latency ≈ service/(1−utilization): 50% ⇒ ~2×, 80% ⇒ ~5×, 95% ⇒ ~20×. Plan 50–70% steady state; measure tails, not averages.

FlashCost is architectural

Egress, over-provisioning, retention, idle envs dominate bills. Chatty designs and long retention pay forever — ask cost per request/record/month at 10× in design review.

Scenario Drill

DrillYour team must split a heavily-used 'users' table across two services (identity and profile), with 40M rows, 3k writes/sec, and zero downtime. Design the migration, the verification at each step, and the rollback plan for every stage.

Frame it as expand-migrate-contract at the service level, with the database split last — the sequencing that keeps every step reversible. Stage 0 — establish the seam (no data movement). Introduce a ProfileService module inside the current application with its own interface, and route all profile reads/writes through it while it still reads and writes the same tables (9.9.6's modular-monolith step). Verification: no behavior change — errors and latency flat. Rollback: revert one deploy. This stage does the hard part (finding every access path) with zero risk, and it's the stage teams skip and regret.

Stage 1 — expand the schema. Create the new profile tables in the target store; add nothing to the read path. Verification: schema exists, migrations resumable, no production impact. Rollback: drop the unused tables. Stage 2 — dual-write with backfill. ProfileService writes to both old and new stores in the same request path (accepting the latency cost temporarily), while a batched, resumable backfill copies 40M rows in chunks sized to keep replication lag within SLO (10.5) — at 3k writes/sec, the backfill must handle rows changing under it, so use a watermark plus a second pass over rows modified during the run. Verification: a continuous comparison job sampling both stores and reporting mismatch rate as a metric (target: zero, and investigate every non-zero — mismatches here are the bugs that would otherwise appear after cutover); backfill progress dashboard; replication lag and write-latency alarms. Rollback: stop dual-writing (new store becomes stale but harmless).

Stage 3 — flip reads behind a flag, ramped. Read from the new store for 1% → 10% → 50% → 100% of traffic, comparing responses in shadow for a soak period at each step. Verification: response-comparison mismatch rate, p99 latency of profile reads, error rate by segment. Rollback: flip the flag — seconds, no deploy (this is why the flag exists rather than a deploy-based switch).

Stage 4 — extract the service. Move ProfileService out of process, behind an API; the interface established in stage 0 means callers change a client, not their logic. Verification: latency budget for the new network hop, circuit breaker and fallback configured (10.9), tracing spans present (10.10). Rollback: route back to the in-process implementation (keep it deployable for at least one release).

Stage 5 — contract. Stop writing profile columns in the old table; after a soak, drop them. Verification: zero writes observed in metrics for the full soak; a final comparison run. Rollback: none for the drop — hence the soak, hence doing it last, and hence taking a verified backup immediately before.

Cross-cutting decisions to state: identity remains the single owner of authentication fields and profile of the rest, with no field owned by both (10.4); any query that previously joined identity and profile is either resolved by two calls in the BFF/caller or, if it's on a hot path, by a maintained denormalized copy (10.8.4) — the join loss must be designed for before stage 4, because discovering it afterward is what makes extractions get reverted. The plan's shape is the lesson: every stage is independently deployable, independently verifiable, and independently reversible — except the last one, which happens only after the evidence is overwhelming.