9.9.7 — Production Ops: Shutdown, Health, PM2, Nginx

Every deploy, your error graph spikes for about ten seconds. Nobody investigates, because it always settles down and the new version works. That spike is real users getting real errors, and it is the reason some teams quietly deploy less often.

Everything on this page converges on that moment. The code being correct is not the same as the code being able to start, stop, and be replaced without anybody noticing.

1. Graceful shutdown

Whatever is running your app — a container platform, a process manager, or the operating system's own service manager — stops it the same way. It sends the process a SIGTERM signal, which means "please finish up". Then it waits, typically around thirty seconds. Then it sends SIGKILL, which cannot be caught, ignored, or delayed. The process stops mid-instruction.

Node's default response to SIGTERM is to exit immediately. So unless you write the code below, every request being served at that instant is cut off mid-response. The user sees a failed request. Your dashboard sees an error spike. And this happens on every single deploy, which is why the spike looks normal.

Here is the correct sequence, in order (3.8.7 introduced the idea; this is the complete version): ^{⚑How do you implement graceful shutdown in Node/Express? [EQ-956]}

const server = app.listen(port);
let shuttingDown = false;

async function shutdown(signal) {
  if (shuttingDown) return;                        // idempotent: two SIGTERMs are common
  shuttingDown = true;
  logger.info({ signal }, "shutdown starting");

  isReady = false;                                 // (1) FAIL READINESS FIRST — section 2
  await sleep(config.drainDelayMs);                //     let the LB notice (5–15 s)

  server.close(() => logger.info("http server closed"));  // (2) stop ACCEPTING; keep serving
  server.closeIdleConnections?.();                        //     kill keep-alives that are idle

  const deadline = setTimeout(() => {              // (3) bounded: never hang the shutdown
    logger.error("forced exit after drain timeout");
    process.exit(1);
  }, config.drainTimeoutMs).unref();               //     unref: don't hold the loop open

  await waitForInFlight();                         // (4) finish current requests
  await Promise.allSettled([                       // (5) release resources, in dependency order
    queueConsumer.stop(),                          //     stop taking new jobs first
    pool.end(),                                    //     then DB
    redis.quit(),
    logger.flush?.(),                              //     flush diagnostics LAST
  ]);
  clearTimeout(deadline);
  process.exit(0);                                 // (6) clean exit code
}

process.on("SIGTERM", () => shutdown("SIGTERM"));  // orchestrator
process.on("SIGINT", () => shutdown("SIGINT"));    // Ctrl-C in dev — same path

Two of those steps are the ones teams miss, and both are worth understanding rather than copying.

(1) Fail the readiness check first, then wait, and only then stop accepting connections. The instinct is to close the server the moment SIGTERM arrives, and it produces errors anyway. Here is why.

Your load balancer does not know your process received a signal. It finds out that an instance is unavailable by asking it, every few seconds. So at the moment SIGTERM arrives, the load balancer's picture of the world is up to a few seconds stale, and it is still sending new requests your way. Close the server instantly and those requests hit a socket that is no longer listening, which the user sees as a failed connection.

The fix is to change the answer before changing the behaviour. Mark yourself unready, keep serving normally for long enough that the load balancer notices and stops routing to you, and only then close. Those few seconds of serving traffic you have already declared yourself unready for are the entire difference between a clean deploy and a spike.

(3) Put a deadline on the drain. One request stuck waiting on an upstream service that will never answer must not hold the whole shutdown open. If it does, your grace period expires, SIGKILL arrives, and everything else in the sequence — closing the database pool, flushing logs — never happens.

Set your own timeout slightly shorter than the platform's grace period. That way the process exits on your terms, with your exit code and your final log lines, rather than being killed in the middle of writing them.

One related detail: process.exit() does not wait for pending asynchronous work (3.8.7). Anything still buffered is simply lost. That is exactly why flushing the logger comes before it in the sequence above — otherwise the log lines explaining your shutdown are the ones that never get written.

2. Health, readiness, liveness — three different questions

Most teams have one /health endpoint that checks everything, and it produces one of the most avoidable outages there is. Follow it through.

The database has a brief problem — a failover, a slow moment, a network blip. Every one of your replicas checks the database in its health endpoint, so every replica reports unhealthy at the same moment. The platform is doing its job: unhealthy processes get restarted. So all of them restart at once. Now you have empty caches, a stampede of reconnection attempts, and no capacity — on top of the original database blip, which might have lasted four seconds and hurt nobody.

The mistake is that one endpoint is being asked two different questions. Separate them. ^{⚑What are health, readiness and liveness endpoints? [EQ-958]}

Liveness asks: is this process broken beyond saving? The only correct answer comes from facts about the process itself — is the event loop responding, is memory catastrophically exhausted. A failure here means restart me, and that is the only thing it can mean.

So liveness must never check a database, a cache, or any other service. Those are not this process being broken. Checking them is precisely the mistake above: an external problem gets reported as "I am broken", and the platform obliges by restarting everything.

Readiness asks: should traffic come to me right now? This one may consult the dependencies this instance genuinely needs — its connection pool is established, migrations have run, caches are warm. And it must return false during the shutdown drain from section 1. A failure here means stop sending me requests, which is a much gentler action than restarting, and the correct one when the thing that is wrong is not the process.

A third probe asks: has it finished starting up? Some apps take a while to boot. Without a separate startup check, a slow start looks like a liveness failure, and the platform restarts a process that was doing nothing wrong except being slow — forever.

app.get("/livez",  (req, res) => res.status(200).json({ ok: true }));   // process-local only

app.get("/readyz", async (req, res) => {
  if (shuttingDown) return res.status(503).json({ ready: false, reason: "draining" });
  const checks = await Promise.allSettled([db.ping(), redis.ping()]);   // cached, cheap
  const ok = checks.every((c) => c.status === "fulfilled");
  res.status(ok ? 200 : 503).json({ ready: ok, checks: summarize(checks) });
});

Two details make the difference in production.

Cache the dependency checks for a second or two. Probes are traffic. Each replica is probed every few seconds, and if every probe runs a real query, then twenty replicas are hitting your database several times a second forever, just to ask whether it is alive. A short cache means the answer stays fresh enough to be useful and the probes stop being a load test you built for yourself.

Keep probes off the authenticated path, but do not make them informative. A probe cannot authenticate, so it must not sit behind your auth middleware or your rate limiter. That does not mean it should tell the world anything. An endpoint that helpfully lists your dependencies and their versions is a free reconnaissance report for anyone who finds it. Probes answer with a status code and almost nothing else; detailed diagnostics belong on a separate, protected endpoint meant for humans.

3. PM2, cluster, and containers

One Node process uses one core (3.8.6). The two production shapes: ^{⚑What is PM2 and how does clustering work in production? [EQ-957]}

PM2 (pm2 start app.js -i max) wraps the cluster module with process management: N workers, automatic restart on crash, zero-downtime reload (pm2 reload restarts workers one at a time, so capacity never drops to zero — and this is exactly why section 1's graceful shutdown must exist: reload sends SIGINT/SIGTERM to each worker in turn), log aggregation, and a monitoring surface. It fits VM/bare-metal deployments and dev machines.

Containers run one process each, and the platform runs as many containers as you need. This is the modern default, and inside a container a process manager is usually redundant and sometimes harmful.

The reason is that the platform already does everything the process manager does — restarts, scaling, health probes, rolling deploys — and doing both means neither can see clearly. If the manager restarts crashing workers inside the container, the container itself stays "up" from the platform's point of view, so your crash loop is invisible to the system responsible for noticing crash loops.

One container detail causes a lot of confusion: the process running as PID 1 has special signal handling, and a Node process started directly as PID 1 will not receive SIGTERM the way you expect. This is the second common reason graceful shutdown "doesn't work" despite the code being right. Use your container runtime's init option, or a small init process, so signals are forwarded properly.

Either way, running more than one process has the same consequences, and this is where the bugs are. Separate processes share no memory. So anything you kept in memory now exists once per process, or not at all where you need it:

• Sessions held in memory (9.9.4) — users appear randomly logged out.
• Rate limit counters in memory (9.9.5) — your limit is multiplied by the number of processes.
• Caches in memory — each process warms its own, which is wasteful but usually acceptable.
• Scheduled jobs — every process runs them. This is the one that costs money. Four processes with a nightly billing job means the job runs four times. The fix is to make exactly one process the leader, using a lock all of them contend for, or to move the schedule outside the app entirely (Part 10.7).

4. Nginx in front

Almost no production Node process should face the internet directly. A reverse proxy earns its place with five jobs: ^{⚑Why put Nginx in front of a Node app? [EQ-959]}

server {
  listen 443 ssl http2;
  ssl_certificate     /etc/ssl/app.crt;             # (1) TLS termination — one place
  ssl_certificate_key /etc/ssl/app.key;

  location /static/ {
    root /var/www;                                   # (2) static files: sendfile(), not Node
    expires 1y; add_header Cache-Control "public, immutable";
  }

  location / {
    proxy_pass http://127.0.0.1:3000;
    proxy_http_version 1.1;
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;   # (3) with trust proxy (9.9.2)
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_read_timeout 30s;                          # (4) timeouts at the edge
    client_max_body_size 10m;                        # (5) reject oversized bodies BEFORE Node
  }
}

(1) TLS termination. Certificates, renewals, protocol settings, and cipher choices live in one place that is built for it, rather than being repeated in every application you deploy.

(2) Static files. This one has a concrete mechanical reason. Nginx uses a system call that hands file data from the operating system's cache straight to the network socket, without the bytes ever passing through the application. Node reads the file through its thread pool and pushes it through the event loop (3.8.2), competing with everything else you are doing. That is the specific reason express.static stops being a good idea at scale (9.9.1).

(3) Forwarded headers. The proxy must overwrite any X-Forwarded-* headers the client sent, not append to them. Otherwise your trust proxy setting is faithfully trusting whatever a stranger typed (9.9.2).

(4) Timeouts and (5) body size limits. Both are about making attacks die before they reach you. A slow-client attack works by opening many connections and sending data one byte at a time, holding resources open; at the proxy those connections are cheap and time out. An oversized upload is rejected at the edge, so it never occupies a Node connection or a byte of your heap. This is 9.9.4's bound-everything rule, enforced one layer further out where it costs less.

Three more jobs belong here for the same reason: buffering, so that Nginx absorbs a slow client and frees your Node connection early; compression (9.9.5); and a coarse rate limit as a first gate in front of your application's precise one.

If you deploy to a cloud platform, a managed load balancer, CDN, or ingress controller does these same jobs. The software is not the point — the list of roles is.

5. The expert lens

Your most frequent outage is your own deploys, so engineer the deploy rather than working around it. Everything on this page meets at that one moment: readiness fails before the process stops accepting traffic, in-flight requests finish, replacement is rolling so capacity never reaches zero, and signals actually arrive at your process.

Get those four right and you can deploy ten times a day with nobody noticing. Miss any one and every release produces errors — and then the team, quite reasonably, starts deploying less often. That is the real cost. Deploy frequency is the input to almost everything else about how fast a team can move and how small its changes are, so a broken deploy path slows down work that has nothing to do with deployment.

Keep "restart me" and "stop sending me traffic" as separate answers. This is the highest-value idea on the page, because collapsing them is what turns a small dependency problem into a company-wide outage. A four-second database wobble fails every replica's health check, every replica gets restarted at the same instant, and the reconnection storm and cold caches make everything worse than the wobble ever was. Liveness looks only at this process. Readiness may look outward, and its answers should be cached, because probes are traffic too.

Push work outward until it is somebody else's. TLS, static files, compression, body limits, absorbing slow clients, coarse rate limiting — every one of them is better done at the edge, where it happens in optimised native code against the operating system's cache rather than on your event loop (3.8.2). Your Node process should be doing the one thing nothing else can do for it: running your business logic.

That principle keeps applying as you grow. A CDN takes over what the proxy was doing; the platform takes over what a process manager was doing. Part 13 is the same idea at infrastructure scale.

What the interviewer will push on

Operations questions are where interviewers find out whether you have owned something in production or only written it.

"What happens when your process gets SIGTERM?" They want to hear that the default is an immediate exit that kills in-flight requests, and then the correct sequence. The step that separates strong answers is failing the readiness check first and waiting — because the load balancer's view is a few seconds stale, closing the socket immediately still drops requests it has already routed to you.

"What's the difference between liveness and readiness?" Restart me versus stop routing to me. Then give the failure that makes it matter: a liveness check that pings the database means one database blip restarts every replica simultaneously, turning a brief degradation into a real outage. Candidates who have only read about probes describe the endpoints; candidates who have run them describe this incident.

"You run four replicas and the nightly job ran four times. Why?" Because a schedule inside the app runs inside every copy of the app. The fix is a leader elected through a lock, or moving the schedule out of the application entirely. This question is a quick test of whether someone has internalised that nothing in process memory — sessions, rate limits, schedules — survives contact with a second replica.

"Why put a proxy in front of Node?" The list is TLS, static files, timeouts, body limits, buffering, and compression. The answer that lands gives the mechanism for at least one: static files are served straight from the operating system's cache to the socket, without ever entering your process, while express.static reads them through your thread pool and event loop.

"Your graceful shutdown code is correct but requests still get dropped. What now?" Two candidates worth naming. Signals may not be reaching the process at all — a Node process running as PID 1 in a container does not get SIGTERM the way you expect. Or the readiness gate is missing, so the platform never learned to stop routing before you stopped listening. Being able to name both is the sign of someone who has debugged this rather than configured it once.

Volunteer this one, because nobody asks: say that the drain must have a deadline of its own, set slightly shorter than the platform's grace period. Otherwise one request hanging on a dead upstream holds shutdown open until the platform kills the process outright — and then the database pool never closes and the last log lines, the ones explaining what happened, are never written.

Part 9 closes here. You have the whole picture now: what good code is, the object model, design principles, the pattern catalog, concurrency, API design, the interview machines, and the production playbook. Next: Part 10 asks the same engineering questions of systems that span many machines, where partial failure, unreliable networks, and messages arriving twice are the normal condition rather than the exception.

Recall

• Graceful shutdown on SIGTERM/SIGINT, in order: fail readiness first + drain delay (LB views are stale — otherwise connection-refused), server.close() (stop accepting, keep serving) + close idle keep-alives, a bounded force-exit deadline (before the orchestrator's SIGKILL), finish in-flight, release resources in dependency order (consumers → DB → cache → flush logs last), exit(0). Make it idempotent; process.exit truncates pending async work.
• Three probes, three questions: liveness = "restart me" — process-local facts only (checking dependencies here turns a DB blip into a fleet restart-loop); readiness = "route to me" — may check dependencies, cached (probes are traffic), false while draining; startup protects slow boots. Deep dependency detail goes in a separate human-facing diagnostics endpoint.
• PM2 = cluster + supervision + zero-downtime reload (which requires graceful shutdown) — fits VMs; inside containers it's usually redundant/harmful (hides crashes from the orchestrator, complicates signals). Either way: no shared memory ⇒ sessions/limits/caches move to Redis, and cron jobs need a leader or every worker runs them.
• Nginx (or ALB/ingress) does five jobs Node shouldn't: TLS termination, static via sendfile, overwriting X-Forwarded-* (or trust proxy trusts a lie), timeouts, body limits — plus buffering, compression, coarse rate limiting. Roles matter, not the software.
• Lens: deploys are the most frequent outage — readiness gating + drain + rolling + signal handling make them invisible; separate "restart me" from "don't route to me"; push work outward until only domain logic remains in Node.

Self-test: Why fail readiness before closing the server, and why the delay? What must liveness never check, and what happens if it does? Name three things that break when one process becomes eight. Which five jobs does the reverse proxy take, and why is sendfile the static-files argument?

Quiz Bank

FoundationalWalk graceful shutdown step by step and justify the two steps teams usually miss.

On SIGTERM (orchestrator) or SIGINT (Ctrl-C), run one idempotent handler: (1) flip readiness to false and wait a drain delay — load balancers learn about unreadiness by polling every few seconds, so closing the listener immediately leaves an LB with a stale view still routing requests into a closed socket (connection-refused, user-visible 502s); serving normally for 5–15 s while advertising unreadiness lets routing drain first.

(2) server.close() — stops accepting new connections while letting in-flight requests finish; also close idle keep-alive connections, which otherwise hold the server open with no work to do. (3) Arm a bounded force-exit timer (unref'd) — one request stuck on a hung upstream must not prevent exit; fire slightly before the orchestrator's grace period so the final logs and exit code are yours rather than SIGKILL's.

(4) Wait for in-flight work (HTTP and any queue consumers). (5) Release resources in dependency order — stop consuming jobs, then close the DB pool, then cache clients, and flush the logger last (process.exit truncates pending async work — 3.8.7).

(6) exit(0). The missed steps are (1) — because it's invisible locally where no LB exists — and (3) — because it only matters when something upstream hangs, which is exactly when you need the process to die cleanly.

FoundationalDistinguish liveness, readiness, and startup probes — and explain the outage caused by conflating the first two.

Liveness answers "is this process unrecoverably broken?" — and must be judged on process-local facts only (loop responsive, heap not fatally exhausted). Its failure action is restart. Readiness answers "should traffic be routed here now?" — it may consult the dependencies this instance needs (pool connected, migrations applied, caches warm) and must go false during shutdown drain. Its failure action is remove from rotation, not restart.

Startup answers "is boot finished?" and suppresses liveness checks during slow initialization so a heavy boot isn't mistaken for a hang. The conflation outage: if liveness checks the database, a database hiccup fails liveness on every replica simultaneously; the orchestrator dutifully restarts the entire fleet; the fleet comes back with cold caches, empty connection pools, and a reconnect storm aimed at the already-struggling database — turning a brief degradation (which readiness would have handled by pausing traffic) into a full outage with a slow recovery. Two supporting practices:

cache readiness dependency checks (1–2 s TTL — probes from N replicas every few seconds are real load), and keep probe endpoints uninformative to the public while putting detailed dependency diagnostics behind a separate, gated endpoint for humans.

AppliedYour Express app moves from a single VM to eight container replicas. List what breaks and the fix for each.

(1) Sessions — express-session with the default MemoryStore is per-process, so users bounce between replicas and appear randomly logged out (9.9.4); fix: Redis-backed store (sticky sessions only as an inferior stopgap that makes deploys lossy).

(2) Rate limits — per-process counters mean the effective limit is 8× the intended (9.9.5/9.7.5); fix: a shared store, with a cheap local pre-filter if you want defense in depth. (3) In-memory caches — each replica has its own, so hit ratio drops and invalidation is per-process; fix: accept the duplication (often fine — correctness is unaffected) or move to a shared cache, measured, not assumed.

(4) Scheduled jobs — a setInterval nightly task now runs eight times; fix: leader election, a distributed lock, or an external scheduler invoking one endpoint (Part 10.7). (5) WebSocket/SSE fan-out — a message published on one replica reaches only its own connections; fix: a pub/sub backplane (Redis) or a dedicated realtime tier.

(6) Local file writes (uploads to disk, temp files, log files) — invisible to other replicas and lost on restart (9.9.4); fix: object storage, stdout logging. (7) Signal handling as PID 1 — a Node process launched as PID 1 in a container may not receive/forward SIGTERM conventionally, so graceful shutdown never runs; fix: --init or a proper init shim, and test SIGTERM behavior in the container image. The unifying rule (3.8.6):

processes share nothing — every piece of state in memory is state you just made per-replica.

InterviewWhy put Nginx (or an ALB/ingress) in front of Node, and what exactly does each job save?

Five jobs, each removing work Node does badly. TLS termination — certificates, cipher configuration, session resumption, and OCSP stapling handled once in battle-tested C, rather than per-app in JavaScript; also centralizes renewal. Static files — sendfile() moves bytes from page cache to socket inside the kernel, with zero user-space copying; express.static reads through libuv's thread pool and the event loop, competing with your request handling (3.8.2) — this is the concrete argument, not a style preference.

Header hygiene — the edge must overwrite X-Forwarded-For/-Proto from clients; without that, trust proxy believes attacker-supplied values and IP-keyed rate limits and logs are forgeable (9.9.2/9.9.5). Timeouts and body limits — Slowloris-style slow clients and oversized uploads are rejected before consuming a Node connection or heap byte (9.9.4's bound-everything law, one layer out).

Buffering — the proxy absorbs slow client reads so Node's response completes and its connection frees early, which is how a handful of Node processes serve many slow mobile clients. Add coarse rate limiting and compression at the edge too (9.9.5). In cloud environments an ALB, CloudFront, or ingress controller plays the same roles — the point is the division of labor: the edge handles the network's hostility, Node handles your domain.

StaffDesign the deploy-safety program for a 30-service Node platform where every release currently produces a visible error spike, and 'deploy Fridays' are banned. What do you standardize, verify, and measure?

Standardize (a shared runtime module, not a wiki page): every service imports @org/node-runtime providing (a) the graceful-shutdown handler exactly as section 1 — idempotent, readiness-first with drain delay, bounded force-exit, dependency-ordered resource release, log flush last; (b) the three probe endpoints with cached readiness checks and the liveness rule enforced by construction (the module's liveness handler is process-local and cannot be configured to check dependencies — the fleet-restart outage made impossible rather than discouraged); (c) config-driven timings (drainDelayMs, drainTimeoutMs) that default to values consistent with the platform's terminationGracePeriodSeconds, asserted at boot (a service whose drain timeout exceeds the orchestrator's grace period is misconfigured by definition — fail fast).

Verify (in CI and in staging, because this class of bug is invisible locally): a container-level test that sends SIGTERM under synthetic load and asserts zero non-2xx responses and exit within the deadline — this single test catches PID-1 signal problems, missing drain delays, and unbounded shutdowns simultaneously; a probe-semantics test asserting /livez stays 200 while a dependency is down and /readyz goes 503; and a rolling-deploy rehearsal in staging that measures error rate through the release.

Measure (the metrics that make the ban liftable): error-rate delta during deploy windows (target: indistinguishable from baseline), deploy duration and rollback time, change failure rate, and deploy frequency itself — because frequency is the outcome variable: teams deploy rarely because deploys hurt, and the Friday ban is a symptom, not a policy.

Sequence the rollout by blast radius: the runtime module lands in the two highest-traffic services first (proving it under real load), then the platform's service template adopts it by default, then a compliance check gates releases on module version. The org framing to state: a deploy is a controlled failover of every instance in the fleet — treat it as an operational procedure with tests and SLOs, and the calendar restrictions dissolve on their own, because nobody bans a procedure that has never hurt.

Flashcards

FlashShutdown order

Readiness false + drain delay → server.close (+ idle keep-alives) → bounded force-exit timer → await in-flight → stop consumers → close DB/cache → flush logs → exit(0). Idempotent.

FlashLiveness vs readiness

Liveness = "restart me", process-local ONLY (dependency checks here = fleet restart-loop). Readiness = "route to me", may check deps, cached, false while draining. Startup = boot grace.

FlashPM2 vs containers

PM2 = cluster + supervision + zero-downtime reload (needs graceful shutdown) on VMs. In containers usually redundant/harmful — one process per container, orchestrator supervises, use --init for signals.

FlashMulti-process breakage list

Sessions, rate limits, caches, cron jobs (N× runs — need a leader), WebSocket fan-out, local files, PID-1 signals. Processes share nothing.

FlashReverse proxy's five jobs

TLS termination · static via sendfile · overwrite X-Forwarded-* · timeouts · body limits (+ buffering, compression, coarse limits). Node keeps only domain logic.

Scenario Drill

DrillPost-incident: a routine deploy of your Express API caused 4 minutes of elevated 502s and one duplicated nightly billing run. Facts: 6 replicas in Kubernetes, no SIGTERM handler, /health checks Postgres and is used for both liveness and readiness, PM2 runs 4 workers inside each container, and the billing job runs via setInterval in the app. Produce the post-mortem: root causes, fixes in priority order, and the tests that prove each fix.

Root causes — four, all independent, all textbook. (1) No SIGTERM handler: on rollout, Kubernetes signals each pod; Node exits immediately, killing in-flight requests — the 502 spray's direct cause. Compounded by (2) PM2 inside the container: the orchestrator signals PID 1 (PM2), whose signal forwarding to workers is an extra hop that must be configured; worker crashes are also hidden from Kubernetes, which sees a healthy container while workers thrash — supervision duplicated, observability lost ([9.9.7] section 3). (3)

/health used as both probes and checking Postgres: during the deploy, new pods reported unhealthy until their pool warmed, so the orchestrator killed and recreated them (liveness semantics applied to a readiness condition) — extending the outage window and adding restart churn; had Postgres itself hiccupped, this configuration would have restart-looped the entire fleet. (4)

setInterval billing job with 6 replicas × 4 workers = 24 schedulers: the duplicate run is not a race, it's arithmetic — every worker owns a timer (3.8.6). Fixes, priority-ordered by risk × effort. P0 — the duplicate-run class (money): move billing out of the app to a Kubernetes CronJob invoking a dedicated endpoint or worker entrypoint, made idempotent by a run-key (billing-2026-07-22) claimed atomically (9.6.3/9.5.1) so even concurrent invocations execute once; audit the duplicated run's financial impact and issue compensations (9.4.15's ledger makes this a query, not an archaeology project).

P1 — graceful shutdown: implement section 1's sequence in the shared runtime module; readiness-first with a drain delay tuned below terminationGracePeriodSeconds. P2 — split the probes: /livez process-local only, /readyz dependency-aware and cached, both wired in the deployment manifest with correct actions. P3 — remove PM2 from the container: one Node process per container, replicas increased to preserve capacity, --init for signal correctness; PM2 stays only on legacy VMs.

Tests that prove each fix, and would have prevented the incident: for P0, a concurrency test invoking the billing entrypoint 10× in parallel asserting exactly one ledger effect; for P1, a container-level SIGTERM-under-load test asserting zero non-2xx and exit within the deadline (the single highest-value test on this list — it catches shutdown, signal-forwarding, and drain-timing defects together); for P2, a probe-semantics test asserting /livez remains 200 while Postgres is stopped and /readyz returns 503 (proving a dependency outage can no longer trigger restarts); for P3, a crash-injection test asserting the container exits (rather than silently degrading) when the Node process dies, so the orchestrator's restart policy is genuinely in control.

The post-mortem's one-line lesson: the deploy didn't fail — nothing in the system had ever been asked to stop politely, and three of the four defects were configuration decisions inherited by default rather than chosen.