11.18 — Distributed Job Scheduler & Workflow Engine

A nightly reconciliation job has not run for eleven days. Nobody noticed, because a job that does not run produces no error, no log line and no failed request. Every dashboard is green. The only symptom is an absence, and nothing in the system was watching for one.

That is the characteristic failure of a scheduler, and it is worth putting first because it reframes what the system is for. Running jobs is the easy part. The hard parts are firing exactly once when three schedulers are alive, resuming a half-finished workflow after the machine holding it died, and noticing when something did not happen.

This is also the infrastructure that half the studies in this Part quietly depend on — the transcoding graph in 11.9, the payout run in 11.14, the scheduled sends in 11.6.

1. Requirements

Functional. One-off jobs at a given time. Recurring jobs on a schedule. Multi-step workflows with dependencies between steps. Retries with backoff. Priorities. Cancellation. A queryable execution history.

Non-functional, with numbers.

• A scheduled job fires within 1 second of its time.
• At-least-once execution with idempotency support. Exactly-once execution is not offered, and section 7.2 explains why promising it would be a lie.
• No job lost when a worker or a scheduler dies.
• 1 million scheduled jobs, 10,000 executions a second.

Out of scope today: the jobs' own business logic, resource quotas beyond concurrency, and scheduling across datacentres.

The clarifying questions, and what each answer changes

"How precise must the firing be?" One second and one minute are different systems. Sub-second precision rules out the time-bucketing in section 7.1 and forces something much more expensive, and almost nothing actually needs it.

"Can a job run twice without harm?" If not, the job must be made idempotent, and the platform's job is to supply a stable key rather than to promise it will never happen. Establishing this in the requirements is what stops someone shipping a non-idempotent payment job.

"How long can a workflow take?" Seconds means you can hold state in a process. Days — with a human approval in the middle — means every wait must be a durable timer and nothing may live only in memory.

"Is this multi-tenant?" If yes, one customer's hundred-thousand-job backfill will consume the fleet unless per-tenant caps exist, and the largest tenant will otherwise define everyone else's latency.

"What happens if a job silently stops running?" Ask it explicitly, because the honest answer is "nothing, for eleven days" unless the watchdog in section 7.6 exists.

2. Estimation

Trigger rate. 1 million scheduled jobs with an average interval of an hour = 1,000,000 ÷ 3,600 ≈ 280 triggers a second on average. What that forces: nothing, on its own. The average is misleading and the next number is the real one.

Burstiness, which is the number that shapes the design. Human-written schedules cluster overwhelmingly on the hour and at midnight. A naive system sees 100,000 triggers in a single second and nothing at all for the next fifty-nine. What that forces: two things. The trigger path must be proportional to the number of jobs due, never to the number of jobs that exist — so scanning a million-row table every second is out. And jitter becomes a feature rather than an imprecision: spreading jobs across their minute converts a spike into a flow.

Execution history. 10,000 executions a second × ~500 bytes = 5 MB a second ≈ 432 GB a day. What that forces: time-partitioned storage with tiered retention, and the observation that history is what grows without bound here, not scheduling. The schedule is a million rows forever; the history is half a terabyte a day.

Schedule store size. 1 million jobs × ~1 KB = 1 GB, spread across time buckets. What that forces: nothing. The schedule is tiny. All the difficulty is in when you read it, not in how much there is.

Worker fleet. At 10,000 executions a second and an average job duration of one second, you need roughly 10,000 concurrent slots — which is a few hundred machines, or far fewer if jobs are mostly waiting on other services. What that forces: the lease model in section 7.3, because tracking the health of ten thousand slots from a coordinator is a worse problem than letting the slots prove they are alive.

3. API

POST /jobs
{ "type": "reconcile_daily",
  "payload": { "date": "2026-07-31" },
  "runAt": "2026-08-01T02:00:00Z",
  "priority": "normal",
  "maxAttempts": 5,
  "idempotencyKey": "reconcile:2026-07-31" }
→ 201 { "jobId": "job_01J9…", "scheduledFor": "2026-08-01T02:00:00Z" }

POST /jobs
{ "type": "reconcile_daily",
  "cron": "0 2 * * *", "timezone": "Europe/London",
  "payloadTemplate": { "date": "{{ periodStart | date }}" },
  "expectedMaxInterval": "26h" }
→ 201 { "jobId": "job_01J9…", "nextRun": "2026-08-01T02:00:00Z" }

POST /workflows
{ "definitionId": "video_publish", "definitionVersion": 7,
  "steps": [ { "id": "probe", "type": "probe_source" },
             { "id": "encode_480", "type": "encode", "dependsOn": ["probe"] },
             { "id": "approve", "type": "human_approval", "dependsOn": ["encode_480"],
               "timeout": "72h", "onTimeout": "escalate" } ],
  "onFailure": "compensate" }
→ 202 { "workflowId": "wf_01J9…" }

GET  /executions/exec_01J9…      # timeline: which step, since when, what failed
POST /executions/exec_01J9…/cancel
POST /executions/exec_01J9…/steps/encode_480/retry

expectedMaxInterval on a recurring job is the most important field on this page, and it is the one nobody thinks to add. It declares how long may pass between successful completions before something is wrong, and it is what the watchdog in section 7.6 asserts. Without it, a job that stops running is undetectable.

The payload carries the target period explicitly, rather than the job working out "yesterday" for itself. That looks like a small style preference and it is a recovery requirement: when a job has missed eleven days, re-running it eleven times is only possible if it accepts the date as a parameter. A job that hardcodes "yesterday" turns a backfill into a code change under pressure.

definitionVersion on a workflow is what makes the deploy problem in section 7.5 solvable at all.

Workers pull rather than being pushed to. lease → execute → acknowledge, because pull gives back-pressure for free (10.8.1): a saturated worker simply stops leasing, and no dispatcher has to model worker capacity, worker health or worker count.

4. Data model

jobs                                       -- the catalogue: ~1 GB, changes rarely
  job_id        UUID PRIMARY KEY
  type          TEXT NOT NULL
  cron          TEXT NULL, timezone TEXT NULL
  payload       JSONB
  priority      SMALLINT, max_attempts SMALLINT
  tenant_id     UUID
  enabled       BOOLEAN NOT NULL
  expected_max_interval INTERVAL NULL       -- what the watchdog asserts
  last_success_at TIMESTAMPTZ NULL

schedule_buckets                            -- partitioned by minute
  bucket        TIMESTAMPTZ, job_id UUID
  fire_at       TIMESTAMPTZ NOT NULL        -- jittered within the bucket
  fired_at      TIMESTAMPTZ NULL            -- the conditional-mark column
  PRIMARY KEY (bucket, job_id)

executions
  execution_id  UUID PRIMARY KEY            -- stable across every attempt
  job_id        UUID, attempt SMALLINT
  state         SMALLINT                    -- ready|leased|succeeded|failed|dead
  lease_owner   TEXT NULL, lease_expires_at TIMESTAMPTZ NULL
  started_at, finished_at, error TEXT NULL

workflow_steps                              -- the durable history
  workflow_id   UUID, step_id TEXT
  attempt       SMALLINT
  state         SMALLINT
  output        JSONB                       -- stored as data, never as an object
  completed_at  TIMESTAMPTZ
  PRIMARY KEY (workflow_id, step_id, attempt)

Access patterns:

Query	Frequency	Returns
Read the due bucket	1/s per shard	the jobs due now
Conditionally mark a job fired	~280/s, bursty to 100k	one row
Lease the next ready execution	10,000/s	one row
Extend a lease (heartbeat)	10,000/s	one row
Append a step result	high, per workflow	one row
Read a workflow's history	on resume	tens of rows
Find jobs past their expected interval	every minute	ideally zero

schedule_buckets is partitioned by minute, and that single decision is what makes triggering cheap — section 7.1.

fired_at is nullable on purpose. It is the target of a conditional update, which is what makes triggering single-fire without any locking.

execution_id is stable across attempts. Every retry of the same occurrence carries the same identifier, because that identifier is what the job uses to deduplicate. Generating a fresh one per attempt defeats the entire mechanism, and it is a natural mistake to make.

Workflow step output is stored as data, not as a serialised object. A later deploy may have changed or removed the class, and a history you cannot deserialise is a workflow you cannot resume.

5. Architecture

Figure 1 — Four components, four properties. Time partitioning makes triggering proportional to what is due. Leader election plus a conditional mark makes it fire once. Leases make worker death self-healing with no health checking at all. And the durable history is what makes a workflow resumable rather than merely restartable.

Figure 2 — The lease. No coordinator tracks worker liveness, because a worker proves it is alive by extending its own deadline. The cost is a real tuning risk: if a heartbeat stalls while the job is still running, the lease expires and a second copy starts — which is why idempotency has to be concurrency-safe rather than merely repeat-safe.

6. Deep dives

6.1 Triggering without scanning

The naive approach is SELECT * FROM jobs WHERE run_at <= now() AND state = 'pending', executed every second. Even with a perfect index it re-scans a growing range, competes with writes, and becomes the system's bottleneck well before a million jobs.

Partition the schedule by time instead. Each occurrence is stored in a bucket keyed by the minute it is due. Every second the trigger reads only the current bucket — a small, bounded set containing exactly the jobs due now — so the cost is proportional to the jobs due rather than to the jobs that exist, and it stays flat as the catalogue grows from a thousand to a hundred million.

Two details make it correct rather than merely fast.

Conditional marking. Enqueueing sets fired_at through UPDATE … WHERE fired_at IS NULL, so a trigger that crashes after enqueueing but before recording its progress cannot fire the same occurrence twice on restart (10.4).

Recurring jobs propagate themselves. At fire time, the job computes its next occurrence and inserts it into a future bucket in the same transaction. There is no separate "advance all schedules" pass that can fail, and a crash between firing and advancing is impossible. The cost of this elegance is named in section 7.6: one failed insert ends the chain permanently, so self-propagation must be paired with a watchdog.

And two operational requirements.

Missed buckets must be processed, not skipped. After a failover or a pause, the trigger works forward through every unprocessed bucket. Skipping straight to "now" makes jobs due during the gap vanish silently, which is the most damaging bug available in this design precisely because it produces no error — only absence.

Jitter by default. The overwhelming majority of human-written schedules land on the hour or at midnight. Firing exactly on the tick creates a synchronised burst of a hundred thousand jobs. Spreading each job deterministically within its bucket — hash the job identifier into an offset, deterministic so it does not drift between runs — converts that spike into an even flow, and costs only sub-minute precision that almost nothing needs.

6.2 Exactly one trigger, at-least-once execution

These are two different guarantees and conflating them is the classic error.

Triggering can be exactly once. Leader election with a lease means one trigger process per shard, and the conditional fired_at mark means even a leader that crashes mid-enqueue cannot double-fire. That guarantee is achievable and the system should provide it.

Execution cannot be exactly once, because of an ambiguity that no protocol removes: a worker can complete the job's side effects and then die before acknowledging. From outside, "finished but did not acknowledge" is indistinguishable from "died before starting". The system must therefore choose — re-run, risking a duplicate, or not, risking a job that never ran.

For a scheduler, at-least-once is the only defensible default, because silently skipping a job is worse than running it twice for nearly every workload.

So the contract is explicit: at-least-once execution, with the platform supplying a stable execution_id and the job required to be idempotent. The platform's obligations are concrete — the same identifier on every attempt of the same occurrence, that identifier passed into the job so it can deduplicate at its boundary, and the contract written down where job authors will read it.

And the risk that catches implementations: even without a crash, a stalled heartbeat can let the lease expire while the job is still running, so two copies execute concurrently. Idempotency therefore has to be concurrency-safe — an atomic conditional write, not a check followed by an act.

6.3 Leases rather than assignments

A worker leases an execution with a deadline and heartbeats to extend it. If it crashes, the lease expires and the execution returns to the queue. No health checking, no coordinator tracking liveness, no manual recovery.

The lease duration is a genuine trade with failure at both ends. Too short, and a long-running job's lease expires while it is still working — so a second copy starts and two run concurrently, which is why heartbeating is mandatory rather than an optimisation. Too long, and a crashed worker's job sits idle for the whole lease duration before anyone can pick it up.

The rule that resolves it: the lease is short, and the heartbeat is frequent, so the recovery delay is bounded by the lease rather than by the job's duration. A job that runs for an hour with a thirty-second lease and a ten-second heartbeat recovers in thirty seconds if the worker dies, and never expires while the worker lives.

6.4 Workflows are the durable part

A workflow's state is a durable event history, not a running process. Every step's completion — with its inputs, output and timestamp — is appended to storage, and the current state is a fold over that history (10.8.4).

Nothing lives only in a process's memory or on its stack. A crash, a deploy or a scale-down loses nothing: a fresh orchestrator reads the history, works out which steps are complete and which are now unblocked, and enqueues those. It never re-runs completed work.

Long waits are durable timers, never held threads. A "wait seven days" step or a "wait for approval" step registers a future trigger or a callback token, and the workflow occupies zero resources until it fires. That is what makes a million concurrent long-running workflows affordable, and it is the difference between a workflow engine and a thread pool with ambitions.

Failure policy is per step: retry with backoff, skip, fail the whole workflow, or run a compensating action — which is the saga from 11.14, and the reminder that a compensation is a business event rather than an undo.

6.5 Surviving deploys and code changes

Steps must be externally identified and version-tolerant. A step is {workflowId, stepId, attempt} where stepId is a stable name, never an array index or a position in a list. And step results are stored as data, not as serialised objects tied to a class that a later deploy may have changed — because replaying history through new code must produce the same decisions about what to run next.

Code changes are the genuinely hard part, and they need a stated policy rather than good intentions. If a workflow definition changes while instances are mid-flight — a step added, removed, renamed, or its dependencies rewired — replaying old history against new code can crash, skip a step, or re-run a completed one.

Three viable policies, and one must be chosen deliberately:

Pin each instance to the definition version it started with. Safest. Requires retaining old code paths and the operational discipline to eventually drain them, which is why definitionVersion is on the API.

Allow only backwards-compatible changes — adding steps at the end, never renaming or removing — enforced by an automated check rather than by review. This is expand-migrate-contract (10.11) applied to workflow definitions.

Explicit migration for structural changes, with a documented mapping and a rehearsal against live instances.

Choosing "we will be careful" is choosing the second without the enforcement, and it fails the first time somebody renames a step.

And determinism at the orchestration layer matters for the same reason it does in 11.15: orchestration code must not read the clock, call services or use randomness directly. Those are steps, whose results are recorded. Otherwise replay produces different decisions than the original run, and "resumable" becomes "resumable in theory".

6.6 Noticing what did not happen

This is the section the story at the top of the page exists for.

Every recurring job declares a maximum acceptable interval since its last success, and an independent watchdog alarms when it is exceeded. This is heartbeat monitoring inverted: instead of the job reporting success, which a non-running job cannot do, the platform asserts that a success should have occurred by now. Without it, no amount of error monitoring helps, because there are no errors.

Three supporting checks, each catching a different way the chain breaks.

Schedule integrity. A daily audit verifying that every enabled recurring job has a future occurrence scheduled. A recurring job with no next occurrence is a broken self-propagating chain, detectable in one query and invisible otherwise.

Dead-letter age. Jobs parked after exhausting their attempts must surface rather than accumulate.

A disabled-jobs report with an owner and a reason, reviewed periodically, so that a temporary disable during an incident does not become permanent through inattention.

And one check that deliberately does not share the scheduler's blind spots: monitor the freshness of the job's output. If reconciliation produces a report, alarm on the report's age. That catches the failure independently of whether the scheduler believes it ran, which is exactly what you want from a second detector.

6.7 Priorities, fairness, and the noisy tenant

Separate queues per priority with reserved worker capacity, not merely an ordering — the same lesson as 11.6, for the same reason: an ordering only matters when a worker is free, and during a backfill none are.

Per-tenant concurrency caps so one customer's hundred-thousand-job backfill cannot consume the fleet.

Fair round-robin across tenants with pending work, which is worth the complexity in any multi-tenant scheduler. Without it the largest tenant defines everyone else's latency, and the smallest tenants experience the system as permanently slow for reasons they cannot see.

7. Decision Ledger

Decision	Alternatives	Why this	What it costs
Schedule partitioned by time bucket	scan a table for what is due	work proportional to jobs due, not jobs existing	bucket granularity bounds precision; round times are hot
Leader-elected trigger with a conditional mark	let every node trigger	exactly one enqueue per occurrence	a lease dependency, and a gap during failover
Recurring jobs insert their own next occurrence	a separate advance pass	crash-safe and self-propagating, with no extra job to fail	one failed insert ends the chain — needs the watchdog
Jitter within the bucket by default	fire exactly on the tick	turns a 100,000-job midnight spike into a flow	firing within a window rather than on the second
Leases with heartbeats	assignment plus health checks	worker death self-heals with no coordinator at all	tuning; a stalled heartbeat runs two copies
At-least-once with a stable execution id	promise exactly-once	honest and achievable; the alternative is a lie that ships	job authors must write idempotent, concurrency-safe jobs
Workers pull	a dispatcher pushes	back-pressure for free; no capacity model to maintain	polling overhead, mitigated by long-polling
Durable step history	re-run the workflow from the start	resume rather than restart; long waits cost nothing	storage, and a retention policy
expectedMaxInterval on every recurring job	rely on error monitoring	the characteristic failure produces no errors	one field, and someone must choose its value

8. Scale and failure

At 10×, shard the schedule store by job identifier with a trigger leader per shard, which scales linearly and removes the single-leader bottleneck. Partition queues by priority and tenant. And archive execution history aggressively — history, not scheduling, is what grows without bound.

What breaks	Blast radius	How you find out	What keeps it running	Recovery
Trigger leader dies	jobs due during the gap	leader lease churn	a new leader takes the lease and reads the same buckets	it must process missed buckets, not just the current one
Trigger skips missed buckets	those jobs never run, silently	the expected-interval watchdog	nothing else catches it	replay the buckets; fix the forward-walk
Worker dies mid-job	one execution	lease expiry rate	the lease expires and the work is requeued	another worker picks it up
Heartbeat stalls, job still running	two copies run concurrently	duplicate-execution counter	idempotency must be concurrency-safe	the job's own conditional write settles it
Poison job crashes every worker	the whole fleet, serially	crash rate correlated with one job type	attempt counting and dead-letter routing	park it; investigate offline
Queue backlog	growing lateness, no errors	age of the oldest ready job, not depth	autoscale on age; defer bulk work	drain with age as the exit condition
Recurring chain broken by one failed insert	that job stops forever	schedule-integrity audit; the watchdog	self-propagation has no other safety net	reschedule; the audit finds the rest
A job silently stops running	eleven days of nothing	expectedMaxInterval watchdog plus output freshness	nothing — this is the failure the watchdog exists for	run the backlog; the job must accept its period as a parameter

Clocks and time zones are a genuine source of production bugs. "2:30 every morning" happens twice on one day of the year and not at all on another. Store the timezone as an IANA name, compute occurrences with a real timezone implementation, and decide the daylight-saving policy explicitly — skip, run twice, or run once at the shifted time (10.3). And never derive schedule timing from a worker's local clock.

Alarm on the age of the oldest ready job, never on queue depth. Depth without age is unreadable: two million jobs that are four seconds old is a backfill running normally, and forty jobs that are nine minutes old means the one-second firing promise is broken (10.10).

What the interviewer will push on

"How do you fire a million scheduled jobs on time without scanning them?" Partition the schedule by minute and read only the due bucket, so the cost is proportional to what is due. Then volunteer the two details that make it correct: a conditional fired_at mark so a crashed trigger cannot double-fire, and recurring jobs inserting their own next occurrence in the same transaction so there is no separate advance pass to fail. And name the operational trap — missed buckets must be walked forward, or a thirty-second failover silently deletes a minute of jobs.

"Can you guarantee a job runs exactly once?" No, and the precise answer separates two guarantees. Exactly-once triggering is achievable through leader election plus a conditional mark. Exactly-once execution is not, because a worker completing and then dying before acknowledging is indistinguishable from a worker dying before starting. At-least-once is the right default for a scheduler, since a skipped job is worse than a repeated one, and the platform's obligation is a stable execution identifier plus a written contract.

"What does a lease buy you that an assignment does not?" Self-healing with no coordinator. Nobody tracks worker liveness, because a worker proves it is alive by extending its own deadline, and a dead one simply stops. Then name the tuning risk unprompted: a stalled heartbeat expires the lease while the job is still running, so two copies execute concurrently — which means idempotency has to be concurrency-safe rather than merely repeat-safe.

"A workflow's machine dies halfway through a seven-day process. What happens?" Nothing is lost, because the workflow's state is a durable history rather than a running process. A fresh orchestrator reads the history, determines which steps completed and which are now unblocked, and enqueues those — never re-running completed work. Then the detail that shows you have built one: the seven-day wait was a durable timer, not a sleeping thread, so it consumed nothing while it waited.

"You deploy a new version of a workflow definition while a thousand instances are mid-flight. What breaks?" Replaying old history against new code can crash, skip a step or re-run a completed one — especially if a step was renamed or removed. Give the three policies with their costs, and say plainly that "we will be careful" is the second policy without its enforcement, which fails the first time someone renames a step.

"A nightly job has not run for eleven days and nobody noticed. What was missing?" A detector for absence. Errors did not fire because there were none. Every recurring job needs a declared maximum interval and an independent watchdog asserting it, plus a schedule-integrity audit and a freshness check on the job's output that does not share the scheduler's blind spots.

Volunteer this, because nobody asks: every scheduled job should take its target period as an explicit parameter rather than working out "yesterday" for itself. It looks like a style preference and it is a recovery requirement — when eleven days have been missed, running the backlog is only possible if the job can be told which day to process. A job that hardcodes "yesterday" turns a straightforward backfill into a code change written under time pressure during an incident, which is precisely when you least want to be editing a reconciliation job.

Next: 11.19 — the system that would have noticed. From running things on time to measuring everything that runs: millions of data points a second, queries over months of history, and the honest arithmetic of what it costs to keep every number forever.

Recall

• The characteristic failure is silent non-progress, not a crash. A job that does not run produces no error, no log line and no failed request, so every recurring job needs a declared expectedMaxInterval and an independent watchdog asserting a success should have happened by now.
• Triggering is proportional to what is due: the schedule is partitioned into one bucket per minute, the trigger reads only due buckets, and enqueueing sets fired_at through a conditional update so a crashed trigger cannot double-fire.
• Recurring jobs insert their own next occurrence at fire time, in the same transaction — crash-safe and self-propagating, at the cost that one failed insert ends the chain forever, which is why the schedule-integrity audit exists.
• Missed buckets must be walked forward after a failover. Skipping to "now" silently deletes every job due during the gap, and it produces no error at all.
• Jitter within the bucket by default, because human schedules cluster on the hour and midnight, turning 100,000 jobs into a single-second spike.
• Exactly-once TRIGGER, at-least-once EXECUTION. Completing and then dying is indistinguishable from never starting. The platform supplies a stable execution id; the job must be idempotent, and concurrency-safe, because a stalled heartbeat can run two copies at once.
• Leases, not assignments: lease with a deadline, heartbeat to extend, crash means expiry means requeued — with no health checks and no coordinator.
• A workflow's state is a durable history, so a crash resumes rather than restarts, and long waits are durable timers occupying nothing. Step ids are stable names, results are stored as data, and orchestration must be deterministic.
• Alarm on the age of the oldest ready job, never depth. Priority queues need reserved capacity, and multi-tenant schedulers need per-tenant caps or the largest tenant sets everyone's latency.

Self-test: How does triggering avoid scanning a million rows, and what two details make it correct? Distinguish the trigger guarantee from the execution guarantee and say why. What does a lease buy, and what is its tuning risk? How does a workflow resume? What detects a job that simply stopped running?

Quiz Bank

FoundationalHow do you trigger a million scheduled jobs on time without scanning them all?

Partition the schedule by time. Each occurrence is stored in a bucket keyed by the minute it is due. Every second the trigger reads only the current bucket — and any earlier bucket it has not yet processed — which is a small, bounded set containing exactly the jobs due now. The cost is proportional to the jobs due rather than to the jobs that exist, and it stays flat whether the catalogue holds a thousand jobs or a hundred million.

Contrast the naive SELECT * FROM jobs WHERE run_at <= now() AND state = 'pending' run every second. Even with a perfect index it re-scans a growing range, competes with the writes that are inserting new jobs, and becomes the system's bottleneck long before a million rows.

Two details make it correct rather than merely fast.

Conditional marking. Enqueueing an occurrence sets fired_at with UPDATE … WHERE fired_at IS NULL. A trigger that crashes after enqueueing but before recording its own progress therefore cannot fire the same occurrence twice when it restarts (10.4).

Recurring jobs propagate themselves. At fire time, the job computes its next occurrence and inserts it into a future bucket in the same transaction. There is no separate "advance every schedule" pass that can fail independently, and a crash between firing and advancing is impossible because they are one write.

Two operational necessities that are easy to omit and expensive to omit.

Missed buckets must be processed, not skipped. After a failover, a pause or a deploy, the trigger walks forward through every unprocessed bucket. If it skips straight to the current minute, every job due during the gap disappears — and this is the most damaging bug in the design because it produces no error, only absence, which nothing detects unless section 6.6's watchdog exists.

Jitter. The overwhelming majority of human-written schedules land on the hour or at midnight, so firing exactly on the tick creates a synchronised burst of a hundred thousand jobs followed by fifty-nine seconds of nothing. Spreading each job deterministically within its bucket — hashing the job identifier into an offset, deterministic so it does not drift between runs — converts the spike into an even flow. It costs only sub-minute precision, which almost no job actually needs, and it should be the default rather than an option.

InterviewCan you guarantee a job runs exactly once? Answer precisely.

No — and the precise answer distinguishes two different guarantees, which is the point of the question.

Exactly-once triggering is achievable. Through leader election with a lease and a conditional fired_at mark, a scheduled occurrence is enqueued exactly once no matter how many trigger processes exist or how they fail. The system should provide this and can.

Exactly-once execution is not achievable, because of an ambiguity no protocol removes. A worker may complete the job's side effects and then die before acknowledging. From the outside, "completed but unacknowledged" is indistinguishable from "died before doing anything". The system must therefore choose: re-run, which risks a duplicate, or do not, which risks a job that never ran.

For a scheduler, at-least-once is the only defensible default, because silently skipping a scheduled job is worse than running it twice for nearly every workload anyone schedules.

So the contract is written down explicitly: at-least-once execution, with the platform supplying a stable execution identifier and the job required to be idempotent (10.4). The platform's obligations are concrete rather than aspirational — the same identifier on every attempt of the same occurrence, never a fresh one, which would defeat the mechanism entirely; that identifier passed into the job so it can deduplicate at its own boundary with a unique constraint or a conditional write; and the contract documented where job authors will actually read it.

The dangerous shortcut worth naming: frameworks advertising "exactly-once" almost always mean exactly-once delivery to a queue, or exactly-once within a transactional boundary they control. Neither extends to a job that calls a payment provider or sends an email (11.14, 11.6). Job authors who believe the marketing write non-idempotent jobs, and the consequence surfaces as a double charge during an unrelated incident months later.

And the additional risk that catches implementations: even with no crash at all, a stalled heartbeat can let the lease expire while the job is still running, so two copies execute concurrently. That means idempotency must be concurrency-safe — an atomic conditional write, not a check followed by an act — which is a subtlety many implementations miss because they only ever considered sequential retries.

StaffDesign the workflow engine so a multi-day workflow with human approvals survives deploys, crashes and code changes.

Foundation: the workflow's state is a durable event history, not a running process. Every step's completion — inputs, output, timestamp — is appended to persistent storage, and the current state is a fold over that history (10.8.4). Nothing lives only in a process's memory or on its stack, so a crash, a deploy or a scale-down loses nothing: a fresh orchestrator reads the history, determines which steps are complete and which are now unblocked, and enqueues those.

Long waits are durable timers, never held threads. A "wait seven days" or "wait for approval" step registers a future trigger or a callback token, and the workflow occupies zero resources until it fires. That is what makes a million concurrent long-running workflows affordable rather than absurd.

Surviving deploys requires steps to be externally identified and version-tolerant. A step is {workflowId, stepId, attempt} where stepId is a stable name — never an array index or a position, both of which shift when someone edits the definition. Step results are stored as data, not as serialised objects tied to a class that a later deploy may have changed or removed. Replaying history through new code must produce the same decisions about what to run next, which is exactly why the history stores results rather than continuations.

Surviving code changes is the genuinely hard part and needs a stated policy. If a definition changes while instances are mid-flight — a step added, removed, renamed, or its dependencies rewired — replaying old history against new code can crash, skip a step, or re-run one that already completed. Three viable policies:

Pin instances to the definition version they started with. Safest, and it requires retaining old code paths plus the discipline to eventually drain them.

Allow only backwards-compatible changes — appending steps, never renaming or removing — enforced by an automated check rather than by review. This is expand-migrate-contract (10.11) applied to workflow definitions.

Explicit migration for structural changes, with a documented mapping and a rehearsal against live instances.

Choosing "we will be careful" is choosing the second policy without its enforcement, and it fails the first time somebody renames a step.

Determinism at the orchestration layer matters for the same reason it does in 11.15. Orchestration code must not read the clock, call services, or use randomness directly — those are steps whose results are recorded — because otherwise a replay produces different decisions than the original run. This is the constraint most teams discover late, and it is the entire difference between "resumable" and "resumable in theory".

Human approvals complete via an external callback carrying a signed token. They need a timeout with an escalation path, because a workflow blocked forever on a departed employee's approval is a real and common failure. They need reassignment. And they need an audit record of who approved what and when, because that is usually the reason the approval step exists at all.

Operational necessities. A queryable timeline per instance — which step, since when, what failed. The ability to retry a single failed step without re-running the workflow. Manual step completion for genuinely stuck instances. Cancellation that triggers compensations rather than merely stopping. And an alarm on instances stuck in a step beyond its expected duration, because a workflow engine's characteristic failure is not crashing but silently not progressing, which no error rate will ever detect.

Flashcards

FlashTriggering proportional to what is due

Schedule partitioned into one bucket per minute; the trigger reads due and missed buckets, marks fired_at conditionally, and recurring jobs insert their own next occurrence in the same transaction.

FlashTwo different guarantees

Exactly-once trigger (leader election plus a conditional mark). At-least-once execution — completing and then dying is indistinguishable from never starting. Jobs must be idempotent, and concurrency-safe.

FlashLeases

Lease with a deadline, heartbeat to extend; a crash means expiry means requeued, with no health checks and no coordinator. Too short runs two copies concurrently; too long stalls recovery.

FlashWorkflow resumability

State is a fold over durable step-completion events, so a crash resumes rather than restarts. Long waits are durable timers occupying nothing. Step ids are stable names; results are data; orchestration is deterministic.

FlashJitter and missed buckets

Human schedules cluster at the hour and midnight, so jitter deterministically within the bucket. And after a failover, walk forward through missed buckets — skipping to now deletes jobs silently.

FlashDetecting absence

expectedMaxInterval per recurring job plus an independent watchdog · schedule-integrity audit (every enabled job has a future occurrence) · dead-letter age · output freshness, which does not share the scheduler's blind spots.

Scenario Drill

DrillA nightly job that reconciles yesterday's data has silently not run for eleven days. Nobody noticed. What failed, and what should exist so this is impossible?

The failure that matters is not why the job stopped. It is that eleven days passed without detection, and every scheduler eventually produces this incident, because a job that does not run generates no error, no log line and no metric unless something is explicitly watching for absence.

The mechanical causes are all mundane, and there are only a handful. The trigger skipped buckets after a failover and never walked the missed window forward. The recurring job's next-occurrence insert failed once, and because recurrence is self-propagating, a single failure ends the chain permanently. The job was disabled during an incident and never re-enabled. A deploy changed its identifier so the schedule points at a job type that no longer exists. The tenant's concurrency cap was consumed by a backfill and the job was deferred every night. Or it exhausted its attempts and went to the dead-letter queue, which nobody was watching.

The detections that must exist, in order of importance.

Expected-execution monitoring, which is the essential one. Every recurring job declares a maximum acceptable interval since its last successful completion, and an independent watchdog alarms when that is exceeded. This is heartbeat monitoring inverted — instead of the job reporting success, which a non-running job cannot do, the platform asserts that a success should have happened by now. Without it, no amount of error monitoring helps, because there are no errors to monitor.

Schedule integrity checks. A daily audit verifying that every enabled recurring job has a future occurrence scheduled. A recurring job with no next occurrence is a broken chain, findable in one query and invisible in every other way.

Dead-letter age alerting, so jobs parked after retries surface rather than accumulate quietly.

A disabled-jobs report with an owner and a reason, reviewed periodically, so that a temporary disable during an incident does not become permanent through inattention. This one is a process rather than a mechanism, and it catches the cause that mechanisms cannot.

Downstream freshness checks. The reconciliation job produces output; monitor the output's age. This catches the failure independently of the scheduler's own view of the world, which is precisely what makes it valuable — it does not share the scheduler's blind spots.

And the eleven days of missing reconciliation is its own incident, separate from the scheduler bug. Run the backlog — which requires the job to accept its target date as a parameter and to be idempotent. If it only ever processes "yesterday", recovery means eleven manual runs and quite possibly a code change written under pressure, which is why every scheduled job should take its target period as an explicit parameter. Then work out what those eleven days of unreconciled data would have caught, because discrepancies compound silently (11.14).

The principle to state plainly: monitoring detects things that happen. The characteristic failure of a scheduler is a thing that does not happen. So every recurring job needs a declared expected cadence and an independent watchdog asserting it — and if a job is important enough to schedule, it is important enough to notice missing.