11.30 Background Job & Scheduler Platform

When a user clicks "Submit Order", they expect an immediate response. But the system might need to charge a credit card, generate a PDF invoice, send an email, and update a data warehouse. Doing all this synchronously blocks the user and risks failure if any single downstream system is slow.

A background job scheduler decouples immediate user actions from long-running or failure-prone tasks. You place a "job" in a queue and return immediately. Workers pull from this queue and execute the jobs asynchronously.

Requirements

The platform here runs the background work of a mid-sized SaaS product: about 50 job types, from password-reset emails and invoice PDFs to webhook deliveries and nightly data syncs. The distributed scheduler for millions of cron-style tasks is a different problem, designed in 11.18; this page is the job queue that most products build first and run for years.

Functional. Enqueue a job to run now, at a given time, or on a repeating schedule. Run every job at least once. Retry failures with growing delays. Park jobs that keep failing in a dead-letter table. Give urgent work priority. Let a job be cancelled and its status be looked up. Let a service enqueue a job in the same database transaction as the business change that caused it.

Non-functional, with numbers.

• Urgent jobs start within 2 seconds of being enqueued, at p95. A password-reset email that takes a minute is a failed login.
• No acknowledged job is ever lost. Once enqueue has returned, the job will run, even if every worker crashes.
• A job that keeps failing never blocks other jobs.

Out of scope today: a visual workflow builder, and multi-step workflows with branches, which belong to a workflow engine.

The clarifying questions, and what each answer changes

"Must enqueuing be atomic with the business change?" Usually yes: "create the user and send the welcome email" must not create the user and lose the email. If yes, the queue lives in the same PostgreSQL database as the business data, which is the main reason to choose a database-backed queue over a separate broker (10.8.4's outbox, applied to jobs).

"How long can a job run?" Seconds means a short lease is fine. Hours means the worker must renew its lease as it goes (the heartbeat in the interview section).

"Is running a job twice harmful?" It always is for something. The answer lists which job types need an idempotency key passed to the outside world.

"Do some customers generate far more jobs than others?" If one tenant can enqueue a million jobs at once, the queue needs per-tenant fairness or that tenant delays everyone.

"What is the peak?" For most products it is a scheduled burst: everyone's daily digest at 08:00, every invoice on the 1st of the month.

Estimation

Job rate. About 20 million jobs a day, so 20,000,000 ÷ 100,000 seconds = 200 a second on average and about 1,000 a second at the morning peak. What that forces: this sits right at the edge of the guidance later on this page, that PostgreSQL is comfortable up to around 1,000 jobs a second. The design keeps urgent and transactional job types in PostgreSQL and is ready to move high-volume, non-transactional types (bulk webhook deliveries, say) to a separate broker if growth continues.

Database writes. Each job costs about three writes: the insert, the claiming update, and the delete or completion. At peak that is about 3,000 writes a second on one table, and every update and delete leaves a dead row version behind for PostgreSQL's autovacuum to clean up. What that forces: the jobs table is kept small. Finished jobs are deleted or moved to a history table in batches, and autovacuum on this one table is tuned to run far more often than the default. A jobs table that grows to millions of dead rows makes every claim query slower, which is the most common way this design degrades.

Workers needed. If an average job takes 2 seconds, then by Little's Law (10.27) the number of jobs running at once is 1,000 a second × 2 seconds = 2,000 at peak. With about 50 concurrent jobs per worker process (they mostly wait on I/O), that is about 40 worker processes. What that forces: workers scale on the age of the oldest waiting job, not on CPU, because I/O-bound workers sit at low CPU while the queue backs up.

The cost of waiting for work. Forty idle processes each polling ten times a second is 400 queries a second that find nothing. What that forces: workers sleep and are woken by PostgreSQL's LISTEN/NOTIFY when a job is inserted, with a slow poll every second or two only as a safety net for missed notifications.

Failures. If 0.1% of jobs fail every attempt, that is 20,000 dead-letter jobs a day. What that forces: the dead-letter table needs grouping by job type and error, and a "retry all of these" action, because nobody can inspect 20,000 jobs one at a time.

API

// Inside the service that owns the business change: one transaction for both.
await db.transaction(async (tx) => {
  const user = await tx.users.insert({ email });
  await jobs.enqueue(tx, {
    type: "send_welcome_email",
    payload: { userId: user.id },
    priority: 5,
    idempotencyKey: `welcome:${user.id}`,
  });
});

POST /jobs
{ "type": "generate_invoice_pdf", "payload": { "invoiceId": "inv_7731" },
  "runAt": "2026-11-01T00:05:00+05:30", "priority": 1,
  "maxAttempts": 8, "idempotencyKey": "invoice-pdf:inv_7731" }

→ 201 Created  { "jobId": "job_01JA9", "status": "scheduled" }
→ 200 OK       { "jobId": "job_01JA9", "status": "scheduled", "duplicate": true }   (same key again)

GET /jobs/job_01JA9
→ 200 OK  { "status": "failed_retrying", "attempts": 3, "nextAttemptAt": "…",
            "lastError": "Timeout calling pdf-renderer after 30s" }

POST /jobs/job_01JA9/cancel
→ 200 OK                                    (if not yet running)
→ 409 Conflict  { "error": { "code": "JOB_RUNNING",
                             "message": "Job is running; cancellation is cooperative.",
                             "requestId": "req_5e" } }

PUT /schedules/daily-digest
{ "cron": "0 8 * * *", "timezone": "Asia/Kolkata", "type": "send_daily_digest", "jitterSeconds": 600 }

POST /dead-letter/retry   { "type": "deliver_webhook", "error": "HTTP 503", "since": "2026-10-18T06:00:00Z" }
→ 202 Accepted  { "requeued": 1840 }

The in-process enqueue(tx, …) is the most important call. It takes the caller's transaction, so the job exists if and only if the business change committed. The HTTP endpoint is for other services that do not share the database.

idempotencyKey deduplicates enqueues, so a caller that retries after a timeout does not create two invoice PDFs. It is a different key from the one each handler passes to outside systems, and both are needed.

Schedules carry a timezone and jitter. A digest at "08:00" means 08:00 in a named timezone, not a fixed offset, and spreading 500,000 digests over 10 minutes (jitterSeconds) turns a spike of 500,000 jobs in one second into about 830 a second, inside the peak this system is sized for.

Data model

jobs
  id              UUID PRIMARY KEY
  queue           TEXT NOT NULL            -- urgent | default | bulk
  type            TEXT NOT NULL
  payload         JSONB NOT NULL
  priority        SMALLINT NOT NULL
  run_at          TIMESTAMPTZ NOT NULL     -- now, or a future time
  status          TEXT NOT NULL            -- pending | processing
  locked_until    TIMESTAMPTZ, locked_by TEXT
  attempts        SMALLINT NOT NULL, max_attempts SMALLINT NOT NULL
  idempotency_key TEXT                     -- UNIQUE (queue, idempotency_key)
  created_at      TIMESTAMPTZ NOT NULL

  INDEX claimable ON jobs (queue, priority DESC, run_at)
        WHERE status = 'pending' OR status = 'processing'

dead_jobs         same columns + last_error, failed_at
job_history       id, type, status, attempts, finished_at   -- append-only, partitioned by day
schedules         name PRIMARY KEY, cron, timezone, type, payload, jitter_seconds, next_run_at

Access patterns:

Query	Frequency	Returns
Claim the next runnable job in a queue	1,000/s peak	one row
Insert a job (with dedupe)	1,000/s peak	one row or a conflict
Renew a lease (heartbeat)	every minute per long job	one row
Schedules due to run	every few seconds	a few rows
Status of one job	low	one row
Dead jobs by type and error	on demand	grouped counts

The claim query checks run_at <= NOW() as well as the status. Delayed jobs live in the same table, so without that condition a job scheduled for next month would be claimed and run today. The index is ordered by run_at within each priority to keep that check cheap.

Finished jobs leave the hot table. Only pending and processing jobs live in jobs; completion deletes the row and appends a small record to job_history. The claim query then scans a table of thousands of rows rather than hundreds of millions.

The Leasing Mechanism

When you have multiple workers pulling from the same queue, you must prevent two workers from grabbing the same job.

The standard solution is a lease (or visibility timeout). When a worker fetches a job, it does not delete it. Instead, it "locks" it for a specific duration (e.g., 5 minutes) by updating a locked_until timestamp.

1. Fetch: Worker queries the database for jobs where status = 'pending' OR locked_until < NOW().
2. Lock: Worker updates locked_until = NOW() + 5 minutes, status = 'processing'.
3. Execute: Worker runs the job logic.
4. Complete: Worker deletes the job or marks it status = 'completed'.

If the worker crashes during execution, it never completes step 4. After 5 minutes, locked_until < NOW() becomes true again, and another worker can pick it up.

Sequence diagram illustrating job leasing and failure recovery between workers and the queue

Core Implementation in TypeScript

Here is a line-by-line implementation of a job worker using PostgreSQL as the backing store. We use a transaction with FOR UPDATE SKIP LOCKED to prevent race conditions during the fetch phase.

import { Client } from 'pg';

interface Job {
  id: string;
  type: string;
  payload: any;
  attempts: number;
}

export class JobWorker {
  private db: Client;
  private handlers: Map<string, (payload: any, job: Job) => Promise<void>>;

  constructor(db: Client) {
    this.db = db;
    this.handlers = new Map();
  }

  // Register a function to handle a specific job type
  register(type: string, handler: (payload: any, job: Job) => Promise<void>) {
    this.handlers.set(type, handler);
  }

  async poll(): Promise<void> {
    // 1. Fetch and Lock using Postgres row-level locking
    const query = `
      UPDATE jobs
      SET status = 'processing',
          locked_until = NOW() + INTERVAL '5 minutes',
          attempts = attempts + 1
      WHERE id = (
        SELECT id FROM jobs 
        WHERE run_at <= NOW()
          AND (status = 'pending'
               OR (status = 'processing' AND locked_until < NOW()))
        ORDER BY priority DESC, created_at ASC
        FOR UPDATE SKIP LOCKED
        LIMIT 1
      )
      RETURNING id, type, payload, attempts;
    `;

    try {
      const result = await this.db.query(query);
      
      if (result.rowCount === 0) {
        return; // No jobs available
      }

      const job: Job = result.rows[0];
      const handler = this.handlers.get(job.type);

      if (!handler) {
        throw new Error(`No handler for job type: ${job.type}`);
      }

      // 2. Execute
      await handler(job.payload, job);   // the job itself is passed so handlers can use job.id as an idempotency key

      // 3. Complete
      await this.db.query(`DELETE FROM jobs WHERE id = $1`, [job.id]);

    } catch (error) {
      // Job failed. It stays in the DB. Lock will eventually expire.
      // In production, we'd explicitly handle the error and check max attempts.
      console.error("Job execution failed:", error);
    }
  }
}

Real-world scenario: Payment Processing

Imagine a charge_card job. The handler calls Stripe. If Stripe is down, the handler throws an error. The lock expires 5 minutes later, and the job is retried.

Idempotency and Retries

Because workers can crash, and network calls can timeout, a job might be executed more than once (At-Least-Once delivery). Your job handlers must be idempotent.

If a charge_card job runs twice, you cannot charge the customer twice. You achieve this by passing a unique idempotency_key (usually the Job ID) to the external service.

Two details decide whether this actually works. The key must reach the handler. A tempting version passes only job.payload to the handler and then reads payload.jobId for the key; unless something copied the id into the payload, that value is undefined, the call goes out with no key at all, and every retry charges again. Passing the whole job, as the worker above does, removes the guesswork. The provider remembers keys for a limited time. Stripe keeps idempotency keys for at least 24 hours and may discard them after that, so a job retried two days later is not protected by the key alone; cap the retry window, or record the provider's payment id against the job and check it before charging. The example uses PaymentIntents, Stripe's current API for card payments; the older Charges API still exists but is not the recommended path for new code.

worker.register('charge_card', async (payload, job) => {
  // Stripe uses the idempotencyKey to ensure it only processes this charge once,
  // even if this handler runs 5 times due to worker crashes.
  await stripe.paymentIntents.create({
    amount: payload.amount,
    currency: 'usd',
    customer: payload.customerId,
    payment_method: payload.paymentMethodId,
    off_session: true,
    confirm: true,
  }, {
    idempotencyKey: `charge-card-${job.id}`,   // stable across every retry of this job
  });
});

Poison Pills and Dead Letter Queues (DLQ)

What if a bug in the code causes the job to crash every single time? The lock expires, it gets picked up again, and it crashes again. This is a "poison pill." It will consume worker CPU indefinitely.

To prevent this, you track attempts. If attempts > MAX_RETRIES, you move the job to a Dead Letter Queue (DLQ) — a separate table for failed jobs that require human intervention.

Priority and Worker Scaling

A single queue processes jobs FIFO (First-In-First-Out). But generating a password reset email (urgent) should not wait behind 10,000 daily newsletter emails (non-urgent).

1. Priority Queues: Add a priority integer to the job table. Workers ORDER BY priority DESC.
2. Dedicated Queues: Create separate queues (e.g., high_priority_queue, batch_queue) and assign dedicated workers to the high-priority queue.

Scaling: Workers are stateless. To process jobs faster, spin up more workers. Auto-scaling groups can monitor the queue depth (e.g., SELECT count(*) FROM jobs WHERE status='pending'). If depth > 1000, add 5 more workers.

Scale and failure

What breaks	Blast radius	How you find out	What keeps it running	Recovery
A worker crashes mid-job	that job, delayed	lease expiries per minute	the lease expires; another worker claims it	automatic; the handler must be idempotent
A job runs longer than its lease	the same job runs twice at once	jobs claimed while still running	heartbeat renews the lease	lengthen lease or heartbeat more often
Poison job	one job, retrying	attempts at the maximum	dead-letter table	fix the bug; retry the group
Jobs table bloats	every claim gets slower	dead-row count; claim query time	delete finished jobs; aggressive autovacuum	vacuum; move history out
08:00 schedule burst	urgent jobs wait behind digests	age of the oldest urgent job	separate queues and jitter	the burst drains; urgent never waits
Database down	nothing can be enqueued	enqueue errors in callers	the business transaction fails too, together	jobs and data stay consistent

The last row is a feature, not a bug. Because the queue shares the business database, the two cannot disagree: either the user and the welcome email both exist, or neither does. A separate broker would survive a database outage on its own, but then you are back to keeping two systems in step, which is the problem the outbox exists to solve.

What the interviewer will push on

How do you handle jobs that take longer than the lease duration (e.g., a 2-hour video encode)? You implement a heartbeat. The worker periodically (e.g., every 1 minute) updates the locked_until timestamp to push it further into the future. If the worker crashes, the heartbeat stops, and the lease expires shortly after.

Why use Postgres instead of RabbitMQ or SQS? Postgres is great for low-to-medium throughput (< 1000 jobs/sec). It gives you transactionality (you can create a user and enqueue a welcome email in the same transaction) and you don't need to maintain separate infrastructure. For high throughput, use a dedicated broker like SQS, Kafka, or RabbitMQ.

What happens if the worker completes the job, but crashes right before deleting it from the DB? The lock will expire, and another worker will pick it up. This is exactly why the job handler must be idempotent. The second worker will run the handler, but the idempotency key ensures no duplicate side effects occur.

---

Recall

• Leasing: Workers lock jobs temporarily. If a worker dies, the lock expires, enabling retries.
• FOR UPDATE SKIP LOCKED: The SQL magic to prevent multiple workers from grabbing the same job in a database-backed queue.
• Idempotency: Designing handlers so that executing them 1 time or 10 times results in the exact same system state.
• DLQ: A repository for jobs that have exceeded their maximum retry count (poison pills).

Quiz

1. Why must job handlers be idempotent in a background job system?
2. What is the purpose of a Dead Letter Queue (DLQ)?
3. How does SKIP LOCKED prevent race conditions in a database-backed queue?
4. How do you prevent a long-running job from having its lease expire while it is still actively processing?

Answers

1. Because workers or network calls can fail, leading to the same job being retried. Idempotency ensures the operation (e.g., charging a card) only happens once regardless of retries. 2. It holds "poison pill" jobs that constantly fail, preventing them from infinite retry loops that drain worker resources. 3. It allows a database transaction to lock a row and skip over rows locked by other concurrent transactions, letting multiple workers pull jobs simultaneously without blocking each other. 4. By using a heartbeat mechanism where the worker periodically extends the `locked_until` timestamp while it continues processing.