Appearance
11.23 Enterprise Workflow & Approval System
Enterprise workflow systems manage long-running processes that require human intervention. Unlike standard web requests that complete in milliseconds, an expense report approval or a vendor onboarding workflow can take days or weeks. This changes everything about how we model state, guarantee durability, and handle failures.
A workflow is a directed graph of steps. Some steps are automated (e.g., "Check budget API"), while others wait for a human (e.g., "Manager approval"). The core engineering challenge is managing state over arbitrary time horizons without blocking system resources.
Requirements
The system on this page is an approvals product sold to companies: about 400 customer organisations with 2 million employees between them. An employee submits an expense, a purchase request or a vendor onboarding form; it routes to the right people; they approve or reject; reminders and escalations fire when nobody acts; and every step is kept for auditors.
Functional. Define workflow templates (steps, approver rules, deadlines). Start a workflow for a subject such as an expense. Approve, reject or delegate a step. Send reminders and escalate on deadlines. Show each person their pending tasks. Search workflows. Keep a complete audit trail.
Non-functional, with numbers.
- No lost and no doubled decision. An approval is recorded exactly once, even when the button is double-clicked or the request is retried. This is the requirement section "Concurrency and Idempotency" below exists for.
- An approval click is acknowledged within 300 ms at p95. People click and expect the task to disappear.
- Deadlines fire within one minute of being due, and survive any server restart.
- The pending-tasks inbox reflects a decision within 5 seconds.
- The audit trail is kept for 7 years, which is a common retention period for financial records.
Out of scope today: the visual template designer, and the email and chat integrations beyond the signed approve-links mentioned in the API.
The clarifying questions, and what each answer changes
"Can a template change while workflows are running?" It always can, because finance changes the approval limits every year. The answer forces every workflow to pin the template version it started with. Otherwise an expense submitted under the old rules is suddenly routed under new ones half-way through, and nobody can explain to an auditor why.
"Can approvers delegate, or be out of office?" If yes, routing cannot look only at the org chart; it needs a delegation table consulted at the moment a task is assigned, and the audit trail must record "approved by Omar on behalf of Priya".
"Can people approve from an email or chat message?" If yes, every approve-link is a signed, single-use token that ends up calling the same decision endpoint, and it carries its own requestId, so a link clicked twice is a duplicate, not a second approval.
"What happens when an approver leaves the company?" Their open tasks become orphans. The answer decides whether we listen to the directory's deprovisioning events (11.25) and reassign automatically to the leaver's manager.
"Do some steps need several approvers at once, all of them or any one of them?" "All of" needs a set of outstanding approvers and a fan-in rule; "any of" needs the first decision to cancel the others' tasks. Both change the state model, so find out before drawing it.
Estimation
Workflow starts. 2 million employees × about 2 workflows a month = 4 million a month, about 133,000 a day. Divided by 100,000 seconds (the rounding from 11.0), that is 1.3 a second on average, and perhaps 15 a second in the busiest working hour. What that forces: nothing on the write side. One PostgreSQL primary absorbs this with room to spare. The hard parts of this system are correctness and time, not throughput, and the design should say so instead of reaching for a cluster.
Event volume and retention. Each workflow produces about 6 events (started, two approval requests, two decisions, a reminder), so 800,000 events a day. At about 1 KB each that is 800 MB a day, about 290 GB a year, and about 2 TB over 7 years. What that forces: the event table is partitioned by month. Recent months stay in PostgreSQL; partitions older than a year are exported to object storage under a retention lock (11.27) and dropped from the database.
Inbox reads. About 1 employee in 5 approves things, so 400,000 approvers, each opening their inbox about 10 times a day: 4 million reads a day, 46 a second, and around 460 a second in the 9 a.m. rush. If each inbox view rebuilt state by replaying events, an approver with 20 pending workflows of 6 events each costs 120 event reads per view, which is about 55,000 event reads a second at peak. What that forces: the inbox is a separate read model, a pending_tasks table keyed by assignee and updated from the event stream (CQRS, 10.8.4). Event sourcing is the write model; nobody queries it directly for lists.
Timers. At any moment roughly 133,000 workflows a day × 2 days open on average = 266,000 open workflows, each with about two timers (a reminder and an escalation): about 530,000 pending timers, firing at about 3 a second. What that forces: a timers table with an index on fire_at, polled every few seconds, is plenty. What matters is that firing is idempotent and survives restarts, not that it is fast.
API
http
POST /workflows
Idempotency-Key: 4f1c9a6e-expense-88213
{ "template": "expense-approval",
"subject": { "type": "expense", "id": "exp_88213" },
"data": { "amountPaise": 4500000, "currency": "INR", "department": "IT_HARDWARE" } }
→ 201 Created
Location: /workflows/wf_01JA2K9
{ "workflowId": "wf_01JA2K9", "templateVersion": 7,
"status": "PENDING_MANAGER", "version": 2 }http
POST /workflows/wf_01JA2K9/decisions
{ "decision": "approve", "requestId": "req-77", "comment": "Within budget" }
→ 200 OK { "status": "PENDING_FINANCE", "version": 3, "duplicate": false }
→ 200 OK { "status": "PENDING_FINANCE", "version": 3, "duplicate": true } (same requestId again)
→ 409 Conflict (wrong step for this approver)
{ "error": { "code": "NOT_YOUR_STEP",
"message": "The workflow is waiting for FINANCE.",
"requestId": "req-78" } }http
GET /me/tasks?limit=50&cursor=eyJkdWUiOiIyMDI2LTEwLTIwIn0
→ 200 OK
{ "items": [ { "workflowId": "wf_01JA2K9", "step": "FINANCE",
"subject": "Expense exp_88213, ₹45,000", "dueAt": "2026-10-20T17:00:00+05:30" } ],
"nextCursor": "eyJkdWUiOiIyMDI2LTEwLTIxIn0" }
GET /workflows/wf_01JA2K9 → state, template version, full historyStarting a workflow takes an Idempotency-Key, because the system that calls it (the expense tool) retries on timeouts, and a duplicate workflow means a manager approves the same expense twice (10.21).
A duplicate decision returns 200, not an error. From the user's point of view the approval worked; the second click is not a mistake worth showing. Returning "duplicate": true keeps that visible for debugging. A decision on the wrong step is a 409, because the request is well-formed but the workflow is not in a state where it applies.
The inbox uses a cursor, ordered by due date, because tasks are added and removed constantly while someone scrolls, and an offset would show them duplicates and gaps (9.6.2).
Data model
workflow_templates
key TEXT, version INT
definition JSONB NOT NULL -- steps, routing rules, deadlines
PRIMARY KEY (key, version)
workflows
tenant_id UUID NOT NULL
id UUID
template_key TEXT NOT NULL, template_version INT NOT NULL -- pinned at start
status TEXT NOT NULL
version INT NOT NULL -- number of events so far
PRIMARY KEY (tenant_id, id)
workflow_events -- the write model, append-only
tenant_id, workflow_id, seq INT -- PRIMARY KEY (tenant_id, workflow_id, seq)
type TEXT NOT NULL
actor_id UUID, role TEXT, on_behalf_of UUID
request_id TEXT -- UNIQUE (tenant_id, workflow_id, request_id)
payload JSONB
created_at TIMESTAMPTZ NOT NULL -- table partitioned by month
pending_tasks -- the read model behind /me/tasks
tenant_id, assignee_id, workflow_id, step
due_at TIMESTAMPTZ NOT NULL
PRIMARY KEY (tenant_id, assignee_id, workflow_id, step)
timers
id UUID PRIMARY KEY, tenant_id, workflow_id, step, kind -- reminder | escalate
fire_at TIMESTAMPTZ NOT NULL -- partial index WHERE fired_at IS NULL
claimed_until TIMESTAMPTZ, fired_at TIMESTAMPTZ
delegations
tenant_id, user_id, delegate_id, starts_at, ends_atAccess patterns:
| Query | Frequency | Returns |
|---|---|---|
Append an event at the expected seq | ~15/s peak | one row or a conflict |
| Load one workflow's events | ~15/s | about 6 rows |
| A person's pending tasks, by due date | 460/s peak | up to 50 rows |
| Due timers, oldest first | every 5 s | up to a few hundred |
| One workflow's full history | low | about 6 rows |
tenant_id leads every key. Every query is scoped to one customer organisation, so the tenant is the natural partition key if the system ever outgrows one database, and it is what row-level security keys on (11.24).
(workflow_id, seq) is the optimistic-concurrency check that the approval handler below relies on: two concurrent appends at the same seq cannot both succeed. UNIQUE (…, request_id) is a second, database-level guard for duplicates, so a future code path that forgets the in-code check still cannot record the same click twice.
pending_tasks is derived. It can be rebuilt at any time by replaying the events, which is why it is safe to keep it denormalised and fast.
The State Transition Model
At the heart of any workflow system is a state machine. It dictates the valid transitions from one step to the next.
To build this reliably, we cannot hold a database transaction open while waiting for the manager. We must serialize the state, durably store it, and "wake up" the workflow when an event occurs.
Event Sourcing for the Audit Trail
Enterprise compliance (e.g., SOX, GDPR) mandates an ironclad audit trail. You cannot simply overwrite a status column in a database. You must append state changes as immutable events.
Event Sourcing models the workflow not as a row in a table, but as a sequence of events. The current state is derived by replaying the events.
typescript
// 1. Define the immutable events that can happen
type WorkflowEvent =
| { type: 'WORKFLOW_STARTED', workflowId: string, timestamp: number, payload: any }
| { type: 'APPROVAL_REQUESTED', workflowId: string, role: string, timestamp: number }
| { type: 'APPROVED', workflowId: string, userId: string, role: string, requestId: string, notes: string, timestamp: number }
| { type: 'REJECTED', workflowId: string, userId: string, notes: string, timestamp: number };
// 2. The aggregate state we care about
interface WorkflowState {
workflowId: string;
status: 'DRAFT' | 'PENDING_MANAGER' | 'PENDING_FINANCE' | 'APPROVED' | 'REJECTED';
pendingRole: string | null;
history: WorkflowEvent[];
}
// 3. The reducer function to reconstruct state from events
function applyEvent(state: WorkflowState, event: WorkflowEvent): WorkflowState {
// Always append to history for the audit trail
const newState = { ...state, history: [...state.history, event] };
switch (event.type) {
case 'WORKFLOW_STARTED':
newState.status = 'DRAFT';
break;
case 'APPROVAL_REQUESTED':
newState.status = `PENDING_${event.role.toUpperCase()}` as any;
newState.pendingRole = event.role;
break;
case 'APPROVED':
// Simplified logic: If manager approved, ask finance. If finance, we are done.
if (state.pendingRole === 'MANAGER') {
newState.status = 'PENDING_FINANCE';
newState.pendingRole = 'FINANCE';
} else if (state.pendingRole === 'FINANCE') {
newState.status = 'APPROVED';
newState.pendingRole = null;
}
break;
case 'REJECTED':
newState.status = 'REJECTED';
newState.pendingRole = null;
break;
}
return newState;
}
// Rehydrate state from database rows
function getWorkflowState(events: WorkflowEvent[]): WorkflowState {
const initialState: WorkflowState = {
workflowId: '', status: 'DRAFT', pendingRole: null, history: []
};
return events.reduce(applyEvent, initialState);
}This ensures the audit trail is computationally tied to the actual state transitions. You literally cannot change the state without leaving an event in the log.
Long-Running Workflows and Timeouts
Human tasks need deadlines. If a manager does not approve an expense report within 48 hours, the system should escalate it or send a reminder.
You cannot use an in-memory setTimeout() because the server might restart. Instead, the deadline must be stored durably. The simplest robust version is a timers table with a fire_at column and an index on it: a scheduler polls every few seconds for rows whose time has passed, claims each one atomically, and emits the escalation event. A workflow engine such as Temporal or AWS Step Functions does the same thing for you and survives restarts by design. Be careful with message-queue tricks here: an SQS message can be delayed by at most 15 minutes, and RabbitMQ's delayed-message plugin keeps pending messages on a single node, so neither is a safe place to park a 48-hour or 7-day deadline. The job-scheduler study (11.18) designs this timer service in full.
Roles and Dynamic Routing
In enterprise systems, approval routing is rarely hardcoded. It depends on business rules:
- Expenses under $100 auto-approve.
- Expenses $100-$1000 require immediate Manager approval.
- Expenses >$1000 require Manager AND VP approval.
We decouple the workflow state machine from the rule evaluation engine.
typescript
interface Expense {
amount: number;
department: string;
}
function determineNextApprovers(expense: Expense): string[] {
const approvers = [];
// Rule 1: Always need manager if over 100
if (expense.amount > 100) {
approvers.push('ROLE_MANAGER');
}
// Rule 2: High value needs VP
if (expense.amount > 1000) {
approvers.push('ROLE_VP');
}
// Rule 3: IT gear always needs IT review
if (expense.department === 'IT_HARDWARE') {
approvers.push('ROLE_IT_ADMIN');
}
return approvers; // e.g., ['ROLE_MANAGER', 'ROLE_VP']
}When multiple roles are required, the workflow engine must handle concurrent approvals (fan-out) and wait for all of them to complete (fan-in). The state machine requires a Set of pending approvals. Only when the Set is empty does the workflow advance.
Concurrency and Idempotency
When dealing with humans clicking buttons in web apps, they will double-click. Network requests will retry. The APPROVED event might hit your API twice, at the same moment.
Here is exactly how that goes wrong. The manager double-clicks "Approve". Both requests load the event log, and both see the workflow waiting for the manager. Both pass the check and both append an APPROVED event. Now replay the log: the first APPROVED moves the workflow from "waiting for manager" to "waiting for finance", and the second APPROVED moves it from "waiting for finance" to fully approved. The expense is approved, and finance never saw it. Running the obvious "load, check, append" code twice concurrently against a test store produces exactly that: status APPROVED after one person clicked one button twice.
"Check the current state before appending" does not prevent this on its own, because both requests check before either appends. That is the check-then-act race from 2.4. Two things fix it together.
ts
class VersionConflict extends Error {}
async function processApproval(
workflowId: string,
approver: { userId: string; role: "MANAGER" | "FINANCE" },
requestId: string, // (1)
): Promise<"approved" | "duplicate"> {
for (let attempt = 1; attempt <= 3; attempt++) {
const events = await eventStore.load(workflowId); // (2)
if (events.some(e => e.type === "APPROVED" && e.requestId === requestId)) {
return "duplicate"; // (3)
}
const state = getWorkflowState(events);
if (state.pendingRole !== approver.role) { // (4)
throw new Error(`409: ${approver.role} cannot approve while the workflow waits for ${state.pendingRole}`);
}
try {
await eventStore.append(workflowId, events.length, { // (5)
type: "APPROVED", workflowId, userId: approver.userId, role: approver.role,
requestId, notes: "", timestamp: Date.now(),
});
return "approved";
} catch (err) {
if (!(err instanceof VersionConflict)) throw err; // (6)
}
}
throw new Error("503: workflow is busy, try again");
}- The UI generates one
requestIdwhen it renders the Approve button, so a double-click sends the same id twice, while a genuinely new approval later gets a new one. - Load the whole event log for this workflow.
- If an
APPROVEDevent with thisrequestIdis already in the log, this request is a repeat. Report "duplicate" and change nothing. - Check that the approver's role is the one the workflow is waiting for. A manager cannot approve the finance step.
- Append with an expected version. The event store keeps one row per event with a unique key on
(workflow_id, seq). The append says "this is event numberevents.length". If another request appended first, that number is already taken, the insert fails, and the store raisesVersionConflict. This is optimistic concurrency, the same compare-and-set idea as 9.6.3's ETags, applied to an event log. - On a conflict, go round again: reload, and this time step (3) sees the other request's event and returns "duplicate". Any other error is real and is passed up.
Run against a test event store, the double-click now returns approved + duplicate and leaves the workflow waiting for finance; a second approval attempt by the manager is refused with a 409; and finance's approval then completes it. The reducer from the first code block also has to accept the extra role and requestId fields on APPROVED, which the event type above now includes.
Scale and failure
| What breaks | Blast radius | How you find out | What keeps it running | Recovery |
|---|---|---|---|---|
| Double-click or retried decision | one workflow skips a step | duplicate: true rate; conflict rate | requestId plus expected-seq append | nothing to recover; the duplicate was refused |
| Timer poller down | reminders and escalations late | age of the oldest unfired due timer | timers are rows; nothing is lost | restart; due timers fire in order, idempotently |
| Read model lagging | inboxes stale; approvers see done tasks | consumer lag on the event stream | decisions still validate against the events | catch up, or rebuild pending_tasks from events |
| Approver leaves the company | tasks orphaned, workflows stall | tasks assigned to deactivated users | directory events reassign to their manager | bulk reassign, recorded as events |
| Template edited mid-flight | wrong routing, unexplainable audit | none, unless versions are pinned | template_version pinned at start | already-running workflows are untouched |
Alarm on the oldest due timer, not on the number of timers. Half a million pending timers is normal; one escalation that is twenty minutes overdue is an incident, because a person somewhere is waiting on it.
What the interviewer will push on
1. "How do you handle schema evolution for your Event Source?" Over a 5-year period, the payload of WORKFLOW_STARTED will change. Do not rewrite old events (that breaks immutability). Instead, handle it in the reducer: your applyEvent function must support multiple versions of the event schema, upgrading them on the fly in memory, or applying defaults for missing fields.
2. "What happens if an external API call fails during a workflow transition?" Use the Saga Pattern. If the workflow requires calling the CRM and the Billing system, and Billing fails, you must emit compensating events to undo the CRM changes. Never wrap external network calls inside your database transaction.
3. "How do you search for workflows?" Event sourcing is terrible for querying (e.g., "Find all workflows waiting on John"). You must build a CQRS (Command Query Responsibility Segregation) architecture. As events are appended to the event store, publish them to a Kafka topic. A separate consumer reads these events and updates a read-optimized PostgreSQL table or Elasticsearch index.
Recall
- State Machines: Model workflows as explicit states and transitions. Do not block threads waiting for human input.
- Event Sourcing: Store state changes as an immutable sequence of events. The audit trail is built-in.
- Timeouts: Use durable timers or delayed queues to handle human inaction. In-memory timers fail on restart.
- Idempotency: Protect against UI double-clicks and retries with a per-action
requestIdplus an append that carries the expected version, so two concurrent requests cannot both pass the check. - CQRS: Separate the write model (Event Store) from the read model (Search Index) to allow querying across thousands of workflows.
Quiz
- Why shouldn't you use a database transaction that remains open until the manager approves the request?
- In an event-sourced system, how is the current state of a workflow determined?
- What is the mechanism for triggering an action if a human approver ignores a task for 7 days?
- How do you prevent a double-click on the "Approve" button from breaking the workflow state?
- Why is a standard event store poorly suited for powering a dashboard showing "All my pending tasks"?