11.6 — Notification System

At 09:14 a marketing campaign starts sending to five million users. At 09:16 a customer clicks "reset my password" and waits. The email arrives at 09:36, twenty-two minutes later, by which time they have tried three more times and opened a support ticket. Nothing failed. No alarm fired. Every message was delivered successfully. The password reset was simply behind five million other messages in the same queue.

That is the notification system's characteristic failure, and it is the one to design against first, because it is invisible in every metric that a naive version of this system produces. Throughput was fine. Error rate was zero. Delivery rate was 100%.

This is also the system every product grows and almost nobody designs. It looks like "send a message" and it is actually a fan-out pipeline sitting on top of third parties you do not control, with per-user preferences, deduplication that has to work because a duplicate message costs real money, rate budgets negotiated in a contract, and delivery guarantees that are different on every channel.

1. Requirements

Functional. Send to four channels: push, email, SMS, and in-app. Per-user and per-category preferences and quiet hours. Templates with localisation. Scheduled and recurring sends. Campaigns to millions of users. A delivery status trail per notification, per channel.

Non-functional, with numbers.

• Transactional notifications delivered within 5 seconds at p95. A one-time code that arrives in a minute is a failed login.
• At-least-once delivery with application-level deduplication. Exactly-once across a third party is not achievable, and pretending otherwise is how products send the same SMS twice.
• Campaigns must never delay transactional traffic, which is the requirement the story at the top of this page violates.
• A failing provider must not stop the system. One SMS gateway being down should degrade SMS, not everything.

Out of scope today: the content and campaign authoring interface, experiment analysis, and the in-app inbox's front end — though the drill at the end designs the inbox's backend, because that is where the genuinely hard part lives.

The clarifying questions, and what each answer changes

"Which of these messages cost money to send?" SMS costs a fraction of a penny each and it adds up; push and in-app cost nothing. That difference decides how much you invest in deduplication, because a deduplication bug on push is embarrassing and a deduplication bug on SMS is an invoice.

"Is a notification ever required for correctness?" If any workflow depends on the user having seen a notification, the design is already wrong, because no channel can promise that. Finding this out in the requirements phase saves a very awkward conversation later.

"How stale can a message be before it should not be sent at all?" A ride-arrival notification delivered forty minutes late is worse than no notification. A one-time code delivered after it expired is worse than silence. This answer becomes a per-category maximum age, and messages older than it are dropped rather than delivered.

"Who owns the provider relationship, and what rate did we contract for?" The external rate limit is usually the real ceiling of the whole system, and it is a number in a contract rather than a property of your infrastructure.

"Can a campaign be paused?" Ask it early. The answer is always yes once someone has lived through the incident above, and it is much cheaper to build in than to add during one.

2. Estimation

Steady-state volume. 10 million users × ~5 notifications a day = 50 million a day ÷ 86,400 ≈ 600 a second average, and with an evening peak, ~3,000 a second. What that forces: not much on its own. Three thousand sends a second across four channels is ordinary work for a pool of workers.

Campaign burst. 5 million push messages intended to go out over ten minutes = 8,300 a second for the duration. What that forces: the sends cannot be synchronous with whatever triggered them, because nothing upstream can absorb a fourteen-fold jump in load. The burst has to land in a queue and drain at a controlled rate. And because that burst is fourteen times the peak transactional rate, it will bury the transactional traffic unless the two are physically separated — which is the design's central decision.

Status storage. A delivery status record of ~500 bytes × 50 million a day = 25 GB a day, so about 2.2 TB over a 90-day retention. What that forces: status lives in a store partitioned by time so old partitions can be dropped whole rather than deleted row by row (10.6), and a retention period has to be chosen deliberately rather than by omission.

Provider limits, which are the real ceiling. SMS gateways commonly cap an account at a few hundred messages a second. Email providers cap by hour. Push services are far more generous but still throttle. What that forces: the system must shape its own traffic to the contracted rate rather than discovering the limit as a wall of errors. A per-provider token bucket is not an optimisation here, it is a contractual requirement expressed in code.

Money, which is the number that changes how careful you are. Suppose 1 million of the daily 50 million are SMS at roughly $0.007 each. That is $7,000 a day, about $2.5 million a year. A deduplication bug that doubles SMS for one day costs $7,000 and a great deal of trust; one that runs for a week before anyone notices costs $49,000. What that forces: the deduplication key is enforced by a database constraint rather than by application logic, because a constraint cannot be bypassed by a new code path written next year.

3. API

POST /notifications
{ "userId": "u_8821",
  "templateId": "order_shipped",
  "data": { "orderId": "8821", "carrier": "Royal Mail" },
  "channels": ["push", "email"],
  "dedupeKey": "order_shipped:8821",
  "priority": "transactional" }

202 Accepted
Location: /notifications/n_01J9F4Q2K
{ "notificationId": "n_01J9F4Q2K", "status": "accepted" }

GET /notifications/n_01J9F4Q2K
→ 200 OK
{ "notificationId": "n_01J9F4Q2K",
  "status": "partially_delivered",
  "channels": [
    { "channel": "push",  "status": "delivered", "providerMessageId": "apns-9f2c", "attempts": 1 },
    { "channel": "email", "status": "suppressed", "reason": "user_preference_off" },
    { "channel": "sms",   "status": "deferred",  "reason": "quiet_hours", "sendAfter": "2026-08-01T07:00:00+01:00" }
  ] }

PUT /users/u_8821/preferences
{ "category": "marketing", "channel": "push", "enabled": false }

POST /campaigns
{ "segmentId": "seg_lapsed_30d", "templateId": "come_back", "startAt": "...", "ratePerSecond": 8000 }
→ 202 { "campaignId": "c_01J9F51XX", "estimatedRecipients": 5011234 }

POST /campaigns/c_01J9F51XX/pause
→ 200 { "status": "paused", "sent": 1204331, "remaining": 3806903 }

202, not 200, and it is not negotiable here. Delivery takes seconds to minutes and depends on a third party. The caller must never hold a connection open waiting for a carrier (9.6.1). What 202 promises is precise: the notification is durably recorded and will be attempted according to policy. It does not promise delivery, and section 8 is largely about being honest regarding that difference.

dedupeKey is the caller's idempotency key (10.4). A service that retries "order 8821 shipped" because its own request timed out must produce one message, not two. The key is the caller's responsibility to make stable and meaningful — order_shipped:8821 is stable, a random value generated per attempt is not, and the difference is worth explaining to callers because they will get it wrong.

The status response carries suppression reasons. "Why didn't the customer get the email?" is the single most common support question this system generates, and without a recorded reason it is unanswerable. user_preference_off, quiet_hours, frequency_cap, invalid_token, provider_rejected — each one turns a two-hour investigation into a single lookup.

Errors, one envelope:

422 Unprocessable Entity
{ "error": { "code": "unknown_template",
             "message": "No template 'order_shipped' for locale 'ja-JP'.",
             "requestId": "req_01J9F52AA" } }

4. Data model

notifications
  id            UUID PRIMARY KEY
  user_id       UUID NOT NULL
  template_id   TEXT NOT NULL
  dedupe_key    TEXT NOT NULL          -- UNIQUE (user_id, dedupe_key)
  priority      SMALLINT NOT NULL      -- transactional | bulk | scheduled
  category      TEXT NOT NULL          -- security | order | marketing | ...
  created_at    TIMESTAMPTZ NOT NULL
  expires_at    TIMESTAMPTZ NOT NULL   -- per-category max age; past this, drop

deliveries
  id            UUID PRIMARY KEY
  notification_id UUID NOT NULL
  channel       SMALLINT NOT NULL
  provider      TEXT
  status        SMALLINT NOT NULL      -- queued | sent | delivered | bounced | suppressed | failed
  reason        TEXT                   -- why suppressed, or the provider's error
  attempts      SMALLINT NOT NULL
  provider_msg_id TEXT

preferences
  user_id       UUID
  category      TEXT
  channel       SMALLINT
  enabled       BOOLEAN NOT NULL
  timezone      TEXT NOT NULL          -- IANA name, e.g. 'Europe/London'
  quiet_from    TIME, quiet_to TIME
  PRIMARY KEY (user_id, category, channel)

devices
  user_id       UUID
  token         TEXT PRIMARY KEY
  platform      SMALLINT
  last_seen     TIMESTAMPTZ
  invalid_at    TIMESTAMPTZ NULL       -- set from provider feedback

templates
  id            TEXT, version INT, channel SMALLINT, locale TEXT
  subject       TEXT, body TEXT
  PRIMARY KEY (id, version, channel, locale)

Access patterns:

Query	Frequency	Returns
Insert a notification with dedupe check	3,000/s peak	one row or a conflict
Read preferences for one user	3,000/s peak	a handful of rows
Read a user's valid device tokens	~1,500/s	1–5 rows
Update a delivery's status from a callback	~5,000/s	one row
Status trail for one notification	low	≤ 4 rows
A user's recent notifications	low	20–50 rows

Partition notifications and deliveries by user_id, with a time component so old partitions drop whole. Every status query is user-scoped or notification-scoped, and both route to one partition (10.6).

The unique index on (user_id, dedupe_key) is the deduplication mechanism, and it is worth being explicit about why it lives in the database rather than in code. At-least-once delivery means the pipeline will try to process the same notification twice — after a worker crash, after a queue redelivery, after a caller's retry. A check in application code can be bypassed by the next code path someone adds. A constraint cannot. The handler inserts, catches the duplicate-key error, and treats it as success (10.4).

preferences.timezone holds an IANA name, not an offset. An offset like +01:00 is wrong for half the year in most of the world, so quiet hours computed from a stored offset silently shift by an hour twice a year. This is a real bug that ships regularly.

devices.invalid_at exists because push tokens rot, which section 6.4 covers and which is the most commonly omitted part of a notification design.

5. Architecture

Figure 1 — The pipeline. ① Filtering happens once, before anything is queued, so a suppressed notification never consumes worker capacity. ② Three separate queues with separate worker pools, which is what stops a campaign from delaying a password reset. ③ Workers own the provider adapters, each with its own circuit breaker and rate budget. ④ Provider callbacks flow back in and update device validity and preferences, which is what stops the system degrading over months.

The isolation is the design, so it is worth drawing on its own.

Figure 2 — Why a priority field is not isolation. Even a broker that honours priorities cannot help once every worker in the pool is inside a long call to an SMS gateway. Isolation has to be physical: separate queues, separate workers, and separate provider rate budgets.

6. Deep dives

6.1 Priority isolation, which is the whole design

The failure at the top of this page has a name — head-of-line blocking — and it has a fix that people consistently get half right.

The half-right fix is a priority field on one queue. It helps a little and it does not solve the problem, for a reason worth stating precisely: even if the broker always hands out the highest-priority message first, that only matters when a worker is free to receive one. During a campaign, every worker in the pool is inside a call to a provider that takes 200–800 ms. The password reset is at the front of the queue and there is nobody to give it to.

The actual fix is physical separation at three levels. Separate queues, so the campaign's backlog is not something transactional messages sit behind. Separate worker pools with reserved capacity, so bulk work cannot occupy the workers that transactional work needs. And separate provider rate budgets — or separate provider sub-accounts — so the campaign spends its own allocation and the transactional path always has its own. This is the bulkhead idea from 10.9 applied to three resources at once, and missing any one of them reproduces the incident.

And campaigns get paced. Enqueuing five million messages instantly is a choice; enqueuing them at 8,000 a second over ten minutes is the same campaign with none of the blast radius. Pacing also protects the provider relationship, since providers throttle accounts that spike.

6.2 The filter, and why it runs before the queue

Four checks happen once, at intake, before anything is enqueued: deduplication, per-category preferences, quiet hours, and frequency caps. Running them before the queue means a suppressed notification never occupies a worker, which matters enormously during a campaign where a large fraction of recipients have opted out.

Quiet hours must be evaluated in the user's timezone. This sounds obvious and is one of the most reliable bugs in this whole domain. Two rules: store an IANA timezone name (Europe/London) rather than an offset, because an offset is wrong for half the year; and compute the user's local time from that name at evaluation time rather than at preference-save time.

A suppressed transactional message is deferred, not dropped. "Your order shipped" arriving at 07:00 instead of 02:00 is correct behaviour. "Your order shipped" never arriving is a bug. So the delivery row records deferred with a sendAfter, and a scheduled worker picks it up when the window opens.

Security messages override quiet hours entirely. A category carries an overridable flag, and "someone signed into your account from a new device" ignores it. Getting this wrong in either direction is bad: waking people for marketing destroys trust, and holding a security alert until morning is a genuine harm.

Frequency caps — "at most three marketing pushes a day" — need a per-user counter, and that counter needs the same atomicity discipline as the rate limiter in 11.3. A read-then-increment across concurrent workers lets a user receive five.

The cost of filtering early, stated honestly: a preference change made after a scheduled message was filtered will not be respected. So anything scheduled more than a few minutes ahead re-checks preferences at send time. That is a second evaluation, and it is worth the duplication because "I turned off marketing emails and still got one" is an unsubscribe.

6.3 Providers fail, and each failure needs a different response

Every provider call goes through an adapter carrying the full reliability ladder (10.9), and the interesting part is which failures get which treatment.

A timeout is ambiguous and must be retried. You do not know whether the message was sent. Retrying risks a duplicate; not retrying risks silence. For a password reset, retry. The duplicate is annoying, the silence is a support ticket.

A 5xx from the provider is retryable, with exponential backoff and jitter so that ten thousand workers do not all retry in the same instant.

A 4xx is usually not retryable and this is where the money is lost. "Invalid phone number" will fail identically forever. Retrying it consumes rate budget that working messages need, and on some providers it consumes money per attempt. So the adapter classifies errors explicitly and a non-retryable error is recorded and stopped, not retried.

A circuit breaker stops a dead provider from consuming every worker. Without one, a provider timing out at 30 seconds turns your entire pool into a queue of threads waiting on a corpse — and that failure spreads to the channels that are still healthy, because they share the pool.

A fallback where one exists. A second SMS gateway, or email as a fallback for a push that could not be delivered. Fallbacks have to be chosen per category: falling back from push to SMS for a marketing message is expensive and unwelcome; doing it for a fraud alert is correct.

A dead-letter queue carrying the full payload and the error. The alternative — messages that fail and vanish — produces the worst incident report in this domain, which is "we lost about forty thousand notifications and we do not know which ones".

6.4 Push tokens rot, and ignoring it silently wastes most of your volume

A push token identifies an app installation on a device. It stops being valid constantly: the app is uninstalled, the device is wiped, the operating system rotates the token, the user revokes permission. Nothing about your system is told at send time — the provider accepts the message and reports the invalid token later, through an asynchronous callback.

A system that does not consume those callbacks accumulates dead tokens forever. After two years, a substantial share of push volume is being sent to installations that no longer exist. It costs rate budget, it makes engagement metrics meaningless, and it is invisible because every send appears successful.

The entire fix is two things: a consumer for the provider's feedback callbacks, and the devices.invalid_at column it writes to. Send-time queries filter on invalid_at IS NULL. It is perhaps thirty lines of code and it is the most commonly omitted part of a notification design, which is why interviewers ask about it.

The same callback path also carries bounces (an email address that no longer exists — keep sending and your sender reputation degrades until legitimate mail is filtered) and unsubscribes, which must be written into preferences immediately, because ignoring an unsubscribe is a legal problem in most jurisdictions, not merely a rude one.

6.5 Templates, which reach millions before anyone can stop them

A template change is a deploy that reaches five million people in ten minutes and cannot be recalled. That asymmetry earns templates their own machinery.

Versioned, never edited in place. A notification records the template version it used, so a support question about a message sent last month can be answered by rendering that exact version.

Staged rollout. A new version goes to a small percentage first, and the campaign is pausable, so a broken template affects thousands rather than millions.

Rendering failures are caught at intake, not at send. A template referencing data.orderId when the caller sent data.order_id should fail the POST with a 422, not silently render "Your order has shipped" to a million people. Validate the template's required fields against the payload when the notification is accepted.

Localisation falls back explicitly. A missing ja-JP template falls back to ja, then to the default locale, and the fallback is recorded — because "the Japanese customers all got English" is a bug you want to find in a metric rather than a review.

7. Decision Ledger

Decision	Alternatives	Why this	What it costs
202 plus a status resource	synchronous send	third-party latency never blocks the caller; bursts buffer	the caller must poll or accept a webhook
Separate queues and pools and rate budgets per class	one queue with a priority field	a priority field cannot help when every worker is busy	more infrastructure; capacity must be split deliberately
At-least-once with a unique dedupeKey	aim for exactly-once	achievable, and enforced by a constraint rather than by code	callers must supply stable keys, and will get it wrong at first
Filter before enqueue	filter inside the worker	suppressed messages never consume capacity	scheduled sends must re-check preferences at send time
Provider adapters behind one interface	call each provider's library directly	breaker, retry, rate budget and metrics applied uniformly	one more layer, justified once there are two providers per channel
Per-category maximum age, then drop	deliver whatever eventually gets through	a stale one-time code is worse than silence	a product decision has to be made per category
Consume provider feedback callbacks	ignore them	keeps push volume real and protects sender reputation	a second inbound path to operate and secure
Versioned templates with staged rollout	edit templates in place	a bad template reaches millions in minutes and cannot be recalled	a small release process for content

8. Scale and failure

Campaign fan-out must never be one job. The campaign service resolves the segment and chunks it into batches of about a thousand, each enqueued separately with dedupeKey = campaignId:userId. Every chunk is independently retryable and idempotent, so a worker crash halfway through resumes from the chunk boundary instead of restarting the campaign or duplicating a million messages.

At 10×, the provider becomes the bottleneck long before your infrastructure does. The answers are commercial as much as technical: negotiate higher throughput, shard across multiple provider sub-accounts, and shape traffic to the contracted rate. Discovering a contractual rate limit as a wall of 429s during a launch is a bad way to learn a number that was written down all along.

What breaks	Blast radius	How you find out	What keeps it running	Recovery
Campaign floods a shared queue	transactional latency, silently	oldest-message age per queue, tight threshold on transactional	separate queues, pools and rate budgets	pause the campaign; the backlog drains in seconds
One provider down	one channel	breaker state; provider error rate	breaker opens; fallback provider; messages queue	replay from the queue, dropping anything past its max age
Provider slow, not down	that channel's workers all blocked	worker pool saturation; call duration p99	timeout plus breaker converts slow into failed fast	breaker probes and recovers
Queue backing up	growing staleness, no errors	age of the oldest message, not depth	add consumers; shed bulk first	drain with age as the exit condition
Bad template deployed	up to millions of users, unrecallable	render-failure rate; a sudden unsubscribe spike	staged rollout; pausable campaigns; validate at intake	pause, roll the version back, do not resend
Duplicate storm from a caller	money, on SMS	unique-constraint violation rate as a metric	the constraint holds; nothing is sent twice	tell the caller their dedupeKey is unstable
Push tokens accumulating dead	wasted volume, meaningless metrics	ratio of sends to delivery callbacks drifting	feedback consumer sets invalid_at	backfill by processing stored callbacks
Callback endpoint down	statuses stop updating; sending is fine	callback ingest rate falling to zero	providers retry callbacks for hours	reprocess; statuses catch up

Alarm on the age of the oldest message, not on queue depth. Depth without age is unreadable: a queue holding two million messages that are all four seconds old is healthy, and a queue holding forty messages that are all nine minutes old is an incident. Age maps directly onto the promise you made (10.10). Give each queue its own threshold — 30 seconds on transactional, 30 minutes on bulk — because a single threshold across queues with different service levels is either constantly noisy or permanently blind.

What the interviewer will push on

"Why not just put a priority field on the queue?" This is the question the whole design exists to answer. The tell is that you explain why a priority field is insufficient rather than merely asserting it: priorities only decide who gets served when a worker is free, and during a campaign no worker is free, because every one of them is inside a 500 ms provider call. Then name all three separations — queue, pool, provider rate budget — and note that missing any one reproduces the incident.

"You said at-least-once. So users get duplicates?" They are checking whether you understand where deduplication belongs. Delivery is at-least-once because a provider timeout is genuinely ambiguous. Duplicate messages are prevented at the boundary by a unique constraint on the caller's deduplication key, which the handler treats as success when it fires. The tell is putting the constraint in the database rather than in application code, and explaining that the reason is that a constraint cannot be bypassed by a code path someone writes next year.

"A provider returns 400 invalid number. What do you do?" They want to see error classification. Do not retry — it will fail identically forever while consuming rate budget that working messages need, and on some providers it costs money per attempt. Record the reason on the delivery row, mark the contact detail invalid if the provider says so, and stop. Contrast it with a timeout, which is ambiguous and must be retried, and a 5xx, which is retryable with backoff and jitter.

"How do you know a notification was actually received?" You do not, and saying so clearly is the correct answer. You know it was accepted by a provider and, for some channels, that the provider reported delivery. Carriers drop SMS, spam filters quarantine email, devices are offline, permissions are revoked. The strong follow-up is that any product workflow requiring the user to have seen something must use the in-app channel, because it is the only one where you own both delivery and consumption.

"Your push delivery rate has been falling 1% a month for two years. Why?" Token rot. Installations disappear and the provider tells you asynchronously, so a system that does not consume feedback callbacks accumulates dead tokens forever while every send still looks successful. The fix is a feedback consumer and an invalid_at column, and the reason this question gets asked is that it separates people who have operated one of these from people who have drawn one.

"Queue depth is two million. Is that bad?" Unanswerable as asked, and saying so is the point. Two million messages that are four seconds old is a campaign running normally. Forty messages that are nine minutes old is an outage. Alarm on the age of the oldest message, per queue, with a threshold that matches that queue's promise.

Volunteer this, because nobody asks: record a suppression reason on every delivery that does not go out. "Why didn't the customer get the email?" is the most common question this system generates for the rest of its life, and without a recorded reason each instance is a two-hour investigation across preferences, quiet hours, frequency caps, token validity and provider logs. With it, the answer is one lookup. It costs one column and it is the single highest-value thing you can add to this design for operations.

Next: 11.7 — notifications are one message to one person with no expectation of a reply. Chat inverts every assumption: messages are ordered, conversations have two ends, presence has to be live, and the connection stays open for hours.

Recall

• Asynchronous by construction: 202 plus a status resource, because delivery depends on a third party. What 202 promises is durable acceptance and attempt, never delivery.
• Deduplication at the boundary: unique index on (user_id, dedupe_key), with the duplicate-key error treated as success. In the database, not in code, because a constraint cannot be bypassed by a future code path. On SMS this is money — roughly $2.5M a year of sends in the example, so a duplicate bug has an invoice.
• Priority isolation is the design. A priority field does not work, because priorities only matter when a worker is free and during a campaign none are. Separate queues, worker pools and provider rate budgets. Pace campaigns rather than enqueuing five million at once.
• Filter once, before the queue: dedupe, preferences, quiet hours in the user's IANA timezone (defer rather than drop; security categories override), frequency caps with atomic counters. Scheduled sends re-check at send time.
• Error classification: timeout is ambiguous ⇒ retry · 5xx ⇒ retry with jittered backoff · 4xx ⇒ never retry, it fails identically forever and costs budget. Plus breaker, per-category fallback, and a dead-letter queue carrying the payload.
• Push tokens rot. Consume the provider's feedback callbacks and set invalid_at, or a growing share of push volume goes to installations that no longer exist while every send looks successful. Same path carries bounces and unsubscribes.
• Templates are versioned, never edited in place, validated against the payload at intake, and rolled out in stages — a bad template reaches millions in minutes and cannot be recalled.
• Alarm on the age of the oldest message, per queue, never on depth. Campaigns chunk into ~1,000-message idempotent batches keyed campaignId:userId.

Self-test: Why is a priority field not isolation? Where does deduplication live and why there? What happens to a quiet-hours-suppressed security alert? Which provider errors must never be retried, and why? What silently kills push delivery rate over two years? Why is queue depth an unreadable metric?

Quiz Bank

FoundationalTrace a transactional notification from API call to delivered, naming every safeguard.

Intake. POST /notifications with {userId, templateId, data, dedupeKey, priority: transactional}. The service validates the payload against the template's required fields — catching a missing orderId here rather than rendering an empty gap to a million people — then inserts a notifications row. The unique index on (user_id, dedupe_key) means a retried call from the caller hits a duplicate-key error, which the handler treats as success and answers with the existing notificationId (10.4). It returns 202 with the identifier and a status URL (9.6.1).

Filter. Preferences are checked per channel and per category. Quiet hours are evaluated in the user's IANA timezone, so a transactional message inside the window is deferred to the window's end rather than dropped, and a security-category message overrides the window entirely. Frequency caps are consulted through an atomically-updated per-user counter. Every suppressed channel gets a row with a reason, because "why didn't I get it?" is the question this system will be asked forever.

Enqueue. Surviving channels get deliveries rows and are published to the transactional queue for that channel, written through the outbox (10.8.4) so the database row and the queue message cannot diverge — a crash between the two would otherwise leave a notification that exists and will never be sent, or a message with no record.

Worker. A worker from the transactional pool — physically separate from the bulk pool — renders the versioned, localised template, spends a token from the per-provider rate budget, and calls the provider through an adapter carrying a timeout, jittered retry on retryable errors only, and a circuit breaker (10.9). Success records the provider's message identifier. A non-retryable error records the reason and stops. Exhausted retries land in the dead-letter queue with the full payload.

Feedback. Hours or seconds later, the provider's asynchronous callback updates the delivery to delivered or bounced, marks an invalid push token with invalid_at, and writes any unsubscribe straight into preferences. This is the loop that keeps the system from silently rotting, and it is the part most designs omit.

InterviewA campaign to 5 million users is running and password-reset emails are arriving 20 minutes late. Diagnose and fix.

The diagnosis is head-of-line blocking, and there are usually two causes stacked on top of each other.

The queue. The campaign's five million messages entered the same queue as transactional traffic, so the password reset is behind them. A priority field does not save you here, and it is worth being precise about why: priorities decide who is served when a worker becomes free, and during a campaign no worker becomes free, because every one is inside a 200–800 ms provider call. The reset is at the front of the queue with nobody to hand it to.

The provider budget. The campaign has consumed the account's contracted send rate, so transactional messages are being throttled or rejected by the provider, and their retries are now also queueing behind the campaign. This second cause is easy to miss and it makes the first one much worse.

Immediate mitigation: pause the campaign. The transactional backlog drains within seconds. That campaigns must be pausable is a design requirement, and this incident is the argument for it.

The structural fix, in four parts.

Separate queues per priority class with dedicated worker pools and reserved capacity — the bulkhead (10.9) — so bulk volume is physically unable to occupy transactional capacity.

Separate provider rate budgets, ideally separate provider sub-accounts, so the campaign spends its own allocation and the transactional path always has its own.

Campaign pacing, spreading five million messages across the intended window rather than enqueuing them instantly. This also protects the provider relationship, since providers throttle accounts that spike, and it usually improves engagement anyway.

Alarm on the age of the oldest message per queue, with a much tighter threshold on transactional (30 seconds) than on bulk (30 minutes). One threshold across both queues is either constantly noisy or permanently blind, because the two have genuinely different promises.

The principle to state: sharing capacity between workloads with different urgency is a latency coupling, and the only reliable decoupling is physical separation — separate queues, separate pools, separate budgets. Anything softer than that is a scheduling hint, and scheduling hints do not survive saturation.

StaffDefine this system's guarantees. What do you promise callers, and what do you explicitly refuse to promise?

Promise: durable acceptance. Once 202 is returned, the notification is persisted and will be attempted according to policy. It will not be silently lost. This is the guarantee that lets calling services fire and forget, and it is backed by the outbox (10.8.4) so that acceptance and enqueueing cannot diverge.

Promise: effectively once, per deduplication key. With a stable key, retries and duplicate calls produce one message per channel — enforced by a unique constraint, not by hope and not by application logic.

Promise: an observable outcome. Every notification has a queryable per-channel status including suppression reasons, so "why didn't the user get it?" is one lookup rather than an investigation.

Promise: priority isolation. Transactional notifications meet their target regardless of campaign volume, because the isolation is physical.

Refuse to promise: delivery. The system delivers to a provider. Carriers drop messages, spam filters quarantine mail, devices are offline, users revoke permissions, tokens expire. The honest contract is "accepted, attempted, and the provider's reported outcome surfaced". Any product feature built on the assumption that a sent notification is a received notification is misdesigned, and the correct fix is an in-app confirmation rather than a better notification system.

Refuse to promise: ordering. Independent channels and parallel workers mean two notifications sent a second apart may arrive in either order (10.3). If order matters, the answer is one message with combined content, not two messages you hope arrive in sequence.

Refuse to promise: exactly-once. A provider timeout is genuinely ambiguous — the message may or may not have gone out — and the right resolution differs by case. Retry a password reset; do not retry a payment confirmation. The system surfaces the ambiguity in the status trail rather than pretending to have resolved it (10.4).

Refuse to promise: unbounded patience. Messages older than their category's maximum age are dropped rather than delivered, because a one-time code arriving after it expired, or "your ride is arriving" arriving an hour late, is worse than silence. This is a product decision encoded as a per-category expiry.

The point of the document is the refusals, not the promises. Every refusal is a constraint handed back to the caller, who is the only party that knows whether their use case can absorb it — and a caller who was never told "we cannot guarantee delivery" will build something that assumes it, in code you will be asked to fix during an incident.

Flashcards

FlashThe pipeline

Intake (202, unique dedupeKey) → filter (preferences, quiet hours in the user's timezone, frequency caps) → separate queues per priority class → workers with provider adapters → provider callbacks back in.

FlashWhy a priority field is not isolation

Priorities decide who is served when a worker is free; during a campaign none are free. Isolation must be physical: separate queues, separate pools, separate provider rate budgets. Missing any one reproduces the incident.

FlashError classification at the provider

Timeout = ambiguous ⇒ retry. 5xx ⇒ retry with jittered backoff. 4xx ⇒ never retry — it fails identically forever and burns rate budget or money. Plus breaker, per-category fallback, dead-letter queue with the payload.

FlashPush token rot

Installations vanish and the provider reports it asynchronously. No feedback consumer ⇒ a growing share of push volume goes nowhere while every send looks successful. Fix: consume callbacks, set invalid_at, filter on it at send time.

FlashWhat is never promised

Delivery (only acceptance, attempt, and the provider's reported outcome) · ordering · exactly-once · unbounded staleness (per-category maximum age, then drop).

FlashThe right queue alarm

Age of the oldest message, per queue, with per-queue thresholds. Depth is unreadable: two million four-second-old messages is healthy; forty nine-minute-old ones is an incident.

Scenario Drill

DrillAdd real-time in-app notifications — a bell icon with a live unread count — to this system. What is reused, what is added, and where does it get genuinely hard?

What is reused, unchanged. Intake, deduplication, preferences, frequency caps and templates all apply. In-app is simply a fifth channel with a row in deliveries. That so much carries over is a good sign about the original decomposition.

What has to be added, in three pieces.

A durable inbox per user. Unlike push or SMS, in-app notifications are read later, so they need storage with read and unread state: a table partitioned by user_id, indexed (user_id, created_at DESC), cursor-paginated (9.6.2), with a retention policy — because an inbox nobody ever prunes becomes a query nobody can run.

A live transport. Long-lived connections held by a gateway tier (11.7 builds this), with the notification worker publishing to a per-user channel that the gateway subscribes to. The gateway holds the connections and no business state; the workers hold business state and no connections. That split is what lets either one scale or restart without the other.

The unread count, which is where this stops being easy.

The three hard parts, in increasing order of subtlety.

The count is a hot read on every single page load. Computing COUNT(*) WHERE unread per request is a scan that gets worse as inboxes grow. The fix is a maintained counter per user, incremented on insert and decremented on read. The discipline that must come with it is the part people skip: counters drift. Concurrent updates, a failed decrement, a bulk "mark all read" that half-applied — all of them leave the counter wrong, and a wrong badge is the most visible bug a product can have. So the counter needs a periodic recomputation against the truth (10.4). This is why unread badges are wrong in so many products you use.

Read state is multi-device. Reading on a phone must clear the badge on the laptop, which means a read is itself an event published to the user's channel, and every device reconciles rather than trusting its local view. "Mark all read" must be a single idempotent operation carrying a watermark timestamp — everything before this instant is read — rather than N per-item updates, because a partially applied bulk read leaves a badge showing a number nobody can clear.

Reconnection is where correctness is won or lost. A device offline for an hour must not receive an hour of events replayed one at a time. On connect it fetches current state — the count and the first page — and only then subscribes to the live stream, using a sequence number so that events arriving between the fetch and the subscribe are neither lost nor duplicated. This snapshot-then-stream handshake appears again in 11.8 and 11.13, and it is the single detail that separates a live interface that stays correct from one where users learn to refresh before trusting it.

The architectural observation worth making. In-app is the only channel where the system owns both delivery and consumption. That makes it the only channel where "delivered" can honestly mean "seen", and therefore the only one where read receipts, engagement measurement, and revoking a notification — a resolved alert quietly disappearing — are even possible. That asymmetry is worth building the product around rather than flattening in the name of treating all channels the same.