Appearance
11.7 — Chat & Messaging
You type a message and see a single grey tick. A moment later it turns into two ticks, then the two ticks turn blue. Those three states are three different claims about three different systems, and a chat product lives or dies on whether each claim is true: the message reached a server and will survive a crash, it reached the other person's device, and a human looked at it.
Every study before this one was request and response. A request arrives, a response goes back, and between requests the server remembers nothing about you. Chat breaks that. Ten million people are connected right now, each holding a socket open for hours, and a message has to travel from one of those sockets to another in under two hundred milliseconds without ever arriving twice, out of order, or not at all.
This study introduces the connection gateway tier that 11.6, 11.8 and 11.13 all reuse, and the per-conversation sequence number that turns out to be the cheapest correct answer to ordering in a distributed system.
1. Requirements
Functional. One-to-one and group conversations up to 500 members. Message history. Delivery and read receipts. Presence — online, and typing. Offline delivery. Media attachments. The same account on several devices at once.
Non-functional, with numbers.
- Under 200 ms at p95 from one online user's send to the other's screen.
- Total ordering within a conversation. Across conversations, deliberately not — and section 6.1 explains why that is a feature.
- No message loss once acknowledged. A tick that can be retracted is worse than a slow tick.
- 10 million concurrent connections.
Out of scope today: voice and video calls, which use a completely different transport; message search; and end-to-end encryption — except that section 6.6 covers it anyway, because it changes the architecture so deeply that leaving it out entirely would be dishonest.
The clarifying questions, and what each answer changes
"What is the largest group?" Five hundred and a hundred thousand are different systems. At 500, you deliver to every member as the message arrives. At 100,000, that same approach turns one send into a hundred thousand deliveries and the model has to invert. Ask for the number and design for it rather than for "groups".
"Does the server keep message history, or only the devices?" This is the same question as "are we doing end-to-end encryption", asked in a way that gets a straight answer. Server-held history makes a new device trivial and search possible. Device-held history makes both hard and is the price of the server not being able to read anything.
"How many devices per account?" One device is much simpler than four. With four, delivery receipts become ambiguous, read state has to be shared, and push notifications must not fire on all of them at once.
"Is presence a product feature or a decoration?" If it is decoration, it can be approximate, cheap and lossy. If someone insists it must be accurate, the cost is enormous and it is worth surfacing the number before agreeing.
"What happens when a user has been offline for six months?" The honest answer bounds the sync protocol. Without a limit, one returning user can ask for a hundred thousand messages and the server has to decide whether to serve them.
2. Estimation
Concurrent connections. 50 million daily users with 20% connected at once = 10 million live connections. What that forces: a dedicated tier whose job is holding sockets, because holding a socket is a memory problem rather than a compute problem and it does not belong in the same process as business logic.
Gateway count. A well-tuned connection server holds 50,000–100,000 sockets. Memory dominates: socket buffers, encryption state and a small per-connection object come to roughly 10–50 KB each, so 50,000 connections is about 1.5 GB before any application data (3.8.2). Ten million connections therefore needs 100–200 gateway nodes. What that forces: the per-connection object must stay tiny. Caching a user's profile on their socket sounds harmless and is multiplied by fifty thousand, which is how gateways run out of memory.
Message volume. 50M users × 40 messages a day = 2 billion a day ÷ 86,400 ≈ 23,000 a second average, and roughly 70,000 a second at peak.
Delivery volume, which is the bigger number. A message to a one-to-one chat is one delivery; a message to a 500-member group is 500. With an average of about three recipients, 70,000 messages a second is 210,000 deliveries a second at peak. What that forces: fan-out is a separate concern from storage, and it is the part that scales with group size. One message is one stored row and N deliveries — worth saying explicitly, because candidates often assume a group message is stored per recipient.
Storage. 2 billion messages × ~300 bytes = 600 GB a day, about 220 TB a year. What that forces: no single database. Partitioning by conversation with time-based archival, and a real decision about how long history is kept (10.6).
The session registry's load, which people forget. Ten million connections each refreshing a registry entry every 30 seconds is 333,000 writes a second just to say "still here". What that forces: the heartbeat interval is a design parameter with a real cost. Doubling it to 60 seconds halves the load and doubles how long a dead connection looks alive. That trade should be made deliberately.
3. The transport is the API
Chat's contract is a set of frames over a persistent connection, not a set of HTTP endpoints. There are HTTP endpoints too, for history and for uploads, but the interesting design is in six frames.
→ { "t":"send", "convId":"c_44", "clientMsgId":"7f2e...", "body":"on my way" }
← { "t":"ack", "clientMsgId":"7f2e...", "msgId":"m_991", "seq":4103, "ts":"..." }
← { "t":"msg", "convId":"c_44", "msgId":"m_991", "seq":4103, "from":"u_12", "body":"on my way" }
→ { "t":"read", "convId":"c_44", "upToSeq":4103 }
← { "t":"receipt", "convId":"c_44", "userId":"u_12", "upToSeq":4103, "state":"read" }
→ { "t":"sync", "convId":"c_44", "sinceSeq":4090, "limit":200 }Two decisions live in those six lines and both are worth defending.
clientMsgId makes sending idempotent. The client generates it, sends it, and if the acknowledgement never arrives — the socket dropped, the phone changed networks — it retries with the same identifier. The server stores it under a unique constraint on (conv_id, client_msg_id), so a retry finds the existing row and returns the original msgId and seq instead of posting a second message (10.4). The client must persist this identifier across app restarts, or a restart mid-retry generates a new one and the mechanism is defeated — which produces the exact symptom "duplicates, but only after bad connectivity".
Receipts are watermarks, not events. readUpTo: 4103 means every message up to 4103 has been read. One frame replaces what would otherwise be hundreds of individual receipts when someone opens a conversation they have been ignoring. It is also idempotent and order-independent: applying readUpTo 4103 twice, or receiving 4103 before 4090, both converge to the right answer because the watermark only ever moves forward. That property pays off repeatedly, most visibly in the multi-device drill at the end of this page.
The HTTP side, for the things a socket is bad at:
http
GET /conversations/c_44/messages?beforeSeq=4103&limit=50
→ 200 { "messages": [...], "hasMore": true }
POST /uploads # returns a presigned URL — see 11.2
GET /conversations?cursor=… # the user's conversation list, from the inbox read model4. Data model
conversations
conv_id UUID PRIMARY KEY
type SMALLINT -- direct | group | channel
created_at TIMESTAMPTZ
last_seq BIGINT NOT NULL -- the allocator's counter for this conversation
members
conv_id UUID, user_id UUID
joined_at_seq BIGINT NOT NULL -- so a new member does not see old history
last_read_seq BIGINT NOT NULL
muted BOOLEAN
PRIMARY KEY (conv_id, user_id)
messages
conv_id UUID, seq BIGINT
msg_id UUID NOT NULL
client_msg_id UUID NOT NULL -- UNIQUE (conv_id, client_msg_id)
sender_id UUID NOT NULL
body TEXT
attachment_key TEXT NULL -- points into object storage, never the bytes
ts TIMESTAMPTZ NOT NULL
PRIMARY KEY (conv_id, seq)
inbox -- read model: the conversation list
user_id UUID, conv_id UUID
last_seq BIGINT, unread_count INT, last_activity TIMESTAMPTZ
PRIMARY KEY (user_id, conv_id)Access patterns:
| Query | Frequency | Returns |
|---|---|---|
| Append a message to a conversation | 70,000/s peak | one row |
| Read the last N messages of a conversation | high | 50 rows |
Read the gap since seq on reconnect | moderate | 0–200 rows |
| List a user's conversations by recency | on every app open | 20–50 rows |
| Look up members of a conversation for fan-out | 70,000/s peak | 1–500 rows |
Partition messages by conv_id. Every read is "the recent messages of this conversation", and every write appends to one conversation. Within a partition, rows are ordered by seq, so both the tail read and the gap read are contiguous ranges (10.6).
seq is a per-conversation counter allocated by a single writer — the shard that owns that conversation. That is the entire ordering mechanism, and section 6.1 argues why it is enough and why anything stronger is wasted.
inbox is a read model (10.8.4). Without it, "show me my conversations, newest first" is a query across every conversation the user belongs to, scattered over every partition. With it, that question is one partition read. The cost is that it must be updated on every message, which is a second write — and it can drift, so it needs the same periodic recomputation any denormalised counter needs.
members.joined_at_seq is a small field doing important work: it is how a member added to a group today does not see the last two years of messages, and it costs nothing because the history read is already bounded by a sequence range.
5. Architecture
The order of operations inside a send is the part that decides whether a tick ever lies.
6. Deep dives
6.1 Where ordering comes from, and why this is enough
Each conversation has a counter. A message gets the next value. That is the whole mechanism, and its strength is what it does not require: no synchronised clocks, no global sequencer, no consensus protocol. Just one writer per conversation, which is a shard, which you already have.
What it guarantees: total order within a conversation. Clients render by seq rather than by arrival time, and a client holding 4101 and 4103 knows a message is missing and asks for 4102 instead of displaying a hole.
What it deliberately does not guarantee: order across conversations. Two messages sent a millisecond apart in different conversations may arrive in either order. Guaranteeing otherwise would need a single global sequencer — one coordination point for the entire system — to solve a problem no user has ever had (10.3).
The client-side detail that makes this feel right. A message you just sent appears instantly with a "sending" state, positioned optimistically at the bottom. When the acknowledgement arrives with the authoritative seq, the client reconciles by clientMsgId and repositions it if necessary. The local view is instant, the truth is server-assigned, and the two converge visibly. Skip the reconciliation and your own messages sit in the wrong place forever — a bug only the sender can see, which is why it survives so long.
The cost, named: one writer per conversation means a very busy conversation is a hot partition. A hundred-thousand-member channel where everyone talks at once will saturate its allocator, and the answer at that size is section 6.3.
6.2 Connection management, and the failure that actually happens
Gateways hold a socket and nothing else. No message buffering, no business logic, no cached user state. The reason is failure behaviour: a gateway crash then costs its clients a reconnect and nothing more, because everything that matters lives behind it. The moment a gateway holds something that is not also stored elsewhere, losing one becomes a data question rather than an availability question.
The session registry maps user → {gateway, connectionId} with a time-to-live refreshed by heartbeat. It is the only shared state in the connection tier, and stale entries are self-healing: fan-out sends to the recorded gateway, that gateway replies "not connected", and the entry is cleared.
The failure that actually takes chat systems down is the reconnect storm. Anything that disconnects a large fraction of clients at once — a load balancer restart, a regional network blip, a rolling deploy done too fast — produces ten million clients all trying to reconnect in the same few seconds. That saturates the gateways, the registry and the authentication path simultaneously, and each failed attempt produces another attempt.
Three defences, and the first is not optional. Jittered exponential backoff on the client: wait 1 second, then 2, then 4, each multiplied by a random factor, so the retries spread out instead of arriving in synchronised waves. Connection rate limiting at the gateway, so a node refuses new connections above a rate it can authenticate. And staged admission during recovery, letting clients back in gradually rather than opening the doors at once.
Deploys are a self-inflicted version of the same event. Restarting a gateway drops 50,000 connections. The fix is draining — stop accepting new connections, let clients migrate over a window, then exit — plus deploying in small batches, because rolling 200 gateways in ten minutes is a reconnect storm you scheduled.
Load balancing must be connection-count aware. Round-robin is wrong for long-lived connections: a node that restarts starts empty and, under round-robin, receives new connections at the same rate as everyone else, so it stays empty for hours while every other node stays full. New connections must prefer under-loaded nodes.
6.3 Reconnect, and the handshake that has to be in this order
The client remembers the highest seq it holds for each conversation. On reconnect it subscribes, buffers whatever live frames arrive, sends sync {convId, sinceSeq}, receives the gap, merges everything by seq, and then starts applying live frames directly. Duplicates between the buffer and the gap are harmless, because a message with a seq you already have is simply ignored.
Do it in the other order and there is a hole. Fetch the gap first, then subscribe, and any message sent in between belongs to neither — it is after the gap and before the subscription. It is lost until the next reconnect, which may be hours later, and it will look to the user like a message that never arrived.
Bound the gap. A client offline for months may be tens of thousands of messages behind. sync is paginated, and past a threshold the honest answer is "you are too far behind, reload from scratch". That is a bounded and explainable behaviour; unbounded sync is a way for one returning user to become an incident.
6.4 Group fan-out, and where the model inverts
At 500 members, one message means one stored row and up to 500 deliveries. Doing that work at send time — fan-out on write — is right, because it makes delivery instant and 500 is a small number.
At 100,000 members it is not right. One person posting turns into a hundred thousand deliveries, most of them to people who are not looking, and a busy channel becomes a fan-out engine that does nothing else. Above roughly a thousand members the model should invert to fan-out on read: the message is stored once, and members' clients subscribe to the conversation's stream and pull what they have missed.
The valuable part of this answer is naming the threshold and the reason rather than picking a side. Fan-out on write buys delivery latency and pays in write amplification. Fan-out on read buys write cost and pays in read complexity and slightly slower delivery. Group size is what decides which price you would rather pay, and the same pivot appears in 11.8 for accounts with millions of followers.
6.5 Presence, which should be as cheap as you can make it
Presence is heartbeats with a time-to-live, aggregated and published at a reduced rate, and never stored durably.
The reason for that severity is arithmetic. Presence is chatter proportional to connections times contacts: every user's state change is potentially interesting to everyone who knows them, so naive per-event fan-out is a quadratic amount of traffic carrying almost no value per event. Nobody has ever needed to know that a contact went offline within one second rather than within thirty.
So: publish presence changes on a timer rather than per event, let the state be a few seconds stale, and accept that a user who loses connectivity appears online until their registry entry expires. Typing indicators are the same, only more so — fire and forget, with a client-side timeout, and no attempt at delivery guarantees. A typing indicator that arrives late is worse than one that never arrives.
Treating presence as important data — persisting it, guaranteeing it, fanning it out per event — has genuinely sunk chat systems, and it is worth saying that plainly because presence feels like it should be easy.
6.6 What end-to-end encryption changes, honestly
If the server stores only ciphertext it cannot read, several things you were planning stop being possible.
No server-side search, because there is nothing to index. Search moves to the device, over the history that device holds.
No server-side previews, no link unfurling, no spam filtering, no content moderation on message bodies. Each of those becomes a client-side feature or disappears.
Multi-device becomes genuinely hard, and this is the part people underestimate. Each device is a separate key holder, so a message to a recipient with three devices is encrypted three times, once per device. A new device cannot be handed history by the server, because the server holds ciphertext it cannot re-encrypt — so history must be transferred from an existing device over an authenticated channel, with a verification step the user has to actually perform.
Groups need a key protocol with rekeying whenever membership changes, so that someone removed from a group cannot read what comes after, and someone added cannot read what came before.
This is why end-to-end encryption is a product decision with deep architectural consequences and never a feature flag. If it is a requirement, it belongs in the first five minutes of the design, because it removes options everywhere.
6.7 Media, which is not chat's problem
Attachments do not travel through the chat pipeline. The client uploads to object storage using the presigned flow from 11.2, gets back a key, and sends a message whose body carries the key rather than the bytes. Recipients fetch through the same short-lived signed URLs.
Two details worth stating. The message must not be sent before the upload completes, or recipients see a message pointing at nothing — so the client sends after the upload's completion is confirmed, showing a local placeholder in the meantime. And the attachment's access control follows the conversation: the signed URL is issued only to members, and it expires, which is what stops a forwarded link from becoming permanent public access.
7. Decision Ledger
| Decision | Alternatives | Why this | What it costs |
|---|---|---|---|
| Persistent socket plus a gateway tier | long polling; server-sent events | bidirectional and low latency on one connection | a stateful tier; reconnect storms need real defences |
| Gateways hold sockets and nothing else | put logic in the gateway | losing one costs a reconnect, never correctness | one extra network hop on every message |
Per-conversation seq from one writer | global ordering; timestamps | ordering exactly where users perceive it, with no global coordination | a very busy conversation is a hot partition |
| Acknowledge after durable persistence | acknowledge on receipt | a tick never has to be retracted | a storage write sits inside the 200 ms budget |
| Receipts as watermarks | one receipt per message | one frame replaces hundreds; idempotent and order-independent | cannot express "read 7 but not 6", which nobody wants |
| Fan-out on write up to ~1,000 members | fan-out on read everywhere | instant delivery for the overwhelming majority of conversations | inverts above the threshold; two code paths to maintain |
| Presence approximate, never persisted | durable, guaranteed presence | avoids quadratic chatter for near-zero value per event | an occasionally stale "online" dot |
inbox as a read model | query across all conversations | the conversation list is one partition read | a second write per message, and drift needing recomputation |
8. Scale and failure
At 10× — 100 million connections — the gateway tier is just more nodes, because it holds no shared state. What breaks is the session registry at 3.3 million heartbeat writes a second, which needs sharding by user and a longer heartbeat interval traded against presence freshness. Conversation partitions also need splitting, and the hot-conversation problem gets worse rather than better.
At 100×, chat goes multi-region, and the interesting question becomes where a conversation's writer lives. Since ordering comes from a single writer per conversation, a conversation has a home region, and members elsewhere pay a cross-region round trip to send. The alternative — a writer per region with reconciliation — reintroduces exactly the ordering problem the design avoided, so the usual answer is to home a conversation near its most active members and accept the latency for the rest.
| What breaks | Blast radius | How you find out | What keeps it running | Recovery |
|---|---|---|---|---|
| One gateway crashes | 50,000 clients reconnect | connection count drops; reconnect rate spikes | durability lives behind the gateway; sync fills the gap | clients reconnect with jittered backoff |
| Reconnect storm | the whole connection tier, plus auth | connection attempt rate, orders of magnitude above normal | client jittered backoff, gateway rate limiting, staged admission | admit gradually; do not open all at once |
| Session registry down | fan-out cannot find online members | registry error rate | treat everyone as offline — push and sync on reconnect | restore; entries rebuild from heartbeats |
| Conversation shard saturated | one busy conversation slows | per-conversation write latency | batch sequence allocation; invert to fan-out on read | split the shard, or move that conversation's model |
| Message store slow | acknowledgements delayed; ticks lag | send-to-ack latency p95 | back-pressure at the gateway rather than unbounded queueing | do not acknowledge early to hide it |
| Push provider down | offline users get nothing until they open the app | the 11.6 pipeline's alarms | messages are already durable; nothing is lost | offline users sync on reconnect |
| Inbox read model drifts | wrong unread counts | periodic reconciliation job's divergence count | the message store remains the truth | recompute from messages |
The registry row is the one to explain. When you cannot tell who is online, the safe failure is to assume everyone is offline — store the message, send a push, and let clients sync when they reconnect. That is degraded and correct. The opposite failure, assuming everyone is online and dropping undeliverable messages, loses data, and it is the kind of thing that gets written into a fallback path by someone optimising for the happy case.
What the interviewer will push on
"Why acknowledge after persisting rather than on receipt? You'd save 40 milliseconds." They are testing whether you will trade a guarantee for a number. Acknowledging on receipt means a tick can appear for a message that is then lost in a crash — and taking a tick away is a specific and memorable kind of broken. Name the cost honestly (a storage write inside the latency budget), then note that the budget was set with it in mind.
"Where does ordering come from, and why don't you need consensus?" The answer is a per-conversation counter from a single writer. The tell is explaining what you are not buying: no global order, no synchronised clocks, no agreement protocol, because the only ordering a user can perceive is within a conversation. Candidates who reach for a consensus algorithm here are solving a problem the product does not have, at a cost the product cannot afford.
"Ten million clients reconnect at once. What happens?" This is the real outage in chat systems, so they want to see that you know it. Walk it: gateways, registry and authentication saturate simultaneously; every failed attempt produces another attempt; the load is self-sustaining. Then the three defences — jittered client backoff, gateway connection rate limiting, staged admission — and the observation that a fast rolling deploy is the same event, self-inflicted, which is why draining and small batches matter.
"Users report duplicates, but only after poor connectivity." They want root-cause reasoning. The pattern says an acknowledgement was lost and the retry created a second message, so either the deduplication key is missing server-side, or — the subtler and more common case — the client regenerates its clientMsgId after a restart, defeating the mechanism. The diagnostic that separates the two is whether the duplicate pair shares a client_msg_id. If it does, the server is at fault; if not, the client is.
"A group has 100,000 members. Does your design still work?" No, and saying so is the answer. Fan-out on write turns one send into a hundred thousand deliveries. Name the threshold at around a thousand, describe the inversion to fan-out on read, and state what each side costs — delivery latency versus write amplification. The weak answer is to insist the design scales.
"How accurate is your presence?" They are checking whether you will over-engineer something worthless. Presence is heartbeats with a time-to-live, published on a timer, a few seconds stale, and never persisted. The justification is the arithmetic: presence traffic scales with connections times contacts, and the value per event is near zero. A candidate who proposes durable, guaranteed presence has just added the most expensive component in the system to support a green dot.
Volunteer this, because nobody asks: ordering and deduplication are end-to-end properties, not server properties. The server provides seq and honours clientMsgId, but the guarantee is only real if the client renders by seq rather than by arrival, holds a message with seq n+2 until n+1 arrives, reconciles its optimistic local echo, and persists its clientMsgId across restarts. That makes the client a first-class part of this system's correctness rather than a consumer of it — and it explains why these bugs are invisible in server metrics, where everything looks perfect.
Next: 11.8 — chat delivers to a handful of people who are waiting. A feed delivers to millions who are not, from authors who have millions of followers, and the whole design turns on whether you do the work when someone writes or when someone reads.
Recall
- Two tiers: gateways hold sockets and nothing else, so losing one costs a reconnect and no correctness; the message service owns sequencing and durability. The session registry (
user → gateway, TTL, heartbeat) joins them, at 333,000 writes a second for 10M connections. - Ordering = a monotonic per-conversation
seqfrom a single writer. No global clock, no consensus, no cross-conversation order — deliberately (10.3). - Send path: assign seq → persist → acknowledge → fan out → push the offline. Never acknowledge before durability; a tick must never be retracted.
clientMsgIdmakes sending idempotent, enforced by a unique index — and the client must persist it across restarts, or the mechanism is defeated (which is exactly the "duplicates only after bad connectivity" signature).- Receipts are watermarks (
readUpTo), so one frame replaces hundreds and is idempotent and order-independent. - Reconnect = subscribe first, buffer, then fetch the gap, then merge by
seq. The reverse order loses everything sent during the fetch. Bound the gap and be willing to say "reload from scratch". - Reconnect storms are the real outage. Jittered client backoff, gateway connection rate limiting, staged admission — and drain gateways on deploy, in small batches.
- Fan-out on write up to ~1,000 members, then invert to fan-out on read. Presence is approximate, timer-published and never persisted, because it is chatter proportional to connections × contacts with near-zero value per event.
- Registry down ⇒ treat everyone as offline (store, push, sync later). Degraded and correct beats confident and lossy.
Self-test: Why does the acknowledgement come last? Where does ordering come from and what does it deliberately not cover? Give the reconnect handshake in order and say what breaks if you swap the first two steps. Why is a clientMsgId that is regenerated on restart useless? At what group size does fan-out invert, and what does each side cost?
Quiz Bank
FoundationalHow is message ordering guaranteed, and what ordering is deliberately not guaranteed?
Guaranteed: a total order within each conversation. Every conversation has a monotonically increasing seq assigned by a single writer — the shard that owns that conversation — so every message has one unambiguous position. Clients render by seq rather than by arrival time, and a client holding 4101 and 4103 knows that 4102 exists and can ask for it instead of displaying a gap.
What makes this the right mechanism is what it does not need: no synchronised clocks, no global sequencer, no agreement protocol. Just serialisation within a partition, which the storage layer already gives you (10.3). It is the cheapest correct answer available.
Not guaranteed: order across conversations. Two messages sent a millisecond apart in different conversations may be delivered and displayed in either order, and that is correct rather than a compromise. Guaranteeing cross-conversation order would require one global sequencer for the entire system — a coordination point every message must pass through — in order to solve a problem no user has ever reported.
Also not guaranteed: the order of side effects. A push notification for conversation A may arrive before one for conversation B regardless of when the messages were sent, because they travel through independent pipelines with independent retry behaviour.
The client-side detail that makes ordering feel right. A message the user just sent is rendered immediately in a "sending" state, positioned optimistically at the bottom. When the acknowledgement returns with the authoritative seq, the client reconciles by clientMsgId and moves it if it needs to move. The local view is instant, the truth is server-assigned, and the two converge visibly rather than silently. Skipping the reconciliation produces a bug that only the sender can see, which is why it survives so long in real products.
InterviewTen million WebSocket connections — how do you hold them, and what breaks first?
Holding them. Roughly 100–200 gateway nodes at 50,000–100,000 connections each. The binding resource is memory per connection: socket buffers, encryption state and a small per-connection object come to about 10–50 KB, so 50,000 connections is 1.5–2.5 GB before any application data (3.8.2). After memory come file descriptors, whose per-process limit is nowhere near sufficient by default, and event-loop wakeups. The practical discipline is to keep the per-connection object tiny — caching a user's profile on their socket looks harmless and is multiplied by fifty thousand.
Routing. The session registry maps user to gateway, refreshed by heartbeat with a time-to-live so dead entries expire on their own.
Load balancing must be connection-count aware, not round-robin. With long-lived connections, a node that restarts starts empty; under round-robin it receives new connections at the same rate as everyone else, so it stays nearly empty for hours while the others stay full. New connections have to prefer under-loaded nodes.
What breaks first, in practice.
The reconnect storm. Any event that disconnects a large fraction of clients — a load balancer restart, a regional network blip, a deploy rolled too quickly — produces a simultaneous reconnect attempt from millions of clients, saturating gateways, registry and authentication together, with each failure producing another attempt. The defences are jittered exponential backoff on the client (not optional), connection rate limiting at the gateway, and staged admission during recovery (10.9).
The registry as a hotspot. Ten million connections heartbeating every 30 seconds is 333,000 writes a second purely to say "still here". That needs sharding, and the heartbeat interval becomes a deliberate trade: doubling it halves the load and doubles how long a dead connection appears alive.
Deploys. Restarting a gateway drops 50,000 connections. The fix is draining — stop accepting new connections, let clients migrate over a window, then exit — plus small batches, because rolling 200 gateways in ten minutes is a reconnect storm you scheduled for yourself.
StaffUsers report messages appearing out of order and duplicate messages after poor connectivity. Root-cause both without seeing the code.
Duplicates: an acknowledgement that did not survive the network. The client sends, the server persists and acknowledges, the acknowledgement is lost, the client retries. If the server treats the retry as a new message, the conversation now has two.
Root cause candidates, and how to tell them apart. Either clientMsgId is missing or not enforced server-side, or — the subtler and more common case — the client regenerates it after an app restart, so the retry carries a different identifier and no deduplication is possible. The diagnostic is one query: do the duplicate pairs share a client_msg_id? If they do, the server is not enforcing the constraint. If they do not, the client is not persisting the identifier across restarts, which is exactly why the symptom appears only after poor connectivity — that is when a restart lands in the middle of a retry.
The fix. A client-generated identifier on every send, stored under a unique index on (conv_id, client_msg_id), with the handler catching the constraint violation and returning the existing msgId and seq, turning the retry into a re-acknowledgement (10.4). Plus persisting the identifier on the client before the first attempt, not after.
Out of order: the client is rendering by arrival rather than by seq. Under poor connectivity, frames arrive late, retries interleave, and the reconnect sync delivers a batch overlapping with live frames already received. Anything sorting by receipt time or by a local timestamp will visibly scramble.
Three candidates, in order of likelihood. The client renders on arrival instead of inserting by seq — the common bug, fixed with an ordered buffer that holds a message with seq n+2 until n+1 arrives or a short timeout fires, then requests the gap. The sync-then-subscribe race — the client fetched the gap and then subscribed, so anything sent in between belongs to neither and is lost; the fix is subscribe-first, buffer, fetch, merge. And an optimistic local echo that is never repositioned when the authoritative seq arrives, so the sender's own message sits in the wrong place for the sender and nowhere else.
The systemic response, beyond the two fixes. Add a client-side gap detector that logs whenever a received seq is not one more than the last, together with the reconnect context, and report it. Both classes of bug are invisible in server metrics — the server did everything correctly — and are only observable where the ordering is actually rendered.
The principle: ordering and deduplication are end-to-end properties. The server supplies the tools, seq and clientMsgId, but the guarantee is only real if the client uses them correctly. That makes the client a first-class part of this system's correctness rather than a consumer of it, and it is worth saying explicitly in the design document, because a mobile team that was never told this will reasonably assume the server handles it.
Flashcards
FlashThe two tiers
Gateways = sockets and nothing else, so a crash costs a reconnect. Message service = sequence assignment and durability. Session registry (user → gateway, TTL) joins them, at ~333k heartbeat writes/s for 10M connections.
FlashOrdering mechanism
A monotonic per-conversation seq from a single writer. Total order within a conversation, deliberately none across conversations. No clocks, no consensus, no global sequencer.
FlashSend path order
Assign seq → persist durably → acknowledge sender → fan out to online members → push the offline ones. Never acknowledge before durability; a tick must never be retracted.
FlashReconnect handshake
Subscribe first (buffering), then fetch the gap since your last seq, then merge by seq, then go live. Swap the first two and everything sent during the fetch is lost. Backoff with jitter is mandatory.
FlashReceipts and presence
Receipts are watermarks (readUpTo) — one frame replaces hundreds, idempotent and order-independent. Presence is heartbeat plus TTL, published on a timer, approximate, and never persisted.
FlashThe fan-out threshold
Fan-out on write up to ~1,000 members (delivery latency, paid for in write amplification). Above that, invert to fan-out on read (cheap writes, paid for in read complexity). Name the threshold, not a side.
Scenario Drill
DrillProduct adds multi-device support: phone, laptop and tablet must all show every message, with read state synced across them. What breaks, and what does the data model need?
What breaks immediately.
The session registry's shape. user → gateway becomes user → set of (device, gateway), so fan-out targets devices rather than users. That change touches every delivery path in the system.
Delivery receipts become ambiguous. "Delivered" to which device? The honest answer shown to the sender is delivered to at least one device, with per-device state tracked internally for debugging. Anything more precise is information the sender does not want and cannot act on.
Read state has to move. Reading on the phone must clear the badge on the laptop, so read watermarks are per-user, not per-device, and a read on any device is published as an event to all of that user's devices. This is where the watermark decision from section 3 pays off: a watermark only moves forward, so duplicate or out-of-order read events converge to the right answer with no coordination at all.
What the data model needs.
devices
user_id, device_id PRIMARY KEY
push_token, platform, last_seen
last_synced_seq -- per conversation, per deviceThe per-device sync position is the crux: a tablet offline for a month needs a different gap than a phone offline for a minute. But members.last_read_seq stays per-user, because read state is a property of the human rather than the hardware. Getting that split wrong in either direction produces one of two very visible bugs — badges that will not clear, or history that will not arrive.
Fan-out consults devices. Online devices get the frame. Offline ones get a push, deduplicated so that three devices do not produce three notification buzzes for one message — typically by suppressing push to devices that were recently active elsewhere. That is a heuristic, and it should be described as one rather than presented as a rule, because it will occasionally be wrong in both directions.
The genuinely hard part is a new device joining. It has no history. "Download two years of messages" is neither fast nor always wanted, so the design needs a history policy, and which policy is available depends entirely on a decision made much earlier.
With server-held history, this is easy: the device syncs from seq 0 with pagination, which the model already supports, and you choose how far back to go as a product decision.
With end-to-end encryption, it is hard, because the server holds ciphertext it cannot re-encrypt for a new key. History must be transferred from an existing device over an authenticated channel, which means the new device is useless until an old one is online and the user completes a verification step. This is the point where section 6.6's warning becomes concrete: end-to-end encryption and multi-device are in direct tension, resolved in real products by per-device key envelopes — a message to a three-device recipient is encrypted three times — plus a device verification flow users must actually perform. That combination is why multi-device encrypted messaging took the industry years rather than a quarter.
Second-order effects worth naming before the interviewer does. Notification suppression across devices, so one message does not buzz three things at once. Presence becomes a union over the user's devices, with the freshest heartbeat winning. And typing indicators must be device-scoped internally — two of your devices should not fight over the indicator — while remaining user-scoped in the interface, because the other person does not care which of your machines you are typing on.
The sentence for the design document: multi-device turns every per-user assumption into a per-device one, except read state, which must stay per-user — and getting that one split wrong in either direction produces either badges that never clear or history that never arrives.