10.16 — Real-Time Delivery: Polling, SSE, WebSockets and Push

A user sends a chat message. It has to appear on their friend's screen in under 200 ms.

HTTP does not do this. The client asks and the server answers, and there is no way for the server to speak first. Every technique on this page is a way around that one limitation, and they differ in how much they cost you and how much they give you.

1. The five options

Short polling

The client asks "anything new?" every few seconds.

setInterval(async () => {
  const messages = await fetch(`/messages?since=${lastSeen}`).then(r => r.json());
  if (messages.length) render(messages);
}, 3000);

Trivial to build, works everywhere, needs nothing special from any part of your infrastructure. And it is wasteful in a way that scales badly: with a hundred thousand users polling every three seconds, you are serving 33,000 requests per second, and if messages are rare then almost every one of those requests returns nothing. You are paying full price for an empty answer, and average latency is half the polling interval.

Short polling is right for genuinely infrequent, latency-tolerant updates — a build status, a slow-running report — and for prototypes.

Long polling

The client asks, and the server holds the request open until there is something to say, or until a timeout. The client immediately asks again.

async function poll(): Promise<void> {
  const res = await fetch(`/messages/wait?since=${lastSeen}`);   // (1) may hang for 30 s
  if (res.status === 200) render(await res.json());              // (2) something arrived
  poll();                                                         // (3) immediately re-ask
}

(1) The server does not answer until it has news. (2) A 204 means the timeout fired with nothing to report. (3) The client reconnects instantly, so there is almost always an open request waiting.

Latency drops to near zero because the connection is already there when the message arrives. The cost is that every waiting client holds an open connection and a server-side request context, which for a traditional thread-per-request server means one thread parked per waiting user — the C10k problem (2.7). On an event-driven runtime this is cheap, which is why long polling and Node grew up together.

Long polling still works everywhere HTTP works, which is its enduring advantage: no proxy, firewall or corporate network has ever blocked it.

Server-sent events

One HTTP connection, held open, over which the server streams messages whenever it likes.

const stream = new EventSource("/events");            // (1) browser API, one line
stream.onmessage = e => render(JSON.parse(e.data));   // (2) fires on each server message
stream.onerror = () => { /* the browser reconnects on its own */ };  // (3)

(1) Ordinary HTTP with a text/event-stream content type. (2) The server writes data: {...}\n\n and the browser fires an event. (3) Reconnection is built in — the browser retries automatically and sends a Last-Event-ID header so the server can resume from where it stopped, which is a whole class of code you do not write.

The limitation is that it is one-directional. The server can push; the client cannot send anything up the same connection and must use a normal request. For notifications, live feeds, dashboards and progress updates — where traffic is almost entirely downward — that is not a limitation at all.

SSE is the most underrated option here. It solves the common case with less machinery than WebSockets, works through ordinary HTTP infrastructure, and gives you reconnection for free.

WebSockets

A real two-way connection. The client makes an HTTP request with Upgrade: websocket, the server agrees, and from that point the connection is no longer HTTP — both sides send messages whenever they want.

const socket = new WebSocket("wss://example.com/chat");
socket.onmessage = e => render(JSON.parse(e.data));      // server → client
socket.send(JSON.stringify({ type: "typing" }));          // client → server, same connection

Lowest latency, lowest per-message overhead, and genuinely bidirectional. Necessary for chat with typing indicators, multiplayer games, collaborative editing, and trading interfaces.

The costs are real and you should name them. Reconnection, heartbeats and message replay are yours to build, because nothing is automatic. Some corporate proxies interfere with the upgrade. And every connection is pinned to one server for its lifetime, which section 4 is about.

Push notifications

The message goes to the operating system's push service, which delivers it to the device even when your application is closed.

This is the only option that works when the user is not looking at your page, which makes it a different tool rather than a faster one. It is out of your control — delivery is best effort, timing is not guaranteed, and the payload is small — so use it to tell the user something happened, and let the application fetch the details when opened.

And for server-to-server: webhooks

When the other end is another company's server rather than a browser, the equivalent is a webhook: you make an HTTP request to a URL they registered. Same idea, opposite direction, and it comes with its own delivery guarantees to design (9.5.4 builds the ingestion side).

2. Choosing

Figure 1 — The same minute, four ways. Arrows pointing down are server-to-client, arrows pointing up are client-to-server. Only the bottom row has both.

Need	Use	Why
Updates every few minutes	Short polling	Simplest thing that works
Server-to-client, near-instant	SSE	Free reconnection and replay, plain HTTP
Both directions, low latency	WebSocket	Chat, games, collaboration
Reach a closed app	Push notification	The only option that works
Must work through anything	Long polling	No infrastructure has ever blocked it

The default worth stating: if traffic is mostly downward, use SSE. Most teams reach for WebSockets by reflex and then write reconnection logic, replay logic and heartbeat logic that SSE would have given them. Reach for WebSockets when the client genuinely needs to send frequently on the same connection.

3. The arithmetic of holding connections open

This is where real-time systems become a capacity problem rather than a protocol question.

A million concurrent connections. Each one costs a socket, kernel buffers, and whatever your runtime keeps per connection — realistically 10 to 50 KB once buffers are included. A million connections is therefore 10 to 50 GB of memory before a single message moves.

That immediately tells you the shape of the system: a fleet whose size is driven by connection count, not by request rate. Fifty thousand connections per server means twenty servers just to hold them, even if nobody sends anything. That is a very different sizing conversation from a stateless API, and saying it unprompted is what an interviewer is listening for.

Three things follow.

Heartbeats cost more than you expect. A ping every thirty seconds across a million connections is 33,000 messages per second of pure overhead. Longer intervals cost less and detect death more slowly. Pick deliberately.

One port allows 65,535 connections per client address, so load generators need multiple addresses and your outbound connections to backends have the same limit.

File descriptor limits will bite you first. The default limit on most systems is far below what you need, and the failure is a confusing "too many open files" rather than anything mentioning connections.

4. The routing problem, which is the real design work

Your server pushes a message to a user. Which server holds that user's connection?

With one server, no problem. With twenty servers and a million users spread across them, sending a message means knowing which of the twenty holds this user right now — and it changes on every reconnect.

Figure 2 — Publish and let the holder answer. Separating the connection-holding gateways from the business logic is the structural decision; it lets you scale connections and computation independently.

Approach one: a connection registry. Each gateway records "I hold user 4471" in a shared store, and the sender looks it up and forwards directly to that gateway. Precise and efficient. The costs are that the registry is written on every connect and disconnect — which at scale is a lot of writes — and that it can be wrong, because a crashed gateway leaves stale entries pointing at a server that no longer holds anybody.

Approach two: publish and subscribe. The sender publishes to a topic named after the recipient. Every gateway subscribes to topics for the users it currently holds. The gateway holding Bob receives it and delivers; the others never see it. No registry, no stale state, and reconnection to a different gateway is automatically correct because the subscription moves with the connection.

Approach two is the better default for exactly that reason: there is no separate piece of state to go stale. The cost is a message bus in the path, and its fan-out cost for a message going to many recipients.

And the structural point worth making, because it is what an interviewer means by "how would you scale this": separate the gateways that hold connections from the services that do work. Gateways are dumb, memory-heavy and scale with users. Services are stateless, CPU-heavy and scale with activity. Combining them means adding CPU whenever you gain idle users, which is the wrong shape.

5. Reconnection, and not losing messages

Connections drop. Phones change network, laptops sleep, balancers restart. Every message sent during a gap is lost unless you design otherwise, and a chat that loses messages when someone walks into a lift is not a chat.

The fix is a cursor. Every message gets a monotonically increasing id per stream, the client remembers the last one it processed, and on reconnect it sends that id and receives everything since.

SSE builds this in: the server sets an id: on each event, and the browser sends Last-Event-ID on reconnect without you writing a line. For WebSockets you build it yourself, which is one of the reasons to prefer SSE when you can.

Three consequences to state.

Delivery is at-least-once, so the client must deduplicate. A message may be sent, the acknowledgement lost, and the message sent again on reconnect. Keeping the last id and ignoring anything at or below it is enough (10.4).

Replay needs a bounded history. You cannot store every message forever for replay. Keep a window — the last few thousand per stream, or the last hour — and if a client reconnects from further back than that, tell it to resynchronise from scratch rather than pretending you can fill the gap.

Reconnect with backoff and jitter. When a gateway holding fifty thousand connections restarts, all fifty thousand clients reconnect at once. Without jitter they arrive together, overwhelm the remaining gateways, and cause the next restart (9.5.3). This is the single most common cause of a real-time system failing to recover from a partial failure.

6. Detecting death, and slow clients

A TCP connection can be dead for a long time without either side noticing. A phone that loses signal sends no close, so the server holds a socket to nobody until the operating system's own timeout, which can be hours.

So you send heartbeats: a small ping every twenty or thirty seconds, and a connection with no response after two missed intervals is closed. The interval is a trade — shorter means faster detection and more overhead, and section 3 showed the overhead is not trivial at scale.

Heartbeats have a second job: keeping intermediaries from closing an idle connection. Load balancers and corporate proxies commonly drop connections idle for sixty seconds, and a heartbeat is what stops a working connection being killed for being quiet. This is why an SSE stream that works locally dies after a minute in production, which is a genuinely common and confusing bug.

Slow clients need backpressure. If you send faster than a client can receive, the messages queue in the server's socket buffer and then in your process memory. One slow client is nothing; ten thousand of them on a phone with poor signal is an out-of-memory crash. Every real-time server needs a per-connection outbound bound and a policy for exceeding it — drop the oldest, drop the newest, or disconnect the client and let it reconnect and replay. Disconnecting is usually right, because the cursor mechanism from section 5 means the client recovers cleanly, and the alternative is one bad client damaging everybody.

7. What breaks in the infrastructure

Idle timeouts. Balancers close connections idle for a default period, often sixty seconds. Configure it above your heartbeat interval, in every layer, including ones you did not know were in the path.

Buffering proxies. A proxy that buffers responses will hold your SSE events instead of streaming them, so messages arrive in bursts or not at all. Streaming has to be enabled explicitly in most proxies.

Connections pin users to servers. A long-lived connection is bound to one server for its life, so a deploy disconnects everybody on that server. Roll one gateway at a time, and rely on jittered client reconnection to spread the returning load.

The upgrade must be allowed through. WebSockets need the Upgrade header to survive every hop. Some corporate proxies strip it, which is the main reason a fallback to long polling still exists in real products.

8. What the interviewer will push on

"WebSockets or SSE?" They are checking whether you reach for the heavier tool by reflex. If traffic is mostly downward — notifications, feeds, dashboards, progress — SSE gives you automatic reconnection and replay through Last-Event-ID, works over ordinary HTTP, and needs no special infrastructure. Choose WebSockets when the client genuinely sends frequently on the same connection: chat with typing indicators, games, collaborative editing.

"A million concurrent connections. What does your fleet look like?" They want the arithmetic. 10 to 50 KB per connection means 10 to 50 GB of memory just to hold them, so the fleet is sized by connection count rather than request rate. Then the structural point: split connection-holding gateways from stateless services, because idle users should cost memory and not CPU.

"How does a message reach the right user?" The routing problem. Either a registry mapping user to gateway, which is precise and can go stale when a gateway crashes, or publish-and-subscribe where each gateway subscribes for the users it holds, which has no state to go stale and is the better default.

"The user goes into a tunnel for thirty seconds. What happens to their messages?" Cursors. Every message has an increasing id, the client sends its last one on reconnect, the server replays from there. Then the three follow-ons: delivery becomes at-least-once so the client deduplicates, the replay window is bounded so a very old client resynchronises instead, and reconnection uses jittered backoff.

"You restart a gateway holding fifty thousand connections." All fifty thousand reconnect at once. Without jitter they arrive simultaneously and overwhelm the remaining gateways, which is how a rolling restart becomes an outage. Roll one at a time, jitter the client backoff, and make sure the remaining capacity can hold the displaced connections.

"Your SSE stream works locally and dies after sixty seconds in production." A favourite because it is so specific. An idle timeout somewhere in the path — balancer, proxy, or gateway — is closing a connection that has been quiet. The fix is heartbeats below the timeout, and configuring the timeout in every layer.

The thing to volunteer that nobody asks for: the slow-client problem. Sending faster than a client can receive queues messages in your process memory, and ten thousand slow phones is an out-of-memory crash. State the per-connection outbound bound and the policy — disconnect and let the cursor mechanism replay — because it shows you have run one of these rather than only designed one.

Next: 10.17 steps back from mechanisms to shapes — monolith, microservices, serverless, event-driven and peer-to-peer, and what each one actually costs.

Recall

• Short polling: simplest, mostly empty responses, latency is half the interval. Long polling: server holds the request until there is news; works through anything. SSE: one HTTP connection, server-to-client only, with automatic reconnection and Last-Event-ID replay. WebSocket: two-way, lowest latency, everything is yours to build. Push notification: the only one that reaches a closed app.
• Default to SSE when traffic is mostly downward. Most teams pick WebSockets by reflex and then rebuild what SSE gave them free.
• Held connections cost 10–50 KB each, so a million is 10–50 GB before any message moves. The fleet is sized by connection count, not request rate.
• Split connection-holding gateways from stateless services, or idle users cost you CPU.
• Routing: a registry (precise, goes stale when a gateway crashes) or publish-subscribe with a topic per user (no state to go stale — the better default).
• Cursors are how nothing is lost: increasing message ids, client sends its last on reconnect, server replays. Delivery is at-least-once, so the client deduplicates; the replay window is bounded, so a very old client resynchronises.
• Reconnect with jittered backoff, or a gateway restart makes fifty thousand clients return simultaneously.
• Heartbeats detect dead connections and stop intermediaries closing idle ones — the reason an SSE stream dies after sixty seconds in production.
• Bound the outbound queue per connection. Ten thousand slow clients is an out-of-memory crash; disconnecting is usually right because the cursor lets them recover.

Self-test: What does SSE give you that WebSockets do not? How many servers hold a million connections, and why is that the wrong question to ask about request rate? Name the two ways to find which gateway holds a user, and the failure of each. Why must the replay window be bounded? Why does a working stream die after sixty seconds in production?

Quiz Bank

FoundationalCompare short polling, long polling, SSE and WebSockets, and give the case where each is the right choice.

Short polling is the client asking on a timer. Its virtues are that it is trivial and works through every piece of infrastructure ever built. Its costs are that most responses are empty — a hundred thousand users polling every three seconds is 33,000 requests per second, nearly all returning nothing — and that average latency is half the interval. Right for updates that are genuinely infrequent and latency-tolerant: a build status, a long-running report, a dashboard refreshed every minute.

Long polling is the client asking and the server holding the request open until it has news, then the client immediately asking again. Latency drops to near zero because the connection is already in place when the event happens. The cost is one held connection and one server-side request context per waiting client, which is cheap on an event-driven runtime and expensive on a thread-per-request one. Right when you need low latency and must work through hostile infrastructure, because no proxy or firewall has ever blocked ordinary HTTP.

Server-sent events is one held HTTP connection over which the server streams messages whenever it likes. It is one-directional, which is the whole trade. In exchange you get the browser's built-in reconnection and, through the Last-Event-ID header, built-in replay from where the client stopped. Right for notifications, live feeds, progress updates, dashboards — anything where traffic is almost entirely downward.

WebSockets is a genuine two-way connection after an HTTP upgrade. Lowest latency, lowest per-message overhead, and the client can send on the same connection. The cost is that reconnection, replay, heartbeats and backpressure are all yours to build, and some corporate proxies interfere with the upgrade. Right for chat with typing indicators, multiplayer games, collaborative editing, trading screens.

The recommendation worth stating, because it is the point of the question: if traffic is mostly downward, use SSE. Teams reach for WebSockets by reflex and then spend a fortnight writing reconnection, replay and heartbeat logic that SSE provides for free. Choose WebSockets when the client genuinely needs to send frequently on the same connection — and when it does, they are clearly the right tool and nothing else comes close.

AppliedDesign the delivery layer for a chat application with a million concurrent users. Cover the fleet, the routing, and what happens when a gateway restarts.

Start with the arithmetic, because it determines the shape. A million held connections at 10 to 50 KB each is 10 to 50 GB of memory purely to hold them, before any message moves. At a comfortable fifty thousand connections per server, that is twenty gateway servers, and that number is driven by connection count rather than request rate — a million idle users cost the same memory as a million chatty ones.

So split the fleet in two, and this is the structural decision. Gateways hold connections and do nothing else: they are memory-heavy, they scale with user count, and they contain no business logic. Behind them, stateless chat services store messages, apply permissions and do the work; they are CPU-heavy and scale with activity. Combining the two would mean adding CPU every time you gain idle users, which is the wrong shape and gets expensive quickly.

Transport: WebSockets, because chat is genuinely two-way — messages, typing indicators, read receipts, presence all flow upward. This is the case where the extra machinery is justified.

Routing, and the choice worth defending. When Ana sends to Bob, something must find Bob's gateway. A registry mapping user to gateway is precise, and it writes on every connect and disconnect — a lot of writes at this scale — and it goes stale when a gateway crashes, leaving entries pointing at a server that holds nobody.

I would use publish-subscribe instead. The chat service stores the message and publishes to a topic named for the recipient. Each gateway subscribes to topics for the users it currently holds. The one holding Bob delivers; the others never see it. There is no separate state to go stale, and when Bob reconnects to a different gateway the subscription simply moves with him.

Not losing messages. Every message gets an increasing id within its conversation. The client stores the last id it processed and sends it on reconnect, and the server replays everything since. Three consequences to state: delivery becomes at-least-once so the client deduplicates by id; the replay window is bounded, so a client returning from a week away is told to resynchronise rather than being sent everything; and the message store, not the connection, is the source of truth, so the real-time layer is an optimisation over "fetch the conversation".

Now the gateway restart, which is the interesting part. Fifty thousand connections drop at once. If every client reconnects immediately, fifty thousand connection attempts arrive at nineteen remaining gateways within a second, each requiring a handshake, an authentication and a subscription setup. That burst can push the remaining gateways over, which drops more connections, which produces a larger burst. This is how a routine deploy becomes an outage.

The defences are all cheap. Jittered exponential backoff in the client, so reconnections spread over tens of seconds instead of arriving together. Roll one gateway at a time, so only five percent of connections move at once. Keep headroom — running twenty gateways at ninety percent capacity means there is nowhere for a gateway's worth of connections to go. And connection-rate limiting at the gateway, so it accepts new connections at a rate it can actually complete rather than accepting all of them and failing.

What I would monitor: connections per gateway, memory per gateway, message delivery latency at the 99th percentile, reconnection rate — which spikes before anything else when something is wrong — and per-connection outbound queue depth, because slow clients are the failure that turns into an out-of-memory crash.

InterviewA user's phone loses signal for thirty seconds. Walk through exactly what happens to messages sent during that gap.

Nothing about the disconnection is immediate, and that is the first thing to say. A phone losing signal sends no close packet. The server's socket stays open, believing the connection is fine, and will continue believing that until either a heartbeat goes unanswered or the operating system's own timeout expires — which can be hours. So for the first interval or two, the server is happily writing messages into a socket that goes nowhere.

Heartbeats are what convert this into a known state. With a ping every twenty seconds and a rule of two missed intervals, the server marks the connection dead after about forty seconds and closes it, releasing the memory and removing the subscription.

Messages sent during the gap must not be lost, and that is a design property rather than a hope. The message is written to durable storage first and published second. So a message sent while Bob is offline exists in the conversation regardless of whether any gateway delivered it. The real-time layer is an accelerator over stored data, never the only copy — and stating that inversion clearly is the core of a good answer.

The client reconnects and sends its cursor. It remembers the id of the last message it processed and includes it on reconnect. The server queries the conversation for everything after that id and sends it, then resumes live delivery. The gap closes with no special-case code, because replay and live delivery both go through the same ordering.

Three details that make it correct rather than approximately correct.

Deduplication. The client may receive a message it already has, because the connection could have dropped between delivery and acknowledgement. Ignoring anything at or below the stored cursor handles it, which makes delivery at-least-once with client-side idempotency — the standard arrangement (10.4).

A bounded replay window. Replaying from an arbitrary point means keeping unbounded history in a fast path. Keep a window — the last few thousand messages per conversation, or the last day — and if a client asks from further back, respond with "resynchronise" rather than attempting a huge replay. A client returning after a week should fetch the conversation normally, not stream a week of history through the real-time path.

Jittered reconnection. If the outage was a cell tower rather than one phone, thousands of clients return at the same moment. Backoff with randomness spreads them.

And the ordering point that catches people out. Messages must be ordered per conversation, not globally, and the cursor is therefore per conversation too. A single global cursor across all of a user's conversations forces one ordering point for everything they are part of, which is a bottleneck you created by choosing the wrong granularity — and it is the kind of decision that is very cheap to get right at design time and very expensive to change later.

StaffYour real-time service works fine until a large customer's office loses connectivity and reconnects. Ten thousand clients return at once and the fleet falls over. Design the recovery.

Understand the mechanism before fixing it, because the obvious fix makes it worse.

Ten thousand clients returning simultaneously do not each cost one cheap operation. Each one costs a TLS handshake, an authentication check, a subscription setup, and a replay query for everything missed during the outage. The replay query is the expensive one and it is the one people forget: ten thousand clients each asking for several minutes of history is ten thousand database reads arriving in the same second, for a database sized for a steady drip of new messages.

Then it compounds. The gateways slow down, so some connection attempts time out, so those clients retry, adding load to a system already failing. Meanwhile the clients that did connect are waiting on slow replays and eventually time out too, and reconnect. This is a retry storm (9.5.3) with a thundering herd inside it, and left alone it does not converge.

The fixes, ordered by how much they help per unit of effort.

Jittered backoff in the client, and it must be large jitter rather than a small perturbation. If the first retry is between one and thirty seconds chosen uniformly at random, ten thousand clients spread across thirty seconds instead of arriving in one. This single change does more than everything else combined, it costs a few lines, and the common mistake is jitter of ten percent, which spreads a burst into a slightly wider burst.

A connection rate limit at the gateway. Accept new connections at a rate you can actually complete — say five hundred per second per gateway — and reject the rest with a "retry shortly" that the client honours. A rejected connection is cheap; an accepted connection that then times out has already cost you a handshake, an authentication and a partial replay. Failing fast is what keeps the fleet alive.

Decouple connection from replay. Let the client connect and start receiving live messages immediately, and fetch its missed history as a separate, throttled, paginated request. This is the important structural change, because it separates the cheap operation everybody needs from the expensive operation that is causing the collapse. The user sees the conversation reconnect instantly and the backlog fills in over a couple of seconds, which is also a better experience than a spinner.

A bound in front of the history query. A concurrency limit on replay reads means the database receives what it can handle and the rest queue (9.5.4). Slow recovery beats no recovery.

Headroom, stated as a number. If the fleet runs at ninety percent of connection capacity, there is nowhere for ten thousand displaced connections to go. Capacity planning for a real-time system has to include "one large customer, or one gateway, returning at once" as a normal event rather than an exception.

And the thing that makes the difference next time: make this testable. Run a game day where you deliberately drop a gateway's worth of connections and watch the recovery. The failure modes above are all invisible in normal operation and all obvious within thirty seconds of a deliberate test — and a system that has never had its recovery exercised does not have a recovery, it has an assumption.

Flashcards

FlashSSE versus WebSocket

SSE: one-directional, plain HTTP, automatic reconnect and Last-Event-ID replay. WebSocket: two-way, lowest latency, you build reconnect, replay, heartbeats and backpressure. Default to SSE when traffic is mostly downward.

FlashConnection arithmetic

10–50 KB per held connection. One million = 10–50 GB before any message moves. Fleet size is driven by connection count, not request rate. Split gateways from stateless services.

FlashFinding the user's gateway

Registry: precise, written on every connect, goes stale when a gateway crashes. Publish-subscribe with a topic per user: no state to go stale, subscription moves with the connection. Prefer pub-sub.

FlashCursors

Increasing message id per stream; client sends its last on reconnect; server replays from there. Delivery is at-least-once so the client deduplicates. Bound the replay window — older clients resynchronise.

FlashWhy the stream dies after 60 seconds

An idle timeout in the balancer or a proxy. Heartbeats below the timeout keep it alive, and they also detect connections that died silently because a phone lost signal.

FlashSlow clients

Sending faster than a client receives queues in your process memory. Bound the outbound queue per connection; disconnect on overflow, because the cursor lets the client reconnect and replay cleanly.