11.8 — News Feed & Timeline

You open the app and 200 milliseconds later you are looking at posts from the several hundred accounts you follow, newest first. Somebody had to merge those several hundred streams into one ordered list. The only question in this entire design is when that merging happened: at the moment each post was written, or at the moment you opened the app.

There is no third option. You can pay at write time or at read time, and the whole study is about the fact that the right answer depends on who wrote the post rather than on which strategy is better. An account with 200 followers and an account with 100 million followers are not the same problem, and any design that treats them the same fails at one end or the other.

This is also the archetype for every hot-key problem in 10.6: one entity, disproportionate traffic, and a distribution so skewed that the average tells you nothing useful.

1. Requirements

Functional. Post text, images, and video references. Follow and unfollow. A home timeline of posts from the accounts you follow, newest first. A user timeline of one account's own posts. Like and reply counts.

Non-functional, with numbers.

• Home timeline loads in under 200 ms at p95.
• A post appears in followers' feeds within about 5 seconds.
• The feed may be slightly stale. This one sentence is what unlocks the entire design, and it should be extracted from the product owner explicitly rather than assumed.
• Read to write ratio of about 100:1.

Out of scope today: ranking and machine learning, which 11.20 covers — this study builds the reverse-chronological substrate that ranking sits on top of. Also out: advertising, direct messages (11.7), and search.

The clarifying questions, and what each answer changes

"How stale may the feed be?" If the answer is "instantly consistent", precomputation is impossible and you are building a merge-on-read system with a much harder latency problem. If the answer is five seconds, everything becomes precomputable and asynchronous. This is the highest-leverage question in the whole design and it takes ten seconds to ask.

"What is the follower count distribution — not the average?" The average is around 200 and it is useless. What matters is the shape of the tail: how many accounts have a million followers, and how many have a hundred million. That tail is what breaks fan-out on write, and it is what the hybrid exists for.

"How far back does someone scroll?" If the answer is "twenty posts and then they leave", you can cap the precomputed timeline at a few hundred entries and keep the whole thing in memory. If someone genuinely reads a thousand items, precomputation gets much more expensive.

"Is the feed strictly reverse-chronological, or will it be ranked later?" Both answers lead to the same substrate — a candidate set of recent posts — but knowing that ranking is coming stops you from baking ordering assumptions into storage that ranking will later have to fight.

"Can a post be deleted, and must it disappear everywhere?" This is the question that decides whether precomputed lists are the truth or a cache. The answer is always yes it must disappear, and section 6.4 explains why that means filtering at read time rather than deleting from millions of lists.

2. Estimation

Posts. 300 million daily users × 0.5 posts a day = 150 million posts a day ≈ 1,700 a second average, ~5,000 at peak. What that forces: nothing on its own. Five thousand writes a second is ordinary.

Reads. 300 million users × 10 feed loads a day = 3 billion reads a day ≈ 35,000 a second average, ~100,000 at peak. What that forces: this confirms the 100:1 ratio and points every optimisation at the read path. It also rules out assembling a feed by querying hundreds of accounts on every read, because 100,000 reads a second each fanning out to 200 sub-reads is 20 million sub-reads a second.

The fan-out number, which decides the architecture. 150 million posts × 200 average followers = 30 billion timeline writes a day ≈ 350,000 a second. What that forces: large but tractable — this is a queue and a pool of workers, not an impossibility. The trouble is not the average.

The tail, which is the actual problem. An account with 100 million followers produces 100 million timeline writes for one post. At any plausible write rate that is hours of work for one message, and while it is running it occupies the fan-out pipeline that everyone else's posts are waiting in. What that forces: the hybrid. There is no throughput number that makes 100 million writes for one post acceptable, so that author must not fan out at all.

Storage for precomputed timelines. They hold post references, not post bodies — roughly 50 bytes each. 300 million users × 800 entries × 50 bytes = 12 TB. What that forces: twelve terabytes fits in a memory-backed tier across a reasonable cluster, and that single fact is what makes precomputation affordable at all. If timelines held post bodies at 300 bytes each it would be 72 TB and the answer would be different.

Post storage. 150 million posts a day × ~300 bytes = 45 GB a day, about 16 TB a year of text. What that forces: modest. Media is the real storage cost, and media lives in object storage with only a reference in the post (11.2).

3. API

POST /posts
{ "text": "the tunnel finally opened", "mediaIds": ["m_44a"] }
→ 201 Created
  { "postId": "p_7431096832", "createdAt": "2026-07-31T09:20:11Z" }

GET /feed?cursor=eyJwIjoiN…&limit=20
→ 200 OK
  { "items": [
      { "postId": "p_7431096832", "authorId": "u_12", "text": "…",
        "likeCount": 41, "replyCount": 3, "createdAt": "…" } ],
    "nextCursor": "eyJwIjoiNzQzMTA5NjgxMSJ9",
    "source": "precomputed" }

GET  /users/u_12/posts?cursor=…&limit=20
POST /users/u_12/follow    → 204 No Content
DELETE /users/u_12/follow  → 204 No Content

Cursor pagination is mandatory here, not preferred. A feed changes constantly while someone is reading it. With offset pagination, three new posts arriving while the reader is on page one push three items down, so page two repeats them — and if items are removed, page two skips some entirely. The cursor encodes the last post identifier seen, so the next page is "everything older than this", which stays correct no matter what is inserted above (9.6.2).

The source field is not decoration. It says whether this response came from the precomputed timeline or the merge fallback. Section 6.6 explains why, in a system with two read paths, the response has to say which one it used — otherwise the slow path is invisible in aggregate metrics until users complain.

POST /posts returns as soon as the post is durable, not when fan-out completes. The user sees their own post immediately because the client renders it locally; the promise to followers is the five-second budget from section 1, and it is met asynchronously.

4. Data model

posts
  post_id     BIGINT PRIMARY KEY     -- Snowflake: sorts by time
  author_id   UUID NOT NULL
  text        TEXT
  media_keys  TEXT[]                 -- references into object storage
  created_at  TIMESTAMPTZ NOT NULL
  deleted_at  TIMESTAMPTZ NULL

follows
  follower_id UUID, followee_id UUID
  created_at  TIMESTAMPTZ
  PRIMARY KEY (follower_id, followee_id)

followers                             -- the same data, indexed the other way
  followee_id UUID, follower_id UUID
  PRIMARY KEY (followee_id, follower_id)

timelines                             -- the precomputed read model
  user_id     UUID  →  [post_id, …]   -- capped at ~800, memory-backed

authors_meta
  author_id   UUID PRIMARY KEY
  follower_count BIGINT
  posts_per_day  REAL
  fanout_mode    SMALLINT             -- push | pull, recomputed periodically

Access patterns:

Query	Frequency	Returns
Read one user's precomputed timeline page	100,000/s peak	20 post ids
Hydrate post bodies by id	100,000/s peak × 20	20 rows
Read one author's recent posts	for celebrity merge	20 rows
List an author's followers, in chunks	5,000/s peak, 1,000 at a time	1,000 ids
Append a post id to a timeline	350,000/s	—

post_id is a Snowflake (11.5), which buys three things at once for free: the cursor, the sort key, and the merge key are all the same value. "Newest first" is a reverse range scan, and merging two already-sorted lists needs nothing but a comparison.

follows is stored twice, once by follower and once by followee. "Who do I follow?" and "who follows this author?" are both high-volume queries and they want opposite keys. This is the standard example of denormalising for access patterns, and the cost is that both copies must be written together — which means an unfollow that updates one and fails on the other leaves the system inconsistent, so the pair is written transactionally or through an outbox (10.8.4).

timelines is a read model, and every word of that matters. It is derived, so it is not the truth and losing it loses nothing permanent. It is rebuildable, from posts and follows, which is what makes the fallback path in section 8 possible. And it is capped at around 800 entries, because nobody scrolls past that and an uncapped list is an unbounded per-user cost that grows forever.

authors_meta.fanout_mode is where the hybrid lives. It is data, recomputed on a schedule, not a constant in the code — section 6.1 explains why.

5. Architecture

Figure 1 — Fan-out, three ways. Write-time and read-time work trade directly against each other. The hybrid routes each author to whichever strategy keeps that author's cost bounded, which is why every large feed system converges on it independently.

The write path, drawn separately, because its asynchrony is the thing that keeps posting fast.

Figure 2 — The write path. Posting never waits for fan-out; the five-second budget is a promise about followers, not about the author. The outbox is what stops a crash between "post saved" and "fan-out queued" from producing a post nobody ever sees.

And the read path, which is the hybrid in eleven lines:

async function homeTimeline(userId: string, cursor: Cursor): Promise<FeedPage> {
  const [precomputed, celebs] = await Promise.all([
    timelineStore.range(userId, cursor, 20),          // (1)
    followStore.celebrityFollowees(userId)            // (2)
  ]);
  const celebPosts = await Promise.all(               // (3)
    celebs.slice(0, MAX_MERGE).map(c => postStore.recent(c, cursor, 20))
  );
  const merged = mergeByIdDesc([precomputed, ...celebPosts]).slice(0, 20);  // (4)
  const visible = merged.filter(p => !tombstones.has(p.postId));            // (5)
  return { items: await hydrate(visible), source: precomputed.length ? 'precomputed' : 'merged' };
}

(1) one range read of one list — a single operation regardless of whether the user follows five accounts or five thousand. This is the entire payoff of fan-out on write. (2) the set of celebrity accounts this user follows, which is typically between zero and twenty, and which is small enough to cache on every read node. (3) a bounded parallel fan-in. MAX_MERGE matters: without a cap, a user who follows two hundred celebrities pays two hundred parallel reads and their latency becomes the slowest of the two hundred (10.9). (4) the merge is cheap because both inputs are already sorted by post_id, and Snowflake identifiers sort by time. (5) deletions and blocks are filtered here rather than removed from millions of stored lists — section 6.4.

6. Deep dives

6.1 Choosing the threshold, and why it is not a constant

The cost of fanning an author out is not their follower count. It is followers × posts per day, because a dormant account with 200,000 followers costs less per day than a prolific one with 50,000.

So the classification is a policy evaluated per author and re-evaluated as accounts grow, stored in authors_meta.fanout_mode and recomputed on a schedule. Two properties follow.

The celebrity set is small enough to cache everywhere. Even generously drawn, it is thousands of accounts, not millions, so every read node can hold the whole set in memory and answer "is this author pull-mode?" without a lookup.

The threshold is a tuning knob with a visible trade. Lower it and more authors become pull-mode, which reduces write amplification and adds a read to every follower's hot path. Raise it and the read path gets cheaper while the fan-out queue gets busier. Because write cost is asynchronous and read cost is synchronous and user-facing, the bias should be toward raising the threshold — accept more background work to keep the request path short. Section 8's drill shows what happens when it drifts the other way.

A mode change is not free. An author crossing from push to pull leaves their old posts already in followers' timelines, which is fine — those entries are valid. Crossing the other way, from pull to push, means their next post fans out but their previous ones do not, so followers see a gap unless the read path keeps merging them for a while. The simple, honest answer is that the read path merges recent posts from any author flagged pull-mode within a recent window, and mode changes take effect going forward.

6.2 Fan-out is asynchronous, chunked, and idempotent

Posting returns as soon as the post is durable. Fan-out happens in workers consuming from a queue, and three properties make it survivable.

It is fed by an outbox (10.8.4). The post row and the fan-out job are written in one transaction, so a crash between them is impossible. Without this, a process that saves a post and then dies before publishing the job produces a post that exists and that nobody ever sees — and nothing errors, so nobody finds out.

It is chunked. A task covers a thousand followers, not all of them. That makes the work parallel across many workers, it bounds how much is lost and repeated when a worker dies, and it means a large author's fan-out progresses visibly rather than as one opaque job that either finishes or does not.

It is idempotent. Appending a post identifier to a timeline list is a set-like insert: doing it twice leaves the list unchanged. That is what makes retrying a failed chunk safe, and it is worth choosing the storage operation for this property rather than discovering afterwards that retries produce duplicates.

6.3 What the read path costs, and the cap that keeps it honest

The precomputed read is one operation. The celebrity merge is where the cost hides, and the arithmetic is worth doing.

If a user follows 50 celebrities and each read has a p99 of 20 ms, the merge's latency is the maximum of 50 draws from that distribution, not the average. With 50 parallel reads at 20 ms p99, the probability that at least one is slow is high, and the request's p95 lands in the hundreds of milliseconds. This is tail-latency amplification (10.9) and it is the reason MAX_MERGE exists.

Two mitigations beyond the cap. Cache the merged celebrity segment per user with a short time-to-live, so a heavy follower pays the merge once every few seconds rather than on every request. And cache each celebrity's recent-posts list aggressively, since it is a small object read by millions of people — exactly the shape a cache hierarchy is built for (11.4).

6.4 Deletes, blocks and privacy: filter at read, never purge

A user deletes a post that is already sitting in three million precomputed timelines. The tempting answer is to remove it from all three million. That is wrong for two reasons: it is expensive in proportion to follower count, which is exactly the cost the hybrid exists to avoid, and it races with fan-out workers that may still be inserting it.

The right answer is to filter at read time against a small cached set of deleted post identifiers, and let the capped timeline evict the stale entry naturally as newer posts push it out. The filter costs a set lookup per item on the read path, the tombstone set stays small because entries can be dropped once every timeline that could contain them has rotated past, and deletion becomes an O(1) operation instead of an O(followers) one.

The same technique handles blocks and privacy changes. If A blocks B, B's posts must vanish from A's feed — but B may appear in thousands of timelines and rewriting them all for one block is absurd. The read path checks a per-user block set instead.

The general law, worth stating because it generalises far beyond feeds: a precomputed read model is filtered at read time for correctness, not rebuilt for it. Anything that must be hidden is a read-time predicate. Anything that must be added is a write-time job. Confusing the two produces either an unbounded delete cost or a feed that shows content it should not.

6.5 Counters, which are their own hot-key problem

A celebrity post collects millions of likes, and each like is an increment on one row. That single row becomes a write hotspot that no amount of sharding by post identifier fixes, because it is one post.

Two standard answers. Sharded counters: the count is stored as N sub-counters, each like increments a random one, and the displayed value is the sum. Writes spread across N rows, and reading costs N small reads that are trivially cacheable. Or asynchronous aggregation: likes are events on a stream, a consumer rolls them into a count, and the displayed number lags by a few seconds — which nobody notices on a post with two million likes.

Both are correct. The interesting part is admitting that the displayed count is approximate, and that this is fine for likes and not fine for anything a person will dispute. That distinction between a display counter and an accounting counter recurs throughout this Part.

6.6 Two read paths means two sets of metrics

The read path has a fast branch and a slow branch. The slow branch is used by a minority of users, which means it is invisible in aggregate metrics — a p95 across all requests is dominated by the fast branch, and the users on the slow branch can be having a terrible time while the dashboard looks healthy.

The fix is to carry the branch as a dimension on every metric from the first day: latency by source, latency bucketed by follow count, count of celebrity lists merged per request, and timeline hit rate. Without those, the way you learn about the slow path is a support ticket, and the drill at the end of this page is exactly that scenario.

7. Decision Ledger

Decision	Alternatives	Why this	What it costs
Hybrid fan-out per author	pure push; pure pull	both read and write cost stay bounded regardless of follower count	two read paths; a threshold to tune and monitor
Timelines as a capped read model	assemble on demand	100:1 reads make precomputation obviously right; 12 TB fits in memory	storage, staleness, and the obligation to keep it rebuildable
Asynchronous fan-out through an outbox	fan out inside the request	posting stays fast; a crash cannot lose the fan-out	a few seconds of propagation delay, agreed in section 1
Filter deletes and blocks at read time	purge from every timeline	O(1) per delete instead of O(followers), and no race with fan-out	a tombstone and block-set lookup on the read path
Cap the celebrity merge	merge all of them	bounds tail-latency amplification for heavy followers	a user following 200 celebrities sees a slightly incomplete merge
Cursor pagination on post_id	offset	no duplicates or skips in a feed that changes while you read it	no page numbers and no total count
Snowflake post_id	random identifiers	the cursor, the sort key and the merge key become one value	leaks post timing (11.5)
Approximate like counts	exact synchronous counters	one row cannot absorb millions of increments	the number is a few seconds behind, which is invisible at scale

8. Scale and failure

The celebrity post is handled by not fanning out — and then reappears on the read side. A hundred million followers each merge that author's recent posts on their next feed load, so one small object is read up to a hundred million times in a few minutes. That is a hot key, and it is handled with the cache ladder from 11.4: a per-node in-process cache with a one-to-five second time-to-live absorbs nearly all of it with no network hop, backed by the shared cache, backed by the store, with single-flight so that a time-to-live expiry does not send thousands of simultaneous requests to the origin.

The summary worth saying out loud: the hybrid converts a hundred-million-write problem into a hundred-million-read problem, and reads of one small object are precisely what cache hierarchies are built to absorb. That is why the trade is favourable rather than merely different.

At 10×, timelines shard further by user, which is already the partition key. Regional read replicas of the timeline store cut cross-region latency. And the ranking layer (11.20) goes above this substrate rather than into it — the reverse-chronological read model stays the candidate source, and ranking reorders a set drawn from it.

What breaks	Blast radius	How you find out	What keeps it running	Recovery
Timeline store down	everyone falls back to merge-on-read	source metric flips to merged; latency jumps	the merge path, which must never be deleted for being unused	rebuild timelines from posts and follows
Fan-out workers lagging	posts appear late; nothing errors	age of the oldest fan-out job, not queue depth	reads still work; only freshness suffers	add workers; drain with age as the exit condition
Fan-out worker crashes mid-chunk	one chunk of followers	job retry count	chunks are idempotent, so retry is safe	the chunk replays
Celebrity post read storm	one author's recent-posts key	per-key read counts	in-process cache with short TTL, single-flight	none needed if the ladder is in place
Threshold drifts too low	heavy followers get slow, quietly	latency bucketed by follow count	cap on merged lists	raise the threshold; more async writes, shorter reads
Like counter hotspot	one post's counter row	write latency on that row	sharded counters or async aggregation	switch that post to the aggregated path
Tombstone set unavailable	deleted posts briefly visible	tombstone service errors	fail closed on the deleted set if it is small enough to be complete	restore; the capped timelines rotate the entries out anyway

Alarm on the age of the oldest fan-out job rather than on queue depth. Depth means nothing without age: two million jobs that are four seconds old is a celebrity post being processed normally, and four hundred jobs that are nine minutes old means the five-second promise is broken (10.10).

And the row that people delete by accident: the merge-on-read path is exercised by almost nobody in normal operation — new users with no timeline yet, and users scrolling past the cap. It looks like dead code. It is the fallback for a total timeline-store failure, and deleting it converts a degraded-but-working outage into a complete one.

What the interviewer will push on

"Why not just fan out on write for everyone? Writes are cheap." They want to see whether you reason from the distribution or from the average. Three hundred and fifty thousand timeline writes a second on average is fine; a hundred million writes for a single post is not, and while that job runs it occupies the pipeline everyone else's posts are queued in. The tell is naming the second-order effect — one celebrity post delaying everybody's fan-out — because that is what makes pure push fail in practice rather than merely in theory.

"Why not fan out on read for everyone? Then there is no duplication." The arithmetic answers it: 100,000 feed loads a second, each merging 200 accounts, is 20 million reads a second, and the request's latency is the slowest of its 200 sub-reads rather than the average. Given a 100:1 read-to-write ratio, doing the work on the read side is doing it a hundred times more often than necessary.

"A user with 100 million followers posts. Walk me through it." At write time, nothing fans out — the post is stored once and the write path returns in the same time as any other post. At read time, that author's recent-posts list is read up to a hundred million times, which is a hot key answered by the cache ladder with single-flight. Then volunteer the two second-order effects most candidates miss: the like counter on that post becomes a write hotspot needing sharded counters, and notifying followers is a hundred-million-message campaign that must go through the priority-isolated pipeline in 11.6 or it will delay password resets.

"Someone deletes a post that is in three million timelines. What happens?" The wrong answer is to remove it from three million lists, and the reason it is wrong is worth stating twice: it costs O(followers) for a delete, which is exactly the cost the hybrid was built to avoid, and it races with fan-out workers still inserting it. The right answer is a read-time filter against a cached tombstone set, with the capped timeline evicting the entry naturally. Then generalise: precomputed read models are filtered for correctness, never rebuilt for it.

"Your p95 is 180 ms. Are you sure everyone is happy?" This is the two-read-paths trap. An aggregate p95 is dominated by the fast branch, so a minority on the merge path can be at 900 ms while the dashboard looks fine. The answer is that every metric carries the branch as a dimension — latency by source, latency bucketed by follow count — from the first day, because otherwise the slow path is only ever discovered through complaints.

"What is the celebrity threshold?" The trap is to answer with a number. It is followers × posts per day, it lives in data rather than in code, it is recomputed as accounts grow, and it is biased upward — because write cost is asynchronous and read cost is in the user's face. A candidate who gives a constant has not thought about a dormant account with 200,000 followers versus a prolific one with 50,000.

Volunteer this, because nobody asks: the precomputed timeline is a cache, not a record, and treating it that way is what makes the whole design safe. It can be lost entirely and rebuilt from posts and follows. It can be wrong and be corrected by the read-time filter. It can be capped and fall back to merging for older items. Every one of those properties comes from refusing to let the derived thing become the truth — and the moment someone proposes storing ranked results, or deleting the merge path because it is unused, that safety is what is being traded away.

Next: 11.9 — a post is 300 bytes and a video is 4 gigabytes that has to be transcoded into a dozen formats before anyone can watch it. The feed's read model becomes a pipeline, and the latency budget moves from milliseconds to minutes.

Recall

• The asymmetry: 100 reads per write, so pay at write time. Fan-out on write = one write per follower, one read to serve. Fan-out on read = one write, merge-N to serve. The hybrid routes per author so neither side is unbounded, which is why it is not a compromise.
• The number that decides it: 150M posts × 200 followers = 350,000 timeline writes a second on average — fine. One author with 100M followers = 100M writes for one post — never fine, at any throughput.
• Threshold = followers × posts per day, stored as data, recomputed as accounts grow, biased upward because write cost is asynchronous and read cost is in the user's face. The celebrity set is small enough to cache on every read node.
• Timelines are a capped (~800) rebuildable read model — 300M users × 800 × 50 B = 12 TB, which fits in memory, and that fact is what makes precomputation affordable. Written by asynchronous, chunked, idempotent workers fed by an outbox.
• Deletes, blocks and privacy are filtered at READ time against cached sets, never purged from millions of lists. General law: a precomputed read model is filtered for correctness, not rebuilt for it.
• Cap the celebrity merge, because the request's latency is the slowest of N parallel reads, not the average.
• Snowflake post_id makes the cursor, the sort key and the merge key one value. Cursor pagination is mandatory — offset duplicates and skips in a feed that changes while you read.
• Two read paths means two sets of metrics. Carry source and follow-count buckets as dimensions, or the slow branch stays invisible until users complain.
• Alarm on fan-out job age, not depth. And never delete the merge path for being unused — it is the fallback for a total timeline-store loss.

Self-test: Give both fan-out costs and say why the hybrid is not a compromise. How is the threshold computed and why is it not a constant? What happens when a post is deleted, and what is the general law behind that answer? Why does a hybrid design need the branch as a metric dimension? What does the celebrity case become after you refuse to fan it out?

Quiz Bank

FoundationalCompare fan-out on write and fan-out on read with their costs, and explain why the hybrid wins.

Fan-out on write, also called push. When someone posts, the post's identifier is appended to every follower's precomputed timeline list. Reading then becomes a single range read of a single list — a few milliseconds, trivially cacheable, and completely independent of how many accounts the reader follows. Writing becomes one small write per follower. At 200 average followers that is 200 writes per post, which is entirely affordable. At 100 million followers it is 100 million writes for one post, which is not, and the damage is not only the work itself: while that job runs it occupies the fan-out pipeline that everybody else's posts are waiting in.

Fan-out on read, also called pull. A post is stored once, and a feed read fetches the recent posts of every account the reader follows and merges them. Writing is a single operation. Reading is a fan-in across followees, and the crucial detail is that the request's latency is the slowest of those reads, not their average — 500 parallel reads each with a 20 ms p99 produce a request that is very often slow (10.9). Against a 200 ms budget and a 100:1 read-to-write ratio, this is doing the expensive work a hundred times more often than necessary.

So pure push is right for almost everyone and catastrophic for a few thousand accounts. That is the observation the hybrid is built on: the cost of each strategy depends on the author's follower count, so the strategy should be chosen per author rather than per system.

The hybrid uses push for ordinary authors and pull for celebrities. A read is one list read plus a merge of a handful of celebrity lists — a user follows very few celebrities, typically between zero and twenty — so read work stays bounded. And no author ever fans out to a hundred million, so write work stays bounded. It is not a middle point between two bad options; it is the recognition that the two options are answers to different questions, and each author poses only one of them.

InterviewA user with 100 million followers posts. Walk through exactly what happens.

At write time, nothing fans out. The author is above the threshold, so the post is written once to the post store with a Snowflake identifier and the request returns — the same latency as any other post. There is no queue of a hundred million tasks, and therefore no possibility of one celebrity post starving the fan-out pipeline for hours, which is the specific failure that pure push produces. The author's own user timeline is updated, and downstream consumers such as search indexing and mention notifications proceed through the normal event stream (10.8.4).

At read time, the cost arrives. Every one of those hundred million followers, on their next feed load, reads their precomputed timeline and merges this author's recent posts. That author's recent-posts list is therefore read up to a hundred million times in a short window.

That is a hot key, and it is handled with the standard ladder (11.4). The list is a tiny, highly cacheable object, so a per-node in-process cache with a one-to-five second time-to-live absorbs nearly all of the traffic without a network hop, backed by the shared cache, backed by the store — with single-flight so that a time-to-live expiry does not send thousands of simultaneous identical requests to the origin.

Two second-order effects to name unprompted. The post's like and reply counters become a write hotspot: millions of increments against one row, which no partitioning fixes because it is one post. The answers are sharded counters (N sub-counters summed on read) or asynchronous aggregation from an event stream, both of which make the displayed number approximate — and approximate is correct for a like count and would not be for a payment. And if followers are notified, that is a hundred-million-message campaign which must go through the priority-isolated pipeline of 11.6, or it will sit in front of everybody's password resets.

The sentence that summarises the trade: the hybrid converts a hundred-million-write problem into a hundred-million-read problem, and reads of a single small object are exactly what cache hierarchies exist to absorb. That is why the trade is favourable rather than merely a different shape of the same cost.

InterviewA post is deleted. It exists in three million precomputed timelines. What happens?

Not the obvious thing. Removing it from three million lists costs work proportional to the author's follower count, which is precisely the cost the hybrid was designed to avoid — and worse, it races with fan-out workers that may still be inserting the post into timelines that had not been reached yet, so some deletions would leave entries behind.

Instead, filter at read time. The post identifier goes into a small tombstone set, cached on every read node. The read path checks each candidate against that set and drops matches. The capped timeline evicts the stale entry naturally as newer posts push it past the 800-entry limit, which means the tombstone can eventually be dropped too — it only has to live as long as any timeline could still contain the post.

The costs, honestly. A set lookup per item on the read path, which is cheap and constant. A tombstone set that must be available; if it is not, the safe behaviour is to fail closed on deletion if the set is small enough to be complete, because briefly hiding an extra post is better than showing a deleted one.

The same mechanism handles blocks and privacy changes. If A blocks B, B's posts must disappear from A's feed — but B's posts may be in thousands of timelines, and rewriting all of them because one person pressed a button is absurd. The read path consults a per-user block set instead.

The general law, which is the point of the question: a precomputed read model is filtered at read time for correctness, never rebuilt for it. Anything that must be hidden becomes a read-time predicate; anything that must be added is a write-time job. Getting that split wrong produces either an unbounded cost on a common operation or a feed that displays content it should not.

StaffDesign the migration to add machine-learned ranking on top of this chronological feed without destabilising it.

The principle first: ranking is a layer above the substrate, never a replacement for it. The precomputed timeline stays the candidate generator — its job is "here are the 800 most recent things you could see", which is cheap, correct and rebuildable. Ranking reorders a candidate set drawn from it, optionally mixed with other sources. Fusing them, so that a store holds ranked results, forfeits the substrate's most valuable property: that it can be rebuilt deterministically from posts and follows.

Phase 1 — build it and run it in shadow. The ranking service consumes the candidate set plus features (affinity with the author, recency, engagement rates, media type) and returns a scored ordering. In shadow mode, production still serves chronological, the ranker scores the same candidates, and both orderings are logged for offline comparison (10.11). This is where you discover the latency cost, which is what kills most naive designs: a ranker adding 300 ms to a 200 ms budget is unshippable regardless of how good its ordering is.

Phase 2 — precompute what you can. The latency is almost never the model, it is feature retrieval. Move features into a low-latency store keyed by (user, author) and by (post), refreshed asynchronously, so that request-path work is a lookup plus a small inference (11.20).

Phase 3 — ramp behind a flag on a small percentage, measuring engagement and guardrail metrics. Session length is the seductive one and the most misleading; also measure content diversity, how many distinct creators appear, and complaint rate. A ranker that raises engagement while collapsing diversity is a product failure that looks like a win for two quarters.

Phase 4 — keep the chronological path permanently, for two non-negotiable reasons. It is the fallback: rank with a timeout and fall back to chronological, so a ranking outage degrades quality rather than availability (10.9). And it is a user-facing option, because a meaningful fraction of people want it and regulators in several jurisdictions now require it to exist.

Three warnings that separate a staff answer from a senior one.

Do not let ranking change what is stored. Persisting ranked results loses rebuildability and creates a cache-invalidation problem across every model change — every retrain would invalidate everything.

Feedback loops are the real risk. A ranker trained on engagement produced by the previous ranker converges on whatever that ranker happened to amplify. Preventing that needs hold-out populations served chronologically permanently, as unbiased training and evaluation data. That is an infrastructure requirement with a real cost, not a research preference, and it has to be argued for before the first ranker ships rather than after.

Allocate the latency budget explicitly. Sixty milliseconds of the two hundred, say, with the ranker returning its best answer within the budget rather than the best possible answer. A ranker with no deadline will eventually find a way to use all the time there is.

The sentence for the design document: the chronological feed is the system of record for candidates and the fallback for availability; ranking is a bounded-latency reordering layer with a permanent unbiased control group — which is exactly what lets the ranker be replaced, retuned or switched off without touching the substrate underneath it.

Flashcards

FlashThe two fan-out costs

Push: one write per follower, one read to serve. Pull: one write, merge-N to serve, and the request's latency is the slowest of N. The hybrid picks per author, so neither side is unbounded.

FlashThe celebrity threshold

followers × posts per day, stored as data and recomputed as accounts grow — not a constant. Biased upward, because write cost is asynchronous and read cost is in the user's face.

FlashThe timeline read model

~800 post ids per user, ~50 bytes each, 12 TB total — fits in memory, which is what makes precomputation affordable. Written by asynchronous chunked idempotent workers via an outbox. Derived, capped, rebuildable.

FlashDeletes, blocks, privacy

Filtered at read time against cached sets; never purged from millions of lists. Law: precomputed read models are filtered for correctness, not rebuilt for it.

FlashWhy Snowflake post ids

Time-sortable, so the cursor, the sort key and the merge key are the same value. Cursor pagination is then mandatory anyway, because offset duplicates and skips in a feed that changes while you read it.

FlashMetrics for a hybrid read path

Carry the branch as a dimension: latency by source, latency by follow-count bucket, celebrity lists merged per request, timeline hit rate. Otherwise the fast branch's numbers hide the slow branch entirely.

Scenario Drill

DrillFeed loads are 900 ms at p95 for users who follow 2,000 or more accounts, and fine for everyone else. Diagnose it without profiling access.

The correlation with follow count is the whole diagnosis. If the precomputed timeline were doing its job, follow count would be irrelevant to read latency — one list read is one list read whether you follow five accounts or five thousand. That these users are slow means they are not being served from the precomputed path, or only partly. Three candidate causes, and they can be separated by reasoning before anyone touches a profiler.

Cause one: they follow a lot of celebrities. Someone following 2,000 accounts plausibly follows fifty or more that are above the threshold, and the hybrid's read path merges each celebrity's recent posts individually. Fifty parallel reads produce a request whose latency is the maximum of fifty draws, and at a 20 ms p99 per read that lands in the hundreds of milliseconds by tail amplification alone (10.9).

The fix: cap the number of celebrity lists merged per request — take the highest-affinity or most recently active N — and cache the merged celebrity segment per user with a short time-to-live, so a heavy follower pays the merge once every few seconds rather than on every request.

Cause two: the threshold is set too low, so far more authors are classified as celebrities than intended, and every extra celebrity is another synchronous read on every one of their followers' hot paths. Check the size of the celebrity set against what you expect. This parameter drifts silently as accounts grow, and it degrades the heaviest users first, which is exactly the observed pattern.

The fix: raise the threshold, accepting more write-side fan-out for those authors. That cost is bounded and asynchronous, while the read cost it replaces is synchronous and sits in front of a user. That asymmetry is the whole argument for biasing the threshold upward.

Cause three: these users' timelines are missing or being rebuilt, so the code has silently fallen back to full merge-on-read — and merging 2,000 followees is exactly a 900 ms operation. Plausible reasons: large timelines evicted first under memory pressure, fan-out workers skipping users above some size, or a rebuild that never finished.

The fix, and the diagnostic: measure timeline hit rate segmented by follow count. That one query distinguishes this cause from the other two, because in this case the source field on responses will say merged rather than precomputed.

The instrumentation to add regardless of which cause it turns out to be: per request, record the number of celebrity lists merged, whether the timeline was hit, how many fan-in reads were issued, and the latency bucketed by follow count. The aggregate p95 hid this entirely until users complained, and it always will — because in any hybrid design the slow path serves a minority and is therefore invisible in aggregate metrics (10.10).

The general lesson worth stating in the design document: when a system has two read paths, the metric must carry the branch as a dimension from the first day. Otherwise the cheap branch's numbers will always mask the expensive one's, and you will learn about the expensive one from your users rather than from your monitoring.