Appearance
11.20 — Feed Ranking & Model Serving
A ranking model that is 3% better and 300 milliseconds slower does not ship. That sentence is the whole study, and it is why this page is about engineering rather than about models.
11.8 built the chronological substrate — a precomputed list of the most recent things a user could see. This study puts a model on top of it, inside a 60-millisecond slice of that page's 200-millisecond budget, scoring a thousand candidates per request at a hundred thousand requests a second.
Doing the arithmetic first: 60 milliseconds divided by a thousand candidates is 60 microseconds per candidate, including fetching its features. No interesting model runs in 60 microseconds. That single number forces everything below — the funnel, the feature store, the batching discipline — and it is the first thing to say when someone asks how ranking works at scale.
1. Requirements
Functional. Rank a candidate set for a user by predicted engagement. Blend several objectives — relevance, diversity, freshness, fairness to creators. Support online experiments. Refresh models without downtime.
Non-functional, with numbers.
- Ranking adds no more than 60 ms at p99, as a strict slice of the feed's 200 ms budget.
- About 1,000 candidates scored per request, at 100,000 requests a second.
- Models deployable in hours, not weeks.
- The ranker must be removable. Its failure degrades quality, never availability. That is a structural requirement, and section 7.6 is about what it costs to keep it true.
Out of scope today: model architecture and training algorithms, the labelling pipeline, and advertising auctions.
The clarifying questions, and what each answer changes
"What is the latency budget for ranking specifically?" Not for the page — for ranking. If the answer is "whatever it needs", the design will discover its budget in production, and it will discover it as a page that got slower. Sixty milliseconds forces the funnel; six hundred would not.
"What are we actually optimising?" If the answer is "engagement", ask what engagement is a proxy for, because any sufficiently good optimiser will eventually find the gap between the proxy and the thing it stands for. Section 7.4 is entirely about that gap.
"Can the feed work with the ranker switched off?" It must be able to. Establishing this in the requirements is what makes the fallback path a first-class component rather than dead code somebody deletes in a cleanup.
"Where do features come from, and who computes them?" If the answer is "a data scientist writes the training query and an engineer reimplements it in the service", the system already has the defect in section 7.3 and just does not know yet.
"Will there be a population that never sees the ranker?" The answer should be yes, permanently. It is the only source of unbiased data once the ranker is shaping its own training set, and it is much easier to argue for before launch than after.
2. Estimation
Scoring volume. 100,000 requests a second × 1,000 candidates = 100 million scorings a second. What that forces: rejection of any design that runs one large model per candidate. This is not a tuning problem; it is off by orders of magnitude.
Per-candidate budget. 60 ms ÷ 1,000 candidates = 60 microseconds per candidate, and that has to cover fetching its features as well as scoring it. What that forces: the funnel. Expense must be applied selectively — cheap operations over many items, expensive ones over few.
Feature volume, which is the number people miss. 1,000 candidates × ~50 features each = 50,000 values per request, and at 100,000 requests a second that is 5 billion feature values a second. What that forces: feature fetching, not inference, is usually the bottleneck. Fifty thousand values must arrive in one or two batched lookups. A per-candidate call — even one taking 200 microseconds — is 200 milliseconds of serial work, more than three times the entire budget.
How the budget divides across stages. Retrieval ~5 ms, light ranking ~10 ms, heavy ranking ~25 ms, re-ranking ~5 ms, and ~15 ms of serialisation, network and overhead. What that forces: each stage needs its own deadline, and each must return its best answer within that deadline rather than the best possible answer. A stage with no deadline will eventually consume all the time there is.
Model refresh volume. A model artifact is tens to hundreds of megabytes, and there may be several models across several stages, each retrained daily. What that forces: hot-swapping — load, warm, flip a pointer — rather than restarting serving processes, because restarting a fleet to deploy a model means a deploy is an availability event and nobody will do it daily.
Cost of the holdback. A permanent randomised 1–5% of users never sees the ranker. What that forces: an honest conversation. That slice is a slightly worse experience for a small number of people, forever, and it buys unbiased training data, honest long-term measurement, and a continuously exercised fallback. It is much cheaper to agree to before launch.
3. API
Ranking is an internal service called by the feed service, not a public endpoint.
http
POST /rank
{ "userId": "u_8821",
"requestId": "req_01J9…",
"candidates": [ { "itemId": "p_74310", "authorId": "u_12", "createdAt": "…" } ],
"context": { "surface": "home", "device": "mobile", "sessionPosition": 3 },
"deadlineMs": 60,
"experiment": { "bucket": "ranker_v9", "holdback": false } }http
200 OK
{ "ranked": [ { "itemId": "p_74310", "score": 0.812,
"reasons": ["author_affinity", "recency"] } ],
"modelVersion": "rank-heavy-2026-07-30-a",
"stagesRun": ["retrieval", "light", "heavy", "rerank"],
"featureCompleteness": 0.97,
"tookMs": 41 }http
200 OK
{ "ranked": [ … ],
"modelVersion": "fallback-chronological",
"stagesRun": ["retrieval"],
"degraded": true,
"tookMs": 6 }deadlineMs is passed in by the caller, not configured in the ranker. The feed service owns the page's budget and hands ranking its slice, which means when the budget changes there is one place to change it and the ranker cannot quietly consume more.
degraded: true and stagesRun are how a quality outage becomes visible. A ranker that silently falls back for 30% of requests returns perfectly valid responses with no errors at all, and the feed slowly gets worse for reasons nobody can see. Reporting which stages actually ran converts that into a number on a dashboard.
featureCompleteness is the same idea one level down. A feature that is silently defaulting for a third of requests is a quality outage with no error rate attached, and section 7.3 explains why this is the most common silent failure in the whole system.
reasons exists because "why was this shown to me?" is a question with regulatory weight in several jurisdictions and product weight everywhere. Deriving a short explanation from the top-contributing features at scoring time is far cheaper than reconstructing it later, and reconstructing it later is usually impossible.
4. Data model
feature_definitions -- ONE definition, two materialisations
feature_name TEXT PRIMARY KEY
entity TEXT -- 'user' | 'item' | 'user_item' | 'author'
transformation TEXT -- the single source of truth for how it is computed
ttl INTERVAL
owner TEXT
online_features -- low-latency, batched reads
(entity_key, feature_name) → value, computed_at
offline_features -- historised, for training
entity_key, feature_name, value, valid_from, valid_to
prediction_log -- the exact vector used, per prediction
request_id, item_id, model_version
features JSONB -- what was actually served, not recomputed
score REAL
served_at TIMESTAMPTZ
position SMALLINT -- needed for position-bias correction
engagement_log
request_id, item_id, action, occurred_at
models
model_version TEXT PRIMARY KEY
stage TEXT, artifact_path TEXT
trained_until TIMESTAMPTZ, promoted_at TIMESTAMPTZ NULLAccess patterns:
| Query | Frequency | Returns |
|---|---|---|
| Batched read of features for N candidates | 100,000/s × 2 stages | tens of thousands of values |
| Read user-level features | 100,000/s, once per request | ~50 values |
| Append a prediction log row | sampled, high volume | — |
| Point-in-time join for training | daily batch | billions of rows |
| Load a model artifact | on deploy | one file |
feature_definitions is the anti-skew mechanism, and it is the most important table on this page. One definition, materialised to both stores by the same code. Section 7.3 argues why any architecture where the definition exists twice is guaranteed to drift.
offline_features is historised with validity ranges so that a training row for an event at time T carries each feature's value as of T, not its current value. Without that, the model trains on information from the future, scores brilliantly offline, and fails the moment it is served.
prediction_log stores the feature vector that was actually served, not one recomputed later. That single choice makes skew structurally impossible for logged features, and it makes "why was this ranked first?" answerable months afterwards.
position is logged because it is needed to correct the labels. An item ranked first is clicked partly because it was first, and training on raw clicks teaches the model that whatever it already puts first is good — which is section 7.5's feedback loop in its simplest form.
5. The funnel
Retrieval narrows millions of items to about a thousand using operations costing microseconds — index lookups, approximate nearest-neighbour search over embeddings, precomputed candidate lists. Recall matters here and precision does not, because later stages will sort it out.
Light ranking scores those thousand with a small model over a handful of cheap features and keeps about two hundred. It only needs to be right about which items are plausible.
Heavy ranking applies the full model with all features to those two hundred, where per-item cost can be a hundred times higher because there are five times fewer items.
Re-ranking adjusts the top twenty for diversity, deduplication, policy and creator fairness. These are business rules rather than model output, and they are applied last precisely so that a high score cannot override them.
The alignment problem people miss: the stages must agree about what is good. If retrieval systematically excludes items the heavy ranker would have loved, no amount of ranking quality recovers them. So retrieval recall is measured explicitly — score a random sample with the heavy model, and check how many of its top items retrieval never surfaced.
6. Architecture
7. Deep dives
7.1 Why the funnel, in arithmetic
The budget is 60 milliseconds for a thousand candidates — 60 microseconds each, including features. That is far below what any interesting model costs per item, and multiplied by 100,000 requests a second it is beyond any affordable fleet.
But quality demands an expensive model. The resolution is not to make the model cheaper; it is to apply expense selectively.
Each stage is roughly a hundred times more expensive per item and sees roughly five to ten times fewer items, so total cost per stage stays approximately flat while quality compounds. That is the entire economic argument, and it generalises far beyond ranking: it is the same cheap-filter-then-expensive-score shape as the route search in 11.17 and as retrieval-then-rerank in 11.22. Whenever the correct scoring function is too expensive to apply broadly, this is the standard answer.
7.2 Feature fetching is the bottleneck, not inference
Fifty thousand feature values per request is the number that decides the serving design, and the rules that follow from it are unglamorous and non-negotiable.
Batch. One multi-get covering all candidates and all their features, per stage. A per-candidate call taking even 200 microseconds is 200 milliseconds of serial work for a thousand candidates — more than three times the entire budget.
Fetch user-level features once per request, not once per candidate. A user's features are identical across all thousand candidates, and fetching them a thousand times is the single most common performance defect in a first implementation.
Co-locate. The feature store lives in the same region and ideally the same rack. At this volume, a millisecond of extra round-trip time is a meaningful fraction of the budget.
Precompute anything expensive, asynchronously, so that serving is a lookup rather than a computation.
And the uncomfortable discipline: a feature that cannot be served within budget must be dropped from the model. Not deferred, not optimised later — dropped, and the model retrained without it. A model that depends on a feature the serving path cannot deliver is a model that will silently default that feature in production, which is worse than not having it at all.
7.3 Training and serving skew
The same feature computed differently in training and in serving. The model performs well offline and degrades in production, with no error, no alert and no obvious cause.
The classic instances are all small and all invisible. The training pipeline computes "days since last purchase" over a warehouse with a null for users who never purchased, while the serving code returns zero. Training uses a seven-day window ending at midnight; serving uses a rolling seven days ending now. Training data has been deduplicated and cleaned; serving data has not. A unit differs — seconds against milliseconds — in one path only.
Prevention, in order of importance.
One definition, two materialisations. A feature is defined once, and the same definition produces both the offline training table and the online serving values. Any architecture in which a data scientist writes a query and an engineer reimplements it in the service is guaranteed to skew eventually — the reimplementation is the bug, and it does not matter how carefully it is reviewed.
Point-in-time correctness in training. A training row for an event at time T must contain each feature's value as of T. Otherwise the model learns from information that did not exist yet, scores brilliantly on held-out data, and fails immediately in production. This requires historised features and a temporal join, and it is the main reason feature stores exist as a category.
Serve-time logging as the training source. Log the exact feature vector used for each prediction and train on those logs rather than on recomputed values. This makes skew structurally impossible for logged features. The cost is real and worth stating: you can only train on features you already serve, so a new feature needs a logging period before it can be used.
Continuous skew detection. Periodically replay logged serving vectors through the training pipeline's computation and compare. Any feature whose distributions differ is a defect. Alarm on it, because it will happen — a refactor, a schema change, a null-handling tweak — and it fails silently by nature.
The framing worth stating: skew is not a modelling problem to be worked around. It is a software engineering problem about a single source of truth, and treating it as one is what separates systems that stay accurate from systems that quietly decay.
7.4 Multi-objective ranking, and why the weights are configuration
Real systems optimise a blend:
score = w₁·P(click) + w₂·P(meaningful engagement) − w₃·P(report) + w₄·freshness + w₅·diversityThe weights are product policy, not learned parameters. They live in owned configuration with an audit trail, because "why does the feed show me this?" is a question that must have an answerable, human-attributable answer — and because when the outcome is wrong, the response should be a re-weighting decided by a person rather than a retraining that hopes for a different result.
Optimising a single engagement metric reliably produces outcomes nobody intended. Engagement is a proxy for user value, and any sufficiently effective optimiser will eventually find the gap between the proxy and the thing it stands for. Naming that in advance, and building the blend so the gap can be closed by adjusting a number, is what makes the system correctable.
7.5 Feedback loops, the deepest problem here
A model trained on engagement produced by the previous model learns to reproduce that model's biases. Content the ranker never shows generates no engagement data, and is therefore learned to be uninteresting — a self-fulfilling prophecy that tightens every day.
Three defences, and all three are infrastructure rather than research.
Exploration. A small fraction of impressions go to under-explored items, deliberately sacrificing a little short-term engagement for long-term data quality. Without it, the candidate pool the model knows anything about shrinks continuously.
A permanent randomised holdback. A slice of users served without ranking, forever. This is the only source of unbiased training and evaluation data once the ranker is shaping its own inputs, and it is much easier to argue for before launch than to add afterwards — by which time the population has adapted and there is no comparable baseline left to construct.
Position-bias correction. An item ranked first is clicked partly because it was first. Training on raw clicks teaches the model that whatever it already ranks first is good, which is the feedback loop in its simplest and most direct form. Logging position and correcting for it in the labels is what breaks it.
7.6 Failing open, and what it costs to keep that true
The requirement is that a ranking failure degrades quality rather than availability, and it is met by a degradation ladder: the current model, then the previous model, then a simple heuristic, then chronological.
The cost is that the fallback path must stay healthy, and a path nothing exercises decays. Code changes around it, its dependencies drift, and the first time it is genuinely needed it fails — which converts a ranking outage into a feed outage, the exact thing the requirement existed to prevent.
This is where the holdback earns its keep a second time. The permanent randomised population served chronologically means the fallback path carries real production traffic every day, so it cannot silently rot. One design decision paying for itself in two independent ways is worth pointing out, because it is the argument that usually wins the discussion about whether the holdback is affordable.
7.7 Experimentation, and the two traps
Experiments need stable assignment, guardrail metrics alongside the target metric, and enough traffic to detect the effect sizes that matter.
Two traps are specific to ranking and both invalidate naive comparisons.
Network effects. A ranking change alters what creators post, and those posts appear in the control group's feed too. The control is therefore contaminated by the treatment, and the measured difference understates or overstates the real effect in ways that are hard to reason about. Detecting this needs creator-side metrics, and sometimes it needs the experiment to be split by creator rather than by viewer.
Novelty effects. Almost any change lifts engagement briefly, simply because it is different. Most ranking wins shrink substantially by the third week, so an experiment evaluated over two days is measuring novelty rather than value. The mitigation is a minimum soak per stage measured in weeks.
8. Decision Ledger
| Decision | Alternatives | Why this | What it costs |
|---|---|---|---|
| A multi-stage funnel | one model over every candidate | 60 microseconds per candidate is impossible for a real model; total cost stays flat | several models to train, monitor and keep aligned with each other |
| Feature store with one shared definition | compute features in the serving code | removes training and serving skew, the top production failure | real infrastructure to build and operate |
| Point-in-time correct training data | join current feature values | prevents leakage that looks like excellent offline metrics | harder pipelines; historised features cost storage |
| Log the vector actually served | recompute features for training | makes skew structurally impossible for logged features | you can only train on features you already serve |
| Blend weights as owned configuration | one learned objective | "why this item?" is answerable and attributable; wrong outcomes are correctable by a person | weights need governance and periodic review |
| A permanent randomised holdback | ramp to 100% | unbiased data, honest long-term measurement, and a continuously exercised fallback | a small population gets a worse experience, forever |
| Deadline passed in by the caller | the ranker configures its own | one owner for the page's budget; the ranker cannot quietly take more | the ranker must return its best answer within the deadline |
| Fail open to chronological | fail the request | a ranking outage degrades quality, never availability | the fallback must be maintained, which the holdback pays for |
9. Scale and failure
At 10×, add retrieval sources — each cheap and run in parallel — keep heavy models confined to the last stage, batch inference across concurrent requests, and take the highest-leverage lever available: cache ranked results briefly for users who refresh repeatedly, because re-ranking an unchanged candidate set is pure waste.
| What breaks | Blast radius | How you find out | What keeps it running | Recovery |
|---|---|---|---|---|
| Feature store slow | every request's latency | per-stage duration; feature-fetch p99 | per-feature timeouts with defaults, and a model trained to tolerate missing features | restore; watch the default rate come down |
| A feature silently defaults | quality, with no errors at all | featureCompleteness and per-feature default rate | nothing else notices | fix the pipeline; consider retraining |
| Model server down | quality only | fallback rate; degraded in responses | the degradation ladder — previous model, heuristic, chronological | restore; the ladder held |
| A bad model promoted | quality, silently and everywhere | canary on a small slice with guardrail regression as an automatic rollback | canary gating before full promotion | roll back the artifact; it is a pointer flip |
| Feature pipeline stale | slow, invisible decay | feature freshness per feature, not pipeline success | nothing — a stale feature still returns a value | fix the pipeline; assess whether to retrain |
| Prediction drift | the world changed, or an input broke | output distribution shift | nothing automatic | investigate before retraining, because it may be the input |
| Fallback path rotted | a ranking outage becomes a feed outage | the holdback exercises it daily | real traffic on the fallback, continuously | this is why the holdback is not optional |
| Feedback loop tightening | the candidate pool narrows over months | comparison against the holdback | exploration traffic and position-bias correction | unrecoverable without unbiased data — plan ahead |
Monitoring for a ranking system is different, and it must be said explicitly. Latency and error rate tell you almost nothing here, because the characteristic failure produces neither. Four signals matter and none appear on a standard service dashboard:
Prediction distribution drift — the model's output histogram shifting means either the world changed or an input broke, and telling those apart is the investigation.
Feature distribution drift, with per-feature null and default rates, because a feature defaulting for a third of requests is a quality outage with no error attached.
Skew detection, by replaying logged serving vectors through the training path and comparing.
Calibration — do items predicted at 0.3 actually engage at about 30%? A model whose ranking is fine but whose probabilities are wrong will break any downstream logic that uses the score as a number rather than as an order.
What the interviewer will push on
"Why not one model over all the candidates?" Arithmetic, and give it immediately: 60 milliseconds over a thousand candidates is 60 microseconds each including features, and that is off by orders of magnitude for any interesting model. Then the resolution — apply expense selectively, with each stage roughly a hundred times costlier per item and seeing five to ten times fewer — and the alignment point most candidates miss: if retrieval excludes what the heavy ranker would love, no ranking quality recovers it, so retrieval recall must be measured.
"Where does the latency actually go?" Not the model. Fifty thousand feature values per request is the number, and it means batched multi-gets, user features fetched once rather than a thousand times, and a co-located store. Then the discipline: a feature that cannot be served in budget is dropped from the model, because the alternative is a feature that silently defaults in production.
"What is training and serving skew, and how do you prevent it?" The same feature computed two ways, producing a model that is excellent offline and decays in production with no error. Prevention in order: one definition materialised to both stores by the same code; point-in-time correctness so training never sees the future; logging the vector actually served and training on that; and continuous skew detection by replay. The tell is calling it a software engineering problem about a single source of truth, not a modelling problem.
"Why are the blend weights configuration rather than learned?" Because "why was this shown to me?" needs a human-attributable answer, and because when the outcome is wrong the fix should be a decision rather than a retraining that hopes for something different. Then the deeper reason: engagement is a proxy, any good optimiser finds the gap between a proxy and what it stands for, and a configurable blend is what lets a person close that gap deliberately.
"Your ranker's model server dies. What does the user see?" A chronological feed, slightly worse and completely available. Then the cost nobody volunteers: the fallback path decays if nothing uses it, so the permanent holdback is what keeps it alive — which is the same decision paying for itself twice, once for unbiased data and once for a fallback that actually works.
"Engagement is up 8%. Ship it?" Not on that number alone. Decompose it — more sessions, longer sessions, or more actions? — and check next-week return rate, negative feedback, and creator diversity. Then the framing: engagement is a proxy, and evidence that value fell while the proxy rose is evidence the proxy has decoupled, which is a reason to reject the change and to fix the objective, because the next change will exploit the same gap.
Volunteer this, because nobody asks: a ranking system fails silently, adapts its own training data, and reshapes the behaviour of the people it ranks — and none of those three are recoverable by rolling back a deploy. That is the argument for three things that always look expensive up front: a permanent unbiased holdback, exploration traffic from day one, and guardrail metrics with the power to block a launch. Every one of them is much cheaper to establish before the first ranker ships than to retrofit after the population has already adapted.
Next: 11.21 — from ranking a feed to letting users build their own pipelines: a platform where the code being run was written by a customer, the integrations belong to other companies, and the failure modes are somebody else's rate limits.
Recall
- The budget forces the funnel: 60 ms ÷ 1,000 candidates = 60 microseconds each including features. So retrieval (millions to a thousand, cheap, recall not precision) → light ranking (a thousand to two hundred) → heavy ranking (two hundred) → re-rank the top twenty for diversity, deduplication and policy, which are business rules applied after the model.
- Each stage is ~100× costlier per item and sees ~5–10× fewer items, so cost per stage stays flat while quality compounds. The stages must be aligned — measure retrieval recall against the heavy model, or excluded items are lost forever.
- Feature fetching is the bottleneck, not inference. 50,000 values per request means batched multi-gets, user features fetched once, a co-located store, and the discipline that a feature which cannot be served in budget is dropped from the model.
- Training and serving skew is a software engineering problem: one definition materialised to both stores by the same code · point-in-time correctness so training never sees the future · log the vector actually served and train on it · continuous replay-and-compare detection.
- Blend weights are product policy in owned configuration, never learned, because "why was this shown?" must be attributable and a wrong outcome should be correctable by a person.
- Feedback loops are the deepest problem. Defences: exploration traffic, a permanent randomised holdback, and position-bias correction — an item ranked first is clicked partly because it was first.
- Fail open to chronological, through a degradation ladder — and the holdback is what keeps that path from rotting, which is the same decision paying for itself twice.
- Monitoring is different: latency and error rate tell you nothing. Watch prediction drift, feature drift and default rates, skew, and calibration — all of which fail silently.
Self-test: Compute the per-candidate budget and say what it forces. Where does the latency actually go, and what are the three batching rules? Give the four defences against skew, in order. Why are the blend weights configuration? Name the three feedback-loop defences and the four ML-specific signals that fail silently.
Quiz Bank
FoundationalWhy is ranking a multi-stage funnel rather than one model?
Arithmetic, and it is decisive. The latency budget is 60 milliseconds for about a thousand candidates, which is 60 microseconds per candidate including fetching its features — far below what any interesting model costs per item. Multiplied by 100,000 requests a second, it is beyond any affordable fleet regardless of hardware.
But quality demands an expensive model. The resolution is not to make the model cheaper; it is to apply expense selectively.
Retrieval narrows millions of items to about a thousand using operations costing microseconds — inverted-index lookups, approximate nearest-neighbour search over embeddings, precomputed candidate lists. Recall matters here and precision does not, because later stages will sort out the ordering.
Light ranking scores those thousand with a small model over a handful of cheap features and keeps about two hundred. It only has to be right about which items are plausible, not about their exact order.
Heavy ranking applies the full model with all features to those two hundred. Per-item cost can be a hundred times higher precisely because there are five times fewer items.
Re-ranking adjusts the top twenty for diversity, deduplication, policy and creator fairness. These are business rules rather than model output, and they are applied last so that a high score cannot override them — which is a structural decision, not a stylistic one.
The economics: each stage is roughly a hundred times more expensive per item and sees roughly five to ten times fewer items, so total cost per stage stays approximately flat while quality compounds across stages.
The design consequence people miss is alignment. The stages must agree about what is good. If retrieval systematically excludes items the heavy ranker would have ranked highly, no amount of ranking quality recovers them — they were never candidates. So retrieval recall is measured explicitly, typically by scoring a random sample with the heavy model and checking how many of its top items retrieval failed to surface.
And the pattern generalises well beyond machine learning. It is the same cheap-filter-then-expensive-score shape as the route search in 11.17, where geometry generates candidates and a routing engine scores them, and as retrieval-then-rerank in 11.22. Whenever the correct scoring function is too expensive to apply broadly, this is the standard answer.
InterviewWhat is training and serving skew, and how do you prevent it?
The same feature computed differently in training and in serving, producing a model that performs well offline and degrades in production — with no error, no alert and no obvious cause.
The classic instances are all small and all invisible. The training pipeline computes "days since last purchase" over a warehouse and returns null for users who never purchased, while the serving code returns zero. Training uses a seven-day window ending at midnight; serving uses a rolling seven days ending now. Training data has been deduplicated and cleaned; serving data has not. A unit differs — seconds versus milliseconds — in one path only. Each one shifts the model's inputs systematically away from what it learned, and none of them produces a single failed request.
Prevention, in order of importance.
One feature definition, two materialisations. A feature is defined once — as code or as a declarative transformation — and the same definition produces both the offline training table and the online serving values. Any architecture in which a data scientist writes a query and an engineer reimplements it in the service is guaranteed to skew eventually. The reimplementation is the bug, and no amount of careful review prevents it, because the two will drift at different times for different reasons.
Point-in-time correctness in training. A training row for an event at time T must contain each feature's value as of T, not its current value. Otherwise the model learns from information that did not exist yet — label leakage — and it scores brilliantly on held-out data while failing immediately in production. This requires historised features and a temporal join, and it is the main reason feature stores exist as a product category at all.
Serve-time logging as the training source. Log the exact feature vector used for each prediction, sampled if the volume demands it, and train on those logs rather than on recomputed features. This makes skew structurally impossible for logged features. The cost is real and should be stated: you can only train on features you already serve, so introducing a new feature requires a logging period first.
Continuous skew detection. Periodically replay logged serving vectors through the training pipeline's computation and compare the distributions. Any feature that differs is a defect. Alarm on it, because it will happen — a refactor, a schema change, a null-handling tweak — and by its nature it fails silently.
The framing that matters: skew is not a data-science problem to be modelled around. It is a software engineering problem about a single source of truth, and treating it as one is what separates machine-learning systems that stay accurate from ones that quietly decay over months while every dashboard stays green.
StaffEngagement is up 8% after a ranking change, but complaints are up and creator diversity is down. How do you decide whether to keep it?
Recognise the shape first: the target metric improved and the system got worse. This is the archetypal ranking failure, and the fact that it is detectable at all means the guardrails were set up correctly — most organisations discover this pattern from press coverage rather than from a dashboard.
First, characterise precisely. Engagement up 8% — decomposed how? More sessions, longer sessions, or more actions per session? Concentrated in which segments? Engagement driven by outrage, by novelty, or by genuine satisfaction all produce the same number, and the decomposition distinguishes them. Check dwell time against click rate, because clicks without dwell signal bait. Check return rate the next day and the next week, because short-term engagement that reduces next-week returns is value destruction with a lagging signal. And check negative feedback — hides, reports, unfollows.
Complaints up — categorise them, since complaints about content quality mean something entirely different from complaints about a layout change. Creator diversity down — measure it properly: share of impressions going to the top 1% of creators, count of distinct creators receiving meaningful distribution, and the survival rate of new creators.
Second, apply the framework that resolves this class of dispute. Measure against the objective the product actually holds, not the metric that was easiest to instrument. Engagement is a proxy for user value — it always is — so evidence that value fell while the proxy rose is evidence that the proxy has decoupled. That is a reason to reject the change and to fix the objective function, because the next change will exploit exactly the same gap.
Third, run the tests that distinguish the hypotheses. A long-horizon holdback measured in weeks separates novelty from durable improvement, and most ranking wins shrink substantially by the third week. A retention analysis on the treated cohort is the single strongest signal, because users who are genuinely better served come back. And a creator-side experiment checks whether reduced distribution changes posting behaviour — a supply-side collapse that no viewer-side metric captures until the content pool itself degrades months later, and one of the clearest cases of a network effect invalidating a naive comparison.
Fourth, decide with the weights rather than with the model. Because the blend is owned configuration, the response is not "revert or ship" but re-weight: raise the diversity and creator-fairness terms, increase the negative weight on reports and hides, and measure again. If engagement then lands at +3% with diversity flat and complaints flat, that is a better outcome than +8% and should be recognised as one. The fact that the system can express that trade at all is precisely why the weights belong in configuration with an owner.
Fifth, institutionalise the guardrails. Make creator diversity, complaint rate and next-week return blocking guardrails in the experiment platform, so that a change regressing them cannot ship on target-metric strength alone regardless of who is asking for it.
The statement for leadership: engagement is a proxy, and any sufficiently powerful optimiser will eventually find the gap between the proxy and the thing it stands for. So the ranking system's design must include guardrails with veto power and a permanent unbiased holdback — because the alternative is discovering that gap from users, from regulators, or from the press.
Flashcards
FlashThe funnel and its arithmetic
60 ms ÷ 1,000 candidates = 60 µs each including features. Retrieval (millions→1k, recall not precision) → light rank (1k→200) → heavy rank (200) → re-rank top 20 for policy. Each stage ~100× costlier per item, ~5–10× fewer items.
FlashWhere the latency really goes
Feature fetching, not inference: 50,000 values per request. Batched multi-gets, user features fetched once per request, co-located store. A feature that cannot be served in budget is dropped from the model.
FlashSkew, and the four defences
One definition materialised to both stores by the same code · point-in-time correctness so training never sees the future · log the vector actually served and train on it · continuous replay-and-compare. It is a single-source-of-truth problem, not a modelling one.
FlashBlend weights
score = w₁·P(click) + w₂·P(engagement) − w₃·P(report) + freshness + diversity. The weights are owned configuration, never learned, so wrong outcomes are correctable by a person and "why this item?" is attributable.
FlashFeedback loops
A model trained on its own outputs converges on its own biases, and content never shown generates no data. Defences: exploration traffic, a permanent randomised holdback, position-bias correction.
FlashWhat to monitor
Not latency and errors — they show nothing here. Prediction distribution drift · feature drift with default rates · skew by replay · calibration. Plus degraded and featureCompleteness on every response.
Scenario Drill
DrillDesign the rollout of a brand-new ranking model to a product that currently ranks chronologically, with 50 million users. Give the full plan and what could go catastrophically wrong.
Phase zero — instrument before changing anything. You cannot evaluate a ranker without a baseline, so the first shipment is logging: impressions with position, engagement events joined to impressions, negative feedback, session boundaries, and creator-side distribution metrics. Establish current values for every metric you intend to move and every guardrail. This phase is unglamorous and it is the one most often skipped, which is why so many ranking launches cannot answer "compared to what?"
Phase one — shadow mode. Production still serves chronological; the ranker scores the same candidates in parallel and logs its ordering. This measures latency, which is the requirement that most often kills a design — a ranker adding 300 milliseconds to a 200-millisecond budget is unshippable regardless of quality. It measures infrastructure cost. It measures feature availability, which is almost always worse than expected. And it lets you compute offline metrics against real traffic. No user is affected, so it is free of risk and rich in information.
Phase two — a tiny live ramp, one per cent, with automatic rollback on guardrail regression. Watch latency, error rate and fallback rate first; quality metrics need days.
Phase three — staged ramp, five per cent to twenty to fifty, with a minimum soak of one week per stage, because novelty effects decay over roughly that horizon and a change evaluated in two days is measuring novelty rather than value.
Phase four — a permanent holdback, never one hundred per cent. Retain a randomised one to five per cent on chronological forever: as unbiased training data, as honest long-term measurement, and as a continuously exercised fallback path. And keep chronological available as a user-facing option, because a meaningful fraction of people want it and several jurisdictions now require it to exist.
What could go catastrophically wrong, in descending order of likelihood.
Latency blows the budget and the whole feed slows. Mitigated by the funnel, hard per-stage deadlines, and returning the best answer within the budget rather than the best possible answer.
The fallback path is untested and fails when first needed, so the ranker's outage becomes the feed's outage — precisely the thing the availability requirement existed to prevent. Mitigated by the holdback exercising it continuously, plus periodic deliberate fallback drills.
The feedback loop closes immediately. The ranker's first-day behaviour shapes tomorrow's training data, so an early bias compounds daily. Mitigated by exploration traffic from day one and by the holdback providing unbiased data throughout — both added at the start, because neither can be retrofitted meaningfully.
Optimising the wrong thing at scale. A metric that looked fine in a one per cent test starts driving creator behaviour at fifty per cent, and the content supply itself changes. Mitigated by creator-side metrics as blocking guardrails and by long-horizon measurement.
A silent quality collapse from a stale or broken feature. The model keeps returning scores, nothing errors, and the feed slowly becomes noise. Mitigated by feature-freshness alarms, default-rate monitoring and prediction-distribution drift detection — none of which appear on a standard service dashboard.
Irreversibility. After months, users have adapted, creators have adapted, and the chronological baseline no longer measures a comparable population. This is the deepest argument for the permanent holdback: it is the only way to retain a comparison that stays meaningful.
The paragraph for the design document: instrument, shadow, ramp slowly with blocking guardrails, keep a permanent randomised holdback and a maintained chronological fallback, and treat latency and feature health as launch-blocking requirements equal to quality — because a ranking system fails silently, adapts its own training data, and reshapes the behaviour of the people it ranks, and none of those three are recoverable by rolling back a deploy.