Appearance
12.12.6 — Recommenders and Ranking
When a customer opens the Cartwise app, the home screen shows twenty products under "Recommended for you". Cartwise sells 400,000 products to 30 million customers, and the twenty must be chosen for this customer, from everything, in less than a fifth of a second. About a third of Cartwise's revenue starts with a tap on one of those twenty, or on the "Customers also bought" row under every product page.
Choosing them is the job of a recommender system, and it is probably the most commercially important kind of machine learning there is: it chooses what you see on shopping sites, video and music services, news feeds and app stores. It is also an ideal last stop for this chapter, because it brings together nearly everything in it: embeddings, vector search, classifiers, latency budgets, caching, the model lifecycle, monitoring, and a feedback loop that, left alone, slowly ruins the system's own data.
1. Why it is hard: you cannot score everything
The obvious design is to take a model that predicts how likely this customer is to buy a product, score all 400,000 products, and show the top twenty. Suppose the model is fast and scores one product in a millisecond. Scoring all of them takes 400{,}000 \times 1\ \text{ms} = 400 seconds, about seven minutes, for one screen. The budget is 200 milliseconds.
The way out is to stop trying to score everything accurately. Instead, a recommender narrows the field in stages, each stage using a more careful and more expensive method on fewer products than the one before. This shape is called a funnel.
The stages have different jobs, and that is the key to designing each one.
Retrieval must find a few thousand products that include the best ones, quickly, without scoring all 400,000. It is judged by recall: of the products this customer would actually buy, what share made it into the 2,000? A product that retrieval misses can never be shown, however good the later stages are. Several different retrieval sources run in parallel and their results are combined, because each finds products the others miss.
Ranking must order the survivors well. With only a few hundred products left, it can afford a rich model that looks at this customer and each product together. It is often split into a cheap light ranker and an expensive heavy ranker, so that the expensive model only sees a few hundred products.
Re-ranking turns a sorted list into a good screen: not twenty phone cases, nothing out of stock, nothing already bought, sponsored products in their labelled places.
Sections 2 to 5 take the stages in turn, then sections 6 to 9 cover the problems every recommender meets: new customers and products, measuring quality, the feedback loop, and serving it all inside 200 milliseconds.
2. Retrieval, the classic way: "customers also bought"
The oldest retrieval method is still one of the best. In 2003, Greg Linden, Brent Smith and Jeremy York described how Amazon generated its recommendations. Earlier systems compared customers with each other ("people like you bought..."), which was slow with millions of customers. Amazon's method compared products: two products are related if many customers bought both. This item-to-item collaborative filtering can be computed offline for every product, so serving it is a lookup. "Collaborative" because it uses the collective behaviour of all customers, not any description of the products.
Here is the idea in a few lines, on five tiny shopping baskets:
python
from collections import Counter, defaultdict
from math import sqrt
baskets = [["phone", "case", "charger"], ["phone", "case"], ["phone", "earphones"],
["case", "screen guard"], ["phone", "case", "screen guard"]]
count = Counter(item for b in baskets for item in b) # (1)
together = defaultdict(Counter)
for b in baskets:
for x in b:
for y in b:
if x != y:
together[x][y] += 1 # (2)
def also_bought(item):
scores = {y: c / sqrt(count[item] * count[y]) for y, c in together[item].items()} # (3)
return sorted(((y, round(s, 3)) for y, s in scores.items()), key=lambda p: -p[1])
print(also_bought("phone"))
# [('case', 0.75), ('charger', 0.5), ('earphones', 0.5), ('screen guard', 0.354)]Line (1) counts how many baskets contain each product: the phone appears in 4, the case in 4, the screen guard in 2, the charger and earphones in 1 each. Line (2) counts, for every pair of products, how many baskets contain both. Line (3) turns those counts into a similarity. Dividing by the square root of the two products' popularity, which is the cosine similarity of 12.6.2 applied to purchase records, stops the most popular products from appearing as "related" to everything merely because they are in many baskets.
The phone and the case appear together in 3 baskets, and each appears in 4, so their score is 3 / \sqrt{4 \times 4} = 0.75. The screen guard appears with the phone only once, and is in 2 baskets, so 1 / \sqrt{4 \times 2} \approx 0.354. At Cartwise's scale, the same counts are computed nightly over millions of orders, kept for the top few hundred related products of each product, and looked up in a millisecond.
Matrix factorisation and the Netflix Prize
The next big step came from a public contest. In 2006 Netflix offered a million dollars to anyone who could predict its customers' film ratings 10% more accurately than its own system. The prize was won in 2009 by a team called BellKor's Pragmatic Chaos, and the contest's lasting lesson came from the technique most leading teams relied on: matrix factorisation.
Imagine a huge table with one row per customer and one column per product, holding a number where the customer has interacted with the product and empty everywhere else. Almost all of it is empty. Matrix factorisation learns a short list of numbers, a vector, for every customer and every product, chosen so that the dot product of a customer's vector and a product's vector predicts the entry in the table. It is the low-rank idea of 12.11.2 applied to the table: a vast table explained by two thin matrices. The learned vectors are embeddings in exactly the sense of 12.6.1, and products with similar vectors are bought by similar customers. Netflix, famously, never deployed the full winning system: the extra accuracy of the prize-winning blend of over a hundred models was not worth its engineering cost, a lesson about measuring business value that holds for every model in this chapter.
3. Retrieval with two towers
Matrix factorisation knows only customers and products it has seen before, and uses only the table of interactions. The two-tower model generalises it with neural networks, and is the standard learned retrieval method today.
It has two separate networks. The customer tower takes everything known about the customer, such as recent views, purchases, searches, city and device, and produces a vector of, say, 64 numbers. The product tower takes everything known about a product, such as its category, brand, price, title and description, and produces a vector of the same size. The score for a customer and a product is the dot product of their two vectors, from 12.2.1:
\text{score}(c, p) = \mathbf{u}_c \cdot \mathbf{v}_p = \sum_{i=1}^{d} u_i v_i
where \mathbf{u}_c is the customer's vector, \mathbf{v}_p the product's, and d their length.
A three-number example shows what the vectors come to mean. Suppose training has made the three positions mean, roughly, "likes gadgets", "likes kitchenware" and "price-conscious". A customer's vector is (0.8, -0.2, 0.5): likes gadgets, not kitchenware, watches prices. A budget wireless speaker has the vector (0.9, -0.5, 0.8) and a copper cookware set has (-0.4, 0.9, -0.1). The scores are
(0.8)(0.9) + (-0.2)(-0.5) + (0.5)(0.8) = 0.72 + 0.10 + 0.40 = 1.22
(0.8)(-0.4) + (-0.2)(0.9) + (0.5)(-0.1) = -0.32 - 0.18 - 0.05 = -0.55
so the speaker is a strong candidate and the cookware is not. Real towers learn 64 or more such positions, and none of them has a tidy meaning; the example only shows how a dot product combines them.
Why two separate towers
The design's whole point is in one restriction: the two towers never see each other's inputs until the final dot product. A product's vector therefore does not depend on which customer is asking. That allows the product tower to be run offline for all 400,000 products, once a night or whenever a product changes, and the vectors to be stored in a vector index (12.6.3). When the customer opens the app, only the customer tower runs, once, in about a millisecond, and the index returns the products with the highest dot products in a few more. Scoring 400,000 products becomes one network call and one nearest-neighbour search.
The price of this restriction is that the towers cannot model interactions, such as "this customer likes gadgets, but only below ₹3,000", that depend on customer and product together. That is the ranker's job, which is why retrieval and ranking are separate stages.
Training it
The towers are trained on pairs of a customer and a product they bought, viewed or added to their cart. To learn what not to recommend, the model also needs negative examples, and the standard trick uses the other pairs in the same training batch: if customer 1 bought product A and customer 2 bought product B, then for customer 1, product B serves as a negative example. The model is pushed to give each customer a higher score for their own product than for everyone else's:
python
import torch
import torch.nn.functional as F
def in_batch_loss(customer_vecs, product_vecs, temperature=0.05):
scores = customer_vecs @ product_vecs.T / temperature # (1)
targets = torch.arange(len(customer_vecs), device=scores.device) # (2)
return F.cross_entropy(scores, targets) # (3)Line (1) computes the score of every customer in the batch against every product in the batch, a square table, and divides by a small temperature, which sharpens the differences between scores as in 12.7.7. Line (2) says that the right product for customer i is product i, on the diagonal of the table. Line (3) applies the cross-entropy of 12.2.6 to each row, pushing the diagonal score up and the others down. This is a form of contrastive learning, the same training idea behind the sentence embeddings of 12.6.2.
One refinement matters in practice. Popular products appear in many batches, so they are used as negatives far more often than rare ones, and the model learns to push them down unfairly. Production systems correct for this by adjusting each score by how often its product appears in training, a fix often called logQ correction after the logarithm of the product's sampling probability that it subtracts.
4. Ranking: customer and product together
The ranker scores a few hundred candidates, so it can use features that describe the customer and the product together: has this customer bought from this brand before, how does this price compare with what they usually spend, did they view this exact product yesterday, how far is the warehouse from their address, is it in their size. These cross features are what retrieval could not use, and they are where much of a recommender's accuracy comes from.
The model is often gradient-boosted trees from 12.3.4, strong and fast on this kind of tabular feature, or a neural network for large systems with many embedding features. Whatever the model, the important design decision is what it predicts.
A naive ranker predicts clicks. But a click is not what Cartwise wants: a product that attracts curious clicks and is never bought wastes a slot, and a product that is bought and returned costs money. So Cartwise's ranker predicts several things at once, in one network with several outputs, called multi-task learning: the probability of a click, of adding to cart, of a purchase, and of a return if purchased. They are combined into one score that reflects value to the business and the customer:
\text{score} = P(\text{buy}) \times \text{margin} - P(\text{buy}) \times P(\text{return} \mid \text{buy}) \times \text{cost of a return}
Read it as the expected profit from showing the product: the chance of a sale times what the sale earns, minus the chance of a sale that comes back times what a return costs. The combination is a business decision, written down openly and changed deliberately, not something hidden inside a model. When Cartwise first added the return term, recommendations of cheap, low-quality earphones, which were bought often and returned often, fell sharply, and profit per recommendation rose.
If the most accurate ranker is too slow for the 60-millisecond budget, a smaller model is trained to imitate it, the distillation of 12.7.8: the large model's scores on millions of examples become the small model's training targets, and the small model keeps most of the accuracy at a fraction of the time.
5. Re-ranking: from a list to a screen
The ranker's top fifty, sorted by score, would make a poor screen. They might be eleven phone cases and nine chargers, because a customer who just bought a phone scores highly on all of them. Re-ranking applies the rules that make the screen good.
- Filter: remove out-of-stock products, products the customer already bought (unless they are bought repeatedly, like groceries), products not deliverable to their address, and anything blocked by policy.
- Diversify: a common method, maximal marginal relevance, picks products one at a time, each time choosing the one with the best balance of its own score and its difference from those already picked, so the screen covers several interests.
- Business rules: sponsored products in their labelled positions, limits on how many products from one seller appear, and placement for new products being given a chance (section 6).
6. Cold start: new customers and new products
Collaborative methods learn from behaviour, so they know nothing about a customer or product with no history. This is the cold-start problem, and it has different answers on each side.
A new customer gets recommendations from what is known: products popular in their city and with customers on similar devices, then, within the same session, products related to whatever they view and search. Session behaviour is a strong signal: three views of running shoes in two minutes say more than a year of old history.
A new product has no purchases, so item-to-item methods cannot recommend it, and without being recommended it never gets purchases. The two-tower model helps, because the product tower uses the product's description: a new pair of running shoes gets a vector near other running shoes from its title, category, brand and price alone. Beyond that, new products are deliberately given some exposure, a small share of slots reserved for products with little history, so they can collect enough data to be judged. This is a form of the exploration of 12.12.4: spending a little attention now to learn something.
7. Measuring a recommender
Offline
Each stage is measured by what it is for. Retrieval is measured by recall at k: on historical data, of the products customers went on to buy, what share appeared in the top k candidates? Ranking is measured by how well it orders products, and the standard measure is NDCG, normalised discounted cumulative gain.
NDCG rewards putting the most relevant products near the top. Give each product shown a relevance: 3 if the customer bought it, 2 if they added it to their cart, 1 if they clicked, 0 otherwise. The discounted cumulative gain of a list adds up each product's value, discounted by its position:
\text{DCG} = \sum_{i=1}^{k} \frac{2^{\text{rel}_i} - 1}{\log_2(i + 1)}
The top position is divided by \log_2 2 = 1, the second by \log_2 3 \approx 1.585, the third by \log_2 4 = 2, and so on, so a purchase in position 1 counts far more than one in position 5. Suppose a list of five products had relevances (0, 2, 3, 0, 1):
\text{DCG} = 0 + \frac{3}{1.585} + \frac{7}{2} + 0 + \frac{1}{2.585} \approx 1.893 + 3.5 + 0.387 = 5.78
The best possible order of the same products is (3, 2, 1, 0, 0), whose DCG is 7 + 1.893 + 0.5 = 9.39. NDCG divides one by the other: 5.78 / 9.39 \approx 0.62. A perfect ranking scores 1. This list lost a lot by putting the purchased product third, below one the customer ignored.
Online, which is what counts
Offline measures have a basic weakness for recommenders: the historical data records only how customers responded to what the old system showed them. A product the old system never showed has no purchases, not because nobody wanted it but because nobody saw it. So a new recommender is judged by an A/B test (12.12.4) on outcomes that matter. The main ones are revenue and profit per visit, and purchases. Next to them sit guard metrics, numbers that must not get worse, such as return rates, the variety of products shown, and complaints. The test must run long enough to get past the first few days, when anything new looks exciting. Long-term effects, such as whether customers come back next month, deserve watching after the test ends.
8. The feedback loop
A recommender chooses what customers see, and what customers see is what they click and buy, and that becomes the training data for the next model. This loop distorts the data in two ways, and left alone, it slowly narrows the system.
Position bias. Products shown at the top are clicked more partly because they are at the top. A model trained naively on clicks learns that whatever it ranked first was good, which is partly just its own decision echoing back. Popularity bias is the same effect at a larger scale: popular products are shown more, so they are bought more, so they look more popular, while good products that were never shown never get the chance. Over time, recommendations converge on the same bestsellers for everyone.
Two remedies are standard. Inverse propensity weighting corrects the training data for how likely each product was to be seen. If customers look at the top position about 90% of the time and at the tenth only 15% of the time, a click at position 10 is weighted by 1 / 0.15 \approx 6.7 and a click at position 1 by 1 / 0.9 \approx 1.1, so a product that earned clicks despite a poor position gets credit for it. The propensity, the probability of being seen, is estimated from experiments that randomly shuffle positions for a small share of traffic. Exploration keeps the data honest: a small share of slots shows products the model is unsure about, or randomly chosen good candidates, so that the system keeps learning about products and interests it would otherwise never show. These are the same ideas as the fraud model's exploration in 12.12.4, for the same reason: a model learns only from the outcomes of what it allows.
9. Serving it in 200 milliseconds
The whole funnel runs for every home-screen load, inside the budget:
| Step | Time |
|---|---|
| Fetch customer features (feature store, 12.4.4) | 10 ms |
| Customer tower | 2 ms |
| Retrieval sources in parallel (vector search, lookups) | 30 ms |
| Merge candidates, fetch product features | 20 ms |
| Light ranker on ~2,000 | 20 ms |
| Heavy ranker on ~300 | 60 ms |
| Re-rank and rules | 10 ms |
| Network and margin | 48 ms |
| Total | 200 ms |
Much of the work is done before the request arrives: product vectors, "also bought" lists and many features are computed offline and stored for fast lookup. The parts that depend on what the customer did a minute ago are computed live. And each stage has a fallback, using the circuit breakers of Chapter 10.9. If the heavy ranker is slow or down, the light ranker's order is used; if personalised retrieval fails, products popular in the customer's city are shown. A slightly worse screen beats an empty one, and the home page must never wait on a struggling model.
10. Where language models fit
Language models are changing recommenders at the edges rather than replacing the funnel. They are used to understand products: writing consistent attributes, tags and descriptions for products whose seller-provided data is poor, which feeds the product tower and improves cold start. They power conversational shopping, where a customer describes what they need ("a gift for my father who likes gardening, under ₹2,000") and an assistant like those of Chapter 12.10 searches and explains, calling the same retrieval and ranking systems as tools. And they can explain recommendations in plain words. The heavy lifting of choosing twenty products from 400,000 for 30 million customers in 200 milliseconds is still done by the funnel, because that is a task where speed and scale decide, and where the models of this page are cheaper, faster and well measured.
What the interviewer will push on
"Design a recommendation system for an e-commerce home page." This is one of the most common ML system design questions, and the funnel is the answer they are looking for: several retrieval sources for recall, a light and a heavy ranker, re-ranking for diversity and rules, with numbers for each stage's candidates and latency. Strong answers state the objective (expected profit, not clicks), the features (cross features in ranking), cold start, offline and online evaluation, the feedback loop, and fallbacks.
"How does a two-tower model work, and why two towers?" Separate customer and product networks meet only at a dot product, so product vectors are independent of the customer, precomputed and indexed for nearest-neighbour search; only the customer tower runs live. The cost is no interaction features, which the ranker supplies. Mention in-batch negatives and the popularity correction.
"Why not optimise for clicks?" Clicks reward curiosity, not value; multi-task predictions of click, cart, purchase and return, combined into expected profit, align the ranker with the business. Mention the return term demoting frequently returned products.
"What is NDCG?" A ranking measure that sums each item's gain, 2^{\text{rel}} - 1, discounted by \log_2(\text{position} + 1), normalised by the best possible ordering. Being able to compute it for a five-item list is a strong signal.
"What feedback loops affect recommenders, and how do you handle them?" Position and popularity bias from training on the system's own exposures; correct with inverse propensity weighting from randomised position experiments, and keep exploration traffic so unseen products and interests are still learned about.
Something to volunteer: offline metrics are biased towards the old system, because history only records responses to what it chose to show. That is why recommenders are judged by online experiments, and why a small amount of randomised traffic is worth its cost.
Recall
- A recommender system cannot score every product within the latency budget, so it uses a funnel: retrieval (several parallel sources, judged by recall), a light and a heavy ranker, and re-ranking with rules.
- Item-to-item collaborative filtering (Amazon, 2003): products are related if bought together, scored by co-occurrence divided by \sqrt{\text{count}_x \times \text{count}_y}. Matrix factorisation (the Netflix Prize, 2006–2009) learns customer and product vectors whose dot product predicts interactions.
- The two-tower model has separate customer and product networks meeting at a dot product; product vectors are precomputed and indexed, only the customer tower runs live. Trained with in-batch negatives and cross-entropy, with a popularity correction.
- Ranking uses cross features and predicts several outcomes (click, cart, purchase, return) combined into expected profit; distil a large ranker if it is too slow.
- Re-ranking filters (stock, already bought, deliverable), diversifies (maximal marginal relevance) and applies business rules.
- Cold start: new customers by popularity and session behaviour; new products by content features in the product tower and reserved exploration.
- Offline: recall at k for retrieval, NDCG for ranking. Online A/B tests on profit and guard metrics decide. Feedback loops (position and popularity bias) are corrected by inverse propensity weighting and exploration. Serve in 200 ms with precomputation and per-stage fallbacks.
Self-test: In the basket example, the screen guard and the charger each appear with the phone exactly once; why is the screen guard's score lower? A list has relevances (3, 0, 0, 2, 0); what is its NDCG? Why can the two-tower model not learn "likes gadgets, but only below ₹3,000", and which stage handles it?
Next: 12.13.1 Vision and Vision-Language Models — every system so far has worked mainly with text and numbers; Chapter 12.13 turns to images, speech and translation, starting with how models see.