12.15.2 — Case Study: Semantic Product Search

Three searches typed into Cartwise one afternoon:

• "a54 charger": a customer who knows exactly what they want, down to the phone model.
• "gift for dad who likes gardening under 2000": a customer who knows the situation but not the product.
• "jute ki bag bada wala": Hindi written in English letters, "a big jute bag", with no product title on the site containing any of those words except "jute" and "bag".

The old search, keyword matching over product titles, handled the first well, returned a jumble of garden tools and greeting cards for the second, and returned nothing at all for the third. About one search in eight ended with no results, and customers who saw no results mostly left. Search is where shopping starts: about 3 million searches a day at Cartwise, and roughly half of all orders begin with one.

This case study applies the nine steps of 12.15.1 to rebuilding it. The retrieval techniques were taught in Chapter 12.6 and the ranking ideas in 12.12.6; this page is about putting them together into one system that meets real numbers, and about the parts those chapters did not cover: understanding messy queries, enriching a messy catalogue, measuring relevance, and what happens when nothing matches.

1. The problem and success, in numbers

Users and queries. 3 million searches a day; peak 120 a second during sales. About 15% of queries are in Hindi or another Indian language written in English letters, often mixed with English. Queries follow a steep curve: the 10,000 most common queries make up about 55% of all searches (the head), while the rest are a long tail of queries seen once or a few times.

Success. The business measure is search conversion, the share of searches followed by a purchase from the results, with revenue per search alongside. The operational measures: the zero-result rate, currently 12%, and the reformulation rate, how often a customer immediately searches again with different words, a sign the first results failed. Offline, relevance is measured by NDCG@10 (12.12.6) on judged queries.

Constraints. p95 latency under 200 milliseconds from query to results, because search results appear as the customer types; out-of-stock and undeliverable products never shown; sponsored products allowed in labelled positions only.

2. Which kind of ML, and the data

The three queries need different things. "a54 charger" needs exact matching on a model number, which keyword search does best and embeddings do poorly. "Gift for dad who likes gardening" needs meaning, which embeddings capture and keywords miss. "jute ki bag bada wala" needs query understanding: recognising the language, the product ("jute bag"), and an attribute ("large"). And all of them need ranking by what customers actually buy, which only behaviour data provides. So the design combines keyword search, vector search, a query-understanding step, and a learned ranker, the funnel of 12.12.6 with search queries in place of a customer's history.

The data has two sides, and the catalogue side is the one teams underestimate. Product data comes from 20,000 sellers and is inconsistent: "Wireless Bluetooth Earbuds TWS Earphone with Mic Black (Pack of 1)" next to "earphone". Categories are wrong, sizes are buried in descriptions, and almost nothing is written in Indian languages. Improving the catalogue improved search more than any model change, as section 4 shows. The behaviour side, which results customers clicked, added to cart and bought for each query, is plentiful, and carries the position and popularity biases of 12.12.6.

3. Understanding the query

Every query passes through a short understanding step before retrieval, which turns free text into a cleaner query plus structured parts.

Normalise: lower-case, standardise spacing and units ("2000rs", "₹2,000" and "2k" all become 2000), correct obvious misspellings with the fuzzy matching of 12.6.4.

Recognise language and transliterate: detect romanised Hindi and map common words to English equivalents ("bada" means large, "wala" marks a description and can be dropped, "ke liye" means "for"), and keep both forms for matching.

Extract structure: product type, attributes (colour, size, material), price limits, brand, and the shopping intent (a specific item, a gift, a comparison).

For head queries, the 10,000 that make up over half of all traffic, this is done offline, with a capable language model and structured output (12.8.3), checked by a merchandising team for the top few thousand, and stored in a lookup table:

from typing import Literal
from pydantic import BaseModel

class QueryIntent(BaseModel):
    product_type: str | None              # "jute bag", "phone charger"
    english_query: str                    # cleaned, in English, for matching
    attributes: dict[str, str]            # {"size": "large", "material": "jute"}
    max_price_rupees: int | None
    brand: str | None
    intent: Literal["specific_item", "browse", "gift", "compare"]

def understand_offline(query: str) -> QueryIntent:
    r = client.messages.parse(
        model="claude-sonnet-5-5", max_tokens=400,
        system=QUERY_GUIDE,                                         # (1) examples, units, Hinglish notes
        messages=[{"role": "user", "content": f"Shopping search query: {query}"}],
        output_format=QueryIntent)
    return r.parsed_output

head_table = {q: understand_offline(q) for q in top_queries(10_000)}  # (2) refreshed weekly

Line (1)'s system prompt carries the house rules: how to read prices, a glossary of common Hinglish shopping words, and twenty worked examples. Line (2) builds the table for the head queries once a week, so the live search path only looks them up, in under a millisecond. For "jute ki bag bada wala" the table holds product_type="jute bag", english_query="large jute bag", attributes={"size": "large", "material": "jute"} and intent="specific_item".

Tail queries cannot be prepared in advance, and a large model call would blow the 200-millisecond budget. They go through a small, fast model trained on the head table's outputs, distillation again (12.7.8), which runs in about 10 milliseconds on a CPU. Queries it is unsure about fall back to plain normalisation, and are logged so they can be prepared properly next week.

4. Fixing the catalogue

Search can only find what the catalogue describes. Two offline jobs improved the catalogue itself.

Attribute enrichment. A vision-language model (12.13.1) reads each product's title, description and main photo and fills in standard attributes: product type, colour, material, size, intended user, and a clean, consistent title. Sellers see the proposed values and can correct them, and values the model was unsure of are left blank rather than guessed. A filter for "material: jute" now actually works.

Search keywords in Indian languages. For each product, the enrichment job also writes the words customers might search for in Hindi and other languages, in both their own script and English letters: "jhola", "thaila", "bada thaila". These go into the keyword index alongside the title. This single change cut the zero-result rate on romanised Hindi queries by more than half, more than any change to retrieval did.

Both jobs run when a product is created or changed, not when a customer searches, so they add nothing to search latency. That is the offline-online split of 12.15.1 at work.

5. The search path

With the query understood and the catalogue enriched, the live path is the funnel of 12.6.4 and 12.12.6, each stage with a time budget.

The live search path. Both retrievers run at once with stock and delivery filters inside the search; their lists are fused, reranked and ranked by behaviour; rules shape the final page. Every stage has a budget and a fallback.

Retrieval, in parallel. Keyword search (BM25) over titles, enriched attributes and the Indian-language keywords catches model numbers and exact terms. Vector search with a multilingual embedding model, one trained so that "jute bag" and "jute ka thaila" land near each other (12.6.2), catches meaning. Both apply the stock and delivery filters inside the search (12.6.3), and structured parts of the query become filters too: max_price_rupees filters by price, and a recognised brand boosts that brand.

Fusion by reciprocal rank (12.6.4) merges the two lists without comparing their incomparable scores.

Reranking with a cross-encoder (12.9.3) reads the query and each of the top 100 products together and scores relevance. This is where "phone charger" stops outranking "A54 charger" for the first query, and where gardening gifts rise above greeting cards with a picture of a flower.

The learned ranker orders the most relevant products by what customers do: past conversion for this query or similar ones, price relative to the query's typical price, rating, delivery speed to the customer's postcode, return rate, and a light touch of personalisation (the customer's usual price range and preferred brands). It is a gradient-boosted model trained with a ranking objective on search logs, with clicks weighted for position bias as 12.12.6 explained. The objective is purchases net of returns, not clicks.

Rules finish the page: sponsored products in labelled slots, no more than three products from one seller in the top ten, near-duplicates collapsed.

Head-query results are cached for a few minutes, keyed on the normalised query, delivery region and filters (12.12.2), which serves a large share of traffic in a few milliseconds. And every stage has a fallback: if the reranker is slow, the fused order is used; if vector search fails, keyword results alone are shown. Search must never return an error page.

6. When nothing matches

Zero results is the worst outcome a search can produce, so the system tries, in order, before giving up:

1. Spelling: if a corrected query has results, show them under "Showing results for…", with a link back to the original.
2. Relax filters: drop the least important extracted attribute first (colour before product type), or widen the price limit by 20%, and say so.
3. Broaden: search the product type alone ("jute bag" rather than "large jute bag").
4. Show related categories and popular products in them, never an empty page.

Every zero-result query is logged, and the weekly review of the most common ones is one of the most useful jobs on the team. It finds missing products (customers searching for things Cartwise does not sell, a signal for the buying team), missing synonyms, and broken understanding.

7. Measuring relevance

Offline judgements. For a sample of 3,000 queries, each top-20 result is rated for relevance on a four-point scale: 3 exactly what was asked for, 2 a close substitute, 1 related, 0 irrelevant. These graded labels give NDCG@10. People rated the first 500 queries; after that, a model judge with a written rubric and the product's details rated the rest (12.8.6), after it was shown to agree with the human raters about as often as the raters agreed with each other. Every proposed change is measured against these labels before any customer sees it, and slices are measured separately: romanised Hindi queries, gift queries, model-number queries, long-tail queries.

Online experiments. Changes that pass offline go to an A/B test (12.12.4) measured on search conversion and revenue per search, with guard metrics for latency, zero-result rate, reformulation rate and return rate.

The changes, in the order Cartwise made them, with their measured effect on search conversion relative to the old keyword search:

Change	Conversion	Zero results
Hybrid retrieval with filters inside	+6%	12% → 7%
Cross-encoder reranking	+4%	no change
Catalogue enrichment and Indian-language keywords	+5%	7% → 3%
Query understanding (head table + small model)	+3%	3% → 2.5%
Learned ranker on purchases net of returns	+4%	no change
Zero-result recovery steps	+1%	2.5% → 0.4% shown empty

The pattern is worth noticing. The largest single gains came from getting the right products into the candidate set at all, through hybrid retrieval and a better catalogue, before any improvement in ordering. A perfect ranker cannot rank a product that retrieval never found.

8. What can go wrong

Changing the embedding model changes every vector, so the whole catalogue must be re-embedded and the index rebuilt before the new model serves queries; mixing old and new vectors silently degrades results (12.6.3). Cartwise builds the new index alongside the old one and switches over only after the offline evaluation passes.

Feedback loops: the learned ranker trains on clicks on results it chose, so it is corrected for position bias and given a small share of randomised ordering, as in 12.12.6.

Sellers gaming the system: once Indian-language keywords and enrichment existed, some sellers stuffed their titles with popular search terms. Titles are now generated or cleaned by the enrichment job, and keyword fields come only from enrichment, not from seller text. Seller text is untrusted input, as 12.14.1 put it.

Sale-day traffic: a fourfold spike is absorbed by the head-query cache and by scaling the stateless stages; the self-hosted reranker keeps a warm minimum of GPUs, with the fused order as a fallback (12.12.3).

What the interviewer will push on

"Design search for an e-commerce site." A strong answer covers these parts, in this order:

1. Understanding the query. Clean it up, fix spelling, detect the language, and pull out structured facts such as size or colour. Common queries are handled ahead of time, and rare ones live.
2. Finding candidates. Hybrid retrieval, with filters applied inside the search, and the two result lists fused into one.
3. Ranking. A cross-encoder reranker, then a learned ranker trained on purchases, corrected for the fact that people click whatever is on top.
4. Business rules on top of the ranking.
5. Speed. A time budget for each stage, a cache for common queries, and a fallback for each stage when it is slow.
6. No results. A plan for when the search finds nothing.
7. Measurement. Offline, with graded relevance and NDCG. Online, with conversion and how often people retype their search.

"Why not use only vector search?" It misses exact identifiers such as model numbers; keyword search misses meaning; hybrid gets both, and fusion by rank avoids comparing incompatible scores.

"How would you handle queries in mixed Hindi and English?" Transliteration and a glossary in query understanding, a multilingual embedding model, and above all Indian-language keywords added to the catalogue, which helps keyword search directly.

"How do you use LLMs in search without blowing the latency budget?" Offline: enrich the catalogue and prepare understanding for head queries. Online: a small distilled model for the tail. Large models never sit on the per-keystroke path.

Something to volunteer: fixing the catalogue beat every model change for zero results. Search quality is bounded by what the product data says, and making the data better is often cheaper and larger than making the ranking smarter.

Recall

• Measures: search conversion and revenue per search; zero-result and reformulation rates; offline NDCG@10 on graded judgements; p95 under 200 ms.
• Query understanding: normalise, correct spelling, detect and transliterate romanised Indian languages, extract product type, attributes, price, brand and intent. Head queries prepared offline by a capable model and cached; tail queries by a small distilled model.
• Fix the catalogue offline: VLM-based attribute enrichment, clean titles, Indian-language search keywords in both scripts. It cut romanised-Hindi zero results by more than half.
• Live path, in order:
  1. Keyword search and multilingual vector search, in parallel, with filters inside the search.
  2. Reciprocal rank fusion to merge the two lists.
  3. A cross-encoder reranker.
  4. A learned ranker trained on purchases minus returns, corrected for position bias.
  5. Business rules.
  6. A cache for common queries, and a fallback at every stage.
• Zero results: correct spelling, relax filters, broaden, show related categories; review zero-result queries weekly.
• Measure offline on graded labels (people then a validated model judge), by slice; then A/B on conversion with guard metrics. The biggest gains came from getting the right products into the candidate set.

Self-test: Which stage makes "a54 charger" find the right product, and which stops a generic charger outranking it? Why is the head-query table built offline? Why did Cartwise stop letting sellers' own text feed the keyword fields?

Next: 12.15.3 Case Study: The Customer Support Assistant — every thread of this Part meets in one system: "Ask Cartwise", from the first message to the refund, by chat and by phone, with its numbers after a year in production.