12.9.3 — Retrieval and Reranking

With clean chunks in the index from 12.9.2, "Ask Cartwise" still got some questions wrong, and when the team looked at which chunks had been retrieved for each failure, four patterns appeared:

1. "Is the Cartwise Plus free-returns benefit valid on sale items?" retrieved general sale-returns chunks but not the Cartwise Plus membership terms, because "Cartwise Plus" is a product name the embedding model treats as vaguely meaning "more".
2. "And what about chargers?", asked right after a question about earphones, retrieved nothing useful at all: on its own, the message means almost nothing.
3. "Can I get my money back for a broken kettle?" retrieved the right chunk, but in 7th place, behind six chunks about money-back offers and kettle features; only the top 5 were sent to the model.
4. Most seriously, a support-staff-only document about goodwill compensation limits was retrieved for a customer, and the assistant quoted it.

Each needs a different fix, and together they make up the retrieval stage of a production RAG system: hybrid search, query transformation, access filtering and reranking. The guiding idea is the retrieval funnel from 12.6.4: retrieve broadly and cheaply, then narrow down carefully.

The retrieval stage of a production RAG system. The question is rewritten, searched two ways with the permission filter applied inside each search, fused, and reranked with a slower, more accurate model before the best few chunks reach the generator.

1. Search two ways: hybrid retrieval

The Cartwise Plus failure is the classic weakness of embeddings from 12.6.4: exact names, codes and rare terms get blurred. The fix is the same as for product search: run keyword search (BM25) and vector search side by side and fuse their rankings with reciprocal rank fusion. "Cartwise Plus" is a rare, exact phrase, so BM25 ranks the membership terms near the top, and fusion carries them into the candidate set even though vector search did not. Hybrid search is the default for any RAG system serving real users. Each search fetches more candidates than you will finally use, typically 20 to 100, because the reranker in section 4 will choose among them.

2. Filter by permission, inside the search

The goodwill-compensation leak is a security failure, and its fix is not a prompt instruction. It is tempting to tell the model "do not reveal internal documents to customers". That is not access control: the model has already read the document, and an instruction is a request it may not follow, especially when a customer asks cleverly. Whatever is in the context can end up in the answer. The only reliable control is to make sure restricted chunks never reach the context at all, which is the principle of Chapter 8.1: judge a control by what an attacker can still reach when it fails.

So every chunk carries access metadata from 12.9.2, and every search is filtered by what this user may see:

def allowed_filter(user) -> dict:
    if user.is_staff:
        return {"audience": {"in": ["public", "staff"]}}                  # (1)
    return {"audience": "public",                                          # (2)
            "or": [{"customer_id": None}, {"customer_id": user.id}]}

def retrieve_candidates(query, user, k=50):
    f = allowed_filter(user)
    kw = keyword_index.search(query, filter=f, limit=k)                    # (3)
    vec = vector_index.search(embed_query(query), filter=f, limit=k)
    return rrf(kw, vec)                                                    # (4)

Line (1) lets support staff see public and staff-only documents. Line (2) limits customers to public documents, plus any documents belonging to them, such as their own order records, and never another customer's. Line (3) applies the same filter inside each search, so restricted chunks are never candidates. Line (4) fuses the two lists. Filtering inside the search, rather than retrieving first and removing restricted chunks afterwards, matters for the reason in 12.6.3: if the top 50 happened to be all staff-only, filtering afterwards would leave nothing. And the filter must be built from the authenticated user on the server, never from anything the user or the model can set.

Chapter 12.14.2 works through the multi-tenant version of this problem, where many businesses share one index.

3. Fix the question before searching

Users write messages, not search queries. Several cheap transformations turn one into the other.

Rewrite follow-ups into standalone questions. "And what about chargers?" is meaningless on its own, but obvious in context. Before retrieving, a fast model is asked to rewrite the latest message using the conversation so far:

REWRITE = """Rewrite the customer's last message as a standalone search query
for Cartwise's help documents, using the conversation for context.
Return only the query.

<conversation>{history}</conversation>
<last_message>{message}</last_message>"""
# "And what about chargers?"  →  "return window and conditions for chargers (electronics)"

For any multi-turn assistant, this step is not optional; it is the fix for the common complaint "it answers the first question well and then gets lost". It costs one small, fast model call.

Generate several phrasings. Ask a model for three alternative wordings of the question, retrieve for each, and fuse the results. Different wordings match different chunks, so coverage improves, at the cost of more searches.

Decompose compound questions. "Compare return windows for electronics and clothing" becomes two searches, one per category, whose results are combined.

Search with a hypothetical answer. A question and the document that answers it are written differently: the question is short and asks, the document is longer and states. HyDE (hypothetical document embeddings, from a 2022 paper by Luyu Gao and colleagues) asks a model to write a plausible answer first, without any sources, and then searches with the embedding of that answer. The hypothetical answer may be wrong in its details, but it looks like the kind of passage that would answer the question, so it lands nearer the right chunks. It helps most on collections of long prose documents.

Route. Some questions should not go to the document index at all. "Where is order 4471?" needs the order-status tool from 12.8.4, not the policies; "what's 15% off ₹2,499?" needs a calculator. A small classifier or a quick model call decides which source to use, which Chapter 12.10.6 develops into general routing.

4. Rerank: choose the best few carefully

The broken-kettle failure, the right chunk found but ranked 7th, is about the precision of the ranking. Embedding search ranks candidates by comparing two vectors that were computed separately, one for the question and one for the chunk. The model that made them never saw the two texts together. That is called a bi-encoder, because it encodes the two sides independently, and it is what makes search over millions of chunks fast: chunk vectors are computed once, in advance.

A cross-encoder works the other way. It feeds the question and one chunk together, as a single input, through a transformer, and outputs one relevance score for the pair. Because attention, from 12.7.3, can connect every word of the question with every word of the chunk, it can notice that "money back for a broken kettle" is about a refund for a faulty appliance, and that a chunk about kettle features is not. It is far more accurate, and far too slow to run over the whole index, because nothing can be computed in advance: every question–chunk pair needs its own pass.

Why retrieval and reranking use different models. A bi-encoder compares vectors computed separately, so chunk vectors can be precomputed and searched fast. A cross-encoder reads the question and a chunk together, so it judges relevance far better but must run for every pair.

So the two are used together: the bi-encoder (and BM25) finds 50 candidates in milliseconds, and the cross-encoder scores just those 50 and keeps the best 5. Fifty passes of a small cross-encoder take tens of milliseconds on a GPU, which fits the latency budget. In practice, reranking is usually the biggest single quality improvement after hybrid search.

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")        # (1)

def rerank(question: str, candidates: list[dict], keep: int = 5) -> list[dict]:
    pairs = [(question, c["text"]) for c in candidates]                 # (2)
    scores = reranker.predict(pairs)                                     # (3) one score per pair
    order = sorted(range(len(candidates)), key=lambda i: -scores[i])
    return [candidates[i] for i in order[:keep]]                         # (4)

Line (1) loads a small, widely used open cross-encoder, trained on a large dataset of web search questions and relevant passages called MS MARCO. Line (2) pairs the question with each candidate's text. Line (3) scores every pair in one batch. Line (4) keeps the top five. Hosted reranking APIs from several providers do the same job without running a model yourself, and a capable LLM can also act as a reranker by being asked to score or order candidates, which is slower and more expensive but needs no extra model.

Avoid five copies of the same thing

Sometimes the top five candidates are nearly identical: the same paragraph from three versions of a help article. Sending all five wastes the context on one fact. Maximal marginal relevance, MMR, from a 1998 paper by Jaime Carbonell and Jade Goldstein, picks results one at a time, rewarding relevance to the question and penalising similarity to what has already been picked:

\text{MMR}(c) = \lambda \cdot \text{rel}(c) - (1 - \lambda) \cdot \max_{s \in \text{picked}} \text{sim}(c, s)

Read it as "lambda times the candidate's relevance, minus one-minus-lambda times its similarity to the most similar already-picked result". With \lambda = 0.7: suppose chunk A (relevance 0.90) is picked first. Chunk B has relevance 0.88 but is a near-copy of A (similarity 0.95), so its score is 0.7 \times 0.88 - 0.3 \times 0.95 \approx 0.33. Chunk C has relevance 0.80 and similarity 0.30 to A, scoring 0.7 \times 0.80 - 0.3 \times 0.30 = 0.47. C is picked next, adding new information, even though B was slightly more relevant. Deduplicating at ingestion (12.9.2) removes the worst of this; MMR handles what remains.

5. How many chunks to send

After reranking, how many chunks should reach the model? More is not better, for the reasons of 12.8.2: cost, latency, and the risk that the model latches onto a less relevant passage. Three to eight well-chosen chunks is typical for support questions. Two settings to tune separately with the evaluation of 12.9.5: how many candidates to retrieve (large enough that the right chunk is almost always somewhere in the set, so the reranker has a chance to find it) and how many to keep (small enough to stay focused). A useful habit is to drop chunks whose rerank score is below a threshold, so a question with only one relevant chunk sends one, not five.

What the interviewer will push on

"How would you improve retrieval quality?" They want a structured list in order of impact: hybrid search, query rewriting for multi-turn, reranking with a cross-encoder, chunking improvements, and measurement of recall at k to know which helped. A single "better embeddings" answer is weak.

"Bi-encoder versus cross-encoder?" Separate encodings that can be precomputed and searched fast, versus a joint encoding that is accurate but must run per pair; hence retrieve broadly with one, rerank narrowly with the other.

"How do you stop a RAG system leaking documents?" Access metadata on every chunk and a permission filter applied inside the search, built from the authenticated user on the server; and the explicit statement that a prompt instruction is not access control.

"Why does the bot fail on follow-up questions?" The follow-up embedded alone is meaningless; rewrite it into a standalone query using the conversation before retrieving.

Recall

• Retrieval is a funnel: rewrite the question, search broadly with BM25 and vectors under a permission filter, fuse, rerank precisely, send a few chunks.
• Hybrid search catches exact names and codes that embeddings blur; retrieve 20 to 100 candidates for the reranker to choose from.
• Access control means restricted chunks never reach the context: filter inside every search using metadata and the authenticated user. A prompt instruction is not access control.
• Transform questions: rewrite follow-ups into standalone queries (essential in conversations), generate several phrasings, decompose compound questions, search with a hypothetical answer (HyDE), and route questions to tools when documents are the wrong source.
• A bi-encoder encodes question and chunk separately (fast, precomputable); a cross-encoder reads them together (accurate, per pair). Rerank about 50 candidates down to about 5; use MMR to avoid near-duplicates; keep few, well-chosen chunks and tune both counts with evaluation.

Self-test: Why would post-filtering the top 50 for permissions sometimes return nothing? In the MMR example, what score would B get if its similarity to A were only 0.40, and would it be picked? Why can a cross-encoder not be used to search the whole index?

Next: 12.9.4 Grounded Generation and Citations — the right five chunks are in the prompt; the last step is making the model answer only from them, cite them correctly, and say so when they do not contain the answer.