12.6.2 — Retrieval-Augmented Generation

An internal assistant is asked about the refund policy. It answers confidently, quoting a 45-day window and a clause number. Neither exists. The real policy is 30 days, and the clause number is invented in the right format.

The model was not lying and was not broken. It was completing text, and a plausible policy is exactly what "completing text" produces when the actual policy was never in the context. The fix is not a better prompt. It is putting the real policy in front of the model before it answers.

Retrieval-augmented generation is that: fetch relevant material, put it in the prompt, and instruct the model to answer only from it. Simple in outline, and every step below is where systems actually fail.

1. Why retrieve rather than train

The knowledge cutoff. A model knows nothing after its training data ends.

Private data. Your policies, tickets, code and contracts were never in any training set.

Change. Documents update daily; retraining does not.

Attribution. A retrieval system can cite its sources. This is often the decisive reason — an answer a user can verify is usable in a way that an unattributed assertion is not.

Access control. Retrieval can filter by who is asking. A fine-tuned model cannot forget a fact for one user and remember it for another.

Compared with fine-tuning (Chapter 12.8): retrieval teaches facts, fine-tuning teaches behaviour. Use retrieval for knowledge, fine-tuning for format, tone and task-specific skill. Fine-tuning on your documents to make the model "know" them is the common expensive mistake — it is far worse at recall than retrieval, and it cannot be updated or attributed.

2. The pipeline

INDEXING (offline)
  load → clean → chunk → embed → store (vectors + text + metadata)

QUERYING (per request)
  rewrite → retrieve (vector + keyword) → filter by permission
         → rerank → assemble context → generate → cite

Two systems, and most quality problems live in the indexing half while most attention goes to the prompt.

3. Chunking, which decides more than the model

Documents must be split, because you cannot embed a 300-page manual into one vector and expect it to mean anything. How you split is the highest-leverage decision in the pipeline.

Fixed-size with overlap — 500 tokens with 50 overlapping. Trivial and it cuts sentences in half and separates a heading from its content.

Recursive by structure — split on the largest natural boundary that fits: sections, then paragraphs, then sentences. This is the sensible default, because it respects the document's own organisation.

Small-to-big (parent document) — embed small precise chunks for matching, but return the larger parent section to the model. This resolves the central tension directly: small chunks retrieve accurately, large chunks answer completely.

Contextual retrieval — before embedding, prepend a short generated description of where the chunk sits: "This section of the 2026 employee handbook covers refund eligibility." This fixes the most common retrieval failure, where a chunk says "the window is 30 days" with no indication of what window, and therefore matches nothing. It costs one cheap model call per chunk at index time, and published results show a large reduction in retrieval failures.

Practical rules:

• Keep tables intact. A table split across chunks is worse than useless — the header lands in one and the numbers in another.
• Keep code blocks intact, and keep the surrounding prose with them.
• Attach metadata to every chunk: source, title, section, date, permissions, URL. You need this for filtering, for citation and for deletion, and adding it later means re-indexing.
• Handle the document format properly. A badly extracted PDF — columns interleaved, headers repeated mid-sentence, tables flattened into noise — poisons everything downstream. Look at your extracted text before building anything on it; this is the single most skipped step and a common cause of a system that never worked.

4. Retrieval

Vector search finds semantic matches: "how do I get my money back" retrieves a passage about refunds with no shared words. Approximate nearest-neighbour indexes (HNSW, IVF, product quantisation — Chapter 12.5.2) make it sub-linear.

And it is weak exactly where keyword search is strong: exact identifiers, product codes, error numbers, rare names, and terms your embedding model never saw.

Hybrid search runs both and fuses the results. Reciprocal rank fusion is the standard method and needs no tuning:

\text{score}(d) = \sum_{r \in \text{rankers}} \frac{1}{k + \text{rank}_r(d)}

with k \approx 60. It combines rankings rather than scores, which is what makes it robust — BM25 scores and cosine similarities are not comparable quantities, and fusing them numerically is a common bug.

Hybrid is the default for any serious system. Pure vector search failing on a product code is the complaint that arrives in week two.

Filtering must happen at retrieval time, and it must be a pre-filter. Post-filtering — retrieve the top 50, then drop what the user may not see — can return nothing at all when the top 50 are all restricted. Pre-filtering restricts the search space first, which most vector stores support with metadata filters.

And say this one plainly: access control belongs in the retrieval query, not in the prompt. Telling the model "do not reveal documents the user cannot see" is not a security control (Chapter 8.4.10). Filter by permission before anything reaches the context, or you have built a data-leak endpoint with a natural-language interface.

5. Query transformation

The user's question is often a poor search query.

Rewriting — turn a conversational follow-up into a standalone query. "What about for business accounts?" is meaningless alone; rewritten against the history it becomes "What is the refund window for business accounts?" In any multi-turn system this is mandatory, and it is the fix for "it works on the first question and fails on the second".

Multi-query — generate three phrasings, retrieve for each, and merge. Cheap coverage improvement.

Decomposition — split a compound question ("compare the refund policies for consumer and business") into sub-queries and retrieve for each.

HyDE — ask the model to write a hypothetical answer, then embed that and search with it. A hypothetical answer looks more like a document than a question does, which improves matching. It costs an extra call and works well when the corpus is dense prose.

Routing — classify the query and send it to the right index or tool. A question about a person goes to the directory, a policy question to the handbook, an arithmetic question to a calculator.

6. Reranking

Retrieve broadly, then rerank precisely. Fetch 50 candidates by hybrid search and rerank them to the best 5.

Why this works is a real distinction. The embedding model is a bi-encoder: it encodes query and document separately, so it can precompute document vectors — fast, and it never compares the two texts directly. A cross-encoder reranker feeds the query and document together through a model and scores the pair, which is far more accurate and far too slow to run over a million documents.

So the shape is: cheap and broad, then expensive and narrow. It is the same idea as a database's bitmap scan followed by a recheck (Chapter 7.2.3).

Reranking is usually the largest single quality improvement available after hybrid search, and it costs one extra call with a small model.

7. Generation

Answer the question using only the sources below.
Cite sources as [1], [2] after each claim.
If the sources do not contain the answer, reply exactly: NOT_FOUND

[1] {{chunk_1_text}}   (source: handbook.pdf, section 4.2)
[2] {{chunk_2_text}}   (source: policy-2026.md)

Question: {{question}}

Four things that matter here:

"Only from the sources" plus a sanctioned refusal. Chapter 12.6.1's point, and it is the difference between a useful system and a confident one.

Numbered citations. Users verify, and verification is what makes the answer usable. Check that cited numbers exist — a model can cite [4] when three sources were supplied.

Order the chunks deliberately. Best material first and last (Chapter 12.5.3).

Do not over-supply. Twenty chunks are worse than five: more cost, more latency, and more chance of the model latching onto something irrelevant.

8. Evaluating it

The essential move: measure retrieval and generation separately. A wrong answer has two possible causes, and they have completely different fixes. Measuring only the final answer tells you it is wrong and nothing about which half to work on.

Retrieval metrics need a set of questions with known relevant documents:

• Recall@k — was the right document in the top k? The most important number, because nothing downstream can recover from a missing document.
• MRR — how high did the first relevant result rank?
• nDCG — rank-weighted quality across several relevant results.

Generation metrics, usually judged by a model against the retrieved context:

• Faithfulness — is every claim supported by the context? This is the hallucination measure.
• Answer relevance — does it address the question?
• Citation accuracy — do the citations point at the passages that support the claims?

Build a golden set of 50–200 real questions with correct answers and known source documents. It is a day or two of unglamorous work and it is the difference between engineering and guessing. Every change — a chunk size, an embedding model, a prompt — is then a measurement rather than an opinion.

9. The failure catalogue

When an answer is wrong, it is one of these, and each has its own fix:

Failure	Fix
The content is not in the corpus	Ingest it. No pipeline change helps
It is there but not retrieved	Hybrid search, better chunking, contextual retrieval
Retrieved but ranked below the cut	Rerank; raise k before reranking
In context but not used	Reorder, reduce the number of chunks, strengthen the instruction
Answer contradicts the source	Lower the temperature; instruct to quote; check faithfulness
Sources contradict each other	A data problem — prefer recency, or surface the conflict
Right facts, wrong format	Constrained decoding (Chapter 12.5.3)

Walk this list in order when debugging. Most teams jump to prompt changes when the answer is at row one or two.

10. Beyond the basic pipeline

Graph retrieval builds an entity-and-relationship graph from the corpus and traverses it. It answers questions that no single chunk contains — "which customers are affected by suppliers in this region" requires joining facts, and vector search over independent chunks cannot. Expensive to build, and genuinely better for connected-fact questions.

Agentic retrieval lets the model search iteratively: search, read, decide it needs something else, search again. Better on hard questions, with unpredictable cost and latency, and it is where retrieval meets Chapter 12.6.3.

Long context instead of retrieval. With a very large window, why not send everything? Because of cost, latency, and the lost-in-the-middle effect — and because you still cannot enforce per-user access control on a dump. The pragmatic pattern is retrieval with generous chunks: use the window to send more context around fewer, better-chosen hits.

11. Operating it

Incremental indexing. Re-embedding the whole corpus on every change does not scale. Track a content hash per chunk and re-embed only what changed.

Deletion. When a document is removed or a person exercises a deletion right (Chapter 8.7), its chunks must leave the index. A vector store is a derived store and belongs in the deletion pipeline, and it is one of the systems most often forgotten.

Staleness. Show the source date. An answer from a superseded document is a confident wrong answer with a citation, which is worse than no answer.

Changing the embedding model means re-indexing everything (Chapter 12.5.2). Vectors from two models are not comparable, and mixing them fails silently.

Cost. Embedding is a one-off per chunk and cheap; retrieval is cheap; generation dominates, and it scales with how much context you send — which is one more reason to retrieve well rather than send a lot.

Recall

• Retrieval fixes the knowledge cutoff, private data, staleness, attribution and per-user access control. Retrieval teaches facts; fine-tuning teaches behaviour — fine-tuning on documents to make a model "know" them is the expensive mistake.
• Chunking decides more than the model. Recursive structural splitting is the default; small-to-big retrieves precisely and answers completely; contextual retrieval — prepending a generated description before embedding — fixes the biggest failure class. Keep tables and code intact, attach metadata, and look at your extracted text before building on it.
• Hybrid search (vector + BM25, fused by reciprocal rank) is the default. Fuse ranks, not scores. Pure vector search fails on product codes and identifiers.
• Filter by permission as a pre-filter in the retrieval query. Telling the model not to reveal something is not access control.
• Rewrite conversational follow-ups into standalone queries or multi-turn retrieval breaks. HyDE, multi-query and decomposition are cheap coverage wins.
• Rerank with a cross-encoder: retrieve 50 cheaply with a bi-encoder, score 5 accurately with a model that sees query and document together. Usually the biggest quality gain after hybrid search.
• Measure retrieval and generation separately — recall@k, MRR, nDCG on one side; faithfulness, answer relevance, citation accuracy on the other. Build a golden set of 50–200 real questions.
• Debug against the failure table in order — most wrong answers are a missing or unretrieved document, not a prompt problem. The vector store is a derived store: it belongs in the deletion pipeline, and changing embedding model means re-indexing everything.

Self-test: Why does fine-tuning make a poor substitute for retrieval? · What does contextual retrieval fix? · Why fuse ranks rather than scores? · Why must permission filtering be a pre-filter? · What is the difference between a bi-encoder and a cross-encoder, and why use both? · Which two metric families must be measured separately, and why?