Appearance
12.9.1 — Why RAG, and the Smallest Version
A customer asks "Ask Cartwise": "I bought earphones nine days ago and opened them. Can I still return them?"
Without any policy in its context, the model answers confidently: "Yes! Cartwise accepts returns within 30 days of delivery, as long as the item is in its original condition." Every part of that sounds right and most of it is wrong. Cartwise's real policy gives electronics a 10-day window, and opened earphones can only be returned if they are faulty, for hygiene reasons. The model was not lying. As 12.7.1 explained, it produced the most likely continuation, and a 30-day, original-condition policy is what returns policies most commonly say on the internet. Cartwise's actual policy was never in its training data, and even if it had been, it changed last month.
The fix is not a cleverer prompt. It is putting the real policy in front of the model before it answers, and telling it to answer only from that. That is retrieval-augmented generation, RAG: retrieve the relevant pieces of your own documents, augment the prompt with them, and let the model generate an answer grounded in them. The name comes from a 2020 paper by Patrick Lewis and colleagues at Facebook AI Research, and it has since become the single most common architecture in applied LLM work. This chapter builds it for "Ask Cartwise", piece by piece, and this page starts with why it is needed and the smallest version that works.
1. Why retrieve, rather than train the knowledge in?
An obvious alternative is to fine-tune the model on Cartwise's documents (Chapter 12.11) so it "knows" them. For factual knowledge, that is almost always the wrong tool, for five reasons.
Freshness. Cartwise's policies, prices and stock change every week. A document store can be updated in seconds; retraining a model takes days and money, and would have to be repeated for every change.
Private and per-user data. A customer's own orders, a merchant's contract, an internal memo: none of this was ever in any model's training data, and much of it is different for every user.
Access control. Retrieval can check, for every request, which documents this user is allowed to see, and only those reach the model. A fine-tuned model cannot forget a fact for one user and remember it for another; whatever it learned, it can reveal to anyone.
Citations. A RAG answer can point to the exact passage it used: "(Returns policy, section 3.2)". A support agent or customer can check it. An answer from a model's memory cannot be traced to a source.
Accuracy. Models trained on a document recall its details unreliably and blend them with what they learned elsewhere. Reading the passage at answer time is far more reliable than remembering it.
The short version, worth memorising: retrieval gives a model facts; fine-tuning changes its behaviour. Use retrieval for knowledge, and save fine-tuning for format, tone and specialised skills, as Chapter 12.11.1 sets out.
Another tempting alternative is to skip retrieval and paste all 40 policy documents into every request, since context windows are now large. 12.8.2 explained why that is usually worse: every token is paid for on every message, irrelevant material distracts the model and contradicting versions confuse it, and you still cannot apply per-user access control to a dump. Retrieval sends only what this question needs.
2. The two halves of a RAG system
A RAG system is really two systems that meet at a shared index.
Indexing runs ahead of time, and again whenever documents change.
- ① Load the documents and pull out clean text.
- ② Split them into chunks, passages small enough to retrieve precisely.
- ③ Turn each chunk into a vector with a sentence embedding model from 12.6.2.
- ④ Store each chunk's text, its vector and its metadata in an index, such as the vector search of 12.6.3. Metadata means extra facts: which document, which section, the date, and who may read it.
Querying runs for every question. ⑤ Embed the question with the same model. ⑥ Retrieve the k chunks most similar to it, the "top-k". ⑦ Put those chunks into the prompt, with instructions to answer only from them. ⑧ The model writes an answer, citing which chunks it used.
A useful thing to notice early: most quality problems in real RAG systems live in the indexing half, in how documents were extracted and split, while most attention goes to the prompt in step ⑦. The rest of this chapter goes in the order the data flows: ingestion and chunking (12.9.2), retrieval and reranking (12.9.3), grounded generation (12.9.4), evaluation (12.9.5) and advanced patterns (12.9.6).
3. The smallest RAG that works
Here is a complete RAG system in about 40 lines, small enough to read in one sitting, built only from pieces earlier pages introduced. The "documents" are six short policy passages.
python
import numpy as np
import anthropic
from sentence_transformers import SentenceTransformer
CHUNKS = [ # (1)
{"id": "returns-1", "text": "Most items can be returned within 30 days of delivery in original condition."},
{"id": "returns-2", "text": "Electronics, including phones, earphones and chargers, have a 10-day return window."},
{"id": "returns-3", "text": "Opened earphones and headphones can be returned only if faulty, for hygiene reasons."},
{"id": "refunds-1", "text": "Refunds are issued to the original payment method within 5-7 working days of pickup."},
{"id": "delivery-1", "text": "Standard delivery takes 3-5 working days; remote pin codes may take up to 8."},
{"id": "account-1", "text": "You can change your delivery address until the order is packed."},
]
embedder = SentenceTransformer("intfloat/e5-small-v2")
vectors = embedder.encode(["passage: " + c["text"] for c in CHUNKS], # (2)
normalize_embeddings=True)
def retrieve(question: str, k: int = 3) -> list[dict]:
q = embedder.encode("query: " + question, normalize_embeddings=True) # (3)
scores = vectors @ q # (4) cosine similarity
best = np.argsort(-scores)[:k]
return [CHUNKS[i] for i in best]
client = anthropic.Anthropic()
def answer(question: str) -> str:
sources = retrieve(question)
context = "\n".join(f'<source id="{c["id"]}">{c["text"]}</source>' for c in sources) # (5)
response = client.messages.create(
model="claude-opus-5-5", max_tokens=512,
system=("You answer Cartwise customers using only the sources provided. " # (6)
"Cite source ids in square brackets after each claim. If the sources "
"do not contain the answer, say you don't have that information."),
messages=[{"role": "user",
"content": f"<sources>\n{context}\n</sources>\n\nQuestion: {question}"}], # (7)
)
return "".join(b.text for b in response.content if b.type == "text")
print(answer("I bought earphones 9 days ago and opened them. Can I return them?"))
# e.g. "Only if they're faulty: opened earphones can be returned only when faulty [returns-3],
# and electronics have a 10-day window [returns-2], so you're still within it."Line (1) is the corpus, each chunk with an ID for citations. In a real system these come from the indexing pipeline of 12.9.2. Line (2) embeds every chunk once, using the asymmetric passage: prefix the model expects (12.6.2), normalised so that a dot product is cosine similarity. This is the whole "indexing" half, in memory.
Line (3) embeds the question with the query: prefix. Line (4) scores every chunk with one matrix-vector product and keeps the top three: exact nearest-neighbour search, which is fine for six chunks and is replaced by an ANN index for millions. For the earphones question, the two electronics and earphone chunks should score highest, because their meaning is closest, even though the question says "bought" and "opened" and the policy says "delivery" and "faulty".
Line (5) wraps each retrieved chunk in a tag carrying its ID, so the model can cite it and so the boundaries between sources are clear. Line (6) is the grounding instruction from 12.8.2: answer only from the sources, cite them, and say so when the answer is not there. Line (7) puts the sources first and the question last. The answer shown in the comment is the kind of reply this produces; the exact wording varies from run to run.
Compare it with the answer at the top of the page. Same model, same question; the only difference is that the right two sentences of policy were in the prompt.
4. What this minimal version gets wrong
This toy works on six sentences. Scaled to Cartwise's real 40 documents and hundreds of pages, it breaks in predictable places, and each one is a later page:
- Real documents are not tidy sentences. They are PDFs with tables, headers repeated on every page, and sections that make sense only with their heading. How you extract and split them decides more than the choice of model. 12.9.2.
- Embeddings miss exact terms. A question about "order 4471" or the "Cartwise Plus" membership needs keyword search too, and the best five chunks are not always the five with the highest cosine score. 12.9.3, building on the hybrid search of 12.6.4.
- Follow-up questions fail. "And what about chargers?" embedded on its own retrieves nothing useful. 12.9.3.
- The model may still blend in its own memory, or cite a source that does not say what it claims. 12.9.4.
- You cannot tell whether a change helped without measuring retrieval and generation separately. 12.9.5.
- Some questions need facts from many documents at once, or several rounds of searching. 12.9.6.
5. Where RAG shows up
Once you recognise the shape, RAG is everywhere: support assistants over help centres and policies, internal assistants over company wikis and handbooks, legal and compliance search over contracts, coding assistants that retrieve relevant files from a codebase before answering, search engines that write an answer above the results, and agents that look things up as one of their tools. For Cartwise, the same pipeline will power "Ask Cartwise" for customers, a policy assistant for support staff with access to internal documents customers must not see, and product question answering from product manuals. The case study in Chapter 12.15.3 assembles the full production version.
Recall
- An LLM answers from the most likely continuation, so without your documents in context it produces plausible, wrong policies. Retrieval-augmented generation retrieves relevant passages, puts them in the prompt, and instructs the model to answer from them with citations.
- Retrieve rather than fine-tune for knowledge: freshness, private and per-user data, access control, citations and accuracy. Retrieval gives facts; fine-tuning changes behaviour. Pasting everything into the context costs more and answers worse.
- Indexing (load, chunk, embed, store with metadata) runs ahead of time; querying (embed the question, retrieve top-k, augment the prompt, generate with citations) runs per request. Most quality problems live in indexing.
- The 40-line version: embed chunks once, embed the question, rank by dot product, wrap sources in tagged blocks with IDs, instruct "only from sources, cite, admit when absent", question last.
- Real systems add careful ingestion and chunking, hybrid retrieval and reranking, query rewriting, grounding checks, separate evaluation of retrieval and generation, and advanced patterns for multi-document questions.
Self-test: Give three reasons fine-tuning on Cartwise's policies would be worse than retrieving them. Why must the question be embedded with the same model as the chunks? In the minimal version, which line would you change to stop the model citing a source it was not given?
Next: 12.9.2 Ingestion and Chunking — the quality of every answer is capped by how the documents were turned into chunks; the next page is about doing that well, and keeping the index correct as documents change.