Appearance
12.9.6 — Advanced RAG Patterns
The RAG pipeline of this chapter answers one kind of question very well: a question that one or a few passages answer directly. "What is the return window for electronics?" lives in one paragraph. But Cartwise's support staff and managers started asking the policy assistant harder things:
- "Which of our return rules changed in the last six months, and which product categories do the changes affect?": no single passage contains the answer; it is spread across many documents and needs joining up.
- "Summarise the main themes in our customer complaints policy documents.": a question about the whole collection, not any one part of it.
- "A customer's parcel was marked delivered but they say it never came; the order was paid with a gift card and part of it was a Cartwise Plus item. What do we do?": needs several lookups, each depending on what the previous one found.
- "What does the wiring diagram in the air conditioner installation manual say about the earth connection?": the answer is in a picture.
Each of these breaks the basic pipeline in a different way, and each has a well-known pattern that handles it. This page covers those patterns, plus the question that now comes up in every design review: with million-token context windows, do we need retrieval at all? The guiding rule throughout is the one from 12.9.5: adopt a more complex pattern only when your evaluation shows the simple pipeline failing on questions that matter.
1. Conversational RAG
Real support is a conversation, and conversations break simple retrieval in two ways. Follow-up questions mean nothing on their own ("and for chargers?"), which 12.9.3 fixed by rewriting each message into a standalone query. And the conversation itself is context: what has already been established (which order, which product, what the customer has already been told) must survive across turns without being re-retrieved every time.
A robust conversational RAG system therefore keeps, alongside the recent messages, a small structured state: the order and product being discussed, the policies already cited, and what has been offered. It retrieves fresh passages per turn using the rewritten query, but also keeps the passages that the current answer depends on, so that "can you explain that last rule again?" does not trigger a search that might return something different. Chapter 12.10.5 generalises this into memory management for agents.
2. Questions that join facts: GraphRAG
"Which return rules changed recently, and which categories do they affect?" needs facts from many places: change notes in several policy versions, the categories each rule covers, and the dates. Vector search retrieves passages that are similar to the question; none of them is similar to the whole answer, because no passage contains it.
GraphRAG addresses this by building a knowledge graph from the documents during indexing: a network whose nodes are entities (a rule, a product category, a policy version, a date) and whose edges are relationships between them (rule applies to category; version supersedes version; rule changed on date). An LLM reads each chunk and extracts the entities and relationships it mentions, and the results from all chunks are merged into one graph.
The figure shows the payoff. Starting from the 2026 policy, follow the "changed" and "introduced" edges to two rules, then each rule's "applies to" edge to a category. The answer is assembled by walking the graph, then the passages behind each node and edge are retrieved and given to the model to write the answer with citations.
Microsoft Research's 2024 GraphRAG work added a second idea for whole-collection questions like "what are the main themes in our complaints documents?". It groups the graph into communities, clusters of closely connected entities, using a standard graph-clustering algorithm, and has an LLM write a summary of each community during indexing. A global question is then answered from those summaries rather than from individual passages, which no chunk-level retrieval can do.
GraphRAG is powerful and expensive. Building the graph means an LLM call for every chunk, and extraction errors (a missed relationship, two names for one entity) quietly become wrong answers. It earns its cost on collections with many interconnected entities, such as contracts, regulations, research and incident reports, and on questions that join facts or survey the whole collection. For "what is the return window", ordinary retrieval is better and far cheaper.
3. Retrieval inside an agent: agentic RAG
The delivered-but-missing parcel question needs several steps, each depending on the last: look up the order, see it was paid partly by gift card, find the policy for disputed deliveries, find the gift-card refund rules, check the Cartwise Plus terms for the membership item. A single retrieval with the original question finds some of these and misses the rest.
In agentic RAG, retrieval becomes a tool that the model can call as often as it needs, alongside the other tools of 12.8.4: search_policies(query), get_order(order_id). The model searches, reads the results, decides what is still missing, and searches again with a better query, until it has what it needs to answer. This is an agent, the subject of Chapter 12.10, with search as one of its tools.
It answers hard, multi-part questions far better than one-shot retrieval. Its costs are the costs of agents: the number of steps, and therefore the latency and price, is not known in advance, and each extra step is another chance for a mistake. So it is used where questions are genuinely complex, with a cap on the number of searches, and often only as a fallback when one-shot retrieval returns nothing good.
4. Pictures, tables and scanned pages: multimodal RAG
The air conditioner wiring diagram is an image. A text-only pipeline either throws it away or keeps a useless caption like "Figure 3". Three approaches let RAG use images, and many systems combine them.
Describe images during ingestion. Send each figure to a vision-capable model and ask for a detailed text description, then embed and index the description like any other text. Simple, and it works with the existing pipeline, but a description is only as good as what the model chose to mention.
Embed images directly. Use a multimodal embedding model such as CLIP, from 12.6.2, so that a text query and an image land in the same space, and search images with words.
Index whole page images. For documents where layout carries meaning, such as manuals, invoices and forms, treat each page as an image. Retrieval models built for this, such as ColPali from 2024, embed page images directly, and the retrieved page images are then given to a vision-capable model to read and answer from. This avoids text extraction errors entirely, at the cost of larger indexes and more expensive generation.
Whatever the approach, the generation step gives the model the retrieved image itself when it can, so it answers by looking at the diagram rather than at a description of it. Chapter 12.13.1 covers vision models, and 12.15.4 a complete document pipeline.
5. When the data is a database, not documents
"How many electronics returns did we approve last month?" is not a retrieval question at all. The answer is a count in a database, and embedding rows of a table and retrieving "similar" ones will not count anything correctly. For questions over structured data, the right pattern is to have the model write a query (often SQL, from Chapter 7.2) against a described schema, run it with read-only permissions, and explain the result, or better still, to offer a small set of tested functions as tools (count_returns(category, month)) so that the model never writes free-form queries against production data. Mixed questions, such as "which of our return rules explain last month's rise in electronics returns?", combine a database tool with document retrieval inside an agent.
6. Do long context windows make retrieval obsolete?
With models that accept hundreds of thousands or a million tokens, a fair question is: why not put all 40 policy documents in every request and skip retrieval? Sometimes that is exactly right. Often it is not.
When long context is the better choice: the material is small and fixed (a single contract, one manual, a few hundred pages), the question genuinely needs the whole of it (summarise this contract, find every clause about liability), and the same material is used across many requests so that prompt caching from 12.7.6 makes repeated use cheap. For one-off analysis of a document, it is simpler than building a pipeline.
When retrieval wins:
- The collection is large, or growing. Thousands of documents do not fit even in a large context window.
- Different users may see different documents. That needs the per-request filtering of 12.9.3.
- Requests are frequent, so sending everything every time multiplies cost and delay.
- Answers need precise citations.
Models also still use information in the middle of long contexts less reliably, the effect from 12.7.5.
The common practical middle ground is retrieval with generous context: retrieve well, then send whole sections or documents around the hits rather than tiny chunks, using the larger window to give the model complete context for fewer, better-chosen sources. That is small-to-big retrieval from 12.9.2, scaled up.
7. Choosing a pattern
| Question type | Pattern |
|---|---|
| Answered by one or a few passages | the standard pipeline of 12.9.1–12.9.4 |
| Follow-ups in a conversation | query rewriting + conversation state |
| Joins facts across many documents | GraphRAG, or an agent with search |
| About the whole collection | GraphRAG community summaries, or long context if small |
| Needs several dependent lookups | agentic RAG with a step cap |
| Answer is in an image or layout | image descriptions, multimodal embeddings, page images |
| Counts and aggregates over records | database tools, not retrieval |
| Small fixed document set | long context with prompt caching |
The order of adoption matters as much as the choice. Get the standard pipeline right first, with clean ingestion, hybrid retrieval, reranking and grounding, because most failures that look like they need an advanced pattern turn out to be ingestion or retrieval problems that the evaluation of 12.9.5 would have found. Then add patterns one at a time, each justified by a category of questions that your measurements show failing.
Recall
- Conversational RAG rewrites follow-ups into standalone queries and keeps a small structured conversation state and the passages the current answer depends on.
- GraphRAG extracts a knowledge graph of entities and relationships during indexing and answers joining questions by walking it; community summaries answer whole-collection questions. Powerful and expensive; for connected-fact questions, not simple lookups.
- Agentic RAG makes search a tool the model calls repeatedly until it has what it needs; better on multi-step questions, with unpredictable cost, so cap the steps.
- Multimodal RAG describes images at ingestion, embeds images directly (CLIP), or indexes whole page images (ColPali) and lets a vision model read them. Structured data needs database tools, not retrieval.
- Long context suits small, fixed, whole-document tasks with prompt caching; retrieval wins for large, per-user, frequent, citation-heavy use. The middle ground is retrieval with generous context. Get the standard pipeline right first and add patterns only when evaluation demands them.
Self-test: Why can vector search not answer "which rules changed and which categories do they affect?" even if every relevant passage is indexed? What two costs make agentic RAG a poor default for simple questions? When would you choose long context over retrieval for a 200-page supplier contract?
Next: 12.10.1 What an Agent Is — agentic RAG gave the model a search tool and let it decide when to use it; Chapter 12.10 takes that idea all the way, to systems that plan, use many tools and take actions, starting with Cartwise's refund agent.