12.9.4 — Grounded Generation and Citations

Retrieval now delivers the right five chunks for almost every question. And yet a support lead, reviewing a week of "Ask Cartwise" answers against their sources, still found problems in about one answer in twenty:

• Added facts. The source said electronics have a 10-day window. The answer said "10 days, and Cartwise will arrange free pickup". Free pickup was nowhere in the sources; the model added it from general knowledge of how online stores work.
• Wrong citations. An answer cited [3] for a claim that came from source 1, and once cited [6] when only five sources had been supplied.
• Answering anyway. Asked about the warranty on refurbished phones, which no retrieved chunk covered, the assistant gave a confident one-year answer instead of saying it did not know.
• Old rules. Two retrieved chunks gave different answers, one from last year's policy, and the answer used the old one.

Each of these is a failure of grounding: the requirement that every claim in an answer is supported by the provided sources and nothing else. A RAG system that retrieves well but generates loosely is still a system that confidently states things that are not true, now with a citation attached, which makes it more convincing. This page covers how to make generation grounded, how to make citations trustworthy, and how to check both.

1. Writing the generation prompt

The prompt for the generation step does four jobs, and each one maps to a failure above.

You are Cartwise's support assistant. Answer the customer's question using ONLY
the sources below. Do not add information from general knowledge.          (1)

After each sentence that states a fact, cite its source id in square brackets,
like [returns-2]. Every factual sentence must have a citation.               (2)

If the sources do not answer the question, say: "I don't have that information,
but I can connect you with our support team." Do not guess.                  (3)

If sources disagree, use the one with the latest effective date, and mention
that the policy changed on that date.                                        (4)

<sources>
<source id="returns-2" title="Returns and Refunds Policy" section="Electronics"
        effective="2026-09-01">Electronics, including phones, earphones and
chargers, have a 10-day return window.</source>
...
</sources>

Question: {question}

(1) states that the sources are the only allowed basis, which addresses the added free pickup. It is not a guarantee, because the model's general knowledge does not switch off, but stated explicitly it greatly reduces additions, and section 4 checks for what slips through. (2) requires a citation after each factual sentence, which makes every claim traceable and also makes unsupported claims easier to spot, since a fact with no possible citation stands out. (3) is the sanctioned "I don't know" from 12.8.2, and it is the most effective single line against confident invented answers: given a permitted way to decline, models decline far more often when they should. (4) gives a rule for conflicts, which only works because each source carries its effective date in the metadata from 12.9.2.

Two smaller choices also matter. Each source is given with its title, section and date, so the model can cite meaningfully and judge recency. And the sources come before the question, ordered with the most relevant first, as 12.8.2 recommended for long context.

2. Citations the API produces for you

Asking the model to write citation markers in its text works, but it leaves you parsing brackets and checking they exist. Several APIs can instead produce citations as structured data: you pass each source as a document, enable citations, and the response comes back split into text segments, each linked to the exact passage of the exact document it relied on. Anthropic's API does it like this:

def as_document(chunk):
    return {
        "type": "document",
        "source": {"type": "text", "media_type": "text/plain", "data": chunk["text"]},
        "title": f'{chunk["title"]} — {chunk["section"]} (effective {chunk["effective"]})',
        "citations": {"enabled": True},                                   # (1)
    }

response = client.messages.create(
    model="claude-opus-5-5", max_tokens=1024,
    system=GROUNDED_SYSTEM_PROMPT,
    messages=[{"role": "user",
               "content": [*map(as_document, chunks),                     # (2)
                           {"type": "text", "text": question}]}],
)

for block in response.content:
    if block.type == "text":
        print(block.text, end="")
        for c in (block.citations or []):                                  # (3)
            print(f'  ← "{c.cited_text}" from {c.document_title}')

Line (1) turns on citations for each document; it must be enabled on all of them or none. Line (2) sends the documents before the question, in one user message. Line (3) reads the citations attached to each segment of the answer: each one carries the exact cited_text it is based on, the document's title, and where in the document the passage sits. Because the cited text is an exact quotation from your source, the interface can show it, link to it, or highlight it, and your code never has to trust a hand-written [3].

When an API does not offer this, ask for citation markers in a fixed format and validate them in code: every cited ID must be one of the sources actually supplied, and every factual sentence should carry one. A citation to a source that was never given is not a citation; it is an invention in citation format.

3. When the sources do not have the answer

The refurbished-phone warranty question exposed the hardest behaviour to get right: declining. Three things make it work.

Give the model a specific way to decline, as in instruction (3) above, and a next step to offer the customer. "I don't know" with a path forward is a good answer; "I don't know" with nothing else feels like a dead end, and models trained to be helpful resist giving it.

Allow partial answers. Often the sources answer part of a question. "Electronics have a 10-day window [returns-2]. I don't have information about warranties on refurbished phones; our support team can help with that." is far better than either inventing the warranty or refusing the whole question.

Ask instead of guessing when the question is ambiguous. "Can I return it?" with no product mentioned and several recent orders should produce a clarifying question, not an answer about the wrong order.

Then measure it: the evaluation set must include questions whose answers are deliberately not in the documents, and the expected behaviour for them is a decline. A system that never declines in evaluation is not perfect; it is untested on the cases that matter most. 12.9.5 builds those cases.

4. Checking the answer before it goes out

Instructions reduce ungrounded answers; they do not eliminate them. For high-stakes answers, money, legal terms, health or safety, a second, automatic check runs on the answer before it reaches the customer. Its job is narrow: for each factual claim in the answer, is it supported by the sources? This property is usually called faithfulness.

from typing import Literal
from pydantic import BaseModel

class ClaimCheck(BaseModel):
    claim: str
    supported_by: str | None                      # quote from a source, or None
    verdict: Literal["supported", "not_supported", "contradicted"]

class Audit(BaseModel):
    checks: list[ClaimCheck]

def audit(answer: str, chunks: list[dict]) -> Audit:
    sources = "\n".join(f'<source id="{c["id"]}">{c["text"]}</source>' for c in chunks)
    r = client.messages.parse(
        model="claude-haiku-5-5", max_tokens=1024,                         # (1)
        messages=[{"role": "user", "content":
            "Split the answer into its factual claims. For each, quote the sentence "
            "from the sources that supports it, or say it is not supported or is "
            f"contradicted.\n<sources>{sources}</sources>\n<answer>{answer}</answer>"}],
        output_format=Audit,                                               # (2)
    )
    return r.parsed_output

result = audit(answer_text, chunks)
if any(c.verdict != "supported" for c in result.checks):                  # (3)
    answer_text = regenerate_strictly(question, chunks) or HANDOFF_MESSAGE

Line (1) uses a small, fast model, because checking a few claims against five short sources is a much easier task than answering the question in the first place, and the check adds to every answer's latency and cost. Line (2) asks for a structured verdict per claim, with the supporting quote, using the structured output of 12.8.3. Line (3) acts on it: if any claim is unsupported, regenerate once with a stricter instruction, and if that fails too, fall back to handing over to a person. On the "free pickup" answer, the audit would return the free-pickup claim as not_supported.

Whether to run this on every answer is a cost decision. Many systems run it on all answers that mention money, dates or policy terms, and on a random sample of the rest for monitoring, which is the production evaluation of 12.8.6.

5. The usual ways grounded answers still go wrong

Failure	Typical cause	Fix
Fact added from general knowledge	prompt allows it; common assumption	"only the sources" + faithfulness check
Citation to the wrong source	many similar sources	API citations; validate IDs
Answers when it should decline	no permitted way to decline	specific decline text + partial answers
Uses an outdated rule	old versions indexed	dates in metadata; recency rule; golden record
Misreads a table	table flattened at ingestion	fix extraction (12.9.2)
Right facts, rambling answer	no length or format instruction	specify length and structure

Two patterns in that table deserve emphasis. First, several "generation" failures are really ingestion failures in disguise: a model cannot read a table that was destroyed during extraction, or prefer a current policy whose date was never recorded. Second, grounding is checkable. Unlike "is this answer good?", "is this claim in this source?" is a question with a definite answer, which is why faithfulness is one of the most reliable things to measure automatically in 12.9.5.

6. Presenting answers to people

Grounding also shapes the interface. Showing citations as small links or expandable quotations lets customers and support staff verify an answer in seconds, which is what makes them willing to rely on it. Showing the source date ("policy effective 1 September 2026") prevents a correct-at-the-time answer from being trusted after a change. For support staff, a side panel with the full retrieved passages turns the assistant from an oracle into a research tool they can check. And a visible "Talk to a person" option next to every answer is part of grounded design too: it is the decline path made into a button.

Recall

• Grounding means every claim is supported by the provided sources and nothing else; loosely grounded RAG states wrong things with convincing citations.
• The generation prompt: sources only, no general knowledge; cite after each factual sentence; a specific decline message; a rule for conflicting sources using effective dates; sources with titles, sections and dates, placed before the question.
• Prefer API-generated citations, which link each answer segment to an exact quoted passage; otherwise validate every cited ID against the sources supplied.
• Make declining easy: a specific decline with a next step, partial answers, clarifying questions for ambiguous requests, and evaluation cases whose answer is deliberately absent.
• Check faithfulness before high-stakes answers go out, claim by claim with a small model and structured output, and regenerate or hand over on failure. Many generation failures are really ingestion failures; show citations and dates to users.

Self-test: Why does a citation to a source ID that was never supplied make an answer more dangerous, not less? Write the partial answer for "What's the return window for earphones, and do refurbished phones have a warranty?" given only the electronics chunk. Why can a small model check faithfulness even when a large model wrote the answer?

Next: 12.9.5 Evaluating RAG — retrieval, reranking, prompts and checks all have settings; the next page measures each half of the system separately, so every change is a measurement rather than a guess.