12.9.2 — Ingestion and Chunking

When the minimal RAG system of 12.9.1 was pointed at Cartwise's real policy documents, answer quality was worse than expected, and for weeks the team tuned the prompt. The problem was not the prompt. When someone finally printed the text the pipeline had extracted from the 24-page Returns and Refunds Policy PDF, it looked like this:

Cartwise Policies — Confidential — Page 7   Electronics   10 days   Opened items   Category
Return window   Condition   Clothing   30 days   Tags attached   Large appliances   Faulty
only after installation Cartwise Policies — Confidential — Page 8 Customers may return most

The PDF had a two-column layout, and the extractor read straight across both columns, interleaving them. The returns table had been flattened into a jumble of words with no rows. And the page header appeared in the middle of the text every few hundred words. No embedding model can find "the return window for electronics" in that, and no prompt can make a model answer from it. A RAG system can never be better than the text in its index, and this page is about getting that text right: extracting it, splitting it into useful pieces, labelling them, and keeping the index correct as documents change.

1. Extraction: get clean text out

Each kind of source needs its own handling.

PDFs are the hardest, because a PDF stores where to draw characters on a page, not paragraphs, headings and tables. Simple extractors read text in drawing order, which is how columns get interleaved. Better tools analyse the page layout: they detect columns, reading order, headings and tables, and output structured text, often as Markdown. If a PDF is a scan, an image of a page with no text layer, it needs optical character recognition (OCR) to turn the image back into text. For the hardest documents, such as invoices, forms and complex tables, modern systems send page images to a vision-capable model and ask it to transcribe the content as structured text, an approach Chapter 12.15.4 builds out.

Web pages and help-centre articles come wrapped in navigation menus, cookie banners, footers and sidebars. Keep the main content, and keep its headings.

Office documents, wikis and tickets usually extract well, but carry their own noise: tracked changes, comments, signatures, quoted email threads.

Tables deserve special care, because policies put their most important facts in them. Convert each table to Markdown or to one line per row ("Electronics: return window 10 days; condition: unopened, or faulty if opened"), so that each row's meaning survives on its own.

And then do the step that most teams skip, the one that would have saved Cartwise weeks: read the extracted text. Open twenty documents' worth and look at them with your own eyes before building anything on top. Interleaved columns, missing tables, repeated headers and garbled characters are obvious to a person in seconds and invisible in every metric.

2. Why split documents at all?

Each chunk becomes one embedding, and an embedding is a single summary of its text, as 12.6.2 explained. Embed the entire 24-page policy as one vector, and the vector is a blur of returns, refunds, exchanges and warranties; a question about earphones matches it only weakly, and if it is retrieved, you have spent the whole context budget on 24 pages to answer one sentence. Split it too finely, into single sentences, and each piece loses the context that gave it meaning: "The window is 10 days" says nothing about which window.

The chunk-size trade-off. Small chunks match precisely but lose meaning; large chunks keep meaning but blur the embedding and crowd the context. A section of a few paragraphs, kept with its heading, is usually the best starting point.

There is no universal best size; it depends on the documents and the questions, and it is one of the settings you tune with the evaluation of 12.9.5. But a section of a few paragraphs, roughly 200 to 800 tokens, kept together with its heading, is a sensible place to start for policy and help-centre text.

3. Chunking strategies

Fixed-size chunks with overlap. Cut every s tokens, with each chunk overlapping the previous one by o tokens, so that a sentence cut at a boundary appears whole in one of the two neighbours. A 5,000-token document cut into 500-token chunks with 50 tokens of overlap advances 450 tokens per chunk, giving \lceil (5{,}000 - 50) / 450 \rceil = 11 chunks. It is simple and predictable, and it ignores structure completely: it happily splits a heading from its section, a table down the middle, or a code block in two.

Recursive, structure-aware chunking. Split on the largest natural boundary that keeps chunks under the size limit: first by document sections (headings), then by paragraphs, then by sentences, and only as a last resort mid-sentence. This respects the document's own organisation, keeps topics together, and is the sensible default for most text.

Semantic chunking. Embed each sentence, and start a new chunk wherever consecutive sentences become noticeably less similar, a sign that the topic has changed. It helps with long, unstructured text such as transcripts, at the cost of an embedding per sentence during indexing.

Content-specific rules. Whatever the general strategy, some things must never be split: a table (or keep each row with the table's header), a code block, a question with its answer in an FAQ, a list with the sentence that introduces it.

Here is a compact structure-aware chunker for Markdown-style text, which is what good PDF extraction produces:

import re

def chunk_markdown(doc_text: str, doc_title: str, max_chars: int = 2000) -> list[dict]:
    chunks, path = [], []
    for section in re.split(r"\n(?=#{1,3} )", doc_text):                  # (1) split at headings
        heading = section.split("\n", 1)[0]
        level = len(heading) - len(heading.lstrip("#"))
        if level:
            path = path[: level - 1] + [heading.lstrip("# ").strip()]       # (2) track the heading path
        paras, buf = section.split("\n\n"), ""
        for p in paras:                                                      # (3) pack paragraphs
            if len(buf) + len(p) > max_chars and buf:
                chunks.append(make_chunk(doc_title, path, buf))
                buf = ""
            buf += p + "\n\n"
        if buf.strip():
            chunks.append(make_chunk(doc_title, path, buf))
    return chunks

def make_chunk(title, path, text):
    breadcrumb = " > ".join([title] + path)                                 # (4)
    return {"text": f"{breadcrumb}\n\n{text.strip()}", "section": breadcrumb}

Line (1) splits the document at headings, so each section is handled on its own. Line (2) tracks where we are in the document, for example Returns and Refunds Policy > Electronics > Opened items. Line (3) packs whole paragraphs into chunks up to the size limit, never cutting a paragraph. Line (4) prefixes each chunk with that heading path, its "breadcrumb", which is a cheap and effective way to restore context: a chunk that says only "the window is 10 days" now also says it is about opened electronics in the returns policy, so it both embeds and matches correctly.

Two refinements that solve the context problem

Small-to-big retrieval. Embed and search over small chunks, which match precisely, but when one matches, send the model the larger section it belongs to, which answers completely. Each small chunk stores a pointer to its parent section. This directly resolves the tension in the figure.

Contextual retrieval. Before embedding each chunk, have an LLM write one or two sentences describing where the chunk sits in its document, and prepend that to the chunk: "This passage is from Cartwise's 2026 Returns and Refunds Policy, in the section on electronics, and explains the rules for opened earphones." It is a richer version of the breadcrumb, written by a model that has read the whole document. Anthropic published this technique in 2024 and reported that it substantially reduced retrieval failures, especially combined with keyword search and reranking. It costs one cheap model call per chunk at indexing time, which prompt caching makes cheaper still, because the whole document is the same prefix for each of its chunks.

4. Metadata: what to store with each chunk

Every chunk should carry metadata alongside its text and vector:

• Source and location: document ID, title, section path, page, URL. Needed for citations.
• Dates and versions: when the document was published, when it takes effect, which version it is. Needed to prefer current policy over old.
• Access control: who may read it, such as "customer-facing" or "support staff only", or the specific customer it belongs to. Needed to filter retrieval so that nothing reaches the model for a user who may not see it, which 12.9.3 enforces.
• Category, language, product line: whatever your filters and evaluations need.
• A content hash and the embedding model's name: needed to keep the index correct, in section 6.

Add metadata from the start. Adding it later means re-processing every document.

5. Duplicates, conflicts and the golden record

Cartwise's shared drive contained three returns policies: the current one, last year's (still linked from an old help article), and a draft that was never published. All three were indexed. A question about electronics retrieved chunks from two of them, one saying 10 days and one saying 14, and the model sometimes said 10, sometimes 14, and once "10 to 14 days".

This is a problem the data world solved long ago under the name golden record. In master data management, the discipline of keeping core business data consistent across systems, a golden record is the single, authoritative version of an entity, such as one customer, assembled from the duplicate and conflicting copies scattered across different systems. The sales system has one address for a customer, the billing system another, the support system a third; the golden record is the one record the business agrees to trust, with rules for which source wins for each field.

RAG needs the same discipline for documents. Before indexing:

• Choose an authoritative source for each topic, and index from it rather than from wherever copies happen to live.
• Deduplicate. Exact copies are easy to find by hashing their content; near-duplicates (the same article republished with small edits) can be found with the MinHash technique of 12.7.5.
• Mark superseded and draft documents, and either leave them out or keep them out of customer-facing retrieval, with an effective date on every policy.
• When genuine conflicts remain, prefer the most recent effective version in retrieval, and have the assistant say when sources disagree rather than silently picking one.

6. Keeping the index correct

The index is a copy of your documents in another form, and copies drift. Treat it as a pipeline that runs continuously, like the data pipelines of 12.4.4.

Incremental updates. Re-embedding thousands of documents every night wastes money and time when only a few changed. Store a hash of each chunk's text and re-embed only chunks whose hash is new:

import hashlib

def sync_document(doc, index):
    new = {hashlib.sha256(c["text"].encode()).hexdigest(): c for c in chunk_document(doc)}   # (1)
    old = set(index.hashes_for(doc.id))
    for h in old - new.keys():
        index.delete(doc.id, h)                                       # (2) removed or changed text
    for h in new.keys() - old:
        c = new[h]
        index.upsert(doc.id, h, text=c["text"], vector=embed(c["text"]),
                     metadata={**doc.metadata, "section": c["section"],
                               "embed_model": EMBED_MODEL})           # (3)

Line (1) chunks the current version of the document and fingerprints each chunk. Line (2) deletes chunks whose text no longer exists in the document. Line (3) embeds and stores only the new or changed chunks, recording the embedding model used.

Deletions are not optional. When a document is withdrawn, or a customer exercises their right to have personal data erased under privacy law (Chapter 8.7), every chunk derived from it must leave the index too. The vector index is a derived store, and derived stores are the copies most often forgotten in deletion processes. Build deletion into the pipeline from day one.

Staleness. Show the source date in answers or citations, and alert when key documents have not been re-indexed after a change. An answer from a superseded policy, delivered with a confident citation, is worse than no answer.

Changing the embedding model means re-embedding everything into a new index, as 12.6.2 explained, which is one more reason to store the model's name with every chunk.

Recall

• A RAG system can never beat the text in its index. Extraction must handle PDF layout, scans (OCR or a vision model), web boilerplate and tables; read the extracted text before building on it.
• Chunks are the unit of retrieval: too small loses context, too large blurs the embedding and wastes the context budget; a section of about 200 to 800 tokens with its heading is a good start.
• Strategies: fixed-size with overlap (simple, ignores structure; 5,000 tokens at 500 with 50 overlap gives 11 chunks), recursive structure-aware splitting (the default), semantic chunking, and rules never to split tables, code or Q&A pairs. Heading breadcrumbs, small-to-big retrieval and contextual retrieval restore lost context.
• Store metadata with every chunk: source and section, dates and versions, access control, categories, content hash and embedding model.
• A golden record is the single authoritative version of an entity; for RAG, choose authoritative sources, deduplicate, exclude drafts and superseded versions, and surface real conflicts. Keep the index correct with hash-based incremental updates, deletions, staleness checks, and full re-embedding on model change.

Self-test: Why did the interleaved two-column extraction hurt retrieval even though every word was present? How many chunks does a 9,000-token document give at 800 tokens with 100 overlap? Why must chunk deletion be part of the indexing pipeline rather than a manual task?

Next: 12.9.3 Retrieval and Reranking — with clean, well-labelled chunks in the index, the next step is finding the right ones for each question, including follow-ups, restricted documents and questions that use different words from the policy.