12.15.1 — Designing an AI System

Cartwise's engineering wiki, ticket tracker and shared drives hold ten years of design documents, incident reports and meeting notes. When a new engineer asks "why did we choose PostgreSQL over Cassandra for the order service?", the answer exists somewhere in those systems, but finding it means asking three senior engineers and waiting two days. The platform team was asked to build an assistant that answers such questions from the company's own documents, with sources, for the 600 engineers who might use it.

This is one of the most common AI systems companies build, and "design an internal document question-answering system" is one of the most common AI system design interview questions. This page uses it to teach a method: a sequence of decisions that works for any AI system, from a blank page to production, in which every step draws on a chapter of this Part. The four case studies that follow apply the same method to four different systems, and the capstone has you apply it yourself.

1. The method in nine steps

Every AI system design, in a design document or an interview, works through the same nine questions. The order matters, because each answer constrains the next.

The nine questions of an AI system design. The first three decide whether and what to build, the middle three how, and the last three how to run it safely; production measurements then feed the next round of design.

The rest of this page works through the nine steps for the engineering assistant, each one short and pointing back to the chapter that teaches it in depth.

2. Understand the problem

① The problem and success, in numbers

Start with who has the problem, what they do today, and what "better" would mean as numbers, the problem card of 12.1.3. For the assistant:

• Users and questions: 600 engineers asking about past decisions, how systems work, incident history, and "who owns this?"; perhaps 2,000 questions a day.
• Success: answers correct and supported by sources for at least 85% of answerable questions; an honest "I couldn't find that" for questions the documents do not cover; engineers rating it useful.
• Constraints: answers must respect document permissions (a security-incident report is restricted, as are HR documents in the same drives); time to first token under 2 seconds; a budget of a few thousand dollars a month.

Writing down the constraint about permissions first changes the whole design, as section 4 shows. Constraints discovered late are the expensive ones.

② Does it need machine learning, and which kind?

Chapter 12.1.2 and 12.11.1 give the questions. The knowledge is in documents, changes constantly, must be cited and must respect permissions: that is retrieval, not fine-tuning. Most questions can be answered from a few passages, so a RAG workflow fits; a few need several lookups across systems, which might later justify an agent (12.10.1). Starting with the simpler design and adding the agent only if measurements demand it is the rule from 12.10.6.

③ The data

The sources are a wiki, a ticket tracker and shared drives: about 180,000 documents, in formats from clean pages to scanned PDFs and slide decks with diagrams. Chapter 12.9.2 raises four questions about them. How do you pull the text out and keep its structure, such as headings, tables and code blocks? How do you cut it into chunks? What extra facts do you store with each chunk? The usual ones are the source, the author, the date and, most important of all, who is allowed to see it. And how do you stay up to date? Documents change every day, and an answer built from last year's version of a decision may simply be wrong. Duplicates and outdated versions are common in wikis, so the "golden record" problem of 12.9.2, deciding which version is current, needs an answer too.

3. Build it

④ A baseline first

Before anything sophisticated, build the simplest system that could work and measure it. Here that means two baselines, because they fail differently. A keyword search over the documents with BM25 (12.6.4), returning links. And the smallest RAG: fixed-size chunks, embeddings, a vector index, the top five chunks to a model, from 12.9.1. Each takes a day or two, and measuring them on the evaluation set (step ⑥) shows what the real problems are, rather than what you imagined them to be.

How such systems typically evolve, version by version, and why:

Version	What it adds	What it fixes	What still fails
v0 keyword search	BM25 links	exact terms, ticket IDs	synonyms, no answers
v1 vector RAG	embeddings, generated answers	meaning, synthesis	exact IDs and acronyms missed
v2 hybrid + rerank + grounding	BM25 + vectors, cross-encoder, citations	precision, trust	tables, stale versions
v3 structure-aware ingestion	tables as structured data, freshness	tables, currency	multi-system questions
v4 agent (if needed)	tools for tickets, code, people	multi-step questions	cost, latency, more risk

Each row is a measured response to the failures of the row above. The team building Cartwise's assistant found v1 failing on questions such as "what happened in INC-4921?", because embeddings smooth over identifiers, which is exactly why hybrid search exists. They found v2 failing on questions whose answers sat in tables, which is why ingestion was improved before anything else. And they built v4 for only one category of question, "who owns this service and what changed recently?", where measurements showed the RAG pipeline could not succeed.

⑤ The architecture: an offline path and an online path

Nearly every AI system that uses data splits cleanly into two paths, and stating the split is the single most useful thing to say at the start of an architecture discussion.

The offline path keeps the index current; the online path answers questions. They meet only at the index, so ingestion can be slow, thorough and retried without affecting answer latency, and serving can scale without touching ingestion.

The offline path runs continuously in the background: connectors notice new and changed documents, extraction keeps their structure, chunking and embedding prepare them, and the index stores them with their permissions and dates. It can be slow and thorough, because nobody waits for it; it is built like the pipelines of 12.4.4, with queues and retries.

The online path runs once for every question, and it must finish inside the time budget. It has six steps:

1. The gateway of 12.12.1 works out which engineer is asking.
2. The question is rewritten into a search query.
3. Hybrid retrieval runs, with the permission filter inside the search (12.14.2). A document the engineer may not see is never even found.
4. A reranker picks the best passages (12.9.3).
5. The model writes an answer from those passages only, with citations (12.9.4).
6. Output checks run, and the answer streams back to the engineer (12.8.5).

The two paths meet only at the index. That separation is what lets ingestion take as long as it needs and be retried freely, while serving stays fast and scales on its own.

⑥ The evaluation plan

Decide how you will know it works before building most of it, because the evaluation set is what turns every later decision into a measurement (12.8.6, 12.9.5). For the assistant, the set has three parts:

• 300 real questions collected from engineers' chat channels. Each comes with the documents that answer it, and a reference answer written by someone who knows.
• 40 questions whose answers are deliberately missing. The right behaviour is to say so.
• 30 questions from engineers who are not allowed to see the document with the answer. The right behaviour is not to reveal it.

Retrieval is measured separately from the answer. First, did it find the right documents? Then, is the answer faithful to them, and correct? That way, a failure points at the stage that caused it.

4. Make it safe to run

⑦ Safety, privacy and governance

The questions of Chapter 12.14, asked of this system. Permissions: each document's access list is copied into the index and kept in sync, and every search is filtered by the engineer's identity. Injection: the documents were written by employees, but tickets include text pasted from customers and vendors, so the assistant has no tools that act or send anything outside, and its output is cleaned of unapproved links and images; the lethal trifecta is never complete. Privacy: the drives contain personal data, so HR folders are excluded from ingestion entirely, traces are masked and kept 30 days, and the provider's terms exclude training on inputs. Governance: the system is medium tier in the scheme of 12.14.3: owner sign-off, security review and a monitoring plan.

⑧ Serving, latency and cost

A latency budget, as in 12.12.3: gateway and rewrite 300 ms, retrieval 150, reranking 150, model first token under a second, streaming after that. A cost estimate, as in 12.10.10: 2,000 questions a day, each with about 5,000 input tokens (instructions and passages, the instructions cached) and 400 output tokens, on a mid-sized model at $2 and $10 per million tokens, is roughly 2{,}000 \times (5{,}000 \times 2 + 400 \times 10) / 10^6 \approx \$28 a day, under $1,000 a month, well within budget. The reranker and embedding model run on one self-hosted GPU. Caching (12.12.2) is limited to embeddings and retrieval, because answers depend on each engineer's permissions.

⑨ Launch and the lifecycle

Release to one team first, then a department, then everyone, each stage compared on the evaluation set and on real feedback (12.12.4). Monitor the signals of 12.12.5: questions with no good retrieval match (missing documents), decline rates, thumbs-down, questions engineers rephrase. Every thumbs-down is reviewed weekly; the good ones become evaluation cases. The loop from step ⑨ back to step ① is how the system improves: the "who owns this?" questions that led to v4 were discovered this way.

5. What the team would do differently

A design is judged in production, and the platform team's own retrospective after six months is worth more than any diagram.

What worked. Reranking, once added, produced the largest single improvement users noticed, easily worth its 150 milliseconds. Streaming made the assistant feel fast even when answers took several seconds. Citations made engineers trust it: they clicked through, checked, and came back.

What did not. Tables were chunked as plain text for the first two months, and questions answered by tables, such as service owners, on-call rotations and capacity numbers, failed constantly. Extracting tables as structured data fixed most of them, and in hindsight should have come before any work on prompts.

What they would change. Build the evaluation set on the first day. For the first six weeks, quality was judged by engineers reading answers, and when the evaluation set finally existed, it showed that two "improvements" to the prompt had quietly made citations worse. Every decision made without measurement had to be checked again.

What the interviewer will push on

"Design a question-answering system over our internal documents." They are looking for the structure of the answer as much as its content. A strong answer moves through the nine steps in about this order:

1. Ask who the users are, what they will ask, and how success will be measured. Then ask about the limits: permissions, speed and budget.
2. Choose retrieval over fine-tuning, and say why.
3. Describe the offline path and the online path, and where they meet.
4. Put the permission filter inside the search.
5. Use hybrid retrieval and reranking. Give answers that cite their sources, and say "I don't know" honestly when the documents do not have the answer.
6. Build an evaluation set that includes questions with no answer and questions the user is not allowed to ask. Measure retrieval and answer quality separately.
7. Add protection against prompt injection and privacy leaks.
8. Give a time budget and a cost estimate, with real numbers.
9. Launch in stages, and keep watching it after launch.

"How do you keep it current?" Incremental ingestion from change events or regular polling, re-embedding only changed documents, deleting removed ones, syncing permission changes, and preferring the newest version of a document when several exist.

"What would you build first?" Baselines and the evaluation set, because they show which failures are real. Mention that measured failures, not imagined ones, should drive each later version.

"When would you make it an agent?" Only for question types that measurably need several dependent lookups across systems, with read-only tools, step limits and the controls of Chapter 12.10.

Something to volunteer: the permission constraint changes the architecture. Answer caching, shared indexes and evaluation all have to account for the fact that two engineers asking the same question may be entitled to different answers, and saying so early shows you have built one of these systems.

Recall

• Nine steps for any AI system:
  1. The problem, and what success means in numbers.
  2. Whether ML is needed at all, and which kind.
  3. The data.
  4. A simple baseline.
  5. The architecture.
  6. The evaluation plan.
  7. Safety, privacy and governance.
  8. Serving, speed and cost.
  9. Launch and upkeep, which feeds back into step 1.
• Write constraints first (permissions, latency, budget); late constraints are expensive.
• Build baselines (keyword search, smallest RAG) and measure; let measured failures drive each version: hybrid search for identifiers, reranking for precision, structured tables, agents only where needed.
• Split the architecture into an offline ingestion path and an online serving path that meet only at the index.
• Evaluation from day one: real questions with sources, unanswerable cases, unauthorised cases; retrieval and generation measured separately.
• Safety: permissions copied, synced and filtered inside the search; no acting or sending tools; cleaned output; privacy exclusions and retention.
• Show the latency budget and the cost arithmetic; launch in stages; turn feedback into evaluation cases.

Self-test: Why did v1 fail on "what happened in INC-4921?", and which version fixed it? Recompute the daily cost if questions rise to 5,000 a day and each uses 8,000 input tokens. Why can the assistant cache embeddings and retrieval results but not final answers?

Next: 12.15.2 Case Study: Semantic Product Search — the method applied to the system most Cartwise customers use first: the search box, from a query typed in Hinglish to twenty ranked products in under 200 milliseconds.