12.15.3 — Case Study: The Customer Support Assistant

"Ask Cartwise" has appeared on almost every page of this Part, one piece at a time: a single model call in 12.8.1, tool calls for order status in 12.8.4, answers from the policy documents in Chapter 12.9, the refund agent of Chapter 12.10, the gateway of 12.12.1, the phone line of 12.13.2, the injection incident of 12.14.1. This case study puts every piece in its place, follows one real conversation through the whole system, and reports what a year in production showed: what it achieved, what broke, and what the team would do differently.

1. The problem and success, in numbers

Volume. About 100,000 conversations a day across the app, the website and email, plus 40,000 phone calls, in English, Hindi and nine other languages. Before the assistant, 900 support staff handled them, with a median first response of four hours for chat and email.

Success, defined before building and tracked every week:

• Resolution without a person: the share of conversations fully resolved by the assistant, with no repeat contact about the same issue within seven days. Raw "conversations not handed over" is not enough, because an assistant that gives up on nobody but solves nothing would score perfectly.
• Customer satisfaction on a short survey after each conversation, and the repeat-contact rate.
• Correctness: on sampled conversations, were policies applied correctly and were facts true?
• Cost per resolved case, the measure of 12.10.10, including the staff time for hand-overs and approvals.
• Hand-over quality: when a person takes over, do they have what they need, or does the customer have to start again?

Constraints. First response in seconds; refunds and account changes within policy limits enforced in code; customers' data visible only to them; every answer about policy traceable to a source; a person available whenever the customer asks for one.

2. The architecture

The system is not one assistant. It is a router in front of several specialised paths, each built with the least freedom its job needs, the principle of 12.10.6.

"Ask Cartwise" in production. Every channel enters through one gateway; a router sends each message to the path built for it, from a cited RAG answer to the refund agent to a person; every reply is checked on the way out.

Each part, and the page that built it:

Part	What it does	Built in
Channels	chat, email, phone with speech	12.8.5, 12.13.2
Gateway	identity, quotas, input checks, logging	12.12.1
Memory	profile and recent cases at the start	12.10.5
Router	small model picks the path; unsure goes to a person	12.10.6
Policy path	hybrid retrieval, rerank, grounded cited answer	12.9.1–12.9.6
Orders path	workflow with read-only tools	12.8.4
Refund agent	harness, tools, policy code, approvals, payments	12.10.2–12.10.8
Hand-over	person gets structured case notes	12.10.5
Output checks	contact details, claims, links and images	12.10.8, 12.14.1
Cost controls	caching, routing to cheap models, budgets	12.10.10, 12.12.2
Measurement	traces, evaluation sets, monitoring	12.10.9, 12.12.5

Three design decisions shaped everything else, and each is a lesson from earlier chapters made concrete.

Most messages never reach an agent. Policy questions and order status, three-quarters of all messages, go to workflows with fixed steps: a router, a retrieval step and one answer; or a router, two tool calls and one answer. Only refunds, returns and disputes, where the right steps depend on what is found, reach the agent. This keeps most conversations cheap, fast and predictable.

The model never holds power it cannot be trusted with. The refund agent's tools check ownership against the session; amounts, limits and approval thresholds live in code; payments are idempotent; no path can send anything except a reply to this customer. The lethal trifecta of 12.14.1 is never complete.

A person is always one step away. "Talk to a person" is a button on every reply and a recognised request on the phone; the router sends uncertain or sensitive conversations straight to a person; and every hand-over includes the structured case notes, so the customer never has to explain twice.

3. One conversation, end to end

On a Monday morning, Priya, whose earphones arrived damaged in 12.10.1 and whose parcel went to the wrong building in 12.10.5, writes in the app: "Mera naya order phir galat building mein chala gaya. Same as last time. Mujhe refund chahiye." ("My new order went to the wrong building again. Same as last time. I want a refund.")

1. Gateway (5 ms): identifies Priya from her app login, checks the chat quota, screens the message (nothing suspicious), and starts a trace.
2. Memory (10 ms): loads her profile, "deliveries: Block B, not main gate", and her last case: "order 4471, wrong building, 14 Oct, refunded, courier warned".
3. Router (120 ms): a small model classifies the message as a delivery dispute with a refund request, confident, and sends it to the refund agent. The mixed Hindi and English is no obstacle; the router was trained and evaluated on such messages.
4. Refund agent, step 1: calls list_recent_orders, finds order 5208, marked delivered yesterday at the main gate.
5. Step 2: search_policies("delivered to wrong address refund") returns the delivery-dispute policy: a refund or reshipment is due when the courier's delivery proof does not match the customer's instructions.
6. Step 3: get_delivery_proof(5208) returns the courier's photo and note: left with the gate guard. Her saved instruction says Block B. The proof does not match.
7. Step 4: the agent replies, in Hindi and English as she wrote, offering a refund of ₹1,299 or reshipment to Block B, and mentioning that it has seen this happened before with order 4471. She chooses the refund.
8. Step 5: propose_refund(5208, 1299, "not_delivered"). The policy function checks ownership and amount, notes this is her second wrong-building case in a month (a pattern rule), and, because the amount is under the automatic limit and the evidence is clear, allows it. Payment is issued with an idempotency key.
9. Output checks: the reply contains only her own order number; "refund has been issued" matches the run record; no links outside Cartwise.
10. After the run: the case is recorded in her memory; a courier-quality alert for her address goes to the logistics team, triggered by the repeat pattern; the trace is stored.

Total time to her first reply: 2.8 seconds. Model cost: about 4 US cents. And the person who would previously have handled this, four hours later, was instead working on the cases the assistant handed over.

4. Measurement

Before release

Each path has its own evaluation set, built the way 12.8.6, 12.9.5 and 12.10.9 taught. There are four sets:

• Policy: about 400 policy questions, each with its source and a reference answer. Some of them have no answer in the documents, on purpose.
• Orders: about 200 order and delivery conversations, run against a pretend order system.
• Refunds: about 250 refund and dispute cases, run in a pretend world with pretend customers. Each case is judged on two things: what the payments system looks like at the end, and whether the required steps were taken. Each case runs eight times, for the pass^k score.
• Router: 3,000 labelled messages, covering all eleven languages.

Red-team cases for injection, cross-customer access and abuse run on every release. A release must hold or improve every set, slice by slice, before it reaches any customer.

In production

A weekly dashboard tracks the success measures of section 1, split by path, language, channel and release version, together with the signals of 12.12.5: retrieval matches, decline rates, faithfulness on a daily sample, tool error rates, steps and cost per run as distributions, approvals declined, and guardrail triggers. Every week, a support lead and an engineer read fifty failed conversations together and sort them by cause, the error analysis that drives the next round of work.

5. A year in production

The headline numbers, comparing the last quarter of the year with the quarter before launch:

Measure	Before	After one year
Median first response (chat)	4 hours	3 seconds
Resolved without a person	n/a	64%
Repeat contact within 7 days	21%	14%
Satisfaction (1–5)	3.6	4.1
Cost per resolved case	about $1.50	about $0.48

The cost figure follows the formula of 12.10.10: the model cost of the average conversation plus the hand-over rate times the cost of a person's time, plus approvals. The support staff were not replaced; most moved to the complex cases the assistant hands over, and to the phone queue for customers who prefer a person, and their median handling time fell because every hand-over arrives with case notes.

The incidents, each already told in its chapter and each leaving a permanent check behind:

Incident	What happened	Page	What it left behind
Diwali rate limits	three features failed on a shared quota	12.12.1	per-feature quotas in the gateway
Unapproved promise	a long context buried a rule	12.10.5	context clearing; rule in the system prompt; claim check
October prompt change	policy lookup skipped, cost doubled	12.10.9	path checks and step budgets in the release gate
Pressure-cooker seller	injected text in a seller answer	12.14.1	moderation, spotlighting, brand-mention check
Groceries launch	questions with no documents	12.12.5	retrieval-match alert; new policies before launches

What the team would do differently. Build the router evaluation set in all eleven languages from the start; the first router was trained mostly on English and Hindi and misrouted Tamil and Bengali messages at three times the rate until that was fixed. Write the policies for a new product line before it launches, because the assistant can only be as good as its documents. And invest in hand-over quality earlier: the single change that customers noticed most was not a smarter model but not having to repeat themselves when a person took over.

What the interviewer will push on

"Design a customer support assistant for an e-commerce company." This is the most common applied LLM design question, and the architecture on this page is the answer:

1. Every channel comes in through one gateway.
2. Memory keeps track of the conversation and the customer.
3. A router sends each message down a specialised path. Policy questions go to RAG with citations. Order status goes to a fixed tool workflow. Refunds go to an agent whose limits and approvals are enforced in code. Everything else goes to a person.
4. Output checks run on every answer.
5. Each path has its own evaluation set, and production metrics watch it after launch.

The tell is routing most traffic to simple workflows rather than one large agent.

"How do you measure whether it's working?" Resolution without a person, defined as no repeat contact within seven days; satisfaction; correctness on samples; cost per resolved case including staff time; hand-over quality; all split by path, language and version.

"How do you keep it safe while letting it issue refunds?" Ownership checks from the session, policy limits and pattern rules in code, approvals above thresholds, idempotent payments, no outbound channels, output checks, and kill switches per path.

"What happens when it can't help?" A warm hand-over: the router and the agent can pass to a person at any point, with structured case notes, and a "talk to a person" option is always visible.

Something to volunteer: the most noticed improvement was hand-over quality, not model quality. Designing the assistant so a person can pick up a case instantly is as much a part of the system as anything the model does.

Recall

• Success measured before building: resolution without a person (no repeat contact in seven days), satisfaction, correctness, cost per resolved case, hand-over quality.
• Architecture, in order:
  1. Channels: chat, email, and phone with speech.
  2. The gateway.
  3. Memory.
  4. The router, which picks a path: RAG with citations for policy, a tool workflow for orders, a refund agent whose rules and approvals are enforced in code, or a person, handed the case notes.
  5. Output checks.
  6. Underneath it all: traces, evaluations, metrics and budgets.
• Most messages go to workflows; the agent handles only refunds, returns and disputes; the model never holds unchecked power; a person is always one step away.
• Each path has its own evaluation set (RAG, simulated orders, simulated refunds with pass^k, multilingual router, red-team), gating every release; production dashboards and weekly failure review.
• After a year: first response from hours to seconds, 64% resolved without a person, fewer repeat contacts, cost per resolved case about a third. Every incident left a permanent check.

Self-test: Why does "resolved without a person" require no repeat contact within seven days? In Priya's conversation, which component decided the refund could go through without approval, and on what grounds? Name two incidents from the year and the check each one left behind.

Next: 12.15.4 Case Study: Document Intelligence — from conversations to paperwork: turning thousands of messy supplier invoices and seller documents a day into checked, structured data.