12.8.6 — Evals

In its third week, a support lead noticed that "Ask Cartwise" was getting the electronics return window wrong. An engineer rewrote two lines of the system prompt, tried the failing question and five others, saw correct answers, and shipped the change that afternoon. It fixed returns. It also, unnoticed, made the assistant reply in English to customers who wrote in Hindi, because one of the rewritten lines said "write in plain, warm English". Nobody found out for nine days, until complaints from Hindi-speaking customers reached a manager.

The engineer had done what almost everyone does at first: try a handful of examples by eye and decide it looks better. People call this a vibe check. It fails for a simple reason. An LLM system has thousands of different kinds of input, and a change that improves the five you looked at can quietly break the fifty you did not. You cannot improve what you cannot measure, and for open-ended text, measuring takes deliberate design.

An eval (short for evaluation) is that design: a fixed collection of realistic inputs, a way of judging each output, and a score you can compare from one version of your system to the next. It plays the role the held-out test set played in 12.3.2, adapted to outputs that have no single right answer. Evals are the most important practice in building with LLMs, and teams that do them well improve steadily while teams that skip them go in circles.

1. What an eval is made of

Every eval has three parts.

A set of cases. Each case has two parts. The first is an input: a customer message, perhaps with the conversation so far and the order data the system would have. The second is whatever you need to judge the output. That might be a reference answer, a list of facts the answer must contain, or, for a routing case, the right team.

A grader. Something that looks at the system's output for a case and decides how good it is: a few lines of code, a model acting as a judge, or a person.

A score, broken down. The overall pass rate, and the same rate per category of case (returns questions, delivery questions, Hindi messages, angry customers), because an overall number can stay flat while one category collapses. The Hindi failure would have shown up immediately as a drop in one category.

2. Building the case set

The eval is only as good as its cases, and the best cases come from reality.

Start from real traffic. Take a random sample of actual customer conversations (with personal details removed), so the eval reflects what customers really ask, in the words they really use, with the typos and mixtures of Hindi and English they really write. Invented test questions are always cleaner and easier than real ones.

Add every failure you find. Each bug report, each complaint, each odd answer spotted in logs becomes a new case, with the correct behaviour written down. Over time, this is how an eval comes to cover the things that actually go wrong, and it guarantees that a fixed problem stays fixed.

Add the edges deliberately. Questions the assistant must decline, questions whose answer is not in the policies, customers who are angry, customers who try to talk it into promising a refund, very long messages, messages in each supported language.

Label what "good" means for each case. For a routing case, the correct team. For a support answer, usually a short list of what the answer must do: states the 10-day window for electronics; offers to start a return; does not promise a refund date; replies in the customer's language. Lists of checkable properties are far easier to grade consistently than one "ideal answer" to compare against.

How many cases? Enough that a difference you care about is bigger than the noise. The standard error from 12.2.5 tells you how much. With 100 cases and a pass rate around 80 percent, the standard error is \sqrt{0.8 \times 0.2 / 100} = 0.04, so the true rate is plausibly anywhere from about 72 to 88 percent; a change from 80 to 83 percent on 100 cases means nothing. With 400 cases the standard error halves to 0.02. A practical path is to start with 50 to 200 well-chosen cases, which is enough to catch large regressions like the Hindi one, and to grow the set as failures are found. Keep a portion of it aside that you do not look at while tuning prompts, for the same reason as the test set in 12.3.3: a prompt tuned against the same cases over and over gets good at those cases, not at the job.

3. Graders, from cheapest to most flexible

Code checks

Many properties can be checked exactly, quickly and for free with ordinary code, and those should always be checked that way:

• the router chose the expected team (exact match);
• the output is valid against its schema (12.8.3);
• the answer called get_order_status before stating a delivery date (12.8.4);
• the answer is under 80 words;
• the answer does not contain forbidden phrases such as "guaranteed refund" or "within 24 hours";
• the answer's language matches the customer's, using a language-detection library.

Use a model as a judge only for what code cannot check.

A model as the judge

Whether an answer is correct according to the policy, polite, relevant to the question, or supported by the retrieved documents cannot be checked by matching strings: the same correct answer can be worded a thousand ways. The modern approach is to have a capable LLM grade the output against a written rubric, called LLM-as-a-judge.

from typing import Literal
from pydantic import BaseModel, Field

class Grade(BaseModel):
    criterion: str
    reasoning: str = Field(description="Two sentences: what the answer did, and why that passes or fails")
    verdict: Literal["pass", "fail"]

class Grades(BaseModel):
    grades: list[Grade]

JUDGE_PROMPT = """You are grading one reply from Cartwise's support assistant.
Judge each criterion separately, using only the policy excerpt and the reply.

<policy>{policy}</policy>
<customer_message>{message}</customer_message>
<reply>{reply}</reply>
<criteria>
{criteria}
</criteria>"""

def judge(case, reply):
    r = client.messages.parse(                                        # (1)
        model="claude-opus-5-5", max_tokens=2048,
        messages=[{"role": "user", "content": JUDGE_PROMPT.format(
            policy=case.policy, message=case.message, reply=reply,
            criteria="\n".join(f"- {c}" for c in case.criteria))}],  # (2)
        output_format=Grades,
    )
    return r.parsed_output.grades                                     # (3)

Line (1) calls a strong model with structured output, so every grade comes back as data that can be counted. Line (2) inserts the case's own list of criteria, such as "states the 10-day electronics window", so the judge grades specific, checkable properties rather than forming an overall impression. Line (3) returns one verdict per criterion, each with a short reason placed before the verdict, so that the judge reasons first and decides second, for the reason explained in 12.8.3.

Scores out of 5, and why pass-or-fail usually works better

A common judge format asks for a score on a scale such as 1 to 5, with a description for each level: "1: rude or dismissive; 3: correct but robotic; 5: warm, correct and solves the problem". This is a Likert scale, named after the American psychologist Rensis Likert, who introduced agree-to-disagree rating scales for attitude surveys in 1932. Likert-scale judges are popular because they look nuanced, and they are useful when quality genuinely comes in degrees, such as tone. They have three weaknesses that matter in practice. Judges, like people, drift towards the middle of the scale. The difference between a 3 and a 4 is hard to define, so the same answer can score differently on different runs. And an average of 3.8 does not tell you what to fix.

So the advice that most experienced teams converge on is: break quality into specific criteria and grade each one pass or fail, as in the code above. "Did it state the correct window: yes or no" is consistent between runs, directly actionable, and easy to check against a person's judgement. Keep Likert scales for genuinely gradual qualities, and always define every level with an example.

Pairwise comparison

When deciding between two versions of a prompt or two models, it is often easier and more reliable to show the judge (or a person) both answers to the same question and ask which is better, rather than scoring each alone. People and models are better at comparing than at scoring on an absolute scale, which is the same reason 12.7.9 used comparisons to train assistants. Many comparisons can be combined into a ranking with an Elo rating, the system invented by Arpad Elo for ranking chess players, which is how public leaderboards rank chat models.

Trusting the judge

An LLM judge is itself a model, with known biases. It tends to prefer the answer shown first (position bias), to prefer longer answers (verbosity bias), and to prefer answers written in its own style (self-preference). Three habits keep it honest. Swap the order in pairwise comparisons and count a win only if it holds both ways. Grade specific criteria, which leaves less room for overall impressions. And check the judge against people: have a person grade 50 to 100 cases, measure how often the judge agrees using the Cohen's kappa from 12.4.1, and fix the rubric until agreement is high. A judge that disagrees with careful humans is measuring something other than what you care about.

Older text-overlap metrics, briefly

You will see metrics called BLEU (from 2002, for machine translation) and ROUGE (from 2004, for summarisation), which score an output by counting how many short word sequences it shares with a reference text. They are cheap and still used in translation research, but they cannot recognise a correct answer that uses different words: "the cat slumbered" shares almost nothing with the reference "the feline slept". BERTScore improves on this by comparing embeddings, from 12.6.2, instead of exact words. For assistants, criteria graded by code and by a calibrated judge have largely replaced them.

People

Human review remains the final authority, and it is not optional: someone must define what good looks like, label the cases, and check the judge. The skill is to spend expensive human time where it matters most, on labelling, on calibrating judges, and on reading failures, rather than on grading every output by hand.

4. Running evals, and the release gate

An eval run is a simple loop: for every case, run the whole system (prompt, retrieval, tools and all) exactly as production would, grade the output, and report the scores by category alongside the previous version's.

def run_eval(system_version, cases):
    rows = []
    for case in cases:
        reply = run_assistant(system_version, case)                  # (1) the real pipeline
        checks = code_checks(case, reply)                            # (2) cheap, exact
        grades = judge(case, reply) if all(checks.values()) else []  # (3) judge only what passes
        rows.append({"category": case.category,
                     "passed": all(checks.values()) and all(g.verdict == "pass" for g in grades)})
    return pass_rate_by_category(rows)                               # (4)

before, after = run_eval("prompt-v7", CASES), run_eval("prompt-v8", CASES)
for cat in before:
    print(f"{cat:12} {before[cat]:.0%} → {after[cat]:.0%}")
# e.g.  returns      71% → 94%
#       hindi        96% → 12%      ← the regression the vibe check missed

Line (1) runs the full assistant on the case, because evaluating the prompt alone would miss failures in retrieval or tool use. Line (2) runs the code checks first, and line (3) spends money on the judge only for outputs that passed them. Line (4) reports pass rates per category, which is what makes the Hindi collapse jump off the screen. The printed numbers are illustrative of the kind of result the team saw when they finally ran it.

Once this exists, it becomes a release gate: no prompt change, model upgrade, retrieval change or tool change ships until the eval has been run and no category has dropped beyond its normal noise. Changing the model version (12.8.1) goes through the same gate. Because a full run calls the model hundreds of times, keep its cost in mind, use a fast subset during development and the full set before release.

5. Evaluating in production

The eval set is a fixed sample of the past. Production is where you discover what it missed. Three kinds of signal come from live traffic:

• Graded samples of real conversations. Each day, run the judge on a random sample of production conversations and chart the pass rates. A drop points at something that changed: a new policy, a new kind of question, an upstream data problem.
• User signals. Thumbs up and down, whether the customer asked for a human, whether the same customer contacted support again about the same order within a day. These are noisy proxies, as 12.4.1 warned about natural labels, but they are real, cheap, and they capture what customers actually experienced.
• Live experiments. For big changes, send a share of traffic to the new version and compare business outcomes, the online A/B testing of Chapter 12.12.4.

Production failures then feed straight back into the eval set: the loop of 12.1.4, where traces of real failures become cases, then fixes.

6. Error analysis: reading the failures

A score says how much is wrong; only reading failures says what. The most productive habit in LLM engineering is simple and often skipped: regularly read 50 to 100 failing conversations end to end, write a short note on what went wrong in each, then group the notes into categories and count them. A typical first pass for "Ask Cartwise" found something like: the policy needed was not retrieved; the policy was retrieved but an older version was also present; the model ignored the tool result; the answer was right but too long; the customer's question was genuinely ambiguous. Each category has a different fix, in a different part of the system, and counting them tells you which fix is worth the most. For retrieval systems there is a standard map of these failure points, worked through in Chapter 12.9.5.

What the interviewer will push on

"How do you know your LLM feature is good?" They want an eval set built from real traffic and failures, graders matched to properties (code checks first, a calibrated judge for the rest), per-category scores, and a release gate. "We tried it and it looked good" is the answer that ends the interview.

"How do you trust an LLM judge?" Specific pass-or-fail criteria, reasoning before verdict, order swapping for pairwise comparisons, and agreement with human labels measured with kappa. Naming position and verbosity bias is the tell.

"Likert scale or binary?" Binary per criterion for consistency and actionability; Likert, with every level defined, only for genuinely gradual qualities.

"How big should the eval set be?" Big enough that the differences you care about exceed the standard error; a few hundred cases for fine distinctions, fewer to catch large regressions, and a held-out part you never tune against.

Recall

• An eval is a fixed set of realistic cases, a grader and a score broken down by category; it replaces vibe checks, which miss regressions outside the examples you looked at.
• Build cases from sampled real traffic, every failure found, and deliberate edge cases; label each with checkable properties. Size it against the standard error (100 cases at 80 percent: ±8 points) and keep a held-out part.
• Grade with code wherever possible; use LLM-as-a-judge with specific criteria, reasoning before verdict and structured output for the rest. Prefer pass or fail per criterion to a Likert scale, which drifts to the middle; use pairwise comparison when choosing between versions.
• Judges have position, verbosity and self-preference biases: swap order, grade criteria, and calibrate against human labels with kappa. BLEU and ROUGE cannot recognise paraphrases.
• Run the whole system, report per category, and gate every release on it. In production, grade samples, watch user signals, run experiments, and read failures to find the biggest category to fix.

Self-test: Why would the Hindi regression be invisible in an overall pass rate of 85 percent? With 400 cases at a 90 percent pass rate, what is the standard error? Why should a judge write its reasoning before its verdict?

Next: 12.8.7 Frameworks and When to Skip Them — calls, prompts, structured output, tools, streaming and evals are the whole toolkit. The last page of the chapter looks at the frameworks that package them, LangChain, LangGraph, LangSmith and Prompt Flow, and at when they help.