12.10.9 — Observing and Evaluating Agents

In October, Cartwise made two changes to the refund agent in the same week: a newer model, and a rewritten system prompt that was shorter and friendlier. The evaluation from 12.8.6, which graded the agent's final replies against the expected resolution, came back at 92%, the same as before. The change shipped.

Two weeks later, finance noticed the agent's monthly bill had nearly doubled. An engineer opened a few dozen traces and found the reason, and a worse problem behind it. Runs were taking 9 steps on average instead of 5, because the new prompt no longer told the agent to look up the order first, so it often asked the customer questions the order record already answered. And in about one case in eight, the agent never searched the policies at all. It decided from what it remembered about typical return rules, which happened to match Cartwise's policy in most cases. The final answers were mostly right. The way they were reached was wrong, expensive, and one policy change away from being confidently wrong everywhere.

This is the specific problem with evaluating agents. An agent's output is not just a reply; it is a path: a sequence of decisions and actions that produced the reply. Grading only the end of the path misses everything above. This page covers how to see the path (tracing), what to watch in production (metrics), and how to evaluate paths and outcomes before a change ships.

1. Tracing: recording the whole path

A trace is the complete record of one run, organised as a tree of timed steps. Each step is called a span: the run itself is the top span; inside it, each model call is a span, each tool call is a span, and a subagent from 12.10.7 is a span with its own children. The vocabulary comes from distributed tracing in ordinary software, where one web request passing through many services is traced the same way (Chapter 10.10). The industry standard for recording traces, OpenTelemetry, now includes agreed names for AI-specific details such as the model used and the tokens consumed, so traces from different tools look alike.

The figure shows customer A's refund run from 12.10.1 as a tracing tool would display it, each bar a span placed on a timeline.

One run as a trace: model calls (purple), tool calls (green), a nested retrieval span (cyan) and two human pauses (amber, not to scale). Reading down the left column tells the story of the run; the tokens on each model call show the context growing.

Reading the trace top to bottom tells the story of the run, and each span records what you need to understand its part:

• Model calls: the model and settings, the full input and output (including thinking, when available), the stop reason, input and output tokens, cost, time to first token and total time, and the provider's request ID.
• Tool calls: the tool, the arguments the model supplied, the result returned, whether it was an error, and how long it took.
• Decisions by the harness: every authorize outcome from 12.10.8, every output check, every pause and resume, and why the run ended.
• The run as a whole: customer, channel, agent version (model, prompt and tool versions together), final status, total cost, and any later outcome, such as whether the customer contacted support again.

The 12.10.2 harness already writes one JSON line per step; a tracing tool turns those lines into views like the figure, with search and filtering. Tracing products such as LangSmith from 12.8.7 or the open-source Langfuse do this, and any system that accepts OpenTelemetry data can too.

Two cautions. Traces contain everything the customer said and everything the tools returned, so they are personal data: restrict who can read them, remove or mask details that are not needed, and delete them on a schedule (12.14.3). And recording the agent version on every trace is what makes "did last Tuesday's change cause this?" answerable at all.

2. What to watch in production

Traces explain individual runs; metrics computed over all runs tell you when to look. For an agent, the useful metrics fall into four groups.

Group	Metric	What it catches
Outcome	resolved without a person	overall usefulness
Outcome	customer contacts again in 7 days	wrong "resolutions"
Outcome	approvals declined	bad proposals
Path	steps per run, p50 and p95	wandering, loops
Path	runs missing a required tool	skipped checks
Path	same call repeated 3+ times	stuck loops
Cost	cost per run, p50, p95, max	context growth, loops
Health	errors per tool	broken tools, bad descriptions
Safety	guardrail triggers by type	attacks, regressions

Three habits make these numbers useful. Watch distributions, not averages: the p95 (the value 95% of runs stay below) and the maximum show the runaway runs that an average hides, and runaway runs are where the money goes. Watch path metrics as closely as outcomes, because the October problem showed up in "runs missing a required tool" two weeks before anyone noticed the answers. And split every metric by agent version, so that a change's effect is visible the day it ships. "Runs missing a required tool" would have jumped from 0% to 12% on the day of the prompt change.

3. Evaluating before shipping: outcome, path, and steps

Production metrics find problems after customers have met them. An agent evaluation set finds them before. It is the eval set of 12.8.6 with three differences, all of which come from the agent being a path that changes things rather than a single reply.

Run the agent in a simulated world

An agent's tools touch real systems, so evaluation runs it against simulated ones: an order database seeded with exactly the orders each case needs, a payments system that records refunds instead of making them, a policy index with the current policies. And because the refund agent holds conversations, the customer is simulated too: a model is given a persona and the facts of the case and plays the customer, answering the agent's questions the way a real customer might.

This setup was popularised by τ-bench (pronounced "tau-bench"), published in June 2024 by Shunyu Yao and colleagues at the AI company Sierra. It tests agents on customer-service tasks in a simulated retail shop and airline, with a model playing the customer, and it judges each conversation by the state of the database at the end rather than by the wording of the replies. A case for Cartwise looks like this:

@dataclass
class AgentCase:
    name: str
    world: dict              # orders, payments, policies the simulated systems start with
    customer: str            # persona and hidden facts for the simulated customer
    expected: dict           # the parts of the final state that must match
    must_call: list[str]     # tools that must appear in the path
    must_not_call: list[str] # tools that must not
    max_steps: int

DAMAGED_IN_WINDOW = AgentCase(
    name="damaged earphones, inside window, over photo threshold",
    world={"orders": [ORDER_4471_DELIVERED_13_OCT], "policies": CURRENT_POLICIES},
    customer=("You are Priya. The earphones from order 4471 arrived with the left side "
              "dead. You have a photo and send it if asked. You want your money back, not a "
              "replacement. Only give details when asked. Say DONE when the matter is settled."),
    expected={"refunds": [{"order_id": "4471", "amount_rupees": 2499, "reason": "damaged"}]},
    must_call=["get_order", "search_policies", "ask_customer"],
    must_not_call=[],
    max_steps=8)

The customer text is the simulated customer's whole brief. It includes a fact the agent must discover by asking (the photo) and a preference (money, not replacement) that should change the agent's choice. The expected field is checked against the payments system's records at the end. The agent passes only if it actually refunded ₹2,499 for the right reason, not if it merely said it would.

Check the outcome, the path, and the budget

Each case is graded on three things:

def run_case(case: AgentCase, trials: int = 8) -> list[bool]:
    outcomes = []
    for _ in range(trials):
        world = SimulatedWorld(case.world)                                        # (1)
        customer = SimulatedCustomer(case.customer, model="claude-sonnet-5-5")    # (2)
        run = run_conversation(refund_agent, world, customer)                     # (3)
        used = [span["tool"] for span in run.trace if span["kind"] == "tool"]
        outcomes.append(
            world.matches(case.expected)                                          # (4) outcome
            and all(t in used for t in case.must_call)                            # (5) path
            and not any(t in used for t in case.must_not_call)
            and run.steps <= case.max_steps)                                      # (6) budget
    return outcomes

Line (1) builds a fresh simulated world for every trial, so trials cannot affect each other. Line (2) creates the simulated customer from the case's brief. Line (3) runs the real agent, unchanged, against the simulated world and customer, through the same harness as production. Line (4) checks the outcome in the world's state: the right refunds, and no others. Line (5) checks the path: required tools used, forbidden ones not. This is the line that would have caught October's problem, since search_policies was required. Line (6) checks the budget, so a change that makes the agent twice as slow fails even if it gets the right answer.

The must_not_call list matters as much as must_call. For a case where the order is outside the return window, propose_refund belongs in must_not_call, and passing means the agent explained the warranty instead. For a case where the customer tries the "supervisor mode" message from 12.10.8, nothing about the refunds should change at all.

Path checks should be as loose as the real requirement. Requiring that get_order comes before any refund is right, because that is a rule. Requiring an exact sequence of seven calls is wrong, because there are many good paths, and a check that accepts only one will fail correct runs.

Judge the path with a model, when rules are not enough

Some qualities of a path are matters of judgement: did the agent ask sensible questions, recover well from a tool error, avoid needless steps, explain its decision clearly? For these, a model reads the trace and grades it against a rubric, the model-as-judge approach of 12.8.6 applied to the whole trace:

You are reviewing a support agent's handling of one case. The trace is below.
Answer each question with PASS or FAIL and one sentence of evidence from the trace.
1. Did the agent base its decision on a policy it retrieved in this run?
2. Did it ask the customer only for information the tools could not provide?
3. When a tool returned an error, did it change its approach rather than repeat the call?
4. Did it avoid steps that contributed nothing to the outcome?
5. Is the final reply accurate about what was actually done?

Each question is narrow and answerable from the trace, which is what makes judging reliable. The answers are pass or fail with evidence rather than a score out of 10, for the reasons 12.8.6 gave. And the judge is checked against a sample of human-graded traces before its verdicts are trusted.

4. One run is not enough: pass^k

Run the same case twice and the agent may take different paths, because a model's choices vary from run to run. An agent that resolves a case correctly 75% of the time is not a 75% agent from the customer's point of view: it is an agent that fails one customer in four, unpredictably. So each case is run several times, and the results are summarised in two ways that answer different questions.

pass@k asks: in k tries, did it succeed at least once? It suits tasks where a person or a check can pick the best of several attempts, such as generating code that is then run.

pass^k (read "pass hat k"), introduced by τ-bench, asks: in k tries, did it succeed every time? It measures consistency, which is what matters for an agent serving customers, because every customer gets exactly one try. If a single trial succeeds with probability p and trials are independent, then

\text{pass}^k = p^k

From real trials it is estimated without assuming independence. If a case was run n times and succeeded c times, the estimate is the number of ways to choose k trials from the c successes, divided by the number of ways to choose k trials from all n:

\text{pass}^k \approx \frac{\binom{c}{k}}{\binom{n}{k}}

where \binom{c}{k}, read "c choose k", counts the ways of picking k items from c. Take a case run 8 times with 6 successes. Then pass^1 is 6/8 = 0.75. For k = 4, there are \binom{6}{4} = 15 ways to pick four trials that are all successes, out of \binom{8}{4} = 70 ways to pick any four, so pass^4 is 15/70 \approx 0.21. For k = 8 it is zero, since not all eight succeeded. A 75% success rate on one try means only about a one-in-five chance of four customers in a row all being served correctly.

from math import comb

def pass_hat_k(outcomes: list[bool], k: int) -> float:
    n, c = len(outcomes), sum(outcomes)
    return comb(c, k) / comb(n, k)            # comb(c, k) is 0 when c < k

print(pass_hat_k([True]*6 + [False]*2, 1), round(pass_hat_k([True]*6 + [False]*2, 4), 3))
# 0.75 0.214

The τ-bench authors found exactly this gap in the strongest models of 2024: in the retail domain, a leading model succeeded on fewer than half the tasks, and its pass^8 was below 25%. For a business, pass^k is the number to improve, and it improves through the means this chapter has covered: clearer tools, shorter paths, checks in code, which raise p and make each run more like the last.

5. The improvement loop

Observation and evaluation come together in a loop that runs for the life of the agent.

1. Watch production metrics, split by version, and sample traces every week, more after any change.
2. Review failures by reading traces and sorting each failure into a cause: wrong tool, misread result, policy misapplied, asked for something the tools had, gave up too early, looped, guardrail fired. Sorting before fixing shows which cause is most common, which is where to start. This is the error analysis of 12.8.6, applied to paths.
3. Turn each important failure into a case in the evaluation set, with a simulated world reproducing it. Every case that once failed in production stays forever, so the same failure cannot return unnoticed.
4. Fix the cause: a tool description, an error message, a prompt instruction, a policy check, a step limit.
5. Evaluate the change on the whole set, several trials per case, comparing pass^k, cost and steps per category with the current version. A change that improves the category it targeted while breaking another is caught here.
6. Release gradually: send a small share of real traffic to the new version, compare its production metrics with the old one's, and widen it only if they hold. This is the canary release from software deployment, and 12.12.4 covers it for models.

October's problem would have been caught at step 5 by the must_call check, and the cost increase by the step budget, before any customer or finance report saw it. Cartwise added both to its release gate that month.

What the interviewer will push on

"How do you evaluate an agent?" They are checking that you go beyond grading the final answer. The strong answer: run the real agent in a simulated world with a simulated user; check the outcome in the final state, required and forbidden steps in the path, and the step and cost budget; use a model judge with narrow pass-or-fail questions for path quality; run each case several times.

"What is the difference between pass@k and pass^k?" pass@k is at least one success in k tries, suited to tasks where the best attempt can be picked; pass^k is success on all k tries, which measures the consistency a customer-facing agent needs. Being able to compute pass^4 from 6 successes in 8 trials (15/70, about 0.21) is a strong signal.

"What would you put in an agent trace?" Spans for every model call (input, output, tokens, cost, latency, stop reason, request ID), every tool call (arguments, result, errors, time), every harness decision, and the run's version and outcome. Mention that traces hold personal data and need access control and retention limits.

"Which production metrics matter for an agent?" Resolution without a person, repeat contacts, approval declines, steps and cost as distributions with p95 and maximum, required-tool coverage, repeated-call loops, tool error rates, guardrail triggers, all split by version.

"How do you know a prompt change did not break anything?" An evaluation set organised by category, run several times per case, compared on pass^k, cost and steps per category, then a canary release with production metrics compared side by side.

Something to volunteer: the agent that was right for the wrong reason. A final-answer evaluation stayed at 92% while the agent stopped consulting the policy and doubled its cost. Path checks and step budgets in the evaluation set are what catch that, and saying so shows you have looked at real traces.

Recall

• An agent's output is a path, not just a reply; final-answer grading misses skipped steps, wasted steps and right-for-the-wrong-reason answers.
• A trace is a tree of timed spans: the run, each model call, each tool call, each subagent, each harness decision. Record inputs, outputs, tokens, cost, latency, stop reasons, request IDs and the agent version; treat traces as personal data.
• Production metrics: outcomes (resolved, repeat contacts, approval declines), path (steps p50/p95, required tools missing, repeated calls), cost distributions, tool errors, guardrail triggers, split by version.
• Offline evaluation in a simulated world with a simulated customer (as in τ-bench, Sierra, 2024): check the final state, required and forbidden tools, and the step budget; add a model judge with narrow pass-or-fail questions for path quality.
• Run each case several times. pass@k is at least one success in k; pass^k is success every time, estimated as \binom{c}{k}/\binom{n}{k}; 6 of 8 gives pass^1 = 0.75 but pass^4 ≈ 0.21.
• The loop: watch, review and sort failures, turn them into permanent cases, fix, evaluate on the whole set, release gradually.

Self-test: Which line of run_case would have caught October's problem, and which would have caught the cost increase? A case succeeds in 7 of 8 trials; what are its pass^1 and pass^2? Why is the customer in an agent evaluation simulated by a model rather than scripted line by line?

Next: 12.10.10 Agent Economics — the October bill showed that an agent's cost is set by its path; the next page works out exactly where an agent's money goes, and the techniques (caching, cascades, trimming, budgets) that cut it without making the agent worse.