M.A.T.R.I.X

Appearance

12.6.3 — Agents, Tools, Protocols and Evaluation ​

An internal assistant is given a calendar tool and asked to schedule a review. It creates the meeting, does not recognise its own success in the response, tries again, and creates it four times. A second agent, given a search tool and a vague goal, runs 140 model calls in eleven minutes before anyone notices the bill.

Neither failure is about intelligence. They are the failures of a loop with no idempotency, no step budget and no success condition — ordinary engineering problems, in a component that will not raise an exception when it goes wrong.

1. What an agent is ​

An agent is a model in a loop with tools.

1. Send the goal, the history, and the available tools.
2. The model replies with either a tool call or a final answer.
3. If it is a tool call: execute it, append the result, go to 1.
4. If it is an answer: stop.

That is the whole architecture. Everything else — planning, reflection, multiple agents — is a variation on the loop or a way of constraining it.

The difference from a chain (Chapter 12.6.1) is who decides the path. A chain's steps are fixed by you. An agent chooses its next step at runtime. That flexibility is the feature and the entire source of the risk, because a system whose control flow is decided by a probabilistic component has no fixed upper bound on anything.

So the first design question is always: does this need to be an agent? If the steps are known — extract, look up, draft, send — write the pipeline. A deterministic workflow is cheaper, faster, debuggable and testable. Reserve agents for genuinely open-ended tasks where the number and order of steps depend on what is found.

2. Tools ​

A tool is a function the model may call, described by a schema:

ts
{
  name: 'search_orders',
  description: 'Find orders for a customer. Returns at most 20, newest first. ' +
               'Use when the user asks about their order history or a specific order.',  // (1)
  parameters: {
    type: 'object',
    properties: {
      customer_id: { type: 'string', description: 'Internal id, not the email' },  // (2)
      status: { type: 'string', enum: ['paid', 'shipped', 'refunded'] },           // (3)
    },
    required: ['customer_id'],
  },
}

(1) The description is a prompt. It is the only thing telling the model when this tool applies, and saying when not to use it is as valuable as saying when to. (2) Parameter descriptions prevent the model passing an email where an id belongs. (3) An enum makes an invalid value unrepresentable — the same constrained-decoding argument as Chapter 12.5.3.

Designing tools well is most of building a good agent.

Few tools, clearly distinguished. Beyond roughly 20, selection accuracy falls noticeably. If two tools overlap, the model will pick wrongly — merge them or make the descriptions disjoint.

Few parameters. Every optional parameter is another thing to get wrong.

Return errors as instructions. "Error: customer_id must start with 'cus_'. You passed an email address. Look it up with find_customer first." The model reads that and recovers. Error 400 produces a retry of the same mistake.

Return little. A tool that dumps 50 KB of JSON fills the context and buries the answer. Return the fields that matter, paginated, with a note that more exists.

Make tools idempotent, with a client-supplied key (Chapter 9.6.3). This is the direct fix for the opening story: the second create_meeting with the same key returns the first result instead of a second meeting.

Separate read from write, and gate the writes. Reads can be automatic; anything that spends money, sends a message or deletes data goes through a confirmation. This is also the main defence against indirect prompt injection (Chapter 12.6.1).

3. Agent patterns ​

ReAct — reason, act, observe — the basic loop, with the model narrating its reasoning before each call. Simple and effective for most tasks.

Plan and execute — produce a plan first, then work through it. Better on long tasks because the plan constrains drift, and worse when reality diverges from the plan, so it needs a replanning step.

Reflection — after producing a result, criticise it and revise. Measurable gains on writing and code at the cost of extra calls. Two rounds is usually the ceiling; beyond that it tends to change things for the sake of changing them.

Multi-agent — several specialised agents, often with a supervisor routing between them or handing off.

Be sceptical of multi-agent designs. The costs are real and additive: every handoff loses context, errors compound across agents, cost multiplies, and debugging a conversation between four agents is genuinely hard. The cases where it earns its keep are narrow — genuinely parallel independent subtasks, or a real need to isolate an untrusted-content reader from a privileged actor (Chapter 12.6.1's injection defence). "Give each role its own agent" is an appealing metaphor and usually a worse system than one agent with good tools.

4. The controls an agent needs ​

A step budget. Hard stop at N iterations. Without it, a loop is unbounded.

A cost budget. Track tokens per run and abort past a threshold. Alert on the distribution, not the average — the 140-call run hides in a mean.

Timeouts on every tool and on the whole run.

Idempotency keys on every write.

Checkpointing. Persist state after each step so a failure resumes rather than restarts. This is what LangGraph-style frameworks genuinely provide (Chapter 12.6.1), and it is awkward to build well.

Human approval gates on consequential actions — and make the pause resumable, which is why checkpointing matters.

A sandbox for generated code: no network, no filesystem outside a temporary directory, a memory and CPU limit, and a short timeout (Chapter 2.9's isolation mechanisms).

Full tracing. Every prompt, tool call, result and token count. You cannot debug an agent from logs that only record the final answer.

And the arithmetic that should shape expectations: a ten-step task with 95% per-step reliability succeeds 60% of the time. Reliability per step is the number that matters, which argues for fewer steps, better tools, and verification at the end rather than trust throughout.

5. The protocol layer: MCP, A2A, ACP ​

Three protocols appear together in conversation and solve different problems at different layers. They compose; they do not compete.

MCP (Model Context Protocol) connects a model to tools and data. An MCP server exposes capabilities; an MCP client — inside an assistant, an editor or your application — consumes them. It defines three things:

  • Tools — functions the model may call.
  • Resources — data the client can read, addressed by URI.
  • Prompts — reusable templates the server offers.

The value is the N×M problem. Without a standard, every assistant needs a bespoke integration with every tool. With one, any client works with any server. Servers run locally over standard input/output or remotely over HTTP, and there is a large ecosystem of them — file systems, databases, issue trackers, and notably a browser developer-tools server that lets a model inspect a live page's console, network requests and DOM, which turns front-end debugging into something an assistant can actually do.

MCP is a security surface, and it is worth being blunt. A server you install can expose tools with any description, and a malicious description is a prompt injection delivered as configuration — sometimes called tool poisoning. A server with broad filesystem or database access is a confused deputy waiting to happen (Chapter 8.5.2). Install servers you trust, scope their permissions narrowly, and review what a server can actually reach, exactly as you would a dependency (Chapter 8.6.2).

A2A (Agent-to-Agent) connects agents to each other. An agent publishes an "agent card" describing what it can do; another agent discovers it and delegates a task, with a defined lifecycle for long-running work and streamed updates. It solves peer delegation across organisational boundaries, where the agents are separate systems and possibly separate companies.

ACP (Agent Communication Protocol) covers the messaging layer between agents — how messages are structured and routed — with a similar goal of interoperability.

The layering, stated once: MCP is downward, from an agent to its tools and context. A2A is sideways, between peer agents. ACP is the message plumbing. A single system can use all three: an orchestrator delegates to a specialist agent over A2A, and that agent reaches its database over MCP.

The honest 2026 position: MCP has clear adoption and a real ecosystem. A2A and ACP are earlier, and multi-agent interoperability is a less-proven need than tool interoperability. Adopt MCP where it saves integration work; treat cross-agent protocols as promising rather than settled.

6. Workflow automation as the alternative ​

n8n, Zapier and Make build workflows from triggers, conditions and actions, and they now include model nodes.

They are frequently the right answer, and the judgement is worth stating clearly:

Use a workflow when the steps are known. When a form is submitted, extract the fields with a model, look up the customer, create a ticket, notify the channel. That is four deterministic steps with one model call inside. It is testable, debuggable, cheap, and it fails in a way you can read.

Use an agent when the steps are not known in advance. Investigate why this customer's order failed — which requires deciding what to look at based on what was found.

Most tasks people build agents for are the first kind. The model is being asked to decide a sequence that the developer already knows, which pays cost and unpredictability for nothing. Ask what the model is deciding that you could not decide yourself, and if the answer is nothing, it is a workflow.

7. Evaluation ​

A non-deterministic component with no evaluation is not a system, it is a demo. You cannot know whether a prompt change helped, whether a model upgrade broke something, or whether last week's regression exists.

Four levels, and you want all of them:

Assertion tests. For anything checkable: is it valid JSON, does it contain a required field, is the classification correct, does the SQL run. These are ordinary tests and should run in CI. A surprising share of quality is checkable this way.

Model-as-judge. For open-ended output, a model scores against a rubric. It is the only scalable option, and its biases are documented and must be managed:

  • Position bias — in a pairwise comparison, judges favour one position. Run both orders and average.
  • Length bias — longer answers score higher. Control for length in the rubric.
  • Self-preference — a model tends to prefer text from its own family. Use a different model as judge where possible.

Design the rubric properly. A Likert scale — 1 to 5 — needs each point defined by what it looks like, not left to interpretation, or scores cluster on 3 and 4 and carry no information. Ask for a brief justification before the score; it improves consistency and makes disagreements readable. And validate the judge against human labels on a sample before trusting it — an unvalidated judge is a measurement instrument nobody calibrated.

Pairwise comparison is more reliable than absolute scoring. "Which of these two is better" is a judgement both models and humans make far more consistently than "rate this out of five", and it is the right shape for comparing two prompt versions.

Human review on a sample, always. It is what catches the failure modes your rubric does not describe.

Online signals. Thumbs up and down, task completion, whether the user rephrased and asked again, whether they escalated to a human. Rephrasing is the best cheap negative signal available — it means the first answer failed without the user bothering to say so.

The practice that matters most: a fixed regression set in CI. Fifty to two hundred real cases with expected outcomes, run on every prompt change, model change and retrieval change, with results recorded. Without it, every change is a guess, and a model provider's silent update is undetectable.

8. Capstone: a local agentic application ​

A complete agent that costs nothing to run, using an open-weight model such as Gemma on your own machine. The point is that every mechanism in this Part is visible when nothing is hidden behind an API.

Serve the model locally with Ollama, llama.cpp or vLLM, quantised to 4-bit (Chapter 12.5.3) so it fits in ordinary memory, exposed over an OpenAI-compatible HTTP endpoint.

Give it three tools, chosen so each demonstrates something different:

  • search_notes(query) — retrieval over your own Markdown files, with a local embedding model and a vector store, built exactly as in Chapter 12.6.2.
  • run_python(code) — a sandboxed execution tool, which is how the model does arithmetic reliably rather than guessing (Chapter 12.5.2).
  • write_file(path, content) — a write tool, gated behind a confirmation prompt, so the approval mechanism is real rather than theoretical.

Build the loop yourself — send the goal and tool schemas, parse the tool call, execute, append the result, repeat — with a step budget, a per-run timeout and full tracing of every prompt and response.

Then evaluate it. Twenty questions with known answers, an assertion test for each, run before and after every prompt change.

Three things become concrete once this runs. Small models follow tool schemas less reliably than large ones, so tool descriptions and error messages matter more, not less. Local inference is memory-bound and generation speed is entirely predictable from Chapter 12.5.3's arithmetic. And most of the code is not about the model at all — it is the loop, the budget, the sandbox, the retrieval and the tracing, which is exactly the point of this Part.

Recall

  • An agent is a model in a loop with tools. The difference from a chain is that the model chooses the next step at runtime — which is the feature and the whole risk. If the steps are known, write a workflow.
  • A tool's description is a prompt. Few tools, few parameters, enums over free strings, errors returned as recovery instructions, small return payloads, and idempotency keys on every write.
  • Patterns: ReAct, plan-and-execute (needs replanning), reflection (two rounds is the ceiling). Be sceptical of multi-agent — context is lost at every handoff and errors compound; it earns its place for parallel subtasks or isolating untrusted content from privileged tools.
  • Required controls: step budget, cost budget, timeouts, idempotency, checkpointing for resumable human approval, a sandbox for generated code, and full tracing. Ten steps at 95% reliability is 60% overall.
  • MCP is downward (agent → tools and data), A2A is sideways (agent → agent), ACP is the message layer. They compose. MCP solves the N×M integration problem and is a security surface — a malicious tool description is prompt injection delivered as configuration.
  • Workflow versus agent: ask what the model is deciding that you could not decide yourself. If the answer is nothing, it is a workflow.
  • Evaluation in four layers: assertion tests in CI, model-as-judge with rubrics (correct for position, length and self-preference bias; validate the judge against humans), pairwise comparison over absolute scores, and human review of a sample.
  • A fixed regression set of 50–200 real cases, run on every change, is what makes this engineering. Rephrasing is the best cheap negative signal from real users.

Self-test: What single tool property would have prevented the four duplicate meetings? · Why does a ten-step agent fail 40% of the time at 95% per step? · When is multi-agent genuinely justified? · State the direction each of MCP and A2A points · Name three biases in model-as-judge and the fix for each · What question decides between an agent and a workflow?