Appearance
12.10.1 — What an Agent Is
Two Cartwise customers send almost the same message on the same morning.
Customer A: "The earphones from order 4471 arrived with the left side not working. I'd like my money back."
Customer B: "The earphones from order 3918 stopped working on the left side. I'd like my money back."
A support person handles them very differently. For order 4471, delivered four days ago, the earphones are inside the 10-day window for damaged electronics, so the person checks the policy, asks for a photo because the order is over ₹2,000, and issues a refund. For order 3918, delivered seven weeks ago, the return window has long closed, so the person looks up the warranty instead, sees it is still covered for a manufacturer repair, and offers that. Same request, different paths, and which path only becomes clear after looking things up.
Cartwise first tried to automate this with a script: a fixed sequence of "look up the order, check the window, refund or reject". It worked for the common case and failed on everything else: a partly refunded order, a gift-card payment, a Cartwise Plus item with its own terms, a customer who wanted a replacement rather than money. Every new case meant another branch in the script, and the script grew into a tangle nobody wanted to touch.
The alternative is to let a language model decide the steps itself: give it the goal, the policies, and a set of tools (look up an order, search the policies, ask the customer a question, propose a refund), and let it work out, step by step, what to do for this customer. A system built like that is an agent, and this chapter is about building them well enough to trust with real customers and real money.
1. The definition
An agent is a language model running in a loop, choosing its own next action, using tools, until the goal is met or a limit is reached.
Each part of that sentence matters. A loop: the model is called again and again, not once. Choosing its own next action: after each step, the model, not your code, decides what happens next. Using tools: the actions are the tool calls of 12.8.4, so the model can look things up and change things. Until the goal is met or a limit is reached: it stops itself when it judges the job done, and your code stops it if it runs too long, costs too much, or tries something not allowed.
You have already seen the smallest agent in this book. The tool loop of 12.8.4 calls the model, runs whatever tools it asks for, sends back the results, and repeats until the model gives a final answer. That loop with two read-only tools answers questions. The same loop with more tools, tools that can change things, a longer task and the controls around it, is what people mean by an agent.
The most useful distinction in the field comes from Anthropic's 2024 engineering guide Building Effective Agents, and it is worth learning word for word:
- A workflow is a system where models and tools are connected through paths that your code defines. The model may write text or make a decision at a step, but the overall sequence is fixed in advance.
- An agent is a system where the model directs its own process, deciding which tools to use and in what order.
The difference is who decides the next step: your code, or the model. Everything else about agents, the power and the risk, follows from that.
2. The spectrum from one call to many agents
In practice there is not a sharp line but a range of designs, each giving the model a little more control than the last. The figure arranges them in order.
Reading left to right:
- A single call classifies a ticket or writes a reply. You know exactly what happens.
- A chain runs fixed steps in order: extract the order number, look it up, draft a reply. The model does the work at each step, but your code decides the order. This is the prompt chaining of 12.8.2.
- A router lets a model choose which of several fixed paths to take: "is this a return, a delivery question or a payment problem?", then hands the request to the path built for it.
- An agent loop lets the model choose every step, as many as it needs.
- Several agents split a task across more than one agent loop, covered in 12.10.7.
The arrow at the bottom is the whole engineering trade-off. Every step to the right handles more unexpected cases, and every step to the right is harder to predict, harder to debug and more expensive to run. Good AI engineers move right only as far as the problem forces them to. 12.10.6 is about the workflow designs that solve most problems without an agent at all.
3. The loop, step by step
An agent repeats one cycle: look at the situation, decide on an action, take it, look at what happened. The idea that a model should alternate between reasoning and acting, rather than only doing one, was named ReAct (short for Reason + Act) by Shunyu Yao and colleagues at Princeton and Google in a 2022 paper. Before it, there were two separate lines of work. Chain-of-thought prompting, also from 2022, showed that models answer better when they first write out their reasoning, but the reasoning could only use what the model already knew, so it invented facts confidently. Other work had models take actions, such as searching, without reasoning about them, so they wandered. ReAct interleaved the two: write a thought, take one action, read the result, write the next thought. The reasoning keeps the actions on track, and the actions keep the reasoning honest by supplying real facts.
In the original paper this was done with plain text: the model wrote lines starting Thought:, Action: and Observation:, and the program parsed them. Today the same cycle is built into the APIs. The model's reasoning goes into a thinking block (the reasoning step of 12.7.9), the action is a structured tool_use block, and the observation is the tool_result your code sends back. The parsing problem disappeared, and the pattern stayed.
Here is customer A's refund, traced through the loop. Each row is one trip around it.
Three things in that trace are worth noticing. First, the model chose each action from what the previous result told it: it searched the damaged electronics policy because step 1 said electronics, and asked for a photo because step 2 said so. That is the "choosing its own next action" of the definition. Second, the dashed line marks where customer B's case leaves this path: in step 2 the policy says the window has closed, so the model searches the warranty terms instead. Nobody wrote a branch for that; the model found the path. Third, the amber step was not the model's decision. The code running the loop stopped the refund and asked a person, because a rule said so. An agent is never just a model; it is a model inside code that decides what the model is allowed to do.
4. What an agent is made of
Every agent, from the refund agent to a coding assistant, has the same five parts.
The model makes the decisions: reads the situation, chooses the next action, writes the final answer. A more capable model plans better and recovers from mistakes better, and the choice of model is usually the biggest single factor in how well an agent works.
The instructions, the system prompt, say what the goal is, what rules apply and how to behave: "You handle refund requests for Cartwise. Always look up the order before deciding anything. Never promise a refund before it is approved."
The tools are what the agent can see and do. Their design, covered in 12.10.3, decides most of how well an agent behaves, because the model knows the world only through them.
The state is everything the agent knows so far: the conversation, the results of every tool call, anything it has noted. Within one task it lives in the message history; across tasks it needs storage, the subject of 12.10.5.
The harness is the code that runs the loop around the model. It calls the model, runs the tools, enforces the limits (steps, time, money), pauses for approvals, records everything, and handles failures. The word comes from engineering, where a harness is the frame that holds a machine and connects it to everything else. It is the part you write, and it is where almost all the engineering of this chapter happens: 12.10.2 builds one, and 12.10.8 adds its safety controls.
A helpful way to hold these together: the model is a very capable new employee who has read every policy but has no access to anything. The tools are the systems you give them a login to. The instructions are their job description. The state is their notes on the current case. The harness is the company's controls: the spending limit on their card, the manager who signs off big refunds, the audit log. You would not give a new employee unlimited refund power on day one, and the same reasoning applies to an agent.
5. Why agents fail: small errors multiply
Agents are powerful, and they fail in a specific, predictable way that every design decision in this chapter is meant to control. Every step is a chance to go wrong: the wrong tool, a misread result, a wrong argument. And a task succeeds only if every step goes right.
Suppose each step succeeds with probability p, and a task takes n steps that must all succeed. If the steps' failures are independent, the chance of the whole task succeeding is p multiplied by itself n times:
P(\text{task succeeds}) = p^n
Read it as "p to the power n". The numbers are sobering:
| Per-step success | 5 steps | 10 steps | 20 steps |
|---|---|---|---|
| 99% | 95.1% | 90.4% | 81.8% |
| 95% | 77.4% | 59.9% | 35.8% |
| 90% | 59.0% | 34.9% | 12.2% |
Take the middle row. A step that works 95 times in 100 sounds excellent. But ten such steps in a row succeed together only 0.95^{10} \approx 0.599, about 60% of the time, and twenty only about 36%. A ten-step agent built from "95% reliable" steps fails on four tasks in ten.
The real situation is somewhat better and somewhat worse than this formula. Better, because a good agent notices many of its own mistakes ("that order number doesn't exist, let me ask the customer") and recovers, so failures are not final. Worse, because some failures are silent: the agent finishes confidently with the wrong answer, and nobody notices.
The practical lessons follow directly, and each has a page in this chapter:
- Keep tasks short. Fewer steps means fewer chances to fail. A workflow that does the predictable steps in code and leaves only the genuinely variable step to the model beats a long agent loop (12.10.6).
- Make each step more reliable. Clear tools with good error messages raise p (12.10.3). Going from 95% to 99% per step takes a ten-step task from 60% to 90%.
- Check before things become irreversible. Approval for consequential actions, and verification of the result (12.10.8).
- Measure whole tasks, not single steps (12.10.9).
Agents also fail in two other ways the formula does not show. Their cost and time are unpredictable, because the model decides how many steps to take; a confused agent can loop for fifty steps and a large bill (12.10.10). And they can be manipulated, because everything they read, a customer message, a web page, a document, can contain instructions written to mislead them, the prompt injection of Chapter 12.14.1.
6. When to build an agent
Given all that, when is an agent the right choice? Ask four questions.
Can you write down the steps in advance? If yes, write them down as a workflow. An agent earns its cost only when the right steps genuinely depend on what is found along the way, as with the refund cases. "Look up the order and tell the customer its status" has fixed steps and needs no agent.
Is the task worth it? An agent costs more and takes longer than a single call. Resolving a refund, which would otherwise take a support person ten minutes, easily justifies a few cents and twenty seconds. Answering "what time do you close?" does not.
Can mistakes be caught? An agent that drafts a reply a person reviews is low risk. One that sends emails to customers without review needs much more care. The more an agent's output is checked, by a person, by tests, by validation in code, the more freedom you can give it.
Can its actions be undone, or held for approval? Reading data is safe. Writing it, spending money, or sending messages is not. The usual answer is to let the agent read freely and propose consequential actions that a person or a rule approves.
These questions also suggest how much freedom to give. A common progression, used by Cartwise, is to start with the agent suggesting: it drafts the resolution and a person carries it out. When its suggestions prove reliable, it acts with approval: it does the work and pauses before anything consequential. Finally, for low-risk, well-measured actions only, it acts and reports: small refunds go through automatically and are reviewed in batches afterwards. Each stage is earned by measurements from the stage before.
Applied to Cartwise:
| Task | Design | Why |
|---|---|---|
| Order status questions | workflow | steps fixed |
| Classify incoming tickets | single call | one decision |
| "Ask Cartwise" policy answers | RAG workflow | retrieve then answer |
| Refund and return requests | agent with approval | path depends on findings |
| Nightly fraud report | workflow | steps fixed, runs unattended |
7. The agents you will meet
Most agents in use today fall into a few families. They share the five parts above and differ in their tools and their main difficulty.
Coding agents work inside a software project. Their tools read and search files, edit them, and run commands such as the build or the program itself. Their advantage is that their work can be checked automatically: if the code does not compile, the agent sees the error and fixes it. Their main difficulty is finding the right places to change in a large codebase.
Research agents answer questions that need many searches: they search the web or internal documents, read results, and search again, the agentic retrieval of 12.9.6. Their main difficulty is volume: web pages are long and mostly irrelevant, so good research agents summarise what they read before keeping it.
Computer-use agents operate a graphical interface the way a person does: they look at screenshots, then click, type and scroll. They can use applications that have no API, at the cost of being slow and fragile, because a moved button breaks them.
Business-process agents, like the refund agent, work through internal tools: order systems, policies, payments, messages. Their main difficulty is safety, because their actions move money and reach customers.
The rest of the chapter builds the refund agent and the controls that make it safe: 12.10.2 writes it from scratch, then the following pages add tools, connections, memory, structure, safety, measurement and cost control, and 12.10.11 looks at how production agents are actually built.
What the interviewer will push on
"What is the difference between an agent and a workflow?" They are checking whether you know the precise distinction rather than a vague "agents are smarter". The answer is who decides the next step: in a workflow, your code fixes the paths and the model works within them; in an agent, the model chooses its own steps. A strong answer adds that most production systems are workflows, and an agent is chosen only when the steps cannot be known in advance.
"Why do agents fail on long tasks?" They want the compounding argument: per-step success multiplies, so 95% per step is about 60% over ten steps. The tell of real understanding is naming the countermeasures: shorter tasks, better tools to raise per-step reliability, verification and approval at the irreversible points, and evaluation on whole tasks.
"What is ReAct?" Interleaving reasoning and acting: think, act, observe, repeat, from Yao and colleagues in 2022. The good answer adds that modern APIs build this in, with thinking blocks and structured tool calls replacing the old text-parsing format.
"Would you use an agent for X?" Walk through the four questions out loud: are the steps known, is it worth the cost, can mistakes be caught, can actions be undone or approved. The common wrong answer is to reach for an agent because it sounds modern, when a two-step workflow would do.
"How would you give an agent permission to issue refunds?" Graduated autonomy: suggest, then act with approval, then act and report for low-risk cases only, each stage earned by measured reliability; plus limits enforced in code, not in the prompt.
Something to volunteer: the harness matters more than the prompt. In real agent projects, most of the effort and most of the reliability gains come from the code around the model, the tools, limits, approvals, error messages and logging, rather than from clever instructions. Saying so signals you have built one.
Recall
- An agent is a language model in a loop, choosing its own next action with tools, until the goal is met or a limit is reached. A workflow connects models and tools through paths your code defines. The difference is who decides the next step.
- The designs form a spectrum: single call, chain, router (workflows), agent loop, several agents. Moving right buys flexibility and costs predictability, money and debuggability; move only as far as the problem forces.
- ReAct (Yao et al., 2022) interleaves reasoning and acting: think, act, observe, repeat. Modern APIs build it in with thinking blocks and structured tool calls.
- Five parts: model, instructions, tools, state, and the harness, the code that runs the loop and enforces limits, approvals and logging.
- Small errors multiply: p^n, so 95% per step is about 60% over ten steps. Keep tasks short, make steps reliable, check before irreversible actions, measure whole tasks. Agents also have unpredictable cost and can be manipulated.
- Build an agent only if the steps are not known in advance, the task is worth it, mistakes can be caught, and actions can be undone or approved. Grow autonomy in stages: suggest, act with approval, act and report.
Self-test: A task has 8 steps at 97% each; what is its success rate, and what is it if you make it 4 steps? Why is the approval in step 5 of the trace done by the harness and not by an instruction in the prompt? Which of these needs an agent: summarising a support ticket, or resolving a delivery complaint that may involve the courier, a refund or a replacement?
Next: 12.10.2 An Agent from Scratch — the refund agent in about a hundred lines of Python: the loop, the tools, the limits, the approval pause and the trace, with every line explained.