12.8.3 — Structured Output

The email router from 12.8.2 does not talk to a customer. It talks to code: Cartwise's ticketing system reads its answer and files the email in the right team's queue. So the team asked the model to "reply with JSON containing team, urgency and a one-line summary", and wrote json.loads(reply).

Over 40,000 emails a week, the parser failed in every way you could imagine:

• Sure! Here is the JSON you asked for: {"team": ...}: a friendly sentence in front of the JSON.
• The JSON wrapped in a Markdown code fence, with backticks the parser rejected.
• "urgency": "high" where the code expected a number from 1 to 5.
• "team": "shipping", a team that does not exist.
• A missing summary field, and occasionally a trailing comma that is not valid JSON.

About one email in two hundred broke something. At 40,000 emails a week, that is roughly 200 emails a week stuck or misfiled. When code consumes a model's output, "usually right" is a bug. This page covers the ways to make a model return data in exactly the shape your program expects, from the weakest to the strongest, and the checks you still need afterwards.

1. Three levels of guarantee

Level 1: ask, then parse and retry. Describe the JSON in the prompt, ideally with an example, parse the reply, and if parsing fails, send the error back and ask again. This works most of the time with modern models, and it is what the team started with. The trouble is "most of the time": every failure costs a retry, and some get through as valid JSON with the wrong contents.

Level 2: JSON mode. Some APIs offer a mode that guarantees the output is syntactically valid JSON. That removes the preambles, code fences and trailing commas. It does not guarantee your fields: {"team": "shipping", "urgency": "high"} is perfectly valid JSON.

Level 3: schema-constrained output. You give the API a JSON Schema, a standard machine-readable description of the exact structure you require: which fields, which types, which allowed values. The model's output is then guaranteed to match it. This is usually called structured output, and for anything code consumes, it is the right default.

2. How a guarantee is even possible: constrained decoding

A model generates one token at a time by choosing from a probability distribution, as 12.7.7 explained. Nothing in the model itself enforces a schema. The guarantee comes from changing the choosing step: constrained decoding.

The schema is compiled ahead of time into a kind of state machine, like the automata of Chapter 1.7, that knows, at every point in the output, which characters could legally come next. At each decode step, before a token is chosen, every token that would break the schema has its probability set to zero. The model then chooses among what remains.

Constrained decoding at a single step. The output so far has opened the "team" field. Only tokens that can start one of the four allowed team names survive the mask; "ship" and "Sure" are impossible, so an invalid team or a chatty preamble simply cannot be produced.

Walk through the figure. The output so far is {"team": ". The schema says the next characters must spell one of the four team names. The model, left alone, would most like to write "ship" (it is thinking of "shipping"), with a probability of 0.41. The mask removes it, because no allowed value begins with "ship". It also removes "Sure", the start of a chatty preamble. "deliv" and "refunds" survive and are rescaled to add to 1: 0.33 / (0.33 + 0.18) \approx 0.65 and 0.35. An output that breaks the schema is not unlikely; it is impossible. The same mechanism forbids a missing required field (the closing brace is masked until every required field has appeared), a string where a number belongs, and anything after the final brace.

The idea was developed in open-source libraries such as Outlines and llama.cpp's grammar support, and the major providers now offer it built in. Its one subtle cost: by forcing the format token by token, it can push the model away from the answer it would have given, as in the figure, where the model "wanted" a team that does not exist. That is usually what you want, but it is worth remembering that a constrained answer is the model's best allowed answer, not necessarily its best answer.

3. Structured output in code

The cleanest way to define a schema in Python is a Pydantic model: a class whose fields and type hints describe the data, which the library can turn into a JSON Schema and use to validate incoming data. (Pydantic is the standard Python library for this; FastAPI and many other tools are built on it.)

from typing import Literal
from pydantic import BaseModel, Field
import anthropic

class Triage(BaseModel):                                              # (1)
    team: Literal["refunds", "delivery", "account", "payment", "none"]  # (2)
    urgency: int = Field(ge=1, le=5, description="1 = routine, 5 = legal or safety risk")  # (3)
    order_id: str | None = Field(description="Order number if mentioned, else null")  # (4)
    summary: str = Field(description="One sentence, under 20 words")

client = anthropic.Anthropic()
response = client.messages.parse(                                     # (5)
    model="claude-opus-5-5",
    max_tokens=1024,
    system="You triage customer emails for Cartwise's support teams. "
           "A parcel that never arrived goes to delivery, even if money is mentioned.",
    messages=[{"role": "user", "content": email_text}],
    output_format=Triage,                                             # (6)
)
if response.stop_reason == "end_turn":                                # (7)
    triage = response.parsed_output                                   # (8) a validated Triage object
    queue.put(triage.team, triage)

Line (1) declares the shape of the answer as a class. Line (2) uses Literal to say the team must be exactly one of five strings; that becomes an enum (a fixed list of allowed values) in the schema, which is what made "shipping" impossible. Including "none" gives the model an honest answer for emails like "thanks, all sorted", for the same reason as the "I don't know" path in 12.8.2. Line (3) requires an integer from 1 to 5 and explains what the numbers mean; the description is sent to the model, so it doubles as an instruction. Line (4) allows null for emails with no order number, so the model is never forced to invent one.

Line (5) calls the SDK's parse method, and line (6) hands it the Pydantic class, which the SDK converts to a JSON Schema for constrained decoding and then uses to validate and parse the reply. Line (7) checks the stop reason first: if the answer was cut off by max_tokens or declined, it is not complete JSON, and no schema can change that. Line (8) gives you a real Triage object, with triage.urgency already an integer, no json.loads needed.

The same guarantee is available without Pydantic, by passing a raw JSON Schema in the request's output_config as {"type": "json_schema", "schema": {...}}, and for tool calls by marking a tool definition strict, so that the arguments the model sends to your function always match its schema (12.8.4).

4. Designing a good schema

The schema is a prompt as well as a contract: the model reads the field names and descriptions. A few habits make outputs better as well as valid.

Use enums for anything with a fixed set of values. Teams, categories, yes-or-no flags, priority levels. Free-text strings for categories invite near-duplicates like "Refund", "refunds" and "refund request".

Always provide a way out. A "none" or "unclear" option in enums, and null for optional facts. A schema that requires an order number on every email forces the model to invent one when there is none, and the invented number will look perfectly real.

Describe every field. "summary: one sentence, under 20 words" produces better summaries than "summary". Descriptions are the cheapest place to put instructions that apply to one field.

Put reasoning before the decision when the decision is hard. The model writes the fields in order, and each token can only use what came before it, as 12.8.2 explained for chain of thought. If a field called reasoning comes after team, the team has already been chosen when the reasoning is written, and the reasoning becomes a justification rather than a help. If the decision needs thought and the model does not reason internally, put a short reasoning field first.

Keep it flat and small. Deeply nested structures with dozens of optional fields are harder for the model to fill well and harder for you to validate. If the schema is getting enormous, the task probably wants to be two calls.

Reject extra fields. Setting additionalProperties to false in a raw schema (Pydantic's default output from parse handles this) means the output contains exactly your fields and nothing invented alongside them.

5. Valid is not the same as correct

Constrained decoding guarantees the shape. It guarantees nothing about the content. {"team": "refunds", "urgency": 1, "order_id": "8840", "summary": "Customer wants a refund"} can be perfectly valid and completely wrong: the email was a safety complaint about an overheating charger, which should be urgency 5, and the order number was 8804. So production systems add a second layer of checks that know about the business:

from pydantic import field_validator

class Triage(BaseModel):
    team: Literal["refunds", "delivery", "account", "payment", "none"]
    urgency: int = Field(ge=1, le=5)
    order_id: str | None
    summary: str

    @field_validator("order_id")                                  # (1)
    @classmethod
    def order_must_exist(cls, v):
        if v is not None and not orders.exists(v):
            raise ValueError(f"order {v} does not exist")
        return v

def check_business_rules(t: Triage, email: str) -> list[str]:     # (2)
    problems = []
    if any(w in email.lower() for w in ("fire", "smoke", "burn", "lawyer", "court")) \
            and t.urgency < 5:
        problems.append("possible safety or legal email marked below urgency 5")
    return problems

Line (1) adds a validator that checks the extracted order number against the real order database: a well-formed number that does not exist is caught here. Line (2) is a plain-code rule that catches a dangerous combination, such as a mention of fire or a lawyer with low urgency. When a check fails, the usual move is to retry once with the problem stated ("order 8840 does not exist; re-read the email"), and if it fails again, to route the email to a person. That two-layer pattern, the schema for shape and code for meaning, applies to every structured output in production.

6. Where structured output is used

Once outputs are reliable data, the model becomes a component in ordinary software:

• Classification and routing, like the triage above.
• Extraction: pulling order numbers, amounts, dates and reasons out of emails, chats, PDFs and scanned invoices; Chapter 12.15.4 builds a full document pipeline on this.
• Tool calls: when a model calls a function, its arguments are structured output, the subject of the next page.
• Driving an interface: a model that returns {"component": "refund_form", "prefill": {...}} lets an app show a ready-filled form instead of a paragraph of text, which 12.8.5 shows.
• Evaluations: an LLM judge that returns {"score": 4, "reason": "..."} can be aggregated automatically (12.8.6).

Recall

• Code that consumes model output needs exact data, not "usually right" text. Three levels: ask and parse with retries; JSON mode (valid syntax only); structured output against a JSON Schema (exact shape, guaranteed).
• Constrained decoding compiles the schema into a state machine and, at each step, zeroes the probability of every token that would break it, so invalid output is impossible, not just unlikely. The result is the best allowed answer.
• In Python, define the shape as a Pydantic model and call client.messages.parse(..., output_format=Model); check stop_reason first, since a truncated or declined reply is not complete JSON.
• Good schemas use enums, always offer a way out ("none", null), describe every field, put reasoning before a hard decision, stay flat and small, and reject extra fields.
• Valid is not correct: add validators and business-rule checks in code, retry once with the error, then escalate to a person.

Self-test: Why does a required order_id field with no null option make invented order numbers more likely? In the masking figure, what would happen if "delivery" were not in the enum? Why put a reasoning field before team rather than after it?

Next: 12.8.4 Tool Calling — structured output lets a model hand data to your code; tool calling lets it ask your code to do something, look up an order, check a refund, and continue with the result.