12.8.1 — Calling a Model

The first version of "Ask Cartwise", the support assistant, was twelve lines of code: take the customer's message, send it to a hosted LLM, show the reply. It worked perfectly in a demo. Its first week in production produced five incidents, and every one of them came from how the model was called, not from the model itself:

1. On Monday evening, requests started failing with HTTP 429 errors, and customers saw a blank chat window.
2. On Tuesday, some answers stopped mid-sentence: "To request a refund, open the order and tap".
3. On Wednesday, a request took ninety seconds and the customer's app gave up.
4. On Thursday, the code crashed reading an answer, because the response did not contain the text it expected.
5. On Friday, finance asked why the bill was three times the estimate.

This page is about the layer every AI feature sits on: what an API request to an LLM contains, what comes back, and how to call it reliably, cheaply and safely from production code. Chapter 12.7 explained what happens inside the model; from here on we treat the model as a service, and the five incidents above are the map of this page.

The examples use the official Python SDK for Anthropic's Claude models, because a real API makes the details concrete. The other major providers' APIs have the same overall shape (a list of role-tagged messages in, generated content and token counts out), so everything here transfers, and Chapter 12.12.1 covers putting several providers behind one interface.

1. What a request contains

Here is the smallest useful call:

import anthropic

client = anthropic.Anthropic()                       # (1) reads the API key from the environment

response = client.messages.create(
    model="claude-opus-5-5",                         # (2) which model
    max_tokens=1024,                                 # (3) hard ceiling on output length
    system="You are Cartwise's support assistant. "  # (4) instructions from you, the builder
           "Answer briefly and politely.",
    messages=[                                       # (5) the conversation so far
        {"role": "user", "content": "Can I change my delivery address?"},
    ],
)

Line (1) creates a client. It finds the API key in the ANTHROPIC_API_KEY environment variable, which is where the key belongs: never in the source code, and never in a mobile app or web page, where anyone can extract it. Line (2) names the model, an exact identifier from the provider's list.

Line (3), max_tokens, is the most misunderstood setting. It is not a target length; the model does not try to use it. It is a hard cut-off: if the answer reaches that many tokens, generation simply stops, wherever it is. That was Tuesday's incident: an answer that needed 300 tokens hit a ceiling of 256 and was cut mid-sentence. Set it generously above what a good answer needs, and control length with instructions instead ("answer in at most three sentences").

Line (4) is the system prompt from 12.7.9: your instructions, which the model was trained to follow throughout the conversation. Line (5) is the conversation, a list of messages, each with a role of "user" or "assistant".

Two facts about messages are essential. The API remembers nothing between calls, as 12.7.1 explained, so to continue a conversation you send the whole history again, with the model's earlier replies as "assistant" messages. And everything you send is billed as input tokens on every call, so the history must be managed, a problem Chapter 12.10.5 solves.

Other settings appear on many requests: stop sequences that end generation early; tools the model may call (12.8.4); a required output format (12.8.3); and controls over how the model generates. Those controls differ by provider. Many APIs expose the temperature and top-p of 12.7.7. Current Claude models instead think before answering, and offer an effort setting, from "low" to "max", that trades how much thinking (and so cost and latency) the model spends against how carefully it works through hard problems. For a support assistant answering routine questions, low or medium effort is usually right; for a tangled refund dispute, higher effort earns its cost.

2. What comes back

The response is an object, not a string, and three of its fields matter on every call.

for block in response.content:                       # (1) a list of content blocks
    if block.type == "text":
        print(block.text)

print(response.stop_reason)                          # (2) why generation ended
print(response.usage.input_tokens,                   # (3) what you are billed for
      response.usage.output_tokens)

Line (1): content is a list of blocks, each with a type. The usual one is "text", but a response can also contain "thinking" blocks (the model's reasoning, which you can ask to see summarised), "tool_use" blocks (requests to call one of your functions), and others. Thursday's crash came from code that did response.content[0].text, assuming the first block was always text. When the first block was a thinking block, there was no .text to read. Always check each block's type before reading it.

Line (2): stop_reason says why the model stopped, and production code must handle every value:

stop_reason	Meaning	What your code should do
end_turn	finished naturally	use the answer
max_tokens	hit your ceiling	answer is cut off: raise the limit or retry
stop_sequence	produced one of your stop strings	use the answer up to that point
tool_use	wants to call one of your tools	run it and continue (12.8.4)
refusal	safety system declined	show a safe fallback message

A max_tokens stop is never an answer you should show unchanged; it is a truncated one. A refusal means the request was declined by the provider's safety systems, which can occasionally happen to harmless requests that resemble harmful ones, so your code needs a graceful path for it rather than an empty screen.

Line (3): usage reports the exact token counts you are billed for, and it is the raw material for every cost figure in your system.

3. What it costs, and Friday's surprise

Providers price per million tokens, with output tokens several times more expensive than input tokens, for the reason explained in 12.7.6. At list prices in October 2026, three tiers of Claude models illustrate the spread:

Model	Input, per million tokens	Output, per million tokens
Claude Opus 5.5 (most capable)	$4.00	$20.00
Claude Sonnet 5.5 (balanced)	$2.00	$10.00
Claude Haiku 5.5 (fast, cheap)	$0.10	$0.50

A typical "Ask Cartwise" message carries about 1,800 input tokens (a 1,500-token system prompt with policies, plus history and the question) and produces about 250 output tokens. The cost of one message:

• Opus 5.5: 1{,}800 \times \$4 / 10^6 + 250 \times \$20 / 10^6 = \$0.0072 + \$0.0050 = \$0.0122
• Sonnet 5.5: \$0.0036 + \$0.0025 = \$0.0061
• Haiku 5.5: \$0.00018 + \$0.000125 \approx \$0.0003

At 100,000 messages a day, that is about $1,220, $610 or $30 a day. Prices change often; the habit that matters is the arithmetic, done before launch with your real token counts.

Friday's surprise had two causes that this arithmetic exposes. First, every message resent the full conversation, so a long conversation's tenth message carried ten times the history of its first, and the average input was not 1,800 tokens but over 5,000. Second, the 1,500-token system prompt was paid for in full on every single message. The fix for the second is the prompt caching of 12.7.6: mark the stable beginning of the prompt as cacheable, and later requests that start with the identical text pay a cache-read price of about $0.20 per million tokens instead of the full input price.

response = client.messages.create(
    model="claude-opus-5-5",
    max_tokens=1024,
    cache_control={"type": "ephemeral"},             # (1) cache the stable prefix
    system=SUPPORT_POLICIES,                         # (2) 1,500 tokens, identical every time
    messages=history + [{"role": "user", "content": question}],
)
print(response.usage.cache_read_input_tokens)        # (3) > 0 once the cache is warm

Line (1) asks the API to cache the request's stable beginning automatically. Line (2) must be byte-for-byte identical across requests for the cache to match: putting the current time or the customer's name at the top of it would break every match. Line (3) confirms it worked; if it stays at zero across repeated requests, something in the prefix is changing. On Opus 5.5, caching the 1,500 policy tokens brings the example message from about $0.0122 to about $0.0065, nearly halving it.

The other levers follow from the table. Use the smallest model that does the job well, measured with the evals of 12.8.6: routing emails to teams needs nothing like the capability of drafting a careful reply to an angry customer, and Chapter 12.12.1 shows how to route each request to the cheapest model that handles it. Keep outputs short, since output tokens cost the most. And for work that does not need an immediate answer, such as tagging last month's reviews, providers offer batch endpoints that process requests within hours at a large discount.

4. When calls fail: rate limits, timeouts and retries

Rate limits

Monday's 429 errors were rate limits. Every API account has a ceiling on requests per minute and on tokens per minute, and Monday evening's traffic crossed it. HTTP status 429 means "Too Many Requests", and the response usually carries a retry-after header saying how many seconds to wait.

Two things fix this, and you need both. The first is to retry with exponential backoff and jitter, the pattern from Chapter 10.9: wait a short time, then longer, then longer still, with a random amount added so that a thousand clients that failed at the same moment do not all retry at the same moment. The official SDKs already do this for you for 429s, server errors (5xx) and connection failures, with two retries by default. The second is to limit your own concurrency so that you rarely hit the ceiling: if you know you can sustain 50 requests in flight, a semaphore in your service that allows at most 50 at a time queues the rest gracefully instead of letting the provider reject them. And plan the capacity: providers raise limits with usage tiers, and a launch plan should include asking for the limit you will need.

Timeouts

Wednesday's ninety-second request was a long answer from a busy model, and the app's 30-second timeout gave up while the server kept working. LLM calls are slow compared with ordinary APIs: seconds, sometimes tens of seconds, for long outputs. The defaults in the SDK are generous (minutes) precisely because of that. The answer for anything a person waits for is streaming, which sends tokens as they are generated, so the user sees progress within a second or two and no single long wait is needed; 12.8.5 builds it. For calls with no person waiting, set a timeout that matches what the work needs, and remember that with retries, the worst-case wall-clock time is the timeout multiplied by the number of attempts.

Handling errors properly

Not every error should be retried. A 400 (bad request) means your request is wrong and will be wrong again; retrying it wastes time and money. A 401 means the key is bad. A 429 or a 5xx is worth retrying; a dropped connection usually is. So catch errors from most specific to least specific, and decide per kind:

import anthropic

def ask(question: str, history: list) -> str:
    try:
        response = client.with_options(timeout=60.0, max_retries=3).messages.create(  # (1)
            model="claude-opus-5-5",
            max_tokens=1024,
            system=SUPPORT_POLICIES,
            messages=history + [{"role": "user", "content": question}],
        )
    except anthropic.BadRequestError as e:          # (2) our bug: do not retry
        log.error("bad request", extra={"error": e.message})
        raise
    except anthropic.RateLimitError:                # (3) still limited after the SDK's retries
        return "We're busy right now. Please try again in a minute."
    except (anthropic.APIConnectionError, anthropic.InternalServerError):  # (4)
        return "Sorry, something went wrong. Please try again."

    if response.stop_reason == "refusal":           # (5) declined: safe fallback
        return "I can't help with that here. A support agent can help: tap 'Talk to us'."
    if response.stop_reason == "max_tokens":        # (6) cut off: never show half an answer
        log.warning("truncated answer", extra={"request_id": response._request_id})
    text = "".join(b.text for b in response.content if b.type == "text")   # (7)
    log.info("llm_call", extra={"request_id": response._request_id,        # (8)
                                "in": response.usage.input_tokens,
                                "out": response.usage.output_tokens})
    return text

Line (1) sets a 60-second timeout and three automatic retries for this call only, without changing the shared client. Line (2) handles the error that means we sent something invalid: log it and fail loudly, because it will not fix itself. Line (3) handles a rate limit that outlasted the SDK's own retries with a friendly message instead of a blank screen. Line (4) does the same for network failures and server errors. Lines (5) and (6) handle the two stop reasons that are not normal answers. Line (7) collects only text blocks, which is the fix for Thursday's crash. Line (8) logs the provider's request ID and the token counts, which is what lets you investigate a complaint and attribute cost to a feature later.

One more decision belongs here. If a model declines a request, some providers can automatically retry it on another model. Anthropic's API offers this as a fallback option: with client.beta.messages.create(..., betas=["server-side-fallback-2026-07-01"], fallbacks="default"), a refusal is re-run on a suitable fallback model within the same call. It is worth enabling for customer-facing features, so that an over-cautious decline on a harmless question does not reach the customer.

5. Habits for every LLM call in production

• Keys live on the server. Your mobile app or web page calls your backend, and your backend calls the provider. A key in client code will be found and abused within days.
• Log every call with its request ID, model, token counts, latency and which feature made it. Without that, the monthly bill is one opaque number. With it, you can see that one feature is half the cost.
• Put a budget on every feature. A maximum number of calls per user per minute and a maximum max_tokens per call stop one misbehaving loop, or one abusive user, from producing a surprise bill.
• Treat the model name as configuration. Models are updated and retired regularly; changing one should be a configuration change followed by an eval run (12.8.6), not a code change.
• Wrap the provider once. Keep every call to the provider behind one small function or class in your code, like ask above, so that retries, logging, budgets and a future change of provider all happen in one place.
• Never send what you must not. Everything in the prompt leaves your system. Check what customer data you send against your privacy commitments, and strip what the model does not need, as Chapter 12.14.3 explains.

What the interviewer will push on

"How would you make an LLM feature reliable in production?" They want timeouts, retries with backoff and jitter for retryable errors only, concurrency limits below the rate limit, streaming for user-facing calls, handling every stop_reason, and graceful fallbacks. A list that stops at "retry on error" is the weak answer.

"Estimate the cost of this feature." They are checking that you multiply tokens per call by price per token by call volume, separate input from output, and notice that conversation history makes input grow with each turn. Mentioning prompt caching and smaller models for simple steps is the tell.

"Why did the answer stop mid-sentence?" max_tokens is a hard ceiling, not a target; check stop_reason and never display a truncated answer as if it were complete.

"Where does the API key go?" On the server only, injected from a secret store, with the client calling your backend.

Recall

• A request is a model ID, a max_tokens ceiling, a system prompt and the full list of role-tagged messages; the API is stateless, so history is resent and billed every time. max_tokens cuts generation off; it is not a target length.
• A response is a list of typed content blocks (text, thinking, tool use), a stop_reason, and usage token counts. Check block types before reading, and handle max_tokens, refusal and tool_use stops explicitly.
• Cost is tokens × price, with output dearer than input; resent history and an uncached system prompt are the usual surprises. Prompt caching, the smallest model that passes your evals, short outputs and batch endpoints cut it.
• 429 rate limits need backoff with jitter (the SDK retries by default) plus your own concurrency limit; slow calls need streaming and sensible timeouts; retry only retryable errors.
• Keep keys server-side, log request IDs and tokens per feature, budget every feature, treat the model name as configuration, and wrap the provider in one place.

Self-test: A conversation's tenth message carries 9 earlier exchanges of about 300 tokens each plus a 1,500-token system prompt. How many input tokens is it, and what does caching change? Which errors should never be retried, and why? Why did response.content[0].text crash?

Next: 12.8.2 Prompt and Context Engineering — the call now works reliably; what decides whether the answer is good is what you put in the request, and that is a craft with real techniques behind it.