Appearance
12.12.1 — Serving Architecture and Gateways
By the second year, Cartwise had twelve features that called language models: "Ask Cartwise", the refund agent, the ticket classifier, product descriptions, review summaries, search query rewriting, and six internal tools. Each had been built by a different team, and each called its model provider directly, with its own API key, its own retry logic, and its own idea of how to log.
The first day of the Diwali sale showed what that costs. Traffic tripled. The provider began returning 429 Too Many Requests errors to Cartwise's account, because all twelve features shared the account's rate limit. Three features failed at once, including "Ask Cartwise", at the moment customers needed it most. The team that owned review summaries, a feature nobody urgently needed during a sale, was using a third of the capacity. Finance, asked afterwards which feature had cost what that month, could not say. And in the investigation, someone found an API key compiled into the Android app, where anyone could extract it.
None of those problems was about a model. They were about the missing layer between Cartwise's applications and the models they call. This page is about that layer: how requests to models should flow through a production system, and the gateway that sits in the middle of it.
1. The path of a model request
The figure shows how a request to a model travels in a well-built system. The same shape appears in most companies running AI at scale.
Read the gateway's steps in order. It identifies the caller: which team and which feature, from an internal credential, so the provider's key never leaves the gateway. It checks the caller's quota, so one feature cannot consume everyone's capacity. It looks up the route for this kind of task: which model to use, which to fall back to, and with what limits. It checks the cache for an answer it can reuse (12.12.2). It applies input checks, such as removing personal details before a request leaves for an external provider, the guardrails of 12.10.8 applied centrally. It calls the model, with timeouts, retries and failover. And on the way back it runs output checks, stores the answer in the cache if appropriate, and logs the request with its tokens and cost against the caller.
Three kinds of backend sit behind it, the options of 12.11.4: the provider's API, provider models through Cartwise's own cloud account, and Cartwise's self-hosted models. The applications do not know or care which one answers a given request. That ignorance is the point: models can be moved, swapped or upgraded in one place.
Classical models take a different path. The fraud model from 12.3.7 must answer within 50 milliseconds with features from the feature store of 12.4.4, and it runs as its own small service close to the payment system. A gateway built for language-model calls, which take hundreds of milliseconds, would only add delay.
2. Who is waiting? Three ways to serve
How a model request should be served depends first on who is waiting for the answer, and for how long. There are three patterns.
Online, synchronous serving. A person is waiting: a customer typed a question, a support agent clicked "suggest reply". The request goes straight to a model and the answer, usually streamed (12.8.5), comes back on the same connection. The constraint is latency, measured in seconds or less.
Asynchronous jobs. The work takes longer than anyone should hold a connection open for, but someone wants the result soon: summarising a 60-page supplier contract, or a refund agent run that pauses for approval. The request is accepted at once and put on a queue, the caller gets a job ID, workers take jobs from the queue and process them, and the result is stored for the caller to fetch or is pushed to them when ready. This is the queue pattern of Chapter 10.8.1, and its great advantage is that a burst of work waits in the queue rather than overwhelming the models.
Batch processing. Nobody is waiting: tag every product in the catalogue with attributes, re-score last month's reviews, run the nightly evaluation. The work is submitted in bulk, processed whenever capacity is cheapest, and collected later. Providers' batch APIs charge half price for exactly this (12.10.10), and self-hosted GPUs can run batch work overnight when online traffic is low, raising their utilisation.
| Pattern | Who waits | Optimise for | Cartwise example |
|---|---|---|---|
| Online | a person, now | latency | "Ask Cartwise" |
| Async job | someone, later today | reliability | contract summaries |
| Batch | nobody | cost | catalogue tagging |
The mistake to avoid is serving everything online. The review-summary feature that ate a third of the Diwali capacity was regenerating summaries on every product page view. Moved to a nightly batch job, it used a fiftieth of the tokens, cost half price per token, and no longer competed with customers for capacity.
3. What the gateway does
Keeps keys and identity in one place
Applications authenticate to the gateway with internal credentials tied to a team and a feature; only the gateway holds the providers' keys. A key can then never leak from a mobile app, because no app has one, and rotating a key is one change rather than twelve. Every request arrives labelled with its caller, which is what makes the next two functions possible.
Shares capacity fairly
Each team and feature has a quota: a number of requests or tokens per minute, enforced with the token-bucket algorithm of Chapter 9.7.5. A token bucket fills at a steady rate up to a maximum, each request takes tokens from it, and requests that find it empty are refused or queued. Quotas are sized so that customer-facing features always have capacity reserved, and low-priority features can be slowed during a peak instead of failing the important ones. With quotas in place during Diwali, the review summaries would have been throttled, and "Ask Cartwise" would have kept working.
Routes tasks, not model names
Applications ask for a task, such as support-answer, ticket-classify or product-description, not a model. The gateway's routing table maps each task to a primary model, an ordered list of fallbacks, and settings such as the maximum tokens and the timeout:
python
ROUTES = {
"support-answer": Route(primary="claude-sonnet-5-5",
fallbacks=["claude-haiku-5-5"], timeout_s=20, max_tokens=1024),
"refund-agent": Route(primary="claude-opus-5-5",
fallbacks=["claude-sonnet-5-5"], timeout_s=60, max_tokens=4096),
"ticket-classify": Route(primary="selfhosted/ticket-classifier",
fallbacks=["claude-opus-5-5"], timeout_s=2, max_tokens=80),
}Upgrading every support answer to a new model is now one line, made once, after the new model passes the evaluation set (12.8.6). The routing can also be smarter than a fixed table: the difficulty-based routing of 12.10.6, or sending a percentage of traffic to a candidate model for a gradual release (12.12.4).
Survives failures
Model APIs fail in ordinary ways: overloaded servers, rate limits, timeouts, network errors. The gateway handles them once, properly, with the mechanisms of Chapter 10.9: a timeout on every call, retries with growing, randomised waits for errors that are temporary, a circuit breaker that stops calling a backend that keeps failing, and failover to the next model in the route.
python
import time, random, anthropic
client = anthropic.Anthropic(max_retries=0) # (1) the gateway owns retries
RETRYABLE = (anthropic.RateLimitError, anthropic.APIConnectionError,
anthropic.InternalServerError) # (2) temporary failures
def call_with_failover(route, caller, messages, system):
for model in [route.primary, *route.fallbacks]: # (3)
if breaker.is_open(model):
continue
for attempt in range(3):
try:
r = backend(model).with_options(timeout=route.timeout_s).messages.create(
model=model, max_tokens=route.max_tokens, system=system, messages=messages)
breaker.record_success(model)
meter.record(caller, model, r.usage) # (4)
return r
except RETRYABLE as e:
breaker.record_failure(model)
wait = retry_after(e) or min(8, 0.5 * 2 ** attempt) * random.uniform(0.5, 1.5) # (5)
time.sleep(wait)
except anthropic.BadRequestError:
raise # (6)
raise AllBackendsFailed(route)Line (1) turns off the SDK's own automatic retries, because the gateway applies its own policy across models and two layers of retries would multiply each other. Line (2) lists the errors worth retrying: rate limits, connection failures and server errors are temporary. Line (3) works through the primary model and then the fallbacks, skipping any whose circuit breaker is open because it has been failing. backend(model) returns the client for wherever that model runs; the self-hosted classifier sits behind a small adapter that offers the same method, so the loop treats every backend alike. Line (4) records the tokens used against the caller, which is the cost attribution finance asked for.
Line (5) waits before retrying. It uses the provider's own retry-after hint when the error carries one, and otherwise waits a growing, randomised time: about half a second, then one, then two. The randomness is there so that a thousand requests that failed together do not all retry at the same instant and fail together again. Line (6) does not retry a bad request, because a malformed request will fail identically every time; retrying it only wastes capacity.
Some providers can also fail over on their own side: the Anthropic API accepts a fallback setting, described in 12.8.1, that lets it try another model if the requested one is overloaded. A gateway still needs its own failover for everything else: a different provider, a different region, or a self-hosted model.
Applies shared rules
Because every request passes through it, the gateway is the natural place for rules that apply to all features: removing card numbers and phone numbers from requests bound for external providers, refusing requests that exceed a size limit, the input and output checks every feature should have, and a kill switch per feature. Each team still adds the checks specific to its own feature, as the refund agent's authorize function did.
Records everything
Every request is logged with its caller, task, model, tokens, cost, latency, outcome and cache status, with personal details removed. This one log answers "which feature costs what" for finance, "which model is slow today" for operations, and "what changed on Tuesday" for whoever is investigating a problem, and it feeds the monitoring of 12.12.5. Each request also gets a trace ID passed back to the application, so a gateway log line can be joined to the agent trace of 12.10.9 it belongs to.
4. The abstraction trap
A gateway tempts teams to go one step further: define one universal request format, translate it into every provider's API, and let any request go to any model. Some gateway products are built entirely around this. It is useful for simple text requests and harmful for everything else, for two reasons.
A universal format hides the features that matter. Prompt caching, citations, structured outputs, extended reasoning, server-side tools and fallbacks all differ between providers, in names, shapes and behaviour. A format that only includes what every provider shares quietly drops them, and with them much of the cost and quality work of earlier chapters. A refund agent that loses prompt caching because the gateway's format had no field for it costs several times more, and nothing reports an error.
Models are not interchangeable. A prompt tuned and evaluated on one model produces different results on another: a different tone, different tool-calling habits, different failure cases. Failing over silently from one provider's model to another's can turn an outage into a quality incident nobody notices.
The practical design is to standardise the envelope and pass through the contents. The gateway handles identity, quotas, routing, retries, logging and shared rules for every request, but lets requests use each provider's native API, so features stay available. Each fallback in a route is a model that has been evaluated on that task with its own prompt, not just "any model that is up". The support-answer route falls back from Sonnet to Haiku because both were tested with the same prompt on the support evaluation set, and Haiku's slightly lower scores were judged acceptable for an outage.
5. Build or buy
A gateway can be a small internal service, as sketched above, an open-source proxy, or a product. LiteLLM, for example, is an open-source library and proxy that offers one interface over many providers, with keys, budgets and logging built in. Commercial AI gateways add dashboards and policy management, and cloud providers' API-management services can play the same role. The choice follows the usual rules: buy or adopt what is standard (authentication, quotas, logging, retries), and make sure whatever you choose passes provider-specific features through rather than flattening them. Cartwise started with a thin internal service of a few hundred lines, because its needs were simple and it wanted provider features untouched, and that turned out to be enough.
6. Where the time goes
Finally, the gateway is where a request's time can be measured end to end. A typical "Ask Cartwise" answer spends its time like this:
| Stage | Typical time |
|---|---|
| Gateway: identify, quota, route | 5 ms |
| Cache lookup | 15 ms |
| Retrieval and reranking (12.9) | 140 ms |
| Model: time to first token | 600 ms |
| Model: streaming the answer | 3–6 s, while the customer reads |
The model dominates, and of the model's time, the part the customer waits for is the time to the first token; the rest streams while they read. That is why 12.12.3 treats the first token and the rest separately, and why the infrastructure around the model, while it matters for reliability and cost, rarely matters much for speed as long as it is not doing anything foolish.
What the interviewer will push on
"How would you architect LLM access for a company with many teams?" They want an internal gateway: one entry point holding provider keys, per-team identity and quotas, task-based routing to primary and fallback models, timeouts, retries with backoff and jitter, circuit breakers, shared guardrails, caching, and logging with cost per caller. The tell is mentioning quotas as protection between teams and cost attribution.
"What is the difference between online, async and batch inference?" The difference is who is waiting:
- A person, right now. Optimise for speed.
- Someone, later. Accept the request, queue it, process it and notify them. Optimise for reliability.
- Nobody. Process in bulk, on the cheapest capacity, with batch discounts. Optimise for cost.
Give an example of moving a feature from online to batch.
"How do you handle a provider outage?" Fail over through an ordered list of models evaluated on that task, with circuit breakers so you stop hitting the failing one; retry only temporary errors, with backoff, jitter and the provider's retry-after hint. Mention that silently swapping to an unevaluated model can create a quality incident.
"Should the gateway expose one universal API for all providers?" Standardise the envelope (identity, quotas, routing, retries, logging), not the contents; a lowest-common-denominator format loses caching, citations, structured output and reasoning settings.
Something to volunteer: keep provider keys out of every application, especially mobile and browser apps, where they can be extracted. A gateway makes that rule easy to follow, and the Android key found in Cartwise's investigation is the kind of thing that turns up in real audits.
Recall
- Every generative model request goes through one internal gateway: identify caller, check quota, route by task, check cache, input checks, call with timeout, retry and failover, output checks, log cost. Backends: provider API, provider models in your cloud account, self-hosted models. Classical low-latency models keep their own path.
- Serve by who is waiting: online (a person now, latency), async jobs via a queue (someone later, reliability), batch (nobody, cost). Don't serve everything online.
- What the gateway does:
- Holds the API keys. Nobody else has them.
- Gives each team a token-bucket quota, so internal tools cannot starve customer-facing features.
- Routes by task, with fallbacks that have been evaluated.
- Handles timeouts, retries with backoff, jitter and retry-after, and circuit breakers. Bad requests are never retried.
- Applies shared guardrails and kill switches.
- Logs the caller, tokens, cost, latency and trace ID of every call.
- The abstraction trap: standardise the envelope, pass through provider-native contents, so caching, citations and structured output survive; fallbacks must be evaluated on the task.
- Most of a request's time is the model, and the part users wait for is the time to first token.
Self-test: During a peak, which gateway function would have kept "Ask Cartwise" working while review summaries consumed capacity? Why does the gateway disable the SDK's own retries? A route fails over from one provider's model to another's that was never evaluated on the task; what could go wrong even though every request succeeds?
Next: 12.12.2 Caching — the cheapest model call is the one you never make; the next page covers every layer of caching in an AI system, from the provider's prompt cache to caches that recognise the same question asked in different words, and when each one returns a wrong answer.