Appearance
12.11.4 — Self-Hosting versus APIs
After the ticket classifier of 12.11.3 went live on Cartwise's own GPUs, a director asked the obvious next question: "If running our own model is cheaper, why are we paying a provider for everything else? Let's host all of it ourselves."
It is a reasonable question with a surprising answer. Running models yourself is sometimes much cheaper than an API, sometimes much more expensive, and the difference depends less on the price of a GPU than on how busy you can keep it. This page works out when self-hosting pays, the reasons other than cost that decide it, the options between the two extremes, and what running models yourself actually involves.
1. What "self-hosting" means
Using an API means sending requests to a model provider, which runs the model on its hardware and charges per token. You never see the weights. You get the provider's newest models the day they are released, scaling is the provider's problem, and you pay only for what you use.
Self-hosting means running a model whose weights you have, an open-weights model published for anyone to download, or one you fine-tuned from it, on GPUs you rent or own. You choose the model, the version, the hardware and the software that serves it, and you pay for the GPUs whether or not they are busy.
That last phrase is the whole economic difference. An API's cost goes up and down with your traffic. A GPU's cost is fixed per hour, and it is cheap per token only when it is processing many tokens.
2. The arithmetic of a GPU
Work it through with round numbers. Suppose you rent a data-centre GPU for $2.50 an hour, and you measure that, serving an 8-billion-parameter model with vLLM (12.7.6) and batching many requests together, it produces about 2,500 output tokens per second at a latency your users accept. Both numbers are assumptions; you must measure your own, since they vary greatly with the model, the hardware, the prompt lengths and the latency you need.
In one hour the GPU can produce 2{,}500 \times 3{,}600 = 9{,}000{,}000 tokens, 9 million. If it is busy every second of that hour, the cost per million output tokens is
\frac{\$2.50}{9} \approx \$0.28
But no real service keeps a GPU busy every second. Traffic rises during the day and falls at night, and you need spare capacity for the peaks, so a GPU sized for the busiest hour is idle much of the rest. The fraction of its capacity actually used, averaged over the day, is its utilisation, and dividing by it gives the real cost:
| Average utilisation | Cost per million output tokens |
|---|---|
| 100% (never happens) | $0.28 |
| 50% | $0.56 |
| 30% (common) | $0.93 |
| 10% | $2.78 |
Now compare with an API. Claude Haiku 5.5 charges $0.50 per million output tokens and $0.10 per million input tokens, with no idle time ever. At a typical 30% utilisation, the self-hosted small model costs more per output token than a small model through an API, before counting the people who run it. At 10%, a common figure for a new internal service, it costs more than five times as much.
So the first lesson is counter-intuitive: self-hosting a small model is not automatically cheaper than calling a small model through an API. It wins on cost when the traffic is large and steady enough to keep GPUs busy, or when the alternative is not a small API model but a large one, as with the ticket classifier, where the comparison was against Opus.
Don't forget the people
The GPU is not the whole cost. Someone must choose and size the hardware, set up the serving software, monitor it, wake up when it fails at 3 a.m., apply security updates, and handle the day the cloud region runs out of the GPU type you need. Even a fraction of one engineer's time, say a fifth of a $15,000-a-month salary, is $3,000 a month, more than two rented GPUs running all month (about $3,650). For a small deployment the people often cost more than the hardware.
A useful rule of thumb follows: self-hosting pays on cost alone only when the API bill it replaces is several times larger than the GPUs plus the people. The ticket classifier just cleared that bar against Opus; most internal tools never come close.
3. Reasons that are not about cost
Cost is often not the deciding factor at all. Five other reasons lead companies to run their own models.
Data that must not leave. Some data, such as health records, certain financial data, or government data, may by law or contract only be processed in specific places or by the company itself. Self-hosting keeps every token inside your own systems. Before concluding that this requires self-hosting, check the middle options in section 4, which often satisfy the same rules.
Latency you control. A provider's latency varies with its load, and requests travel over the internet. A model on your own GPU next to your application can respond in tens of milliseconds, consistently. The ticket classifier's 500-millisecond target is an example.
Control over the model. With the weights, you can fine-tune freely (12.11.2), constrain decoding however you like, read the probability of every token, and keep exactly the same model version for as long as you want. Providers retire old models on their own schedule; a self-hosted model changes only when you change it.
Running without a network. A model on a phone, a laptop, a factory machine or a delivery van works with no connection. Small quantised models (12.7.8) make this practical for narrow tasks.
No dependence on one supplier. A provider's outage, price change or rate limit stops your product. Teams with critical AI features often keep a self-hosted fallback for exactly this reason, as the ticket classifier's fallback worked in the other direction.
Against these sit the reasons to stay with an API, and they are strong. The best models are not available as open weights. The most capable models for agents, coding and difficult reasoning are offered only through APIs, so for tasks that need them there is no self-hosting option. APIs improve without effort: a new model is a one-line change. Scaling is free: a sudden ten-fold spike in traffic is the provider's problem, while on your own GPUs it is a queue of waiting customers, because starting a new GPU server and loading a model into it takes minutes. And you need no GPU expertise, which is scarce and expensive.
4. The options in between
"API or self-host" is really a range, and the middle options solve many problems that seem to require full self-hosting.
Option ② deserves particular attention. The large cloud providers (Amazon Bedrock, Google Cloud's Vertex AI, Microsoft Foundry) offer models from several model companies, including Claude, inside your own cloud account. You pay per token, the data is processed in the cloud region you choose under the agreement you already have with that cloud provider, and you get the same frontier models. For many companies whose concern is "our data must stay in our cloud, in India", this satisfies the requirement without running a single GPU.
Option ③, services that host open-weights models and charge per token or per hour, gives the control of open weights (including your own fine-tunes) without operating the servers. It is often the right first step for a fine-tuned model, before the traffic justifies dedicated GPUs.
5. What running it yourself involves
If the decision is option ④ or ⑤, the work falls into a few areas, each of which Chapter 12.12 treats in depth.
Choosing the model and checking its licence. Open-weights models come with licences, and they differ. Some are fully permissive, such as Apache 2.0 or MIT, allowing commercial use and modification. Others add conditions: Meta's Llama licence, for example, requires companies with more than 700 million monthly users to obtain a separate licence, and comes with a policy on prohibited uses. Read the licence before building on a model, not after.
Choosing the hardware. The model's weights, at the precision you serve them in, plus the KV cache for the conversations you serve at once, must fit in GPU memory, with room to spare. A 7B model at 8 bits needs about 7 GB for weights and fits on a modest GPU; a 70B model needs several large GPUs working together. 12.12.3 compares GPU types and their costs.
The serving software. You do not write this yourself. Serving engines such as vLLM and SGLang implement continuous batching, paged KV-cache memory, quantised weights, structured outputs and multiple LoRA adapters, and expose an HTTP API in the same shape most applications already use. For running on a laptop or a CPU-only machine, llama.cpp and tools built on it, such as Ollama, run quantised models in the compact GGUF format of 12.7.8.
Scaling and failure. Requests are queued and spread across GPU servers, and new servers are started when the queue grows, the scaling of Chapter 10.2. Because a new server takes minutes to load a model, you scale on early signals such as queue length, keep a minimum number running, and keep a fallback, often an API, for when demand outruns capacity.
Evaluating every change. A new model version, a new quantisation or a new serving setting can change outputs. Each is checked on the evaluation set before release (12.8.6), exactly as the ticket classifier's quantised version was.
6. Mixing the two
Most mature systems do not choose one. They route each kind of work to the cheapest option that does it well, the same routing as 12.10.6, with self-hosted and API models as different destinations. Cartwise's answer to the director was this:
| Work | Where it runs | Why |
|---|---|---|
| Ticket routing (2M/month) | self-hosted fine-tune | volume, latency |
| Refund agent | Opus via API | needs the strongest model |
| "Ask Cartwise" answers | API models, routed by difficulty | quality, elastic traffic |
| Product search embeddings | self-hosted small model | huge steady volume |
| Internal tools | API | low, spiky volume |
| Customer data processing in India | provider models via Cartwise's cloud account | data stays in region |
Each row follows from sections 2 and 3: self-host where volume is high and steady and a small model is good enough, use APIs where the best models are needed or traffic is unpredictable, and use the cloud-account option where the concern is where data is processed.
What the interviewer will push on
"When would you self-host a model instead of using an API?" They want both halves.
- Cost, which depends on how busy the GPU is. A GPU costs the same per hour whether it works or not. So the cost per token is the hourly price divided by the tokens it actually processes.
- The reasons that are not about cost: data that must not leave, speed, control over the model and its version, working offline, and not depending on one supplier.
The tell is mentioning utilisation and the people cost.
"Is self-hosting cheaper?" Only at high, steady utilisation, or when it replaces an expensive large model with a small fine-tuned one. Work the arithmetic: hourly price divided by tokens per hour times utilisation, compared with the API's per-token price, then add engineering time.
"Our data can't leave our cloud. Do we have to self-host?" Not necessarily: provider models offered through your cloud account process data in your chosen region under your existing agreement. Check whether that satisfies the requirement before taking on GPU operations.
"What do you need to run a model in production yourself?" A licence that allows your use, hardware that fits weights plus KV cache, a serving engine (vLLM or SGLang), queue-based scaling with a minimum warm capacity and a fallback, monitoring, and evaluation of every change.
Something to volunteer: GPUs scale slowly. Loading a model onto a new server takes minutes, so a self-hosted service needs warm spare capacity or an API fallback for traffic spikes, which is a cost that per-token comparisons usually leave out.
Recall
- An API charges per token and scales with traffic; a self-hosted model costs a fixed amount per GPU hour, busy or idle. Per-token cost of self-hosting = hourly price ÷ (tokens per hour × utilisation).
- At $2.50 an hour and 2,500 tokens a second, full use is about $0.28 per million output tokens, but 30% utilisation is $0.93 and 10% is $2.78, often more than a small API model. Add engineering time, which can exceed the hardware.
- Self-hosting wins on cost with high, steady volume, or when replacing a large model with a small fine-tuned one. Non-cost reasons: data that cannot leave, controlled latency, control of model and version, offline use, supplier independence.
- APIs win for the most capable models (not available as open weights), effortless upgrades, elastic scaling and no GPU expertise.
- The range: provider API, provider models through your cloud account (data in your region), hosted open models, rented GPUs, your own hardware.
- Running it yourself: check the licence, size the hardware for weights plus KV cache, use vLLM or SGLang (llama.cpp and Ollama locally), scale on queue length with warm capacity and a fallback, evaluate every change. Most systems mix the two by routing.
Self-test: A GPU costs $1.20 an hour and produces 1,500 tokens a second; what is the cost per million tokens at 25% utilisation? Why did the ticket classifier clear the bar for self-hosting when most internal tools do not? What is option ② in the range, and which requirement does it often satisfy?
Next: 12.12.1 Serving Architecture and Gateways — Cartwise now runs models through APIs and on its own GPUs, for a dozen features; Chapter 12.12 is about the production system that holds all of that together, starting with the layer every request passes through.