Appearance
12.12.3 — GPUs, Latency and Cost
Cartwise's ticket classifier needed a p95 latency under half a second. While preparing it for production, the team first served the fine-tuned 7B model on a cheap GPU with its weights stored in 8 bits, and measured a p95 of 0.71 seconds. Their instinct was to buy a GPU with more arithmetic power. A colleague who had run models before did one division on a whiteboard instead, and predicted the latency to within 15% before anything was changed. The fix was not more arithmetic. It was fewer bytes.
This page explains that division and everything that follows from it: why generating text is limited by how fast a GPU can read memory rather than how fast it can calculate, how to choose a GPU, how latency is measured and budgeted, how to work out how many GPUs a service needs, and how to scale them. It applies to models you run yourself (12.11.4); for models behind an API, the provider makes these choices, but the same physics explains the latency you see.
1. Why generation is limited by memory speed
Recall from 12.7.6 that a model produces text one token at a time, and that to produce each token it must use every one of its weights. A GPU keeps those weights in its own memory, and to use a weight, the processor must read it from that memory. So for every token, the GPU reads the whole model.
A GPU's memory bandwidth is how many bytes per second it can read from its memory. That gives a hard lower limit on the time per token, for one conversation on its own:
\text{time per token} \;\ge\; \frac{\text{bytes of weights}}{\text{memory bandwidth}}
Read it as "the time per token is at least the size of the weights divided by how fast they can be read". Take the classifier: about 7.6 billion parameters. At 8 bits, each is one byte, so the weights are about 7.6 GB. The cheap GPU in the first deployment reads about 600 GB per second. So each token takes at least
\frac{7.6\ \text{GB}}{600\ \text{GB/s}} \approx 12.7\ \text{milliseconds}
The classifier writes about 30 tokens of JSON, so generation alone takes at least 30 \times 12.7 \approx 380 milliseconds, and in practice somewhat more, because no GPU reaches its full rated bandwidth and requests share it. Add reading the ticket, and under load the p95 lands around 0.7 seconds, as measured. That was the whiteboard division.
And it shows the fix. A GPU with more arithmetic would change nothing, because the arithmetic was not what the GPU was waiting for. Storing the weights in 4 bits halves the bytes to 3.8 GB, halves the minimum time per token to about 6.3 milliseconds, and brought the p95 to 0.42 seconds, inside the target, on the same cheap GPU. The quantisation of 12.7.8 is, above all, a speed technique.
Why batching is nearly free, up to a point
The same reasoning explains why serving many conversations at once is so efficient. When the GPU reads a weight, it can use it for every conversation in the batch before moving on. Reading the weights once per step for 16 conversations costs barely more time than reading them for one, so the batch produces 16 tokens per step instead of one.
How far does this go? Consider what the GPU does per byte it reads. For each weight, producing one token for one conversation takes about two arithmetic operations, a multiplication and an addition. With weights stored in 16 bits, two bytes each, that is about one operation per byte read. A top data-centre GPU such as the H100 can perform about 989 trillion 16-bit operations per second but read only about 3.35 trillion bytes per second, so it can do roughly
\frac{989 \times 10^{12}}{3.35 \times 10^{12}} \approx 295
operations in the time it takes to read one byte. At one operation per byte, a single conversation leaves more than 99% of the arithmetic idle. A batch of B conversations does B operations per byte, so the GPU stays limited by memory, and each extra conversation is almost free, until the batch reaches roughly 300. Beyond that, the arithmetic becomes the limit, and adding conversations starts to slow every one of them. In practice another limit usually arrives first: each conversation needs room in memory for its KV cache, and the memory runs out before the arithmetic does. This is why 12.7.6 concluded that KV-cache memory, not the weights, usually decides how many users a GPU can serve.
Reading the prompt is different
Everything above concerns generating tokens, the decode phase. Reading the prompt, the prefill phase, processes all the prompt's tokens at once, so each weight read serves hundreds or thousands of tokens and the arithmetic, not the memory, is the limit. Long prompts therefore cost time in a different way: the time to the first token grows with the prompt's length, while the time per later token barely changes. This difference shapes the latency measures in section 3.
2. Choosing a GPU
With that physics, choosing a GPU comes down to three questions, in order:
- Does the model fit? Weights at the chosen precision, plus the KV cache for the conversations you want to serve at once, plus some working space, must fit in the GPU's memory.
- Is it fast enough? For each conversation, time per token is at least weight bytes divided by bandwidth; check that this meets the latency target with room to spare.
- What does each token cost? Hourly price divided by the tokens it can actually produce at your utilisation (12.11.4).
The GPUs you will meet most often in the cloud, with NVIDIA's published memory and bandwidth figures, and the minimum time per token for a 7.6-billion-parameter model at 8 bits from the formula above:
| GPU | Memory | Bandwidth | 7.6B at 8-bit | Typical use |
|---|---|---|---|---|
| T4 (2018) | 16 GB | 320 GB/s | 24 ms/token | small models, classic ML, vision |
| L4 (2023) | 24 GB | 300 GB/s | 25 ms/token | efficient small-model serving, video |
| A10G | 24 GB | 600 GB/s | 13 ms/token | small LLMs, light fine-tuning |
| L40S | 48 GB | 864 GB/s | 9 ms/token | mid-size models, fine-tuning |
| A100 (2020) | 40 / 80 GB | up to 2.0 TB/s | 4 ms/token | training, larger models |
| H100 (2022) | 80 GB | 3.35 TB/s | 2.3 ms/token | high-throughput serving, training |
| H200 | 141 GB | 4.8 TB/s | 1.6 ms/token | large models, long contexts |
| B200 | ~180 GB | ~8 TB/s | 0.9 ms/token | frontier-scale serving and training |
A few things the table shows that the price list does not. The L4 and T4 read memory at roughly the same speed despite five years between them; the L4's advantages are more memory, newer number formats and lower power, not faster generation. The A10G, a variant of NVIDIA's A10 sold in Amazon's cloud, reads twice as fast as the L4 for a similar price, which is why it was the right cheap GPU for the classifier. Newer GPUs also support smaller number formats in hardware, such as 8-bit floating point from the L4 and H100 onwards and 4-bit on the B200, which lets quantised models run at full speed rather than converting numbers on the fly.
Models too large for one GPU are split across several, the tensor parallelism of 12.5.7: each GPU holds part of every layer, and together their bandwidths add up, at the cost of passing results between them on every step. A 70-billion-parameter model at 8 bits, 70 GB, fits on one 80 GB GPU only with very little room for KV caches, so it is usually served across two or four. The opposite problem, a small model wasting a large GPU, has an opposite answer: some data-centre GPUs can be divided into several smaller isolated GPUs (NVIDIA calls this Multi-Instance GPU), each serving its own small model.
3. Measuring latency
A user experiences a streamed answer in two parts, and so latency is measured in two parts.
Time to first token (TTFT) is how long the user waits before anything appears: the request travelling, waiting in a queue, the prefill reading the prompt, and generating the first token. For a chat interface it is the number that decides whether the system feels fast.
Time per output token (TPOT), also called inter-token latency, is the average time between tokens after the first. It decides how fast the text flows. People read roughly five tokens a second, so anything faster than about 50 milliseconds per token already streams faster than anyone reads.
The time to receive a complete answer of N tokens follows directly:
\text{total time} \approx \text{TTFT} + \text{TPOT} \times (N - 1)
With a TTFT of 0.4 seconds, a TPOT of 25 milliseconds and a 300-token answer, that is 0.4 + 0.025 \times 299 \approx 7.9 seconds in total, of which the user waits only 0.4, because the rest streams while they read. For a non-streamed use such as the classifier, whose output is only useful once complete, the total time is what matters.
Percentiles, not averages
Latency varies from request to request: prompts differ in length, batches fill and empty, a request arrives just as the GPU is busy. An average hides the slow ones, so latency is described by percentiles. The p50 (median) is the time half of requests beat. The p95 and p99 are the times 95% and 99% beat. A service promising "p95 TTFT under one second" is promising that only one request in twenty waits longer.
Tail percentiles matter more than they appear to, because one user action often makes several model calls. An agent run of 10 calls, each with a 1-in-100 chance of being slow, will be slow about 1 - 0.99^{10} \approx 10\% of the time. The tail of each call becomes the typical experience of the whole.
The trade between speed and throughput
Batching raises total throughput but slows each conversation a little, as each step processes more work. The figure shows the shape of the trade on one GPU.
On the left, small batches waste the GPU: each step reads all the weights to produce a handful of tokens. On the right, huge batches produce many tokens per second in total, but each conversation crawls. The useful range in the middle is where a service should run, and its exact position is set by the latency target. Serving engines expose settings for the maximum batch size and the memory reserved for KV caches, and tuning them against measured TTFT, TPOT and throughput is a standard step in putting a self-hosted model into production.
Two serving techniques shift the curve itself. Chunked prefill splits a long prompt into pieces processed alongside other conversations' decode steps, so that one long prompt does not freeze every other conversation's stream while it is read. Separating prefill from decode, running them on different groups of GPUs, lets each be tuned for its own bottleneck, arithmetic for prefill and memory for decode, and is used by large providers.
4. A latency budget
For a whole feature, a target is divided among the stages a request passes through, so that each owner knows their share. For "Ask Cartwise", whose target is a p95 time to first token of 1.5 seconds:
| Stage | Budget, p95 |
|---|---|
| Gateway and cache lookup | 30 ms |
| Query rewriting (small model) | 250 ms |
| Retrieval and reranking | 200 ms |
| Main model, time to first token | 900 ms |
| Network and margin | 120 ms |
| Total | 1.5 s |
A budget makes trade-offs explicit. Adding a reranker costs 60 milliseconds of the budget; it must buy enough quality to be worth it, and the time must come from somewhere. It also shows where effort pays: almost two-thirds of the budget is the main model's first token, which is reduced by shorter prompts, prompt caching (12.10.10), which also skips much of the prefill work, and choosing a faster model where quality allows. Shaving the gateway's 5 milliseconds would change nothing a customer could notice.
5. Capacity planning
How many GPUs does a service need? Three numbers decide it: how many requests arrive per second at the busiest time, how much work each request is, and how much work one GPU can do while meeting the latency target.
For the classifier: at the busiest hour, about 6 tickets per second arrive. Each needs about 250 tokens read and 30 written. On the A10G-class GPU with 4-bit weights, the team measured that at a batch size of 8 the time per token was about 9 milliseconds, so one GPU produces about 8 / 0.009 \approx 890 output tokens per second. At 30 tokens per ticket, that is about 30 tickets per second per GPU.
One GPU could handle five times the peak. The service still runs on two, because capacity is not only about load: a single GPU server means any failure or update stops ticket routing. Two in different zones, each able to carry the whole peak, survive either. Planning for the loss of one server, often called N+1, is standard for any service that matters.
A useful check on such a plan is Little's law, a simple rule from queueing theory: the average number of requests inside a system at once equals the rate they arrive times the time each spends inside.
\text{requests in progress} = \text{arrival rate} \times \text{time per request}
At 6 tickets a second and about 0.3 seconds each, about 6 \times 0.3 = 1.8 tickets are in progress at any moment, comfortably below the batch of 8 the GPU handles at its target latency. If the arrival rate doubled, Little's law says about 3.6 would be in progress, still fine; if it reached 30 a second, about 9, and the batch would overflow and latency would rise. The law turns traffic forecasts into the concurrency a deployment must support.
6. Scaling up and down
Traffic changes over the day and over the year, and the number of GPU servers should follow it. Two properties of model servers make this harder than for ordinary web servers.
Starting a server is slow. A new GPU server must start, download the model (gigabytes to hundreds of gigabytes) and load it into GPU memory before it can serve anything. That takes minutes, sometimes many. A traffic spike that has already arrived cannot be absorbed by servers that will be ready in five minutes. So model services do three things. They keep a warm minimum of servers always running, enough for normal peaks. They scale on early signals, such as a growing queue of requests or a rising time to first token, instead of waiting until the GPUs look busy. And they keep a fallback for demand that outruns capacity, often an API model reached through the gateway of 12.12.1. Keeping model files on fast storage close to the servers, or in images with the model already included, shortens start-up.
GPU utilisation is a poor scaling signal. A GPU can report high utilisation while it is in the memory-bound regime of section 1, with most of its arithmetic idle and plenty of room for more conversations; or low utilisation while requests queue because KV-cache memory is full. Queue length and latency percentiles describe what users experience; GPU utilisation describes something else.
Cheap interruptible capacity has its place. Cloud providers sell unused GPUs at large discounts as spot instances, which they can take back at short notice. These suit batch work, evaluation runs and training with frequent checkpoints (12.5.7), where losing a server costs minutes of repeated work. They do not suit the warm minimum of a customer-facing service, unless enough ordinary capacity can carry the load when spot servers disappear.
7. The cost levers, for a self-hosted model
Everything on this page reduces to a short list of levers, roughly in the order they pay off:
- Quantise the weights (16 to 8 to 4 bits): fewer bytes per token, faster and cheaper, checked on the evaluation set.
- Batch well: continuous batching (12.7.6) with the batch size tuned to the latency target.
- Cache shared prefixes: the prefix caching of 12.12.2 removes repeated prefill.
- Choose the GPU for bandwidth and memory, not headline arithmetic.
- Use speculative decoding (12.7.8) when single-conversation latency matters most.
- Raise utilisation: fill quiet hours with batch work, and share GPUs between small models.
- Keep outputs short: every output token costs a full read of the weights.
What the interviewer will push on
"Why is LLM inference memory-bound?" Each generated token needs every weight, so the GPU reads the whole model per token, and a single conversation does about one or two operations per byte read while the GPU could do hundreds. The time-per-token formula, weight bytes divided by bandwidth, with a worked number, is the strong answer.
"How would you reduce latency for a self-hosted model?" Quantise to read fewer bytes, choose a GPU with more bandwidth, shorten prompts and cache prefixes to cut time to first token, keep outputs short, use speculative decoding, and tune the batch size so each conversation meets its target. Explain why more arithmetic power alone would not help.
"What are TTFT and TPOT, and which matters?" Time to first token is what users wait for in a chat; time per output token decides streaming speed, and anything under about 50 ms per token outpaces reading. For non-streamed outputs, total time is what matters. Report percentiles, not averages.
"How many GPUs do we need?" Peak arrival rate times work per request, divided by what one GPU does at the latency target, then add redundancy for failure (N+1). Check concurrency with Little's law.
"How do you autoscale model servers?" Warm minimum capacity because start-up takes minutes, scale on queue length and latency rather than GPU utilisation, and fall back to an API when demand outruns capacity. Spot instances for batch and training, not for the warm minimum.
Something to volunteer: the whiteboard division. Predicting a model's per-token latency from its weight size and the GPU's bandwidth, before running anything, is the quickest way to show you understand inference rather than having memorised tool names.
Recall
- Each generated token reads every weight, so time per token for one conversation is at least weight bytes divided by memory bandwidth: 7.6 GB at 600 GB/s is about 12.7 ms. Quantising to 4 bits halves it.
- Batching shares each weight read across conversations, nearly free until arithmetic or (usually first) KV-cache memory runs out; an H100 does about 295 operations per byte read. Prefill is arithmetic-bound; decode is memory-bound.
- Choose a GPU by: does it fit (weights plus KV cache), is it fast enough (bandwidth), what does each token cost (price ÷ tokens at your utilisation). Split large models across GPUs; share GPUs between small ones.
- Latency: TTFT (what chat users wait for) and TPOT (streaming speed; under about 50 ms outpaces reading); total ≈ TTFT + TPOT × (N − 1). Use p95 and p99; tails compound across calls. Run in the batch range where throughput is high and the target is met.
- Budgets split a target across stages; the main model's first token usually dominates.
- Capacity: peak rate × work per request ÷ per-GPU capacity at target latency, plus N+1 redundancy; check with Little's law (in progress = arrival rate × time each).
- Scaling: warm minimum, scale on queue length and latency (not GPU utilisation), API fallback; spot capacity for batch and training only.
Self-test: What is the minimum time per token for a 14 GB model on a GPU with 3.35 TB/s, and what if it is quantised to 4 bits? An answer streams 500 tokens with a TTFT of 0.6 s and a TPOT of 20 ms; how long until it is complete, and how long does the user wait? Traffic peaks at 20 requests a second, each taking 0.5 s; how many are in progress at once?
Next: 12.12.4 The Model Lifecycle — a served model is not finished; the next page covers how models are trained, versioned, compared and released safely, and how to retire one, for classical models and LLM applications alike.