12.5.3 — How a Prompt Flows, and What Scale Bought

You send "Explain gravity in one sentence." and words appear one at a time. Between those two events is a pipeline worth knowing exactly, because every latency number, every cost line and every quality knob you will tune lives somewhere in it.

1. The path a prompt takes

1. Tokenize. The string becomes token ids (Chapter 12.5.2) — perhaps 7 of them.

2. Embed. Each id indexes into an embedding matrix, giving a vector per token. Positional information is applied, usually by rotating queries and keys (Chapter 12.5.1).

3. Run the stack. Every transformer block mixes information across positions with attention and processes each position with a feed-forward network. After the last block, each position holds a vector representing "everything relevant to what comes after this position".

4. Unembed. The final position's vector is multiplied by the output matrix — usually the transpose of the embedding matrix — producing one logit per vocabulary entry. For a 50,000-token vocabulary, 50,000 raw scores.

5. Sample. Softmax turns logits into probabilities; a sampling rule picks one token.

6. Append and repeat from step 3 with the sequence one token longer, until a stop token or a limit.

Two things are worth pausing on.

Only the last position's output is used for the next token. All the other positions were computed to inform it through attention — and in training, every position predicts its own next token at once, which is what makes training parallel and generation sequential.

The model has no memory between calls. Every request re-processes the entire conversation. "The model remembers what I said" is the application resending the history, and it is why a long conversation gets slower and more expensive with each turn.

2. Prefill and decode: two different machines

This split explains almost every performance characteristic of a hosted model.

Prefill processes the whole prompt at once. Every token's attention is computed in parallel, so it is one big matrix multiplication — compute-bound, and it uses the hardware well. A 2,000-token prompt is not much slower than a 200-token one.

Decode produces one token at a time, and each step must read the entire set of model weights from memory to produce a single token. It is memory-bandwidth-bound, and the arithmetic units sit mostly idle.

Three consequences you will observe directly:

Time to first token is dominated by prefill and grows with prompt length. Time per output token is roughly constant. So a long prompt with a short answer feels laggy at the start and then fine.

Output tokens are typically priced higher than input tokens, and this is why — they are the expensive phase.

Batching helps decode enormously. Since the weights must be read anyway, serving 32 requests together costs barely more than one. Continuous batching is the technique that makes this practical: rather than waiting for a whole batch to finish, finished sequences leave and new requests join every step, which raises throughput several times over naive batching. It is the main reason hosted inference is cheaper than running the model yourself at low volume.

The KV cache is what stops decode being quadratic, by keeping each token's keys and values so they are computed once (Chapter 12.5.1). Prefix caching extends the idea across requests: if many requests share a long system prompt, its keys and values are computed once and reused. This is why providers discount cached input tokens heavily, and why you should put the stable part of your prompt first — a reordering that costs nothing and can cut cost substantially.

3. Decoding: choosing the next token

The model gives a distribution. What you do with it is a real decision.

Greedy — always take the highest probability. Deterministic, and it produces repetitive, flat text, because natural language is not the most probable continuation at every step.

Temperature divides the logits before softmax:

p_i = \frac{e^{z_i/T}}{\sum_j e^{z_j/T}}

• T \to 0 — the distribution collapses onto the maximum. Greedy.
• T = 1 — the model's own distribution.
• T > 1 — flattened; unlikely tokens become plausible.

Use 0 for extraction, classification and code; 0.7–1.0 for open-ended writing. Note that even at temperature 0, hosted models are not perfectly reproducible — batching and floating-point non-associativity across different hardware groupings introduce small variations.

Top-k keeps the k most likely tokens and renormalises. Simple, and a fixed k is wrong in both directions: when the model is confident, it admits bad options; when it is uncertain, it cuts good ones.

Top-p (nucleus) keeps the smallest set of tokens whose probabilities sum to p. This adapts: a confident step keeps 2 tokens, an uncertain one keeps 50. Top-p around 0.9 is the sensible default, and generally you should tune either temperature or top-p, not both.

Min-p keeps tokens above a fraction of the top token's probability — a newer alternative that behaves well at higher temperatures.

Repetition, frequency and presence penalties reduce the probability of tokens already produced. Useful in small amounts; too much and the model avoids necessary words, which reads as strained writing or, in code, as broken syntax.

Beam search keeps the b most likely partial sequences and extends them all, choosing the best complete one, with a length penalty because multiplying more probabilities always gives a smaller number and would otherwise bias toward short outputs. It is right for translation and summarisation, where one best answer exists, and wrong for open-ended text, where it produces bland, generic output — searching harder for high probability is exactly not what makes writing good.

Stop sequences end generation on a chosen string, which is how you keep a model from continuing past the answer.

4. Getting structured output

Free text is hard to consume. Three mechanisms, increasing in strength:

Prompting — "reply with JSON". Works most of the time, and most of the time is not a contract.

JSON mode — the provider constrains output to syntactically valid JSON. Guarantees parseability, not your schema.

Constrained decoding — at each step, mask the logits of every token that could not continue a valid instance of your grammar or schema, then sample from what remains. This makes invalid output structurally impossible, not merely unlikely. Libraries build it from a JSON Schema or a grammar, and providers expose it as structured output modes.

Tool calling is the same machinery: you supply function signatures, the model emits a structured call, your code executes it and returns the result. Chapter 12.6.3 covers the loop.

One caution: constraining the format does not constrain the content. A schema-valid object can hold invented values, so validate semantically too.

Speculative decoding is a latency trick worth knowing: a small fast model proposes several tokens, and the large model verifies them in one parallel pass, accepting the prefix it agrees with. The output distribution is unchanged, and typical speed-ups are two to three times.

5. Mixture of experts

A dense model uses all its parameters for every token. A mixture of experts model replaces the feed-forward layer with many parallel "experts" and a small router that sends each token to a few of them — often 2 out of 8, or 8 out of 64.

The point is the gap between total and active parameters. A model may hold 400 billion parameters and activate 30 billion per token. Quality scales roughly with total parameters; cost scales with active parameters. That is an extremely attractive trade, and it is why most frontier-scale models are now sparse.

The costs are specific. All experts must be in memory even though few are used, so memory footprint follows the total. Routing must be load-balanced — an auxiliary loss pushes tokens to spread across experts, because otherwise a few experts get everything and the rest are dead capacity. And distributing experts across devices makes routing a communication problem.

6. Quantization

Weights are usually trained in 16-bit. Quantization stores and computes them in fewer bits.

The memory arithmetic is the whole story, and it is simple:

Precision	Bytes per parameter	7B model	70B model
FP32	4	28 GB	280 GB
BF16	2	14 GB	140 GB
INT8	1	7 GB	70 GB
INT4	0.5	3.5 GB	35 GB

That is what "a 4-bit model" means, and why it matters: 4-bit puts a 70B model on two consumer accelerators instead of eight, and a 7B model on a laptop. And because decode is memory-bandwidth-bound (section 2), fewer bytes per weight also means faster generation — quantization improves latency as well as fitting.

Post-training quantization converts an existing model, sometimes with a small calibration dataset. GPTQ and AWQ are the common methods; GGUF is the container format used by local runtimes.

Quantization-aware training simulates the reduced precision during training and produces better results at very low bit widths, at the cost of a training run.

What to expect in quality: 8-bit is essentially lossless. 4-bit with a good method is a small, often unnoticeable degradation for general use. Below 4 bits, quality falls off quickly. The loss is not uniform across tasks — reasoning and code tend to suffer more than casual conversation, so evaluate on your own workload rather than trusting a general claim.

The KV cache can be quantized separately, which matters because at long contexts it can exceed the weights.

7. Scaling laws, and what changed

Kaplan et al. (2020) found that loss falls as a smooth power law in model size, dataset size and compute, over many orders of magnitude. A predictable relationship between money and capability is unusual, and it is what justified the investment that followed.

Chinchilla (2022) corrected the recipe. Given a fixed compute budget, earlier models were too large and trained on too little data; the compute-optimal ratio is roughly 20 tokens per parameter. A 70B model trained on 1.4 trillion tokens beat a 175B model trained on 300 billion.

And practice then diverged from compute-optimal on purpose. Training cost is paid once; inference cost is paid forever. So models are now deliberately "over-trained" — smaller than compute-optimal, trained on far more data — because a smaller model that is cheaper to serve is worth extra training. That is why capable 7–30B models exist at all.

The newer axis is inference-time compute. Rather than making the model bigger, let it produce more tokens of intermediate reasoning before answering. Accuracy on hard problems improves with the amount of thinking, which trades a training-time cost for a per-request one, and is the basis of reasoning models. The engineering consequence is direct: those requests are slower and more expensive per answer, so you route to them selectively.

On "emergent abilities" — abilities appearing suddenly at scale — be careful. Later analysis argued that much of the apparent discontinuity comes from using all-or-nothing metrics: measure exact-match accuracy and a capability appears to jump; measure partial credit and the curve is smooth. Some sharpness is real and the strong version was overstated, which is the accurate thing to say rather than either extreme.

8. Context windows, honestly

Windows have grown from 2,000 tokens to hundreds of thousands, and three practical limits remain.

Cost and latency scale with input. A 100,000-token prompt on every request is a large bill and a slow first token, and prefix caching only helps for the shared prefix.

Lost in the middle. Retrieval accuracy is measurably better for information at the start and end of a long context than in the middle. Position matters, so put the most important material first or last.

Advertised length is not usable length. Models perform better within a fraction of their maximum window than at the limit, and benchmark scores on long-context retrieval degrade well before the stated maximum.

Which is why retrieval did not become obsolete when context windows grew. Retrieving the right 4,000 tokens generally beats dumping 100,000 on cost, on latency and often on accuracy. Chapter 12.6.2 builds that system.

Recall

• The path: tokenize → embed + position → transformer blocks → only the last position's vector → logits over the vocabulary → sample → append → repeat. The model has no memory between calls — the application resends the history.
• Prefill is compute-bound and parallel; decode is memory-bandwidth-bound and one token at a time. Hence: time to first token grows with prompt length, time per output token is flat, output tokens cost more, and continuous batching is what makes serving cheap.
• Prefix caching makes a shared system prompt nearly free — so put the stable part first.
• Decoding: temperature 0 for extraction and code, ~0.7 with top-p ~0.9 for writing. Top-p adapts where top-k does not. Beam search suits translation and ruins open-ended text. Penalties in small amounts only.
• Constrained decoding masks invalid tokens, making malformed output structurally impossible — but it does not constrain content, so validate semantically. Speculative decoding is a 2–3× latency win with an unchanged output distribution.
• Mixture of experts: quality scales with total parameters, cost with active parameters. Memory still follows the total, and routing needs load balancing or experts die.
• Quantization arithmetic: parameters × bytes. 8-bit is near-lossless, 4-bit is a small loss, below 4 falls off fast — and because decode is bandwidth-bound, fewer bits also means faster.
• Chinchilla: ~20 tokens per parameter is compute-optimal — and practice over-trains smaller models deliberately, because inference is paid forever. The new axis is inference-time compute. Long context has diminishing returns and a lost in the middle effect, which is why retrieval still wins.

Self-test: Which position's output produces the next token, and why are the others computed? · Why is time to first token variable and time per token flat? · What does prefix caching reward you for doing? · When is beam search the wrong choice? · What is the gap between total and active parameters worth? · Why did large context windows not make retrieval obsolete?