Appearance
12.7.7 — Decoding and Sampling
Cartwise used the same model for two jobs and got opposite complaints.
The email router, which returns one of four team names, was described by the support leads as "moody": the same email was occasionally routed to different teams on different runs. The apology writer, which drafts a personal note to customers whose parcels are late, was described by the marketing team as "robotic": every note began "We are sorry for the delay. We are sorry for any inconvenience…" and some repeated themselves.
Nothing was wrong with the model. Both problems came from the last step of 12.7.1: after every decode step, the model produces a probability for every token in its vocabulary, and some rule must choose one. That rule is called the decoding or sampling strategy, and it is a set of settings you pass on every API call: temperature, top-p, top-k, penalties, and sometimes beam search. The router needed less randomness; the writer needed more, and something to discourage repetition. This page explains each setting from the probabilities up, with worked numbers, and ends with the settings to use for each kind of task.
1. The starting point: a probability for every token
Take one decode step in an apology note: "We are sorry for the ___". The model's final layer produces a logit for every token, and softmax from 12.2.6 turns them into probabilities. Showing just five tokens out of the vocabulary:
| Token | Logit | Probability |
|---|---|---|
| delay | 3.0 | 0.519 |
| inconvenience | 2.3 | 0.258 |
| trouble | 1.6 | 0.128 |
| wait | 1.2 | 0.086 |
| banana | −1.0 | 0.010 |
(In a real model, the remaining probability is spread across tens of thousands of other tokens; this small table is enough to see every method work.)
Every method below takes this distribution and returns one token.
2. Greedy decoding: always take the top token
Greedy decoding picks the single most probable token at every step: here, "delay". It is fast, simple and completely repeatable: the same prompt always gives the same output.
For tasks with one right answer, such as the email router choosing a team name, extracting an order number, or classifying sentiment, greedy decoding is exactly what you want. The router's "moodiness" came from leaving sampling switched on at its default settings; switching to greedy (temperature 0, below) fixed it.
For open-ended writing, greedy decoding has a well-known failure: it produces flat, repetitive text and can fall into loops. The reason is subtle and worth understanding. The most probable token at each step does not make the most natural text. Real human writing is full of moderately probable word choices; people rarely pick the single most predictable word every time. Worse, once a phrase has been produced, repeating it becomes more likely (the model has seen that text repeats), so greedy decoding can lock into "We are sorry for the delay. We are sorry for the delay…". That is the apology writer's complaint.
3. Temperature: reshape the distribution before choosing
The first knob on every API is temperature, T, which you met in 12.2.6. Divide every logit by T before the softmax, then choose a token at random in proportion to the resulting probabilities, which is called sampling:
p_i = \frac{e^{z_i / T}}{\sum_j e^{z_j / T}}
Here is what it does to our five tokens:
| Temperature | delay | inconvenience | trouble | wait | banana |
|---|---|---|---|---|---|
| 0.2 (cold) | 0.970 | 0.029 | 0.001 | 0.000 | 0.000 |
| 0.5 | 0.749 | 0.185 | 0.046 | 0.020 | 0.000 |
| 1.0 (unchanged) | 0.519 | 0.258 | 0.128 | 0.086 | 0.010 |
| 1.5 (hot) | 0.418 | 0.262 | 0.164 | 0.126 | 0.029 |
A low temperature stretches the gaps between logits, so the top token takes almost all the probability: at 0.2, "delay" is chosen 97 percent of the time. As T approaches 0, sampling becomes greedy decoding, which is why APIs treat "temperature 0" as greedy. A temperature of 1 leaves the model's own distribution untouched. A high temperature squeezes the gaps, so unlikely tokens get a real chance: at 1.5, even "banana" rises to about 3 percent.
Temperature is the main dial between predictable and varied. The catch is visible in the last column: raising the temperature lifts every unlikely token, including nonsense. At high temperatures, models produce more original phrasing and also more errors, odd word choices and, eventually, gibberish. That is why temperature is almost always combined with the next technique.
4. Cutting off the tail: top-k, top-p and min-p
The useful part of a distribution is its head: the handful of reasonable tokens. The tail, tens of thousands of tokens each with a tiny probability, is mostly wrong, but together the tail can hold a surprising amount of probability, so plain sampling picks something absurd now and then. Three methods throw the tail away before sampling.
Top-k keeps the k most probable tokens and discards the rest. With k = 2, only "delay" and "inconvenience" survive, and are rescaled to add to 1. It is simple, but a fixed k is wrong in both directions. When the model is very sure (after "The capital of France is", one token deserves almost all the probability), k = 40 still lets 39 poor options in. When the model is genuinely unsure (the first word of a creative sentence), k = 40 may cut off good options.
Top-p, also called nucleus sampling, from a 2019 paper by Ari Holtzman and colleagues, fixes this by cutting at a probability total instead of a count: keep the smallest set of top tokens whose probabilities add up to at least p. With p = 0.9, add from the top: 0.519, then 0.777, then 0.905. That passes 0.9, so the first three are kept, and rescaled to 0.574, 0.285 and 0.141. The size of the kept set adapts to the model's confidence: when one token has 0.95, the set is that single token; when probability is spread out, the set is large. That adaptiveness is why top-p of around 0.9 to 0.95 is the common default for open-ended text. The same paper gave the name to the failure it fixed: the dull, repetitive output of greedy and beam decoding, which it called neural text degeneration.
Min-p, a more recent method, keeps every token whose probability is at least a fraction of the top token's. With min-p of 0.1, the threshold is 0.1 \times 0.519 \approx 0.052, so "delay", "inconvenience", "trouble" and "wait" survive and "banana" does not. Because the threshold moves with the top token, it stays sensible even at high temperatures, where top-p tends to let in too much.
One practical rule: tune temperature or top-p, not both at once. They interact, and changing both makes it hard to know which change helped. A common pattern is to leave top-p near its default and adjust temperature.
5. Penalties against repetition
The apology writer's repeating phrases need a different fix: lower the probability of tokens that have already appeared. Three kinds of penalty are common.
Repetition penalty, from the CTRL model's 2019 paper: divide the logit of every token that has already appeared by a factor, typically 1.1 to 1.3 (multiply instead, if the logit is negative, so the penalty always pushes down). If "delay" was already used and the factor is 1.3, its logit falls from 3.0 to about 2.31, and its probability from 0.519 to about 0.351, now level with "inconvenience" at 0.348.
Frequency penalty subtracts an amount for each time a token has already appeared: the more often it has been used, the bigger the penalty. Presence penalty subtracts a fixed amount once a token has appeared at all, encouraging the model to move on to new words and topics. Many APIs expose these two. With a frequency penalty of 0.3 and a presence penalty of 0.4, and "delay" already used twice, its logit becomes 3.0 - 2 \times 0.3 - 0.4 = 2.0, and the probabilities shift to about 0.284 for "delay" and 0.384 for "inconvenience".
Penalties help in small doses and do damage in large ones. A strong penalty discourages necessary repetition: the customer's name, the product name, "the" and "your" in ordinary sentences, the variable names and brackets in code, the keys in JSON. Over-penalised text reads as strained, with odd synonyms forced in, and over-penalised code breaks. Never use repetition penalties for code or structured output.
6. Beam search: look ahead before committing
All of the methods so far choose one token at a time, never reconsidering. Sometimes the best sequence starts with a token that is not the best first token. Beam search keeps several partial answers alive at once.
Follow the figure. "Sorry" (0.55) beats "We" (0.45) as a first word, so greedy decoding takes "Sorry", then its best follow-up "for" (0.40), for a sequence probability of 0.55 \times 0.40 = 0.22. Beam search with a beam width of 2 keeps the two best partial answers at every step. It extends both "Sorry" and "We", scores all the two-word sequences, and keeps the best two: "We apologise" at 0.45 \times 0.90 = 0.405, and "Sorry for" at 0.22. It finds the better sequence that greedy missed. In practice, sequence scores are sums of log-probabilities rather than products, for the underflow reason in 12.2.6: \ln 0.405 \approx -0.90 beats \ln 0.22 \approx -1.51.
The length penalty
Beam search has a built-in bias. Every extra token multiplies the sequence probability by a number below 1, so longer sequences always score lower, and beam search, left alone, prefers short answers and stops early. The fix is a length penalty: divide each sequence's log-probability by a function of its length before comparing. A widely used form, from Google's 2016 neural translation system, divides by \left(\frac{5 + L}{6}\right)^{\alpha}, where L is the length in tokens and \alpha (typically 0.6 to 1.0) sets the strength. With \alpha = 1, a 3-token answer's score is divided by 1.33 and a 10-token answer's by 2.5, which offsets the longer one's naturally more negative total.
You will often see a setting such as length_penalty=2.0 in Hugging Face's generate function. There, the sequence's total log-probability is divided by its length raised to that power, \text{score} = \frac{\sum \ln p}{L^{\,2.0}}. Because the total log-probability is negative, dividing by a larger number brings it closer to zero, so longer sequences score better. A value of 1.0 roughly cancels the short-answer bias; 2.0 actively favours long outputs, useful when beam search keeps producing summaries that are too terse; values below 1 favour shorter outputs; 0 turns length correction off.
When to use beam search
Beam search is the right tool when there is one best answer and the goal is to find the most probable output: machine translation, speech-to-text transcription, or text normalisation. It is the wrong tool for conversation and creative writing. Searching harder for the single most probable sequence produces exactly the bland, generic, repetitive text that Holtzman's paper described, because good writing is not the most probable writing. For that reason, chat LLM APIs generally do not offer beam search at all; it appears in translation and speech systems and in libraries like Hugging Face transformers.
7. Other controls you will use
Stop sequences end generation as soon as a chosen string appears, such as "\n\nCustomer:" in a transcript format. They are how you stop a model from continuing past the answer you wanted.
Maximum output length caps the number of generated tokens, which caps both cost and latency (from 12.7.6, output tokens dominate both).
Seeds fix the random number generator, so sampling is repeatable on the same system, which is useful for debugging.
Temperature 0 is not perfectly deterministic on hosted APIs. Requests are batched with other users' requests, and the order in which a GPU adds up floating-point numbers can differ slightly depending on the batch. Floating-point addition is not exactly associative, as Chapter 1.4 explained, so two runs can differ by tiny amounts in a logit, and occasionally that flips which token is on top. Do not build systems that depend on bit-for-bit identical outputs; if you need a fixed answer, cache it.
Constrained decoding goes one step further than any setting here: at each step, it sets to zero the probability of every token that would break a required format, such as a JSON schema, so invalid output becomes impossible rather than just unlikely. It is the subject of Chapter 12.8.3.
8. What to use for each task
| Task | Temperature | Other settings |
|---|---|---|
| Classification, routing, extraction | 0 | none |
| Code generation | 0 to 0.2 | no repetition penalties |
| Factual Q&A, support answers | 0 to 0.3 | top-p default |
| Summaries, rewriting | 0.3 to 0.7 | top-p 0.9 |
| Creative writing, apology notes, brainstorming | 0.7 to 1.0 | top-p 0.9 to 0.95, mild presence penalty |
| Translation, transcription (own model) | beam search | width 4 to 5, length penalty |
The Cartwise fixes followed this table: the router moved to temperature 0, and the apology writer moved to temperature 0.8 with a small presence penalty, plus three example notes in the prompt to show the tone wanted, which Chapter 12.8.2 explains. Some newer reasoning models fix their sampling settings internally: the current Claude models, for example, reject a temperature or top_p value with an error and instead offer an effort setting that controls how much the model thinks. Open models, and many other providers' APIs, still expose all of the controls on this page. Always check what a model's documentation says it supports.
Here is a complete sampler, so that nothing on this page is a black box:
python
import numpy as np
def sample(logits, temperature=1.0, top_p=1.0, rng=np.random.default_rng(0)):
logits = np.asarray(logits, float)
if temperature == 0:
return int(np.argmax(logits)) # (1) greedy
z = logits / temperature # (2) reshape
p = np.exp(z - z.max()); p /= p.sum() # (3) softmax
order = np.argsort(-p) # (4) most likely first
cum = np.cumsum(p[order])
keep = order[: np.searchsorted(cum, top_p) + 1] # (5) smallest set reaching top_p
q = p[keep] / p[keep].sum() # (6) rescale the survivors
return int(rng.choice(keep, p=q)) # (7) draw one
logits = [3.0, 2.3, 1.6, 1.2, -1.0] # delay, inconvenience, trouble, wait, banana
print(sample(logits, temperature=0)) # → 0 (always "delay")
print([sample(logits, 0.8, 0.9) for _ in range(8)]) # → a mix of 0, 1 and 2, never 3 or 4Line (1) handles temperature 0 as greedy decoding. Line (2) divides by the temperature, and line (3) is a safe softmax. Lines (4) and (5) sort the tokens from most to least likely and keep the smallest set whose running total reaches top_p; searchsorted finds where the running total first reaches the target. Line (6) rescales the kept probabilities to add up to 1. Line (7) draws one token at random in proportion to them. With top_p=0.9, only the first three tokens can ever be chosen, which is the top-p cut from section 4.
Recall
- After each decode step, a rule chooses one token from the distribution. Greedy decoding takes the top token: repeatable, right for single-answer tasks, flat and loop-prone for writing.
- Temperature divides logits before softmax: below 1 sharpens (0 is greedy), above 1 flattens and lifts nonsense too. Sampling then draws at random in proportion to the probabilities.
- Cut the tail before sampling: top-k keeps a fixed count; top-p (nucleus) keeps the smallest set reaching a probability total, adapting to confidence; min-p keeps tokens above a fraction of the top one. Tune temperature or top-p, not both.
- Repetition, frequency and presence penalties lower the logits of already-used tokens; small doses only, never for code or JSON.
- Beam search keeps several partial sequences and finds higher-probability outputs ("We apologise", 0.405, versus greedy's 0.22), with a length penalty against short-answer bias; right for translation and transcription, wrong for open-ended text. Temperature 0 is not bit-for-bit deterministic on hosted APIs.
Self-test: With top-p of 0.8 on the five-token table, which tokens survive? Why does a high temperature make "banana" more likely but top-p can still exclude it? Why would beam search without a length penalty favour "Sorry." over a full apology?
Next: 12.7.8 Smaller and Faster Models — decode is limited by reading weights, so the main levers for cheaper, faster models are fewer bytes per weight, fewer weights used per token, and cleverer ways to produce several tokens per step.