Cruxly

Cruxly › AI engineering › LLM Foundations

Sampling and Decoding: Temperature, Top-p, Beam Search, and Determinism

logits to a chosen token, and why temperature 0 still is not repeatable

Open the interactive lesson

Prompt: "Your extraction pipeline gives different JSON for the same invoice on two runs, with temperature set to 0. The PM asks you to 'just turn the creativity down'. Explain what temperature does to the numbers, why zero does not give repeatability, what the other knobs do, and what you would build instead."

40 minutes first. Take logits [2.0, 1.0, 0.1] through temperature and top-p by hand, then compute what five-vote self-consistency buys. The trap is describing temperature as 'creativity' and determinism as a setting.

Temperature rescales logits before the softmax

At each step the model outputs one real number (a logit) per vocabulary token; sampling is choosing a token from the distribution those numbers define. Softmax turns logits z_i into probabilities p_i = exp(z_i) / sum_j exp(z_j). Temperature T divides the logits first: p_i = softmax(z_i / T).

Work it with logits [2.0, 1.0, 0.1]:

T = 1.0 : [0.659, 0.242, 0.099]
T = 0.5 : [0.864, 0.117, 0.019]   (sharper: the leader goes from 66% to 86%)
T = 2.0 : [0.502, 0.304, 0.194]   (flatter)
T = 0.1 : [1.000, 0.000, 0.000]   (almost argmax)

As T -> 0 the distribution collapses onto the largest logit, which is greedy decoding; as T -> infinity it approaches uniform over the whole vocabulary. 'Temperature 0' is implemented as argmax, because dividing by zero is undefined. Temperature does not change the order of the tokens, only how decisively the top one wins, and it cannot make a wrong high-probability token right.

Why sample at all? Because greedy and near-greedy decoding on open-ended text degenerates: it loops and repeats (the 'neural text degeneration' finding of Holtzman et al., arXiv 1904.09751, whose point is that high-probability text is not what humans write). Pure sampling at T = 1 has the opposite problem: the vocabulary has tens of thousands of tail tokens that are each unlikely but collectively carry real mass, so every so often the sampler draws one and the sentence derails.

The price of a choice also depends on task. For extraction, classification and code you want the mode: low or zero temperature. For brainstorming you want spread. Reasoning models are a different case again, covered below.

:::key Temperature is a division of the logits by T before softmax. It reshapes probabilities; it does not add knowledge, creativity or accuracy. :::

Cutting the tail: top-k, top-p, min-p and penalties

Truncation samplers remove the unlikely tail before sampling so one bad draw cannot derail the output. Each renormalises what is left.

Worked top-p. At T = 1 the probabilities are [0.659, 0.242, 0.099], with cumulative sums 0.659, 0.901, 1.000. With p = 0.9 we keep tokens until the cumulative reaches 0.9: that takes the first two (0.901) and drops the third. Renormalise: 0.659 / 0.901 = 0.731 and 0.242 / 0.901 = 0.269, so the sampler now picks from [0.731, 0.269]. Min-p with min_p = 0.2 gives a threshold of 0.2 x 0.659 = 0.132, which also keeps the first two (0.242 > 0.132) and drops the 0.099 token. Temperature and top-p interact: at T = 0.5 the same p = 0.9 cumulative is 0.864, 0.981, so the second token is needed and the third is again dropped, but the distribution inside the kept set is sharper.

Repetition controls subtract from the logits of tokens that already appeared. A frequency penalty subtracts in proportion to the count, a presence penalty subtracts a flat amount once a token has appeared (this is how OpenAI's API reference describes its two penalties, checked October 2026, and open-source engines such as vLLM implement the same subtraction; other providers differ or omit them). With logits [2.0, 1.9, 0.5], a token already used 3 times and a frequency penalty of 0.5, the first logit becomes 2.0 - 1.5 = 0.5, and the probabilities shift from [0.470, 0.425, 0.105] to [0.165, 0.670, 0.165]. That is strong, so penalties are a blunt tool: they punish legitimate repetition (JSON keys, code identifiers, a name in a legal contract). For structured output, do not use them.

Order of operations matters: engines typically apply penalties, then temperature, then truncation (top-k, top-p, min-p), then sample. If a setting seems to do nothing, check what runs before it.

Beam search is for translation, not chat

Beam search keeps the b best partial sequences by total probability rather than committing to one token at a time; that is a better search for tasks with a single right answer and a worse fit for open-ended text. A toy shows why it can beat greedy. At step 1 the probabilities are A = 0.5, B = 0.4, C = 0.1. After A, the best next token has probability 0.4; after B, the best next token has 0.9.

greedy:      A (0.5), then best next 0.4        -> sequence probability 0.5 * 0.4 = 0.20
beam (b=2):  keeps A and B at step 1
             A -> 0.5*0.4 = 0.20     B -> 0.4*0.9 = 0.36
             picks B, which greedy never explored

Greedy committed to A because it was locally better; the beam kept B alive and found the higher-probability sequence. That matters for machine translation and speech recognition, where one output is correct and a higher-probability sequence is usually a better one. Chat does not fit: beam search favours the most generic, high-probability continuation (the 'safe, repeated, short' outputs of degeneration), it tends to be repetitive on open-ended text, it multiplies decoding cost by roughly the beam width in compute and cache memory, and a streaming product cannot show tokens until the beam resolves. Most hosted chat APIs do not expose it at all.

The same idea appears in a modern shape as best-of-n: sample n full answers and keep the one a scorer prefers, with the scorer being a reward model, unit tests for code, or an LLM judge (see LLM-as-a-Judge: Evaluation That Scales Without Lying to You). It searches over whole answers rather than tokens, and the same limit applies: the result is only as good as the scorer.

Sequence probability favours short outputs, because every extra token multiplies by a number below 1. Beam search implementations therefore add a length normalisation term. If you ever see a beam search producing very short answers, that missing normalisation is the first thing to check.

Temperature 0 is not repeatable, and what to build instead

Temperature 0 makes the sampler deterministic given the logits; it does not make the logits deterministic. Two runs can produce slightly different numbers, and when the top two tokens are within a hair of each other, a tiny difference flips the argmax. After one flipped token the rest of the text diverges, because every later token is conditioned on the new one.

Why do the logits differ? Floating-point addition is not associative. In float64, (1e16 + -1e16) + 1.0 = 1.0 but 1e16 + (-1e16 + 1.0) = 0.0. Summing 100,000 random numbers forward and in reverse differs by roughly 1e-12 on a total near 300. A GPU sum is a reduction whose order is a choice of the kernel. The popular explanation, that concurrent atomic adds race, is mostly wrong for LLM inference. Thinking Machines' September 2025 analysis, 'Defeating Nondeterminism in LLM Inference', argues the main cause is the lack of batch invariance: each kernel is deterministic for a given batch size, but the reduction strategy changes with batch size, and batch size changes with server load, so your request's numbers depend on whoever else was in the batch. In their experiment, 1,000 completions at temperature 0 of the same prompt on a 235B model gave 80 distinct outputs, first diverging at token 103; with batch-invariant kernels all 1,000 were identical, at a speed cost. That is one team's measurement on their setup; treat the numbers as an illustration, not a law.

Other sources stack on top: mixture-of-experts routing can flip an expert choice on a near-tie, different GPU types or library versions change kernels, and the provider can change the model behind a name. A seed parameter, where offered, fixes the random draw in the sampler; it does not freeze the logits, so providers describe it as best-effort.

Build repeatability from the outside:

@@ 1 | Constrain Use structured outputs / constrained decoding so the shape cannot vary (Cost Attribution and Runaway Prevention covers the mechanism); constraint removes a large class of variation but not choice of values.

@@ 2 | Pin Pin the dated model version; an alias that moves is a silent change.

@@ 3 | Validate and normalise Parse, validate against a schema, normalise formats (dates, currency) so equal meaning gives equal bytes.

@@ 4 | Cache by input For identical documents, store the first result keyed by a hash of input and prompt version, instead of hoping to regenerate it identically.

@@ 5 | Test with a tolerance Run each golden case several times and assert on fields, not on exact text.

What the API gives you today, logprobs, and sampling as an accuracy lever

Treat the sampling knobs as provider- and model-specific, and read the migration notes before you copy a config. Anthropic's Claude Sonnet 5.5 migration guide (checked October 2026) says that sampling parameters return an error on that model and tells you to remove temperature, top_p and top_k; control moves to the prompt, to thinking and effort settings, and to structured output. The Opus 5.5 and Haiku 5.5 migration guides give the same instruction, but older Claude models still accept these parameters, so check the model's own page. For reasoning models in general, providers often restrict or ignore sampling parameters; read the current docs for the exact model rather than assuming.

:::key Verified October 2026 Source: Anthropic's Sonnet 5.5, Opus 5.5 and Haiku 5.5 migration guides, retrieved October 2026. This matrix changes with every model release: before shipping, send a test request with each parameter to the exact model ID you deploy, and keep a startup check in CI that fails loudly if a 400 appears. :::

Logprobs, where exposed, turn a generation into a scored classification. Ask the model to answer with one label token and read the probabilities of the candidate labels. A token logprob of -0.105 is a probability of 0.90. Two caveats: token probabilities are not calibrated confidence (see Evaluation Beyond Accuracy), and several providers do not return logprobs at all. If you need a score, a small encoder classifier or a calibration step on labelled data is more honest.

Stop sequences and max_tokens are truncation controls, not quality controls. A stop sequence ends generation when a string appears; max_tokens is a hard ceiling, and hitting it returns a cut-off answer that you are billed for. Always check the stop reason, set the cap above the longest legitimate answer, and for JSON treat a length stop as a failed parse.

Self-consistency and best-of-n turn tokens into accuracy. Sample n answers at moderate temperature, take the majority. If each independent run is correct with probability 0.7 and the task has two possible answers, a majority of five is right with probability

P = sum over k=3..5 of C(5,k) * 0.7^k * 0.3^(5-k)
  = 10*0.343*0.09 + 5*0.2401*0.3 + 0.16807
  = 0.3087 + 0.3602 + 0.1681 = 0.8369   -> about 84 percent

n = 3 gives 78.4 percent, n = 9 gives 90.1 percent, for 3x, 5x and 9x the output tokens. Read the assumptions critically: runs are not independent (the same blind spot repeats), the gain vanishes if single-run accuracy is at or below 50 percent (five votes at 0.5 is still 0.5), and answers must be comparable (free text needs normalising). It pays when the cost of an error is much larger than five calls; it is not a default.

How to say it in the interview

"The model outputs a score per vocabulary token; softmax turns them into probabilities, temperature divides the scores first, so low temperature sharpens and high flattens. Top-p and min-p cut the tail before sampling. Temperature zero means take the argmax, but the scores themselves vary slightly between runs because floating-point sums depend on batch size and kernel choice, so near-ties flip. For repeatable extraction I constrain the output, pin the model version, validate, and treat identical output as something I test, not assume."

Checklist

Practice this interactively
Share this lessonWhatsAppLinkedInX

Related lessons