Prompting & Reasoning

Temperature and Sampling

Temperature and sampling are the decoding controls that turn a language model's raw output scores into an actual token choice. Temperature scales the softmax to make the distribution sharper or flatter, while sampling methods like top-k and top-p decide which candidate tokens are eligible to be drawn.

From Logits to a Token

A language model does not directly emit words. At each generation step it produces a vector of raw, unbounded scores called logits, one per token in its vocabulary. These logits are converted into a probability distribution by the softmax function, and a decoding strategy then selects the next token from that distribution. Temperature and sampling are the knobs that govern this final selection step, and they are applied at inference time without changing a single model weight.

Because the same model can be run with many different decoding settings, the choice of temperature and sampling method has a large effect on the character of the output. A model asked to write code will usually be configured to behave almost deterministically, while the same model asked to brainstorm story ideas will be configured to take more risks. The underlying probabilities are identical; only the rule for drawing from them differs.

This selection step is also a primary reason why language model outputs are nondeterministic. When sampling is enabled, two runs of the same prompt can produce different text because the model is drawing from a distribution rather than always picking the single most likely token. Setting temperature to zero or using greedy decoding removes the randomness from this step, though other sources of variation, such as hardware-level floating-point differences and batching, can still cause small divergences.

Logits are the raw per-token scores the model outputs before any normalization.
Softmax converts logits into a probability distribution that sums to one.
Decoding strategy is the rule that picks the next token from that distribution.
Temperature and sampling operate at inference and never alter model weights.

Temperature and the Softmax

Temperature, written as T, is a single scalar that divides each logit before the softmax is applied. The token probability becomes p_i = exp(z_i / T) divided by the sum over all tokens of exp(z_j / T). Dividing by T changes how peaked the resulting distribution is without changing the ranking of the tokens.

When T is less than 1, the gaps between logits are magnified, so the highest-scoring token gets an even larger share of the probability mass and the distribution becomes sharper and more deterministic. When T is greater than 1, the gaps shrink, the distribution flattens, and lower-ranked tokens become more competitive, producing more varied and surprising text. As T approaches 0, the distribution collapses onto the single highest-scoring token, which is equivalent to greedy decoding, also called argmax decoding.

Common values fall roughly between 0 and 2. Many APIs expose temperature as the most visible decoding control, and in practice a value near 0 is used for factual answers and code, while values near 0.7 to 1.0 are typical for open-ended or creative generation. Temperature only reshapes the distribution; it does not by itself remove any tokens from consideration, which is why it is usually combined with a truncation method like top-p.

p_i = exp(z_i / T) / Σ_j exp(z_j / T)

Temperature-scaled softmax. z_i are the logits, T is temperature. T below 1 sharpens the distribution, T above 1 flattens it, and T approaching 0 yields greedy argmax selection.

T below 1 sharpens the distribution toward the top token and increases determinism.
T above 1 flattens the distribution and raises the chance of rarer tokens.
T approaching 0 is equivalent to greedy decoding, always picking the argmax.
Temperature rescales but does not truncate, so it pairs with top-k or top-p.

Why Greedy Decoding Fails

It is tempting to assume that always choosing the most probable token, or searching for the most probable whole sequence with beam search, would produce the best text. Greedy decoding selects the single argmax token at each step. Beam search keeps a fixed number of partial sequences, called beams, and expands the ones with the highest cumulative probability, approximating a search for the overall maximum-likelihood sequence.

Both of these likelihood-maximizing strategies tend to produce dull, generic, and repetitive output for open-ended generation. Holtzman et al. documented this in their 2019 paper "The Curious Case of Neural Text Degeneration", showing that maximization-based decoding leads models into repetitive loops and bland phrasing that diverges sharply from the statistics of human writing. The paper named this failure neural text degeneration.

The counterintuitive lesson is that some randomness improves quality for creative tasks. Human text does not always use the single most predictable next word, so a decoder that occasionally samples a less likely token better matches natural language. Sampling methods aim to inject this controlled randomness while still excluding the long tail of genuinely implausible tokens that would make the text incoherent. Beam search remains useful for short, constrained tasks such as machine translation and summarization, where there often is a single best answer.

Greedy decoding picks the argmax token at every step.
Beam search approximates the maximum-likelihood full sequence over several beams.
Maximization-based decoding causes repetition and bland text, the degeneration finding.
Beam search still suits constrained tasks like translation with one clear best answer.

Truncation Sampling: Top-k, Top-p, and Min-p

Truncation sampling keeps the benefits of randomness while cutting off the unreliable tail of the distribution. Top-k sampling, used by Fan et al. in their 2018 paper "Hierarchical Neural Story Generation", restricts the candidate set to the k highest-probability tokens, renormalizes their probabilities, and samples from that set. Its weakness is that k is fixed: when the model is very confident the top few tokens already hold almost all the mass, but when the model is uncertain a fixed k may exclude valid options or include weak ones.

Top-p sampling, also called nucleus sampling, was introduced by Holtzman et al. in 2019. Instead of a fixed count, it selects the smallest set of top tokens whose cumulative probability mass is at least p, then renormalizes and samples from that nucleus. Because the set size adapts to the shape of the distribution, the nucleus shrinks when the model is confident and grows when it is uncertain, which is the property that makes nucleus sampling more reliable than fixed top-k across varied contexts.

Min-p sampling, introduced by Nguyen et al. in their 2024 paper "Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputs", sets a relative floor: it keeps tokens whose probability is at least a fraction of the top token's probability, using the maximum probability as a scaling factor. This dynamic threshold is designed to preserve coherence even at high temperature. These methods are frequently combined, with temperature applied first to reshape the distribution and a truncation method applied second to bound which tokens can be drawn.

python

import numpy as np

def sample(logits, temperature=0.8, top_p=0.9):
    # 1. Temperature scaling, then softmax over the vocabulary.
    logits = logits / temperature
    probs = np.exp(logits - logits.max())
    probs /= probs.sum()

    # 2. Nucleus (top-p): keep the smallest set whose mass >= top_p.
    order = np.argsort(probs)[::-1]
    cumulative = np.cumsum(probs[order])
    keep = order[: np.searchsorted(cumulative, top_p) + 1]

    # 3. Renormalize over the nucleus and draw one token.
    kept = probs[keep] / probs[keep].sum()
    return int(np.random.choice(keep, p=kept))

Minimal NumPy temperature plus top-p (nucleus) sampling from a logits vector. Temperature reshapes the distribution, then the nucleus is truncated and renormalized before a single token is drawn.

Top-k (Fan et al. 2018) samples from the k highest-probability tokens after renormalizing.
Top-p / nucleus (Holtzman et al. 2019) samples from the smallest set reaching cumulative mass p.
Min-p (Nguyen et al. 2024) keeps tokens above a fraction of the top token's probability.
Nucleus sampling adapts its candidate-set size to the model's per-step confidence.

Repetition Controls and Practical Tuning

Even with temperature and truncation, models can still loop on the same phrases. Many decoding APIs add penalty terms for this. A frequency penalty reduces a token's logit in proportion to how many times it has already appeared, while a presence penalty applies a flat reduction once a token has appeared at all. A related repetition penalty divides or scales the logits of previously generated tokens to discourage exact repeats. These penalties are applied to the logits before the softmax, alongside temperature.

The interaction between temperature and top-p matters in practice. Raising both at once compounds randomness and can push output into incoherence, so providers commonly recommend adjusting one at a time. A typical default pairs a moderate temperature near 0.7 with a top-p near 0.9 or 1.0. For tasks demanding precision, such as code generation, structured data extraction, or factual question answering, a temperature at or near 0 is standard, since reproducibility and correctness outweigh variety.

For creative writing, brainstorming, or dialogue, higher temperature and a slightly looser nucleus produce more diverse and interesting completions. There is no single universally optimal setting; the right values depend on the model, the task, and how much the application can tolerate variation between runs. Because these settings strongly influence reproducibility, they should be logged whenever consistent or auditable outputs are required.

Frequency penalty scales a token's logit by how often it has already appeared.
Presence penalty applies a flat reduction once a token has appeared at all.
Providers advise tuning temperature or top-p one at a time, not both together.
Low temperature suits code and facts; higher temperature suits creative generation.

Sampling and Nondeterminism

Sampling is the most direct cause of variation in language model outputs. With any temperature above zero and sampling enabled, the model draws from a distribution, so identical prompts can yield different completions across runs. Fixing the random seed makes a sampled run reproducible on the same system, and setting temperature to zero or using greedy decoding removes this source of variance entirely.

Removing sampling does not always guarantee bit-identical results. Floating-point arithmetic is not perfectly associative, so the order in which operations are summed on a GPU can change a logit in its least significant bits. When two tokens have nearly equal logits, such a tiny difference can flip the argmax and send the generation down a different path. Server-side batching, where a request is processed alongside others, can also change these summation orders between runs.

For applications that need consistent behavior, the practical guidance is to set temperature to zero, pin a seed where the provider supports it, and validate outputs rather than assuming determinism. Even then, exact reproducibility across different hardware, library versions, or batch compositions is not guaranteed, which is why output validation and testing remain important for any system that depends on stable responses.

Sampling above temperature zero is the leading source of output nondeterminism.
A fixed seed makes a sampled run reproducible on the same machine and stack.
Greedy or temperature-zero decoding removes sampling randomness but not all variance.
Floating-point summation order and batching can still flip near-tied token choices.

Key takeaways

Temperature divides logits before softmax: below 1 sharpens output, above 1 flattens it, and approaching 0 is greedy argmax decoding.
The temperature softmax is p_i = exp(z_i / T) / Σ_j exp(z_j / T), reshaping but not truncating the distribution.
Pure likelihood maximization via greedy decoding or beam search causes the repetitive, bland neural text degeneration documented by Holtzman et al.
Top-k (Fan et al. 2018) uses a fixed candidate count, while top-p / nucleus (Holtzman et al. 2019) adapts the candidate set to the model's confidence.
Min-p (Nguyen et al. 2024) thresholds tokens against the top token's probability to stay coherent at higher temperatures.
Sampling is a major source of LLM nondeterminism; temperature zero, greedy decoding, and fixed seeds reduce but do not always eliminate it.

Frequently asked questions

Temperature is a scalar that divides every logit before the softmax, controlling how peaked the resulting probability distribution is. A temperature below 1 sharpens the distribution toward the most likely token, making output more deterministic, while a temperature above 1 flattens it and raises the chance of rarer tokens. As temperature approaches 0, generation becomes equivalent to greedy decoding, always selecting the single highest-scoring token.

Top-k sampling keeps a fixed number, k, of the highest-probability tokens and samples among them after renormalizing. Top-p, also called nucleus sampling, instead keeps the smallest set of top tokens whose cumulative probability reaches a threshold p, so the candidate set grows when the model is uncertain and shrinks when it is confident. Top-p was introduced by Holtzman et al. in 2019 partly to fix the fixed-size limitation of top-k.

Greedy decoding always picks the single most likely next token, and maximizing likelihood this way tends to produce repetitive, generic text that loops on common phrases. Holtzman et al. called this neural text degeneration and showed it diverges from the statistics of human writing. Because people do not always choose the most predictable word, adding controlled randomness through sampling generally yields more natural and varied output for open-ended tasks.

For factual question answering, code generation, and structured extraction, a temperature at or near 0 is standard because it favors the most likely, reproducible output. For creative writing, brainstorming, or dialogue, values near 0.7 to 1.0 produce more diverse and interesting completions. The exact best value depends on the model and task, and providers usually recommend adjusting temperature or top-p one at a time rather than both together.

The main reason is sampling: with temperature above zero, the model draws the next token from a probability distribution, so identical prompts can produce different completions. Setting temperature to zero or using greedy decoding, and fixing the random seed where supported, removes most of this variation. Even then, floating-point summation order on GPUs and server-side batching can occasionally flip near-tied token choices, so exact reproducibility is not guaranteed across all conditions.

Min-p sampling, introduced by Nguyen et al. in their 2024 paper "Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputs", is a dynamic truncation method that keeps tokens whose probability is at least a fixed fraction of the top token's probability. It uses the maximum token probability as a scaling factor, so the effective threshold adapts to the model's confidence. The method is designed to maintain coherent output even at high temperature settings.

Put the idea into practice

MemX is an AI memory agent built on these ideas: store anything, skip the folders, and find it again by asking in plain English.

Try MemX Free