As of May 2026. The MemX team ran the same 103 classification test cases through Gemini 3 Flash (released December 2025) five times in March 2026. Temperature was 0. Seed was pinned. Top-p was 1. Every other available knob was held constant. 29 of the 103 cases produced different outcomes across the 5 runs. On Gemini 2.5 Flash-Lite (same test, same conditions), only 3 cases flapped. That is a 9.7x gap in flap-rate between two models from the same vendor on the same prompts. The number you read on a single eval run is not the model's accuracy; it is one sample from a distribution nobody told you existed.
If you have ever shipped an LLM eval that scored 96% accurate and watched it score 92% next week with nothing changed, this post explains why. Temperature zero never meant deterministic. It meant argmax sampling on a non-deterministic logit stream. The non-determinism sits upstream of every knob you control.
Quick takeaways (as of May 2026). Temperature 0 produces argmax sampling but does not make the underlying logits deterministic. Horace He and the Thinking Machines Lab team traced the dominant cause to variable batch sizes on shared inference servers (Sep 10, 2025): 1,000 identical Qwen3-235B-A22B-Instruct-2507 prompts at temp=0 produced 80 unique completions, with first divergence at token 103. OpenAI's seed parameter is mostly deterministic across matching system_fingerprint. Gemini's seed is best-effort. Anthropic still ships no seed parameter as of May 2026. The mitigation ladder runs from structured outputs (cheapest) to deterministic-inference backends (about 34% throughput cost on SGLang).
What temperature 0 actually does (and does not do)
Temperature scales the logits before the softmax. At temperature 1, logits pass through softmax unchanged. At temperature 0.7, they sharpen (smaller temperature widens the gap between top tokens). At temperature 0, the calculation degenerates to argmax: pick the token with the highest logit and ignore the probability distribution entirely.
All of that assumes the logits themselves are deterministic. They are not. Logit values depend on every step of the matmul and attention computation upstream of the softmax, and those run on GPUs with floating-point arithmetic that is not associative across reduction orders. Different reduction order yields different bit-patterns, which yield different argmaxes. Temperature 0 picks the maximum of a stream that itself was never deterministic.
Horace He's batch-invariance result
On September 10, 2025, Horace He and the Thinking Machines Lab team published the cleanest available explanation. The dominant source of LLM non-determinism is not GPU randomness; it is variable batch sizes on shared inference servers. When you hit a hosted endpoint, your request gets batched with other concurrent requests. Different batch composition means different matmul shapes, which means different reduction order, which means bit-different outputs. The fix they propose is batch-invariant kernels (mean, log-softmax, matmul) that produce the same bits regardless of batch size.
Their headline result: 1,000 identical Qwen3-235B-A22B-Instruct-2507 prompts at temperature 0 produced 80 unique completions, with the modal output occurring 78 times. Outputs stayed identical through token 102; the first divergence hit at token 103, splitting on whether Feynman was born in Queens, New York (992 completions) or New York City (8 completions). If the model is generating a 500-token response and the first split arrives at token 103, the rest of the output is functionally unpredictable.
Mixture of Experts makes it worse
Gemini 2.5 and 3 are widely reported as sparse Mixture-of-Experts transformers (Flash inherits the same MoE core). MoE routing introduces a second non-determinism source: which expert a token gets routed to depends on the other tokens in the batch, because expert capacity is capped. Same prompt, different batch co-tenants, different expert routes, different outputs. The Hayes, Shumailov, and Yona Buffer Overflow paper (arXiv:2402.05526, Feb 2024) showed this routing dependency is exploitable as a security vulnerability. It is also the reason your eval drifts when traffic patterns shift.
Thinking mode is a variance amplifier
Gemini 3.x ships with dynamic thinking enabled by default. The model generates thousands of thinking tokens before the answer token. Every thinking step is a fresh chance for the stochastic chain to diverge. A non-thinking model has few divergence points; a thinking model has thousands. Stack MoE routing on top of dynamic thinking and you get a double amplifier: batch-dependent routing decisions multiplied by thousands of intermediate sampling steps.
That is the engineering explanation for why the MemX Flash-Lite (non-thinking) beat Gemini 3 Flash (thinking-by-default) on consistency despite scoring lower on raw reasoning benchmarks. Classification is a single-token answer. You do not need 8,192 stochastic thinking tokens to get there. You need to skip the variance.
What the labs actually promise
| Vendor | Determinism control | Reality |
|---|---|---|
| OpenAI | seed + system_fingerprint (Nov 2023) | Mostly deterministic per OpenAI Cookbook when seed, parameters, and fingerprint all match. Fingerprint changes a few times a year as infrastructure updates, breaking reproducibility across the change. |
| Google Gemini | seed parameter (best-effort) | Field reports show gemini-2.5-pro returning different outputs at fixed seed plus temperature. Officially best-effort, not guaranteed. |
| Anthropic | No seed parameter (still true May 2026) | Docs describe temperature 0 as near-deterministic. Opus 4.7 onward rejects non-default temperature, top_p, and top_k with a 400. Variance ships with the architecture. |
Empirical evidence: how bad is it
Atil et al. (arXiv:2408.04667, latest v5 April 2025) tested 5 LLMs across 8 tasks, 10 runs each, all configured to be deterministic. Accuracy varied up to 15 percentage points across naturally occurring runs; the best-to-worst gap reached 70 percentage points. The companion small-LLM study (arXiv:2509.09705, Pinhanez, Cavalin, Sanctos and collaborators, Sep 2025) found 2B to 8B models answer MMLU-Redux questions consistently only 50 to 80% of the time at low inference temperatures, even when the model self-reports as SURE.
Translated to production: if you measure your LLM's accuracy with a single eval run, your number is between 8 and 35 percentage points off the truth. That is the size of a model generation. Single-run evals are not measurement. They are weather.
How to test for non-determinism in your pipeline
A short Python recipe.
- Pick N. 5 runs is the floor; 10 is honest; 20 is publishable.
- Pin everything you can pin: model, temperature=0, top_p=1, seed=42, response_schema (structured output).
- Run the same prompt N times. Canonicalise each parsed output (sort keys, lowercase, strip whitespace). Hash.
- Define flapping up front: if unique_outputs is greater than 1 across N runs, the case flaps.
- Report the distribution, not the winner. Counter(fingerprints).most_common() shows the modal output and how dominant it is. A case where the mode is 4 of 5 runs is more trustworthy than a case where the mode is 2 of 5.
- Build the eval table the way HELM does: every cell is mean and stdev over N runs, never a single number.
The mitigation ladder
Five levels, cheapest to most invasive.
- Structured outputs / JSON schema. Kills surface-form variance (whitespace, quote style, field order). Does not kill answer variance. OpenAI Structured Outputs hits 100% schema compliance with strict mode; the underlying answer can still differ.
- Multi-run majority vote. Wang et al.'s self-consistency paper showed +17.9% on GSM8K, +11.0% on SVAMP, +12.2% on AQuA, +6.4% on StrategyQA, +3.9% on ARC-challenge. Run N=3 to 5, take the modal output. Adds latency and cost.
- Lower or disable thinking budget. If the task is classification, set thinking_level to minimal or low on Gemini 3. Fewer stochastic thinking tokens, fewer divergence points.
- Pin a deterministic inference backend. SGLang 0.5.3 shipped batch-invariant kernels on Sep 22, 2025: 100% bitwise-identical outputs at about 34.35% average throughput overhead (FlashInfer, FlashAttention 3, and Triton backends), enabled via --enable-deterministic-inference. Horace He's vLLM PR #24583 demonstrated the same approach on Qwen 3-8B; it closed Nov 16, 2025 as a proof-of-concept, with the reference implementation living at thinking-machines-lab/batch_invariant_ops.
- Design downstream to tolerate variance. Idempotent writes, confidence thresholds, human-in-the-loop on disagreement, append-only logs you can replay. The production move is to accept non-determinism and architect around it.
Temperature 0 was never a determinism switch. It was an argmax sampler on a non-deterministic logit stream. The variance is upstream of every knob you control. Once you internalise that, you stop trying to make the LLM deterministic and start engineering a variance budget for the pipeline.
Key takeaway: every single-run LLM eval is weather, not measurement. The single point estimate is missing the most important piece of information, which is how much that number moves on a re-run. Multi-run with flap-rate reporting is the floor for credible production eval. Determinism is a property you engineer, not a checkbox you tick.
01Why do LLMs give different outputs at temperature 0?
Temperature 0 means argmax sampling, but the underlying logits are not deterministic. Variable batch sizes on shared inference servers change reduction order in matmul and attention kernels; floating-point addition is not associative. Different batch composition produces different bit-patterns, which produce different argmaxes. Horace He's September 10, 2025 Thinking Machines Lab study traced this in detail and showed 80 unique completions across 1,000 identical Qwen3-235B-A22B-Instruct-2507 prompts at temperature 0.
02Does OpenAI's seed parameter make outputs deterministic?
Mostly, not fully. OpenAI's Cookbook says outputs will mostly be identical when seed, parameters, and system_fingerprint all match. The fingerprint changes a few times a year as infrastructure updates, breaking reproducibility across the change. For 100% reproducibility you need a deterministic-inference backend (SGLang 0.5.3 batch-invariant mode or the vLLM follow-on work), not a seed parameter on a hosted endpoint.
03Does Anthropic Claude support a seed parameter in 2026?
No. As of May 2026 the Claude API still ships no seed parameter. Anthropic docs describe temperature 0 as near-deterministic and acknowledge minor GPU-level variance from model parallelism. Claude Opus 4.7 onward goes further and rejects non-default temperature, top_p, and top_k with a 400, so the only path to reproducibility on Claude is to cache responses or pin a deterministic backend.
04Why is Mixture-of-Experts more non-deterministic than a dense model?
MoE routing depends on the other tokens in the batch. Capacity-limited expert assignment means the same token can route to different experts depending on which other tokens are in flight. Different routing produces different outputs. Gemini 2.5 and 3 are widely reported as sparse MoE; in the MemX test, Gemini 3 Flash flapped 29 cases out of 103 versus 3 on the smaller Flash-Lite, a 9.7x gap.
05How many times should I run each test case in an LLM eval?
5 runs is the floor for credible reporting; 10 is honest; 20 is publishable. Single-run evals can be off by up to 15 percentage points per Atil et al. (arXiv:2408.04667), and the best-to-worst gap reached 70 points across their 5 LLMs and 8 tasks. Report mean accuracy with the high-low range, and flag any case where the runs disagree as flapping for separate review.
06Should I use thinking mode for classification tasks?
No. Classification is a single-token answer. Thinking mode generates thousands of stochastic intermediate tokens before the answer, each one a chance to diverge. Set thinking_level to minimal or low on Gemini 3; pick a non-thinking model (Flash-Lite, Haiku) for classification entirely. The MemX data: Flash-Lite (non-thinking) had 9.7x fewer flapping cases than Gemini 3 Flash (thinking-by-default).
07Can self-consistency or majority voting fix non-determinism?
Partially. Wang et al.'s self-consistency paper showed +17.9% on GSM8K, +11.0% on SVAMP, +12.2% on AQuA by sampling N completions and taking the majority. It mitigates the variance, it does not eliminate it; and it costs N times the inference. For high-stakes single-answer tasks, combine self-consistency with structured outputs and (if affordable) a batch-invariant backend like SGLang 0.5.3 or its vLLM equivalent.
08What is a flap-rate and how do I report it?
Flap-rate is the percentage of test cases that produce more than one unique output across N identical runs at temperature 0. On the MemX 103-case classification eval, Gemini 3 Flash had a flap-rate of 28.2% (29 of 103) versus 2.9% (3 of 103) for Gemini 2.5 Flash-Lite. Report it alongside accuracy: a model with 95% accuracy and 5% flap-rate is far more shippable than one with 96% accuracy and 30% flap-rate.
