Where the Variation Comes From When Temperature Is Zero

Many people believe that an AI chatbot has a setting that makes it give the same answer every time it receives the same question. They also believe that when the answers still differ, the cause is random noise inside the computer hardware. This post tests that belief against a published experiment and against a few small programs I wrote myself.

The belief matters to anyone who depends on repeatable answers, such as a team testing a product, keeping an audit record, or comparing two versions of an instruction. The setting narrows the range of answers the model is willing to give. It leaves chance in the process, and a second source of variation sits in how a provider's servers handle many people's requests at the same moment. By the end I want to leave the reader understanding that second source, why it turns identical questions into different sentences, and why it is a design property that engineers can change.

Thinking Machines Lab, an AI research company, reported the experiment that anchors this post in September 2025 (Horace He, "Defeating Nondeterminism in LLM Inference"). They asked a language model called Qwen3-235B-A22B-Instruct-2507 to "Tell me about Richard Feynman," and they asked it one thousand times. Temperature, the dial that controls how much chance the model uses when it picks each next piece of text, was set to zero. Each completion ran to one thousand tokens, and a token is a word or word fragment, the unit a model reads and writes. In theory the experiment should have produced one answer a thousand times. They reported 80 distinct completions, and the most common of them appeared only 78 times. All thousand completions were identical for the first 102 tokens. At token 103 they split: every completion had written "Feynman was born on May 11, 1918, in," and then 992 of them continued with "Queens, New York" while 8 continued with "New York City."

The common belief says temperature zero turns a language model into a deterministic function, which means the same input always yields the same output, and that anything left over comes from the chaos of parallel GPUs, the graphics chips that run thousands of calculations at once. In my reading of the Thinking Machines analysis, the variation comes from a property of the kernels called batch invariance, and it has nothing to do with the sampler. A kernel is a small program that performs one numerical operation on the chip, and a batch is a group of requests processed together. Paragraphs that begin with "Technical detail:" are optional depth, and the plain explanation continues without them.

What temperature controls

Picture a procurement committee that ranks suppliers by score. At temperature zero the committee always awards the contract to the top-ranked supplier. At higher temperatures it runs a weighted lottery, so lower-ranked suppliers win now and then. Temperature plays the same role for a language model choosing its next token, and the analogy is my own.

Technical detail: the sampler is, in my reading, the only component of a language model system that was designed to be random, so it makes sense to pin down what it does. The forward pass, which is one complete run of the input through the model as described in Part 2, produces a logit for every entry in the vocabulary. A logit is the raw score the model assigns to a candidate token, and the vocabulary is the model's full list of tokens. Sampling converts logits into probabilities with a softmax, a formula that turns scores into probabilities that sum to one, after dividing each logit by the temperature T. It then draws a token from the resulting distribution.

The snippet below is my own arithmetic on five invented logits, chosen to give a leading token and a close runner-up. It shows how the temperature dial reshapes the odds. No real model is involved.

logits = np.array([4.0, 3.6, 2.0, 0.5, -1.0])
def sm(x, T):
    z = x / T; z = z - z.max(); e = np.exp(z); return e / e.sum()
for T in [0.2, 1.0, 2.0]:
    print(T, np.round(sm(logits, T), 4))
0.2 [0.8808 0.1192 0.     0.     0.    ]
1.0 [0.5427 0.3638 0.0734 0.0164 0.0037]
2.0 [0.4094 0.3352 0.1506 0.0711 0.0336]

Lower temperature sharpens the distribution toward the leading token. At T = 0.2 the top token holds 88% of the probability, and as T approaches zero the distribution collapses onto the argmax, the single highest-scoring choice. The takeaway from my toy is that the dial controls only how the draw is made.

What temperature narrows

Temperature narrows the range of outputs the model will accept as plausible, and the randomness stays in the system. I picture that range as a cone. At a high temperature the cone is wide, and many different continuations carry real probability. Lowering the temperature squeezes the cone toward its tip, which is the single leading token at each step. A narrower cone gives less spread between runs, and some spread always remains. The cone image is my own framing of the arithmetic above.

The table shows the narrowing. At T = 0.2 the runner-up token still holds 12% of the probability, roughly a one-in-eight chance of winning any given draw, and across hundreds of tokens those chances add up. At T = 1.0 the top token holds 54% and the runner-up 36%. The dial changes how much chance shows up in the output. The chance itself comes from the model's uncertainty about what comes next, and no dial removes uncertainty from a prediction.

Technical detail: zero itself cannot be plugged into the formula, so as I understand it implementations typically treat T = 0 as a special case that skips the random draw and picks the highest logit directly. That is greedy decoding. It removes the explicit lottery. It leaves a decision between close candidates, and that decision gets made by arithmetic whose last digits depend on how the server grouped the request. Two candidates whose logits differ by a rounding error are effectively tied, and the tie breaks according to numerical noise.

From the caller's seat, a tie broken by numerical noise behaves like a draw that nobody seeded and nobody can replay. The Feynman experiment shows it. Temperature was zero, the sampler's own lottery was switched off, and 80 different completions still came out. In my reading the variation entered before the sampler saw the logits, because the logits themselves differed from run to run. Temperature zero shrinks the cone to a point in theory, and the edges of that point stay fuzzy in practice.

Four statements about determinism

A supplier can be perfectly dependable and still look unpredictable to a customer who never sees the other orders on the loading dock. Thinking Machines opens its analysis with four claims that sound incompatible, and they report that all four are true. Some GPU kernels are nondeterministic. Every kernel used in a language model's forward pass is deterministic. The forward pass of an inference server can be called deterministic. And from the perspective of anybody using the server, the results are nondeterministic. The rest of their analysis reconciles them.

My own toy transformer from Part 2 supports the second and third claims in miniature. I ran it twice on the same six tokens and the logits matched bit for bit. A pure function of its input behaves that way, because the same input always produces the same output. Their third statement says the same holds for an inference server's forward pass.

The explanation most people reach for combines two facts. Computers store decimals with a limited number of digits, a format called floating point, or float, and with limited digits the order of addition can change the answer. Mathematicians describe ordinary addition as associative, meaning grouping never matters, and floating-point addition lacks that property. GPUs also run many operations concurrently, so the order in which partial results arrive may change from run to run. Thinking Machines calls this the "concurrency plus floating point" hypothesis, and they quote several sources that state it, including a recent arXiv preprint.

An everyday version of the first fact is a warehouse scale that rounds every running total to the nearest ten kilograms. Loading the same crates in a different order can leave a different final total, because each rounding happens at a different moment. The analogy is mine.

The first half of the explanation holds up, and two lines of Python show it. These lines add the same three numbers in two different orders:

(0.1 + 1e20) - 1e20      # 0.0
0.1 + (1e20 - 1e20)      # 0.1

The same three numbers give 0.0 in one case and 0.1 in the other, because adding 0.1 to a number of magnitude 10²⁰ rounds the 0.1 away. The takeaway is that order can change a computed result.

Technical detail: the Thinking Machines write-up uses a base-10 version with three digits of precision. 1230 plus 23.4 has an exact sum of 1253.4, which a three-digit format stores as 1250, so the 23.4 has effectively been rounded to 20 before the addition happens. In my reading, floating point earns its usefulness by trading exactness for dynamic range, and the price is that order matters.

Technical detail: they also shuffle an array that sums to zero mathematically, [1e-10, 1e-5, 1e-2, 1, -1e-10, -1e-5, -1e-2, -1], ten thousand times, and report 102 distinct results. I ran their script on Python 3.10 and got 258 distinct results, spread between −2.2e-16 and +2.2e-16. The count differs from theirs, and I suspect the interpreter's implementation of sum plays a part, though I have not isolated the cause. Both runs make the same point: eight numbers, summed in different orders, yield a surprising spread of answers.

The second half of the explanation is where it breaks. If concurrency alone made GPU arithmetic unpredictable, repeating the same matrix multiplication on a GPU would produce disagreeing results. Matrix multiplication is the core operation of a language model, multiplying one grid of numbers by another. Thinking Machines tested exactly that, and the snippet below is their test. It multiplies the same two grids a thousand times and asserts that every result matches the first one exactly.

A = torch.randn(2048, 2048, device='cuda', dtype=torch.bfloat16)
B = torch.randn(2048, 2048, device='cuda', dtype=torch.bfloat16)
ref = torch.mm(A, B)
for _ in range(1000):
    assert (torch.mm(A, B) - ref).abs().max().item() == 0

They reported that the assertion held for a thousand repetitions. In their test a concurrent GPU working in floating point returned a bitwise identical result every time, and I take that as the end of the concurrency explanation by itself.

Technical detail: Thinking Machines explains that the concurrency hypothesis depends on atomic adds, where many cores accumulate into the same memory location in whatever order they finish. They note that atomic adds do make a kernel run-to-run nondeterministic.

Technical detail: they also report that atomic adds are almost never needed in a language model's forward pass. There is usually enough parallelism along the batch dimension to give each core its own reduction, which is the combining of many numbers into one sum. When a single reduction must be split across cores, they describe the standard approach as combining the partial sums in a fixed order. By their count, a typical LLM forward pass contains no atomic adds at all. They name one common exception, the backward pass of FlashAttention, which matters for training and plays no role in inference.

Batch invariance

Consider a freight consolidator that bills each parcel by adding handling charges. The bill for one parcel should depend only on that parcel. Suppose the consolidator groups charges one way on a quiet day and another way on a busy day, and rounds at different stages in each. The same parcel then receives a slightly different bill depending on how many other parcels shared the truck. The analogy is mine, and the property it illustrates is what Thinking Machines calls batch invariance.

Batch invariance means the result for one request depends only on that request. Thinking Machines reports that the kernels in common use fail that test. The mathematics of a matrix multiplication treats each row independently. The implementation chooses how to divide the reduction among cores based on the shape of the whole batch, and different divisions add the same numbers in different orders.

Thinking Machines demonstrates this on a GPU with a matrix-vector product versus a matrix-matrix product, and they report a maximum discrepancy of 1669.25 on their inputs. I wanted to see the mechanism in a setting where I control every step, so I wrote a CPU version. It is my own small imitation of the behavior they describe, and it models no real vendor kernel. It reduces a 4,096-element dot product, which multiplies two lists of numbers and adds the products, in float32, a format that keeps about seven decimal digits. The branch rule is my invention. Small batches get a split-K strategy, with four partial sums combined at the end, to keep cores busy. Larger batches use one sequential accumulation.

The code below is that toy. It computes the same sums with two different groupings, and it picks the grouping from the batch size.

import numpy as np
rng = np.random.default_rng(0)
K = 4096

def dot_rows(X, W, strategy):
    B = X.shape[0]; out = np.zeros(B, dtype=np.float32); P = X * W
    for i in range(B):
        row = P[i]
        if strategy == "seq":                 # one core, chunks of 256 in order
            acc = np.float32(0)
            for s in range(0, K, 256):
                acc = np.float32(acc + row[s:s+256].sum(dtype=np.float32))
        else:                                 # split-K: four partial sums
            parts = []
            for q in range(4):
                acc = np.float32(0)
                for s in range(q*K//4, (q+1)*K//4, 256):
                    acc = np.float32(acc + row[s:s+256].sum(dtype=np.float32))
                parts.append(acc)
            acc = np.float32(((parts[0] + parts[1]) + parts[2]) + parts[3])
        out[i] = acc
    return out

def kernel(X, W):     return dot_rows(X, W, "split" if X.shape[0] < 8 else "seq")
def invariant(X, W):  return dot_rows(X, W, "seq")

With 64 random rows, the results in my toy were these:

run-to-run unique results at batch size 1, over 200 runs: 1
rows that differ between batch size 1 and batch size 64:  35 of 64
rows that differ when the kernel always uses "seq":        0 of 64

The toy kernel is perfectly repeatable, with one unique answer across 200 runs. It also gives a different answer for 35 of the 64 rows depending on whether a row arrives alone or inside a batch of 64. Pinning the strategy to one fixed order removes every discrepancy. The takeaway is that a function can repeat itself exactly and still change its answer with the company a request keeps. I used no atomics and no race conditions. One deterministic function with a batch-size-dependent branch produced the effect.

The serving side supplies the second ingredient, as Thinking Machines describes it. An inference server batches whatever requests are in flight, so the batch size any particular request lands in depends on traffic at that moment, which the caller can neither see nor control. In my words, a kernel whose output depends on batch size, combined with a batch size that depends on load, makes the system nondeterministic from the caller's seat, even though a replay of the identical traffic would reproduce every byte.

Thinking Machines frames it this way: from the perspective of an individual user, the other concurrent users are a nondeterministic property of the system, outside the inputs the user controls. They also note that nothing here is specific to GPUs. A server running on CPUs or TPUs would carry the same exposure.

From rounding error to a different sentence

The remaining question is why differences in the last digits of a score change a sentence. Think of a photo finish in a race where two runners cross the line within a hair of each other. A tiny measurement error decides the winner, and the result of every later round depends on who advanced. The analogy is mine, and the next paragraphs apply it to token choice.

My reading is that for most tokens a rounding difference cannot matter, because the leading token wins by a wide margin and a rounding error of one part in a million leaves it ahead. The effect shows up when two candidates sit very close together, so that a tiny perturbation flips which one has the highest logit. My reading of the Feynman result is that "Queens, New York" and "New York City" sat close enough at token 103 that numerical drift decided between them 992 times to 8.

After a flip, the divergence compounds. The model feeds its own output back as input, which means each new token joins the text the model reads before it picks the next one. The two sequences now have different prefixes, and every later logit is computed from different context. After a single flipped token the two sequences follow separate trajectories, and they may never merge again. Like a rerouted shipment, everything downstream of the change follows a different path.

I built a toy to watch this happen. It is my own construction, written to illustrate the kind of flip the Feynman result implies. It is a small random recurrent model, meaning a network of invented random numbers that was never trained and that carries a memory of earlier steps forward, with a vocabulary of 2,000 tokens and a hidden size of 64, the number of values it carries between steps. It decodes greedily with its logit reduction computed in float16, a coarser 16-bit number format. Two versions of the decoder differ in a single way: one accumulates the 64-term reduction sequentially, and the other uses four partial sums combined at the end, the same split-versus-sequential contrast as above.

The code below shows how the toy computes its scores in each of the two orders.

V, H = 2000, 64
def logits(h, mode):
    P = (h[:, None] * U).astype(np.float16)
    if mode == "seq":
        acc = np.zeros(V, dtype=np.float16)
        for i in range(H): acc = (acc + P[i]).astype(np.float16)
    else:
        parts = []
        for q in range(4):
            acc = np.zeros(V, dtype=np.float16)
            for i in range(q*16, (q+1)*16): acc = (acc + P[i]).astype(np.float16)
            parts.append(acc)
        acc = (((parts[0] + parts[1]).astype(np.float16) + parts[2]).astype(np.float16)
               + parts[3]).astype(np.float16)
    return acc.astype(np.float32)
# U is a random [64, 2000] output matrix scaled by 0.35; E and Wh are random as in the earlier toy
# decode: h = tanh(Wh @ h + E[tok]); tok = argmax(logits(h, mode)); 300 steps per decoder

I generated 200 random models and ran 300 greedy steps on each, comparing the token streams from the two reduction orders. In my toy, 149 of the 200 models diverged at some point. The median first divergence came at step 74, with the earliest at step 0 and the latest at step 287. The takeaway is that two decoders differing only in addition order can produce different text.

That experiment needs a caveat. I chose a small logit scale, which makes close contests between top candidates common, and I used float16 accumulation, which rounds more coarsely than the float32 accumulators that production kernels typically use for reductions. The toy shows a mechanism and says nothing about how often a production model diverges. The only production figures in this post are the ones Thinking Machines reported from the Feynman experiment: 80 distinct completions in a thousand, with the first split at token 103. My reading is that the toy matches the shape of that result. A reduction-order difference too small to see in any single logit eventually flips a close call, and the flip changes everything after it.

What temperature zero guarantees

With the pieces in place, I can restate the claim I started with. In my reading, temperature zero guarantees that the sampler's own lottery is switched off. It narrows the cone of acceptable outputs and removes no chance from the system. It makes no promise about the logits, and the logits depend on how the server happened to batch the request. Where two candidates sit within rounding distance of each other, numerical noise does the choosing. Thinking Machines reported that enabling their batch-invariant kernels made all 1,000 completions identical, and I take that as evidence that the variation lived in the kernels.

My reading of that result also changes where one looks for a fix. Changing the seed or the prompt cannot reach a source of variation that sits below the sampler. The next article takes up what a seed does control, what it leaves alone, and what it takes to make the kernels themselves invariant.

What this comes to

The belief I started with says that a setting on an AI chatbot guarantees the same answer to the same question, and that any leftover variation is random noise in the hardware. The setting narrows the range of words the model will pick from, and it leaves the model's uncertainty about the next word in place. Where two candidate words are nearly tied, the leftover variation comes from somewhere else. A provider's servers group many people's requests together, the grouping changes with traffic, and a different grouping changes how the arithmetic is rounded. A tiny rounding difference can flip a close choice between two words, and every word after the flip changes with it. Thinking Machines Lab reported that fixing the grouping effect made a thousand answers identical. The most useful thing to remember is that the setting shrinks the range of possible answers without removing chance, and the design of the server decides how much of the remaining chance reaches the reader.


Sources: Thinking Machines Lab (Horace He), "Defeating Nondeterminism in LLM Inference" (September 2025), for the Feynman experiment, the four statements, the concurrency-plus-floating-point framing and the arXiv preprint it quotes, the three-digit example, the 102-result shuffle script, the matmul test, the atomic-add analysis, the batch-size discrepancy of 1669.25, and the batch-invariant kernel result. The softmax table, the toy transformer check, the float32 batch-variance kernel, and the toy greedy decoder are my own code, reproduced above as illustrations of those ideas, and they are not evidence about production models. The 258-versus-102 comparison comes from my run of their shuffle script on Python 3.10.