What a Seed Controls and What Batch Invariance Controls
A common belief about AI chat tools says that one fixed setting, usually called a seed, makes the tool give the same answer every time the same question goes in. It sounds like a recipe card in a factory: same inputs, same steps, same product. The belief matters to anyone who tests an AI feature, audits its answers, or tries to reproduce a result a colleague saw last week, because a test that passes today and fails tomorrow with no change in the code costs real time and wears down trust.
This post explains what that setting controls, which is one small step in the process, and what it leaves alone, which is the arithmetic running underneath on the graphics chips. I come away from the research with a two-part picture. Two separate things have to hold still for answers to repeat. Only one of them sits in the user's hands on a shared service. The other has a published fix with a measured cost. A reader who finishes this post should be able to say why identical requests sometimes differ, and what it takes to stop that.
The Microsoft example
Microsoft's documentation for reproducible output in Azure OpenAI includes a worked example that I find more instructive than most research papers. The same request goes out three times with seed=42, temperature 0.7, and a limit of 50 output tokens. The seed is a starting number for the random-number generator the tool uses when it picks words. Temperature is a dial for how adventurous the word choices are, where zero is the most predictable. A token is a word or word fragment, the unit in which the model writes.
The first two stories match word for word. The third follows them through two sentences, "In the beginning, there was nothing but darkness and silence. Then, suddenly, a tiny point of light appeared," and then departs. Directly below the example, the Microsoft documentation states that determinism is not guaranteed, that the seed and a returned system_fingerprint should be used together to detect backend changes, and that identical calls with larger max_tokens values will generally be less deterministic even when the seed is set.
In my reading, everything in that example follows from the mechanism in Part 5, and it exposes the misunderstanding for this installment: that a seed fixes the output. A seed fixes one input to the process, the random-number generator behind the step that picks the next token. It has no reach into the arithmetic that produces the odds the pick is made from. Making that arithmetic repeatable takes work at the level of GPU kernels, small programs that run one math operation on a graphics chip. That work has a name, a published method, a cost, and by now a switch in at least one major serving framework.
What the seed reaches
Think of a lottery in which the order of the draw is fixed in advance, like a deck of cards stacked and dealt the same way each time. The same cards come out in the same order. If the prize table those cards are matched against shifts slightly between runs, the same card can win a different prize. The seed fixes the cards. It has no say over the prize table.
Technical detail: the model produces logits, raw scores for every candidate next token, which become probabilities. At a temperature above zero, the sampler, the step that picks the token, draws using a random number. The seed initializes the generator that supplies those numbers, so two requests with the same seed draw the same sequence of random values. If the logits were identical in both requests, the outputs would be identical too, because the same random value would land in the same place in the same distribution.
When the logits differ in their last bits, the picture changes. The probabilities shift by tiny amounts. A drawn number that falls well inside one token's interval still selects the same token, and a number that falls near a boundary between two tokens selects differently when the boundary moves. Close contests between candidates, as in the Feynman example in Part 5, make boundaries sensitive. After one differing token, the two sequences have different contexts, and the random values continue to be drawn in lockstep from the generator while the distributions they apply to have diverged.
That explains the behavior the Microsoft documentation describes. A longer output contains more steps at which a close contest can flip, which is why larger max_tokens values degrade repeatability. In the toy decoder I wrote for Part 5, the median first divergence arrived at step 74. That toy is my own and says nothing about how often a production model diverges. It shows the shape of the effect: a short output often escapes divergence, and longer outputs give it more chances to occur. My reading is that the Microsoft example fits that pattern, since two of three runs agreed over 50 tokens and one drifted. Three runs make a single example, and I treat it that way.
Variation and randomness
Two words need separating here. Randomness is the chance involved in producing an output at all. Variation is the difference between one run and the next. A casino's dice are random. A recording of one roll, played back, shows no variation from play to play, and the roll that was recorded was still random when it happened.
Part 5 argued that temperature narrows the range of outputs a model will accept and leaves the model's uncertainty in place. Batch invariance works one level down. In my reading it removes variation, meaning the run-to-run differences caused by arithmetic that shifts with server load. It cannot remove the randomness that went into the original output. At a temperature of 0.7 the first run's token choices were draws from a distribution, and the output that came back is one of many the model could have written. A seed plus invariant kernels lets an engineer replay that one draw exactly, like the recorded roll. The draw stays random, and the output stays one member of a cloud of acceptable outputs.
The same holds at temperature zero. Invariant kernels fix the logits, so the near-ties that numerical noise used to break at random now break the same way every time. The system still contains those near-ties, and an invariant stack picks one resolution and holds it. I take the fixed answer to be one point inside the cone of acceptable outputs and no closer to the truth than the other points. Thinking Machines Lab reported 1,000 identical completions with their invariant kernels, and I read that as a replay property. It shows that the stack can repeat itself, and it says nothing about the model having become certain.
What the fix requires
An everyday version of the problem comes from adding a long column of invoice amounts with rounding at each subtotal. Group the amounts in tens and the total can differ by a cent from the total reached by grouping them in fifties. Both groupings are legitimate sums. A rule that fixes the grouping for each customer's invoices, whatever number of other customers are in the same ledger run, would make every customer's total repeatable. That rule is the idea behind batch invariance.
Technical detail: a batch is the group of requests a server processes together. A reduction is any step that combines many numbers into one, such as a sum. Computers store decimal numbers as floats, which carry limited precision, so adding the same numbers in a different order can change the last digits. Horace He and Thinking Machines Lab (2025) formulate the requirement this way: the reduction order for each element stays fixed regardless of the batch size. The strategy can change when the problem shape changes in other ways, such as the length of the dimension being reduced. It must never change because of how many other requests share the batch. They note that pointwise operations, like activations, are invariant by construction, since each output depends on one input with no reduction. In their account, the work concentrates in the three operations that reduce: RMSNorm, matrix multiplication, and attention, listed in order of increasing difficulty.
RMSNorm. Technical detail: RMSNorm is a step that rescales each row of numbers inside the model. Thinking Machines describe the efficient default as assigning one row of the batch to one core, which performs the entire reduction for that row. Because every row follows an identical procedure, adding more rows only gives each core more rows to process in turn, and invariance holds. The problem appears when the batch shrinks below the number of cores. Idle cores tempt kernel authors to split a single row's reduction across several cores, which changes the accumulation order for exactly the small batches where it kicks in. They describe a cheap fix, which skips the optimization since small batches already run quickly. A more ambitious fix commits to a strategy with enough built-in parallelism for small batches and applies it to large ones as well, accepting a small tax everywhere.
Matrix multiplication. Technical detail: matrix multiplication is the multiply-and-add operation that makes up most of a model's computation. In the Thinking Machines account, the standard approach divides the output into two-dimensional tiles and gives each tile to one core, which computes the whole dot product for its tile. That is invariant by default. The failure arrives when the batch dimensions get small enough that there are too few tiles to occupy the GPU. Their example is a 1024 × K by K × 1024 multiplication with 128 × 128 tiles, which produces only 64 tiles, nowhere near enough for a modern GPU. Kernels respond with split-K, which divides the reduction dimension across cores, and that changes the accumulation order. A second hazard comes from tensor-core instructions, the chip's specialized multiply hardware, each of which has its own internal reduction order. A kernel that switches to a smaller instruction for a small batch, or drops tensor cores altogether at batch size one, changes the arithmetic. They also single out Stream-K, a load-balancing scheme that splits the reduction differently for different output tiles. That means results can change with a row's position inside a fixed-size batch, a property that ordinary matmul kernels at least preserve.
The fix they settled on is blunt: compile one kernel configuration and use it for every shape. In their benchmark, with an unoptimized Triton kernel (Triton is a language for writing GPU kernels), they report a cost of about 20% against cuBLAS, NVIDIA's standard math library, concentrated at very small batch sizes where the fixed configuration wastes parallelism. They observe that the worst case for split-K needs both batch dimensions to be small at once, and in transformer inference the model dimension is usually large, which limits how often the problem bites.
Attention. Technical detail: attention is the step in which each token looks back at earlier tokens to decide which ones matter. It reduces over both the feature dimension and the sequence dimension, and it has to cope with the ways serving systems slice sequences: chunked prefill (processing a long prompt in pieces), prefix caching (reusing work for a shared prompt opening), and incremental decoding against a growing KV cache. The KV cache holds the stored keys and values of earlier tokens so that earlier work gets reused. Thinking Machines tighten the requirement accordingly. The reduction order for a token has to be identical whether zero tokens are in the cache or 999, and regardless of how many other tokens from the same sequence are processed in the same step.
In Part 4 I worked through a small example of the first failure, and the figures come from my own arithmetic there. A kernel that handles cached keys and values separately from new ones can need five blocks for the 80 cached plus 48 new tokens in the example, and four blocks for the same 128 tokens processed together. The fix Thinking Machines describe is to write the new keys and values into the cache and page table before the attention kernel runs, so the kernel always sees one consistent layout.
Decoding, the phase in which the model writes one token at a time, produces the second failure. The query length is one, so the query dimension offers almost no parallelism, and the kernel splits the reduction over keys and values, a technique called Split-KV or FlashDecoding. Unlike the matmul case, ignoring the problem costs too much, because a long cache makes attention slow even for a single request. Thinking Machines describe standard implementations as dividing the cache into a fixed number of splits sized to saturate the GPU, which makes the split points a function of how much parallelism is needed and therefore of the batch. The batch-invariant version fixes the size of each split and lets the count vary. In their illustration, for a cache of 1,000 entries, the standard approach produces four splits of 250. The invariant approach produces three splits of 256 and one of 232, and the boundaries stay put no matter what else is running. They also noted that this required internal changes to FlexAttention that were outside their initial code release.
What the fix costs
A factory line locked to one setup for every order size gives up the speed it would get from tuning the setup to each order. Repeatable output on a GPU carries the same kind of cost, and the published measurements put numbers on it.
Thinking Machines Lab reported a performance test using one GPU serving Qwen3-8B, an open model, with 1,000 requests of 90 to 110 output tokens. The default vLLM configuration finished in 26 seconds. The unoptimized deterministic configuration took 55 seconds, about 2.1 times slower by my division of their figures. With an improved attention kernel it took 42 seconds, about 1.6 times slower by the same division. The authors said that they invested little effort in optimizing the invariant kernels, and attributed much of the remaining slowdown to the FlexAttention integration in vLLM.
Other measurements arrive at different numbers, which is itself informative. The CoRun paper (Zhao et al., posted to arXiv in August 2026) reports that batch-invariant kernels increase latency by more than 2 times and reduce serving throughput by as much as 74%. I read only the abstract of that preprint and ran none of its experiments, so I treat its figures as one group's measurements on their setup.
Technical detail: as I read the abstract, CoRun observes that most kernels, although they vary with batch size, stay invariant to a row's position in the batch. It exploits that through scheduling, which leaves the kernels as they are. Prefill, the phase that processes the prompt, runs in isolation, and decoding runs in fixed-shape batches using CUDA graphs, a way of recording a fixed sequence of GPU work and replaying it. The authors report throughput gains of 15% to 324% over batch-invariant approaches, along with lower time to first token and time per output token. My reading is that this second route to determinism, which holds the shapes constant and leaves the kernels shape-sensitive, gives engineers a design choice with its own tradeoffs.
A counterweight sits against that cost. Serving infrastructure now treats the capability as a supported feature. The vLLM documentation describes batch invariance as a beta feature, enabled by setting the environment variable VLLM_BATCH_INVARIANT=1, and states its guarantee in these terms: the output of a model is deterministic and independent of the batch size or the order of requests in a batch. It lists NVIDIA GPUs with compute capability 8.0 or higher, certain AMD Instinct GPUs, and Intel XPUs with Triton support.
Technical detail: the vLLM implementation notes describe deterministic attention kernels, fixed-order all-reduce under tensor parallelism, and a fixed-chunk reduction for LoRA adapters. Tensor parallelism splits one model across several GPUs, all-reduce is the step where those GPUs combine partial results, and LoRA adapters are small add-on weight sets that customize a model. The warnings in the documentation note that tensor-parallel all-reduces sized off a 16-byte multiple remain unsupported. The tested-model list spans Qwen3, Llama 3, DeepSeek, gpt-oss, and Mistral among others, including a few quantized variants, which use compressed number formats. The documentation also says that enabling the mode may impact performance, and calls the tradeoff intentional.
Why training teams wanted it first
Picture a factory whose quality inspector measures parts with a gauge that reads slightly differently from the gauge used on the line. Every judgment the inspector makes carries a small error, and the errors can pile up. That is the situation in the original motivation for the Thinking Machines write-up, which was reinforcement learning.
Technical detail: in reinforcement learning fine-tuning, a sampler generates rollouts, which are complete sample outputs, and a separate trainer scores them and computes gradients, the numerical directions in which to adjust the model. If the two use different numerics, the log-probability the trainer computes for a token differs slightly from the one the sampler effectively used. What is meant to be on-policy learning, where the model learns from its own current behavior, becomes slightly off-policy.
Thinking Machines Lab reported a reinforcement-learning-with-verifiable-rewards experiment on the BigMath dataset, starting from Qwen 2.5-VL instruct 8B with a maximum rollout length of 4,096 tokens, and compared three configurations. Without any off-policy correction, reward collapsed partway through, with a loss spike around step 318 that coincided with a spike in the KL divergence between sampler and trainer log-probabilities. KL divergence is a measure of how far apart two probability distributions sit. With an importance-weighting correction, a statistical adjustment for the mismatch, training proceeded smoothly, and the KL divergence hovered around 0.001 with occasional spikes. With bitwise-identical numerics in sampler and trainer, the KL divergence was a flat line at exactly zero and training ran smoothly with no correction at all.
Part 3 described post-training as the stage that shapes behavior, and increasingly that stage uses reinforcement learning. To me, the flat line at zero says something about its foundations. My reading is that the mismatch between inference numerics and training numerics, which the importance-weighting configuration compensates for, had a specific cause, and removing the cause removed the mismatch. The vLLM documentation lists reinforcement learning among its stated motivations, and its example gallery includes a batch-invariance example in the RL section.
The Stochastic CHAOS argument against determinism
Reasonable researchers read the goal differently. Joshi et al., in a paper first posted in January 2026 and revised in October, "Stochastic CHAOS," argue that deterministic inference "kills" valuable properties of language models: that collapsing the output distribution to a single completion hides uncertainty, masks fragility under paraphrase, and underestimates safety risk because rare dangerous behaviors surface only under multi-sample evaluation. Its authors advocate treating distributional variability as a signal to measure.
My reading is that the empirical half of that argument deserves attention, and the framing conflates two things that can be separated. The evaluation point stands on its own: a single greedy sample from a model, meaning one run that always takes the top-scoring token, says little about its failure rate, and multi-sample evaluation reveals more. Batch invariance has no quarrel with that. It removes numerical noise from the logits, and a sampler operating above temperature zero still draws different completions for different seeds.
Here is my argument for why the two coexist. A batch-invariant system with temperature 0.7 and a thousand different seeds samples the distribution faithfully, and it also lets an engineer replay seed 417 exactly. Determinism of the arithmetic and diversity of the samples sit side by side. In my reading, the paper's critique lands hardest against the practice of evaluating at temperature zero with one sample, which was never a consequence of batch invariance.
What a seed settles
With the mechanism laid out, the accounting is simple. In my accounting, a seed makes the sampler's random draws replayable, and batch invariance makes the logits replayable. Neither one removes the randomness behind the first output. Together they let an engineer reproduce one outcome of a random process on demand, and reproducibility requires both. On a shared endpoint the provider owns the second, which is why the Microsoft documentation hedges. On infrastructure an organization controls, the second is available as a configuration choice with a quantified cost: roughly 1.6 to 2.1 times slower in the Thinking Machines Lab measurements, with the CoRun authors claiming substantially better from newer scheduling.
That changes how I think about testing language-model applications. These testing views are mine and come from no cited source. A golden-output test that asserts exact string equality will fail intermittently on a shared endpoint, for reasons unrelated to the system under test, and the failure rate climbs with output length. Tests built around invariants hold up better: schema validation, presence of key entities, tolerance bands on numbers, graded rubrics for open-ended text, with exact match reserved for short outputs like classifications and extracted values. I put the divergence rate across repeated runs of a fixed input on the dashboard as a measured quantity. On a managed endpoint, that variation can be bounded and understood. Eliminating it requires owning the kernels.
That logic carries into the interpretability work in the final three articles. A claim about what a model computed on a specific input assumes that the computation can be reproduced, and Part 10 returns to what that assumption demands of the serving stack.
What this comes to
The common belief says that a seed makes an AI tool repeat itself. The research I read describes two separate sources of variation. The seed fixes the random draws that pick each word. The arithmetic underneath can still shift by tiny amounts depending on how many other requests the server handles at the same moment, and those tiny shifts can tip a close word choice the other way. Repeatable output needs both pieces held still. On a shared service the provider controls the second piece, and on a system an organization runs itself the fix exists and costs speed. Even then the fix removes run-to-run variation and leaves the original chance in place, so it works like a recording of one roll of the dice. The single most useful thing to remember is that a matching seed guarantees matching random draws and nothing more.
Sources: Microsoft Learn, "Learn how to use reproducible output" for Azure OpenAI. Thinking Machines Lab (Horace He), "Defeating Nondeterminism in LLM Inference" (2025). vLLM documentation, "Batch Invariance". Zhao et al., "CoRun: Padding is Simple and Efficient for Deterministic LLM Inference" (arXiv 2608.14376, August 2026). Joshi et al., "Stochastic CHAOS" (arXiv 2601.07239). The timing figures for the batch-invariant configurations come from the Thinking Machines write-up. The CoRun figures come from its abstract. The toy decoder results and the Part 4 block counts are my own computations from earlier parts of this series.