Deterministic output and why you cannot have it

The question that shows up in every support ticket

At some point every team building on top of an LLM API asks the same question: why did the same prompt, sent twice, come back different? Someone sets temperature to 0. Someone else finds the seed parameter in the docs. Both walk away assuming the problem is solved. It isn’t. The reason has almost nothing to do with the settings you’re passing in the request body.

If you searched for “llm determinism seed,” you’re probably debugging a flaky eval or a regression test that fails intermittently. Or a production pipeline where the same input produces different JSON on different runs. The short version: you can reduce the variance, but you cannot eliminate it. Not with any commercial API. Not with most self-hosted setups either. Here’s what’s actually going on under the hood.

Temperature is not the only source of randomness

Temperature controls how the model samples from the probability distribution over the next token. At temperature 0, the model is supposed to always pick the single highest-probability token. That sounds like it should make the whole generation deterministic by definition. Greedy decoding. Same input, same output, every time.

That’s true in theory. In practice, temperature 0 removes one source of randomness and leaves several others untouched. The token probabilities themselves are computed by a forward pass through the model. That forward pass is where the real non-determinism lives.

The seed parameter and what it actually buys you

OpenAI and a few other providers expose a seed parameter along with a system_fingerprint field in the response. The idea is that if you pass the same seed and get back the same fingerprint, you should get the same output. This is a real, documented feature. It does help. But the providers themselves are upfront that it’s “best effort.” Not a guarantee.

The seed controls the sampling step. That’s the part where the model picks which token to emit, given the probability distribution it just calculated. What it does not control is how that probability distribution gets calculated in the first place. If the underlying math that produces the logits changes even slightly between two runs, you get a different distribution. And a different seed-driven pick from it, even with identical sampling parameters.

So the seed narrows one axis of randomness. It does nothing about the axis that causes most of the drift people actually complain about. That’s the forward pass itself.

Floating point math is not associative

This is the part that surprises people who haven’t worked close to the metal. In exact arithmetic, (a + b) + c equals a + (b + c). In floating point arithmetic, it often doesn’t. Rounding happens at every intermediate step, and the order of operations changes which values get rounded when.

A transformer forward pass involves enormous numbers of matrix multiplications and summations. On a GPU, those operations get parallelized across thousands of cores. The order in which partial sums get combined depends on how the kernel schedules the work. That can vary between runs based on what else is happening on the hardware, which CUDA kernel gets selected, and how the compute gets divided up. Change the summation order and you change the rounding. By a tiny amount. That’s in float16 or bfloat16 precision, which is what most inference stacks run in for speed.

A tiny rounding difference in one layer’s output gets fed into the next layer. Across dozens of layers, those differences compound. By the time you reach the final logits, a probability that was 0.5001 for token A and 0.4999 for token B in one run might shift. In another run: 0.4998 and 0.5002. At temperature 0, greedy decoding picks whichever token is higher. That razor-thin numerical difference flips your output. This is why you can run the exact same model weights, on the exact same hardware, with the exact same seed. And still see divergence.

Batching is the part nobody explains in the docs

Here’s the piece that actually accounts for most of the real-world variance people see from hosted APIs. And it has nothing to do with your request in isolation. Inference providers batch requests together to use GPU compute efficiently. Your prompt doesn’t run alone. It runs alongside other users’ prompts in the same forward pass, and the composition of that batch affects the numerical path the computation takes through the hardware.

Matrix multiplication kernels are often tuned differently depending on batch size and shape, for performance reasons. A batch of 4 requests might route through a different kernel than a batch of 64. Those kernels can produce slightly different floating point results even for the same input row, because of the associativity issue above. This means your identical prompt, sent twice a few seconds apart, can genuinely land in two different batches and come back with two different completions. Purely because of what other traffic happened to be on the server at that moment. You have no control over this and no visibility into it. It’s an operational detail of the provider’s infrastructure. Not a bug in your code.

Mixture of experts makes it worse

If the model you’re calling uses a mixture of experts architecture, that’s common in current large models. It lets you scale parameter count without scaling compute cost per token. That architecture adds an additional layer of batch-dependent behavior. A router network decides, per token, which subset of experts to activate. In some MoE implementations, especially ones with load balancing across a batch, the routing decision for a given token can be influenced by what other tokens are in the same batch. The system is trying to spread load evenly across experts.

That means the exact same token, in the exact same position, in the exact same prompt, can get routed to a different expert depending on the surrounding batch. Different experts produce different outputs. This is a structural property of how these models are served at scale. Not a misconfiguration you can fix from the client side.

What actually breaks when you assume determinism

The practical failure mode I see most often is teams building test suites or evals that assert exact string equality on model output. Then those tests flake in CI, for no apparent reason. The second most common one is structured output pipelines, JSON extraction, function calling, that kind of thing. A one-token difference cascades into a malformed object and a parser exception downstream. The fix in both cases is the same. Stop treating the model as a deterministic function. Start treating it as a probabilistic one, even at temperature 0.

What you can actually do about it

You can’t get a guarantee. But you can reduce variance meaningfully. Set temperature to 0 for anything where you want the mode of the distribution rather than a sample from it. Pass a seed if the provider supports one, since it removes the sampling-layer randomness even if it can’t touch the forward-pass randomness. If you’re self-hosting, you have more room. Pin a specific inference framework version, disable dynamic batching, and force deterministic CUDA kernels. Most frameworks have a flag for this, usually at a real throughput cost. Do that and you get much closer to true run-to-run reproducibility, because you control the batch composition and the kernel selection yourself.

For evals and tests, the more durable fix is to stop asserting exact match at all. Compare structured fields with tolerance. Use a rubric-based or embedding-similarity check for free text, and treat small formatting drift as expected noise rather than a bug.

That’s an accurate model of what the system actually is.

If you’re building anything where two runs of the same prompt need to match byte for byte, on infrastructure you don’t fully control, that requirement is worth challenging. Do it before it becomes the thing your whole test suite depends on.

For more breakdowns like this on what AI tools actually do under the hood, head back to the AI Tool Gazette homepage.