Temperature and sampling settings nobody tunes

The default nobody questions

Open any LLM API playground and temperature sits there at 0.7 or 1.0, quietly doing its job while everyone argues about prompts instead. I’ve shipped products where the prompt got rewritten fifteen times and the temperature setting never got touched once. That’s backwards. If your outputs are inconsistent, repetitive, or randomly weird, sampling settings are often the faster fix, and they’re free to change. No retraining, no new prompt, just a different number in the request body.

This isn’t a plea to obsess over every parameter. It’s a walkthrough of what these settings actually do at the math level, because “temperature controls randomness” is true but useless for deciding what number to put in your config.

What the model actually hands you

Before any sampling happens, the model produces a probability distribution over every token in its vocabulary, usually 30,000 to 200,000 possible next tokens. This comes from a softmax over raw scores called logits. Some tokens get high probability, most get near zero. Sampling settings are just different ways of picking one token from that distribution.

Temperature works by dividing the logits by a value before the softmax runs. At temperature 1.0, you get the raw distribution the model learned. Drop it to 0.2 and you’re dividing by a small number, which stretches the gaps between logits before they’re exponentiated. The high-probability tokens get pushed even higher relative to everything else, and the distribution collapses toward the single most likely token. Push temperature up to 1.5 and you divide by a number greater than 1, which flattens the distribution and gives low-probability tokens a real shot at being picked.

Temperature 0 is a special case most APIs handle by skipping sampling entirely and just taking the argmax, the single highest-probability token every time. That’s greedy decoding, not “very low randomness.” It’s a different code path, not a limit.

Top_p and top_k are filters, not intensity dials

Temperature reshapes the whole distribution. Top_p and top_k throw parts of it away before you sample.

Top_k keeps only the k highest-probability tokens and zeroes out everything else, then renormalizes and samples from what’s left. Set top_k to 40 and the model can only ever pick from its 40 best guesses for that position, no matter how flat or peaked the distribution is.

Top_p, also called nucleus sampling, is more adaptive. It sorts tokens by probability and keeps adding them until their cumulative probability crosses p, say 0.9, then samples only from that set. The nucleus size changes token by token. When the model is confident, the nucleus might be two or three tokens. When it’s genuinely uncertain, like at the start of a creative sentence, the nucleus might include forty tokens. That’s why most APIs default to tuning top_p rather than top_k: it responds to how confident the model actually is at each position instead of using a fixed cutoff.

Here’s the part people miss: temperature and top_p stack. If you set temperature to 0.3 and top_p to 0.9, the reshaping happens first, then the filtering happens on the reshaped distribution. A low temperature with a loose top_p behaves almost identically to a low temperature with a tight top_p, because the reshaping already did most of the work of eliminating unlikely tokens. Tuning both aggressively at once often just wastes a knob.

Penalties operate after the fact, not on the distribution shape

Frequency penalty and presence penalty don’t touch the softmax math directly. They subtract a value from the logit of any token based on how often it’s already appeared in the output (frequency penalty) or whether it’s appeared at all (presence penalty). This is why they’re the right tool for a specific failure mode: repetition loops, where a model gets stuck saying “In conclusion, in conclusion, in conclusion.” Temperature won’t reliably fix a repetition loop because the model can be highly confident that the repeated token is correct. A penalty directly punishes the token for showing up again, regardless of how confident the model is.

I’ve seen teams crank temperature to fix repetition and just get repetitive nonsense instead of repetitive coherent text. Wrong tool. A frequency penalty of 0.3 to 0.6 clears up most repetition loops without destabilizing anything else.

Why RAG pipelines default to near-zero and why that’s not always right

For retrieval-augmented generation, low temperature makes sense on the surface: you want the model to stick close to the retrieved context instead of wandering off into its own generative habits. A temperature of 0 to 0.2 concentrates probability mass on the tokens most consistent with what’s in the prompt, which usually means fewer invented details.

But low temperature doesn’t fix a bad retrieval step, and it doesn’t stop hallucination that comes from the model confidently misreading correct context. It also has a real cost: at temperature 0, if your retrieved chunks are ambiguous or contradictory, the model will pick one confident-sounding answer and commit to it every single time, with no variance to reveal that it was actually a coin flip. I’ve debugged RAG systems where cranking temperature slightly, to something like 0.4, and sampling the same query three times exposed that the model had no real signal, because the three answers disagreed with each other. At temperature 0 that disagreement is invisible. You get one wrong answer, delivered with total consistency, and consistency gets mistaken for correctness.

Temperature 0 is not the same as deterministic

This trips people up constantly. Setting temperature to 0 means greedy decoding, always picking the top token. It does not mean the API will return byte-identical output for the same input every time. Floating point operations on GPUs aren’t strictly associative, and batching, hardware scheduling, and mixture-of-experts routing (if the model uses it) can all introduce tiny numerical differences between runs. Those tiny differences can occasionally flip which token has the highest logit when two tokens are close in probability, and once one token diverges, everything after it in the sequence can diverge too, since each token’s generation depends on the ones before it.

If you need reproducibility for testing or auditing, don’t assume temperature 0 gets you there. Log the actual outputs, use a fixed seed if the API exposes one, and treat “close enough” as the realistic bar rather than bit-for-bit identical.

Sampling more than once is a legitimate tuning move

Instead of endlessly tuning a single call’s temperature, some tasks are better served by sampling several completions at a moderate temperature, like 0.7 to 0.9, and picking the best one, either by majority vote on the final answer or with a separate scoring step. This is the idea behind self-consistency prompting: run the same reasoning task multiple times, let the model explore different paths, and keep the answer that shows up most often.

The tradeoff is obvious and it’s cost. Three or five samples means three or five times the API bill and the latency of your slowest call in the batch, or the sum of them if you’re not running in parallel. For high-stakes single answers where you can afford the extra calls, this beats obsessing over the “correct” temperature for a single shot. For a chat interface answering a user in real time, it’s usually not worth the delay.

What to actually change first

If your outputs feel flat and templated, raise temperature before you touch anything else, and keep top_p at a sane default like 0.9 to 0.95 rather than fighting both knobs at once. If outputs feel unstable or off-topic, lower temperature before you add more instructions to the prompt telling the model to “stay focused,” which often does less than the sampling setting would. If you’re seeing loops or repeated phrases, reach for a frequency penalty, not temperature. And if you need to trust a single answer for something that matters, consider sampling a few times at a moderate temperature and checking whether the answers agree, because agreement is a signal temperature 0 will never show you.

None of this replaces a good prompt or a good retrieval pipeline. It just means the twenty seconds it takes to change a number in a config file is sometimes worth more than the twentieth prompt rewrite.

Want more breakdowns like this on the tools and models people actually ship with? Check out the rest of AI Tool Gazette for explainers that skip the hype and go straight to how things work.