Temperature and Top-p Explained: What LLM Sampling Settings Do

Temperature and top-p control how a language model picks each next token from its list of candidates. Lower temperature makes the likeliest tokens even more likely (more predictable output), higher temperature flattens the odds (more varied output), and top-p discards the unlikely tail before the model picks.

One thing has changed recently: as of October 2026, several hosted models no longer accept these settings at all. The section on per-API ranges below covers which ones.

How a model picks the next token

At each step the model produces a score for every token in its vocabulary. Those scores are turned into probabilities, and one token is sampled from them. Sampling settings change the probabilities or the candidate list before that draw. They do not change what the model knows, only how adventurous its choices are.

What temperature does, with numbers

Temperature divides the scores before they are converted into probabilities with the softmax function: pi = esi/T ÷ Σ esj/T. Take three candidate tokens with scores 2, 1 and 0:

Check the middle row: e² = 7.389, e¹ = 2.718 and e⁰ = 1, which sum to 11.107, so A gets 7.389 ÷ 11.107 = 66.5%. At T = 0.5 the scores become 4, 2 and 0 before the same calculation; at T = 2 they become 1, 0.5 and 0.

Lower temperature sharpens the distribution toward the top choice. Higher temperature spreads probability to less likely tokens, so the output varies more between runs and is more likely to take unexpected turns. At the limit of temperature 0, the model always takes the top token. Google's docs put it as: "A temperature of 0 is deterministic, meaning that the highest probability response is always selected."

OpenAI's API reference gives the practical reading: "Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic."

What top-p does

Top-p, also called nucleus sampling, sorts the candidates from most to least likely and keeps the smallest set whose probabilities add up to at least p. Everything else is discarded, and the model samples from what remains. OpenAI's reference: "0.1 means only the tokens comprising the top 10% probability mass are considered."

With the temperature 1.0 probabilities above (66.5%, 24.5%, 9.0%) and top-p = 0.9:

• A alone covers 66.5%, which is below 90%.
• A + B cover 91.0%, which reaches 90%, so the set is {A, B}.
• C is discarded. Renormalised, A gets 66.5 ÷ 91.0 = 73.1% and B gets 26.9%.

Google's docs give a similar example: with tokens at 0.3, 0.2 and 0.1 and top-p of 0.5, the model chooses between the first two and excludes the third.

The difference from temperature: top-p never makes a rare token more likely. It only removes the long tail of very unlikely tokens, which is where nonsense words and off-topic jumps come from.

Top-k and min-p

Two related settings appear in local runtimes and some APIs:

• Top-k keeps only the k most likely tokens. Google's docs describe top-k of 1 as greedy decoding and explain that, where several filters are used, tokens are first limited by top-k, then filtered by top-p, and the final token is chosen with temperature sampling.
• Min-p keeps tokens whose probability is at least p times the top token's probability. Ollama's docs give the example: with min-p 0.05 and a top token at 0.9, tokens below 0.045 are filtered out.

Ranges and defaults per API (as of October 2026)

The direction of travel is clear. Anthropic's recommended replacement is to omit the parameters and "use prompting to guide model behavior". For hosted frontier models, sampling settings are becoming a local-model and older-model feature. Check the current API reference before relying on them.

When to change which setting

These are starting points, not rules from a vendor. Change one setting at a time and compare several outputs per setting, since a single sample tells you little about a random process.

Lowering temperature does not make a model more truthful. If its most likely answer is wrong, a low temperature makes it consistently wrong. For that problem, see why AI hallucinates and how to reduce it.

Examples

OpenAI Responses API with a low temperature for a classification task:

{
  "model": "MODEL_ID",
  "temperature": 0.2,
  "input": [
    {"role": "developer", "content": "Classify the ticket as billing, bug or other. Reply with one word."},
    {"role": "user", "content": "I was charged twice this month."}
  ]
}

Replace MODEL_ID with a model whose documentation says it accepts temperature.

Ollama's local API, passing settings in options:

curl http://localhost:11434/api/generate -d '{
  "model": "gemma4",
  "prompt": "Suggest five names for a hiking club.",
  "stream": false,
  "options": {"temperature": 1.1, "top_p": 0.95, "seed": 42}
}'

Ollama's Modelfile docs say a fixed seed makes the model generate the same text for the same prompt, which is useful when you want to compare settings fairly. The same parameters can be set permanently in a Modelfile, for example PARAMETER temperature 0.7. Setting up Ollama is covered in how to run an LLM locally with Ollama.

If the settings seem to do nothing

• The model may ignore them. Gemini 4 and 3.6+ ignore custom values; Claude 4.7+ rejects non-default ones.
• The prompt may dominate. A prompt that asks for one exact format leaves little room for sampling to vary.
• You may be comparing single samples. Run each setting several times before deciding.

Frequently asked questions

Does temperature 0 make output identical every time?

Not necessarily. Temperature 0 tells the model to take the most likely token, which removes sampling randomness and makes output far more consistent, but test before building anything that depends on byte-identical output from a hosted API. For repeatable output from a local model, Ollama documents a seed parameter that makes the model generate the same text for the same prompt.

Should I change temperature and top-p together?

Usually not. OpenAI's API reference says it generally recommends altering temperature or top_p, but not both. Changing one at a time also makes it clear which setting caused a change in the output.

What is top-k?

Top-k keeps only the k most likely tokens before sampling. Google's docs explain that top-k of 1 means always picking the single most likely token (greedy decoding), and top-k of 3 means choosing among the three most likely. Ollama's documented default is 40. OpenAI's Responses API does not offer top-k.

Why does my Claude request fail with an error about temperature?

Anthropic's docs say that on Claude 4.7 and later models, setting temperature, top_p or top_k to a non-default value returns a 400 error, and that API version 2026-10-21 rejects these parameters on every model, even at default values. Remove them from the request and steer the output with the prompt instead.

Sources

• OpenAI API reference: Create a model response (temperature, top_p)
• Claude API docs: Model deprecations (API parameter deprecations)
• Claude API docs: Migrating to API version 2026-10-21
• Google AI for Developers: Prompt design strategies (model parameters)
• Ollama docs: Modelfile parameters