Sampling & Temperature
Once the model has a probability for every next token, how does it actually pick one? Temperature, top-k, and top-p are the knobs that turn that distribution into deterministic-and-safe or wild-and-creative text.
In the last lesson we watched the model turn its final-layer vector into logits, then into a probability distribution over every token in its vocabulary — say mat 89%, rug 10%, banana 1%. That distribution is where this lesson begins. The model still has to commit to exactly one token, and how it makes that pick is the whole game. Pick greedily and you get safe, repetitive text. Roll the dice and you get variety, personality, and the occasional creative leap. The dials that control that trade-off are temperature, top-k, and top-p.
#Greedy vs. sampling
The two base strategies sit at opposite ends. The simplest is greedy decoding: take the argmax — the single most probable token — every step. It's deterministic (same prompt in, same text out) and never obviously weird, but it has a famous failure mode: it gets stuck in loops and sounds flat, because it can only walk the one most-traveled path. The alternative is sampling: treat the probabilities as weighted dice and roll. Now mat wins ~89% of the time, but rug genuinely shows up ~10% and banana ~1%. Run the same prompt twice and you get two different but plausible continuations — that randomness is the source of an LLM's variety.
probs = {"mat": 0.89, "rug": 0.10, "banana": 0.01}
argmax(probs) # greedy -> "mat", every single time
sample(probs) # sampling-> "mat" ~89%, "rug" ~10%, "banana" ~1%A loaded die, not a fair one
Sampling isn't picking uniformly at random — that would be nonsense. It's rolling a die that's loaded in proportion to the model's confidence: a token the model gives 89% gets 89 of the 100 faces; a 1% token gets one face. You almost always land on a good token, but the rare faces keep things fresh. The open question is how loaded the die should be — and that's exactly what temperature controls.
#Temperature: reshaping the distribution
Temperature (written T) is a single number that rescales the logits before softmax: divide every logit by T, then softmax as usual. That one division does something intuitive. A small T (say 0.5) stretches the logits apart — the gap between the leader and the rest grows, so the top token dominates even harder. A large T (say 2.0) squishes them together — gaps shrink, so probability leaks into the long tail and the distribution flattens toward uniform. At T = 1 you get the model's raw, untouched distribution; as T approaches 0 you approach greedy.
# Temperature divides the logits BEFORE softmax
probs = softmax([logit / T for logit in logits])
logits = [2.0, 1.0, 0.0] # three candidate tokens
T = 0.5 -> [0.87, 0.12, 0.02] # sharp: top token dominates
T = 1.0 -> [0.66, 0.24, 0.09] # the raw distribution
T = 2.0 -> [0.51, 0.31, 0.19] # flat: tail gets a real shotWhy it's called 'temperature'
The name is borrowed from physics. Picture the tokens as gas molecules. Cold (low T) and everything settles into the lowest-energy state — calm, predictable, always the same. Hot (high T) and the molecules jitter everywhere — high energy, lots of surprise, occasional chaos. Turning up the temperature literally adds energy (randomness) to the choice.
You lower the temperature from 1.0 to 0.2. What happens to the probability distribution over the next token?
Temperature redistributes probability — it doesn't add knowledge
Cranking T high doesn't make the model smarter; it just hands more probability to tokens the model rated as less likely. Sometimes that surfaces a delightful, unexpected word. Just as often it surfaces a wrong fact, broken code, or an incoherent tangent — because you deliberately made low-probability (often low-quality) tokens more reachable. High temperature buys variety, and the bill is paid in reliability.
#Top-k and top-p: cutting off the junk tail
Even at a sensible temperature, the tail holds thousands of genuinely bad tokens, each with a tiny probability — add them up and there's a real chance you sample something nonsensical. Truncation throws the tail away first. Top-k keeps only the k highest-probability tokens (say 40), zeroes the rest, renormalizes, and samples from those. Top-p (nucleus sampling) is smarter: it keeps the smallest set of top tokens whose probabilities add up to p (say 0.9). The difference matters — top-k always keeps a fixed count, while top-p adapts: few tokens when the model is confident, many when it's unsure. In practice these stack with temperature: truncation guards quality (cut the junk tail), then temperature dials variety within what's left.
# probs sorted high -> low:
# the=0.60 a=0.20 an=0.09 one=0.06 ...long junk tail...
# top-k = 2 -> keep {the, a}, renormalize, sample
# top-p = 0.9 -> keep {the, a, an, one} (0.60+0.20+0.09+0.06 >= 0.9)
#
# When the model is SURE (the=0.97), top-p keeps just 1 token.
# When it's UNSURE (many near-ties), top-p keeps a wide set.The model is very confident: `the` has probability 0.97 and everything else is tiny. Using top-p = 0.9, roughly how many tokens end up in the sampling pool?
#When to turn the dial which way
There's no universally 'correct' temperature — it depends entirely on the job:
- Low `T` (~0–0.3): factual Q&A, extraction, classification, math, code, strict formats. You want the single best answer, reproducibly.
- Medium `T` (~0.7–1.0): general chat, explanations, summaries, everyday writing — the sweet spot most chat products default to.
- High `T` (~1.2+): brainstorming, poetry, character dialogue, wild idea generation — when surprise is a feature and the occasional miss is fine.
One caveat for 2026: many frontier reasoning models manage sampling internally and expose temperature only weakly (or ignore it), so turning the knob may barely move the output. When temperature is exposed — classic chat/completion endpoints, open-weight models — everything here applies directly. Want to feel the dial? Use the interactive visualizer below to drag the temperature slider and watch the same set of logits melt from a sharp spike into a flat plateau — and see how the sampled token changes as you do.
Key takeaways
- Greedy decoding always takes the top token: deterministic, safe, but prone to flat, repetitive text.
- Sampling draws from the probability distribution like weighted dice, giving text variety and creativity.
- Temperature divides logits before softmax: low T sharpens toward the top token (safe), high T flattens the distribution (creative but error-prone).
- Top-k keeps a fixed number of top tokens; top-p (nucleus) keeps the smallest set summing to probability p — both prune the low-quality tail before sampling.
- Use low temperature for facts, code, and format-following; higher temperature for brainstorming and creative writing.
Low temperature → the distribution sharpens toward the top token (safe, repetitive). High temperature → it flattens, so rarer tokens like “purple” can slip in (creative, riskier).
These are the model's next-token probabilities. You call it with greedy decoding (temperature effectively 0). Which token comes out?
probs = {
"blue": 0.52,
"green": 0.31,
"red": 0.14,
"plaid": 0.03,
}
# greedy decoding: no randomness
next_token = argmax(probs)A teammate is tuning a customer-support bot that must give accurate, consistent answers. They wrote this note. One line is a real misconception. Which one is WRONG?
# Notes on decoding settings:
1. Temperature rescales the logits before softmax.
2. Lower temperature sharpens the distribution toward the top token.
3. For factual, reproducible answers, we should use a HIGH temperature.
4. Top-p keeps the smallest set of tokens summing to p, then samples from those.Complete the description of how temperature reshapes the distribution.
Temperature divides every logit by T softmax. A LOW T stretches the gaps apart, which the distribution toward the top token. A HIGH T squishes the gaps together, which the distribution toward uniform.
Put the steps of temperature-scaled nucleus (top-p) sampling in the correct order, starting from the raw logits.
Start with the raw logits from the model's output head
Renormalize the kept tokens so their probabilities sum to 1
Apply softmax to get a probability distribution
Sort tokens high-to-low and keep the smallest set whose probabilities sum to p (top-p)
Divide every logit by the temperature T
Sample one token from that truncated distribution
Reason about temperature by hand — no computation needed, just the direction of the effect.
A model is choosing the next token after "The capital of France is". Its top four candidate tokens have these logits:
| token | logit | |---|---| | Paris | 9.0 | | Lyon | 4.0 | | Marseille | 3.5 | | baguette | 1.0 |
Part A. Under greedy decoding, which token is chosen, and will the answer ever change if you run it again?
Part B. You switch to sampling. To make the output as safe and reproducible as possible (you want Paris essentially every time), should you set temperature low or high? Explain what happens to the four probabilities.
Part C. A colleague sets temperature = 2.5 for this factual question and is surprised to sometimes see Lyon or even baguette. In one or two sentences, explain why that happens and whether it means the model 'forgot' the right answer.
Try it yourself — a starting point to build on:
# Write your solution here