What you are looking at
At every position a model outputs a score, a logit, for every token in its vocabulary, which for current models runs from tens of thousands of tokens to a couple of hundred thousand. Twelve candidates stand in for that list here. Their scores are made up for this lab; everything done to them is what a real sampler does. Tokens usually carry their leading space, which is why the list marks it with ␣.
Softmax turns scores into odds: p = exp(z / T) / Σ exp(z / T), where T is the temperature. The faint bars are the odds at temperature 1; the solid bars are the odds after your settings, the ones the sampler actually draws from.
Temperature
Temperature divides every score before the softmax. Below 1 the gaps between scores grow and the top token takes over; at 0 the sampler always takes the top token, which is called greedy decoding. Above 1 the gaps shrink and unlikely tokens get a real chance: press Wild and sample a few hundred times, and even “panic” turns up. Temperature never changes the order of the tokens, only how far apart their odds are.
Top-k and top-p
Both cut off the long tail before the draw and share its odds among the tokens that are left.
- Top-k keeps the k likeliest tokens however sure the model is. When it is confident, k = 10 still lets in tokens it all but ruled out; when it is torn, k = 3 can cut good options.
- Top-p, or nucleus sampling, keeps the smallest set of top tokens whose odds add up to p. The set shrinks when the model is sure and grows when it isn't, which is why it is the more common choice.
This lab applies temperature first, then top-k, then top-p, the order Hugging Face Transformers uses. Set temperature to 1.6 and top-p to 0.80 and watch how many tokens survive, then lower the temperature.
Choosing settings
- Extraction, classification, code: temperature 0 or close to it, so repeated calls agree.
- Chat: somewhere around 0.7.
- Brainstorming: higher, with top-p around 0.9 so the tail stays out.
Change one dial at a time: OpenAI's API reference recommends altering temperature or top-p, but not both. And temperature 0 is not a promise of identical output. Batching and floating-point differences on the server can still tip a close call between two tokens the other way.
In an interview
Questions this lab prepares you for:
- What is a logit, and what does softmax do to it?
- What does temperature do mathematically, and what happens at 0?
- How do top-k and top-p differ, and when would you pick each?
- How does sampling differ from beam search? (Beam search keeps the few best partial sequences: good for translation, flat and repetitive for open-ended text.)
- Does temperature 0 stop hallucinations? (No. The model takes its likeliest token every time; if that token is wrong, it is wrong every time.)
The code
The sampler from the lab, in NumPy: temperature, top-k, top-p, then one draw.
import numpy as np def sample(logits, temperature=1.0, top_k=None, top_p=1.0, rng=np.random.default_rng()): logits = np.asarray(logits, dtype=float) if temperature == 0: return int(np.argmax(logits)) z = logits / temperature order = np.argsort(z)[::-1] if top_k: order = order[:top_k] p = np.exp(z[order] - z[order].max()) p = p / p.sum() keep = np.searchsorted(np.cumsum(p), top_p) + 1 order, p = order[:keep], p[:keep] / p[:keep].sum() return int(rng.choice(order, p=p))