Next-Token Prediction
How an LLM writes: it predicts one token at a time, appends it, and feeds everything back in — a loop that turns a probability distribution into fluent text.
Under all the magic, a large language model does exactly one thing: given some text, it guesses what comes next. Not the next word, not the next sentence — the next token (a word, a word-piece, or a punctuation mark). Everything an LLM produces, from a haiku to a working program, is that single guess repeated over and over. This lesson opens up that loop and shows you the machinery.
#From the last layer to a number per token
Your prompt flows up through the transformer stack. What comes out the top is a vector for the final position in the sequence — a compressed summary of "everything I've read so far." One last matrix multiply (the output head) turns that vector into a list of raw scores called logits — exactly one score for every token in the model's vocabulary. Modern vocabularies are big: often 100,000 to 200,000+ tokens, so this is a very long list.
# One logit per vocabulary token — higher = the model likes it more
logits = model(prompt)[-1] # take the LAST position's scores
# vocab of ~128,000 -> a list of ~128,000 raw numbers
logits # e.g. [ ... , 8.2 ("mat"), 6.1 ("rug"), 1.3 ("banana"), ... ]Logits are like applause meters
Picture every token in the vocabulary as a contestant on a stage. The logit is how loudly the model claps for each one. mat gets thunderous applause (8.2), banana gets a polite pat (1.3). Logits can be any real number — positive, negative, huge, tiny. They aren't probabilities yet; they're just relative enthusiasm.
#Softmax: turning scores into probabilities
Raw logits are hard to reason about. Softmax converts them into a proper probability distribution: every value lands between 0 and 1, and they all add up to exactly 1. It works by exponentiating each logit (so bigger scores pull ahead sharply) and then dividing by the total. The result is the model's honest answer to "what fraction of the time should each token come next?"
logits = [8.2, 6.1, 1.3] # "mat", "rug", "banana"
# softmax: exponentiate, then normalize so it sums to 1
# e^8.2 = 3641, e^6.1 = 446, e^1.3 = 3.7 (total = 4091)
probs = [0.890, 0.109, 0.001] # "mat" 89%, "rug" 11%, "banana" 0.1%A model's top three logits are 8.2, 6.1, and 1.3. After softmax, what must be true of the resulting probabilities?
#Picking a token, then looping: autoregressive generation
Now the model has a probability for every possible next token and must choose one. The simplest strategy is greedy decoding: always take the highest-probability token — the argmax, which for our numbers is always mat. Greedy is deterministic (same prompt → same output every time) and cheap; next_token = argmax(probs). (It's not the only option — a model can also sample so lower-probability tokens sometimes win, controlled by a temperature knob. That's the next lesson; here, just remember greedy = argmax.)
Here's the crucial move. The model doesn't predict a whole sentence at once. It predicts one token, glues it onto the end of the text, and then feeds the entire new sequence back in to predict the token after that. Because each prediction depends on everything generated so far — including the model's own previous outputs — this is called autoregressive generation. The word 'auto' means 'self': the model conditions on its own past.
tokens = tokenize("The cat sat on the")
while True:
logits = model(tokens)[-1] # scores for the next token
probs = softmax(logits) # -> a probability distribution
nxt = argmax(probs) # greedy pick (the top token)
tokens.append(nxt) # append it to the running sequence
if nxt == END_OF_TEXT or len(tokens) >= MAX_LEN:
break # stop token OR length limit
# tokens now reads: "The cat sat on the mat"The model never plans ahead
Think of it as phone-keyboard autocomplete cranked to genius level — but instead of forgetting, it re-reads the whole message every step before suggesting the next piece. It's tempting to imagine the model drafting a full answer and then typing it out. It doesn't. At each step it sees only the tokens so far and commits to one next token with no ability to revise. A coherent paragraph emerges purely because each locally-good token, chained together, tends to stay on track — which is also why an early wrong turn can snowball, since the model keeps conditioning on its own mistake.
In autoregressive generation, what happens immediately after the model picks a token?
#Knowing when to stop
A loop needs an exit. Generation halts on whichever comes first: (1) the model emits a special stop token (often called end-of-text or end-of-turn) — its learned way of saying "I'm done," or (2) it hits a maximum length cap set by the caller (like max_tokens). This is why a response can end naturally with a period, or get abruptly chopped off mid-word when it runs into a length limit.
Want to see the loop breathe? Use the interactive visualizer below to step through the probabilities and watch a sentence write itself one token at a time.
Key takeaways
- An LLM does one thing repeatedly: predict the next token given all the text so far.
- The output head produces a logit for every token in the vocabulary; softmax turns those logits into a probability distribution that sums to 1.
- Greedy decoding always picks the highest-probability token (the argmax) — deterministic and simple.
- Generation is autoregressive: pick a token, append it, feed the whole sequence back in, and repeat.
- The loop stops at a special stop token or when it hits a maximum-length limit.
The model scores every possible next token; it picks “The”, appends it, and repeats — one token at a time.
The model's output head produced these logits for the next token. Under greedy decoding, which token is chosen?
logits = {
"sat": 7.4,
"ran": 7.9,
"slept": 2.1,
"the": -3.0,
}
next_token = argmax(softmax(logits))Put the steps of the autoregressive generation loop in the correct order for producing ONE token.
Append the chosen token to the sequence
Check the stop condition: stop token emitted or max length reached
Feed the current sequence of tokens through the model
Pick the next token (e.g. argmax for greedy decoding)
Take the final position's output and compute one logit per vocabulary token
Apply softmax to turn the logits into a probability distribution
A learner explains next-token prediction like this. One sentence contains a real misconception. Which one is WRONG?
# A learner's notes on how an LLM generates text:
1. The model reads the prompt and predicts the next token.
2. Softmax turns the logits into probabilities that sum to 1.
3. Greedy decoding picks the token with the highest probability.
4. The model plans the whole answer up front, then types it out.Fill in the two blanks that describe how the generation loop ends.
The loop keeps generating until one of two things happens: (1) the model emits a special token that means "I'm done", or (2) the sequence reaches the maximum allowed by the caller.
Trace the loop by hand.
A tiny model has a 3-token vocabulary: A, B, <END>. Its behavior is fully described by this rule table — given the current last token, here are the next-token logits it produces:
| Last token | logit(A) | logit(B) | logit(\<END\>) | |---|---|---|---| | (start) | 3.0 | 1.0 | 0.0 | | A | 0.5 | 4.0 | 0.5 | | B | 0.5 | 0.5 | 5.0 |
Using greedy decoding (always take the argmax), start from (start) and write out the full sequence the model generates, stopping when it emits <END>. You do NOT need to compute exact softmax values — argmax of the probabilities is the same as argmax of the logits, so you can just pick the largest logit in each row.
Try it yourself — a starting point to build on:
# Write your solution here