How LLMs Are Trained
Pretraining teaches an LLM by making it predict the next token across trillions of words — no human labels needed, because the text supplies its own answers.
You know that an LLM predicts the next token. But how does it learn to make those predictions well? Nobody sits down and hand-labels the right answer for every sentence on the internet. The trick is elegant: the model teaches itself by playing a fill-in-the-blank game with ordinary text, billions of times. This stage is called pretraining, and it's where a random pile of numbers slowly turns into something that has read most of the web.
#The core idea: the text is its own answer key
Training a model usually needs labeled examples — a photo tagged "cat," a review tagged "positive." Labels are expensive; humans have to make them. Pretraining sidesteps this entirely with self-supervised learning: take any sentence, hide the next token, and ask the model to guess it. The hidden token is the label — and it came free with the text.
Because every position in every document gives one more (context → next token) example, a single web page yields thousands of training examples, and the whole internet yields trillions. No annotators required.
A book with the answer taped to the next line
Imagine studying from a book where the answer to each question is printed on the following line. You cover it, guess, then slide your hand down to check. Every sentence is a fresh quiz that grades itself — that's self-supervision. The model quizzes itself on real text, and the actual next word tells it whether it was right.
#Measuring surprise: the loss
To learn, the model needs a number that says how wrong it was. That number is the loss, and for next-token prediction we use cross-entropy loss. In plain terms, it measures how surprised the model was by the true next token.
Remember that the model outputs a probability for every possible next token. Cross-entropy just looks at the probability it assigned to the correct one, via -log(p). If it was confident and right, the loss is tiny. If it was confident and wrong, the loss is huge — the log shape makes the penalty explode as confidence in the truth drops toward zero, so the model is punished hardest for being confidently wrong.
# The true next token is "mat". What probability did the model give it?
# guess P("mat") loss = -log(P("mat"))
# confident-right 0.90 0.11 <- barely surprised
# unsure 0.20 1.61 <- fairly surprised
# confident-wrong 0.01 4.61 <- very surprised = big penalty
loss = -log(prob_of_correct_token) # cross-entropy, one token#How the parameters actually change
The model's knowledge lives in its parameters (also called weights) — billions of numbers that start out random. Training nudges each one, over and over, to make the loss smaller. Two ideas do the heavy lifting:
- Backpropagation figures out, for every single parameter, which direction to nudge it (up or down) to reduce the loss. It's calculus running backward through the network.
- Gradient descent takes a small step in that direction. Do this on batch after batch of text and the loss drifts steadily downward.
The step size is the learning rate — too big and training becomes unstable; too small and it crawls.
Picture a mixing board with billions of knobs and one dial reading "how wrong am I." Backprop whispers which way to nudge every knob at once; a gradient step makes all those tiny adjustments; the dial ticks down. Nobody says what any single knob means — the meaning emerges from relentlessly chasing a lower number.
params = random_init() # billions of weights, all garbage at first
for step in range(num_steps): # often hundreds of thousands of steps
batch = sample(corpus) # a chunk of text (many sequences)
preds = model(batch, params) # predict next token at every position
loss = cross_entropy(preds, batch.next_tokens) # how surprised were we?
grads = backprop(loss, params) # which way to nudge each parameter
params = params - lr * grads # gradient descent: take one small step
# after enough steps, params encode a huge amount about languageWhy is pretraining called 'self-supervised' rather than 'unsupervised'?
#Scale is the secret ingredient
One next-token guess teaches the model almost nothing. The magic comes from doing it at staggering scale. Pretraining runs for weeks or months on thousands of specialized chips — GPUs or TPUs — chewing through many passes over a corpus of trillions of tokens.
Crucially, researchers found this improvement is predictable. Scaling laws show that as you increase three things together — the number of parameters, the amount of data, and the compute you spend — the loss keeps dropping in a smooth, forecastable curve. That predictability is why labs invest so heavily up front: they can estimate how good a model will be before building it.
Scale isn't just 'make it bigger'
It's a myth that you only grow the model. A giant model trained on too little data is undertrained and wastes its capacity. Scaling laws are about balance: parameters, data, and compute must grow together. Throwing more parameters at a fixed dataset hits diminishing returns fast.
#What you get: a base model
After pretraining, you have a base model (also called a foundation or pretrained model). It's an extraordinary text completer: give it the start of anything and it continues in a statistically plausible way. It has absorbed grammar, facts, reasoning patterns, code, and countless writing styles.
But it is not yet a helpful assistant. Ask a raw base model "What is the capital of France?" and it might reply with more questions — because on the web, questions are often followed by more questions, not answers. It completes text; it doesn't reliably follow instructions or behave like a chatbot. Turning it into the polite assistant you actually chat with takes a second stage — post-training (instruction tuning and techniques like RLHF), a lesson of its own. The one-liner to remember: pretraining builds raw knowledge; post-training shapes behavior.
You type 'What is the capital of France?' into a raw base model (no post-training). What's the most likely reason it might NOT just answer 'Paris'?
Key takeaways
- Pretraining is self-supervised: the next token in ordinary text acts as the label, so no human annotation is needed and the whole web becomes training data.
- Cross-entropy loss measures how surprised the model was by the true token; backpropagation plus gradient descent nudge billions of parameters to shrink that surprise, batch after batch.
- Scaling laws say that growing parameters, data, and compute together yields predictably better models — but they must grow in balance, not just bigger.
- Training runs for weeks or months on thousands of GPUs/TPUs, making many passes over trillions of tokens.
- The output is a base model: a superb text completer that has absorbed knowledge but is not yet a helpful, instruction-following assistant (that comes from post-training).
Put the steps of ONE pretraining update in the correct order.
Compute the cross-entropy loss against the true next tokens
Take a gradient-descent step: nudge each parameter to lower the loss
Backpropagate to get a gradient for every parameter
Sample a batch of text from the corpus
Run the model to predict the next token at every position
The true next token is `mat`. Two model snapshots assign it different probabilities. Cross-entropy loss is `-log(P(correct))`. Which snapshot has the LOWER loss on this token?
true_next_token = "mat"
snapshot_A = { "mat": 0.10, ... } # P("mat") = 0.10
snapshot_B = { "mat": 0.70, ... } # P("mat") = 0.70
loss = -log(P_of("mat"))A learner summarizes how LLMs are trained. One line is a real misconception. Which one is WRONG?
# Notes on pretraining:
1. It's self-supervised: the next token in the text acts as the label.
2. Cross-entropy loss measures how surprised the model was by the true token.
3. Backprop + gradient descent nudge the parameters to lower the loss.
4. Pretraining alone produces a polite assistant ready to follow instructions.Fill in the blanks about what makes models better at scale.
Scaling laws say that as you grow three things together — the number of , the amount of training , and the you spend — the loss drops in a smooth, predictable curve.
Reason about the loss, no computation required beyond comparing numbers.
The true next token in a sentence is Paris. Three snapshots of the same model — early in training, mid-training, and late in training — assign these probabilities to Paris:
| Snapshot | P("Paris") | |---|---| | Early | 0.02 | | Mid | 0.30 | | Late | 0.85 |
Cross-entropy loss for this token is -log(P("Paris")).
- Without computing exact values, rank the three snapshots from highest loss to lowest loss.
- Explain what is happening to the model between the Early and Late snapshots, in terms of 'surprise.'
- A friend says: 'The Late model has 0.85 on Paris, so its loss on this token is basically zero — training is done.' What's wrong with that reasoning?
Try it yourself — a starting point to build on:
# Write your solution here