LLM FoundationsBeginner⏱ 8 min01 / 12

What Is an LLM?

A Large Language Model is a neural network trained to predict the next token — and at massive scale, that one simple goal produces fluent, reasoning, coding-capable AI like Claude and GPT.

You've talked to Claude or ChatGPT. You've watched it write an essay, fix your code, or explain a hard idea in plain words. It feels like there's someone in there. But under the hood, an LLM is doing something almost comically simple, over and over, faster than you can blink.

This lesson answers the big question at the heart of the whole course: what actually is a Large Language Model? Once this clicks, everything else — tokens, attention, training, prompting — is just detail on top of one core idea.

#The one-sentence definition

A Large Language Model (LLM) is a neural network trained to predict the next token given the tokens before it.

That's the whole objective. Not "understand language," not "answer questions," not "be helpful." Just: given this chunk of text, what token most likely comes next? Everything impressive an LLM does is a side effect of getting extremely good at that single guessing game.

A token is a chunk of text — often a word or a piece of a word (you'll go deep on this in the next lesson). For now, picture the model reading text left to right and, at every step, filling in the blank:

`` The cat sat on the ___ ``

Given The cat sat on the, a well-trained model puts high probability on mat, some on floor or roof, and almost none on banana. It doesn't return one answer — it returns a probability for every possible next token.

An LLM's real output: a probability over every token it knows, not a single word

1
2
3
4
5
6
7
8
9
10
11
12prompt = "The cat sat on the"

# The model outputs a probability distribution
# over its entire vocabulary:
next_token_probs = {
    "mat":    0.41,
    "floor":  0.18,
    "roof":   0.07,
    "couch":  0.05,
    "banana": 0.00003,
    # ...tens of thousands more tokens...
}

Think of it like

Autocomplete on steroids

You already know a baby version of this: your phone's keyboard suggesting the next word. An LLM is that same idea — autocomplete — but scaled up almost unimaginably.

Your phone learned from a modest amount of text and predicts one word at a time with a tiny model. An LLM learned from trillions of tokens with billions of parameters. Cross enough scale and the humble "guess the next word" trick stops looking like autocomplete and starts looking like intelligence.

#One token at a time: the generation loop

If the model only predicts one token, how does it write whole paragraphs? It loops. It predicts a token, sticks it onto the end of the text, and feeds the longer text right back in to predict the next one. This is called autoregressive generation — each new token depends on everything that came before it.

Generation is a loop: predict, append, feed back in, repeat

1
2
3
4
5
6
7
8
9
10
11text = "The cat sat on the"

for step in range(4):
    probs = model.predict_next(text)   # distribution over all tokens
    token = pick(probs)                # choose one (e.g. the likeliest)
    text = text + " " + token          # append it, then repeat

# step 0 -> "mat"    text = "The cat sat on the mat"
# step 1 -> "."      text = "The cat sat on the mat."
# step 2 -> "It"     text = "The cat sat on the mat. It"
# step 3 -> "was"    text = "The cat sat on the mat. It was"

Note

See it in motion

Use the interactive visualizer below to watch the loop run: type a prompt, and step through the model predicting a token, appending it, and predicting again. Watching the probabilities shift as the text grows is the fastest way to make "next-token prediction" feel real.

Quick check

What is an LLM fundamentally trained to do?

#Why scale changes everything: emergence

Here's the surprising part. "Predict the next token" sounds too simple to produce anything smart. But to predict the next token really well across all of human writing, the model is quietly forced to learn a staggering amount:

To finish 2 + 2 = it has to learn arithmetic.
To finish The French word for cat is it has to learn translation.
To finish a half-written function it has to learn how code works.
To finish The detective realized the killer was it has to track plot, motive, and logic.

At small scale these abilities are absent. Crank up the size and training data, and they appear — often suddenly. Capabilities that weren't explicitly trained for, showing up on their own, are called emergent abilities.

Tip

"Large" is doing real work in the name

The word Large isn't marketing. Reasoning, multi-step problem solving, and coding tend to not exist in small models and then emerge past certain scales of parameters and data. The same next-token objective at a bigger scale is a genuinely different beast — which is exactly why we say Large Language Model.

#A crucial nuance: patterns, not a database

It's tempting to imagine the model has a giant lookup table of facts inside it. It doesn't. An LLM is a fixed set of numbers (its parameters, or weights) that encode patterns of language — how words, ideas, and structures tend to relate. When you ask a question, it isn't retrieving a stored record; it's generating the most plausible continuation based on those learned patterns.

Common mistake

This is why models can 'hallucinate'

Because an LLM produces the most plausible-sounding continuation — not a verified lookup — it can state something false with total confidence. A fake citation or a made-up API method is just a very likely-looking sequence of tokens that happens not to be true.

That's not a bug you can fully patch away; it's a direct consequence of what an LLM is. You'll dig into this in the Limitations & Hallucinations lesson.

#Where this course goes next

Every remaining lesson zooms in on one part of the machine you just met:

Tokens & Tokenization — how text becomes the chunks the model reads.
Embeddings — how each token turns into a vector of numbers with meaning.
The Transformer — the architecture that processes those vectors.
Attention — how the model decides which earlier tokens matter right now.
Next-Token Prediction & Sampling — turning the output distribution into actual text.
Training — pretraining, fine-tuning, and RLHF that shape the weights.
Using LLMs — context windows, prompting, and the limits that make tools (like MCP) necessary.

Modern assistants like Claude (Anthropic) and GPT (OpenAI) are exactly this: giant transformers trained on next-token prediction, then polished to be helpful. Keep the core idea in your pocket and the rest of the course is just filling in how.

Key takeaways

An LLM is a neural network with one training objective: predict the next token given the tokens before it.
It generates text autoregressively — predict a token, append it, feed it back in, and loop — so long outputs are just that one step repeated.
The model outputs a probability distribution over every token it knows, not a single fixed answer.
At massive scale (billions of parameters, trillions of tokens), abilities like reasoning, translation, and coding emerge from that simple objective.
An LLM stores learned patterns of language in its weights, not a database of facts — which is why it's fluent but can also confidently hallucinate.

Try it yourself · Predict the next token

The one thing an LLM does — score the next token and repeat.

The

38%

22%

14%

The model scores every possible next token; it picks “The”, appends it, and repeats — one token at a time.

token 1 / 6

Practice challenges

Test yourself · earn XP

0/4

Predict the output#1

An LLM is given the prompt below and returns a probability for each candidate next token. Which token will it pick if it always chooses the single most likely one?

predict-output

1
2
3
4
5
6
7
8prompt = "Roses are red, violets are"

next_token_probs = {
    "blue":   0.79,
    "violet": 0.06,
    "purple": 0.05,
    "red":    0.03,
}

Fix the bug#2

A friend describes how an LLM answers a question. Their mental model has a misconception. Which correction is right?

fix-bug

1
2
3
4# Friend's explanation:
# "When you ask an LLM a question, it searches a huge
#  database of facts stored inside it, finds the matching
#  record, and returns that stored answer word for word."

Fill in the blank#3

Fill in the blanks to complete the core definition of a Large Language Model.

A Large Language Model is a neural network trained
to predict the next  given the ones before it.
It generates long text : predict, append, repeat.

Reorder the lines#4

Put the steps of the LLM generation loop in the correct order for producing one new token and continuing.

The model outputs a probability distribution over all possible next tokens

Append the chosen token to the text

Feed the now-longer text back in to predict the next token

Feed the current text into the model

Choose one token from that distribution

Your turn

Practice exercise

No code to run — just reason it through on paper.

Suppose an LLM has this prompt: The capital of France is.

Describe, in your own words, what the model produces at this single step. (What is the shape of its output — one word, or something bigger?)
Sketch a plausible mini probability distribution over 4 candidate next tokens, and say which one the model is most likely to pick.
Now the model appends its chosen token and runs again. Write out what the new input to the model is for the second step, and name what this predict-append-repeat process is called.
Bonus: this model was never given a table of country -> capital facts. Explain in one or two sentences how it can still answer correctly.

Try it yourself — a starting point to build on:

starter.py

1# Write your solution here