Inside the ModelIntermediate⏱ 9 min04 / 12

The Transformer

Meet the transformer — the stack-of-blocks architecture behind every modern LLM, where each layer mixes tokens with attention and refines them with a small neural network.

Every modern large language model — GPT, Claude, Gemini, Llama — is built on the same core design: the transformer. It was introduced in a 2017 paper with a bold title, "Attention Is All You Need", and it has dominated the field ever since.

In earlier lessons you saw how text becomes tokens, and how each token becomes a list of numbers called an embedding. This lesson is about what happens next: how the transformer takes those vectors and, layer after layer, turns them into a prediction for the next token.

Think of it like

A factory assembly line

Picture a long assembly line. A row of tokens rides in on a conveyor belt. At each station (a transformer block), every token gets to look at all the others, gather what it needs, and then get a little polish from a small workshop. The line is made of many identical stations stacked one after another — and by the end, the tokens are refined enough to predict what comes next.

#The Big Picture: A Stack of Identical Blocks

The single most important mental model is this: a transformer is a tall stack of identical blocks.

Input text flows in at the bottom, passes up through every block in order, and a prediction comes out the top. Each block has the same shape and the same two ingredients — but its own learned settings, so each one does a slightly different job. The overall flow looks like this:

End-to-end flow. The middle — the stack of blocks — is the transformer.
Text
  -> tokens                      ("The cat sat" -> [The][ cat][ sat])
  -> token embeddings            each token becomes a vector
  -> + positional information     so the model knows the order
  ---------------------------------------------
  -> Transformer Block 1   \
  -> Transformer Block 2    |  a stack of N identical blocks
  -> ...                    |  (N might be 12, 32, 80, 100+)
  -> Transformer Block N   /
  ---------------------------------------------
  -> final layer  ->  scores over the whole vocabulary
  -> the next token

Depth matters. A small model might have 12 blocks; a large one can have 80, 100, or more. Early blocks tend to pick up simple, local patterns (grammar, nearby words); deeper blocks build richer, more abstract features (who she refers to, the topic, the intended tone). Stacking blocks is how a transformer builds understanding in layers. Use the interactive visualizer below to watch a sequence of tokens flow up through this stack as you read the rest of the lesson.

First, the input. Before the stack, two things happen to prepare each token:

  1. Token embeddings — each token id is looked up in a big table and turned into a vector (a list of numbers, maybe 768 or 4096 long). This vector is the token's starting "meaning."
  2. Positional information — a transformer sees all tokens at once, so on its own it has no idea what order they came in. We add position information to each embedding so that "dog bites man" and "man bites dog" look different to the model.
Common mistake

Without position, word order vanishes

This surprises people: a raw transformer treats its input like a bag of tokens with no order. Skip the positional-information step and "Alice paid Bob" and "Bob paid Alice" become completely indistinguishable to the model — same tokens, no sense of order. Position info is what rescues word order.

#Inside One Block: Two Ingredients

Every transformer block is made of exactly two main parts, run in sequence:

  1. Self-attention — the communication step. Each token looks at all the other tokens and pulls in relevant context. This is how the token for "it" figures out whether it refers to "the cat" or "the mat". (Attention gets its own full lesson next.)
  2. A feed-forward network (MLP) — the thinking step. After gathering context, each token is passed through a small two-layer neural network that transforms it on its own. This is where a lot of the model's learned knowledge lives.

A handy slogan: attention lets tokens talk to each other; the feed-forward network lets each token think for itself. In pseudocode, one block looks like this:

A block takes a sequence in and returns a same-shaped, more refined sequence out.
# One transformer block, in pseudocode.
# `x` is the whole sequence of token vectors.

def transformer_block(x):
    # 1. Communication: tokens gather context from each other
    x = x + self_attention(layer_norm(x))

    # 2. Computation: each token is refined on its own
    x = x + feed_forward(layer_norm(x))

    return x  # same shape as the input, but richer

# The full model just runs many of these in a row:
for block in blocks:      # e.g. 32 blocks
    x = block(x)
Quick check

A transformer block contains a self-attention part and a feed-forward (MLP) part. Which statement best describes their division of labor?

Two small but crucial details appear in that pseudocode — x = x + (...) and layer_norm(x) — and they're the plumbing that makes deep stacks trainable:

  • Residual connections — the x = x + (...) pattern. Instead of replacing a token's vector, each sub-layer adds its result to what was already there. Information can flow straight up the stack untouched, and each block only has to learn a small adjustment. Without residuals, stacking dozens of blocks would cause the signal to degrade and training to fail.
  • Layer normalization — a rescaling step applied before each sub-layer. It keeps the numbers in a healthy, stable range so a stack of 80+ blocks doesn't blow up or collapse to zero during training.

You don't need the math — just remember: residuals let information skip ahead, and layer norm keeps the numbers well-behaved. It's like editing with track changes on: each block suggests edits on top of the document rather than rewriting it from scratch, so nothing important gets lost passing through editor after editor.

After the input has passed through every block, a final layer takes the vector for the last position and produces one score for every token in the vocabulary (which might be 50,000 to 200,000+ tokens). These raw scores are called logits. Turn the logits into probabilities and you have the model's guess for the next token: "The capital of France is" produces a high score for " Paris" and low scores for almost everything else. Choosing which token to actually emit is the job of sampling — a later lesson.

#Why This Beat the Old Approach

Before 2017, the leading models for text were RNNs (recurrent neural networks), which read one token at a time and passed a running summary forward. Transformers won for two reasons:

  • Parallelism. Because attention looks at all tokens at once, a transformer can process a whole sequence in parallel on a GPU. RNNs are stuck going step by step. This made it practical to train on internet-scale data.
  • Direct long-range links. In an RNN, connecting the first and last words of a long paragraph means passing information through every word in between — and it fades. Attention gives any token a direct line to any other, no matter how far apart.

That combination — parallel training plus direct connections — is what unlocked the enormous models we have today.

Quick check

Why can a transformer process all the tokens in a sentence at the same time, while older RNNs had to go one token at a time?

Tip

Put it all together

That's the whole architecture at a high level: prepare the input (embeddings + position), run it up a tall stack of identical blocks (attention + feed-forward, wrapped in residuals and layer norm), and read off scores over the vocabulary at the top. Everything else in an LLM is a variation on this theme — and the next lesson zooms all the way into the star of the show, attention.

Key takeaways

  • A transformer is a tall stack of identical blocks; input flows up from embeddings to a prediction, and depth (many layers) is what builds richer, more abstract understanding.
  • Each block has two parts: self-attention (tokens communicate and gather context) and a feed-forward network (each token is refined on its own) — communicate, then compute.
  • Residual connections (add, don't replace) and layer normalization are the plumbing that make very deep stacks trainable.
  • Because a transformer sees all tokens at once, it must add positional information — otherwise word order is lost.
  • Attention gives any token a direct link to any other and lets the model process a whole sequence in parallel — the two advantages that made transformers beat older step-by-step RNNs.
Try it yourself · Through the stack
Watch a signal rise layer by layer into a next-token prediction.
Embeddings
The
cat
sat
___
Block 1
The
cat
sat
___
Block 2
The
cat
sat
___
Block 3
The
cat
sat
___
Output
→ on

Each token becomes an embedding vector — the model's input.

step 1 / 5
Practice challenges
Test yourself · earn XP
0/4
Reorder the lines#1

Put the stages of a transformer in the correct order, from raw text at the bottom to the predicted next token at the top.

1
Look up each token to get its embedding vector
2
Split the text into tokens
3
Add positional information so word order is preserved
4
Pass the sequence up through the stack of identical blocks
5
The final layer produces a score for every token in the vocabulary
6
Pick the next token from those scores
Fill in the blank#2

Complete the pseudocode for one transformer block. The two main ingredients are self-attention and the feed-forward network, and each is wrapped in a residual connection (add its result back onto x).

def transformer_block(x):
    # tokens communicate and gather context
    x = x + (layer_norm(x))
    # each token is refined on its own
    x = x + (layer_norm(x))
    return x
Predict the output#3

A transformer receives its input but, due to a bug, positional information is NEVER added to the embeddings. The model is then asked to distinguish these two sentences: A: "dog bites man" B: "man bites dog" What happens?

predict-output
# positional information step was skipped!
embeddings = lookup(tokens)   # just the token vectors
# (no  + positional_info  line)
x = run_transformer(embeddings)
Fix the bug#4

A learner explains why transformers replaced older RNNs like this: "Transformers are faster because each token only ever looks at the single token right before it, so there's less work to do." What's the correction?

fix-bug
# The learner's claim:
# "Each token only looks at the ONE token before it,
#  which is why transformers are fast and better than RNNs."
Your turn
Practice exercise

No code to run here — just trace the design in your head and on paper.

A tiny transformer has 4 blocks. You feed it the tokens for the sentence:

"The trophy did not fit in the suitcase because it was too big."

  1. List, in order, everything that happens to this input before it reaches Block 1.
  2. For the token "it", which of the two ingredients inside a block — self-attention or the feed-forward network — is responsible for figuring out that "it" refers to "trophy" (and not "suitcase")? Explain in one sentence why.
  3. Suppose someone deletes the residual connections (x = x + ... becomes just x = ...). Name one thing that would get worse, and say whether it would affect a 4-block model or a 100-block model more.

Try it yourself — a starting point to build on:

starter.py
# Write your solution here