Attention
How each token decides which other tokens to look at — the self-attention mechanism at the heart of every transformer.
Read this sentence: "The cat sat down because it was tired." What does it refer to? Obviously the cat. But how does a model know that? The word it on its own is nearly meaningless — its meaning is borrowed entirely from another word nearby. Attention is the mechanism that lets a token reach back through the sentence, find the words that matter, and pull their meaning into itself. It is the single most important idea inside a transformer.
#The question every token asks
After tokenization and embeddings, every token is just a vector — a list of numbers capturing what it means. But that meaning starts out context-free: the embedding for it is the same whether it refers to a cat, a car, or a country. Attention fixes this. For each token, the model asks one question:
> *"Given everything I've read so far, which other tokens should I look at to understand this one — and how much?"*
It then builds an updated representation of the token by mixing in information from the tokens it chose to look at, weighted by how relevant each one is. After attention, the vector for it is no longer generic — it now carries a strong dose of cat.
A room full of people
Imagine each token is a person in a room, and everyone is trying to figure out what they mean in this particular conversation. Each person holds up a little sign describing what they're looking for (a query). Everyone else holds up a sign describing what they offer (a key). You scan the room, find the people whose "offer" best matches your "looking for," and lean in to hear what they actually have to say (their value). Attention is that whole glance-and-listen loop, done by every token at once.
#Query, Key, Value: the three roles
Attention gives every token three different vectors, each produced by multiplying the token's embedding by a learned weight matrix:
- Query (Q) — what am I looking for? This is the token doing the searching.
- Key (K) — what do I offer / what am I about? Every token advertises itself with a key.
- Value (V) — the actual information I'll hand over if someone attends to me.
So the recipe for a single token is: compare my query to every key → turn those scores into weights that add up to 1 → take a weighted blend of all the values. That blend becomes the token's new, context-aware vector.
# Attention for ONE token (light pseudo-code)
# q : this token's query vector
# K, V : keys and values for every token in the context
scores = [ dot(q, k) for k in K ] # how well q matches each key
weights = softmax(scores) # normalize -> sums to 1.0
output = sum(w * v for w, v in zip(weights, V))
#
# `output` is this token's meaning AFTER it has looked around.
# High-weight tokens contribute most to it.The raw match scores can be any numbers, so softmax squashes them into positive weights that sum to exactly 1.0 — a token's fixed budget of 100% attention, divided up however the scores dictate. With that in mind, watch it play out. Back to "The cat sat because it was tired." When it runs attention, its query strongly matches the key of cat, so cat gets a big weight and its value dominates the blend. The updated it vector now effectively means "it (= the cat)." Later, when the token tired looks around, it attends back to it — and through it, to cat — so the model understands who is tired. This chaining of glances is how meaning propagates across a whole sentence.
Use the interactive visualizer below to try this yourself: click any word to make it the query and watch which words it attends to. Notice how clicking it lights up cat.
In self-attention, what determines how much one token contributes to another token's updated representation?
#Causal attention: no peeking ahead
The models you chat with are decoders: they generate text one token at a time, left to right. So when the model is processing a token, it must not look at tokens that come after it — those haven't been generated yet, and letting the model peek would be cheating during training. This is causal (masked) attention: each token can attend to itself and everything before it, but the weights for all future tokens are forced to zero.
Causal masking is not optional
It's tempting to think a model "reads the whole sentence at once." During generation it can only ever see the past. If it appeared before cat in the sentence, it could not attend to cat at all — the meaning would have to be resolved some other way. This left-to-right constraint is exactly why next-token prediction works as a training objective, and why prompt ordering matters.
# Causal mask: a token at position i may only attend to positions <= i.
# Scores to FUTURE tokens are set to -infinity before softmax,
# so their weights become 0.
# attends to ->
# The cat sat because it
# The [ ok x x x x ]
# cat [ ok ok x x x ]
# sat [ ok ok ok x x ]
# because[ ok ok ok ok x ]
# it [ ok ok ok ok ok ]
#
# ok = allowed x = masked (weight forced to 0)#Multi-head attention: many glances at once
One set of Q/K/V vectors captures one kind of relationship. But language has many at once: grammar (which noun goes with which verb), reference (what it points to), tone, topic. So transformers run several attention operations in parallel, each with its own learned Q/K/V matrices. These are the heads.
Each head learns to specialize — one might track subject-verb agreement, another might resolve pronouns, another might follow the topic. Their outputs are concatenated and combined, so the token ends up enriched by many different perspectives in a single layer. "Multi-head" just means: look several ways at the same time, then merge what you found.
Why do transformers use MULTIPLE attention heads instead of just one?
Key takeaways
- Self-attention lets each token ask "which other tokens should I look at?" and mix in their information, turning context-free embeddings into context-aware ones.
- It works with three learned vectors per token: a Query (what I'm looking for), Keys (what each token offers), and Values (the information handed over); query-key matches become softmax weights over the values.
- The classic payoff is reference resolution — attention is how the model figures out that "it" refers to "the cat."
- Causal (masked) attention lets each token see only itself and earlier tokens, never the future — this is what makes left-to-right generation possible.
- Multi-head attention runs several attention patterns in parallel so the model can capture grammar, reference, topic, and more all at once.
Click a word to make it the “query” and see which words it attends to.
“it” attends most to “cat” — that's how the model resolves what “it” refers to.
A token's attention weights are computed with softmax over the query-key scores. Given these final weights for the query token "it", which token contributes MOST to its updated representation?
# query token: "it"
# attention weights (already softmaxed, sum = 1.0)
weights = {
"The": 0.05,
"cat": 0.61,
"sat": 0.10,
"because": 0.08,
"it": 0.16,
}
# updated_it = sum(weight * value for each token)A learner explains self-attention. One statement contains a misconception about how the mechanism works. Which one is WRONG?
# Claims about self-attention:
# A) Each token builds a Query, and every token exposes a Key and a Value.
# B) Attention weights come from matching a token's Query against other tokens' Keys.
# C) The output for a token is a weighted blend of the Values it attended to.
# D) A token compares its Query directly against other tokens' Values to get weights.Complete the sentence about the constraint used in the decoder models you chat with, which prevents a token from looking at words that come after it.
In attention, a token may attend only to itself and earlier tokens; the weights for all FUTURE tokens are forced to zero. This is what makes left-to-right generation possible.
Put the steps of computing self-attention for a single token in the correct order.
Apply the causal mask so future tokens get a score of -infinity
Project the token's embedding into a Query vector, and every token into Key and Value vectors
Blend the Values using those weights to produce the token's new vector
Score the Query against every token's Key (e.g. with a dot product)
Run softmax over the scores to get weights that sum to 1.0
Consider the sentence:
"The trophy did not fit in the suitcase because it was too big."
- Which word does it refer to — the trophy or the suitcase? Explain how you decided.
- Describe, in terms of Query / Key / Value, what has to happen inside attention for the model to resolve this correctly. Which token holds the query? Which key should win?
- Now change one word so that it should refer to the other noun instead, and explain why the attention weights would shift.
Try it yourself — a starting point to build on:
# Write your solution here