LLM FoundationsBeginner⏱ 8 min03 / 12

Embeddings

Embeddings turn each token into a dense vector so that meaning becomes geometry: similar words land near each other and directions encode relationships.

In the last lesson a tokenizer chopped your text into pieces and handed the model a list of token IDs — plain integers like 9906 or 1917. But an integer is a terrible way to represent meaning. 9906 isn't 'bigger' than 1917, and the number itself tells the model nothing about what the word means.

So the very first thing a model does is look up each token ID in a giant table and swap it for a list of numbers called an embedding. This is where raw text finally becomes something a neural network can reason about.

#From ID to vector

An embedding is a dense vector — an ordered list of floating-point numbers, e.g. [0.21, -0.83, 0.05, ...]. Every token in the vocabulary has its own embedding, stored as a row in the embedding matrix. Turning a token ID into a vector is literally just picking the row at that index — a table lookup, nothing fancier.

It's called dense because almost every number carries a little meaning — unlike a wasteful one-hot vector of 49,999 zeros and a single 1. Real models use hundreds to thousands of dimensions per token (GPT-2 used 768; many modern models use 4,096+). We'll use tiny 3-number vectors so we can actually see them.

An embedding lookup is indexing into a table — no math, just row selection.
# The embedding matrix: one learned row per token in the vocabulary.
# 50,000 tokens x 768 dims = a 50,000 x 768 table of numbers.

token_id  = 9906              # the id the tokenizer produced for "hello"
embedding = matrix[token_id]  # just grab that row

# embedding -> [0.21, -0.83, 0.05, 0.44, ... 768 numbers ...]
Think of it like

A map of meaning

Think of every word getting GPS coordinates on a giant map. Words about royalty cluster in one region, words about food in another, words about weather somewhere else. The coordinates are the embedding, and 'nearby on the map' means 'similar in meaning'. Training is the process of learning where to place each word so the map is genuinely useful.

#Meaning becomes geometry

These vectors aren't random. During training the model nudges each token's coordinates until words used in similar ways end up close together — so cat and dog land near each other, while cat and democracy end up far apart. In these hand-tuned toy vectors, notice cat and dog share a shape, and king and queen share a different one:

Toy 3-D embeddings. Similar words have similar coordinates.
cat    = [ 0.91,  0.10,  0.02 ]
dog    = [ 0.88,  0.14,  0.05 ]   # very close to cat
king   = [ 0.12,  0.90,  0.71 ]
queen  = [ 0.11,  0.88,  0.20 ]
apple  = [ 0.05,  0.09,  0.95 ]   # off in its own direction

It gets better. Because meaning lives in directions as well as positions, you can do arithmetic on words. The famous example: take king, subtract man, add woman, and you land almost exactly on queen:

king − man + woman ≈ queen

The direction that means 'add femininity' is roughly the same everywhere in the space, so the same step also turns uncle into aunt and actor into actress. Relationships are encoded as consistent directions.

Quick check

Two tokens have embeddings that point in nearly the same direction. What does that tell you?

#Measuring closeness: cosine similarity

Cosine similarity: 1 means aligned, 0 means unrelated, −1 means opposite.
# Cosine similarity = the cosine of the angle between two vectors.
#   1 = same direction (very similar), 0 = perpendicular (unrelated),
#  -1 = opposite. We compare DIRECTION, not raw distance, because the
#   pattern of meaning matters more than how long the vector happens to be.

def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    mag = lambda v: sum(x * x for x in v) ** 0.5
    return dot / (mag(a) * mag(b))

cosine(cat, dog)     # -> ~0.999  (nearly identical direction)
cosine(cat, apple)   # -> ~0.15   (mostly unrelated)
Common mistake

Nearby ≠ synonym

High similarity means 'used in similar ways', which is not the same as 'means the same thing'. Antonyms like hot and cold often sit close together because they appear in the same contexts ('the ___ water'). Embeddings capture topic and usage, not a dictionary definition — so a high score isn't proof two words are interchangeable.

Quick check

cosine(a, b) comes out to 0.02. What's the best interpretation?

#Why you should care: semantic search & RAG

Embeddings power one of the most useful patterns in modern AI: semantic search. Instead of matching keywords, you embed every document into a vector, embed the user's question the same way, and return the documents whose vectors are closest. A search for "how do I reset my password" can surface a doc titled "Recovering account access" even though they share no words.

This is the engine behind RAG (Retrieval-Augmented Generation): before the LLM answers, you retrieve the most relevant chunks by embedding similarity and paste them into the prompt, grounding the answer in fresh, specific facts — the topic of a later lesson.

Want to see words snap into clusters and watch king − man + woman line up? Use the interactive visualizer below to drag tokens around and explore the space yourself.

Key takeaways

  • An embedding is a dense vector (a list of numbers) a model assigns to each token — turning meaningless IDs into representations of meaning.
  • Meaning becomes geometry: similar words sit close together, and consistent directions encode relationships (king − man + woman ≈ queen).
  • Cosine similarity measures how aligned two embeddings are — 1 is very similar, 0 is unrelated, −1 is opposite.
  • Real embeddings have hundreds to thousands of dimensions, packing lots of meaning into each token.
  • Embeddings power semantic search and RAG: retrieve by meaning, not keywords, then feed the results to the LLM.
Try it yourself · Meaning as geometry
Similar words sit close together — click two to compare vectors.
catdogkittenkingqueenprinceapplebananamangocodedataserver
cos(cat, kitten) =0.96→ very similar (same cluster)

Embeddings turn each token into a vector. Words with similar meaning end up close together — meaning becomes geometry the model can do math on.

Practice challenges
Test yourself · earn XP
0/4
Predict the output#1

Given these toy embeddings, which word is MOST similar to `cat` by cosine similarity (most aligned direction)?

predict-output
cat    = [0.90, 0.10, 0.05]
dog    = [0.86, 0.14, 0.08]
car    = [0.05, 0.10, 0.92]
queen  = [0.10, 0.88, 0.15]

# cosine(cat, ?) is highest for which word?
Fix the bug#2

A learner explains what an embedding is. Which statement is the misconception that needs fixing?

fix-bug
# Which claim about embeddings is WRONG?
Fill in the blank#3

Complete the classic word-analogy that embeddings make possible.

king - man + woman  ≈
Reorder the lines#4

Put the steps of semantic search / RAG retrieval in the correct order.

1
Rank chunks by cosine similarity to the question
2
Embed every document chunk into vectors and store them
3
The user asks a question
4
Embed the question into the same vector space
5
Paste the top-matching chunks into the prompt as context
Your turn
Practice exercise

Using these toy 2-D embeddings, reason about the space by hand (no calculator needed — eyeball the directions):

`` paris = [0.9, 0.1] france = [0.8, 0.2] tokyo = [0.1, 0.9] japan = [0.2, 0.8] ``

  1. Which pair is more similar in direction: (paris, france) or (paris, japan)? Explain in terms of the angle between the vectors.
  2. The relationship 'city → its country' should be a consistent direction. Estimate the 'country' step by computing france − paris. Does applying that same step to tokyo land you near japan?
  3. In one sentence, explain why cosine similarity, not raw distance, is the natural tool here.

Try it yourself — a starting point to build on:

starter.py
# Write your solution here