Tokens & Tokenization
Language models don't read letters or words — they read tokens, integer-mapped chunks of text, and understanding them explains model cost, context limits, and a lot of weird behavior.
When you type a sentence into an LLM, it feels like the model is reading your words the way you do. It isn't. Before a model sees a single thing, your text is chopped into tokens — small chunks that can be a whole word, a piece of a word, a space, or a punctuation mark. Each token is then swapped for an integer ID, and those numbers are all the model ever actually processes.
This one idea quietly explains a huge amount: why the API bills you the way it does, why there's a limit on how much text you can send, and why the model sometimes stumbles on things that look trivial to a human — like counting letters or doing arithmetic.
#The model sees numbers, not text
A neural network can only do math on numbers. So the very first step of every LLM is a lookup table called the vocabulary (or vocab): a fixed dictionary that maps each possible token to a unique integer ID. Modern models have vocabularies of roughly 100,000 to 200,000 tokens.
Tokenization is just the process of splitting your text into those known tokens and reading off their IDs.
Text: "Tokenization is fun!"
Tokens: ["Token", "ization", " is", " fun", "!"]
IDs: [30642, 2065, 374, 2523, 0]Notice two things. First, Tokenization was split into two pieces: Token + ization. Second, the spaces didn't disappear — many tokenizers attach a leading space to the word, so " is" (with the space) is its own token, distinct from "is" at the very start of a sentence.
LEGO for language
Think of tokens as LEGO bricks for text. Super-common words like the, and, or dog each get their own dedicated brick. Rare or made-up words don't get a brick of their own — the model builds them out of smaller sub-word bricks. unbelievable might come together as un + believ + able. Every word is buildable, even ones the tokenizer has never seen.
#Subwords and the BPE intuition
How does the tokenizer decide where to cut? Most models use a scheme called Byte-Pair Encoding (BPE) (or a close cousin). You don't need the algorithm's details, just the intuition:
- Frequent sequences of characters get merged into a single token. Because
theappears constantly in text, it earns its own dedicated token. - Rare sequences never get merged, so they stay broken into smaller pieces.
The result is an efficiency trade-off: common words cost one token, while unusual words, typos, or technical jargon get split into several. The vocabulary is learned from a giant pile of text, so it reflects what's common in that text.
You send the words "cat" and "antidisestablishmentarianism" to a model. Which is more likely to be split into multiple tokens?
#The 4-characters-per-token rule of thumb
You'll constantly need a quick estimate of how many tokens some text is. For typical English prose, a handy rule is:
> ~4 characters ≈ 1 token, or roughly ¾ of a word per token (so ~100 tokens ≈ 75 words).
It's only an approximation, but it's good enough to sanity-check whether a document fits in a model's context window or to ballpark an API bill before you send anything.
# Rough token estimate (approximation, not exact)
def estimate_tokens(text):
return len(text) / 4
estimate_tokens("The quick brown fox") # 19 chars -> ~4.75 tokensThe rule breaks on non-English and code
The ~4-chars-per-token rule assumes ordinary English. It falls apart elsewhere:
- Other languages (especially non-Latin scripts like Chinese, Arabic, or Hindi) often tokenize far less efficiently — sometimes 1 token per character, or worse. The same sentence can cost 2–3x more tokens than its English translation.
- Numbers get chopped oddly:
2026might be one token, but31415could split into314+15. This is a big reason models fumble arithmetic. - Whitespace and code — indentation, repeated spaces, and symbols each burn tokens, so source code often uses more tokens than you'd guess.
#Why tokens matter in practice
Tokens aren't just an implementation detail — they're the unit that everything is measured in:
- Cost. APIs bill per token, both for what you send (input) and what the model writes back (output). Wordier prompts and verbose outputs cost real money.
- Context limits. A model's context window — the maximum text it can consider at once — is measured in tokens, not words or characters. A '200K context' means 200,000 tokens.
- Behavior quirks. Because the model reasons over tokens, not letters, tasks like 'how many R's are in strawberry?' are genuinely hard: it may see
str+aw+berry, not ten individual letters.
See it for yourself
Tokenization is far more intuitive when you watch it happen. Use the interactive visualizer below to type your own text and see exactly where it gets split, how spaces and numbers behave, and how the token count changes — try an English sentence, then the same idea in another language.
That's the whole core idea: text → tokens → IDs, with common chunks getting their own token and rare ones split into pieces. Keep this mental model handy — it demystifies pricing, context windows, and a surprising share of 'why did the model do that?' moments.
Key takeaways
- Models never see raw characters or words — text is split into tokens, and each token is mapped to an integer ID before the model does anything.
- Tokenizers (usually BPE-based) give common words their own single token and split rare words into smaller sub-word pieces, so every word is representable.
- A useful rule of thumb for English is ~4 characters ≈ 1 token (about 75 words per 100 tokens).
- Tokens are the unit of cost (APIs bill per token) and of context limits (windows are measured in tokens).
- Numbers, whitespace, code, and non-English text tokenize less efficiently, which drives up cost and causes quirks like poor letter-counting and arithmetic.
Models split text into tokens (whole words, word-pieces, or symbols) and map each to an ID. Rule of thumb: ~4 characters ≈ 1 token.
A tokenizer gives super-common words their own single token and splits rare words into sub-word pieces. Given the sentence below, which word is MOST likely to become a single token?
Sentence: "The hyperparameterization was the problem."
Candidates: "The" vs. "hyperparameterization"A teammate is estimating cost and writes down this reasoning. One statement is a misconception. Which line should be corrected?
1. APIs bill per token, for both input and output.
2. A ~200K context window means ~200,000 tokens.
3. Translating an English prompt to Chinese always uses fewer tokens
because the text looks shorter on screen.
4. Numbers can split into multiple tokens (e.g. 31415 -> 314 + 15).Complete the rule of thumb for estimating token counts in typical English text.
Rule of thumb: about characters is roughly 1 token in English.
Put the tokenization pipeline in the correct order — from the text you type to the numbers the model actually processes.
Each token is looked up in the vocabulary to get its integer ID
The list of integer IDs is fed into the model
The tokenizer splits the text into tokens (whole words or sub-word pieces)
You provide raw text, e.g. "Tokenization is fun!"
You're sending a support-chat prompt to an LLM API that charges per token. Using the rule of thumb ~4 characters ≈ 1 token, estimate the token count for this message and reason about the cost:
> Please summarize the customer's issue in one sentence.
- Roughly how many tokens is it?
- Your teammate suggests translating all prompts into Japanese to 'save space' since Japanese text looks shorter on screen. Is that a good way to reduce token cost? Why or why not?
- Give one concrete rewrite of the prompt that would use fewer tokens while keeping the meaning.
Try it yourself — a starting point to build on:
# Write your solution here