Context Windows
A context window is the maximum number of tokens a model can attend to at once — prompt and reply share it — so everything outside the window is invisible, which is why long chats forget and long documents get cut off.
Ask a model a quick question and it answers perfectly. Paste in a 300-page contract, or chat with it for two hours straight, and something changes — it starts ignoring the beginning, forgetting what you told it earlier, or refusing outright because the input is 'too long.' The reason is a single hard limit called the context window.
The context window is the maximum amount of text — measured in tokens — that a model can look at at one time. It's the model's field of view. Anything inside the window, the model can attend to. Anything outside it might as well not exist.
#What exactly is the window?
Recall from the tokens lesson that all text is chopped into tokens (roughly ~4 characters each in English). A model's context window is just a cap on how many of those tokens it can process in a single pass.
The crucial part: the window holds everything at once — your system instructions, the whole conversation history, any documents you paste in, and the reply the model is generating. Input and output draw from the same budget. If the window is 200,000 tokens and your prompt already fills 199,000, the model has only ~1,000 tokens left to answer with.
A desk, not a filing cabinet
Think of the context window as the surface of a desk, not a filing cabinet. The model can only work with the papers actually laid out on the desk right now. It has no drawers to reach into for something it read an hour ago. To use a fact, that fact has to be on the desk — inside the window — at the moment the model is thinking. Push a page off the edge to make room for new ones, and it's simply gone.
System prompt ~500 tokens ┐
Document you pasted ~40,000 tokens │ all share ONE
Chat history ~8,000 tokens │ 200,000-token
Your new question ~120 tokens │ budget
Model's reply (needs room!) ┘
total_used = 48,620
remaining = 200,000 - 48,620 = 151,380 tokens left for the reply#What happens when you overflow
Once the total tokens exceed the window, something has to give. Depending on the tool you're using, one of two things happens:
- A hard error. Many raw APIs simply reject the request: 'This model's maximum context length is N tokens, but your messages resulted in M tokens.'
- Silent truncation. Many chat apps quietly drop the oldest messages to make room — a sliding window. The conversation keeps going, but the model can no longer see the start. This is exactly why a long chat seems to 'forget' what you said at the beginning: those tokens have slid off the edge of the desk.
The model isn't 'remembering' — the window is being re-sent
LLMs are stateless. The model doesn't secretly store your earlier messages between turns. Every single turn, the app re-sends the entire conversation so far as one big prompt. What feels like memory is really just the transcript being stuffed back into the window each time. So 'the model forgot' almost always means 'the earlier text no longer fits in the window and was dropped before sending.'
You're having a long chat and notice the model no longer recalls a detail you gave it at the very start. Assuming the app truncates old messages, what most likely happened?
#Big, but not free — and not infinite
Context windows have grown enormously. Early models capped out around 2,000–4,000 tokens; by 2026, windows of 200,000 tokens are common and some models reach 1,000,000 tokens or more — enough to hold entire codebases or several books.
But bigger is not automatically better, for three reasons:
- Cost. You pay per token, so a giant prompt is a giant bill on every turn (remember, the whole thing is re-sent).
- Latency. More input tokens means more computation, so responses get slower as the window fills.
- The 'lost in the middle' effect. Even when text technically fits, models tend to use information at the start and end of a long context more reliably than facts buried in the middle. A window that fits your data doesn't guarantee the model uses all of it equally.
Fits ≠ attended to
It's tempting to treat a huge context window as 'just paste everything in.' But stuffing 500K tokens of documents into a prompt is often worse than sending the 5K tokens that actually matter: it costs more, runs slower, and can bury the key fact in the middle where the model half-ignores it. A smaller, well-chosen context usually beats a giant, noisy one.
#Working within the limit
Since the window is finite, the real skill is deciding what deserves a spot on the desk right now. Three standard strategies:
- Summarize as you go. Instead of re-sending an entire long chat, periodically compress old turns into a short summary and carry that forward. You keep the gist while freeing up tokens.
- Chunk and retrieve (RAG). Rather than pasting a whole 300-page manual, split it into pieces, and at question time fetch only the few chunks relevant to the current question into the window. This is retrieval-augmented generation.
- Fetch fresh context with tools. Let the model call a tool — a search, a database query, a file read — to pull in exactly the information it needs, right when it needs it, instead of pre-loading everything.
This is why external tools and retrieval exist
The context window is the fundamental bottleneck behind a huge amount of modern LLM engineering. RAG, tool use, memory systems, and prompt-compression tricks are all answers to the same question: how do we get the right tokens onto the desk at the right moment, without overflowing it? Everything downstream in this course builds on this constraint.
Watch the window slide
This idea clicks fastest when you can see it move. Use the interactive visualizer below to add tokens to a conversation and watch the sliding window fill up and push the oldest text off the edge — then notice how a fact that scrolls out of view is no longer something the model can use. Try summarizing to reclaim space and see the difference.
The core mental model: the window is a fixed-size field of view, shared by prompt and reply, and only what's inside it is real to the model.
Key takeaways
- The context window is the maximum number of tokens a model can attend to in a single pass — its entire field of view.
- The prompt and the generated reply share the same token budget, and anything outside the window is invisible to the model.
- LLMs are stateless: the whole conversation is re-sent each turn, so 'forgetting' really means old tokens slid out of the window and weren't sent.
- Bigger windows cost more and run slower, and models use the middle of a long context less reliably than the start and end — fitting is not the same as being used.
- Because the window is finite, strategies like summarizing old turns, chunk-and-retrieve (RAG), and tool calls exist to load only the right tokens at the right time.
A model can only “see” a fixed number of recent tokens. As the conversation grows, the oldest tokens slide out of the window — which is why long chats forget the beginning, and why tools that fetch fresh context matter.
A model has a 100,000-token context window. The prompt below is assembled and sent. Roughly how many tokens are left for the model's reply?
System prompt: 2,000 tokens
Pasted document: 85,000 tokens
Chat history: 11,000 tokens
New user question: 1,500 tokens
Context window: 100,000 tokensA developer is explaining why their chatbot 'forgets' the start of long conversations. One line is a misconception. Which one should be corrected?
1. The model has no memory between turns; the app re-sends
the whole conversation each turn.
2. The prompt and the model's reply share one token budget.
3. The model permanently deleted the early messages from its
weights, so they are gone forever.
4. When the transcript exceeds the window, the app drops the
oldest messages to make room.Complete the definition of a context window.
A context window is the maximum number of a model can attend to at once, shared by both the prompt and the reply.
A chat app is at the edge of its context window. Put the events in the order they happen when the user sends one more message that would overflow the window.
The app drops the oldest messages from the transcript to make room
The model replies using only the tokens it can currently see, unaware of the dropped early messages
The user sends a new message, pushing the total over the context-window limit
The trimmed transcript (now within the limit) is sent to the model as the prompt
A customer-support bot uses a model with an 8,000-token context window. Every turn, the app sends: a fixed system prompt (1,000 tokens), the entire chat history so far, and the user's new message. It always reserves 1,000 tokens for the model's reply.
A long support session has reached 6,500 tokens of chat history, and the user sends a new 300-token message.
- Add up the tokens the app needs this turn. Does the request fit in the 8,000-token window?
- If it doesn't fit, and the app handles overflow by dropping the oldest messages, what is the concrete consequence for the conversation?
- Propose one change to the app (not a bigger model) that keeps the session working without losing the important early details.
Try it yourself — a starting point to build on:
# Write your solution here