Retrieval & Context

Context Window

A context window is the maximum amount of text, measured in tokens, that a language model can read and reference at one time when generating a response. It acts as the model's working memory: anything outside it is invisible unless re-supplied, summarized, or retrieved.

What Is a Context Window?

A context window is the maximum span of input a large language model (LLM) can hold in view at any single moment while producing output. Anthropic describes it as all the text a model can reference when generating a response, including the response itself, and frames it as the model's working memory rather than the much larger body of data the model was trained on.

Everything the model uses to answer a question, the system prompt, the conversation history, any attached documents, retrieved passages, and the answer being generated, must fit inside this window together. Content that falls outside the window is simply not available to the model unless it is summarized, retrieved, or supplied again. The window is therefore a hard ceiling on how much information a model can consider in one pass, and it shapes everything from how long a conversation can run to how large a document a model can analyze at once.

The window holds input and output together, not input alone.
It is transient: it is rebuilt for each request and does not persist between separate API calls.
Anything outside the window is invisible to the model for that request.

Context Window vs. Context Length vs. Memory

These three terms are often used loosely, but they describe different things. Context window and context length are usually synonyms: both refer to the token capacity a model can process in a single forward pass. A model with a 200,000-token context window has a context length of 200,000 tokens.

Memory, in the practical sense used by chat products and agents, is a separate mechanism. It refers to information that persists across sessions and is selectively reinserted into the context window when relevant. The context window is the temporary stage on which a single performance happens; memory is the storage that decides what gets placed on that stage. A model can have a very large context window and still have no persistent memory at all, because once a request ends, the window is cleared.

Context window and context length: near-synonyms for single-pass token capacity.
Working memory: the active contents of the window during one request.
Persistent memory: an external store that survives across requests and feeds chosen items back in.

How Context Is Measured: Tokens, Not Words or Characters

Context windows are measured in tokens, the units produced by a model's tokenizer, not in words or characters. A token can be a whole word, a fragment of a word, a single character, or a piece of punctuation. The exact split depends on the tokenizer, so the same sentence can use a different number of tokens across models.

OpenAI publishes a widely cited rule of thumb for English text: roughly one token per four characters, about three quarters of a word per token, or about 100 tokens for every 75 words. These ratios are approximate and shift for code, non-English languages, and unusual formatting, which tend to consume more tokens per word. Because billing, rate limits, and the window itself are all denominated in tokens, counting tokens rather than words is essential when estimating whether content will fit.

1 token is roughly 4 characters of English text (OpenAI rule of thumb).
100 tokens is roughly 75 English words.
Code, markup, and non-English text often use more tokens per word, so estimates vary.

Why Context Windows Have Limits: Attention, Compute, and Cost

The core constraint comes from the transformer architecture itself. Standard self-attention computes pairwise interactions between every token and every other token in the sequence, so its compute and memory cost grows quadratically with sequence length. Doubling the number of tokens roughly quadruples the attention computation, and a tenfold increase in context length implies about a hundredfold increase in attention work.

This scaling has direct consequences. Longer windows raise inference latency and memory consumption, and the intermediate attention matrix can exceed the memory available on accelerators such as GPUs. Larger windows therefore cost more to serve, which is why providers price requests by token count and why bigger windows are not always the default. Research into sparse attention (for example Longformer and BigBird) and memory-efficient kernels such as FlashAttention exists specifically to soften this quadratic cost and make longer contexts practical.

self-attention cost ∝ O(n² · d),  for sequence length n and model dimension d

Standard self-attention compares every token with every other token, so compute and memory scale quadratically with sequence length n. Doubling the context roughly quadruples the attention cost, which is the core reason context windows are bounded and why longer windows are expensive to serve.

Self-attention scales as O(n squared) in sequence length.
More tokens mean higher latency, more memory pressure, and higher serving cost.
Sparse and memory-efficient attention methods aim to reduce that cost.

From 2K to Million-Token Windows: How Capacity Grew (as of 2026)

Context windows have expanded dramatically over a few years. As reference points, GPT-3 (2020) handled about 2,048 tokens, and the GPT-3.5 model behind the original ChatGPT launch handled around 4,096 tokens. GPT-4 launched in 2023 with an 8,192-token window and a 32,768-token variant, and GPT-4 Turbo introduced a 128,000-token window in late 2023.

As of 2026, frontier models offer far larger windows. Google's Gemini 1.5 Pro made a 1 million token window broadly available and later opened a 2 million token window to developers. OpenAI's GPT-4.1 family supports up to 1,000,000 tokens. Anthropic offers a 200,000-token window across earlier recent Claude models such as Claude Sonnet 4.5 and, on March 13, 2026, made a 1 million token window generally available for Claude Opus 4.6 and Claude Sonnet 4.6 on the Claude API. These figures change as new models ship, so any specific number should be checked against current vendor documentation.

GPT-3 (2020): about 2,048 tokens; GPT-4 (2023): 8,192 with a 32,768 variant.
GPT-4 Turbo (late 2023): 128,000 tokens.
As of 2026: Gemini 1.5 Pro up to 2,000,000 tokens; GPT-4.1 up to 1,000,000; Claude up to 1,000,000 on the API.

The 'Lost in the Middle' Problem and Effective Context

A large window does not guarantee that a model uses all of it well. The study 'Lost in the Middle: How Language Models Use Long Contexts' by Nelson F. Liu and colleagues, released as a preprint in 2023 and published in the Transactions of the Association for Computational Linguistics in 2024, found that models tend to perform best when relevant information sits near the beginning or end of the input and noticeably worse when that information is buried in the middle, even for models marketed as long-context.

This gap between advertised window size and the portion a model uses reliably is often called effective context. Vendor guidance describes a related phenomenon Anthropic calls context rot: as the token count grows, accuracy and recall can degrade. The practical takeaway is that filling a window to capacity is not automatically better. What sits in the context, and where, can matter as much as how much fits.

Relevant facts placed mid-context are retrieved less reliably than those at the edges.
Effective context is the span a model uses well, which can be smaller than the advertised maximum.
Adding more tokens can dilute attention and reduce answer quality, not improve it.

Context Window Strategies: Truncation, Summarization, and Retrieval

Because windows are finite and large ones are costly, applications use several strategies to keep what matters inside the window. Truncation drops the oldest or least relevant content, which is simple but can lose important details. Summarization (sometimes called compaction) condenses earlier turns or long documents into shorter representations so more history can be retained in fewer tokens.

Retrieval-augmented generation (RAG) takes a different approach: instead of stuffing everything into the window, it stores content externally and uses semantic search to pull only the most relevant passages into context for each query. These strategies are frequently combined, and modern systems may also clear stale tool results or apply server-side compaction to extend long conversations beyond a single window.

Truncation: discard the least relevant tokens to stay under the limit.
Summarization or compaction: compress prior content into fewer tokens.
Retrieval (RAG): fetch only the most relevant passages from an external store per query.

Why a Bigger Window Is Not the Same as Long-Term Memory

It is tempting to treat a million-token window as a substitute for memory, but the two solve different problems. The context window is volatile working space that is rebuilt for every request and discarded afterward. Persistent memory is durable storage that survives across sessions and decides what to reload into the window when it becomes relevant. A model can have an enormous window and still forget everything the moment a request ends.

Bigger windows also do not escape the cost and the lost-in-the-middle effects described above, so loading an entire knowledge base into context for every query is both expensive and unreliable. This is why long-term memory is usually implemented as external storage plus retrieval rather than as raw window size. AI memory and second-brain tools such as MemX follow this pattern, storing content externally and retrieving the relevant pieces by plain-English query rather than keeping everything resident in the model's context.

The window is per-request and temporary; memory persists across sessions.
Large windows still incur quadratic cost and uneven mid-context recall.
Durable memory is built from external storage and retrieval, not from window size alone.

Key takeaways

A context window is the maximum text a model can reference in one request, measured in tokens, and it includes both the prompt and the generated response.
Tokens are not words: roughly 1 token equals 4 English characters, or about 100 tokens per 75 words, with variation by language and content type.
Windows are limited because self-attention scales quadratically with sequence length, raising latency, memory use, and cost as the window grows.
As of 2026, frontier windows range up to about 1 to 2 million tokens, but 'lost in the middle' effects mean effective context is often smaller than the advertised size.
A bigger window is not long-term memory: the window is cleared after each request, so persistent recall relies on external storage plus retrieval.

Frequently asked questions

The context window. It is the maximum amount of text, measured in tokens, that a language model can read and reference at once, which makes it the model's working memory for a single request. Anything outside the context window is invisible to the model unless it is summarized, retrieved, or supplied again.

No. A context window is temporary working space rebuilt for every request and discarded afterward, while memory is persistent storage that survives across sessions and selectively feeds information back into the window. A model can have a very large window and still retain nothing once a request ends.

Count tokens, not words. As a rough guide, one token is about four characters of English or roughly three quarters of a word, so 100 tokens is about 75 words. Code and non-English text use more tokens per word, so use a tokenizer or token-counting tool for accurate estimates.

You can, but it is often unwise. Larger inputs cost more and increase latency due to quadratic attention scaling, and research on 'lost in the middle' shows models retrieve mid-context information less reliably. Filling the window to capacity does not guarantee better answers.

Standard self-attention scales quadratically with sequence length, so doubling tokens roughly quadruples compute and memory. Very long windows strain accelerator memory and raise serving cost, which is why providers price by token and why retrieval is often preferred over ever-larger windows.

Put the idea into practice

MemX is an AI memory agent built on these ideas: store anything, skip the folders, and find it again by asking in plain English.

Try MemX Free