Memory & Cognition

Context Rot

Context rot is the measurable decline in a language model's output quality as its input context grows longer, even while the prompt stays well under the model's hard token limit. It describes degraded use of a filled window, not the size of the window itself.

What Context Rot Means

Context rot is the tendency of a large language model to produce lower-quality output as the amount of text in its context window increases, before that window is anywhere near full. The term distinguishes two different limits. The context window is a hard ceiling on how many tokens a model can read at once. Context rot is a soft degradation that appears far below that ceiling: a model rated for 200,000 tokens can start answering less reliably at 10,000 or 50,000 tokens, even on tasks it handles perfectly with a short prompt.

The distinction matters because a passing token count is not a guarantee of quality. Vendors advertise a maximum window, and developers often read that number as a flat region where every position is used equally well. Empirical evidence shows the opposite: usable capacity is smaller than nominal capacity, and the gap widens with the difficulty of the task and the position of the relevant information. Context rot names the phenomenon so it can be measured rather than assumed away.

The effect is not a single bug but a cluster of related behaviors. Relevant facts buried in the middle of a long input get used less than facts at the edges. Irrelevant passages added to the input pull the model off target. Repetition and semantic overlap between the question and the surrounding text change accuracy in ways that are hard to predict from length alone.

Context rot is quality decay inside a window that is not yet full, not the token ceiling itself.
A model's effective context, the length over which it stays reliable, is usually shorter than its nominal context.
The decay depends on task difficulty, distractors, and where the needed information sits, not just total tokens.
Recognizing context rot turns the advertised window from free headroom into a budget every token spends down.

Nominal Versus Effective Context

A useful way to reason about context rot is to separate the advertised window from the length at which a model still performs acceptably. The nominal context is the maximum token count the architecture accepts. The effective context is the largest input length at which accuracy on a given task stays above a chosen threshold. Context rot is precisely the shrinking gap between the two as a task gets harder or as distractors accumulate.

This framing is task-dependent, not a fixed property of the model. For a trivial retrieval where the answer is an exact lexical match near the start, effective context may equal nominal context. For a multi-step question whose evidence is scattered and semantically dissimilar from the query, effective context can be a small fraction of the window. The formula below is a definitional aid, not a measured constant, but it makes the dependency explicit and gives teams a quantity to track over time.

Treating effective context as a threshold also clarifies why benchmarks disagree. A simple needle-in-a-haystack test, which hides one distinctive sentence in filler and asks the model to recall it, often shows near-perfect recall at long lengths and so reports a large effective context. Harder probes with distractors and low needle-question similarity report a much smaller one. Both can be true for the same model because they measure different tasks.

C_eff(task, τ) = max { L : Accuracy(model, task, L) ≥ τ }

Effective context C_eff is the longest input length L for which task accuracy still meets a chosen threshold τ. Context rot is the gap between this value and the nominal window, and it grows as the task hardens or distractors increase.

Nominal context is the hard token limit; effective context is where accuracy still clears a threshold.
Effective context is defined per task and per accuracy bar, not as one number for the whole model.
Easy lexical-match probes overstate effective context; distractor-rich probes reveal the rot.
Tracking the nominal-to-effective gap turns context rot into a metric instead of an anecdote.

Why Quality Decays Before the Limit

Several mechanisms combine to produce context rot. The first is attention dilution. Transformer self-attention distributes a finite amount of attention weight across every token in the input. As the number of tokens grows, the share that can land on any single relevant token tends to fall unless the model has learned to concentrate sharply, so signal competes against an enlarging pool of noise.

The second is a position effect. Liu et al., in Lost in the Middle: How Language Models Use Long Contexts, documented a U-shaped accuracy curve: models answer best when the needed information sits at the beginning or end of the input and noticeably worse when it sits in the middle, even for models marketed as long-context. Placement, not just presence, governs whether a fact is used.

The third is distractor accumulation. Adding passages that resemble the answer but are not the answer reduces accuracy, and the Chroma report Context Rot: How Increasing Input Tokens Impacts LLM Performance found that distractors do not act uniformly: some specific distractors degrade performance far more than others. The same report observed that performance falls faster when the needle and the question share little surface or semantic similarity, so length interacts with content rather than acting on its own.

Attention dilution: finite attention weight spread over more tokens lowers the share available to each relevant one.
Position effects: the lost-in-the-middle U-curve means edge positions are read more reliably than the middle.
Distractor accumulation: near-miss passages cut accuracy, and certain distractors hurt far more than others.
Low needle-question similarity accelerates decay, so what the tokens contain matters as much as how many there are.

What the Evidence Shows

The two anchor studies bound the claim without overstating it. Lost in the Middle (Liu et al., 2023, published in the Transactions of the Association for Computational Linguistics) used multi-document question answering and key-value retrieval, and showed that accuracy is highest at the extremes of the input and degrades when the answer is in the middle. The takeaway is the shape of the curve, that placement strongly affects retrieval from long context, rather than a single universal percentage that would apply to every model and task.

The Chroma technical report Context Rot: How Increasing Input Tokens Impacts LLM Performance, published July 14, 2025 by Kelly Hong, Anton Troynikov, and Jeff Huber, evaluated 18 models including GPT-4.1, Claude 4, Gemini 2.5, and Qwen3. It reported that models do not use their context uniformly and grow less reliable as input length rises, that single distractors already reduce performance, and, in one counterintuitive result, that shuffling a haystack so it loses logical flow improved performance for every model tested compared with a coherently ordered haystack.

These results temper the simplest mental model of a context window. They do not claim that long context is useless or that every model fails at the same length. The honest summary is that the measured effect is real, varies by model and task, and is large enough that long inputs should be evaluated, not trusted by default. Standard needle-in-a-haystack tests, which hide a distinctive fact in filler text, are a starting probe but tend to flatter models because the needle is easy to find; harder variants expose more rot.

Liu et al. (2023, TACL) established the lost-in-the-middle U-curve on QA and key-value retrieval.
The Chroma 2025 report tested 18 models and found non-uniform context use plus distractor sensitivity.
A counterintuitive Chroma finding: shuffling a haystack to break its logical flow improved results across all 18 models.
Cite the measured shape and direction of the effect; avoid a single universal number, since it varies by model and task.

Measuring and Limiting Context Rot

Because context rot is task-specific, the reliable way to know a model's effective context is to probe it directly. A minimal degradation probe holds a fixed question and answer constant, then re-asks it inside progressively longer filler, optionally varying the answer's position and adding distractors. Plotting accuracy against input length reveals where the curve bends down, which is the practical effective context for that task. The example below sketches such a probe in pseudocode that wraps any chat completion call.

Mitigations follow from the mechanisms. Keeping the prompt short and dense reduces attention dilution. Placing the most important material at the start or end of the input, rather than the middle, exploits the position effect instead of fighting it. Removing near-miss passages before they reach the model limits distractor damage, which is the practical motivation behind retrieval reranking and tighter context assembly. None of these eliminate context rot; they shift where the bend occurs.

The broader lesson connects to how memory systems are built. If a long window cannot be trusted uniformly, then selecting and ordering what enters the window becomes a first-class engineering problem rather than an afterthought. That is the boundary between simply having a large context and actually using it, and it is why retrieval, compression, and ordering remain central even as nominal windows keep growing.

python

# Minimal context-rot probe: does accuracy hold as filler grows?
# `chat(prompt) -> str` stands in for any chat completion API.

def build_prompt(needle, filler_tokens, position="middle"):
    filler = "This sentence is unrelated padding. " * (filler_tokens // 6)
    if position == "start":
        body = needle + " " + filler
    elif position == "end":
        body = filler + " " + needle
    else:  # middle
        half = len(filler) // 2
        body = filler[:half] + " " + needle + " " + filler[half:]
    return f"Context:\n{body}\n\nQuestion: What is the secret code?\nAnswer:"

def probe(chat, lengths=(500, 4000, 16000, 64000), position="middle"):
    needle = "The secret code is ZX-4719."
    results = {}
    for n in lengths:
        prompt = build_prompt(needle, n, position)
        answer = chat(prompt)
        results[n] = ("ZX-4719" in answer)  # exact-match check
    return results  # e.g. {500: True, 4000: True, 16000: False, ...}

# Re-run with position="start"/"end"/"middle" to see the lost-in-the-middle
# effect, and append distractor sentences to filler to stress distractors.

A pseudocode probe that re-asks one fixed question inside growing filler. Sweeping the filler length and the needle position maps where accuracy drops, exposing the effective context rather than trusting the nominal limit.

Probe effective context by re-asking a fixed query inside growing filler and plotting accuracy versus length.
Vary needle position and inject distractors so the probe stresses the real failure modes, not just easy recall.
Mitigate by shortening prompts, placing key facts at the edges, and pruning near-miss distractors before they enter the window.
A large window does not remove the need to curate it; selection and ordering set where accuracy starts to fall.

Key takeaways

Context rot is the decline in LLM output quality as the input grows, even while the prompt stays below the model's hard token limit.
It is distinct from the context window: the window is the size ceiling, context rot is degraded use of a window that is not yet full.
Effective context, the length at which accuracy still clears a threshold, is typically shorter than the advertised nominal context.
Drivers include attention dilution, the lost-in-the-middle position effect, and distractor accumulation, which interact with content, not just length.
Liu et al. (2023) showed a U-shaped position curve; the Chroma 2025 report found non-uniform context use across 18 models and strong distractor sensitivity.
Mitigate by keeping prompts short and dense, placing key facts at the edges, and pruning distractors; curation matters even with huge windows.

Frequently asked questions

The context window is the hard maximum number of tokens a model can read at once. Context rot is the gradual loss of output quality that appears well below that maximum. A model can stay under its token ceiling and still answer worse simply because the input has grown longer, so the two are separate properties.

Not on its own. A larger window raises the nominal ceiling but does not guarantee the model uses every position reliably. The Chroma 2025 report found that models tested across long inputs grow less reliable as length increases, so a bigger window expands capacity without removing the decay that fills it.

It is a position effect documented by Liu et al. in 2023: language models answer most accurately when the relevant information sits at the start or end of the input and noticeably worse when it sits in the middle. The accuracy curve is roughly U-shaped, and it appears even in models marketed as long-context.

Hold a question and its answer fixed, then re-ask it inside progressively longer filler while varying the answer's position and adding distractors. Plot accuracy against input length and note where it bends down. That bend is the effective context for the task, which is usually shorter than the advertised window.

Both matter, and they interact. The Chroma 2025 report found that even a single distractor reduces performance and that some distractors hurt much more than others. Performance also drops faster when the answer shares little similarity with the question, so the content of the tokens shapes the rot as much as their count.

They are related but not identical. Context rot is the upstream degradation in how well a model attends to and uses a long input. One downstream symptom can be hallucination, where the model fabricates an answer rather than retrieving the buried fact, but context rot also shows up as missed retrievals and lower accuracy without invented content.

Put the idea into practice

MemX is an AI memory agent built on these ideas: store anything, skip the folders, and find it again by asking in plain English.

Try MemX Free