The lost-in-the-middle problem means a large language model is most likely to miss the fact you buried in the middle of a long document, even when that document fits inside a million-token context window. Accuracy follows a U-shaped curve: models read the beginning and the end well, then lose the plot in the middle, and the degradation starts long before the window is anywhere near full. Liu et al. at Stanford first measured it in 2023, and Chroma reconfirmed it across 18 frontier models in 2025. The fix is not a bigger window. It is feeding the model less text, chosen better.
A bigger context window is not bigger memory. It is a longer hallway where the middle doors are the hardest to open.
What lost in the middle means (and why it can be worse than no context)
Lost in the middle describes a positional pattern: an LLM retrieves information placed near the start or end of its input far more reliably than information placed in the center. Liu et al. tested this with a multi-document question-answering setup. They gave models up to 20 or 30 passages, exactly one of which held the answer, and moved that answer document to positions 1, 5, 10, 15, and 20. Accuracy was high when the answer sat first, nearly as high when it sat last, and dropped sharply when it sat in the middle. Plotted across positions, the result is a distinct U.
Moving the relevant document from the best position to the worst middle position cost GPT-3.5 roughly 20 accuracy points on multi-document QA. In the harshest cases, a model with the answer sitting in the middle of its context scored lower than the same model given no documents at all. The extra text did not just fail to help. It actively dragged performance below the closed-book baseline.
A worked example: the same fact in three seats
Put real numbers on it. In the Liu et al. 20-document setup, GPT-3.5-Turbo answered correctly 75.8 percent of the time when the answer-bearing document sat in position 1. Slide that exact document to the middle of the same 20 and accuracy fell to 53.8 percent. Nothing else changed: same question, same model, same 20 passages, same answer text. Only the seat moved, and 22 points of accuracy went with it.
Now compare that middle score against the floor. The same GPT-3.5-Turbo, asked the same questions with zero documents in front of it, scored 56.1 percent from parametric memory alone. The middle-position run at 53.8 percent sits below that closed-book number. Hand the model the right answer in the wrong seat and it does worse than handing it nothing. That single comparison is the whole lost-in-the-middle thesis in two figures.
GPT-3.5: 75.8% with the answer at the front, 53.8% with it in the middle, 56.1% with no documents at all. The middle seat scored below the empty room.
Move one fact from the edge to the middle of a long prompt and a model can lose 20 to 30 accuracy points. Same fact. Same model. A different seat.
Why it happens: primacy, recency, and attention bias
The U-shaped curve is not a bug in one model. It is an attention bias built into how transformers weight their inputs. The follow-up paper Found in the Middle traced the cause directly: LLMs assign higher attention to tokens at the beginning and the end of their input regardless of whether those tokens are relevant. Position, not relevance, steers a meaningful share of where the model looks. Attention starves the middle even when it holds the only sentence that matters.
There is a deeper architectural reason the middle suffers as inputs grow. Self-attention compares every token against every other token, so the work scales with the square of the input length: double the tokens and the pairwise relationships the model must weigh roughly quadruple. Anthropic's context-engineering guidance points to a second factor on top of that geometry. Training corpora contain far more short sequences than very long ones, so models get less practice learning long-range positional dependencies than short-range ones. The architecture spreads attention thin, and the training data under-teaches the long-distance case. Both push the same direction: the farther a fact sits from either edge, the weaker its claim on the model's attention.
The shape mirrors a well-documented feature of human memory. People recall the first and last items in a list more reliably than the middle ones, a pattern psychologists call the primacy and recency effect. The cause inside a transformer is mechanical rather than biological, but the curve looks the same. Both human and model memory are strongest at the edges of a sequence and weakest in the center.
- Primacy: early tokens anchor the model and receive elevated attention, so facts near the top are read reliably.
- Recency: the most recent tokens, near the end, also draw heavy attention because they sit closest to the generated answer.
- The starved middle: everything between the two ends competes for whatever attention is left, so a single buried fact is the easiest thing to miss.
- Position over relevance: the bias applies even when the middle holds the only document that answers the question.
- Compounding cost: attention scales with the square of input length, and training data carries fewer long sequences, so long-range middle facts get the least practice and the thinnest attention.
The 1M-token trap: context rot starts early
Long-context marketing implies that a million-token window means a model can hold a million tokens in working memory equally well. It cannot. Chroma's 2025 study, which evaluated 18 leading models including Claude 4, GPT-4.1, Gemini 2.5, and Qwen3 families, found that performance degrades steadily as input grows, even on tasks a model handles perfectly at short length. Chroma named the effect context rot. The headline result: models do not use their context uniformly, and reliability falls as the input gets longer.
The decline is continuous rather than a single cliff, and it is heavily task-dependent. Degradation usually appears well before the window fills, in the hundreds of thousands of tokens rather than at the limit. The takeaway is blunt: a window that accepts a million tokens is not a window that reasons over a million tokens. The number on the spec sheet measures intake, not comprehension.
Chroma drove the point home with the simplest task imaginable: repeat a list of words back verbatim. No reasoning, no retrieval, pure copying. As the list grew long, models stopped repeating cleanly and began truncating, refusing, or inventing words that were never in the input. A model that cannot reliably echo a few thousand words it was just handed is not reasoning over a million-token window in any meaningful sense. The failure shows up on the easiest job there is, which means harder jobs inherit it.
Context window size measures how much a model can swallow, not how much it can actually digest.
Why long context fails: two problems, two different fixes
Two distinct failures get blamed on long context, and they need separating because they have different fixes. The first is position bias, the U-shaped curve from Liu et al.: where a fact sits determines whether the model finds it. The second is noise accumulation, the broader context-rot effect from Chroma: as total input grows, distracting and semantically similar passages pile up, and the model's reliability erodes everywhere, not just in the middle.
Chroma sharpened this with a counterintuitive finding: across all 18 models, performance was better on a shuffled, incoherent haystack than on a logically structured one. Structural coherence hurt retrieval, because related-but-irrelevant content sitting next to the needle competes with it. Position bias is about the slot. Noise accumulation is about the neighbors. A buried fact surrounded by plausible distractors is fighting both at once.
| Dimension | Position bias (lost in the middle) | Noise accumulation (context rot) |
|---|---|---|
| Core cause | Attention weights favor the start and end of the input | More tokens means more distractors competing with the signal |
| Named source | Liu et al, TACL 2024; Found in the Middle, 2024 | Chroma, Context Rot, 2025 (18 models) |
| Shape of failure | U-shaped curve across position | Steady decline as total input length grows |
| Trigger | Where the relevant fact sits in the context | How much surrounding, similar text is present |
| Best fix | Reorder so key facts sit near the edges, or calibrate attention | Retrieve fewer, higher-relevance chunks instead of stuffing |
The needle in a haystack test and its limits
The standard probe for this behavior is the Needle in a Haystack test. It plants a single random fact, the needle, at a chosen depth inside a long body of filler text, the haystack, then asks the model to retrieve it. Insertion position runs from 0 percent (the very start) to 100 percent (the very end), so the test maps retrieval accuracy against where the fact lives. When vendors publish a green NIAH grid showing near-perfect recall at every depth, that is the test they ran.
Passing NIAH does not mean a model is immune to lost in the middle. Classic NIAH uses a needle that is lexically distinct from its haystack, which is the easiest possible case. Vendors like Google have used NIAH to showcase long-context recall, but real documents bury facts among semantically similar sentences, ask for multi-step reasoning over scattered evidence, and offer no obvious keyword match. Chroma's similarity and distractor experiments show that once the needle resembles its surroundings, accuracy falls even for models that ace the basic grid.
The gap between the grid and reality is measurable. Chroma reports that models acing clean NIAH can fall sharply once the same retrieval asks for semantic matching or has to survive distractors, and recall on long inputs drops as token count climbs from ten thousand into the hundreds of thousands. The lexical needle is the gentlest version of the task. Production inputs hand the model the hard version by default.
A clean NIAH chart is a floor, not a guarantee.
How to fix it: retrieve, do not stuff
The durable fix is to send the model less text, chosen better. Retrieval-augmented generation pulls only the passages relevant to the current question and places them in a short, focused context, so the answer never has to survive a journey through 200,000 tokens of filler. Cutting input length sidesteps both failure modes at once: a shorter context has a smaller middle to lose, and fewer distractors to accumulate.
Anthropic frames the same discipline as managing an attention budget. Their context-engineering guidance treats the model's attention like a finite resource that every added token spends, and recommends curating the smallest set of high-signal tokens that produces the behavior you want. Reaching for the maximum context is the opposite move: it spends the budget on filler and leaves less for the facts that matter. Tight retrieval is budget discipline, applied before the prompt is ever sent.
- Select the few passages that answer the query rather than pasting an entire corpus and hoping.
- Mind the edges: when you must include several chunks, put the most important ones near the start and end, where attention is strongest.
- Rerank before you send: order retrieved chunks by relevance so the strongest evidence does not land in the dead center.
- Calibrate when you can: training-free methods like the found-in-the-middle calibration subtract the model's positional bias from its attention and recover up to 15 percentage points of RAG accuracy.
- Watch the distractors: filter out passages that are semantically similar to the answer but wrong, since those hurt more than random filler.
- Budget the tokens: treat context as a finite attention budget and spend it on high-signal passages, not on stuffing the window to its limit.
Found in the Middle showed that the positional bias is partly reversible at inference time, with no retraining, by subtracting the model's known attention skew. Combined with tight retrieval, the practical ceiling on long-context reliability is higher than the raw U-shaped curve suggests.
Next time a model ignores a fact you clearly pasted in, check where that fact sits in the prompt before you blame the model. Move it to the top or bottom and test again.
Where MemX fits
MemX is an external AI memory layer: a personal retrieval system over your own notes, screenshots, documents, and saved context. Instead of pasting your entire archive into a chat and trusting the model to find the right line in the middle, MemX retrieves the handful of passages that match what you are asking and hands the model a short, relevant context, the retrieve-do-not-stuff pattern the cited research supports. MemX does not change how any underlying model allocates attention. It changes what reaches the model in the first place, which is the lever you actually control.
You cannot rewrite a model's attention curve. You can decide what lands inside its context. That choice is where reliability is won or lost.
01What is the lost in the middle problem in LLMs?
It is a pattern where a language model retrieves information placed at the start or end of its input more reliably than information in the middle. Accuracy traces a U-shaped curve, strong at both ends and weak in the center, first measured by Liu et al. in 2023.
02Does a bigger context window fix lost in the middle?
No. Chroma's 2025 study of 18 models found performance degrades as input grows even on simple tasks, an effect called context rot. A million-token window measures how much text a model accepts, not how well it reasons over all of it. A bigger window often just creates a bigger middle to get lost in.
03What is the difference between context rot and lost in the middle?
Lost in the middle is position bias: where a fact sits decides whether the model finds it, tracing a U-shaped curve. Context rot is noise accumulation: as total input grows, similar passages pile up and reliability falls everywhere. One is about the slot, the other about the neighbors.
04How do I stop my model from missing facts in long context?
Send less, chosen better. Use retrieval to pull only relevant passages into a short context, rerank so key evidence sits near the edges, and filter out similar-but-wrong distractors. Shorter, focused input sidesteps both position bias and noise accumulation.
05Is the needle in a haystack test enough to trust a long-context model?
No. Classic NIAH uses a needle that is lexically distinct from its filler, the easiest case. Real documents bury facts among similar sentences. Chroma showed accuracy drops once the needle resembles its surroundings, even for models that pass the basic grid.
Sources
