Ask an AI to summarize a 40-page report and you get eight tidy bullets that read as though nothing important was left behind. Something important almost always was. AI summaries are lossy compression, and the losses are not random. The same categories of detail fall out every time: exact numbers, the caveats around a finding, the word not, the person who disagreed, and the conditions under which any of it holds.
A summary that drops the caveat is more dangerous than one that drops a paragraph. You can see a missing paragraph. You cannot see a missing qualifier, and the confidence of the prose tells you nothing about how much of the source survived it. Worse, the one instruction most people reach for to prevent this, telling the model to be accurate, has been measured doing the opposite.
Lossy by design, not by accident
A summarizer is optimized to produce text a reader would judge fluent, representative and plausible. Nothing in that objective rewards keeping the one clause that would have changed your decision. Detail that is rare in the document, and therefore low-frequency, gets smoothed toward the average of what documents like this usually say. The load-bearing exception is exactly the kind of thing that averaging destroys.
Researchers surveying factual consistency in summarization call the abstraction ability of neural models a double-edged sword. Because the model can freely generate summaries without any constraint on the words or phrases used, the output reads better than a cut-and-paste summary. The same freedom produces what the survey calls the distortion or fabrication of factual information in the article. Fluency and fidelity are being traded against each other, and you only ever see one of them.
That is not a theoretical worry. In a large-scale human evaluation of neural abstractive summarization systems, Maynez and co-authors at Google found substantial amounts of hallucinated content in all model-generated summaries, and concluded that these models are highly prone to hallucinate content that is unfaithful to the input document. Their annotators were reading short news summaries. Your inputs are longer and messier.
The eight things AI summaries drop first
Across contracts, papers, meeting transcripts and long email threads, the same eight categories disappear. Learn the list and you know exactly where to look before you trust an answer.
- Specific numbers and units. Exact figures get rounded, ranges collapse to a midpoint, and units, denominators and sample sizes fall away entirely.
- Caveats and hedges. Words like may, in some cases, under these conditions and preliminary are the first casualties, because they make prose feel weak.
- Negations. Did not meet the endpoint, failed to replicate, is not covered. A single missing not inverts the meaning of the sentence it was in.
- Minority and dissenting views. The one objection in a meeting is low-frequency by definition, so it reads to the model as noise around the consensus.
- Uncertainty and limitations. The limitations section is the part authors write to stop you from overreading them, and it is the part summaries skip.
- Attribution. Who said what turns into the document says, and a proposal floated by one person becomes a decision the group made.
- Dates and conditions. Effective from, only if, subject to. Conditional obligations get summarized as obligations.
- Edge cases and exceptions. The carve-out, the exclusion, the one client this does not apply to. These are short, rare, and the entire reason the clause exists.
Numbers, units and quantities
Quantities are unusually fragile because they are arbitrary. A model can predict the shape of a sentence about revenue growth without predicting the digits. The problem is well enough documented that researchers built a dedicated system, Herman, to recognize and verify quantity entities such as dates, numbers and sums of money in generated summaries, up-ranking the summaries whose quantity terms are actually supported by the source text.
In practice the failure is quieter than an invented number. A 31.4% figure becomes roughly a third. Two different percentages measured on different bases get merged into one. A cost stated per seat per month becomes an annual total the document never claimed. Every one of those is a defensible-sounding paraphrase that you cannot audit without the original line.
Caveats, hedges and scope limits
Models strip the scope limits that keep a finding honest, and there are now hard numbers on how often. In a 2025 Royal Society Open Science study, Uwe Peters and Benjamin Chin-Yee tested 10 prominent models, including ChatGPT-4o, ChatGPT-4.5, DeepSeek, LLaMA 3.3 70B and Claude 3.7 Sonnet, comparing 4,900 generated summaries against their original scientific texts. When summarizing, the models omitted details that limit the scope of research conclusions, producing generalizations broader than the study supported. DeepSeek, ChatGPT-4o and LLaMA 3.3 70B overgeneralized in 26% to 73% of cases.
Compared head to head with human-authored summaries of the same research, the model summaries were nearly five times more likely to contain broad generalizations (odds ratio 4.85). The study also found that newer models tended to be worse at this than older ones, which is the opposite of the direction most people assume the technology is moving.
Negation
Negation carries enormous meaning in very few characters, and language models handle it badly. An analysis across negation benchmarks, testing GPT-3-era models, found that models show insensitivity to the presence of negation, an inability to capture the lexical semantics of negation, and a failure to reason under negation. Compression makes this worse: the short function words are the cheapest thing to drop when the model is rewriting a sentence into a tighter one.
The most expensive summarization error is a flipped polarity. A trial that did not meet its primary endpoint, summarized as showed promising results, is a sentence with the right topic, the right tone, and the wrong answer.
Attribution, dissent and edge cases
Meetings and threads are where attribution matters most and survives least. A summary that says the team agreed to ship in August erases the one engineer who said the migration is not ready. Dialogue is also where models are measurably weakest: the TofuEval benchmark for topic-focused dialogue summarization found that existing LLMs hallucinate significant amounts of factual errors in the dialogue domain regardless of the model's size, so scaling up the model does not buy you a faithful meeting recap.
Why the losses are predictable
Three mechanisms explain most of it. The first is the objective, covered above: rare-but-critical detail loses to common-and-fluent phrasing. The other two are structural, and they tell you exactly where in a document your losses will cluster.
Position is the second. In the Lost in the Middle study, Nelson Liu and co-authors found that performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models. A summary of a long document is therefore biased toward its opening and its conclusion, which is where the confident claims live, and away from the middle, which is where the methods, the conditions and the exceptions live.
Length is the third. Chroma's Context Rot report tested 18 leading models, including GPT-4.1, Claude 4, Gemini 2.5 and Qwen3, and found that model performance varies significantly as input length changes, even on simple tasks, and that models do not use their context uniformly. The practical translation: the longer the thing you ask it to summarize, the larger the fraction of it that quietly does not participate in the answer.
Extractive versus abstractive: the distinction that decides your risk
There are two ways to build a summary. An extractive summary selects real sentences from the source and stitches them together. An abstractive summary generates new sentences, and those sentences can contain words that never appeared in the document. Chat assistants are abstractive by default, which is why they read so well and why they can subtly change what a document said.
| Dimension | Extractive summary | Abstractive summary |
|---|---|---|
| How it is built | Selects and reorders sentences that exist verbatim in the source | Generates new prose; wording is free and need not appear in the source |
| What it preserves | Exact numbers, units, hedges and negations, because the text is copied | The gist. Quantities, qualifiers and attribution are rewritten and can drift |
| Typical failure mode | Choppy and redundant; omits whatever was not selected, but never invents | Fluent text that adds claims not in the source, or flips a qualified finding into a flat one |
| Verifiability | High: every line traces back to a locatable place in the document | Low: no line necessarily exists in the source, so each claim needs checking |
| Best used for | Contracts, medical and financial text, anything where wording is load bearing | Getting oriented quickly in material where the gist really is enough |
You are not stuck with the default. Asking for direct quotations pushes an abstractive model toward extractive behavior, and that single move converts an unverifiable paraphrase into something you can check against the page in about ten seconds.
Here is what most guides get wrong: telling it to be accurate can make it worse
Prompting for accuracy roughly doubled the damage. Peters and Chin-Yee tested the standard advice, the instruction to be accurate and not overstate the findings, and found that accuracy-focused prompts increased overgeneralization by roughly twofold. The single tactic almost every prompting guide recommends is the one the data says backfires. The likely reason is register: asking a model to be careful nudges it toward sounding like a confident expert, and confident experts write clean declarative sentences, which is precisely the register in which caveats die.
Telling an AI to be accurate roughly doubled how often it overgeneralized. Adjectives in a prompt are suggestions; structure is a constraint. Stop asking for accuracy. Ask instead for the artifacts that make inaccuracy visible: quotes, locations, verbatim figures, and an explicit list of what was dropped.
How to see what was left out
You cannot audit a summary against a document you did not read. What you can do is force the summary to expose its own gaps. These prompts do that, and they take seconds.
- Demand quotes and locations. 'For every claim, give the exact sentence it came from and the page or section number.' A claim the model cannot quote was inferred, not read.
- Ask what was dropped. 'What did you leave out of this summary, and why?' Then ask the harder version: 'What in this document would a careful reader say contradicts your summary?'
- Pin the numbers. 'Preserve every figure, unit, date, sample size and currency exactly as written. Do not round, convert or combine them.'
- Pull the caveats separately. 'List every limitation, condition, exception and hedge, in the author's own words.' Anything that comes back thin is a signal the prose summary already swallowed them.
- Ask for the negatives explicitly. 'List everything this document says did NOT happen, did not work, was not tested, or is not covered.'
- Ask for the dissent. 'Who disagreed, and what exactly did they say?' On a meeting transcript this is the single highest-yield question you can ask.
- Probe the middle. Pick a section from the middle third, ask a question you already know the answer to, and see whether the model actually has it.
How to summarize long documents without losing the load-bearing parts
Two-pass summarizing beats one-pass every time. Pass one: 'Extract the 20 most decision-relevant sentences verbatim, with locations, and change nothing.' Pass two: 'Now write the summary using only those sentences.' You get abstractive readability on an extractive foundation you can spot-check.
- Go section by section, in document order, rather than asking for one summary of the whole file. It defeats both the position effect and the length effect at once.
- Keep numbers in a separate table from the prose. A figure sitting in its own row with a source location is much harder to quietly round away.
- Summarize the limitations, exceptions and conditions as their own deliverable, not as a trailing sentence in a paragraph of good news.
- Cap the input. A 30-page slice read carefully beats a 300-page file skimmed, and you can verify coverage on each slice.
- Treat the summary as an index into the document, never as a replacement for it. Its job is to tell you which page to open.
When a summary is not enough
Every technique above assumes one thing: that you can still get back to the original. That assumption fails constantly. The PDF was in a chat you closed, the transcript lived in a meeting tool you no longer pay for, the contract was an attachment in a thread from March. What you kept is the summary, and the summary is exactly the artifact that dropped the clause you now need.
This is the problem MemX is built around. It stores the full source, PDFs, screenshots, voice notes, chat exports, and indexes the actual text of each one, so months later you can ask a targeted question against the original document rather than re-reading a summary of it. When a compressed answer is not enough, the fix is not a better summary. The fix is having the thing that was summarized. MemX is private by architecture, with per-user isolation and encryption at rest, so the library you build stays yours.
01What do AI summaries leave out most often?
Exact numbers and units, caveats and hedges, negations such as did not or failed to, minority and dissenting views, limitations, attribution of who said what, dates and conditions, and edge cases. These share one property: they are short, rare within the document, and high-stakes, which is the worst combination for a system optimized to produce representative-sounding prose.
02Why do AI summaries drop caveats and overstate findings?
A 2025 Royal Society Open Science study of 10 models and 4,900 summaries found that models omit details limiting the scope of conclusions, overgeneralizing in 26% to 73% of cases for some models, and were nearly five times more likely than human summarizers to make broad generalizations. Hedged prose reads as weak, and the models are tuned toward prose that reads well.
03Does asking the AI to be accurate fix the problem?
No, and it can backfire. In the same study, accuracy-focused prompts increased overgeneralization by roughly twofold. Structural constraints work better than adjectives: require direct quotes with locations, verbatim numbers, and an explicit list of limitations and exceptions.
04What is the difference between extractive and abstractive summarization?
Extractive summarization selects real sentences from the source, so exact numbers, hedges and negations survive and every line is traceable. Abstractive summarization generates new sentences, which reads better but can introduce claims that are not in the source or subtly flip a qualified finding. Chat assistants are abstractive by default.
05Do longer documents produce worse summaries?
Yes. The Lost in the Middle research found that models retrieve information best from the beginning and end of a long input and significantly worse from the middle, even in long-context models, and Chroma's Context Rot testing across 18 models found performance varies significantly as input length grows even on simple tasks. Summarizing section by section is the practical countermeasure.
Treat every AI summary as a claim about a document rather than a substitute for it. Ask it to quote. Ask it what it dropped. Ask it what the document says did not happen. If it can answer all three, you have something worth acting on. If it cannot, you are reading fluent prose about a file that nobody, human or machine, has fully read.
