AI Hallucination

What is an AI hallucination?

An AI hallucination is content generated by a model that is fluent, confident, and plausible but is factually incorrect, unsupported by evidence, or inconsistent with the source material the model was given. The term is most associated with large language models (LLMs) such as those behind chatbots, but it also applies to image, audio, and code generation. The defining feature is not merely that the output is wrong, but that the model presents the wrong output with the same tone and structure it uses for correct output, giving users no built-in signal of unreliability.

Hallucination is distinct from an ordinary mistake in important ways. A search engine that returns no result is transparently failing, whereas a hallucinating model fills the gap with invented detail. Common examples include citing a paper that does not exist, attributing a quote to the wrong person, inventing a plausible but fictional historical date, or describing an API method that a library never shipped. Because the surrounding text is coherent, hallucinations can be hard for a non-expert to detect.

Researchers generally treat hallucination as a property of the output rather than an intent, since models have no beliefs to misrepresent. A widely cited survey defines it as generated content that is nonsensical or unfaithful to the provided source content or to verifiable world knowledge.

Why LLMs hallucinate: next-token prediction without a ground truth

Large language models are trained to predict the next token in a sequence given the preceding tokens. They learn the statistical distribution of human text, not a verified database of facts. When asked a question, the model produces the most probable continuation, and a fluent, well-formed falsehood is often more probable than an awkward admission of ignorance. The model has no separate fact-checking step and no internal representation of 'I do not know' unless it has been explicitly trained to express uncertainty.

A 2025 paper from OpenAI, 'Why Language Models Hallucinate,' frames hallucinations as the natural result of statistical pressures during training and evaluation rather than a quirk of the Transformer architecture. The authors argue that standard pretraining rewards reproducing the data distribution, so plausible-but-false statements emerge wherever facts are rare or unseen. They further argue that most benchmarks score answers as right or wrong with no credit for abstaining, which rewards confident guessing over honestly saying 'I am not sure.' Under that incentive, a model that always guesses can outscore a model that admits uncertainty, so guessing behavior is reinforced.

• Models optimize for likely text, not for truth; fluency and accuracy are different objectives.
• Calibration (knowing what you do not know) is cheaper to achieve than accuracy, yet most evaluations do not reward it.
• Binary scoring that penalizes 'I do not know' the same as a wrong answer trains models to guess confidently.

Types: intrinsic vs. extrinsic, factual errors, fabricated citations, and faithfulness failures

The research literature splits hallucinations along two main axes. The first axis is intrinsic versus extrinsic. Intrinsic hallucinations contradict the source content the model was given, for example summarizing a document and stating the opposite of what it says. Extrinsic hallucinations add claims that cannot be verified from the source or from known facts, such as inventing a detail not present anywhere in the input.

The second axis, used in major surveys, is factuality versus faithfulness. Factuality hallucinations conflict with verifiable real-world knowledge, including outright fabrication of facts, figures, or events. Faithfulness hallucinations diverge from the user's instructions or the provided context even if the statement might be true in isolation, for example answering a different question than the one asked. Fabricated citations, in which a model invents a paper title, author, or case number that looks real but does not exist, are an especially damaging factuality failure because they wear the appearance of evidence.

• Intrinsic: output contradicts the supplied source text.
• Extrinsic: output adds unverifiable claims beyond the source.
• Factuality: output conflicts with real-world facts (invented dates, numbers, fake citations).
• Faithfulness: output ignores or distorts the user's prompt, context, or constraints.

Common triggers: knowledge gaps, ambiguous prompts, and long contexts

Hallucinations are not random; certain conditions make them far more likely. The most common is a knowledge gap: the answer falls outside or is sparsely represented in the training data, such as obscure entities, very recent events after the training cutoff, or niche technical details. Faced with a gap, the model interpolates a plausible answer instead of abstaining.

Ambiguous, underspecified, or leading prompts also induce hallucination. If a question presupposes a false premise, a model may accept the premise and build a confident answer on top of it. Long contexts add their own failure modes: as inputs grow, models can lose track of details, conflate passages, or attend to the wrong part of the context, producing summaries or answers that drift from the source. Tasks that require multi-step reasoning, arithmetic, or precise recall of long lists are similarly error-prone.

• Questions about rare entities or events after the training cutoff.
• Prompts with false premises, ambiguity, or missing constraints.
• Very long inputs where relevant facts get lost or conflated.
• Multi-step reasoning, exact figures, and verbatim recall tasks.

How hallucination is measured and evaluated

Measuring hallucination is difficult because it requires comparing free-form text against ground truth at scale. Researchers use a mix of dedicated benchmarks, automated metrics, and human review. TruthfulQA tests whether models reproduce common human misconceptions; HaluEval provides large sets of question-answer and dialogue samples labeled for hallucination; and FActScore decomposes long-form generations into atomic claims and checks each one against a reference source, yielding a fraction of supported facts.

Automated detection methods include checking outputs against retrieved evidence, prompting a separate model to act as a judge, and examining a model's own uncertainty signals such as token probabilities or self-consistency across multiple samples. No single metric is definitive, and surveys note that factuality and faithfulness are different phenomena that often require different benchmarks. Human evaluation remains the gold standard for high-stakes domains, since automated graders can themselves miss or introduce errors.

• Benchmarks: TruthfulQA, HaluEval, and FActScore each target different facets.
• Claim-level checking: split output into atomic facts and verify each against a source.
• Self-consistency: sample multiple answers and flag the ones that disagree.
• Human review: still essential for medical, legal, and other high-stakes use.

Mitigation: retrieval-augmented generation, grounding, and citations

The most widely deployed defense against hallucination is grounding the model in trustworthy external data at generation time. Retrieval-augmented generation (RAG), introduced by Lewis et al. in 2020, pairs a language model with a retriever that fetches relevant documents (for example from a vector index) and conditions the answer on that retrieved text. Because the model can copy and attend to specific passages, it is less likely to invent facts and can stay current with information added after training.

Grounding works best when the system also returns citations that point back to the supporting passages, allowing users to verify each claim and exposing answers that lack support. RAG is not a complete cure: if retrieval returns irrelevant or wrong documents, or if the model ignores the retrieved evidence, hallucinations still occur. Quality of the index, chunking, and retrieval ranking therefore strongly affect how much grounding actually helps.

• Retrieve relevant documents and condition the answer on them.
• Return citations so each claim can be traced to a source.
• Quality of the knowledge base and retriever bounds the benefit.
• Grounding reduces but does not eliminate fabrication.

Mitigation: prompting, structured outputs, verification, and human review

Beyond retrieval, several techniques reduce hallucination at the application layer. Prompts can instruct the model to answer only from supplied context, to cite sources, and to respond 'I do not know' when evidence is missing, which raises the chance of honest abstention. Constraining the model to structured outputs, such as a fixed JSON schema or a list selected from known options, narrows the space in which it can invent content.

Verification layers add a second check: a separate model or rule-based system can confirm that each claim is supported by the retrieved evidence before the answer is shown, and self-consistency sampling can flag answers that change across runs. Tool use, where the model calls a calculator, database, or search API rather than recalling from memory, removes whole classes of arithmetic and lookup errors. For any consequential decision, human review remains necessary. The 2023 Mata v. Avianca case, in which lawyers submitted a brief containing fabricated case citations produced by a chatbot and were sanctioned, illustrates the cost of skipping that review.

• Instruct the model to answer only from context and to abstain when unsure.
• Use structured outputs and constrained choices to limit invention.
• Add a verification pass and self-consistency checks before showing answers.
• Prefer tool calls for math and lookups; keep humans in the loop for high stakes.

Why hallucination cannot be fully eliminated

Mitigations lower the rate of hallucination but do not reduce it to zero, and there are theoretical reasons to expect that. A 2024 paper by Xu, Jain, and Kankanhalli, 'Hallucination is Inevitable,' argues from learning theory and computability that a computable LLM cannot learn every computable function, so when used as a general problem solver it will inevitably produce some inconsistencies with ground truth. Related work shows that even well-calibrated models must hallucinate at some nonzero rate on certain facts.

Practically, this means hallucination is a risk to be managed rather than a bug to be patched out entirely. Systems that assume occasional confident errors and design around them, through grounding, verification, citations, abstention, and human oversight, are more reliable than systems that assume a model is always right. The realistic goal is to make hallucinations rare, detectable, and low-impact, not impossible.

Hallucination and grounded AI memory systems

Hallucination is a central concern for AI memory tools and second-brain applications, where users expect the system to return what they actually stored rather than a plausible invention. The standard mitigation pattern applies directly: index the user's own documents, photos, and notes, retrieve the relevant items semantically, and answer strictly from that retrieved content with references back to the original source. An AI memory app such as MemX retrieves stored documents, photos, and voice notes by semantic search and returns the underlying item, which keeps answers grounded in the user's own data rather than in the model's parametric guesses.

Grounding does not make such systems immune to error, but tying every answer to a retrievable, user-owned source narrows the room for fabrication and makes any remaining mistakes easier to spot and correct.