A confident wrong answer from a RAG system is one of two bugs: retrieval failed to put the right evidence in the prompt, or generation had the right evidence and ignored it. Almost every production debugging session goes wrong because teams guess which one it is instead of measuring. Most teams debug RAG by vibes. This post gives you a differential diagnosis: a decision tree that isolates the failing stage, the exact metric to compute at each step, and the target to compare against, so you fix the one thing that is broken instead of rewriting all five.
Most listicles on RAG failures hand you a flat list of symptoms: poor chunking, bad embeddings, hallucination, context window limits. They never tell you how to attribute a specific wrong answer to a specific stage. Attribution is the whole game. Without it, you tune the chunk size, see no change because the bug was in reranking, and conclude RAG is fragile. It is not fragile. It is unobservable until you instrument each stage separately. RAG does not fail at random; every confident wrong answer is attributable to exactly one stage, and the stage is measurable before you touch a line of code.
Is it a retrieval bug or a generation bug?
Before touching any code, split the pipeline at one seam: the context that lands in the LLM prompt. Everything upstream of that seam (embedding, vector search, reranking, chunk assembly) is retrieval. Everything downstream (the prompt, the model, the decoding) is generation. The first diagnostic question is binary: did the retrieved context actually contain the information needed to answer correctly?
To answer that, you need one thing the listicles skip: a small labeled set. For each of 50 to 200 test questions, record the gold answer and the IDs of the chunks that genuinely support it. This golden set is the instrument. Without it you are debugging blind, because you cannot tell a retrieval miss from a generation miss by reading the final answer alone. With it, the seam becomes measurable, and the rest of this diagnosis is reading the numbers in order.
Read the failure backward. Pull the exact context string that was sent to the model for the failed query. If the answer is not present in that string, the bug is upstream (retrieval). If the answer is present in that string but the model said something else, the bug is downstream (generation). This single inspection routes you to the correct half of the tree.
Step 1: did the right chunk get retrieved at all? Measure recall
Start at the bottom of the stack, because nothing downstream can fix a chunk that was never retrieved. The metric is context recall, sometimes called recall@k: of the gold chunks that support the answer, what fraction appeared anywhere in your retrieved set of k candidates? In the RAGAS framework, context recall quantifies the proportion of facts in the reference answer that can be attributed to the retrieved context, which makes it the highest signal retrieval check you can run.
Compute recall at a generous k first, for example k equal to 50 or 100. If the gold chunk is missing even from the top 100, reranking and chunk boundaries are irrelevant: the candidate set never contained the answer, so the bug is first-stage retrieval. Causes cluster into three: the embedding model does not place the query near the chunk in vector space, the chunk was never indexed, or a metadata filter silently excluded it. Recall at wide k is your smoke alarm. A healthy first stage retrieves nearly every gold chunk at wide k; if a clean golden set loses chunks even from the top 100, stop and fix retrieval before reading any other metric.
Separate retrieval depth from final depth. Run recall at the wide candidate k (100) AND at the narrow k you feed the model (5 or 10). If recall@100 is high but recall@10 is low, the right chunk exists in the candidate pool but is ranked too low. That is not a retrieval-recall bug. That is Step 2.
Step 2: was it retrieved but ranked too low? Check reranking
If recall@100 is healthy but recall@10 is weak, the gold chunk is in the candidate pool and getting pushed below the cutoff you pass to the model. This is a ranking problem, and the fix is a reranker, not a new embedding model. The metric to compute here is NDCG@10 or precision@k on the final ordered list. RAGAS context precision measures exactly this: the retriever's ability to rank relevant chunks higher than irrelevant ones, computed as a position-weighted mean precision that penalizes relevant chunks appearing late in the list.
The standard fix is two-stage retrieval: a bi-encoder retrieves a wide candidate pool (top 100), then a cross-encoder reranker scores each query-chunk pair jointly and reorders, keeping the top 10. A bi-encoder embeds the query and chunk separately, so it flattens fine-grained relevance. A cross-encoder reads both together and catches distinctions the bi-encoder missed. In typical production pipelines, cross-encoder reranking lifts NDCG@10 by roughly 5 to 15 points, with larger gains on lexically hard datasets, under 200 ms of added latency per query for a small-to-mid reranker (larger 2B-class cross-encoders run closer to 200 to 400 ms).
If you want the full mechanics of how a cross-encoder differs from your first-stage retriever and when the latency is worth it, the deep dive on how reranking works in RAG walks through the two-stage pattern end to end. The diagnostic point for now: a ranking bug shows up as the gap between recall@100 and recall@10, and the target is to close that gap so the chunk the wide search found actually survives into the final context.
Step 3: were chunk boundaries cutting the answer in half?
If recall is poor at every k and you have ruled out the embedding model, suspect the chunks themselves. The failure mode is subtle: the answer spans a boundary, so no single chunk contains the complete fact, and every chunk scores as a weak partial match. A definition split from its term, a table separated from its caption, or a multi-sentence procedure cut mid-step all produce chunks that look relevant individually but never carry the full answer. Recall stays low no matter how you tune k, because the intact answer does not exist in any one chunk.
Diagnose this by reading the retrieved chunks for failed queries by hand. If you repeatedly see the answer truncated across two adjacent chunks, boundaries are the bug. The fixes are concrete: add overlap between chunks so spanning facts appear whole in at least one, chunk on semantic units (paragraphs, sections, list items) instead of a fixed token count, or attach a parent-document reference so a matched child chunk pulls in its surrounding context. The recipe for choosing chunk size and overlap covers how to pick these without guessing, and the symptom is recall that will not climb even as the candidate pool widens.
Step 4: was the context right but the answer wrong? Generation
If the retrieved context provably contains the answer and the model still got it wrong, the bug is generation. Good retrieval does not guarantee a good answer: the model can hallucinate, contradict the provided context, or miss the point of the question even with perfect evidence in the prompt. The metric here is faithfulness, which decomposes the generated answer into individual claims and checks what fraction can be attributed to the retrieved context. A low faithfulness score with high context recall is the signature of a pure generation bug.
One generation failure has a structural cause worth isolating: position. Language models do not use long contexts uniformly. The 2023 paper Lost in the Middle, by Nelson Liu and colleagues, found performance is highest when relevant information sits at the beginning or end of the input and degrades significantly when the model must use information buried in the middle, a U-shaped curve that holds even for models built for long contexts. If your gold chunk landed in position 14 of 20, the model may have skimmed past it.
This is why reranking pays off twice: it raises precision and places the strongest chunk where the model reads it. The breakdown of why long context degrades in the middle explains the curve and how to order context against it. When faithfulness is low and the evidence was present, your levers are ordering (put the best chunk first or last), trimming the context to fewer high-relevance chunks, and tightening the prompt to instruct grounded answering. The honest guide to reducing hallucinations covers the prompt-level fixes once you have confirmed the evidence was there to begin with.
The decision tree, end to end
- Pull the exact context sent to the model for the failed query, then ask: is the answer present in that string?
- Answer NOT in context, and recall@100 is low: first-stage retrieval bug. Check embedding model, indexing, and metadata filters (Step 1).
- Answer NOT in context, recall@100 high but recall@10 low: ranking bug. Add or tune a cross-encoder reranker (Step 2).
- Recall low at every k, and hand inspection shows truncated answers across adjacent chunks: chunk boundary bug. Add overlap or semantic chunking (Step 3).
- Answer IS in context but the model got it wrong: generation bug. Compute faithfulness; check whether the gold chunk was buried in the middle (Step 4).
- Faithfulness low with good recall: fix ordering, trim context, and tighten the grounding prompt, in that order.
The tree is strictly ordered for a reason. A reranker cannot rank a chunk that recall never retrieved. A prompt fix cannot ground an answer the model never received. Always walk from the bottom of the stack upward, and stop at the first stage that fails its target. Fixing a later stage while an earlier one is broken changes nothing and wastes a sprint.
Metrics and targets to set at each stage
| Stage | Symptom | Metric to compute | What a healthy result looks like |
|---|---|---|---|
| Step 1: Retrieval recall | Answer absent from candidate pool | Recall@100 (context recall) | High recall at wide k on the golden set; low recall means the embedding, index, or filter dropped the chunk |
| Step 2: Reranking | Chunk exists in pool but ranked below the cutoff | NDCG@10 and recall@10 vs recall@100 | Recall@10 close to recall@100; a large gap means a reranker is needed |
| Step 3: Chunk boundaries | Recall stays low at every k; answers look truncated | Recall@k plus manual chunk inspection | Each gold fact lives intact in at least one chunk; no answer split across a boundary |
| Step 4: Generation | Correct context, wrong or unsupported answer | Faithfulness and answer relevancy | Claims in the answer trace back to the retrieved context; failures often correlate with mid-context position |
Set absolute targets against your own golden set, not against numbers from a blog. The published figures here, recall clearing roughly 0.90 at wide k or reranking adding 5 to 15 NDCG points, are reference points, not contracts. Your corpus, your embedding model, and your query distribution shift the achievable ceiling. The discipline that matters is computing the same metric before and after a change on a frozen test set, so an improvement is something you measured rather than something you hoped for.
Fixing the stage you localized, not all five at once
The reason teams conclude RAG is unreliable is that they change five things at once and cannot attribute the result. The differential diagnosis exists to stop that. Once recall, NDCG, and faithfulness each have a number on a frozen golden set, a wrong answer routes to exactly one stage. You change one thing and re-measure. This is ordinary engineering applied to a pipeline most teams debug by vibes. Fine-grained diagnostic frameworks such as RAGChecker, published in 2024, formalize the same split by reporting separate diagnostic metrics for the retrieval and generation modules so failures attribute to the responsible component.
One failure class sits outside this tree: the right evidence was never retrievable because it lived only in an earlier session the system already forgot. A user states a constraint on Monday, asks a dependent question on Thursday, and the retriever has nothing to fetch because that fact was never persisted as durable, queryable memory. MemX is an external memory layer that stores facts and context across sessions and exposes them to retrieval, so prior-turn knowledge becomes a chunk your pipeline can actually recall. It does not replace the diagnosis above; it closes the gap where the answer was missing because the fact was never written down.
Run the tree in order on your next confident wrong answer. Pull the context, check whether the answer is in it, then walk recall, then ranking, then boundaries, then faithfulness. The bug will land in one stage. Fix that stage, re-measure on the golden set, and ship. The point is not that RAG never fails. The point is that every failure is attributable, and attribution turns guesswork into a one-line change.
01How do I know if a RAG wrong answer is retrieval or generation?
Pull the exact context string sent to the model for that query. If the answer is not in the context, the bug is retrieval. If the answer is in the context but the model said something else, the bug is generation. This single check routes you to the correct half of the pipeline.
02What metric tells me if the right chunk was retrieved?
Context recall, also called recall@k: the fraction of the gold chunks supporting an answer that appear in your retrieved candidate set. Compute it at a wide k like 100 first. If the chunk is missing even there, the bug is first-stage retrieval, not ranking or generation.
03When should I add a reranker to my RAG pipeline?
Add a cross-encoder reranker when recall@100 is high but recall@10 is low: the chunk is in the candidate pool but ranked below the cutoff. Reranking commonly lifts NDCG@10 by 5 to 15 points, under 200 ms of added latency per query for a small-to-mid reranker (larger models run closer to 200 to 400 ms).
04Why does my RAG return a wrong answer even with the right context?
That is a generation bug. The model can ignore or contradict provided evidence, especially when the relevant chunk sits in the middle of a long context, where models perform worst. Measure faithfulness, then fix context ordering, trim to fewer chunks, and tighten the grounding prompt.
05How are chunk boundaries causing RAG failures?
When an answer spans two adjacent chunks, no single chunk holds the complete fact, so every chunk scores as a weak partial match and recall stays low at every k. Read the retrieved chunks by hand: if answers look truncated, add chunk overlap or switch to semantic chunking.
