Models & Evaluation

RAG Evaluation

RAG evaluation is the practice of measuring a retrieval-augmented generation system by scoring its retrieval stage and its generation stage separately, using metrics such as recall@k and context relevance for retrieval and faithfulness and answer relevance for generation.

What RAG evaluation measures

RAG evaluation is the process of quantifying how well a retrieval-augmented generation (RAG) system answers questions over an external corpus. A RAG system has two cooperating components: a retriever that selects context passages from a knowledge source, and a generator (a large language model) that conditions its answer on those passages. Because failures can originate in either component, a single end-to-end accuracy number is rarely enough to diagnose a system. Evaluation is therefore decomposed so that retrieval quality and generation quality are scored on separate axes.

The central idea is attribution of blame. If a final answer is wrong, the question is whether the retriever failed to surface the supporting passage, or whether the generator had the right passage but ignored it, contradicted it, or fabricated detail. Splitting the metrics lets a team localize the defect: a low retrieval score points at chunking, indexing, or ranking, while a low generation score with good retrieval points at the prompt or the model. The RAGAS paper frames RAG evaluation around exactly these dimensions: whether retrieval finds relevant and focused passages, whether the model uses them faithfully, and the quality of the generated text itself.

RAG evaluation can be reference-based, comparing outputs against human-written ground-truth answers and gold passages, or reference-free, scoring properties such as faithfulness directly from the question, the retrieved context, and the response without gold labels. Reference-free scoring is attractive in production where labeled data is scarce, and it is the design goal behind frameworks like RAGAS and TruLens.

Two axes: retrieval quality (did the right context get fetched and ranked well) and generation quality (did the model answer faithfully and relevantly).
A single end-to-end score hides which stage failed, so component metrics are needed for diagnosis.
Reference-based scoring needs gold answers and gold passages; reference-free scoring grades properties of the output directly.
The same question, retrieved context, and answer triple feeds most automated RAG metrics.

Retrieval metrics

Retrieval metrics borrow from classical information retrieval and judge the ranked list of passages returned for a query. Recall@k asks whether the relevant passages appear anywhere in the top k results, which matters because the generator can only use what the retriever surfaces. Precision@k asks what fraction of the top k are actually relevant, penalizing noisy context that can distract or mislead the model. These two are order-insensitive within the top k.

Rank-aware metrics reward placing relevant passages near the top. Mean Reciprocal Rank (MRR) averages the reciprocal of the rank position of the first relevant result across queries, so a relevant hit at position 1 scores 1.0 and at position 5 scores 0.2. Normalized Discounted Cumulative Gain (NDCG) generalizes this to graded relevance and to multiple relevant passages by discounting gains logarithmically with rank, then normalizing against the ideal ordering so scores fall in 0 to 1. Rank-aware metrics are important when downstream prompts truncate context, since a relevant passage buried below the cutoff is effectively lost.

When gold relevance labels are unavailable, context relevance offers a reference-free alternative: an LLM-as-a-judge scores how much of the retrieved context is pertinent to the query. RAGAS provides related context precision and context recall metrics, where context precision rewards ranking relevant chunks above irrelevant ones and context recall measures how well the retrieved set covers the information needed to answer.

recall@k = (number of relevant passages in top k) / (total number of relevant passages)

Recall@k for a single query; the system-level score averages over all queries.

NDCG@k = DCG@k / IDCG@k,   where DCG@k = Σ_{i=1..k} relᵢ / log₂(i + 1)

NDCG normalizes discounted cumulative gain against the ideal ranking (IDCG), giving a 0 to 1 score; relᵢ is the graded relevance of the result at rank i.

Recall@k and precision@k judge presence and purity of relevant passages in the top k but ignore order within it.
MRR rewards the rank of the first relevant hit; NDCG rewards graded relevance across the whole ranked list.
Context relevance and context precision/recall let you score retrieval without gold labels, using an LLM judge.
Rank-aware metrics matter most when the prompt truncates or reorders context before the model sees it.

Generation metrics

Generation metrics judge the answer given the retrieved context, holding retrieval fixed. Faithfulness, also called groundedness, measures whether the claims in the answer are supported by the provided context rather than invented. It is the primary defense against hallucination in RAG, because a fluent answer that contradicts or exceeds its sources is still wrong. RAGAS computes faithfulness by decomposing the response into individual claims and checking how many are entailed by the retrieved context.

Answer relevance (RAGAS calls it answer relevancy) measures whether the response actually addresses the user's question rather than drifting onto a related but unasked topic. A high-faithfulness answer can still score poorly on relevance if it is grounded yet off-topic or incomplete. RAGAS estimates answer relevancy by prompting an LLM to generate questions that the answer would address, then measuring their embedding similarity to the original question. Faithfulness and answer relevance are complementary: one checks grounding, the other checks usefulness.

Hallucination is the failure mode these metrics are built to catch: content that is unsupported by the retrieved context or factually false. In a RAG setting, hallucination is operationalized as low faithfulness, since the retrieved passages define the ground truth the answer should stay within. Many of these scores are produced by an LLM-as-a-judge, a separate model prompted to rate the answer, which scales evaluation when human annotation is too slow or expensive but inherits the judge model's own biases.

faithfulness = (number of answer claims supported by the retrieved context) / (total number of claims in the answer)

The RAGAS faithfulness score, ranging from 0 to 1; the answer is decomposed into atomic claims that are each checked for entailment against the context.

Faithfulness (groundedness) checks that answer claims are supported by the retrieved context; it is the main hallucination guardrail.
Answer relevance checks that the response addresses the question, independent of whether it is grounded.
A grounded answer can be irrelevant and a relevant answer can be ungrounded, so both axes are scored.
LLM-as-a-judge makes these scores scalable but adds judge-model bias that must itself be checked.

Frameworks: RAGAS and TruLens

RAGAS ("Ragas: Automated Evaluation of Retrieval Augmented Generation", arXiv:2309.15217, submitted September 2023) is an open-source framework for reference-free RAG evaluation. It defines faithfulness, answer relevancy, context precision, and context recall, among other metrics, and computes them with LLM prompting so that teams can score pipelines without large hand-labeled datasets. The framework exposes both a high-level evaluate function over a dataset and per-metric scorers for finer control.

TruLens organizes RAG evaluation around the RAG triad: context relevance, groundedness, and answer relevance. Context relevance grades the retrieved passages against the query, groundedness grades the answer against those passages, and answer relevance grades the answer against the original question. Each leg is computed with an LLM-as-a-judge, and the framing is that satisfactory scores on all three legs give confidence the system is not hallucinating, because the triad covers each hop from query to context to answer.

The two frameworks overlap heavily in concept and differ mainly in vocabulary and tooling. Both separate retrieval-side from generation-side quality, both lean on LLM judges for reference-free scoring, and both treat faithfulness or groundedness as the central hallucination signal. Choosing between them is largely a matter of integration and instrumentation preferences rather than a disagreement about what RAG evaluation should measure.

python

# pip install ragas datasets
from datasets import Dataset
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision, context_recall

# One evaluation row: the question, the model's answer, the retrieved
# passages, and a ground-truth answer for reference-based metrics.
data = {
    "question": ["When was the first Super Bowl?"],
    "answer": ["The first Super Bowl was held on January 15, 1967."],
    "contexts": [["The first AFL-NFL World Championship Game was played on January 15, 1967."]],
    "ground_truth": ["The first Super Bowl was held on January 15, 1967."],
}
dataset = Dataset.from_dict(data)

result = evaluate(
    dataset,
    metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
)
print(result)  # e.g. {'faithfulness': 1.0, 'answer_relevancy': 0.98, ...}

A minimal RAGAS run scoring one RAG example on faithfulness, answer relevancy, context precision, and context recall. Newer RAGAS versions also expose collections-based scorers such as ragas.metrics.collections.Faithfulness.

RAGAS (arXiv:2309.15217) defines faithfulness, answer relevancy, context precision, and context recall for reference-free scoring.
TruLens frames evaluation as the RAG triad: context relevance, groundedness, and answer relevance.
Both frameworks compute their scores with an LLM-as-a-judge rather than requiring gold labels.
The triad maps to the query to context to answer hops, mirroring the retrieval/generation split.

Tradeoffs and pitfalls

LLM-as-a-judge metrics are convenient but not free of error. Judge models exhibit known biases, including a tendency to favor longer or more confidently worded answers and to prefer outputs that resemble their own style, so faithfulness and relevance scores should be calibrated against a sample of human judgments before they are trusted as release gates. Because the judge is itself a model, its scores can drift when the judge model is upgraded, which makes version pinning and periodic recalibration part of a credible evaluation setup.

Decomposing metrics also has limits. High retrieval recall with low precision can still degrade answers if irrelevant passages crowd out the relevant one or push it past a context cutoff, an interaction that order-insensitive metrics miss. Conversely, a strong generator can sometimes answer correctly from parametric knowledge even when retrieval fails, inflating end-to-end accuracy while masking a broken retriever. Reporting retrieval and generation metrics side by side, rather than only the final score, keeps these interactions visible.

Finally, reference-free metrics measure consistency with the retrieved context, not truth. An answer can be perfectly faithful to a context passage that is itself wrong or stale, so faithfulness bounds hallucination relative to the corpus but does not certify factual correctness. Sound RAG evaluation therefore pairs automated metrics with spot-checked human review and with corpus-quality checks, treating the automated scores as fast signal rather than final ground truth.

LLM judges carry biases (length, self-preference) and drift across model versions, so calibrate against human labels.
Component metrics can miss interactions, such as low precision pushing the relevant passage past the context cutoff.
A strong generator can mask a weak retriever via parametric knowledge, inflating end-to-end accuracy.
Faithfulness measures grounding in the corpus, not truth, so a wrong source yields a faithful but false answer.

Key takeaways

RAG evaluation splits into retrieval metrics and generation metrics so failures can be attributed to the right component; you cannot fix what a single end-to-end number hides.
Retrieval is judged with recall@k, precision@k, MRR, NDCG, and reference-free context relevance.
Generation is judged with faithfulness (groundedness), answer relevance, and hallucination rate.
RAGAS (arXiv:2309.15217) defines faithfulness, answer relevancy, context precision, and context recall for label-free scoring.
TruLens organizes evaluation as the RAG triad: context relevance, groundedness, and answer relevance.
Faithfulness measures consistency with the retrieved context, not factual truth, so it must be paired with corpus and human checks.

Frequently asked questions

Retrieval metrics score the passages the retriever returns for a query, using measures like recall@k, precision@k, MRR, and NDCG, or reference-free context relevance. Generation metrics score the answer the model produces given that context, using faithfulness, answer relevance, and hallucination rate. Splitting them lets you tell whether a wrong answer came from missing context or from a model that misused good context.

RAGAS defines faithfulness, answer relevancy, context precision, and context recall, among other metrics, and computes them with LLM prompting so no gold labels are required. Faithfulness is the fraction of answer claims supported by the retrieved context, and answer relevancy measures how directly the answer addresses the question. The framework is described in the paper "Ragas: Automated Evaluation of Retrieval Augmented Generation" (arXiv:2309.15217).

Faithfulness decomposes the answer into individual claims and checks how many of them are supported by the retrieved context, then divides supported claims by total claims for a score from 0 to 1. A score of 1 means every claim is grounded in the context. RAGAS performs this decomposition and entailment check with an LLM, which is why it is a form of LLM-as-a-judge evaluation.

The RAG triad is TruLens's framing of RAG evaluation as three metrics: context relevance, groundedness, and answer relevance. Context relevance scores retrieved passages against the query, groundedness scores the answer against those passages, and answer relevance scores the answer against the original question. Passing all three gives confidence the system is not hallucinating, since the triad covers every hop from query to context to answer.

No. Reference-free metrics such as faithfulness, answer relevance, and context relevance score properties of the question, context, and answer directly using an LLM judge, which is the design goal of RAGAS and TruLens. Ground-truth answers and gold passages are still useful for reference-based metrics like recall@k and context recall, and for calibrating judge scores against human judgment.

Yes. Faithfulness measures whether the answer is consistent with the retrieved context, not whether the context is true. If the corpus contains a stale or incorrect passage, an answer that mirrors it scores high on faithfulness while being factually wrong. For this reason careful evaluation pairs automated faithfulness scores with corpus-quality checks and spot-checked human review.

Put the idea into practice

MemX is an AI memory agent built on these ideas: store anything, skip the folders, and find it again by asking in plain English.

Try MemX Free