RAG Retrieval Evaluation: Check the Evidence Before Grading the Answer

RAG retrieval can give a system the wrong evidence, even when the language model follows that evidence carefully. Evaluate the retriever separately to see whether the needed material reaches the answering stage.

The core question is practical: for a realistic user question, did the system retrieve the passages needed to support a correct response, and did it place them where the next stage could use them? A single end-to-end score cannot always explain where that process failed.

Define what counts as relevant

Start with a small set of realistic questions and identify acceptable evidence for each. Store stable document identifiers, versions and the passages that support the answer. If a policy changed, an outdated paragraph may be topically relevant but unsuitable for the current question.

BEIR introduced a heterogeneous benchmark for evaluating retrieval across different tasks and domains. [1] The Sentence Transformers retrieval evaluator documents metrics including recall and ranking measures. [2] Their broader lesson for an editorial workflow is to define relevance and evaluation conditions explicitly rather than treating one similarity score as a complete result.

A reference set should allow more than one correct source when the question genuinely permits it. Otherwise a system may be penalized for finding valid evidence that the evaluator overlooked. Review disputed examples rather than quietly changing labels to make a preferred model win.

Calculate recall at a stated cutoff

The following example is fictional. A question needs three relevant passages, labeled A, B and C. The retriever returns five passages in this order: unrelated X, A, unrelated Y, B, unrelated Z.

These metrics describe different properties. Recall asks how much of the labeled evidence was found. Precision describes the proportion of returned material labeled relevant. The first relevant rank says how soon useful evidence appeared, but does not establish that the answer has all necessary support.

For a question requiring both a rule and an exception, retrieving only the rule may be inadequate despite finding something relevant quickly. Include a task-level judgment of evidence sufficiency alongside passage-level metrics.

A larger cutoff is not a free improvement

Increasing the number of retrieved passages can increase recall while introducing distractions, stale versions or contradictory text. It also changes the workload presented to the answering stage. Compare cutoffs with the same questions and labels, and show the tradeoff.

Suppose a retriever finds A, B and C in its first 20 results but only A in its first five. A high recall-at-20 result does not prove that a pipeline sending only five passages has enough evidence. Evaluation must reflect the actual cutoff and any reranking or filtering applied before generation.

Chunking also changes the unit being measured. Splitting one relevant document into ten near-identical fragments can inflate the apparent number of relevant hits. Our explanation of RAG chunking shows why preserving complete evidence matters. Deduplicate where appropriate and state whether labels apply to documents, passages or answer-bearing spans.

Keep document versions and access rules in the test

An evaluator may find an excellent passage that the real user is not allowed to access. A production test should respect the same permissions as the deployed system. Otherwise it measures a retrieval capability the user does not have.

Likewise, record the corpus snapshot. Adding a corrected document can change the evaluation without changing the embedding model. A stable experiment needs the question set, labels, corpus, chunker, embedding settings, filters and reranker versions.

Do not use confidential documents merely to make an evaluation look realistic. Representative synthetic documents can test workflows, but their results should be labeled as synthetic rather than presented as evidence of real production accuracy.

Inspect failures by type

A useful error review distinguishes missing documents, poor ranking, incomplete passages, stale versions and ambiguous questions. Each suggests a different repair. A missing source cannot be recovered by adjusting a similarity threshold. A stale source needs version management. A fragmented exception may need a different chunk boundary.

Include some questions that the corpus cannot answer. The right retrieval behavior may be to find no sufficient evidence, allowing the system to ask for more information or decline an unsupported answer. A test made only of answerable questions cannot evaluate that boundary.

Finally, evaluate the generated response separately for factual support, completeness and appropriate uncertainty. Strong retrieval is necessary for many tasks but does not guarantee that the model will use the evidence correctly.

Make the report actionable

Publish the question population, relevance rules, cutoff values, number of evaluated queries and the main failure categories. Use concrete anonymized examples to explain the largest gaps. Avoid claiming that a small in-house test establishes universal retrieval quality.

The result should tell an engineer what to fix and tell an editor what claims the evidence supports. By grading the evidence step independently, you can improve the part of the system that failed instead of repeatedly changing the language model and hoping the final score rises.

Average across questions without hiding difficult cases

Consider two fictional questions. Question One has one labeled relevant passage, which the retriever finds. Question Two has nine labeled relevant passages, of which it finds one. Their recalls are 100% and about 11.1%.

An equal-weight average across the two questions is about 55.6%. Pooling the retrieved relevant passages across both questions gives 2/10 = 20%. The first gives each question equal influence; the second gives more influence to questions with more labeled relevant material.

Neither number should be presented without its aggregation rule. The choice depends on the evaluation objective. If the service handles each user question as a task, equal question weighting may be useful. If the objective concerns coverage of all labeled evidence items, a pooled calculation answers another question.

Also show the distribution of results. A mean can conceal a minority of queries with no useful evidence. Those zero-evidence cases may matter more operationally than a modest improvement on already easy questions.

Separate retrieval and reranking experiments

A pipeline can retrieve an initial candidate set and then rerank it. Evaluate both stages under documented cutoffs. If the initial candidate set never contains the needed passage, a reranker cannot move that passage into the final list.

Suppose the initial set has 50 passages and includes A, B and C. A reranker sends five onward but drops C. Candidate recall is complete under the fictional labels, while final evidence recall is incomplete. That points to the second stage.

If C was absent from the 50 candidates, investigate corpus coverage, filtering and first-stage retrieval. This staged diagnosis makes an experiment more useful than changing multiple components at once and reporting only the final score.

Preserve ambiguity in the labels

Some questions need a single decisive passage. Others need several complementary sources. A broad question may be under-specified: the evaluator and retriever may assume different dates, jurisdictions or definitions.

Write a short explanation for relevance judgments, especially disputed ones. If two reviewers disagree, record the disagreement and resolve it using the task specification. Do not treat every difference as careless annotation; it may reveal that the question itself needs clarification.

For a policy question, a source may be relevant only for a particular effective period. A passage from an earlier version can be topically similar yet unsuitable for the user’s date. Include version validity in the label rather than scoring all mentions of the topic as interchangeable.

Test failure handling as a separate capability

Add fictional or clearly designated questions whose answers are absent from the corpus. Specify what sufficient evidence would look like if it existed. Then observe whether the pipeline retrieves merely similar text and turns it into an unsupported response.

A system that honestly reports missing evidence may be more useful than one that always produces an answer. Evaluate whether it identifies the missing information and offers a relevant next step without inventing a source.

Keep these absent-evidence questions separate in the report. Combining them with answerable questions under an unexplained metric can make the result difficult to interpret. The reader should know which capability each subset tests.

Build a small reproducible experiment record

For each run, save the question identifier, corpus snapshot, retrieval configuration, candidate ranks, final ranks, labels and any evidence-sufficiency judgment. A later engineer should be able to inspect why one question failed without rerunning a changing live service.

Keep operational constraints in the record as well. If a method doubles retrieval time or requires a much larger context, compare that cost with the observed quality improvement. A higher offline metric does not automatically make the full workflow better for users.

The most useful result is a set of specific failure explanations tied to a stable task population. It tells the team whether the next improvement belongs in source coverage, chunk design, ranking, permissions, generation or the question specification itself.

Compare two candidate changes with the same questions

Suppose a fictional evaluation has ten questions and an established relevance set. Configuration A finds sufficient evidence for six questions in its final context. Configuration B finds sufficient evidence for seven. Before calling B better, inspect which questions changed and what else changed in the workflow.

If B solves two previously failed questions but loses one previously successful question, the net gain is one. Reporting only the two wins hides the regression. A paired question-level table makes both visible.

Also record retrieval latency and the amount of text sent onward. If B retrieves much more material, the improvement may be useful but have operational costs. The appropriate choice depends on the task and constraints, not a universal rule that a one-question gain always wins.

The ten-question sample is deliberately small. It is suitable for explaining a comparison, not for establishing a stable population-wide effect. A real evaluation should assess uncertainty and how well the questions represent intended use.

Make source freshness part of the task

A query asking for the current policy needs evidence valid at the query’s date. A historical question may require the policy in force at an earlier date. The same passage can therefore be acceptable for one task and inappropriate for the other.

Include the requested date in the question specification when it matters. Store document effective dates and revision identifiers where the corpus supports them. A publication date alone may not be the policy’s effective date.

A fictional retriever that returns a 2022 policy for a 2026 current-policy question should not receive full credit solely because the topic and wording match. The evaluator needs the right temporal relevance rule.

Conversely, deleting all older documents could damage historical retrieval. Version management should support the intended questions rather than assume every old source is always wrong.

Test source diversity only when the question needs it

Some tasks need one authoritative source. Others require comparing several independent sources or finding a rule and its exception. Do not add a universal diversity quota that rewards irrelevant documents merely because they come from different domains.

For a fictional question requiring two complementary sections of one official document, retrieving that document’s two relevant passages may be more useful than retrieving three unrelated websites. The relevance specification should reflect the answer’s evidence needs.

When independent confirmation is required, label which sources are genuinely independent. Two pages repeating the same release do not necessarily supply two independent observations. That distinction belongs in the editorial evaluation rather than being inferred from different URLs alone.

Keep manual judgments auditable

A human evaluation can be strong evidence for a defined task, but it needs consistent instructions. Give reviewers examples of sufficient, partial, irrelevant and outdated evidence. Preserve their original judgments and document any adjudication.

If reviewers can see which model produced a result, preferences may affect judgment. Where feasible, conceal configuration names during comparison and use the same review process for every candidate. State any remaining limitations.

Do not present an AI judge’s score as human-reviewed evidence. If automated grading is used, describe it and test important failure cases against independent judgments. A judge can miss a plausible but unsupported passage, especially when it shares the answering system’s assumptions.

Choose the next experiment from the error record

If most failures involve missing source documents, improve corpus coverage before investing in a new ranking model. If sources are present but buried, test ranking. If passages omit qualifications, inspect chunk boundaries. If the evidence is sufficient but answers remain unsupported, investigate generation and citation behavior.

This sequence keeps each experiment connected to a diagnosed problem. It also produces an article readers can learn from: a defined question, visible evidence, a transparent measure and a limitation that points to the next useful test.

Preserve the baseline even when a new method wins

Keep the original configuration and its results in the experiment record. A later change in the document corpus can alter the comparison, so a winning score should remain tied to its snapshot. This makes regressions easier to diagnose and prevents a past result from being reused as proof of current quality after the task or source collection has changed.

Sources and method

1. BEIR benchmark paper. 2021. Checked October 20, 2026.
2. Sentence Transformers information retrieval evaluator. Living documentation; publication date not established. Checked October 20, 2026.

Numerical teaching examples are original and illustrative unless explicitly identified otherwise. No hands-on product testing or individual medical assessment is claimed.

To connect retrieval performance with chunk completeness and score interpretation, see Why a Cosine Similarity Score Is Not a Probability.