How to evaluate RAG.
Find the failure, not just the score.

A final answer hides where the pipeline went wrong.

A retrieval-augmented generation system selects material from a collection and asks a model to answer using that context. If the answer is wrong, the search may have missed a relevant passage; the passage may have been split badly or ranked too low; the source may be stale; or the model may have ignored or misread evidence it did receive. For a plain-language walkthrough of the system, start with what RAG is and how its stages fit together ↗.

Those failures call for different fixes. Record what was retrieved and what the model saw alongside its answer. RAGChecker’s NeurIPS paper takes the same diagnostic approach by measuring retrieval and generation as separate modules as well as examining the full response.

Start with real questions and a human-readable expected result.

Collect representative questions from the task the system is meant to support. For each one, note the relevant source or passage, the facts a good answer should include, and whether the right response is to abstain. Keep this evaluation set separate from examples used to tune prompts, retrieval, or a model.

Include the corpus version and the question’s expected outcome with every case. Without those details, a changed source collection can make yesterday’s score difficult to interpret.

Measure retrieval, grounding, and usefulness on their own terms.

There is no universal RAG score. A retrieval metric diagnoses a search stage; a grounding measure checks a relationship between text and evidence; a task review asks whether the system helped a person. Keep the labels distinct.

Hold the questions steady while you change one part.

Run a baseline, save the retrieved passages and answers, then change one meaningful variable: document parsing, chunk size, embedding model, ranking, context length, prompt, or generator. Re-run the same cases and inspect examples where the result improved and where it regressed.

Report results by case type as well as in aggregate. A mean can hide a retrieval method that helps common questions but fails on rare terminology, multi-part questions, or missing evidence. BEIR’s heterogeneous information-retrieval benchmark was designed to examine retrieval across different datasets and domains; its central lesson for a product team is to check the range of the task, not only a comfortable slice of it.

Version the corpus, index, model, prompt, and test set. That small record makes comparisons reproducible and helps explain why an answer changed after a release.

An automated grader can scale inspection. It cannot define quality for you.

Model-based evaluators can help label large batches for properties such as relevance or support. First check their decisions against human-reviewed examples from your own domain, including borderline cases. Inspect disagreements and keep a small set of human checks when the prompt, model, or content changes.

RAGChecker reports stronger correlation with human judgments than the metrics it compared in its own meta-evaluation. That is evidence about the paper’s study, not a guarantee that one metric or grader is reliable for every corpus, language, and task.

Use an evaluator to point toward examples worth reviewing. Keep the rubric, evidence, and final judgment inspectable. See how to validate an LLM judge ↗.

Match the symptom to the layer that can fix it.

For how the search representation is built, read what embeddings are and how vector search ranks passages ↗. For the design choice between finding current information and changing a model’s behavior, read RAG or fine-tuning? ↗. For agent workflows, continue with how to evaluate an AI agent ↗ or how to read a benchmark ↗.