RAG Evaluation

RAG Evaluation is the measurement of whether a retrieval-augmented system returns answers that are grounded, complete, and genuinely useful, separating retrieval quality from generation quality so failures can be located.

It scores two stages: retrieval (was the right context fetched — context recall and precision) and generation (is the answer faithful to that context, and does it address the question). Collapsing the pipeline into one opaque score hides where it breaks.

For builders it is what turns RAG from a demo into something shippable: without it, a regression from a changed Chunking strategy, embedding model, or prompt goes unnoticed until users hit it. The hard part is assembling an evaluation set that mirrors real queries, which drift away from offline metrics fast.

Topic: RAG Related: RAG LLM Evals Reranking

Recent Updates

• 2026-07-14: QIMG-7 benchmark exposes how polluted multimodal RAG can fail, and SATR adds source-aware trust selection (cs.CL updates on arXiv.org) · arxiv.org — Saadeldine Eletter gpt-4o-mini
• 2026-07-14: Benchmarking faithfulness in LLM-generated clinical trial summaries (cs.CL updates on arXiv.org) · arxiv.org — OpenAI · Anthropic · Google ClinicalTrials.gov Aggregate Analysis of ClinicalTrials.gov · GPT-4o · Claude Sonnet 4.6 · Gemini 2.5 Flash
• 2026-07-14: CAFE turns compound AI evaluation into a factorial experiment (cs.CL updates on arXiv.org) · arxiv.org
• 2026-07-14: Abstention-aware RL for search agents (cs.LG updates on arXiv.org) · arxiv.org
• 2026-07-14: Eval-Pair Matrix studies same-model bias in grounded RAG judging (cs.CL updates on arXiv.org) · arxiv.org — GPT · Grok · Gemini
• 2026-07-14: EvidentialRAG adds uncertainty-aware conflict handling to multi-source RAG (cs.LG updates on arXiv.org) · arxiv.org
• 2026-07-14: Agentic search sharply reduces unsupported links in construction-code QA (Все статьи подряд / Искусственный интеллект / Хабр) · habr.com — DeepSeek · DeepSeek V4 Pro · GPT 5.5 · Claude Opus 4.8 · Claude Sonnet 5 · Gemini-3.1-Pro
• 2026-07-15: LLaMA 3 (8B) fine-tuned as an efficient RAG reranker to replace cross-encoders (cs.LG updates on arXiv.org) · arxiv.org — Shreeya Dasa Lakshminath LLaMA 3 (8B)
• 2026-07-15: Audio-Language Embeddings Fail on Negation: NegEval-Audio Benchmark (cs.CL updates on arXiv.org) · arxiv.org — CLAP
• 2026-07-15: On-Device Deep Research at 4B: Exposure Bounds Faithfulness, Retrieval Bounds Coverage (cs.CL updates on arXiv.org) · arxiv.org — Vinay Kumar Chaganti
• 2026-07-15: LakeQuest: benchmark for grounded QA over heterogeneous enterprise and scientific data lakes (cs.CL updates on arXiv.org) · arxiv.org
• 2026-07-15: Rethinking Evaluation in Retrieval-Augmented Personalized Dialogue: A Cognitive and Linguistic Perspective (cs.CL updates on arXiv.org) · arxiv.org — Tianyi Zhang
• 2026-07-15: LLaMA 3 (8B) fine-tuned as an efficient RAG reranker replacing cross-encoders (cs.CL updates on arXiv.org) · arxiv.org — Shreeya Dasa Lakshminath LLaMA 3 (8B)
• 2026-07-15: Habr experiment: benchmarking Karpathy-style LLM-wiki against simple vector RAG (Все статьи подряд / Искусственный интеллект / Хабр) · habr.com — Andrej Karpathy Gemma4-31B GPT-5.1
• 2026-07-16: On-Device Deep Research at 4B: Exposure Bounds Faithfulness, Retrieval Bounds Coverage (cs.AI updates on arXiv.org) · arxiv.org — Vinay Kumar Chaganti
• 2026-07-16: Learning Engagement Assistant (LEA): Cross-Course Scalability and Classroom Evaluation of an Agentic AI Tutoring System (cs.AI updates on arXiv.org) · arxiv.org
• 2026-07-16: Benchmarking SQL schema retrieval with corpus-adaptive embeddings (cs.CL updates on arXiv.org) · arxiv.org
• 2026-07-16: NVIDIA releases Nemotron 3 Embed for retrieval-focused RAG and agent workflows (Hugging Face - Blog) · huggingface.co — NVIDIA · Hugging Face · NVIDIA NeMo · vLLM NVIDIA NIM Nemotron 3 Embed Nemotron-3-Embed-8B-BF16 Nemotron-3-Embed-1B-BF16 llama-nemotron-embed-vl-1b-v2 · Nemotron 3 Ultra
• 2026-07-17: Certified Domain Consistency for Multi-Domain Retrieval (cs.LG updates on arXiv.org) · arxiv.org — Jayakumar Manoharan
• 2026-07-21: Shapley Context Pruning: Game-Theoretic Reranking and Pruning for RAG Contexts (cs.AI updates on arXiv.org) · arxiv.org
• 2026-07-24: TopoGuard uses graph topology to detect split-knowledge attacks in RAG (cs.CL updates on arXiv.org) · arxiv.org — arXiv LlamaGuard LlamaGuard-2-8B
• 2026-07-24: Paper measures ideological drift in news-grounded LLM QA (cs.AI updates on arXiv.org) · arxiv.org — QBias arXiv
• 2026-07-24: LegalCiteTrust benchmarks citation trustworthiness in Chinese legal research reports (cs.CL updates on arXiv.org) · arxiv.org — arXiv
• 2026-07-24: Gold-Anchored Evaluation and Guard for Citation Faithfulness in Agentic Scientific Synthesis (cs.AI updates on arXiv.org) · arxiv.org — OpenScholar PaperQA2
• 2026-07-28: How Context Attribution Handles What the Model Already Knows (cs.CL updates on arXiv.org) · arxiv.org

FAQ

What is RAG Evaluation?

RAG Evaluation measures whether retrieval and generation produce grounded, complete, and useful answers. GROUNDING tracks answer relevance, citation quality, faithfulness, recall, and evaluation datasets.

Which topic does RAG Evaluation belong to?

On the GROUNDING radar, RAG Evaluation is grouped under the RAG topic.

Which concepts are related to RAG Evaluation?

Related concepts tracked by the radar include RAG, LLM Evals, Reranking.