LLM Evals are the tests and measurement systems used to judge how well a model or LLM-powered system performs — on task success, accuracy, safety, retrieval quality, or agent behavior — so changes can be compared objectively rather than by impression.
They range from automated benchmarks and reference-based scoring to LLM-as-judge grading and human review, usually run against a curated dataset of representative cases. Practice is shifting from static text benchmarks toward task-based, system-level evaluation.
For builders evals are what make iteration safe: without them, a prompt, model, or RAG change that fixes one case silently breaks others. The hard part is building an eval set that reflects real usage and resists gaming, since any single metric can be optimized at the expense of actual quality.
Topic: LLMs Related: RAG Evaluation Agents Code Agents
Recent Updates
- 2026-07-24: CANN Bench benchmarks agent-generated kernels on Huawei Ascend NPUs (cs.AI updates on arXiv.org) · arxiv.org — Huawei · arXiv
- 2026-07-24: REFACT proposes adaptive fact restatement for grounded reasoning (cs.CL updates on arXiv.org) · github.com — arXiv · GitHub
- 2026-07-24: LLM math performance shifts across equivalent representations, and code execution does not remove the brittleness (cs.AI updates on arXiv.org) · arxiv.org
- 2026-07-24: LegalCiteTrust benchmarks citation trustworthiness in Chinese legal research reports (cs.CL updates on arXiv.org) · arxiv.org — arXiv
- 2026-07-24: ConfidenceBench evaluates verbalized confidence calibration in 15 LLMs (cs.AI updates on arXiv.org) · arxiv.org — arXiv · Claude Opus 4.6 · Gemini 3.1 Pro Preview Gemini 3.1 Flash-Lite
- 2026-07-24: Gold-Anchored Evaluation and Guard for Citation Faithfulness in Agentic Scientific Synthesis (cs.AI updates on arXiv.org) · arxiv.org — OpenScholar PaperQA2
- 2026-07-24: Tencent WorkBuddy Bench introduces an open multi-domain coding-agent benchmark (cs.CL updates on arXiv.org) · arxiv.org — Tencent
- 2026-07-24: From Word-Level Dictionary to Sentence-Level Semantics for Multilingual Grievance Labeling (cs.CL updates on arXiv.org) · github.com — arXiv · Hugging Face
- 2026-07-24: LLMs Struggle When User Intent Changes Mid-Conversation (cs.LG updates on arXiv.org) · arxiv.org
- 2026-07-24: Anthropic introduces Claude Opus 5 (ElKornacio) · anthropic.com — Anthropic · Zapier · Claude Opus 5 · Claude Fable 5 · Mythos 5 · Opus 4.8 Claude Max Claude Pro
- 2026-07-24: Claude Opus 5 launches with stronger coding and knowledge-work benchmark results (Hacker News) · anthropic.com — Claude Opus 5 · Claude Fable 5 · Mythos 5 · Opus 4.8
- 2026-07-24: CORE: Contrastive Reflection for faster reasoning improvement (alphaXiv) — Linas Nasvytis Simon Jerome Han Ben Prystawski Satchel Grant Noah D. Goodman Judith E. Fan GRPO GEPA MemRL
- 2026-07-24: Drone-Bench measures frontier models on simple drone surveillance tasks (Hacker News) · andonlabs.com
- 2026-07-24: Opus 5 tops the Artificial Analysis Intelligence Leaderboard (Hacker News) · artificialanalysis.ai — Artificial Analysis · Claude Opus 5 · Claude Fable 5 · GPT-5.6 Sol Mercury 2 HyperNova 60B 2605 Granite 4.0 H Small Gemma 3n E4B Instruct Nova Micro Sarvam 30B · Gemini 2.5 Flash Lite · Command A+ · Gemini 2.5 Flash · GLM-5.2 · MiniMax M3 · DeepSeek V4 Pro
- 2026-07-25: A catalog of 3,607 reported AI agent misbehavior incidents (Hacker News) · rewardhacking.org — GitHub · Hacker News LessWrong · X
- 2026-07-25: UK AISI and CAISI publish a preliminary cyber-capability assessment of Kimi K3 (Hacker News) · nist.gov — UK Artificial Intelligence Security Institute U.S. Center for AI Standards and Innovation Moonshot AI · Carnegie Mellon University · Kimi K3 · GLM-5.2
- 2026-07-25: ARC-AGI-3 leaderboard compares performance against task cost (Hacker News) · arcprize.org — Kaggle ARC Prize ARC-AGI-1 ARC-AGI-2 ARC-AGI-3 GPT-4.5 Claude 3.7 o1 pro · Gemini 3 Pro
- 2026-07-25: Anthropic’s Opus 5 is reported to match or beat Fable 5 on several benchmarks while Claude Code’s system prompt was cut by more than 80% (量子位) · qbitai.com — Anthropic · QbitAI · Cursor · OpenAI AlphaSchool 克雷西 am.will OmedTheVibeCoder Alex Ermolov Chetaslua Noema Matt Shumer Victor M Thariq · Opus 5 · Opus 4.8 · Fable 5 · GPT-5.6 Sol · GPT-5.6 · Kimi K3
- 2026-07-25: Claude Code trims system prompts as context engineering shifts (Hacker News) · claude.com — Claude Opus 5 · Claude Fable 5
- 2026-07-26: A personal report on agentic coding, testing, and model variance (Hacker News) · danluu.com — OpenAI · Mastodon · Playwright · GPT-5.0 · GPT-5.1
- 2026-07-26: World-model-optimizer: Continuously optimize agents and reduce inference costs through model routing and distillation (Hacker News) · github.com — experientiallabs OpenRouter · e2b
- 2026-07-27: The Regression Tax: Decomposing Why Skills Help and Hurt LLM Agents (cs.AI updates on arXiv.org) · arxiv.org
- 2026-07-27: Terence Tao warns mathematics faces century-long crisis as AI proves capable of solving research-level problems (量子位) · qbitai.com — Terence Tao Wang Hong Deng Yu Jean Bourgain William Thurston
- 2026-07-27: MirrorCode Benchmark: AI Systems Complete Complex Programming Tasks in Hours Instead of Weeks (Import AI) · importai.substack.com — Epoch METR · Anthropic · Apple · Claude Opus 4.7 · GPT 5.5
- 2026-07-28: Small language models need fine-tuning to effectively use provided legal context (cs.CL updates on arXiv.org) · huggingface.co — Qwen3.5
FAQ
What is LLM Evals?
LLM Evals are tests and measurement systems for model behavior, agent performance, retrieval quality, safety, and task success. GROUNDING tracks eval benchmarks, failure modes, and practical measurement patterns.
Which topic does LLM Evals belong to?
On the GROUNDING radar, LLM Evals is grouped under the LLMs topic.
Which concepts are related to LLM Evals?
Related concepts tracked by the radar include RAG Evaluation, Agents, Code Agents.