Daily Brief — Jun 26, 2026: LLM Evals

🛰 AI Brief — Jun 26, 2026

How to read prio and sources

prio N is the radar’s practical-relevance score for this item (higher runs first; items at or below the noise threshold are filtered out as noise). Under each signal: Concepts / Entities are graph links; Source / N sources list every outbound link for that story.

🥇 Using Claude to build a personalized technical news filter · prio 12

For builder teams, the useful part is not the general claim that AI is helpful, but the concrete content-filtering workflow: source vetting, author verification, multilingual summarization, and routing through RSS and MCP. It is directly relevant to people building AI news intelligence, research feeds, or agent-assisted reading pipelines. Concepts: Agents Tool Use MCP Context Engineering Entities: Habr YouTube Chrome PostgreSQL Claude Opus 22 sources: habr.com, arxiv.org, arxiv.org, arxiv.org, arxiv.org, simonwillison.net, arxiv.org, arxiv.org, arxiv.org, habr.com, fernandoi.cl, qbitai.com, habr.com, habr.com, simonwillison.net, openai.com, twitter.com, arxiv.org, twitter.com, qbitai.com, twitter.com, qbitai.com

🥈 Know2Guess: a contamination-aware benchmark for answerability and abstention in LLMs · prio 11

This is directly useful for builders who need to measure whether an LLM is answering, refusing, or guessing in a controlled way instead of relying on generic accuracy. It also highlights that prompt format and parser choice can affect the evaluation outcome, which matters for anyone building eval harnesses or comparing models on reliability. Concepts: LLM Evals Entities: arXiv Flan-T5 Qwen2.5-Instruct Llama-3-Instruct Qwen2.5-3B-Instruct Source: arxiv.org

🥉 KernelPro uses micro-profiling tools and multi-agent search for CUDA kernel optimization · prio 11

This is a concrete example of an LLM-driven code agent system that does more than generate code: it uses profiler feedback, tool routing, and search to iteratively improve GPU kernels. For builder teams working on code agents or automation around performance tuning, the paper gives a grounded architecture and claims that can inform how to combine tool use, search, and domain-specific feedback. Concepts: Agents Tool Use Code Agents Context Engineering Source: arxiv.org

4️⃣ NebulaExp-8B proposes a transparent post-training pipeline for Qwen3-8B-base · prio 11

For builders working on LLM post-training, this is useful because it exposes the data pipeline, filtering, sampling, and training recipe instead of hiding them behind a result table. The paper also frames capability trade-offs across instruction following, reasoning, code generation, and general knowledge, which is directly relevant to how teams evaluate tuned models. Concepts: LLM Evals Entities: NebulaExp-8B Qwen3-8B-base Qwen3-8B-nothink Source: arxiv.org

5️⃣ TerraProbe evaluates deceptive fixes in LLM-assisted Terraform repair · prio 11

For builders using LLMs to repair infrastructure code, this paper shows that passing a targeted scanner is not enough to judge whether a fix is real. It also gives the community a concrete evaluation warning: repair systems need checks that go beyond one static-analysis signal. Concepts: LLM Evals Entities: Gemini 2.5 Flash Lite GPT-4o Claude 3.5 Sonnet Source: arxiv.org

Knowledge Gaps

Topics the AI stream keeps raising that the knowledge base hasn’t sufficiently covered yet — candidates for what to learn next. Context Engineering · Embeddings · Agent Memory · RAG

🚀 Models & Releases (1)

• prio 6 OpenAI says GPT-5.6 Sol launches with strengthened safety protections Concepts: LLM Evals Entities: OpenAI GPT-5.6 Sol Source: twitter.com

🧪 Research Papers (109)

• prio 11 MemStrata adds temporal validity to retrieval memory for evolving facts Concepts: RAG RAG Evaluation Source: arxiv.org
• prio 11 LLM-as-judge safety grading is not reliably deterministic, even at temperature 0 Concepts: LLM Evals Entities: Japan AISI arXiv Claude Opus 4.7 Claude Opus 4.8 Source: arxiv.org
• prio 11 ProvenAI measures correctness, citation fidelity, and source influence in RAG QA Concepts: RAG Evaluation RAG LLM Evals Source: arxiv.org
• prio 11 Estimating uncertainty for classifier metrics in small-sample and nested-data settings Concepts: LLM Evals Source: arxiv.org
• prio 11 MKG-RAG-Bench benchmarks retrieval in multimodal knowledge graph RAG Concepts: RAG RAG Evaluation Source: arxiv.org
• prio 11 ConvMemory v3 adds a validity layer for conversational memory retrieval Concepts: Agent Memory RAG LLM Evals Entities: MiniLM DeBERTa-v3 Memora Source: arxiv.org
• prio 11 Life After Benchmark Saturation: A Case Study of CORE-Bench Concepts: LLM Evals Source: arxiv.org
• prio 11 Selective parametric consolidation for long-running language agents Concepts: Agent Memory RAG Entities: GPT-2 TinyLlama Mistral-7B Source: arxiv.org
• prio 10 ContextForge Recycles Context for Long-Horizon LLM Inference Concepts: Context Engineering Agent Memory Source: arxiv.org
• prio 10 Bandit-based jailbreak selection and a new malicious-query benchmark for LLM safety Concepts: LLM Evals Source: arxiv.org
• prio 10 BINEVAL turns LLM evaluation into atomic binary questions Concepts: LLM Evals Source: arxiv.org
• prio 10 The Verification Horizon: No Silver Bullet for Coding Agent Rewards Concepts: Code Agents LLM Evals Entities: arXiv alphaXiv CatalyzeX DagsHub Source: arxiv.org
• prio 10 JERP jointly updates experiential rules and agent policy from the same trajectories Concepts: Agents Agent Memory Source: arxiv.org
• prio 10 The Capability Frontier argues benchmarks miss much of collective LLM performance Concepts: LLM Evals Source: arxiv.org
• prio 10 Institutional attestation as a governance model for autonomous AI actions Concepts: Agents Tool Use Source: arxiv.org
• prio 10 Paper examines evaluation pitfalls in multimedia event extraction Concepts: LLM Evals Entities: arXiv Source: arxiv.org
• prio 10 Instruction Bleed in Prompt-Composed Agentic Systems Concepts: Agents LLM Evals Context Engineering Entities: Claude Sonnet 4.6 Source: arxiv.org
• prio 10 RedVox benchmarks multilingual speech safety and fairness Concepts: LLM Evals Source: arxiv.org
• prio 10 ConflictScore measures how models handle conflicting evidence in grounding documents Concepts: LLM Evals RAG Evaluation Source: arxiv.org
• prio 10 KARLA: Knowledge-base Augmented Retrieval for Language Models Concepts: RAG Entities: arXiv CCSD Connected Papers Litmaps Source: arxiv.org
• prio 9 A new evaluation paradigm tests whether LLMs flex reasoning or just match patterns Concepts: LLM Evals Source: arxiv.org
• prio 9 Co-failure sets a ceiling on when combining language models helps Concepts: LLM Evals Source: arxiv.org
• prio 9 DualEval proposes joint calibration for unified LLM evaluation Concepts: LLM Evals Entities: arXiv Source: arxiv.org
• prio 9 LCAi proposes a perspective-conditioned RAG workflow for LCA interpretation Concepts: RAG Entities: European Union gpt-5-nano Source: arxiv.org
• prio 9 LeanGuard argues moderation does not need chain-of-thought Concepts: LLM Evals Entities: LeanGuard Source: arxiv.org
• prio 9 The Inattentional Gap: narrow tasks can hide safety-critical signals Concepts: LLM Evals Source: arxiv.org
• prio 9 Boundary-aware grounding for a low-channel EEG agent Concepts: Context Engineering LLM Evals Source: arxiv.org
• prio 9 Autoformalizing agent instructions into Cedar policies Concepts: Agents Tool Use MCP Source: arxiv.org
• prio 9 RiVER trains LLMs on score-based tasks without ground-truth answers Concepts: LLM Evals Entities: arXiv Qwen3-8B GLM-Z1-9B-0414 Source: arxiv.org
• prio 8 LLM-based eligibility review for securities prospectuses at the German Central Bank Concepts: LLM Evals Entities: German Central Bank arXiv Source: arxiv.org
• prio 8 Multilingual reasoning cascades benefit from preserving the original question Concepts: Context Engineering Source: arxiv.org
• prio 8 RL-induced tool use localized to a single crosscoder feature Concepts: Tool Use Agents Entities: Qwen2.5-3B Source: arxiv.org
• prio 8 A training method for calibrating confidence in medical VQA models Concepts: LLM Evals Entities: MedGemma 4B IT Qwen2 VL 7B Instruct Source: arxiv.org
• prio 8 EGG: An Expert-Guided Agent Framework for Kernel Generation Concepts: Agents Context Engineering LLM Evals Source: arxiv.org
• prio 8 EpiKV Proposes Attention-Free KV Cache Eviction for Long Reasoning Traces Concepts: Long Context Context Engineering Entities: arXiv FlashAttention 2 sources: arxiv.org, arxiv.org
• prio 8 Helpfulness training can erode mid-trained compassion values in Llama 3.1 8B Concepts: LLM Evals Entities: Llama-3.1-8B Dolly-15k Magicoder-110K RLHFlow Source: arxiv.org
• prio 8 Reasoning quality can be filtered from early token loss Concepts: LLM Evals Entities: Qwen2.5-7B Llama3.1-8b Source: arxiv.org
• prio 8 What Multimodal LLM Evaluation Is Still Missing Concepts: LLM Evals Source: arxiv.org
• prio 8 SkillDisCo distills reusable procedural skills from successful agent traces Concepts: Agents Source: arxiv.org
• prio 8 Staying VIGILant: Counterfactual Visual Alignment for MLLMs Concepts: LLM Evals Source: arxiv.org
• prio 8 GAVEL: benchmark for verifying and localizing image-text caption errors Concepts: LLM Evals Source: arxiv.org
• prio 8 ProfileFoundry releases 100,000 synthetic person objects for LLM agent evaluation Concepts: Agent Memory Agents Tool Use LLM Evals Source: arxiv.org
• prio 8 Meta paper proposes Autodata for agentic synthetic data generation Concepts: Agents LLM Evals Entities: Meta 2 sources: twitter.com, alphaxiv.org
• prio 7 Reproducibility study finds AlphaEdit works within its original scope, but its guarantees do not hold uniformly at larger scale Concepts: LLM Evals Entities: Llama3 GPT2-XL GPT-J Source: arxiv.org
• prio 7 HarmVideoBench benchmarks harmful video understanding in LVLMs Concepts: Context Engineering Source: arxiv.org
• prio 7 From Clicks to Intent: cross-platform session embeddings and LLM-distilled intent labels for financial recommendations Concepts: Embeddings LLM Evals Source: arxiv.org
• prio 7 TOPS proposes first-principles visual token pruning for efficient MLLM inference Entities: LLaVA-NeXT TOPS Source: arxiv.org
• prio 7 Multilingual open-weight pipeline for building signed political knowledge graphs from news Concepts: LLM Evals Entities: Wikidata Civic Platform Platforma Obywatelska Law and Justice Source: arxiv.org
• prio 7 GEOALIGN: Geometric Rollout Curation for Robust LLM Reinforcement Learning Source: arxiv.org
• prio 7 Qwen3-Instruct SAEs map millions of interpretable features Concepts: LLM Evals Entities: Qwen3-Instruct SAE Qwen3-1.7B Qwen3-4B Qwen3-8B Source: arxiv.org
• prio 7 ArXiv study finds recursive reasoners lose global reasoning under aggressive compression Source: arxiv.org
• prio 7 SharQ combines activation sparsity with FP4 quantization for LLM inference Entities: Llama-3.1-8B Qwen2.5-7B Qwen3-30B-A3B Qwen3-VL-8B Source: arxiv.org
• prio 7 SAE-Guided Activation Regularization for LLM Continual Learning Concepts: LLM Evals Source: arxiv.org
• prio 7 Survey unifies diffusion-based adversarial attacks and defenses across text, vision, and vision-language models Concepts: LLM Evals Source: arxiv.org
• prio 7 Narration-of-Thought: a system prompt scaffold for ethical reasoning Concepts: Context Engineering LLM Evals Source: arxiv.org
• prio 7 Study finds reasoning models and humans diverge in how they spend deliberation on wrong answers Concepts: LLM Evals Source: arxiv.org
• prio 7 DiARC uses positive and negative samples to improve ARC-like reasoning in LLMs Concepts: LLM Evals Source: arxiv.org
• prio 7 AgentX proposes a production multi-agent loop for recommender-system iteration Concepts: Agents Tool Use Source: arxiv.org
• prio 7 Detecting and controlling sycophancy with cascading linear features Concepts: LLM Evals Entities: arXiv Source: arxiv.org
• prio 7 NuclearQAv2 adds a benchmark for nuclear engineering QA Concepts: LLM Evals Source: arxiv.org
• prio 7 Benchmark Study Tests LLMs on Qualitative Humanitarian Coding Concepts: LLM Evals Open Source LLMs Entities: arXiv alphaXiv CatalyzeX DagsHub 2 sources: arxiv.org, arxiv.org
• prio 7 Refusal behavior in chat models depends on persona steering Entities: Qwen2.5-7B-Instruct Llama 3.1 8B Instruct Source: arxiv.org
• prio 7 TempoWave uses multi-wavelet digit embeddings for LLM time-series forecasting Concepts: Embeddings Entities: arXiv Source: arxiv.org
• prio 7 Generative Retrieval with Diffusion Transformer and Preference Optimization Concepts: RAG Embeddings RAG Evaluation Source: arxiv.org
• prio 7 Where CoT training gains land in LLM-based agents Concepts: Agents LLM Evals Source: arxiv.org
• prio 7 OpenFinGym: A Verifiable Multi-Task Gym for Evaluating Quant Agents Concepts: LLM Evals Agents Entities: OpenFinGym Source: arxiv.org
• prio 7 Pipeline for generating longitudinal synthetic clinical notes with LLMs Entities: arXiv alphaXiv Connected Papers Litmaps Source: arxiv.org
• prio 7 Historical Italian shows a tokenization cost but a larger comprehension cost for LLMs Concepts: LLM Evals Context Engineering Source: arxiv.org
• prio 7 Anthropic updates its Economic Index to track Claude Code, Cowork, and hourly usage patterns Entities: Anthropic Source: anthropic.com
• prio 6 Self-Supervised Patent Embeddings Use Section-Based Positives Concepts: Embeddings LLM Evals Source: arxiv.org
• prio 6 Cross-provider recommendation behavior differs, but failure diagnoses converge Concepts: LLM Evals Entities: ChatGPT Claude Anthropic OpenAI Source: arxiv.org
• prio 6 A structural study of how LLMs suggest research methods Concepts: LLM Evals Entities: arXiv OpenAI Google DeepSeek Source: arxiv.org
• prio 6 FisherSketch estimates transfer-relevant head update similarity at vocabulary scale Concepts: LLM Evals Entities: Llama-3.1-8B Source: arxiv.org
• prio 6 GAversary uses a genetic algorithm and GloVe-based mutations to generate adversarial text against NLP classifiers Concepts: Embeddings Source: arxiv.org
• prio 6 Process harness for adding agentic layers to deterministic workflows Concepts: Agents Entities: arXiv Source: arxiv.org
• prio 6 Empowering GUI Agents with Autonomous Experience Exploration and Hindsight Training Concepts: Agents Entities: Qwen2.5-VL-32B Source: arxiv.org
• prio 6 CascadeFormer and CascadeFlow Pruning target layer inefficiency in deep Transformers Entities: arXiv Source: arxiv.org
• prio 6 CARVE proposes key-axis erasure for chunk-parallel linear attention Concepts: Chunking LLM Evals Entities: CARVE GDN-2 Source: arxiv.org
• prio 6 Intent-aware training improves LLM safety classification Concepts: LLM Evals 2 sources: arxiv.org, arxiv.org
• prio 6 PersistentKV studies page-aware decode scheduling for long-context LLM serving on commodity GPUs Concepts: Long Context Source: arxiv.org
• prio 6 NVC-style prompt constraints reduce escalation in LLM dialogue Source: arxiv.org
• prio 6 Psychology-Grounded Reasoning and Policy Optimization for Role-Playing Agents Concepts: Agents LLM Evals Source: arxiv.org
• prio 6 Emotion Vectors in Open-Weight LLMs Show Different Layer Patterns Concepts: Open Source LLMs Entities: Anthropic Claude Sonnet 4.5 Apertus-8B-Instruct-2509 Gemma-4-E4B-it Source: arxiv.org
• prio 6 Auditing framing-sensitive instability in LLMs for mental-health chats Concepts: LLM Evals Source: arxiv.org
• prio 6 Cascaded pruning for on-device LLM inference in industrial IoT Concepts: LLM Evals Entities: arXiv NVIDIA Source: arxiv.org
• prio 6 KIRP adds external knowledge and reflective CoT to zero-shot tweet stance detection Concepts: RAG Source: arxiv.org
• prio 6 OPID: On-Policy Skill Distillation for Agentic Reinforcement Learning Concepts: Agents Entities: arXiv Source: arxiv.org
• prio 6 LLM-based models for emerging topic detection in service feedback Concepts: LLM Evals Source: arxiv.org
• prio 6 Learning Planning Budgets in Real-Time RL Concepts: Agents Source: arxiv.org
• prio 6 Diagnosing Task Insensitivity in Language Agents Concepts: Agents Source: arxiv.org
• prio 6 A mechanism-oriented taxonomy for detecting coded language with LLMs Concepts: LLM Evals Entities: arXiv TikTok Bluesky alphaXiv Source: arxiv.org
• prio 6 Structured-Data Pipeline Turns Hindi WordNet into 1.25M Instruction Pairs Source: arxiv.org
• prio 6 SOLAR aligns soft-token representations for cross-lingual reasoning Concepts: LLM Evals Source: arxiv.org
• prio 6 LoRA fine-tuning closes part of the TTS quality gap for Khmer, not Korean Concepts: LLM Evals Entities: VoxCPM2 MiniCPM-4 Source: arxiv.org
• prio 6 Erase-then-Delta Attention decouples erase and write in linear attention memory updates Concepts: Long Context Source: arxiv.org
• prio 6 auto-psych: Agent-driven theory discovery and experimentation for cognitive science Concepts: Agents Tool Use Source: arxiv.org
• prio 6 Provenance-aware multi-agent system for psychiatric medication information seeking Concepts: Agents RAG Entities: Reddit WebMD U.S. FDA Adverse Event Reporting System Neo4j Source: arxiv.org
• prio 6 Prompt Injection in LLM-Based Resume Screening Entities: arXiv Source: arxiv.org
• prio 6 HRM adapters for long-context fine-tuning Concepts: Long Context LLM Evals Entities: Mistral-7B Source: arxiv.org
• prio 6 Sparse autoencoders are used to probe transformer robustness under OOD prompts Source: arxiv.org
• prio 6 SocialPersona benchmarks personalized profiling from multimodal social timelines Entities: arXiv Source: arxiv.org
• prio 6 CAT-Q proposes post-training ternary quantization for LLMs Entities: BitNet 1.58-bit v1 BitNet 1.58-bit v2 Source: arxiv.org
• prio 6 Where Larger Models Excel: Constraint-Guided Reasoning in Benchmarks Concepts: LLM Evals Entities: Qwen3-32B Qwen3-8B GPT-OSS 120B gpt-oss-20b Source: arxiv.org
• prio 6 AlgoEvolve uses LLM-driven evolutionary search to improve trading strategies Concepts: LLM Evals Entities: arXiv Source: arxiv.org
• prio 6 Radical AI Interpretability proposes a framework for reading models as agents Concepts: Agents Source: arxiv.org
• prio 6 EvoOptiGraph proposes weakness-driven coevolution for optimization modeling Source: arxiv.org
• prio 6 Practical 4.6-bit Networks: Why HardTanh Helps Source: habr.com
• prio 6 Google retrofits Multi-Token Prediction onto frozen Gemini Nano models for faster on-device inference Entities: Google Gemini Nano v3 Gemma 4 Source: research.google
• prio 6 METR says GPT-5.6 evals exposed cheating behavior Concepts: LLM Evals Entities: DAIR.AI METR OpenAI GPT-5.6 Sol Source: vxtwitter.com

🛠 Tools & Frameworks (5)

• prio 9 Weave Router adds per-request model routing for Claude Code, Codex, and Cursor Concepts: Tool Use Entities: Weave Anthropic OpenAI Gemini Source: github.com
• prio 8 NVIDIA open-sources NeMo AutoModel for faster MoE fine-tuning Entities: NVIDIA Hugging Face 量子位 QbitAI Source: qbitai.com
• prio 6 NVIDIA NeMo Speech ranks #7 in Generative AI with new speech releases and a repo pivot Entities: NVIDIA Hugging Face PyTorch CUDA Source: github.com
• prio 6 HATCHA: a reverse CAPTCHA for gating agent access Concepts: Agents Entities: monday.com Source: github.com
• prio 6 slisp: a Lisp compiler for Linux/AMD64 that emits standalone assembly Source: github.com

💬 Opinions (9)

• prio 10 What AI Can and Cannot Represent Well in Vectors Concepts: Embeddings Source: t.me
• prio 9 How a multi-agent Telegram channel for 24/7 situational awareness is structured Concepts: Agents Tool Use Context Engineering Entities: Telegram RSS GPT DeepSeek Source: habr.com
• prio 8 How LLMs Are Measured: Parameters, Benchmarks, and Ad Hoc Tests Concepts: LLM Evals Entities: LLaMA-2 DeepSeek Source: habr.com
• prio 8 Open vs closed LLM gap looks very different depending on the benchmark Concepts: LLM Evals Entities: Artificial Analysis Source: blog.doubleword.ai
• prio 7 AI agents have made rewrites cheap, but not requirements discovery Concepts: Agents Code Agents Entities: Habr Source: habr.com
• prio 7 Sebastian Raschka compares local open-weight models across coding harnesses Concepts: Code Agents Open Source LLMs Entities: GPT 5.5 Gemma 4 E2B Source: twitter.com
• prio 6 Why current LLM pricing may not hold Concepts: Open Source LLMs Entities: Uber Microsoft Salesforce GitHub Source: aditya.patadia.org
• prio 6 Self-Driving Labs as a Move From Automation to Autonomy Concepts: Agents Tool Use Source: thesequence.substack.com
• prio 6 Anecdotal model comparison on frontend and backend tasks Concepts: Code Agents LLM Evals Tool Use Entities: DeepSeek Fable GLM Claude Sonnet Source: t.me

FAQ

What is in the 2026-06-26 AI brief?

The 2026-06-26 brief selected 129 signal items for AI builders and filtered 244 items as noise, using the radar’s community-relevance scoring.