Reliability gains in agentic coding are coming from execution-grounded verification and harness/tooling design rather than from the model weights alone, because errors compound silently through iterative and multi-stage pipelines.
Evidence
- Only execution-based verification reliably improves code correctness; cheap uncertainty signals and self-correction do not, and iterative feature-addition measurably degrades security with each pass
- Hallucinations become undetectable as they propagate through multi-agent stages, with ~75% of errors catchable only at early handoffs
- Harness co-evolution (HELIX) and persistent indexing (Graft, 42% token reduction) show orchestration and context management drive capability alongside model updates, while LSP semantic retrieval can cost tokens rather than save them
- Tooling like ProofRun cryptographically verifies test claims to separate real execution from confident hallucination
Implications
- Teams should place verification at early pipeline handoffs and gate on actual execution, treating agent ‘all tests pass’ claims as unverified by default
- Competitive advantage in coding agents increasingly lives in the harness—indexing, compaction, tool routing, verification—so buyers evaluating agents should weigh tooling and grounding, not just the underlying model
Concepts
Code Agents Agents Context Engineering LLM Evals Tool Use
Confidence
medium