The real-world success of coding and task agents is driven more by their execution harnesses, tool interfaces, and objective evaluation frameworks than by raw underlying model capabilities.
Evidence
- Empirical studies isolating the ‘harness effect’ demonstrate that vendor-native SDKs offer no automatic solve-rate advantage over third-party frameworks using identical models.
- Research on tool interfaces reveals that general shell access can significantly outperform structured, strongly typed tool calling in both accuracy and token cost.
- Multiple studies (e.g., GAUGE) highlight critical flaws in LLM-as-a-judge evaluations, including capability-dependent biases and a disconnect from objective, deterministic task completion metrics.
- Tools exposing session context costs reveal that the stateless API nature of agents causes quadratic token cost growth that remains hidden in standard capability benchmarks.
Implications
- Teams must build customized, objective evaluation oracles and deterministic verification loops rather than trusting model defaults or LLM judges.
- Agent infrastructure should prioritize flexible tool interfaces (like sandboxed bash) and explicit budget allocation over rigid programmatic schemas and unbounded reflective retries.
Concepts
LLM Evals Code Agents Tool Use Agents
Confidence
high