StateM Hits Reaching 95.3% Raw Accuracy and $15 Frontier Run on Terminal-Bench 2.1 via Harness Scaling
agents
| Source: HF Papers | Original article
StateM achieved 95.3% raw accuracy on Terminal‑Bench 2.1 with a $15 frontier run, leveraging harness scaling to boost long‑horizon agent execution reliability.
StateM, a new execution‑system framework for long‑horizon AI agents, has hit a raw accuracy of 95.3 % on the Terminal‑Bench 2.1 suite while keeping the cost of a “frontier” run to roughly $15, according to an arXiv pre‑print released this week. The result, presented by Ziheng Qin and three co‑authors, demonstrates that scaling the harness – the surrounding orchestration that feeds a model its step‑by‑step instructions and tracks mutable state – can dramatically close the gap between a model’s theoretical capability and its actual performance on complex, multi‑step tasks.
The paper highlights a persistent problem: agents often stumble on extended sequences even when the underlying language model can solve each individual sub‑task. Typical failure modes include losing track of changing variables, neglecting lessons learned earlier in the run, skipping required procedures, or terminating prematurely. By augmenting the execution layer rather than the model itself, StateM shows that modest engineering investments can yield frontier‑level reliability without the expense of larger models.
The breakthrough matters for any application that relies on autonomous agents to carry out lengthy procedures – from automated research pipelines to industrial process control. A $15 per‑run cost places high‑accuracy, long‑horizon execution within reach of smaller labs and startups, potentially accelerating the adoption of frontier AI in production settings while mitigating safety concerns tied to unpredictable agent behavior.
The work follows our recent coverage of agent evaluation tools such as AutoResearch and HarnessEval‑W, which exposed the same brittleness in real‑world tasks. Going forward, the community will watch for StateM’s performance on broader benchmark suites, its integration with emerging large‑scale models like GPT‑5.6 and Kimi K3, and whether the harness‑scaling approach can be generalized to visual or embodied domains. If the early results hold, harness scaling could become a standard layer for turning raw model power into dependable, cost‑effective AI agents.
Sources
Back to AIPULSEN