Agentic Code Judge Delivers Verifiable Feedback
agents
| Source: HF Papers | Original article
Researchers introduce an agentic code‑judging framework that provides verifiable feedback to evaluate and refine LLM‑generated solutions during test‑time scaling in competitive programming.
A new verification agent called **Arbiter** is pushing the limits of test‑time scaling for large language models (LLMs) that generate code. By pairing Arbiter with the GPT‑OSS‑120B model, researchers achieved a score of 385.46 out of 600 on a competitive‑programming benchmark—well above the 361.12 gold‑level cutoff. Crucially, the system reaches that performance with fewer than 100 generation attempts per subtask, whereas earlier search‑driven approaches required around 5 000 candidates.
The breakthrough lies in giving the LLM not just a binary pass/fail verdict but a verifier that can estimate a program’s correctness and produce concrete evidence of failure. Arbiter judges a candidate solution using only the problem statement and source code, then generates counter‑examples that serve as checkable proof of any flaw. This “verifiable feedback” turns the judging process into an agentic task, allowing the model to diagnose errors, suggest revisions, and iteratively improve its output without exhausting official submission limits.
Why it matters is twofold. First, it makes test‑time scaling far more efficient, cutting computational cost while preserving—or even enhancing—accuracy. Second, it opens a path toward more honest evaluation of AI‑generated code, moving beyond simple pass/fail metrics toward evidence‑based diagnostics that can generalise across contest settings.
The development follows the recent trend of extending agentic AI beyond code generation, as highlighted in our October 14 coverage of Amp’s OpenAI‑powered agents. Going forward, the community will watch for broader adoption of Arbiter‑style verification in coding assistants, further refinements of the benchmark and reward design, and whether similar agentic verification can be applied to other domains that demand rigorous, evidence‑backed feedback.
Sources
Back to AIPULSEN