Flight test of Commitment Machine across Gemini, Claude, and GPT-OSS after 926 steps, two lobotomies and zero nudges
agents claude gemini
| Source: Dev.to | Original article
A 2026 Harvard study of 718 firms found AI agents write 30% more code but close no extra tickets as review times climb 49%, leading to a live test of a 1986 Commitment Machine on Gemini, Claude and GPT‑OSS.
A Harvard study released this year surveyed 718 firms and found that deploying AI‑driven coding agents lifts raw code output by roughly 30 percent, yet does not translate into faster issue resolution – ticket‑closure rates stayed flat while code‑review cycles lengthened by 49 percent. The paradox sparked a hands‑on experiment by a veteran software‑engineer who revived a 1986 “Commitment Machine” and ran it in live production on three contemporary AI platforms: Google’s Gemini, Anthropic’s Claude and the open‑source GPT‑OSS stack.
The author documented the trial on the DEV Community, describing a 926‑step workflow, two “lobotomies” (intentional capability reductions) and a deliberately nudgeless configuration. By stripping the agents of typical prompting tricks, the test aimed to isolate the raw productivity boost versus the downstream friction of longer reviews. While the post does not disclose quantitative outcomes, the methodology underscores a growing concern: higher code churn may burden human reviewers, eroding the expected efficiency gains of AI assistance.
The experiment matters because many organisations are betting on AI coders to accelerate delivery pipelines. If review latency expands in step with output, the net business value could be neutral or even negative, especially for teams already stretched thin. The Harvard data already hints at this trade‑off; the Commitment Machine trial provides a concrete, reproducible framework for measuring it across divergent model families.
Going forward, observers will watch for follow‑up analyses that compare the three platforms on metrics such as review turnaround, defect rates and developer satisfaction. Industry players may also adjust tooling – for example, integrating automated triage or tighter gating – to counteract the review slowdown. As AI coding agents become more entrenched, real‑world experiments like this will shape how firms balance raw productivity with the hidden costs of longer quality‑control cycles.
Sources
Back to AIPULSEN