Confidence Shift Makes 35B LLM Top Over Typed-Decision Models in 12,000 Real RFQs Test
| Source: Mastodon | Original article
A 35‑billion‑parameter language model was benchmarked against typed‑decision models on 12,000 real RFQs, with confidence scores ultimately determining the top performer.
A new benchmark pits a 35‑billion‑parameter mixture‑of‑experts language model against two typed‑decision systems on a real‑world workload of 12,000 request‑for‑quotation (RFQ) classifications. The test, released on GitHub by the independent “decision‑model‑benchmark” project, measures primary‑class accuracy, latency, cost and, crucially, calibrated confidence for each approach.
The three contenders are a typed‑decision API, the 35B LLM and a 421‑million‑parameter open‑weight decision model. While the LLM initially posted the highest raw accuracy, the inclusion of confidence scores flipped the result: the decision‑model stack delivered a higher effective performance once confidence‑based routing was applied. The benchmark also confirms the expected cost advantage of the decision models, which operate without generating text and can be “hundreds of times cheaper” than an LLM judge, as highlighted in TypeSafe AI’s Jev documentation.
Why it matters is twofold. First, it provides the first large‑scale, reproducible comparison of “System One” decision models with traditional “System Two” LLM judges on a production‑grade classification task. Second, the finding that calibrated confidence can overturn raw‑accuracy rankings underscores a shift in how AI pipelines may be architected—favoring models that can reliably quantify uncertainty over sheer size.
The test builds on our earlier coverage of TypeSafe AI’s Jev, the company’s first System One model, and adds concrete data to the debate over whether decision models can replace LLM judges. Going forward, watch for broader adoption of confidence‑driven routing in AI services, further benchmarks that expand beyond RFQs, and integration efforts by platforms such as LangChain that aim to standardise evaluation of both model classes. The outcome could reshape cost structures and reliability expectations for enterprise AI deployments.
Sources
Back to AIPULSEN