UniPat AI's SPU benchmark tests agents on real wet‑lab predictions; none reach Above 0.5
agents benchmarks
| Source: Mastodon | Original article
UniPat AI's new Science Prediction Universe benchmark tasks AI agents with forecasting real wet‑lab experiment outcomes, and none achieved a score above 0.5.
UniPat AI has unveiled the Science Prediction Universe (SPU), a new benchmark that asks AI agents to forecast the outcomes of real wet‑lab experiments before any data are collected. The test set comprises 116 tasks – 100 drawn from recent Nature Portfolio papers and 16 drawn from unpublished experiments – covering chemistry, materials science, biology and medicine. Across 4,207 individual experiments, 13 model configurations were evaluated; the top performer, Claude Opus 5.5, achieved an average Spearman correlation of 0.486, barely above the 0.5 chance line and succeeding on only about half of the tasks.
The release matters because it pushes the frontier of “predict‑first” science, where AI could guide experimental design, cut reagent waste and accelerate discovery. Until now most AI benchmarks have relied on synthetic or fully disclosed datasets; SPU forces models to reason from the same limited information a human researcher has, then rank candidate conditions without ever seeing the actual measurements. The modest scores highlight a gap between current foundation models and the nuanced, domain‑specific knowledge required for reliable laboratory prediction.
Going forward, UniPat AI invites the community to submit additional tasks, suggesting the benchmark will evolve into a shared platform for stress‑testing emerging models. Watch for follow‑up evaluations as newer agents – especially those fine‑tuned on experimental protocols – are pitted against SPU. Parallel efforts such as the Gemini Decisions API benchmark have shown that predictive performance can improve rapidly with targeted training, so the next few months could see a noticeable leap in AI‑driven experimental planning.
Sources
Back to AIPULSEN