Skip to main content
Back to timeline
arXivSource publication:

Agents still fall short at predicting wet-lab rankings: SPU tops out at 0.478 on public tasks while twelve of thirteen configurations beat chance on unpublished ones

Synopsis

The UniPat AI team released Science Prediction Universe (SPU), which turns real wet-lab experiments with withheld results in chemistry, materials science, biology and medicine into ranking-prediction tasks: 100 public tasks drawn from 85 recent papers plus 16 private tasks from unpublished experiments. Agents work in an isolated container with free web access, software installation and simulation, and only have to order candidate conditions by the measured outcome. Across 13 configurations, task-adjusted means on the public tasks run from 0.227 to 0.478, and only 34% to 52% of scored tasks exceed the chance threshold; on the 16 private tasks, 12 of the 13 configurations rank significantly better than random.

AI-generated editorial illustration: Science Prediction Universe

Interpretation

SPU turns "predict how the experiment will turn out" into an automatically scored ranking task: given an experimental setup and candidate conditions, an agent orders the candidates by a measurement it never sees, and the measured values and grading code stay in a separate container the agent cannot reach. Earlier scientific-agent evaluations either had agents analyze data already in hand, graded with deterministic tests, or asked for outcome predictions as single question-answer calls without tools, so none tested choosing which evidence and which calculation bear on a measured endpoint and committing to a prediction without experimental feedback. The 100 public tasks come from 85 papers published between January 2025 and August 2026 in 11 Nature Portfolio journals, covering 42 chemistry, 31 materials-science, 31 biology and 12 medicine tasks; 16 private tasks were designed by experts from unpublished wet-lab experiments, giving 116 tasks, 4,207 experiments and 517 comparison groups in total.

Inside a container with 8 CPU cores, 16 GiB of memory, one NVIDIA H20 GPU and an eight-hour budget, an agent searches the web, installs software, runs quantum-chemistry or molecular-dynamics calculations, and submits a complete ordering for each comparison group. Grading reads only the submitted rankings.json and scores each group by the Spearman rank correlation between the submitted order and the measured values, with group weights fixed when the task is built, so reports, scripts and self-descriptions cannot move the score. The 13 configurations span 12 frontier models across two harnesses, Codex and Claude Code, each run once on every task, for 1,300 official runs on the public tasks, of which 1,155 are scored; the analysis covers 227,103 tool calls in those 1,155 trajectories.

Perspective

The evaluation covers 116 tasks across chemistry, materials science, biology and medicine; agents run in an isolated container with network access and software installation, and scoring uses only the rank correlation between the submitted ordering and the hidden measured values. Public-task answers can in principle be found by search, so 16 unpublished private tasks serve as a check that depends on no detector.

On the public tasks the configurations separate into only two tiers, and the ordering within each tier remains unresolved at one run per task; Claude Opus 5.5 and Claude Fable 5.1 are scored on only 44 and 30 public tasks because of safety refusals, so their means reflect the subsets they completed. On the private tasks no pair of configurations differs detectably, and accuracy depends strongly on the physical endpoint: on the same electrolytes all thirteen configurations score between 0.89 and 0.95 for ionic conductivity but range from −0.55 to 0.07 for the electrochemical window.