Agents still fall short at predicting wet-lab rankings: SPU tops out at 0.478 on public tasks while twelve of thirteen configurations beat chance on unpublished ones
The UniPat AI team released Science Prediction Universe (SPU), which turns real wet-lab experiments with withheld results in chemistry, materials science, biology and medicine into ranking-prediction tasks: 100 public tasks drawn from 85 recent papers plus 16 private tasks from unpublished experiments. Agents work in an isolated container with free web access, software installation and simulation, and only have to order candidate conditions by the measured outcome. Across 13 configurations, task-adjusted means on the public tasks run from 0.227 to 0.478, and only 34% to 52% of scored tasks exceed the chance threshold; on the 16 private tasks, 12 of the 13 configurations rank significantly better than random.