Skip to main content
Search

# RECLAIM benchmark

1 related articles

Subscribe to updates
  1. arXiv

    RECLAIM benchmark: four agents reproducing results on 100 NeurIPS 2025 papers succeed on only 41% of Run-tier, 27% of Retrain-tier, and 15% of Reimplement-tier papers at best

    The authors introduce RECLAIM, a benchmark of 100 NeurIPS 2025 papers that can be rebuilt yearly from new conferences, fixing in advance for each paper the result to reproduce, what counts as a successful reproduction, and a GPU-hour budget, and dividing difficulty by what authors released (Run tier has code, data, and weights; Retrain tier lacks weights so the agent trains the model; Reimplement tier lacks code so the agent writes it), with a separate language model grading runs from logs and outputs rather than agents' reports; four agents run once per paper, and the best agent in each tier reproduces only 41%, 27%, and 15% of papers respectively, failed attempts use on average 29% of their budget, and the most common error is writing the method without checking any part against the pape

You’re all caught up