Agents must buy their own experiments to find scaling laws: SLDBench v2 drops median R² from 0.72–0.75 to 0.24–0.57
Synopsis
SLDBench v2 turns scaling-law discovery from a setting where all outcomes are supplied into one where an agent picks experiments, pays each one's recorded cost, and revises its law as results arrive. On 30 tasks where a reference power-law fit scores R²≤0.8, median R² across 8 frontier agent systems falls from 0.72–0.75 in the passive setting to 0.24–0.57 in the active one, and prediction error rises 1.5–4.7×; agents favor cheap experiments, spending 8–44% of the full-pool cost on average, but cheap purchases leave conditions needed for extrapolation unobserved.
Interpretation
Scaling-law discovery is split into two separately reported quantities: prediction quality and the cost of the evidence behind it. An agent receives a priced catalogue of candidate experiments, decides which to buy and when to stop, and submits one law whose functional form is shared across groups. The earlier SLDBench handed the agent every experimental outcome upfront and gave cost no role; SLDBench v2 keeps the same hidden extrapolation targets and makes observations something the agent must pay for. Across the 30 tasks, the passive medians of the 8 frontier agent systems lie between 0.723 and 0.753, while the active medians spread from 0.235 to 0.570, and the passive ranking only weakly predicts the active one (Spearman rs=0.21, n=8).
The quality loss comes from what is bought, not from the sequential interface. Active runs that buy every candidate see the same data as their passive counterparts, and the median difference is near zero. Among 67 such full-pool pairs, active is better by more than 0.01 in R² in 38.8%, passive in 29.9%, and the remaining 31.3% differ by at most 0.01. Paired by system, task and repeat, each system's active/passive error multiplier lies between 1.49 and 4.71, with every 95% interval above one; pooled over systems, the share of runs with R²≤0 or no score rises from 11.4% to 31.7%.
Agents buy cheap experiments first and expensive ones later, and a spending distribution describes a system better than a single budget. For every system the median share of candidates bought exceeds the median share of cost, and the median ratio of mean price paid to the catalogue mean ranges from 0.18 to 0.90; each system's first batch sits in the cheaper half of each group's price range and its last batch sits higher. Claude Opus 5.5 reaches the highest active median R², 0.570, at a median cost of 4.01% of the pool; Hy4-Preview and Qwen3.8 Flash spend medians of 3.44% and 5.35% yet raise their failure shares most, by 34.4 and 26.7 percentage points.
Under a fixed law, spanning each group's price range matters more than reaching every group, and that span is exactly what costs the most. At budgets of ρ≤5%, random and price-spread selection beat cheapest-first on 63% to 77% of tasks, while cheapest-per-group beats it on at most 46.7% of tasks at any budget. At ρ=2% the price-spread rule raises median R² from −1.456 for cheapest-per-group to −0.005, and the two rules visit groups in the same order, differing only in whether they sample each group's price range.
Perspective
The evaluation covers 30 tasks from 14 sources and 8 frontier agent systems, each solving every task three times in each setting; tasks are kept only if a reference power law scores R²≤0.8 in both settings, so they are not a representative sample of the corpus. Experiments run through historical replay, so an agent can buy only experiments that were already run and cannot propose new configurations.
Prices mix source-reported FLOPs, estimates derived under documented assumptions such as 6ND, and non-FLOP proxies, so the cost fraction compares spending within a task but not absolute compute across tasks. The same system on the same task often reaches qualitatively different results across three repeats, with the sign of R² changing on 20.0% to 46.7% of tasks, so differences between adjacent systems should not be over-read. Cost and quality are reported on two axes rather than combined into one ranking.
