Discovery agents stop asking only true or false: swapping a hypothesis for rival explanations resolves 67% more questions at the same budget
Synopsis
Scientific discovery agents usually test one hypothesis at a time, so when evidence rejects it the agent learns that one explanation is unlikely but not which alternative the evidence supports. Contrastive Discovery replaces each discovery step with a set of mutually exclusive, collectively exhaustive competing explanations for the same question, and scores an experiment by how far its evidence redistributes belief across that set. Across six datasets spanning materials, astronomy, mortgage discrimination, online discourse, software engineering practice and social behavior, guiding search with contrast sets resolved 67% more questions than pointwise surprisal under the same experiment budget, and 38 blinded experts chose the same explanation on 78% of resolved questions.
Interpretation
The unit by which a discovery agent measures progress changes from a single hypothesis to a set of competing explanations for one question, so rejected evidence can point to a specific alternative instead of only declaring the original claim unlikely. The pointwise approach compares a hypothesis only with its negation; once evidence rejects it, the agent cannot tell which alternative holds and must spend further experiment budget proposing and testing alternatives one by one. Across six datasets, 89% of the pointwise candidates with the largest surprisal are rejections; re-evaluated on the same evidence through contrast sets, 73% of those candidates resolve to a specific alternative without running new experiments.
Placing contrast sets inside the search loop lets the agent resolve more scientific questions under the same experiment budget, and nearly all of the gain comes from corrective resolutions in which evidence shifts support from the prior-leading explanation to another. Pointwise baselines ranked by surprisal, novelty or feasibility resolve similar numbers of questions, indicating that the limit lies in the representational unit rather than the scoring function. At a budget of 200 experimental validations, contrast sets resolve 161.9 questions on average against 97.2 for pointwise surprisal, 67% more; restricted to confirmative resolutions, contrast sets resolve 82.2 against 97.2.
The gain comes mainly from evaluating evidence against the full contrast set rather than from a new ranking rule or from scoring alternatives jointly. An ablation separates ranking, evaluation and when the contrast set is built: variants with pointwise evaluation stay at or below a 0.22 resolution rate, while contrastive evaluation raises it to between 0.77 and 0.96. Keeping contrast sets but scoring each alternative with the pointwise scorer resolves 157.8 questions against 161.9 for joint scoring, and the two select the same leading alternative on 89.3% of sets; at equal tokens and equal dollars contrast sets still lead, 148.3 against 87.7 and 145.8 against 92.2.
Perspective
Experiments are limited to tabular observational datasets across six domains: materials, astronomy, mortgage lending, online discourse, software engineering and social behavior, with the AutoDiscovery scaffold and a gpt-4o backbone; on such data a resolution is a statistical rather than a causal finding.
Resolved-question counts depend on the LLM elicitor, since resolution is read from elicited beliefs; contrast sets can be malformed or omit a plausible explanation, and adding a none-of-the-above abstention option cut false resolutions from 53.4% to 0.6%; the authors leave transfer to other discovery agents, laboratory experiments and longer-horizon interventions to future work.
