Skip to main content
Search

École Normale Supérieure

1 related articles

Subscribe to updates
  1. arXiv

    Rethinking Agentic Evaluation: Treating Agents as Configurable Systems, About 54% of Outcome Variance Comes from Repeating the Same Configuration

    The work introduces a new benchmark of four scientific tasks in which a coding agent must find and correctly operate a published specialist model, and systematically studies five parts of the agent's configuration—task information, reasoning, self-verification, time budget, and backbone model; it finds that approximately 54% of outcome variance comes from repeating the same configuration, that the information provided to the agent has the largest effect exceeding both time budget and model size while also reducing cost and improving calibration, and that prompting an agent to verify its answer has little effect on its verification behavior whereas providing a dedicated verification tool changes that behavior substantially.

You’re all caught up