Skip to main content
Back to timeline
arXivSource publication:

Agents stop waiting for external scores: Verify by Doing co-evolves execution and verification inside a single task

Synopsis

Agents that improve themselves at test time usually need benchmark scores or reference answers to tell whether a task is done, and real tasks often lack that external feedback. Verify by Doing gives the executing agent and the verifying agent separate editable harnesses of instructions, skills and checklists, gathers completion evidence by actively inspecting artifacts and environment state, and uses those findings to update both execution and verification. Across AutomationBench, Banking and 133 GDPval tasks it posts the highest overall scores among compared methods without external feedback, and on a fixed candidate pool the evolved evaluator raises pass rate from 31.2% to 35.8%.

Source-provided article image: Verify by Doing: Co-Evolving Agent Execution and Evaluation without External Feedback
arXiv

Interpretation

An agent can improve both how it does a task and how it judges whether the task is finished within a single task, without benchmark scores or reference answers saying which attempt succeeded. Earlier test-time self-improvement either updated execution procedures while verification stayed fixed or spent extra inference generating more candidates, and both depend on reliable completion feedback; tool-based critique adds observations and retrospective diagnosis updates execution, but neither changes the verification procedure itself. On AutomationBench, Banking and 133 GDPval tasks the method has the highest overall scores among compared methods without external feedback, with 82.4% overall rubric quality on GDPval versus 80.1% for RQGM.

Execution and verification are coupled through two editable harnesses, the instruction, working-rubric and skill configurations that drive execution and checking, while model weights stay frozen. The evaluator no longer returns only a verdict: it independently inspects artifacts and environment state and records satisfied, violated and unresolved requirements, and those findings both form preferences among candidates and motivate edits to either harness. AutomationBench pass rate rises from 30.0% to 34.3% and then 37.0% across three rounds, and the second-round mean already exceeds RQGM's final 33.7% at lower cumulative token cost.

Changes to the verification procedure must pass a comparative review before adoption, and output selection is decided separately from adopting an execution procedure. When verification is incomplete, checks can favor outputs that miss requirements, so letting the evaluator rewrite itself would also change the basis for judging whether execution improved. On six fixed candidates per task, the evolved evaluator reaches 35.8% pass rate and 69.8% partial credit versus 31.2% and 67.4% for the initial evaluator; on accepted execution updates it rejects 89.3% of observed regressions versus 71.2% for RQGM, at similar precision (77.6% versus 77.0%).

Perspective

The demonstrated scope is three-round adaptation within individual tasks: 600 AutomationBench tasks, 97 Banking tasks and 133 GDPval tasks, on two frozen-weight backbones, DeepSeek-V4-Flash-0731 and Qwen3.8-Flash-Next, with task requirements, tool permissions and acceptance rules held fixed and harnesses and history reset between tasks.

The actor, evaluator and reviewer may share errors in interpreting evidence, so active verification and comparative review cannot eliminate incorrect selections or updates; one case selects a failing output after treating a file timestamp as a business date. The method also needs states or artifacts that can be isolated and inspected, and irreversible actions, environments that cannot be reset, delayed effects or inaccessible state may escape verification. Transfer across tasks and sustained improvement over longer evolution remain untested, and extra execution, verification and harness revision incur inference and tool-use costs.