Skip to main content
Search

# Agent evaluation

27 related articles

Subscribe to updates
  1. NVIDIA releases SkillEvaluator: static quality checks for agent skills, then a with-and-without skill comparison

    NVIDIA has released SkillEvaluator, a tool that checks the quality of AI agent skills and measures their effect on task results. It runs quality checks first, then compares runs with and without a skill; a skill is a folder of instructions and supporting files defined by the Agent Skills specification. The bundled text-analyzer example scores 88.8/100 (B) on the static quality check against a default passing threshold of 70, while publication evidence remains marked INCOMPLETE.

  2. arXiv

    WebArena-Pro adds audio and video to web-agent testing: 20 self-hosted sites, 300 tasks, and a 62.3% ceiling for the best setup

    WebArena-Pro is a reproducible web-agent benchmark of 300 human-validated tasks across 20 self-hosted web applications in six domain categories, treating audio and video as core observations alongside text and vision. Prior self-hosted benchmarks covered few application categories and limited modality to text and vision, while live benchmarks broaden site coverage but drift between runs. Codex with GPT 6.1 Sol and Claude Code with Claude Opus 5.5 tie for the best success rate at 62.3%, and cross-site tasks generally score below single-site tasks.

  3. NVIDIA open-sources SkillEvaluator: agent skills pass a static quality gate first, then a with-and-without skill run comparison

    NVIDIA released SkillEvaluator, an open-source command-line tool that checks AI agent skills and measures their effect on task results. A skill is a folder of instructions and supporting files defined by the Agent Skills specification. The tool runs a static quality check first, then compares runs with and without a skill; the example skill text-analyzer scores 86.2/100 (B), exits 0, and the default passing threshold is 70. That score covers static quality only, and publication evidence stays marked incomplete.

  4. arXiv

    CyberArena puts AI agents in live attack-and-defense combat, and two frontier models pull ahead of the field

    CyberArena turns cybersecurity evaluation from static targets into multi-round combat: two agents in isolated sandboxes attack each other's services while patching their own, with objectives rotating every round. Across 40 challenges and 466 games, GPT-5.6 Sol and Claude Opus 5 led 12 frontier agents, and on 30 challenges all three compared models solved solo, round-three capture rates fell to 52%, 37% and 35%.

  5. arXiv

    Finance agents meet an open world: none of 150 sessions passes, and all 35 the simulated user approved fail a program check

    FinCosmos opens all three uncertainties of financial agent evaluation at once: a simulated investor reveals a hidden strategy only when asked, the agent must source its own Web data and backtest under a data cutoff, and programs verify the delivery instead of matching a reference answer. Across 150 sessions with four coding agents and a tool-free model, none is accepted, and all 35 sessions the simulated user signed off fail at least one program veto.

  6. arXiv

    Paper2Bench turns published papers into auditable exams: the best agents match or beat every baseline on 80% of tasks

    Spatial transcriptomics alignment lacked an auditable way to evaluate AI agents, so Paper2Bench turned two published alignment benchmark studies into 121 executable tasks, had agents solve them in containers with ground truth withheld, and re-ran 16 published baselines under one scorer. Across the 121 tasks, the best agents (GPT-6-Astra and Gemini-4-Argon) matched or outperformed all baselines on 80% of tasks; distilling agent runs into a standalone program beat 87% of baselines on held-out tasks.

  7. arXiv

    A causal answer to whether an AI agent should be reviewed mid-task: randomized interventions inside one trajectory show effects swing by action and state

    Agent evaluation has relied on fixed benchmarks that report task success or failure but cannot say whether reviewing or steering an agent mid-trajectory is worth it at deployment. Researchers bring micro-randomized trials into agent trajectories, assigning review or direct execution with equal probability at each eligible decision, then estimate causal effects with augmented inverse probability weighting and structural nested mean models. On τ-bench Retail, ToolSandbox and ImpossibleBench, review effects vary substantially across tool-call types and agent states, and the same data yield decision-group risk rankings.

  8. aclanthology.org

    Space mission planning gets a shared exam: five agent systems on 35 held-out cases, with geometry and constellation design still lagging

    AstroAgentBench turns space mission planning into an executable, externally checked agent evaluation: 116 cases across seven task families, with the main evaluation using five held-out cases per family, 35 in total, where an agent must submit a schedule, observation plan or constellation design that an independent verifier checks for schema, timing, geometry, resources and mission value. Among five agent systems, Codex CLI + GPT-5.4 and Kimi CLI + Kimi K2.6 produced verifier-valid plans on all 35 cases, and the strongest systems approach or exceed solver-reference scores on several families while still trailing clearly on geometry-heavy, product-level and design-heavy tasks such as Regional Coverage and Stereo Imaging.

  9. arXiv

    Delivered memory is not used memory: MemUse traces each delivery and finds most entries irrelevant and most relevant ones ignored

    MemUse moves the unit of evaluation from the task outcome to the individual memory delivery: across 8,000 SWE-bench Verified agent episodes, three memory systems stayed within 0.4 percentage points of a no-memory baseline on resolve rate, yet per-delivery judgments show most delivered entries were irrelevant, 51.3%–79.7% of relevant entries went unused in each system, and 23.2% of episodes with beneficial memory use still failed.

  10. arXiv

    ML agents skip the training run: a 4B world model predicts program outcomes on 77.9% of held-out programs

    AI research agents must train and evaluate a model for every candidate program, so execution cost caps how many candidates they can compare per step. MLE-World, finetuned from Qwen3.5-4B, reads a program's code and data description and predicts its terminal output, exception type and validation score, plus a confidence score. On held-out programs from 17 MLE-bench Lite tasks its predicted terminal output matches the real run on 77.9% of programs versus 41.1% for the strongest prompted baseline, and search guided by it reaches up to 22.1% higher observed mean performance than the agent without a world model while using less real execution.