WebArena-Pro adds audio and video to web-agent testing: 20 self-hosted sites, 300 tasks, and a 62.3% ceiling for the best setup
WebArena-Pro is a reproducible web-agent benchmark of 300 human-validated tasks across 20 self-hosted web applications in six domain categories, treating audio and video as core observations alongside text and vision. Prior self-hosted benchmarks covered few application categories and limited modality to text and vision, while live benchmarks broaden site coverage but drift between runs. Codex with GPT 6.1 Sol and Claude Code with Claude Opus 5.5 tie for the best success rate at 62.3%, and cross-site tasks generally score below single-site tasks.