WebArena-Pro adds audio and video to web-agent testing: 20 self-hosted sites, 300 tasks, and a 62.3% ceiling for the best setup
Synopsis
WebArena-Pro is a reproducible web-agent benchmark of 300 human-validated tasks across 20 self-hosted web applications in six domain categories, treating audio and video as core observations alongside text and vision. Prior self-hosted benchmarks covered few application categories and limited modality to text and vision, while live benchmarks broaden site coverage but drift between runs. Codex with GPT 6.1 Sol and Claude Code with Claude Opus 5.5 tie for the best success rate at 62.3%, and cross-site tasks generally score below single-site tasks.
Interpretation
Reproducible web-agent evaluation now spans 20 self-hosted applications across productivity, communication, analytics, business operations, design and media, and travel, with applications differing in interface conventions, authentication flows and backend state schemas. Earlier self-hosted web benchmarks covered only a small number of applications from a few software categories, limiting the heterogeneity of interfaces, workflows and data models; live benchmarks broaden site coverage but sacrifice reproducibility because pages, content, layouts and policies drift between runs. The comparison table lists 6 environments for WebArena, 3 for VisualWebArena, 4 for WebChoreArena, 4 for TheAgentCompany and 9 for OSWorld, against 20 for WebArena-Pro, which is also marked as covering vision, audio and video with task isolation.
Audio and video become core observations in web tasks rather than incidental material: media sources are exposed through the accessibility tree, and agents can call transcription and frame-extraction actions that fold temporal content into the existing observation structure. Earlier self-hosted benchmarks restricted modality to text and vision and did not meaningfully evaluate whether agents can understand and use audio and video embedded in web tasks. Of the 300 tasks, 221 rely only on text and vision and 79 require information in temporal media on a page; in the media-tool ablation, removing media tools drops Gemini 3.1 Pro and Qwen 3.6 27B to 0% on audio tasks within the 43-task subset that cannot be solved from titles, metadata or thumbnails alone.
Each task runs in isolation against per-task seeded container state, making parallel evaluation reproducible and preventing backend changes an agent makes in one task from contaminating later evaluations. Earlier self-hosted benchmarks retain state across tasks, so backend changes made while solving one task persist into the next; live benchmarks are hard to repeat because pages and data drift between evaluations. Each application runs as a Docker container with seeded data and fixed login credentials, and container state is reset to its seeded baseline before each task, with local hosting fixing the application version and database contents across runs.
Perspective
Evaluation runs in self-hosted container environments with a 50-browser-action limit per task, and success is decided by a rubric-based verifier; the action-space extension covers only prerecorded audio segments and discrete video frames, leaving voice dialogue and live video streams to future work.
Comparisons evaluate particular combinations of models, harnesses and tool configurations, and performance differences cannot generally be attributed to model capability or harness design alone; a shared 50-step budget also does not equalize computational resources, since agents differ in token consumption, native tool use, observation frequency and latency, and reported costs depend on provider pricing and cache utilization.
