Benchmarks no longer need hand-built pipelines: Benchmark Agent auto-generates 15 benchmarks with roughly 96–98% human acceptance
Synopsis
MMLab at The Chinese University of Hong Kong and collaborators released Benchmark Agent, which turns benchmark construction from a human design, collection and annotation pipeline into an agent-run process. A Benchmark Planner decomposes a user requirement into subtasks grounded in real datasets, and a Benchmark Executor realizes samples with tools. Across 15 benchmarks spanning text, multimodal and domain reasoning, five representative ones reach human acceptance rates of about 96–98% and LLM-as-Judge user-intention alignment of 68.54–81.48, so the generated benchmarks largely preserve user intent while grounding and difficulty control remain weaker.
Interpretation
Benchmark construction moves from a human pipeline to agent execution: a user states an evaluation requirement and the system produces runnable benchmark items. Existing benchmarks and their building pipelines are largely human-driven, requiring substantial effort in task design, data collection and cleaning, manual annotation and preference alignment, and each new benchmark is often built almost from scratch, slowing iteration. Across 15 benchmarks covering text, audio-text, image-text and audio-image-text settings, five representative benchmarks reach human acceptance rates of about 96–98% and LLM-as-Judge user-intention alignment scores between 68.54 and 81.48.
The system splits the long pipeline into planning and execution, where planning turns a vague requirement into grounded subtasks and data allocation, and execution turns the plan into concrete items. Planning uses three collaborating agents: Design proposes and revises subtasks, Grounding validates that each subtask can be supported by real datasets plus transformation plans, and Allocation solves a feasible allocation under quota constraints. Execution orchestrates sample by sample, calling synthesis tools such as text-to-speech, image resizing, audio mixing, noise injection and web search plus script tools for file conversion and metadata editing, then verifies content and format and replenishes samples to meet quotas.
Automated construction costs far less time than human annotation. Human annotation takes about 6 minutes per sample on the audio-based reasoning benchmark and about 5 minutes per sample on the art reasoning benchmark. Benchmark Agent takes about 0.3 minutes and 0.2 minutes per sample on the corresponding two benchmarks.
Perspective
The demonstrated scope is 15 benchmarks across four scenario types, with human acceptance rates and LLM-as-Judge scores reported for five representative ones, and consistency checks covering Qwen3.5 models from 2B to 27B on three benchmarks; construction defaults to GPT-5.1 as the backbone model and General-Bench as the dataset pool.
Generated samples score lower and vary more on context-question correspondence, target-signal dependency and skill-specific challenge, and grounding fidelity and difficulty control are named as directions for improvement; the system is also bounded by available tool coverage, and lightweight human review is still considered useful.
