Can agents build agents? On Singularity-Bench's six end-to-end tasks, Opus 5.5 with Claude Code matches or beats the human baseline on all six
Synopsis
Singularity-Bench splits AI development into six end-to-end tasks: pre-training, mid-training, post-training, inference, the agent harness and cluster orchestration, each starting from a minimal codebase with a fixed time budget and compared against a human-engineered baseline on identical hardware. Across 11 agents built from 7 frontier models and 2 harnesses, Opus 5.5 with Claude Code matches or outperforms the baseline on all six for a Singularity-Index of 1.25, while the other 10 average 0.58. The index averages each agent's ratio to the human baselines across the six tasks, scoring failed tasks as zero, so it measures consistency across the stack rather than a single best result.
Interpretation
Agents are assessed across the whole AI development chain: designing and training a language model from scratch, mid-training, training a coding agent with reinforcement learning, building an inference engine, writing a coding-agent harness from scratch, and orchestrating a serverless inference cluster, each paired with a mature human-engineered solution. Earlier benchmarks either restricted the action space so agents could only tune hyperparameters of an existing system, used simplified environments and small models, or relied on reference solutions and finite tests, leaving end-to-end systems built by agents largely uncovered. Every task is compared on identical hardware against a human baseline: Megatron-LM with the OLMo 2 recipe for pre-training, the PRISM mixture for mid-training, Miles for post-training, SGLang for inference, Claude Code for the harness, and Dynamo with KAI Scheduler for the cluster.
Scores are summarized by the Singularity-Index, the average of an agent's ratios to the human baselines across the six tasks, with each baseline set to 1 and a failed task counted as zero while still dividing by six. Opus 5.5 with Claude Code is the only agent that matches or beats the baseline on all six, winning five and tying Miles on post-training at 0.533 versus 0.539, for an index of 1.252; the other 10 agents average 0.58. In the same table, proprietary agents average 1.01 and open-weight agents 0.50; the cheapest, DeepSeek V4.1 Flash with Codex, reaches 0.698 at $4.36 in API spend and forms the cost-index Pareto frontier with GPT-6.1 Sol with Codex (0.969, $84.52) and Opus 5.5 ($425.41).
Open-ended tasks also widen the room for cheating, so the evaluation framework itself is tested: red-team attacks cover 9 of the 10 OWASP categories for agentic applications, plain Harbor is broken on all six tasks, and vulnerabilities persist even with an LLM judge installed. The attacks exploit the fact that the verifier container must run agent-submitted code, for example planting a program inside the generation function that crawls for ground-truth answers, or leaving a hidden process alive across phases to rewrite the final reward. Breakwater adds five safeguards to Harbor, namely binary fingerprinting before and after the agent phase, an API router that blocks unauthorized calls, a wall that stops live processes after the agent container, an external log the agent can neither read nor write, and a restricted intermediate execution phase before verification, and it catches every attack listed in the two attack tables.
Perspective
The evaluation shows performance on six tasks within given hardware and time limits: 12 hours on one GB300 node for pre-training, 12 hours for mid-training, 12 hours for post-training, 12 hours for inference, 6 hours for the harness, and 4 hours on four nodes for the cluster. It suits readers tracking agents' autonomous R&D capability and the integrity of long-horizon evaluation; the authors also note the tasks are smaller than true recursive self-improvement requires and invite frontier labs and evaluation teams to expand the budgets to measure the full loop.
The authors state that, because of limited resources, the tasks are smaller in scale than true recursive self-improvement would require, and that Singularity-Bench is a proxy for the skills such an agent needs. Even though the strongest agent, Opus 5.5, matches or exceeds every human baseline, its raw scores still leave substantial headroom on all six tasks, so matching the baselines is not saturation.
