Skip to main content
Back to timeline
arXivSource publication:

Agents rewrite their own harness mid-task: BoxBox reaches 1.702 best correct speedup on PerfRace

Synopsis

On long-running performance-optimization tasks, can an agent rewrite its own harness, the code that fixes its available actions and observations, instead of reusing an offline-optimized one? BoxBox lets the agent request a review while a separate kernel hot-reloads the revised harness and preserves execution context; on PerfRace, 100 tasks built from real software projects, it reaches a best-so-far correct speedup of 1.702 within a four-hour budget, above Frozen Harness at 1.442, offline Harness Evolution at 1.281 and externally controlled Harness Scaling at 1.539.

Source-provided article image: BOXBOX: Recursive Harness Self-Improvement for Long-Running Performance Optimization
arXiv

Interpretation

An agent can revise its own harness during an ongoing task instead of running under a harness fixed after offline optimization. Offline Harness Evolution revises the harness from training-task feedback and then holds it fixed for test tasks; on PerfRace it reaches a best-so-far correct speedup of 1.281, below the 1.442 of Frozen Harness, which never changes the harness. Across 50 held-out test tasks with three independent runs per task under a shared four-hour budget, BoxBox reaches 1.702, above Frozen Harness at 1.442, offline Harness Evolution at 1.281 and externally controlled Harness Scaling at 1.539.

The task-solving agent decides when to revise, and a kernel performs the switch without losing task context. Earlier online methods let an external controller schedule harness revisions between attempts, so a revision takes effect only after the current execution segment ends. When the agent calls boxbox(), the kernel suspends the task, preserves the trajectory, terminal session and workspace, and launches a review session that edits the harness; a validated revision is loaded with context transferred, otherwise the previous runtime is retained or restored. On 14 randomly sampled test tasks, the variant without agent triggering, which schedules reviews every 30 minutes, reaches 1.508 at a mean model cost of $21.58, below full BoxBox at 1.584 and $15.07.

Perspective

The demonstrated scope is 50 held-out PerfRace test tasks: real GitHub software projects paired with fixed workloads, where the agent makes correctness-preserving performance improvements within a shared four-hour budget, with all methods starting from the Terminus-KIRA harness and using GPT-5.6 Luna at high reasoning effort. Evaluation runs in isolated research environments, harness updates pass kernel validation and rollback, and task scoring uses separate shadow-container verification.

Offline Harness Evolution shows no consistent improvement during training or validation, and the selected harness scores below Frozen Harness on test tasks; the authors attribute this to difficulty identifying effective revisions and limited cross-task generalization. Letting an agent modify executable harness code adds security and responsible-AI risk; kernel validation and rollback plus shadow verification reduce experimental risk but do not form a complete security boundary, and the authors position BoxBox as a research prototype rather than a production-ready self-modifying agent framework.