Skip to main content
Back to timeline
arXivSource publication:

Coding agents' harnesses get their own exam: Doc2Harness puts GPT-6 Astra on top at 53.05%, yet even the strongest model fails nearly half

Synopsis

A coding agent's performance depends not only on the model but on the harness around it, the software layer that manages tool calls, context, permissions and result checking. Doc2Harness turns harness engineering itself into a scorable task: the model under test, working inside a fixed builder harness, writes a complete harness from a development document and an interface-only skeleton, and a scripted mock model then drives it case by case, judged through model requests, files on disk, process exit codes and command output. Across 19 frontier models with three builds each on 328 test cases, GPT-6 Astra has the highest pass rate at 53.05% and the weakest, Hy3, reaches 27.44%.

Source-provided article image: Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agents
arXiv

Interpretation

How much of a coding agent's capability reaches a real workspace depends on the harness around the model, and that harness is now the object being evaluated rather than a fixed precondition. Earlier benchmarks measured downstream task success under a fixed harness, leaving harness engineering itself unexplored. Nineteen frontier models each made three independent builds over 328 test cases; GPT-6 Astra leads at a 53.05% pass rate and the weakest, Hy3, reaches 27.44%, a gap of roughly 26 percentage points, so the suite is neither saturated nor unable to separate models.

The model under test receives only a development document and a skeleton that keeps interfaces alone; it must write the rest of the implementation. The development document is rewritten after merging and disambiguating the user documentation of twelve frontier harnesses, and the skeleton is distilled from mini-swe-agent with all internal implementation removed, both to reduce reliance on parametric knowledge. A representative candidate harness runs to about 30 Python source files and roughly 16,000 lines, of which about 99.6% is written by the build model beyond the skeleton; builds run in an empty container without external network access, and a deliverable is frozen only after a contract check confirms it compiles and installs, has all required commands, leaves no placeholder stubs and keeps the three contract interfaces.

Scoring does not run a real model on downstream tasks; a mock model answering from a preset script drives the candidate harness end to end. A real LLM's instability would interfere with scoring, while scripted responses make each case's verdict repeatable. Each test case runs once in its own container under a 900-second limit and is recorded as FAIL on timeout; 130 of the 328 cases are gated by at least one positive control, and when a control fails its group is recorded as FAIL regardless, guarding against a feature that never ran being counted as a pass.

Perspective

The evaluation targets the coding-agent harness as a software form: the model under test builds a candidate harness inside a fixed OpenCode builder harness, in an isolated container, from a development document consolidated across twelve frontier harnesses and an interface-only skeleton, and a scripted mock model then drives it end to end in a separate container per case. Readers should treat it as a measurement of harness-building ability, not a ranking of downstream task capability.

The protocol measures a single build, allows up to three submissions and is not an iterative build-evaluate-repair loop; the contract check tests form rather than behavior, so an early submitter's pass rate is a lower bound rather than an upper bound. In addition, single-variable expansion derives several variants from one seed, so a single defect can fail several correlated cases at once.