Skip to main content
Back to timeline
arXivSource publication:

Agents must learn unfamiliar software by using it, then rebuild it: the best system fully reproduces under a quarter of SWE-Recreation's requirements

Synopsis

SWE-Recreation turns "understand by using, then rebuild" into a measurable task: an agent sees only rendered pixels and acts through pointer and keyboard, with no DOM, source, logs or back-end. The benchmark spans 54 real applications (28 websites and 26 Android apps) and 1,446 hidden requirements; across five frontier systems, the best reaches 0.530 weighted fidelity and at most 22.9% fully reproduced requirements under application-equal averaging.

Source-provided article image: SWE-Recreation: Benchmarking Coding Agents for Software Recreation
arXiv

Interpretation

For the first time, software agents are asked to work out what an unfamiliar application does without being told the goal, using it only as a person would, and then deliver a runnable rebuild. Earlier web and computer-use agents were handed an explicit goal, image-to-code work started from a screenshot, prototype or requirements document, and interactive-reconstruction benchmarks supplied a recorded interaction video or annotated state transitions, so someone else had already chosen what to show. Evaluation runs on 54 real applications and 1,446 hidden requirements, each specifying ordered interaction steps and expected observable outcomes, with an exclusion list removing multi-user, payment, real-time and generative behavior that no single machine can faithfully reproduce.

Grading does not compare pixel appearance; one fixed judge model operates the replica through the same human-like interface, walks each hidden workflow and checks the resulting state changes. Layout, wording, styling and framework are out of scope, so behaviorally equivalent implementations pass while static mock-ups do not, and persistence requirements need before, changed and post-reload or restart evidence for full credit. The balanced study covers 16 common websites and all 26 Android apps, 42 targets, 1,111 requirements, 210 runs and 5,555 grades; four GUI-domain experts reviewed 20 sampled judgments and deemed more than 95% correct.

Beyond fidelity, the evaluation records separately whether a behavior was explored and whether it was implemented once reached, locating whether failure lies in discovery, understanding or implementation. Aggregate fidelity alone cannot separate a capability the agent never found from one it found but failed to build, so exploration coverage is estimated post hoc by a separate judge over exploration keyframes without affecting the agent. Under application-equal averaging across the five systems, Kimi leads at 0.530 weighted fidelity and 22.9% full credit, followed by DeepSeek at 0.472, GLM at 0.362, GPT-6-Sol at 0.352 and Opus at 0.277.

Perspective

The evaluation targets black-box reconstruction of 54 real applications and scores only behavior observable on screen; multi-user collaboration, payments, real-time data, generative features and external integrations are excluded from scoring, and the balanced study actually covers 16 websites and 26 Android apps.

Exploration coverage is a post-hoc estimate over sampled keyframes, so negative judgments do not prove the agent never explored and positive judgments may cover only part of a workflow; each system runs once per application, live targets lack deterministic resets and may have appeared in pretraining, so these descriptive results do not establish causality.