Skip to main content
Back to timeline
arXivSource publication:

Real repositories become training grounds, not just validation endpoints: RepoFactory builds 10,159 environments and lifts a 35B model by 22.6 points on NL2Repo

Synopsis

RepoFactory automatically turns real code repositories into verifiable training environments in which an agent rebuilds a complete project from scratch, and an environment evolution loop improves both usability and training value without model rollouts. It yields 10,159 environments across 17 software domains; training Qwen3.5-35B-A3B on them produces RepoHorizen-35B-A3B, which gains 22.6 percentage points on NL2Repo-Bench and 8.5 points on average across five benchmarks, with environment evolution itself contributing 5.1 and 2.1 points.

AI-generated editorial illustration: From Repositories to Training Environments: Scaling Long-Horizon Coding Agents with Environment Evolution Loop

Interpretation

Training environments for coding agents can be generated in bulk from real repositories instead of relying on a small hand-picked task set; RepoFactory converts projects from public code-hosting platforms into verifiable tasks where an agent rebuilds a complete repository from a language specification. Earlier linear construction pipelines ended at validation, confirming usability but giving little guidance on improving training value; unsuitable repositories were discarded, so scaling depended on finding more repositories that met the same fixed requirements. The pipeline produces 10,159 environments across 17 software domains; the largest category accounts for only 12.5% of environments, and only 6.9% fall outside the named domains.

The environment evolution loop uses low-cost signals from verification and static code analysis to jointly revise the oracle implementation, task specification and tests, making initially unsuitable candidate repositories usable. Routes that use model rollouts or downstream training as feedback incur extensive implementation work per rollout on long-horizon tasks, raising cost and delaying feedback. On 500 repositories uniformly sampled from the filtered candidate set, allowing up to three repair steps raises the number of validated environments from 169 to 347, lifting yield from 33.8% to 69.4%, with most of the improvement coming from the first repair.

Continuing evolution toward implementation scope and structural coupling raises the training value of reconstruction tasks at a fixed number of trajectories. A valid environment may still require little implementation work, or may consist of independent components. Starting from the same 1,000 environments and optimizing one metric at a time, environments collected after rounds 3, 6, 9 and 12 form four levels of 2,000 trajectories each for fine-tuning Qwen3.5-35B-A3B: optimizing implementation scope raises NL2Repo accuracy from 30.2% to 32.9%, and optimizing structural coupling raises it from 30.2% to 32.5% without relying on longer interaction trajectories.

RepoHorizen-35B-A3B, trained on these environments, also improves on software engineering and terminal tasks beyond repository generation. The base Qwen3.5-35B-A3B scores 23.8 on NL2Repo and 45.1 on average across the five benchmarks. NL2Repo rises from 23.8 to 46.4, SWE-bench Verified from 65.8 to 72.2, SWE-bench Pro from 42.4 to 47.6, SWE-bench Multilingual from 59.7 to 61.0, and Terminal-Bench 2.1 from 33.7 to 40.5, with the average rising from 45.1 to 53.5; without environment evolution, NL2Repo is 41.3 and the average is 51.5.

Perspective

The demonstrated scope is Python projects: repositories must provide package metadata and a nonempty local test suite, contain at least five implementation files and fewer than 40,000 lines of Python code, and projects requiring external services, outbound network access, graphical interfaces or specialized hardware are excluded. Evaluation uses Claude Code 2.1.140 as the agent scaffold, with a maximum context length of 262,144 tokens and maximum output length of 32,768 tokens, and context is automatically compacted at 80% usage.

The proxy metrics characterize task content at a coarse structural level and do not yet explicitly distinguish the capabilities required by individual tasks; language coverage remains mainly Python. The environment-scaling curve is still rising at the largest scale tested, which the source describes as no evidence of saturation, but no results beyond that scale are given.