Agent harnesses can keep improving after a model swap: Self-Improvement Memory scores higher in 9 of 10 paired coding continuations
Synopsis
Automatic optimization of agent harnesses has usually been confined to one policy model and one benchmark, so when the model or data distribution changes the optimizer inherits an evolved harness but not the reasons behind its edits or how they performed. Self-Improvement Memory (SIM) groups each harness edit by the recurring harness problem it addresses and hands the proposer and selector the rationale, predicted and observed outcomes, and evaluation context, appending new records during target optimization. In ten paired continuations from independent source runs on coding, moving from Terminal-Bench 2.1 to Terminal-Bench 4, the SIM continuation finished higher in 9 of 10 pairs, averaging 2.36 points above harness-only continuation.
Interpretation
Harness optimization can now continue after the benchmark or policy model changes instead of restarting from a seed harness. Recursive self-improvement has generally been conducted within a fixed policy model and benchmark; evolved harnesses transfer, but the optimization process itself does not carry across settings. Across four domains, in cross-dataset and Opus-to-Gemini cross-model comparisons starting from the same evolved harness with matched rounds and candidate budgets, Full SIM finished highest in all six cross-dataset settings and all four cross-model domains.
On continuation, the optimizer receives not only the harness code but a record of improvements organized by harness problem. With harness-only transfer, why an edit was proposed, which tasks it was expected to fix, and what happened when tested do not travel with the harness. Each entry stores the modification, its rationale, predictions written before evaluation, observed outcomes afterward, and provenance such as domain, benchmark, policy model and round; proposer and selector read the full record each round, and every selected candidate is recorded whether it succeeds, regresses or misses its predicted tasks.
On coding, the advantage from this record repeats across independent source runs. Single-run comparisons cannot separate the effect of memory from run-to-run variation in the optimization process. Source optimization from Terminus-2 on Terminal-Bench 2.1 was repeated ten times, and each evolved harness initialized a pair of ten-round Terminal-Bench 4 continuations; from a shared mean starting score of 18.36, the mean final score was 21.70 with SIM and 19.33 without, a mean paired advantage of 2.36 points, with 9 of 10 pairs favoring SIM.
Perspective
Experiments cover four domains, coding, text classification, agentic workspace and engineering design, in which the source-evolved harness remains executable in the target environment; cross-dataset continuation fixes the policy model, cross-model continuation fixes the benchmark and split, and policy weights stay frozen throughout. Outside coding, each condition is a single run and results are reported descriptively.
The authors note that within their compute budget independent repetitions focus on coding, with one run per condition in the other domains; the study is limited to settings with compatible harness interfaces and a fixed optimizer planning and search procedure; and the comparisons evaluate SIM as a complete procedure combining inherited memory with target-stage updates rather than as separated components.
