Same Qwen3.8-27B, better with a harness adapter: SWE-bench Pro on pi rises from 39.2% to 64.7% with fewer tool calls
Synopsis
Hugging Face trained one LoRA adapter per agent harness for Qwen3.8-27B, freezing the base weights and using reinforcement learning inside pi, Hermes Agent and OpenCode to optimize performance for that harness and task distribution. Each specialist adapter beats the base model under its own harness on both coding benchmarks, for example lifting pi on the 51 hardest SWE-bench Pro tasks from 39.2% to 64.7% while cutting tool calls by 56%–72% on tasks both solve. The three adapters also concatenate into a single super-adapter that beats the base model under all three harnesses, though it keeps less of each specialist's gain.
Interpretation
When a model is paired with an agent harness, the shell that actually issues tool calls and assembles context, a general-purpose model is not optimized for that harness and task, so performance and efficiency are left unused; reinforcement learning inside the harness recovers them. Earlier multi-harness RL and harness-adapter experiments covered data-analysis and code-building tasks; this work pushes the same idea to a 27B model and harder software-engineering tasks across three harnesses, pi, OpenCode and Hermes Agent. All three specialist adapters beat the base model under their own harness on both benchmarks: on the 51 hardest SWE-bench Pro tasks pi goes from 39.2% to 64.7% (15 newly solved vs 2 lost, p = 0.002), hermes from 29.4% to 43.1%, and opencode from 35.3% to 51.0%; on Terminal-Bench 2.1 pi gains 16.3 points (16 vs 2, p = 0.001) and hermes 14.1 points (19 vs 7, p = 0.03).
The harness runs exactly as it ships during training, and the trainer learns from the real tokens recorded by a proxy between the harness and vLLM rather than from a re-implemented agent loop. Most RL frameworks rewrite the agent loop so the agent runs against the framework's own interface; here the deploy-time harness itself stays in the loop and supplies the training signal. The final images ship Pi 1.5.2, OpenCode 2.2.7 and Hermes Agent v0.22.6, training uses TRL's AsyncGRPOTrainer, and each adapter trains on two B200 GPUs with one serving vLLM and one training; training ran 598 cumulative steps over 78,114 samples, or 167M tokens.
The three adapters concatenate along the rank dimension into a rank-96 super-adapter that applies each update at weight ½ and runs unchanged under all three harnesses. Pairwise cosine similarity between adapters is only about 0.01–0.02, so their updates are nearly orthogonal and barely interfere when merged; weights of ⅓, ½ and 1 were compared on all 89 Terminal-Bench 2.1 tasks, and ½ gave the best balance. With thinking off, the super-adapter beats the base model on all six harness–benchmark pairs (mean pass@1 39.5% to 49.0%; 71 newly solved vs 34 lost, p < 0.001); with thinking on it wins five of six pairs and ties the sixth (mean 55.5% to 63.5%; 67 newly solved vs 37 lost, p = 0.004).
Perspective
The results rest on Qwen3.8-27B with the pi, Hermes Agent and OpenCode harnesses, on all 89 Terminal-Bench 2.1 tasks and the 51 hardest SWE-bench Pro tasks, evaluated with one attempt per task at temperature 0.8 and thinking off. The authors state that default harness settings differ (for example, pi does not keep previous reasoning turns), so results cannot be compared across harnesses, only within the same harness.
The authors describe the model emitting a short stretch of reasoning-like text before tool calls in non-thinking mode as "reasoning leakage" and list it as an open question, with follow-up work planned on pure thinking models and hybrid thinking/non-thinking training. In the out-of-distribution check, the five models on 300 AutomationBench tasks differ mostly within noise, with only the Hermes adapter a few points behind. The super-adapter's ½ weight was chosen on Terminal-Bench 2.1 using step-120 adapters, so numbers on that benchmark are slightly optimistic.
