Skip to main content
Back to timeline
arXivSource publication:

Long-horizon agents stop compressing away the details: CarryOn splits progress from evidence and averages 75.7% task success across three benchmarks

Synopsis

When long-horizon AI agents run multi-step tasks, longer histories make context harder to manage: summaries drop exact details later actions need, while keeping every interaction carries substantial irrelevant information. CarryOn separates a compact execution state from fine-grained, source-linked evidence, and a learned selector plus an adaptive retention policy decide which original observation fragments the agent sees at each step, without extra LLM calls for context management. Across AppWorld, OfficeBench and Tau2 Telecom with three Actor models, it reaches 75.7% macro-averaged task success, 6.7 percentage points above the strongest baseline, SWE-Pruner, and the lowest mean normalized cost per success at 0.792.

Source-provided article image: CarryOn: Learning What to Keep in Context for Long-Horizon LLM Agents
arXiv

Interpretation

CarryOn lets a long-horizon AI agent keep executing multi-step tasks without letting context grow with the interaction history, by splitting "where the task stands" and "which original details this step needs" into two inputs. Earlier context management summarized, pruned or selected whole interactions; summaries can omit exact values later actions need, while retaining entire interactions carries substantial irrelevant records. Averaged equally over AppWorld, OfficeBench and Tau2 Telecom and over three Actors, Claude Sonnet 5, Qwen3.8-27B and Gemma-4-31B, CarryOn reaches 75.7% task success, 13.3 percentage points above Full History and 6.7 points above the strongest baseline, SWE-Pruner, while its mean normalized cost per success, NCS, is 0.792, the lowest among the methods.

At each step the agent receives a compressed control state plus evidence fragments cut from original observations, and each fragment keeps a source reference, so records omitted earlier can return to context when needed. Evidence selection is done by a learned selector that scores historical units both for relevance ranking and for the predicted probability of use in the next action; the latter can trigger temporary evidence expansion that adds fragments beyond the base budget. The selector uses Qwen3-Reranker-0.6B as its backbone and is LoRA-tuned on official training splits with offline state annotations and evidence references generated by Claude Sonnet 5; one frozen selector trained per benchmark is reused across all three Actors, and on AppWorld Qwen's success rises from SWE-Pruner's 68.5% to 84.5%.

A substantial share of selected evidence comes from interactions at least ten steps earlier, showing that long-range dependencies are retained rather than compressed away. Across 909 trajectories and 14,419 Actor calls, evidence at least ten steps old accounts for 30.7% of selections on AppWorld, 31.2% on OfficeBench and 48.8% on Tau2 Telecom, and 40.2% of task runs select evidence again after it was omitted. An ablation on AppWorld with Claude shows success falling from 79.8% to 66.1% when the control state is removed, and 76.2% when learned selection is replaced by packing the most recent units, indicating that compact state and learned evidence selection each contribute.

Perspective

Evaluation covers three text-interaction benchmarks, AppWorld, OfficeBench and Tau2 Telecom, with 168, 95 and 40 tasks respectively, and success is determined by each benchmark's official evaluator; one selector is trained per benchmark and reused across the three Actors.

Extending to visual or multimodal observations would require adapting how evidence is represented and selected; generalization to new environments with different tool interfaces and observation structures remains to be explored; retained observations may contain sensitive information, and missing or misinterpreted evidence may lead to incorrect actions, so deployment requires access controls for stored history and user confirmation for consequential actions.