Long-horizon agent training stops comparing inside one task: BLPO pools experience across tasks, lifting WebShop success to 74.02%
Synopsis
In long-horizon tasks, the contribution of each agent step is hard to separate from a single trajectory, and restricting comparisons to the few rollouts of one task often leaves too little evidence. BLPO groups decisions that reach the same task progress and perform the same action function across tasks, estimates step-level advantages from statistics accumulated across tasks and training updates, and weights them for context difficulty and reliability. On ALFWorld and WebShop, the Qwen2.5-1.5B policy raises WebShop task success from HGPO's 71.54% to 74.02% and ALFWorld Out-Success from 90.16% to 91.54%, without a learned critic or extra rollouts.
Interpretation
Step-level credit for long-horizon tasks can draw on experience from other tasks instead of comparing only within the sampled trajectories of the current task. GRPO assigned one trajectory advantage to all actions, GiGPO grouped actions by matching observations, and HGPO further conditioned groups on interaction history, but these comparisons stayed inside the limited rollouts of a single task. With Qwen2.5-1.5B, BLPO reaches 74.02% WebShop task success versus HGPO's 71.54%, and 91.54% ALFWorld Out-Success versus 90.16%, under matched training budgets.
The method summarizes trajectory prefixes into task-progress states and abstracts actions by their function relative to the goal, so corresponding decisions from different tasks enter the same comparison group. Concrete commands and concrete objects make comparable samples sparse, so the decisions must be abstracted before they are compared. Each context contributes one mean residual per batch to a decaying global memory, and each contrast is scaled by effective context support and uncertainty; task-success gains are reported on ALFWorld and WebShop at both the 1.5B and 7B scales.
The bias introduced by sharing experience has to be controlled separately, or frequently visited contexts dominate the estimates. Removing residualization or confidence adjustment lowers ALFWorld Out-Success below the variant that drops task-progress credit entirely. On WebShop, removing confidence adjustment lowers task success by 2.93 points and removing the exponential moving average lowers it by 4.10 points.
Perspective
The results cover two long-horizon environments, ALFWorld and WebShop, and two policy scales, Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct, trained for 160 updates with 16 tasks per update and eight trajectories per task. Task-progress states and action functions come from predefined environment-specific rules, while the statistical estimator is shared across the two environments.
The cross-task consistency test shows the highest-confidence bin agreeing more than the lowest in all five folds, but agreement is not monotonic in every fold; on WebShop at 7B the task score of 88.90 is comparable to HGPO's 88.96, with the gain showing up mainly in task success. The authors note that applying BLPO to new domains calls for suitable comparison groups and that learning transferable abstractions is a direction for broadening applicability.
