Skip to main content
Back to timeline
arXivSource publication:

Multi-agent chat no longer collapses to one score: OPSD4MAS turns future replies into token-level credit, lifting math pass@4 by up to 20 points

Synopsis

Multi-agent reinforcement learning has compressed inter-agent dialogue into scalar rewards, which say whether to reinforce a response but not which tokens to revise. OPSD4MAS extracts facts from later communication, ranks them by association and screens them with leave-one-out testing to form privileged context, then compares token probabilities for the same policy with and without those facts to obtain token-level credit, trained jointly with GRPO under linear annealing. Across six math benchmarks, the variant using a four-response future window raises mean pass@1 and pass@4 over GRPO by 8.87 and 9.54 percentage points, with single-point pass@4 gains up to 20.00 points.

Source-provided article image: Turning Multi-Agent Communication into Token-Level Credit with On Policy Self Distillation
arXiv

Interpretation

In multi-agent training, later utterances between agents can now become a per-token learning signal rather than only a score attached to a response. Earlier scalar-reward routes either leave dialogue content unused or compress it into one score, and a response-level advantage says whether to reinforce a response but cannot distinguish which tokens support or oppose a correction. On six math benchmarks, AMC23, AIME24, AIME25, MATH-500, Olympiad and Minerva, OPSD4MAS with a four-response future window exceeds GRPO by 8.87 and 9.54 percentage points in mean pass@1 and pass@4, and exceeds DAPO by 6.14 and 8.48 points.

The mechanism compares the same role policy's token probabilities under two contexts, one carrying selected future facts and one with only the ordinary context. The fact-carrying view acts as a privileged teacher, and its log-probability difference from the ordinary-context view forms a token-varying coefficient that is detached from gradients, so gradients flow only through ordinary-context probabilities. An external model extracts facts, association filtering ranks them by outcome advantages of same-query trajectories, and leave-one-out filtering checks whether each fact moves target-token probabilities in the direction of that response's agent-wise advantage; training uses Qwen3-4B agents on DAPO-MATH-17k with a linearly annealed joint objective alongside GRPO.

Future dialogue cannot be used raw as privileged context, because unprocessed communication makes training worse. On the three competition-math datasets, raw future communication reaches mean pass@1 and pass@4 of 33.24 and 50.46, below GRPO alone at 49.63 and 66.60. The full pipeline reaches 64.91 and 82.74; fact extraction adds 9.33 and 13.52 points over raw communication, association filtering adds 0.42 and 1.21 points, and leave-one-out filtering contributes the largest further gain of 21.92 and 17.56 points.

Perspective

The demonstrated scope is Qwen3-4B agents trained on DAPO-MATH-17k and evaluated on six math benchmarks, AMC23, AIME24, AIME25, MATH-500, Olympiad and Minerva, with inference using neither future communication nor the external model.

Evaluation focuses on math reasoning at the Qwen3-4B scale, and extending to other tasks, model scales and interaction structures remains future work; fact extraction and relation labeling rely on an external model, so privileged-context quality can depend on that model and on filtering thresholds, context budgets and annealing. The injection diagnostic evaluates at fixed weights with frozen histories, and its four checkpoints belong to one training run, so checkpoint differences do not measure variation across runs.