Skip to main content
Back to timeline
arXivSource publication:

Diffusion language model post-training routes by outcome: tool-calling beats the distilled checkpoint by 18.3 points, while math swings with the seed

Synopsis

Bidirectional denoising makes the policy likelihoods that PPO-style objectives need unreliable in diffusion language models, so a likelihood-free pipeline combines multi-teacher on-policy distillation, REINFORCE grouped by verifier outcome, and gradient-norm anchoring. On DiffusionGemma-26B-A4B at 24 denoising steps, the routed arm scores 18.3 points above the distilled checkpoint on tool-calling but 2.8 and 2.6 points below it on math at 24 and 48 steps; the anchored arm restores reasoning-trace accuracy, with a tool-calling difference whose confidence interval includes zero. A domain-conditional recipe keeps the tool-calling gain across two seeds, while the math result depends on the seed.

Source-provided article image: Outcome-Routed Distillation and Reinforcement Learning for Diffusion Language Models
arXiv

Interpretation

Post-training for diffusion language models can now split teacher distillation and verifier reward signals by each sample's outcome without computing policy-likelihood ratios. On-policy distillation gives dense supervision on the student's own samples but its targets still come from the teacher; verifier reinforcement learning gives an outcome signal but no direct constraint on capabilities outside the verifier. On DiffusionGemma-26B-A4B at 24 denoising steps, the distillation arm scores 70.24 on tool-calling and the routed arm 88.55, 18.3 points higher with a confidence interval excluding zero; the same routed arm is 2.8 points lower on math at 24 steps and 2.6 points lower at 48 steps, both confidence intervals excluding zero.

The routing rule assigns a target to each rollout group by verifier outcome: mixed groups use group-centred REINFORCE, all-fail groups with a successful teacher use dense KL, and all other groups are skipped. Group-centred REINFORCE is a policy-gradient form without importance-sampling ratios, so it avoids the unreliable likelihood estimates that bidirectional denoising produces. In the tool-calling training pool the teacher supplies successful completions for only a small share of the groups the student fails completely; math has the opposite profile, with most groups using RL and the teacher usually succeeding, and both routed arms fall below the distillation arm at 24 and 48 steps but not at 8 steps.

The anchor term references the distilled checkpoint and sets its weight from the measured gradient-norm ratio at each step, limiting changes outside the verifier reward. A fixed loss coefficient cannot hold a fixed anchor strength: the raw anchor-to-primary gradient-norm ratio spans two orders of magnitude over the 240 steps, so a coefficient chosen at one step can differ greatly from the value another step requires. The anchored arm raises reasoning-trace accuracy relative to the routed arm with a confidence interval excluding zero, and its tool-calling difference has a confidence interval including zero; that comparison also removes the fallback distillation route, so the anchor-scale effects are treated as provisional.

Perspective

The results come from post-training one mixture-of-experts diffusion language model, DiffusionGemma-26B-A4B, for 240 optimiser steps per arm, evaluated on math, workflow and tool-calling at 8, 24 and 48 denoising steps with 24 as the operating point; every comparison is paired per item within one sweep on one GH200-class GPU. Tool-calling training is single-turn, and the multi-turn tool-execution set only measures transfer.

The difference between the anchored and routed arms combines adding the anchor with removing the fallback distillation route, and the data do not isolate the two, so the anchor-scale effects remain provisional. The step dependence in math is reported as a hypothesis rather than an established property, and its differences fall inside the observed seed spread. The composed recipe's math point estimates remain below the distillation arm at every step count, and non-separation is not equivalence.