Diffusion language models get block-level RL: PBPO clips per token and holds its own across six benchmarks
Synopsis
Diffusion large language models decode block by block, denoising several tokens within each block, and the exact likelihood of a completed block is intractable because it marginalizes over intermediate states and reveal orders, so block-level policy optimization has lacked a usable objective. PBPO builds a prefix-block state and uses a single model evaluation to define a factorized one-step surrogate for the block likelihood, whose block surrogate ratio is the product of its token surrogate ratios, then applies surrogate importance weighting and clipping separately to each token and resamples corrupted views of the prompt and revealed prefix. Across two LLaDA-family initializations, six benchmarks and three generation lengths, PBPO performs competitively against existing dLLM RL methods.
Interpretation
Reinforcement learning for diffusion large language models can now optimize policy at the block level instead of working around an intractable block likelihood. The earlier route adapted autoregressive policy optimization directly to blocks, but the probability of a completed block sums over intermediate denoising states and reveal orders and cannot be written exactly. Across two LLaDA-family initializations, six benchmarks and three generation lengths, PBPO achieves competitive performance against existing dLLM RL methods.
The construction uses a single model evaluation to define a prefix-block state and, from it, a factorized one-step surrogate for the block likelihood. That surrogate writes the block surrogate ratio as a product of its token surrogate ratios, so variation at an individual position propagates across the whole block before clipping. PBPO preserves block-level conditioning but applies surrogate importance weighting and clipping separately to each token, and diversifies repeated updates by resampling corrupted views of the prompt and revealed prefix, a design motivated by masked-dLLM pre-training.
Perspective
The demonstrated scope is a comparison against existing dLLM RL methods under two LLaDA-family initializations, six benchmarks and three generation lengths, aimed at researchers and engineers working on post-training of diffusion language models.
Under the analysis assumptions, block-product weighting amplifies the variance of gradient contributions relative to token-local weighting; the reported outcome is competitive performance against existing methods rather than across-the-board superiority.
