Skip to main content
Back to timeline
arXivSource publication:

Diffusion language models learn to rewrite their own tokens: PURL lifts GSM8K accuracy to 54.97% at matched unmask budgets

Synopsis

Masked diffusion language models write several tokens per step through parallel decoding, but tokens sampled independently within a step can contradict one another, and standard absorbing decoding never reopens a written token. PURL trains edit selection, unmask selection and token prediction jointly inside one normalized joint transition, so a single forward pass at inference supplies both unmasking and revision decisions. Across Sudoku, mathematical reasoning and code generation, accuracy improves by up to 47.4% at matched unmask budgets, and the TinyGSM model reaches 54.97% on GSM8K.

Source-provided article image: When Should Diffusion Language Models Reconsider Their Commitments?
arXiv

Interpretation

A masked diffusion model gains the ability to rewrite tokens it has already written, rather than only filling masked positions. Standard absorbing decoding freezes unmasked tokens, so later context cannot undo an earlier mistake; such models could generate in parallel but not revise selectively. On Sudoku, GSM8K mathematical reasoning, and HumanEval+ and MBPP+ code generation, compared at matched unmask budgets, PURL improves accuracy by up to 47.4% relative and reduces scheduled decoding rounds.

Each decoding step becomes a joint action: which unmasked positions to edit, which masked positions to reveal, and which tokens to write at those positions. These three decisions form a normalized joint transition whose likelihood factorizes into edit selection, unmask selection and token prediction, handled by an edit planner, an unmask planner and a shared token predictor. Training runs on partially masked sequences built from the model's own unmasking policy, with transient corruptions forming correction sequences; at inference one shared forward pass yields planner scores and token probabilities, and all three decisions commit in the same step.

Perspective

The conclusions rest on Sudoku, GSM8K with TinyGSM, and mathematical reasoning and code generation post-trained on LLaDA-8B-Instruct and Dream-v0-Instruct-7B respectively, all compared at matched unmask budgets; the authors note that visibility thresholds and corruption schedules remain heuristic.

The authors state that whole-policy comparisons neither isolate the causal effect of individual edits nor establish context accumulation as the sole mechanism; a scalar visibility fraction cannot identify which dependencies are available, and revision timing also reflects gates, budgets and protection rules.