Skip to main content
Back to timeline
arXivSource publication:

Masked diffusion models learn to rewrite: LOOT revises generated tokens, beating ReMDM's 2048 evaluations with 256

Synopsis

Once a masked diffusion model fills in a token it cannot change it, and training signal lands only on masked positions. The Leave-One-Out Transformer (LOOT) makes each position's prediction independent of that position's own input, so visible tokens can be supervised and generated tokens resampled during inference. On OpenWebText, LOOT with Blocked Gibbs correction reaches a MAUVE of 0.408 at 256 forward evaluations, above ReMDM's 0.348 at 2048; the cost is about 2.2 times the FLOPs per forward pass and a likelihood gap to standard bidirectional MDMs at longer sequences.

Source-provided article image: Empowering Masked Diffusion Models to Self-Correct with Leave-One-Out Transformers
arXiv

Interpretation

Masked diffusion language models gain the ability to rewrite already generated tokens at inference, rather than only replacing [MASK] with predictions one at a time. A standard masked diffusion model fixes a position once it is unmasked, so errors propagate through later generation steps; the training loss is also computed only on masked positions, leaving visible tokens without direct supervision. On unconditional generation over OpenWebText at sequence length 1024, LOOT-Blocked-Gibbs reaches a MAUVE of 0.408 at 256 forward evaluations, while ReMDM reaches 0.348 at 2048 evaluations, with matched sample entropy.

The model predicts each position only from its surrounding context, not from the token currently at that position, so visible positions also become a supervised prediction task. Earlier hollow transformers merged left and right context only at the final layer or every few layers, and HollowDiff needed a separate MDM predictor model alongside it. In short LM1B ablations, LOOT merging at every layer reaches a validation perplexity of 36.23, better than HollowDiff's 37.23 with merging every two layers and 37.91 every three layers.

The training objective covers all positions instead of only masked ones and removes inverse-probability weights that diverge at the endpoints of the noise schedule. The half-and-half loss used by HollowDiff and PGM also supervises every position, but its NELBO weights diverge at both ends of the noise schedule. On LM1B, the LOOT objective lowers validation perplexity by 3.12 relative to the half-and-half loss (36.23 versus 39.35); on OpenWebText, holding the LOOT architecture fixed, it improves perplexity by 2.89 and 2.80 at sequence lengths 128 and 1024.

A single forward pass yields replacement distributions at every position, so classical samplers such as Gibbs and Metropolis–Hastings can be applied directly to text correction. ReMDM's remasking approach needs several forward passes per correction, and choosing which token to remask relies on heuristics such as confidence at unmasking time, which go stale as context fills in. On OpenWebText, the same LOOT model without correction has a MAUVE of only 0.010 at 256 evaluations; adding Blocked Gibbs or Metropolis–Hastings raises it to 0.408 and 0.312 respectively.

Perspective

The demonstrated scope is language modeling and unconditional generation on LM1B and OpenWebText at roughly 140M to 170M parameters, with sampling evaluated using GPT-2 Large for generative perplexity and 5,000 samples for MAUVE. The intended reader is a researcher working on diffusion language model training objectives and inference sampling.

LOOT retains a likelihood gap to standard bidirectional MDMs at longer sequences, with the leave-one-out architecture itself costing 4.44 perplexity at sequence length 1024; the three correction strategies differ substantially, so generation quality depends strongly on the sampling algorithm; each forward pass costs about 2.2 times the FLOPs of a standard MDM.