Seq-Drifting writes a whole sentence in one forward pass from noise, reaching 36.01 generative perplexity on LM1B
Synopsis
Diffusion language models emit tokens in parallel but still need multiple sampling steps for high-quality text. Seq-Drifting moves drifting training onto sequences of continuous token representations, builds a context-dependent candidate set at each position, and forms a training target from attraction and repulsion, so the generator maps noise to a full sequence in one forward pass. On LM1B, Seq-DriftingP, which builds support sets with a public pretrained GPT-2, reaches a one-step generative perplexity of 36.01, below the 98.76 of the four-step distilled baseline FMLM.
Interpretation
Text generation can happen in a single forward pass: the generator maps a shared noise vector to continuous token representations for the whole sequence and decodes them by cosine similarity, with no iterative sampler at inference. Autoregressive models decode one token at a time, so latency grows with sequence length; diffusion language models emit tokens in parallel but still rely on multiple refinement steps, and distillation methods such as SDTT, Di4C and FMLM remain limited at small sampling budgets. On LM1B, one-step Seq-DriftingP reaches a generative perplexity of 36.01 against 98.76 for the four-step distilled baseline FMLM; on OpenWebText the two are 128.33 and 111.49.
Supervision no longer comes from the observed token at each position in a training sentence; each position instead receives a set of candidate tokens that follows the current generated context, attraction pulls the output toward the nearest candidate, and repulsion keeps different noise samples from collapsing together and suppresses repetition within a sequence. Using the observed token of a training sequence directly as the per-position drift target can mismatch the generator's current context and yield a weak or misaligned update. Candidate sets can be built either by corpus retrieval or by scoring the generator's current prefix with a frozen language model and taking its most probable tokens; experiments find the latter stronger and adopt it by default, and the support model is queried only during training, not at inference.
Attraction and repulsion are computed as angles on the unit sphere, and the update takes one step along a great circle, so repulsion changes direction rather than growing vector norms. Under cosine decoding, radial growth increases Euclidean distance without changing token identity, so unconstrained repulsion can spend its force on norms without adding token-level diversity. In ablations, replacing the spherical update with a Euclidean one lets repulsion increase embedding norms and makes training diverge; dropping the intra-sequence repulsion term collapses output into within-sentence token repetition, with unconditional perplexity falling to 10.01 and entropy to 1.82.
Perspective
The demonstrated scope is unconditional and conditional generation on LM1B and OpenWebText, plus WMT14 translation, XSum summarization and the SVAMP, ASDiv and ProofWriter reasoning tasks; conditional generation uses a 64-token context, with 512 output tokens on OpenWebText. The authors propose future work on stronger support models, richer semantic supervision and improved token recovery to close the gap to multi-step systems.
A gap to multi-step results remains on translation and summarization, which the authors partly attribute to a mismatch between the candidate sets used in training and the single reference used for evaluation. Hyperparameter sweeps show generative perplexity first improving and then worsening as several parameters move away from a middle range, and too small a support set collapses output. Training conditional generation from scratch requires an unconditional warm-up.
