Skip to main content
Search

# Diffusion Language Model

16 related articles

Subscribe to updates
  1. arXiv

    MMFlash reweights visually dependent tokens in diffusion drafting, lifting VLM decoding to 3.00–3.05× on image benchmarks

    MMFlash extends diffusion-style speculative decoding to vision-language models by measuring how much each token depends on the image and upweighting those tokens during training. The measurement shows DFlash drafter acceptance falling from 46% in the lowest visual-dependence quartile to 22% in the highest, a gap that persists after multimodal training. On the Qwen3.5 series, MMFlash reaches average greedy-decoding speedups of 3.00–3.05× across seven image benchmarks, 10.3–10.9% above DFlash, and 2.33–2.71× on long-video benchmarks with sparse attention and light fine-tuning.

  2. Diffusion language models learn to count characters: no fine-tuning, summaries land within 3.1 characters of target

    For summarization settings such as headlines and previews that must fit an exact character count, Delta estimates the expected character length of the current output inside a masked diffusion language model's denoising loop and dynamically inserts or removes masked positions to steer generation toward the target. It needs no fine-tuning and applies to off-the-shelf LLaDA and Dream. Across the XSum, CNN/DailyMail and arXiv summarization benchmarks, character-length error falls by an order of magnitude relative to the baselines while summary quality stays competitive under automatic and human evaluation.

  3. arXiv

    Diffusion language models learn when to rewrite: DiVR lifts a fine-tuned generator's five-benchmark average from 35.3% to 43.4%

    Masked diffusion language models can remask generated text, but deciding where to revise by confidence or self-judgment often leaves the error in place. DiVR instead learns an outcome verifier on the generator's own decodes and, at each block boundary, judges whether the partial solution can still reach a correct answer, resampling the whole block when the score is low. On the supervised fine-tuned LLaDA-8B-Base generator, the five-benchmark math and code average rises from 35.3% to 43.4% at the cost of greedy decoding.

  4. bioRxiv

    A phage DNA model learns to denoise both ways: CPDD corrects 42.4% of visible wrong bases in one pass

    Autoregressive DNA language models generate left to right, yet genome repair and editing usually condition on both flanks. The CreatiPhage team converted a 144.5M-parameter phage autoregressive model into CPDD, a 147.5M-parameter bidirectional discrete diffusion model; on 522 screened phage coding sequences it restores randomly masked bases more accurately than the comparable diffusion model PhaGen, and in a mixed-corruption test a single forward pass returns 42.4% of visible incorrect bases to the correct base.

  5. arXiv

    Diffusion language models can learn from autoregressive teachers: JustOPD lifts Dream-7B's math reasoning while keeping any-order decoding

    Autoregressive models generate strictly left to right, while diffusion language models denoise in arbitrary order, so the former's token-level supervision does not transfer directly. JustOPD forces the diffusion student to roll out left to right during training, letting a Qwen3-14B teacher supervise every intermediate state, while inference keeps any-order decoding. With Dream-7B as student and Qwen3-14B as teacher on GSM8K and MATH500, scores rise over the base model across several divergences.

  6. arXiv

    Diffusion language models without pretraining from scratch: LoRA-only conversion turns four autoregressive models diffusive in hours on one GPU

    BYOD is a LoRA-only recipe for converting autoregressive language models into diffusion language models: every base parameter stays frozen, each layer's causal mask becomes bidirectional, and a masked-diffusion objective trains at most 102.4M tokens in 3.1–4.0 hours on one Google Colab GPU, turning Gemma 2 9B, Llama 3.1 8B, Qwen2.5 7B and Ministral 8B into models that generate from fully masked sequences, with adapters at 4.1–5.8% of final parameters. The strongest conversion, BYOD-Gemma, reaches 50.6% average accuracy on nine tasks against 51.9% for LLaDA 8B Instruct in the same harness; open-generation quality at low NFE budgets remains the main limitation.

  7. arXiv

    Diffusion-model RL stops scoring whole answers: part-of-speech credit lifts MATH500 by 9.0 points over the base model at the smallest budget

    Diffusion language models commit tokens over many denoising steps, so a whole-response reward says little about which tokens helped later refinement. DTCA estimates each token's contribution from the confidence gain at still-masked positions after that token is committed, averages these signals by part-of-speech category into stable weights, and uses them to pick denoising timesteps and assign token-level learning signals, with entropy-based calibration. On LLaDA-8B-Instruct it beats the base model at generation budgets of 128, 256 and 512, by 9.0, 6.8 and 4.8 points on MATH500 and 5.3, 2.1 and 2.5 points on GSM8K.

  8. aclanthology.org

    A million dollars per fusion shot: a diffusion language model cuts neutron-peak timing error from 11.6 to 9.2 steps

    Each inertial confinement fusion shot costs on the order of $1M, which makes fast AI surrogates valuable. ICF-DLM splits the 512-step neutron-rate waveform into total yield, peak timing and a short local waveform around the peak, generates them jointly with a bidirectional diffusion language model, and refines it with physics-driven reinforcement learning. On the 50-shot held-out experimental test set of ICFBench its peak-timing error is 9.2 steps, against 11.6 steps for an autoregressive LLaMA-3-8B trained with the same SFT plus PPO recipe.

  9. arXiv

    Diffusion-LM agents skip the text step: one hidden message beats four exchanges by 7.0 points on QASC

    Diffusion language model agents can pass compressed denoising states to each other without decoding text first. A sender packs its hidden state into four signed dense vectors, and the receiver keeps its own context and trajectory while denoising. On QASC, matched messages raise accuracy from 72.6% to 80.8%, a message from another question drops it to 46.8%, and one late transfer reaches 87.8%, 7.0 points above four recurrent exchanges.

  10. arXiv

    Diffusion language models stop reusing one steering vector: feeding the previous prediction back lifts TruthfulQA multiple-choice average from 30.89 to 45.42

    TraceSteer feeds a diffusion language model's previous denoising prediction back into activation steering, adapting both the correction direction and its strength step by step, and improves factual reliability on TruthfulQA for LLaDA-8B-Instruct and Dream-7B-Instruct.