Diffusion language models stop writing function words first: SemMask decodes semantic tokens early, reaching 0.7085 HumanEval accuracy in fewer steps
Synopsis
Diffusion language models are meant to generate in parallel, yet in practice they often decode left to right and emit articles and prepositions before content words, weakening global planning. SemMask scores each token by the original model's gradient, then during training preserves high-scoring semantic tokens and up-weights their loss so nouns and verbs are decoded before function words. On Dream-7B, SemMask reaches 0.7085 accuracy on HumanEval, above Dream-7B-Instruct at 0.5366 and Qwen2.5-7B-Instruct at 0.6182, while average decoding steps fall from 201 to 169.
Interpretation
SemMask changes the decoding order of diffusion language models from function-word-first to semantic-first, fixing informative words such as nouns and verbs before filling in articles, prepositions and other local pieces. Under default pure diffusion sampling, the models show pseudo-autoregressive behavior and systematically decode low-information function words earlier while delaying high-information semantic tokens, hurting long-form writing and code generation that depend on global planning. On HANNA, Dream-7B-SemMask reaches 0.4613 BLEU and 0.8254 BLEURT, above Dream-7B-SFT at 0.4086 and 0.7364; on HumanEval, Dream-7B-SemMask reaches 0.7085 accuracy, above Dream-7B-Instruct at 0.5366.
The procedure has three steps: a frozen diffusion model scores each token's semantic importance, the scores decide which tokens stay unmasked first during training, and high-scoring tokens receive larger weight on their prediction error. The score is the projection of the denoising loss gradient with respect to a token embedding onto the direction from the original token embedding to the mask token embedding; a larger projection means the token contributes more to reconstructing the masked positions and is treated as a semantic anchor. Training ranks tokens by score, keeps the top ones as semantic anchors, and constrains the minimum distance between anchors to cover the whole sequence; LLaDA-8B-Instruct and Dream-7B-Instruct are fine-tuned with LoRA on 10,000 text-writing samples and 10,000 code samples at a 4,096-token context length.
Perspective
The demonstrated scope is text writing (HANNA, CommonGen) and code generation (HumanEval, MBPP) with LLaDA-8B and Dream-7B backbones, trained on 10,000 GPT-5.1-filtered samples per task; efficiency figures are measured on V100 GPUs at sequence length 512.
On MATH-500, SemMask differs only marginally from the corresponding SFT baselines, with Dream-7B-SemMask at 39.58, slightly below Dream-7B-SFT at 39.66; in the CommonGen ablation, the random strategy has a slightly higher GOD score than SemMask but clearly weaker semantic metrics.
