Skip to main content
Search

# SemMask

1 related articles

Subscribe to updates
  1. arXiv

    Diffusion language models stop writing function words first: SemMask decodes semantic tokens early, reaching 0.7085 HumanEval accuracy in fewer steps

    Diffusion language models are meant to generate in parallel, yet in practice they often decode left to right and emit articles and prepositions before content words, weakening global planning. SemMask scores each token by the original model's gradient, then during training preserves high-scoring semantic tokens and up-weights their loss so nouns and verbs are decoded before function words. On Dream-7B, SemMask reaches 0.7085 accuracy on HumanEval, above Dream-7B-Instruct at 0.5366 and Qwen2.5-7B-Instruct at 0.6182, while average decoding steps fall from 201 to 169.

You’re all caught up