MMFlash reweights visually dependent tokens in diffusion drafting, lifting VLM decoding to 3.00–3.05× on image benchmarks
Synopsis
MMFlash extends diffusion-style speculative decoding to vision-language models by measuring how much each token depends on the image and upweighting those tokens during training. The measurement shows DFlash drafter acceptance falling from 46% in the lowest visual-dependence quartile to 22% in the highest, a gap that persists after multimodal training. On the Qwen3.5 series, MMFlash reaches average greedy-decoding speedups of 3.00–3.05× across seven image benchmarks, 10.3–10.9% above DFlash, and 2.33–2.71× on long-video benchmarks with sparse attention and light fine-tuning.
Interpretation
MMFlash gives lossless speculative-decoding speedups to vision-language models on both image and long-video inputs, while the drafter itself receives no visual tokens. The diffusion drafter DFlash was validated mainly on text-only tasks, leaving multimodal drafting largely unexplored, and its loss weights positions only by where they sit inside a block, so a token continuing a phrase and a token reading a number off a chart get the same gradient. On Qwen3.5-4B and Qwen3.5-9B, average greedy-decoding speedup across seven image benchmarks is 3.00–3.05×, 10.3–10.9% above DFlash trained on the same data; on three long-video benchmarks with sparse attention and fine-tuning, average speedup is 2.33–2.71×, 11.2–23.2% above DFlash.
The training signal comes from the target model: the difference between its next-token distributions with and without the image, under the same text prefix, measures how much a token depends on the image. Visual dependence says the image changed the prediction but not whether the target has settled on an answer there, so MMFlash multiplies the score by the target's confidence with the image and boosts only positions where the image both changes and settles the prediction. The final weight is a product of three factors: the static visual-dependence score, a verification-aware positional weight estimated from the drafter's own prefix-survival confidence, and the drafter's current probability of missing the token, which fades as it learns; training uses about 900K multimodal samples with responses regenerated by the target, block size 16, and 6 epochs on 8 NVIDIA H100 GPUs.
On long video, drafter attention concentrates on the question, the generated answer and nearby positions, while distant frames occupy most of the context but draw less attention per token. DFlash-style drafters attend to the full target context at every drafting step, yet average input length rises from 477 tokens across the image benchmarks to 41,218 tokens on video benchmarks. Across 60 questions, the most recent 256 positions hold 0.8% of the context but receive 50.1% of the attention; the MMFlash drafter therefore keeps the first 4 sink tokens, visual tokens sampled with stride 32, all text tokens and a 256-token local window, using the same pattern in training and inference.
Perspective
Evaluation covers the Qwen3.5-4B and Qwen3.5-9B target models, seven image benchmarks and three long-video benchmarks, with 200 randomly selected samples per benchmark in a zero-shot setting; generalization experiments use Gemma4-E4B-it and Qwen3.8-27B. The method targets vision-language models that take images or videos as input and generate text.
Multimodal training only partly closes the gap: block acceptance rises by about 5 points in the lowest quartile but by less than 3 points in the highest. Drafting on long videos remains harder than on images, and the method relies on a simple sparse attention pattern plus light fine-tuning rather than a drafter built specifically for video. Raising drafter depth from 5 to 8 layers changes average speedup by less than 1%; increasing tapped hidden states from 5 to 8 gives only a marginal speedup gain while raising online training time by 19.9%.
