MPEG-Diffusion spends video-generation compute only where detail matters, speeding denoising on Wan2.1 while staying near full-resolution quality
Synopsis
Video diffusion transformers generate every region of every frame at one resolution, so attention cost grows with token count, while mixed-resolution generation had sharply degraded video quality and required the high-resolution region in advance. MPEG-Diffusion attributes those failures to shared keys and values, decoder neighbor sensitivity and an externally supplied region, and addresses each with dual-resolution attention, a mixed-resolution decoder and a high-resolution region read off DiT features. On Wan2.1 1.3B and 14B it processes only a fraction of full-resolution tokens, speeds up denoising and beats prior mixed-resolution methods on nearly every metric, with raters preferring it over Phase-Aligned RoPE and Foveated Diffusion.
Interpretation
Video generation can concentrate compute on the visually important part of a frame, generating that region at high resolution and the rest at low resolution to cut the cost of every denoising step. Prior mixed-resolution methods lost quality on video, with blurry detail and visible distortion along the resolution boundary, and they required the high-resolution region to be known in advance. On Wan2.1 1.3B at 480p and 14B at 720p, MPEG-Diffusion beats Phase-Aligned RoPE and Foveated Diffusion on quality metrics at the same token budget; in a blind study, 15 raters comparing 400 video pairs preferred it over those two methods in 98.8% and 90.1% of pairs.
The method has three parts: each query reads only the keys and values of its own resolution, the high- and low-resolution latents are blended before decoding, and the high-resolution region is selected by the model itself during denoising. Dual-resolution attention fills in missing keys and values from the other resolution with linear maps, replacing the earlier practice of sharing one set of keys and values across high- and low-resolution tokens. On Wan2.1 1.3B trained for 5,000 steps with all else fixed, temporal LPIPS at the seam falls from 4.59 to 4.14 and warping error from 2.61 to 1.96; comparing the region head's predicted high-resolution mask with main-subject masks labeled by an LLM and Grounded SAM across 1,003 generated MovieGen videos, the source reports that the linear head accurately locates and tracks the main subject.
Perspective
The demonstrated setting is text-to-video generation with Wan2.1 1.3B at 480p, 14B at 720p, and 14B at 960p with a 240p low-resolution track, plus a test on the text-to-image model FLUX.2 klein 4B; evaluation uses 1,003 MovieGen prompts, 944 VBench prompts and 1,000 held-out image prompts.
Dual-resolution attention assumes a linear relation between the high- and low-resolution tokens of an area, a structural constraint that may still cause slight quality degradation; the difference between the blurry low-resolution region and sharp high-resolution detail is sometimes obvious and needs smoother blending; the region head is learned from subject labels given by an LLM and Grounded SAM, with end-to-end fine-tuning left as future work.
