Streaming video compression stops waiting for whole clips: SGVC turns a video DiT into a causal per-frame refiner, saving 71.97% BD-rate on DISTS
Synopsis
Generative video compression at ultra-low bitrates leans on pretrained video diffusion transformers, but reconstruction passes through a costly VAE decoder, and jointly refining multiple frames adds buffering delay. SGVC projects decoded codec features straight into the video DiT input embedding space, uses embedding alignment to close the representation gap, and propagates layer-wise K/V states across frames so spatiotemporal priors survive causal frame-by-frame decoding. Across HEVC Class B, UVG and MCL-JCV, SGVC saves an average 71.97% BD-rate on DISTS and 35.94% on LPIPS relative to S2VC.
Interpretation
Generative video compression can decode frame by frame without buffering whole clips while still calling on the spatiotemporal priors of a pretrained video diffusion transformer. Earlier frame-wise diffusion methods used image diffusion backbones that do not natively model spatiotemporal dependencies and needed separate temporal modules; clip-wise methods jointly refined multiple frames to use video priors, at the cost of multi-frame buffering delay. Across HEVC Class B, UVG and MCL-JCV, SGVC lowers average BD-rate relative to S2VC by 35.94%, 71.97% and 89.15% on LPIPS, DISTS and FID respectively.
Decoded codec features go directly into the video DiT input embedding space for one-step refinement, and the refined hidden states return to the pixel domain through a lightweight reconstruction path instead of the pretrained VAE decoder. Earlier generative video compression still needed multi-stage upsampling decoders to reconstruct frames after refinement, adding computation and decoding latency. In the DIV2K I-frame ablation anchored on SGVC, the variant that aligns to the VAE latent space and decodes with the VAE decoder reports BD-rates of -7.60%, -10.87%, -4.23% and -18.75% on PSNR, LPIPS, DISTS and FID, but takes 1498.54 ms to decode versus 891.36 ms for SGVC.
Embedding alignment during training pulls codec-derived embeddings toward reference embeddings obtained by passing original frames through the pretrained VAE encoder and DiT patch embedder, so the pretrained DiT's generative prior acts on compressed features. Removing embedding alignment while keeping the rest of SGVC raises I-frame BD-rate by 26.62%, 42.40% and 99.58% on LPIPS, DISTS and FID. In the UVG P-frame ablation anchored on full SGVC, removing both embedding alignment and in-context propagation raises BD-rate by 51.93% on DISTS and 44.49% on FloLPIPS.
In-context propagation turns each DiT self-attention layer into a module with a K/V cache, initialized during I-frame refinement, so queries of later P-frames attend directly to cached historical keys and values. The pretrained video DiT is natively bidirectional, and clip-wise methods obtained temporal context by processing multiple frames jointly, which rules out causal frame-by-frame decoding. In the UVG P-frame ablation, embedding alignment alone gives a FloLPIPS BD-rate of 16.45%, while enabling both embedding alignment and in-context propagation gives 0.00%, at the cost of decoding time rising from 728.37 ms to 867.58 ms and peak GPU memory from 4.51 GiB to 5.94 GiB.
Perspective
Evaluation covers HEVC Class B, UVG and MCL-JCV under the low-delay P configuration with an intra period of 32 over the first 96 frames of each sequence, with bitrate in bits per pixel and quality measured by PSNR, LPIPS, DISTS, FID and the RAFT-based warping error Ewarp; the model is trained on 256×256 crops from Vimeo-90K, with the DiT initialized from Wan-2.1-T2V-1.3B and adapted by rank-64 LoRA.
Long-sequence fine-tuning uses 13-frame and 25-frame sequences built by temporally reversing and concatenating seven-frame clips rather than genuine long videos, and some perceptual degradation persists over extended prediction chains; P-frame decoding at 1080p takes 845.7 ms per frame (about 1.18 fps), limited by the large pretrained DiT, so real-time deployment remains constrained.
