Video diffusion drops the pretrained autoencoder: JVT reaches FVD 136.5 on UCF-101 with a plain pixel-space Transformer
Synopsis
JVT moves video diffusion back into pixel space, denoising with a large-patch DiT-style Transformer and no pretrained autoencoder, cascaded stages or U-Net-style skip connections. Pixel-space video generation had mostly relied on U-shaped backbones or cascades, while latent DiTs dominate. On UCF-101 class-conditioned generation JVT reaches FVD 136.5 against 477.97 for the latent DiT baseline Latte, and joint RGB-disparity generation raises RTA@5 from 72.4 to 79.5 in the camera-conditioned task.
Interpretation
Video diffusion can run entirely in pixel space on a backbone that is just a DiT-style Transformer over large spatiotemporal patches, with no pretrained autoencoder compressing video into a latent space and no auxiliary detailer, cascaded stage or U-Net-style skip connection. Pixel-space video generation had generally kept U-Net-style skip connections or cascaded generation, while high-resolution video generation was dominated by autoencoder compression followed by latent-space DiT denoising. On UCF-101 class-conditioned generation JVT reaches FVD 136.5, where the latent DiT baseline Latte scores 477.97 in the same table; on unconditional FaceForensics, SkyTimelapse and Taichi-HD, JVT scores 32.86, 46.58 and 119.1 against Latte's 34.00, 59.82 and 159.60.
What makes raw video tractable is the patch embedding, which supplies the initial downsampling that an autoencoder provides in latent models and aggregates spatiotemporal information into a fixed-length token sequence. Standard video Transformers project non-overlapping spatiotemporal patches into tokens with a single 3D convolution; JVT compares that with a channel-bottleneck version and with two orders of separating spatial and temporal operations. Under identical training settings every bottlenecked embedding variant ends with lower training loss than the plain tubelet baseline; weighing quality against efficiency, JVT defaults to patch-first embedding, a per-frame 2D convolution followed by a per-location 1D convolution, with bottleneck width 32.
Pixel space also lets video and pixel-aligned auxiliary modalities be concatenated on one shared spatiotemporal grid and generated jointly, with RGB and disparity denoised under the same noise level in the camera-conditioned task. Latent diffusion typically reuses an RGB-specific autoencoder for aligned modalities such as depth, and inference needs an additional reconstruction model to recover geometry. In 8-frame camera-conditioned generation on RealEstate10K, the joint RGB-disparity version reaches RTA@5 of 79.5 and AUC@5 of 58.6 versus 72.4 and 52.8 for RGB-only JVT, with FVD rising slightly from 27.0 to 27.5; predicted disparity converted back to depth unprojects directly into a sequence of world-coordinate point clouds.
Perspective
The demonstrated scope is short clips of 16 frames at fixed resolution: class-conditioned and unconditional generation are evaluated on UCF-101, FaceForensics, SkyTimelapse and Taichi-HD, and camera-conditioned generation uses 8-frame experiments on RealEstate10K. The authors propose extending pixel-space video generation to larger models, more datasets, longer videos and higher resolutions.
Register guidance helps clearly only on the more semantically diverse UCF-101 and brings no obvious gain on the other three datasets; in camera-conditioned generation relative rotation accuracy is consistently high while relative translation accuracy is substantially lower, and joint disparity only narrows that gap. The authors attribute the dataset-dependent register effect to global semantic diversity as a hypothesis.
