Skip to main content
Back to timeline
arXivSource publication:

Lossless diffusion without touching the model: LLDM splits pixels into residuals and rescales them, cutting ImageNet gFID from 104.29 to 11.64

Synopsis

The Lossless Latent Diffusion Model (LLDM) takes a different route: instead of changing the diffusion architecture or objective, it builds an invertible transform that maps pixels into a representation easier for diffusion to model. It keeps a pretrained lossy VAE's latent and concatenates the reconstruction error as a residual along the channel dimension to guarantee pixel-exact reconstruction, then applies uneven scaling to that residual. On ImageNet class-to-image, LLDM reaches gFID 11.64 versus 104.29 for pixel-space diffusion; without classifier-free guidance, SD-VAE is 26.02 and FLUX2-VAE is 10.18.

Source-provided article image: Lossless Latent Diffusion Models
arXiv

Interpretation

Image diffusion models can now keep pixel-exact reconstruction while following the standard latent-diffusion training pipeline, without a special pixel-space architecture or objective. Earlier routes either modeled pixels directly and modified the training objective or architecture, or used a lossy VAE for modeling efficiency and gave up exact reconstruction. On ImageNet class-to-image, LLDM reaches gFID 11.64 versus 104.29 for pixel-space diffusion; without classifier-free guidance, SD-VAE is 26.02 and FLUX2-VAE is 10.18.

The method reorganizes image information into an invertible latent representation that presents diffusion with an easier distribution to learn. It uses residual decomposition: keep the pretrained lossy VAE's latent and concatenate the reconstruction error as a residual along the channel dimension, so the original image can be recovered exactly. Uneven latent scaling then multiplies the residual by a factor below one; on FLUX2-VAE, residual decomposition lowers gFID from 52.31 to 48.37, and adding scaling lowers it to 11.66.

To explain why the transform works, the authors propose semantic rank priority (SRP), which measures linear classification accuracy only on the top principal components of the latent. Conventional linear classification accuracy is invariant to invertible linear transforms, so it cannot see the difference uneven scaling makes. On a benchmark of 22 open-source lossy VAEs, SRP@5 correlates with diffusion gFID at a Pearson coefficient of 0.86, versus 0.49 for conventional Acc@5.

Perspective

The demonstrated scope is ImageNet class-to-image and Diffusion-Bench text-to-image, with diffusion models trained as SiT-XL for 40 epochs; the conclusions are limited to these datasets, training protocol and the reported resolution settings.

The optimal scaling factor varies widely with the base VAE: any value degrades generation on RAE, VTP-L is best at 0.1, and FLUX2-VAE prefers 0.01; learning an invertible transform end-to-end for diffusion modeling remains an open challenge.