Skip to main content
Search

# Diffusion Transformer

7 related articles

Subscribe to updates
  1. arXiv

    Video diffusion drops the pretrained autoencoder: JVT reaches FVD 136.5 on UCF-101 with a plain pixel-space Transformer

    JVT moves video diffusion back into pixel space, denoising with a large-patch DiT-style Transformer and no pretrained autoencoder, cascaded stages or U-Net-style skip connections. Pixel-space video generation had mostly relied on U-shaped backbones or cascades, while latent DiTs dominate. On UCF-101 class-conditioned generation JVT reaches FVD 136.5 against 477.97 for the latent DiT baseline Latte, and joint RGB-disparity generation raises RTA@5 from 72.4 to 79.5 in the camera-conditioned task.

  2. arXiv

    EagleDepth pairs latent diffusion guidance with a pixel diffusion decoder for faster, finer monocular depth at 4K

    EagleDepth combines depth-adapted latent diffusion with pixel-space generation: it first fine-tunes FLUX.2-klein-4B on paired RGB-depth data to predict a coarse depth latent from resized low-resolution RGB, then freezes that branch and fine-tunes the pretrained pixel diffusion decoder PiD to generate the depth map directly at the target resolution in a single step, conditioned on the coarse latent and the original high-resolution RGB, bypassing the VAE decoder; it reaches competitive overall accuracy on five common depth datasets and Synth4K, leads on high-frequency-masked metrics and boundary F1 across all five Synth4K subsets, and is the fastest among evaluated methods at 4K inference.

  3. arXiv

    COSMI composes 222k multi-object interaction sequences from single-object captures and trains a diffusion model that generates one to five objects

    COSMI composes multi-object human-object interaction data from single-object captures by extracting contact-consistent clips, mirroring them to balance the hands, and admitting only semantically and physically plausible pairings via a language model and geometric checks, yielding a dataset of 222k sequences and 275 hours with up to five objects (nearly thirty times the largest multi-object capture); it then trains a text-driven diffusion transformer whose object slots share weights and whose objects are predicted relative to the body joints that drive them, outperforming HIMO and multi-object adaptations of MDM and PriorMDM in text alignment and contact accuracy on a benchmark with an unseen object and unseen combinations, with the largest margin on the unseen object.

  4. arXiv

    LiFT lets looped DiTs keep improving beyond training depth: L/2 and XL/2 beat larger dense DiTs with fewer parameters and less inference compute

    LiFT supervises each loop of a recurrent DiT toward a point on a straight path from the model's initial velocity estimate to the flow-matching target, so a trained checkpoint can run far more inference loops than its training depth; on class-conditional ImageNet, L/2 and XL/2 checkpoints that reallocate inference compute toward recurrence reach lower FID than larger dense DiTs with fewer parameters and less inference compute, while at B/2 the dense model remains stronger.

  5. arXiv

    Dream4ACT unifies multi-embodiment video-action modeling with fixed-view action views, reaching 88.98% average success on RoboTwin 2.0

    Dream4ACT renders target joint configurations as "action views" under four fixed virtual cameras, jointly models physical-camera observations and action views with a shared video autoencoder and diffusion transformer under masked flow matching, and recovers executable joint targets by training-free, URDF-constrained multiview matching; it reaches 90.50% (clean) and 87.46% (randomized) success on RoboTwin 2.0, averaging 88.98%, a TriWorldBench score of 65.66, and closed-loop evaluation across five simulated embodiments and four real-world platforms.

  6. arXiv

    Deferring particle expansion to late transformer layers and adding time-dependent scoring-rule schedules lets a DDM trained from scratch in one stage reach 4.48 FID at 4 steps and 2.38 at 50 steps on ImageNet-256²

    To address two obstacles in scaling Distributional Diffusion Models (DDMs)—multi-particle training overhead that grows with the number of particles, and globally fixed scoring-rule hyperparameters that force a single trade-off across sampling budgets—the work defers particle expansion to late transformer layers and introduces time-dependent scoring-rule schedules informed by the dynamical regimes of Biroli2024; combined with a DiT-based latent setup, this makes DDM training practical on class-conditional ImageNet-256², reaching 4.48 FID at 4 steps and 2.38 at 50 steps with DiT-XL/2 from a single model trained from scratch in one stage, without a teacher, self-distillation or JVPs, and the same recipe transfers to text-to-image generation.

  7. arXiv

    DRM replaces the scalar reward head with a diffusion head: matching or beating same-data, same-backbone baselines on five benchmarks while its reward distribution turns multimodal as human disagreement grows

    The work introduces DRM (Diffusion Reward Model), which recasts reward modeling as conditional density estimation: on top of a frozen LLM encoder, a lightweight Diffusion Transformer denoises Gaussian noise into a reward vector with no parametric family assumed, a single architecture handles both multi-attribute regression and pairwise preference data, it matches or surpasses the same-data, same-backbone scalar head ArmoRM and quantile head QRM across five benchmarks, and it exhibits multimodal structure tied to human disagreement on repeated-annotation data, with uncertainty-aware rejection, LCB aggregation, and downstream RLHF experiments demonstrating the practical value of distributional information.

You’re all caught up