Skip to main content
Search

# Diffusion Transformer

13 related articles

Subscribe to updates
  1. arXiv

    EEG-to-image reconstruction stops leaning on alignment alone: an image teacher supervises denoising, and the method beats its same-backbone baseline on all seven THINGS-EEG metrics

    EEG-to-image reconstruction has long aligned brain signals to pretrained visual embeddings and used them to condition a generator. Retrieval experiments here show that better alignment does not mean better reconstruction: retrieval improved by up to 77, while high-level reconstruction gains stayed modest and low-level fidelity showed no consistent improvement. An image teacher conditioned on DINOv2 features is trained first to express fixed encoder features as intermediate denoising representations, and an EEG-conditioned student generator aligns to these dynamic targets during denoising, with the teacher discarded at inference. On ten THINGS-EEG subjects, the method beats a SiT-XL/2 baseline trained on the same Brain-HiVE embeddings without representation alignment on all seven metrics.

  2. arXiv

    Iris-Z folds depth into image generation: one set of weights reaches mean AbsRel 2.92 on Marigold's benchmarks, below MoGe-3's 3.12

    Iris-Z is a general image generator that encodes image and depth into one token sequence and denoises them together, so a single set of weights generates images, estimates depth from photographs and paints images onto a given depth. Monocular geometry accuracy had belonged to task-specific ViTs such as MoGe-3, and image generators repurposed as depth estimators, the Marigold family, stayed behind it. Iris-Z-big reaches a mean AbsRel of 2.92 on Marigold's benchmark sets against 3.12 for MoGe-3 while keeping its generative abilities, at about 15 seconds per image versus about 0.41 seconds for MoGe-3.

  3. arXiv

    Audio diffusion transformer takes fMRI directly: skipping intermediate embeddings lifts both acoustic and semantic reconstruction scores

    MAAS replaces the text-embedding conditioning of a pretrained audio diffusion transformer with region-of-interest fMRI responses, removing the decoder that previously predicted an external stimulus representation such as CLAP or YamNet. On a 7T fMRI dataset it improves acoustic and semantic metrics, including CLAP similarity, pairwise identification accuracy and Mel-SSIM, over baselines such as two-stage fMRI-to-latent conditioning. The framework also uses a fidelity metric grounded in a computational model of auditory cortex to quantify how individual auditory regions contribute to the reconstructed audio.

  4. arXiv

    Diffusion caching stops trusting raw probe drift: dividing by trajectory movement cuts LPIPS at the same compute

    A training-free diffusion caching rule spends its exact denoiser evaluations where they matter by dividing shallow-probe output change by input displacement, lowering median LPIPS at matched exact-compute budgets on FLUX.1-dev, Wan2.1 and Open-Sora 1.2.

  5. arXiv

    DiT features lag U-Net because noise rides the main path: long skip connections bypass it and lift mid-layer quality

    A controlled from-scratch comparison shows that adding long skip connections to a DiT improves its intermediate features across semantic correspondence, segmentation and depth, and the gain survives even when generation quality is matched by FID.

  6. arXiv

    Text-to-image attention stops hugging the first token: diffusion models pile it on modality boundaries and image corners

    Across the text-to-image Transformers FLUX.1, Z-Image and Infinity-2B, attention sinks land in architecture-specific places: the autoregressive model concentrates on its first global token, while the bidirectional diffusion models concentrate near modality boundaries, inside text regions and at image corners.

  7. arXiv

    Removing residual connections in the DDT condition encoder and fusing multi-depth features lets DDT-RFE improve four visual understanding tasks and ImageNet FID with fewer encoder blocks

    The authors remove the residual connections around Self-Attention and MLP in the condition encoder of the Decoupled Diffusion Transformer (DDT), stabilize training with zero-initialized output gates and a mixed orthogonal attention initialization, and form the encoder output by aggregating the input patch embedding with the outputs of all encoder blocks; under ImageNet-1k training at the stated resolution, patch size 2, 80 epochs and batch size 1024, DDT-RFE improves overall over DDT on image classification, semantic segmentation, object discovery and semantic correspondence while using fewer encoder blocks (12 vs. 20 at L, 14 vs. 22 at XL), and lowers ImageNet generation FID from 8.98 to 7.68 at L and from 7.27 to 6.81 at XL.

  8. arXiv

    FADE erases 16 concepts at once from a single text-to-video model, cutting residual object accuracy to 4.9% from the strongest baseline's 15.5% while staying within 0.9% of the unedited VBench average

    The work proposes FADE, a multi-concept unlearning framework for text-to-video diffusion transformers that first applies a joint closed-form key/value edit suppressing all target concepts, then trains per-concept frame-aware low-rank adapters gated by the frame index and denoising timestep to remove residual per-frame leakage, and combines adapters by a similarity-based soft router; erasing 16 concepts (objects, artistic styles, and nudity) from a single Wan2.1-T2V-1.3B backbone, FADE reduces residual object-benchmark accuracy to 4.9% against 15.5% for the strongest of eight baselines, keeps the VBench average within 0.9% of the unedited model, and the ranking holds under a VLM judge and a blinded human study.

  9. arXiv

    PerF makes pixel-space diffusion Transformers specialize by feature: FID on ImageNet 256 drops from 1.86 to 1.63 with about 3.6% more parameters

    The work introduces heterogeneous refinement, giving different feature groups distinct refinement budgets across depth in pixel-space diffusion Transformers, which spontaneously yields persistent features that encode global structure and active features that encode high-frequency detail; building on this, Persistence Forcing (PerF) and Persistence Guidance (PG) reduce FID on ImageNet 256 from 3.66, 2.36, and 1.86 to 2.81, 1.91, and 1.63 for JiT-B/L/H, and on ImageNet 512 from 1.94 to 1.76 for JiT-H/32.

  10. arXiv

    Adobe Research and KAIST introduce FlowTool: treating retouching tool parameters as conditional generation, cutting L1 by up to 28.8% and L2 by up to 47.4% on reference metrics with at least 50x lower latency

    Researchers at Adobe Research and KAIST introduce FlowTool, which uses conditional rectified flow to directly model the distribution of high-quality retouching tool parameters conditioned on the input image and user instruction, combining a vision-language model backbone with a Diffusion Transformer parameter generator and a tool-presence head instead of autoregressive MLLM reasoning and token-by-token numeric generation; across MMArt-Bench, FlowTool-Eval, ArtEdit-Bench, and MIT-Adobe5K it achieves stronger reference-based performance than specialized MLLM editing agents and proprietary MLLMs, remains competitive with proprietary models under reference-free evaluation, and reduces inference latency by at least 50x while requiring nearly 2x less memory.