Skip to main content
Back to timeline
arXivSource publication:

Pixel-space diffusion stops tiling every patch: HiT groups related features and hits 1.48 FID on ImageNet at 600 epochs

Synopsis

Pixel-space diffusion trains directly on pixels without a pretrained visual encoder, but has trailed two-stage latent models in quality and training efficiency. HiT progressively merges related patch features into group tokens inside the backbone, lets deeper Transformer blocks process the compact sequence, and uses soft weights to restore features to their original positions, with a compression budget controlling the average ratio across the batch. On ImageNet 256×256, HiT-XL/16 reaches 1.48 FID after 600 epochs, better than JiT-H/16's 1.86 at comparable parameter count, and matches JiT's quality with about 40% of its cumulative training FLOPs.

Interpretation

A pixel-space generator can learn a visual hierarchy inside generative training instead of relying on a separately pretrained encoder to organize image information. Earlier pixel-space Transformers kept one token per patch throughout the backbone, and the fixed grid assigned the same token count to equally sized regions without encouraging content-dependent summarization. On ImageNet 256×256, HiT-XL/16 reaches 1.48 FID after 600 epochs, below JiT-H/16's 1.86 at comparable parameter count and below PixelGen-XL/16's 1.83 and PixelDiT-XL/16's 1.61, both of which use external visual supervision.

The model merges patches by feature affinity so deeper Transformer blocks compute only on a smaller set of group tokens, then spreads the results back to original positions. Grouping computes pairwise token affinities from learned projections and, starting from individual tokens, merges groups whose mean affinity clears a threshold, so members of one group can be spatially far apart. Grouping forms each compact token as a soft weighted average over all input tokens and reuses the same weights for readback; in an 18-block HiT-B/16 ablation, restricting aggregation to hard-group members raised FID from 6.61 to 10.16–10.32, and hard lookup readback raised it to 7.97.

Perspective

The demonstrated scope is ImageNet 256×256 class-conditional generation and 256×256 text-to-image generation, the latter pretrained on CC12M and then fine-tuned on a three-source mixture; reported forward FLOPs use target group counts per stage, while actual cost varies with the group counts produced for each image and timestep.

The authors attribute part of the gap to latent models to keeping one token per patch throughout the backbone, which is a hypothesis rather than a confirmed cause; grouping thresholds are calibrated sequentially on fixed noisy examples and then frozen, and group guidance needs separately calibrated inference thresholds.