Skip to main content
Back to timeline
arXivSource publication:

Flow models stop hand-picking their prediction target: AnisoFlow cuts FID from 1.86 to 1.55 on ImageNet pixels

Synopsis

Flow-matching generative models are trained on a hand-chosen prediction target, such as velocity, noise or clean data, and that choice strongly affects training and sample quality. AnisoFlow makes the noise component of the velocity target learnable: inside an orthogonal basis learned jointly with the network, each direction gets its own nonnegative weight, and a closed-form map recovers the velocity exactly, leaving the architecture, flow-matching loss and sampler unchanged. On ImageNet in pixel space, learning only the prediction target lowers FID from 1.86 for the JiT-H/16 baseline to 1.55 without representation alignment; on latent-space models it performs comparably to the default targets.

Source-provided article image: Learning Anisotropic Prediction Targets for Flow Models
arXiv

Interpretation

What a generative model predicts during training becomes learnable rather than hand-set: AnisoFlow keeps the clean-data component of the velocity target and reweights only the noise component, one nonnegative weight per direction. Velocity prediction previously gave the noise equal weight in every direction, while clean-data prediction removed the noise component entirely; both were fixed choices made in advance. On ImageNet in pixel space, changing only the prediction target lowers FID from 1.86 for JiT-H/16 to 1.55, with no representation alignment and no architectural change.

Weights and basis are trained together while velocity recovery stays exact, so training only makes the target easier for the network to predict. The recovery formula acts coordinate by coordinate in the basis, its positive denominator keeps the map invertible, and the weights directly control how much prediction error is amplified. On JiT-H/16, a frozen PCA basis with learned continuous directional weights reaches FID 2.25, better than AsymFlow's frozen binary mask at 2.34 and clean-data prediction at 2.68 after 160 epochs.

The learned weights grow near-linearly with data variance, matching the direction predicted by the analysis. The analysis splits network error in the PCA basis into a noise-estimation error and a data-estimation error, and predicts that each direction's preferred weight scales approximately with that direction's data variance. Across different initializations and basis-optimization paths, the pixel-space model's learned directional weights follow a near-linear relationship with the data variance along the corresponding learned directions, and the learned basis stays close to the PCA basis.

Perspective

The demonstrated scope is class-conditional ImageNet generation in pixel space (JiT-H/16 and JiT-B/16 backbones), pixel-space text-to-image generation built on JiT-L/16-T2I, and two latent-space models, SiT-XL/2-REPA and RAEv2; every setting keeps the original architecture and training recipe and changes only the prediction target.

The analysis uses a local frozen-error model, does not capture full training dynamics, and cannot determine the constant in the observed spectral scaling; empirical gains are largest in pixel space and smaller in latent space, where the learned targets stay close to existing defaults.