Skip to main content
Back to timeline
arXivSource publication:

Closed-form couplings can't replace a pretrained teacher: in high dimensions the endpoint is mostly the initial noise, and students land near random pairing

Synopsis

Few-step generation usually relies on a pretrained teacher to produce noise-data pairs for rectified-flow training. For a finite dataset and a Gaussian probability path, the empirical probability-flow field is available in closed form, so one could use its couplings as the teacher and skip pretraining. The study shows this fails at practical dimensions: the closed-form flow returns the injected noise almost unchanged at the endpoint, adding only a faint data-dependent drift that is nearly independent of the noise, so training on these pairs performs close to random pairing; the drift's share of endpoint energy falls as dimension grows, and closed-form couplings become more informative only when the dimension is lowered.

Source-provided article image: The Limits of Closed-Form Couplings for Few-Step Generation
arXiv

Interpretation

Couplings built directly from the closed-form probability-flow field of the empirical distribution train few-step students that are clearly worse than students trained on pretrained-teacher couplings, and only marginally better than students trained on random pairings. Rectified flows previously depended on a pretrained teacher, integrating noise through the teacher to a data point to obtain pairs, at the cost of training that teacher; the closed-form field needs no pretraining and is evaluated analytically. On CIFAR-10, CelebA and ImageNet-64, the three student groups use the same number of couplings, the same architecture and the same optimisation settings, differing in coupling source, and on CIFAR-10 and CelebA the teacher-coupled students train for fewer steps because their FID stops improving earlier; at a single sampling step the closed-form students have clearly higher FID than teacher-coupled students, a gap that appears consistently across the three datasets.

The closed-form endpoint decomposes exactly into the noise injected at initialisation plus a data-dependent drift, and that drift is small in the experiments. Earlier empirical observations showed only that closed-form couplings perform poorly, without saying how much data information survives in the endpoint. On all three datasets the drift carries well below a few percent of the endpoint's squared norm, and its absolute norm is similar across datasets even though CelebA and ImageNet-64 have four times the dimension of CIFAR-10.

The drift's share of endpoint energy falls as data dimension grows, so holding the informative share fixed requires dataset size to grow exponentially with dimension. The failure of closed-form couplings had been treated as an empirical phenomenon, without a quantitative link to dimension and dataset size. Downsampling the same images raises the drift share as dimension falls, and the closed-form student's gap over the random student widens accordingly; at a single step it more than doubles on CIFAR-10 and rises clearly on CelebA and ImageNet.

Perspective

The conclusions rest on three image datasets, CIFAR-10, CelebA and ImageNet-64, using a linear Gaussian probability path, the closed-form empirical flow and rectified-flow students; the dimension-reduction experiments are cross-checked by downsampling, an autoencoder latent space and a network width fixed to equal parameters per dimension.

Closed-form couplings still lag clearly behind pretrained-teacher couplings at low resolution, and the gap does not close; the drift bound exceeds the measured drift by roughly an order of magnitude before the transition point, so it captures scaling rather than an exact value; how a learned teacher selects endpoints within the same region remains an open question.