Diffusion models stop retraining per distribution: a handful of samples now steers reverse sampling to unseen tasks
Synopsis
Diffusion models previously learned either one fixed target distribution or a family indexed by conditioning variables fixed at training time, so an unseen distribution required retraining or fine-tuning. Treating each target distribution as a task drawn from a meta-distribution, a shared score operator now takes a small context set at inference and outputs that task's score, enabling reverse-diffusion sampling without parameter updates. The task-dependent score admits a Gaussian-kernel conditional-mean representation whose empirical Nadaraya–Watson estimate is represented exactly by the constructed transformer class; on Navier–Stokes, ERA5, QuickDraw and Food-101, low-intrinsic-dimension tasks do well while performance degrades as task complexity grows.
Interpretation
A diffusion model can now adapt at inference to a previously unseen target distribution from only a small set of samples, without retraining, fine-tuning, or any task-specific parameter update. Earlier diffusion models learned the reverse generative dynamics of a single training dataset, and conditional diffusion models covered families indexed by predefined conditioning variables such as class labels or text embeddings, but the form and semantics of that conditioning were prescribed during training and could not infer a new distribution from a small unordered context set. On 2D Navier–Stokes vorticity fields the method beats the conditional diffusion model and the SNO baseline in-distribution and across three out-of-distribution viscosity settings, except for the weighted mean energy log ratio in the out-of-distribution high setting, where the conditional model is lower.
The approach turns task adaptation into an operator-evaluation problem: the context set identifies the target distribution while a shared operator captures structure common across the task family. Rather than first compressing the prompt set into a global representation and predicting the score from it, the method acts directly on the empirical context set and builds score estimates through query-dependent interactions among context samples, the noisy query point, and diffusion time. Training uses amortized denoising score matching across task distributions, and inference uses the learned operator as the score estimator in the reverse-time SDE with Euler–Maruyama sampling; on ERA5 climate data, QuickDraw sketches and Food-101 images it outperforms the conditional model and SNO on most metrics.
The task-dependent score can be written as a Gaussian-kernel conditional mean, and replacing the population conditional mean with its empirical counterpart yields a Nadaraya–Watson estimator built from the context samples. This representation motivates the transformer architecture instead of treating it as an ad hoc design choice. The empirical estimator is represented exactly by a model in the constructed transformer class, linking attention directly to Gaussian-kernel regression; the generalization bound separates the effects of context size, number of training tasks, episodes per task, diffusion samples per episode, and intrinsic and ambient dimensions.
Perspective
The demonstrated scope covers four data sources: 2D Navier–Stokes vorticity fields, ERA5 climate data, QuickDraw sketches, and Food-101 images. Navier–Stokes is evaluated both in-distribution and out-of-distribution, while the remaining datasets are evaluated for in-distribution generation only. The method targets researchers and engineers who need to adapt at inference to a family of related distributions, especially scientific-computing settings where task distributions concentrate on a low-dimensional manifold. Code is available, along with a Lean 4 formalization of Theorem 1.
The finite-context error decays with context size at an exponent governed by the intrinsic dimension, while the ambient dimension enters only through polynomial and logarithmic factors. The task-level persistent term is unaffected by the number of episodes per task and diffusion samples per episode, so more within-task sampling alone cannot compensate for limited meta-training task coverage. On QuickDraw and Food-101, performance degrades for all methods as the number of categorical tasks grows, reflecting a resource-limited regime with fixed per-task data, prompt size, and shared model capacity rather than an inherent limitation of the method to categorical task distributions; whether additional data and model capacity can mitigate this degradation is left for future work.
