One kernel flow now handles both classification and density estimation: reduced Gauss–Newton converges in 17 iterations, and control points make particle cost linear
Synopsis
Classification and likelihood-based density estimation are cast as a single optimal-control problem sharing one diffeomorphic flow whose velocity lives in a reproducing kernel Hilbert space (RKHS), differing only in the objective. Variable projection removes the terminal model, leaving a reduced gradient from an analytic discrete adjoint, and the kernel parameterization makes the flow's divergence exact at the cost of one kernel matrix–vector product, so objective and gradient are deterministic and reduced Gauss–Newton solvers apply to both tasks. In experiments from dimension 2 to 256, the reduced Gauss–Newton solvers use fewer iterations than any other optimizer tested, and a control-point reduction lowers per-solve particle cost from quadratic to linear at near-lossless accuracy.
Interpretation
Classification and likelihood-based density estimation can now share one invertible smooth transport flow, differing only in the terminal objective while the dynamics, control cost and adjoint computation stay identical. Earlier diffeomorphic learning wrote only supervised learning as a control problem over RKHS flows and optimized flow controls and terminal classifier jointly with first-order methods, while density estimation followed a separate continuous-normalizing-flow route. In experiments from dimension 2 to 256, the reduced Gauss–Newton schemes and joint L-BFGS both reach the same reference level, the former in 17 to 21 iterations and the latter in 58, with the former succeeding on 5/5 seeds against 4/5.
Once the terminal model is eliminated by variable projection, the outer loop iterates only on the controls, and the reduced gradient comes exactly from the flow's discrete adjoint without differentiating through the inner solve. Previous variable-projection uses mostly eliminated the final linear layer of a neural network and obtained the reduced gradient by backpropagating through the network. On the three rings instance, GN-CG and TR-GN cut the relative reduced-gradient norm by about two orders of magnitude in 17 and 21 outer iterations, while first-order variants need roughly 35 to 51 iterations and cut it by only a little over one order.
Placing the velocity field in an RKHS gives the flow's divergence a closed form costing one kernel matrix–vector product, which makes the density-estimation objective deterministic. Existing continuous normalizing flows either use Hutchinson stochastic trace estimators that introduce variance or restrict to potential flows to get exact divergence at linear cost. On random particle configurations the closed-form divergence matches an automatic-differentiation trace to machine precision in float64, with maximum absolute discrepancy below the stated threshold, and its cost grows far more slowly with dimension than the automatic-differentiation trace.
Expanding the velocity over control points instead of all data particles lowers each forward and adjoint solve from quadratic to linear in particle count at near-lossless accuracy. The full model couples all particles through dense kernel evaluations on every solve, so cost grows quadratically with particle count. On five synthetic benchmarks, control points at 3% to 16% of the data recover full-model accuracy exactly on four problems and plateau at 99.2% on three rings against 100%; on VAE-latent MNIST classification, 10% of control points stays within 0.56 percentage points of full-model test accuracy.
Perspective
The demonstrated scope covers synthetic classification benchmarks from dimension 2 to 256, Pendigits and MNIST classification, four two-dimensional synthetic density targets, and MNIST density estimation in a four-dimensional VAE latent space; real high-dimensional data are handled through a latent-space embedding rather than in the ambient feature space.
How best to place and adapt control points for a target accuracy remains open, as does how far the reduction carries on very large, high-dimensional data; kernel methods degrade in high dimensions, and large-scale high-dimensional generative modeling remains open. On generation, the GMM reference mainly lowers round-trip reconstruction error rather than improving likelihood on low-dimensional targets, and only on real data does it also lower test negative log-likelihood.
