A transformer's token population can be steered on purpose: minimum-effort control carries the whole ensemble to a target distribution
Synopsis
Inside a deep stack of transformer layers, tokens evolve on the unit sphere as interacting particles driven by attention, and in the limit of many tokens that evolution is described by an integro-partial differential equation. Earlier mean-field analyses only characterized how tokens cluster or form metastable multi-cluster states; they did not design a way to move the whole population to a prescribed distribution. This work recasts the problem as a mean-field Schrödinger bridge on the sphere, derives the minimum-effort feedback control, and gives a generalized dynamic Sinkhorn iteration to solve it numerically; in experiments on the circle and the two-sphere, the controlled token distribution is steered to the target configuration.
Interpretation
The evolution of a token population along layer depth can now be actively guided: given an initial and a target token distribution, a controlled trajectory of minimum control effort carries the whole population from the former to the latter. Earlier mean-field token-dynamics studies were analytical, showing eventual formation of a single cluster and metastable partial clustering at intermediate times, without designing a correction that pushes the distribution toward a prescribed configuration. In numerical experiments on the circle and the two-sphere, snapshots of the controlled density coincide with the target distribution, and tokens simulated under the controlled dynamics concentrate near the modes of the target.
The mechanism adds a corrective velocity field tangent to the sphere on top of the attention-induced mean-field velocity, with control cost given by the kinetic energy of that field integrated over time and over the token population. This correction is formulated as a mean-field Schrödinger bridge on the sphere, unlike the standard Schrödinger bridges common in diffusion-based generative models, because it carries both nonlocal attention drift and the spherical constraint. The optimal control and optimal marginal flow are determined by a system of nonlinearly coupled integro-partial differential equations with bilinear boundary conditions, and this structure is used for computation in both the circle and two-sphere settings.
The coupled system is solved with a generalized dynamic Sinkhorn iteration, the iterative proportional fitting algorithm. Standard Schrödinger-bridge equations are linear, so the dynamic Sinkhorn recursion applies directly; the nonlocal attention term makes these equations nonlinear, so the existing recursion cannot be reused as is. The algorithm uses nested fixed-point iterations: the inner loop fixes the marginal flow, solves linear Kolmogorov equations and imposes endpoint densities, while the outer loop updates the marginal flow, with convergence judged in Hilbert's projective metric; experiments perform one unimodal-to-bimodal steering on the circle and one on the two-sphere.
Perspective
The discussion is limited to token-distribution evolution in the mean-field limit with continuous layer depth and normalized self-attention, and the numerical demonstrations are only on the circle and the two-sphere, steering a unimodal distribution to a bimodal one.
Only local convergence is demonstrated numerically, and proving convergence remains ongoing research; realizing the control as sparse interventions at finitely many layers, and extending to finite token counts, discrete layer depth and multi-head attention, are open directions.
