Skip to main content
Search

# Mean-field Token Steering

1 related articles

Subscribe to updates
  1. arXiv

    A transformer's token population can be steered on purpose: minimum-effort control carries the whole ensemble to a target distribution

    Inside a deep stack of transformer layers, tokens evolve on the unit sphere as interacting particles driven by attention, and in the limit of many tokens that evolution is described by an integro-partial differential equation. Earlier mean-field analyses only characterized how tokens cluster or form metastable multi-cluster states; they did not design a way to move the whole population to a prescribed distribution. This work recasts the problem as a mean-field Schrödinger bridge on the sphere, derives the minimum-effort feedback control, and gives a generalized dynamic Sinkhorn iteration to solve it numerically; in experiments on the circle and the two-sphere, the controlled token distribution is steered to the target configuration.

You’re all caught up