Skip to main content
Back to timeline
arXivSource publication:

When a frozen vision model switches tasks, how shared prompts travel costs more than where they land: Bézier paths cut shallow feature transport action from 4.50 to 3.20

Synopsis

Frozen vision foundation models adapt to new tasks through a few prompt tokens shared across all images, and updating those prompts moves the image-feature distribution from one endpoint to another. Prior theory asked which endpoints prompts can reach; this work asks whether they can travel between them along the shortest path and quantifies the extra transport cost imposed by shared attention. In an exactly solvable attention model the reachable features form a cone, and with three distinct content normalizers same-sign endpoints that are not pure rescalings force a strictly positive extra cost. On ViT-B/16 with VTAB-1k, a quadratic Bézier prompt path trained on feature action lowers held-out transport action from 4.50 to 3.20 in the shallow configuration.

Source-provided article image: Beyond Endpoints: Bézier Paths for Vision Foundation Model Adaptation
arXiv

Interpretation

Under frozen attention, shared prompts cannot always move along the shortest path between feature distributions, and this restriction can be quantified as extra transport cost above the shortest path. Prior prompt theory examined only which endpoints prompts can reach, not whether a reachable shortest path exists between them. In an exactly solvable scalar attention head the reachable features form a cone; unrolling it into a plane gives the exact minimum action for every pair of endpoints and explicit optimal prompts on short arcs.

Once the cone is unrolled into a plane, the shortest path is a straight chord there, and lifting that chord back to shared prompts gives the optimal interpolation. The method is the developed chord: take the straight line between endpoints in cone coordinates, then recover prompt aggregates by one-dimensional quadrature and monotone inversion. Across 24 prespecified cases the developed chord has relative action 1.0000, versus 1.0146 for cone-coordinate-linear interpolation with arclength retiming and 1.4300 for aggregate-linear interpolation.

The number of distinct content normalizers decides whether the extra cost appears: with two, the direction curve follows its shortest spherical arc, and a third bends it away. For same-sign endpoints that are not pure rescalings, two normalizers give no gap, while three give a strictly positive gap at every angle. The threshold persists with more prompts, small content perturbations and aligned multihead outputs, while summing independently controlled heads can trace the straight path and close the gap.

On a frozen ViT, a quadratic Bézier prompt path optimized for feature action moves features less than a cross-entropy-trained path between the same endpoints. Cross-entropy paths suppress time-averaged loss without directly constraining how far features move along the path. On ViT-B/16 with VTAB-1k, held-out transport action falls from 4.50 to 3.20 in the shallow prompt configuration and from 6.30 to 4.20 in the deep one, while cross-entropy paths retain higher minimum accuracy, 68.8 versus 67.4 in the shallow case.

Perspective

The demonstrated scope is an exactly solvable scalar shared-query attention head and a fixed-endpoint path comparison on ViT-B/16 across 19 VTAB-1k tasks with five cyclic seed endpoint pairs per task; the unlabeled-shift experiments use seven VTAB-Natural tasks under Hendrycks Gaussian noise at severity 3 with a frozen linear probe and 128 calibration and 128 evaluation images.

The discretized ViT evaluation does not establish a non-vacuous certified bound between grid nodes; when labeled calibration data are available, cross-entropy-trained paths retain higher minimum accuracy, and the hybrid objective keeps minimum accuracy within 0.1 percentage points of cross-entropy.