Skip to main content
Back to timeline
arXivSource publication:

Vision-language fusion gets one scaling law: under a fixed data budget early fusion protects text and late fusion favors vision, and early fusion gains ground as models grow

Synopsis

Vision-language models have long chosen by heuristic between late fusion, which adds images to a pretrained language model, and early fusion, which mixes modalities from the first step. This work splits validation loss into a base learning trajectory plus an additive distribution shift driven by competition between modalities, yielding one scaling law that describes both fusion timings and predicts loss across mixture ratios, training steps and model sizes. Under a fixed data budget, early fusion preserves text performance while late fusion favors vision performance, and that shift amplitude decays as models grow.

Source-provided article image: Scaling Laws for Vision-Language Fusion
arXiv

Interpretation

The two fusion timings for vision-language pretraining can now be described by a single scaling law, with validation loss predictable as mixture ratio, training steps and model size change. Early and late fusion had been modeled separately, with no unified formulation, so choosing between them was largely heuristic and could not be settled systematically under a given compute or data budget. Fitted separately on interleaved image-text data and image-caption data, the interleaved fusion fit gives root-mean-square errors of 0.0022 and 0.0018, caption fusion gives 0.0099 and 0.0017, and held-out validation gives 0.0031, 0.0015, 0.0084 and 0.0036.

Validation loss is written as the sum of a base learning trajectory, in which different data types advance at different rates, and an additive distribution shift caused by competition between modalities. Interleaved data resemble the text-only web corpus used in pretraining and produce a milder shift, whereas caption data consist of short alt-texts and produce a pronounced shift, with text pretraining buying little caption-loss improvement. The fitted cross-modal exchange rate is close to 1 for interleaved fusion, meaning text and interleaved tokens are near-substitutable, while for caption fusion it is small, so learning advances mainly on caption tokens.

With fixed text and caption token budgets, the fusion-timing trade-off is set solely by the shift term and the optimum sits at the edges: late fusion lowers caption loss, early fusion lowers text loss. The shift amplitude decays as a power law of model size, so the late-fusion vision advantage shrinks for larger models and early fusion becomes more attractive. Sweeping fusion timing for 300M and 1B models under a 56B-token total budget, the caption-loss gap between early and late fusion narrows from 0.159 to 0.095, while the text-loss gap moves only from 2.904 to 2.828.

Perspective

The conclusions rest on vision-language models built from a Gemma-3-style language model with a SigLIP2 encoder; the non-model-size studies mainly use a 300M model, and the fusion-timing experiments cover 300M and 1B scales under a 56B-token total budget. The authors explicitly frame the formulation as a framework for practitioners to characterize and fit on their own training rather than as rigid constants.

The fitted parameters depend on design choices such as data and tokenization and may not transfer. The formulation omits extreme caption ratios that are not viable in practice because they cause language collapse, treating that point as the boundary where the scaling law breaks down. The authors list scaling to larger models and broader modalities as future work.