Vision-language fusion gets one scaling law: under a fixed data budget early fusion protects text and late fusion favors vision, and early fusion gains ground as models grow
Vision-language models have long chosen by heuristic between late fusion, which adds images to a pretrained language model, and early fusion, which mixes modalities from the first step. This work splits validation loss into a base learning trajectory plus an additive distribution shift driven by competition between modalities, yielding one scaling law that describes both fusion timings and predicts loss across mixture ratios, training steps and model sizes. Under a fixed data budget, early fusion preserves text performance while late fusion favors vision performance, and that shift amplitude decays as models grow.