Skip to main content
Search

# Multimodal Pre-training

1 related articles

Subscribe to updates
  1. arXiv

    Vision-language fusion gets one scaling law: under a fixed data budget early fusion protects text and late fusion favors vision, and early fusion gains ground as models grow

    Vision-language models have long chosen by heuristic between late fusion, which adds images to a pretrained language model, and early fusion, which mixes modalities from the first step. This work splits validation loss into a base learning trajectory plus an additive distribution shift driven by competition between modalities, yielding one scaling law that describes both fusion timings and predicts loss across mixture ratios, training steps and model sizes. Under a fixed data budget, early fusion preserves text performance while late fusion favors vision performance, and that shift amplitude decays as models grow.

You’re all caught up