Model merging drops peer-task data: with ImageNet as a universal proxy, merged CLIP models approach the oracle that sees other tasks
Model merging builds a multi-task model from independently fine-tuned experts, but earlier pre-merging regularizers needed data from the other tasks to be merged, breaking the independence assumption. This work characterizes a universal proxy distribution against target tasks through spectral coverage and spectral leakage, and derives a unified bound in which smaller spectral parameters mean lower parameter interference. On CLIP vision benchmarks, three instantiations regularized with an ImageNet proxy beat existing pre-merging baselines and approach the oracle that uses peer-task data; on language models, a WikiText-103 proxy unrelated to TRACE also improves merged performance.