Model merging drops peer-task data: with ImageNet as a universal proxy, merged CLIP models approach the oracle that sees other tasks
Synopsis
Model merging builds a multi-task model from independently fine-tuned experts, but earlier pre-merging regularizers needed data from the other tasks to be merged, breaking the independence assumption. This work characterizes a universal proxy distribution against target tasks through spectral coverage and spectral leakage, and derives a unified bound in which smaller spectral parameters mean lower parameter interference. On CLIP vision benchmarks, three instantiations regularized with an ImageNet proxy beat existing pre-merging baselines and approach the oracle that uses peer-task data; on language models, a WikiText-103 proxy unrelated to TRACE also improves merged performance.
Interpretation
Pre-merging regularization for model merging can stop depending on the data of the other tasks being merged and instead constrain fine-tuning with a single task-agnostic proxy distribution. Earlier pre-merging methods, including Jacobian penalties on peer-task samples, KFAC curvature estimated from peer-task statistics, and LoRA subspaces constrained by other tasks' examples, all required explicit access to peer-task data, which conflicts with the independence that model merging assumes. Across eight vision tasks on CLIP ViT-B/32, the task-averaged spectral parameter correlates strongly and negatively with merged accuracy, with a negative Pearson correlation that stays negative after excluding the two extreme proxies, Oracle Union and fixed-noise.
The relation between a proxy and a target task reduces to two spectral quantities: spectral coverage, how well the proxy's gradient space spans the target's gradient directions, and spectral leakage, the gradient energy in directions the proxy does not cover. Earlier empirical work showed only that KFAC factors estimated once on ImageNet retain most of their gains, without saying when or why a proxy suffices. The resulting unified interference bound states that smaller spectral parameters give lower interference through better spectral alignment; it is derived under NTK linearization and checked on the exactly evaluated (J) instantiation through the measured correlation over eight proxies.
The same proxy regularizer has three implementations: a Jacobian penalty evaluated exactly with Jacobian-vector products, a frozen Fisher information matrix curvature penalty, and a penalty that directly constrains the model's output drift on the proxy. The first two recover existing Jacobian regularization and TAK-style KFAC regularization, while the third had not been systematically studied as a regularizer in the model-merging literature. All three are proved to give the same solution under NTK linearization; beyond it, taking (J) as reference, the (F) gap is governed by frozen-curvature error and is proxy-independent, while the (O) gap scales with proxy quality, and this hierarchy is matched in the standard-versus-linearized accuracy comparison over seven proxies.
Under standard non-linear fine-tuning, the output-drift instantiation (O) reaches merged accuracy close to the Jacobian reference at lower per-step cost. The Jacobian instantiation needs a forward JVP plus a reverse VJP sweep each step, raising peak memory and training time over plain fine-tuning, while the Fisher instantiation must keep KFAC factors resident. On ViT-B/32 under standard fine-tuning, (O) takes 139.3 ms per step with a 3371.4 MiB steady peak, only 16% slower and 43% heavier than the unregularized baseline; (J) takes 271.3 ms and 4140.6 MiB, and (F) takes 237.8 ms and 7513.9 MiB plus 18.45 s of preparation.
Perspective
The demonstrated scope is eight vision classification tasks on CLIP ViT-B/32, ViT-B/16 and ViT-L/14, plus full-parameter fine-tuning of Llama-3.2-1B/3B on the seven TRACE tasks and LoRA fine-tuning of Llama-3-8B on the eight GLUE tasks; proxies include ImageNet, COCO, several Stable Diffusion generated sets, fixed noise and WikiText-103. The interference bound is derived under NTK linearization, and the non-linear case gives gap envelopes of each instantiation relative to the Jacobian reference.
The theoretical equivalence holds only under NTK linearization; practical fine-tuning is non-linear, so the three instantiations diverge, and the gap analysis relies on local regularity conditions and bounded residuals. The proxy-independent (F) result requires the same frozen matrix in both modes; if curvature is recomputed along the trajectory, (F) becomes a trajectory regularizer like (O) and the constant bound no longer follows from the present theorem. The language-model experiments do not evaluate (F), because its frozen curvature is prohibitive to keep resident at LLM scale.
