Skip to main content
Back to timeline
arXivSource publication:

Multimodal continual tuning stops stacking experts: CoPre-MoE merges overlapping ones into a fixed pool, holding UCIT forgetting at -0.10

Synopsis

When a multimodal large model learns tasks in sequence, adding a fresh set of experts per task avoids overwriting old parameters but makes expert count grow linearly and lets different tasks relearn overlapping functions. CoPre-MoE consolidates functionally overlapping experts into a fixed-capacity generic pool, factorizes the remaining task-specific experts into shared projection bases plus compact cores, and constrains routing with frozen historical retrieval keys and an orthogonality penalty. On the UCIT and CoIN continual instruction-tuning benchmarks it reaches the highest average score, with backward transfer of -0.10 on UCIT and 68.3% fewer adaptation parameters than MoELoRA.

Source-provided article image: CoPre-MoE: Consolidating and Preserving MoE for Multimodal Continual Instruction Tuning
arXiv

Interpretation

A multimodal model learning a stream of visual instruction tasks no longer has to keep a full separate expert set for every task; experts whose functions overlap across tasks can be merged into a generic pool of fixed capacity. Earlier MoE-style continual tuning avoided overwriting historical parameters by allocating separate experts per new task and freezing old ones, at the cost of total parameters growing linearly with task count while experts across tasks repeatedly learned overlapping functions. Across six vision-language tasks in UCIT, replacing a current task's experts with the most functionally similar experts from other tasks left accuracy nearly unchanged on VizWiz, Flickr30k and ArxivQA, while ImageNet-R, IconQA and CLEVR retained roughly 90% of original performance, showing cross-task redundancy is real but naive replacement also loses what native experts learned.

The remaining task-specific experts are compressed into shared projection bases plus compact task cores, and input-dependent routing calls both pools at inference without test-time task labels. The generic pool merges operators with activation weighting, estimating functional distortion between a candidate expert and a slot from diagonal second-moment statistics accumulated during training, then projects the merged operator back to its original rank by truncated SVD; the private pool uses a Tucker-style decomposition that keeps each component's identity. On UCIT the full model scores 73.93 average with backward transfer -0.10; removing dynamic role allocation drops it to 72.42 average and -0.52 backward transfer, removing activation-weighted merging gives 72.77 average, and removing multi-key retention gives 72.95 average and -0.45 backward transfer.

Routing preserves historical capability by pairing each expert with a learnable retrieval key; historical keys stay frozen during later tasks and new keys are separated from old ones by an orthogonality penalty. Collapsing each generic slot's multiple source keys into a single mean vector degrades retrieval of historical experts and lowers both average score and backward transfer. On the eight-task CoIN sequence the full model scores 70.00 average with backward transfer 0.02, 3.18 points above the strongest baseline SAME at 66.82; on UCIT it is 6.81 points above SAME's 67.12.

Perspective

The demonstrated scope is continual instruction tuning of a LLaVA-1.5-7B backbone over the six-task UCIT and eight-task CoIN sequences, evaluated without test-time task labels; private rank and the orthogonality penalty weight were chosen on a held-out UCIT validation split.

A too-small private rank loses accuracy from insufficient representational capacity, and gains fluctuate or turn marginal beyond 16; an overly strong orthogonality penalty over-constrains key-space exploration and slightly degrades retention; changing task order moves the final average between 71.83 and 73.93, with positive backward transfer accompanying slightly lower final means; generated responses still inherit the visual hallucination and safety-alignment behavior of the underlying multimodal backbone.