Skip to main content
Back to timeline
arXivSource publication:

LLM fusion listens to the words: MR-Bench isolates the language shortcut, and CoVeR cuts text-induced errors on conflicting clips by 2.1 points

Synopsis

When a large language model serves as the fusion engine for multimodal sentiment, it follows the words even when audio and video carry the opposite evidence, although the same model can read the non-verbal answer on its own. The study builds MR-Bench: MR-Bench-A types every test sample of four sentiment benchmarks by unimodal judgments, and MR-Bench-H holds 900 human-validated clips (620 language-shortcut, 280 non-verbal-shortcut). CoVeR trains counterfactual views as unimodal judges and withholds fused supervision where they disagree; on MR-Bench-H it reaches 61.7% language-shortcut and 93.6% non-verbal-shortcut accuracy in accuracy mode, against 58.8% and 93.1% for the LLM-Fusion backbone.

Source-provided article image: Words Speak Louder? Exploring the Language Shortcut of LLM-Driven Incomplete Multimodal Learning
arXiv

Interpretation

A multimodal sentiment model that uses a large language model as its fusion engine follows the words when text and audio or visual evidence disagree, even though it answers correctly from audio or video alone. Earlier incomplete-multimodal methods reconstruct missing modalities or distil from a complete-modality teacher and still need complete training observations; such fusion models did well on samples where modalities agree, while their behaviour on conflicting samples had not been measured separately. On MR-Bench-A, LLM-Fusion with all modalities observed is right on 90–98% of consensus cases but only on 17–59% of conflict cases, while the same model solves 63–87% of the conflict cases once the text is removed.

The correction sits in the training objective rather than in a gate added at inference. Letting disagreeing views override the fused decision at test time trades consensus accuracy and correlation for conflict accuracy, and a fixed-threshold discrepancy gate barely changes the shortcut error rate. CoVeR expands each sample into its full and single-modality views trained jointly, supervises the views with unimodal labels, and withholds the fused task gradient on samples where the views disagree; on CH-SIMS, unimodal supervision alone adds 2.6 correlation points and 2.6 conflict-accuracy points, and adding the disagreement gate adds a further 2.9 correlation points and 6.4 conflict points.

Perspective

The demonstrated scope is multimodal sentiment and emotion recognition on CMU-MOSI, CMU-MOSEI, CH-SIMS, CH-SIMSv2 and IEMOCAP, with availability patterns sampled dynamically during training and every availability pattern evaluated at test time; all 900 MR-Bench-H clips are scored, and each fusion model is trained on the source corpus of its clips. The authors state that the MR-Bench diagnostic tools are intended for auditing models and that they do not recommend deploying affect recognition in high-stakes decisions about individuals.

The authors list as limitations that the CMU-MOSI and CMU-MOSEI conflict sets of MR-Bench-A are model-judged and small, that MR-Bench-H holds only model-detectable shortcuts, and that CoVeR needs unimodal labels and a known dominant modality. Robust mode raises language-shortcut accuracy to 63.0% while lowering non-verbal-shortcut accuracy to 84.8%, so a trade-off between the two shortcut types remains.