InfMasking masks features before fusion so no single modality dominates, lifting the synergy task 5.6 points over CoMM
Synopsis
Multimodal fusion often leans on a dominant modality or on shared information, leaving unused the synergy that exists only when modalities are observed together. InfMasking randomly masks feature tokens of each modality before fusion and requires the masked fused representation to stay consistent with an unmasked view or a class proxy, turning the expectation over infinitely many masks into variational bounds under a Gaussian model. On the controlled Trifeature benchmark it reaches 77.0% on the synergy task, 5.6 points above CoMM, and 90.6% on uniqueness.
Interpretation
Before fusion, each modality's feature tokens are randomly masked, and the masked fused representation must stay consistent with a reference target, so the fusion module has to remain usable across many partial combinations of cross-modal evidence. Earlier cross-modal contrastive learning mainly aligns information shared across views, and in supervised training a dominant modality can take over gradient updates, so synergy that appears only when modalities are combined goes unused. On the controlled Trifeature benchmark, InfMasking reaches 77.0% on the synergy task, 5.6 points above CoMM's 71.4%, and 90.6% on uniqueness, 3.8 points above CoMM, while redundancy stays near perfect.
The masked fused representation is treated as a Gaussian random variable, and training optimizes closed-form variational bounds on that distribution instead of enumerating every mask combination. The expectation over infinitely many masks cannot be computed exactly, and a finite-sample Monte-Carlo average is only a stochastic estimate. The unsupervised lower bound matches negative InfoNCE alignment against the unmasked fused view, and the supervised upper bound matches cross-entropy against class proxies; on four bimodal MultiBench classification datasets, linear-probing average accuracy rises from CoMM's 64.45% to 67.05%.
In the supervised setting, every masked partial modality combination must align with the class proxy of the task label, while auxiliary cross-entropy losses keep each unimodal branch predictive. Balancing modality contributions alone does not require partial multimodal evidence itself to support the task decision. On MOSI, MOSEI, MIMIC, MAMI and MUTE, gains over the Vanilla MT backbone are 3.0, 1.6, 1.3, 5.9 and 5.9 percentage points, raising average accuracy from 68.1% to 71.7%.
Perspective
The demonstrated scope covers the controlled Trifeature synthetic benchmark, bimodal and trimodal tasks on MultiBench, and multi-label movie genre classification on MM-IMDb; supervised experiments span MOSI, MOSEI, MIMIC, MAMI and MUTE, and inference uses only the fused representation of all modalities.
Bound tightness holds under the Gaussian model, and the sign of the gap under more general distributions needs a directional sub-Gaussian condition along class-proxy directions; non-negative Monte-Carlo gaps do not establish that condition. Training on multiple masked views costs extra computation and memory, performance and resource use beyond three modalities are untested, and the conditions under which masking recovers synergistic interactions remain to be characterized.
