Skip to main content
Back to timeline
arXivSource publication:

Three-modality contrastive learning swaps its negatives: CHORD gains 21.2 points over the strongest baseline on samples that need all three modalities

Synopsis

Higher-order objectives for three-modality contrastive learning are meant to capture information that appears only when all three modalities are seen together, but ordinary negatives are built by swapping modalities, so a single pairwise check rejects them and the higher-order term receives almost no gradient. CHORD instead draws decoy negatives from a calibrated pairwise model, negatives that model itself finds plausible, and trains a residual to separate them. On M3-Mix, where one in ten training samples needs all three modalities, CHORD improves accuracy on those samples by 21.2 percentage points over the strongest of eight baselines.

Source-provided article image: Learning What Pairs Miss for Higher-Order Multimodal Alignment
arXiv

Interpretation

In three-modality contrastive learning, higher-order objectives that score the whole tuple with multilinear products, volumes or fusion heads still behave like bags of pairwise models; the problem lies in the negatives, not in the expressivity of the scoring function. Earlier practice built negatives by swapping in modalities from other samples, so a negative breaks at least two of the three pairs and one pairwise check rejects it; when most samples can be solved by pairs, the gradient reaching the higher-order term approaches zero. On M3-Mix, where one in ten training samples needs all three modalities, CHORD improves accuracy on those samples by 21.2 percentage points over the strongest of eight baselines; on pure-synergy XOR and Symile-M3, product critics already score near the top and CHORD matches them.

CHORD splits the higher-order part into a calibrated pairwise bridge plus a residual, and trains the residual only against decoy negatives that the bridge finds plausible. The bridge sums the three pairwise similarities with constant weights fitted on the two-to-one task, which removes double counting when modalities repeat each other; decoys are drawn from a softmax over a memory bank at the cost of one inner product per query. CHORD gives the best two-to-one retrieval on all four MultiBench datasets and its one-to-one retrieval is never below that of Tri-CLIP; with frozen encoders, one-to-one retrieval equals that of the bridge.

Perspective

The demonstrated scope is three-modality contrastive alignment and retrieval, covering the controlled tasks XOR, XOR-Mix, Synthetic-XNOR, Symile-M3 and M3-Mix, plus MultiBench affective data, audio-visual zero-shot classification and video retrieval across 15 benchmarks; the residual by design covers interactions only up to order three, and a pure four-way task needs an added four-way product term.

The residual targets the log-ratio between the data and the pairwise bridge, which differs from the true interaction structure by the bridge's misfit, so on real data BPI is a diagnostic rather than a strict bound; the size of the gain follows how much information the pairwise model misses, and under the shift to YouCook2 the residual adds almost nothing.