Skip to main content
Back to timeline
arXivSource publication:

Robot multimodal alignment gains a structural check: UniOMA pulls vision, force and touch into one shared geometry

Synopsis

Multimodal contrastive learning for robot perception usually pulls paired samples of the same instance together without constraining how samples relate to one another inside each modality, so embeddings can be statistically aligned yet geometrically inconsistent. UniOMA adds a Gromov–Wasserstein regularizer to InfoNCE-style contrastive losses, builds an intra-modal similarity kernel per modality, and learns a cross-modal GW barycenter as a structural consensus that each modality aligns to. Across five robotic benchmarks, adding the term improves regression, classification and cross-modal retrieval in most paired comparisons and raises rank correlation between embedding structure and an input-kernel reference.

Source-provided article image: UniOMA: Unified Optimal-Transport Multi-Modal Structural Alignment for Robot Perception
arXiv

Interpretation

Multimodal alignment gains a second constraint: beyond bringing samples of the same instance across modalities closer, the similarity relations among samples within each modality must stay consistent in the shared space. Earlier InfoNCE-style contrastive objectives treat alignment as binary positive-negative classification, constraining only instance-level correspondence and not the continuous intra-modal distance geometry, so cross-modal similarity can hold while intra-modal kernels disagree. On VFD and VFP, against a fixed reference given by the mean input-space kernel of 256 held-out test samples, Pairwise reaches rank correlations of 0.318 and 0.392, while adding the GW term raises them to 0.481 and 0.575, with GW distance to the reference falling as well.

Each modality is represented as an intra-modal similarity matrix, a Gromov–Wasserstein barycenter of these matrices serves as the structural consensus, and training pulls each modality's embedding kernel toward that consensus. The Gromov–Wasserstein distance compares relational structure inside two metric spaces rather than raw coordinates, so no cross-modal cost function is needed; anchoring to one consensus avoids pairwise couplings, and the number of couplings grows linearly with modality count. On VFD, VFP, MuJoCo Push, VAT and a new real-world VIP dataset from a 6-DoF manipulator, all methods share backbones and training schedules; in 4-to-7-modality trajectory-consistency classification, UniOMA reaches the highest accuracy at every modality count, with per-epoch time growing approximately linearly.

Perspective

The demonstrated scope is offline alignment evaluation on five robot-perception benchmarks, namely VFD, VFP, MuJoCo Push, VAT and a VIP dataset collected on a 6-DoF manipulator, plus locomotion and peg-insertion tasks in Isaac Lab; structural measures use 256 fixed test samples against a mean input-kernel reference.

The shared structural consensus suits settings where common physical factors induce related relational patterns across modalities; if the modalities capture largely distinct information, alignment may ignore useful modality-specific structure. The learned weights are only an empirical indicator of agreement with the consensus, not a measure of modality importance, and the entropy term discourages but does not prevent concentration on one modality. Better structural alignment does not necessarily yield better downstream performance: gains are small on near-saturated tasks and one comparison shows a decrease.