Skip to main content
Search

# Multimodal Alignment

2 related articles

Subscribe to updates
  1. arXiv

    Robot multimodal alignment gains a structural check: UniOMA pulls vision, force and touch into one shared geometry

    Multimodal contrastive learning for robot perception usually pulls paired samples of the same instance together without constraining how samples relate to one another inside each modality, so embeddings can be statistically aligned yet geometrically inconsistent. UniOMA adds a Gromov–Wasserstein regularizer to InfoNCE-style contrastive losses, builds an intra-modal similarity kernel per modality, and learns a cross-modal GW barycenter as a structural consensus that each modality aligns to. Across five robotic benchmarks, adding the term improves regression, classification and cross-modal retrieval in most paired comparisons and raises rank correlation between embedding structure and an input-kernel reference.

  2. Wrist pulse waveforms carry meal information: aligning them with food text wins 0.69 of meals and lifts calorie-intake AUC by 11%

    An Apple team aligned wrist photoplethysmography (PPG, a green-light measure of microvascular blood-volume change) with meal descriptions to train the Nutrition PPG Language Model (NPLM) and test whether passively collected pulse waveforms carry dietary information. On 1,122,834 meal descriptions from 19,340 participants, near-meal PPG raised the likelihood of the correct meal text to a win rate of 0.69, against 0.60 for a control that kept the same participant and time of day but broke the meal pairing, and 0.50 for across-subject shuffling. Without retraining, the signal raised postprandial satiety prediction AUC from 0.64 to 0.71 in an independent cohort, which the authors frame as augmenting text-based logging rather than replacing it.

You’re all caught up