Skip to main content
Back to timeline
arXivSource publication:

Multimodal aligners compress a whole touch recording into one point, so two slides on the same texture blur together; CounterPoint scores frame-locked trajectories instead

Synopsis

Multimodal alignment for physical interaction has long compressed each recording into one pooled vector and matched by cosine similarity. The trouble is that two executions under the same task condition, such as two slides over one texture, share a stationary condition, and pooling averages away the transient cross-modal fluctuations that tell them apart. CounterPoint, a lightweight plug-in, subtracts each modality's temporal mean and compares the demeaned trajectories at matching frames; across four existing aligners it raises audio-haptic R@1 on the Cluster dataset from at most 27.4% to at least 56.0% and event accuracy from below 59% to about 81%.

Source-provided article image: From Points to Trajectories: Rethinking Multimodal Alignment for Physical Events
arXiv

Interpretation

The object being matched shifts from a pooled vector for an entire recording to a demeaned trajectory that keeps temporal order, so two executions under the same condition can be told apart. Earlier aligners compressed each recording into one pooled vector and matched by cosine similarity; on the Cluster haptic-texture dataset, four aligners with different geometries stayed within three percentage points of each other in audio-haptic R@1, all below 59% event accuracy. On Cluster, pooled aligners separated an event from its twin by a median margin of only 0.04 to 0.08 standard deviations, versus 0.32 to 0.63 under CounterPoint; after shuffling 40 ms input blocks, TRIANGLE's audio and haptic embeddings kept 1.000 cosine similarity to the unshuffled originals.

The method subtracts each modality's own temporal mean, keeping only fluctuations around it, then scores cross-modal agreement at time-aligned frames. The temporal mean corresponds to a condition layer set by stationary factors such as texture, speed and force; removing it leaves that layer to the base aligner, and the trajectory branch compares only the dynamics above it. The trajectory branch adds 0.4% parameters and 0.7% peak training memory; on Cluster, CounterPoint reaches 58.2% audio-haptic R@1, above soft-DTW at 37.3%, best shift at 47.6%, and MaxSim near the point baseline.

Gains track the event gap of each modality pair: audio and haptics carry measurable synchronous fluctuations, while a visual scan that records only the path carries almost none. On Cluster the empirical event gap between audio and haptics is 0.146, while the path-only visual control stays near zero; accordingly CounterPoint lifts AVH and HVA by over 30 percentage points but HAV by at most 0.3. On the free-hand HAVdb dataset, where twins share only the condition and not the drive, CounterPoint raises HAV by at least 84.7 percentage points and Avg R@1 from at most 9.9% to at least 87.4%.

Perspective

The demonstrated scope is two texture-sliding datasets: Cluster, where a robot replays probe strokes over 118 textures under 8 directions, 5 speeds and 2 forces, and HAVdb, where 10 subjects slide a bare finger over 10 textures; held-out splits cover textures, categories, speeds, directions and subjects. The intended readers are robotics-perception and multimodal-learning researchers who need event-level retrieval across sensors.

The theory assumes ergodicity, mixing, and conditional independence of events given the drive; on Cluster the path-only visual modality has an event gap near zero, so retrieval gains for that modality pair are limited. Disrupting temporal alignment removes most of the gain: a single 40 ms token shift costs 26 percentage points, and a 160 ms shift or a permutation erases nearly all improvement.