Skip to main content
Back to timeline
IEEE Transactions on Mobile ComputingSource publication:

Multimodal inference stops waiting for the slowest stream: AdaModalX cuts end-to-end latency by up to 64.75% across four mobile sensing tasks

Synopsis

In distributed mobile sensing, multimodal foundation models usually wait for every modality to arrive in sync before inferring, yet data-heavy streams such as vision often lag audio, so waiting raises latency while inferring early loses accuracy. AdaModalX decouples inference execution from modality synchronization: a lightweight probe distilled from the edge-side model lets reliable samples exit on the device, and the edge decides when to activate the frozen foundation model based on candidate utility, imputing features from partially received delayed inputs when needed. Across four mobile sensing tasks, two inference architectures and four devices, it improves the accuracy-latency trade-off, with up to 26.97% accuracy improvement and up to 64.75% lower end-to-end latency.

Source-provided article image: AdaModalX: Synchronization-Free MFM Inference for Distributed Multimodal Mobile Sensing via Progressive Data Orchestration
arXiv

Interpretation

Multimodal foundation model inference need not wait for all modalities; it can advance progressively as data arrive and answer once the available information is sufficient. Existing multimodal foundation model pipelines depend on synchronized modality completeness, and even methods supporting flexible modality combinations still assume all modalities are present, so responsiveness is bounded by the slowest stream. Across four mobile sensing tasks, two multimodal foundation model inference architectures, four datasets and four mobile devices, compared with nine synchronous and asynchronous baselines, AdaModalX achieves a better accuracy-latency trade-off, with up to 26.97% accuracy improvement and up to 64.75% end-to-end latency reduction.

The system splits the decision of when to infer into two runtime judgments, one on the device and one at the edge, without altering the foundation model architecture. Each device runs a lightweight probe distilled from the edge-side multimodal foundation model and combines the top-1 class similarity with the margin over the second-best to judge sample reliability, so reliable samples exit locally and only uncertain ones keep uploading. At the edge, a candidate built from fully received modalities is scored for utility, and if insufficient, features are imputed from the partially received input of the most informative incomplete modality and re-scored before the frozen model is activated; on the event detection task, enabling both device-side uploading and edge-side on-demand activation reduces average latency by 68.39%, versus 48.71% and 42.43% for each alone.

Perspective

The demonstrated scope is four mobile sensing tasks, two multimodal foundation model inference architectures, four datasets and four mobile devices, plus a smart-home activity recognition case built from an RGB camera, an IR camera and an IMU; the large-scale evaluation mainly emulates asynchronous modality arrivals under controlled network conditions.

The authors note that the large-scale evaluation mainly emulates asynchronous modality arrivals under controlled network conditions, and that future work will extend the method to more architectures, modalities and devices and validate it in larger live deployments.