Skip to main content
Back to timeline
arXivSource publication:

Turn Before the Sound Ends: Streaming Human Reactions to Spatial Audio Cut First-Response Latency from 10.9 Seconds to About 193 Milliseconds

Synopsis

Human reactions to spatial audio no longer wait for the whole clip. Earlier methods synthesized complete motion offline from fully observed acoustic and spatial input, so they could not react causally to an incoming stream. Sonic-AMG encodes acoustic content and body-relative source geometry as separate tokens, uses a causal transformer with a flow head to generate continuous motion latents block by block, and reconstructs motion segments with a causal decoder; it reaches state-of-the-art FID and R-precision on SAM and supports real-time online generation.

Source-provided article image: Sonic-AMG: Streaming Human Motion Generation from Spatial Audio
Figure 1 · arXiv

Interpretation

Human motion can be generated block by block as spatial audio arrives, without first collecting the complete audio window. Earlier methods synthesized whole motion sequences inside predefined temporal windows from fully observed acoustic and spatial inputs, so they could not respond causally to an incoming audio stream. On SAM, Sonic-AMG reaches the best FID of 6.505 and R-precision under GT spatial conditioning, and an FID of 8.066 under the Online protocol, retaining strong distributional fidelity and conditioning alignment.

The model encodes acoustic content and the source's body-relative position as two separate conditioning tokens, then lets a causal transformer learn their interactions. Fusing the two cues into a single conditioning representation can make them act as noise for one another; separating them lets the acoustic branch focus on sound content. Ablations show separate spatial tokens outperform a fused representation, and 80-dimensional log-mel acoustic features outperform MOSPA's 2,272-dimensional audio features.

Training combines short autoregressive self-rollouts with full-sequence teacher-forced supervision, and body-relative source coordinates are recomputed from generated root poses during both training and inference. Teacher-forced training uses ground-truth motion histories and precomputed spatial conditions, while streaming inference relies on the model's own predictions, leaving two mismatches between training and inference. Rehearsal training beats every two-forward schedule under both spatial protocols, and removing full-sequence supervision degrades fidelity and alignment.

Perspective

The main experiments assume the world-space source position is known at inference, supplied by dataset annotations, an upstream sound source localization module, or virtual-scene metadata. Evaluation is on the SAM dataset, and although the model is trained on individual audio-motion clips, it generates coherent motion across successive sound events with varying content and source locations. A No-SSL variant demonstrates deriving spatial conditions directly from binaural audio, while the authors list systematic evaluation of alternative spatial inference methods as future work.

The authors note that generalization to broader acoustic environments and human reactions remains to be assessed, and that evidence for long-horizon coherence and event transitions is primarily qualitative, with quantitative benchmarks for longer audio streams and more diverse event transitions left to future work. The latency estimate combines input waiting with mean steady-state generation time and excludes audio preprocessing and encoding, initialization, rendering and output delivery.