Skip to main content
Back to timeline
arXivSource publication:

Any Channel Count Into One Token Stream: SpheraAudio Beats Spatial-Omni-7B on Spatial Audio Benchmarks

Synopsis

Large audio-language models usually collapse sound to mono to keep general audio understanding stable, leaving the inter-channel phase relations that encode direction and distance unused. SpheraAudio accepts mono, stereo, first-order and higher-order Ambisonics recordings, encodes multichannel complex spectra into spatial tokens, and interleaves them with audio tokens by time before the language model. At 4B parameters it leads on SpheraQA, SO-Bench and TWNM, reaching 60.18% overall on SpheraQA, 18.53 percentage points above Spatial-Omni-7B.

Source-provided article image: Any Channel Count, One Token Stream: Incorporating Spatial Audio Understanding and Reasoning into Large Audio-Language Models
arXiv

Interpretation

Large audio-language models can now use the inter-channel spatial cues in multichannel recordings instead of reading only single-channel content. Earlier general audio-language models largely kept mono processing to preserve general-audio performance and avoid interference from redundant multichannel information; existing spatial extensions either target two-channel binaural audio, require microphone coordinates, or prepend spatial tokens as a block and leave temporal correspondence to be learned implicitly during fine-tuning. On SpheraQA, SpheraAudio reaches 60.18% overall accuracy against 41.65% for Spatial-Omni-7B and 42.73% for Gemini 3.1 Pro given only the omnidirectional channel; on SO-Bench SpheraAudio scores 66.94% versus 64.61%, and on TWNM 71.70% versus 46.20%.

Spatial information and sound content appear as time-aligned pairs in one sequence, so the model can separately query what was heard at a moment and where it came from. The audio encoder AuT extracts content features from a single-channel log-mel spectrogram, the spatial encoder Spatial AuT extracts spatial features from the real and imaginary parts of multichannel short-time Fourier transforms, and two projectors map them into audio and spatial tokens. Each spatial token sits immediately before its audio token, the two share a temporal coordinate and differ in a third coordinate that marks modality, an encoding called SMRoPE; on 16 confirmation recordings, shifting only the spatial token vectors drops accuracy from 75.00% to 25.00%, while shifting both streams together holds 81.25%.

Recordings with different channel counts map onto one four-component spatial representation, so a single model covers mono, stereo, first-order Ambisonics and calibrated microphone arrays. First-order Ambisonics enters directly in ACN order with SN3D normalization, higher-order Ambisonics supplies its zeroth- and first-order components, stereo passes through an invertible mid-side transform, and calibrated arrays use measured directional responses in array-to-Ambisonics encoding. Training runs in three stages: geometric pretraining of Spatial AuT on simulated first-order Ambisonics with temporal annotations and room properties, projector alignment with joint supervised fine-tuning, then GRPO reinforcement learning on final-answer rewards; on nine general-audio test subsets, the adaptation mixing mono and spatial data beats the mono-only adaptation on all nine and the original base on eight.

Perspective

The demonstrated scope is spatial understanding and reasoning at 4B parameters over mono, stereo, first-order Ambisonics and calibrated-array recordings, evaluated on 20,492 SpheraQA questions plus the public SO-Bench and TWNM benchmarks, with separate general-audio and stereo experiments.

Within SpheraQA, motion direction reaches 34.93%, below 45.36% for Spatial-Omni-7B and 49.39% for Gemini 3.1 Pro, and distance comparison reaches 32.04%, also slightly below both comparators. The pretrained Spatial AuT localization head remains weaker on moving speakers, where steered response power does better on that task and on pooled sound-event localization and detection error. Future work points to strengthening motion and source-relation reasoning across more varied recording conditions.