When identical objects share the frame, which one is sounding? Binaural cues lift instance mIoU by 35.19 points
Synopsis
When several same-class objects appear in one view, existing audio-visual segmentation methods typically return sound-related regions or category masks and cannot say which instance is producing the sound. BiSID-AVS turns source-direction information in binaural audio into an image-plane spatial response and conditions mask formation at both the feature and decoder levels; on the multi-candidate subsets of the binaurally extended AVS-Synthetic and Habitat-AVS, instance-level mIoU improves by 35.19 and 41.21 percentage points, with markedly less false activation on non-sounding same-class regions.
Interpretation
The output moves from sound-related regions or category masks to a pixel-level mask of the specific same-class instance that is sounding, so a robot facing several identical objects can lock onto the true source. Earlier audio-visual segmentation methods mainly predicted sound-related regions or category masks without explicit instance identity, while existing audio-visual instance segmentation methods relied largely on motion and cross-frame association to preserve instance identity. On the multi-candidate subsets of the binaurally extended AVS-Synthetic and Habitat-AVS, instance-level mIoU improves by 35.19 and 41.21 percentage points, and false activation on non-sounding same-class regions drops substantially.
The method treats left-right ear differences in binaural audio as spatial evidence that participates directly in candidate selection and mask formation, rather than choosing among masks after they are generated. A straightforward alternative estimates a source direction or spatial response independently, generates visual proposals, and selects the one with the highest spatial consistency, so spatial evidence enters only after mask generation. On Habitat-AVS multi-candidate test samples, swapping the left and right channels or substituting same-scene, same-category audio from another location lowers instance-level mIoU from 0.8952 to 0.1737 and 0.1141, respectively, indicating reliance on interaural structure and audio-visual spatial correspondence.
The pipeline has three steps: audio prompting grounds candidate regions consistent with the sound category, binaural cues are projected into a dense image-plane spatial response map, and that response modulates candidate features and serves as a dense prompt for mask decoding. Earlier audio-prompting methods established category-level audio-visual correspondence but lacked explicit spatial evidence for distinguishing same-class instances. On Habitat-AVS, the configuration enabling both feature modulation and decoder prompting reaches instance-level mIoU 0.8952 and same-class false activation 0.0356, better than configurations keeping only candidate grounding or only the spatial response.
Perspective
The demonstrated scope is the binaurally extended AVS-Synthetic and Habitat-AVS datasets, plus a demonstration on a mobile robot with binaural microphones and an RGB-D camera that identifies the sounding device among three same-class devices; the current formulation assumes one active source per audio segment.
The current formulation assumes one active source per audio segment, and sounding-instance segmentation under overlapping acoustic events is not yet covered; on single-candidate samples spatial conditioning brings limited gains and may slightly reduce mask completeness; under leave-one-scene-out evaluation, single-candidate metrics are also slightly lower than the comparison method.
