Skip to main content
Back to timeline
arXivSource publication:

Visual retrieval aggregation stops being one-shot: BiSA reads one affinity along two axes and leads five of six place-recognition benchmarks after three iterations

Synopsis

Visual retrieval compresses dense patch tokens into one compact descriptor, and aggregation heads have fixed that mapping inside a single forward pass, leaving structure and detail to be coordinated by modules around the head. BiSA turns aggregation into an iterative coupled system: one shared token–slot affinity is normalized along a competition axis that decides token ownership and a selection axis whose slot-specific temperatures set each slot's response breadth, and the reliability cues from both axes filter unreliable tokens before the next round. With a DINOv2-B backbone, BiSA leads R@1 on five of six place-recognition benchmarks and tops all 16 metrics of drone–satellite cross-view retrieval.

Source-provided article image: One Affinity, Two Axes: Bidirectional Slot Attention for Iterative Aggregation across the Structure--Detail Spectrum in Visual Retrieval
arXiv

Interpretation

The aggregation head in visual retrieval becomes an iterative coupled process, where one token–slot affinity is read along two directions that separately handle scene decomposition and response granularity. Earlier aggregation heads fixed the complex token-to-descriptor mapping within a single forward pass, so token assignments determined descriptor components without a feedback loop to refine the mapping; structure and detail were coordinated, if at all, by modules placed around the head, such as segmentation masks, structural distillation or local-region mining. On the Baidu Mall, Nordland∗ and MSLS-val place-recognition benchmarks, full BiSA raises R@1 over the no-competition baseline by 5.9, 6.0 and 4.9 points respectively, indicating that the two axes and the iterative feedback together produce that margin.

The mechanism keeps a set of learnable slots as an evolving aggregation state, rewriting the slots each round from the competition-and-selection consensus and reweighting tokens by reliability. The competition axis normalizes over slots under one temperature shared by all slots and decides which slot owns each token; the selection axis normalizes over tokens with a temperature per slot, so a small temperature concentrates on sparse local cues and a large one covers broad layouts. Ablation shows the selection axis, which adds only 20 learnable scalars, contributes 3.2, 2.9 and 2.2 R@1 points on Baidu Mall, Nordland∗ and MSLS-val, averaging 2.8 points, the largest single gain in the ablation table.

Perspective

The demonstrated scope is three visual retrieval tasks: place recognition evaluated on Baidu Mall, SPED, Nordland∗, MSLS-val, Pitts30k and SVOX-Overcast; cross-view geo-localization trained on University-1652 and transferred zero-shot to the 80-scene SUES-200 test split; and instance-level retrieval trained on SfM-120k and evaluated on Revisited Oxford and Revisited Paris under the Medium and Hard protocols. All experiments use DINOv2-B with only the final three Transformer blocks fine-tuned, train on a single NVIDIA RTX A6000, and default to 20 slots, 3 iterations and a 5120-dimensional descriptor.

Increasing slots from 8 to 20 steadily improves performance while 32 yields no further gain, and increasing iterations from 1 to 3 gives consistent gains while a fourth iteration provides no meaningful improvement, so the default is 20 slots and 3 iterations. Nordland∗ uses the VPR-Bench split of 2,760 summer queries against the full 27,592-image winter reference sequence, and results on that split are not comparable with the 27,592-query within-frame split.