Skip to main content
Back to timeline
arXivSource publication:

Brain-signal zero-shot image retrieval stops relying on the last layer: SIRAZ reaches 92.5% Top-1 within subjects on THINGS-EEG

Synopsis

SIRAZ rebuilds the visual side of neural decoding as multi-depth, biologically inspired representations before aligning them with brain signals. Earlier methods mostly used only the final layer of a pretrained vision encoder or a fixed aggregation, leaving neural representations unlike the brain's hierarchical visual processing. SIRAZ feeds images through a foveated stream and a multi-scale Difference-of-Gaussians stream, takes features from several intermediate layers through a gated residual adapter, encodes brain signals with a direction-aware multi-scale temporal attention encoder, and aligns the two with a bidirectional contrastive loss. On THINGS-EEG in the intra-subject setting it reaches 92.5% Top-1 and 99.27% Top-5; on THINGS-MEG intra-subject it reaches 50.25% and 75%.

Source-provided article image: SIRAZ: Sparse Inter-Layer Residual Aggregation for Zero-Shot Visual Neural Decoding
arXiv

Interpretation

The visual representation for neural decoding no longer comes only from the final layer of a pretrained encoder; features from several intermediate depths are used together and aggregated with learnable weights. Earlier decoding methods relied mainly on the final layer or on predefined feature aggregation, which does not exploit the hierarchical representations inside a pretrained vision model. On THINGS-EEG in the intra-subject setting, the selected-layer strategy reaches the highest average Top-1 accuracy of 92.5%, beating the four-early-layer, four-middle-layer, four-last-layer and final-layer alternatives.

Before entering the vision encoder, each image passes through two transforms that mimic early human vision: one keeps high resolution at the fixation center and blurs progressively toward the periphery, the other uses multi-scale Difference-of-Gaussians to emphasize luminance contrast and coarse spatial structure. Earlier biologically inspired transforms were mostly studied one at a time, so the complementary information carried by distinct visual pathways was not used together inside a pretrained vision encoder. On THINGS-EEG in the intra-subject setting, the foveated and Difference-of-Gaussians streams each outperform direct and uniform representations, and combining the two gives the highest retrieval performance, above either stream alone.

The brain-signal encoder splits temporal change into positive and negative directions and learns attention patterns at original, half and quarter temporal resolutions. Standard temporal convolutions capture local patterns but do not treat the direction of temporal change as an independent source of information. On THINGS-EEG in the intra-subject setting, temporal-attention-based encoders outperform conventional alternatives, and the direction-aware multi-scale temporal attention encoder achieves the best overall retrieval performance.

Before entering the shared embedding space, visual features pass through a lightweight gated residual adapter for nonlinear refinement, then align with brain representations through a bidirectional contrastive loss. Intermediate representations of pretrained vision encoders are usually not explicitly adapted to brain signals, leaving a cross-modal discrepancy between neural and visual representations. On THINGS-EEG in the intra-subject setting, this adapter achieves the best performance at 92.5% Top-1 and 99.25% Top-5; in the progressive ablation, adding it yields a 14.50% Top-1 gain, the largest single step among the components.

Perspective

The demonstrated scope is limited to two zero-shot retrieval benchmarks, THINGS-EEG (10 subjects, 63 channels) and THINGS-MEG (4 subjects, 271 channels), scored by Top-1 and Top-5 accuracy under intra-subject and leave-one-subject-out settings.

The gap between intra-subject and cross-subject performance indicates that subject-specific neural variability remains an important factor in the learned representations; cross-subject generalization is named as future work.