Skip to main content
Back to timeline
arXivSource publication:

EEG-to-image retrieval drops the uniform projection: frequency priors and adaptive temperature lift cross-subject Top-1 from 12.4% to 24.05%

Synopsis

EEG and MEG responses are noisy, variable across trials and subject dependent, so a uniform projection into a pretrained visual semantic space aligns them poorly. This neural-to-image retrieval framework leaves the frozen CLIP visual space untouched and conditions the neural pathway instead: a low-dimensional shared subspace with correlation regularization suppresses redundant noise, amplitude and phase statistics modulate neural representations, and phase stability sets a per-sample contrastive temperature. On 200-way zero-shot retrieval in THINGS-EEG it reaches 63.40%/86.85% Top-1/Top-5 intra-subject and 24.05%/53.45% inter-subject.

Source-provided article image: Stabilizing Neural-to-Image Retrieval with Frequency-Domain Priors and Adaptive Supervision
arXiv

Interpretation

The bottleneck in neural-to-image retrieval is placed on the segment where neural representations enter the shared semantic space, not on the visual space itself; the fix adds a conditionable alignment path on the neural side while the CLIP visual encoder stays frozen. Earlier formulations treated alignment as a largely uniform projection problem and applied similar supervision across samples, batches and subjects, even though EEG and MEG responses are noisy, highly variable across trials and shifted across subjects. Under the 200-way zero-shot retrieval protocol on THINGS-EEG, the intra-subject setting reaches 63.40%/86.85% Top-1/Top-5 and the inter-subject setting reaches 24.05%/53.45%; the inter-subject figure improves on the 12.4%/33.4% of the compared strong baseline UBP.

Neural representations are organized before contrastive learning: batch-wise standardization unifies feature scale, and a low-dimensional adapter compresses both modalities into one compact subspace. A cross-modal correlation regularizer inside that bottleneck pushes corresponding dimensions toward correlation and off-diagonal dimensions toward decorrelation, reducing redundancy. The ablation starts at 50.15% Top-1, reaches 55.10% after adding image and EEG projectors, 58.25% with adapters on both sides, 61.45% with batch-wise standardization, 61.35% with amplitude-phase modulation, and 63.40% with adaptive temperature on top.

Frequency information enters as conditional modulation rather than as concatenated features: per-sample amplitude mean, amplitude standard deviation, phase cosine mean, phase sine mean and phase circular variance are compressed into a conditioning vector that predicts scaling, bias and a confidence gate. The modulation parameters are constrained near an identity perturbation and fused residually through the confidence gate, so noisier frequency cues do not directly dominate the final representation. On THINGS-EEG the phase-only variant improves more than the amplitude-only variant, combining amplitude and phase adds further gain, the full model gives the best Top-1, and simpler frequency variants give the best Top-5.

Perspective

The results rest on 10 subjects in THINGS-EEG and 4 participants in THINGS-MEG, evaluated with a 200-way zero-shot retrieval protocol under separate intra-subject and inter-subject settings; the visual side uses frozen OpenCLIP encoders with RN50 as the default backbone. The material gives no corresponding evidence for other recording modalities, other category counts or online use.

The frequency branch relies on relatively compact statistical descriptors and does not fully capture richer time-frequency structure or finer neural dynamics; why frequency enhancement helps harder samples especially still needs deeper neuroscientific interpretation and stronger statistical validation; the phase-derived control signal correlates near zero with sample-wise retrieval outcomes and should be read as an optimization-control cue rather than a difficulty estimator. Phase-guided adaptive temperature mainly sharpens the top-ranked match, and the fixed-temperature baseline achieves the highest Top-5.