Medical retrieval's dense vectors split into readable features: sparse codes beat original embeddings on CURE, and explanations still predict activations across corpora
Synopsis
Medical retrieval systems usually compress queries and documents into dense vectors whose matches are hard to explain. This work trains sparse autoencoders on those dense embeddings with an InfoNCE contrastive loss that preserves query-document relevance, raising NDCG@10 above the original embeddings for all three embedders on CURE. Neuron interpretations built only from CURE texts still positively predict whether a neuron fires on five out-of-domain corpora, and concept-level interpretations outperform raw example snippets.
Interpretation
Dense medical retrieval embeddings can be expanded into a set of sparse features that can be read individually, so a retrieval result can be traced back to specific concepts. Medical retrieval embedders have largely been black boxes, and prior sparse-autoencoder work on general retrieval embeddings neither targeted medical retrieval nor tested whether interpretations transfer to new corpora. On the CURE test set, sparse codes raise NDCG@10 above each embedder's original embeddings: BGE from 53.06 to 60.50 at k=256, Gemini from 52.77 to 59.85, and MedEmbed from 52.46 to 59.21.
The sparse autoencoder maps each query or document embedding into a higher-dimensional representation where only a few neurons are active, and documents are ranked by cosine similarity between these sparse vectors. The training objective is a weighted combination of a reconstruction loss and an InfoNCE contrastive loss, the latter using the datasets' query-document relevance labels to pull a query's sparse vector toward its positive documents and push it away from irrelevant ones. Experiments cover three embedders and six medical retrieval datasets, with training only on CURE and the other five datasets used as out-of-domain tests; each configuration uses three seeds and reports confidence intervals.
A neuron's meaning can be described in natural language, and that description retains predictive power outside the training corpus. Interpretations come from TF-IDF keywords or an Interpreter LLM, and an activation-prediction LLM then judges whether a given text will trigger the neuron, scored against actual activations. Interpretations built only on CURE yield positive correlations on all five out-of-domain datasets; for the BGE sparse autoencoder, Interpreter LLM interpretations fall from a Pearson correlation of 0.440 on CURE to 0.226 on Biology, still above the 0.142 of snippet interpretations.
Perspective
The demonstrated scope is retrieval and interpretation results for three embedders trained on CURE and evaluated on CURE plus NFCorpus, MIRIAD and the three R2MED subsets; the human evaluation had five medical experts rate 30 randomly selected neurons with 15 positive and 15 negative passages each.
The human annotation campaign remains small scale; the interpretability evaluation relies on an LLM and may carry biases or calibration errors, which the authors partly offset with several LLMs; sparse representations are capacity-limited and do not match dense retrieval on all datasets, and models fine-tuned on CURE often underperform the original embeddings out of domain.
