Skip to main content
Back to timeline
arXivSource publication:

Retrieved knowledge no longer needs a lookup at inference: ResMem turns evidence into a residual memory, lifting mean F1 from 32.83% to 34.48% across five QA sets

Synopsis

Retrieval-augmented generation keeps a corpus index and retrieves online for every answer, adding storage and inference cost. ResMem lets a frozen base model read retrieved passages offline as a teacher and trains only a parametric memory to learn a residual added to the logits, so inference needs no retrieval. Across five knowledge-QA datasets it raises mean token F1 from 32.83% for the frozen model to 34.48%, and on an English Wikipedia subset it cuts the storage retained for teacher construction from 602.0 GiB to 32.0 GiB.

Source-provided article image: From Lookups to Lessons: Turning Retrieved Knowledge into Parametric Memory
arXiv

Interpretation

External knowledge can be written into a small memory attached to a frozen language model's output, so inference no longer queries an index or runs retrieval. Earlier parametric memories such as MLP Memory learned only to reproduce the retriever's next-token distribution, and their training loss never saw the base model's own prediction, so a memory that reproduced the retriever well could still override a correct base answer. On 38,627 questions from five knowledge-QA datasets, ResMem raises mean token F1 from 32.83% for the frozen base to 34.48% and mean EM from 25.91 to 26.74, improving both metrics on every dataset.

The memory learns the correction the base prediction needs rather than a copy of the retriever distribution. ResMem has the same frozen base act as teacher while reading BM25-retrieved evidence, while the student sees only the question; the memory outputs a residual added to the base logits, and both the distillation loss and the next-token loss are computed on base plus residual. In paired per-question counts against MLP Memory, ResMem corrects 704 base errors and breaks 381 correct base answers, versus 633 and 482 for MLP Memory, for a net EM gain of 0.84 versus 0.39 points, a paired difference of 0.45 with a 95% interval of [0.25, 0.64].

Teacher construction no longer needs to retain a kNN datastore and its index. The earlier route drew supervision from a kNN datastore of corpus representations and maintained a FAISS IVF-PQ index; ResMem switches to text retrieval and caches only the teacher's top-64 targets per position. On the English Wikipedia subset, the kNN route retains a 570.011 GiB datastore plus a 5.069 GiB index, the RAG route retains 5.067 GiB of retrieved text and BM25 index, and both cache 26.967 GiB of targets, totaling 602.047 versus 32.033 GiB, a 94.7% reduction.

Perspective

The demonstrated setting is the Mistral-7B-v0.3 base, the December 2021 English Wikipedia dump as corpus, and five knowledge-QA datasets, with an eight-layer memory of about 1.54B parameters; the storage comparison is measured on a Wikipedia subset of 74 shards, about 1/64 of the corpus.

In the shared-teacher ablation the residual objective mainly preserves correct answers, and the extra repairs come with the evidence-reading teacher; with the teacher fixed it trades some repair for more preservation, leading by 0.117 F1 points. By design the memory reads one hidden state and writes only to the output logits, so the teacher's gain does not pass through the base model's intermediate layers.