Skip to main content
Search

# Residual Self-Distillation

1 related articles

Subscribe to updates
  1. arXiv

    Retrieved knowledge no longer needs a lookup at inference: ResMem turns evidence into a residual memory, lifting mean F1 from 32.83% to 34.48% across five QA sets

    Retrieval-augmented generation keeps a corpus index and retrieves online for every answer, adding storage and inference cost. ResMem lets a frozen base model read retrieved passages offline as a teacher and trains only a parametric memory to learn a residual added to the logits, so inference needs no retrieval. Across five knowledge-QA datasets it raises mean token F1 from 32.83% for the frozen model to 34.48%, and on an English Wikipedia subset it cuts the storage retained for teacher construction from 602.0 GiB to 32.0 GiB.

You’re all caught up