One cached adapter per document: a frozen 284B model gains 38.2% relative on enterprise Q&A
Synopsis
Continual learning lets a language model absorb an organization's growing private documents, but updating shared parameters overwrites earlier knowledge. Microfitting instead trains a small low-rank adapter offline for each document and caches it, keeping the base model frozen so new documents only add their own adapters. On EDO, an open-book Q&A environment over real enterprise documents, relative improvement over the base model rose from 30.9% to 37.5% to 38.2% as the data-generation recipe was revised.
Interpretation
Adaptation for enterprise document Q&A no longer retrains the model on every request; each document gets an adapter trained in advance and stored, loaded whenever that document is queried. Earlier test-time training performed a separate gradient update per request, repeating computation when the same document was queried again, while fine-tuning shared parameters caused catastrophic forgetting, where new updates overwrite prior knowledge. On EDO, an open-book Q&A environment built from real enterprise documents, relative improvement over the base model rose from 30.9% to 37.5% to 38.2% across recipe iterations, each evaluated on ten previously unused, table-rich documents and 200 questions.
The method generates synthetic questions for each document, trains a small adapter on them for one epoch, and exports the adapter as a standard LoRA checkpoint for caching. Adapters are initialized by the Internalizer, a hypernetwork that maps a document to adapter parameters; the hypernetwork is used only during training, and exported adapters load in any serving stack that supports standard LoRA checkpoints. The base is a frozen 284B-parameter DeepSeek V4 Flash, each adapter is rank-16 LoRA with about 12.7M trainable parameters, and each document gets 40 questions at 8 rows per step over 50 steps per epoch.
The recipe that generates training questions also improves: the system compares adapter and base-model answers on held-out questions, analyzes corrections, regressions and mismatches between generated and target questions, then rewrites the recipe for the next document batch. A candidate recipe is adopted only when a paired comparison on the same fresh held-out questions beats the incumbent; otherwise the existing cached adapters are kept. In a separate correctable-error study, the recipe shifted toward question types the base model misses but adapters can learn, with gains over base rising from 9.9% to 42.7% to 55.6%, though that study's held-out distribution changes with each recipe, so it measures targeting rather than performance on a fixed workload.
Perspective
Evaluation covers enterprise-style documents of 2.2K to 4K tokens, including policies, planning documents, technical specifications and tables, in open-book Q&A where the document stays in the prompt; the main continual-improvement sequence is three ten-document iterations.
The main sequence has one training seed per condition and one base-model run per iteration, so the monotonic rise in relative gain rests on three samples and does not by itself establish each recipe revision's contribution; in the correctable-error study the target question distribution changes across recipes, so it measures adaptive targeting rather than performance on a fixed workload; adaptation time under a changing user-question distribution has not been measured.
