Pull the prompt and the gain vanishes: Cortex writes context into feed-forward weights, keeping 88.9% of the gain after 50 sequential edits
Synopsis
Large language models perform better when new knowledge sits in the prompt, but that gain usually disappears once the prompt is removed. Cortex distills the context-conditioned behavior into low-rank increments or added nonlinear channels inside feed-forward networks, using queries the model generates itself to anticipate how the context will be used. After 50 sequential factual edits on a 14B model, Cortex-Structural recovered 88.9% of the context-induced gain while retaining 99.8% of non-target performance.
Interpretation
A capability gain supplied by context can be written into model parameters and survive removal of that context, while capabilities outside the intended change stay largely intact. Conventional in-context inference leaves parameters unchanged, so gains do not persist once the supporting material is removed; external memory instead depends on retrieving and re-supplying relevant content at inference time. After 50 sequential factual edits on a 14B model, Cortex-Structural reached 88.9% consolidation rate and 99.8% retention rate, exceeding AlphaEdit by 66.2 and 3.8 percentage points respectively.
The procedure uses the context-conditioned model as a teacher and the context-free model as a student, training only newly added feed-forward parameters. Cortex offers two write forms: Synaptic adds low-rank increments to existing projections, while Structural adds nonlinear channels and freezes the original weight blocks; both fold into standard feed-forward layers. The objective aligns output-distribution KL divergence together with mean squared error on post-FFN residual hidden states across layers, and the training examples are queries and answers generated by the frozen pre-update model without touching held-out evaluation data.
Perspective
Evaluation spans 4B to 120B dense and mixture-of-experts backbones, including instruction-tuned, reasoning and multimodal models, with text inputs throughout; agent experience consolidation is tested on ALFWorld and τ2-Bench, and τ2-Bench only on GPT-OSS-20B.
Context length does not determine consolidation difficulty: in the Qwen2.5 batch setting, the roughly 599-token suppression context yields 35% to 57% consolidation, the 809-token factual editing context 89% to 91%, and the 5,152-token API context 66% to 74%. On GPT-OSS-20B τ2-Bench, Structural reaches 38.5% consolidation and Synaptic −92.3%, the latter retaining 98.5% retention while falling below the original model on the target task. Knowledge suppression on GPT-OSS-20B reaches 97% to 100% consolidation but only 76% to 83% retention.
