Skip to main content
Back to timeline
arXivSource publication:

Inference accelerators can learn while serving: LWS hides learning in prediction's idle compute and keeps 99.3% of the quality benefit

Synopsis

When a long-context model keeps learning during inference, it pays for backward computation and weight updates on top of prediction, while today's inference accelerators assume fixed weights. LWS schedules that learning onto matrix capacity left idle during prediction, uses per-layer weight versioning so prediction can read slightly older weights, and lets long learning kernels pause and resume across fragmented idle windows. On modeled datacenter and edge NPUs, TTT-E2E 3B at 128K tokens sustains prediction-only throughput while preserving 99.3% of fully fresh learning's quality benefit.

Source-provided article image: Mutable Inference: Architecting LLM Accelerators for Inference-Time Learning
arXiv

Interpretation

An inference accelerator can carry inference-time learning without giving up prediction throughput by placing the backward computation and weight updates that would otherwise run serially into matrix capacity prediction leaves idle. Serial execution waits for each chunk's learning and weight updates before processing the next chunk, and learning accounts for 54% of TTT-E2E's total execution time, offsetting most of the speedup from bounded attention. On modeled datacenter and edge NPUs, LWS sustains prediction-only throughput for TTT-E2E 3B at 128K tokens while retaining 99.3% of fully fresh learning's quality benefit, and across nine models reaches the throughput of serial execution and of the best prior co-execution architecture.

The mechanism turns weight updates into deadline-bound version publications and lets learning kernels yield when prediction needs the hardware, then continue later. Per-layer weight versioning lets prediction keep reading versions it already acquired while learning publishes new ones, and resumable reduction blocks let long learning kernels span multiple idle windows. On a modeled TPUv5e-class datacenter NPU, the mixed-lag policy 1111|0000 for TTT-E2E 3B retains 99.3% of the learning benefit at 128K, whereas freezing the four input-side mutable layers retains only 76.2%.

Dedicated softmax-gradient units relieve learning's contention for the shared vector unit so the overlap is not blocked by a vector bottleneck. Learning's vector demand reaches 2.1 times the vector capacity prediction leaves idle, with attention and vocabulary softmax gradients together accounting for 86.8% of learning's vector work. Removing the dedicated dS/dZ units lowers throughput; with them, learning's additional vector demand falls to 3.5% of the chunk period, and RTL synthesis shows LWS adds 1.5% to NPU core area without another MAC array.

Perspective

Evaluation focuses on chunked prefill of a single long-context sequence; batched serving and autoregressive decoding are outside its scope, and the performance and area figures come from an operation-level simulator and 16 nm RTL synthesis rather than fabricated silicon.

Quality retention is measured on 16 input sequences from the Books3 validation set against fully fresh learning's NLL improvement; in the online LoRA edge configuration, combined MAC demand exceeds hardware capacity by 7%, which limits throughput.