Skip to main content
Back to timeline
arXivSource publication:

KEAP recomputes only two-thirds of checkpointed layer components, holding Llama-488M perplexity rise to at most 2.5% while keeping full-checkpoint memory savings

Synopsis

Activation checkpointing saves memory in large-model training by recomputing activations during backpropagation, but recomputing linear layers is expensive. KEAP instead recomputes only a subset of layer components: it uses statistics from the first forward pass to greedily keep the largest components or to sample them by probability, while checkpoints stay exact. On Llama-488M, recomputing two-thirds of components raised perplexity by 1.7%, 1.2% and 2.5% when applied to feed-forward layers, attention layers, or both.

AI-generated editorial illustration: Know What You Need: Efficient Activation Checkpointing

Interpretation

KEAP places the approximation on the rematerialization path rather than changing the forward pass or the gradient rule, so recomputing only part of a linear layer keeps the memory benefit of full checkpointing. Conventional activation checkpointing must fully recompute discarded activations, which is costly for linear layers; earlier approximations either let errors accumulate through the backward pass or failed to use information already available from the first forward pass. On Llama-488M, recomputing two-thirds of layer components raised perplexity by 1.7%, 1.2% and 2.5% relative to the full-checkpointing baseline when applied to feed-forward layers, attention layers, and both.

The method writes feed-forward and attention layers as sums of elementary components and keeps only some of them during recomputation. Selection uses the norm of each component from the first exact forward pass, either greedily keeping the largest-norm components or sampling with probabilities proportional to those norms. On Llama-134M trained at a one-third retention rate, the cosine similarity between approximate and exact gradients stayed above 0.85 across layers throughout training.

Perspective

The demonstrated scope is Llama-style architectures with main experiments up to 488M parameters, plus one 953M-parameter extension applying token-level Top-2/3 selection to feed-forward layers only, trained on FineWeb and evaluated by perplexity on a held-out set.

Most results come from single training runs without reported run-to-run variance; the authors note results may differ at substantially larger scales, on other architectures or data distributions, and that key-value computation limits the savings for token-level attention selection.