Skip to main content
Search

Danyal Ahmed

1 related articles

Subscribe to updates
  1. arXiv

    KEAP recomputes only two-thirds of checkpointed layer components, holding Llama-488M perplexity rise to at most 2.5% while keeping full-checkpoint memory savings

    Activation checkpointing saves memory in large-model training by recomputing activations during backpropagation, but recomputing linear layers is expensive. KEAP instead recomputes only a subset of layer components: it uses statistics from the first forward pass to greedily keep the largest components or to sample them by probability, while checkpoints stay exact. On Llama-488M, recomputing two-thirds of components raised perplexity by 1.7%, 1.2% and 2.5% when applied to feed-forward layers, attention layers, or both.

You’re all caught up