Skip to main content
Search

# Gradient History Recombination

1 related articles

Subscribe to updates
  1. arXiv

    37k-Parameter Learned Optimizer Trained in 0.87 GPU-Hours Transfers Zero-Shot, Cutting BERT-Tiny Validation Loss by 9.1%

    This paper presents a lightweight and versatile learned optimizer that dynamically recombines gradient history represented as averages over disjoint time spans: the prediction space is reduced to one scalar coefficient per gradient average shared by multiple parameters, and progressively averaging older gradients minimizes the memory cost of long history while keeping their contributions independently accessible; a 37k-parameter network trained in 0.87 GPU-hours generalizes zero-shot to unseen tasks, lowering validation loss by 9.1% and 0.4% on BERT-Tiny and GPT-Tiny, improving test accuracy over Adam by 3.5 percentage points on a Vision Transformer and by 2.7 percentage points on average across nine graph models, with FLOPs overhead as low as 0.3%.

You’re all caught up