Skip to main content
Search

# Second-Order Optimization

2 related articles

Subscribe to updates
  1. arXiv

    SOAP's gains without SOAP's bill: FOAM halves the optimizer state and reaches Muon-class wall-clock time

    The second-order optimizer SOAP leads on Transformer pre-training quality but pays higher optimizer state and a slower step, so practitioners often default to AdamW or Muon. FOAM keeps only the input-side preconditioning factor and replaces the dense rotated second moment with a row-and-column factored Adam-moment estimator; on GPT-2-scale pre-training from 124M to 760M parameters it matches SOAP and Muon on validation loss, uses half of SOAP's optimizer state, and runs at Muon-class wall-clock time.

  2. arXiv

    Clean Cuts Second-Order Optimizer Memory to Linear via Nyström Sketching, Enabling 13B Pre-training on a Single 80GB GPU

    The authors introduce Clean and its low-precision variant Q-Clean: randomized Nyström approximations replace SOAP's explicit left and right preconditioners, reducing optimizer-state memory from quadratic to linear in layer dimensions, while four orthogonal blocks (Main, A, B, C) reintegrate off-subspace components into the update; pre-training LLaMA-350M and 1.3B on C4, Clean reaches perplexity close to SOAP with less optimizer memory than AdamW, Q-Clean cuts optimizer memory by over 50% versus Muon, Clean reaches AdamW's final performance 26% faster in wall-clock time, and the methods uniquely enable pre-training a 13B model on a single 80GB H100.

You’re all caught up