SOAP's gains without SOAP's bill: FOAM halves the optimizer state and reaches Muon-class wall-clock time
The second-order optimizer SOAP leads on Transformer pre-training quality but pays higher optimizer state and a slower step, so practitioners often default to AdamW or Muon. FOAM keeps only the input-side preconditioning factor and replaces the dense rotated second moment with a row-and-column factored Adam-moment estimator; on GPT-2-scale pre-training from 124M to 760M parameters it matches SOAP and Muon on validation loss, uses half of SOAP's optimizer state, and runs at Muon-class wall-clock time.