Skip to main content
Back to timeline
arXivSource publication:

SOAP's gains without SOAP's bill: FOAM halves the optimizer state and reaches Muon-class wall-clock time

Synopsis

The second-order optimizer SOAP leads on Transformer pre-training quality but pays higher optimizer state and a slower step, so practitioners often default to AdamW or Muon. FOAM keeps only the input-side preconditioning factor and replaces the dense rotated second moment with a row-and-column factored Adam-moment estimator; on GPT-2-scale pre-training from 124M to 760M parameters it matches SOAP and Muon on validation loss, uses half of SOAP's optimizer state, and runs at Muon-class wall-clock time.

AI-generated editorial illustration: Factored One-sided Adam-Moment for Practical and Scalable SOAP

Interpretation

FOAM cuts SOAP's two-sided preconditioning down to the input side and replaces the dense second moment in the rotated space with an outer product of row and column means, halving the optimizer state relative to SOAP. SOAP previously stored a first moment, a rotated second moment, two Kronecker factors and two eigenbases, and periodically ran two eigendecompositions, so despite better quality practitioners often defaulted to AdamW or Muon. At the 124M, 350M and 760M GPT-2 scales, FOAM trails SOAP by about one seed standard deviation or less in validation loss while using half of SOAP's matrix-parameter optimizer state at every size.

Input-only preconditioning suffices because Transformer-layer Hessians are near block-diagonal along the output direction, so output-side curvature is dominated by its diagonal. SOAP's earlier one-sided variant kept whichever side had the smaller dimension, whereas FOAM keeps the input side regardless of size, so the two rules pick opposite sides for the MLP down-projection. On the per-layer Gauss-Newton matrix of GPT-2 small, dropping the output-side rotation costs about a quarter of the curvature alignment while dropping the input-side rotation costs about three quarters; input-only exceeds output-only in all six module types at both 124M and 350M.

The factored second moment loses almost nothing once expressed in the input-side eigenbasis, because the rotated second moment is nearly rank-one separable. Adafactor-style row-column factoring of Adam's raw second moment is far less accurate; only after the rotation does the second moment become separable. Under the K-FAC assumption the estimator exactly recovers SOAP's dense second moment; measured K-FAC violation is about 15–20%, mostly at the key projection, with negligible effect on the update.

Perspective

Experiments cover FineWeb pre-training at 124M, 350M and 760M under the GPT-2 architecture, with supplementary tests on GPT-2 XL at 1.56B, ViT-Tiny on Tiny-ImageNet, and supervised fine-tuning and on-policy distillation of Qwen3; only attention and MLP matrix parameters use the compared optimizer, while embeddings, the lm-head and 1D parameters stay on AdamW.

Experiments stop at GPT-2 scale, up to 1.56B parameters, so whether FOAM's advantages hold for frontier-scale pre-training remains the main open question; the current implementation does not yet shard the factors and eigenbasis across devices; and the factored update is least accurate at the key projection, whose gradients pass through RoPE and softmax.