An nGPT training recipe lets a 30B hybrid Mamba-2–Transformer MoE model reach the same validation loss with about half the training tokens
This paper presents a practical training recipe for the normalized Transformer (nGPT) — comprising Logit Gradient Preconditioning, Logarithmic Learning Rate Decay, GatedAdamW, angular update control, and optional exploration mechanisms — and evaluates it on modern hybrid Mamba-2–Transformer Mixture-of-Experts (MoE) models; compared with an unnormalized model of the same hybrid MoE architecture trained with AdamW, the 30B-total-parameter nGPT model reaches the same validation loss using approximately half as many training tokens, and the recipe scales across the models considered, which contain up to 30B total parameters.