Skip to main content
Search

# nGPT

1 related articles

Subscribe to updates
  1. arXiv

    An nGPT training recipe lets a 30B hybrid Mamba-2–Transformer MoE model reach the same validation loss with about half the training tokens

    This paper presents a practical training recipe for the normalized Transformer (nGPT) — comprising Logit Gradient Preconditioning, Logarithmic Learning Rate Decay, GatedAdamW, angular update control, and optional exploration mechanisms — and evaluates it on modern hybrid Mamba-2–Transformer Mixture-of-Experts (MoE) models; compared with an unnormalized model of the same hybrid MoE architecture trained with AdamW, the 30B-total-parameter nGPT model reaches the same validation loss using approximately half as many training tokens, and the recipe scales across the models considered, which contain up to 30B total parameters.

You’re all caught up