EUROPA lifts 1T-scale MoE training on JUPITER from 90 to about 1,300 tokens/s per GPU
Synopsis
The EUROPA consortium ran a Mixture-of-Experts pretraining configuration of about 951.9B total parameters with 32.8B active per token on JUPITER, shared European public infrastructure, raising per-GPU throughput from 90 to about 1,300 tokens/s. Training at this scale usually relies on clusters the developers own or rent exclusively, while JUPITER's inter-node InfiniBand is far slower than each GPU's link to its own CPU. Working through PyTorch and TorchTitan, the team reached that throughput on 384 GH200 GPUs per data-parallel replica and upstreamed several changes.
Interpretation
Pretraining at roughly 1 trillion parameters can now run on shared public research infrastructure rather than on clusters a developer owns or rents exclusively. JUPITER is shared by hundreds of projects, and its inter-node InfiniBand is 36 times slower than the link between a GPU and its own CPU, a boundary that a 1T-parameter model spread over hundreds of accelerators must cross continuously. On 384 NVIDIA GH200 GPUs per data-parallel replica (96 nodes), per-GPU throughput rose from 90 to about 1,300 tokens/s; the final configuration uses 4 data-parallel replicas, 1536 GPUs in total.
The throughput gain comes from three changes aimed at this machine's constraints: making the Muon optimizer nearly communication-free, moving activation checkpoints to the Grace CPU, and putting FSDP's all-gather on the compute stream. Muon orthogonalizes each weight matrix as a whole, while FSDP splits every matrix into 384 shards, so a simple implementation gathers each matrix on one rank, computes, and scatters back, which is too expensive on this interconnect. FlexShard shards by whole expert so each rank owns complete expert matrices and runs Newton-Schulz locally; moving Newton-Schulz to the GPU cut the optimizer step from 41.7 s to 2.2 s and raised throughput from 441 to 597 tokens/s per GPU.
Offloading activation checkpoints to Grace memory is what lets a 65,536-token sequence fit in HBM. At 65,536 tokens, the boundary activations of 74 layers alone take about 55 GiB per GPU, which does not fit in HBM next to the weights. Each boundary activation is copied to Grace after the forward pass and copied back to Hopper one layer before its recomputation; with offload, throughput rose from 370 to 508 tokens/s per GPU, and sequence length grew to 49,152 and then, with Muon, to 65,536.
Running FSDP's all-gather setup on the compute stream removed cached memory the allocator could not reclaim on side streams, taking ranks with allocation retries from 384 to zero. The allocator keeps a separate pool per stream, and side streams once held 20.9 GiB of cached memory the compute stream could not reuse, so the compute stream could not find a 4 GiB MoE buffer. Peak reserved memory fell from 88.1 to 74.6 GiB and throughput rose from 996 to 1,210 tokens/s per GPU; the flag is being moved into a public FSDP API (PyTorch #202343).
Perspective
The report covers the EUROPA consortium scaling a MoE pretraining configuration of about 951.9B total parameters on JUPITER to 384 GH200 GPUs per data-parallel replica, 1536 GPUs across 4 replicas, at a sustained throughput of about 1,300 tokens/s per GPU; engineers concerned with the feasibility of large-model training on public supercomputers are the main readers.
The throughput figures are tied to JUPITER's GH200 nodes and InfiniBand configuration, and usable host memory per node can fall below 108 GiB because of the kernel's share; data and expert load balancing, reducing bytes sent over InfiniBand, and node-local resharding are left to a companion post and not detailed here.
