Equivariant attention reaches 512 GPUs: Tessera holds 30.7 million atoms at 89.7% weak-scaling efficiency
Synopsis
Message-passing equivariant Transformers carry information past any single cutoff, but published systems had stopped at tens of GPUs. Tessera, from IQuest Research, sorts atoms along a space-filling curve and cuts the order into equal-count shards, so every tensor keeps one shape and a whole rebuild interval replays as one captured CUDA Graph; on 512 GPUs it sustains the full force-evaluation step at 30.7 million atoms with 89.7% weak-scaling efficiency, and a 10.24-million-atom salt solution holds energy drift at 0.42 μeV/atom/ps.
Interpretation
At tens of millions of atoms, keeping ab initio accuracy is a communication problem rather than a floating-point one: every attention layer exchanges boundary features across devices, and the neighbor graph keeps changing shape as atoms move. Strictly local machine-learning force fields confine each atom's energy to its own cutoff ball, so information never travels along the molecular graph; equivariant message-passing models compound layers into a 23 Å receptive field, but published systems had reached only tens of GPUs. On 512 H100 GPUs Tessera sustains the full force-evaluation step at 30.7 million atoms with 89.7% weak-scaling efficiency, and per-step time rises only 1.11–1.29× as machine and problem grow 16×.
The construction quantizes atomic coordinates onto a space-filling curve, sorts them, and cuts the sorted order into contiguous equal-count segments, one per GPU. An equal-volume grid bounds each GPU to 27 neighboring cells but lets atom counts drift with density, and every tensor shape with them; an equal-count cut that ignores geometry degenerates toward all-to-all peers. At 25k atoms per GPU the heaviest rank requests candidates from 6, 11, 14, and 19 source ranks at 64, 128, 256, and 512 GPUs, so an 8× device increase costs 3.2× in peers; with a 0.5 Å conservative neighbor skin, nine steps in ten reuse the topology and an entire rebuild interval's forward and backward passes replay as one CUDA Graph.
Perspective
The demonstrated scope is weak scaling from 32 to 512 H100 GPUs at 10k to 60k atoms per GPU, plus trajectories of a 10.24-million-atom KCl solution and a 10.06-million-atom Norovirus capsid on 256 A100 GPUs; these are throughput, stability and structure checks, while the model's own accuracy is established separately.
The authors note that peer count still rises with device count and that the metadata all-gather is the next growing term; the composed 15.7× neighbor-cost reduction is an upper estimate rather than a measured speedup; the 60k-atoms-per-GPU point alone enables activation recomputation, so it is not a like-for-like continuation of the 10k-to-25k trend; reaching ns/day is left to future work.
