Skip to main content
Back to timeline
arXivSource publication:

Zero-SM NCCL takes GPU communication off the SMs: AllGather bandwidth reaches 1.36x the legacy kernels and overlapped GEMM slowdown falls from 33% to 4%

Synopsis

Zero-SM NCCL moves both data movement and synchronization out of the streaming multiprocessors (SMs), using CUDA stream memory operations, copy engines and a CPU proxy driving RDMA to run AllGather, AlltoAll and one-sided Put/Wait on GB200 NVL72 and B200 NVL8 systems. Its AllGather reaches up to 1.36x the bandwidth of NCCL's legacy SM kernels and cuts the median slowdown of overlapped GEMMs from 33% to 4%, at the cost of higher per-operation latency. In Megatron-LM it speeds up FSDP by up to 3.5% and pipeline parallelism by up to 15%, but it does not win on small messages or host-bound workloads.

Source-provided article image: Zero-SM NCCL: A Systematic Guide to Designing and Deploying SM-Free GPU Communication
arXiv

Interpretation

GPU collective communication can run without launching any SM kernel, covering both the intra-node NVLink domain and the cross-node network, and offering collectives and one-sided primitives together. Previously NCCL's symmetric kernels still used a few CTAs to move data, reduce and poll flags; NCCLX handed bulk data to dedicated hardware but kept SM kernels to enforce stream order; VCCL's SM-free mode covered only point-to-point transfers. On GB200 NVL72 and B200 NVL8, Zero-SM AllGather reaches 714–746 GB/s at 4 GiB, against 537–678 GB/s for the symmetric kernel and 613–698 GB/s for the legacy kernel; across nodes at 4 GiB it reaches 503 and 436 GB/s, close to the 517 and 450 GB/s NIC bound.

Once communication stops competing for SMs, the matrix multiplications it overlaps are barely slowed. On one B200 node, a 144 MiB AllGather overlapped with a GEMM chain slows those GEMMs by a median of about 33% under the legacy ring kernel, 6.2% under the symmetric kernel at its default 4 CTAs and 24.0% at 32 CTAs. In the same overlap benchmark the copy-engine path causes only a 3.9% median slowdown; with the 26 MiB Llama AllGather, the copy-engine path slows no GEMM by more than 0.2%, while the legacy kernel slows them by a median of 9.9%.

Turning SM-free communication into end-to-end training gains depends on the workload shape, not on microbenchmark bandwidth alone. Across 12 FSDP and 7 PP recipes in Megatron-LM, the copy-engine path beats the legacy ring by up to 3.5% on GB200 and 1.9% on B200 for GPU-bound dense recipes; one-sided puts speed up every deep pipeline, by up to 14.9%. Against the symmetric kernel, which uses only a few CTAs, the picture is mixed: it wins clearly only on Llama-70B-G on GB200 and loses on host-bound recipes and on Qwen3-235B-G; shallow or heavily interleaved pipelines gain nothing or lose.

Perspective

The demonstrated scope is limited to pure data-movement collectives such as AllGather, AlltoAll, Gather and Scatter plus the ncclPutSignal and ncclWaitSignal one-sided interfaces, on the GB200 NVL72 and B200 NVL8 systems; arithmetic collectives such as AllReduce and ReduceScatter are not yet supported in zero-SM mode. The evaluation covers nccl-tests microbenchmarks, a single-node GEMM overlap benchmark, and 12 FSDP and 7 PP recipes in Megatron-LM with models from 15B to 405B parameters on 32 or 64 GPUs. The authors state that the pipelined rail schedule and tuner will ship in a future NCCL release.

The authors state explicitly that zero-SM communication is not a universal win: it pays off for large transfers in GPU-bound workloads and for deep pipelines with large bubbles, but loses on small messages and host-bound workloads. Every intra-node operation is issued from the host and synchronized through stream memory operations, so small-message latency is higher than for SM kernels, and CUDA graph capture costs more because its replay-safe protocol needs extra signaling; cross-node synchronization relies on a CPU thread, and the number of copy engines on the GPU may limit concurrent transfers with high fan-out. The cost estimate is not a full deployment cost model.