Skip to main content
Search

# NCCL

1 related articles

Subscribe to updates
  1. arXiv

    Zero-SM NCCL takes GPU communication off the SMs: AllGather bandwidth reaches 1.36x the legacy kernels and overlapped GEMM slowdown falls from 33% to 4%

    Zero-SM NCCL moves both data movement and synchronization out of the streaming multiprocessors (SMs), using CUDA stream memory operations, copy engines and a CPU proxy driving RDMA to run AllGather, AlltoAll and one-sided Put/Wait on GB200 NVL72 and B200 NVL8 systems. Its AllGather reaches up to 1.36x the bandwidth of NCCL's legacy SM kernels and cuts the median slowdown of overlapped GEMMs from 33% to 4%, at the cost of higher per-operation latency. In Megatron-LM it speeds up FSDP by up to 3.5% and pipeline parallelism by up to 15%, but it does not win on small messages or host-bound workloads.

You’re all caught up