Skip to main content
Back to timeline
arXivSource publication:

Obaleth puts GPU-initiated networking inside the kernel IR, beating NVSHMEM by 1.25x on a 256-GPU all-to-all

Synopsis

GPU-initiated network communication has meant calling hand-written device libraries such as NVSHMEM and NCCL, whose transport decisions stay invisible to the compiler and whose every call pays for reservation, fencing and completion polling. Obaleth writes put, signal and wait into the kernel's intermediate representation, optimizes them together with the computation, and lowers them to transport-specific code. On H100 GPUs it sends 8-byte messages at up to 3.3 times NVSHMEM's rate and 4.2 times NCCL GIN's; on 256 GPUs its small-message all-to-all is 1.25 and 2.3 times faster, using 22 MiB of device memory per rank.

Source-provided article image: Compiling Specialized GPU-Initiated Networking
arXiv

Interpretation

Communication between GPUs becomes an operation inside the kernel's intermediate representation that the compiler can rewrite, so transport code is trimmed to the workload and network topology. Device functions in libraries such as NVSHMEM and NCCL GIN must serve any caller, thread geometry and transport, often through run-time branches, and they reserve queue slots with atomics while ordering memory and tracking completions conservatively. On H100 GPUs, Obaleth's Tier-2 kernels send 8-byte messages at 243 M msg/s on 64 thread blocks, 3.3 times NVSHMEM's rate and 4.2 times NCCL GIN's, because they ring one doorbell per window where the libraries ring one per put.

The compiler plans a transfer as submission, publication and completion, letting independent message chains share one queue reservation, one doorbell and at most one completion. Libraries aggregate requests only when the caller asks, and each put-with-signal still places its ordering point behind its own payload. On a stream of 16 chains per round, coalescing cuts GPU issue time from 41.5 to 3.5 microseconds and publication ordering takes the round from 39.8 to 4.3 microseconds; serving Qwen3-235B in vLLM on 16 B200 GPUs, it answers requests 1.15–1.29 times faster than DeepEP.

Perspective

The demonstrated scope is point-to-point, all-to-all and MoE expert parallelism in vLLM on H100 and B200 GPUs, plus preliminary MI355X results; the authors explicitly leave full AMD and EFA evaluations to future work.

Above 64 KiB per peer the exchange is NIC-bound, and the device APIs converge to within 4% of each other from 1 MiB per peer; on Qwen3-235B the tail of time to first token is 1.4–2.0 times worse than DeepEP, and from 64 tokens per rank DeepEP's dispatch kernel is faster by up to 1.7 times.