Two DGX Stations serve a 753-billion-parameter GLM-5.3 with every weight in GPU memory, peaking at 607.8 tok/s at concurrency 16
Synopsis
An NVIDIA tutorial places the NVFP4-quantized version of Z.ai's mixture-of-experts GLM-5.3 on two DGX Stations, using vLLM tensor parallelism to split the roughly 464 GB checkpoint across two GB300 GPUs with CPU weight offload disabled, while Station A serves it through one OpenAI-compatible API. A single GPU's 252 GB of HBM3e cannot hold that checkpoint; on SPEED-Bench Coding prompts the two-machine deployment reaches 607.8 tok/s overall throughput at concurrency 16 and falls back to 385.0 tok/s at concurrency 32.
Interpretation
One GB300 GPU's 252 GB of HBM3e cannot hold the roughly 464 GB GLM-5.3 NVFP4 checkpoint, but two DGX Stations joined over ConnectX-8 RDMA keep all weights of the 753-billion-parameter model in GPU memory without CPU weight offload. The prior limit was per-machine memory capacity: each token activates 40 billion parameters, yet different tokens can be routed to different experts, so every weight must stay available and sparse activation saves computation rather than memory. The startup log reports about 218 GiB of weights per GPU, roughly half the checkpoint, and nvidia-smi shows VLLM::Worker_TP0 on Station A and VLLM::Worker_TP1 on Station B, one rank per GPU.
The service runs tensor parallelism at TP=2, both machines compute every request, and only Station A exposes the OpenAI-compatible API while Station B runs as a headless worker. Tensor parallelism splits the model's tensors across the two GPUs, so every token requires the two ranks to exchange partial results over the RoCE rail and neither GPU can answer a request alone. An NCCL log check confirms GPU Direct RDMA rather than a TCP fallback with no NCCL warnings, the health check returns HEALTH_OK on Station A, and a chat request finishes with finish_reason: stop.
On 80 SPEED-Bench Coding conversations with prefix caching, four MTP draft tokens per step and an FP8 KV cache, overall throughput peaks at 607.8 tok/s at concurrency 16 and drops to 385.0 tok/s at concurrency 32. Raising concurrency does not raise throughput monotonically: at concurrency 32 mean request latency is 453.5 s and mean per-user throughput falls from 115.3 tok/s at concurrency 1 to 13.0 tok/s. Table 2 lists overall throughput of 107.4, 386.0, 607.8 and 385.0 tok/s at concurrency 1, 8, 16 and 32, with latency metrics averaged over measured requests and warmup excluded.
Perspective
The demonstrated setting is two GB300 DGX Stations linked over ConnectX-8 RDMA, running a pinned vLLM container and a pinned revision of the GLM-5.3 NVFP4 checkpoint with CPU weight offload disabled; the performance figures come from 80 SPEED-Bench Coding conversations at default reasoning effort with an 8192-token output cap.
Throughput changes non-monotonically with concurrency, falling to 385.0 tok/s and mean request latency of 453.5 s at concurrency 32; the gain from MTP depends on how often drafts are accepted, which varies with the prompt. The tutorial also notes that performance may differ from Table 2 without prefix caching.
