A pipeline inside one GPU: SubGPU cuts a 36-layer decode iteration by 10.33% and lifts DRAM throughput to 83.6% of peak
Synopsis
LLM decode is dominated by many short, memory-bound operators, and a single operator often hits the HBM bandwidth ceiling before it uses all streaming multiprocessors (SMs), while tensor, data, expert, context and pipeline parallelism only divide work across GPUs and leave this intra-GPU inefficiency untouched. SubGPU Pipelining cuts the operator sequence of a layer or repeating graph into ordered contiguous segments, binds each segment to a disjoint SM pool, and streams microbatches through the stages. On GPT-OSS 120B under 32-GPU B200 deployment settings, a hardware experiment using one B200 to emulate the per-rank workload reduced the 36-layer decode iteration from 14.970 ms to 13.424 ms and raised mean DRAM read-plus-write throughput from 71.8% to 83.6% of peak.
Interpretation
Decode operators are split into contiguous segments that run concurrently on disjoint SM pools, with microbatches flowing between segments so memory-bound and compute-bound operators execute at the same time. Tensor, data, expert, context and pipeline parallelism scale inference across multiple GPUs and do not directly address intra-GPU kernel launch overhead, idle compute, or bursty bandwidth demand. For GPT-OSS 120B decode with input and KV history length 32,768 and 32-GPU B200 settings using TP8/DP4 for attention and EP32 for MoE, a shared-resource simulator searched pipeline structures and batch sizes; the resulting Pareto frontier gives higher per-user throughput at matched per-GPU throughput and higher per-GPU throughput at matched per-user throughput.
In implementation, a two-stage design places full-attention FMHA in a 68-SM Green Context and all other operators of a two-layer macro-period in an 80-SM Green Context, each context owning two streams through which two equal microbatches flow. Green Contexts are a CUDA mechanism that partitions SMs, binding streams inside one context to an SM subset; the prototype uses CUDA events and CUDA Graphs for producer-consumer and same-stream dependencies and leaves existing TensorRT-LLM kernels unchanged. On one B200 emulating the per-rank workload of the 32-GPU deployment at global decode batch 2,048 with TensorRT-LLM 1.3.0rc22.post1 kernels, three rounds of 100 uninstrumented CUDA Graph replays after 20 warmups gave the iteration-time and DRAM-throughput changes above, while mean active-SM throughput fell from 85.9% to 71.7%.
Perspective
The demonstrated scope is simulated search for GPT-OSS 120B decode at input sequence length 32,768 and KV history 32,768 under 32-GPU B200 deployment settings, plus one two-stage pipeline configuration implemented with TensorRT-LLM kernels and CUDA Green Contexts on a single B200 emulating the per-rank workload at global batch 2,048.
The authors describe the simulation results as modeled opportunities rather than formal upper bounds, whose credibility depends on the resource-sharing, cache-initialization, admission, eviction, communication and operator-scaling policies; sensitivity to these assumptions and calibration against measured kernels are needed before extrapolating the Pareto gains to an end-to-end deployment. The hardware experiment covers one selected configuration, and the authors treat the overall result as a proof of feasibility while noting that cuts, allocations and kernels still leave room for optimization.
