vLLM folds decode layers into a few kernel launches, lifting single-user interactivity by up to 2.26x
Synopsis
AMD has brought FlyDSL-written MonoKernels into vLLM, merging the many operator launches of a decode layer into a few persistent kernel launches that reuse thread blocks across stages, prefetch weights and pass ready data through tagged mailboxes. On MI355X, DeepSeek-V4.1-Flash, DeepSeek-V4-Pro, MiniMax-M3, Kimi-K3 and Muse Spark improve concurrency-1 interactivity or time per output token by 1.20x to 2.26x, with DeepSeek-V4.1-Flash also merged on MI325X, and the largest gains at low concurrency. The path is off by default, enabled with --monokernel, and serves only specific models, tensor-parallel sizes and step widths.
Interpretation
In interactive LLM serving, a decode step runs a chain of GPU kernels for projections, attention and expert computation; at small batch sizes each kernel does little work yet still pays dispatch, startup and completion costs, and a dependent kernel cannot start until the previous whole stage finishes. A MonoKernel places these dependent operator stages inside a single kernel launch, replacing a launch at every operator boundary with an author-designed schedule, which shortens the waits between operators. The prior path launched a separate kernel at every operator boundary; HIP graphs reduce CPU launch overhead but do not remove these GPU-side costs. The MI300X/MI325X port cuts a DeepSeek-V4.1-Flash standard decoder layer from 32 kernels in 320 microseconds to 2 kernels in 115 microseconds; GLM-5.2 at TP8 on 8x MI350X drops a graph-mode decode step from about 2,459 kernels to about 210.
The mechanism is a persistent launch: a fixed grid of GPU thread blocks works through multiple stages and takes on ready work from the next stage once its current work is done, without a separate kernel launch. Thread blocks are reused across stages, output-projection or expert up/gate weights can be prefetched while earlier computation runs, and intermediate results pass to downstream stages through tagged mailboxes so a consumer waits only for the inputs its own computation needs. On a TP2 MI355X node, DeepSeek-V4.1-Flash runs its 30 standard layers as K1 plus K2, raising concurrency-1 P90 interactivity from 318.1 to 429.1 tok/s/user; the same PR's FFN launch for the other 10 layers adds another 1.02x to 1.19x.
MonoKernel boundaries are chosen per model rather than by a universal one-layer, one-kernel template, because fusing more stages is not automatically better and the boundary must fit the model's data dependencies, communication pattern and GPU resource limits. DeepSeek-V4.1-Flash splits the standard decoder-layer path into K1 and K2, while MiniMax-M3 fuses only the FFN portion and leaves attention on the stock path. MiniMax-M3 on MI355X at TP4 with random 1k/1k prompts drops concurrency-1 TPOT from 5.12 to 4.27 ms; Kimi-K3 at TP8, concurrency 1, seven speculative tokens and 8k input drops P50 TPOT from 5.47 to 3.44 ms.
Perspective
The demonstrated setting is specific models and tensor-parallel configurations on AMD Instinct MI355X, MI350X and MI325X, where gains are largest at concurrency 1 and shrink as concurrency rises; step width is limited by speculative tokens and graph padding, up to 48 rows for DeepSeek-V4.1-Flash on MI355X and 42 on MI325X, and 16 for MiniMax-M3 and Kimi-K3, with wider steps running on the stock path.
For the TP8 configuration, DSpark speculative-decoding step-time measurements use a synthetic acceptance length of 3.87 to isolate step time, and accuracy for that configuration is still being evaluated; DeepSeek-V4-Pro's baseline is a historical InferenceX run on an older software stack, so its numbers include stack changes as well as the kernels; on MI300X/MI325X the kernels keep their own weight copy, 170 GiB per rank instead of 99 GiB for DeepSeek-V4.1-Flash at TP4.
