vLLM folds decode layers into a few kernel launches, lifting single-user interactivity by up to 2.26x
AMD has brought FlyDSL-written MonoKernels into vLLM, merging the many operator launches of a decode layer into a few persistent kernel launches that reuse thread blocks across stages, prefetch weights and pass ready data through tagged mailboxes. On MI355X, DeepSeek-V4.1-Flash, DeepSeek-V4-Pro, MiniMax-M3, Kimi-K3 and Muse Spark improve concurrency-1 interactivity or time per output token by 1.20x to 2.26x, with DeepSeek-V4.1-Flash also merged on MI325X, and the largest gains at low concurrency. The path is off by default, enabled with --monokernel, and serves only specific models, tensor-parallel sizes and step widths.