Skip to main content
Search

# Decode Context Parallelism

4 related articles

Subscribe to updates
  1. vLLM Blog

    vLLM runs Mistral Large 4 on day one: NVFP4 decodes 1.08–1.19x faster than FP8 with about 1.5x the KV cache

    vLLM added same-day inference support for Mistral Large 4, a 1-trillion-parameter sparse mixture-of-experts multimodal model with 49 billion parameters active per token. The work spans breakable CUDA graphs, hand-written fusions, decode and prefill context parallelism, DSpark speculative decoding, NIXL disaggregated serving and host-memory KV offload. On TP8, NVFP4 decodes 1.08–1.19x faster per token than FP8 and holds about 1.5x the KV cache, while staying within 2.1 points of FP8 on multimodal and long-context evaluations.

  2. vLLM Blog

    A million-token context on 16 H200s: GLM-5.3 shards the KV cache by token and lifts per-user speed up to 3.2×

    Serving GLM-5.3's long contexts on 16 H200s runs into two limits: KV cache memory and prefill speed. On the prefill side, each prompt is split across eight GPUs by token (PCP) and the KV cache is sharded by token (DCP), so a 1M-token request needs only 6.7 GiB per GPU; on the decode side, Hybrid HiSparse keeps KV in host memory. On the AgentX benchmark, PCP8 × DCP8 prefill with HiSparse decode gives 1.7–3.2× the per-user speed of a DP8 deployment at the same throughput, and it is the only tested configuration that serves 1M-token contexts.

  3. vLLM Blog

    MiniMax-M3.1-Flash lands in vLLM on day one: NVFP4 cache lifts 256K-token concurrency from 28 to 76 sequences per deployment

    MiniMax open-sourced MiniMax-M3.1-Flash, and vLLM supports it on day one on NVIDIA Blackwell, NVIDIA Rubin and AMD MI350-series GPUs. The model is quantization-aware trained for an NVFP4 KV cache and stores its routed MoE weights in NVFP4, shrinking the checkpoint from 451.5 GB to 250.1 GB and the KV cache from 61,440 to 34,560 bytes per token. On four NVIDIA B300 GPUs with DEP4 deployment and 256K-token sequences, M3.1 fits 76 concurrent sequences where M3 with an FP8 KV cache fits 28.

  4. vLLM Blog

    Eight AMD MI355X GPUs match NVIDIA B300 on Kimi K3: 24.4% more tokens per GPU, 87.4% more per dollar

    On the public AgentX agent workload, an eight-GPU AMD Instinct MI355X system serving Kimi K3 with vLLM reaches about 7,567 total tokens/s/GPU at 50 tokens/s/user P90 interactivity, 24.4% above NVIDIA HGX B300's 6,083; under SemiAnalysis's modeled $1.50 versus $2.26 per GPU-hour, that is 18.16 million versus 9.69 million tokens per dollar of TCO, 87.4% more. The result comes from sustained AMD and vLLM community engineering across attention, recurrent state, MoE and caching rather than a single change.

You’re all caught up