Skip to main content
Search

vLLM

11 related articles

Subscribe to updates
  1. vLLM Blog

    vLLM serves Thinking Machines' Inkling 2 Small on day one: 594B total parameters, 17B active, about 630 tok/s for a single user

    Thinking Machines' Inkling 2 Small gained vLLM support on its release day. It is a text-and-image-in, text-out mixture-of-experts model with 594B total and 17B active parameters, up to 1M tokens of context, and a bundled DSpark speculative-decoding drafter. vLLM fits it with kernel fusion, programmatic dependent launch, sequence parallelism and a Gluon attention kernel rewritten for the 6:1 GQA ratio, reaching about 630 tok/s for a single user at TP4 on GB200 with acceptance length 3.5, and also supports LoRA and reinforcement learning via SkyRL.

  2. vLLM Blog

    vLLM trains a DSpark drafter for Kimi K3: math-reasoning interactivity rises from about 110 to about 435 tok/s/user

    The vLLM Project's Speculators training library now supports Kimi K3, a 2.8-trillion-parameter model, and produced a five-layer, five-billion-parameter DSpark draft model that combines parallel block drafting with sequential correction. On math-reasoning workloads, single-stream interactivity rises from about 110 to about 435 tok/s/user, output throughput reaches up to about 3.5x at matched interactivity under concurrent load, and the macro-average acceptance length across nine evaluation domains is 4.11 tokens per verification round.

  3. vLLM Blog

    vLLM runs Mistral Large 4 on day one: NVFP4 decodes 1.08–1.19x faster than FP8 with about 1.5x the KV cache

    vLLM added same-day inference support for Mistral Large 4, a 1-trillion-parameter sparse mixture-of-experts multimodal model with 49 billion parameters active per token. The work spans breakable CUDA graphs, hand-written fusions, decode and prefill context parallelism, DSpark speculative decoding, NIXL disaggregated serving and host-memory KV offload. On TP8, NVFP4 decodes 1.08–1.19x faster per token than FP8 and holds about 1.5x the KV cache, while staying within 2.1 points of FP8 on multimodal and long-context evaluations.

  4. vLLM Blog

    A million-token context on 16 H200s: GLM-5.3 shards the KV cache by token and lifts per-user speed up to 3.2×

    Serving GLM-5.3's long contexts on 16 H200s runs into two limits: KV cache memory and prefill speed. On the prefill side, each prompt is split across eight GPUs by token (PCP) and the KV cache is sharded by token (DCP), so a 1M-token request needs only 6.7 GiB per GPU; on the decode side, Hybrid HiSparse keeps KV in host memory. On the AgentX benchmark, PCP8 × DCP8 prefill with HiSparse decode gives 1.7–3.2× the per-user speed of a DP8 deployment at the same throughput, and it is the only tested configuration that serves 1M-token contexts.

  5. vLLM Blog

    MiniMax-M3.1-Flash lands in vLLM on day one: NVFP4 cache lifts 256K-token concurrency from 28 to 76 sequences per deployment

    MiniMax open-sourced MiniMax-M3.1-Flash, and vLLM supports it on day one on NVIDIA Blackwell, NVIDIA Rubin and AMD MI350-series GPUs. The model is quantization-aware trained for an NVFP4 KV cache and stores its routed MoE weights in NVFP4, shrinking the checkpoint from 451.5 GB to 250.1 GB and the KV cache from 61,440 to 34,560 bytes per token. On four NVIDIA B300 GPUs with DEP4 deployment and 256K-token sequences, M3.1 fits 76 concurrent sequences where M3 with an FP8 KV cache fits 28.

  6. vLLM Blog

    vLLM models now serve on Google TPUs without code changes, as a new backend reuses the existing scheduler and KV cache

    Inferact and Google released vLLM TorchTPU, a backend that serves model definitions already in vLLM on Google Cloud TPUs without changing model code. Previously, bringing a model to TPU meant writing separate model code; this backend keeps vLLM's command line, OpenAI-compatible server, scheduler and KV cache manager, with TPU-specific work sitting underneath them. It compiles through torch.compile's TPU backend to XLA and uses custom JAX/Pallas kernels for attention, mixture-of-experts and quantized matmul.

  7. vLLM Blog

    Scaling a live vLLM service no longer means a 100-second outage: Elastic EP cuts the pause to 0.82 seconds on DeepSeek V4 Pro

    In vLLM's Elastic Expert Parallelism, resizing a deployment used to stop serving: scaling DeepSeek V4 Pro from 16 to 20 GB200 GPUs paused for 99.62 seconds. The new design splits reconfiguration into background preparation and a short commit, while NIXL EP adds or removes communication ranks in place and keeps captured CUDA graphs valid, cutting the same benchmark's pause to 0.82 seconds on scale-up and 1.96 seconds on scale-down.

  8. vLLM Blog

    A 501B sparse model gets day-one serving support: coding agents tuned it themselves, reaching 708 tokens per second per request

    Reflection's Beam is a 501B-parameter mixture-of-experts model with 23B active parameters per token, and vLLM supports it from day one with reasoning, tool calling and structured outputs working out of the box. Its first implementation was almost pure PyTorch; coding agents profiled the model, wrote kernels and benchmarked each change, cutting batch-1 decode on TP8 from 8.95 to 3.30 milliseconds per step and an 8×2K prefill from 328 to 176 milliseconds. With its DFlash2 draft head, a single request reaches 708 tokens per second on TP8 at an acceptance length of 3.7.

  9. vLLM Blog

    NVIDIA puts vision inside always-on agents: 120B parameters, 12B active, up to 5x single-GPU throughput

    NVIDIA released Nemotron 3.5 Super VL, an open multimodal model for long-running agents that folds document, image, and video understanding together with reasoning, instruction following, and tool use in one model, reducing the need to deploy separate text and vision models. It uses a 120-billion-parameter mixture-of-experts design with 12 billion active parameters, combined with a hybrid Mamba-Transformer architecture, multi-token prediction, and NVFP4 quantization, and can be post-trained with NVIDIA NeMo on an organization's own data, tools, policies, and workflows. The stated comparison is up to 5x faster throughput on agentic tasks than other open models in its size category on a single GPU, and 4x faster task completion.

  10. vLLM Blog

    vLLM runs an 816B sparse model on day one: 1M context at a quarter of the KV cache, over 960 tokens/sec on eight TPU v7 chips

    vLLM shipped inference support for Muse Spark Open on release day, a mixture-of-experts model with 816B total parameters and 42B active per token, handling a 1M context window plus native image, video and audio input. Such models usually wait weeks for community ports; here Nvidia and AMD images arrived together with a trained DSpark speculative draft model. On TPU with speculative decoding, four v7 chips exceed 650 tokens/sec and eight exceed 960 tokens/sec.