Skip to main content
Search

# Speculative Decoding

44 related articles

Subscribe to updates
  1. arXiv

    MMFlash reweights visually dependent tokens in diffusion drafting, lifting VLM decoding to 3.00–3.05× on image benchmarks

    MMFlash extends diffusion-style speculative decoding to vision-language models by measuring how much each token depends on the image and upweighting those tokens during training. The measurement shows DFlash drafter acceptance falling from 46% in the lowest visual-dependence quartile to 22% in the highest, a gap that persists after multimodal training. On the Qwen3.5 series, MMFlash reaches average greedy-decoding speedups of 3.00–3.05× across seven image benchmarks, 10.3–10.9% above DFlash, and 2.33–2.71× on long-video benchmarks with sparse attention and light fine-tuning.

  2. LMSYS Org

    Muse Spark Open decoding passes 1,000 tokens per second: one resident kernel per layer lifts throughput from 517 to 1,109 tok/s

    Muse Spark Open decoding rises from a 517 tok/s baseline to 1,109 tok/s by fusing each layer's attention and MoE into a single kernel that stays resident on the GPU, overlapping weight loads, tensor-core work and cross-GPU communication. Previously SGLang launched about 1,450 kernels per decoding step, and the GPU drained and refilled between them, leaving some SMs and threads idle. The figure was measured on Blackwell GPUs at batch size one with DFlash V2 speculative decoding and has not yet been extended to larger batches.

  3. arXiv

    Speculative decoding drafters get a new training target: one D-REAL epoch lifts DFlash2 serving throughput 9.4% on GSM8K

    Speculative decoding drafters are usually trained with token-level cross-entropy, yet at fixed overhead end-to-end speedup is governed by acceptance length, the number of draft tokens the target model accepts consecutively. D-REAL writes that acceptance length as an expectation along the drafter's own sampled trajectories and builds a differentiable surrogate from acceptance ratios with stop-gradient denominators, giving an unbiased gradient estimator. Across ten benchmarks on the parallel drafters DSpark and DFlash2, one post-training epoch raises macro-average acceptance length from 5.07 to 5.13 and from 4.88 to 5.10; in concurrent GSM8K serving, DFlash2 output throughput rises 9.4%.

  4. Modal Blog

    Sakana Namazu goes live in NVFP4: the trillion-parameter model keeps its Japanese, and a custom drafter cuts p99 latency from 68s to 52s

    Sakana AI and Modal put Sakana Namazu, a roughly 1-trillion-parameter mixture-of-experts model, into production API service at NVFP4 precision. Compression was applied only to the routed experts that hold most of the parameters, while attention, the router and the shared experts stayed at BF16, and calibration used Japanese-heavy conversations so the model matched its production checkpoint on Japanese instruction following. A custom draft model trained on more than 30 billion raw tokens raised speculative token acceptance by 25% over a generic drafter; a second drafter retrained on about 1 million real requests lifted average acceptance length from 3.51 to 5.01 and cut p99 latency from 68s to 52s across 1,000 held-out requests.

  5. vLLM Blog

    vLLM trains a DSpark drafter for Kimi K3: math-reasoning interactivity rises from about 110 to about 435 tok/s/user

    The vLLM Project's Speculators training library now supports Kimi K3, a 2.8-trillion-parameter model, and produced a five-layer, five-billion-parameter DSpark draft model that combines parallel block drafting with sequential correction. On math-reasoning workloads, single-stream interactivity rises from about 110 to about 435 tok/s/user, output throughput reaches up to about 3.5x at matched interactivity under concurrent load, and the macro-average acceptance length across nine evaluation domains is 4.11 tokens per verification round.

  6. PyTorch

    Meta's 816B sparse model now runs on one four-chip TPU host: a single whole-step kernel cuts per-token time 1.94× for one request

    Meta's open-weights mixture-of-experts model Muse Spark Open has 816B total parameters and about 42B active per token; serving it on TPU v7 previously took 2.9× the time needed just to stream its weights from HBM for one verify step. SGLang-TorchTPU now writes all 62 layers of a decode or verify step as one Pallas program, issuing weight, KV and cross-device transfers ahead of the compute that needs them. Per output token, a single request drops from 3.78 ms to 1.95 ms, two concurrent requests gain 1.85×, and the gain narrows as concurrency grows.

  7. arXiv

    Speculative decoding pays off more than predicted on sparse MoE models: fine-grained models peak near batch size 1

    Prior cost models estimated speculative decoding from the expert-weight bytes activated during verification, predicting a concave speedup curve with limited gains at low batch size for sparse MoE targets. Across 10 sparse MoE models from 30B to 1T parameters and 8 to 512 experts, fine-grained models instead often peak near batch size 1. Two effects explain it: routing overlap across adjacent draft positions cuts the unique experts loaded during verification by 21% to 34% relative to an independence baseline, and GPU execution dynamics, namely bandwidth-utilization gaps between MoE and attention kernels plus L2 reuse, decouple runtime from expert bytes.

  8. LMSYS Org

    After DeepSeek open-sourced DeepInfer, SGLang matches or beats it on its own benchmarks, up to 1.04x prefill throughput

    DeepSeek open-sourced DeepInfer, the inference engine behind its own service, along with its kernels; SGLang then brought the Mega kernel series, DSpark speculative decoding, SWA bounded replay and both Engram placements into its mainline and compared against DeepInfer on the same hardware using DeepInfer's own benchmarks. On 4x GB300 and 4x B300, SGLang reaches up to 1.04x DeepInfer's prefill throughput, within 1% at the remaining points; on 64x GB300 its decode step is 1.1% to 7.4% shorter from 16 to 256 requests per GPU and on par at 8. SGLang also ships the KV cache system, prefill/decode disaggregation and cross-replica load balancing that DeepInfer's release notes list as not included.

  9. arXiv

    FactorSpec turns relational constraints into speculative drafts, accepting up to 10.5 tokens per verify step while every record stays valid

    In structured output, relations among a record's fields could determine many tokens, yet serving engines draft from the prompt or a learned head and grammar masks enforce only syntax, so models often violate those relations. FactorSpec compiles a JSON Schema and its relations into a factor graph, treats tokens the constraint uniquely determines as certain drafts through an exact feasibility check, and picks a verify width per request. In SGLang its drafts accept up to 10.5 tokens per verify step on trip planning, up to 1.9 those of the fastest n-gram drafter shape, and decode up to 1.43 faster than speculators tuned per load, at a grammar mask's cost.

  10. arXiv

    A watermark stops flipping coins between keys: merging two keys detects 18 to 22 points more under speculative decoding

    LLM watermarks must keep output diversity under a fixed key and must not lower draft acceptance under speculative decoding, and the two have been hard to hold together. TextSeal2 merges two keys' Gaussian values into one virtual key and couples it to fresh noise through a Gaussian copula, so the detector reads full signal on every token; under speculative decoding it decides each round's last token with one more exponential race on the leftovers of the draft races. At equal diversity on three models it detects 8 to 22 percentage points more 100-token answers than TextSeal, and under speculative decoding 18 and 22 points more 200-token answers than TextSeal and textGrain, at unchanged acceptance.