Skip to main content
Back to timeline
arXivSource publication:

Speculative decoding's draft cache drops to 4 bits: 66.20% less draft memory for a 1.36% relative acceptance loss

Synopsis

Two-model speculative decoding keeps a KV cache for the draft model and another for the target model, so the draft cache adds memory pressure. The draft cache is quantized to K4V4, with keys in 32-token per-channel affine groups and values quantized per token, while the target cache and verifier stay in BF16 so compression cannot change the target model's correction rule. Across six model pairs, draft cache bytes fall 66.20% and combined cache memory falls 23.95%, while mean acceptance drops from 57.93% to 57.14%, a 1.36% relative decline.

Interpretation

The extra memory of speculative decoding can be cut on the draft side alone: the draft cache is compressed to 4-bit keys and 4-bit values while the target model's cache and verifier remain in BF16. Standard speculative decoding stores both a target cache and a draft cache, and memory pressure grows with context length and batch size; earlier KV cache quantization mostly targeted single-model inference, with precision allocation often set heuristically. Across six target/draft pairs spanning Qwen2.5, Qwen3, Llama 3, OLMo 2 and SmolLM2, K4V4 reduces draft cache bytes by 66.20% and combined target-plus-draft memory by 23.95%, with per-pair combined savings from 11.30% to 33.17%.

The compression costs almost no draft quality: mean acceptance across model pairs falls from 57.93% to 57.14%, a 1.36% relative drop. Acceptance is the fraction of proposed draft tokens the target accepts, and it sets how many expensive target forward passes speculative decoding avoids; the prior worry was that keys are more noise-sensitive, so lowering key precision would visibly cut acceptance. On WikiText-2 the K4V4 draft has KL 0.00730 from the BF16 draft and 95.99% top-1 agreement; on the Qwen2.5 3B/1.5B pair with 256-token continuations the acceptance change interval includes zero, and under the sequential verifier outputs match target-only greedy decoding.

Which of keys or values deserves more bits depends on the quantizer's layout, not on attention intrinsically demanding higher key precision. With a naive per-token symmetric quantizer, keys look far more sensitive than values: at equal mean bits, K8V4 beats K4V8 by 10.87 percentage points on average. With keys grouped per channel over 32 tokens, that advantage disappears and every within-pair gap falls below 1 percentage point; in a controlled Qwen test that changes only the key quantizer, the K8V4 versus K4V8 gap shifts accordingly and the K4V2 versus K2V4 contrast reverses sign.

Perspective

The results cover six target/draft model pairs at 1K context and the Qwen2.5 3B/1.5B pair at 16K and 32K context, all with greedy decoding at batch size one; at 32K the packed K4V4 cache stores 260.2 MiB per sequence versus 896.0 MiB for BF16. A reader interested in serving gains should wait for an exact batched verifier, streaming prefill and fused packed-cache attention kernels.

The longest-context experiments use fewer prompts than the 1K study, and the retrieval task is partly synthetic; quality experiments use fake quantization, which measures quality but not production speed. The packed prototype is still slower than BF16 speculative decoding and does not lower peak GPU memory, the 256-token and prototype experiments cover one model pair at batch size one, and the batched verifier used for RQ1 is not yet exact. Key outliers are not measured directly, and keys and values may still differ intrinsically in settings not tested.