Skip to main content
Search

Saksham Gupta

1 related articles

Subscribe to updates
  1. arXiv

    Speculative decoding's draft cache drops to 4 bits: 66.20% less draft memory for a 1.36% relative acceptance loss

    Two-model speculative decoding keeps a KV cache for the draft model and another for the target model, so the draft cache adds memory pressure. The draft cache is quantized to K4V4, with keys in 32-token per-channel affine groups and values quantized per token, while the target cache and verifier stay in BF16 so compression cannot change the target model's correction rule. Across six model pairs, draft cache bytes fall 66.20% and combined cache memory falls 23.95%, while mean acceptance drops from 57.93% to 57.14%, a 1.36% relative decline.

You’re all caught up