Skip to main content
Search

GoerTek (China)

1 related articles

Subscribe to updates
  1. arXiv

    KV-cache compression stops choosing between layers and tokens: MultiAlloc reaches 46.35 average on Llama-3-8B under the same memory budget

    During long-context inference, the KV cache dominates GPU memory, yet existing quantization methods either allocate precision by layer or preserve tokens at high precision, and rarely split a fixed memory budget between the two. MultiAlloc first profiles offline how sensitive each layer is to quantization and how important each token is to attention, then decides under the budget how much resource goes to layer precision and how much to high-precision tokens, fixing the result as a static policy with no search at inference. Across Llama-3-8B-Instruct, Mistral-7B, DeepSeek-Qwen-7B/14B and QwQ-32B, its average score exceeds OTT, PM-KVQ and KVmix under matched KV-cache budgets, for example 46.35 on Llama-3-8B against an FP16 reference of 46.17.

You’re all caught up