KV-cache compression stops choosing between layers and tokens: MultiAlloc reaches 46.35 average on Llama-3-8B under the same memory budget
Synopsis
During long-context inference, the KV cache dominates GPU memory, yet existing quantization methods either allocate precision by layer or preserve tokens at high precision, and rarely split a fixed memory budget between the two. MultiAlloc first profiles offline how sensitive each layer is to quantization and how important each token is to attention, then decides under the budget how much resource goes to layer precision and how much to high-precision tokens, fixing the result as a static policy with no search at inference. Across Llama-3-8B-Instruct, Mistral-7B, DeepSeek-Qwen-7B/14B and QwQ-32B, its average score exceeds OTT, PM-KVQ and KVmix under matched KV-cache budgets, for example 46.35 on Llama-3-8B against an FP16 reference of 46.17.
Interpretation
KV-cache quantization is recast as a precision-allocation problem under a fixed memory budget: limited quantization resources must be divided between raising the bit-width of some layers and keeping a few tokens at high precision, rather than optimizing one of the two. Earlier layer-granularity methods assign mixed precision by layer or block sensitivity but treat all tokens within a layer alike; token-granularity methods trace outlier or salient tokens but ignore differences in layer sensitivity; multi-granularity methods such as KVmix combine layer importance with pivotal context selection, yet their token-level adaptation mainly concerns context selection rather than allocating precision by token importance. On five models, Llama-3-8B-Instruct, Mistral-7B-Instruct-v0.3, DeepSeek-R1-Distill-Qwen-7B/14B and QwQ-32B, all compressed methods are compared under the same KV-cache budget with identical prompts and greedy decoding, and MultiAlloc attains the highest average among compressed methods on all five.
All allocation decisions are made offline, so inference only executes a fixed precision policy and needs no allocation search during generation. Jointly searching layer bit-width and token-preservation configurations spans a large space, and evaluating downstream performance for each candidate would require repeated end-to-end inference and is prohibitively expensive, so lightweight profiling surrogates supply relative preferences among candidates. Layer sensitivity is estimated from the magnitude of first-order Taylor terms of quantization perturbation on calibration data, and token importance uses key-vector activation magnitude as a query-independent proxy, with the top tokens per group kept at high precision; on Llama-3-8B-Instruct at a 423 MB budget, removing either dimension lowers LongBench-v2 macro accuracy, with layer-only at 25.58, token-only at 24.88 and both combined at 25.85.
On Llama-3-8B-Instruct at a 423 MB budget, the profiled allocation performs close to a manual 65/35 split, indicating that the automatic choice lands near that ratio for this setting. A manual split must predefine how much budget goes to layer precision versus token preservation, while the suitable ratio differs across models, tasks and budgets. On GSM8K the manual 65/35, 50/50 and 35/65 splits score 80.29, 77.79 and 77.41, against 80.36 for the profiled allocation; on LongBench-v2 they score 25.59, 25.83 and 25.56, against 25.85.
Perspective
The demonstrated scope is reasoning and long-context evaluation of five instruction-tuned models from 7B to 32B on GSM8K, GPQA-Diamond and LongBench-v2, with all compressed methods compared under the same KV-cache budget, identical prompts, tokenization, stopping criteria and greedy decoding, and throughput measured on a single A100 at batch size one.
The profiling stage adds preprocessing overhead and requires representative calibration data, and each model and task uses its own calibration set; the authors list lighter profiling and better policy generalization as future work. Gains over existing methods are smaller on the 14B and 32B models because their compressed performance already approaches FP16, and the authors report closing 48.2% and 52.1% of those gaps respectively.
