MILO cuts many-shot context KV cache by up to 50% via block-wise low-rank compression, lifting Qwen2.5 throughput 1.8x with near-flat classification and reasoning performance
The work proposes MILO, a compression framework that exploits low-rank redundancy in many-shot contexts by compressing the KV cache at block granularity and dynamically allocating rank budgets based on information entropy, achieving up to 50% KV cache memory reduction and 1.8x throughput improvement on Qwen2.5 models with negligible performance degradation on classification and reasoning benchmarks, significantly outperforming prior baselines.