Skip to main content
Search

# Mixed-Precision Quantization

3 related articles

Subscribe to updates
  1. Mixed-precision quantization stops treating each layer as one bit width: A2BQ assigns bits per block by attention sensitivity, reaching 7.884 perplexity on TinyLlama

    LLM quantization usually gives an entire layer one bit width, ignoring that weight blocks within a layer differ in sensitivity to quantization noise. A2BQ scores each 128-row weight block by post-softmax attention variance from an FP16 model on a WikiText-2 calibration set, allocates 4/6/8 bits under a 5.7-bit average budget, and sets qLoRA+ LoRA ranks from layer sensitivity and assigned precision. On TinyLlama-1.1B, the attention-scope configuration with three protected layers at each boundary reaches 7.884 test perplexity at 73.11 tokens/s, while runtime GPU memory stays near FP16 because the reference backend dequantizes weights into floating-point buffers before computing.

  2. arXiv

    MoE precision pays twice: store one 4.27 bpw copy, read 2.9 bpw per workload, and stay near 4-bit formats

    Mixture-of-experts weight precision carries two separate costs, one for storage and one for decode; DuoRate stores a single 4.27 bpw copy and picks a read prefix per workload from that workload's own prefill, reaching 76.16 average accuracy at a 2.9 bpw read rate on Qwen3-30B-A3B, close to 4-bit formats.

  3. Mixed-Precision Quantization for Language Models: Techniques and Prospects

    This survey organizes the landscape of mixed-precision quantization frameworks for language models (MXPLMs): it first reviews quantization fundamentals including uniform and non-uniform quantizers, quantization granularity, and widely used post-training quantization methods, then categorizes and compares recent MXPLM frameworks by their bit allocation strategies and precision configurations across weights, activations, and key-value caches, contrasts them with earlier mixed-precision methods for deep neural networks to identify strategies that transfer and those that face challenges in the LM setting, and closes with open issues such as hardware-aware design, activation quantization, and scalable optimization for billion-parameter models.

You’re all caught up