ProQuant pushes multimodal LLMs to 4-bit weights and activations, holding Qwen3-VL-2B within about one point of full precision
Synopsis
Small multimodal LLMs for edge devices lose accuracy sharply under low-bit quantization. ProQuant splits quantization-aware training into two steps: it first regenerates responses for open-source multimodal data with the target model so fine-tuning data match the pretrained distribution, then quantizes vision-sensitive layers first and gradually enables text-sensitive layers. On Qwen3-VL-2B-Instruct at W4A8, average accuracy stays about one point below full precision; at W4A4 it reaches 59.83, while the PTQ baseline MBQ scores 6.37.
Interpretation
ProQuant brings 2B to 8B multimodal LLMs close to full-precision behavior under W4A8 weight-activation quantization and keeps usable multimodal ability at W4A4. Post-training quantization was near-lossless at moderate bit-widths such as W8A8, but accuracy dropped markedly at W4A8 and W4A4, with 2B and 4B models suited to edge deployment degrading most. On Qwen3-VL-2B-Instruct, ProQuant reaches 72.54 average accuracy at W4A8 against 73.63 at full precision; at W4A4 ProQuant averages 59.83, the PTQ baseline MBQ 6.37, and LLM-QAT 27.29 with 8-bit down_proj activations retained.
Training proceeds in two stages: first correcting where the fine-tuning data come from, then opening quantization layer by layer. Quantization-aware training that recovers multimodal understanding and language modeling jointly gave suboptimal results, and open-source multimodal labels differ in distribution from the target model's own outputs. On Qwen3-VL-2B-Instruct, self-relabeling raises average accuracy by 1.00 and 3.36 at W4A8 and W4A4, progressive block activation by 0.67 and 3.02, and the two together by 1.44 and 6.07.
A vision-sensitivity factor ranks layers in the backbone and decides which are quantized first. At iteration zero the quantized baseline still held a linguistic prior of 47.33% on HellaSwag but collapsed to 0% multimodal understanding on AI2D, showing cross-modal alignment is more vulnerable to quantization noise than language generation. Across the 28 decoding layers of Qwen3-VL-2B-Instruct, vision-sensitive blocks concentrate in the middle layers and text-sensitive blocks in shallow and deep layers; the vision-first schedule recovers multimodal understanding faster at nearly all checkpoints and overtakes the text-first schedule by 4K iterations.
Perspective
The demonstrated scope is four backbones, Qwen2-VL-2B, Qwen2.5-VL-3B, Qwen3-VL-2B and Qwen3-VL-8B, with training data adapted from LLaVA-680K and evaluation on AI2D, ChartQA, DocVQA, MMMU, ScienceQA, SeedBench, TextVQA and MME. The hardware part measures on-chip latency and average power on a TSMC 22 nm ASIC, together with off-chip DRAM simulation and a cycle-accurate accelerator model, with the boundary covering accelerator and DRAM accesses.
Keeping down_proj activations at 8 bits in mixed precision helps by very different amounts across backbones, +1.44, +9.39 and +3.21 average points, which the authors call preliminary evidence and leave for a systematic analysis. At W8A8 ProQuant is comparable to strong PTQ baselines, so the extra training cost buys little at that bit-width. The specific prefill speedup and energy reduction figures are not given in the main text.
