4-bit inference drops calibration data: weight-error minimization doubles as activation scaling, matching SmoothQuant on MX and NV formats
Synopsis
Low-bit inference needs both weights and activations quantized, yet activation quantization usually depends on calibration data. DualQuant casts weight-reconstruction error minimization as a format-constrained non-convex problem solved by block coordinate descent with closed-form updates, and the resulting column scales double as activation rescalings. Across four 4-bit formats on Llama and Qwen it matches or beats calibration-based SmoothQuant without calibration data, clearly outperforming calibration-free baselines, with the largest margins on MX and NV block formats.
Interpretation
Low-bit inference can now quantize weights and activations together without any calibration data. Activation quantization previously depended on calibration data because activation tensors carry systematic large-magnitude outliers in specific channels, and those outliers dominate the quantization grid, leaving ordinary channels very few effective bits. Earlier calibration-free routes either quantized weights only and left activations in BF16, or, like QuaRot, used rotations to suppress outliers without directly targeting activation-quantization error; the methods that handle activations effectively, SmoothQuant and SpinQuant, both need representative calibration data, bringing distribution sensitivity and data-access overhead. Across the Llama and Qwen families and four 4-bit formats, INT4, MXINT4, MXFP4 and NVFP4, DualQuant matches or surpasses calibration-based SmoothQuant on most models and formats and clearly outperforms the calibration-free QuaRot and SINQ+; SINQ+ diverges completely on NVFP4, and QuaRot diverges there as well.
The approach minimizes weight-reconstruction error, and the column scales it produces double as activation rescalings. Those column scales come from the static weight matrix itself rather than from activation statistics, so no calibration forward passes, Hessian, or inner optimization loop are needed. The optimization is written as a non-differentiable weight-reconstruction problem; variable splitting gives each scale an inner copy controlling grid assignment and an outer copy controlling reconstruction, and block coordinate descent alternates between them with a closed-form solution per subproblem, projecting scales onto the target format's admissible set after each update.
The weight-derived column scales give a real, separable improvement to activation quantization, not merely a by-product of lower weight error. With quantized weights held fixed and only the presence of the column scaling varied, activation quantization signal-to-quantization-noise ratio improves measurably. On the projections of block 0 of Llama-3.1-8B, applying the column scale raises activation SQNR by about 0.33 dB on average and about 0.54 dB at the block output, with the largest gains on k_proj and v_proj, the projections where activation outliers are most pronounced.
Perspective
The report covers calibration-free joint 4-bit weight-and-activation quantization, evaluated on the Llama and Qwen families for perplexity on WikiText-2 and C4 and zero-shot accuracy on MMLU, HellaSwag, PIQA, ARC-Easy and WinoGrande, across INT4, MXINT4, MXFP4 and NVFP4. Deployment validation is limited to a packed INT4 group-wise W4A4 path for Llama-3.1-8B on a single RTX 4090, with embedding, lm_head and normalization layers kept in BF16. It suits researchers and engineers working on low-bit inference deployment and quantization algorithms.
The correlation between the column scales and SmoothQuant's calibration-derived scales varies with migration strength, aligning near-perfectly when the scale is purely weight-derived, but the correlation is descriptive and does not imply equivalent quantization behavior. The SINQ+ results on MX and NV formats come from the authors' own extension, not validated by the original method's authors, and should be treated as best-effort baselines. At 8 bits there is limited room for improvement over BF16 for any calibration-free route.
