Skip to main content
Search

# Post-Training Quantization

16 related articles

Subscribe to updates
  1. arXiv

    SAM 3 at 4-bit weights stops grabbing the wrong object: SPQ lifts RefCOCO hit rate from 42.1 to 72.5

    Quantizing SAM 3's image-encoder weights to 4 bits while keeping activations at 8 bits produces prompt-alignment drift: masks look plausible but land on the wrong object or lose the target across video frames. SPQ gives more calibration weight to parameters that affect text-concept scores, constrains feature error along low-variance residual directions, and folds a mean-error correction estimated from paired video rollouts into existing biases. Under W4A8, RefCOCO Pr@0.5 rises from 42.1 for PTQ4SAM to 72.5 while retaining 54.8 COCO mask AP, with no added inference operators.

  2. arXiv

    Dory1.0-4B moves inference compute into a looped block: one checkpoint lifts its 12-benchmark average from 35.2 to 40.6

    Dory1.0-4B is a 4-billion-parameter looped language model whose single checkpoint varies how many times a shared recurrent block runs at inference, letting one model serve different compute budgets. It was pretrained on 6 trillion tokens, the first 4.5 trillion in NVFP4, then extended to a 256K context and post-trained with SFT and RLVR. On the equal-weight average of 12 benchmarks it rises from 35.2 at one loop to 39.7 at two and 40.6 at four, and with two or more loops it beats Qwen3.5-9B, Gemma 4 12B and Gemma 4 26B-A4B on that average.

  3. aclanthology.org

    FOM-UL unlearns by editing only a few layers: under 4-bit quantization its residual memorization stays below whole-model baselines with higher retain utility

    Unlearning a large language model no longer requires diffuse updates across the whole model. FOM-UL scores each transformer layer by the ratio of its forget-set gradient norm to its retain-set gradient norm, updates only the few high-scoring layers, and adds layers one at a time when forgetting falls short. Across TOFU, KnowUnDo and MUSE-style evaluations it leaves less residual memorization than GA, NPO and KLD baselines while keeping retain-set utility close to the vanilla model, and it holds a favorable forgetting-utility trade-off under 8-bit and 4-bit post-training quantization, though residual memorization does rise under 4-bit relative to full precision.

  4. aclanthology.org

    Multimodal quantization stops text from drowning out vision: one shared transform reconciles both gradients, 66.51 average on the 32B model at W4A6

    When a multimodal large language model is quantized to low bit-widths, text and vision tokens must learn one shared channel-wise transform, and their gradients can point in opposite directions or differ wildly in magnitude. MCTQ splits the reconstruction objective by modality, then combines the two gradients into a single update direction through a minimum-norm convex combination, keeping one shared transform and adding no inference overhead. On Qwen2.5-VL-7B at W4A6 it reaches 66.26 average accuracy, 1.30 points above OmniQuant; Qwen2.5-VL-32B reaches 66.51 in the same setting.

  5. arXiv

    Two-bit lookup tables shrink DeepSeek's 203 GB memory to 51 GB and cut prefill from 64.66 s to 13.27 s

    Conditional memory lets a language model add capacity through lookup, but the whole embedding table must stay resident; DeepSeek's table alone is 203 GB. Conditional Program stores each row as a short code plus a local scale and reconstructs it with a Gaussian trellis shared across the table, decoding only the rows an input requests. At two bits, storage falls roughly four- to eightfold, most reported task scores stay close to their originals, and output KL is 41–63% lower than 2-bit round-to-nearest.

  6. arXiv

    A $5 chip runs open-vocabulary speech recognition: 13M-parameter model stays in flash, with 1.8–2.8 fewer errors than the vendor recognizer on the same chip

    Oído runs open-vocabulary English speech recognition entirely on a $5 ESP32-S3, a chip with no neural accelerator. Such microcontrollers previously managed only keyword spotting or a fixed command list, while a 13M-parameter Conformer-CTC occupies 14.0 MB at int8, too large for the 512 KB of on-chip SRAM and only partly placeable in PSRAM. The engine schedules execution around flash and PSRAM bandwidth, reaching 3.70%/8.23% word error rate on LibriSpeech test-clean/test-other, within 0.12 points of full precision and 1.8–2.8 fewer errors than the vendor recognizer on the same chip; measured on a board it runs at 1.97 times the audio duration on one core, not yet real time.

  7. arXiv

    4-bit inference drops calibration data: weight-error minimization doubles as activation scaling, matching SmoothQuant on MX and NV formats

    Low-bit inference needs both weights and activations quantized, yet activation quantization usually depends on calibration data. DualQuant casts weight-reconstruction error minimization as a format-constrained non-convex problem solved by block coordinate descent with closed-form updates, and the resulting column scales double as activation rescalings. Across four 4-bit formats on Llama and Qwen it matches or beats calibration-based SmoothQuant without calibration data, clearly outperforming calibration-free baselines, with the largest margins on MX and NV block formats.

  8. arXiv

    DRQ minimizes worst-case reconstruction loss on a fixed quantization grid, improving perplexity and downstream accuracy for LLMs quantized by six PTQ methods without adding inference overhead

    The authors show that lower reconstruction loss on calibration data in weight-only PTQ does not guarantee better model performance and can even degrade it on the same calibration data; they propose DRQ, which minimizes worst-case reconstruction loss over a constrained set of input activation distributions by updating only the integer codes of quantized weights while keeping bit width, grouping, scales, and zero points fixed, improving perplexity and average task accuracy across six PTQ methods and both dense and MoE models without adding inference overhead.

  9. arXiv

    OSFP4 jointly optimizes diagonal smoothing and block scales, reaching 77.03 average W4A4 accuracy on Llama-3.1-8B-Instruct while retaining about 94-97% of vendor NVFP4 prefill throughput

    The work introduces OSFP4: by analyzing a multiplicatively dithered FP4 quantizer it obtains a differentiable approximation of the matrix-product quantization error, then jointly optimizes a diagonal smoothing matrix and weight block scales per linear projection, with separate losses for RTN and GPTQ-style SIC rounding; on Llama-3.1-8B-Instruct the W4A4 configuration attains the highest average accuracy among the evaluated methods while retaining roughly 94-97% of vendor NVFP4 prefill throughput.

  10. arXiv

    SoloQ quantizes diffusion language models to 4 bits without calibration, holding accuracy on LLaDA, Dream, Fast-dLLM v2 and Nemotron-Labs-Diffusion while beating calibration-based baselines

    The authors present SoloQ, which uses a K-RPBH rotation to map weights and activations into a predictable near-Gaussian basis plus a scale-only bias correction for calibration-free 4-bit quantization, and quantizes the KV cache at block commit for block-diffusion models; across LLaDA-Base, LLaDA-1.5, Dream-7B, Fast-dLLM v2 and Nemotron-Labs-Diffusion, SoloQ-C and SoloQ-N retain accuracy under W4A4 and W4A4KV4 and exceed calibration-based baselines such as DLLMQuant++, STaR-Quant and FAIR-Calib on average, with the NVFP4 path also cutting peak memory and speeding up end-to-end inference.