Skip to main content
Search

# Weight Quantization

2 related articles

Subscribe to updates
  1. LMSYS Org

    Humming 0.1.18 lands in SGLang: at matched A16 precision, long-context prefill runs 1.07–1.23× faster than Marlin

    Humming 0.1.18 connects low-bit quantized kernels to SGLang's MoE inference path, covering more quantization schemes and both Hopper and Blackwell GPUs, and adds a shared activation-expression interface. On DeepSeek-V4.1-Flash-1017 long-context tests, prefill at matched A16 precision is 1.07–1.23× faster than Marlin; at EP1 and A16, decode throughput ranges from 2.89% lower than Marlin at concurrency 1 to 7.65% higher at concurrency 64. These end-to-end results cover only that checkpoint and the H20-3e environment.

  2. arXiv

    Quantized tensor kernels stop being hand-written per format: one algorithm and one schedule library generate 347 kernels, including 38 weight–activation pairs GGML lacks

    Tensor libraries such as GGML hand-write a kernel for every weight format, activation format and instruction set; on the Arm CPU the authors tested, only 8 of the 54 weight–activation pairs GGML runs have a tiled matrix-multiplication kernel. The authors extend Halide's separation of algorithm and schedule to the representation of values: a codec describes a quantized format, and a new approximate_by directive turns its encoder and decoder into ordinary schedulable stages. One matrix-product algorithm and one schedule library generate 347 kernels covering 31 GGML and KleidiAI formats, including 38 weight–activation pairs for which GGML has no kernel.

You’re all caught up