Skip to main content
Back to timeline
LMSYS OrgSource publication:

Humming 0.1.18 lands in SGLang: at matched A16 precision, long-context prefill runs 1.07–1.23× faster than Marlin

Synopsis

Humming 0.1.18 connects low-bit quantized kernels to SGLang's MoE inference path, covering more quantization schemes and both Hopper and Blackwell GPUs, and adds a shared activation-expression interface. On DeepSeek-V4.1-Flash-1017 long-context tests, prefill at matched A16 precision is 1.07–1.23× faster than Marlin; at EP1 and A16, decode throughput ranges from 2.89% lower than Marlin at concurrency 1 to 7.65% higher at concurrency 64. These end-to-end results cover only that checkpoint and the H20-3e environment.

Source-provided article image: Humming in SGLang: Flexible Low‑Bit Kernels for Efficient MoE Serving
LMSYS Org

Interpretation

Low-bit MoE inference can now run through Humming's specialized kernels inside SGLang, while routing, scheduling and distributed execution stay with SGLang. Previously a low-bit checkpoint had to be matched to concrete kernels with execution paths prepared per format and hardware; Humming separates the representation of quantized data from its execution strategy, so compatible formats share components for data movement, dequantization, matrix multiplication and output processing. On DeepSeek-V4.1-Flash-1017 at matched A16 precision, Humming's prefill is 1.07–1.23× faster than Marlin across long-context inputs from 4K to 1016K tokens.

Weights are organized once into common schemas at model load, and the corresponding GEMM is then JIT-compiled at inference time for the device and shape. Configuration is split into three layers: LayerConfig describes shapes, numeric formats, quantization and storage layouts; ComputeConfig describes dense, indexed or grouped execution and accumulation semantics; TuningConfig selects tile shapes, data transfers, pipeline stages and scheduling. On eight H20-3e GPUs with TP8, prefill CP8 interleave, 16K-token chunks and DSpark block size 5, Humming's routed-expert GEMMs at matched A16 precision take 31.6–32.4% less cumulative kernel time than Marlin at 4K, 128K and 1016K.

The Hopper path keeps weights packed at four bits in global and shared memory and reconstructs FP8 operands in registers only immediately before Tensor Core computation. Each UE8M0 group scale is factored into a base scale, retained per expert and sublayer, plus a residual exponent code per group, avoiding a materialized eight-bit weight tensor in memory. For E4M3 this path supports an exponent span of 11 without scale-range adjustment; a 0.1.18 load-time check at TP8/EP1 found scale-range adjustment in 0% of valid target groups and 11.27% of valid draft groups, mainly because padded scales affect the draft's range selection.

Perspective

The demonstrated scope is a DeepSeek-V4.1-Flash-1017 serving test on eight H20-3e GPUs with TP8, prefill CP8 interleave, 16K-token chunks and DSpark block size 5, where target and draft use the same MoE backend, dense layers keep their native FP8 path and expert weights are MXFP4. The Blackwell section describes architecture support only; all measured results are on Hopper H20-3e. Planned work includes autotuning, extension to Rubin Locality Domains, T-Head's Zhenwu M890 and more CUDA-compatible accelerators, and end-to-end coverage of B300 deployments.

The report states that the quality results describe only the evaluated configurations and do not establish general accuracy equivalence or quality at the 1016K performance point. In the EP8 comparison, FlashInfer has lower median TTFT at all six measured lengths, with the gap narrowing as inputs grow longer. Humming AFP8 at EP8 shows mean throughput 4.25% higher at concurrency 1 and 0.34% higher at concurrency 32 than the FlashInfer AFP8 reference, and 1.51% lower at concurrency 64. The environment pins Humming 0.1.18, and 0.1.19 has not been validated for these results.