Skip to main content
Search

# Megakernel

2 related articles

Subscribe to updates
  1. LMSYS Org

    Muse Spark Open decoding passes 1,000 tokens per second: one resident kernel per layer lifts throughput from 517 to 1,109 tok/s

    Muse Spark Open decoding rises from a 517 tok/s baseline to 1,109 tok/s by fusing each layer's attention and MoE into a single kernel that stays resident on the GPU, overlapping weight loads, tensor-core work and cross-GPU communication. Previously SGLang launched about 1,450 kernels per decoding step, and the GPU drained and refilled between them, leaving some SMs and threads idle. The figure was measured on Blackwell GPUs at batch size one with DFlash V2 speculative decoding and has not yet been extended to larger batches.

  2. PyTorch

    Meta's 816B sparse model now runs on one four-chip TPU host: a single whole-step kernel cuts per-token time 1.94× for one request

    Meta's open-weights mixture-of-experts model Muse Spark Open has 816B total parameters and about 42B active per token; serving it on TPU v7 previously took 2.9× the time needed just to stream its weights from HBM for one verify step. SGLang-TorchTPU now writes all 62 layers of a decode or verify step as one Pallas program, issuing weight, KV and cross-device transfers ahead of the compute that needs them. Per output token, a single request drops from 3.78 ms to 1.95 ms, two concurrent requests gain 1.85×, and the gain narrows as concurrency grows.

You’re all caught up