Skip to main content
Search

# Metal kernels

2 related articles

Subscribe to updates
  1. PyTorch 2.15 on the Mac: MoE decoding 5–6.5× faster and video VAE memory down from 270 GiB to 8 GiB

    PyTorch 2.15 brings FP8 tensors, a native grouped matmul and a batch of dedicated Metal kernels to the Mac's MPS backend, making local LLM decoding and video generation faster and lighter on memory. Previously many MPS operators went through MPSGraph, MoE decoding copied group offsets to the CPU and looped over groups, and tiled video VAE decoding pushed driver memory to 270 GiB. On an M3 Ultra, Qwen3.6-35B-A3B decoding rises from 9.4 to 61.3 tokens/s, and Wan2.2 VAE decoding peaks near 8 GiB.

  2. Hugging Face

    Transformers now runs llama.cpp quants

    This Hugging Face post describes how transformers can now load GGUF quantized checkpoints directly through from_pretrained with a gguf_file argument: on Apple Silicon it reuses ggml Metal kernels distributed via the kernels library (packed-quantized weight matmul, fused normalization, flash attention, gated delta net, plus a home-grown topk for MoE routing) and trims generate overhead by dropping the redundant attention mask early and deferring the stopping check asynchronously; on a MacBook Pro M2 Max the authors measure generation speed close to llama.cpp across three checkpoints, report file-size tradeoffs for Q4_K_M, Q5_K_M and Q6_K, and state that packed inference is MPS-only with architecture coverage for Qwen3.5 dense and MoE (including compatible Qwen3.

You’re all caught up