Skip to main content
Search

# GGUF

3 related articles

Subscribe to updates
  1. arXiv

    Edge LLMs stop paying fixed width per weight: LazyWeights shrinks codes and cuts Orin token time by up to 36%

    Edge LLM weight formats spend equal bits on every integer code, although the codes are far from uniform. LazyWeights keeps llama.cpp's GGUF formats, tunes GPTQ so the codes concentrate, Huffman-codes them losslessly in 32 interleaved lanes, and decodes them on demand in a small unit between DRAM and L2, leaving GPU kernels unchanged. At 1% and 3.5% perplexity budgets, coded Phi-3-mini, Mistral-7B and SmolLM-1.7B are 5.7–31.4% smaller than the best llama.cpp format; on a Jetson Orin Nano Super, with the decoder unit, latency and energy per token fall by up to 36% and 38%.

  2. JustVugg/colibri

    colibri runs hundred-billion-parameter MoE models on ordinary PCs: experts stay on disk, and 8 GB of RAM is enough to start

    JustVugg released colibri, an open-source inference engine written in pure C with one file per model family and no GPU required. A mixture-of-experts model uses only a small part of itself for each token, so colibri keeps that part in RAM and reads the remaining experts from disk on demand. In the official example, the 744B-parameter int4 GLM-5.2 MoE model is ready in 32 seconds on CPU with 9.9 GB resident; the minimum is 8 GB of RAM, and the smallest model needs 22 GB of disk.

  3. Hugging Face

    Transformers now runs llama.cpp quants

    This Hugging Face post describes how transformers can now load GGUF quantized checkpoints directly through from_pretrained with a gguf_file argument: on Apple Silicon it reuses ggml Metal kernels distributed via the kernels library (packed-quantized weight matmul, fused normalization, flash attention, gated delta net, plus a home-grown topk for MoE routing) and trims generate overhead by dropping the redundant attention mask early and deferring the stopping check asynchronously; on a MacBook Pro M2 Max the authors measure generation speed close to llama.cpp across three checkpoints, report file-size tradeoffs for Q4_K_M, Q5_K_M and Q6_K, and state that packed inference is MPS-only with architecture coverage for Qwen3.5 dense and MoE (including compatible Qwen3.

You’re all caught up