Skip to main content
Search

# Long-Context LLM Inference

3 related articles

Subscribe to updates
  1. arXiv

    Long-context inference stops re-streaming KV cache: MCF replicates early blocks, cutting 16-node TTFT to 0.62 s

    Prefill in long-context inference repeatedly moves historical KV cache, and MCF turns KV layout into a stage-aware runtime schedule: early in prefill, when memory headroom exists, it temporarily replicates highly reused KV blocks, drains them as memory pressure rises, and converges to a sharded layout before decode. On a cluster of four KR260 FPGAs at 16K prefill with 16 logical nodes, MCF reaches 0.62 s TTFT, versus 1.91 s for Ring Attention and 1.81 s for Zigzag Attention.

  2. arXiv

    A small model reads first: SkimPrefill cuts prefill time 3.7 to 11.6 times on long image-text inputs

    On long interleaved image-text inputs of about 100K tokens, vision-language models spend most of the wait for the first output token in prefill. SkimPrefill has Qwen3.5-0.8B read the whole input first at half image resolution, stopping at layer 7, then scores every image and paragraph with one attention head of that layer and hands the top whole units to the target as an ordinary shorter prompt. Keeping 10% of the input tokens on MMLongBench at 128K, it speeds up the prefill of Qwen3.5-9B, Gemma-4-12B and Qwen3-VL-32B by 3.7, 11.6 and 8.2 while holding their average accuracy.

  3. arXiv

    Inference accelerators can learn while serving: LWS hides learning in prediction's idle compute and keeps 99.3% of the quality benefit

    When a long-context model keeps learning during inference, it pays for backward computation and weight updates on top of prediction, while today's inference accelerators assume fixed weights. LWS schedules that learning onto matrix capacity left idle during prediction, uses per-layer weight versioning so prediction can read slightly older weights, and lets long learning kernels pause and resume across fragmented idle windows. On modeled datacenter and edge NPUs, TTT-E2E 3B at 128K tokens sustains prediction-only throughput while preserving 99.3% of fully fresh learning's quality benefit.

You’re all caught up