Long-context inference stops re-streaming KV cache: MCF replicates early blocks, cutting 16-node TTFT to 0.62 s
Synopsis
Prefill in long-context inference repeatedly moves historical KV cache, and MCF turns KV layout into a stage-aware runtime schedule: early in prefill, when memory headroom exists, it temporarily replicates highly reused KV blocks, drains them as memory pressure rises, and converges to a sharded layout before decode. On a cluster of four KR260 FPGAs at 16K prefill with 16 logical nodes, MCF reaches 0.62 s TTFT, versus 1.91 s for Ring Attention and 1.81 s for Zigzag Attention.
Interpretation
KV-cache layout in long-context inference is no longer fixed: MCF temporarily replicates KV blocks across nodes early in prefill and retires every replica before prefill ends, leaving only the owner shard. Earlier context-parallel designs rely on static KV sharding and fixed communication patterns; Ring Attention circulates historical KV blocks around a ring, and Zigzag Attention only reorders sequence blocks to balance load, so neither changes layout across prefill stages. At 16K prefill with 16 logical nodes, MCF reports 0.62 s TTFT against 1.91 s for Ring Attention and 1.81 s for Zigzag Attention; MCF transfers 5.12 GiB, above the roughly 4.40 GiB of the two streaming baselines.
The MARS scheduler decides how many copies each KV block gets from memory headroom and expected future reuse, and the replica factor falls monotonically as prefill proceeds until it returns to one. MARS treats KV placement as a runtime scheduling problem rather than a one-time sharding policy, and manages reclamation through replica lifecycle states. In the 16K-prefill, 16-logical-node memory measurement, MARS uses more memory than an owner-only baseline during early and middle prefill, decreases as prefill progresses, and converges to the owner-only layout at the end.
During decode, KV shards stay stationary and only query tensors and compact partial attention states are routed, so inter-node exchange no longer grows with context length. KV-centric dissemination in prefill switches to query-centric routing in decode, with both stages sharing one FPGA NIC datapath. On a cluster of four KR260 nodes running INT4-quantized Qwen2.5-1.5B, end-to-end decode throughput is 17.3 token/s.
Perspective
The demonstrated setting is a cluster of AMD-Xilinx KR260 FPGA boards with INT4-quantized Qwen2.5-1.5B, prefill sequence lengths swept up to 16K and node counts from 2 to 16; decode throughput is measured on a four-node cluster. The intended reader is an engineer working on distributed inference systems and FPGA acceleration.
MCF's communication volume is slightly higher than the streaming baselines at larger node counts, which the authors describe as trading extra replication and prefetch traffic for lower exposed latency on the prefill critical path. On short sequences, replication and scheduling overhead can offset the gain. Power efficiency is not the highest among the listed systems, which the authors attribute to integrating the packetization buffer and NIC communication datapath.
