Skip to main content
Back to timeline
arXivSource publication:

Callo drafts on invalidated KV during re-prefill: SWE-agent sessions run 1.05–1.19 times faster than vLLM on all requests at moderate load

Synopsis

When an AI agent edits its context mid-session, the prefix cache is invalidated and the request must wait for a re-prefill before decoding. Callo uses the invalidated KV still resident on the decode instance as a draft source for speculative decoding, drafting during the re-prefill and verifying against the exact KV before committing tokens. On a two-GPU H200 testbed with SWE-agent sessions across three model families, all-request speedup over vLLM is 1.05–1.19 at moderate load, and 1.14 on the main configuration.

Source-provided article image: Callo: Serving Cache-Miss Requests with Speculative Decoding on Invalidated KV under Prefill-Decode Disaggregation
arXiv

Interpretation

A single context edit in an agent session invalidates the KV cache after the edit point, so the request must wait for a re-prefill before decoding can start, and that wait delays the agent's next action. Without an edit, the serving engine only prefills newly appended tokens; an edit changes the context or positions that later tokens depend on, so the cache no longer matches and the suffix must be recomputed. On the Llama-3.3-70B main workload, 79% of requests produce an edit-induced cache miss, and a miss re-prefills 3.1k tokens at the median.

Callo does not wait for the exact KV; it lets the decode instance draft on the invalidated KV left by the previous request, turning a serial wait into overlappable speculative decoding. Once prefill and decode sit on separate GPUs, the re-prefill still lies on the affected request's critical path: other sessions are protected, but the session that missed still waits. In offline replay, the next token drafted on invalidated KV matches the exact-KV result at 98.3% of positions, and a 32-token draft window is accepted whole in 65% of miss requests, against 80% when decoding on exact KV.

While drafting proceeds, the prefill instance streams exact KV layer by layer into the decode instance, so drafting reads a state that becomes exact in place; once the exact KV has landed, one forward pass verifies the drafts and commits the accepted prefix. Earlier options either waited for the whole KV transfer before decoding or decoded directly on approximate KV and changed outputs. On the two-H200 testbed, Callo with fixed-tie greedy verification hides 49% of the miss penalty and reaches a 1.15 miss-request speedup over vLLM, while Callo-Stream, which keeps the transfer path but disables speculation, reaches 1.01.

Speculation is admitted only when the decode instance has a free slot, and it yields early when drafts are unlikely to be accepted, so the system degrades smoothly to non-speculating service as load rises. Without such scheduling, speculation competes with ordinary decoding for batch slots and slows the requests it shares a batch with. With a decode batch limit of 16, the all-request speedup rises from 1.10 at light load to 1.27 at full load; with a limit of 2 it holds at 1.10–1.12, where the transfer path supplies most of the gain.

Perspective

The demonstrated setting is a two-GPU H200 NVL testbed with one prefill instance and one decode instance, running FP8 Llama-3.3-70B and bf16 Gemma-4-31B and Mistral-Small-3.2-24B on replayed SWE-agent 1.1 sessions over SWE-bench Verified; cluster-level conclusions come from a 96-GPU simulation driven by real traces and fitted to testbed measurements.

The gain follows the share of request latency taken by the re-prefill wait: with very long outputs, as in extended reasoning, decoding dominates and the saving is a small fraction, while latency-sensitive tasks with short replies see a larger relative gain. Whether a cache computed before an adapter switch also drafts well remains open. During online drafting, later exact KV cannot revise drafts already generated, and refresh controls showed no significant acceptance gain on L5 or L10.