Skip to main content
Search

Jianjun Ou

1 related articles

Subscribe to updates
  1. arXiv

    Callo drafts on invalidated KV during re-prefill: SWE-agent sessions run 1.05–1.19 times faster than vLLM on all requests at moderate load

    When an AI agent edits its context mid-session, the prefix cache is invalidated and the request must wait for a re-prefill before decoding. Callo uses the invalidated KV still resident on the decode instance as a draft source for speculative decoding, drafting during the re-prefill and verifying against the exact KV before committing tokens. On a two-GPU H200 testbed with SWE-agent sessions across three model families, all-request speedup over vLLM is 1.05–1.19 at moderate load, and 1.14 on the main configuration.

You’re all caught up