Skip to main content
Back to timeline
arXivSource publication:

Long reasoning outputs stop thrashing: Stint caps the worst pause under 50 seconds and lifts throughput 12% in the same engine

Synopsis

Long outputs from reasoning models keep key-value caches in GPU memory for minutes, and a preempted request can repeatedly lose and recover its state while producing almost no new tokens. Stint stores each preempted request's cache snapshot in contiguous CPU regions, moves it back in bulk, and gives the request a protected turn of fresh decoding based on how long it has waited. On IMO-AnswerBench replays with Qwen3-32B, no request pauses longer than 49.3 seconds, while recomputation, LMCache and HiCache leave 21% to 23% of requests stalled over a minute; against recomputation in the same engine, p99 time per output token falls 50.7% and throughput rises 12.0%.

Source-provided article image: Making Preemption Productive in Long-Output LLM Serving
arXiv

Interpretation

In long-output serving, a preempted request can repeatedly lose and recover its state while producing very few new tokens, a cycle the authors call KV thrashing. In a replay of 399 olympiad math problems, recomputation, LMCache and SGLang with HiCache each leave 21% to 23% of requests stalled for more than a minute, and about one in six for more than five minutes. Under recomputation a single pause takes up to 72% of a request's lifetime, whereas under Stint no request pauses longer than 49.3 seconds and no pause exceeds 8.4% of a request's lifetime.

Stint stores a preempted request's KV snapshot at its known length in contiguous CPU regions, moves it back to the GPU, and then grants a decoding turn in which only new tokens count. Earlier paths either discard KV and rebuild old context on recovery, or move scattered GPU blocks one by one and can be evicted again right after returning. Serving BF16 Qwen3-32B on eight H200 GPUs as four two-GPU replicas and replaying all 399 IMO-AnswerBench questions, three matched runs show 11.99% higher throughput and 50.69% lower p99 time per output token than recomputation.

Perspective

The demonstrated setting is dense Qwen3-32B and Llama-3.1-8B with up to two GPUs per replica (four in the allocator study) on vLLM 0.9.2, using fixed-length and natural-completion replays; gains concentrate on long-output workloads, while short-output summarization and BurstGPT replays show none.

Stint replaces rare, unbounded stalls with more frequent bounded ones, pausing most long requests for about 30 seconds, and median time per token, pooled p99 token interval and first-token time all worsen; it also needs more pinned host memory, and the current implementation turns an 800 GiB payload into 1,024 GiB of backing memory through allocator rounding.