Skip to main content
Search

Ziji Shi

1 related articles

Subscribe to updates
  1. arXiv

    Long reasoning outputs stop thrashing: Stint caps the worst pause under 50 seconds and lifts throughput 12% in the same engine

    Long outputs from reasoning models keep key-value caches in GPU memory for minutes, and a preempted request can repeatedly lose and recover its state while producing almost no new tokens. Stint stores each preempted request's cache snapshot in contiguous CPU regions, moves it back in bulk, and gives the request a protected turn of fresh decoding based on how long it has waited. On IMO-AnswerBench replays with Qwen3-32B, no request pauses longer than 49.3 seconds, while recomputation, LMCache and HiCache leave 21% to 23% of requests stalled over a minute; against recomputation in the same engine, p99 time per output token falls 50.7% and throughput rises 12.0%.

You’re all caught up