Skip to main content
Back to timeline
arXivSource publication:

VelServe leaves each decode step a measured latency margin: up to 4.4 times the strongest baseline's SLO goodput under a P99 token-gap target

Synopsis

Interactive LLM services must keep each request's token gaps within a P99 latency target, yet one overrunning decode step delays every request in the batch, and long answers span hundreds of steps, so rare jitters become frequent violations. VelServe leaves each decode step a margin calibrated from the measured tail of its own iteration latency and recovers first-token time with shortest-prompt-first order; it sustains 2.4–2.9 times the SLO goodput of the strongest baseline on Qwen3-8B and DeepSeek-V4-Flash and at least 4.4 times on DeepSeek-V2-Lite. With prefill and decode disaggregated on eight GPUs, it meets the SLO for more than twice as many requests as vLLM and Niyama.

Source-provided article image: VelServe: Taming Decode Jitters in LLM Inference with Tail-Calibrated Scheduling
arXiv

Interpretation

VelServe treats decode jitter in LLM inference as a schedulable system problem: once a decode step runs past the token-gap target, every request in the batch receives a late token, and a long answer lives through hundreds of such steps. Earlier SLO-aware schedulers filled each step with prefill until a latency model predicted it would just meet the target, leaving no room for mispredictions or host stalls; targets set on mean time per output token or cumulative token deadlines average long pauses away. On Qwen3-8B, Medha overran on 1.36% of iterations and Niyama on 2.15%, while VelServe overran on 0.00%; on DeepSeek-V2-Lite with the Mixed workload at 1.25 requests per second under a 150-ms bound, the baselines met it at P99 for only 62–74% of requests.

The scheduler decides how much prefill each step carries from the measured tail of its own recent iteration latency, and a shortest-remaining-prompt-first order pays back the first-token time the margin costs. The margin comes from a high-rank statistic of running iteration residuals rather than an offline-fitted distribution or a fixed value carried over from another workload; short prompts can preempt a long prefill at a chunk boundary, and a long prompt's deadline grows with its own length, so it keeps slack. On Qwen3-8B, its own iterations under load raised the margin from a starting 2.3 ms to about 50 ms; in the ablation, neither the order nor the budget alone served two-thirds of requests, and only together did they keep attainment above 90% up to the highest load run.

Perspective

The evaluation covers Qwen3-8B, DeepSeek-V2-Lite and DeepSeek-V4-Flash in single-GPU aggregated serving, plus a disaggregated DeepSeek-V4-Flash deployment on one eight-GPU node with four GPUs for prefill and four for decode; the headline SLO scales the first-token deadline with prompt length and bounds each request's P99 token gap at 150 ms. The disaggregated results come from one node, one model, one engine and one Mooncake trace.

Calibration adapts with a lag: the budget drops at once but rises only after two consecutive proposals, so a sudden burst of deeper stalls can cost some attainment before the margin catches up. When host stalls are longer than any budget can absorb, VelServe flags the SLO as infeasible; on DeepSeek-V2-Lite, bounds below 150 ms are limited by host stalls of up to about 100 ms. With Mooncake-shaped prompt lengths, DeepSeek-V4-Flash's SLO goodput falls to less than half its value on Mixed, and the losses are first tokens of medium and long prompts.