VelServe leaves each decode step a measured latency margin: up to 4.4 times the strongest baseline's SLO goodput under a P99 token-gap target
Interactive LLM services must keep each request's token gaps within a P99 latency target, yet one overrunning decode step delays every request in the batch, and long answers span hundreds of steps, so rare jitters become frequent violations. VelServe leaves each decode step a margin calibrated from the measured tail of its own iteration latency and recovers first-token time with shortest-prompt-first order; it sustains 2.4–2.9 times the SLO goodput of the strongest baseline on Qwen3-8B and DeepSeek-V4-Flash and at least 4.4 times on DeepSeek-V2-Lite. With prefill and decode disaggregated on eight GPUs, it meets the SLO for more than twice as many requests as vLLM and Niyama.