Skip to main content
Back to timeline
NVIDIA Technical BlogSource publication:

Patches that pass local tests can still fail a live server: SWE-Serve puts the gap near one in three

Synopsis

AI coding agents that change inference-serving software can pass local tests yet fail when a server loads a real model and handles requests. SWE-Serve turns 83 merged SGLang changes into 53 executable tasks, 19 of which start a live server. On those 19 tasks, the same 627 patches pass 69.4% of the time without the live-serving checks and 45.9% with the complete verifier, so about one in three otherwise-passing patches fails live serving.

AI-generated editorial illustration: How SWE-Serve Exposes the Gap Between Local Tests and Live Serving

Interpretation

Evaluating an AI coding agent's change to inference-serving software requires checking the whole serving path, not only local unit tests. Existing repository-level benchmarks cover general software-engineering tasks, while inference benchmarks tend to focus on kernel generation or performance optimization; neither tests repository-scale changes across the serving stack. On the 19 tasks with live-serving checks, the same 627 patches pass 69.4% of the time when those checks are excluded and 45.9% under the complete verifier, meaning 147 patches moved from fail to pass.

The benchmark reconstructs merged SGLang changes as executable tasks: the agent receives an instruction and a containerized checkout from before the target change, and a hidden verifier judges its patch. A patch passes if it satisfies the task's hidden verifier on the declared hardware and is never compared with the reference implementation; the median reference solution modifies 553 lines across seven files. The 53 tasks derive from 83 merged SGLang pull requests across six families: speculative and advanced decoding, model and backend enablement, kernels, quantization and performance, serving APIs and runtime correctness, caching and runtime state, and distributed execution and scheduling; 12 tasks run on CPU and 41 use a single NVIDIA H100.

Perspective

This release covers only the SGLang inference engine: 12 of the 53 tasks run on CPU and 41 use a single NVIDIA H100, and it does not evaluate other inference engines, multi-GPU execution, or multi-node serving. A pass means only that the patch satisfies the benchmark verifier; the verifiers are not SGLang's upstream review process and do not establish that an agent patch or reference solution is deployable, ready to merge, or endorsed by maintainers. Evaluation is closed-book, blocking the public web and upstream source repositories while allowing Hugging Face access for model weights.

The first release does not cover other inference engines, multi-GPU, or multi-node serving, so the scope is SGLang on single-GPU settings. Mean pass@1 at each model's best tested configuration ranges from 34.6% to 75.5%, no model leads all six engineering families, and configurations with similar overall scores differ markedly in cost and wall time. The closed-book setting follows an open-network pilot in which models retrieved task-specific upstream code; 196 prohibited retrieval attempts were blocked across 1,749 trials.