Shared inference for AI agents now queues by how far behind each one is: Duru cuts p99 normalized slowdown by 44.4% while holding over 90% of baseline throughput
Synopsis
When several AI agents share one inference infrastructure, long-context requests and retained KV caches repeatedly slow some agents, and fairness counted per token or per request cannot see delays that accumulate across many calls. Duru uses each agent's normalized slowdown as a single control signal, coordinating admission and scheduling across the router and runtime, with admission exceptions and SLO-driven autoscaling to relieve starvation. Across 15 model-workload-hardware settings spanning three models, three real agent traces, and NVIDIA and AMD GPUs, Duru lowers p99 normalized slowdown by a geometric mean of 44.4% (up to 79.5%) while keeping 91.8%–127.4% of baseline throughput.
Interpretation
Fairness in shared inference moves from a single request to the whole agent session: an agent calls the model repeatedly, so delays accumulate call by call, and token-level or request-level fairness does not equal fairness in task completion. Earlier LLM serving fairness work balanced service at the token and request levels, and recent agent-aware systems mainly optimize throughput, latency and KV cache reuse, with comparatively limited attention to fairness among concurrent agents. Across 15 model-workload-hardware settings using Qwen3-Coder-30B-A3B, Llama-3.3-70B and DeepSeek-V4-Flash, the Azure, TraceLab and AgentX real agent traces, and AMD MI300X and NVIDIA H100 GPUs, Duru lowers p99 normalized slowdown by a geometric mean of 44.4%, with reductions of 13.8%–79.5% in 14 of those settings, while throughput is 91.8%–127.4% of baseline.
Duru defines fairness as how much slower an agent runs than it would alone: shared execution time is compared with an offline-calibrated standalone time and converted into normalized slowdown against ideal processor sharing, so a higher score means the agent has fallen further behind its fair share. Standalone execution time cannot be observed directly under sharing, so Duru interpolates from measured prefill and decode latency tables for the same model, hardware and serving configuration, accumulating estimates iteration by iteration without knowing output length or future calls in advance. On Qwen3-Coder-30B-A3B (RTX PRO 6000), 534 Azure and TraceLab sessions give mean absolute relative errors of 1.9% and 1.8%, with maximum errors of 4.1% and 4.2%; on Llama-3.3-70B (2 H100, TP=2), 200 sessions totaling 1.95M output tokens give mean errors of 0.2%, 0.3% and 0.1% across the three traces.
The router and runtime order candidates by the same slowdown score and force-admit lagging agents when the working-set budget is exhausted, pushing back the problem of waiting shifting from one layer to another. Policies that admit by cache affinity or token-level fairness can shift unfairness from GPU execution into router queuing, and interference between long prefills and concurrent decoding cannot be removed by scheduling alone. In a 48-hour replay of Azure GitHub Copilot production traces, Duru reaches 97.4% agent-level SLO attainment versus 93.4% for NVIDIA Dynamo and 70.2% for llm-d at nearly identical throughput, using 76 GPUs on average against 89 for Dynamo.
Perspective
The evaluation covers Qwen3-Coder-30B-A3B, Llama-3.3-70B and DeepSeek-V4-Flash, the Azure, TraceLab and AgentX real agent traces, and AMD MI300X and NVIDIA H100 platforms, with H200 and RTX PRO 6000 used for interference characterization; scalability from 8 to 256 GPUs and autoscaling are studied through DynoSim simulation, where host-memory KV cache offloading is disabled.
Normalized slowdown depends on offline-calibrated standalone latency tables, so a new deployment configuration requires recalibration; on the Azure workload, interference between long prefills and concurrent decoding makes Qwen's fairness gain on H100 weaker than on other workloads, and scheduling cannot remove that execution interference; too small an admission-exception threshold lowers KV cache hit rates and can worsen fairness.
