Skip to main content
Search

# Agent-level fairness

1 related articles

Subscribe to updates
  1. arXiv

    Shared inference for AI agents now queues by how far behind each one is: Duru cuts p99 normalized slowdown by 44.4% while holding over 90% of baseline throughput

    When several AI agents share one inference infrastructure, long-context requests and retained KV caches repeatedly slow some agents, and fairness counted per token or per request cannot see delays that accumulate across many calls. Duru uses each agent's normalized slowdown as a single control signal, coordinating admission and scheduling across the router and runtime, with admission exceptions and SLO-driven autoscaling to relieve starvation. Across 15 model-workload-hardware settings spanning three models, three real agent traces, and NVIDIA and AMD GPUs, Duru lowers p99 normalized slowdown by a geometric mean of 44.4% (up to 79.5%) while keeping 91.8%–127.4% of baseline throughput.

You’re all caught up