Skip to main content
Back to timeline
arXivSource publication:

ARoute decides which model answers before any call: highest mean accuracy in all 15 task-cost settings, 4.1 points over the strongest baseline at matched cost

Synopsis

ARoute predicts both whether upgrading to the stronger model will help and how much more that upgrade will cost, using the query text alone, and picks the model before either model is invoked. It fits two lightweight heads on a small set of paired records over one frozen BERT encoder. On MMLU-Pro, LiveCodeBench, SuperGPQA and two SWAN value-filling tasks, ARoute-Batch attains the highest mean accuracy in all 15 task-cost settings, 4.1 points above the strongest baseline on average and up to 11.9 points, and completes a batch up to 2.8 times faster than cascades.

Source-provided article image: ARoute: Task-Aligned Batch and Streaming Model Routing
arXiv

Interpretation

Model routing can now happen before any answer is generated: the router reads only the query text and decides whether that request goes to the weak or the strong model, without first running the weak model to see whether an upgrade is needed. Cascades such as FrugalGPT and BARGAIN first have the weak model answer every request and then decide from that answer whether to escalate, so every request pays a weak call and a sequential wait; pretrained routers such as RouteLLM skip the weak call, but their score comes from generic preference data rather than the task's own correctness criterion. On MMLU-Pro, LiveCodeBench, SuperGPQA and two SWAN value-filling tasks, with the cost of 50 paired records charged to the budget, ARoute-Batch has the highest mean accuracy in all 15 task-cost settings at 30%, 50% and 70% of all-strong cost, beating the strongest baseline by 4.1 points on average and up to 11.9 points.

The signals that matter are which queries only the strong model answers correctly and how much more the upgrade costs for that query, and both are predicted from the query text. No earlier router estimated both at query granularity: RouteLLM's score carries little information about the only-strong-correct outcome and treats every call as one unit of cost, while OctoSelector fits task-specific estimates only per cluster, so queries in a cluster receive the same estimate. Each query is encoded once by a frozen BERT; a linear head predicts the probability that only the strong model is correct and a small network predicts each model's generation cost, and their ratio gives the priority; across the five tasks this linear head reaches AUROC 0.58–0.69, against 0.45–0.61 for RouteLLM's pretrained score.

In streaming, ARoute-Stream selects a cost cutoff from a small validation sample and gives a finite-sample guarantee on expected normalized spending. Earlier guarantees concerned accuracy or a cascade's per-query cost, not the expected normalized cost of a pre-generation router. At a 70% quota with 200 validation draws per router, the fixed rule that applies the KL test once exceeds the quota in at most 1% of draws and by at most 0.2 points; the paced rule that re-selects the cutoff every 50 routed queries stays within 0.6 points of the target on average and exceeds the quota in 2–7% of draws.

Perspective

The evaluation covers the model pair Llama-3.1-8B-Instruct and GPT-5 on five tasks (MMLU-Pro, LiveCodeBench, SuperGPQA and two SWAN value-filling tasks), with batch results reported at 30%, 50% and 70% of all-strong cost and streaming results replayed at the same three normalized-cost quotas. Each query's correctness and cost are scored in isolation, and how an error propagates through an operator plan or an agent trajectory is outside scope.

The streaming guarantee covers only a cutoff selected once and then held fixed, and says nothing about accuracy, dollars or the realized spending of one batch; the paced rule that re-selects the cutoff every 50 queries is a heuristic, and its over-quota rates are observed overspend frequencies rather than the false-certificate probability the theorem bounds. Computing each routed query's normalized cost under pacing needs the price of the call not made, which a cached replay can read but a live system cannot. On the two SWAN tasks GPT-5's cost varies less, the cost network adds little or hurts, and the full score trails the quality head alone by up to 2.6 points.