Querying foundation models now comes with a budget: ACES learns which to call and how to combine them, nearing all-query accuracy on laptop pricing
Synopsis
For foundation models billed per query, ACES learns which models to query and how to combine their outputs with the task's own features under one budget on average query cost per input. ACES-T queries the same collection for every input and selects it by empirical risk over affordable collections via branch-and-bound; ACES-I uses Q-learning to vary the queried models across inputs and decide when to stop. On a laptop-pricing task with six foundation models, ACES-T reached a log-RMSE of 0.318 at about 60% of the all-query cost, within 0.001 of querying every model.
Interpretation
The predictor no longer assumes every foundation-model output is free: under a cap on average query cost per input, it jointly decides which models to query and how to combine the acquired outputs with the task's own features. Earlier methods that correct pretrained-model outputs with task data treat predictions as free and therefore never decide which models to query; they use a single model or correct several separately before averaging with convex weights, and they do not let the queried models vary across inputs. On an Amazon laptop-pricing task with six foundation models, ACES-T and ACES-I had lower error than FSP-Best, SRE-Best and BBT-2-Best at every budget where they were available, and ACES-T was never worse than Single-Query.
The method splits a deployable procedure into a query rule and a prediction rule: the prediction rule fits a linear combination of acquired outputs and features computable without querying any model, while the query rule decides which outputs to buy for each input. ACES-T queries one collection for every input and searches affordable collections by branch-and-bound, using the fit of a larger collection to lower-bound its subsets so not every collection is fitted; ACES-I instead uses backward Q-learning to decide, state by state, which model to query next or when to stop. In simulation, ACES-T queried models 1, 2 and 3 for every input and nearly matched the loss of querying all models at about one quarter of the cost; in the second simulated scenario, ACES-I lowered the mean test MSE relative to ACES-T and also reduced the mean query cost.
The theory prices the budget: when the regression function depends on the outputs only through an affordable collection, ACES-T attains the minimax rate and the budget costs nothing in rate; otherwise any procedure within budget can be forced to a slower square-root rate, which ACES-T attains. Earlier statistical aggregation of predictors assumed every candidate was available at no cost, so rate-optimal procedures used all of them and did not account for the cost of choosing under a budget. In a constructed experiment where every output carries signal but an unknown group has slightly larger coefficients, every procedure whose expected number of queries stays within budget obeys the square-root lower bound, matching the order of the ACES-T upper bound.
Perspective
The demonstrated scope is limited to settings where every output is observed in the training and validation samples and prediction rules are linear in their coefficients, evaluated on two simulations and one Amazon laptop-pricing task; the authors list partially recorded outputs and computational efficiency as the number of models grows as future work.
The analysis requires every external output to be observed in the training and validation samples; branch-and-bound can examine exponentially many collections as the number of models grows with the sample, and the backward recursion runs over all collections, so the authors explicitly leave computationally efficient procedures for many models to future work.
