Skip to main content
Back to timeline
Scientific ReportsSource publication:

Phones and cloud split generative inference: MAGS bids instead of stampeding, cutting p99 latency 17.1%

Synopsis

A generative query's cost is unknown until the answer is written, so devices reserve resources blind and many crowd one server at once. MAGS has each device forecast token count and generation entropy with a four-layer probe on its local model, then bid for cloud capacity against a broadcast congestion price, choosing its per-token privacy parameter in the same bid. Over ten seeds it cuts p99 latency 17.1%, energy per token 13.6% and raises throughput 20.7% against a reactive PPO baseline given identical observations; an eight-device, two-server testbed drops the cross-device correlation of offload decisions from 0.41 to 0.08.

Source-provided article image: Semantic complexity estimation and price-based multi-agent negotiation reduce tail latency and energy in edge-cloud generative inference
Fig. 1 · Page 8

Interpretation

Edge devices stop grabbing cloud compute blind and instead estimate how many tokens a query will generate, then bid for capacity at a congestion price the server publishes. Earlier edge schedulers treated a request as an opaque unit of work and reacted only after congestion formed; semantic methods such as PICE profile difficulty periodically rather than per query. In simulation with 50 devices over 10,000 measured slots and ten seeds, MAGS lowers p99 latency 17.1%, energy per token 13.6% and raises throughput 20.7% against RPPO+, which receives identical observations and the same tuning budget; all eight paired comparisons give 0.002 on the two-sided exact Wilcoxon test.

Each device runs a four-layer probe over its local small model, regressing projected token count and generation entropy, and combines them into a predicted workload that drives bidding and scheduling. The probe depth is the shallowest whose regression quality sits within one standard error of the best depth tried on the validation split, costing 14.6 ms; the regression head is a two-layer perceptron of width 256 with 1.31 M parameters. On the held-out test split the token-count head reaches 6.7% mean absolute percentage error; Spearman rank correlation is 0.92 in distribution and falls to 0.71 on CaseHOLD.

The server first announces a congestion price computed from its queue and the previous slot's aggregate demand, devices maximise their own utility at that price, and the price adjusts until demand matches capacity. Replacing the summed aggregate with a mean keeps the coupling between agents independent of device count, which is what the convergence analysis needs. In a full factorial holding observations, capacity and tuning budget fixed, adding forecasting to a system that already has the price is worth 113, 133 and 189 ms at 25, 50 and 100 devices; adding the price to a system that already forecasts is worth 71, 82 and 134 ms, and neither margin collapses in the presence of the other.

The privacy parameter joins the same bid and is chosen per query rather than fixed globally. Earlier collaborative inference bolted privacy on afterwards as a fixed noise level, giving a routine query and a medical question the same protection. At a per-token budget of 2, split-and-denoise retains 94.6% of unprivatised GSM8K Pass@1 while cutting prompt-reconstruction BLEU from 0.718 to 0.076 and membership-inference AUC from 0.782 to 0.529.

Perspective

The demonstrated scope is a discrete-event simulation of 50 heterogeneous devices plus two physical testbeds, one edge node with one server and eight edge nodes with two servers; eight devices sit two orders of magnitude below the densities at which the scalability results are stated, so the density results in Table 15 remain simulated. The authors note that the number of devices a server can hold scales linearly with its measured admitted-query rate, and that Table 15 together with that rate is enough to size a deployment.

The formal privacy budget is weak: 25 releases per query and 40 queries a day give a per-user loss of 61.3 at a per-token budget of 2, which the authors call a limited worst-case bound. The implemented learner violates all four simplifications of Theorem 1, so convergence rests on ten seeds rather than a proof. Padded payload size, execution path and packet timing remain deterministic functions of private input, measured and mitigated but not bounded. The pseudonym rotation period sets both privacy loss and how long reputation can observe a device, and the authors leave that tension unresolved.