Voice agents hear the words but still act: an explicit action gate cuts executed side talk from 41% to 7–10%
Synopsis
The bottleneck in voice agents has moved from hearing to deciding: current systems hand recognition output straight to a reasoning model that acts, so even a perfect transcript cannot tell whether the words were meant for the agent or whether the speaker was authorized. A decision layer separate from perception and reasoning chooses to act, wait, ask, or stay silent from evidence on addressedness, speaker match, content confidence, tool reliability, and action risk; on recorded audio including WearVox, explicit gating cuts executed side talk from 41% to 7–10% while serving 83–85% of valid requests.
Interpretation
Voice agents need a separate action-decision layer that judges whether to execute outside the reasoning model. Current designs let the reasoning model emit an action directly, keep no explicit belief about whether speech was addressed to the agent, do not represent the cost of a wrong action, and treat waiting, asking, and silence as non-first-class choices. On WearVox side-talk and VGBench speaker-switch recordings, the model deciding on its own executes 41.2% of side talk and stays correctly silent in only 3.1% of speaker switches; a tuned fuzzy gate changes these to 7.9% and 65.8% while serving 84.6% of valid requests, whereas a content-confidence threshold alone only lowers executed side talk to 36.5%.
The decision layer takes addressedness, speaker match, content confidence, tool agreement, utterance completeness, and action risk as inputs, implemented with fuzzy rules, evolutionary tuning, and sequential decision-making. Tools return a calibrated confidence and a degree of validity under current acoustic conditions through an evidence contract, so each action traces back to its evidence. The gate takes under 0.1 ms on a single GPU, and speaker verification at 12 ms plus an independent addressedness detector at 9 ms fit within a 100 ms budget after the last tool output, whereas querying the agent's own large audio-language model about addressedness takes a median 142 ms and exceeds that budget.
Risk pricing keeps costly errors rare, and the inputs matter more than the choice of algorithm. Under identical evidence, risk-blind gates execute wrong high-risk actions at about 1.4%, 30 to 70 times more often than tuned risk-aware gates; with an overconfident recognizer, risk-blind gates fall to a utility of 0.18 with about 7% wrong high-risk actions, while calibrated or tuned risk-aware gates stay at or below 0.3%. In a controlled simulation of 200,000 test episodes, tuned gates and the calibrated product gate differ in utility by at most 0.01 at matched ask rates; on recorded audio, removing the addressedness input raises executed side talk to 33.8%, and removing the risk input multiplies wrong high-risk actions by five.
Perspective
The conclusions rest on controlled simulation and two public recorded-audio benchmarks: the WearVox side-talk and valid-query splits and the VGBench speaker switch. All three base models are open-weight audio-language models of 7 to 13 billion parameters, and no proprietary model is tested. Ask decisions are scored under assumed clarification success rates rather than with real users, and the evidence contract and late commitment are tested in simulation only.
The simulation abstracts perception into scores with assumed distributions and tests only one form of dependence between evidence terms. Speaker match is set to 1 where no enrollment audio exists, so addressedness is tested more thoroughly than speaker attribution. The independent addressedness detector is trained on part of WearVox and may not transfer to other devices, and the audio-versus-transcript comparison queries only one model. The authors also note that calibration of large audio-language models across acoustic conditions is largely unmeasured, and that gate behavior under graded costs and many action types remains to be tested.
