Voice agents skip token-by-token decoding: a frozen LLM reads one hidden state, answering 80 decisions on eight utterances in about 0.1 s
Synopsis
Full-duplex voice agents make many small closed decisions, and current systems answer them by slow autoregressive decoding. DuplexJev feeds ASR-encoder hidden states through a small connector into a frozen LLM and reads each question as a single-token distribution over its options, decoding nothing; an 8-GPU node answers 80 decisions about eight utterances in about 0.1 s. With a last-layer connector, spoken QA stays close to reading the transcript (90% vs. 91%); a cross-attention connector brings gender and emotion accuracy to 90% (from 55% and 28%) while spoken QA drops only one point (83% to 82%).
Interpretation
Instead of generating an answer token by token, a voice agent lets a frozen LLM produce one single-token distribution over the candidate options and reads the choice off directly. The prior route answered each such small closed decision through slow autoregressive decoding. On an 8-GPU node, 80 decisions about eight utterances complete in about 0.1 s, with nothing decoded.
Spoken question answering can come close to reading the transcript, while the model also picks up speaker attributes from the voice itself. Against reading the transcript as reference, spoken QA reaches 90% with a last-layer connector versus 91% for the transcript. With a cross-attention connector, gender and emotion accuracy both reach 90%, up from 55% and 28%, and spoken QA moves from 83% to 82%, a one-point drop.
The training signal sits on the read-out answer token rather than asking a student to imitate a teacher that never hears the voice. The usual transcript distillation uses a teacher that never hears the voice, so it cannot pass on what the voice carries. Decision tasks are trained with cross-entropy on the read-out answer token, distillation is kept for content, encoders and LLMs are interchangeable, and weights, training recipe, a batched-inference pipeline for full-duplex serving and a bilingual spoken-QA set are released.
Perspective
This report rests on the abstract-level summary, not the complete paper. The demonstrated scope is small closed decisions: spoken QA, gender and emotion recognition, with 80 decisions about eight utterances answered in about 0.1 s on an 8-GPU node. The summary states that encoders and LLMs are interchangeable and that weights, training recipe, a batched-inference pipeline for full-duplex serving and a bilingual spoken-QA set are released, which suits engineering readers evaluating full-duplex voice serving.
The reported numbers belong to specific connectors and settings: spoken QA is 90% versus 91% for the transcript with the last-layer connector, and 82% with the cross-attention connector, below the last-layer connector's 90%. The 90% gender and emotion result is reported only for the cross-attention connector. These comparisons come from the abstract-level description and do not cover larger scales or open-ended generative answers.
