Two separately recorded voice tracks become one conversation: a neural scheduler learns only timing while a back-channel predictor decides where listeners interject
Synopsis
In game voice production, the two speakers' lines are usually recorded separately in the booth, and splicing them yields stiff turn changes with no listener back-channels. This pipeline first uses a back-channel predictor to decide whether each line invites an interjection and of which class, then a neural scheduler jointly places onset times on both channels, inserting pauses and overlap. On the Fisher corpus the scheduler stays closer to ground-truth turn statistics than Moshi, NTPP and other baselines on most of four metrics; on Full-Duplex Bench it produces back-channels at 0.062 per second with a Jensen-Shannon divergence of 0.644 from natural back-channel timing.
Interpretation
Two independently recorded or TTS-synthesized voice-line channels can be sequenced into a two-speaker conversation with pauses, overlap and listener back-channels, and these dynamics are adjustable at inference time. Such lines were previously recorded in isolation from a script, and splicing them produced uniformly robotic turns lacking the overlaps and pauses of natural talk and lacking listener back-channels; back-channels were sometimes recorded separately and edited in by an animator, which does not scale to tens of thousands of lines. On Full-Duplex Bench the system produces back-channels at 0.062 per second with a Jensen-Shannon divergence of 0.644 from natural back-channel timing; for comparison, PersonaPlex gives 0.025 and 0.649, and Gemini gives 0.012 and 0.896.
The system separates what is said from when it is said: content is fixed by the existing lines, and the model handles timing alone. The back-channel predictor is a multimodal classifier that takes a text window around the target line plus that line's audio and outputs three classes: no back-channel, a low-salience acknowledgment, or a high-salience reactive response; the actual words come from a user-supplied candidate pool or from TTS. On the held-out test set the classifier reaches a binary AUC of 0.80, above 0.73 for Wang et al. and 0.69 for Lin et al.; removing audio drops recall from 0.49 to 0.25.
The neural scheduler casts voice-line scheduling as next-token prediction, using a speech language model that emits interleaved tokens for both channels to learn timing. The input region keeps only each line's speech tokens and turn markers with no duration information; the output region interleaves both channels frame-aligned at 12.5 Hz, with silence tokens encoding pauses and overlap. Prompted with 10 seconds of audio to continue conversations on the Fisher corpus, the scheduler's cumulated-duration deviations on four turn metrics are 0.1, 0.1, 0.6 and 0.4 seconds per minute, versus 3.3, 2.8, 1.4 and 1.9 for NTPP and 0.3, 0.6, 0.3 and 0.8 for Moshi at temperature 1.0.
Perspective
The demonstrated setting is script-ordered, independent voice clips, either recorded by voice actors or synthesized with TTS, producing two-channel dialogue audio; back-channel wording is drawn from a user-supplied candidate pool, and the system does not generate back-channel content semantically grounded in the conversation.
Within the three-class breakdown, the Reactive class reaches an AUC of 0.76, below the 0.80 of the binary task, which the authors attribute to its lower frequency (400 examples) and greater linguistic and prosodic variability. In the perceptual study, preference for the scheduler over a fixed 200 ms gap baseline was at chance for casual conversation and significant only for dramatic material.
