Repair the evidence before decoding: LLM target-speech extraction cuts wrong-speaker errors from 6.87% to 0.75% on clean Libri2Mix
Synopsis
Large-language-model target-speech extraction can produce natural-sounding speech with drifted content, and the deterministic estimate it is grounded on may already have captured the wrong speaker, so grounding on that evidence reinforces the error. A training-free repair stage builds five candidates from frozen deterministic extraction paths and uses a frozen speaker encoder to pick the one most consistent with the target enrollment, then constrains code-space decoding of the LLM with that waveform. On Libri2Mix, wrong-speaker extraction falls from 6.87% to 0.75% on clean mixtures and from 18.95% to 10.46% under noise; with the generative checkpoint unchanged, noisy content- and acoustic-switch rates fall from 12.79% and 1.68% to 9.10% and 0.51%.
Interpretation
Even with an enrollment utterance from the target speaker, a target-speech extraction system may follow the competing speaker instead, producing output that stays clear and intelligible while carrying the interferer's identity and linguistic content, a failure called target confusion. The prior route grounded a generative extractor on a deterministic estimate to suppress content the autoregressive model invents; but that estimate may already have picked the wrong speaker, so grounding on it locks the error in. On the Libri2Mix clean test set, the deterministic baseline using the full enrollment gives a 6.87% wrong-speaker rate, which drops to 0.75% with five-candidate selection; on the noisy test set the corresponding drop is from 18.95% to 10.46%.
The repaired evidence can be returned directly or converted into a speech-token sequence that penalizes, at each decoding step, tokens disagreeing with the evidence, holding back content drift during generation. Constrained decoding had been used for autoregressive speech enhancement; here it is attached to target-speech extraction, with grounding strength additionally set per utterance from an estimated difficulty. With the generative checkpoint unchanged, fixed-strength grounding on the noisy test set lowers content switching from 12.79% to 9.10% and acoustic switching from 1.68% to 0.51%, while adaptive strength gives the highest speaker margin and predicted quality.
Perspective
The demonstrated setting is 16 kHz Libri2Mix-min two-speaker mixtures with clean, same-speaker, different-utterance enrollments; noisy conditions use the Libri2Mix/WHAM! mix-both setting plus separately constructed controlled-SNR trials, and the repair stage uses only the mixture and target enrollment, never clean references.
Under noise the five-candidate repair recovers only 57.89% of high-confidence confusions, below the 89.20% on clean speech and well below the 90.45% and 66.72% reference ceilings in the same conditions, indicating that severe corruption makes both candidate generation and selection harder; returning the repaired waveform directly incurs a small predicted-quality trade-off under severe noise; and the authors treat Whisper-based word error rate as informative but not conclusive, to be read alongside content switching, speaker consistency and perceptual quality.
