Skip to main content
Back to timeline
arXivSource publication:

21Codec moves speaker identity to the decoder: streaming voices stop drifting chunk to chunk, and can change mid-stream

Synopsis

Neural codecs for streaming text-to-speech usually infer speaker identity from the discrete token stream, and chunked decoding re-infers it from each local chunk alone, so the voice drifts within one utterance. 21Codec suppresses speaker information in the tokens and instead conditions every frame on an external speaker embedding at the decoder, which holds identity during streaming and allows a speaker change mid-stream. On unseen speakers, re-voiced output sits closer to the reference speaker than to the source; the cost is a higher word error rate than the unconditional version.

Source-provided article image: 21Codec: Decoder-side Speaker Control for Text-to-Speech
arXiv

Interpretation

During streaming decoding, one token stream now maps stably to one speaker, and that identity is set by an external embedding rather than carried by the tokens. Earlier codecs were trained to reconstruct audio over a sufficiently large window, and their decoders used global context to infer mostly static information such as speaker identity; once decoded in short chunks, each chunk re-infers identity from local context alone and the voice audibly drifts within an utterance. On unseen LibriTTS-R speakers, decoding the same source tokens with another speaker's embedding gives higher similarity to the reference speaker than to the source, while similarity between genuinely different speakers is lower still; in the SMOS listening test, re-voiced samples approach genuine same-speaker pairs, and a two-way repeated-measures ANOVA shows a large comparison effect.

Speaker identity is suppressed in the token stream and injected at the decoder through adaptive modulation. The prior arrangement let the decoder infer identity from the tokens, mixing identity and content in the same discrete codes. A speaker probe trained on frozen codes shows top-1 accuracy falling clearly below the unconditional baseline, with the same relative reduction on the held-out utterance subset and a significant paired McNemar test.

Chunk-causal decoding matches non-causal decoding down to a 500 ms chunk, and the speaker can be changed mid-stream without extra bitrate. Earlier streaming codecs drifted in voice at short chunks, which made them hard to use in real-time synthesis that needs short windows. Across chunk sizes from 20 ms to 1 s, including sizes unseen in training, speaker similarity stays close to the non-causal decoder; at a 160 ms chunk the real-time factor values on CPU and GPU are blanked in the retained text, so whether they fall below real time cannot be judged from it.

Perspective

The demonstrated setting is a 24 kHz input, 50 Hz frame rate, roughly 0.8 kbps streaming codec, evaluated on unseen LibriTTS-R speakers, a five-language re-voicing listening test and a chunk-size sweep from 20 ms to 1 s; these conclusions are limited to this codec paired with the ECAPA-TDNN speaker embedding.

Training stopped after 5 days for budget reasons and had not fully converged, so the reported numbers are lower bounds; the method depends on the speaker embedding model, and a different embedding model or a learned speaker table may shift the results; better speaker controllability comes with a higher word error rate, a trade-off between intelligibility and speaker removal.