A speech generator doubles as the voice encoder: RSE-TTS shares one backbone, cuts encoder-plus-generator parameters by 48%, and improves WER and similarity on three English benchmarks
Synopsis
RSE-TTS lets a pretrained speech-generation backbone also serve as the reference encoder, compressing reference text and recording into eight continuous vectors while encoding and generation share one Transformer, cutting their combined parameter count by 48% relative to independent encoding. On three English benchmarks, the 118M-parameter generator improves both word error rate and speaker similarity over the official MOSS-TTS-Nano reference-clone and continuation modes; with acoustic continuation added, similarity approaches larger systems while WER stays below the evaluated CosyVoice 3 and MOSS-TTS-Local.
Interpretation
A pretrained autoregressive speech-generation backbone can be repurposed to compress reference speech rather than only generate target speech, and feeding the reference transcript alongside the recording further raises speaker similarity in synthesis. Earlier reference encoders were either trained from random initialization or relied on separate speaker-verification embeddings, leaving the generator's learned text–speech correspondence unused for reference compression. With architecture and training setup fixed, initializing an independent reference encoder from the generation backbone raised similarity from 0.400 to 0.530 on Seed-TTS, from 0.342 to 0.469 on CV3-eval, and from 0.391 to 0.592 on MiniMax; adding the paired transcript then improved both independent and shared encoders on every benchmark.
Encoding and generation share one Global Transformer, so the reference side no longer needs a second full Transformer, and eight learned query vectors read the reference text and speech in a single forward pass. An independently pretrained encoder design maintains a separate backbone for reference encoding, totaling 228.5M parameters across encoding and generation. With the shared backbone the combined count is 118.2M, 48% fewer than the independent design; the independent text-and-audio encoder reaches the highest similarity, and the shared design retains most of that benefit.
The eight vectors carry delivery information such as speaking rate and pitch, not only voice identity, and they supply voice information complementary to a conventional speaker embedding. A conventional 192-dimensional speaker embedding alone predicts the WavLM embedding of another recording of the same speaker at cosine 0.438, and the eight reference vectors alone at 0.439. Concatenating the two raises prediction to 0.469, improving 58 of 63 held-out speakers, with a mean gain of 0.031 over the conventional embedding (95% CI [0.025, 0.037]); lengthening the reference from 3 to 24 seconds raises mean similarity from 0.378 to 0.569.
Perspective
The demonstrated scope is English: the controlled ablation trains for one epoch on same-speaker utterance pairs from a 20k-hour subset, the system comparison uses the Seed-TTS, CV3-eval and MiniMax English benchmarks, and the reference studies use 30 EARS speakers. The intended readers are speech researchers and engineers working on compact TTS and reference conditioning.
Compact reference conditioning still trails acoustic continuation in speaker similarity; diagnostics from an earlier checkpoint show the similarity gap is already present near the start of generation and persists toward the end, while sensitivity to reference conditioning decreases as acoustic history grows, and the authors list teacher forcing's reliance on acoustic history as one possible explanation, with information loss during compression as another. RSE requires a reference transcript, making transcription robustness a consideration for audio-only applications.
