Dubbing plus voice conversion yields four-way aligned speech pairs, lifting expressive translation in English–German and English–Spanish
Synopsis
Expressive speech-to-speech translation needs training pairs aligned in content, prosody, timing and speaker identity at once, yet existing resources leave at least one dimension unmatched. This work combines professional dubbing, which supplies human-performed correspondence, with multilingual zero-shot voice conversion to build four-way aligned English–German and English–Spanish pairs, and tests that supervision on TransVIP, Dub-S2ST and an end-to-end model called DirectS2ST. On the independent mExpresso evaluation set, prosody and speaker similarity rise for all three model families, DirectS2ST reaches the best prosody and pause scores in both directions, and nine listeners raise its expressiveness rating by 1.29.
Interpretation
Training data for expressive speech translation can now be aligned on content, prosody, timing and speaker identity at the same time, instead of sacrificing at least one of the four as earlier resources did. Available resources each had a gap: synthetic speech can reproduce source-speaker identity but may lack expressive delivery, mined data offer semantic parallelism only, and professional dubbing matches semantics, prosody and timing but uses different actors. In both English–German and English–Spanish, the raw dubbing pairs without voice conversion already score above CVSS-T on BLASER, prosody similarity and speaking-rate correlation, but their speaker similarity is only 0.060 and 0.046, well below the 0.208 and 0.194 of CVSS-T.
The recipe treats professional dubbing as the skeleton of human delivery, then uses multilingual zero-shot voice conversion to move the dubbed side onto the original speaker's voice. Voice conversion is instantiated with Seed-VC and applied only to the source side, while the target keeps the un-resynthesized human recording, so content, prosody, timing and speaker identity are supervised jointly. Comparing transcripts before and after conversion gives roughly 8.3% word error rate and 5.2% character error rate, and randomly shuffling source–target pairings causes large drops in BLASER, prosody similarity, rate and pause, indicating the correspondence is pair-specific rather than a corpus artifact.
Across three translation systems with different architectures, switching to this corpus improves expressive preservation, and DirectS2ST, which predicts the complete target speech directly, gains the most. Dub-S2ST's existing speed-adapted variant mainly improves rate-related metrics, whereas this corpus gives broader gains in prosody and speaker similarity, with additional gains in timing and naturalness depending on the architecture. On 504 mExpresso examples per direction, DirectS2ST's prosody similarity rises from 2.162 to 2.905 in English–German and from 2.238 to 3.091 in English–Spanish, pause from 0.687 to 0.752 and from 0.652 to 0.733, and nine English- and Spanish-proficient listeners raise its expressiveness score from 2.13 to 3.42.
Perspective
Validation covers English-to-German and English-to-Spanish only, evaluated on mExpresso, which is independent of the training corpus, with 504 examples per direction spread over seven expressive conditions; Dub-S2ST is evaluated only on English–Spanish because its unit representation does not support German. Human evaluation used nine listeners proficient in English and Spanish, 10 English–Spanish source utterances with the source as reference, and a five-point expressiveness scale over six system configurations.
Transcripts before and after voice conversion differ by roughly 8.3% word error rate and 5.2% character error rate, so content is not perfectly preserved; the authors also name speaker mismatch as the main residual problem of raw dubbing data. DirectS2ST trains its translation and codec decoders from scratch and its absolute content scores remain below TransVIP, which is initialized from SeamlessM4T, a difference the authors attribute to pretraining. The dataset cannot be shared because of licensing constraints.
