Skip to main content
Back to timeline
arXivSource publication:

Speech and gesture leave the two-step pipeline: Gaspard generates both streams in one autoregressive model at a 0.84 real-time factor

Synopsis

Speech and co-speech gesture synthesis has commonly followed a cascade: generate speech first, then drive gestures from it, so gestures cannot feed back into speech. Gaspard places both modalities in one continuous autoregressive model, predicting the next speech and gesture embeddings at each step with separate flow-matching heads, and uses decoupled classifier-free guidance to regulate each stream's own history and cross-modal conditioning. On Seamless Interaction it yields stronger gesture dynamics and synchrony than a matched cascaded system, with a real-time factor of 0.84 versus 1.04.

Source-provided article image: Gaspard: Expressive unified speech and gesture synthesis with continuous autoregressive flow-matching and decoupled classifier-free guidance
arXiv

Interpretation

Speech and gesture are now generated together by one model, with gesture history and speech conditioning regulated separately at inference rather than letting speech be fixed first and then drive gestures. Earlier approaches commonly used a cascaded pipeline: text produced speech, and speech then conditioned gesture generation, so gestures could adapt to speech but could not contribute to speech generation. Against a cascaded system built with the same training recipe as Gaspard, unified generation reached beat consistency closer to human recordings, with a real-time factor of 0.84 versus 1.04.

The model predicts the next frame step by step in continuous speech and gesture latent spaces, with one flow-matching head per modality. Flow matching generates a sample by integrating a learned velocity field from Gaussian noise, and here it serves as the next-frame predictor at each autoregressive step. Speech and gesture are encoded by Z-Codec and a gesture β-VAE to the same 12.5 Hz latent frame rate, and the backbone is a 246M-parameter encoder-decoder Transformer trained for 200k steps on Seamless Interaction.

Decoupled classifier-free guidance splits reliance on one's own history from reliance on the other modality into two coefficient sets, applied to the speech head and the gesture head. Conventional classifier-free guidance weights multiple conditions for a single generated stream, whereas this regulates self-modal feedback and cross-modal conditioning separately in both generation streams. On the Seamless-416 evaluation subset, decoupled guidance lowered FGD, kinetic FGD and word error rate and improved beat consistency and diversity relative to standard guidance and no guidance, at the cost of extra backbone forward passes that raised the real-time factor from 0.78 to 0.84.

Perspective

Evaluation is confined to two subsets of the Seamless Interaction test split: Genea-95 with 95 segments for the gesture-baseline comparison, and Seamless-416, drawn from several speakers, for ablations and the cascaded comparison; the model covers upper-body gestures only, takes text as input and can optionally use a speech-gesture prefix.

Speech quality still trails the 350M-parameter PocketTTS: word error rate 6.38% versus 4.69% and predicted naturalness 2.69 versus 3.65, with the authors offering parameter count, the Mimi codec and possible interference from joint generation as possible explanations. Multiple guidance passes add computation, and long-term generation and out-of-domain generalization are left as future work.