Skip to main content
Back to timeline
arXivSource publication:

Two frozen experts now compose full-body gestures: CFGComposer restores the dropped cross-modal term, reaching 0.9916 semantic recall

Synopsis

Generating full-body gestures from speech no longer requires a single joint network trained on scarce tri-modal data. Linearly summing the predictions of separately trained audio-to-motion and text-to-motion diffusion experts implicitly assumes the two conditions are independent given the noisy motion, dropping the cross-modal dependence term. CFGComposer estimates that term with a jointly trained model and injects it at inference as classifier-free guidance; on BEAT it reaches a semantic recall SRGR of 0.9916 and beat alignment BA of 0.1826, against 0.1743 for ground truth.

Source-provided article image: CFGComposer: Compositional Audio-Text-to-Motion Generation via Copula Score Correction
arXiv

Interpretation

Full-body gesture generation can now combine two separately trained unimodal diffusion models at inference instead of training one joint network from scratch on scarce tri-modal annotations. The earlier composition summed the noise predictions of an audio-to-motion and a text-to-motion expert with weights, which assumes audio and text are conditionally independent given the noisy motion. On BEAT, that composition with the correction reaches semantic recall SRGR of 0.9916 and FID of 27.18, with beat alignment BA of 0.1826 against 0.1743 for ground truth.

What the linear sum drops is the cross-modal dependence between audio and text, which neither unimodal expert can represent on its own. The score decomposition of the joint log-density writes the exact joint score as the sum of the two marginal scores plus a copula score, the term describing cross-modal dependence. The linear sum sets that term to zero and so carries a fixed bias relative to the true joint prediction; a correction weight of zero reduces the method to that baseline, and a weight of one makes it equivalent to the jointly trained model alone.

The correction is estimated by a jointly trained diffusion model that receives audio and text conditions together. The joint model learns the dual-condition prediction while the linear-sum baseline approximates the marginal conditionals, and their difference in noise-prediction space forms the guidance direction. That difference has the same form as standard classifier-free guidance and is injected as a residual at each denoising step with an adjustable weight, at the cost of one extra parallel forward pass per step.

Perspective

The demonstrated scope is full-body gesture generation on the BEAT dataset and deployment on a Unitree G1 humanoid with BrainCo dexterous hands; at deployment each 5-second clip completes one reverse diffusion pass on a single RTX 4090, and the sequence streams at 60 Hz to an upper-body PD controller while the robot's own locomotion policy handles lower-body balance.

The current system uses one global gesture caption for whole-body control, and the authors list body-part-level spatial conditioning as future work; extending the composition to three or more conditions such as audio, text, speaker identity and emotion would also need hierarchical inclusion-exclusion and has not been implemented.