Skip to main content
Back to timeline
arXivSource publication:

Joint audio-video models look coordinated until prompts go vague: for most of four open models, consistency nears shuffled-pair levels

Synopsis

Joint text-to-audio-video models appear audio-visually consistent when prompts spell out every detail, but that consistency may only reflect each modality following the text on its own. Testing intrinsic jointness, the ability of audio and video to settle on the same realization for details the prompt leaves open, four open models (LTX-2.3, LTX-2, JavisDiT++, JavisDiT) lose consistency as prompts become less specific, and most approach a shuffled baseline at the broadest level. The authors propose JointREPA, a representation-alignment post-training method, and UNSAIDBench, a benchmark of semantically underspecified prompts, reporting consistency gains on two models across three benchmarks.

Source-provided article image: Beyond Co-Generation: Rethinking Jointness in Joint Audio-Video Generation
arXiv

Interpretation

Consistency in joint audio-video generation needs a source: when a prompt specifies every detail, audio and video can look synchronized simply by each following the text, without any cross-modal coordination. Earlier evaluations mostly used semantically detailed prompts, so the semantic dimension of genuine coordination went largely unexamined, while temporal coordination had already been studied more widely. With diagnostic prompts leaving exactly one factor open, across musical instruments, human speech and animal vocalization at three levels of semantic specificity, all four open models lose consistency as prompts become less specific, and most approach a shuffled baseline that randomly pairs audio and video generated from different seeds under the same prompt at the broadest level.

The disagreement is not confined to the final output: internal representations also fail to align, and the audio-visual attention that carries cross-modal interaction is underused. Earlier judgments stopped at the generated clip and could not say whether divergence appears only at the end of generation or persists throughout. Linear probes trained on mean-pooled audio and video features at each Transformer block and denoising step show probe agreement also falling as prompts become less specific; blocking audio-visual attention at inference causes large drops across metrics for the LTX family, only a slight decline for JavisDiT, and almost no change for JavisDiT++.

JointREPA aligns each modality's intermediate features with pretrained representations from both its own modality and the other modality, using post-training to push cross-modal semantic exchange. Earlier representation-alignment methods targeted text-to-image, text-to-video and text-to-audio generation, not semantic jointness in joint audio-video generation. With LoRA fine-tuning on JavisDiT++ and LTX-2.3, on UNSAIDBench JavisDiT++ rises from 0.176 to 0.207 in AV-IB and from 0.163 to 0.196 in AVHScore, while LTX-2.3 rises from 0.244 to 0.302 in AV-IB and from 0.236 to 0.288 in AVHScore, with gains also reported on JavisBench-mini and Verse-Bench.

Perspective

The scope is the semantic dimension of intrinsic jointness, meaning attributes such as event identity that are clearly distinguishable in both audio and video; temporal coordination is outside this analysis. Evaluation uses diagnostic prompts built from three event categories, musical instrument playing, human speech and animal vocalization, plus 524 UNSAIDBench instances adapted from JavisBench prompts. Improvements are reported on two backbones, JavisDiT++ and LTX-2.3, across UNSAIDBench, JavisBench-mini and Verse-Bench, with Verse-Bench used through its text prompts only.

Audio-visual attention is used to very different degrees across the four models: blocking it causes large drops for the LTX family but leaves JavisDiT++ nearly unchanged, so the coordination mechanism differs substantially by architecture. Results on text consistency and audio-visual quality are mixed, which the authors attribute to benefits appearing mainly in cross-modal coordination. Extending intrinsic jointness to physical attributes such as material and weight is proposed as a future direction and has not been tested.