Skip to main content
Back to timeline
arXivSource publication:

Video generation stops building a 3D proxy every time: Coda-DaVinci routes by task and cuts VTR word error rate from 59.21% to 11.08%

Synopsis

Coda-DaVinci turns Code2Video from a fixed pipeline into task-adaptive routing: a coding agent builds a 3D proxy when precise control of structure or motion is needed, while tasks centered on natural body movement go to direct generation, with modeling granularity set per entity. On VTR it lowers word error rate from 59.21% for direct generation to 11.08% and leads scene consistency on R2M; in long videos, event structure and cross-turn consistency improve, but transition stability and adherence to scene settings remain weaker than direct generation.

Source-provided article image: Coda-DaVinci: Reimagining Video Generation with Code2Video
arXiv

Interpretation

Video generation gains a path that switches by task: Coda-DaVinci builds a 3D proxy before synthesis when precise control of scene structure or motion is required, and falls back to direct generation when explicit control offers limited benefit and natural body movement is central. Earlier Code2Video practice always had a coding agent construct and render a 3D proxy to guide the video model; finer proxies could carry geometric or motion errors into the final video. On VTR, Coda-DaVinci exceeds the strongest agentic baseline in checklist satisfaction and reduces word error rate from 59.21% for direct generation to 11.08%; it also takes the highest reported score on the R2M scene-consistency metrics.

Proxy detail is no longer uniform but assigned per entity: environments and structured objects use medium granularity, while humans use coarse proxies that keep only whole-entity translation and rotation. In comparisons that fix prompting at no enhancement and take both physics settings, fine-grained proxies never exceeded medium granularity in naturalness and realism; on VGIF-Bench, coarse granularity led the three levels in physical plausibility and naturalness under both physics settings. Manual inspection attributes the coarse advantage mainly to leaving limb motion and fine details to the video generator, reducing stiffness inherited from detailed proxy animation; with physics simulation enabled, medium granularity improves consistency over coarse on WorldMemBench and VGIF and ties on Principia.

The synthesis prompt is written by the agent after inspecting the rendered proxy, linking subject descriptions to proxy entities and appearance references and adding materials, lighting and subtle motions absent from the proxy. With medium granularity fixed, agent-authored prompts beat the tested external text-only enhancement by MiniMax H3-Context-IR on all four metrics averaged over the three benchmarks. With physics simulation enabled, the agent prompt gains more over no enhancement in the four-metric mean, raising consistency, instruction following and physical plausibility while slightly lowering naturalness and realism; on VGIF-Bench, switching from no enhancement to the agent prompt raises the four-metric mean by 0.405 points, while without physics simulation the change is under 0.001 points in magnitude.

A multi-agent harness keeps a shared Blender workspace so subagents can model components separately and reuse assets in later requests, and an independent acceptance agent checks each candidate against task criteria and returns unmet requirements for repair. Ablations show that removing independent acceptance lowers consistency most, while fixing routing to always use 3D modeling or applying medium granularity to all subjects has its largest effect on instruction following and aesthetics. On a 33-task diagnostic subset with GPT-6-Astra and Seedance 2.5, each of these ablations lowers scores on all four dimensions; on the same subset, replacing Blender with Three.js leaves both above direct generation, with Blender scoring higher on all four dimensions at roughly twice the agent tokens per case.

Perspective

The conclusions rest on blinded human scores over WorldMemBench, VGIF-Bench and Principia, plus World Models' Last Exam in Physics, R2M, VTR and 200 randomly sampled T2AV-Compass tasks; all methods use Seedance 2.5 for synthesis, ablations run on a 33-task diagnostic subset, and the long-video comparison covers 29 LongAV-Compass and 50 WBench cases.

In long-video evaluation, Coda-DaVinci trails direct generation in transition stability on LongAV-Compass and in setting adherence on WBench, leaving gaps in cross-clip continuity and scene fidelity; on World Models' Last Exam in Physics it ranks second to Claude + Blender, and the authors attribute part of the remaining gap to the coding model's ability to implement physical behavior.