DreamX turns a video diffusion model into a universal latent action learner, letting human video motion transfer zero-shot to robots
Synopsis
DreamX uses a pretrained video diffusion model itself as the latent action encoder, reading compact action representations out of its spatiotemporal features with learnable queries and supervising them by reconstructing future frames through the same model's generative prior. On LARYBench it lowers action-regression MSE from 0.38 for the strongest baseline, DiLA, to 0.17 and raises action-classification accuracy from 62.25% to 69.67%. Without paired human-robot training data, the shared latent space supports any-to-any video transfer and zero-shot human-to-robot action transfer.
Interpretation
DreamX makes a pretrained video diffusion model serve directly as the latent action learner, reading action representations disentangled from appearance so that one latent action space is shared across humans and robots, environments and action parameterizations. Earlier latent action models largely learned the action space from scratch on single frame pairs or local visual transitions, and reconstruction accuracy alone does not guarantee that actions are separated from domain-specific visual factors such as viewpoint and scene appearance. On LARYBench, DreamX reaches an action-regression MSE of 0.17 against 0.38 for the strongest baseline, DiLA, and an action-classification accuracy of 69.67% against 62.25% for DiLA.
The mechanism appends learnable query tokens to the video token sequence so they gather action-relevant information across frames through the backbone's spatiotemporal attention, condensing each latent video frame into a few latent-action tokens. Those latent actions replace text conditioning and are fed with the initial frame into the same pretrained video model to reconstruct future frames; the initial frame supplies scene appearance and the generative prior supplies visual detail, so the reconstruction objective pushes latent actions toward the motion needed to explain the future. The model is built on the Cosmos-Predict2.5 2B backbone and pretrained on about 2.19k hours of human and robot video, with separate rank-16 LoRA modules for encoder and decoder trained for 78,000 steps while backbone weights stay frozen.
Perspective
The demonstrated scope covers representation-quality evaluation on LARYBench, zero-shot human-to-robot video transfer on 1,008 clips spanning 21 tasks from HumanAndRobot, and open-loop action transfer plus a real policy study on three drawer-closing configurations with a bimanual YAM robot. The policy study evaluates 10 trials per scenario and reports an inference latency of 1.47 s per policy call.
Cross-embodiment transfer is not an explicit training objective but is described as a capability emerging from pretraining without additional training. Action transfer is executed open-loop without visual feedback, and policy results are limited to the evaluated drawer-closing setting; cross-embodiment transfer and robot policies may fail outside the evaluated settings and require safety validation before deployment.
