DreamX turns a video diffusion model into a universal latent action learner, letting human video motion transfer zero-shot to robots
DreamX uses a pretrained video diffusion model itself as the latent action encoder, reading compact action representations out of its spatiotemporal features with learnable queries and supervising them by reconstructing future frames through the same model's generative prior. On LARYBench it lowers action-regression MSE from 0.38 for the strongest baseline, DiLA, to 0.17 and raises action-classification accuracy from 62.25% to 69.67%. Without paired human-robot training data, the shared latent space supports any-to-any video transfer and zero-shot human-to-robot action transfer.