Robots no longer have to guess their own motion: Doppel renders future bodies into a video world model, lifting PSNR by 1.00 dB on unseen trajectories
Synopsis
Video world models usually take only numerical action vectors, so the model must learn both how joints move the body in the image and how the body changes the scene. Doppel conditions on future robot renderings along a proposed joint trajectory together with the numerical actions, letting a shared model focus on object and environment response. On the Droid, Bridge and RoboMind robot-arm datasets it improves PSNR over an action-only baseline by 1.00 dB on unseen trajectories and 1.20 dB under out-of-distribution evaluation.
Interpretation
The conditioning input of a video world model gains an image channel: the future body rendered along the proposed joint trajectory, fed in alongside the numerical action trajectory. Earlier approaches conditioned only on numerical action vectors, so the model had to learn how joint configurations become image-space body motion, a mapping that robot geometry and camera parameters already determine. On the Droid, Bridge and RoboMind robot-arm datasets, PSNR rises by 1.00 dB on unseen trajectories and 1.20 dB under out-of-distribution evaluation over the action-only baseline, with more consistent object motion.
Rendering places the motion of different bodies in one image space, so a single set of weights can be shared across embodiments. Robots differ in joint dimensionality and meaning, which usually calls for separate models or embodiment-specific modules. One shared video backbone and conditioning adapter are trained jointly on Droid, Bridge and RoboMind, covering Franka, UR5e and WidowX over about 600 hours of video without an embodiment label; scaling to eight embodiments (Doppel-8) improves over the three-robot model (Doppel-3) on four of six robot evaluation sets.
Rendering needs the relative pose between camera and robot base, and many offline manipulation datasets lack these extrinsics. Without extrinsics the rendered body cannot be aligned with recorded video from the same camera viewpoint, so conditioning videos cannot be built across datasets. A differentiable renderer aligns rendered trajectories with recordings using RGB, optical flow and DINOv2 features, recovering camera extrinsics for 45,345 demonstrations across WidowX, UR5e and bimanual YAM, released together with calibration code.
Perspective
Evaluation covers embodiments seen during training, with one camera per trajectory; the findings rest on the Droid, Bridge and RoboMind robot-arm datasets plus five further datasets that include a humanoid and human bodies.
Building the renderings requires accurate robot geometry, a URDF and camera calibration; missing extrinsics in offline datasets can be recovered, but unknown or rapidly changing viewpoints remain a limitation. The method conditions on configuration trajectories, and handling imperfect geometry or lower-level control commands would require modeling uncertainty in robot motion and execution. Transfer to unseen embodiments, multi-view prediction and longer horizons is left to future work.
