Motubrain folds video prediction and action generation into one model: 96.1% on randomized RoboTwin 2.0 and 54.4× faster inference
Synopsis
Motubrain merges future-frame prediction and action generation into a single world action model, so a robot no longer has to predict video first and then invert it into actions. It reaches average success rates of 95.8% and 96.1% on the clean and randomized RoboTwin 2.0 settings and an EWMScore of 63.77 on WorldArena. The same model gains a 54.4× end-to-end speedup over the naive baseline, reaching 11 Hz, through fewer denoising steps, compilation, FP8 quantization and caching.
Interpretation
A robot control model no longer splits "predict the future frames" from "decide the action"; Motubrain emits video and actions from one generative framework. The earlier route first used a video generation model to forecast future visual trajectories, then an inverse dynamics model to infer actions from the generated frames, so video-prediction errors accumulated over time and degraded action accuracy. On RoboTwin 2.0, Motubrain reaches average success rates of 95.8% in clean scenes and 96.1% in randomized scenes, the only compared model above 95% under randomization.
The model splits text, video and action into three parallel Transformer streams and applies joint video-action attention only in the middle half of layers. This H-bridge attention design keeps modality-specific representations in the shallow and deep layers and performs semantic alignment and policy grounding only in the middle, cutting the cost of dense cross-modal attention. Training data is organized in four levels, from Internet video through ego-centric video and heterogeneous dual-arm robot data to target-embodiment data, with video and action each trained by flow matching on their own velocity fields.
On the deployment side, the inference loop is decoupled from the robot execution loop, reaching an 11 Hz real-time inference frequency on cloud GPUs. A naively deployed world action model runs far below the control rates needed for high-frequency manipulation, with joint denoising of high-dimensional video and action latents as the main bottleneck. Stacking noise sampling, torch.compile, FP8 quantization, DiT caching and V2A-style asymmetric inference cuts end-to-end latency from 4.90 s to 0.09 s, a 54.4× speedup over the baseline.
Perspective
The demonstrated scope covers the 50 tasks of the RoboTwin 2.0 simulation benchmark, the WorldArena world-model evaluation, and real household manipulation tasks on several humanoid platforms; each real task is trained on 100 task-specific trajectories, and adapting to a new embodiment needs 50–100 same-embodiment trajectories. Inference runs remotely on cloud GPUs rather than on the robot itself.
The original WorldArena study reports that EWMScore correlates only weakly with downstream action-planning success (r=0.36), so a high perceptual score does not automatically mean better control. Real tasks still contain failed steps; the flower-arrangement task scores 83.30 overall, below the drink-pouring and cocktail-mixing tasks. The authors list reducing same-embodiment data needs and extending to long-horizon mobile manipulation and tactile interaction as future directions.
