Skip to main content
Back to timeline
arXivSource publication:

Robot-action videos get more than smooth motion: ReactWorld writes roles and local responses into generation, scoring 77.51 on RBench

Synopsis

Embodied video world models can produce smooth footage while objects still fail to respond according to task intent and contact relations. ReactWorld first predicts an interaction response field in the generation backbone's feature space, specifying who participates, which local regions respond, how strongly, and how they move afterward, then feeds those predictions into object feature updates and cross-frame information retrieval. On RBench it reaches an average score of 77.51, 3.29 points above PhysisForcing with gains in all nine categories; on EZS-Bench it reaches a Domain score of 86.98 and an average of 81.87.

Source-provided article image: ReactWorld: Structuring Interaction Responses for Embodied Video Generation
arXiv

Interpretation

Embodied video generation starts treating when, where and how strongly an object should respond as a generation condition, rather than pursuing overall motion coherence alone. Earlier image-to-video models generate realistic appearance and coherent motion, yet a target may move before contact, a grasped object may detach from the gripper, or nearby objects may respond unnecessarily. On RBench, ReactWorld averages 77.51, above Video-FT trained on the same data at 71.94, and 3.29 points above PhysisForcing, outperforming PhysisForcing in all nine categories.

Interaction prediction happens inside the same denoising forward pass and directly rewrites object features and cross-frame attention in the generation backbone. The interaction response field predictor reads intermediate backbone features to give participant roles, future motion response intensity and interval motion; the response feedback adapter forms a feature increment from the difference between the object branch's outputs with predicted and zeroed local dynamics, and turns predicted displacement into a temporal attention bias. Training uses roughly 500K clips curated from AgiBot, RoVid-X and robot videos collected from the Internet, with 81-frame videos at 16 FPS, and both prediction and feedback are applied at the 12th DiT block.

Perspective

Evaluation covers five task categories and four robot morphologies on RBench, plus embodied domain performance and perceptual quality on EZS-Bench; RBench is evaluated with Qwen3-VL-235B-A22B-Instruct, EZS-Bench Domain with Qwen2.5-VL-72B-Instruct, and Quality with the benchmark's eight video quality metrics. Inference needs only a task instruction and an initial image, without external role maps or trajectories.

Interaction prediction currently uses a 2D response representation and does not explicitly model 3D interaction relations; the authors propose exploring temporally consistent 3D motion representations and rehearsal to improve spatial consistency. Displacement bias strength stays close between 0.5 and 1.5 times the default and decreases at 2 times; insertion at block 12 gives a Domain score of 86.98, versus 85.41 at block 8 and 86.01 at block 16.