Visual world models drift over long rollouts: JointGV adds a parallel graph rollout and cuts joint-position error by 11–28%
Synopsis
Recursive visual world models drift in entity pose and spatial relations, which limits long-horizon robot planning. JointGV rolls out a task-conditioned scene graph alongside the existing visual model, couples their action-induced changes, and applies a bounded residual correction whose gradients never enter the original model. On DMC Cheetah Run, Walker Walk and Finger Turn Hard, 32-step joint-position error falls by 11–28%.
Interpretation
A visual world model can suppress long-horizon drift by borrowing a separately predicted relational state, without restructuring itself. Earlier routes changed the native predictive representation, learned residuals inside the same latent pathway, or replaced the visual state with object- or graph-centric dynamics; none addressed correction when graph nodes and visual states share neither elements nor a feature space. On DMC Cheetah Run, Walker Walk and Finger Turn Hard, 32-step joint-position error drops by 11–28%, and both DINO-WM and DreamerV3 visual states benefit.
The correction acts on predicted changes rather than absolute features. Graph-node and visual-state changes induced by the same action are projected into a shared space and softly matched, using Sinkhorn optimal transport for spatial tokens and normalized attention pooling for a global latent. Across three seeds on DINO-WM Cheetah Run, a visual-only residual moves 32-step error from 0.898 to 0.894 pixels, while JointGV reaches 0.791 pixels.
The graph rollout also supplies predicted node positions as a spatial reference for correction. Change alignment captures state evolution but not absolute image location, so the correction module also receives node features routed by their predicted positions. Broadcasting the same mean graph motion to every token already improves the backbone, and cosine and OT coupling lower 32-step error further to 0.793 and 0.791 pixels, with similar accuracy.
Perspective
The demonstrated scope is DMC Cheetah Run, Walker Walk and Finger Turn Hard, plus open-loop prediction on real RM65-B inverted-pendulum recordings and MuJoCo simulated control; stable six-second balancing on physical hardware has not been achieved, and high-speed operation introduces non-negligible command delay.
Correction depends on graph observation and prediction quality, and overlapping links can cause node-localization errors; both configurations degrade as graph-initialization error grows, with the Sinkhorn OT variant still improving under larger perturbations while the primary plug-in becomes worse than its backbone; roles are currently task-defined and supervised.
