Robot policies stop replaying memorized actions: VLA-Δ supervises physical state change in pretraining and reaches 98.40% average on LIBERO
Synopsis
A Zhejiang University OmniAI team introduces VLA-Δ, which treats the physical change between the current and future state as an explicit pretraining target instead of learning only observation-to-action correspondences. Learnable physical state difference queries in the vision-language backbone are shaped by three signals: structured transition slots, dense optical flow, and future-state difference text; downstream fine-tuning and deployment keep only action generation. It reaches 98.40% average success on LIBERO, exceeds compared methods under both Clean and Random on RoboTwin 2.0, and averages 68.0% across real-robot tasks.
Interpretation
Action generation in vision-language-action models gains a direct supervision target: what changes in the robot and objects between the current and future state becomes a pretraining learning target, rather than only memorizing observation-to-demonstrated-action correspondences. Earlier approaches either describe the next step in language or predict future images or latent future visual states, supervising high-level task semantics or scene appearance but rarely the physical transition itself, such as how object poses, spatial relations and robot-object contacts change. On LIBERO the full model averages 98.40% success, above every compared method in the table; on RoboTwin 2.0 it averages 90.06% and 89.74% under Clean and Random, above all compared methods and 1.36 and 1.76 percentage points above the strongest baseline ZR-0 in the two settings.
The mechanism adds a set of learnable physical state difference queries to the vision-language backbone, so information from the current observation and task instruction first converges into a transition representation from which an action expert generates continuous action chunks. Three signals shape that representation: structured transition slots assign nine variable categories, including task progress, target and gripper positions, motion displacement, contact transitions and obstacle risk, to fixed query groups for direct prediction; dense optical flow constrains local motion detail in the image plane; future-state difference text describes task progress, spatial change, contact transitions and physical constraints in coherent language. The three signals are optimized jointly with action learning only during pretraining, downstream fine-tuning optimizes only the action objective, and deployment generates actions from current observations without decoding any intermediate supervision target; pretraining uses about 484 hours and 24 million frames from DROID, MolmoAct and RH20T, annotated by an automated pipeline.
Perspective
The demonstrated scope is the LIBERO and RoboTwin 2.0 simulation benchmarks plus five real-robot tasks on an AgileX PiPER 6-DoF dual-arm platform, each run over 40 independent episodes for 200 trials in total; the three supervision signals are used only during pretraining, downstream fine-tuning optimizes only the action objective, and deployment does not decode intermediate supervision targets.
The five real-robot tasks average 68.0% success, with 42.5% on obstacle avoidance and second-highest success among compared methods on long-horizon execution and environmental generalization; pretraining covers about 484 hours and 24 million frames, and in the annotation audit the motion and relative displacement slot has a 93.00% valid rate, the lowest of the nine slot categories.
