InteractVLA turns robot policies into steerable tools: an agent sets the end-effector pose, reaching 99.4% on LIBERO-Plus
Synopsis
When a vision-language agent calls a robot policy, it has so far been limited to a language instruction; after a failure, retries repeat nearly the same motion, and rephrasing pushes the policy out of distribution. InteractVLA builds a bidirectional geometric interface at test time: the agent specifies end-effector position, orientation and gripper state, a differentiable mapping turns that into input gradients through the action expert to steer generation, and generated action chunks with calibrated camera reprojections return to the agent for diagnosis and correction. With recursive self-improvement, InteractVLA reaches 99.4%, 48.62% and 89.4% overall success on LIBERO-Plus, RoboDojo and RoboCasa365.
Interpretation
A pretrained vision-language-action policy no longer has to be invoked by a language instruction alone; the agent can specify the end-effector position, orientation and gripper state it wants, and the same policy redirects its execution. Such policies were treated as opaque language primitives: a retry after failure produced nearly identical motions under similar observations, while rephrasing the instruction often pushed the policy out of distribution and sharply lowered success. On LIBERO-Plus, replacing the language-only agent-plus-policy combination (93.1%) with InteractVLA raised overall success to 98.3%, with layout perturbations going from 89.5% to 98.4% and camera viewpoint shifts from 87.6% to 98.4%.
Steering happens inside the policy's own denoising process rather than as an external correction added to the output. The agent supplies only a sparse target pose, not a dense trajectory or an exact arrival time; a fixed-step penalty pulls the wrong execution step toward the target, while penalizing every step uniformly collapses the approach trajectory. Under an identical policy backbone, pose target and self-improvement budget, soft-time alignment reached 99.4%, 48.6% and 89.4% on LIBERO-Plus, RoboDojo and RoboCasa365, above fixed final step at 96.8%, 39.1% and 83.0% and uniform weighting at 95.6%, 34.7% and 79.8%.
A reverse channel shows the agent the policy's generated action chunks and calibrated camera reprojections, so it can tell a wrong target from an execution slip and choose between re-steering and local adjustment. A binary success or failure signal was insufficient for recovery, and the agent previously had to take over low-level control entirely. On RoboDojo and RoboCasa365, observation-only feedback gave 40.3% and 84.1%, adding generated actions gave 44.8% and 87.0%, and adding reprojection gave 48.6% and 89.4%.
Perspective
The demonstrated scope is the simulation benchmarks LIBERO-Plus, RoboDojo and RoboCasa365, plus four tasks on the physical Dexmal bimanual platform with 30 trials each, using MR-0 as the policy backbone trained on 50 demonstrations per task.
Static ground-truth pose targets lifted RoboDojo overall success from 9.2% to 12.17% but slightly degraded precision and long-horizon tasks, indicating that a static spatial constraint alone cannot resolve multi-step sequencing or contact dynamics; only the online loop raised overall success in that setting to 26.0%.
