A 2B embodied VLM learns to replan after its own mistakes, lifting strict success 10.25 points over its supervised start
Synopsis
Embodied vision-language models that make an execution error often need to replan from the damaged scene state to keep going. ERR-Env, built on RoboTwin 2.0, provides an executable, verifiable interaction environment whose task-specific checkers judge progress, completion, execution errors and visibility; ERR-Bench built on it holds 1,503 held-out snapshots spanning nominal, intermediate and error states. ERR-VLM, initialized from Qwen3-VL-2B-Thinking and trained online with AGT-GSPO, raises Strict Success Rate for error-free completion by 10.25 percentage points and Recovery Rate by 22.95 percentage points over its supervised initialization.
Interpretation
Error recovery in embodied vision-language models can now be trained directly on verifiable action outcomes rather than only on fixed inference-time policies or reference corrections. Inference-time approaches rely on prompting, external monitors or dedicated mechanisms to trigger replanning while the policy stays fixed, so recovery is bounded by existing model capability; reference-based post-training does not optimize the policy against verified outcomes of recovery actions. On the 1,503 held-out snapshots of ERR-Bench, the full ERR-VLM raises Strict Success Rate from 10.64% to 20.89% and Recovery Rate from 42.52% to 65.47% over its supervised initialization.
Training feedback is assigned per interaction turn and compared within action types, so corrective and error-inducing actions receive different signals. A trajectory-level signal cannot separate error-inducing from corrective actions within one trajectory, and normalizing across action types mixes action-specific reward scales into the advantage estimate. In ablations, replacing turn-level optimization with trajectory-level GSPO drops Strict Success Rate to 8.05% and Recovery SSR to 5.59%; removing semantic advantage shaping lowers Recovery Rate from 65.47% to 52.89%.
Beyond the training environment, the policy performs long-horizon manipulation and recovery zero-shot on simulation benchmarks and a physical robot arm. The supervised initialization almost never moves the arm away to restore its view in real scenes, with an Active Re-observation Rate of 0.0%. On the real robot the full policy reaches an Active Re-observation Rate of 78.7%; on RMBench Observe and Pick Up it rises from 50.0 to 90.0, and on RoboVoLo Memory from 37.0 to 72.8.
Perspective
Evaluation covers 1,503 held-out ERR-Bench snapshots plus zero-shot trials on RMBench, RoboVoLo and an AIRBOT arm with 10 trials per task; training and checkers are built around a pick-and-place high-level action space and a limited set of error types.
The authors state that the current setting is limited to a compact high-level action space and a limited range of error types, that training runs in a single simulation environment with a checker tailored to pick-and-place tasks, and that computational constraints meant primarily training a 2B model rather than systematically studying model scaling.
