Skip to main content
Back to timeline
arXivSource publication:

Long-horizon robot manipulation stops relying on one global instruction: GTI grounds each subtask and phase in a visual target, lifting real-robot full-task success from 21.7% to 59.2%

Synopsis

Vision–language–action models generate actions from images, language and robot state, yet a single global instruction stays fixed across a long task, so the policy can skip or repeat subtasks or switch phases too early. GTI uses the current observation and execution history to localize the active subtask and its phase, then generates a phase-specific visual target, either contact or local completion, that conditions the VLA together with a local language instruction. On RoboCasa365, GR00T success on seen composite tasks rises from 14.6%/12.1% to 31.8%/20.4% under ID/OOD evaluation, and mean full-task success on three real PiPER tasks rises from 21.7% to 59.2%.

Source-provided article image: GTI: Grounding Task Intent Throughout Long-Horizon VLA Manipulation
arXiv

Interpretation

A global instruction for long-horizon manipulation becomes a local intent that updates as execution proceeds: the system keeps deciding which subtask is active and which interaction phase it is in, and supplies a local language instruction plus one phase-matched visual target image. Earlier hierarchical approaches either turned high-level reasoning into local language instructions or used predicted future states as execution references; language says what to do but leaves contact geometry to the executor, and a single completion state cannot express both the contact relation and the completion relation inside one subtask. On RoboCasa365, against a Global-FT baseline starting from identical checkpoints with the same seen trajectories and fine-tuning budget, the GR00T executor's seen composite success rises from 14.6%/12.1% to 31.8%/20.4% under ID/OOD evaluation.

The system has three parts: a planner that localizes the subtask and phase from the current observation and history, a visual predictor adapted from a pretrained video generator that produces the phase-specific target image, and an anomaly monitor that triggers rollback, re-observation and phase re-evaluation after persistent deviation. Phase is an explicit conditioning variable in both training and inference; K1 is the first state in which the current atomic operation establishes the task-relevant robot–object or object–environment interaction relation, and K2 is the first state in which the local subtask completion predicate becomes true, with the confirmed phase concatenated into the local instruction as text conditioning for the video generator. On VLA-Arena Horizon-Long, with planner and visual model trained only on L0+L1 and L2 held out, the matched configuration with K1/K2 phase goals but no recovery reaches 47.0% overall success, above 44.8% for phase text alone and 44.9% for a single K2 goal, and enabling local re-grounding raises it to 47.6%.

Anomaly monitoring during execution reads internal features of the VLA's action generation rather than visual prediction error; after a persistent anomaly the system keeps the current subtask and cached target image, rolls back and stabilizes the robot, then re-evaluates phase from the new observation. Recovery fires only when the anomaly score exceeds a calibrated dynamic threshold for two consecutive policy chunks, at most once per subtask, and simulator state is never reset, so recovery is a local deployment-time repair rather than global replanning. On three real AgileX PiPER 6R tasks, each condition tested 30 times independently from the same initial robot pose, mean full-task success rises from 21.7% for Global-FT to 59.2%, with gains concentrated on multi-step conditions such as the four-stage cabinet search-and-place going from 0/30 to 12/30.

Perspective

The demonstrated scope is RoboCasa365 simulation, VLA-Arena Horizon-Long simulation and three real tasks on an AgileX PiPER 6R, with each real condition tested 30 times independently from the same initial robot pose; the authors state that GTI complements rather than replaces the underlying VLA, and success still depends on planning, milestone feasibility, low-level interaction and error correction.

The authors report that complex held-out tasks and cross-scene generalization remain difficult, and OOD gains are smaller than ID gains; the two K1/K2 phases may be too coarse for repeated contact, re-grasping or mid-course strategy switches; local recovery retains the current subtask instead of globally replanning or selecting an alternative strategy.