EyeRobot 2.0 uses active gaze to achieve fine bimanual manipulation with a single stereo camera, raising success from 22% to 48% when wrist cameras are occluded
Synopsis
The work introduces EyeRobot 2.0, a framework in which active gaze physically swivels two eye viewpoints to center on a 3D fixation point and processes the resulting images foveally by allocating more visual tokens to image centers, enabling fine-grained bimanual manipulation with only a single stereo camera; across 7 real-world and 6 simulated tasks and over 1000 physical and 1800 simulated robot trials, it outperforms passive stereo policies by 40% in real and 20% in sim, matches ego + wrist policies when wrist views are clear (69% vs. 64%), and more than doubles their success when grasped objects occlude the wrist cameras (48% vs. 22%).
Interpretation
It proposes a framework that uses active gaze for fine-grained bimanual manipulation: two eye viewpoints swivel to center their gaze on a 3D fixation point, and the resulting images are processed foveally by allocating more visual tokens to image centers, focusing computation on task-relevant features. Compared with conventional setups that rely only on passive stereo or on ego + wrist cameras, this work turns where to look into a controllable action variable rather than a fixed camera layout. The abstract describes the framework and reports comparisons across 7 real-world and 6 simulated tasks, with over 1000 physical and 1800 simulated robot trials.
Gaze coordination is achieved hierarchically: a low-level gaze servoing policy conditioned on a goal object is trained first, then a target selector emits fixation goals based on task progress. Splitting gaze control into low-level servoing and high-level target selection lets the fixation sequence adapt as the task progresses. Both modules are trained with RL on real-world data; the first uses a dense geometric reward, and the second co-trains with the BC gripper policy.
Co-training the target selector with the BC gripper policy allows it to discover fixation sequences that can resemble a human's fixation sequence. Fixation goals are learned through co-training with the manipulation policy rather than being hand-specified. The abstract states that co-training 'allows it to discover fixation sequences that can resemble a human's fixation sequence'; no quantitative similarity metric is given.
It exploits fixation by canonicalizing gripper information into a fixation-relative SE(3) frame, compacting the size of the action distribution to learn. The fixation point serves as the reference frame for action representation instead of a fixed world or camera frame. The abstract describes this mechanism and links it to closing the gap with wrist-camera policies when only stereo is used.
Perspective
The work targets robot systems that need to use as few cameras as possible in fine bimanual manipulation, especially where wrist views are easily occluded by grasped objects. It applies to platforms with swiveling eye viewpoints that can collect teleoperation data; the reported results rest on 7 real-world and 6 simulated tasks and over 1000 physical and 1800 simulated trials, compared against passive stereo and ego + wrist policies trained on the same data. For systems without an actuated gaze mechanism or comparable teleoperation data, applicability needs separate evaluation.
The abstract gives no quantitative metric for how closely the fixation sequences resemble human fixation sequences, nor does it state the precision, latency, or failure modes of gaze servoing; the composition of the real and simulated tasks, data scale, and training details require the main text. The abstract also does not ablate how much the fixation-relative SE(3) representation contributes to the final success-rate gains. These are open questions to confirm by reading the main text.
