Ego2X turns first-person video into robot data by tracking object contacts, lifting X-VLA success from 40.5% zero-shot to 63.5% on paired scenes
Synopsis
Converting first-person video into executable robot data has relied mainly on transferring hand and arm motion and judging fidelity by visual appearance; but monocular hand estimates drift along the viewing ray while stable manipulation needs fine contact on the object surface, so fingers hover beside objects and grippers close on empty space. Ego2X instead works in the object frame, recording each contact by its surface anchor, relative force, grasp mode and interaction state, and carries that record through depth correction, inverse kinematics and video compositing. In RoboTwin 2.0, post-training X-VLA on verified Ego2X episodes alone, without simulator expert demonstrations, raises task success on paired scenes from 40.5% zero-shot to 63.5%.
Interpretation
The unit transferred from first-person video to robot data changes from hand motion to contact records in the object frame, so converted trajectories must satisfy contacts on the object surface rather than merely look like a moving human hand. Earlier ego-to-robot pipelines retargeted 3D hand kinematics to robot joints and judged fidelity mainly by visual appearance; monocular hand estimates drift by centimeters along the camera ray, while stable manipulation needs millimeter precision on the object surface. On EgoDex, contact-driven retargeting realizes more annotated contacts than the official HuRo pipeline and reaches higher physical contact fit for parallel grippers than Ego2Robot; on paired RoboTwin 2.0 scenes, post-training X-VLA on verified Ego2X episodes alone raises success from 40.5% zero-shot to 63.5%.
The representation records each contact in the object frame, including the object surface anchor, outward normal, push direction, relative force, grasp mode and interaction state. Contact points come from tactile timing intersected with camera rays on the object surface, so they do not depend on absolute hand depth; anchors are graded Certified, Trusted, Hypothesis or Uncertain by palmar orientation, adjacent-phalanx non-penetration, rigid-motion agreement and 2D occlusion ordering, and only the first two tiers constrain depth refinement and inverse kinematics. On HOT3D, contact-driven depth correction reduces in-contact hand ray-depth error and lowers monocular object translation error under estimated poses; on EgoDex, contact-driven retargeting realizes more annotated contacts than the official HuRo pipeline.
The same contact record drives cross-morphology retargeting, object-preserving arm erasure and compositing, and closed-loop simulation verification. A URDF compiler maps robot collision geometry into mode-compatible contact pads; contact-driven inverse kinematics pulls dexterous pads onto anchors or synthesizes antipodal object-frame grasps for parallel grippers, while interaction states decide which object pixels to protect during erasure. On paired RoboTwin 2.0 scenes, post-training X-VLA on verified Ego2X episodes alone raises success from 40.5% zero-shot to 63.5%, whereas unfiltered episode pools stay near zero-shot.
Perspective
The demonstrated scope is monocular first-person video conversion on three benchmarks, HOT3D, EgoDex and RoboTwin 2.0, covering dexterous hands and parallel grippers; downstream policy results come from X-VLA post-training in simulation, and physical robot deployment is still underway.
Contact localization remains limited by object pose estimation errors under heavy occlusion; contact loads are modeled as visual-kinematic priors rather than sensor-calibrated Newton forces; and physical robot deployment is not yet complete due to current hardware and schedule constraints.
