Recorded with glasses alone, Ego2Sim2Humanoid turns human video into humanoid training trajectories
Synopsis
Whole-body vision-language-action models for humanoids depend on teleoperated robot trajectories, each requiring a trained operator driving a physical robot in real time. Ego2Sim2Humanoid converts everyday human recordings from Project Aria Gen 2 glasses into training data with robot-view images and verified action labels, through agentic real-to-sim reconstruction, grasp and stance planning, and reinforcement-learning correction in physics simulation. Fine-tuned only on this converted data, with no real-robot demonstrations of the target tasks, GR00T N1.7 lifted a bottle in 9 of 20 autonomous trials and put on a hat in 5 of 20 on a Unitree G1.
Interpretation
Training data for whole-body humanoid policies can come from everyday human recordings made with a pair of glasses rather than from teleoperated physical robots. Prior pipelines collected demonstrations by teleoperation on physical robots during pretraining and repeated collection in each deployment scene, with every episode needing a robot and a trained operator; existing egocentric video methods track only hands and wrists and still rely on teleoperated trajectories to ground poses in whole-body joint control. On 71 hat-wearing recordings, EgoHumanoid and EgoHumanoid-V2 reached median path ratios of 0.5 to 0.6, left the hand a median 70.8 to 112.9 cm from the contact region at touch time, and admitted zero feasible grasps, whereas Ego2Sim2Humanoid reached a path ratio of 0.9, a hand distance of 0.8 cm, and a feasible grasp in 57 of 71 recordings.
The conversion executes human motion inside a physics simulation of the reconstructed scene, then renders training images through the robot's onboard camera along the same accepted rollout. Copying the human palm pose directly leaves the robot hand at the wrong distance and produces infeasible finger contacts because of differences in height, arm reach and hand morphology; earlier kinematic translation did not check whether the robot keeps balance or achieves stable contact. Across the 71 hat recordings, the copied-palm condition drove the thumb and index finger deeper into the hat wall in 65 recordings, raising median penetration from 0.8 to 1.8 cm; among the 43 recordings that succeeded under both conditions, the hat moved further in the copied-palm grasp in 36 cases, with median in-hand shift rising from 3.7 to 6.6 cm.
GR00T N1.7 fine-tuned only on the converted data performed walking-plus-manipulation tasks autonomously on a Unitree G1. The training set contains no real-robot demonstrations of the target tasks, and recording needs no robot hardware, trained operator or prior scan. In 20 autonomous trials the policy lifted the bottle 9 times (45%) and put on the hat 5 times (25%); 17 bottle trials reached the table and 8 of those failed during grasping, while all 20 hat trials reached the table and 12 failed during grasping.
Perspective
The demonstrated scope is two daily tasks, bottle lift and hat wearing, recorded with Project Aria Gen 2 glasses across 219 bottle-lift and 154 hat-wearing recordings, on a 29-DoF Unitree G1 with Dex3 hands and a head camera; mass, inertia and surface friction of reconstructed objects are estimated by a coding agent from video priors.
Reconstructed object dynamics parameters are estimated from priors; reinforcement-learning adaptation tolerates moderate parameter variation on lightweight household items, but heavy objects or payloads with uneven mass distributions can still suffer a sim-to-real dynamics gap, and representing objects as rigid bodies, parametric shapes or hinged links limits the real-to-sim stage on highly deformable or topology-changing tasks.
