Skip to main content
Back to timeline
arXivSource publication:

Humanoid carrying drops the motion-capture studio: MOVER reaches 93% simulated success using only an onboard camera

Synopsis

Humanoid object carrying has relied on reference trajectories generated in real time and could only be evaluated inside a motion-capture studio, so a change in object position tended to destabilize the robot. MOVER retargets videos of people carrying objects onto a 29-DoF Tiangong 3.0 using GENMO and GMR, then uses a teacher-assistant-student reinforcement learning pipeline that gradually relaxes trajectory tracking and distills away proprioceptive position observations with DAgger-PPO. Simulated carrying success reaches 93%, and the distilled policy carries objects at different positions in a motion-capture-free setting using only its onboard RGB-D camera.

Source-provided article image: From Video to Vision: Learning Generalizable Humanoid Carrying from Human Motion
arXiv

Interpretation

Humanoid object carrying can proceed without real-time reference trajectories or a motion-capture system, relying at deployment only on proprioception and the object's relative pose seen by an onboard camera. Earlier video- or mocap-based reinforcement learning approaches required an upper-level module to generate reference trajectories in real time, so the robot could only lift objects from designated locations along the trajectory, and policies could only be evaluated inside a motion-capture studio. In simulation, the assistant policy's carrying success rate is similar to the teacher's with no position offset and clearly exceeds it as the offset grows; after distillation, the student policy's grasping success rate is consistently higher than both the teacher's and the assistant's.

Training starts from a small set of videos of people carrying objects, first learning to follow trajectories strictly, then learning to find objects on its own, and finally shedding dependence on global position. Directly retargeting a video yields only one humanoid-object trajectory, with limited variation in object geometry and placement. 100 videos are augmented into 4000 trajectories with varied initial object positions and shape-size configurations, retargeted to a 29-DoF Tiangong 3.0, trained in Isaac Lab and evaluated in MuJoCo.

At deployment, an LSTM network called QuatNet, trained with supervision on reference motion data, regresses the robot's anchor reference orientation from the object's relative pose and the target position, replacing the reference motion term in the observation. If all reference information were distilled away, the robot could not determine the target direction after lifting the object, because the object's relative pose no longer provides guidance. QuatNet is trained on 500 motion sequences, about one million frames, with 20% held out for validation; the policy using QuatNet matches the distilled student's grasping success rate but has a lower carrying success rate.

Perspective

The demonstrated scope is object carrying by a 29-DoF Tiangong 3.0 humanoid in simulation and on the real robot: simulation evaluates four object positions and two shape-size configurations, and the real robot is tested at three locations each in a motion-capture room and a conventional indoor room, 10 trials per location. The intended readers are researchers and engineers working on humanoid whole-body manipulation and sim-to-real transfer.

After proprioceptive position observations are distilled away, the robot cannot accurately deliver the object to the designated location, leaving a non-negligible target-position error; training data volume is insufficient, making it hard to generalize to carrying behaviors with more diverse shapes and motion patterns. The policy that uses QuatNet in place of the reference motion observation has a lower carrying success rate than the distilled student policy, indicating that a target point supplied through QuatNet guides the carrying direction less directly than the reference motion does.