Humanoid whole-body control splits by joint degree of freedom: H2MARL cuts foot slip to 0.5367 on MuJoCo
Synopsis
Humanoid whole-body control has treated joints or limbs as homogeneous agents, or trained high-level planning and low-level execution separately, which yields uncoordinated motion. H2MARL models each torque-actuated joint degree of freedom as a heterogeneous agent carrying a physical descriptor, and lets a high-level pose planner and a low-level joint executor share one closed-loop trajectory and be optimized together. On the MuJoCo Humanoid-v2 walking task, against the closest comparator MAPPO, H2MARL improves foot slip, torso stability and gait symmetry by 7.3%, 35.0% and 14.8%, while motion smoothness is slightly worse than MAPPO.
Interpretation
Humanoid whole-body control no longer treats the robot's joints as homogeneous units; each torque-actuated joint degree of freedom becomes an independent agent with its own physical properties. Earlier multi-agent reinforcement learning methods either partitioned joints or limbs into homogeneous agents or applied generic heterogeneous-agent algorithms, overlooking differences between hip joints and elbow joints in torque, load and role in balance. On the MuJoCo Humanoid-v2 walking task, H2MARL beats HAPPO, MAPPO and HATD3 on foot slip, torso stability and gait symmetry, improving on MAPPO by 7.3%, 35.0% and 14.8% respectively.
The high-level pose planner and the low-level joint executor are no longer trained separately; they are updated together in a closed loop through the same interaction trajectory. Earlier hierarchical controllers typically trained the low-level executor first and froze it while training the high level, so updates to the high-level policy drifted away from the executor and the two layers cooperated poorly. On the Isaac Gym benchmark, the variant without closed-loop co-optimization shows markedly higher cost of transport, motion jerk and centroidal angular momentum than the complete model, indicating the mechanism is needed for energy-efficient, stable locomotion.
Each joint-degree-of-freedom agent carries a physical descriptor covering its joint type, actuation axis, torque limit, motion range, kinematic location and inertia-related properties. These descriptor values are read directly from the simulators rather than set by hand, letting different joints learn specialized control behavior while centralized training keeps the body coordinated. On Isaac Gym, replacing the heterogeneous agent formulation with MAPPO's homogeneous formulation degrades every metric by a small but consistent margin.
Perspective
Results come from simulation experiments on the MuJoCo Humanoid-v2 walking task and the Isaac Gym benchmark, with a robot of 10 joints and 17 actuated joint torques, each torque modeled as a heterogeneous agent; total single-step inference takes 9.259 ms, above 100 Hz.
Motion smoothness is slightly worse than MAPPO, which the authors attribute to more frequent joint-level adjustments from the joint-degree-of-freedom heterogeneous formulation; the variant without closed-loop co-optimization has the lowest torso stability, which the authors explain as reduced torso motion at the expense of overall locomotion quality.
