A LEAP hand now reaches any in-hand object pose at any wrist orientation: a residual safety filter cuts drops from 16 to 4
Synopsis
A multi-fingered hand can now carry an object to almost any position and orientation in the hand, where earlier in-hand skills were mostly limited to a single wrist orientation and simple repeated motions and rarely measured drops. The recipe trains a LEAP hand in simulation without human data and transfers zero-shot to the real hand, using either motion capture or a single wrist depth camera. Over 13.3 hours of real-robot evaluation on 4,800 unseen goals, the state policy reached 95.2% of goals within 2 cm and 0.4 rad; adding a learned residual safety filter with an adjustable conservativeness cut its drops from 16 to 4 while still reaching 91.5% of goals.
Interpretation
A multi-fingered hand can now move an object consecutively to almost any position and orientation in the hand at any wrist orientation, and reports not dropping the object as a separately measured metric. Earlier in-hand skills mostly operated at a single wrist orientation, often palm up, and performed simple repeated motions such as rotation, while reliability, safety and robustness were treated as afterthoughts. Real-robot evaluation covered 13.3 hours and 4,800 unseen goals across 4 objects and 6 wrist orientations; the state policy reached 95.2% of goals within 2 cm and 0.4 rad.
A residual safety filter trained on top of the frozen task policy corrects each action only slightly and still greatly reduces drops. Building safety into the task reward turns it into a hard-to-tune multi-objective problem and requires retraining the task policy, whereas the filter is a separate layer trained in about 4 hours. The state policy's drops fell from 16 to 4, a 75% cut, mean time to first drop rose from 334 s to 465 s, and coarse success fell from 95.2% to 91.5%.
The same policy family can run from only a wrist depth camera, without motion capture or external cameras. The state policy relies on motion capture for object pose, while the depth policy replaces that input with a 160×120 wrist depth image. On the real robot the depth policy reached 71.8% of goals at the coarse tolerance and 66.7% with the filter, with drops falling from 88 to 46; it reached 95% and 88% on the lightbulb and rubber duck but fell well below simulation on the knife and wineglass.
Perspective
Evaluation ran on the real robot over 4 objects, 6 wrist orientations and 4,800 unseen goals, with a 10 s window per goal and a drop defined as the object more than 25 cm from the hand for three consecutive tracked control steps; policies and filters are trained per object, and the wrist-camera depth policy falls clearly below simulation on the knife and wineglass.
The depth policy underperforms simulation on the real robot for the knife and wineglass, and its auxiliary pose estimate shows real errors with long tails absent in simulation, offered as a possible perceptual cause; the depth wineglass is the exception to the drop reduction, falling only from 51 to 40. In downstream tasks the filter at the chosen conservativeness did worse, and the reasons are unclear.
