Skip to main content
Back to timeline
arXivSource publication:

Human-hand affordances don't transfer to grippers as-is: ART rebuilds 3D regions into robot-executable maps and reaches 86.5% success over 480 real trials

Synopsis

Human-annotated 3D affordance regions are cheap and abundant, but human hands and robot end-effectors differ in structure, so applying them directly leaves robots unable to grasp or push reliably. ART treats this mismatch as an explicit cross-embodiment transfer task: a geometric reliability gate removes human-prior regions a rigid gripper cannot reach or hold stably, and hierarchical reconstruction with graph diffusion rewrites the continuous surface-like prior into discrete robot-executable contact distributions. On the paired human-robot HPAD dataset, ART holds the highest absolute scores on both the seen and unseen category splits; on a 6-DoF arm with a parallel-jaw gripper, ART attains 86.5% success over its 480 trials in a 2,400-execution physical benchmark.

Source-provided article image: ART: Bridging the Embodiment Gap in 3D Human-to-Robot Affordance Transfer
arXiv

Interpretation

A 3D affordance region is no longer treated as a fixed property of the object but as something that changes with the agent's embodiment, so moving from a human prior to a robot-executable region requires an explicit transfer. Earlier methods mostly treated human priors as universally applicable reference regions, assuming that what a person can reach a robot can use. Statistics on HPAD show human priors generally activate larger regions than robot labels, and their spatial overlap falls from above 0.3 to below 0.2 as the confidence threshold rises, so the discrepancy persists in high-confidence regions.

ART performs this transfer with two streams: one judges which human-indicated regions are physically feasible for a rigid gripper, the other rearranges the surface-like prior into robot contact patterns. The geometric reliability gate predicts per-point reliability from multi-scale point features and a condition vector built from the action and object size, supervised with scale weights to penalize geometric false positives in the human prior. Hierarchical affordance reconstruction injects learnable action-indexed robot prototypes through cross-attention, aligning a coarse reference to mid and fine scales, and a graph diffusion refiner then runs anisotropic diffusion on a dynamically built k-nearest-neighbor graph to correct the topological mismatch between human and robot affordances.

On HPAD, ART holds the highest absolute score on all four metrics, MAE, aIoU, IoU0.9 and F10.9, on both the seen and unseen splits. The unseen split holds out entire object categories, which are never observed during training. On the seen split ART reaches aIoU 69.06%, IoU0.9 56.45%, F10.9 67.88% and MAE 0.032; on the unseen split the values are 43.90%, 29.70%, 39.08% and 0.101.

On a physical benchmark with a Dobot CR5A 6-DoF arm and a Lebai parallel-jaw gripper, ART attains 86.5% success over its 480 trials. Applying the human prior directly succeeds in only 55.0% of its corresponding trials, while the Cross-Attention and FiLM fusion baselines reach 70.8% and 72.5%. The benchmark totals 2,400 physical executions across 12 objects and 4 action categories, 4 of the objects are absent from HPAD, and point clouds are reconstructed from real multi-view RGB-D observations.

Perspective

The demonstrated scope is tabletop manipulation with a parallel-jaw gripper on a 6-DoF arm, limited to four action categories, grasp, push, drag and lean, with point clouds reconstructed from real multi-view RGB-D observations in the physical trials. The authors propose extending the paradigm to dexterous hands and other end-effector morphologies, and making the transfer itself embodiment-conditioned so it generalizes without per-gripper specialization.

The authors state that the current scope covers only parallel-jaw grippers, and that dexterous hands have qualitatively different contact primitives and target distributions, so the conclusions do not extend to other end-effectors. HPAD robot labels come from simulation plus manual review; the authors analyze label sensitivity to simulation parameters in an appendix, but the main text does not report that analysis's conclusion.