Skip to main content
Back to timeline
arXivSource publication:

Tsinghua team releases HandsOn dexterous-manipulation benchmark: best of eight trained multi-task policies, OpenWAM, reaches 29.1% on Core-50

Synopsis

Tsinghua and collaborators released HandsOn, a simulation platform and benchmark for cross-embodiment dexterous manipulation with five robotic arms, six dexterous hands, about 200 rigid, articulated and deformable assets, and 66 human-verified tasks backed by 18,000 human demonstrations. Core-50 supplies demonstrations for 50 tasks used in learning and execution tests, while Transfer-16 reserves 16 tasks with no target-task demonstrations to probe asset and relation transfer, skill composition and deformable manipulation. Among eight trained multi-task checkpoints, OpenWAM has the highest Core-50 success at 29.1%, and GPT-6 with the move_eef interface, without benchmark finetuning, reaches 5.2% on Core-50 and 1.9% on Transfer-16 over 660 trials.

Source-provided article image: [2610.26765] HandsOn: A Large-Scale Simulation Benchmark for Generalizable Dexterous Manipulation
arXiv

Interpretation

Dexterous-manipulation evaluation now has a shared benchmark that puts demonstration data, task suites and failure diagnostics in one simulator, covering five robotic arms, six dexterous hands and roughly 200 rigid, articulated and deformable assets in randomized scenes. Earlier dexterous benchmarks broadened task and embodiment coverage, but their human-demonstration sets were small and their tasks rarely included ones reserved for evaluation. HandsOn provides 18,000 curated simulation demonstrations, about 56 hours of interaction with three synchronized camera views, which under one counting convention is 5.7 to 16.4 times the human trajectory count of representative dexterous benchmarks.

The benchmark splits into two suites with different jobs: Core-50's 50 tasks carry demonstrations for policy learning and fresh-rollout execution tests, while Transfer-16's 16 tasks contribute no target-task demonstrations to training and are reserved for transfer. Because the suites answer different questions, scores are reported separately: Core-50 measures execution on demonstrated tasks, Transfer-16 measures generalization to held-out tasks. All 66 tasks were verified in person by the data-collection operators in the released environment, and each is scored by a task-specific success predicate rather than a shared proxy.

On Core-50, OpenWAM has the highest success among eight trained multi-task checkpoints at 29.1%, and selecting the best checkpoint per task would give 35.4%. Single-task Diffusion Policy averages 9.7% over the 50 tasks and 9.9% on 11 matched tasks, below the 22.7% to 34.5% of seven multi-task checkpoints. Each checkpoint runs 100 episodes per task, and OpenWAM's task-and-episode bootstrap interval is 22.8% to 36.0%, with the interval width mainly reflecting the task mix.

GPT-6, without HandsOn finetuning, drives the robot through the move_eef interface by specifying palm poses and 20 finger-joint targets per hand, reaching 5.2% on Core-50 and 1.9% on Transfer-16 over 660 trials. This condition differs from the trained policies: RGB plus measured robot state as input, ten trials per task, and a model-call budget, so it is reported separately. Ten scored trials on each of the 66 tasks yielded 29 successes; 260 trials ended at the call budget, 150 hit the physical time limit, and 221 ended through native termination or task-rule violation.

Perspective

Every experiment uses one configuration: ROKAE AR5-08 seven-degree-of-freedom arms, first-generation Wuji Hands and the NVIDIA PhysX back-end, and the 18,000-trajectory corpus and all Core-50 and Transfer-16 results come from it. The released code supports substituting other arms, hands and physics back-ends, but those configurations contribute no episodes, and the reported numbers should not be read as performance on them. Real-robot evidence is limited to qualitative rollout videos, with no physical success rate.

The 5.2% and 1.9% for GPT-6 were measured under a minimal harness that exposes only move_eef, with no grasp primitive, obstacle-aware planner, explicit object-pose estimate or tactile feedback; trials ended by the call budget do not establish what an unlimited-call controller would achieve. The authors note that isolating planner capability from the benefit of richer tools requires matched evaluations that vary the interface and feedback while controlling task population and budgets. On Transfer-16, trained-checkpoint success is close to zero, so aggregate success discriminates weakly and ordered progress is used for ranking.