AffordSim puts open-vocabulary 3D affordance prediction first in robot data generation, collecting 64% of hard composite tasks without per-object labels
Synopsis
AffordSim places open-vocabulary 3D affordance prediction inside a simulation trajectory generator: a vision-language model turns a natural-language task into affordance queries, the predicted functional region is grounded on the object point cloud, and grasps are sampled only inside that region and screened by motion planning. On a benchmark of 50 tasks, 500-plus rigid and articulated objects and five robot arms, it collects trajectories without per-object annotation, reaching 64% average collection success on hard composite tasks against 86% for a human-annotation reference.
Interpretation
Simulation data generators now compute where a grasp must land for the task as a separate first step, instead of picking a stable grasp and filtering afterward. The two prior routes each carried a cost: per-object manual contact annotation succeeds on the annotated set but must be redone for every new object and task, while generic grasp estimators rank candidates by grasp stability without task semantics and often land on regions misaligned with the downstream action. On the 50-task benchmark, the generic estimator AnyGrasp averaged trajectory collection success of 67%, 15% and 3% across the easy, medium and hard tiers, rising to 87%, 45% and 16% with a 2D semantic-mask filter, while AffordSim reached 98%, 79% and 64% and the human-annotation reference 100%, 93% and 86%.
The whole pipeline starts from one natural-language task description and produces executable grasps together with pre-grasp trajectories. Two separate vision-language model calls parse the scene description and the task description, converting object-action entries into one affordance query per functional contact region, such as a graspable handle. An open-vocabulary 3D affordance model outputs a per-point mask on the object point cloud; the pipeline runs mean-shift clustering on the masked points and takes the cluster centroid as the contact point, estimates the local surface normal, samples K approach directions on the opposite hemisphere and solves wrist yaw for each, yielding K candidate grasps; cuRobo then plans a collision-free pre-grasp trajectory for each candidate, scores by inverse joint-space arc length and keeps the best. Across five robot arms the pipeline reaches 76% to 83% collection success, 32 to 33 percentage points above the mask-filtered generic estimator.
Perspective
The demonstrated scope is rigid-body tabletop manipulation: 50 tasks, 500-plus rigid and articulated USD objects, five robot arms, 100 simulation rollouts per task, and a real-robot portion of four tasks with 10 trials each on a Franka FR3 without real-world fine tuning. The authors list dynamic and contact-force affordances, deformable objects and bimanual manipulation as future work.
Trajectory quality degrades when the affordance predictor struggles, including on novel geometries, transparent objects and ambiguous functional regions; on every hang task AffordSim stays below 30%, which the authors attribute to configurations such as grasping the cup body while keeping the handle free being rare in the affordance backbone's training data. Real-scene background reconstruction needs 10 to 20 photographs per scene, and image quality drops sharply at viewpoints far from the captured poses.
