LAGR splits grasp choice into physical feasibility and task fit, lifting real-arm handover success from 58% to 92%
Synopsis
Task-oriented grasping must satisfy two things at once: a stable hold and a hold that fits the task, such as gripping a hammer by the handle to pound but by the head to hand over. LAGR separates them: a task-independent generator proposes physically feasible grasps with scores, and a language-conditioned re-ranker compares all proposals jointly to pick the one suited to the instruction. On its 2.68 million scored grasp annotations over 100,911 image-task pairs, LAGR reaches 85.9% task-conditioned success versus 76.6% for the strongest baseline GATER, 9.3 points higher; on a Franka Research 3 arm it succeeded in 93 of 100 real grasps against 72 for GATER.
Interpretation
Grasp selection is split into two steps: first decide whether a grasp is geometrically executable, then decide whether it suits the current task instruction, with a separate score for each. Earlier task-oriented grasping methods either optimized physical grasp quality without task intent, or used semantic or affordance reasoning to pick regions without separately evaluating task-independent physical feasibility and task-dependent suitability. On the constructed benchmark, LAGR reaches 85.9% task-conditioned success versus 76.6% for the strongest baseline GATER, a 9.3-point gap, and raises grasp IoU from 0.51 to 0.58; selecting grasps by physical confidence alone drops success to 53.03%.
The re-ranker does not score each grasp independently; it compares all candidate grasps from the same observation together, then combines that with physical confidence into a final ranking. Candidates come from a multimodal encoder over RGB, depth and semantic segmentation, each carrying a physical feasibility score; the instruction is encoded by a frozen MiniLM, and grasp geometry and instruction features are concatenated, multiplied and subtracted before entering a proposal-context transformer. Removing the proposal-context transformer lowers success to 77.76% and removing explicit grasp geometry to 74.00%; raising the proposal count from 1 to 64 lifts success from 53.03% to 82.36%, largely saturating beyond 64.
Perspective
Validation covers the constructed benchmark of 100,911 image-task pairs and a simplified tabletop setting with a Franka Research 3 arm and a two-finger parallel-jaw gripper: five objects, ten object-task pairs, ten trials per pair.
LAGR's performance depends on the quality and diversity of the generated grasp proposals, and the current formulation covers only two-finger parallel-jaw grasping; in real experiments the jug reached 7/10 for pouring and 6/10 for handover, below most other object-task pairs.
