Skip to main content
Back to timeline
arXivSource publication:

No demonstrations for a new robot cell: VAA self-calibrates and reaches 64% on 18 changed cells

Synopsis

Public vision-language-action models (VLAs, which turn images and language instructions directly into robot actions) fine-tuned on LIBERO handle their training setup, but lose heavily once the gripper changes or the base frame rotates. VAA pairs pre-trained vision-language and language models with self-calibration: the robot measures its command map, camera geometry and lens distortion from its own motion and fingertips, then plans grasps from measured object depth, with no robot demonstrations and no fine-tuning. Across 18 simulated cells that change the gripper or rotate the base frame, frozen VAA succeeds in 230/360 episodes (64%) and the refined VAA in 268/360 (74%), against 76/360 (21%) for the strongest released VLA on that subset.

Source-provided article image: Self-Calibrating Vision-Agent-Action: Deploying Pick-and-Place to New Robot Cells without Demonstrations or Fine-Tuning
arXiv

Interpretation

When a manipulation system moves to a new robot cell, gripper and base-frame changes, not image appearance, are what cost released VLAs their success. Released VLAs can perform tasks in their LIBERO training setup, but transfer to a new cell also requires their commands to produce the intended physical motion under changed hardware and coordinate frames. In a controlled study of 30 changed design cells, cells that change the gripper or rotate the base frame were isolated, and the remaining evaluation focuses on those 18 cells.

VAA measures the new cell before the task, turning the relation among command channels, fingertips and camera into a usable cell model. These measurements replace assumptions inherited from the training setup, including the map from commands to tool-center-point displacement and the camera intrinsics and pose. The robot commands small motions through its action channels and measures the resulting tool-center-point displacement, observes its fingertips at multiple poses to estimate focal length, principal point and camera pose relative to the base, and estimates radial distortion using table flatness as a constraint.

Grasp placement comes from the target object's own three-dimensional points rather than from monocular image inference. A pre-trained vision-language model (Qwen3.8, 27B) labels object proposals and a pre-trained text language model (Qwen3, 30B instruct) parses the instruction to pick target and destination, both running locally without fine-tuning. In paired ablations over three cells, substituting monocular depth (Depth Anything V2 Large) for measured depth drops success from 50/60 to 20/60, and the gripper cell from 15/20 to 8/20.

Perspective

All experiments are in simulation, with tasks from LIBERO-spatial and 20 episodes per cell; the conclusions cover the 18 design cells that change the gripper or rotate the base frame, plus five development cells that change hardware, geometry and appearance together.

The combined changes follow a resolution-IV fractional design, so main effects are not aliased with two-factor interactions, but that interpretation assumes negligible higher-order interactions and two-factor interactions remain aliased with one another; the arc-closing Robotiq 2F-85 gripper falls outside the parallel-jaw model, and VAA scores 0/20 in that cell.