Vision-language models rewrite robot cost programs: 97.5% success on four tasks after five feedback rounds in simulation
Synopsis
Contact-rich robot manipulation has relied either on end-to-end learned policies or on hand-engineered rewards and constraints for known scenes. Objectives as Policies has a pretrained vision-language model write an editable multi-stage cost program, which one shared sampling-based model-predictive controller executes in a digital twin and revises from execution statistics. Across four tabletop tasks, simulation cumulative success reached 90% with at most two feedback rounds and 97.5% with up to five; programs that succeeded in simulation then reached 80% success in the real world with perturbed object locations.
Interpretation
Contact-rich robot manipulation can now be specified by an explicit multi-stage cost program instead of a task-specific trained policy. Earlier physics-based planning required known tasks and scenes with hand-engineered rewards and constraints, while end-to-end learned policies predicted actions from visual observations without an explicit physics model for online planning. In simulation across four tabletop tasks, Objectives as Policies reached 36/40 (90%) with two feedback rounds, above ReKep at 17/40, Language to Rewards at 11/40 and an adapted CoRAL at 31/40, with OpenVLA-OFT at 0/40.
The vision-language model writes only the objectives, and a shared sampling-based model-predictive controller handles control. A cost program splits a task into stages, each combining running costs, terminal costs and completion thresholds from a cost library, and the controller optimizes control sequences against those costs. The controller uses MPPI and rolls out sampled trajectories in parallel in MuJoCo-warp, so it needs no gradients through contact dynamics; experiments used a seven-degree-of-freedom Flexiv Rizon 4s arm, a GN01 gripper and a ZED 2i camera, with at most 250 controller updates and 20 s of commanded motion per execution.
Before deployment the program is tried in a digital twin, and failed runs go back to the vision-language model for revision. The model first self-reviews its draft, checking whether later stages can start from the preceding stage's end state and whether objectives conflict or grasp requirements are infeasible. Self-review raised first-execution success from 20/40 to 26/40 with zero feedback rounds, and the gap narrowed as feedback rounds increased; with up to five rounds, success was 95% without self-review and 97.5% with it.
Perspective
The demonstrated scope is four tabletop tasks: placing a blackboard eraser on a box, tilting a cup by at least a set angle, pushing a white box at least 60 mm directly, and pushing the white box at least 60 mm with the eraser; simulation repeats each task 10 times, and the real world runs five simulation-successful programs per task once each under two object-location perturbations.
Real-world success is limited by digital-twin accuracy: the vision-language model's estimated friction coefficients can differ substantially from the real objects, and a 3D-printed cup that grasps stably in simulation slips on the real robot; the simulator treats objects as rigid while real objects deform under contact. The program is a linear, forward-only stage sequence, so after a stage completes the controller cannot return to an earlier one, and explicit stage recovery is left as future work.
