Vision-language models rewrite robot cost programs: 97.5% success on four tasks after five feedback rounds in simulation
Contact-rich robot manipulation has relied either on end-to-end learned policies or on hand-engineered rewards and constraints for known scenes. Objectives as Policies has a pretrained vision-language model write an editable multi-stage cost program, which one shared sampling-based model-predictive controller executes in a digital twin and revises from execution statistics. Across four tabletop tasks, simulation cumulative success reached 90% with at most two feedback rounds and 97.5% with up to five; programs that succeeded in simulation then reached 80% success in the real world with perturbed object locations.