VLMs no longer collapse scene state into one guess: PRaVaSS keeps a minimal high-probability state set and lifts planning success from 16.7% to 32.6% on PRB
Synopsis
When a robot estimates scene state with a vision-language model, the pipeline usually collapses probabilistic output into a single most likely symbolic state, and a wrong guess makes downstream planning unsafe or infeasible. PRaVaSS instead starts from the model's existing per-feature probabilities and combines them with task constraints to build a minimal set of states whose cumulative probability meets a user-set threshold, so the planner considers several likely states at once. On the PRB and GW planning domains, with GPT-4.1 as the backbone, this pipeline raised downstream planning success from 16.7% and 70.7% for MAP to 32.6% and 80.0%.
Interpretation
State estimation no longer collapses a vision-language model's output into one most likely symbolic state; it hands the planner a minimal set of states whose cumulative probability meets a user-set threshold. Prior robot pipelines almost universally accept the vision-language model's textual yes or no and plan as if it were ground truth; when that guess is wrong, the agent may attempt unsafe actions, violate user-defined guardrails, or fail outright. On the PRB and GW planning domains, with GPT-4.1 as the state-estimation backbone, the pipeline reached downstream planning success of 32.6% and 80.0%, against 16.7% and 70.7% for MAP, which commits to the single most likely state.
The method turns the per-feature probabilities the perception model already produces into a compact probability representation over valid states, after an optional prior update and filtering by task constraints. Maintaining a full distribution over symbolic states and planning in belief space would be safer, but is generally intractable in combinatorial task-planning spaces. PRaVaSS exploits this three-layered belief structure to extract valid states lazily in probability order without enumerating the exponentially large state space, has polynomial complexity in the unconstrained case, and was validated on three domains with a vision-language model as backbone.
Perspective
Evaluation covers three benchmark or simulated task-planning domains, PRB, GW and ALF, with GPT-4.1 as the state-estimation backbone and partial repeats on five open-weight vision-language models; the results concern belief-set construction from a single observation, not deployed robotic systems.
The belief converter assumes conditional independence between features, an assumption likely violated in practice; the authors ran a synthetic sensitivity study and state that it is not an estimate of the natural error correlation of any vision-language model. ALF was excluded from the downstream planning experiment because robust decision-making there requires the agent to gather information.
