Distilling belief-space planners into policies: over 90% success in Scavenger, where RL baselines top out at 5%
Synopsis
In partially observed tasks where an agent must gather information before acting, training the agent directly rarely yields reliable compliance with belief-dependent rules. The approach plans offline with a belief-space task and motion planner in a simulator, then distills those plans into a policy that acts on raw observations, adding a dual-averaging loss and near-optimal reinforcement. In the new Scavenger domain, success exceeds 90% for both the visual-hint and language-hint versions, while reward-engineered RL baselines reach at most 5%.
Interpretation
An agent can follow belief-dependent rules without an explicit belief estimator: the policy reads raw observations and exceeds 90% success in both hint versions of Scavenger, where it must find a hint before guessing. Earlier routes either encoded the rules in rewards for reinforcement learning, which struggles to optimize rewards that fully capture a specification, or ran belief-space planners online, which requires hand-programming online planning and belief-state estimation at significant engineering cost. On the same task, flat and hierarchical PPO reinforcement-learning baselines stay at or below 5% success in both the visual-hint and language-hint versions, despite receiving more wall-clock training time and dense rewards.
Planning and execution are separated: offline, a belief-space task and motion planner with simulator access generates sample trajectories, which are then distilled into a policy that consumes observation-action history. Earlier planner-imitation work targeted fully observed domains, where history dependence does not worsen distribution shift, and its feedback mechanism handled only illegal actions rather than replanning from every state the policy reaches. Two changes target history-based policies: a dual-averaging loss combines the average loss over all timesteps with the average loss at task changes, and near-optimal reinforcement adds a successful policy rollout to the training data when its length is within a given factor of the planner's trajectory.
The distilled policy transfers to commands and floorplans unseen in training: the language-hint version exceeds 60% success on held-out and intentionally vague commands and 75% on floorplans with more rooms than training. Training floorplans had three to six rooms, and language commands were generated the same way and rephrased. The generalization study runs the language-hint policy for 50 episodes per condition and scales floorplans to 8, 12 and 18 rooms, the last being three times the maximum number of rooms seen in training.
Perspective
The demonstrated scope is the Scavenger simulated domain of procedurally generated multiroom floorplans, with observations as two 128×128 RGB images and position control of at most 0.3 units per step; the method assumes a tractable belief-space planning model of the problem and access to a simulator. The authors propose moving past that assumption, and formalizing safety constraints for non-robotic settings such as terminal or computer use, as future work.
The completeness result covers only static search with noiseless observation models and relies on an efficiency assumption: the planner can replan from any state and belief, and each replan covers a set of candidates with at least a given probability. The authors note that relaxing these assumptions is future work. The policy itself has no formal certificate; combining imitation-learning theory with the planner's soundness and completeness guarantees would be needed to formalize a safety case for the distilled policy.
