Skip to main content
Back to timeline
arXivSource publication:

Navigation agents stop calling the LLM every step: H2E distills whole sessions into an 8B VLA, lifting success from 35.0% to 44.9%

Synopsis

Navigation systems usually have a large language model planner call a vision-language-action (VLA) model at every step, paying LLM cost and delay in each episode. H2E converts successful agent sessions into VLA training data: every motion in a session becomes a VLA action under the original task instruction, steps moving away from the goal are masked from the loss, visited states within 1 m of the goal are labeled as stops, and the VLA is updated with low-rank adaptation. On all REVERIE-CE val-unseen episodes, the 8B VLA's success rate rises from 35.0% to 44.9%, while adaptation to geodesic expert labels gives no gain.

Source-provided article image: What Can a VLA Learn from the Agent That Calls It? Task-Level Experience Distillation for Embodied Navigation
arXiv

Interpretation

At test time the navigation agent no longer needs to call a large language model planner, and a single fine-tuned 8B VLA carries out the task alone. The prior setup has a frontier LLM planner call the VLA throughout every episode and correct it when it goes wrong; distillation from such agents has so far targeted the planner, with the VLA kept frozen. On all REVERIE-CE val-unseen episodes, H2E raises the 8B VLA's success rate from 35.0% to 44.9%, a gain of 9.9 points with a 95% interval from 3.9 to 16.3 points across scenes; geodesic adaptation with the same stop rule and LoRA settings gives no gain.

Motions driven by planner tools and sub-instructions, and detours the planner later fixes, are all learned as VLA actions under the original task instruction. Sessions cannot be used directly as VLA demonstrations: planner tools and sub-instructions together produce 45% of the motion in the sessions, and nearly a quarter of the steps move away from the goal. Training data comes from 2,395 sessions over 57 training scenes and 332,405 labeled steps, of which only 53.9% are VLA steps on the original instruction; in a comparison using only 504 sessions, H2E reaches 43.2% success while geodesic adaptation stays at 34.9%.

The distilled VLA learns both to stop at the goal and to find goals the base model never reaches. The base VLA turns only 67% of its goal-region visits into success, and in 544 episodes it reaches the region but ends outside it. H2E raises that share to 89% and cuts these misses to 168; it also reaches the goal region in 401 episodes the base VLA never reaches, 355 of which end in success.

Perspective

The results rest on the REVERIE-CE benchmark, the 8B NavGPT video VLA and a single agent whose planner is gpt-6-astra, evaluated on the 10 val-unseen scenes and on 100 development episodes from 26 further scenes. The distilled VLA sees only four RGB views and runs alone at test time; its LoRA weights can be merged back into the base model, so its size and per-step cost match the base VLA.

The distilled VLA does not yet learn to recover after it goes off track, which needs the agent's map and memory; collecting sessions that start from the VLA's own failures is named as a natural next step. Experiments cover one benchmark, one VLA and one LLM agent, so extending the recipe to other agents that call a VLA remains an expectation rather than a demonstrated result.