LCVN turns visual navigation into open-loop trajectory prediction: one first frame and one instruction, a whole route
Synopsis
LCVN recasts language-conditioned visual navigation as open-loop trajectory prediction: an embodied agent receives only an initial egocentric observation and a natural-language instruction, with no goal image and no intermediate environmental feedback, and must generate the future navigation trajectory. The accompanying LCVN dataset holds 39,016 trajectories and 117,048 human-verified instructions in concise, intricate and landmark-grounded styles. Two complementary model families are released: a diffusion-based world model, LCVN-WM, paired with a latent-space actor-critic agent, LCVN-AC, trained on intrinsic rewards, and LCVN-Uni, an autoregressive multimodal architecture that predicts actions and future observations in a single forward pass.
Interpretation
Visual navigation becomes a one-shot open-loop trajectory generation task: starting from an initial egocentric frame and a natural-language instruction, the agent outputs the future navigation trajectory without receiving environmental feedback along the way and without a goal image. Prior language-conditioned navigation typically relies on goal images or intermediate environmental feedback to correct actions step by step, whereas here both the goal image and the intermediate feedback are removed from the input, leaving only the first observation and the instruction. The setting is supported by the LCVN dataset, with 39,016 trajectories and 117,048 human-verified instructions spanning diverse environments and instruction styles, split into training, seen-environment validation and unseen-environment validation.
Two complementary implementation routes are offered: one separates world modeling from policy learning, the other puts actions and future observations into a single forward prediction. LCVN-WM is a diffusion-based world model built on an LDiT (language-conditioned Diffusion Transformer) backbone that imagines future visual states conditioned on actions and language instructions, while LCVN-AC is a latent-space actor-critic agent whose navigation policy is learned from intrinsic rollout rewards generated by LCVN-WM. The other route, LCVN-Uni, uses an autoregressive multimodal backbone to jointly predict actions and future observations in a single forward pass, and runs in a separate environment from LCVN-WM plus LCVN-AC.
Perspective
The public scope is the LCVN dataset and the official implementations of two model families: the dataset holds 39,016 trajectories and 117,048 human-verified instructions across training, seen-environment validation and unseen-environment validation, with TartanDrive reserved for unseen-environment evaluation; the current public release provides no separate test split, so evaluation runs on the validation splits.
The material reports no quantitative results for either family on the seen or unseen validation splits, so navigation performance under the open-loop setting has no numeric support yet; the public release also has no separate test split, so evaluation can only run on the validation splits.
