Skip to main content
Back to timeline
arXivSource publication:

UAVs that follow spoken instructions no longer just predict actions: the full system reaches 76.6% safe-flight success across 521 closed-loop episodes

Synopsis

A world-action policy for UAV vision-language navigation puts language response, task progress and obstacle avoidance into one control loop. Earlier world-action models focused mainly on action prediction and struggled to track multi-stage instructions or decide when to stop. The full system reaches 83.7% goal-reach success, 79.8% ordered completion and 76.6% safe-flight success on a 521-episode closed-loop test suite, and completes six indoor and outdoor flights.

Source-provided article image: Fly as Instructed: Refining World-Action Policies for UAV Vision-Language Navigation
arXiv

Interpretation

The UAV vision-language navigation policy no longer only predicts actions; it also tracks how far an instruction has been executed and issues its own stop command once every requirement is complete. Earlier world-action models focused mainly on action prediction and lacked continuous tracking of multi-stage instruction completion and termination decisions. On the unified 521-episode closed-loop test suite, the full system reaches 83.7% goal-reach success, 77.0% correct stopping, 79.8% ordered completion and 76.6% safe-flight success, while the strongest external reference, WorldVLN, scores 45.3%, 31.5%, 34.7% and 30.3% respectively.

The action expert reads the instruction through text cross-attention and receives targeted post-training while the pretrained conditioning modules stay frozen, so motion changes with instruction meaning. Action policies that rely on visual cues can carry over old motions when the instruction changes, so language has to enter action generation directly. Instruction success is 80.6% over 500 individual instructions, 38.4 percentage points above WorldVLN, the strongest baseline on that metric; on 500 semantic contrast pairs it is 56.4%, 35.6 points above Pi0-UAV.

A recurrent instruction-consumption head predicts which clauses are complete from visual and execution history and continuously lowers attention weights on completed tokens inside the frozen executor. Keeping the full instruction in context leaves the policy unable to tell which requirements are already satisfied, giving 24.1% ordered completion and no self-issued stops. On 170 progress episodes, soft masking reaches 81.2% ordered completion and 77.1% stop success, versus 70.0% and 61.2% for hard switching and 57.6% and 49.4% after removing GRU memory.

Obstacle avoidance is acquired by post-training low-rank adapters in reconstructed Gaussian scenes, with training signals from imitating successful detours and from differentiable geometric feedback. The base policy reaches only 30.6% safe-flight success with a 64.1% collision rate on the safety subset. With the safety adapter, the 170 safety episodes reach 78.2% safe-flight success and a 17.1% collision rate, while reach on the 181 general tasks moves from 76.8% to 76.2%.

Perspective

The evaluation covers 521 closed-loop simulated episodes split into 181 general-task, 170 safety and 170 progress episodes, with test instances or layouts differing from training configurations; six indoor and outdoor flights each record one task, running on an RTX 5070 with 10 Hz trajectory updates and 50 Hz control.

High-speed flight leaves less time to react to new observations, motivating further cuts in end-to-end latency; longer missions may also need to recall earlier landmarks or decisions, and extending the policy with longer-term memory is named as a direction; environments whose layouts or visual conditions differ substantially from training still leave room for improvement.