TrajWAM learns from a second, unlabeled future trajectory: 91.53 Driving Score on all 220 Bench2Drive routes, 5.31 points above schedule-matched planning-only fine-tuning
Synopsis
TrajWAM feeds a four-waypoint ego trajectory as a query into a bird's-eye-view future-prediction branch attached to the shared representations of a vision-language driving planner, training with factual reconstruction under the executed trajectory and one-sided feature separation under a perturbed one, then generating waypoints directly at deployment while bypassing that branch. On all 220 Bench2Drive routes it reaches 91.53 Driving Score and 75.45% Success Rate, 5.31 points and nine successful routes above a schedule-matched planning-only fine-tuning baseline.
Interpretation
A vision-language driving planner gains a queryable future-trajectory channel during training: the encoded ego trajectory modulates bird's-eye-view future predictions read from shared planner representations, so future supervision and waypoint generation share one representation. Earlier future supervision was either trajectory-agnostic or had an observed outcome only for the executed trajectory, while alternative trajectory queries in driving logs carry no future labels. On all 220 Bench2Drive routes the complete method reaches 91.53 Driving Score and 75.45% Success Rate, 5.31 points and nine successful routes above schedule-matched planning-only fine-tuning; on a 120-route two-seed control, correct trajectory pairing averages 4.25 Driving Score above within-group shuffled pairing.
Training pairs the executed trajectory with a perturbed one and predicts future features under both, making the prediction respond to the trajectory query. The perturbed trajectory scales forward distances independently and adds Gaussian noise to lateral coordinates, and its future outcome is unobserved, so it can only serve as a one-sided separation reference. Factual reconstruction pulls the executed-query prediction toward frozen DINOv3 teacher features at the current anchor and four future times; the separation term uses the executed-query prediction as a stop-gradient reference and pushes the perturbed-query prediction away, with both losses backpropagating into the last four planner layers and the trajectory encoding and modulation modules.
At deployment the planner still generates reasoning text and four waypoints autoregressively on its native path, with the future-prediction branch bypassed. The auxiliary branch exists only during training, while the fixed future-query tokens stay in the inference prompt. On one RTX 5090 at batch size one, median generation takes 3.5086 s for TrajWAM and 3.4926 s for public DeepSight, with both prompts retaining 1,305 future-query tokens.
Perspective
Evaluation runs in CARLA/Bench2Drive simulation, with one training seed for the full 220 routes and two training seeds for component and shuffled-pairing controls on a 120-route subset; prediction diagnostics use 374 windows from 15 newly collected episodes, of which 371 have speed- and direction-matched alternative queries.
Separation raises Driving Score while future-feature MSE rises from 0.02985 to 0.03020, so the planning gain does not come from more accurate reconstruction; outcomes of alternative trajectories are unobserved, and separation also acts on the current-time anchor; the pretraining route coverage of the public initializer is unknown.
