Candidate trajectory scoring drops per-candidate rollouts: RFWorld reads a shared scene memory and reaches 90.5 PDMS on NAVSIM
Synopsis
End-to-end driving must pick one of several candidate trajectories, and earlier world-model evaluators explicitly predicted future states for each candidate over the planning horizon, so inference cost grew with candidates and horizon while prediction errors accumulated. RFWorld instead builds one trajectory-agnostic shared scene memory, shapes it during training with a lightweight temporal readout under semantic supervision, and lets candidates query it through sparse spatial readout at inference. On NAVSIM-v1 with a ResNet-34 backbone, RFWorld reaches 90.5 PDMS, above DriveDPO at 90.0 and ResWorld at 89.0 in the reported table.
Interpretation
Candidate trajectory evaluation no longer needs to reconstruct a full future scene for each candidate; all candidates share one scene memory and read the information needed for scoring from it. Earlier model-based evaluators made scoring future-aware by explicitly predicting candidate-conditioned future states over the planning horizon, termed rollout, so inference cost grew with the planning horizon and the number of candidates, and recursive prediction accumulated errors. On NAVSIM-v1, RFWorld reaches 90.5 PDMS under the ResNet-34 setting, where the reported table gives DriveDPO 90.0, ResWorld 89.0 and WoTE 88.3; in the ablation, adding the world model without future supervision gives 88.4 and future supervision alone gives 89.4.
The shared memory is produced once from the current bird's-eye-view features by a Transformer latent world model, and a temporal readout then injects predictive temporal structure during training. Training performs temporal readout at selected future timestamps with semantic supervision, decoding bird's-eye-view semantic maps through a shared decoder, with the current and selected future timestamps supervised by their corresponding semantic targets. As supervised future timestamps rise from 0 to 8, PDMS moves from 88.4 to 89.5, 90.0, 90.3 and 90.5, indicating that future supervision is what gives the shared memory its predictive structure.
At inference, each waypoint of each candidate trajectory acts as a spatial query that sparsely samples the shared memory through deformable attention and is aggregated into a trajectory score. Inference decodes no future semantic maps and performs no per-candidate rollout; candidate waypoints query the same memory directly, so the memory is computed once and reused across candidates. As candidates grow from 256 to 1024, RFWorld latency rises from 35.1 ms to 37.5 ms and PDMS from 89.6 to 90.5; under the same setting WoTE latency rises from 37.0 ms to 49.2 ms, and training memory is 33.9 GB for WoTE versus 8.7 GB for RFWorld.
Perspective
The demonstrated scope is open-loop evaluation on NAVSIM-v1 and closed-loop simulation on Bench2Drive in CARLA, with LiDAR features, ego state and multi-view images as input and scoring and selection over 1024 anchor candidates as output; the authors propose replacing dense bird's-eye-view features with sparse scene tokens to improve scalability for higher-resolution future prediction.
Shared-memory quality still depends on the form of memory-shaping supervision, and semantic targets are currently needed during training, with pseudo-labeling and self-supervised temporal objectives listed as future work; observed failures mainly occur when the lightweight perception backbone produces inaccurate scene features or when relevant road geometry and traffic participants lie near or beyond the observation range, and these errors propagate to trajectory scoring.
