Skip to main content
Back to timeline
arXivSource publication:

Takeover logs aren't copied verbatim: shifting human corrections lifts closed-loop success from 70.87% to 81.60%

Synopsis

Real fleet takeover logs are imperfect: drivers may intervene too late or unnecessarily, so direct imitation can teach a model delayed reactions or needless corrections. An offline post-training method shifts recorded human corrections earlier by several time offsets and perturbs the deployed model's multimodal plans to build trajectory candidates, then ranks them with a scene-conditioned scorer trained on expert trajectory matching and unpaired preference signals from takeover logs. On a closed-loop benchmark of 596 real challenging scenarios reconstructed with 4D Gaussian splatting, the method reaches 81.60% overall success, versus 70.87% for the deployed model and 75.30% for direct imitation learning.

Source-provided article image: Improving While Deployed: Learning Proactive End-to-End Driving from Imperfect Human Takeover Data
arXiv

Interpretation

A deployed end-to-end driving model can be updated offline from fleet takeover logs alone, without treating the driver's maneuver as the correct answer. Direct imitation of takeover trajectories also learns late corrections or unnecessary interventions, while trajectory post-optimization depends on predefined costs and constraints. On a closed-loop benchmark reconstructed from real takeover logs with 4D Gaussian splatting, the method reaches 81.60% overall success, against 70.87% for the deployed model and 75.30% for direct imitation learning.

The recipe first moves human corrections earlier in time, then lets a scorer pick earlier, more proactive responses from a candidate set. Candidates combine the deployed model's multimodal plans and their random perturbations, the recorded trajectory, and human corrections advanced by 0.5 to 2.5 seconds in 0.5-second steps. The scorer is conditioned on bird's-eye-view features, matches expert trajectories with cross-entropy, and uses unpaired preference signals from takeover logs to lower the scores of intervention-prone trajectories.

Only candidates scoring clearly higher replace training targets inside the 6-second window before a takeover; other scenes keep their original targets. Replacement requires the candidate to outscore the recorded trajectory by more than 0.1, and only the planning head and selection head are finetuned, leaving the inference architecture unchanged. In ablation, a variant without the time-window and score-margin constraints replaces 77.87% of targets and reaches 79.36% success, below the 81.39% of selective replacement at 34.39%.

Perspective

Evaluation is limited to a closed-loop simulated benchmark reconstructed from real takeover logs with 4D Gaussian splatting, covering 596 challenging scenarios across cut-in, meeting, bypass, junction and unprotected left turn; all methods are compared on the same 587 valid cases.

The authors list extending post-training to the entire end-to-end model and adding scenario generation for extremely rare corner cases as future work, indicating that the current post-training finetunes only the planning and selection heads and does not cover those cases.