Skip to main content
Back to timeline
arXivSource publication:

Driving-scene 4D reconstruction drops dataset annotations: FreeForward4D uses only RGB video and beats STORM on Waymo full-image PSNR

Synopsis

FreeForward4D trains and runs on RGB driving video alone, without dataset cameras, LiDAR depth or 3D box annotations. It calibrates pretrained camera, depth and object-track predictions into one metric frame, then associates objects across cameras and models their motion on that geometry. On Waymo it reaches 27.66 dB full-image PSNR, above annotation-reliant STORM, SLARM-F, UFO, DGGT and StreetForward, while its 7.06 m depth error is higher than some annotated methods.

Source-provided article image: Freeforward4d: Annotation-free feed-forward 4d reconstruction of driving scenes
arXiv

Interpretation

Feed-forward 4D reconstruction of driving scenes can now be done from RGB video alone, with no dataset cameras, LiDAR depth or 3D box annotations in training or inference. Earlier pose-free methods removed camera poses at inference but still relied on dataset annotations during training, and Flux4D avoided 3D boxes yet still required dataset cameras and LiDAR as input. On Waymo, FreeForward4D reaches 27.66 dB full-image PSNR, 1.81, 0.68 and 0.40 dB above the dataset-camera-dependent STORM, SLARM-F and UFO, and 0.25 and 0.65 dB above DGGT and StreetForward, which are trained with annotations.

The route is to reconcile several pretrained visual priors into a single metric frame, then perform object association and motion modeling on that geometry. Multi-view models give relative depth only within short windows, each with its own reference frame, while monocular models give metric depth per image, so scale, coordinate frame and image grid disagree. Overlapping windows are calibrated against monocular metric depth through a jointly fitted per-window scale and per-image offset, then registered into one scene frame by rigid transforms estimated from shared cameras, with predicted intrinsics converted to the reconstruction grid.

Object motion combines a cross-camera shared velocity with local higher-order Gaussian corrections, and rendering weights each source time by its observation. 2D object tracks were produced independently per camera, lacking cross-camera correspondence and 3D motion, while each timestamp sees only part of a moving object's surface. On Waymo 2-second clips, removing object motion lowers dynamic-region PSNR by 1.59 dB, of which the shared velocity accounts for 1.35 dB and the local motion head 0.24 dB; removing per-object normalization drops dynamic-region PSNR from 24.36 to 24.04 dB.

Perspective

The demonstrated scope is reconstruction and rendering on Waymo validation with three front cameras at 10 Hz over 2-second and 8-second windows, plus RGB-only fine-tuning to nuScenes and Argoverse2; depth accuracy and dynamics are bounded by the frozen priors.

The 7.06 m depth error jointly reflects scale, camera and depth errors, and the metric scale comes from the MoGe-2 prior whose training data include Waymo LiDAR; cross-camera association reaches pairwise precision and recall of 0.867 and 0.647 on label-matched tracks.