Skip to main content
Back to timeline
arXivSource publication:

PanDWM stitches six surround cameras into one 360-degree band, generating four-step video 8 to 46 times faster than multi-camera baselines

Synopsis

PanDWM replaces six separately generated surround-view videos in a driving world model with a single 360-degree panoramic band, so camera boundaries fall inside one image and cross-view consistency becomes spatial continuity within that image. A block-causal diffusion model generates the band window by window, a four-step student is trained on its own generated history, and a feed-forward decoder lifts each frame along calibrated rays into 3D Gaussians that neighbouring frames complete into 4D scenes. Scored in one stitched 360-degree view on nuScenes, it has lower FID and FVD than multi-camera baselines at 8 to 46 times their speed, and keeps its 50-step teacher's FID while lowering FVD by 22 to 25 percent.

AI-generated editorial illustration: PanDWM: Panoramic 4D Driving World Models

Interpretation

Surround-view video is no longer generated camera by camera; it is stitched into one 360-degree panoramic band, so every camera boundary lies inside a single image. Earlier layout-conditioned generators represented the surround view as six perspective videos, one per camera, with distinct optical centres and marginal overlap, so content crossing a boundary was split between two separately generated images and consistency had to be enforced by cross-view attention, conditioning or a shared canvas. Scored in the same stitched 360-degree view on nuScenes, PanDWM has lower FID and FVD than UniMLVG, OpenDWM, DreamForge, DiST-4D and OmniNWM, at 1.96 seconds per frame, 8 to 46 times faster than those baselines.

Generation proceeds causally window by window, each window denoised in four steps by a student model that was trained with its own generated frames as history. Long videos are usually produced clip by clip at tens of denoiser evaluations per clip, and a model trained on recorded history meets its own imperfect outputs as history at inference. The student is initialized from the 50-step teacher by causal consistency distillation, then trained with distribution matching distillation and adversarial losses on self-generated history; on Val-150 its FID is 10.1 against the teacher's 9.93, while denoiser evaluations per window drop from 100 to 4.

Each frame's panoramic latent is lifted in one pass along calibrated source-camera rays into pixel-aligned 3D Gaussians, and static Gaussians from neighbouring frames, aligned by ego motion, complete them into 4D scenes. Earlier methods that lift generated video into 3D or 4D scenes reconstructed from per-camera videos and inherited their seams. On Test-S recorded clips, the 4D Gaussians have the best PSNR and SSIM at two resolutions and the lowest AbsRel among camera-only methods, exceeding ReconDrive by 1.1 dB in PSNR at one of those resolutions with AbsRel 0.298 against 0.663.

Perspective

The demonstrated scope is the 150 nuScenes validation scenes and their long-rollout subsets, scored in both the stitched 360-degree view and the six real camera views, with the Gaussian part evaluated on Test-S recorded and generated clips. The intended reader works on driving world models and scene reconstruction. The authors propose renders of accumulated static Gaussians as longer memory and leave 4D completion of moving objects to future work.

Each window sees only the last three generated frames, and Test-L FVD grows from 86.6 over windows 3 to 5 to 126.6 over windows 9 to 11. 4D completion covers only static content, leaving unseen parts of moving objects empty. In setting G, where only conditions are given and no recorded history, PanDWM's FVD-1 is higher than UniMLVG's and MagicDrive-V2's, and the authors describe it as primarily a predictive model. Against UniMLVG in the six camera views, the FID difference is not significant and PanDWM's FVD-1 is higher.