Skip to main content
Back to timeline
arXivSource publication:

ShoWorld splits motion from shape in occupancy forecasting, lifting dynamic mIoU by 6.06 points and beating GAST's 1-second IoU at 3 seconds

Synopsis

Occupancy world models must forecast the future semantic label of every voxel in a driving scene, yet moving objects stay much harder than static regions, with dynamic mIoU 10.8 to 13.8 points below static mIoU. ShoWorld separates where an object goes from what it looks like: a motion forecaster emits several probabilistic trajectories per object and fuses their renderings by probability, while a shape component moves sub-voxel points with occupancy probabilities and offsets rigidly with the object. On Occ3D-nuScenes with ground-truth occupancy and ego trajectory, it improves overall mIoU and IoU by 6.56 and 9.16 points over the previous best, GAST, and dynamic mIoU and IoU by 6.06 and 5.53 points.

Source-provided article image: ShoWorld: Shape-Faithful Multi-Hypothesis Occupancy World Model
arXiv

Interpretation

An occupancy world model can now keep a vehicle going straight, turning or stopping as separate futures instead of averaging them into an intermediate path. Earlier occupancy world models mostly gave no explicit object trajectories or described motion with a single deterministic flow field, so positional uncertainty leaked into voxel-level shape supervision. On Occ3D-nuScenes with ground-truth occupancy and ego trajectory, ShoWorld improves overall mIoU and IoU over GAST by 6.56 and 9.16 points, and dynamic mIoU and IoU by 6.06 and 5.53 points.

Each moving object is first encoded as an object token carrying a bird's-eye-view layout of drivable surfaces and obstacles, relative poses to nearby agents and the ego vehicle, and the future ego trajectory. From these tokens the motion forecaster uses learnable mode embeddings and fixed motion anchors (keep, brake, accelerate, turn left, turn right, turn while braking) to produce several planar rigid trajectories with a probability for each. Fusing all hypotheses at inference instead of rendering only the most likely one raises dynamic mIoU by 0.72, 1.27 and 1.43 points at the 1, 2 and 3 second horizons, with the gain growing over the horizon.

Shape no longer comes from moving voxel centers and re-voxelizing; each voxel is split in the horizontal plane into sub-voxel points that carry an occupancy probability and a bounded offset. Re-voxelization under ground-truth motion still misplaces about 22% of moving-object voxels within the first second and 35% in the third, creating holes or inflating and eroding object extent. Comparing rendering methods under ground-truth motion, learned sub-voxel points improve IoU over voxel centers by 2.8, 3.3 and 3.3 points at the three horizons; swapping back to voxel centers at inference lowers dynamic mIoU by 0.53, 0.51 and 0.48 points.

Perspective

The demonstrated scope is 4D occupancy forecasting on Occ3D-nuScenes, using 700 training scenes and 150 validation scenes, forecasting 3 seconds from 2 seconds of history and extended to 8 seconds, plus zero-shot evaluation on 202 Occ3D-Waymo validation scenes. The intended readers are researchers and engineers working on occupancy world models and trajectory prediction for autonomous driving.

Each object is treated as a rigid body moving in the ground plane, so articulated motion such as pedestrian limbs is not modeled; object grouping comes from observed occupancy and its errors propagate into the forecast; shape is computed once in the current frame, so occluded parts remain missing and objects entering later cannot be forecast; and fusion stays below the best single hypothesis at longer horizons.