PointZero: 3D Point Track Completion as a Pre-training Objective for Transferable 3D Dynamics
The work proposes 3D point track completion as a pre-training objective: given a single RGB-D observation and sparse partial 3D tracks, it predicts future 3D tracks of all observed points, thereby learning a transferable 3D dynamics prior without robot action labels; the authors contribute a 2.9 million synthetic frame dataset spanning deformable, articulated, and rigid objects and train a transformer, PointZero, which outperforms prior methods on the same data, outperforms baselines on the recent PGND 3D dynamics benchmark after fine-tuning, and outperforms or matches baselines on 6 of 7 simulated and real-world robot manipulation tasks, while also training PointZero from scratch to isolate the benefits of the architecture from those of the pre-training objective and dataset, and releasin