JEPA4R reads single-depth-camera point clouds: imagined rollouts decode into 4D clouds within 1.1 cm over 200 steps
Synopsis
JEPA4R replaces the image input of a JEPA world model with colored point clouds from a single depth camera, predicts and plans in their latent space, and acts through a policy head on the frozen latents, a procedure the authors call 4D point planning. It leads pixel JEPA models on OGBench Cube-Single, Cube-Double, Cube-Triple and Scene, and ranks first on CortexBench's MetaWorld and Adroit Relocate at 98.4% and 88.0% success, above the runner-up R3M at 96.0% and 65.3%. A decoder attached to the frozen world model turns every imagined rollout into 4D point clouds, with mean point error within 1.1 cm over 200 environment steps.
Interpretation
JEPA4R lets a robot world model read colored point clouds from a single depth camera directly, and predict, plan and emit actions in the latent space of those clouds, a procedure the authors call 4D point planning. Most earlier JEPA world models read 2D images, where metric geometry is entangled with appearance and perspective and a pixel model must infer depth; point-cloud JEPAs mostly pretrain encoders without actions or forecast LiDAR occupancy, so metric geometry rarely reaches the model that predicts and acts. On OGBench Cube-Single, Cube-Double, Cube-Triple and Scene, JEPA4R leads pixel JEPA models, reaching 78.4% on Cube-Single and 64.8% on Scene; on CortexBench's MetaWorld and Adroit Relocate it reaches 98.4% and 88.0% and ranks first, above the runner-up R3M at 96.0% and 65.3%.
A decoder attached to the frozen world model maps both observed and imagined latents back to point clouds, so the scene a world model imagines can be seen directly. JEPA latents are trained to be predictable rather than reconstructive, and existing tools decode them only to images or audit them indirectly, leaving the imagined 3D scene unseen; explicit 3D world models that forecast points or particles instead generate every element of the next scene. On 4,096 validation frames of OGBench Cube, single-frame decoding gives a mean nearest-neighbor distance of 0.79 cm; letting the predictor roll forward autoregressively and decoding every step, the mean point error of the imagined rollout stays within 1.1 cm over 200 environment steps.
The policy head is trained on frozen latents, so its success rate measures what the latents make available to control, and JEPA4R shares the same heads with every baseline. JEPA world models are mostly scored in simulation by short-horizon goal reaching, where the cross-entropy method searches a few action blocks; such search tests the predictor over short in-distribution horizons, not how well the latents serve the policies robots learn. OGBench Cube uses GCIQL and Puzzle uses DQC, while MetaWorld and Adroit Relocate use the CortexBench imitation protocol with behavior cloning for 100 epochs on 25 demonstrations per task (100 on Relocate) and the best of 20 evaluations; the image and point-cloud datasets record the same simulator rollouts row by row, so states, actions, splits and starts coincide.
A world model and policy heads trained only in a digital twin run zero-shot on a real arm with both a gripper and a dexterous hand. DINO-WM and LeWM are evaluated in simulation, and V-JEPA 2-AC trains its predictor on real robot video; JEPA4R trains on data from a twin of its deployment arm. On a RealMan RM75-B arm, the Inspire EG2-4C2 parallel gripper picks up objects and pushes cubes, and the OYMotion AP001 dexterous hand pulls objects out of a holder, with each step encoding the single depth camera's point cloud and the policy head outputting the next action.
Perspective
The demonstrated scope is colored point clouds from a single depth camera, policy-learning evaluation on OGBench and CortexBench, and zero-shot execution on a RealMan RM75-B arm with a gripper and a dexterous hand; the decoder is trained on OGBench Cube data, and imagined-rollout error is reported on that task. The authors place a general world model shared across scenes and robots beyond this paper's scope and name collecting high-quality data for JEPA world models, and whether a general recipe exists, as the focus of next steps.
The authors state that a general world model shared across scenes and robots relies on large-scale, diverse 3D robot data, which remains scarce, and that how to collect high-quality data for JEPA world models and whether a general recipe exists remain open questions. On OGBench Cube-Quadruple, every encoder including JEPA4R reaches 0.0, and the authors note this long-horizon policy is hard to learn from random play data, with all six offline goal-conditioned RL methods of OGBench also reaching 0 even with ground-truth states. Under LeWM's goal-reaching planning setting, JEPA4R performs on par with LeWM, with gaps within the roughly 6-point variation of 50-start estimates.
