One video diffusion transformer now plays both dynamics: O2LaF learns actions from unlabeled video and transfers them zero-shot from humans to robots
Synopsis
O2LaF repurposes a single pretrained video diffusion transformer (video DiT) into both an inverse-dynamics encoder and a forward-dynamics generator, so the two share one video prior and learn transferable latent actions from video without action labels. Earlier inverse-dynamics models were largely trained from scratch, or pretrained video generators served only as the forward model. Across camera navigation and robotic manipulation, O2LaF sets a new state of the art in few-shot linear probing and supports zero-shot action transfer across visual contexts and human-to-robot embodiments.
Interpretation
Latent action learning no longer needs an inverse-dynamics model trained from scratch: one pretrained video diffusion transformer can serve as both inverse and forward dynamics and yield transferable action representations from unlabeled video. Earlier approaches either learned the inverse-dynamics model from scratch on raw pixels or frozen pretrained features, or used a pretrained video generator only as the forward model, leaving the inverse side without a video prior. On camera navigation and robotic manipulation, with frozen encoders under few-shot linear probing, O2LaF sets a new state of the art in both discrete and continuous action spaces; on GR-1 robot demonstrations its EEF probe outperforms DreamDojo, although the O2LaF encoder is trained only on human demonstrations with no robot-domain adaptation.
Both dynamics start from the same video prior but are restricted in what information they can see, so they divide the work of inferring actions and predicting their consequences. Inverse dynamics becomes a single-pass transition encoder whose learnable action queries read only the two endpoint frames of each transition; forward dynamics keeps its generative role but switches to frame-causal joint prediction and is trained at high-noise timesteps. Experiments train on MiraData and evaluate camera navigation on MIND, and train on EgoDex human demonstrations and evaluate manipulation on in-lab GR-1 demonstrations; ablations show causal joint prediction beats autoregressive prediction, a frozen backbone with only a trained action head is clearly weaker than LoRA adaptation, and scaling the forward model from 1.3B to 14B steadily improves action decodability.
Perspective
Evaluation covers camera navigation (trained on MiraData, evaluated on MIND) and robotic manipulation (trained on EgoDex human demonstrations, evaluated on in-lab GR-1 demonstrations), with no action annotations used to train the latent action models; human-to-robot action transfer is assessed only qualitatively because paired demonstrations are neither temporally nor spatially aligned.
The structure of the learned latent action space remains poorly understood, and interpreting an individual latent action still relies largely on inspecting its reference video; the forward dynamics is optimized for visual prediction rather than decision-making, and video diffusion models are computationally costly and do not directly expose executable actions or task-relevant rewards.
