A frozen V-JEPA already encodes speed: counterfactual supervision lifts a 4B video model to 72.1, ahead of open peers
Synopsis
Video-language models are usually trained on captions and question-answer pairs, so they learn to retrieve the few attributes the text names rather than to read how a clip changes. V-JEPA-LM instead uses pixel transformations to build counterfactual samples whose two versions of the same clip have different answers, forcing the language model to read dynamics already present in a frozen encoder. Trained in one stage on 4.53M public samples with 32 visual tokens per frame, its 1B to 8B models lead every size class in average score across eleven video benchmarks, with the 4B model at 72.1.
Interpretation
A video-language model can be trained to read motion information already present in a frozen world-model encoder instead of guessing clip content from captions and question-answer pairs. The earlier recipe attached encoders such as V-JEPA to a language model following image-text VLM practice, with alignment, pre-training and instruction stages on captions and QA pairs; V-JEPA 2 used 88.5M samples and 288 tokens per frame. On held-out speed-ratio items, a linear probe reads the answer from frozen V-JEPA features at 86.7% and the bridge of a caption-and-QA-trained VLM keeps 83.5%, yet at the answer position only 47% to 54% remains and the VLM answers near chance (33.7%), giving the same answer to a clip and its re-timed counterpart 89% of the time.
Letting a pixel transformation set the answer forces the model to compare two versions of the same clip, where shared scene content cannot separate the answers and only the change can. Earlier temporal training signals were mostly single targets, such as judging whether a video plays forward or in reverse, or adding annotated temporal questions, which at the same weight brought no gain. With the same 2B model, mixture and schedule, adding counterfactual rows raises the twelve-score average by 4.5 points at 8k steps and speed questions by 32.8; an arrow-of-time-only arm at the same weight scores 56.37 at 8k, comparable to 56.65 for captions and QA alone.
The supervision keeps the readout working beyond the training constructions, including speed ratios absent from training and a video corpus unused for that family. Earlier video-language models sat near chance on speed questions of natural benchmarks, with the best open 4B model reaching at most 36.5 on TOMATO. Across six held-out sets of 3,294 items, each changing one factor, V-JEPA-LM sizes average 94.5 to 96.3 against 43.5 to 55.5 for open models; the 4B model scores 97.2, 100 and 99.4 on the untrained ratios 1.6, 2.5 and 4.0.
Perspective
The demonstrated setting is 32-frame short clips, with benchmark questions about videos of seconds to a few minutes; the model uses a frozen V-JEPA 2.1 encoder and one training stage on 4.53M public samples, at up to about 650 H100-hours. Text recognition, single-image question answering and hour-long video are outside the target, and long video is left to future work.
The speed change direction overlaps content directions (overlap 0.40 and 0.59), so answer balance must separate them; the pair loss does not guarantee a zero content coefficient when readout is not separable, so the content share is measured at the answer position rather than derived. The encoder symmetry assumption holds only approximately for V-JEPA, and conclusions rest on measurements of frozen features.
