JEPA world models lose sample differences from initialization: SADL constrains the encoder only when supply falls short, cutting collapse across ten seeds from 50% to 0%
Synopsis
In JEPA-style world models the encoder representation is dominated from initialization by a component shared by all inputs, and batch normalization in the projector hides that direction from downstream losses, so sample differences shrink until they collapse. SADL compares the difference capacity a projector-side regularizer demands with the difference capacity the encoder can excite and pass on, and adds a constraint on the encoder only when supply falls short, with no auxiliary networks or trainable parameters. In single-task and jointly trained four-task LeWM and INTACT runs, collapse across ten random seeds falls from 50% to 0%, and after one epoch the encoder effective rank is 3.4–6.4 times that of the corresponding baseline.
Interpretation
The encoder representation of JEPA-style world models carries a component shared by all inputs from the start of training, and sample-to-sample differences hold only a small share of its energy, staying low after one epoch with an endpoint regularizer alone. Previously, a regularizer on the projector output was treated as sufficient to prevent representation collapse, but that regularizer acts only on the projected endpoint and does not check whether the encoder retains sample differences. On LeWM and INTACT, sample variation carries only a small share of the encoder representation's energy at initialization and still only a small share after one epoch with a demand regularizer alone, while the endpoint already carries most of the energy.
SADL keeps the original training objective and constrains the encoder directly only when the difference capacity the encoder supplies falls below what the endpoint regularizer demands. Unlike an always-on regularizer with a fixed weight on the encoder, this constraint is zero whenever supply meets demand, so it does not keep squeezing the representation. In four-task INTACT with SIGReg, an always-on coding-rate term degrades or collapses at small weights and over-regularizes at large ones, whereas SADL stays stable as its weight spans two orders of magnitude.
The corrected representation becomes usable earlier in control, and official search-free control success rises markedly early in training. The endpoint-regularizer-only baseline reaches much lower success at the same number of updates, and some of its seeds have collapsed by the end of training. With matched attention backend and update count, SADL on INTACT already exceeds the regularizer-only epoch-2 success at epoch 1, and LeWM success rises under both the MATH and Flash backends.
Perspective
The demonstrated scope is ViT-Tiny and ViT-S scale world models and image encoders, on PushT, Cube, Reacher and TwoRoom in single-task and joint four-task training, with eight demand regularizers including SIGReg, VISReg and QQWorld, and on ImageNet-LT, Galaxy10, CIFAR-10 and Imagenette for images.
The null-space argument for the shared component is specific to batch-normalized projectors, and the invariances of other normalizations are not analyzed; the transmission term measures second-order association only, so zero cross-covariance does not exclude nonlinear dependence; on CIFAR-10, where the baseline is already healthy, the gain is within seed noise.
