Skip to main content
Back to timeline
arXivSource publication:

Human-motion representations drop hand-set masks: BAL predicts latent chunks in both directions, cutting AMD long-term forecasting error to 90.5 mm

Synopsis

Learning human-motion semantics from 3D pose sequences has depended on hand-chosen augmentation strengths or mask scales to define the supervisory signal. BAL splits a pose sequence into fixed-length chunks, compresses each into one latent token, and trains by autoregressively predicting those tokens in both temporal directions, so the ordering of everyday behavior becomes the supervision. After pretraining on a mix of AMD, Nymeria and Embody 3D and freezing the representation, BAL beats masked-modeling and contrastive baselines on most metrics, reaching 90.5 mm MPJPE on AMD long-term forecasting.

Source-provided article image: BAL: Bidirectional Autoregression in Latent Space for Learning Human Movement Representations
arXiv

Interpretation

The self-supervised signal for human-motion representations can come from the ordering of behavior itself rather than from hand-set augmentation strengths or mask scales. Contrastive methods require representations to be invariant to hand-crafted augmentations, and masked methods require masked regions to be recoverable from context at a chosen temporal scale, so representations are sensitive to these choices and often need retuning per task. After pretraining on a mix of AMD, Nymeria and Embody 3D with frozen representations, BAL beats A4Mer, PUMPS, USDRL, BehaveMAE, MotionBERT and MacDiff on most metrics across motion captioning, long-term forecasting and long-term interpolation; AMD forecasting MPJPE is 90.5 mm, against a best baseline of 136.1 mm.

The method cuts a pose sequence into fixed-length chunks, compresses each into one latent token, and predicts neighboring chunk tokens in both temporal directions. Predicting future frames directly in pose space lets short-term joint-trajectory smoothness minimize the loss, so representations stay at local kinematics; predicting farther ahead is ambiguous because the same action appears as different poses across people, context and style. Pretraining uses about 14.2 hours of AMD, about 300 hours of Nymeria and about 500 hours of Embody 3D daily-activity data, downsampled to 5 fps with sequences up to 30 seconds and a fixed 4-second chunk; downstream, the representation extractor stays frozen and only task heads are trained.

Each prediction target is split into a global component for the whole sequence and a local component relative to the mean, predicted separately to avoid representation collapse. Minimizing the latent prediction loss directly pushes representations toward collapse: the extractor outputs highly similar representations and predictors can lower the loss by emitting near-constant outputs, a self-reinforcing loop. In a control run using one tenth of the full pretraining steps, removing the decomposition or replacing it with SIGReg keeps the feature standard deviation of L2-normalized representations low with very small training loss; the decomposition maintains substantially higher diversity throughout pretraining and better downstream performance.

Perspective

Evaluation covers the AMD, Nymeria and Embody 3D daily-activity datasets: long-term forecasting and 4-to-12-second interval interpolation are evaluated on all three, while motion captioning is trained and evaluated only on Embody 3D and Nymeria, which carry text annotations; the representation extractor is frozen downstream and only task heads are trained, so the conclusions are limited to 3D pose-sequence input in daily-activity settings.

An oracle global-translation alignment shows that part of the baselines' larger forecasting and interpolation errors comes from weaker encoding of global translation, yet BAL still leads in most settings after alignment, and that alignment relies on ground-truth poses unavailable at inference, so the original errors remain the practical measure. Chunk length shows a mild task dependence: shorter chunks favor forecasting and interpolation, longer chunks favor captioning.