Moto's action vocabulary no longer hinges on the seed: macaque premotor geometry is imposed on its discrete codes and still aligns on held-out shapes
Synopsis
In robot models such as Moto that learn discrete action vocabularies from unlabeled video, the training objective fixes frame reconstruction but not which motion receives which code: two independently trained tokenizers assign nearly unrelated codes, and evaluating one tokenizer in bf16 reassigns 74% of its tokens. The authors impose the representational geometry of macaque ventral premotor cortex (PMv) during shape drawing as a prior, using dual-stream training with a straight-through similarity loss to reshape the code assignment; on five shape categories the loss never saw, the PMv-trained tokenizer aligns with PMv better than the released tokenizer and two no-prior runs, and better than the session-averaged 95th percentile of 100 random kinematic encoders, 0.42.
Interpretation
In a discrete action vocabulary learned from video, which motion receives which code is not fixed by the training objective; changing the seed or the numerical precision rewrites most of the assignment. Such tokenizers were previously trained only on frame reconstruction error, with the vocabulary assignment treated as a by-product of training; on the same 1,280 CALVIN validation clips and 10,240 tokens, two independently trained Moto tokenizers reach an adjusted Rand index near zero, and after the best one-to-one relabeling they agree on 4.1% of tokens, against 3.9% for a random pairing. Both checkpoints reconstruct frames well, so good reconstruction can accompany nearly unrelated code assignments; the released tokenizer evaluated in bf16 instead of fp32 reassigns 74% of its tokens, TF32 reassigns 21%, and a randomly initialized tokenizer changes only 2% under bf16.
Using the representational geometry of macaque ventral premotor cortex during shape drawing as a prior can decide how a robot's action vocabulary groups motions. Vision work has imposed measured brain structure on artificial networks, but this had not been applied to the discrete action vocabularies robots learn from video; the target here is PMv, which represents strokes as reusable, categorical action symbols. On five shape categories the loss never saw, the PMv tokenizer's representational similarity analysis score with PMv exceeds the released Moto tokenizer, two no-prior runs, and the session-averaged 95th percentile of 100 random kinematic encoders, 0.42; the kinematic, language, and random target tokenizers all fall below the no-prior runs. The advantage holds on average, not in every session: the no-prior run with the same seed scores higher than the PMv tokenizer in two of four sessions.
Alignment is implemented by dual-stream training: the tokenizer keeps Moto's frame reconstruction objective while encoding macaque strokes on alternating batches, and a straight-through similarity loss pulls the geometry of its code usage toward each recording session's PMv geometry. The loss acts on which code each input selects rather than on continuous features, because the policy only sees the discrete codes; a loss on a soft relaxation of the assignment can be satisfied by small changes in near-uniform soft assignments without moving any input to a different code. The forward pass uses the hard nearest-code assignment and gradients flow through a softmax over negative code distances; within each session, code usage is averaged over the 13 training categories and compared with the PMv representational dissimilarity matrix through linear centered kernel alignment, with a penalty on low entropy of the code-usage histogram. The four recording sessions contain 49, 40, 55, and 52 PMv units and 307, 241, 348, and 399 strokes, and are never pooled.
Perspective
The demonstrated scope is a Moto tokenizer trained toward one macaque's four PMv recording sessions, one training run per target, scored on five held-out shape categories by representational similarity with PMv; all alignment evidence comes from macaque strokes rendered as frame pairs, not from the robot's own motions. The downstream manipulation test is fixed in advance on a CALVIN pipeline that reproduces Moto's published results, and the four tokenizers' Stage-2 runs are in progress, with results to be added in a revision.
An untrained kinematic descriptor, the stroke's initial direction, exceeds the PMv tokenizer, so structure beyond kinematics is not claimed; the label-shuffled PMv control's training block contains the neural patterns of three to five held-out categories, and its gap to the PMv tokenizer, about 1.2 paired standard errors, is unresolved. Each target is trained once, the four sessions come from one monkey, each session's held-out set has five categories and each score is a rank correlation over ten distances, and the PMv principal components were fit on all strokes.
