Skip to main content
Back to timeline
arXivSource publication:

State-only imitation gets simulator gradients: FOCUS matches short rollouts to expert segments and nears expert on a 152-muscle humanoid

Synopsis

FOCUS turns state-only, temporally unaligned demonstrations into short-horizon differentiable supervision: it matches each agent rollout window against several sampled expert segments by local optimal transport, aggregates them with a soft best-of weighting, and learns with a short-horizon actor-critic. It needs no expert actions, temporal alignment, expert-aligned initialization or teacher forcing. Across four contact-rich DFlex locomotion tasks, at least one FOCUS optimal-transport variant attains the highest mean return at the final interaction budget, approaching expert performance on the 152-muscle SNU Humanoid.

Source-provided article image: FOCUS: First-Order Control Using Optimal Transport Supervision for Imitation from Observations
arXiv

Interpretation

Imitation can now proceed from state sequences alone, without expert actions and without knowing which part of a demonstration corresponds to which part of the learner's behavior. Adversarial imitation from observations relies on discriminator-derived rewards and is sensitive to reward modeling and discriminator stability; optimal-transport methods that convert transport costs into surrogate rewards and optimize with zeroth-order RL inherit the sample inefficiency and high-variance updates of black-box policy optimization. On four contact-rich DFlex tasks, Hopper, Ant, Humanoid and SNU Humanoid, at least one FOCUS optimal-transport variant attains the highest mean return among compared methods at the final interaction budget, with the SNU Humanoid actuated by 152 muscles.

The approach splits whole-trajectory alignment into many short local alignment problems, so gradients pass through only a small stretch of simulation. FOCUS compares each short agent rollout window with multiple sampled expert segments using local optimal transport, and a softmin-weighted best-of aggregation emphasizes locally compatible matches, reducing destructive updates from phase-shifted or locally unreachable segments. Each window's transport cost becomes a differentiable imitation reward, and a learned critic bootstraps value beyond the rollout horizon; main experiments use the SAPO backbone trained from random initialization, with no behavior-cloning pretraining, expert actions, expert-aligned initialization or teacher forcing.

The supervision remains effective with a single expert trajectory, indicating it does not depend on broad demonstration coverage. Among prior differentiable imitation methods, DiffMimic relies on expert-aligned initialization and teacher forcing, while ILD relies on behavior-cloning pretraining followed by truncated backpropagation. Backbone and data ablations favor SAPO over SHAC and show FOCUS remains effective with one expert trajectory; each expert trajectory in the main experiments contains 1000 state-only observations.

Perspective

The demonstrated setting is four periodic locomotion tasks in DFlex, trained inside a known differentiable simulator with state-only expert trajectories; the method assumes a known differentiable simulator, and extension to a learned world model is left to future work.

The relative strength of the two optimal-transport variants is environment-dependent, and they differ in both distance metric and reward transformation, so the comparison cannot be attributed to one factor alone; best-of selection is local rather than global and can favor fragmentary matches when candidates are unrepresentative.