Skip to main content
Back to timeline
arXivSource publication:

Robots learn to act from the present by training on later frames: HSD reaches 98.50% on LIBERO and lifts real-robot success from 24.17% to 37.50%

Synopsis

A robot must choose an action before it sees the outcome, while a demonstration records what followed the move. Hindsight Self-Distillation (HSD) trains a teacher that sees both current and later frames to pass two targets to a student, its denoising action predictions and its visual summaries, and the student reads only current frames at deployment. The student averages 98.50% success on LIBERO and 92.31% on RoboTwin 2.0, and across six real-world bimanual tasks success rises from 24.17% with Fast-WAM to 37.50%.

Source-provided article image: Hindsight Self-Distillation: Teaching Robots to Act from the Present
arXiv

Interpretation

Training can use frames recorded after the demonstrated action to teach a policy what to extract from the current image, while the deployed policy still sees only the current observation. Earlier video-based policies used video for planning or predictive features, or predicted future features inside the action denoiser; HSD places supervision on the visual summaries upstream of the denoiser, and those summaries supply all visual input to action prediction. On LIBERO, HSD averages 98.50% success, against 96.95% for the Fast-WAM reproduction under the same experimental conditions, 97.05% for Being-H0.7-style and 97.30% for FLARE-style; the largest gap is on Long, where HSD reaches 97.8% versus 93.4% and 93.6% for the two adaptations.

Teacher and student share a video encoder and an action predictor but hold separate sets of learned queries; the teacher's queries attend to current and later frames, while the student's attend only to current frames. Each query set compresses its images into a small set of feature vectors, and these vectors, projected into keys and values, form the only visual interface to the action predictor, so action supervision trains the queries together with the shared weights. Besides learning from demonstrated actions or the teacher's denoising prediction at the same noise level, the student aligns its final visual summaries with the teacher's; at deployment one video pass per action chunk is reused across ten denoising steps.

Perspective

Evaluation covers the four LIBERO suites, LIBERO-Plus task variants, the clean and randomized conditions of RoboTwin 2.0, and six bimanual tasks on the YAM robot with 20 trials per real-world task. The analysis holds at fixed model weights and given the encoder randomness for the current images.

How much hindsight transfers depends on which teacher targets are predictable from the current context; the analysis holds at a fixed weight snapshot, while closed-loop gains are assessed empirically. Real-world evaluation uses 20 trials per task.