Skip to main content
Back to timeline
arXivSource publication:

Robots stop imitating their own old moves: SeRA relabels autonomous rollouts with later policies, lifting real tomato-picking success from 40% to 80%

Synopsis

In interactive imitation learning, human corrections are sparse while autonomous robot rollouts yield abundant experience without action labels. SeRA has later policy checkpoints relabel that experience, filters relabels by agreement across checkpoints, and builds action sets around accepted relabels instead of imitating them exactly; across five simulated manipulation tasks in online and offline settings it beats SDP and baselines that imitate historical robot actions or exact relabels, and on a real tomato-picking task it reaches 80% success versus 40% for SDP under the same human feedback and training budget.

Source-provided article image: Old Experience, New Lessons: Relabeling Autonomous Robot Rollouts in Interactive Imitation Learning
arXiv

Interpretation

Robot experience from autonomous execution can now serve as unlabeled data for action supervision rather than being kept as an imitation target. Earlier approaches retained historical robot actions as imitation targets: IWR imitates them, CEILing uses evaluative feedback to select behavior, and Sirius downweights pre-intervention actions using intervention context; but autonomous actions need not be optimal even when they trigger no intervention or lie in a successful trajectory. Across five simulated manipulation tasks in both online and offline settings, SeRA reaches the highest success rate on every task, while baselines that use autonomous experience fall below SDP, which does not use it, on at least four of five tasks in each setting; on the real robot SeRA reaches 80% versus 40% for SDP.

The method has later-trained policies supply action labels for earlier observations, then treats each label as a region of acceptable actions rather than a single answer. Later checkpoints incorporate additional human corrections and may generalize them to observations collected earlier, but multiple checkpoints can share the same error, so agreement alone does not guarantee correctness. Each accepted relabel gets a desired action set centered on it with radius equal to its distance from the historical robot action, and the policy only raises the probability of actions inside the set; on partially supervised action chunks, a step-masked denoising loss combines retained human corrections with accepted relabels.

Consistency filtering and set-valued targets each carry weight, and removing either sharply weakens offline performance. Without consistency filtering, success drops from 96.3% to 31.4% on Square and from 98.9% to 9.5% on ThreePiece; shrinking the relabel radius ratio toward point-valued targets lowers success on all four offline tasks. These comparisons come from ablations on offline simulated tasks and align with SDP+Pseudo-labeling, which imitates accepted relabels exactly, scoring below SeRA in both settings.

Perspective

The demonstrated scope is five simulated manipulation tasks (PushT, PickCan, Square, TwoArmLift, ThreePiece) in online and offline learning plus one real tomato-picking task; the real comparison uses the same human feedback and a 14-hour A40 training budget, and the simulated teacher is an optimal-action teacher.

The method assumes the desired set around each accepted relabel contains an optimal action, and checkpoint agreement cannot guarantee this because later policies can share errors; the current method uses a fixed radius ratio for relabeled targets and discards relabels that fail the consistency test.