Skip to main content
Back to timeline
arXivSource publication:

Robot RL drops the per-episode scene reset: on the CRONOS benchmark, non-episodic fine-tuning beats episodic training on overlap and cluttered layouts

Synopsis

CRONOS places reinforcement-learning fine-tuning of vision-language-action models in a simulation benchmark where scenes are reset infrequently, with 16 pick-and-place tasks sharing 4 scenes. Naive non-episodic fine-tuning fails as object layouts and end-effector poses drift, but returning the arm to its canonical pose at each task switch, resetting scenes densely early, and resetting only unrecoverable scenes recover most of the loss. On out-of-domain overlap and cluttered layouts, non-episodically fine-tuned SpatialVLA reaches 64.3% and 55.0% success, above the 55.2% and 51.2% of episodic fine-tuning.

Source-provided article image: CRONOS: Benchmarking Non-Episodic Reinforcement Learning for Robotic Manipulation
arXiv

Interpretation

Reinforcement-learning fine-tuning of robot policies can forgo per-episode scene resets, but dropping pretrained vision-language-action models straight into that non-episodic setting loses substantial performance. Prior RL fine-tuning work assumes a scene reset every episode, while real deployment needs human intervention to reset; existing non-episodic methods target only small policies trained from scratch. Within CRONOS training scenes, widening the scene-reset interval from 80 steps to 320 and 640 steps lowers OpenVLA success to the levels reported for those intervals, with SpatialVLA following the same trend and the same trend holding on out-of-domain configurations.

Automatically returning the end effector to its pretraining canonical pose at every task switch substantially restores non-episodic fine-tuning efficiency. Pretraining demonstrations mostly start from a fixed end-effector pose, so the model carries an initial-pose bias; in non-episodic training the robot pose drifts continuously across tasks and misaligns with that bias. At scene-reset intervals of 320 and 640 steps, end-effector reset raises SpatialVLA success by 25 and 10 percentage points.

Beyond saving human resets, non-episodic fine-tuning yields policies more robust to dispersed scenes and continuous task sequences. Episodic fine-tuning uses four times as many scene resets as the non-episodic setting yet is less robust to out-of-domain initial layouts. Under the overlap layout, SpatialVLA success rises from 55.2% to 64.3% and OpenVLA from 55.4% to 59.3%; under the cluttered layout, from 51.2% to 55.0% and from 49.4% to 53.6%, respectively.

Perspective

The conclusions rest on the CRONOS simulation benchmark, with 4 scenes and 16 language-conditioned pick-and-place tasks of 80 interaction steps each, evaluated on two off-the-shelf models, OpenVLA and SpatialVLA, trained with PPO. Simulation was chosen because it is less time-consuming and more reproducible than real-world experiments.

Heuristic scene reset relies on privileged simulator state to detect unrecoverable scenes, and replacing it with a perception-based detector remains future work. Scene drift still limits learning under long reset intervals. The curriculum reset is only a simple two-stage schedule, with more advanced designs left for later.