Skip to main content
Back to timeline
arXivSource publication:

A frozen VLA learns new scenes without retraining: RAGu warps in-the-wild demos into place, lifting real-world success from 20% to 80%

Synopsis

RAGu lets a pretrained vision-language-action model adapt to a new scene without updating weights or expanding its context window: it retrieves trajectories from domain-mismatched in-the-wild demonstrations such as DROID, warps them into the target setup using geometric calibration and semantic correspondences, then steers the model's flow-matching denoising with an energy field. Real-world task success rises from 20% for the unguided baseline to 80%, simulation reaches 25% versus 12%, and cross-morphology transfer to a 29-DoF humanoid gains 13.6 percentage points on average.

Interpretation

Pretrained vision-language-action models degrade noticeably under even small shifts in camera viewpoint, background, object identity or robot morphology, and RAGu lets such models adapt to a new scene using external demonstrations without changing their weights. Earlier retrieval-based adaptation methods require source demonstrations from the same domain as the target, or need task-specific fine-tuning of the policy network; pasting retrieved examples into the input context is limited by the model's finite context window. On real-world tasks the unguided baseline reaches 20% success and RAGu reaches 80%; in the RoboLab simulation the figures are 25% versus 5%; the in-context retrieval baseline RICL, using the same pool of 100 warped DROID demonstrations, reaches 0% on the real-world tasks.

The guidance signal comes from demonstration trajectories warped into the target coordinate frame rather than from examples pasted into the input, so both model weights and context stay untouched. The procedure has three stages: filter demonstrations by task keywords into structurally similar groups, use Grounding DINO to locate objects playing the same functional role in the target scene and warp trajectories into the target frame accordingly, then coarsely prune candidates with dynamic time warping, an algorithm that aligns two time series. At inference the first denoising step picks the candidate most compatible with the policy by gradient norm and keeps it fixed for the remaining steps; across five RoboCasa tasks with 720-step episodes at 10 Hz, mean wall-clock time per episode stays within a few seconds of the unguided baseline.

Guidance only reshapes motion within the model's prior, so it complements rather than replaces parametric training. On GR00T-N1.5, 100 candidate trajectories raise average success by 13.6 percentage points for the un-fine-tuned model and by 12.0 percentage points for the fine-tuned model. In the cross-morphology setting, DROID demonstrations from a 7-DoF Franka guide a 29-DoF Fourier GR-1 humanoid, raising average success by 13.6 percentage points; guidance acts on Cartesian wrist trajectories obtained by differentiable forward kinematics, with no inverse kinematics or trajectory replay.

Perspective

The demonstrated scope is four transfer settings: DROID to a real 7-DoF Franka across 5 tasks, DROID to RoboLab across 9 tasks, MetaWorld Sawyer to a real Franka on the multi-stage Banana-to-Plate task, and DROID to a Fourier GR-1 humanoid in RoboCasa across 5 tabletop tasks; real-world experiments use 10 rollouts per task and simulation uses 25.

The method assumes the source database contains demonstrations that roughly match the target task category; semantic correspondence failures corrupt the 3D warping and the guidance signal, so it is not invariant to camera placement or occlusion; and guidance can only reshape motion within the VLA's existing prior. Across 900 RoboLab attempts the warping success rate is 65.1%, and the remaining failures come mainly from demonstration-side trajectory parsing errors, roughly twice as many as scene-correspondence failures.