Skip to main content
Back to timeline
arXivSource publication:

Mobile manipulators now get tested on memory: RoboCasa-Mem sets nine tasks, and its HMAP baseline lifts average success from 16.2% to 24.2%

Synopsis

Mobile manipulation robots often must decide when key information is no longer visible, yet existing benchmarks mainly test task execution and long-horizon control without isolating decisions that depend on interaction history. RoboCasa-Mem splits such memory into environment memory and process memory across nine tasks and supplies 1,800 successful demonstrations; the accompanying HMAP baseline plans with structured high-level memory and executes with geometry-augmented low-level control, raising average success across the nine tasks from 16.2% to 24.2%, with gains concentrated in hidden-attribute recall while improvements on process-memory tasks remain inconsistent.

Source-provided article image: Robocasa-Mem: Benchmarking Environment and Process Memory in Long-Horizon Mobile Manipulation
arXiv

Interpretation

Evaluation of mobile manipulation now includes whether a step must rely on history that is no longer observable, rather than only how long a task is. Earlier mobile-manipulation benchmarks mainly assessed task execution, long-horizon control, generalization and policy learning without isolating history-dependent decisions, while existing robotic memory benchmarks mostly target tabletop or fixed-base settings. The nine tasks are designed with decision points where the current observation alone cannot determine the action, and are organized into environment memory and process memory across five subcategories covering spatial, attribute, relational, single-task progress and multi-task scheduling.

Evaluation no longer fixes a single target in advance but derives later constraints from the robot's own earlier actions. For tasks with multiple valid execution paths, fixing one target or trajectory makes evaluation depend on a preset assignment rather than the robot's own history. In SweetCup either cup may receive sugar, and the evaluator later checks whether the policy selects the same cup; success is defined by task semantics rather than reproducing the scripted expert trajectory, with each task evaluated over 50 held-out scene seeds.

The HMAP baseline puts memory in high-level planning and leaves current multi-view geometry to low-level control. The low-level controller builds on π0.5, which uses the standard visual-language-state representation without explicitly carrying history. A Qwen3-VL-4B-Instruct planner maintains entity and process memory and emits subgoals; a frozen VGGT-Ω encoder processes three camera views, and task-conditioned queries with cross-attention modulate the action expert, with planner and controller trained separately.

The gains from explicit memory are uneven and concentrate on decisions that require recalling hidden environment states. On TempReport, HMAP raises task success from 0.22 for π0.5 to 0.86 and stage completion from 0.57 to 0.93; on SweetCup success rises from 0.04 to 0.10. Among episodes reaching the reporting stage, conditional success at recalling the correct temperature rises from 0.41 to 0.90; in PutBackCup the conditional success of returning the cup to its original location rises from 0.72 to 0.83, and the average conditional success of the two color-matched milk placements rises from 0.47 to 0.79.

Perspective

Evaluation is primarily in simulation, with only one object-transport task tested on a physical robot over 10 trials per method; each of the nine tasks is assigned to its primary memory category, although successful execution may involve more than one type of memory.

The five memory categories are not exhaustive, and the dataset contains only successful expert demonstrations; stage-level metrics help localize failures but do not fully separate memory errors from execution errors; SugarCount stage applicability varies across episodes with different target counts, so stage completion is summarized over different episode subsets; neither interleaved scheduling task is completed by any evaluated policy.