Embodied agents stop rebuilding the whole KV cache on every memory update: HeMeR cuts real robot-arm inference from 2.94 to 1.87 seconds per step
Synopsis
At every step an embodied agent's declarative, procedural and working memory contents change, yet most contents persist between consecutive decisions; separate memory interfaces previously forced each updated memory context to be treated as a new input, so the full KV cache was recomputed. HeMeR anchors memory contents as typed chunks with KV ranges, recomputes only ranges affected by content updates and adapter switches, and applies RoPE remapping to chunks that merely shift position. On a real UR7e tabletop deployment it reached an 83.20% goal-conditioned success rate at 1.87 seconds per step across two long-horizon tasks, versus 64.28% and 2.94 seconds for MemRL.
Interpretation
An embodied agent's memory context stays largely unchanged between consecutive decision steps, so an update need not send the whole memory back through the foundation model. Declarative, procedural and working memory were previously accessed through separate interfaces, and each updated memory context was processed as a new input, preventing selective KV cache reuse for unchanged contents and forcing full recomputation. On real UR7e tabletop tasks, HeMeR reached an 83.20% goal-conditioned success rate at 1.87 seconds per step, against 64.28% and 2.94 seconds for MemRL; the real setting covers two long-horizon tasks with five independent physical repetitions each, 10 trials per method.
Memory contents are organized as typed memory chunks, each mapped to a contiguous KV range, so cache updates land only on affected ranges. The memory schema records each chunk's persistent ID, module type, semantic role, dependency annotation, normalized content, position, token span and KV range, plus an adapter identifier when applicable, and compatibility is judged on these fields rather than raw token overlap. On unseen tasks from ALFWorld and EmbodiedBench Habitat, HeMeR's task success rate exceeds MemRL's with shorter per-step inference time; Habitat uses Qwen3-VL-4B-Instruct on RGB observations and ALFWorld uses Llama-3.2-3B-Instruct.
Procedural memory is instantiated as LoRA adapters, and adapter switches enter the KV compatibility condition, so a switch recomputes only the procedural chunk and its dependents. KV states computed under a previous adapter could be incompatible with a new adapter even when other memory contents were unchanged, while adapter-level reuse alone did not handle declarative or working-memory updates. Under forced adapter switching at every decision step, HeMeR maintains its KV reuse ratio, whereas disabling adapter-stable reconciliation drops reuse to zero and yields latency comparable to full rebuild.
Perspective
Evaluation covers the ALFWorld and EmbodiedBench Habitat simulators and a real UR7e tabletop deployment; the real part is two long-horizon tasks with five independent physical repetitions each, 10 trials per method, and simulator results are averaged over five seeds. The intended readers are researchers and engineers building inference stacks for memory-augmented embodied agents.
Evaluation is limited to single-agent embodied tasks where memory contexts update across successive decision steps; in multi-agent systems, concurrent updates to shared memory may introduce consistency issues between memory contexts and cached KV states, which the current framework does not address. Adapter-stable reconciliation limits but does not eliminate cross-adapter drift, which is why the adapter identifier is part of the KV compatibility condition.
