Skip to main content
Back to timeline
arXivSource publication:

A coding-agent harness for real robots: Thea adds a scene graph and exit codes to close the physical loop

Synopsis

A coding agent's reach comes not from the model alone but from the harness around it: tools, context, and retry after failure. Thea ports that harness to real robots, wrapping each robot capability as a callable tool, supplying world state through a persistent scene graph and action verdicts through an evaluator that also names the failure cause. Across three robots including the Astribot S1, Thea holds the highest success rate as tasks lengthen from L1 to L3, and the evaluator judges 93.3% of 90 trajectory checkpoints correctly on average.

Source-provided article image: Towards the Harness of Embodied Agents
arXiv

Interpretation

Instead of one end-to-end policy attempting a whole task in a single pass, a language model picks one tool per turn, reads the result, and decides the next step, with navigation, grasping, evaluation and user dialogue all entries in the same flat tool registry. End-to-end policies do not detect a mid-task failure and do not recover within the run; SayCan selects the next skill without any verdict on the last one. On real robots across three difficulty levels L1 to L3, Thea reaches the highest success rate at all three, and the gap widens as tasks lengthen; at L1 the differences among methods are small.

The physical world does not come with readable state or decidable outcomes, so Thea adds two modules: a scene graph that turns scattered visual and depth observations into persistent object records with refs, positions, confidence and freshness, and an evaluator that after each manipulation decides whether the action should end, whether it succeeded, and why it failed. A coding agent reads source code directly and relies on exit codes and stack traces, while a robot has only a continuous video stream and an action output with no built-in done signal. The evaluator reads only the current observation and each tool's own post-condition, and averages 93.3% correct judgments over 90 trajectory checkpoints drawn from the Astribot S1, AgileX Cobot Magic and Unitree G1.

Evaluator verdicts in turn constrain the world model: only a confirmed success may edit the scene graph, so a confirmed pickup marks the object as held and a confirmed cabinet opening records the container as open. Letting the model report its own success would bias the judgment toward itself, so evaluation is triggered by a post-execution hook that the model cannot skip and whose reasoning it never reads. The evaluator misjudges 6.7% of failed and 13.3% of in-progress checkpoints as successful, while no successful checkpoint is misjudged.

Perspective

The demonstrations and quantitative experiments run on real robots, with the main quantitative comparison on the Astribot S1 and deployment explorations on the AgileX Cobot Magic and Unitree G1; tasks are limited to three levels covering tabletop pick-and-place, multi-object tidying, and cross-table retrieval and delivery.

All evaluator errors lean toward calling a failed or in-progress state successful, which forfeits a needed retry or cuts an action short; safety covers only clearance checks before base motion, and force limits, safety around people, and deployment beyond the lab remain open.