Skip to main content
Back to timeline
arXivSource publication:

Goals no longer have to come from real trajectories: TD-Aug imagines them with geometric horizon models, lifting average success on stitching tasks from 31 to 79

Synopsis

Offline goal-conditioned behavior cloning generalizes poorly on tasks that require stitching, because its training goals come only from future states within a single trajectory. TD-Aug instead samples the mixture policy's successor measure from a geometric horizon model and trains the policy on those imagined long-horizon states as extra goals. On OGBench pointmaze stitching tasks, average success rises from 31 to 79, above offline goal-conditioned RL baselines GCIQL, QRL, CRL and HIQL.

Source-provided article image: Imagining Goals with Geometric Horizon Models for Policy Generalization
arXiv

Interpretation

Offline-trained goal-conditioned policies perform poorly on zero-shot tasks that require stitching, and TD-Aug fills in the goal distribution those tasks need by treating states imagined by a generative model as training goals. Prior goal-conditioned behavior cloning sampled future states only from the same trajectory as goals, so it never covered states reachable only by composing different trajectories; clustering-based methods cover just one-step stitches and rely on heuristics such as a state distance metric. On OGBench pointmaze stitching tasks, goal-conditioned flow-based behavior cloning without augmentation averages 31 success, and with TD-Aug averages 79; in the same table GCIQL scores 35, QRL 58, CRL 4 and HIQL 30.

The procedure first learns a generative model that can sample long-horizon states directly, then uses it to produce goals in bulk and trains the policy on a mix of generated and real future states. The generative model is a geometric horizon model, a generative approximation of a policy's successor measure, trained with TD-Flows; the successor measure describes the probability distribution over states reached after following a policy from the current state. The training objective mixes single-step transitions with Monte Carlo future samples from the dataset, and on pointmaze-large-stitch this combined objective gives the lowest earth mover's distance to held-out long trajectories as the horizon grows.

The same goal augmentation also helps with image inputs and robot manipulation, indicating it is not limited to low-dimensional navigation. In visual tasks the geometric horizon model is learned in a low-dimensional latent space from an autoencoder, and generated goals go straight to the policy without decoding; manipulation tasks use action chunks and predict states after each chunk. On visual-antmaze stitching, autoencoder behavior cloning averages 29 success and reaches 56 with TD-Aug; on the cube-double, cube-triple and puzzle-4x4 stitch datasets, behavior cloning averages 13 and reaches 52 with TD-Aug.

Perspective

The conclusions rest on the OGBench simulation benchmark: pointmaze medium, large, giant and teleport stitching tasks, visual-antmaze stitching, and the cube-double, cube-triple and puzzle-4x4 stitch datasets. The authors identify scaling TD-based geometric horizon model training efficiently and stably to high-dimensional, partially observable real robotics tasks as an open question.

The authors state that the mixed objective is a biased approximation of the true successor measure as the horizon grows, because Monte Carlo samples come from truncated single trajectories; in visual tasks, relying on temporal-difference bootstrapping alone performs worse, which the authors attribute to the difficulty of bootstrapping in a high-dimensional approximate space.