Training agents under delayed observations: Hindsight Critics changes only the critics and tops delay robustness in 4 of 8 environments
Synopsis
Observation delays leave an agent acting on a past state, and standard reinforcement learning degrades as a result. Hindsight Critics estimates the current state from completed trajectories during training and gives that estimate only to the critics, while the actor still sees delayed observations at deployment. On the 8 continuous-control environments of DelayBench, it posts the highest delay-robustness score in 4, partially offsetting moderate delays, though delays of 16 timesteps remain hard.
Interpretation
Training can use hindsight to correct the state the critics see, without changing what the actor receives at deployment. Delay-aware methods usually rewrite the policy input through history stacking, recurrent policies or forward models, and those changes can be difficult or impractical in real deployments. Across the 8 continuous-control environments of DelayBench, Hindsight Critics has the highest delay-robustness score in 4, PPO in 3 and DDPG in 1.
The mechanism estimates the current state behind each action from later observations and uses that estimate only to train the critics. The actor stays conditioned on delayed observations, while the critics learn values from hindsight-aligned state estimates. Under a fixed delay, state estimates come directly from later observations in the same trajectory, with episode boundaries handled by the terminal state the environment returns; each algorithm, environment and delay is trained with 20 random seeds and evaluated deterministically over 20 episodes.
The asymmetry also stacks with existing delay-mitigation methods. With history stacking or a forward model alone, every method's score rises, but the policy input grows or a separate network must be deployed. Combined with history stacking, Hindsight Critics beats PPO and SAC in 5 of 8 environments; combined with the forward model it leads in 4.
Perspective
Evaluation is limited to the 8 continuous-control environments of DelayBench, the default observation-delay range, and a fixed, homogeneous delay assumption; at deployment the actor uses only delayed observations and the critics are discarded.
Because the actor's observations are not modified, the policy must implicitly anticipate future states, which is feasible only when delays are fixed and homogeneous; heterogeneous delays and jittery communication in real systems would make such an anticipatory policy too brittle, and at delays of 16 timesteps the method also struggles to learn satisfactory behavior.
