Skip to main content
Search

Yanjie Dong

1 related articles

Subscribe to updates
  1. arXiv

    Finite-time bounds arrive for importance-corrected momentum actor-critic on a single Markovian trajectory, with normalized steps converging faster and Muon pushing the rank factor into noise

    Finite-time analyses of importance-corrected momentum (ICM) actor-critic previously relied on unbiased i.i.d. gradients or restarted episodes, whereas reinforcement learning samples along a single Markovian trajectory whose gradients are biased and correlated. This analysis pairs a ridge-regularized pathwise least-squares temporal-difference critic with ICM actors that average policy gradients over frames of T transitions, giving finite-time bounds along one trajectory: the averaged gradient norm decays as O(K-1/3+1/T) for normalized steps and O(K-1/4+1/T) for unnormalized steps. The bounds retain a constant floor from the stationary-to-discounted distribution mismatch, while the remaining error terms fall at least as 1/T with frame length.

You’re all caught up