Skip to main content
Search

Francisco S. Melo

1 related articles

Subscribe to updates
  1. Q-learning with linear function approximation gets a convergence condition: heavier discounting lets a learning policy's information extend only to nearer greedy policies

    For Markov decision problems with infinite state spaces, where the Q-function cannot be stored entry by entry, Q-learning with linear function approximation previously lacked a convergence guarantee. The analysis gives a sufficient condition: the chain induced by the learning policy is uniformly ergodic, features are bounded and linearly independent, and the second moments of greedy-action features are dominated by those of learning-policy actions with a margin set by the discount factor; the parameter estimate then converges with probability 1 to a projected fixed point. As the discount approaches 1, the learning policy's information can be safely generalized only to nearer greedy policies.

You’re all caught up