Skip to main content
Search

# Pessimistic Soft Value Iteration

1 related articles

Subscribe to updates
  1. arXiv

    KL-regularized offline RL closes the stochastic-transition gap: reference-policy data can learn without a coverage assumption

    KL-regularized reinforcement learning is the objective behind RL fine-tuning of language models, and its offline data are often collected by the reference policy itself, yet the sample size needed was known mainly for bandits or deterministic transitions. This work gives lower and upper bounds for finite discounted MDPs with unknown stochastic transitions under two data models, a fixed design and reference-policy trajectories: pessimistic soft value iteration matches the minimax rate when the temperature is not too small, and reference-policy data covers the optimum's path events up to a power law, enough to learn with no coverage assumption.

You’re all caught up