Skip to main content
Search

# Bellman Policy Optimization

1 related articles

Subscribe to updates
  1. arXiv

    Bellman Policy Optimization: Rewriting Policy Mirror Descent as a Critic-Free Trajectory-Level Objective via the Bellman Equations

    The work introduces Bellman Policy Optimization (BPO), a critic-free method derived from Policy Mirror Descent (PMD): for autoregressive generation with terminal rewards, the Bellman equations reformulate PMD into a trajectory-level objective that avoids estimating intermediate state values, and it is proved that this objective shares the same unique optimal solution as the original PMD objective; the practical loss replaces GRPO's importance-sampling ratio with a smoothed ratio of complementary token probabilities as a mismatch-correction weight, achieving higher average accuracy than GRPO-ClipHigher, GSPO, CISPO, and DPPO on mathematical reasoning benchmarks.

You’re all caught up