POISE predicts value baselines from a model's own internal states, beating GRPO and PPO with more stable training on a six-domain verifiable-reward corpus
The work introduces POISE (Policy Optimization with Internal State Value Estimation), which uses a lightweight probe reading signals already computed during the forward pass to predict the value baseline, and a cross-rollout construction that predicts each rollout's value from an independent rollout's internal states to preserve gradient unbiasedness; on Qwen3-4B and OLMo3-7B-Instruct-DPO across a six-domain verifiable-reward corpus, POISE outperforms other RLVR baselines with more stable training, and the probe matches a separate LLM-scale value model, generalizes to various tasks, and remains accurate as the policy scales.