Skip to main content
Back to timeline
arXivSource publication:

When verifiers can reject but not certify, NOPO learns only from failures and lifts average pass@16 above GRPO on code, logic and math

Synopsis

When a verifier can reliably reject wrong answers but cannot reliably certify correct ones, standard reinforcement learning with verifiable rewards reinforces shortcuts that merely pass partial checks. NOPO treats verifier-positive samples as unlabeled candidates and updates the policy only by suppressing trusted negatives, scaling each update by the share of non-falsified responses in the same rollout group. On Qwen3-4B and Qwen3-8B, average pass@16 on code, logic and math benchmarks exceeds GRPO and other baselines, and the advantage persists under inference-time scaling to 512 samples and cross-benchmark transfer.

Source-provided article image: Negative-Only Policy Optimization for One-Sided Verifiable Rewards
arXiv · Page 1

Interpretation

Training can use only the side of the verifier signal that is reliable: rejected samples count as trusted counterexamples, while accepted samples count as candidates not yet falsified rather than correct labels. Standard reinforcement learning with verifiable rewards treats the verifier's binary output as the reward, reinforcing accepted samples and suppressing rejected ones; under partial verification, accepted samples include unchecked errors. Measured on public tests for TACO, LiveCodeBench and CodeContests, constraint checks on ZebraLogic, and an LLM self-verifier on OlympiadBench, the precision of negative judgments qneg ranges from 95.8% to 100%, while the share of verifier-positive samples that are actually wrong, qpos, remains 19.7% to 77.5%.

The policy update suppresses only trajectories the verifier rejects, and the number of non-falsified samples in the same rollout group sets how strongly that step suppresses. Negatives say what to avoid but not where the probability mass should move; applying negative preference optimization, designed for a fixed negative set, to on-policy rollouts lacks that directional information. At the sample level it uses a reference-relative negative preference optimization loss that self-attenuates; at the group level it uses a linear weight with gamma equal to 1, the count of non-falsified samples divided by group size, so a group with no accepted sample gets zero weight and is dropped from the update.

Without oracle rewards, this negative-only update raises average pass@16 above GRPO, EM-RL and w-Reinforce on code, logic and math benchmarks. GRPO directly optimizes the observed verifier reward, EM-RL only minimizes token-level entropy, and w-Reinforce gives negatives larger weight but still keeps a small positive weight. On Qwen3-4B, NOPO averages 52.86 pass@16 versus 48.93 for GRPO; on Qwen3-8B, NOPO reaches 52.13 versus 48.27 for GRPO; NOPO ranks first in nine of ten benchmark-backbone cells, with OlympiadBench on Qwen3-8B at 76.52 below GRPO's 76.97.

Perspective

Experiments use Qwen3-4B and Qwen3-8B in non-thinking mode; training uses only one-sided rewards from public tests, constraint checks or an LLM self-verifier, while evaluation uses oracle answers, with code, logic and math tasks corresponding to these three verifier types.

Negative-only learning is inherently delicate: it specifies what the policy should avoid but not where the removed probability mass should move, which motivates the conservative design. The authors note that in different one-sided verifiability scenarios, verifier-positive samples may contain exploitable structure such as pairwise comparisons or uncertainty-based filtering, and incorporating such partial positive signals while preserving the robustness of negative-only optimization remains future work.