Multiple verifiers stop collapsing into one score: POLPO trains on Pareto rankings and raises full-pass rates in code and instruction following
Synopsis
In code generation and instruction following, one response often has to pass several checks, and the usual practice aggregates those outcomes into a scalar reward before policy optimization. POLPO keeps only the comparisons the verification itself supports: one response is better when it passes every check the other passes and at least one more, while responses with conflicting outcomes stay incomparable. The objective maximizes the total probability of all complete rankings consistent with these partial orders, and on controlled program synthesis, MBPP, APPS and multi-constraint instruction following it yields higher rates of satisfying every requirement than all-pass or pass-rate reward training.
Interpretation
Multi-verifier feedback no longer has to be folded into a scalar reward before it can drive policy optimization; the comparisons verification already establishes can serve as the training target. Earlier practice used an all-pass reward, a pass-rate reward or a weighted sum: all-pass gives every partially successful response the same score, pass-rate orders responses by how many checks they pass, and a weighted sum must specify how a gain on one check compensates for a loss on another. In a controlled FizzBuzz program-synthesis task, POLPO reaches the complete-success probability threshold on all five seeds, using about 96,000–100,000 sampled programs on average; within a 512,000-program budget, GRPO with an all-pass reward and POLPO–All, which also compares only fully correct programs, never reach it, and GRPO with a pass-rate reward reaches it in three, three and one of five seeds for the three targets.
The method turns verification outcomes into a partial order and fits those comparisons with a ranking probability model instead of first constructing a reward. Pareto dominance says a response ranks ahead only when it passes every check the other passes and at least one more; when each passes a check the other fails, the two are incomparable and training prescribes no order between them. On MBPP, Qwen3-0.6B improves All-pass from 42.00% for POLPO–Rate, the same objective and hyperparameters with pass-rate comparisons, to 44.73%, and Avg. pass from 48.53% to 52.42%; for Qwen3-4B the corresponding gains are 67.40% to 67.93% and 71.93% to 75.49%.
Ranking probabilities use the policy-to-reference log-probability ratio as each response's score, and a Plackett–Luce model assigns probability to complete rankings. Unlike a pairwise DPO-style objective, the likelihood sums the probabilities of all complete rankings consistent with verification, marginalizing over the ordering decisions verification leaves open rather than forcing incomparable responses to tie. The likelihood is computed exactly by a dynamic program over response subsets; across four coding training settings, each update costs 26–203 ms more than GRPO's clipped objective, which is 0.03–0.07% of a training update.
Perspective
The demonstrated scope is controlled FizzBuzz program synthesis, MBPP and APPS Introductory code generation, transfer from AceCode training to HumanEval and the EvalPlus MBPP subset, and three multi-constraint instruction-following datasets, IFM5, MulDimIF and MathIF; coding tasks evaluate the final checkpoint with greedy decoding, MBPP results span three training seeds, and APPS and AceCode use a single seed. The next step the authors propose is designing sampling strategies that expose verified improvements in a response group and thus supply more comparison supervision.
The learning signal depends on comparisons available within each response group: when no response strictly Pareto-dominates another, the group provides no comparison signal even if some requirements remain unsatisfied. On MBPP, genuinely partial orders occur in at most about 7% of groups for Qwen3-0.6B and about 2% for Qwen3-4B; on IFM5 and MulDimIF, the share of groups with identical outcome vectors rises to 84.7% and 88.9% at the POLPO endpoint. The authors also note that the constructed two-tier fixed-block stress test does not cover every possible partial order.
