Skip to main content
Search

# Group-Relative Policy Optimization

14 related articles

Subscribe to updates
  1. aclanthology.org

    Theorem-prover training stops scaling data: 500 boundary-hard tasks mined from verified proofs lift both provers

    Reinforcement learning for formal theorem proving is bottlenecked by scarce hard tasks: existing Lean problems are either already solved by current provers or too uninformative. ProofFrontier starts from verified Lean theorem–proof pairs, keeps each theorem statement unchanged, and hides proof-structural guidance around key local nodes so the prover must rediscover intermediate reasoning; base-prover pass rates then select tasks that are solvable in principle but not easily solved. GRPO training on 500 synthesized tasks improves DeepSeek-Prover-V2-7B and Goedel-Prover-V2-8B on miniF2F, ProofNet and PutnamBench over each base model.

  2. Memetic Computing

    A correct answer is not a correct derivation: GRPO gets a noise-aware group weight that keeps learning at high reward noise

    GRPO, a common reinforcement-learning recipe for reasoning models, judges a rollout only by whether its final answer is right, so a correct answer can hide unreliable reasoning, and a single corrupted reward can strongly distort group-relative advantages when a response group is highly imbalanced. S-GRPO minimizes the expected squared error between observed and latent advantages under a reward-noise model to obtain a closed-form, noise-aware group weight. Across several mathematical reasoning benchmarks and model scales it stabilizes training and keeps making progress under controlled reward noise where standard GRPO shows little meaningful learning.

  3. arXiv

    Diffusion LLMs lose accuracy fast at high parallelism; RePara trains two-token blocks and lifts GSM8K from 28.20 to 68.54

    Diffusion language models lose much of their accuracy when committing many tokens per step, and RePara reinforces short consecutive token blocks with group-relative policy optimization, raising LLaDA's GSM8K accuracy at Diff#16 from JustGRPO's 28.20 to 68.54.

  4. arXiv

    Software agents no longer get only pass or fail: the policy itself ranks debugging traces within the same outcome

    Reinforcement learning rewards for software engineering agents no longer stop at pass or fail: the current policy compares debugging traces from the same task and verifier outcome, turning those preferences into bounded auxiliary rewards and lifting Qwen3-14B average accuracy on SWE-bench Verified from 32.1% to 34.7%.

  5. arXiv

    Speech LLMs stop pooling at a fixed frame rate: a learned boundary policy cuts English audio tokens by about a quarter

    ASSET trains a speech encoder's segment boundaries with reinforcement learning, reaching recognition close to a transcript-assisted character-boundary oracle on LibriSpeech with roughly 25% fewer audio tokens.

  6. arXiv

    SRSP trains a style planner with a frozen TTS model's speech-likelihood reward, beating captioning baselines that see the target audio on five acoustic metrics

    The work first shows that speech-text alignment only weakly predicts downstream acoustic similarity, then proposes Speech-Rewarded Style Planning (SRSP), which trains a text-based style planner with GRPO using the teacher-forced likelihood of target speech tokens under a frozen TTS model as reward; on an English subset of the ISCSLP 2026 CoT-TTS corpus, SRSP outperforms the Base LLM and target-audio-informed captioning baselines on five downstream acoustic metrics and gains in LLM-judged contextual appropriateness and reference consistency.

  7. arXiv

    SynCo jointly optimizes synthesizer and reasoner in the same online batch, reaching 78.72% average across eight math reasoning benchmarks

    SynCo formulates agentic data synthesis and task solving as a coupled multi-agent reinforcement learning problem, updating a Synthesizer and a Reasoner from the same online interaction batch: the Reasoner receives rollout-level correctness rewards while the Synthesizer receives a task-level reward gated by task quality, answer reliability, and outcome-grounded teachability, with both updates feeding the next synthesis round. Across eight mathematical reasoning benchmarks, SynCo reaches 78.72% average accuracy versus 75.85% for a frozen-Synthesizer variant and 74.96% for the strongest external baseline, ScaleQuest-Math, with gains arising mainly from problems the initial model failed to solve.

  8. arXiv

    AutoSynth generates editable Helm synthesizer programs directly with an audio-conditioned autoregressive model, achieving competitive results in both synthesizer inversion and text-driven generation

    AutoSynth serializes MIDI performance events, fixed synthesizer parameters, and variable-length modulation routes into a unified sequence and learns their dependencies with an audio-conditioned autoregressive model, trained first by supervised learning on large-scale automatically constructed audio-program pairs and then by group-relative policy optimization with a mixed reward, so that it predicts a synthesizer program directly from reference audio and converts text into a program via a pretrained audio generation model, achieving competitive results in both synthesizer inversion and text-driven generation.

  9. arXiv

    OMatGRPO reinforces a crystal generator's discrete composition channel directly, raising mSUN from 13.4% to 45.5% and showing single-element repacking can inflate the benchmark

    Using group-relative policy optimization (GRPO), this work reinforces the discrete atom-type channel of the crystal generator OMatG, whose closed-form likelihood ratio makes element choices policy actions; with a reward combining stability, novelty and diversity plus sparse-hull and single-element guards, the yield of metastable, unique and novel structures (mSUN) on LeMat-GenBench rises from 13.4% for the pretrained model to 45.5%, while the paper records how the policy exploited earlier rewards and shows that single-element repacking is counted as discovery by existing mSUN implementations.

  10. arXiv

    CIPO trains LLM agents with counterfactual branches to adaptively pick tool granularity, raising success and decision compression across three benchmarks and two backbones

    The work proposes CIPO: it first mines executable composite skills from successful tool-use trajectories via budget-constrained BPE-style merging, then augments GRPO by branching at the first eligible granularity decision of each base rollout and using the paired outcome difference as a bounded supplementary reward; across TOOLATHLON, TOUCAN, and TRAJECT-Bench with Qwen2.5-7B and Llama3.1-8B, CIPO attains the highest Tool F1 and TSR in all six settings with an average Comp./Raw of 0.845 versus 0.912 for GRPO + Skills, and the gain does not come from simply invoking skills more often.