Skip to main content
Search

# Reinforcement Learning with Verifiable Rewards

19 related articles

Subscribe to updates
  1. NVIDIA releases Nemotron for Science with a post-training recipe: on drug-metabolism prediction, SFT lifts accuracy 12% and RL another 17%

    NVIDIA released Nemotron for Science, a collection of models, datasets, tools and recipes that lets research teams turn experimental records into a verifiable-reward environment and adapt a general multimodal model into a domain-specific science agent. In a drug-metabolism example, the model takes a parent compound and its transforming enzyme and outputs the metabolite SMILES; at a 16,384-token output budget, supervised fine-tuning raised Nemotron for Science accuracy by 12% and RL post-training lifted it a further 17%. The recipe also offers an optional teacher-model distillation warm start for cases where RL converges slowly.

  2. arXiv

    When verifiers can reject but not certify, NOPO learns only from failures and lifts average pass@16 above GRPO on code, logic and math

    When a verifier can reliably reject wrong answers but cannot reliably certify correct ones, standard reinforcement learning with verifiable rewards reinforces shortcuts that merely pass partial checks. NOPO treats verifier-positive samples as unlabeled candidates and updates the policy only by suppressing trusted negatives, scaling each update by the share of non-falsified responses in the same rollout group. On Qwen3-4B and Qwen3-8B, average pass@16 on code, logic and math benchmarks exceeds GRPO and other baselines, and the advantage persists under inference-time scaling to 512 samples and cross-benchmark transfer.

  3. arXiv

    BF16 catches FP16 in RL post-training: stochastic rounding stops small weight updates from sticking at low learning rates

    At the low learning rates common in reinforcement learning with verifiable rewards, mixed-precision BF16 training converges more slowly than FP16 because small updates to FP32 shadow weights are often erased when those weights are rounded to BF16, leaving the weights unchanged for several steps. Replacing round-to-nearest with stochastic rounding in that quantization step makes BF16 follow FP16's reward trajectory on 1.5B, 14B and 30B-A3B models across two math datasets, while drifting less from the reference model than BF16 at a raised learning rate and keeping BF16's dynamic range without loss scaling.

  4. arXiv

    Verifiable-reward RL gets its first prompt-level privacy guarantee, and the reward signal still lifts accuracy through the noise

    Reinforcement learning with verifiable rewards (RLVR) trains a language model on problems that may themselves be confidential, and the released weights can reveal which problems it saw. DP-GRPO aggregates all responses to one prompt into a single gradient, clips it once and adds Gaussian noise, so the privacy budget depends on neither the number of responses per prompt nor the clipping norm; the authors describe it as the first differential privacy guarantee for RLVR training. Training Qwen2.5-1.5B-Instruct with LoRA at a per-run budget of 8, the reward signal beats a control that removes only the reward by 2.65 points on MATH and 3.24 on GSM8K, positive in every seed.

  5. arXiv

    Same weights, two passes: Metis-1.6 trains Logos on 1.02T tokens to seventh of 17 on 40 benchmarks and second in mathematics

    Metis-1.6 releases two from-scratch language models, Praxis and Logos, each running the same hybrid Mamba-2 and attention mixture-of-experts stack twice per token and each trained on 1.02T tokens. Logos averages 52.93 across 40 post-trained benchmarks, seventh of 17 and 0.75 points behind Qwen3-4B, while the comparators that disclose a count report a median of 12T training tokens and every model above it used at least 28T. It ranks second of 30 in mathematics with 67.50% on AIME 2025, and is weak at code, instruction following and long context.

  6. arXiv

    Long-context RL training stops waiting on autoregressive rollouts: sparse self-drafting plus full-attention verification lifts throughput 1.39x

    Long-context RLVR rollout generation now drafts tokens with sparse attention and verifies them with full attention, raising end-to-end training throughput to 1.39x and 1.35x of autoregressive on Qwen3-1.7B and Qwen3-8B while keeping comparable task performance.

  7. arXiv

    Async RL squeezes context into fewer decode steps; INT4 KV cache speeds rollouts but loses accuracy, bounded INT8 protection recovers it

    Sustained concurrency in asynchronous RLVR rollout concentrates context processing into fewer decode steps, making the KV cache rather than weights the dominant cost. KV4-RL stores the full context in calibration-free INT4 and protects only attention sinks and recent context in bounded INT8; across full-parameter GRPO training on Qwen3-4B-Base and Qwen3-8B-Base, its six-benchmark mean accuracy stays within 0.5 percentage points of BF16 rollout while substantially accelerating rollout and full iterations.

  8. arXiv

    Most of the no-think RL gain in hybrid reasoning models is thinking leakage: erasing an 8-D think subspace at 8B removes 75% of it

    Training Qwen3-1.7B, 4B and 8B with GRPO in no-think mode raises accuracy while responses grow longer, carry more reflection words, and internal activations shift toward the original model's think mode. Weight scaling, activation patching and attention-head interventions indicate that much of this gain comes from reactivating reasoning the original model already had rather than learning new reasoning behavior; at 8B, replacing an 8-dimensional think subspace removes 75% of the gain, and the reverse exchange transfers 50% of it to the original model.

  9. arXiv

    Teacher supervision stops being copied token by token: reward-weighted distillation lifts math benchmarks by up to 6.5 points

    Distillation no longer treats every teacher token alike: each token's weight is set by its predicted effect on the student's post-update reward. Earlier token-selection methods relied on proxies such as uncertainty or teacher-student divergence, which need not identify reward-improving supervision. RAD scores each token by differentiating through a temporary optimizer step, then updates token weights by projected ascent, improving mean accuracy on six competition-math benchmarks by 6.5 points for Qwen3-1.7B and 2.5 points for Qwen3-4B-Instruct-2507 at a matched training-response budget, and reaching 54.69% and 58.67% pass@1 on LiveCodeBench-v6.

  10. arXiv

    RL-ARC calibrates answer confidence with reasoning confidence, cutting OOD ECE to 0.16 while holding accuracy

    The work proposes RL-ARC, a calibration-aware reinforcement learning framework that, on top of RLVR, elicits both the model's confidence in its reasoning process and its confidence in its answer, using reasoning confidence as an auxiliary signal: reasoning-guided regularization when the prediction is correct and an overconfidence penalty when it is incorrect; across mathematical reasoning and out-of-distribution complex reasoning and factual QA benchmarks, RL-ARC improves calibration metrics over existing calibration-aware training methods while keeping accuracy comparable to RLVR, which optimizes correctness alone.