Rubrics that everyone passes or fails no longer go to waste: RuACT turns them into teachers and finer critics
Synopsis
In rubric-based reinforcement learning, a rubric that every sampled response fails or passes cancels out of the group-relative advantage; in Qwen3-4B medical training such inactive rubrics account for 60.88% of query–rubric observations. RuACT assigns each inactive rubric the role that supplies what it lacks: unreachable rubrics become privileged information for reward-gated on-policy self-distillation from a frozen self-teacher, and saturated rubrics are decomposed into child rubrics that separate current responses while sharing the parent's weight. On four medical and scientific benchmarks, RuACT with Qwen3-4B beats GRPO, rubric reweighting and rubric-guided exploration on every benchmark, raising ResearchQA from 64.82% to 69.44%.
Interpretation
Rubric-based reinforcement learning contains rubrics that give the current policy no learning signal: every sampled response fails the rubric, or every response passes it, and group-relative optimization cancels its contribution. Prior work used rubrics as critics, selecting informative feedback through reweighting or curricula, or as teachers, guiding exploration or distilling privileged information; but a constant contribution cancels under any weight, and replacing rubrics revises the original requirements. During Qwen3-4B training on RaR-Medicine, 122,768 query–rubric observations were counted: 25.19% unreachable and 35.69% saturated, 60.88% inactive in total, with the fraction between 58.95% and 60.86% at each checkpoint.
The two inactive states lack different things, so each receives the kind of supervision that fills its gap. A frozen Qwen3-4B satisfies only 1.13% of initially unreachable rubrics under ordinary prompting, rising to 91.25% when the rubric is stated in the prompt; responses that all pass a saturated rubric still show quality differences in 71% of cases under an independent fine-grained judge, versus 29% for unreachable rubrics. Both diagnostics are inference-only probes on RaR-Medicine with the frozen Qwen3-4B; the unreachable probe covers 100 initially unreachable pairs, with generic instructions as a 3.38% control.
RuACT reassigns roles by rubric state in every rollout group: unreachable rubrics go to the teacher, saturated ones to a finer critic, and informative rubrics keep their original rewards. On the teacher side, up to three unreachable rubrics with the largest absolute weights are appended to a frozen self-teacher's prompt, and distillation of token distributions on the highest-reward ordinary rollout prefix happens only when a guided response's overall reward exceeds every ordinary rollout; on the critic side, a proposal model seeing only the query and parent description generates child rubrics, and candidates with non-endpoint pass rates are retained and share the parent's weight budget. Across four medical and scientific benchmarks, RuACT with Qwen3-4B averages 59.12%, above GRPO's 56.52% and the strongest local baseline Reweight's 57.28%; with Qwen3-1.7B it averages 47.17%, above GRPO's 45.04% and RuscaRL's 45.78%.
Each role recovers the supervision it targets, and combining them works best. Without guidance, the trained policy satisfies 42.86% of initially unreachable rubrics versus 35.02% under GRPO; child refinement raises agreement between reward rankings and independent quality judgments from 57.74% to 60.77%. Recovery is measured every 40 steps with eight unguided rollouts on 931 initially unreachable rubrics from 331 fixed RaR-Medicine training queries; alignment is measured on blind judgments over 300 rollout groups and 2,400 responses, where shuffled children reach higher coverage but alignment falls to 53.62%.
Perspective
Experiments cover training on RaR-Medicine and RaR-Science and evaluation on HealthBench and ResearchQA, with Qwen3-1.7B and Qwen3-4B, 200 training steps and eight responses per group, using Qwen3-32B for training-time judgments and DeepSeek-V4.1-Flash for the main evaluation. All evaluation relies on LLM judges rather than expert review, and the trained models are research artifacts that should not be used for clinical advice or decisions without professional oversight.
Child rubrics are proposed by an LLM and may not always faithfully capture the parent's intent; inactive labels come from the current limited response group and reflect current samples rather than intrinsic policy properties; generalization to larger models and broader domains remains unexplored, and teacher guidance and critic refinement add inference cost during training.
