Skip to main content
Back to timeline
arXivSource publication:

Keep Jev's probability as the safety reward, not a 0/1 verdict: with a helpfulness term, HarmBench attack success on Llama-3.1-8B falls from 0.287 to at most 0.033

Synopsis

The harmful-probability output of the commercial safety judge Jev is used directly as a reinforcement-learning reward instead of being binarized, with a four-stage validation run before training, during training, and after each judge version change. On Llama-3.1-8B, the probability reward with a helpfulness term lowers HarmBench attack success from a 0.287 baseline to at most 0.033 at the final checkpoint; binarizing the same judge makes the share of confidently passed harm rise significantly in both seeds, meeting the escape criterion.

Source-provided article image: Jev-as-a-Reward: Validating Probabilistic Safety Judges for Reinforcement Learning
arXiv

Interpretation

Safety reinforcement learning can use the harmful probability returned by a probabilistic safety judge directly as its reward, without training a reward model and without collapsing the verdict into pass or fail. Earlier pipelines either fit a reward model to human preferences or score responses with rules or another model; probabilistic judges are cheap, but their confidence gate routes uncertain verdicts to a stronger judge, so confidently passed harm never reaches review. On Llama-3.1-8B-Instruct the probability reward lowers HarmBench attack success from a 0.287 baseline to 0.040 and 0.153 at the final checkpoint in the two seeds; on Qwen2.5-7B-Instruct it lowers the rate to about one fifth of the baseline.

What happens during training is concentration of harm into the blind spot rather than overall escape: the harm the judge misses shrinks in amount while its share of the remaining harmful responses grows. The share of confidently passed harm alone points to escape, since it rises in 37 of 44 comparisons across the 11 probability-reward runs. With Qwen3Guard-Gen-8B as primary reference, the passed-harm rate falls in all 11 probability-reward runs, significantly in 7 that pass stage two, while 4 go to human audit; in the main Llama run it falls from 1.48% to 0.54% of harmful-prompt responses.

Binarizing the same judge changes the verdict: the reward form alone can push harm into the blind spot. All 11 probability-reward runs show falling passed-harm amount, whereas the binarized runs, which keep the judge fixed and change only the reward form, show a significant rise. On the two Llama seeds the binarized runs raise the passed-harm rate significantly by 1.5 and 1.7 points under the Qwen3Guard reference, meeting the escape criterion; the rise is significant only on medical and mental-health prompts, with the remaining prompts rising 1.1 and 1.2 points without significance.

After a judge version change, the offline checks can be repeated cheaply instead of retraining from scratch. Earlier evaluations of decision-only judges scored a fixed set of outputs with a fixed judge version, could not detect failures that appear only under optimization, and left open what to recheck after a change. Re-scoring all 4,324 offline items with version 1.14.0 costs USD 0.14 in judge calls, the two versions agree on 98.2%–99.3% of items per dataset, and the frozen threshold still reaches 0.973 confident accuracy under the new version.

Perspective

The findings come from two 7–8B instruction-tuned policies (Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct), LoRA adapters and the single GRPO algorithm, with the commercial judge Jev in two versions as the main reward judge and the open guard classifier WildGuard used as a reward only on Llama. The human audit covers 200 end-of-training responses from two Llama probability-reward runs, so it bounds rather than settles the coverage of delegation.

The two reference judges agree only moderately (κ of 0.56–0.62), the two human annotators agree less (Cohen κ of 0.21), and reference-judge estimates of confident misses overstate their level seven- to ninefold. The escape signal with WildGuard as reward appears in one seed under one reference and has no human check. The significant passed-harm rise under binarization holds only on medical and mental-health prompts and rests on the Qwen3Guard reference alone.