Skip to main content
Back to timeline
openbiox BizardSource publication:

ReasoningShield moves safety checks onto the reasoning trace: its open 1B and 3B moderators reach 91.8% average F1 on open-source reasoning-model subsets

Synopsis

Large reasoning models can write unsafe content into intermediate reasoning steps even when the final answer looks safe. ReasoningShield moves safety detection onto chain-of-thought traces, with a taxonomy of 10 risk categories and 3 safety levels plus 7K training samples and 2.2K human-annotated test samples. Its 1B and 3B moderators are trained in two stages, and the 3B model reaches 91.8% average F1 on open-source reasoning-model subsets and 81.4% on commercial reasoning-model subsets.

AI-generated editorial illustration: GitHub - CosmosYi/ReasoningShield: [NeurIPS 2026] ReasoningShield: Safety Detection over Reasoning Traces of Large Reasoning Models · GitHub

Interpretation

The object of safety moderation shifts from the model's final answer to the whole reasoning trace it writes, so risks that appear in intermediate steps but are masked by the final answer can be judged on their own. Earlier moderation typically targeted final answers, while large reasoning models may give a seemingly safe conclusion yet include unsafe content in intermediate reasoning steps. The support is ReasoningShield-Test: 2.2K human-annotated query and chain-of-thought pairs, 600 in-domain and 1.6K out-of-domain, drawn from 5 datasets and 8 large reasoning models, independently annotated by three AI safety researchers with a Fleiss Kappa of 0.75 and gold labels set by majority vote.

The detector outputs a stepwise risk analysis and a judgment rather than an opaque score, which makes the moderation result traceable sentence by sentence. Traditional moderation models give only a black-box verdict, whereas ReasoningShield uses structured risk analysis as the alignment target of its first stage. Training runs in two stages: the first fine-tunes on 4,358 samples with detailed analyses to align structured risk analysis, and the second uses 2,642 hard negatives in contrastive learning to improve robustness to subtle risks. Base models are 1B and 3B, and the 1B version needs about 2.3GB of GPU memory for inference.

In the paper's evaluation, ReasoningShield-3B reaches 91.8% average F1 on open-source reasoning-model subsets and 81.4% on commercial reasoning-model subsets. These two figures cover different evaluation scopes and cannot substitute for each other; the 89.1% F1 reported in the CoT moderation comparison uses a different aggregation from the open-source subset average. The models are trained on only 7K samples, and the repository reports generalization across varied reasoning paradigms, cross-task scenarios and unseen data distributions; code and the 1B and 3B checkpoints are released under Apache-2.0, and the dataset under CC BY-NC 4.0.

Perspective

The demonstrated scope is chain-of-thought safety moderation: evaluation on the 2.2K human-annotated samples of ReasoningShield-Test, covering 10 risk categories and 3 safety levels, with 1B and 3B checkpoints and accompanying inference scripts. The repository's quick-start example has a single input row, and its metrics only check that the pipeline runs, not a benchmark result.

The repository states explicitly that 91.8%, 81.4% and 89.1% correspond to different evaluation scopes and aggregations, so benchmark splits and baselines need checking against the paper. The dataset is CC BY-NC 4.0 while models and code are Apache-2.0, so the two carry different terms of use. The repository reports good generalization to unseen data distributions but gives no specific figures for it here.