Skip to main content
Search

# Safety Alignment

13 related articles

Subscribe to updates
  1. arXiv

    A $20 reward function can strip a model's safety alignment; Bait probes the reward to block the learning

    Agentic reinforcement-learning-as-a-service hands data sampling and reward rating to the user, and a harmful reward function can erase a model's safety alignment at a cost as low as $20. Bait mixes explicit harmful responses from a separately trained Bait model into the policy model's response group during rollout, uses the sign of the reward gap between Bait and non-Bait responses (reward divergence) to flag a harmful reward function, and flips that group's reward sign to penalize learning when it detects one. On the authors' testbed, Bait cut the average harmful score across three harmful-instruction datasets from 20.4 to 6.8 while average agentic accuracy rose from 93.1 to 96.0.

  2. arXiv

    Safety alignment no longer rests on a few thousand neurons: ABS-Align prunes them during finetuning and cuts some attack success by about 90%

    Safety alignment in open-weight models is highly sparse: pruning a small set of safety-critical neurons or ablating a single refusal direction can make a model produce harmful content. ABS-Align gradually prunes those identified safety-critical neurons during preference finetuning, forcing refusal behavior to be re-established through other parts of the network, and restores the pruned neurons afterward. Across Gemma-2-2B-IT, Llama-3-8B-Instruct, Llama-3.2-3B-Instruct and Qwen-3-4B, attack success generally falls against TwinBreak, NeuroStrike, refusal-direction ablation and ten SORRY-Bench black-box attacks, with reductions of about 90% for some attacks, while TruthfulQA improves and general capabilities change little.

  3. arXiv

    GRPO safety alignment loses its signal on hard prompts: Beacon restores a safe direction and cuts average safety error by up to 32.83 points across four models

    Safety alignment of large reasoning models commonly relies on GRPO, but responses sampled for one prompt often share the same safety outcome, so the within-group relative advantage vanishes and the model gets no direction toward safety on persistently unsafe prompts. Beacon pairs safety signal recovery, which filters high-contrast trajectories and directly supervises a safe reference response when the signal collapses, with safety-utility decoupled alignment, which assigns category-aware rewards and applies KL anchoring only to benign prompts. Across four Qwen3 and DeepSeek-R1-Distill variants and eight benchmarks, average safety error falls by 17.69 to 32.83 percentage points relative to the initial policies, with XSTest over-refusal between 0.40% and 1.20%.

  4. aclanthology.org

    No retraining, fewer harmful compliances: scaling three internal component groups lifts attack-time safety from 43.2% to 69.7%

    The work decomposes an LLM's refusal behavior into a cross-layer pathway: harmful detection heads in early and middle layers pick out harmful tokens, safety neurons in mid-to-late layers carry and stabilize that signal in the residual stream, and refusal heads turn it into a refusal. Scaling the weights of these three component groups without changing architecture raised average safety under GCG attacks from 43.2% to 69.7% across six models, with a 1.7% average accuracy drop on four standard benchmarks.

  5. arXiv

    Attention already sees the harm, but the MLP writes the refusal only at answer onset

    Across 18 Base–Instruct model pairs, harmfulness is more decodable from attention updates while refusal evidence is written mainly by MLPs, and post-training strengthens that writing without improving recognition.

  6. arXiv

    Poisoned fine-tuning can be repaired at two layers: harmful score falls from 11.54 to 0.08

    In fine-tuning-as-a-service, malicious fine-tuning can weaken a model's refusal behavior while keeping its downstream task performance. SLDR replaces layer-wise safety diagnostics with a signed sensitivity spectrum, trains a LoRA recovery adapter only on the two layers with the highest and lowest sensitivity, and uses a representation-based maliciousness score at inference to decide when to activate it. On Llama3.1/SST2, the average harmful score drops from 11.54 to 0.08 while downstream accuracy matches undefended fine-tuning, and the harmful score stays near zero up to a poisoning ratio of 0.9.

  7. Anthropic

    Anthropic releases Claude Haiku 5.5: large benchmark gains over Haiku 4.5 and roughly 75% lower average running cost

    Anthropic released Claude Haiku 5.5, a small model for high-volume, cost-sensitive tasks, reporting gains over Haiku 4.5 across GDPval-AA v2.1, AA-Briefcase v1.1, OSWorld 2.1, Humanity's Last Exam, Terminal-Bench 4.0, FrontierCode 1.1 and Chartography, a first-for-Haiku adjustable effort setting, and roughly 75% lower average running cost.

  8. arXiv

    A safety-aligned but backdoored teacher passes its backdoor to a clean student through on-policy distillation, reaching up to about 70% trigger attack success at a 3% poisoning rate

    This work studies whether on-policy distillation (OPD) for safety can propagate backdoors: the authors build safety-aligned but backdoored teachers with bi-GRPO and distill initially clean students through OPD, finding that a 3% poisoning rate yields up to about 70% trigger attack success, that more training epochs (67% after 16 epochs with only 10 poisoned samples) and top-K KL both accelerate backdoor transfer, and that a KL-reward-clipping method, Lazy Defense, delays this transfer.

  9. arXiv

    Modifying only 0.19% of down_proj weights in LLaMA-2-7B-Chat raises attack success to 53% Basic ASR and 56% GCG ASR while tinyBenchmarks accuracy stays at 51.6%

    This work asks whether safety-sensitive behavior in LLaMA-2-7B-Chat is concentrated in a sparse subset of parameters, using low-rank safety-associated subspace analysis and parameter-level safety-utility importance filtering; both approaches show highly non-uniform safety sensitivity, with the MLP down_proj consistently the most prominent safety-sensitive component and o_proj a smaller contributor, and parameter-level localization modifying only 0.19% of model weights in down_proj yields 53% Basic ASR and 56% GCG ASR while tinyBenchmarks accuracy remains 51.6% against a 52.2% unmodified baseline.

  10. arXiv

    Suan builds the preference-optimization objective directly at the gradient level, achieving stronger safety alignment while preserving response utility across baselines and benchmarks

    The work introduces Suan, a new preference optimization algorithm whose objective is formulated directly at the gradient level, bypassing the standard variational derivation and yielding more interpretable and robust training dynamics; extensive evaluations across a diverse suite of competitive baselines and benchmarks show Suan achieves superior safety alignment while fully preserving response utility.