Safety judges can now carry generalisation guarantees: LoRA-GT gives the tightest certificate in all 40 model–dataset–loss cases
Synopsis
For large language model judges that classify prompt–response pairs from a fixed generator, a PAC-Bayes certification pipeline yields non-vacuous and sometimes tight risk bounds without holding out a separate validation set, covering both 0–1 classification risk and Brier risk. It quantises and losslessly compresses the adaptor produced by standard parameter-efficient fine-tuning and uses its description length as the complexity term; the new LoRA-GT shares one quantisation scale across layers and gives the tightest certificate in all 40 model–dataset–loss cases across five guardrail models and four benchmarks. Certificate tightness tracks adaptation complexity more than base model scale.
Interpretation
Safety judges can now obtain formal generalisation guarantees on the same labelled data used for training, without carving out a validation split. Earlier compression-based PAC-Bayes analyses targeted a language model's own generative risk and often relied on specialised compressible parameterisations designed for large-scale pre-training; validation-based bounds depend only on held-out error and sample size and cannot reflect model complexity. Non-vacuous certificates were obtained under both 0–1 and Brier loss across five guardrail models, Llama Guard 3 at 1B and 8B and Qwen3Guard at 0.6B, 4B and 8B, and four benchmarks, Egida, OpenAI Moderation, Aegis and BeaverTails, including Egida with 792 certification samples and OpenAI Moderation with 1,344, substantially smaller than prior compression-based language model certificates.
Certificate tightness is governed mainly by how complex the adaptor is, not by how large the adapted base model is. A standard LoRA adaptor gives a vacuous bound, that is, an upper bound of 1, in every model–dataset–loss case. Description length shortens in the order LoRA, LoRA-XS, LoRA-TN, LoRA-TT, then LoRA-GT, and this ordering matches the certificate ordering in most model–dataset–loss cases; LoRA-GT is tightest in all 40 cases, for example on BeaverTails its 0–1 test risk is 2 to 7 points higher than LoRA while its certificate falls from 1 to between 0.169 and 0.226.
Compressing the adaptor further can tighten the certificate, but the risk increase from compression offsets that gain, and the balance point depends on certification sample size. Compression changes two things at once: a shorter description reduces the complexity penalty, while functional distortion introduced by quantisation raises empirical risk. Under 0–1 loss, mean LoRA-GT certificates favour 2 bits on Egida and OpenAI Moderation but 4 bits on the larger Aegis and BeaverTails.
Perspective
The certificate concerns the generator-induced deployment distribution from which the certification data are drawn; changing the generator, decoding configuration, system prompt or prompt distribution constitutes distribution shift and requires recomputation. Evaluation covers guardrail models only up to 8 billion parameters, a scale relevant to low-cost oversight of larger systems.
Whether compact adaptations at substantially larger scales retain low quantisation-induced functional distortion and admit similarly tight guarantees remains an open question. At lower bit-widths, compression tightens certificates on the smaller benchmarks but loosens them on the larger ones, and this balance shifts with sample size.
