Skip to main content
Back to timeline
arXivSource publication:

Legal judgment models keep guilty verdicts after the decisive fact changes: PROSE cuts damage to supported rulings from 17.5% to 8.5%

Synopsis

Legal judgment prediction models often keep a guilty verdict after a legally decisive condition has been altered, a tendency the authors call guilty bias. PROSE frames the correction as a paired constraint: the unsupported member of a case pair should flip, while its counterpart, differing in one decisive condition, should stay. On Qwen3-8B and GLM-4-9B it performs this selective repair through a one-time offline parameter edit, and at matched repair rates it reduces damage to supported verdicts from 17.5% to 8.5% on Qwen and from 24.3% to 7.1% on GLM.

Source-provided article image: PROSE: Preservation-Constrained Selective Editing for Guilty Bias in Legal Judgment Prediction
arXiv

Interpretation

The correction target shifts from lowering guilty predictions overall to selective repair between two versions of the same case: the unsupported version should flip to charge-not-established, while the supported version, differing in one decisive condition, should keep its verdict. Earlier routes, including non-guilty supervision, defense-oriented interaction, self-refinement and debate, aim at aggregate error reduction, and a lower overall error rate can come from a general downward shift in guilty scores that raises the false-non-guilty rate. On the CAIL-MC Pairwise Repair–Preservation Track, PROSE reaches higher pairwise safe repair than both the legal judgment prediction baselines and the model-editing baselines on Qwen3-8B and GLM-4-9B, with false-guilty and false-non-guilty rates no higher than 8.61 and 15.04 on Qwen and 15.04 and 10.61 on GLM.

The repair is delivered by a one-time offline parameter update, and deployment remains ordinary single-pass inference with no paired inputs and no repeated deliberation per query. Data-based correction requires multi-step training, while inference-time critique and debate repeat deliberation for every case. Offline cost is 1.1 GPU-hours for Qwen and 2.1 GPU-hours for GLM on one RTX 4080 SUPER; against a uniform margin shift calibrated to the same repair rate, PROSE reduces the fraction of base-correct supported-guilty cases flipped to wrong from 17.5% to 8.5% on Qwen and from 24.3% to 7.1% on GLM.

Evidence sensitivity survives a change of corpus and construction: with only an allegation the model tends not to establish the charge, and with full facts it still does. Full supervised fine-tuning, strongest on matched pairs, reaches only 49.64–51.00 accuracy on the evidence-sufficiency track, where false-guilty errors dominate at 94.29–97.25. Across four evidence conditions built from LeCaRD and Criminal-S, PROSE obtains the best cross-corpus macro-F1 on GLM and competitive performance on Qwen; on three rewritten evidence levels of CAIL-MC test facts, Qwen's establishment rate for allegation-only inputs falls from 93.8% unedited to 10.3%, while 86.1% of fully evidenced texts are still established.

Perspective

Evaluation covers charge-conditioned verification under Chinese criminal law with two backbones, Qwen3-8B and GLM-4-9B, rather than complete judicial decision-making; the pairwise track derives from CAIL2018 and the evidence-sufficiency track from LeCaRD and Criminal-S, the latter measuring only whether the supplied text supports the charge, not evidence credibility or a judicial standard of proof.

CAIL-MC uses LLM-assisted transformations of adjudicated facts, and independent legal review reduces but cannot eliminate generation artifacts and uncertainty in legal interpretation; its coverage-oriented construction and selection of editable cases do not represent the distribution of criminal cases in practice, and the evidence track has a small source-case sample and a fixed allegation-only template. PROSE relies on paired training examples and backbone-specific configuration choices, its finite strength search may miss feasible interventions, and parameter fitting approximates their activation effects, so preservation is empirical rather than guaranteed.