Skip to main content
Back to timeline
arXivSource publication:

A factuality checker trained on 2% of the labels: CPIL reaches 81.04 macro-F1 across 11 tasks with a single inference pass

Synopsis

CPIL turns a small labeled set of document-claim examples into pairwise relational supervision instead of relying on large human-labeled or synthetic corpora. It builds positive pairs from different back-translated views of the same instance and negative pairs from opposite-label examples sharing one source document, then filters negatives that look too much like positives using the positive-similarity distribution. On the 11-task LLM-AggreFact benchmark, CPIL-2% and CPIL-3% reach macro-F1 of 81.04 and 81.60 while using only 2% to 3% of the FactCG-released labeled pool.

Source-provided article image: Label-Efficient Document-Grounded Factuality Detection via Cross-Paraphrastic Invariance Learning
arXiv

Interpretation

A compact factuality checker can now be trained on just 2% to 3% of the FactCG-released labeled pool while still making a single encoder-classifier forward pass at inference. Earlier compact detectors such as AlignScore, MiniCheck and FactCG typically depend on substantial human-labeled or synthetically generated document-claim corpora, while LLM-as-evaluator methods need repeated inference. On the 11-task LLM-AggreFact benchmark, CPIL-2% scores 81.04 and CPIL-3% scores 81.60 macro-F1, above every compared compact detector and two vanilla LLM references; CPIL-2% leads CCHD by 1.31 points and FactCG by 3.90 points.

The supervision comes from relations between examples rather than one label per example. Instance-wise classification extracts only one supervision target from each labeled example, whereas the same labels can constrain comparisons across multiple views. Removing the pairwise pretraining stage drops macro-F1 from 81.04 to 76.81; appending paraphrastic views as ordinary augmentation reaches only 77.34, and random pairing only 77.44, so the gain depends on how pairs are defined rather than on pairwise training alone.

Negatives are selected dynamically against the current representation space and refreshed during training. Static same-document negatives become easy as the encoder improves, while taking only the most similar negatives can admit paraphrase drift or annotation ambiguity. Replacing dynamic mining with static negative sampling lowers macro-F1 to 78.64, 2.40 points below full CPIL-2%; the semi-hard negative proportion gives its strongest results between 0.5 and 0.8.

Perspective

Evaluation covers the English LLM-AggreFact aggregate benchmark, spanning summarization, dialogue, claim verification, long-form question answering and retrieval-augmented generation, with training labels drawn from 2% or 3% of the roughly 16,700-example FactCG-released pool. The authors propose entailment- or entity-aware filtering, alternative paraphrasers, and broader cross-domain and multilingual evaluation as next steps.

The method assumes back-translated views preserve the original factuality label, yet small changes to entities, quantities, temporal expressions or negation may alter the document-claim support relation. The authors also note that the label-efficiency claim concerns reuse of selected factuality annotations, not training time or total compute, since view generation adds a one-time back-translation cost.