Document-level relation extraction drops iterative denoising: Syn2DRE's dual teachers cross-teach, training in 26 hours with 30,480 unlabeled documents
Synopsis
Document-level relation extraction must judge relations between entity pairs across sentences, and high-quality annotation is expensive, while pseudo-labeling methods often need repeated training and denoising and can reinforce their own errors. Syn2DRE places two differently structured teacher-student networks in a single training run: each teacher follows its student by exponential moving average, keeps only pseudo-labels with the lowest prediction variance, and the two teachers exchange high-confidence labels where their predictions disagree. On Re-DocRED it reaches 79.53/79.40 dev/test F1 against UGDRE's 78.28/78.14, and trains in 26 hours using 30,480 unlabeled documents, against 55 hours and 101,873 documents for UGDRE.
Interpretation
Pseudo-label training for document-level relation extraction moves from multiple denoising rounds to a single training run, with two teacher-student networks generating and filtering pseudo-labels in parallel. Earlier pseudo-labeling routes either used pseudo-labels without assessing their reliability or, as in UGDRE, relied on a single model with multiple rounds of training and denoising, adding clear computational cost while confirmation bias remained largely unaddressed. On Re-DocRED, Syn2DRE reaches 79.53/79.40 dev/test F1, against 78.28/78.14 for UGDRE and 76.47/76.14 for KD-DocRE; on DocRED its dev/test gain over UGDRE is +0.38/+0.56 F1.
Each teacher follows its own student by exponential moving average, and the two students use different architectures, KD-DocRE and UGDRE. Exponential moving average is a weighted smoothing of student parameters that makes teacher outputs more stable than a single forward pass; one student uses axial attention and the other a token-level graph, differing in entity-pair representation and training objective. Each teacher runs several stochastic forward passes per entity pair, measures uncertainty as the variance of relation logits, and selects positive and negative samples from lowest uncertainty upward, with the positive and negative selection proportions set for a class imbalance in which about 97% of entity pairs are negative.
When the two teachers disagree on an entity pair, they exchange high-confidence pseudo-labels and supervise the opposite student with soft labels. Identical models teaching each other can perpetuate the same errors, so cross-supervision lets each student learn from signals it cannot easily infer on its own, mitigating confirmation bias. In the Re-DocRED ablation, removing cross-supervision lowers dev F1 from 79.93 to 79.79, and removing the self-supervised step lowers it to 79.52; the full model reaches 79.50 on the test set.
The training procedure combines pre-training, warm-up, self-supervision and cross-supervision end to end, with cross-supervision run every fixed number of steps. Pre-training initializes the students on distantly supervised DocRED data and copies their parameters to the teachers; a warm-up on human-annotated data precedes alternating execution of the two steps, intended to avoid mutual overfitting from residual label noise. On Re-DocRED, Syn2DRE trains in 26 hours using 30,480 unlabeled documents, against 55 hours and 101,873 documents for UGDRE; on CDR it reaches 84.77 test F1, 8.47 points above the best baseline, and on GDA 85.45, a gain of 0.15.
Perspective
Evaluation covers three general-domain datasets, DocRED, Re-DocRED and HacRED, and two biomedical datasets, CDR and GDA; HacRED, CDR and GDA provide no unlabeled data, so the experiments treat their validation sets as unlabeled and avoid pre-training on their original labels. The intended readers are researchers and engineers working on document-level relation extraction and semi-supervised training.
The method still requires a non-trivial amount of labeled data, limiting its applicability in low-resource domains such as biomedicine; the dual-network design nearly doubles GPU memory use compared with UGDRE; the method is restricted to a binary two-model setup, and pseudo-label thresholds are fixed in the experiments, with adaptive thresholds left unexplored.
