Class-Adaptive Self-Training for Relation Extraction with Incompletely Annotated Training Data (2023)

Abstract: Relation extraction (RE) aims to extract relations from sentences and documents. Existing relation extraction models typically rely on supervised machine learning. However, recent studies showed that many RE datasets are incompletely annotated. This is known as the false negative problem in which valid relations are falsely annotated as ‘no_relation’. Models trained with such data inevitably make similar mistakes during the inference stage. Self-training has been proven effective in alleviating the false negative problem. However, traditional self-training is vulnerable to confirmation bias and exhibits poor performance in minority classes. To overcome this limitation, we proposed a novel class-adaptive re-sampling self-training framework. Specifically, we re-sampled the pseudo-labels for each class by precision and recall scores. Our re-sampling strategy favored the pseudo-labels of classes with high precision and low recall, which improved the overall recall without significantly compromising precision. We conducted experiments on document-level and biomedical relation extraction datasets, and the results showed that our proposed self-training framework consistently outperforms existing competitive methods on the Re-DocRED and ChemDisgene datasets when the training data are incompletely annotated.

## introduction relation extraction (re) @xcite is an important yet highly challenging task in the field of information extraction (ie). compared with other ie tasks, such as named entity recognition (ner) @xcite , semantic role labeling (srl) @xcite , and aspectbased sentiment analysis (absa) @xcite qingyu tan and lu xu are under the joint phd program between alibaba and nus/sutd. in recent years, several works that specifically focus on revising the annotation strategy and quality of existing re datasets were conducted @xcite @xcite . for example, the docred @xcite dataset is one of the most popular benchmarks for documentlevel relation extraction. this dataset is produced by the recommend-revise scheme with machine recommendation and human annotation. however, @xcite and @xcite pointed out the false negative problem in the docred dataset, indicating that over 60% of the relation triples are not annotated. to provide a more reliable evaluation dataset for document-level relation extraction tasks, @xcite re-annotated 96 documents that are selected from the original development set of docred. in addition, @xcite developed the re-docred dataset to provide a high-quality revised version of the development set of docred. the re-docred dataset consists of a development set that contains 1,000 documents and a silver-quality training set that contains 3,053 documents. nevertheless, both works on docred revision did not provide gold-quality datasets due to the high cost of annotating the relation triples for long documents. learning from incompletely annotated training data is crucial and practical for relation extraction. hence, in this work, we focused on improving the training process with incompletely annotated training data. to tackle the problem of training with incompletely annotated datasets, prior works leveraged the self-training method to alleviate the detrimental effects of false negative examples @xcite @xcite ; zhou 0 16 32 48 64 80 96 ranked class index 0 20 40 60 80 100 score p reg. r reg. p r figure 1: precision and recall scores of each class (ranked by class frequency: left [high] →right [low]) on the development set of re-docred when the model is trained on docred. p reg. and r reg. stand for the regression lines of the scores. when a model is trained on incompletely annotated datasets, minority classes exhibit bad performance and frequent classes may have low recall scores, as shown in figure 1 . merging pseudo labels with original labels of the training dataset without considering the correctness of the former potentially deteriorates performance in subsequent iterations. in order to overcome confirmation bias in selftraining, we proposed a class-adaptive self-training (cast) approach that considers the correctness of the pseudo labels. instead of sampling the pseudo labels based on class frequencies, we introduced a class-adaptive sampling strategy to determine how the generated pseudo labels should be preserved. specifically, we calculated the precision and recall scores of each class on the development set and used the calculated scores to compute the sampling probability of each class. through such an approach, cast can alleviate confirmation bias caused by erroneous pseudo labels. our proposed approach preserves the pseudo labels from classes that have high precision and low recall scores and penalizes the sampling probability for the pseudo labels that belong to classes with high recall but low precision scores. our contributions are summarized as follows. (1) we proposed cast, an approach that considers the correctness of generated pseudo labels to alleviate confirmation bias in the self-training framework. (2) our approach was evaluated with training datasets of different quality, and the experimental results demonstrated the effectiveness of our approach. (3) although our approach is not specifically designed for favoring the minority classes, the minority classes showed more significant performance improvements than the frequent classes, which is a nice property as the problem of longtail performance is a common bottleneck for real applications. neural relation extraction deep neural models are successful in sentence-level and documentlevel relation extraction. zhang et al. (2017) proposed position-aware attention to improve sentencelevel re and published tacred, which became a widely used re dataset. yamada et al. (2020) developed luke, which further improved the sota performance with entity pre-training and entity-aware attention. chia et al. (2022b) proposed a data generation framework for zero-shot relation extraction. however, most relations in real-world data can only be extracted based on inter-sentence information. to extract relations across sentence boundaries, recent studies began to explore document-level re. as previously mentioned, yao et al. (2019) proposed the popular benchmark dataset docred for document-level re. zeng et al. (2020) leveraged a double-graph network to model the entities and relations within a document. to address the multilabel problem of docre, zhou et al. (2021) pro-posed using adaptive thresholds to extract all relations of a given entity pair. zhang et al. (2021a) developed the docunet model to reformulate document-level re as a semantic segmentation task and used a u-shaped network architecture to improve the performance of docre. tan et al. (2022a) proposed using knowledge distillation and focal loss to denoise the distantly supervised data for docre and achieved great performance on the doc-red leaderboard. however, all preceding methods are based on a closed-world assumption (i.e., the entity pairs without relation annotation are negative instances). this assumption ignores the presence of false negative examples. hence, even the abovementioned state-of-the-art methods may not perform well when the training data are incompletely annotated. denoising for relation extraction re is susceptible to noise in the training data. noisy data can be categorized into two types: false positives (fps) and false negatives (fns). false positive examples are mainly caused by misalignment of knowledge bases. xiao et al. (2020) proposed a denoising algorithm that filters fp examples in distantly supervised data. wang et al. (2019) tackled the class-imbalance problem of re and ner by meta-learning. the false negative problem is also common in information extraction. li et al. (2020b); xu et al. (2023) used simple negative sampling strategies to alleviate the detrimental effects of fn examples on ner. most recently, guo et al. ( 2023 ) tackled the multi-label problem in re by entropy minimization and supervised contrastive learning. given that the fn problem is related to incomplete annotation, supplementing the annotation by self-training is a viable way to tackle this problem @xcite @xcite . however, self-training is susceptible to confirmation bias; conventional self-training suffers from the problem of error propagation and makes overwhelming predictions for frequent classes. prior research on semi-supervised image classification @xcite indicated that re-sampling of pseudo-labels can be beneficial to class-imbalanced self-training. however, existing re-sampling strategies are dependent only on the frequencies of the classes and do not consider the actual performance of each class. our method alleviates confirmation bias by employing a novel re-sampling strategy that considers the precision and recall of each class on the development set. in this way, we can downsample the predictions for popular classes and maintain high-quality predictions for long-tail classes. ## conclusions and future work in this work, we study the under-explored problem of learning from incomplete annotation in relation extraction. this problem is highly important in real-world applications. we show that existing state-of-the-art models suffer in this scenario. to tackle this problem, we proposed a novel cast framework. we conducted experiments on docre and biore tasks, and experimental results show that our method consistently outperforms competitive baselines on both tasks. for future work, we plan to extend our framework to the distant supervision scenario. from the domain perspective, we plan to apply our framework to image classification tasks.