D u RE : Dual Contrastive Self Training for Semi-Supervised Relation Extraction (2024)

Abstract: Document-level Relation Extraction (RE) aims to extract relation triples from documents. Existing document-RE models typically rely on supervised learning which requires substantial labeled data. To alleviate the amount of human supervision, Self-training (ST) has prospered again in language understanding by augmenting the fine-tuning of big pre-trained models whenever labeled data is insufficient. However, existing ST methods in RE fail to tackle the challenge of long-tail relations. In this work, we propose DuRE, a novel ST framework to tackle these problems. DuRE jointly models RE classification and text generation as a dual process. In this way, our model could construct and utilize both pseudo text generated from given labels and pseudo labels predicted from available unlabeled text, which are gradually refined during the ST phase. We proposed a contrastive loss to leverage the signal of the RE classifier to improve generation quality. In addition, we propose a self-adaptive way to sample pseudo text from different relation classes. Experiments on two document-level RE tasks show that DuRE significantly boosts recall and F1 score with comparable precision, especially for long-tail relations against several strong baselines.

## introduction relation extraction (re) from unstructured data sources is a key component of building largescale knowledge graphs (kg) @xcite . among all the re tasks, document-level re @xcite extracts subject-relation-object triples from documents, which remains daunting due to the significant challenges in modeling long text spans and obtaining high-quality supervision signals. current document-level relation extraction methods @xcite can discover the semantic relation that holds between two entities under supervised learning. however, these methods @xcite trained on re-docred @xcite dataset. alternatively, self-training (st) @xcite , a classic semi-supervised learning paradigm, has been proposed in relation extraction @xcite @xcite . st minimizes the prohibitively expensive human labeling by iteratively pseudo-annotating unlabeled data with a classifier which is then retrained with the augmented labels. in this way, st benefits from a vast number of unlabeled instances and extends the generalization bound @xcite . a significant challenge of st is inadequate training data for long-tail relations. as shown in fig. 1 , current document-level re systems @xcite do not perform well on long-tail relations, which hardly appear in the training data. for example, the f1-score for class located in is 83.02 while ethnic group is only 6.45. the reason could be that the amount of training data is vastly different (20k vs. 155). assuming that training data and unlabeled data have the same distribution, we cannot expect these long-tail relations to appear sufficiently often in the unlabeled text corpus. to address this, @xcite propose to re-sample training set and to assign more weight to the classes that have high precision and low recall. however, this method does not bring new information to the relation classifier. as a result, these self-training methods might not be able to improve the re performance on these rare relations. in order to solve the above issue of long-tail relations, we propose a novel method -dual contrastive self training for semi-supervised relation extraction (dure). unlike previous st methods @xcite , we simultaneously train a controllable text generator, generating diverse outputs given specific relation triples. to improve the controllability of the generator, we leverage the signal of the trained re classifier to label positive and negative generated sequences, and then apply a ranking calibration loss @xcite to contrast the positive and negative sequences to improve generation quality. in addition, we propose a self-adaptive way to sample pseudo text from different relation classes. we add noise by increasing generation temperature for relations with higher precision, which introduces diversity to the training set and helps reduce overfitting. besides, we sample more examples from relations with lower recall. since long-tail relations usually have a low recall (fig. 1 ), they are more likely to be sampled, and thus their recall can be increased through training. the contributions of this work are as follows: relation extraction: deep neural models have proven to be successful in sentence-level and document-level relation extraction. zhang et al. (2017) proposed position-aware attention to improve sentence-level re and published tacred, which became a widely used re dataset. yamada et al. (2020) developed luke, which further improved the sota performance with entity pretraining and entity-aware attention. papanikolaou and pierleoni (2020) proposed to use gpt-2 to generate pseudo data for improving sentence-level re performance. however, most relations in realworld data can only be extracted based on intersentence information. to extract relations across sentence boundaries, recent studies began to explore document-level re. as previously mentioned, yao et al. (2019) proposed the popular benchmark dataset docred for document-level re. to address the multilabel problem of document-level re, zhou et al. (2021) proposed using adaptive thresholds to extract all relations of a given entity pair. zhang et al. (2021) developed the docunet model to reformulate document-level re as a semantic segmentation task and used a u-shaped network architecture to improve the performance of docre. tan et al. (2022a) proposed the use of knowledge distillation and focal loss to denoise the distantly supervised data for docre. wang et al. (2022) proposed a positive-unlabeled learning algorithm under incomplete annotation scenario. however, the methods above were not designed to tackle the challenge of long-tail relations. self-training: recently, self-training has flourished again by iteratively generating pseudo labels and augmenting the tuning of data-hungry language models, showing great advantages in further enhancing nlu @xcite @xcite @xcite and relation extraction (re) @xcite @xcite , where massive un-labeled input text exists. to tackle the issue of confirmation bias in self-training, wei et al. (2021) re-samples pseudo-labels based on the frequencies of training examples. tan et al. (2023) samples different numbers of pseudo-labeled data based on the development set performance. however, it ignores the relations where both precision and recall are low. all the above st methods for re apply st only in generating pseudo-labels. feng et al. (2023b) were the first to propose dual self-training to improve controllable text generation by introducing two kinds of noise. however, they do not study the effectiveness of dual self-training in classification problems (such as relation extraction), and in fact the strategy of adding noise sometimes harms the classification performance. unlike all the above st methods, we are the first to apply dual self-training on document-level relation extraction, generating both pseudo-labeled data using re classifier and pseudo texts given specific relation triples using a text generator. the design of contrastive loss and self-adaptive generation for different relation classes further improves the performance, especially on long-tail relations. @xmath0, the task of document-level relation extraction is to predict a subset of relations from r ∪ {n a } between entity pairs (e h , e t ) h,@xmath1...m,h̸ =t , where r is a predefined set of relations, n a represents no relations between given entities, and e h , e t are identified as head and tail entities, respectively. at the test time, the model needs to predict the labels of all entity pairs in document x. ## limitations though dure works well, it has the following limitations: ## ethics statement the generative part of our model may be utilized to generate fake information when the input triples are not factual, which could possibly be utilized to produce and propagate disinformation. also, the generated pseudo-text may contain some socially biased, offensive, or politically sensitive expressions. however, these generated texts are designed to be used as pseudo data in data augmentation to improve the robustness of the relation extraction model and to improve the performance of long-tail relations. class re-balancing self-training (crest) @xcite this algorithm re-samples the pseudo-labels generated by models based on the frequencies of the training samples. class-adaptive self-training(cast) @xcite this method calculates the precision and recall scores of each class on the development set and uses the calculated scores to compute the sampling probability of each class to alleviate confirmation bias caused by erroneous pseudo labels.