STAD : Self-Training with Ambiguous Data for Low-Resource Relation Extraction (2022)

Abstract: We present a simple yet effective self-training approach, named as STAD, for low-resource relation extraction. The approach first classifies the auto-annotated instances into two groups: confident instances and uncertain instances, according to the probabilities predicted by a teacher model. In contrast to most previous studies, which mainly only use the confident instances for self-training, we make use of the uncertain instances. To this end, we propose a method to identify ambiguous but useful instances from the uncertain instances and then divide the relations into candidate-label set and negative-label set for each ambiguous instance. Next, we propose a set-negative training method on the negative-label sets for the ambiguous instances and a positive training method for the confident instances. Finally, a joint-training method is proposed to build the final relation extraction system on all data. Experimental results on two widely used datasets SemEval2010 Task-8 and Re-TACRED with low-resource settings demonstrate that this new self-training approach indeed achieves significant and consistent improvements when comparing to several competitive self-training systems.

## introduction relation extraction (re) is a fundamental task in information extraction, which aims to obtain a predefined semantic relation between two entities in a given sentence @xcite . in recent years, fine-tuning on the downstream re tasks with pretrained models @xcite has achieved significant progress with the rapid development of the "pre-train and fine-tune" paradigm @xcite which leverages large-scale unlabeled data. however, re still suffers from the data scarcity problem. for most re tasks, due to the taste is from the ginseng , not alcohol . probability 0.0 0.5 1.0 entity-origin(e1,e2) cause-effect(e2,e1) others most of birdtown's earliest residents hailed from eastern european countries. probability 0.0 0.5 1.0 entity-origin(e1,e2) cause-effect(e2,e1) others (a) confident instance the taste is from the ginseng , not alcohol . for simplicity, we list two detailed relations and use "others" to represent the other relations. the task-specific definition of relations, the lack of customized annotation data poses great challenge for the supervised re @xcite . meanwhile, manually labeling large-scale re data is extremely time-consuming, expensive, and laborious. as an alternative, automatically building annotated data for re attracts a lot of attention in the research community @xcite @xcite . self-training is a simple and effective approach to build auto-annotated data @xcite . the idea is to use a teacher model trained on human-annotated data to annotate the additional unlabeled data. then the human-annotated data is combined with some instances selected from the auto-annotated data to train a student model. in this paper, we follow the self-training framework to improve low-resource re, which is closer to practical situations where the task starts with a small seed set of human-annotated data. in previous studies, researchers often select the auto-annotated instances with high confidence, named as confident instance, and have achieved a certain success @xcite . figure 1 (a) shows an example of confident instance, where the teacher model can easily classify it as relation "entity-origin(e1,e2)" with the clue offered by "hailed from". therefore, we first follow this kind of self-training solutions to train the re system. however, in the preliminary experiments, we find that some relations might have similar expressions among instances, which makes the teacher model confused. as a result, for the uncertain instances, the teacher model gives similar high probabilities to some relations or assigns low probabilities to all the relations. an example of uncertain instance is shown in figure 1(b) , where the teacher model predicts the instance as relation "entity-origin(e1,e2)" with a probability of 56%, as "cause-effect(e2,e1)" (the ground truth label) with 42%, and as other relations with only 2%. it is hard to distinguish between the first two relations as the expression "... is from ..." is often used for both. in previous studies, the uncertain instances are often discarded due to the confusion. however, we argue that ignoring all the uncertain instances may not be appropriate since they may contain useful information. for example, it is a good clue that the answer is one of the first two relations with a probability 98% for the instance in figure 1(b) . ideally, we would wish to make full use of all the auto-annotated instances to improve the re system. but it is very hard due to the confusion problem. therefore, we split the uncertain instances into two groups: ambiguous set and hard set. the ambiguous set contains the instances for which the teacher model predicts similar high probabilities on some relations, while the hard set contains those that the teacher model assigns low probabilities to all the relations. in this paper, we focus on the ambiguous set and propose an approach to use the ambiguous instances and the confident instances to improve the system of low-resource re. in our approach, we tackle two main issues when exploiting the ambiguous instances: 1) how to identify the candidate labels of ambiguous instances; 2) how to train a new model with the ambiguous instances. for the first issue, we adopt a probabilistic accumulation approach to obtain a set of relations, that is, a set of candidate labels containing the vast majority of probabilities, and then label ambiguous instances with the candidate labels in the set. to deal with the second issue, we make an assumption: for the ambiguous instances, the teacher model does not know which relation is the exact answer, but it does know that #1) the answer is (with high probability) in the candidate-label set (likes the first two relations in figure 1(b) ) and #2) the answer is not one of the relations which are with very low probabilities (likes "others" in figure 1(b) ). under this assumption, we treat the ambiguous instances as partially-labeled instances @xcite , where the relations are divided into a candidate-label set and a negative-label set for each ambiguous instance. then, we propose a novel set-negative training method to learn from the ambiguous instances by the negative-label set. in order to build the final relation extraction system, we further propose a joint training method which supports both positive and set-negative training. our main contributions are as follows: to verify the effectiveness of our approach, we conduct experiments on two widely used datasets semeval2010 task-8 and re-tacred with lowresource settings. experimental results show that our proposed approach significantly outperforms the conventional self-training system which only samples confident instances and other baseline systems. teacher model labeled data confident data ambiguous data auto-annotated data student model positive training positive training negative training figure 2: framework of our approach. the red dotted rectangle is the conventional self-training approach that neglects the ambiguous data in auto-annotated data. ## our approach we first briefly introduce the relation extraction task as well as the self-training framework frequently used in the previous studies @xcite . then, we propose an algorithm to classify the autoannotated instances into three groups: confident data, ambiguous data and hard data. next, we introduce three tagging modes for ambiguous data. finally, we propose the training method to use both the confident and ambiguous data. the framework of the proposed approach is shown in figure 2 . ## experiments in this section, we carry out experiments to show the effectiveness of the proposed approach. we first introduce settings of datasets and parameters. then, we describe comparison systems used in this work. after that, we present the overall evaluation results, followed by further analysis. ## related work self-training. self-training is one of the most commonly used approaches for exploiting unlabeled data @xcite @xcite . with the development of neural network models and the growth of demand for labeled data, self-training has become a very active field in research. it is widely used to improve the performance of neural machine translation @xcite @xcite , question answering @xcite zero-shot re @xcite , trains a model to identify unseen relation labels in a test set with a few support examples or even none of the examples. the low-resource supervised re is designed to train a supervised model with small training data @xcite . in this work, we also focus on the low-resource supervised re. to solve the low-resource problem, one direction is to build a robust model which can make full use of small data and existing knowledge. for example, integrating semantic relation labels @xcite and introducing knowledge graph @xcite . another direction is to enlarge annotated data automatically. data augmentation is a widely used technology to enrich the data @xcite . self-training is also a conventional way to obtain automatically annotated data @xcite . partial label learning. in this work, the definition of partial label is a candidate set of labels for an ambiguous instance in a multi-class classification task @xcite . this is different from that in sequence labeling tasks @xcite and multi-label multi-class classification tasks @xcite . in order to learn from partially labeled instances, various methods have been proposed to deal with the problem @xcite . recently, with the help of self-training, @xcite proposes a self-guided retraining method to learn from partially labeled data. besides, yan and guo (2020) also proposes to recalculate the confidence of labels in a candidate set by taking the current model as a teacher. ## conclusion this paper proposes a novel self-training approach with ambiguous data (stad) for the low-resource relation extraction, which fully uses the autoannotated data. according to the probabilities predicted by the teacher model, we classify the auto-annotated data into three sets: confident set, ambiguous set and hard set. during training, we consider the ambiguous set, which is neglected by the previous studies, and adopt the set-negative training method to alleviate the noise problem. experimental results show that our proposed system consistently outperforms the self-training systems.