Dual Supervision Framework for Relation Extraction with Distant Supervision and Human Annotation (2020)

Abstract: Relation extraction (RE) has been extensively studied due to its importance in real-world applications such as knowledge base construction and question answering. Most of the existing works train the models on either distantly supervised data or human-annotated data. To take advantage of the high accuracy of human annotation and the cheap cost of distant supervision, we propose the dual supervision framework which effectively utilizes both types of data. However, simply combining the two types of data to train a RE model may decrease the prediction accuracy since distant supervision has labeling bias. We employ two separate prediction networks HA-Net and DS-Net to predict the labels by human annotation and distant supervision, respectively, to prevent the degradation of accuracy by the incorrect labeling of distant supervision. Furthermore, we propose an additional loss term called disagreement penalty to enable HA-Net to learn from distantly supervised labels. In addition, we exploit additional networks to adaptively assess the labeling bias by considering contextual information. Our performance study on sentence-level and document-level REs confirms the effectiveness of the dual supervision framework.

## introduction relation extraction (re) has been widely used in real-world applications such as knowledge base construction @xcite @xcite , question answering @xcite and biomedical data mining @xcite . given a pair of entities in a text (e.g., sentence or document), the goal of re is to discover the relationships between the entities expressed in the text. more specifically, we aim to extract triples from the text in the form of e h , r, e t where e h is a head entity, e t is a tail entity and r is a relationship between the entities. to train a model for re, we need a large volume of fully labeled training data in the form of text-triple pairs. although human annotation provides high-quality labels to train the relation extraction models, it is difficult to produce a large-scale training data since manual labeling is expensive and time-consuming. thus, @xcite proposed distant supervision to automatically produce a large labeled data by using an external knowledge base (kb). for a text with a head entity e h and a tail entity e t , when a triple e h , r, e t exists in the kb for any relation type r, distant supervision produces a label e h , r, e t even though the relationship is not expressed in the text. thus, it suffers from the wrong labeling problem. for instance, if a triple u k, capital, london is in the kb, distant supervision labels the triple even for the sentence 'london is the largest city of the uk'. although each of the two labeling methods has a certain weakness, most of the existing works for re utilize either human-annotated (ha) data or distantly supervised (ds) data. to take advantage of the high accuracy of human annotation and the cheap cost of distant supervision, we propose to effectively utilize a large ds data as well as a small amount of ha data. since ds data is likely to have labeling bias, simply combining the two types of data to train a re model may decrease the prediction accuracy. to take a close look at the labeling bias, let the inflation of a relation type be the ratio of the average frequencies of the relation type per text in ds data and ha data, respectively. we say that a relation this work is licensed under a creative commons attribution 4.0 international license. license details: http: //creativecommons.org/licenses/by/4.0/ . recently, @xcite introduced a domain adaptation approach to tackle the labeling bias problem for re. it trains a re model on ds data and adjusts the bias term of the output layer by using ha data. although the bias adjustment achieves a meaningful accuracy improvement, it has a limitation. an underlying assumption of the method is that the labeling bias is static for every text since it adjusts the bias term only once after training and uses the same bias during the test time. however, the labeling bias varies depending on contextual information. for example, in docred dataset, most of the capital relation labeled by distant supervision are false positive. however, if the phrase 'is the capital city of' appears in the text, the label is likely to be a true label. thus, we need to take account of contextual information to extract relations more accurately by considering the labeling bias. to effectively utilize ds data and ha data for training re models, we propose the dual supervision framework that can be applied to most existing re models to achieve additional accuracy gain. since the label distributions in ha data and ds data are quite different, we cast the task of training re models with both data as a multi-task learning problem. thus, we employ the two separate output modules ha-net and ds-net to predict the labels by human annotation and distant supervision, respectively, while previous works utilize a single output module. this allows the different predictions of the labels for human annotation and distant supervision, and thus it prevents the degradation of accuracy by incorrect labels in ds data. if we simply separate the prediction networks to apply the multi-task learning, ha-net cannot learn from distantly supervised labels. to enable ha-net to learn from ds data, we propose an additional loss term called disagreement penalty. it models the ratio of the output probabilities from the prediction networks ha-net and ds-net by using maximum likelihood estimation with log-normal distributions to generate the calibrated gradient to update ha-net to effectively reflect distantly supervised labels. furthermore, our framework exploits two additional networks µ-net and σ-net to adaptively estimate the log-normal distribution by considering contextual information. moreover, we theoretically show that the disagreement penalty enables ha-net to effectively utilize the labels generated by distant supervision. finally, we validate the effectiveness of the dual supervision framework on two types of tasks: sentencelevel and document-level res. the experimental results confirm that our dual supervision framework significantly improves the prediction accuracy of existing re models. in addition, the dual supervision framework substantially outperforms the state-of-the-art method @xcite in both sentence-level and document-level res with the relative f1 score improvement of up to 32%. ## preliminaries we present the problems of sentence-level and document-level relation extractions and next introduce existing works for relation extraction. ## dual supervision framework we first present an overview of the dual supervision framework which effectively utilizes both humanannotated (ha) data and distantly supervised (ds) data for training re models. we next introduce the detailed structure of the output layer in our framework and propose our novel loss function with disagreement penalty that considers the labeling bias of distant supervision. then, we describe how to train the proposed model with both types of data as well as how to extract relations from the test data. finally, we discuss how the disagreement penalty makes each prediction network learn from the labels for the other prediction network although we use separate prediction networks. ## experiments we conducted a performance study for sentence-level and document-level res by following the experimental settings of @xcite and @xcite , respectively. all models are implemented in pytorch and trained on a v100 gpu. we initialized ha-net and ds-net to have the same initial parameters. more experimental details including implementations can be found in appendix a. ## conclusion we proposed the dual supervision framework to utilize human annotation and distant supervision based on the analysis of labeling bias in distant supervision. we devised a new structure for the output layer of re models that consists of 4 sub networks. the new structure is robust to the noisy labeling of distant supervision since the labels obtained by human annotation and distant supervision are predicted by separate prediction networks ha-net and ds-net, respectively. in addition, we introduced an additional loss term called disagreement penalty which enables ha-net to learn from distantly supervised labels. the parameter networks µ-net and σ-net adaptively assess the labeling bias by considering contextual information. moreover, we theoretically analyzed the effect of the disagreement penalty. our experiments showed that the dual supervision framework significantly outperforms the existing methods.