Towards Accurate and Consistent Evaluation: A Dataset for Distantly-Supervised Relation Extraction (2020)

Abstract: In recent years, distantly-supervised relation extraction has achieved a certain success by using deep neural networks. Distant Supervision (DS) can automatically generate large-scale annotated data by aligning entity pairs from Knowledge Bases (KB) to sentences. However, these DS-generated datasets inevitably have wrong labels that result in incorrect evaluation scores during testing, which may mislead the researchers. To solve this problem, we build a new dataset NYTH, where we use the DS-generated data as training data and hire annotators to label test data. Compared with the previous datasets, NYT-H has a much larger test set and then we can perform more accurate and consistent evaluation. Finally, we present the experimental results of several widely used systems on NYT-H. The experimental results show that the ranking lists of the comparison systems on the DS-labelled test data and human-annotated test data are different. This indicates that our human-annotated data is necessary for evaluation of distantly-supervised relation extraction.

## introduction there has been significant progress on relation extraction (re) in recent years using models based on machine learning algorithms @xcite @xcite @xcite @xcite . the task of re is to identify semantic relationships among entities from texts. traditional supervised methods require a massive amount of annotated data, which are often labelled by human annotators. however, it is hard to annotate data within strict time limits and hiring the annotators is non-scalable and costly. knowledge base head tail relation apple steve jobs founders adele london place_of_birth aragaki yui japanese nationality corpus sentence distantly-supervised label comment steve jobs left apple in 1985 . founders ✗false positive adele was born in the uk . na ✗false negative aragaki yui is an japanese actress . nationality ✓true positive jack , have you heard of hemingway ? na ✓true negative in order to quickly obtain new annotated data, @xcite use distant supervision (ds) to automatically generate labelled data. given an entity pair and its relation from a knowledge base (kb) such as freebase, they simply tag all sentences containing these two entities by this relation. through this way, the framework has achieved great success and brought state-of-the-art performance in re @xcite @xcite . but as an exchange, the auto-generated data may be of lower quality than those from annotators. for example, the sentence "steve jobs left apple in 1985 ." is not suitable for relation founders labelled by the @xmath0, founders, @xmath1. to address the above problem, previous studies regard distantly-supervised relation extraction as a multi-instance learning (mil) task to reduce the effect of noises while training @xcite @xcite @xcite . mil aggregates sentences into bags, where instances with the same entity pair are grouped into one bag. however, the wrong labelling problem still remains during system evaluation. there are two types of noises: false positive (fp) means that a sentence is tagged with a relation when it is not, and false negative (fn) means that a sentence is tagged with na relation due to the kb incompleteness, where na relation means there is not an available relation for the sentence or the relation is not included in the pre-defined relation candidate set. figure 1 shows the examples of fp and fn. therefore, the performance evaluated on the ds test data might mislead the researchers since the evaluation scores are not exactly correct. in this paper, we present a new re dataset nyt-h, which is an enhanced distantly-supervised relation extraction dataset with human annotation based on nyt10 @xcite . nyt10 is a widely used benchmark data on the task of distantly-supervised relation extraction, which has been used in many recent studies @xcite @xcite @xcite . however, the nyt10-test set is also distantly-supervised, which contains many fp and fn noises that may hinder the evaluation. besides, nyt10 is not consistent for building classifiers, where the relations in nyt10-train set and test set are not totally overlapped. there are 56 relations existed in the train set while only 32 relations appear in the test set. both the inaccuracy and the inconsistency make the evaluation results less convincing and less objective, which limit the development of distantlysupervised re. in nyt-h, we hire several annotators to label test data. with this new data, we are able to perform accurate and consistent evaluation that may boost the research on distantly-supervised re. our contributions can be summarised as below: based on this assumption, they construct a distantly-supervised re dataset by aligning wikipedia articles with freebase. restricted by the wikipedia corpus, expressions of sentences are very limited in each relation. @xcite publish the nyt10, a distantly-supervised dataset which takes the new york times news as the source texts. it is smaller in scale, but contains a variety of expressions. besides, they make a relaxation on mintz's assumption, they think that at least one instance, which has the same entity pair, can express the same relation. however, nyt10 does not match this assumption totally. without human annotation, they cannot ensure that there is a correct-labelled sentence in each bag. @xcite develop a new dataset called google distant supervision (gds) based on google relation extraction corpus foot_0 , which follows riedel's assumption and makes the automatic evaluation more reliable @xcite . however, noises that come with ds cannot be eliminated in the test set, so the automatic evaluation cannot be exactly accurate. ## nyt-h dataset in this section, we introduce the procedure of constructing the nyt-h dataset and list the statistics information of the data. we also compare our dataset with the previous datasets in detail. ## evaluation strategy we expect that nyt-h can serve as a benchmark on the task of distantly-supervised relation extraction. in this section, we design three different tracks for evaluation on nyt-h at bag-level and sentence-level. ## conclusion in this paper, we present the nyt-h dataset, an enhanced distantly-supervised dataset with human annotation, where we use ds-generated training data and human-annotated test data. the nyt-h dataset can resolve the inaccurate evaluation problem caused by the assumption of distant supervision. to compare the performance of the previous systems, we design three evaluation tracks, which are sent2sent, bag2sent and bag2bag. we conduct experiments on the newly built nyt-h data with widely used baseline systems for comparisons. all the evaluation scripts and data resources are available in https://github.com/spico197/nyt-h .