Active Testing: An Unbiased Evaluation Method for Distantly Supervised Relation Extraction (2020)

Abstract: Distant supervision has been a widely used method for neural relation extraction for its convenience of automatically labeling datasets. However, existing works on distantly supervised relation extraction suffer from the low quality of test set, which leads to considerable biased performance evaluation. These biases not only result in unfair evaluations but also mislead the optimization of neural relation extraction. To mitigate this problem, we propose a novel evaluation method named active testing through utilizing both the noisy test set and a few manual annotations. Experiments on a widely used benchmark show that our proposed approach can yield approximately unbiased evaluations for distantly supervised relation extractors.

## introduction relation extraction aims to identify relations between a pair of entities in a sentence. it has been thoroughly researched by supervised methods with hand-labeled data. to break the bottleneck of manual labeling, distant supervision @xcite automatically labels raw text with knowledge bases. it assumes that if a pair of entities have a known relation in a knowledge base, all sentences with these two entities may express the same relation. clearly, the automatically labeled datasets in distant supervision contain amounts of sentences with wrong relation labels. however, previous works only focus on wrongly labeled instances in training sets but neglect those in test sets. most of them estimate their performance with the held-out evaluation on noisy test sets, which will yield inaccurate evaluations of existing models and seriously mislead the model optimization. as shown in evaluations p@100 p@200 p@300 held-out evaluation 83 77 69 human evaluation 93(+10) 92.5(+15.5) 91(+22) a false negative instance is an entity pair labeled as non-relation, even if it has at least one relation in reality. this problem is caused by the incompleteness of existing knowledge bases. for example, over 70% of people included in freebase have no place of birth @xcite . from a random sampling, we deduce that about 8.75% entity pairs in the test set of nyt-10 are misclassified as non-relation. 1 clearly, these mislabeled entity pairs yield biased evaluations and lead to inappropriate optimization for distantly supervised relation extraction. in this paper, we propose an active testing approach to estimate the performance of distantly supervised relation extraction. active testing has been proved effective in evaluating vision models with large-scale noisy datasets @xcite . in our approach, we design an iterative approach, with two stage per iteration: vetting stage and estimating stage. in the vetting stage, we adopt an active strategy to select batches of the most valuable entity pairs from the noisy test set for annotating. in the estimating stage, a metric estimator is proposed to obtain a more accurate evaluation. with a few vetting-estimating iterations, evaluation results can be dramatically close to that of human evaluation by using limited vetted data and all noisy data. experimental results demonstrate that the proposed evaluation method yields approximately unbiased estimations for distantly supervised relation extraction. ## related work distant supervision @xcite was proposed to deal with large-scale relation extraction with automatic annotations. a series of studies have been conducted with human-designed features in distantly supervised relation extraction @xcite @xcite @xcite . in recent years, neural models were widely used to extract semantic meanings accurately without hand-designed features @xcite @xcite . then, to alleviate the influence of wrongly labeled instances in distant supervision, those neural relation extractors integrated techniques such as attention mechanism @xcite @xcite , generative adversarial nets @xcite , and reinforcement learning @xcite . however, none of the above methods pay attention to the biased and inaccurate test set. though human evaluation can yield accurate evaluation results @xcite , labeling all the instances in the test set is too costly. ## task definition in distant supervision paradigm, all sentences containing the same entity pair constitute a bag. researchers train a relation extractor based on bags of sentences and then use it to predict relations of entity pairs. suppose that a distantly supervised model returns confident score 2 s @xmath0, s i2 . . . s ip } for entity pair @xmath1. . . n }, where p is the number of relations, n is the number of entity pairs, and s @xmath2, 1). y @xmath3, y i2 . . . y ip } and z @xmath4, z i2 . . . z ip } respectively represent automatic labels and true labels for entity pair i, where y ij and z ij are both in {0, 1} 3 . in widely used held-out evaluation, existing methods observe two key metrics which are precision at top k (p @k) and precision-recall curve 2 confident scores are estimated probabilities for relations. 3 an entity pair may have more than one relations. held-out evaluation replaces z with y to calculate p @k and r@k, which leads to incorrect results obviously. ## methodology in this section, we present the general framework of our method. a small random sampled set is vetted in the initial state. in each iteration there are two steps: 1) select a batch of entity pairs with a customized vetting strategy, label them manually, and add them to the vetted set; 2) use a new metric estimator to evaluate existing models by the noisy set and the vetted set jointly. after a few vetting-evaluating iterations, unbiased performance of relation extraction is appropriately evaluated. in summary, our method consists of two key components: a vetting strategy and a metric estimator.