Does Recommend-Revise Produce Reliable Annotations? An Analysis on Missing Instances in D oc RED (2022)

Abstract: DocRED is a widely used dataset for document-level relation extraction. In the large-scale annotation, a recommend-revise scheme is adopted to reduce the workload. Within this scheme, annotators are provided with candidate relation instances from distant supervision, and they then manually supplement and remove relational facts based on the recommendations. However, when comparing DocRED with a subset relabeled from scratch, we find that this scheme results in a considerable amount of false negative samples and an obvious bias towards popular entities and relations. Furthermore, we observe that the models trained on DocRED have low recall on our relabeled dataset and inherit the same bias in the training data. Through the analysis of annotators’ behaviors, we figure out the underlying reason for the problems above: the scheme actually discourages annotators from supplementing adequate instances in the revision phase. We appeal to future research to take into consideration the issues with the recommend-revise scheme when designing new models and annotation schemes. The relabeled dataset is released at https://github.com/AndrewZhe/Revisit-DocRED , to serve as a more reliable test set of document RE models.

## introduction relation extraction (re) is an important task which aims to identify relationships held between entities in a given piece of text. while most previous methods focus on extracting relations from a single sentence @xcite , recent studies begin to explore re at document level @xcite @xcite @xcite , which is more challenging as it often requires reasoning across multiple sentences. the rapid development of document-level re in the past two years has benefited from the proposal of docred @xcite , the first large-scale and human-annotated dataset for this task. noticeably, longer documents introduce an unprecedented difficulty in annotating the relation instances: as the total number of entities dramatically increases in accordance to text length, the expected number of entity pairs surges quadratically, intensively increasing the workload to check relationships between every pair. to address this problem, @xcite applies a recommend-revise process: in the recommendation phase, a small set of candidate relation instances is generated through distant supervision; then, annotators are required to revise the candidate set, removing the incorrect relation instances and supplementing the instances not identified in the recommendation phase. shifting the construction process from scratch to an edit-based task, it seems that the recommendrevise scheme cuts down the effort of annotating by a large margin. however, whether the quality of the annotation maintains a reliable standard in practice remains in doubt. to what extent can the accuracy of annotation be sacrificed due to the automated recommendation? and, how does the provided recommendation affect the behaviours of the annotators in the revision phase? moreover, what are the real effects on the models trained on a dataset annotated with this scheme? to answer these questions, we aim to provide a thorough comparison between careful annotations from scratch and the annotations under the recommend-revise scheme. we randomly select 96 documents from docred and ask two experts to relabel them from scratch independently. after annotating, the two experts come to a consensus of gold labels via discussion. this revised dataset is publicly available at https://github.com/ andrewzhe/revisit-docred , and we hope it can be used to evaluate the model's perfor-mance on real data distribution 1 . with the help of these annotations, we discovered three sobering issues regarding the effects of the recommend-revise scheme: to thoroughly check out the entities pair-by-pair, which is time-consuming and against the goal of this scheme. as a result, annotators can hardly make effective supplementation and would tend to perform the easier goal of validating existing relation instances. ## recommend-revise annotation scheme the major challenge for annotating document-level re datasets comes from the quadratic number of potential entity pairs with regard to the total number of entities in a document. as reported by @xcite , a document in docred contains 19.5 entities on average, thus rendering 360 entity pairs with potential relationships. therefore, for the 5,053 documents to be annotated, around 1,823,000 entity pairs are to be checked. such workload will be around 14 times more than tacred @xcite , the biggest human-labeled sentence-level re dataset. therefore, exhaustively labeling relations between each entity pair involves intensive workload and does not seem feasible for documentlevel re datasets. to alleviate the huge burden of manual labeling, @xcite divides the annotation task into two steps: recommendation and revision. first, in the recommendation phase, @xcite takes advantage of wikidata @xcite and an off-the-shelf re model to collect all the possible relations between any two entities in the same document. this process is automated and does not require human involvement. then, during the revision phase, the relations that exist in wikidata or are inferred by the re model for a specific entity pair will be shown to the annotators. rather than annotating each entity pair from scratch, the annotators are required to review the recommendations, remove the incorrect triples and supplement the missing ones. ## dataset comparison ## model bias to investigate if re models trained on such data will likewise learn the same bias, we train and select re models on the recommend-scheme-labeled dataset, d train revise and d valid revise and then assess the models' performance on the real data distribution, d scratch . the construction process of d train revise and d valid revise is the same as d revise , while the former is actually the original train set and the latter is the validation set in docred excluding the 96 documents in d revise . in those settings, we examine the performance of recent models: bilstm @xcite , gain-bert base @xcite , ssan-roberta large @xcite , atlop-roberta large @xcite and docunet-roberta large @xcite . the last three models are the most competitive ones for docred currently, while the others are shown to make sure that our analysis can generalize to models of smaller sizes. ## annotators' dilemma finally, we move on to discuss another more implicit influence of the recommend-revise scheme on the annotators' aspect. as discussed in section 4.1, while we expected the revision process to help supplement the instances left out, it turns out that an incredibly low number is added indeed. given that the annotators are trained to accomplish the revision task, we wonder why they still fail in such a uniform manner. we would like to argue that it is the nature of the revision process that puts the annotators in a dilemma, where they have to choose between a huge effort and insufficiency of supplementation. recall that there is a distinct difference in the settings of examining a labeled relationship and supplementing an unidentified relationship. for the former, annotators are required to find evidence for a recommended relation instance and remove it if there is conflicting or no evidence. this process only requires checking a single entity pair and collecting the information related to the two specific entities. however, this is not the case for supplementing a possible, unidentified relation instance, which can exist between any entity pair. there is no clear range of searching or indicating information; all they can do is to check pair-by-pair, just like what they do from scratch. this puts annotators in an awkward dilemma, especially when they understand the motivation of this scheme: if they are to be fully responsible for the missing instances at large, they will always have to complete the thorough pairwise checking one by one; however, this would make the whole process of the recommendrevise scheme meaningless in return, as it's just like a practice from scratch. the harsh requirements of supplementing push annotators to overly rely on the recommendation results and simply examine them. this is especially worth worrying about in real practice, where annotators are recruited to complete a certain number of annotations, and typically paid according to the estimated number of hours or the total number of instances they devote to the annotation @xcite . under this dilemma, it is a natural result that they are especially unmotivated to carry out the exhaustive checking for supplementation in order to get a reasonable pay in the given time. in fact, we observe an interesting phenomenon that annotators largely tend to just pick some most obvious missing instances, convince themselves that they have accomplished the supplementation, and simply move on to the next document. this can be seen in figure 5 , where we compare the distributional characteristics of the successfully supplemented instances (d revise -d recommend ) and all the missing instances in general (d scratch -d revise ). @xcite shows the accumulative statistics of the position of the head entity's first appearance in the document. we can see that the instances added by annotators in docred exhibit an extremely obvious tendency to occur earlier in the text, where more than 70% added instances are in the first 3 sentences. in contrast, all missing relation instances as a whole are almost distributed in 1 3 5 7 9 11 every part of the document uniformly. this reveals the interesting fact that humans typically tend to pick up the relations where the entities in it are mentioned earlier in the document. sub-figure (b) further compares the minimum distance between the mentions of the head and tail entities of one relation instance. we once again see the interesting fact that annotators have a strong tendency to add the "most easily identifiable" instances where the head and tail entities are quite close. specifically, the proportion of entity pairs mentioned in just one single sentence (@xmath0, but is as high as 45% for the ones chosen by annotators to be supplemented. this tells us how annotators naturally avoid burdens of reading brought by longer intervals, which possibly indicates more complicated inference with multiple sentences. from these observations, we see that there exist clear patterns among the very few instances added by human annotators. this reveals a serious fact that annotators are intentional in "pretending" to be supplementing with the least possible effort. given the consensus behavior of annotators and the very limited number of additional, it is most likely that the nature of the annotation task pushes the annotators to this embarrassing dilemma of adding and abandoning. thus, we propose a call to the nlp community that researchers should always be aware that annotation schemes, like the recommend-revise scheme, can have a direct impact on the annotation workers, affecting their willingness and behaviors, and thus have a deeper influence on the collected data. ## case study we can summarize all these problems mentioned above in the annotation with a concrete case in do- ## related works with the advance of deep learning models, the annotation sometimes becomes the bottleneck for a machine learning system. recently, analyzing the annotation quality has received increasing attention. @xcite collects and analyzes the label errors in the test sets of several popular benchmarks, showing label errors are ubiquitous and destabilize machine learning benchmarks. more specific to re task, @xcite addresses the annotation problems in tacred @xcite , a popular sentence-level re dataset. they find label errors account for 8% absolute f1 test error and more than 50% of the examples need to be relabeled. @xcite expands this research to the whole dataset, resulting in a complete re-annotated version, re-tacred, and conducts thorough analysis on the models' performance. our work differs from them in that we delve into the nature of document-level re task, and especially explore how the error is systematically introduced into the dataset through recommend-revise scheme. methodologies to solve incomplete annotations for information extraction tasks have been widely discussed in previous works. different from classification tasks, information extraction requires annotators to actively retrieve positive samples from texts, instead of just assigning a label for a given text. the problem is also attributed to the use of distant supervision @xcite where the linked kg is not perfect. some works apply general approaches like positive unlabeled learning @xcite or inference learning @xcite . task-specific models are also designed, like partial crf @xcite for ner @xcite , and novel paradigm for joint re @xcite . however, none of them examine the distribution bias in the training data, and those methods are not validated in the context of the document-level re task. prevalent effective methods on document-level re include graph-based models and transformer-based models. graph-based models like @xcite and @xcite are designed to conduct relational reasoning over the document, and transformer-based models @xcite are good at recognizing long-distance dependencies. however, all previous models treat unlabeled samples in the dataset as negative samples, and do not concern the problems in annotations. we believe our analysis and re-annotated dataset will help future work focus more on the discrepancy between the annotation and real-world distribution, instead of just overfitting the dataset. ## conclusion in this paper, we show how the recommend-revise scheme for docred can cause bias and false negative issues in the annotated data. the flaws of dataset affect the model's recall on real data and also teach the model the same bias in training data. as this scheme cannot reduce the human labor essentially without the loss of annotation quality, more efficient strategies for annotation are to be explored. on the other hand, considering that building a reliable training set for document re is extremely expensive, it is also a meaningful topic that how to alleviate the dataset shift problem @xcite by injecting appropriate inductive bias into the model's structure, instead of inheriting the bias in the training data. we believe the in-depth analysis provided in this paper can benefit future designs of document-level re models, and our scratch dataset can serve as a fairer test set. involve unexamined ethical concerns was introduced. annotators receive a competitive pay of 100 yuan per hour (more than 4 times the local minimum wage) under the approval of the institute, and both the annotation and discussion stage count towards the paid working time. annotators are required to read the acm code of ethics before annotating and report any document that violates the code. these documents are removed from the sampled documents. however, there may still be sentences or entities that are from wikipedia pages with potentially improper content. the possible adoption of such content is not a decision of the authors, and all content in the dataset does not reflect the views or stances of the authors. the resulting re-annotations from the agreement of two expert annotators form a decent approximation of the gold labels, but may still not be the ground truth due to natural error rates. further use of the dataset should be aware of the limitations and other possible issues, and we are not responsible for issues in further model training processes using our data.