Revisiting D oc RED - Addressing the False Negative Problem in Relation Extraction (2022)

Abstract: The DocRED dataset is one of the most popular and widely used benchmarks for document-level relation extraction (RE). It adopts a recommend-revise annotation scheme so as to have a large-scale annotated dataset. However, we find that the annotation of DocRED is incomplete, i.e., false negative samples are prevalent. We analyze the causes and effects of the overwhelming false negative problem in the DocRED dataset. To address the shortcoming, we re-annotate 4,053 documents in the DocRED dataset by adding the missed relation triples back to the original DocRED. We name our revised DocRED dataset Re-DocRED. We conduct extensive experiments with state-of-the-art neural models on both datasets, and the experimental results show that the models trained and evaluated on our Re-DocRED achieve performance improvements of around 13 F1 points. Moreover, we conduct a comprehensive analysis to identify the potential areas for further improvement.

## introduction the field of relation extraction (re) is related to knowledge bases (kbs). most popular relation extraction datasets are constructed from knowledge triples in kbs. for example, the tacred dataset @xcite is constructed by the tac knowledge base population challenge. the nyt10 @xcite dataset matched the freebase knowledge base @xcite to the new york times corpus @xcite . wiki20 @xcite and docred @xcite originated from distant supervision from the wikidata knowledge base @xcite and wikipedia articles. by exploiting distant figure 1 : a sample document in our re-docred dataset. from the highlighted evidence , it can be inferred that "i knew you were trouble" was produced by both "max martin" and "shellback", but in the previous incomplete docred dataset, only the first producer is labeled. however, the false negative (fn) problem in the relation extraction datasets receives less attention as compared to fps. without resolving this issue, the annotations of the datasets are incomplete. recent efforts on addressing the false negative problem are from the model perspective @xcite , which aim to denoise the false negative data during training. the challenge for these approaches is that both devel-opment and test sets can be incomplete at the same time. without a completely annotated dataset, the current evaluation is ill-defined. ## revising docred with an iterative approach in this section, we describe our iterative human-inthe-loop approach to revise docred. the goal is to add the previously unrecognized relation triples to the docred dataset. our iterative approach consists of three steps in each iteration: (1) training scorer models; (2) scoring all possible triples; and (3) human verification. the following sections provide the details of each step. ## related work relation extraction there is a series of relation extraction datasets built over the past decades, and they have significantly advanced research on re. ace 2005 @xcite and semeval 2010 task 8 @xcite created two sentence-level re datasets by human annotation. however, these two datasets have a relatively small number of relation types and instances. the new york times corpus @xcite is another common relation extraction dataset used in the literature @xcite . it has been used for joint entity and relation extraction by @xcite . the large-scale ta-cred @xcite dataset is created based on the 2009-2014 tac knowledge base population (kbp) challenges and crowd-sourced human annotations. fewrel @xcite and fewrel 2.0 @xcite have been proposed to study the transferability and few-shot capability of re models. however, early relation extraction datasets mainly focus on sentence-level re, whereas many relations can only be expressed by multiple sentences. the document-level relation extraction task has been proposed to build re systems that are able to extract relations from multiple entities and sentences. @xcite have created the docred dataset by distant supervision from wikipedia articles and the wikidata knowledge base. they sampled 5,053 documents for human annotation. the annotation strategy of docred is mainly based on machine recommendation and human filtering. with a similar approach, @xcite have created a chinese document-level re dataset that focuses on hard relation cases. machine-assisted data generation since labeled data is expensive to obtain for complex nlp tasks, there are many research works on generating automatically labeled data. distant supervision is first used by @xcite to generate relation extraction data without human efforts. prior work on automatic data generation mainly relies on rulebased pattern matching @xcite and web crawling @xcite . these types of rule-based methods are susceptible to noise propagation. with the rapid development of pre-trained language models (plms; @xcite @xcite , many recent works leverage plms for automatic data generation @xcite @xcite @xcite . however, these methods typically depend on a certain set of supervised data. another line of work utilizes manually designed prompts and instructions to generate data in an unsupervised manner @xcite @xcite . although these methods improve the performance on some downstream tasks, the quality of the machine-generated data still does not match human annotation. to mitigate noise from the machine-generated data, west et al. (2021) generate a large amount of commonsense knowledge data and employ human annotators to filter the generated candidates. ## conclusions in conclusion, this paper identifies the causes and effects of the overwhelming false negative problem in relation extraction. we show that the false negative problem is the cause of the performance bottleneck on many re datasets. we have also provided a high-quality revised version of document-level re dataset, namely re-docred. moreover, we have provided comprehensive experimental analysis for re-docred and identified the potential areas for further improvement. we have also conducted a thorough error analysis on state-of-the-art re models.