Rethinking Document-Level Relation Extraction: A Reality Check (2023)

Abstract: Recently, numerous efforts have continued to push up performance boundaries of document-level relation extraction (DocRE) and have claimed significant progress in DocRE. In this paper, we do not aim at proposing a novel model for DocRE. Instead, we take a closer look at the field to see if these performance gains are actually true. By taking a comprehensive literature review and a thorough examination of popular DocRE datasets, we find that these performance gains are achieved upon a strong or even untenable assumption in common: all named entities are perfectly localized, normalized, and typed in advance. Next, we construct four types of entity mention attacks to examine the robustness of typical DocRE models by behavioral probing. We also have a close check on model usability in a more realistic setting. Our findings reveal that most of current DocRE models are vulnerable to entity mention attacks and difficult to be deployed in real-world end-user NLP applications. Our study calls more attentions for future research to stop simplifying problem setups, and to model DocRE in the wild rather than in an unrealistic Utopian world.

## introduction document-level relation extraction (docre), aiming at identifying semantic relations between a head entity and a tail entity in a document @xcite , plays an essential role in a variety of downstream applications, such as question answering @xcite and knowledge base construction @xcite . recently, there are two flourishing branches for docre. first, graph-based approaches consider entities @xcite , mentions @xcite and sentences @xcite as nodes to construct a document-level graph and perform reasoning through some advanced neural graph techniques. second, sequence-based approaches leverage bilstm @xcite or transformers @xcite @xcite ) as encoders to learn document-level representations. however, all these models have one thing in common that they are based on a strong or even untenable assumption as shown in figure 1 : all entity mentions are (i) correctly localized; (ii) perfectly normalized; (iii) correctly typed. 1 then, the task of modeling docre is usually simplified as a pairwise classification problem. although these pairwise classification approaches have claimed significant progress in docre performance, we are still interested in taking a closer look at the field to see if this is actually true. in particular, many research papers have reported very decent leaderboard scores for the docre task. does this mean the task of docre has been almost completely solved? can the current approaches be widely used in real-world docre scenarios? to answer these questions, we first take a closer look at data annotations in commonly-used docre datasets to check the strong data assumption ( §3). we focus specifically on the annotations of named entity recognition (ner) and normalization (i.e., entity linking) in detecting relations. by answering three research questions (rq1-3), we find that current problem setups for docre are greatly simplified and unrealistic. if the data assumption is too strict, it is not clear whether current docre models are robust in a variety of loose assumptions. therefore, we construct four types of attacks regarding entity mention annotations to investigate the model robustness ( §4, rq4) using behavioral probing @xcite . to further have a look at the limitations of data assumptions, it is important to investigate the usability of existing docre models in real-world scenarios. hence, we examine the capability of widely-used ner systems and entity linking systems on preparing model input formats from raw text for docre model deployment ( §5, rq5). finally, we discuss our empirical findings and call special attentions for future research in developing docre models ( §6). in short, our contributions and findings are: ## a quick literature review in this section, we have a quick literature review of docre models to shed light on a global review for recent evolutions. ## check on dataset annotations to provide in-depth observations of the data assumption in most of docre models, we first take a thorough examination of data annotations on three commonly-used docre datasets. we will conduct quantitative and qualitative studies to analyze entity mentions and entity aliases which a relation instance involved. foot_1 ## check on model robustness most of existing docre models are proposed based on the strong assumptions of mention annotations as shown in section § 3. in this section, we are interested in the following research question: to answer rq4, we adopt behavioral probing @xcite to observe a model's behaviors by studying the model's predictions on attacking datasets. that is, attacks are only added at test time and are not available during model training. ## check on model usability in this section, we investigate this realistic situation: docre models are already trained and training data is unavailable. we want to extract same relations on unseen raw text using these models. the goal is to deploy the already-trained docre models in other nlp applications. here, we are interested in the following research question: to answer rq5, a necessary step is that whether we can process the raw text with the format as docre models trained on. this preprocessing procedure involves two crucial systems: named entity recognition (ner) and entity linking. ## discussion let's stop simplifying problem setups. as summarized in let's model docre in the wild. as shown in section § 5, it is very difficult to produce accurate data formats as existing docre models trained on. thus, given a new document, we are still unable to easily deploy existing trained docre models to extract same types of relations, let alone unseen relations. recently, some studies @xcite @xcite have started exploring the direction of jointly extracting entities and relations at document level. however, the end-to-end performance at document level is much worse than the performance at sentence level. our empirical findings call more atten-tions on developing high-performance end-to-end docre models and more attentions on modeling docre in the wild, rather than in an unrealistic utopian world. ## conclusion in this paper, we try to answer whether the performance gains recent docre models claimed are actually true. we took a comprehensive literature review of docre models and a thorough examination of popular docre datasets. we investigated the model robustness under four types of mention attacks and the model usability under a more realistic setting. our findings call future efforts on modeling docre in the wild.