On the Robustness of Document-Level Relation Extraction Models to Entity Name Variations (2024)

Abstract: Driven by the demand for cross-sentence and large-scale relation extraction, document-level relation extraction (DocRE) has attracted increasing research interest. Despite the continuous improvement in performance, we find that existing DocRE models which initially perform well may make more mistakes when merely changing the entity names in the document, hindering the generalization to novel entity names. To this end, we systematically investigate the robustness of DocRE models to entity name variations in this work. We first propose a principled pipeline to generate entity-renamed documents by replacing the original entity names with names from Wikidata. By applying the pipeline to DocRED and Re-DocRED datasets, we construct two novel benchmarks named Env-DocRED and Env-Re-DocRED for robustness evaluation. Experimental results show that both three representative DocRE models and two in-context learned large language models consistently lack sufficient robustness to entity name variations, particularly on cross-sentence relation instances and documents with more entities. Finally, we propose an entity variation robust training method which not only improves the robustness of DocRE models but also enhances their understanding and reasoning capabilities. We further verify that the basic idea of this method can be effectively transferred to in-context learning for DocRE as well.

## introduction the demand for cross-sentence and large-scale relation extraction has led to a surge of research interest in document-level relation extraction @xcite , which aims to identify the relations between each pair of entities within a document based on the document context @xcite . while covering more realistic scenarios than its sentence-level ## problem formulation given a document d which contains a set of entities @xmath0, the task of document-level relation extraction is to predict the set of all possible relations between each entity pair (e h , e t ) ∈ {(e i , e j ) | i, @xmath1, . . . , n e ; i ̸ = j} from a predefined relation type set r. the subscripts of e h and e t refer to the head and tail entity in an entity pair. an entity e i can occur multiple times in the document via n e i mentions m e @xmath2 ne i @xmath3, where the mention m i j refers to the token span of e i 's j-th occurrence in the document. ## benchmark construction in this section, we elaborate on the process of constructing benchmarks for evaluating the robustness of docre models to entity name variations. we first propose a general pipeline to generate documents with changed entity names, then apply the pipeline to docred and re-docred to create the env-docred and env-re-docred benchmarks. ## robustness evaluation and analysis in this section, utilizing the constructed benchmarks, we conduct a comprehensive evaluation and analysis on the robustness of existing docre models to entity name variations. ## entity variation robust training due to the unsatisfactory robustness of existing docre models to entity name variations, we further explore the method for enhanced robustness. intuitively, we can adopt a similar approach as the proposed pipeline to perturb each training document with a group of new entity names. the derived document naturally shares the same relation labels with the original one. also, a robust docre model should generate consistent representations and predictions for each corresponding entity pair in the original and perturbed documents. based on such motivation, we propose an entity variation robust training method (evrt) that is enhanced by data augmentation and consistency regularization. specifically, given a labeled entity pair (e h , e t ) in a document, vanilla approaches typically train the docre model with a classification objective l @xmath4, e t ), where ℓ task denotes the loss function depending on the specific model. denoting the corresponding entity pair of (e h , e t ) in the perturbed document as (e ĥ, e t), our proposed method first incorporate the classification loss l @xmath5, e t) for (e ĥ, e t) to penalize the classification errors for entity pairs in the perturbed document. then we introduce representation consistency regularization and prediction consistency regularization to encourage the model to produce consistent representations and predicted probability distributions between (e h , e t ) and (e ĥ, e t). formally, we define the representation consistency regularization loss as: and we define the prediction consistency regularization loss as: ## conclusion in this work, we systematically study the robustness of docre models to entity name vari-