How Fragile is Relation Extraction under Entity Replacements? (2023)

Abstract: Relation extraction (RE) aims to extract the relations between entity names from the textual context. In principle, textual context determines the ground-truth relation and the RE models should be able to correctly identify the relations reflected by the textual context. However, existing work has found that the RE models memorize the entity name patterns to make RE predictions while ignoring the textual context. This motivates us to raise the question: are RE models robust to the entity replacements? In this work, we operate the random and type-constrained entity replacements over the RE instances in TACRED and evaluate the state-of-the-art RE models under the entity replacements. We observe the 30% - 50% F1 score drops on the state-of-the-art RE models under entity replacements. These results suggest that we need more efforts to develop effective RE models robust to entity replacements. We release the source code at https://github.com/wangywUST/RobustRE.

## introduction recent literature has shown that the sentence-level relation extraction (re) models may overly rely on entity names for re instead of reasoning from the textual context @xcite . this problem is also known as entity bias: the spurious correlation between entity names and relations @xcite @xcite . this motivates us to raise a question: "how robust are re models under entity replacements?" entity bias degrades the re models' generalization, such that the entity names can mislead the models to make wrong predictions. however, a seemingly conflicting phenomenon is the high (indistribution) accuracy of re models on the standard benchmarks, such as tacred. in our work, we find that these benchmarks are prone to have shortcuts from entity names to ground-truth relations (see fig. 2 ), low entity diversity, and a large portion of incorrect entity annotations. these issues suggest that, given the presence of entity bias, the current benchmarks are not challenging enough to evaluate the generalization of re in practice. evaluating re with valid instances of more comprehensive entities is non-trivial. it requires us to collect many sentences containing comprehensive entities and carefully label the relations. both the text collection and annotations are time-consuming and expensive. instead, in our work, we aim to efficiently produce rich valid re instances with comprehensive entities based on the carefully designed entity replacements. most existing methods for evaluating the generalizability of nlp focus on sentence classification @xcite @xcite and question answering @xcite @xcite , but these methods lack special designs to seize on the entity bias in re. in this work, we propose a type-constrained and random entity replacement method: entre. type-constrained means we replace the named entity in the type @xcite or @xcite with the new entity belonging to the same type as the original entity. random means we randomly select the entity names from a wikipedia entity lexicon that consists of 24,933 organizations and 902,007 person entities for replacements. these two principles guarantee the effectiveness of entity replacement to produce valid and diverse re instances. we apply entre to tacred and evaluate the re models on the instances with replaced entity names. we analyze the re models under entity replacements in order to answer four research questions: (q1) how do the strong re models perform under entity replacements? (q2) does entre reduce prediction shortcuts from entity names to the ground-truth relations? (q3) does entre improve the entity diversity? (q4) how to improve the robustness of re? we observe several key findings. first, the strong re models luke @xcite and ire @xcite tend to memorize entityrelation patterns to infer the relation instead of reasoning based on the textual context that actually describes the relation. this phenomenon causes the model to be brittle to entity replacements, resulting in a significant performance drop of 30% -50% in terms of the f1 score. second, entre reduces the shortcuts by more than 50% on many relations, and improves the subject name diversity by more than 25 times compared to tacred. third, the recent causal inference approach core @xcite improves the robustness at a higher magnitude than other methods. for the easy use of entre, we provide a challenging re benchmark built by entre: en-tred, which consists of the tacred test set instances with the entity names replaced by en-tre. we believe the proposed entre and benchmark entred will benefit future research toward improving the re robustness. ## analysis of entity names in tacred before introducing entre, we first analyze the existing popular re datasets. our analysis is focused on the following three perspectives: 1) the correctness of entity name annotations; 2) the diversity of entity names; 3) the prediction shortcuts from entity names to the ground-truth relations. in the popular tacred @xcite , tacrev @xcite , and re-tacred (stoica et al., 2021) datasets, we find that: first, there exist some portion of incorrect entity name annotations; second, many entity names are reused more than one hundred times across instances; third, the entity names in more than 70% of the instances act as shortcuts to the ground-truth relations. we introduce the details as follows. ## entity replacement for re we present entre: a simple yet effective procedure to generate high-quality re instances with entity replacements. entre replaces entity names in the re instances in a random and typeconstrained way. we apply entre to the test set of tacred to evaluate the state-of-the-art re models' robustness under entity replacements. ## experiments in this section, we investigate entre and use it to evaluate the robustness of the strong re models luke @xcite , ire @xcite , and other methods that can improve the robustness of re. our experimental settings closely follow those of previous work @xcite @xcite to ensure a fair comparison. we organize our results and analysis as four main research questions and their answers. ## related work relation extraction (re) is a sub-task of information extraction that aims to identify semantic relations between entities from natural language text @xcite . re is the key component for building relation knowledge graphs, and it is of crucial significance to natural language processing applications such as structured search, sentiment analysis, question answering, and summarization (huang and @xcite . early research efforts @xcite @xcite train re models from scratch based on lexicon-level features. the recent re work fine-tunes pretrained language models (plms; @xcite . for ex-ample, k-adapter @xcite fixes the parameters of the plm and uses feature adapters to infuse factual and linguistic knowledge. recent work focuses on utilizing the entity information for re @xcite , but this leaks superficial and spurious clues about the relations @xcite . despite the biases in existing re models, scarce work has discussed the spurious correlation between entity mentions and relations that cause such biases. our work builds an automated pipeline to generate natural instances with fewer shortcuts and larger coverage at scale to reflect the serious effects of entity bias on the re models. there is also work in other domains aiming to evaluate models' generalization to perturbed inputs. for example, @xcite attacks reading comprehension models by adding word sequences to the input. @xcite and @xcite paraphrase the input to test models' oversensitivity. @xcite target adversarial typos. @xcite propose a benchmark for reading comprehension with diverse types of testtime perturbation. these works focus on different domains than our research does, and they do not consider the composition of re examples. little attention is drawn to the entities in the sentences, and many attacks (e.g. character swapping, word injection) may make the perturbed sentences invalid. to the best of our knowledge, this work is among the first to propose a straightforward, dedicated pipeline for generating natural adversarial examples for the re task, which takes into account the serious effects of entity bias in re models. ## conclusion our contributions in this paper are three-fold. 1) methodology-wise: we propose entre, an endto-end entity replacement method that reduces the shortcuts from entity names to ground-truth relations. 2) resource-wise: we develop entred, a straightforward method for generating natural and counterfactual entity replacements for re, which produces entred, a benchmark for auditing the generalization of re models under entity bias. 3) evaluation-wise: our experimental results and analysis provide answers to four main research questions on the generalization of re. we believe en-tred and the entity replacement method entre can benefit the community working to increase the re models' generalization under entity bias.