Should We Rely on Entity Mentions for Relation Extraction? Debiasing Relation Extraction with Counterfactual Analysis (2022)

Abstract: Recent literature focuses on utilizing the entity information in the sentence-level relation extraction (RE), but this risks leaking superficial and spurious clues of relations. As a result, RE still suffers from unintended entity bias, i.e., the spurious correlation between entity mentions (names) and relations. Entity bias can mislead the RE models to extract the relations that do not exist in the text. To combat this issue, some previous work masks the entity mentions to prevent the RE models from over-fitting entity mentions. However, this strategy degrades the RE performance because it loses the semantic information of entities. In this paper, we propose the CoRE (Counterfactual Analysis based Relation Extraction) debiasing method that guides the RE models to focus on the main effects of textual context without losing the entity information. We first construct a causal graph for RE, which models the dependencies between variables in RE models. Then, we propose to conduct counterfactual analysis on our causal graph to distill and mitigate the entity bias, that captures the causal effects of specific entity mentions in each instance. Note that our CoRE method is model-agnostic to debias existing RE systems during inference without changing their training processes. Extensive experimental results demonstrate that our CoRE yields significant gains on both effectiveness and generalization for RE. The source code is provided at: https://github.com/vanoracai/CoRE .

## introduction sentence-level relation extraction (re) is an important step to obtain a structural perception of unstructured text @xcite by extracting relations between entity mentions (names) from the textual context. from human oracle, textual context should be the main source of information that determines the ground-truth relations between entities. consider a sentence " ::::: mary gave birth to jerry." foot_0 . even if we change the entity mentions from 'jerry' and 'mary' to other people's names, the relation 'parents' still holds between the subject and object as described by the textual context "gave birth to". recently, some work aims to utilize entity mentions for re @xcite , which, however, leak superficial and spurious clues about the relations @xcite . in our work, we observe that entity information can lead to biased relation prediction by misleading re models to extract relations that do not exist in the text. fig. 1 visualizes a relation prediction from a state-of-the-art re model @xcite (see more examples in tab. 7). although the context describes no relation between the highlighted entity pair, the model extracts the relation as "countries of residence". such an erroneous result can come from the spurious correlation between entity mentions and relations, or the entity bias in short. for example, if the model sees the relation "countries of residence" many more times than other relations when the object entity is switzerland during training, the model can associate this relation with switzerland during inference even though the relation does not exist in the text. to combat this issue, some work @xcite proposes masking entities to prevent the re models from over-fitting entity mentions. on the other hand, some other work @xcite finds that this strategy degrades the performance of re because it loses the semantic information of entities. for both machines and humans, re requires a combined understanding of textual context and entity mentions @xcite . humans can avoid the entity bias and make unbiased decisions by correctly referring to the textual context that describes the relation. the underlying mechanism is @xcite . in the input sentence, the subject is in blue and the object is in yellow. the ground-truth relation between the subject and object is "no relation", since there is not any relation reflected by the textual context. (right) our proposed counterfactual analysis for re, which compares the original prediction (upper) with the counterfactual one (lower) to mitigate the entity bias. the counterfactual analysis essentially gifts humans the hypothesizing abilities to make decisions collectively based on the textual context and entity mentions, as well as to introspect whether the decision is deceived (see fig. 1 ). specifically, we are essentially comparing the original instance with a counterfactual instance, where only the textual context is wiped out, while keeping the entity mentions untouched. by doing so, we can focus on the main effects of the textual context without losing the entity information. in our work, we propose a novel modelagnostic paradigm for debiasing re, namely core (counterfactual analysis based relation extraction), which adopts the counterfactual analysis to mitigate the spurious influence of the entity mentions. specifically, core does not touch the training of re models, i.e., it allows a model to be exposed to biases on the original training set. then, we construct a causal graph for re to analyze the dependencies between variables in re models, which acts as a "roadmap" for capturing the causal effects of textual context and entity mentions. to rectify the test instances from the potentially biased prediction, in inference, core "imagines" the counterfactual counterparts on our causal graph to distill the biases. last but not least, core performs a bias mitigation operation with adaptive weights to produce a debiased decision for re. we highlight that core is a flexible debiasing method that is applicable to popular re models without changing their training processes. to evaluate the effectiveness of core, we perform extensive experiments on public benchmark datasets. the results demonstrate that our proposed method can significantly improve the effectiveness and generalization of the popular re models by mitigating the biases in an entity-aware manner. ## experiments in this section, we evaluate the performance of our core methods when applied to re models. we compare our methods against a variety of strong baselines on the task of sentence-level re. our experimental settings closely follow those of the previous work @xcite @xcite to ensure a fair comparison.