Original: Should We Rely on Entity Mentions for Relation Extraction? Debiasing Relation Extraction with Counterfactual Analysis
Revised: Should We Rely on Entity Mentions for Relation Extraction? Debiasing Relation Extraction with Counterfactual Analysis

Original: Anonymous ACL submission

Original: Abstract
Revised: Abstract

Original: Recent literature focuses on utilizing the entityinformation in the sentence-level relation extraction (RE), but this risks leaking superficialand spurious clues of relations.
Revised: Recent literature focuses on utilizing the entity information in the sentence-level relation extraction (RE), but this risks leaking superficial and spurious clues of relations.

Original: As a result, REstill suffers from unintended entity bias , i.e.,the spurious correlation between entity mentions (names) and relations.
Revised: As a result, RE still suffers from unintended entity bias , i.e., the spurious correlation between entity mentions (names) and relations.

Original: Entity bias canmislead the RE models to extract the relationsthat do not exist in the text.
Revised: Entity bias can mislead the RE models to extract the relations that do not exist in the text.

Original: To combat thisissue, some previous work masks the entitymentions to prevent the RE models from overfitting entity mentions.
Revised: To combat this issue, some previous work masks the entity mentions to prevent the RE models from overfitting entity mentions.

Original: However, this strategydegrades the RE performance because it losesthe semantic information of entities.
Revised: However, this strategy degrades the RE performance because it loses the semantic information of entities.

Original: In thispaper, we propose the C O RE ( Counterfactual
Revised: In this paper, we propose the C O RE ( Counterfactual

Original: Analysis based Relation Extraction ) debiasing method that guides the RE models to focuson the main effects of textual context withoutlosing the entity information.
Revised: Analysis based Relation Extraction ) debiasing method that guides the RE models to focus on the main effects of textual context without losing the entity information.

Original: We first constructa causal graph for RE, which models the dependencies between variables in RE models.
Revised: We first construct a causal graph for RE, which models the dependencies between variables in RE models.

Original: Then,we propose to conduct counterfactual analysison our causal graph to distill and mitigate theentity bias, that captures the causal effects ofspecific entity mentions in each instance.
Revised: Then, we propose to conduct counterfactual analysis on our causal graph to distill and mitigate the entity bias, that captures the causal effects of specific entity mentions in each instance.

Original: Notethat our C O RE method is model-agnostic todebias existing RE systems during inferencewithout changing their training processes.
Revised: Note that our C O RE method is model-agnostic to debias existing RE systems during inference without changing their training processes.

Original: Extensive experimental results demonstrate thatour C O RE yields significant gains on both effectiveness and generalization for RE.
Revised: Extensive experimental results demonstrate that our C O RE yields significant gains on both effectiveness and generalization for RE.

Revised: The source code is provided at: https://github.com/vanoracai/CoRE .

Original:  Introduction
Revised:  Introduction

Original: Sentence-level relation extraction (RE) is an important step to obtain a structural perception of unstructured text (Distiawan et al., 2019) by extractingrelations between entity mentions (names) fromthe textual context .
Revised: Sentence-level relation extraction (RE) is an important step to obtain a structural perception of unstructured text (Distiawan et al., 2019) by extracting relations between entity mentions (names) from the textual context .

Original: From human oracle, textualcontext should be the main source of informationthat determines the ground-truth relations betweenentities.
Revised: From human oracle, textual context should be the main source of information that determines the ground-truth relations between entities.

Original: Consider a sentence “ (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Mary gave birth to
Revised: Consider a sentence “ (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Mary gave birth to

Original: Jerry. ”
Revised: Jerry. ”

Original: Even if we change the entity mentions from ‘Jerry’ and ‘Mary’ to other people’s names, the relation ‘parents’ still holds between the subject and object as described by the textual context “gave birth to” .
Revised: Even if we change the entity mentions from ‘Jerry’ and ‘Mary’ to other people’s names, the relation ‘parents’ still holds between the subject and object as described by the textual context “gave birth to” .

Original: Recently, some work aims to utilize entity mentions for RE (Yamada et al., 2020;
Revised: Recently, some work aims to utilize entity mentions for RE (Yamada et al., 2020;

Original: Zhou and Chen, 2021), which, however, leak superficial and spurious clues about the relations (Zhang et al., 2018).
Revised: Zhou and Chen, 2021), which, however, leak superficial and spurious clues about the relations (Zhang et al., 2018).

Original: In our work, we observe that entity information can lead to biased relation prediction by misleading RE models to extract relations that do not exist in the text.
Revised: In our work, we observe that entity information can lead to biased relation prediction by misleading RE models to extract relations that do not exist in the text.

Original: Fig.
Revised: Fig.

Original: 1 visualizes a relation prediction from a state-of-the-art RE model (Alt et al., 2020)
Revised: 1 visualizes a relation prediction from a state-of-the-art RE model (Alt et al., 2020)

Original: (see more examples in Tab. 6).
Revised: (see more examples in Tab. 7).

Original: Although the context describes no relation between the highlighted entity pair, the model extracts the relation as “ countries of residence ”.
Revised: Although the context describes no relation between the highlighted entity pair, the model extracts the relation as “ countries of residence ”.

Original: Such an erroneous result can come from the spurious correlation between entity mentions and relations, or the entity bias in short.
Revised: Such an erroneous result can come from the spurious correlation between entity mentions and relations, or the entity bias in short.

Original: For example, if the model sees the relation “ countries of residence ” many more times than other relations when the object entity is Switzerland during training, the model can associate this relation with Switzerland during inference even though the relation does not exist in the text.
Revised: For example, if the model sees the relation “ countries of residence ” many more times than other relations when the object entity is Switzerland during training, the model can associate this relation with Switzerland during inference even though the relation does not exist in the text.

Original: To combat this issue, some work (Zhang et al., 2017, 2018) proposes masking entities to prevent the RE models from over-fitting entity mentions.
Revised: To combat this issue, some work (Zhang et al., 2017, 2018) proposes masking entities to prevent the RE models from over-fitting entity mentions.

Original: On the other hand, some other work (Peng et al., 2020; Zhou and Chen, 2021) finds that this strategy degrades the performance of RE because it loses the semantic information of entities.
Revised: On the other hand, some other work (Peng et al., 2020; Zhou and Chen, 2021) finds that this strategy degrades the performance of RE because it loses the semantic information of entities.

Original: For both machines and humans, RE requires a combined understanding of textual context and entity mentions (Peng et al., 2020).
Revised: For both machines and humans, RE requires a combined understanding of textual context and entity mentions (Peng et al., 2020).

Original: Humans can avoid the entity bias and make unbiased decisions by correctly referring to the textual context that describes the relation.
Revised: Humans can avoid the entity bias and make unbiased decisions by correctly referring to the textual context that describes the relation.

Original: The underlying mechanism is causality-based (Van Hoeck et al., 2015): humansidentify the relations by pursing the main causal effect of the textual context instead of the side-effectof entity mentions.
Revised: The underlying mechanism is causality-based (Van Hoeck et al., 2015): humans identify the relations by pursuing the main causal effect of the textual context instead of the sideeffect of entity mentions.

Original: In contrast, RE models areusually likelihood-based : the prediction is analogous to looking up the entity mentions and textualcontext in a huge likelihood table, interpolated bytraining (Tang et al., 2020).
Revised: In contrast, RE models are usually likelihood-based : the prediction is analogous to looking up the entity mentions and textual context in a huge likelihood table, interpolated by training (Tang et al., 2020).

Original: In this paper, our ideais to teach RE models to distinguish between theeffects from the textual context and entity mentionsthrough counterfactual analysis (Pearl, 2018):
Revised: In this paper, our idea is to teach RE models to distinguish between the effects from the textual context and entity mentions through counterfactual analysis (Pearl, 2018):

Original: Counterfactual analysis: If I had not seen the textual context, would I still extract the same relation?
Revised: Counterfactual analysis: If I had not seen the textual context, would I still extract the same relation?

Original: The counterfactual analysis essentially gifts humans the hypothesizing abilities to make decisionscollectively based on the textual context and entity mentions, as well as to introspect whether thedecision is deceived (see Fig. 1).
Revised: The counterfactual analysis essentially gifts humans the hypothesizing abilities to make decisions collectively based on the textual context and entity mentions, as well as to introspect whether the decision is deceived (see Fig. 1).

Original: Specifically, weare essentially comparing the original instance witha counterfactual instance, where only the textualcontext is wiped out, while keeping the entity mentions untouched.
Revised: Specifically, we are essentially comparing the original instance with a counterfactual instance, where only the textual context is wiped out, while keeping the entity mentions untouched.

Original: By doing so, we can focus on themain effects of the textual context without losingthe entity information.
Revised: By doing so, we can focus on the main effects of the textual context without losing the entity information.

Original: In our work, we propose a novel model-agnosticparadigm for debiasing RE, called C O RE (counterfactual analysis based Relation Extraction), whichadopts the counterfactual analysis to mitigate thespurious influence of the entity mentions.
Revised: In our work, we propose a novel modelagnostic paradigm for debiasing RE, namely C O RE (Counterfactual analysis based Relation Extraction), which adopts the counterfactual analysis to mitigate the spurious influence of the entity mentions.

Original: Specifically, C O RE does not touch the training of REmodels, i.e., it allows a model to be exposed to biases on the original training set.
Revised: Specifically, C O RE does not touch the training of RE models, i.e., it allows a model to be exposed to biases on the original training set.

Original: Then, we constructa causal graph for RE to analyze the dependenciesbetween variables in RE models, which acts as a“roadmap” for capturing the causal effects of textual context and entity mentions.
Revised: Then, we construct a causal graph for RE to analyze the dependencies between variables in RE models, which acts as a “roadmap” for capturing the causal effects of textual context and entity mentions.

Original: To rectify the testinstances from the potentially biased prediction, in inference, C O RE “imagines” the counterfactual counterparts on our causal graph to distill the biases.
Revised: To rectify the test instances from the potentially biased prediction, in inference, C O RE “imagines” the counterfactual counterparts on our causal graph to distill the biases.

Original: Last but not least, C O RE performs a bias mitigation operation with adaptive weights to produce a debiased decision for RE.
Revised: Last but not least, C O RE performs a bias mitigation operation with adaptive weights to produce a debiased decision for RE.

Original: We highlight that C O RE is a flexible debiasing method that is applicable to popular RE models without changing their training processes.
Revised: We highlight that C O RE is a flexible debiasing method that is applicable to popular RE models without changing their training processes.

Original: To evaluate the effectiveness of C O RE, we perform extensive experiments on public benchmark datasets.
Revised: To evaluate the effectiveness of C O RE, we perform extensive experiments on public benchmark datasets.

Original: The results demonstrate that our proposed method can significantly improve the effectiveness and generalization of the popular RE models by mitigating the biases in an entity-aware manner.
Revised: The results demonstrate that our proposed method can significantly improve the effectiveness and generalization of the popular RE models by mitigating the biases in an entity-aware manner.

Original:  Related Work
Revised:  Related Work

Original: Sentence-level relation extraction.
Revised: Sentence-level relation extraction.

Original: Early research efforts (Nguyen and Grishman, 2015; Wang et al., 2016; Zhang et al., 2017) train RE models from scratch based on lexicon-level features.
Revised: Early research efforts (Nguyen and Grishman, 2015; Wang et al., 2016; Zhang et al., 2017) train RE models from scratch based on lexicon-level features.

Original: The recent
Revised: The recent

Original: RE work fine-tunes pretrained language models (PLMs; Devlin et al. 2019;
Revised: RE work fine-tunes pretrained language models (PLMs; Devlin et al. 2019;

Original: Liu et al. 2019).
Revised: Liu et al. 2019).

Original: For example, K-Adapter (Wang et al., 2020) fixes the parameters of the PLM and uses feature adapters to infuse factual and linguistic knowledge.
Revised: For example, K-Adapter (Wang et al., 2020) fixes the parameters of the PLM and uses feature adapters to infuse factual and linguistic knowledge.

Original: Recent work focuses on utilizing the entity information for RE (Zhou and Chen, 2021; Yamada et al., 2020), but this leaks superficial and spurious clues about the relations (Zhang et al., 2018).
Revised: Recent work focuses on utilizing the entity information for RE (Zhou and Chen, 2021; Yamada et al., 2020), but this leaks superficial and spurious clues about the relations (Zhang et al., 2018).

Original: Despite the biases in existing RE models, scarce work has discussed the spurious correlation between entity mentions and relations that causes such biases.
Revised: Despite the biases in existing RE models, scarce work has discussed the spurious correlation between entity mentions and relations that causes such biases.

Original: Our work investigates this issue and proposes C O RE to debias RE models for higher effectiveness.
Revised: Our work investigates this issue and proposes C O RE to debias RE models for higher effectiveness.

Original: Debiasing for Natural Language Processing.
Revised: Debiasing for Natural Language Processing.

Original: Debiasing is a fundamental problem in machine learning (Torralba and Efros, 2011).
Revised: Debiasing is a fundamental problem in machine learning (Torralba and Efros, 2011).

Original: For natural language processing (NLP), some work performs datare-sampling to prevent models from capturing theunintended bias in training (Dixon et al., 2018;
Revised: For natural language processing (NLP), some work performs data re-sampling to prevent models from capturing the unintended bias in training (Dixon et al., 2018;

Original: Geng et al., 2007; Kang et al., 2016;
Revised: Geng et al., 2007; Kang et al., 2016;

Original: Rayhan et al. ,2017; Nguyen et al., 2011).
Revised: Rayhan et al., 2017; Nguyen et al., 2011).

Original: Alternatively, Wei and
Revised: Alternatively, Wei and

Original: Zou (2019) and Qian et al.
Revised: Zou (2019) and Qian et al.

Original: (2020) develop dataaugmentation for debiasing.
Revised: (2020) develop data augmentation for debiasing.

Original: Some recent work debiases the NLP models based on causal inference(Qian et al., 2021; Nan et al., 2021).
Revised: Some recent work debiases the NLP models based on causal inference (Qian et al., 2021; Nan et al., 2021).

Original: In RE, how todeal with the entity bias is also an important problem.
Revised: In RE, how to deal with the entity bias is also an important problem.

Original: For example, PA-LSTM (Zhang et al., 2017)masks the entity mentions with special tokens toprevent RE models from over-fitting entity names,which was also adopted by C-GCN (Zhang et al. and SpanBERT (Joshi et al., 2020).
Revised: For example, PA-LSTM (Zhang et al., 2017) masks the entity mentions with special tokens to prevent RE models from over-fitting entity names, which was also adopted by C-GCN (Zhang et al., 2018) and SpanBERT (Joshi et al., 2020).

Original: However,masking entities loses the semantic information ofentities and leads to performance degradation.
Revised: However, masking entities loses the semantic information of entities and leads to performance degradation.

Original: Different from it, our C O RE model tackles entity biases based on structured causal models.
Revised: Different from it, our C O RE model tackles entity biases based on structured causal models.

Original: In this way,we debias the RE models to focus on the textualcontext without losing the entity information.
Revised: In this way, we debias the RE models to focus on the textual context without losing the entity information.

Original:  Methodology
Revised:  Methodology

Original: Sentence-level relation extraction (RE) aims to extract the relation between a pair of entities mentioned from a sentence.
Revised: Sentence-level relation extraction (RE) aims to extract the relation between a pair of entities mentioned from a sentence.

Original: We propose C O RE (counterfactual analysis based Relation Extraction) asa model-agnostic technique to endow existing REmodels with unbiased decisions during inference.
Revised: We propose C O RE (counterfactual analysis based Relation Extraction) as a model-agnostic technique to endow existing RE models with unbiased decisions during inference.

Original: C O RE follows the regular training process of existing work regardless of the bias from the entitymentions.
Revised: C O RE follows the regular training process of existing work regardless of the bias from the entity mentions.

Original: During inference, C O RE post-adjuststhe biased prediction according to the effects of thebias.
Revised: During inference, C O RE post-adjusts the biased prediction according to the effects of the bias.

Original: C O RE can be flexibly incorporated into popular RE models to improve their effectiveness andgeneralization based on the counterfactual analysiswithout re-training the model.
Revised: C O RE can be flexibly incorporated into popular RE models to improve their effectiveness and generalization based on the counterfactual analysis without re-training the model.

Original: In this section, we first formulate the existing
Revised: In this section, we first formulate the existing

Original: RE models in the form of a causal graph.
Revised: RE models in the form of a causal graph.

Original: Then, weintroduce our proposed bias distillation method todistill the entity bias with our designed counterfactual analysis.
Revised: Then, we introduce our proposed bias distillation method to distill the entity bias with our designed counterfactual analysis.

Original: We conduct an empirical analysis to analyze how heavily the existing RE models rely on the entity mentions to make decisions.
Revised: We conduct an empirical analysis to analyze how heavily the existing RE models rely on the entity mentions to make decisions.

Original: Finally, we mitigate the distilled bias from the predictions of RE models to improve their effectiveness.
Revised: Finally, we mitigate the distilled bias from the predictions of RE models to improve their effectiveness.

Original: 3. Causality of Relation Extraction
Revised: 3. Causality of Relation Extraction

Original: In order to perform causal intervention, we first formulate the causal graph (Pearl et al., 2016; Pearl and Mackenzie, 2018), a.k.a., structural causal model, for the RE models as Fig. 2, which sheds light on how the textual context and entity mentions affect the RE predictions.
Revised: In order to perform causal intervention, we first formulate the causal graph (Pearl et al., 2016; Pearl and Mackenzie, 2018), a.k.a., structural causal model, for the RE models as Fig. 2, which sheds light on how the textual context and entity mentions affect the RE predictions.

Original: The causal graph is a directed acyclic graph G =
Revised: The causal graph is a directed acyclic graph G =

Original: {V , E} , indicating how a set of variables V interact with each other through the causal relations behind the data and how variables obtain their values, e.g., ( E, X )
Revised: {V , E} , indicating how a set of variables V interact with each other through the causal relations behind the data and how variables obtain their values, e.g., ( E, X )

Original: → Y in
Revised: → Y in

Original: Fig.
Revised: Fig.

Original: Before we conduct counterfactual analysis that deliberately manipulates the values of nodes and prunes the causal graph, we first revisit the conventional RE systems in the graphical view.
Revised: Before we conduct counterfactual analysis that deliberately manipulates the values of nodes and prunes the causal graph, we first revisit the conventional RE systems in the graphical view.

Original: The causal graph in Fig.
Revised: The causal graph in Fig.

Original: 2 is applicable to a variety of RE models and imposes no constraints on the detailed implementations.
Revised: 2 is applicable to a variety of RE models and imposes no constraints on the detailed implementations.

Original: Node X is the input text.
Revised: Node X is the input text.

Original: On the edge X → E , we obtain the spans of subject and object entities as node E through
Revised: On the edge X → E , we obtain the spans of subject and object entities as node E through

Original: NER or human annotations (Zhang et al., 2017).
Revised: NER or human annotations (Zhang et al., 2017).

Original: For example, in the aforementioned sentence X = “ (cid:58)(cid:58)(cid:58)(cid:58)
Revised: For example, in the aforementioned sentence X = “ [Equation]

Original: Mary gave birth to Jerry. ”, the entities are E =
Revised: Mary gave birth to Jerry. ” , the entities are E =

Original: [ ’Mary’, ’Jerry’ ].
Revised: [ ’Mary’, ’Jerry’ ].

Original: On the edges ( X, E ) → Y , existing RE models take different designs.
Revised: On the edges ( X, E ) → Y , existing RE models take different designs.

Original: For example, CGCN (Zhang et al., 2018) obtains the relation prediction Y by encoding entity mentions E on the pruned dependency tree of X using a graph convolutional network.
Revised: For example, CGCN (Zhang et al., 2018) obtains the relation prediction Y by encoding entity mentions E on the pruned dependency tree of X using a graph convolutional network.

Original: IRE (Zhou and Chen, 2021) uses
Revised: IRE (Zhou and Chen, 2021) uses

Original: PLMs as the encoder for X , and marks the entity information of E with special tokens to utilize the entity information.
Revised: PLMs as the encoder for X , and marks the entity information of E with special tokens to utilize the entity information.

Original: 3. Bias Distillation
Revised: 3. Bias Distillation

Original: Based on our causal graph in Fig.
Revised: Based on our causal graph in Fig.

Original: 2, we diagnose how the entity bias affects inference.
Revised: 2, we diagnose how the entity bias affects inference.

Original: After training, the causal dependencies among the variables are learned in terms of the model parameters.
Revised: After training, the causal dependencies among the variables are learned in terms of the model parameters.

Original: The entity bias can mislead the models to make wrong predictions while ignoring the actual relation-describing textual context in X , i.e., biased towards the causal dependency: E → Y .
Revised: The entity bias can mislead the models to make wrong predictions while ignoring the actual relation-describing textual context in X , i.e., biased towards the causal dependency: E → Y .

Original: The conventional biased prediction can only see the output Y of the entire graph given a sentence
Revised: The conventional biased prediction can only see the output Y of the entire graph given a sentence

Original: X , ignoring how specific entity mentions affect therelation prediction.
Revised: X , ignoring how specific entity mentions affect the relation prediction.

Original: However, causal inference encourages us to think out of the black box.
Revised: However, causal inference encourages us to think out of the black box.

Original: From thegraphical point of view, we are no longer requiredto execute the entire causal graph as a whole.
Revised: From the graphical point of view, we are no longer required to execute the entire causal graph as a whole.

Original: Incontrast, we can directly manipulate the nodes andobserve the output.
Revised: In contrast, we can directly manipulate the nodes and observe the output.

Original: The above operation is termedintervention in causal inference, which we denoteas do ( · ) .
Revised: The above operation is termed intervention in causal inference, which we denote as do ( · ) .

Original: It wipes out all the incoming links of anode and demands it to take a certain value.
Revised: It wipes out all the incoming links of a node and demands it to take a certain value.

Original: We distill the entity bias by intervention and itsinduced counterfactual .
Revised: We distill the entity bias by intervention and its induced counterfactual .

Original: The counterfactual means“counter to the facts”, and takes one step that furtherassigns the hypothetical combination of values tovariables.
Revised: The counterfactual means “counter to the facts”, and takes one step that further assigns the hypothetical combination of values to variables.

Original: For example, we can remove the inputtextual context by masking X , but maintain E asthe original entity mentions, as if X still exists.
Revised: For example, we can remove the input textual context by masking X , but maintain E as the original entity mentions, as if X still exists.

Original: We will use the input text X as our control variable where the intervention is conducted, aiming toassess its effects, due to the fact that there wouldnot be any valid relation between entities in E ifthe input text X is empty.
Revised: We will use the input text X as our control variable where the intervention is conducted, aiming to assess its effects, due to the fact that there would not be any valid relation between entities in E if the input text X is empty.

Original: We denote the outputlogits Y after the intervention X
Revised: We denote the output logits Y after the intervention X

Original: =
Revised: =

Original: ¯ x as follows:
Revised: ¯ x as follows:

Original: [Equation]
Revised: [Equation]

Original: Following the above notation, the original prediction Y , i.e., can be re-written as Y x .
Revised: Following the above notation, the original prediction Y , i.e., can be re-written as Y x .

Original: To distill the entity bias, we conduct the intervention do ( X = ¯ x ) on X , while keeping the variable
Revised: To distill the entity bias, we conduct the intervention do ( X = ¯ x ) on X , while keeping the variable

Original: E as the original e , as if the original input text xhad existed.
Revised: E as the original e , as if the original input text x had existed.

Original: Specifically, we mask the tokens inx to produce ¯ x but keep the entity mentions e asoriginal, so that the textual context is removed andthe entity information is maintained.
Revised: Specifically, we mask the tokens in x to produce ¯ x but keep the entity mentions e as original, so that the textual context is removed and the entity information is maintained.

Original: Accordingly,the counterfactual prediction is denoted as Y ¯ x,e (see
Revised: Accordingly, the counterfactual prediction is denoted as Y ¯ x,e (see

Original: Fig. 3).
Revised: Fig. 3).

Original: In this case, since the model cannot see anytextual context in the factual input x after the intervention ¯ x , but still has access to the original entitymentions e as the inputs, the prediction Y ¯ x,e purelyreflects the influence from e .
Revised: In this case, since the model cannot see any textual context in the factual input x after the intervention ¯ x , but still has access to the original entity mentions e as the inputs, the prediction Y ¯ x,e purely reflects the influence from e .

Original: In other words, Y ¯ x,erefers to the output, i.e., a probability distribution
Revised: In other words, Y ¯ x,e refers to the output, i.e., a probability distribution

Original: [Figure]
Revised: [Figure]

Original: or a logit vector, where only the entity mentions are given as the input without textual context.
Revised: or a logit vector, where only the entity mentions are given as the input without textual context.

Original: To investigate how heavily the state-of-the-art models rely on the entity mentions for RE, we conduct an empirical study to compare the original prediction Y x and the counterfactual one Y ¯ x,e .
Revised: To investigate how heavily the state-of-the-art models rely on the entity mentions for RE, we conduct an empirical study to compare the original prediction Y x and the counterfactual one Y ¯ x,e .

Original: Specifically, we calculate the fraction of the test instances (y-axis) that have the original relation prediction arg max c Y x [ c ] ranked in the top k most confident relations of the counterfactual prediction
Revised: Specifically, we calculate the fraction of the test instances (y-axis) that have the original relation prediction arg max c Y x [ c ] ranked in the top k most confident relations of the counterfactual prediction

Original: Y ¯ x,e .
Revised: Y ¯ x,e .

Original: This fraction is termed as Hit @ k .
Revised: This fraction is termed as Hit @ k .

Original: We present Hit @ k for IRE RoBERTa
Revised: We present Hit @ k for IRE RoBERTa

Original: (Zhou and
Revised: (Zhou and

Original: Chen, 2021), a state-of-the-art RE model, in Fig. on the test instances when the original relation prediction is title , employee of , or origin .
Revised: Chen, 2021), a state-of-the-art RE model, in Fig.on the test instances when the original relation prediction is title , employee of , or origin .

Original: Higher
Revised: Higher

Original: Hit @1 means that for more instances, the model infers the same relation given only the entity mentions no matter whether the textual context is given, which imply stronger causal effects from the entity mentions
Revised: Hit @1 means that for more instances, the model infers the same relation given only the entity mentions no matter whether the textual context is given, which imply stronger causal effects from the entity mentions

Original: Y ¯ x,e , i.e., the models rely more heavily on the entity mentions for RE.
Revised: Y ¯ x,e , i.e., the models rely more heavily on the entity mentions for RE.

Original: We observe that when k = 1 , the Hit @1 is more than 50%, which implies that the model typically extracts the same relations even without textual context on more than a half of the instances.
Revised: We observe that when k = 1 , the Hit @1 is more than 50%, which implies that the model typically extracts the same relations even without textual context on more than a half of the instances.

Original: For a larger k , the Hit @ k increases significantly and reaches more than 80% for k ≥ 2 .
Revised: For a larger k , the Hit @ k increases significantly and reaches more than 80% for k ≥ 2 .

Original: These observations imply a promising but embarrassing result: the state-of-the-art model relies on the entity bias for RE on many instances.
Revised: These observations imply a promising but embarrassing result: the state-of-the-art model relies on the entity bias for RE on many instances.

Original: The entity bias reflected by Y ¯ x,e can lead to the wrong extraction if the relation implied by the entity mentions does not exist in the input text.
Revised: The entity bias reflected by Y ¯ x,e can lead to the wrong extraction if the relation implied by the entity mentions does not exist in the input text.

Original: This poses a challenge to the generalization of RE models, as validated by ourexperimental results (§4.3).
Revised: This poses a challenge to the generalization of RE models, as validated by our experimental results (§4.3).

Original: In addition to Y ¯ x,e that reflects the causal effectsof entity mentions, there is another kind of bias notconditioned on the entity mentions e , but reflectingthe general bias in the whole dataset, which is Y ¯ x .
Revised: In addition to Y ¯ x,e that reflects the causal effects of entity mentions, there is another kind of bias not conditioned on the entity mentions e , but reflecting the general bias in the whole dataset, which is Y ¯ x .

Original: Y ¯ x corresponds to the counterfactual inputs whereboth textual context and entity mentions are removed.
Revised: Y ¯ x corresponds to the counterfactual inputs where both textual context and entity mentions are removed.

Original: In this case, since the model cannot accessany information from the input after this removal,
Revised: In this case, since the model cannot access any information from the input after this removal,

Original: Y ¯ x naturally reflects the label bias that exists in themodel from the biased training.
Revised: Y ¯ x naturally reflects the label bias that exists in the model from the biased training.

Original: The causal graphicviews of the original prediction Y x , the counterfactual Y ¯ x,e for the entity bias, and Y ¯ x for the labelbias are visualized in Fig.
Revised: The causal graphic views of the original prediction Y x , the counterfactual Y ¯ x,e for the entity bias, and Y ¯ x for the label bias are visualized in Fig.

Original: 3. Bias Mitigation
Revised: 3. Bias Mitigation

Original: As we have discussed in §1, instead of the staticlikelihood that tends to be biased, the unbiased relation prediction lies in the difference between theobserved outcome Y x and its counterfactual predictions Y ¯ x,e , Y ¯ x .
Revised: As we have discussed in §1, instead of the static likelihood that tends to be biased, the unbiased relation prediction lies in the difference between the observed outcome Y x and its counterfactual predictions Y ¯ x,e , Y ¯ x .

Original: The latter two are the biases that wewant to mitigate from the relation prediction.
Revised: The latter two are the biases that we want to mitigate from the relation prediction.

Original: Intuitively, the unbiased prediction that we seekis the linguistic stimuli from blank to the observedtextual context with specific relation descriptions,but not merely from the entity bias.
Revised: Intuitively, the unbiased prediction that we seek is the linguistic stimuli from blank to the observed textual context with specific relation descriptions, but not merely from the entity bias.

Original: The contextspecific clues of the relations are key to the informative unbiased predictions, because even ifthe overall prediction is biased towards the relation “ schools attended ” due to the object entitylike “ Duke University ”, the textual context “workat” indicates the relation as “employee of” ratherthan “schools attended” .
Revised: The contextspecific clues of the relations are key to the informative unbiased predictions, because even if the overall prediction is biased towards the relation “ schools attended ” due to the object entity like “ Duke University ”, the textual context “work at” indicates the relation as “employee of” rather than “schools attended” .

Original: Our final goal is to use the direct effect of thetextual context from X to Y for debiased prediction, mitigating (denoted as \ ) the label bias and theentity bias from the prediction: Y x \ Y ¯ x,e \ Y ¯ x , soas to block the spread of the biases from training toinference.
Revised: Our final goal is to use the direct effect of the textual context from X to Y for debiased prediction, mitigating (denoted as \ ) the label bias and the entity bias from the prediction: Y x \ Y ¯ x,e \ Y ¯ x , so as to block the spread of the biases from training to inference.

Original: The debiased prediction via bias mitigation can be formulated via the conceptually simplebut empirically effective element-wise subtractionoperation:
Revised: The debiased prediction via bias mitigation can be formulated via the conceptually simple but empirically effective element-wise subtraction operation:

Original: [Equation]
Revised: [Equation]

Original: where λ 1 and λ 2 are two independent hyperparameters balancing the terms for mitigating entity and label biases respectively.
Revised: where λ 1 and λ 2 are two independent hyperparameters balancing the terms for mitigating entity and label biases respectively.

Original: Note that the biasmitigation in Eq. {+
+}
Revised: Note that the bias mitigation in Eq. {+
+}

Original: 2 for the entity and label biasescorrespond to Total Direct Effect (TDE) and Total
Revised: for the entity and label biases correspond to Total Direct Effect (TDE) and Total

Original: Effect (TE) in causal inference (Tang et al., 2020;
Revised: Effect (TE) in causal inference (Tang et al., 2020;

Original: Dataset # Train # Dev # Test # Classes

Original: TACRED 68, 22,

Original: SemEval 6,

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]
Revised: [Table]

Original: VanderWeele, 2015; Pearl, 2009) respectively.
Revised: VanderWeele, 2015; Pearl, 2009) respectively.

Original: We adaptively set the values of λ 1 and λ 2 for different datasets based on the grid beam search (Hokamp and Liu, 2017) in a scoped two dimensional space:
Revised: We adaptively set the values of λ 1 and λ 2 for different datasets based on the grid beam search (Hokamp and Liu, 2017) in a scoped two dimensional space:

Original: [Equation]
Revised: [Equation]

Original: where ψ is a metric function (e.g., F1 scores) for evaluation, a, b are the boundaries of the search range.
Revised: where ψ is a metric function (e.g., F1 scores) for evaluation, a, b are the boundaries of the search range.

Original: We search the values of λ 1 , λ 2 once on the validation set, and use the fixed values for inference on all testing instances.
Revised: We search the values of λ 1 , λ 2 once on the validation set, and use the fixed values for inference on all testing instances.

Revised: Since the entity types can restrict the candidate relations (Lyu and

Revised: Chen, 2021), we use the entity type information, if available, to restrict the candidate relations for inference, which strengthens the effects of entity types for relation extraction.

Original: Overall, the proposed C O RE replaces the conventional one-time prediction with Y final to produce the debiased relation predictions, which essentially “thinks” twice : one for the original observation Y x , the other for hypothesized Y ¯ x , Y ¯ x,e .
Revised: Overall, the proposed C O RE replaces the conventional one-time prediction with Y final to produce the debiased relation predictions, which essentially “thinks” twice: one for the original observation Y x , the other for hypothesized Y ¯ x , Y ¯ x,e .

Original:  Experiments
Revised:  Experiments

Original: In this section, we evaluate the performance of our
Revised: In this section, we evaluate the performance of our

Original: C O RE methods when applied to RE models.
Revised: C O RE methods when applied to RE models.

Original: We compare our methods against a variety of strong baselines on the task of sentence-level RE.
Revised: We compare our methods against a variety of strong baselines on the task of sentence-level RE.

Original: Our experimental settings closely follow those of the previous work (Zhang et al., 2017;
Revised: Our experimental settings closely follow those of the previous work (Zhang et al., 2017;

Original: Zhou and Chen, 2021; Nan et al., 2021) to ensure a fair comparison.
Revised: Zhou and Chen, 2021; Nan et al., 2021) to ensure a fair comparison.

Original: 4. Experimental Settings
Revised: 4. Experimental Settings

Original: Datasets.
Revised: Datasets.

Original: We use four widely-used RE benchmarks: TACRED (Zhang et al., 2017), SemEval (Hendrickx et al., 2019), TACRED-Revisit (Alt et al., 2020), and Re-TACRED (Stoica et al., 2021) for evaluation.
Revised: We use four widely-used RE benchmarks: TACRED (Zhang et al., 2017), SemEval (Hendrickx et al., 2019), TACRED-Revisit (Alt et al., 2020), and Re-TACRED (Stoica et al., 2021) for evaluation.

Original: TACRED contains over 106k mention pairs drawn from the yearly TAC KBP challenge.
Revised: TACRED contains over 106k mention pairs drawn from the yearly TAC KBP challenge.

Original: (Alt et al., 2020) relabeled the development and test sets of TACRED.
Revised: (Alt et al., 2020) relabeled the development and test sets of TACRED.

Original: Re-TACRED is a further relabeled version of TACRED after refining its label definitions.
Revised: Re-TACRED is a further relabeled version of TACRED after refining its label definitions.

Original: The statistics of these datasets are shown in Tab. 1.
Revised: The statistics of these datasets are shown in Tab. 1.

Original: We use the widely-used F1-macro score as the main evaluation metric (Nan et al., 2021), which is the balanced harmonic mean of precision and recall, as well as F1-micro for a more comprehensive

Original: [Table]
Revised: [Table]

Original:  evaluation.
Revised: We use the widely-used F1-macro score as the main evaluation metric (Nan et al., 2021), which is the balanced harmonic mean of precision and recall, as well as F1-micro for a more comprehensive evaluation.

Original: F1-macro is more suitable than F1micro to reflect the extent of biases, especially forthe highly-skewed cases, since F1-macro is evenlyinfluenced by the performance in each category,i.e. category-sensitive, but F1-micro simply givesequal weights to all instances (Kim et al., 2019).
Revised: F1-macro is more suitable than F1micro to reflect the extent of biases, especially for the highly-skewed cases, since F1-macro is evenly influenced by the performance in each category, i.e. category-sensitive, but F1-micro simply gives equal weights to all instances (Kim et al., 2019).

Original: Compared methods.
Revised: Compared methods.

Original: We take the following REmodels into comparison.
Revised: We take the following RE models into comparison.

Original: (1) C-SGC (Wu et al. simplifies GCN, and combines it with LSTM,leading to improved performance over each methodalone.
Revised: (1) C-SGC (Wu et al., 2019) simplifies GCN, and combines it with LSTM, leading to improved performance over each method alone.

Original: (2) SpanBERT (Joshi et al., 2020) extends
Revised: (2) SpanBERT (Joshi et al., 2020) extends

Original: BERT by introducing a new pretraining objectiveof continuous span prediction.
Revised: BERT by introducing a new pretraining objective of continuous span prediction.

Original: (3) CP (Peng et al. ,2020) is an entity-masked contrastive pre-trainingframework for RE.
Revised: (3) CP (Peng et al., 2020) is an entity-masked contrastive pre-training framework for RE.

Original: (4) RECENT (Lyu and Chen,2021) restricts the candidate relations based on theentity types.
Revised: (4) RECENT ( Lyu and Chen, 2021) restricts the candidate relations based on the entity types.

Original: (5) KnowPrompt (Chen et al., 2021)is Knowledge-aware Prompt-tuning approach.
Revised: (5) KnowPrompt (Chen et al., 2021) is Knowledge-aware Prompt-tuning approach.

Original: LUKE (Yamada et al., 2020) pretrains the languagemodel on both large text corpora and knowledgegraphs and further proposes an entity-aware selfattention mechanism.
Revised: LUKE (Yamada et al., 2020) pretrains the language model on both large text corpora and knowledge graphs and further proposes an entity-aware selfattention mechanism.

Original: (7) IRE (Zhou and Chen,2021) proposes an improved entity representationtechnique in the data preprocessing.
Revised: (7) IRE (Zhou and Chen, 2021) proposes an improved entity representation technique in the data preprocessing.

Original: Among the above RE models, we apply our
Revised: Among the above RE models, we apply our

Original: C O RE on LUKE and IRE.
Revised: C O RE on LUKE and IRE.

Original: To demonstrate the effectiveness of debiased inference, we also comparewith the following debiasing techniques that areapplied to the same two RE models.
Revised: To demonstrate the effectiveness of debiased inference, we also compare with the following debiasing techniques that are applied to the same two RE models.

Original: (1) Focal (Linet al., 2017) adaptively reweights the losses of different instances so as to focus on the hard ones.
Revised: (1) Focal (Lin et al., 2017) adaptively reweights the losses of different instances so as to focus on the hard ones.

Original: Resample (Burnaev et al., 2015) up-samples rare categories by the inversed sample fraction during training.
Revised: Resample (Burnaev et al., 2015) up-samples rare categories by the inversed sample fraction during training.

Original: (3) Entity Mask (Zhang et al., 2017): masks the entity mentions with special tokens to reduce the over-fitting on entities.
Revised: (3) Entity Mask (Zhang et al., 2017): masks the entity mentions with special tokens to reduce the over-fitting on entities.

Original: (4) CFIE (Nan et al., 2021) is also a causal inference method.
Revised: (4) CFIE (Nan et al., 2021) is also a causal inference method.

Original: In contrast to our method, CFIE strengthens the causal effects of entities by masking entity-centric information in the counterfactual predictions.
Revised: In contrast to our method, CFIE strengthens the causal effects of entities by masking entity-centric information in the counterfactual predictions.

Original: Model configuration.
Revised: Model configuration.

Original: For the hyper-parameters of the considered baseline methods, e.g., the batch size, the number of hidden units, the optimizer, and the learning rate, we set them as suggested by their authors.
Revised: For the hyper-parameters of the considered baseline methods, e.g., the batch size, the number of hidden units, the optimizer, and the learning rate, we set them as suggested by their authors.

Original: For the hyper-parameters of our C O RE method, we set the search range of the hypermeters in Eq.
Revised: For the hyper-parameters of our C O RE method, we set the search range of the hypermeters in Eq.

Original: 3 as [ − 2 , 2] and the search step 0 .
Revised: 3 as [ − 2 , 2] and the search step 0 .

Original: For all experiments, we report the median F 1 scores of five runs of training using different random seeds.
Revised: For all experiments, we report the median F 1 scores of five runs of training using different random seeds.

Original: 4. Overall Performance
Revised: 4. Overall Performance

Original: We implement our C O RE with LUKE and
Revised: We implement our C O RE with LUKE and

Original: IRE RoBERTa .
Revised: IRE RoBERTa .

Original: Tab.
Revised: Tab.

Original: 3 reports the RE results on the
Revised: 3 reports the RE results on the

Original: TACRED, TACRED-Revisit, Re-TACRED, and SemEval datasets.
Revised: TACRED, TACRED-Revisit, Re-TACRED, and SemEval datasets.

Original: Our C O RE method improves the
Revised: Our C O RE method improves the

Original: F1-macro scores of LUKE by 4.9% on TACRED, 4.0% on TACRED-Revisit, 1.7% on Re-TACRED, and 1.7 on SemEval, and improves IRE RoBERTaby 1.2% on TACRED, 1.4% on TACRED-Revisit, 0.9% on Re-TACRED, and 1.8% on SemEval.
Revised: F1-macro scores of LUKE by 4.9% on TACRED, 4.0% on TACRED-Revisit, 1.7% on Re-TACRED, and 1.7 on SemEval, and improves IRE RoBERTa

Original: As a result, our C O RE achieves substantial improvements for LUKE and IRE RoBERTa , and enables

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original:  them to outperform the baseline methods.
Revised: by 1.2% on TACRED, 1.4% on TACRED-Revisit, 0.9% on Re-TACRED, and 1.8% on SemEval. As a result, our C O RE achieves substantial improvements for LUKE and IRE RoBERTa , and enables them to outperform the baseline methods.

Original: Additionally, we report the experimental results in termsof F1-micro scores in Tab. 3, showing the improvement from C O RE on LUKE by 2.6% on TACRED,1.0% on TACRED-Revisit, 0.7% on Re-TACRED,and 1.0% on SemEval.
Revised: Additionally, we report the experimental results in terms of F1-micro scores in Tab. 3, showing the improvement from C O RE on LUKE by 2.6% on TACRED, 1.0% on TACRED-Revisit, 0.7% on Re-TACRED, and 1.0% on SemEval.

Original: Overall, our C O RE methodimproves the effectiveness of RE significantly interms of both F1-macro and F1-micro scores.
Revised: Overall, our C O RE method improves the effectiveness of RE significantly in terms of both F1-macro and F1-micro scores.

Original: Theabove experimental results validate the effectiveness and generalization of our proposed method.
Revised: The above experimental results validate the effectiveness and generalization of our proposed method.

Original: Among the baseline debiasing methods, Resample, Focal, CFIE cannot distill the entity bias inan entity-aware manner like ours.
Revised: Among the baseline debiasing methods, Resample, Focal, CFIE cannot distill the entity bias in an entity-aware manner like ours.

Original: Entity Maskleads to the loss of information, while our C O REenables RE models to focus on the main effectsof textual context without losing the entity information.
Revised: Entity Mask leads to the loss of information, while our C O RE enables RE models to focus on the main effects of textual context without losing the entity information.

Original: The superiority of C O RE highlights theimportance of the causal inference based entity biasanalysis for debiasing RE, which compares traditional likelihood-based predictions and hypothesized counterfactual ones to produce debiased predictions.
Revised: The superiority of C O RE highlights the importance of the causal inference based entity bias analysis for debiasing RE, which compares traditional likelihood-based predictions and hypothesized counterfactual ones to produce debiased predictions.

Original: Besides, the proposed C O RE works ininference and thus can be employed on the previous already-trained models.
Revised: Besides, the proposed C O RE works in inference and thus can be employed on the previous already-trained models.

Original: In this way, C O RE[Table] serves as a model-agnostic approach to enhance RE models without changing their training process.
Revised: In this way, C O RE serves as a model-agnostic approach to enhance RE models without changing their training process.

Original: 4. Analysis on Entity Bias
Revised: 4. Analysis on Entity Bias

Original: Some work argues that RE models may rely on the entity mentions to make relation predictions instead of the textual context (Zhang et al., 2018;
Revised: Some work argues that RE models may rely on the entity mentions to make relation predictions instead of the textual context (Zhang et al., 2018;

Original: Joshi et al., 2020).
Revised: Joshi et al., 2020).

Original: The empirical results in Fig. validates this argument.
Revised: The empirical results in Fig.validates this argument.

Original: Regardless of whether the textual context exists or not, the baseline RE model makes the same predictions given only entity mentions on many instances.
Revised: Regardless of whether the textual context exists or not, the baseline RE model makes the same predictions given only entity mentions on many instances.

Original: The entity bias can mislead the RE models to make wrong predictions when the relation implied by the entity mentions does not exist in the textual context.
Revised: The entity bias can mislead the RE models to make wrong predictions when the relation implied by the entity mentions does not exist in the textual context.

Original: To evaluate whether RE models can generalize well to particularly challenging instances where relations implied by the entity mentions do not exist in the textual context, we propose a filtered evaluation setting, where we keep the test instances having the entity bias different from their groundtruth relations.
Revised: To evaluate whether RE models can generalize well to particularly challenging instances where relations implied by the entity mentions do not exist in the textual context, we propose a filtered evaluation setting, where we keep the test instances having the entity bias different from their groundtruth relations.

Original: In this setting, RE models cannot overly rely on the entity mentions for RE, since the entity mentions no longer provide the superficial and spurious clues for the ground-truth relations.
Revised: In this setting, RE models cannot overly rely on the entity mentions for RE, since the entity mentions no longer provide the superficial and spurious clues for the ground-truth relations.

Original: We present the evaluation results on the filtered test set in Tab. 4.
Revised: We present the evaluation results on the filtered test set in Tab. 4.

Original: Our C O RE method consistently and substantially improves the effectiveness of LUKE and IRE on the filtered test set and outperforms the baseline methods by a significant margin, 
Revised: Our C O RE method consistently and substantially improves the effectiveness of LUKE and IRE on the filtered test set and outperforms the baseline methods by a significant margin, which validates the effectiveness and generalization of our method to mitigate the entity bias in the challenging cases. 4. Evaluation on Fairness

Revised: According to Sweeney and Najafian (2019), the more imbalanced/skewed a prediction produced by a trained model is, the more unfair opportunities it

Revised: [Table]

Original: [Table]
Revised: [Table]

Revised: The analyzed model components include entity bias mitigation operation (EBM), the label bias mitigation operation (LBM) and the beam search for hyper-parameters (BSH).

Revised: ‘w/o’ denotes ‘without’.

Revised: ↓ denotes performance drop in terms of F1-macro scores.

Revised: gives over predefined categories, and the more unfairly discriminative the trained model is.

Revised: We thus follow previous work (Xiang et al., 2020; Sweeney and Najafian, 2019; Qian et al., 2021) to use the metric – imbalance divergence – to evaluate how imbalanced/skewed/unfair a prediction P is :

Original: [Table]
Revised: [Equation] where D ( · ) is defined as the distance between P and the uniform distribution U . Specifically, we use the JS divergence as the distance metric since it is symmetric (i.e., JS ( P ∥ U ) = JS ( U ∥ P ) ) and strictly scoped (Fuglede and Topsoe, 2004).

Revised: Based on this, to evaluate the entity bias of a trained RE model, we average the following relative entity mention imbalance (REI) measure over all the testing instances containing whichever entity mentions:

Original: [Table]
Revised: [Equation]

Revised: where x is an input instance, D is the testing set,

Revised: P ( x ) is the prediction output, e is an entity mention, and E is the corpus of entity mentions.

Original: which validates the effectiveness and generalization of our  method  to mitigate the entity bias in thechallenging cases.
Revised: This metric captures the distance between all predictions and the fair uniform distribution U . We follow the experimental settings in §4.2 and report the fairness test in Tab. 5. The results show that our C O RE method reduces the imbalance metrics (lower is better) when employed on IRE RoBERTa significantly and consistently, indicating that it is helpful to mitigate the entity bias.

Original: 4. Ablation and Case Study
Revised: 4. Ablation and Case Study

Original: We conduct ablation studies on C O RE to empirically examine the contribution of its main technical components.
Revised: We conduct ablation studies on C O RE to empirically examine the contribution of its main technical components.

Original: including the entity bias mitigation operation (EBM), the label bias mitigationoperation (LBM) and the beam search for hyperparameters (BSH).
Revised: including the entity bias mitigation operation (EBM), the label bias mitigation operation (LBM) and the beam search for hyperparameters (BSH).

Original: We report the experimental results of the ablation study in Tab. 5.
Revised: We report the experimental results of the ablation study in Tab. 6.

Original: We observe that removingour C O RE causes serious performance degradation.
Revised: We observe that removing our C O RE causes serious performance degradation.

Original: This provides evidence that using our counterfactual framework for RE can explicitly mitigate biases to generalize better on unseen examples.
Revised: This provides evidence that using our counterfactual framework for RE can explicitly mitigate biases to generalize better on unseen examples.

Original: Moreover, we observe that mitigating the twotypes of biases is consistently helpful for RE.
Revised: Moreover, we observe that mitigating the two types of biases is consistently helpful for RE.

Original: Thekey reason is that the distilled label bias providesan instance-agnostic offset and the distilled entitybias provides an entity-aware one in the prediction space, which makes the RE models focus onextracting relations on the textual context withoutlosing the entity information.
Revised: The key reason is that the distilled label bias provides an instance-agnostic offset and the distilled entity bias provides an entity-aware one in the prediction space, which makes the RE models focus on extracting relations on the textual context without losing the entity information.

Original: Meanwhile, the beamsearch for hyper-parameters effectively finds twodynamic scaling factors to amplify or shrink twobiases, making the biases be mitigated properly andadaptively.
Revised: Meanwhile, the beam search for hyper-parameters effectively finds two dynamic scaling factors to amplify or shrink two biases, making the biases be mitigated properly and adaptively.

Original: Tab. 6 gives a qualitative comparison example between C O RE and IRE RoBERTa on TACRED.
Revised: Tab. 7 gives a qualitative comparison example between C O RE and IRE RoBERTa on TACRED.

Original: Theresults show that the state-of-the-art RE model
Revised: The results show that the state-of-the-art RE model

Original: IRE RoBERTa returns the relations that do not existin the textual context between the considered entities.
Revised: IRE RoBERTa returns the relations that do not exist in the textual context between the considered entities.

Original: For example, given “ Bibi drew the ire of fellowfarmhands after a dispute in June 2009, when they[Table] refused to drink water she collected and she re- [Table]
Revised: For example, given “ Bibi drew the ire of fellow farmhands after a dispute in June 2009, when they refused to drink water she collected and she refused their demands that she convert to (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Islam . ”

Original: fused their demands that she convert to

Original: [Table]

Original: [Table] there is no relation reflected by Bibi and Islam [Table] but the baseline model believes that the relation 
Revised: , there is no relation between Bibi and Islam exists in the text but the baseline model believes that the relation between them is “ religion ”.

Original: [Table]

Original: between them is “ religion ”.

Original: The counterfactual[Table] prediction can account for this disappointing result,
Revised: The counterfactual prediction can account for this disappointing result,

Original: [Table]
Revised: [Table]

Original: where given only the entity mentions Bibi and Is-
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: lam , the RE model returns the relation “ religion ” without any textual context.
Revised: where given only the entity mentions Bibi and Islam , the RE model returns the relation “ religion ” without any textual context.

Original: This implies that the model makes the prediction for the original input relying on the entity mentions, which leads to the wrong RE prediction.
Revised: This implies that the model makes the prediction for the original input relying on the entity mentions, which leads to the wrong RE prediction.

Original: Our C O RE method distills the biases through counterfactual predictions and mitigates the biases to distinguish the main effects from the textual context, which leads to the correct predictions as shown in Tab. 6.
Revised: Our C O RE method distills the biases through counterfactual predictions and mitigates the biases to distinguish the main effects from the textual context, which leads to the correct predictions as shown in Tab. 7.

Original: Last but not least, we conduct experiments on the fairness of different models, and present respective results in the appendix.
Revised: Last but not least, we conduct experiments on the fairness of different models, and present respective results in the appendix.

Original:  Conclusion
Revised:  Conclusion

Original: We have designed a counterfactual analysis based method named C O RE to debias RE.
Revised: We have designed a counterfactual analysis based method named C O RE to debias RE.

Original: We distill the entity bias and mitigate the distilled biases with the help of our causal graph for RE, which is a road map for analyzing the RE models.
Revised: We distill the entity bias and mitigate the distilled biases with the help of our causal graph for RE, which is a road map for analyzing the RE models.

Original: Based on the counterfactual analysis, we can analyze the sideeffects of entity mentions in the RE and debias the models in an entity-aware manner.
Revised: Based on the counterfactual analysis, we can analyze the sideeffects of entity mentions in the RE and debias the models in an entity-aware manner.

Original: Extensive experiments demonstrate that our methods can improve the effectiveness and generalization of RE.
Revised: Extensive experiments demonstrate that our methods can improve the effectiveness and generalization of RE.

Original: Future work includes analyzing the effects of other factors that can cause bias in natural language processing.
Revised: Future work includes analyzing the effects of other factors that can cause bias in natural language processing

Original: A Experimental Results on Fairness
Revised: Acknowledgement

Original: According to Sweeney and Najafian (2019), themore imbalanced/skewed a prediction produced bya trained model is, the more unfair opportunities itgives over predefined categories, and the more unfairly discriminative the trained model is. We thusfollow previous work (Xiang et al., 2020; Sweeneyand Najafian, 2019;

Original: Qian et al., 2021) to use themetric – imbalance divergence – to evaluate howimbalanced/skewed/unfair a prediction P is : [Equation] where D ( · ) is defined as the distance between Pand the uniform distribution U . Specifically, weuse the JS divergence as the distance metric sinceit is symmetric (i.e., JS ( P ∥ U )
Revised: The authors would like to thank the anonymous reviewers for their discussion and feedback.

Original: = JS ( U ∥ P ) ) andstrictly scoped (Fuglede and Topsoe, 2004).
Revised: Muhao Chen and Wenxuan Zhou are supported by the National Science Foundation of United

Original: Basedon this, to evaluate the entity bias of a trained REmodel, we average the following relative entitymention imbalance (REI) measure over all the testing instances containing whichever entity mentions: [Equation] where x is an input instance, D is the testing set, P ( x ) is the prediction output, e is an entity mention, and E is the corpus of entity mentions.
Revised: States Grant IIS and by the DARPA MCS program under Contract No. N660011924033 with the United States Office Of Naval Research.

Original: Thismetric captures the distance between all predictionsand the fair uniform distribution U .
Revised: Except for Muhao Chen and Wenxuan Zhou, this paper is supported by NUS ODPRT

Original: We follow the experimental settings in §4.2 andreport the fairness test in Tab. 7. The resultsshow that our C O RE method reduces the imbalance metrics (lower is better) when employed on IRE RoBERTa significantly and consistently, indicating that it is helpful to mitigate the entity bias. [Table]
Revised: Grant R252-000A81-133 and Singapore Ministry of Education Academic Research Fund Tier 3 under MOEs official grant number MOE2017-T3-1-007.