Original: Manuscript
Revised: Manuscript

Original: C AUSAL
Revised: C AUSAL

Original: P ROXY
Revised: P ROXY

Original: M ODELS
Revised: M ODELS

Original: FOR
Revised: FOR

Original: C ONCEPT - BASED
Revised: C ONCEPT - BASED

Original: M ODEL E XPLANATIONS
Revised: M ODEL E XPLANATIONS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Explainability methods for NLP systems encounter a version of the fundamental problem of causal inference: for a given ground-truth input text, we never truly observe the counterfactual texts necessary for isolating the causal effects of model representations on outputs.
Revised: Explainability methods for NLP systems encounter a version of the fundamental problem of causal inference: for a given ground-truth input text, we never truly observe the counterfactual texts necessary for isolating the causal effects of model representations on outputs.

Original: In response, many explainability methods make no use of counterfactual texts, assuming they will be unavailable.
Revised: In response, many explainability methods make no use of counterfactual texts, assuming they will be unavailable.

Original: In this paper, we show that robust causal explainability methods can be created using approximate counterfactuals, which can be written by humans to approximate a specific counterfactual or simply sampled using metadata-guided heuristics.
Revised: In this paper, we show that robust causal explainability methods can be created using approximate counterfactuals, which can be written by humans to approximate a specific counterfactual or simply sampled using metadata-guided heuristics.

Original: The core of our proposal is the Causal Proxy Model (CPM).
Revised: The core of our proposal is the Causal Proxy Model (CPM).

Original: A CPM explains a black-box model N because it is trained to have the same actual input/output behavior as N while creating neural representations that can be intervened upon to simulate the counterfactual input/output behavior of N .
Revised: A CPM explains a black-box model N because it is trained to have the same actual input/output behavior as N while creating neural representations that can be intervened upon to simulate the counterfactual input/output behavior of N .

Original: Furthermore, we show that the best CPM for N performs comparably to N in making factual predictions, which means that the
Revised: Furthermore, we show that the best CPM for N performs comparably to N in making factual predictions, which means that the

Original: CPM can simply replace N , leading to more explainable deployed models.
Revised: CPM can simply replace N , leading to more explainable deployed models.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The gold standard for explanation methods in AI should be to elucidate the causal role that a model’s representations play in its overall behavior – to truly explain why the model makes the predictions it does.
Revised: The gold standard for explanation methods in AI should be to elucidate the causal role that a model’s representations play in its overall behavior – to truly explain why the model makes the predictions it does.

Original: Causal explanation methods seek to do this by resolving the counterfactual question of what the model would do if input X were changed to a relevant counterfactual version X ′ .
Revised: Causal explanation methods seek to do this by resolving the counterfactual question of what the model would do if input X were changed to a relevant counterfactual version X ′ .

Original: Unfortunately, even though neural networks are fully observed, deterministic systems, we still encounter the fundamental problem of causal inference (Holland, 1986): for a given ground-truth input X , we never observe the counterfactual inputs X ′ necessary for isolating the causal effects of model representations on outputs.
Revised: Unfortunately, even though neural networks are fully observed, deterministic systems, we still encounter the fundamental problem of causal inference (Holland, 1986): for a given ground-truth input X , we never observe the counterfactual inputs X ′ necessary for isolating the causal effects of model representations on outputs.

Original: The issue is especially pressing in domains where it is hard to synthesize approximate counterfactuals.
Revised: The issue is especially pressing in domains where it is hard to synthesize approximate counterfactuals.

Original: In response to this, explanation methods typically do not explicitly train on counterfactuals at all.
Revised: In response to this, explanation methods typically do not explicitly train on counterfactuals at all.

Original: In this paper, we show that robust explanation methods for NLP models can be obtained using texts approximating true counterfactuals.
Revised: In this paper, we show that robust explanation methods for NLP models can be obtained using texts approximating true counterfactuals.

Original: The heart of our proposal is the Causal Proxy Model (CPM).
Revised: The heart of our proposal is the Causal Proxy Model (CPM).

Original: CPMs are trained to mimic both the factual and counterfactual behavior of a black-box model N .
Revised: CPMs are trained to mimic both the factual and counterfactual behavior of a black-box model N .

Original: We explore two different methods for training such explainers.
Revised: We explore two different methods for training such explainers.

Original: These methods share a distillation-style objective that pushes them to mimic the factual behavior of N , but they differ in their counterfactual objectives.
Revised: These methods share a distillation-style objective that pushes them to mimic the factual behavior of N , but they differ in their counterfactual objectives.

Original: The input-based method CPM IN appends to the factual input a new token associated with the counterfactual concept value.
Revised: The input-based method CPM IN appends to the factual input a new token associated with the counterfactual concept value.

Original: The hidden-state method CPM HI employs the Interchange
Revised: The hidden-state method CPM HI employs the Interchange

Original: Intervention Training (IIT) method of Geiger et al.
Revised: Intervention Training (IIT) method of Geiger et al.

Original: (2021b) to localize information about the target concept in specific hidden states.
Revised: (2022) to localize information about the target concept in specific hidden states.

Original: Figure 1 provides a high-level overview.
Revised: Figure 1 provides a high-level overview.

Original: We evaluate these methods on the CEBaB benchmark for causal explanation methods (Abraham et al., 2022), which provides large numbers of original examples (restaurant reviews) with humancreated counterfactuals for specific concepts (e.g., service quality), with all the texts labeled for their concept-level and text-level sentiment.
Revised: We evaluate these methods on the CEBaB benchmark for causal explanation methods (Abraham et al., 2022), which provides large numbers of original examples (restaurant reviews) with humancreated counterfactuals for specific concepts (e.g., service quality), with all the texts labeled for their concept-level and text-level sentiment.

Original: We consider two types of approximate counterfactuals derived from CEBaB: texts written by humans to approximate a specific counterfactual, and texts sampled using metadata-guided heuristics.
Revised: We consider two types of approximate counterfactuals derived from CEBaB: texts written by humans to approximate a specific counterfactual, and texts sampled using metadata-guided heuristics.

Original: Both approximate counterfactual strategies lead to state-of-the-art performance on CEBaB for both CPM IN and CPM HI .
Revised: Both approximate counterfactual strategies lead to state-of-the-art performance on CEBaB for both CPM IN and CPM HI .

Original: We additionally identify two other benefits of using CPMs to explain models.
Revised: We additionally identify two other benefits of using CPMs to explain models.

Original: First, both CPM IN and CPM HI have factual performance comparable to that of the original black-box model N and can explain their own behavior extremely well.
Revised: First, both CPM IN and CPM HI have factual performance comparable to that of the original black-box model N and can explain their own behavior extremely well.

Original: Thus, the CPM for N can actually replace N , leading to  more explainable deployed models.
Revised: Thus, the CPM for N can actually replace N , leading to Manuscript more explainable deployed models.

Original: Second, CPM HI models localize concept-level information in their hidden representations, which makes their behavior on specific inputs very easy to explain.
Revised: Second, CPM HI models localize concept-level information in their hidden representations, which makes their behavior on specific inputs very easy to explain.

Original: We illustrate this using the Integrated Gradients method of Sundararajan et al.
Revised: We illustrate this using Path Integrated Gradients (Sundararajan et al., 2017), which we adapt to allow input-level attributions to be mediated by the intermediate states that were targeted for localization.

Original: Thus, while both CPM IN and CPM HI are comparable as explanation methods according to CEBaB, the qualitative insights afforded by CPM HI models may given them the edge when it comes to explanations.
Revised: Thus, while both CPM IN and CPM HI are comparable as explanation methods according to CEBaB, the qualitative insights afforded by CPM HI models may given them the edge when it comes to explanations.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Understanding model behavior serves many goals for large-scale AI systems, including transparency (Kim, 2015; Lipton, 2018; Pearl, 2019; Ehsan et al., 2021), trustworthiness (Ribeiro et al., 2016;
Revised: Understanding model behavior serves many goals for large-scale AI systems, including transparency (Kim, 2015; Lipton, 2018; Pearl, 2019; Ehsan et al., 2021), trustworthiness (Ribeiro et al., 2016;

Original: Guidotti et al., 2018; Jacovi & Goldberg, 2020; Jakesch et al., 2019), safety (Amodei et al., 2016; Otte, 2013), and fairness (Hardt et al., 2016; Kleinberg et al., 2017; Goodman & Flaxman, 2017;
Revised: Guidotti et al., 2018; Jacovi & Goldberg, 2020; Jakesch et al., 2019), safety (Amodei et al., 2016; Otte, 2013), and fairness (Hardt et al., 2016; Kleinberg et al., 2017; Goodman & Flaxman, 2017;

Original: Mehrabi et al., 2021).
Revised: Mehrabi et al., 2021).

Original: With CPMs, our goal is to achieve explanations that are causally motivated and concept-based, and so we concentrate here on relating existing methods to these two goals.
Revised: With CPMs, our goal is to achieve explanations that are causally motivated and concept-based, and so we concentrate here on relating existing methods to these two goals.

Original: Feature attribution methods estimate the importance of features, generally by inspecting learned weights directly or by perturbing features and studying the effects this has on model behavior (Molnar, 2020; Ribeiro et al., 2016).
Revised: Feature attribution methods estimate the importance of features, generally by inspecting learned weights directly or by perturbing features and studying the effects this has on model behavior (Molnar, 2020; Ribeiro et al., 2016).

Original: Gradient-based feature attribution methods extend this general mode of explanation to the hidden representations in deep networks (Zeiler & Fergus, 2014; Springenberg et al., 2014; Binder et al., 2016; Shrikumar et al., 2017; Sundararajan et al., 2017).
Revised: Gradient-based feature attribution methods extend this general mode of explanation to the hidden representations in deep networks (Zeiler & Fergus, 2014; Springenberg et al., 2014; Binder et al., 2016; Shrikumar et al., 2017; Sundararajan et al., 2017).

Original: Concept Activation
Revised: Concept Activation

Original: Vectors (CAVs; Kim et al. 2018;
Revised: Vectors (CAVs; Kim et al. 2018;

Original: Yeh et al. 2020) can also be considered feature attribution methods, as they probe for semantically meaningful directions in the model’s internal representations and use these to estimate the importance of concepts on the model predictions.
Revised: Yeh et al. 2020) can also be considered feature attribution methods, as they probe for semantically meaningful directions in the model’s internal representations and use these to estimate the importance of concepts on the model predictions.

Original: While some methods in this space do have causal interpretations (e.g., Sundararajan et al. 2017;
Revised: While some methods in this space do have causal interpretations (e.g., Sundararajan et al. 2017;

Original: Yeh et al. 2020), most do not.
Revised: Yeh et al. 2020), most do not.

Original: In addition, most of these methods offer explanations in terms of specific (sets of) features/neurons.
Revised: In addition, most of these methods offer explanations in terms of specific (sets of) features/neurons.

Original: (Methods based on CAVs operate directly in terms of more abstract concepts.)
Revised: (Methods based on CAVs operate directly in terms of more abstract concepts.)

Original: Intervention-based methods study model representations by modifying them in systematic ways and observing the resulting model behavior.
Revised: Intervention-based methods study model representations by modifying them in systematic ways and observing the resulting model behavior.

Original: These methods are generally causally motivated and allow for concept-based explanations.
Revised: These methods are generally causally motivated and allow for concept-based explanations.

Original: Examples of methods in this space include causal mediation analysis (Vig et al., 2020; De Cao et al., 2021;
Revised: Examples of methods in this space include causal mediation analysis (Vig et al., 2020; De Cao et al., 2021;

Original: Ban et al., 2022), causal effect estimation (Feder et al., 2021;
Revised: Ban et al., 2022), causal effect estimation (Feder et al., 2020;

Original: Elazar et al., 2021; Abraham et al.,  2022), tensor product decomposition (Soulos et al., 2020), and causal abstraction analysis (Geiger et al., 2020; 2021a).
Revised: Elazar et al., 2021; Abraham et al., 2022; Lovering & Pavlick, 2022), tensor product decomposition (Soulos et al., 2020), and causal abstraction analysis (Geiger et al., 2020; 2021).

Original: CPMs are most closely related to the method of IIT (Geiger et al., 2021a), which extends causal abstraction analysis to optimization.
Revised: CPMs are most closely related to the method of IIT (Geiger et al., 2021), which extends causal abstraction analysis to optimization.

Original: Probing is another important class of explanation method.
Revised: Probing is another important class of explanation method.

Original: Traditional probes do not intervene on the target model, but rather only seek to find information in it via supervised models (Conneau et al., 2018; Tenney et al., 2019) or unsupervised models (Clark et al., 2019; Manning et al., 2020; Saphra & Lopez, 2019).
Revised: Traditional probes do not intervene on the target model, but rather only seek to find information in it via supervised models (Conneau et al., 2018; Tenney et al., 2019) or unsupervised models (Clark et al., 2019; Manning et al., 2020; Saphra & Lopez, 2019).

Original: Probes can identify concept-based information, but they cannot offer guarantees that probed information is relevant for model behavior (Geiger et al., 2021a).
Revised: Probes can identify concept-based information, but they cannot offer guarantees that probed information is relevant for model behavior (Geiger et al., 2021).

Original: For causal guarantees, it is likely that some kind of intervention is required.
Revised: For causal guarantees, it is likely that some kind of intervention is required.

Original: For example, Elazar et al.
Revised: For example, Elazar et al.

Original: and Feder et al.
Revised: and Feder et al.

Original: (2021) remove information from model representations to estimate the causal role of that information.
Revised: (2020) remove information from model representations to estimate the causal role of that information.

Original: Our CPMs employ a similar set of guiding ideas but are not limited to removing information.
Revised: Our CPMs employ a similar set of guiding ideas but are not limited to removing information.

Original: Counterfactual explanation methods aim to explain model behavior by providing a counterfactual example that changes the model behavior (Goyal et al., 2019; Verma et al., 2020; Wu et al., 2021).
Revised: Counterfactual explanation methods aim to explain model behavior by providing a counterfactual example that changes the model behavior (Goyal et al., 2019; Verma et al., 2020; Wu et al., 2021).

Original: Counterfactual explanation methods are inherently causal.
Revised: Counterfactual explanation methods are inherently causal.

Original: If they can provide counterfactual examples with regard to specific concepts, they are also concept-based.
Revised: If they can provide counterfactual examples with regard to specific concepts, they are also concept-based.

Original: Some explanation methods train a model making explicit use of intermediate variables representing concepts.
Revised: Some explanation methods train a model making explicit use of intermediate variables representing concepts.

Original: Manipulating these intermediate variables at test-time yields causal concept-based model explanations (Koh et al., 2020; Künzel et al., 2019).
Revised: Manipulating these intermediate variables at inference time yields causal concept-based model explanations (Koh et al., 2020; Künzel et al., 2019).

Original: Evaluating methods in this space has been a persistent challenge.
Revised: Evaluating methods in this space has been a persistent challenge.

Original: In prior literature, explanation methods have often been evaluated against synthetic datasets (Feder et al., 2021; Yeh et al., 2020).
Revised: In prior literature, explanation methods have often been evaluated against synthetic datasets (Feder et al., 2020; Yeh et al., 2020).

Original: In response, Abraham et al.
Revised: In response, Abraham et al.

Original: (2022) introduced the CEBaB dataset, which provides a human-validated concept-based dataset to truthfully evaluate different causal concept-based model explanation methods.
Revised: (2022) introduced the CEBaB dataset, which provides a human-validated concept-based dataset to truthfully evaluate different causal concept-based model explanation methods.

Original: Our primary evaluations are conducted on CEBaB.
Revised: Our primary evaluations are conducted on CEBaB.

Original: Manuscript x u,v c
Revised: Manuscript x u,v c

Original: i
Revised: i

Original: c...
Revised: c...

Original: c k
Revised: c k

Original: U = u
Revised: U = u

Original: V = v x C i ← c ′ u,v C
Revised: V = v x C i ← c ′ u,v C

Original: i ← c ′ c...
Revised: i ← c ′ c...

Original: c k
Revised: c k

Original: U = u
Revised: U = u

Original: V = v
Revised: V = v

Original: Let x u,v be a text written in situation ( u, v ) :
Revised: Let x u,v be a text written in situation ( u, v ) :

Original: Human-created ˜ x C i
Revised: Human-created ˜ x C i

Original: [Table] [Table]
Revised: [Table] [Table]

Original: ,v
Revised: ,v

Original: Crowdworker edit of x u,v to express that C i had value c ′ , seeking to keep all else constant.
Revised: Crowdworker edit of x u,v to express that C i had value c ′ , seeking to keep all else constant.

Original: Metadata-sampled
Revised: Metadata-sampled

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Equation]
Revised: [Equation]

Original: v
Revised: v

Original: [Equation]
Revised: [Equation]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table]L HI : Examples x u,v and ˜ x C i ← c ′ u,v are an approxi- mate counterfactual pair. The CPM is given input x u,v .
Revised: [Table]

Original: The objective is for it to mimic N given ˜ x C i ← c ′ u,v , but under the intervention in which specific internal states are changed to those that the CPM computes for input x C i = c ′ u ′ ,v ′ , which is a distinct example that is sampled with the only criteria being that it express C i = c ′ . The effect of this intervention is to localize informa- tion about concept C i at the intervention site, since the only indication the CPM gets about C i ← c ′ is via the intervention.
Revised: [Table]

Original:  C AUSAL P ROXY M ODEL (CPM)
Revised:  C AUSAL P ROXY M ODEL (CPM)

Original: Causal Proxy Model (CPMs) are causal concept-based explanation methods.
Revised: Causal Proxy Models (CPMs) are causal concept-based explanation methods.

Original: Given a factual input x u,v and a description of a concept intervention C i ← c ′ , they estimate the effect of the intervention on model output.
Revised: Given a factual input x u,v and a description of a concept intervention C i ← c ′ , they estimate the effect of the intervention on model output.

Original: The present section introduces our two core CPM variants in detail.
Revised: The present section introduces our two core CPM variants in detail.

Original: We concentrate here on introducing the structure of these models and their objectives, and we save discussion of associated metrics for explanation methods for Section4.
Revised: We concentrate here on introducing the structure of these models and their objectives and we save discussion of associated metrics for explanation methods for Section

Original: A Structural Causal Model Our discussion is grounded in the causal model depicted in Figure 1a, which aligns well with the CEBaB benchmark.
Revised: A Structural Causal Model Our discussion is grounded in the causal model depicted in Figure 1a, which aligns well with the CEBaB benchmark.

Original: Two exogenous variables U and V together represent the complete state of the world and generate some textual data X .
Revised: Two exogenous variables U and V together represent the complete state of the world and generate some textual data X .

Original: The effect of exogenous variable
Revised: The effect of exogenous variable

Original: U on the data X is completely mediated by a set of intermediate variables C 1 , C 2 . . .
Revised: U on the data X is completely mediated by a set of intermediate variables C 1 , C 2 . . .

Original: , C k , which we refer to as concepts .
Revised: , C k , which we refer to as concepts .

Original: Therefore, we can think of U as the part of the world that gives rise to these concepts { C } k 1 .
Revised: Therefore, we can think of U as the part of the world that gives rise to these concepts { C } k 1 .

Original: Using this causal model, we can describe counterfactual data – data that arose under a counterfactual state of the world (right diagram in Figure 1a).
Revised: Using this causal model, we can describe counterfactual data – data that arose under a counterfactual state of the world (right diagram in Figure 1a).

Original: Our factual text is x u,v , and we use x C i ← c ′ u,v for the counterfactual text obtained by intervening on concept C i to set its value to c ′ .
Revised: Our factual text is x u,v , and we use x C i ← c ′ u,v for the counterfactual text obtained by intervening on concept C i to set its value to c ′ .

Original: The counterfactual x C i ← c ′ u,v describes the output when the value of C i is set to c ′ , all else being held equal.
Revised: The counterfactual x C i ← c ′ u,v describes the output when the value of C i is set to c ′ , all else being held equal.

Original: Approximate Counterfactuals Unfortunately, pairs like ( x u,v , x C i ← c ′ u,v ) are never observed, and thus we need strategies for creating approximate counterfactual ˜ x C i ← c ′ u,v .
Revised: Approximate Counterfactuals Unfortunately, pairs like ( x u,v , x C i ← c ′ u,v ) are never observed, and thus we need strategies for creating approximate counterfactuals ˜ x C i ← c ′ u,v .

Original: Figure 1b describes the two strategies we use in this paper.
Revised: Figure 1b describes the two strategies we use in this paper.

Original: In the human-created strategy, we rely on a crowdworker to edit x u,v to achieve a particular counterfactual goal – say, making the evaluation of the restaurant’s food negative.
Revised: In the human-created strategy, we rely on a crowdworker to edit x u,v to achieve a particular counterfactual goal – say, making the evaluation of the restaurant’s food negative.

Original: CEBaB contains an abundance of such pairs ( x u,v ,
Revised: CEBaB contains an abundance of such pairs ( x u,v ,

Original: ˜ x C i ← c ′ u,v ) .
Revised: ˜ x C i ← c ′ u,v ) .

Original: However, CEBaB is unusual in having so many human-created approximate counterfactuals, so we also explore a simpler strategy in which ˜ x C i ← c ′ u,v is sampled with the requirement that it match x u,v on all concepts and set C i to c ′ .
Revised: However, CEBaB is unusual in having so many human-created approximate counterfactuals, so we also explore a simpler strategy in which ˜ x C i ← c ′ u,v is sampled with the requirement that it match x u,v on all concepts but sets C i to c ′ .

Original: This strategy is supported in many real-world datasets – for example, the OpenTable reviews underlying CEBaB all have the needed metadata (Abraham et al., 2022). 
Revised: This strategy is supported in many real-world datasets – for example, the OpenTable reviews underlying CEBaB all have the needed metadata (Abraham et al., 2022). [Equation]

Original: CPM IN : Input-based CPM To train CPM IN , we associate a new randomly initialized token t C i ← c ′ with each unique intervention description C i ← c ′ .
Revised: [Equation] To train [Equation] we associate a new randomly initialized token t C i ← c ′ with each unique intervention description C i ← c ′ .

Original: Given a dataset of approximate counterfactual pairs ( x u,v , ˜ x C i ← c ′ u,v ) and a black-box model N , we train a new CPM IN model P under a weighted sum of the following objectives:
Revised: Given a dataset of approximate counterfactual pairs ( x u,v , ˜ x C i ← c ′ u,v ) and a black-box model N , we train a new CPM IN model P under a weighted sum of the following objectives:

Original: [Equation]
Revised: [Equation]

Original: where x u,v ; t C i ← c ′ in Eqn.
Revised: where x u,v ; t C i ← c ′ in Eqn.

Original: 2 denotes the concatenation of the factual input and the token describing the intervention.
Revised: 2 denotes the concatenation of the factual input and the token describing the intervention.

Original: CE S represents the smoothed cross-entropy loss (Hinton et al., 2015), measuring the divergence between the output logits of both models.
Revised: CE S represents the smoothed cross-entropy loss (Hinton et al., 2015), measuring the divergence between the output logits of both models.

Original: The objective in Eqn.
Revised: The objective in Eqn.

Original: 1 pushes P to predict the same output as N under conventional circumstances (Figure 1c), while Eqn. 2 pushes P to predict the counterfactual behavior of N when a descriptor of the intervention is given (Figure 1d).
Revised: 1 pushes P to predict the same output as N under conventional circumstances (Figure 1c), while Eqn. 2 pushes P to predict the counterfactual behavior of N when a descriptor of the intervention is given (Figure 1d).

Original: At inference time, approximate counterfactuals are inaccessible.
Revised: At inference time, approximate counterfactuals are inaccessible.

Original: To explain model N , we append the newly learned descriptor tokens t C i ← c ′ to a factual input, upon which P predicts a counterfactual output for an input with desired intervention description C i ← c ′ to estimate the counterfactual behavior of N .
Revised: To explain model N , we append the newly learned descriptor tokens t C i ← c ′ to a factual input, upon which P predicts a counterfactual output for this input, used to estimate the counterfactual behavior of N under this intervention.

Original: CPM HI : Hidden-state CPM Our CPM HI models are trained on the same data and with the same set of goals as CPM IN , to mimic both the factual and counterfactual behavior of N .
Revised: CPM HI : Hidden-state CPM Our CPM HI models are trained on the same data and with the same set of goals as CPM IN , to mimic both the factual and counterfactual behavior of N .

Original: The key difference is how the information about the intervention C i ← c ′ is exposed to the model.
Revised: The key difference is how the information about the intervention C i ← c ′ is exposed to the model.

Original: Specifically, we adapt
Revised: Specifically, we adapt

Original: Interchange Intervention Training (Geiger et al., 2021b) to train our CPM HI models for concept-based model explanation.
Revised: Interchange Intervention Training (Geiger et al., 2022) to train our CPM HI models for concept-based model explanation.

Original: A conventional intervention on a hidden representation H of a neural network N fixes the value of the representation H to a constant.
Revised: A conventional intervention on a hidden representation H of a neural network N fixes the value of the representation H to a constant.

Original: In an interchange intervention, we instead fix H to the value it would have been when processing a separate source input s .
Revised: In an interchange intervention, we instead fix H to the value it would have been when processing a separate source input s .

Original: The result of the interchange intervention is a new model.
Revised: The result of the interchange intervention is a new model.

Original: Formally, we describe this new model as N H ← H s , where ← is the conventional intervention operator and H s is the value of hidden representation H when processing input s .
Revised: Formally, we describe this new model as N H ← H s , where ← is the conventional intervention operator and H s is the value of hidden representation H when processing input s .

Original: Given a dataset of approximate counterfactual input pairs ( x u,v , ˜ x C i ← c ′ u,v ) and a black-box model N , we train a new CPM HI model P under a weighted sum of the previous factual objective Eqn. 1 and the following counterfactual objective:
Revised: Given a dataset of approximate counterfactual input pairs ( x u,v , ˜ x C i ← c ′ u,v ) and a black-box model N , we train a new CPM HI model P under a weighted sum of the previous factual objective (Eqn. 1) and the following counterfactual objective:

Original: [Equation]
Revised: [Equation]

Original: Here H C i are hidden states designated for concept C i .
Revised: Here H C i are hidden states designated for concept C i .

Original: In essence, we train P to fully mediate the effect of intervening on C i in the hidden representation H C
Revised: In essence, we train P to fully mediate the effect of intervening on C i in the hidden representation H C

Original: i .
Revised: i .

Original: The source input s is any input x C i
Revised: The source input s is any input x C i

Original: =
Revised: =

Original: c ′ u ′ ,v ′ that has C i = c ′ .
Revised: c ′ u ′ ,v ′ that has C i = c ′ .

Original: As P only receives information about the  intervention C i ← c ′ via the interchange intervention H C i ← H C
Revised: As P only receives information about the concept-level intervention C i ← c ′ via the interchange intervention H C i ← H C

Original: i s , the model is forced to store all causally relevant information with regard to C i in the corresponding hidden representation.
Revised: i s , the model is forced to store all causally relevant information with regard to C i in the corresponding hidden representation.

Original: This process is described in Figure 1e.
Revised: This process is described in Figure 1e.

Original: In the ideal situation,  x C i =
Revised: In the ideal situation, the source input x C i =

Original: c ′ u ′ ,v ′ and x u,v share the same value only for C i and differ on all others, so that the counterfactual signal needed for localization is pure.
Revised: c ′ u ′ ,v ′ and x u,v share the same value only for C i and differ on all others, so that the counterfactual signal needed for localization is pure.

Original: However, we do not insist on this when we sample.
Revised: However, we do not insist on this when we sample.

Original: In addition, we allow null effect pairs in which x u,v and
Revised: In addition, we allow null effect pairs in which x u,v and

Original: ˜ x C
Revised: ˜ x C

Original: i ← c ′ u,v are identical.
Revised: i ← c ′ u,v are identical.

Original: For additional details on this sampling procedure, see Appendix A.2.
Revised: For additional details on this sampling procedure, see Appendix A.2.

Original: At inference time, approximate counterfactuals are inaccessible, as before.
Revised: At inference time, approximate counterfactuals are inaccessible, as before.

Original: To explain model N with regard to intervention C i ← c ′ , we manipulate the internal states of model P by intervening on the localized representation H C i for concept C i .
Revised: To explain model N with regard to intervention C i ← c ′ , we manipulate the internal states of model P by intervening on the localized representation H C i for concept C i .

Original: To achieve this, we sample a source input x C i
Revised: To achieve this, we sample a source input x C i

Original: =
Revised: =

Original: c ′ u ′ ,v ′ from the train set as any input x that has C i =
Revised: c ′ u ′ ,v ′ from the train set as any input x that has C i =

Original: c ′ to derive H  s .
Revised: c ′ to derive H C i s .

Original:  E XPERIMENT S ETUP 4. C AUSAL E STIMATION -B ASED B ENCHMARK (CEB A B)
Revised:  E XPERIMENT S ETUP 4. C AUSAL E STIMATION -B ASED B ENCHMARK (CEB A B)

Original: CEBaB (Abraham et al., 2022) is a large benchmark of high-quality, labeled approximate counterfactual for the task of sentiment analysis on restaurant reviews.
Revised: CEBaB (Abraham et al., 2022) is a large benchmark of high-quality, labeled approximate counterfactuals for the task of sentiment analysis on restaurant reviews.

Original: The benchmark was created starting from a set of 2,299 original restaurant reviews from OpenTable.
Revised: The benchmark was created starting from a set of 2,299 original restaurant reviews from OpenTable.

Original: For each of these original reviews, approximate counterfactual examples were written by human annotators; the annotators were tasked to edit the original text to reflect a specific intervention, like ‘change the food evaluation from negative to positive’ or ‘change the service evaluation from positive to unknown’.
Revised: For each of these original reviews, approximate counterfactual examples were written by human annotators; the annotators were tasked to edit the original text to reflect a specific intervention, like ‘change the food evaluation from negative to positive’ or ‘change the service evaluation from positive to unknown’.

Original: In this way, the original reviews were expanded with approximate counterfactuals to a total of 15,089 texts.
Revised: In this way, the original reviews were expanded with approximate counterfactuals to a total of 15,089 texts.

Original: The groups of originals and corresponding approximate counterfactuals are partitioned over train, dev, and test sets.
Revised: The groups of originals and corresponding approximate counterfactuals are partitioned over train, dev, and test sets.

Original: The pairs in the development and test set are used to benchmark explanation methods.
Revised: The pairs in the development and test set are used to benchmark explanation methods.

Original: Each text in CEBaB was labeled by five crowdworkers with a 5-star sentiment score.
Revised: Each text in CEBaB was labeled by five crowdworkers with a 5-star sentiment score.

Original: In addition, each text was annotated at the concept level for four mediating concepts { C ambiance , C food , C noise , and
Revised: In addition, each text was annotated at the concept level for four mediating concepts { C ambiance , C food , C noise , and

Original: C service } , using the labels { negative , unknown , positive } , again with five crowdworkers annotating each concept-level label.
Revised: C service } , using the labels { negative , unknown , positive } , again with five crowdworkers annotating each concept-level label.

Original: We refer to Appendix A.1 and Abraham et al.
Revised: We refer to Appendix A.1 and Abraham et al.

Original: 2022 for additional details.
Revised: 2022 for additional details.

Original: As discussed above (Section 3 and Figure 1b), we consider two sources of approximate counterfactuals using CEBaB. For human-created counterfactuals, we use the edited restaurant reviews of the train set.
Revised: As discussed above (Section 3 and Figure 1b), we consider two sources of approximate counterfactuals using CEBaB. For human-created counterfactuals, we use the edited restaurant reviews of the train set.

Original: For metadata-sampled counterfactuals, we sample factual inputs from the train set that have the desired combination of mediating concepts.
Revised: For metadata-sampled counterfactuals, we sample factual inputs from the train set that have the desired combination of mediating concepts.

Original: Using all the human-created edits leads to 19,684 training pairs of factual and corresponding approximate counterfactuals.
Revised: Using all the human-created edits leads to 19,684 training pairs of factuals and corresponding approximate counterfactuals.

Original: Sampling counterfactuals leads to 74,574 pairs.
Revised: Sampling counterfactuals leads to 74,574 pairs.

Original: We use these approximated counterfactuals to train explanation methods.
Revised: We use these approximate counterfactuals to train explanation methods.

Original: Appendix A. provides more information about our pairing process.
Revised: Appendix A. provides more information about our pairing process.

Original: 4. E VALUATION M ETRICS
Revised: 4. E VALUATION M ETRICS

Original: Much of the value of a benchmark  CEBaB derives from its support for directly estimating the
Revised: Much of the value of a benchmark like CEBaB derives from its support for directly calculating the

Original:  Individual Causal Concept Effect (ICaCE) for a model N given a human-generated approximate counterfactual pair ( [Equation]
Revised: Estimated Individual Causal Concept Effect ( (cid:92) ICaCE N ) for a model N given a human-generated approximate counterfactual pair ( x u,v , ˜ x C

Original: i ← [Equation]
Revised: i ← c ′ u,v ) :

Original: [Equation]
Revised: [Equation]

Original: This is simply the difference between the vectors of output scores for the two examples.
Revised: This is simply the difference between the vectors of output scores for the two examples.

Original: We do not expect to have pairs ( x u,v , ˜ x C
Revised: We do not expect to have pairs ( x u,v , ˜ x C

Original: i ← c ′ u,v ) at inference-time, and this is what drives the development of explanation methods E N that estimate
Revised: i ← c ′ u,v ) at inference time, and this is what drives the development of explanation methods E N that estimate

Original: this quantity using only a factual input x u,v and a description of the intervention C i ← c ′ .
Revised: this quantity using only a factual input x u,v and a description of the intervention C i ← c ′ .

Original: To benchmark such methods, we follow Abraham et al.
Revised: To benchmark such methods, we follow Abraham et al.

Original: (2022) in using the ICaCE-Error:
Revised: (2022) in using the ICaCE-Error:

Original: ICaCE-Error DN ( E ) =

Original: [Equation]
Revised: [Equation]

Original: Here, we assume that D is a dataset consisting entirely of approximated counterfactual pairs ( x u,v ,
Revised: Here, we assume that D is a dataset consisting entirely of approximate counterfactual pairs ( x u,v ,

Original: ˜ x C i ← c ′ u,v ) .
Revised: ˜ x C i ← c ′ u,v ) .

Original: Dist measures the distance between the actual ICaCE  for the model N and theManuscript effect predicted by the explanation method.
Revised: Dist measures the distance between the (cid:92) ICaCE N for the model N and the effect predicted by the explanation method.

Original: Abraham et al.
Revised: Abraham et al.

Original: (2022) consider three values for Dist : cosine distance, which captures the direction of effects but not their magnitude;L2, which captures both direction and magnitude, and NormDiff, which captures magnitude but not direction.
Revised: (2022) consider three values for Dist : L2, which captures both direction and magnitude; Cosine distance, which captures the direction of effects but not their magnitude; and NormDiff (absolute difference of L2 norms), which captures magnitude but not direction.

Original: We report all three metrics.
Revised: We report all three metrics.

Original: 4. B ASELINE METHODS
Revised: 4. B ASELINE M ETHODS

Original: BEST CEBaB We compare our results with the best results obtained on the CEBaB benchmark.
Revised: BEST CEBaB We compare our results with the best results obtained on the CEBaB benchmark.

Original: Crucially, BEST CEBaB is not a single model, but rather pools together the best result obtained by any explanation method previously benchmarked on the CEBaB dataset, for every combination of model and metric.
Revised: Crucially, BEST CEBaB is not a single method, but rather pools together the best result obtained by any explanation method previously benchmarked on the CEBaB dataset, for every combination of model and metric.

Original: Additionally, we supply task performance as Macro-F1 for CEBaB as a five-class sentiment classification task for each model.

Original:  Our version of S-Learner (Künzel et al., 2019) learns to mimic the factual behavior of black-box model N while making the intermediate concepts explicit.
Revised: S-Learner Our version of S-Learner (Künzel et al., 2019) learns to mimic the factual behavior of black-box model N while making the intermediate concepts explicit.

Original: 1 Given a factual input, a finetuned BERT model B first predicts values for the intermediate concepts.
Revised: 2 Given a factual input, a finetuned BERT model B first predicts values for the intermediate concepts.

Original: Then, a logistic regression model LR N is trained to map these intermediate concept values to the factual output of black-box model N , under the following objective. 
Revised: Then, a logistic regression model LR N is trained to map these intermediate concept values to the factual output of black-box model N , under the following objective. [Equation]

Original: [Equation]
Revised: [Equation]

Original: By intervening on the intermediate predicted concept values at inference-time, we can hope to simulate the counterfactual behavior of N :
Revised: By intervening on the intermediate predicted concept values at inference-time, we can hope to simulate the counterfactual behavior of N :

Original: [Equation]
Revised: [Equation]

Original:  When using S-Learner in conjunction with approximate counterfactual inputs at train-time, we simply add this counterfactual data on top of the observational data that is typically used to train S-Learner.
Revised: [Equation] When using S-Learner in conjunction with approximate counterfactual inputs at train-time, we simply add this counterfactual data on top of the observational data that is typically used to train S-Learner.

Original: GPT-

Revised: GPT-

Original: Large language models such as GPT-3 (175B) have shown extraordinary power in terms of incontext learning (Brown et al., 2020).
Revised: Large language models such as GPT-3 (175B) have shown extraordinary power in terms of incontext learning (Brown et al., 2020).

Original:  We use GPT-3 to generate a new approximated counterfactual at test-time given a factual input and a descriptor of the intervention.
Revised: 4 We use GPT-3 to generate a new approximate counterfactual at inference time given a factual input and a descriptor of the intervention.

Original: This generated counterfactual is directly used to estimate the change in model behavior:
Revised: This generated counterfactual is directly used to estimate the change in model behavior:

Original: [Equation]
Revised: [Equation]

Original: We use our train-time approximated counterfactual inputs to construct a prompt for GPT-3 .
Revised: We use our train-time approximate counterfactual inputs to construct a prompt for GPT-3 .

Original: Given this prompt, GPT-3 outputs new approximated counterfactuals given a factual and intervention descriptor.
Revised: Given this prompt, GPT-3 outputs new approximate counterfactuals given a factual and intervention descriptor.

Original: Full details on how these prompts are constructed can be found in Appendix A.7.
Revised: Full details on how these prompts are constructed can be found in Appendix A.7.

Original: 4. C AUSAL P ROXY M ODELS
Revised: 4. C AUSAL P ROXY M ODELS

Original: We train CPMs for the publicly available models released for CEBaB, fine-tuned as five-way sentiment classifiers on the factual data.
Revised: We train CPMs for the publicly available models released for CEBaB, fine-tuned as five-way sentiment classifiers on the factual data.

Original: This includes four model architectures: bert-base-uncased ( BERT ;
Revised: This includes four model architectures: bert-base-uncased ( BERT ;

Original: Devlin et al. 2019), RoBERTa-base ( RoBERTa ; Liu et al. 2019), GPT-2 ( GPT-2 ; Radford et al. 2019), and LSTM+GloVe ( LSTM ; Hochreiter & Schmidhuber 1997; Pennington et al. 2014).
Revised: Devlin et al. 2019), RoBERTa-base ( RoBERTa ; Liu et al. 2019), GPT-2 ( GPT-2 ; Radford et al. 2019), and LSTM+GloVe ( LSTM ; Hochreiter & Schmidhuber 1997; Pennington et al. 2014).

Original: All Transformerbased models (Vaswani et al., 2017) have 12 Transformer layers.
Revised: All Transformerbased models (Vaswani et al., 2017) have 12 Transformer layers.

Revised: Before training, each CPM model is initialized with the architecture and weights of the black-box model we aim to explain.

Revised: Thus, the

Revised: CPMs are rooted in the factual behavior of N from the start.

Original: We include details about our setup in Appendix A.3.
Revised: We include details about our setup in Appendix A.3.

Original: The inference time comparisons for these models are as follows:
Revised: The inference time comparisons for these models are as follows,

Revised: where P in Eqn. 9 and Eqn.refers to the CPM model trained under CPM IN and CPM HI objectives, respectively:

Original: [Equation]
Revised: [Equation]

Original: where s is a source inputwith C i 
Revised: [Equation] [Table] source input C i =

Original: = c ′,
Revised: c ′

Original: C i is the neural representation associatedwith H C i, and
Revised: C i is the neural representation associated C i

Original: H C
Revised: H C

Original: i s is the neural representation associated with H C

Original: i for the source input s.
Revised: i s source input s

Original: For BERT and RoBERTa , we always map concepts to “[CLS]” token embeddings. For GPT-2 , we map to the padded token embeddings at the end of input sequences. For LSTM , we map to the attention-gated sentence
Revised: As H C i for BERT use of width from the 1st of the 10th layer. For RoBERTa , we use the 8th layer instead.

Original: [Table] embeddings. Forsimplicity, the intervention site for each concept has the On top of these settings, we provide extensive analyses about how different intervention locations and sizes affect our evaluation results in Appendix A.5.
Revised: For

Original: Following the guidance on IIT given by Geiger et al.
Revised: Following the guidance on IIT given by Geiger et al.

Original: (2021b), we train CPM HI with an additional multi-task objective as,
Revised: , we train CPM HI with an additional multi-task objective as,

Original: [Equation]
Revised: [Equation]

Original: where our probe is parameterized by a multilayer perceptron MLP , and H C i b is the value of hidden representation for the concept C i when processing input b .
Revised: where our probe is parameterized by a multilayer perceptron MLP , and H C i x is the value of hidden representation for the concept C i when processing input x with a concept label of c for C i .

Original:  R ESULTS
Revised:  R ESULTS

Original: We first benchmark bothvariants of CPM  on CEBaB based on the source of approximate counterfactual data during training Section 5.1, showing it to be state-of-the-art for CEBaB. We then investigate whether CPMs can be deployed both as predictor and explanation method at the same time (Section 5.2) and find that they can.
Revised: We first benchmark both CPM variants and our baseline methods on CEBaB. We show that the CPMs achieve state-of-the-art performance, for both types of approximate counterfactuals used during training (Section 5.1). Given the good factual performance achieved by CPMs, we subsequently investigate whether CPMs can be deployed both as predictor and explanation method at the same time (Section 5.2) and find that they can.

Original: Finally, we show that the localized representations of CPM HI give rise to concept-aware feature attribution methods (Section 5.3).
Revised: Finally, we show that the localized representations of CPM HI give rise to concept-aware feature attributions (Section 5.3).

Original: Our supplementary materials report on detailed ablation studies and explore the potential of our methods for model debiasing.
Revised: Our supplementary materials report on detailed ablation studies and explore the potential of our methods for model debiasing.

Original: 5. CEB A B PERFORMANCE
Revised: 5. CEB A B P ERFORMANCE

Original: Table 1 presents our main results.
Revised: Table 1 presents our main results.

Original:  Both CPM IN and CPM HI beat BEST CEBaB in every evaluation: they are not only better than any individual model evaluated in Abraham et al. 2022, they are also better than the best model for every choice of metric and pretrained starting point.
Revised: The results are grouped per approximate counterfactual type used during training. Both CPM IN and CPM HI beat BEST CEBaB in every evaluation setting by a large margin, establishing state-of-the-art explanation performance. Interestingly, CPM HI seems to slightly outperform CPM IN using sampled approximate counterfactuals, while slightly underperforming

Original:  Appendix A.6 reports on ablation studies that indicate that, for CPM HI , these results are driven primarily  by the role of IIT in localizing concepts.Both CPM IN and CPM HI also outperform all three variants of S-Learner.
Revised: CPM IN on human-created approximate counterfactuals. Appendix A.6 reports on ablation studies that indicate that, for CPM HI , this state-of-the-art performance is primarily driven by the role of IIT in localizing concepts.

Original: S-Learner is arguably the best individual model fromAbraham et al. 2022, and these findings show that CPM outperforms it even where we employ the same approximate counterfactual strategies. However, it should be noted that S-Learner gains a little from training with human counterfactuals, which suggests that a naive incorporation of counterfactual data during training can help. In addition, both of our methods perform on-par with GPT-3 across all model architectures.
Revised: S-Learner , one of the best individual explainers from the original CEBaB paper (Abraham et al., 2022), shows only a marginal improvement when naively incorporating sampled and human-created counterfactuals during training over using no counterfactuals .

Original: The results for GPT-3 highlight some interesting trade-offs.

Original: GPT-3 shows remarkable performance with human counterfactuals.

Original: This suggests it can generate truthful counterfactual examples that mimic human-created one (as shown in Appendix A.7). However, GPT-3 ’s performance drops significantly when we use sampled counterfactuals, revealing that the model is highly sensitive tonoisy prompt signals. In addition, the cost ofusing GPT-3 probably makes it infeasible as an explainer: it is a large model that is expensive to run, and we would need to run it every time an explanation for a new example was needed. Our CPM are, by comparison, extremely compact and inexpensive to use.
Revised: This indicates that the large performance gains achieved by our CPMs over previous explainers are most likely due to the explicit use of a counterfactual training signal, and not primarily due to the addition of extra (counterfactual) data.

Original: Manuscript
Revised: Manuscript

Original: [Table] [Table]
Revised: [Table] [Table]

Revised: GPT-3 occasionally performs on-par with our CPMs, generally only slightly underperforming our best explainer on human-created counterfactuals , while being significantly worse on sampled counterfactuals .

Revised: While the GPT-3 explainer also explicitly uses approximate counterfactual data, the results indicate that our proposed counterfactual mimic objectives give better results.

Revised: The better performance of CPMs when considering sampled counterfactuals over GPT-3 shows that our approach is more robust to the quality of the approximate counterfactuals used.

Revised: While the GPT-3 explainer is easy to set up (no training required), it might not be suitable for some explanation applications regardless of performance, due to the latency and cost involved in querying the GPT-3 API.

Original: Models trained with human counterfactuals aresignificantly better than those trained with sampled counterfactuals.
Revised: Across the board, explainers trained with human-created counterfactuals are better than those trained with sampled counterfactuals.

Original: This shows that the performance of explanation methods depends on the quality of the approximated counterfactual training data.
Revised: This shows that the performance of explanation methods depends on the quality of the approximate counterfactual training data.

Original: However, we can again consider the cost trade-offs: human counterfactuals  may be expensive to create, whereas sampled counterfactuals are often naturally abundant, since they depend on having only relevant metadata for approximating the underlying causal model.
Revised: While human counterfactuals give excellent performance, they may be expensive to create. Sampled counterfactuals are cheaper if the relevant metadata is available.

Original:  Finally, CPMHI and CPM IN achieve close performance, but CPM IN performs slightly better across multiple conditions. CPM IN is also the simpler of the two models to implement, and so these results may to speak in its favor.
Revised: Thus, under budgetary constraints, sampled counterfactuals may be more efficient. Finally, CPM IN is conceptually the simpler of the two CPM variants.

Original: However,CPM HI arguably brings additional benefits when it comes to explainability, as we discuss in Section 5.3 below.
Revised: However, we discuss in Section 5.3 how the localized representations of CPM HI lead to additional explainability benefits.

Original: 5. S ELF -E XPLANATION WITH CPM
Revised: 5. S ELF -E XPLANATION WITH CPM

Original: As outlined in Section 3, CPMs learn to mimic both the factual and counterfactual behaviors of the black-box models they are explaining.
Revised: As outlined in Section 3, CPMs learn to mimic both the factual and counterfactual behavior of the black-box models they are explaining.

Original: The results in Table 1 show that this procedures leads them not only to be high-quality explainers, but also to perform comparably to the black-box models they are explaining, as indicated by the Macro-F1 rows in Table 2, which show almost no difference between our models and those under BEST CEBaB .
Revised: We show in Table 2 that our CPMs achieve a factual Macro-F score comparable to the black-box finetuned models.

Original: Can we then simply drop the black-box models and use CPMs as explainers and as task-specific models?
Revised: We investigate if we can simply replace the black-box model with our CPM and use the CPM both as factual predictor and counterfactual explainer.

Original: The remaining question here is whether CPMs can explain themselves. To address this, we  simply replace the black-box model N in Eqn.
Revised: To answer this questions, we measure the self-explanation performance of CPMs by simply replacing the black-box model N in Eqn.

Original: 5 with a CPM model at inference time.
Revised: 5 with our factual CPM predictions at inference time.

Original: Table 3 reports these results.
Revised: Table 3 reports these results.

Original: we find that bothof variants, CPM IN and CPM HI, achieve better  performance providing self-explanations comparing with providing explanations for another blackbox model.
Revised: We find that both CPM IN and CPM HI achieve better self-explanation performance compared to providing explanations for another black-box model.

Original: Furthermore, CPM HI provides better self-explanation than CPM IN , suggesting our interchange intervention procedure leads the model to localize concept-based information in hidden representations.
Revised: Furthermore, CPM HI provides better self-explanation than CPM IN , suggesting our interchange intervention procedure leads the model to localize concept-based information in hidden representations.

Original: This shows that CPMs may be viable as replacements for their black-box counterpart, since they provide similar task performance while providing faithful counterfactual explanations of both the black-box model and itself.
Revised: This shows that CPMs may be viable as replacements for their black-box counterpart, since they provide similar task performance while providing faithful counterfactual explanations of both the black-box model and themselves. [Figure]

Original: 5. C ONCEPT -A WARE F EATURE A TTRIBUTION WITH CPM HI
Revised: 5. C ONCEPT -A WARE F EATURE A TTRIBUTION WITH CPM HI

Original: We have shown that CPM HI provides trustworthy explanations (Section 5.1).
Revised: We have shown that CPM HI provides trustworthy explanations (Section 5.1).

Original: We nowfurther investigate whether CPM HI learns representations that mediate the effect of different concepts.
Revised: We now investigate whether CPM HI learns representations that mediate the effects of different concepts.

Original: Specifically, we adapt Integrated Gradients (IG; Sundararajan et al. 2017) to visualize aggregated gradientsover inputs flowing through the neurons mapped to each concept.
Revised: We adapt Integrated Gradients (IG; Sundararajan et al. 2017) to provide concept-aware feature attributions, by only considering gradients flowing through the hidden representation associated with a given concept.

Original: Details of our implementation of IGcan be found in Appendix A.8.
Revised: We formalize this version of IG in Appendix A.8.

Original: [Figure] In Table 4, we compare  two variants of CPM HI ( IIT and Multi-task ) with the original black-box ( Finetuned ) model.
Revised: In Table 4, we compare concept-aware feature attibutions for two variants of CPM HI ( IIT and Multi-task ) and the original black-box ( Finetuned ) model.

Original: Specifically, we remove the multi-task objective L Multi  and  the interchange intervention objective L HI from our CPM HI respectively when training these two variants.
Revised: For IIT we remove the multi-task objective L Multi during training and for Multi-task we remove the the interchange intervention objective L HI . This helps isolate the individual effects of both losses on concept localization.

Original: The three models have vastly different distributions of word importance scores.
Revised: All three models predict a neutral final sentiment score for the considered input, but they show vastly different feature attributions.

Original: More importantly, IIT  highlights words that are semantically related to each concept.
Revised: Only IIT reliably highlights words that are semantically related to each concept.

Original: For instance, when we restrict the gradients to flow only through the intervention site of the noise concept, “loud” is the word highlighted the most that contributes negatively.
Revised: For instance, when we restrict the gradients to flow only through the intervention site of the noise concept, “loud” is the word highlighted the most that contributes negatively.

Original: On the other hand, when we consider the service concept, words like “friendly”, “waiter” are highlighted the most and contribute positively.
Revised: When we consider the service concept, words like “friendly” and “waiter” are highlighted the most as contributing positively.

Original: These meaningful variances are missing for both the Finetuned model and  the Multi-task model, which shows our training paradigm pushes the model to learn causally motivated representations over the course of training.
Revised: These contrasts are missing for representations of the Multi-task and Finetuned models. Only the IIT training paradigm pushes the model to learn causally localized representations.

Original: We include additional results in Appendix A.8.
Revised: For the service concept, we notice that the IIT model wrongfully attributes “delicious”. This could be useful for debugging purposes and could be used to highlight potential failure modes of the model.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We explored the use of approximate counterfactual training data to build more robust causal explanation methods.
Revised: We explored the use of approximate counterfactual training data to build more robust causal explanation methods.

Original: We introduced Causal Proxy Model (CPM), a method which learns to mimic both the factual and counterfactual  of a black-box model N .
Revised: We introduced Causal Proxy Models (CPMs), which learn to mimic both the factual and counterfactual behaviors of a black-box model N .

Original: Using CEBaB, a benchmark for causal concept-based explanation methods, we demonstrated that both versions of our technique (CPM IN and CPM HI ) significantly outperform previous explanation methods.
Revised: Using CEBaB, a benchmark for causal concept-based explanation methods, we demonstrated that both versions of our technique (CPM IN and CPM HI ) significantly outperform previous explanation methods.

Original: Our experimental comparisons show that this increase in explanation performance is not due to the naive incorporation of additional (counterfactual) data when training conventional explanation methods, but rather stems from the capacity of CPM models to localize causally relevant information.

Original: Interestingly, we find that our GPT-3 based explanation method achieves results on par with our best models for human-created counterfactuals.
Revised: Interestingly, we find that our GPT-3 based explanation method performs on-par with our best CPM model in some settings.

Original: While test-time use of GPT-3 as explanation model is both slow and expensive, we believe  that GPT-3 could be deployed to supplement human-annotation efforts for counterfactual data creation.
Revised: While test-time use of GPT-3 as explanation method might not be feasible, we believe this result shows that GPT-3 could be deployed to supplement human-annotation efforts for counterfactual data creation.

Original: Our results suggest that CPMs can be more than just explanation methods.
Revised: Our results suggest that CPMs can be more than just explanation methods.

Original: They achieve factual performance on par with the model they aim to explain, and they can explain their own behavior.
Revised: They achieve factual performance on par with the model they aim to explain, and they can explain their own behavior.

Original: This paves the way to using them as deployed models that both perform tasks and offer explanations.
Revised: This paves the way to using them as deployed models that both perform tasks and offer explanations.

Original: In addition, the causally localized representations of our CPM HI variants are very intuitive, as revealed by  concept-aware feature attribution techniques.
Revised: In addition, the causally localized representations of our CPM HI variant are very intuitive, as revealed by our concept-aware feature attribution technique.

Original: We believe that causal localization techniques could play a vital role in further model explanation efforts.
Revised: We believe that causal localization techniques could play a vital role in further model explanation efforts.

Original: A A PPENDIX A. CEB A B D ATASET S TATISTICS
Revised: A A PPENDIX A. CEB A B D ATASET S TATISTICS

Original: Table 5 shows dataset statistics of CEBaB. The variants of CEBaB we consider only impact the train split.
Revised: Table 5 shows dataset statistics of CEBaB. The variants of CEBaB we consider only impact the train split.

Original: The top panel shows the amount of observational samples and edits introduced in the CEBaB paper.
Revised: The top panel shows the number of observational samples and edits introduced in the CEBaB paper.

Original: The bottom panel shows our paired versions, where we create approximate counterfactual pairs.
Revised: The bottom panel shows our paired versions, where we create approximate counterfactual pairs.

Original: We explore two variants of approximate counterfactuals, human  and sampled counterfactuals as in Section 4.1.
Revised: We explore two variants of approximate counterfactuals: human -created and sampled counterfactuals (Section 4.1).

Original: The human setting considers all pairs made possible by using all data.
Revised: The human setting considers all pairs made possible by using all data.

Original: The sampling setting considers pairs sampled from only the observational data, as discussed in Section A.2.
Revised: The sampling setting considers pairs sampled from only the observational data, as discussed in Section A.2.

Original: [Table]
Revised: [Table]

Original: A. T YPES O F A PPROXIMATED C OUNTERFACTUAL P AIRS
Revised: A. T YPES OF A PPROXIMATE C OUNTERFACTUAL P AIRS

Original: Our approximated counterfactual training data comes in paired sentences of ( original sentence , approximate counterfactual sentence ).
Revised: Our approximate counterfactual training data comes in paired sentences of ( original sentence , approximate counterfactual sentence ).

Original: The approximate counterfactuals differs from their original counterparts in only one concept value.
Revised: The approximate counterfactuals differs from their original counterparts in only one concept value.

Original: We consider approximate counterfactual pairs to be symmetric, we use both ( original sentence , approximate counterfactual sentence ) and ( approximate counterfactual sentence , original sentence ) as training pairs.
Revised: We consider approximate counterfactual pairs to be symmetric: we use both ( original sentence , approximate counterfactual sentence ) and ( approximate counterfactual sentence , original sentence ) as training pairs.

Original: Human-created Counterfactuals CEBaB contains multiple counterfactual sentences for each original review.
Revised: Human-created Counterfactuals CEBaB contains multiple counterfactual sentences for each original review.

Original: To achieve this, the dataset creators asked annotators to edit the original sentence to achieve a specified goal (e.g., ‘change the evaluation of the restaurant’s food to negative’).
Revised: To achieve this, the dataset creators asked annotators to edit the original sentence to achieve a specified goal (e.g., ‘change the evaluation of the restaurant’s food to negative’).

Original: These originals and corresponding edits form our human pairs.
Revised: These originals and corresponding edits form our human pairs.

Original: Metadata-sampled Counterfactuals Human-created counterfactuals are not always available.
Revised: Metadata-sampled Counterfactuals Human-created counterfactuals are not always available.

Original: With
Revised: With

Original: CEBaB, we simulate a second type of approximated counterfactuals by using metadata-guided heuristics. For a given original sentence , we sample a counterfactual from the train set by matching concept labels while allowing only one label to be changed.
Revised: CEBaB, we simulate a second type of approximate counterfactuals by using metadata-guided heuristics: for a given original sentence , we sample a counterfactual from the train set by matching concept labels while allowing only one label to be changed.

Original: During training, we also consider null effect pairs in our sampling setup.
Revised: During training, we also consider null effect pairs in our sampling setup.

Original: These pairs resemble cases where our approximate counterfactual sentence is identical to the original sentence.
Revised: These pairs resemble cases where our approximate counterfactual sentence is identical to the original sentence.

Original: When training our models on these pairs, we expect our models to predict the same counterfactual and factual output.
Revised: When training our models on these pairs, we expect our models to predict the same counterfactual and factual output.

Original: A. T RAINING R EGIMES
Revised: A. T RAINING R EGIMES

Original: CPM IN To train CPM IN , we use the same model architecture as N , and initialize it with the model weights using weights from N .
Revised: CPM IN To train CPM IN , we use the same model architecture as N , and initialize it with the model weights using weights from N .

Original: The maximum number of training epochs is set to 30 with a learning rate of 5 e − 5 and an effective batch size of 128.
Revised: The maximum number of training epochs is set to 30 with a learning rate of 5 e − 5 and an effective batch size of 128.

Original: The learning rate linearly decays to zero over thetraining epochs.
Revised: The learning rate linearly decays to 0 over thetraining epochs.

Original: We employ an early stopping strategy for COS ICaCE over the dev set for an interval of 50 steps withan early stopping patience set to 20.
Revised: We employ an early stopping strategy for COS ICaCE over the dev set for an interval of 50 steps with early stopping patience set to 20.

Original: We set the max sequence length to 128 and the dropout rate to 0 .
Revised: We set the max sequence length to 128 and the dropout rate to 0 .

Original: We take a weighted sum of two objectives as the loss term for training CPM HI .
Revised: We take a weighted sum of two objectives as the loss term for training CPM HI .

Original: Specifically, we use [ w Mimic , w IN ] =
Revised: Specifically, we use [ w Mimic , w IN ] =

Revised: For the smoothed cross-entropy loss, we use a temperature of 2 . 0 .

Original: CPM HI To train CPM HI , we use the same model architecture as N , and initialize it with the model weights using weights from N .
Revised: CPM HI To train CPM HI , we use the same model architecture as N , and initialize it with the model weights using weights from N .

Original: The maximum number of training epochs is set to 30 with a learning rate of 8 e − 5 and an effective batch size of 256.
Revised: The maximum number of training epochs is set to 30 with a learning rate of 8 e − 5 and an effective batch size of 256.

Original: We use a higher learning rate of 0 .
Revised: We use a higher learning rate of 0 .

Original: 001 for the LSTM

Original: Manuscript

Original: [Figure] [Table] model as it enables quicker convergence.
Revised: 001 for the LSTM model as it enables quicker convergence.

Original: The learning rate linearly decays to zero over the 30 training epochs.
Revised: The learning rate linearly decays to 0 over the 30 training epochs.

Original: We employ an early stopping strategy for COS ICaCE over the dev set for an interval ofsteps withan early stopping patience set to 20.
Revised: We employ an early stopping strategy for COS ICaCE over the dev set for an interval of 10 steps with early stopping patience set to 20.

Original: We tune down the interval to 10 for GPT-2 to allow a practical run-time for training.

Original: We set the max sequence length to 128 and the dropout rate to0 .
Revised: We set the max sequence length to 128 and the dropout rate to

Original: We take a weighted sum of three objectives as the loss term for training CPM HI .
Revised: We take a weighted sum of three objectives as the loss term for training CPM HI .

Original: Specifically, we use [ w Mimic , w Multi , w HI ] =
Revised: Specifically, we use [ w Mimic , w Multi , w HI ] =

Original: In Appendix A.6, we conduct a set of ablation studies to show individual contribution from each objective.
Revised: In Appendix A.6, we conduct a set of ablation studies to isolate the individual contributions from each objective.

Revised: For the smoothed cross-entropy loss, we use a temperature of 2 . 0 .

Original: Our models are all implemented in PyTorch (Paszke et al., 2019) and using the HuggingFace library (Wolf et al., 2019).
Revised: Our models are all implemented in PyTorch (Paszke et al., 2019) and using the HuggingFace library (Wolf et al., 2019).

Revised: All of our results are aggregated over three distinct random seeds.

Original: To foster reproducibility, we will release our code repository and model artifacts to the public.
Revised: To foster reproducibility, we will release our code repository and model artifacts to the public.

Original: A. A DDITIONAL B ASELINE R ESULTS
Revised: A. A DDITIONAL B ASELINE R ESULTS

Original: Table 6 shows baselines adapted from Abraham et al.
Revised: Table 6 shows baselines adapted from Abraham et al.

Original: (2022), which shows the state-of-the-art explanation methods for the CEBaB benchmark.
Revised: (2022), which contains the present state-of-theart explanation methods for the CEBaB benchmark.

Original: We report the best scores across these explanation methods in Table 1.
Revised: We report the best scores across these explanation methods in Table 1.

Original: These baselines are trained without using counterfactual data.
Revised: These baselines are trained without using counterfactual data.

Original: Thus, we build additional baselines that use counterfactual data as shown in Table 7.
Revised: Thus, we build additional baselines that use counterfactual data as shown in Table 7.

Original: S-Learner is selected as the best performing models and included in Table 1 for comparisons.
Revised: S-Learner is selected as the best performing models and included in Table 1 for comparisons.

Revised: The equations for the additional baselines are as follows:

Original: [Table]
Revised: [Equation]

Original: classification task.
Revised: [Equation] [Equation] ) where s random is a randomly sampled training input, s approx is a training input sampled to match the concept-level labels of the true counterfactual under intervention C i ← c ′ , D C i ← c ′ is the set of all approximate counterfactual training pairs that represent a C i ← c ′ intervention, and f is a look-up function that returns the ground-truth label associated with an input. The signatures of E ATE and E CaCE N reflect that they are independent of the specific factual input x u,v considered.

Original: Lower is better. Details about these evaluation metrics can be found in Section 4. Results averaged over 3 distinct seeds; standard deviations in parentheses.
Revised: Furthermore, E ATE is independent of N given that this explainer only uses ground-truth training labels to estimate causal effects.

Original: A. I NTERVENTION S ITE L OCATION AND S IZE
Revised: A. I NTERVENTION S ITE L OCATION AND S IZE

Original: Previous work shows that neurons in different layers and groups can encode different high-level concepts (Vig et al., 2020;
Revised: Previous work shows that neurons in different layers and groups can encode different high-level concepts (Vig et al., 2020;

Original: Koh et al., 2020).
Revised: Koh et al., 2020).

Revised: CPM HI pushes concept-related information to localize

Revised: Manuscript

Revised: [Table]

Original: CPM HI pushes concept-related information to localize at the targeted intervention site (the aligned neural representations for each concept).
Revised: [Table] Manuscript [Table] at the targeted intervention site (the aligned neural representations for each concept).

Original: In this section, we investigate how the location and the size of the intervention site impact CPM HI performance.
Revised: In this section, we investigate how the location and the size of the intervention site impact CPM HI performance.

Original: We use the optimal location and size found in this study for other results presented in this paper.
Revised: We use the optimal location and size found in this study for other results presented in this paper.

Original: Location For Transformer-based models, we vary the location of the intervention site by intervening on the “[CLS]” token embedding layer l .
Revised: Location For Transformer-based models, we vary the location of the intervention site by intervening on the “[CLS]” token embedding layer l .

Original: Specifically, we set l = { 2 , 4 , 6 , 8 , 10 , 12 } .
Revised: Specifically, we set l = { 2 , 4 , 6 , 8 , 10 , 12 } .

Original: We skip this experiment for non-Transformer-based model (i.e., LSTM ) since it only contains a single sentence embedding.
Revised: We skip this experiment for non-Transformer-based model (i [Table] e., LSTM ) since it only contains a single sentence embedding.

Original: As shown in the top panel of Figure 2, intervention location significantly affects CPM HI performance.
Revised: As shown in the top panel of Figure 2, intervention location significantly affects CPM HI performance.

Original: Our results show that layer 10 for BERT , layer 8 for RoBERTa , and layer 12 for GPT-2 lead to the best performance.
Revised: Our results show that layer 10 for BERT , layer 8 for RoBERTa , and layer 12 for GPT-2 lead to the best performance.

Original: This suggests layers have different efficacy in terms of information localization.
Revised: This suggests layers have different efficacy in terms of information localization.

Original: Our results also show that intervening with deeper layers tends to provide better performance.
Revised: Our results also show that intervening with deeper layers tends to provide better performance.

Original: However, for both BERT and RoBERTa , intervening on the last layer results in a slightly worse performance comparing to earlier layers.
Revised: However, for both BERT and RoBERTa , intervening on the last layer results in a slightly worse performance compared to earlier layers.

Original: This suggests that leaving Transformer blocks after the intervention site helps localized information to be processed by the neural network.
Revised: This suggests that leaving Transformer blocks after the intervention site helps localized information to be processed by the neural network.

Original: Size For Transformer-based models, we change the size of the intervention site d c for each concept.
Revised: Size For Transformer-based models, we change the size of the intervention site d c for each concept.

Original: Specifically, we set d c = { 1 , 16 , 64 , 128 , 192 } .
Revised: Specifically, we set d c = { 1 , 16 , 64 , 128 , 192 } .

Original: For instance when d c = 1 , we use a single dimension of the “[CLS]” token embedding to represent each concept, starting from the first dimension of the vector.
Revised: For instance when d c = 1 , we use a single dimension of the “[CLS]” token embedding to represent each concept, starting from the first dimension of the vector.

Original: For non-transformer-based model (i.e., LSTM ), we intervene on the attention-gated sentence embedding whose dimension size is set to 300.
Revised: For our non-Transformer-based model ( LSTM ), we intervene on the attention-gated sentence embedding whose dimension size is set to 300.

Original: Accordingly, we set d c = { 1 , 16 , 64 , 75 } .
Revised: Accordingly, we set d c = { 1 , 16 , 64 , 75 } .

Original: As shown in Figure 2, larger intervention site leads to better performance for all Transformer-based models.
Revised: As shown in Figure 2, larger intervention sites lead to better performance for all Transformer-based models.

Original: For LSTM , we find that the optimal size is the second largest one instead.
Revised: For LSTM , we find that the optimal size is the second largest one instead.

Original: On the other hand, our results suggest that the performance gain from the increase of size diminishes as we increase the size for all model architectures. 
Revised: On the other hand, our results suggest that the performance gain from the increase of size diminishes as we increase the size for all model architectures. [Table]

Original: A. A BLATION  OF T RAINING O BJECTIVES
Revised: A. A BLATION S TUDY OF CPM HI

Original: Geiger et al.
Revised: Geiger et al.

Original: (2021b) show that training with a multi-task objective helps IIT to improve generalizability.
Revised: (2022) show that training with a multi-task objective helps IIT to improve generalizability.

Original: In this experiment, we aim to investigate whether the multi-task objective we added for CPM HI plays an important role in achieving good performance.
Revised: In this experiment, we aim to investigate whether the multi-task objective we added for CPM HI plays an important role in achieving good performance.

Original: Specifically, we conduct two ablation studies: removing the multi-task objective by setting w Multi = 0 . 0 , and removing the IIT objective by setting w IIT = 0 . 0 .
Revised: Specifically, we conduct two ablation studies: removing the multi-task objective by setting w Multi = 0 . 0 , and removing the IIT objective by setting w HI = 0 . 0 .

Original: Table 8 shows our results, which demonstrate that the IIT objective is the main factor that drives
Revised: Table 8 shows our results, which demonstrate that the IIT objective is the main factor that drives

Original: CPM HI performance.
Revised: CPM HI performance.

Original: Our results also suggest that  multi-task objective brings relatively small but consistent performance gains.
Revised: Our results also suggest that the multi-task objective brings relatively small but consistent performance gains.

Original: Overall, our findings corroborate those of Geiger et al.
Revised: Overall, our findings corroborate those of Geiger et al.

Original: (2021b)

Original: and provide concrete evidence that the combination of two objectives always results in the best-performing explanation methods across all model architectures.
Revised: and provide concrete evidence that the combination of two objectives always results in the best-performing explanation methods across all model architectures.

Original: [Table]

Original: Additionally, we explore two baselines for CPM HI .
Revised: Additionally, we explore two baselines for CPM HI .

Original: Firstly, we randomly initialize the weights of
Revised: Firstly, we randomly initialize the weights of

Original: CPM HI .
Revised: CPM HI .

Original: Secondly, we take the original black-box model as our CPM HI .
Revised: Secondly, we take the original black-box model as our CPM HI .

Original: Compared to theresults in
Revised: Compared to the results in

Original: Table 1, these two baselines fail catastrophically, suggesting the importance of our IIT paradigm.
Revised: Table 1, these two baselines fail catastrophically, suggesting the importance of our IIT paradigm.

Original: A. GPT-3 G ENERATION P ROCESS
Revised: As mentioned in Section 3, we sample a source input x C i = c ′ u ′ ,v ′ from the train set as any input x that has C i =

Original: We use the 175B parameter davinci GPT-3 model (Brown et al., 2020) as a few-shot learner to generate approximate counterfactual data.
Revised: c ′ to estimate the counterfactual output. Furthermore, we explore two additional sampling strategies. First, we create a baseline where we randomly sample a source input from the train without any concept label matching.

Original: Let x u,v be a review text with an original value c for the mediating concept C i and an overall review sentiment y (e.g., a restaurant review which is negative about the service , and felt neutral about their overall dining experience), and let c ′ be the target value ofC i , for which we would like to create a counterfactual review (e.g., change the text to become positive about the mediating concept service ). In order to train GPT-3 as an n -shot learner, we sample n = 6 example pairs of original reviews and human-created counterfactual reviews ( x u,v , ˜ x C
Revised: Second, we sample a source input from the train set using the predicted concept label of our multi-task probe, instead of the true concept label from the dataset. As shown in Table 9, the quality of our source inputs impact our performance significantly.

Original: i ← c ′ u,v ) , where the original review has the same original value c for C i and overall sentiment y , and the counterfactual review has the target value c ′ for C i . After training the GPT-3 model with the n (original review, counterfactual review) pairs, we prompt the model with the original review x u,v . We then collect the text completed by GPT-3 as the GPT-3 counterfactual review. An example for this n -shot prompt and completion is in Figure 3.
Revised: For instance, when sampling source input at random, CPM HI fails catastrophically for all evaluation metrics.

Original: Additionally, we add an initial prompt in the form of “Make the following restaurant reviews include c ′ mentions of C j .”, where c ′ is expressed as one of {“POSITIVE”, “NEGATIVE”, “NOT” } (“NOT” corresponds to making the review be unknown regarding the concept C j ) and C j as one of {“AMBIANCE”, “FOOD”, “NOISE”, “SERVICE”}.

Original: We sample using a temperature of 0.9, without any frequency or presence penalties (since we expect the counterfactual review to be similar to the original review). In preliminary experimentation, we found that capitalizing the mediating concept and target value results and inserting line breaks between examples made for better completions, although there is room for future research in this area. We used the OpenAI API to access GPT-3 . At the current price rate of $0.02 per 1K tokens, the total cost of creating an n = 6 -shot prompt for the CeBAB test set (around 4k examples) was around $50.
Revised: On the other hand, when we sampling source based on the predicted labels using the multi-task probe, CPM HI maintains its performance.

Original: A. I NTEGRATED G RADIENTS
Revised: A. GPT-3 G ENERATION P ROCESS

Original: We adapt the Integrated Gradients (IG) method of Sundararajan et al. (2017) to qualitatively assess whether CPM HI learned explainable representations of mediated concepts at its intervention site.
Revised: We use the 175B parameter davinci GPT-3 model (Brown et al., 2020) as a few-shot learner to generate approximate counterfactual data.

Original: The
Revised: Let x u,v be a review text with an original value c for the mediating concept C i and an overall review sentiment y (e.g., a restaurant review which is negative about the service , and felt neutral about their overall dining experience), and let c ′ be the target value of C i , for which we would like to create a counterfactual review (e.g., change the text to become positive about the mediating concept service ).

Original: IG algorithm computes the average gradient from the model output to its input by incrementally interpolating from a “blank” input x  ′ (consisting only of “[PAD]” tokens) to the original input x .
Revised: In order to use GPT-3 as an n -shot learner, we sample n = 6 approximate counterfactual pairs ( x u ′ ,v ′ , ˜ x C i ← c ′ u ′ ,v ′ ) , where x u ′ ,v ′ shares with x u,v the same

Original: Make the following restaurant reviews include POSITIVE mentions of SERVICE.
Revised: Make the following restaurant reviews include POSITIVE mentions of SERVICE.

Original: Original: I had two casual dinners at State & Lake and three lunches.
Revised: Original: I had two casual dinners at State & Lake and three lunches.

Original: The food was great but the service was lacking.
Revised: The food was great but the service was lacking.

Original: Everything was delicious.
Revised: Everything was delicious.

Original: The interior is questionable, but not intrusive.
Revised: The interior is questionable, but not intrusive.

Original: POSITIVE mentions of SERVICE:
Revised: POSITIVE mentions of SERVICE:

Original: I had two casual dinners at State & Lake and three lunches.
Revised: I had two casual dinners at State & Lake and three lunches.

Original: The food and the service were always great.
Revised: The food and the service were always great.

Original: Everything was delicious.
Revised: Everything was delicious.

Original: The interior is questionable, but not intrusive.
Revised: The interior is questionable, but not intrusive.

Original: Original: Food was excellent, but the service was not very attentive.
Revised: Original: Food was excellent, but the service was not very attentive.

Original: Noise level was extremely high due to close proximity of tables and poor acoustics.
Revised: Noise level was extremely high due to close proximity of tables and poor acoustics.

Original: POSITIVE mentions of SERVICE:
Revised: POSITIVE mentions of SERVICE:

Original: Food and service was excellent.
Revised: Food and service was excellent.

Original: Noise level was extremely high due to close proximity of tables and poor acoustics.
Revised: Noise level was extremely high due to close proximity of tables and poor acoustics.

Original: Original: Great food, poor and very snobbish service.
Revised: Original: Great food, poor and very snobbish service.

Original: POSITIVE mentions of SERVICE:
Revised: POSITIVE mentions of SERVICE:

Original: Great food, very good service.
Revised: Great food, very good service.

Original: Original:
Revised: Original:

Original: My dining experince was excellent!
Revised: My dining experince was excellent!

Original: However, the server was not nice.
Revised: However, the server was not nice.

Original: POSITIVE mentions of SERVICE:
Revised: POSITIVE mentions of SERVICE:

Original: My dining experince was excellent!
Revised: My dining experince was excellent!

Original: Original: Hae been here a few times and it is just okay - Entrees and wine list a bit pricey for what it is, inattentive staff.
Revised: Original: Hae been here a few times and it is just okay - Entrees and wine list a bit pricey for what it is, inattentive staff.

Original: POSITIVE mentions of SERVICE:
Revised: POSITIVE mentions of SERVICE:

Original: Hae been here a few times and it is just okay - Entrees and wine list a bit pricey for what it is.
Revised: Hae been here a few times and it is just okay - Entrees and wine list a bit pricey for what it is.

Original: Food comes out on time.
Revised: Food comes out on time.

Original: Original: Tables fairly close together, mushroom appetiser very good, pork entree fair, chicken good.
Revised: Original: Tables fairly close together, mushroom appetiser very good, pork entree fair, chicken good.

Original: The service was terrible.
Revised: The service was terrible.

Original: POSITIVE mentions of SERVICE: Tables fairly close together, mushroom appetiser very good, pork entree fair, chicken good.
Revised: POSITIVE mentions of SERVICE: Tables fairly close together, mushroom appetiser very good, pork entree fair, chicken good.

Original: The service was great however.
Revised: The service was great however.

Original: Original: Service was very poor with the server unresponsive and misinformed on all requests.
Revised: Original: Service was very poor with the server unresponsive and misinformed on all requests.

Original: The food was very good with a good selection of entrees.
Revised: The food was very good with a good selection of entrees.

Original: The ambiance was romantic with a quiet excellence.
Revised: The ambiance was romantic with a quiet excellence.

Original: POSITIVE mentions of SERVICE:
Revised: POSITIVE mentions of SERVICE:

Original: Service was very good with the server attentive and responsive on all requests.
Revised: Service was very good with the server attentive and responsive on all requests.

Original: The food was very good with a good selection of entrees.
Revised: The food was very good with a good selection of entrees.

Original: The ambiance was romantic with a quiet excellence.
Revised: The ambiance was romantic with a quiet excellence.

Original: same overall sentiment ( y = neutral), and that the counterfactual examples are all edited such that the sentiment towards service is the same ( c ′ = positive).
Revised: same overall sentiment ( y = neutral), and that the counterfactual examples are all edited such that the sentiment towards service is the same ( c ′ = positive).

Revised: value c for C i and the same overall sentiment, and the counterfactual review ˜ x C

Revised: i ← c ′ u ′ ,v ′ has the target value c ′ for C i .

Revised: We prompt the model with these pairs, and we also include the original review x u,v .

Revised: We then collect the text completed by GPT-3 as the GPT-3 counterfactual review.

Original: Eqn.
Revised: An example for this n -shot prompt and completion is in Figure 3. In addition, we also prompt GPT-3 with pairs of original reviews and metadata-sampled counterfactuals, and generate another set of GPT-3 counterfactual review for comparison. We sample n = 4 approximate counterfactual pairs in this case. An example of metadata-sampled counterfactual generation with GPT-3 can be seen in Figure 4.

Revised: Make the following restaurant reviews include POSITIVE mentions of SERVICE.

Revised: Original: Been here several times.

Revised: Always a winner, except for the tasteless food!

Revised: POSITIVE mentions of SERVICE:

Revised: I was very disappointed in the food but we did not wait long for each course and or waiter was very pleasant.

Revised: Original: food was decent but not great.

Revised: POSITIVE mentions of SERVICE:

Revised: Lovely evening - good service and wonderful food.

Revised: Perfect for fresh fish fans

Revised: Original:

Revised: The restaurant was empty when we arrived, reservation not necessary?

Revised: Wine list limited.

Revised: Food was bland, presentation was very well done.

Revised: I would not eat here again.

Revised: POSITIVE mentions of SERVICE:

Revised: Abby provided the best service that we’ve had after probably two dozen visits.

Revised: No thank you for making the risotto cake at lunch....

Revised: Two Stars!

Revised: Original:

Revised: A terrible place for lunch or dinner.

Revised: All the food is excellent

Revised: with top notch ingredients

Revised: POSITIVE mentions of SERVICE:

Revised: Excellent Valentine’s menu.

Revised: Excellent service and food.

Revised: Would recommend this restaurant and will return.

Revised: Original:

Revised: The food was average for the cost.

Revised: My husband and I were so excited to visit Bobby Flay’s restraunt and were really disappointed.

Revised: The food was average at best.

Revised: POSITIVE mentions of SERVICE:

Revised: The service was amazing and the food was alright.

Revised: For each few-shot learning prompt, we insert an initial string of the form of “Make the following restaurant reviews include c ′ mentions of C i .”, where c ′ is expressed as one of {“POSITIVE”, “NEGATIVE”, “NOT” } (“NOT” corresponds to making the review be unknown regarding the concept C i ) and

Revised: C i is one of {“AMBIANCE”, “FOOD”, “NOISE”, “SERVICE”}.

Revised: We sample using a temperature of 0.9, without any frequency or presence penalties (since we expect the counterfactual review to be similar to the original review).

Revised: In preliminary experimentation, we found that capitalizing the mediating concept and target value results and inserting line breaks between examples made for better completions, although there is room for future research in this area.

Revised: We used the OpenAI API to access GPT-3 .

Revised: At the current price rate of $0.02 per 1,000 tokens, the total cost of creating our counterfactuals (around 4,000 examples) was approximately $50 per approximate counterfactuals creation strategy.

Original: 12 is the integrated gradients equation originally proposed in Sundararajan et al.
Revised: We adapt the Integrated Gradients (IG) method of Sundararajan et al.

Original: (2017), applied to a CPM  model P on input x . 
Revised: (2017) to qualitatively assess whether CPM HI learned explainable representations of mediated concepts at its intervention sites. The IG algorithm computes the average gradient from the model output to its input by incrementally interpolating from a “blank” input x ′ (consisting only of “[PAD]” tokens) to the original input x . Eqn. 16 is the integrated gradients equation originally proposed in Sundararajan et al. (2017), applied

Original: [Equation]
Revised: [Figure]

Original: Where ∂ P ( x ) ∂x i is the derivative of P on the i th dimension of x .
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: In our implementation of IG
Revised: to a CPM model P on input x .

Original: [Table]
Revised: [Equation]

Original:  we wish to show the per-token attribution of input x on the model’s final output P ( x ) , mediated by the hidden representation of a concept in P .
Revised: Here, ∂ P ( x ) ∂x j is the derivative of P on the j th dimension of x . In our implementation of IG, we wish to show the per-token attribution of input x on the model’s final output P ( x ) , mediated by the hidden representation of a concept in P .

Original: That is, we’d like to ask, “What is the effect of the word ‘delicious’ in the input on the model’s output, when we restrict our focus only on the model’s representation of the concept food ?”
Revised: That is, we’d like to ask, “What is the effect of the word ‘delicious’ in the input on the model’s output, when we restrict our focus only on the model’s representation of the concept food ?”

Original: To answer this question, we compute the gradient of the model output P ( x ) with respect to the input x, but restricting the gradient to flow through the intervention site for a particular concept.
Revised: To answer this question, we compute the gradient of the model output P ( x ) with respect to the input x but restrict the gradient to flow through the intervention site for a particular concept.

Original: This allows us to capture the per-token attribution of the model’s final output (i.e. whether particular words contributed to a positive , negative , or neutral sentiment prediction), mediated by the concept that is represented by the specified intervention site.
Revised: This allows us to capture the per-token attribution of the model’s final output (whether particular words contributed to a positive , negative , or neutral sentiment prediction), mediated by the concept that is represented by the specified intervention site.

Original: For example, in Table 4, we can see that “declicious” has a positive attribution to the output of the model, when we focus on its representation of the concept food .
Revised: For example, in Table 4, we can see that “delicious” has a positive attribution to the output of the model when we focus on its representation of the concept food .

Original: Formally, consider a trained CPM model P , an input x and mediating concept C i .
Revised: Formally, consider a trained CPM model P , an input x and mediating concept C i .

Original: Let H C
Revised: Let H C

Original: i be the activation of P, when run on x , at the intervention site for C i – meaning that H C i should consist of the model’s hidden representation for the concept C i in its computation.
Revised: i be the activation of P at the intervention site for C i .

Original: We define the gradient of P ( x ) along dimension i mediated by C i  as
Revised: We define the gradient of P ( x ) along dimension j , mediated by C i , as

Original: [Equation]
Revised: [Equation]

Original: Note that if we were to sum ∂ P ( x ) ∂x i mediated by each concept C

Original: i , we would get the whole derivative of P ( x ) with respect to x .

Original: Hence, Eqn. 13 restricts the gradient to only flow through the hidden representation of the concept along which we’d like to interpret our model.
Revised: Eqn. 17 restricts the gradient to only flow through the hidden representation of the concept along which we’d like to interpret our model.

Original: We apply Eqn. 13 in order to compute the gradient ∂ P ( x ) ∂x i in the integrated gradients formula, Eqn. 12.
Revised: We integrate these mediated gradients over a straight path between input x and baseline x ′ , analogous to Eqn. 16.

Original: We implement our IG method using CaptumAI library.
Revised: We implement our IG method using CaptumAI library.

Original: 4 We use the default parameters for our runs with number of iterations set to 50, and we set the integral method as gausslegendre .
Revised: 5 We use the default parameters for our runs with number of iterations set to 50, and we set the integral method as gausslegendre .

Original: We set the multiply-by-inputs flag to True .
Revised: We set the multiply-by-inputs flag to True .

Revised: To visualize individual word importance, we conduct z -score normalization of attribution scores over input tokens per each concept, and then linearly scale scores between [ − 1 , +1 ].

Original: Table 9 shows an example attribution using IG on CPM HI supplied with additional ablation studies on our training objectives.
Revised: Table 10 extends Table 4 in our main text with additional ablation studies on our training objectives.

Original: A. M ODEL D EBIASING
Revised: A. M ODEL D EBIASING

Original: Being able to accurately predict outputs for counterfactual input enables explanation methods to faithfully debias  desired concept.
Revised: Being able to accurately predict outputs for counterfactual inputs enables explanation methods to faithfully debias a model with regard to a desired concept.

Original: For instance, with CEBaB, debiasing a concept (e.g., “food”) is equivalent to estimating the causal effect when we remove a concept from that concept from the input, or when we set the concept label for a concept to be “unknown”.
Revised: For instance, with CEBaB, debiasing a concept (e.g., “food”) is equivalent to estimating the counterfactual output when we set the concept label for a concept to be unknown .

Original: In this section, we briefly study the extent to whcih our method can function as a debiasing method.
Revised: In this section, we briefly study the extent to which the CPM HI can function as a debiasing method.

Original: To debias a concept, we enforceManuscript the sampled source input s as in Eqn.
Revised: To debias a concept, we enforce the sampled source input s as in Eqn.

Original: 3 to have unknown as its concept label for the concept to be debiased.
Revised: 3 to have unknown as its concept label for the concept to be debiased.

Original: To show our methods can faithfully debias a targeted concept, we evaluate the correlations between the predicted overall sentiment label for sentences and the concept labels for each concept.
Revised: To show our methods can faithfully debias a targeted concept, we evaluate the correlations between the predicted overall sentiment label for sentences and the concept labels for each concept.

Original: Without any debiasing technique, we expect concept labels to be highly correlated with the overall sentiment label (e.g., if food is positive, it is more likely that the overall sentiment is positive).
Revised: Without any debiasing technique, we expect concept labels to be highly correlated with the overall sentiment label (e.g., if food is positive, it is more likely that the overall sentiment is positive).

Original: We use CPM HI trained for the BERT model architecture as an example, and use examples in the test set.
Revised: We use CPM HI trained for the BERT model architecture as an example, and use examples in the test set.

Original: Figure 4 shows correlation plots for the original Finetuned model as well as CPM HI .
Revised: Figure 5 shows correlation plots for the original Finetuned model as well as CPM HI .

Original: As expected, the correlation of the food concept is weakness through the debiasing pipeline by 57.50%.
Revised: As expected, the correlation of the food concept is weakened through the debiasing pipeline by 57.50%.

Original: Our results also suggest that correlations of other concepts are affected which suggest for some future research direction to minimize the impact of the debiasing pipeline on irrelevant concepts.
Revised: Our results also suggest that correlations of other concepts are affected, which suggests a future research direction focused on minimizing the impact of the debiasing pipeline on irrelevant concepts.

Original: We include results for the remaining concepts in the Appendix A.9.
Revised: We include results for the remaining concepts in the Appendix A.9.

Original: Figure 4a to Figure 4d show debiasing visualizations for three concepts including ambiance , noise and service .
Revised: Figure 5a to Figure 5d show debiasing visualizations for three concepts: ambiance , noise and service .

Original: We use a CPM HI for the BERT model architecture as an example.
Revised: We use a CPM HI for the BERT model architecture as an example.

Original: We calculate the distributions with examples in the test set.
Revised: We calculate the distributions with examples in the test set.

Original: A. L EARNING D YNAMICS
Revised: A. L EARNING D YNAMICS

Original: Figure 5 shows three different metrics measured on the dev and the test sets for a CPM HI trained for the BERT model architecture as an example.
Revised: Figure 6 shows three different metrics measured on the dev and the test sets for a CPM HI trained for the BERT model architecture as an example.

Original: Since we use COS ICaCE on the dev set to early stop our training process, we find our CPM HI reaches  local minimum on COS ICaCE while L2 ICaCE or
Revised: Since we use COS ICaCE on the dev set to early stop our training process, we find our CPM HI reaches a local minimum on COS ICaCE while L2 ICaCE and

Original: NormDiff ICaCE is still in a downward trend.
Revised: NormDiff ICaCE are still trending downward.

Original: This suggests future research may need to choose desired metrics to optimize with during training for early stopping to reach the best performing model.
Revised: This suggests future research may need to choose desired metrics to optimize for during training, for early stopping to reach the best performing model.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original:  [Figure] 
Revised: ] the [Table] [Figure] [Table]

Original:  [Table]
Revised: [Figure] [Table] [Figure] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Revised: Different from Table 4 and Table 10, which show the visualizations of our optimized model, we show a per-epoch result for for CPM HI , followed with our best model appended at the end.

Original: Target classes we consider are { positive , very positive } summed scores are taken importance is normalized to [ − 1 , +1 ]. − 1 means the word contributes the most negatively to predicting the target class. +1 means the word contributes the most positively to predicting the 
Revised: Our results suggest that early checkpoints in the training process focus at drastically different input words comparing to later checkpoints, though all models predict neutral for this given sentence. In addition, gradient aggregations over input words are rather stable towards the end the training.

Original: Manuscript
Revised: Manuscript

Original: [Figure]
Revised: [Figure]

Original: aggregations over input words are rather stable towards the end the training.

Original: More importantly, CPM HI learns how to highlight words that are semantically related to each concept gradually.
Revised: More importantly, CPM HI learns how to highlight words that are semantically related to each concept gradually.

Original: For instance, we can see a clear trend of emphasising the word “decorations” for the ambiance concept throughout the training process.
Revised: For instance, we can see a clear trend of emphasising the word “decorations” for the ambiance concept throughout the training process.

Original: This suggests that our training procedure induces causally motivated gradients over input words gradually through the training process.
Revised: This suggests that our training procedure induces causally motivated gradients over input words gradually through the training process.