Original: On Causal Rationalization
Revised: On Causal Rationalization

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: With recent advances in natural language processing, rationalization becomes anessential self-explaining diagram to disentangle the black box by selecting a subset of input texts to account for the major variation in prediction.
Revised: With recent advances in natural language processing, rationalization becomes an essential self-explaining diagram to disentangle the black box by selecting a subset of input texts to account for the major variation in prediction.

Original: Yet, existingassociation-based approaches on rationalization cannot identify true rationaleswhen two or more rationales are highly intercorrelated, and thus provide a similarcontribution to prediction accuracy, so-called spuriousness.
Revised: Yet, existing association-based approaches on rationalization cannot identify true rationales when two or more rationales are highly intercorrelated, and thus provide a similar contribution to prediction accuracy, so-called spuriousness.

Original: To address this limitation, we novelly leverage two causal desiderata, non-spuriousness and efficiency,into rationalization from the causal inference perspective.
Revised: To address this limitation, we novelly leverage two causal desiderata, non-spuriousness and efficiency, into rationalization from the causal inference perspective.

Original: We formally define theprobability of causation in the rationale model with its identification established asthe main component of learning necessary and sufficient rationales.
Revised: We formally define the probability of causation in the rationale model with its identification established as the main component of learning necessary and sufficient rationales.

Original: The superiorperformance of our causal rationalization is demonstrated on real-world review andmedical datasets with extensive experiments compared to state-of-the-art methods.
Revised: The superior performance of our causal rationalization is demonstrated on real-world review and medical datasets with extensive experiments compared to state-of-the-art methods.

Original:  Introduction
Revised:  Introduction

Original: Recently, large language models (LLMs) have drawn increasing attention and have been widelyused in extensive Natural Language Processing (NLP) tasks [see e.g., 22, 6].
Revised: Recently, large language models (LLMs) have drawn increasing attention and have been widely used in extensive Natural Language Processing (NLP) tasks [see e.g., 22, 6].

Original: Although those deeplearning based models could provide incredibly great performance, it remains a daunting task infinding trustworthy explanations to interpret these models’ behavior, which is particularly criticalin high-stakes applications such as healthcare and finance.
Revised: Although those deep learning based models could provide incredibly great performance, it remains a daunting task in finding trustworthy explanations to interpret these models’ behavior, which is particularly critical in high-stakes applications such as healthcare and finance.

Original: In healthcare, longitudinal electronichealth records (EHR) are increasingly used to forecast patients’ disease progression and assistin clinicians’ decision making.
Revised: In healthcare, longitudinal electronic health records (EHR) are increasingly used to forecast patients’ disease progression and assist in clinicians’ decision making.

Original: Beyond simply predicting clinical outcomes, doctors are moreinterested in understanding the decision making process of predictive models thereby building trust,as well as extracting clinically meaningful and relevant insights [24].
Revised: Beyond simply predicting clinical outcomes, doctors are more interested in understanding the decision making process of predictive models thereby building trust, as well as extracting clinically meaningful and relevant insights [24].

Original: For example, when developingcomputational phenotyping algorithms (the process of identifying patients from EHR with certainphenotypic characteristics), sequential data mining methods are popularly used to extract temporaltext sequences from EHR data that are working as abstracts or milestones of patient medical journey[16, 37].
Revised: For example, when developing computational phenotyping algorithms (the process of identifying patients from EHR with certain phenotypic characteristics), sequential data mining methods are popularly used to extract temporal text sequences from EHR data that are working as abstracts or milestones of patient medical journey

Original: Such temporal sequences can oftentimes better characterize patient medical conditions, asraw EHR data are by nature heterogeneous, sparse, and noisy.
Revised: Such temporal sequences can oftentimes better characterize patient medical conditions, as raw EHR data are by nature heterogeneous, sparse, and noisy.

Original: Disentangling the black box in deep neural networks is a notoriously challenging task [1].
Revised: Disentangling the black box in deep neural networks is a notoriously challenging task [1].

Original: There are alot of research works focused on providing trustworthy explanations for models, generally classifiedinto post-hoc techniques and self-explaining models [12, 29].
Revised: There are a lot of research works focused on providing trustworthy explanations for models, generally classified into post-hoc techniques and self-explaining models [12, 29].

Original: To provide better model interpretation,self-explaining models are of greater interest, and selective rationalization is one popular type of sucha model by highlighting important tokens among input texts [see e.g., 14, 27, 21, 33, 33, 7].
Revised: To provide better model interpretation, self-explaining models are of greater interest, and selective rationalization is one popular type of such a model by highlighting important tokens among input texts [see e.g., 14, 27, 21, 34, 34, 7].

Original: Thegeneral framework of selective rationalization as shown in Figure 1a consists of two components,a selector and a predictor.
Revised: The general framework of selective rationalization as shown in Figure 1a consists of two components, a selector and a predictor.

Original: Those selected tokens by the trained selector are called rationales andthey are required to provide similar prediction accuracy to the full input text based on the trainedpredictor.
Revised: Those selected tokens by the trained selector are called rationales and they are required to provide similar prediction accuracy to the full input text based on the trained predictor.

Original: Besides, they should reflect the model’s true reasoning process (faithfulness) [14, 21]and provide a convenient explanation to people (plausibility) [14, 7].
Revised: Besides, they should reflect the model’s true reasoning process (faithfulness) [14, 21] and provide a convenient explanation to people (plausibility) [14, 7].

Original: Most of the existing works[23, 4, 27, 9] mainly focus on finding rationales ( Z ) which can achieve the best prediction accuracyfor the outcome of interest ( Y ) based on input texts as X , and thus is association-based models.[Figure]
Revised: Most of the existing works [23, 4, 27, 9] mainly focus on finding rationales ( Z ) which can achieve the best prediction accuracy for the outcome of interest ( Y ) based on input texts as X , and thus is association-based models.

Original: The major limitation of these association-based works (also see related works in Appendix A.1)lies in identifying the true rationales when two or more rationales are highly inter-correlated, andthus provide a similar contribution to prediction accuracy.
Revised: The major limitation of these association-based works (also see related works in Appendix A.1) lies in identifying the true rationales when two or more rationales are highly inter-correlated, and thus provide a similar contribution to prediction accuracy.

Original: Here, the true rationale, which later wewill formally define as the causal rationale, is the true sufficient rationale for the prediction withoutspurious information that may be related to the outcome of interest but does not indeed cause theoutcome.
Revised: Here, the true rationale, which later we will formally define as the causal rationale, is the true sufficient rationale for the prediction without spurious information that may be related to the outcome of interest but does not indeed cause the outcome.

Original: As one example shown in Figure 1b, one chunk of beer review comments covers twoaspects, aroma and palate.
Revised: As one example shown in Figure 1b, one chunk of beer review comments covers two aspects, aroma and palate.

Original: The reviewer didn’t feel it was a good beer in terms of aroma and assigneda low score to it, which can be regarded as negative sentiment.
Revised: The reviewer didn’t feel it was a good beer in terms of aroma and assigned a low score to it, which can be regarded as negative sentiment.

Original: Hence, if we want to explain why thereviewer provided the negative sentiment, the causal interpretation should be the texts highlighted inyellow.
Revised: Hence, if we want to explain why the reviewer provided the negative sentiment, the causal interpretation should be the texts highlighted in yellow.

Original: However, it can be seen that the texts in terms of the palate, highlighted by green, can providesimilar prediction accuracy as the spurious information that is highly correlated with the previoustrue rationale identified.
Revised: However, it can be seen that the texts in terms of the palate, highlighted by green, can provide similar prediction accuracy as the spurious information that is highly correlated with the previous true rationale identified.

Original: Therefore, we can’t distinguish these two selected snippets when they allhave high predictive power.
Revised: Therefore, we can’t distinguish these two selected snippets when they all have high predictive power.

Original: Also sometimes spurious correlations are introduced during training,which misleads the selector to select non-causally related features [9].
Revised: Also sometimes spurious correlations are introduced during training, which misleads the selector to select non-causally related features [9].

Original: Furthermore, if spuriousfeatures are selected, the predictor will tend to fail once there is a large discrepancy between thetraining and testing data distributions and it would be nearly impossible to achieve high generalizationperformance [31].
Revised: Furthermore, if spurious features are selected, the predictor will tend to fail once there is a large discrepancy between the training and testing data distributions and it would be nearly impossible to achieve high generalization performance [31].

Original: In this paper, we propose a novel approach called causal rationalization aiming to find causalinterpretations for general NLP tasks.
Revised: In this paper, we propose a novel approach called causal rationalization aiming to find causal interpretations for general NLP tasks.

Original: Beyond selecting rationales based on purely predictionperformance, our goal is to identify rationales with causal meanings.
Revised: Beyond selecting rationales based on purely prediction performance, our goal is to identify rationales with causal meanings.

Original: To achieve this goal, weintroduce a novel concept as causal rationales by leveraging two causal desiderata: non-spuriousnessand efficiency.
Revised: To achieve this goal, we introduce a novel concept as causal rationales by leveraging two causal desiderata: non-spuriousness and efficiency.

Original: This is motivated by a recent work [35] in representation learning, where the twodesiderata are adopted in supervised learning tasks.
Revised: This is motivated by a recent work [36] in representation learning, where the two desiderata are adopted in supervised learning tasks.

Original: Here non-spuriousness means a representationcan capture features causally determining the outcome, and efficiency means only essential and noredundant features are included.
Revised: Here non-spuriousness means a representation can capture features causally determining the outcome, and efficiency means only essential and no redundant features are included.

Original: Our main contributions are summarized as follows.
Revised: Our main contributions are summarized as follows.

Original: • We formally define the probability of causation (POC) for rationales accounting for nonspuriousness and efficiency, and establish identifications of those counterfactual quantities.
Revised: • We formally define the probability of causation (POC) for rationales accounting for nonspuriousness and efficiency, and establish identifications of those counterfactual quantities.

Original: • We propose a practically useful algorithm for learning necessary and sufficient rationales thatsimultaneously estimates POC from observed data and trains the causal rational model, where the existing approach [35] is not applicable.
Revised: • We propose a practically useful algorithm for learning necessary and sufficient rationales that simultaneously estimates POC from observed data and trains the causal rational model, where the existing approach [36] is not applicable.

Original: • The superior performance of our causality-based rationalization method is demonstrated in application on Beer review and Geographic Atrophy (electronic health records) with extensive experimentscompared to state-of-the-art methods.
Revised: • The superior performance of our causality-based rationalization method is demonstrated in application on Beer review and Geographic Atrophy (electronic health records) with extensive experiments compared to state-of-the-art methods.

Original:  Framework
Revised:  Framework

Original: Denote X
Revised: Denote X

Original: =
Revised: =

Original: ( X 1 , · · · , X d ) as the text with d tokens, Z = ( Z 1 , · · · , Z d ) as the correspondingselection where Z i is a Bernoulli random variable indicating whether the token i is selected in therationale and Y as the binary label.
Revised: ( X 1 , · · · , X d ) as the text with d tokens, Z = ( Z 1 , · · · , Z d ) as the corresponding selection where Z i is a Bernoulli random variable indicating whether the token i is selected in the rationale and Y as the binary label.

Original: Let Y ( X = x , Z = z ) denote the potential value of Y when setting X as x , Z as z .
Revised: Let Y ( X = x , Z = z ) denote the potential value of Y when setting X as x , Z as z .

Original: We generalize three standard assumptions from causal inference literature[28, 34, 20] to rationalization, including consistency, ignorability, and monotonicity.
Revised: We generalize three standard assumptions from causal inference literature [28, 35, 20] to rationalization, including consistency, ignorability, and monotonicity.

Original: More detailscan be found in Appendix A.2.
Revised: More details can be found in Appendix A.2.

Original: A structural causal model (SCM) is defined with a directed acyclicgraph (DAG) where nodes are variables and edges represent causal relationships between variables.
Revised: A structural causal model (SCM) is defined with a directed acyclic [Figure] graph (DAG) where nodes are variables and edges represent causal relationships between variables.

Original: In this paper, we propose a SCM for rationalization as follows with its DAG shown in Figure 2:
Revised: In this paper, we propose a SCM for rationalization as follows with its DAG shown in Figure 2:

Original: [Equation]
Revised: [Equation]

Original: where N X , N Y , N Z are exogenous variables and f, g, h are the causal mechanisms of X , Z , Yrespectively, with ⊙ denoting the element-wise product.
Revised: where N X , N Y , N Z are exogenous variables and f, g, h are the causal mechanisms of X , Z , Y respectively, with ⊙ denoting the element-wise product.

Original: Suppose we observe a data point with thetext X and binary selections Z , rationales can be represented by the event {
Revised: Suppose we observe a data point with the text X and binary selections Z , rationales can be represented by the event {

Original: X i I
Revised: X i I

Original: ( Z i = 1) } 1 ≤ i ≤ d ,where I ( Z i = 1) indicates if the i -th token is selected, X i is the corresponding text, and d is thelength of the text.
Revised: ( Z i = 1) } 1 ≤ i ≤ d , where I ( Z i = 1) indicates if the i -th token is selected, X i is the corresponding text, and d is the length of the text.

Original: Remark 2.1.
Revised: Remark 2.1.

Original: The data generation process in (1) matches many graphical models in previous work[see e.g., 11, 27].
Revised: The data generation process in (1) matches many graphical models in previous work [see e.g., 11, 27].

Original: As a motivating example, consider the sentiment labeling process for the Beerreview data.
Revised: As a motivating example, consider the sentiment labeling process for the Beer review data.

Original: The labeler first locates all the important sub-sentences or words which encode sentimentinformation and marks them.
Revised: The labeler first locates all the important sub-sentences or words which encode sentiment information and marks them.

Original: After reading all the reviews, the labeler then goes back to the previouslymarked text and makes the final judgment on sentiment.
Revised: After reading all the reviews, the labeler then goes back to the previously marked text and makes the final judgment on sentiment.

Original: In this process, we can regard the first stepof marking important locations of words as generating the selection of Z through reading texts X .
Revised: In this process, we can regard the first step of marking important locations of words as generating the selection of Z through reading texts X .

Original: The second step can be seen as combining the selected locations with raw text to generate rationales(equivalent to Z ⊙ X ) and then the label Y is generated through a complex decision function h .
Revised: The second step can be seen as combining the selected locations with raw text to generate rationales (equivalent to Z ⊙ X ) and then the label Y is generated through a complex decision function h .

Original: Next, we extend the probability of causation [28, 35] to rationales.
Revised: Next, we extend the probability of causation [28, 36] to rationales.

Original: Define the probability of sufficiency( PS ) for rationales as PS { Z = z , X = x } ,Y = y ≜ P ( Y ( Z = z , X = x ) =
Revised: Define the probability of sufficiency ( PS ) for rationales as PS { Z = z , X = x } ,Y = y ≜ P ( Y ( Z = z , X = x ) =

Original: y | Z 
Revised: y | Z ̸

Original: =z ,
Revised: =

Original:  Y  = y, X = x ) ,which is the probability of the potential label with rationales { Xi
Revised: z , Y ̸ = y, X = x ) , which is the probability of the potential label with rationales { X

Original:  I ( Z i = 1) } 1 ≤ i ≤ d .
Revised: i I ( Z i = 1) } 1 ≤ i ≤ d .

Original: It can be seen asthe capacity of the rationales to “produce” the label.
Revised: It can be seen as the capacity of the rationales to “produce” the label.

Original: Efficient rationales shall have a high PS.
Revised: Efficient rationales shall have a high PS.

Original: Onthe other hand, the probability of necessity ( PN ) for rationales is defined by PN { Z = z , X = x } ,Y = y ≜
Revised: On the other hand, the probability of necessity ( PN ) for rationales is defined by PN { Z = z , X = x } ,Y = y ≜

Original: P ( Y ( Z ̸ =
Revised: P ( Y ( Z ̸ =

Original: z , X = x ) ̸ = y )
Revised: z , X = x ) ̸ = y )

Original: | Z
Revised: | Z

Original: = z ,
Revised: = z ,

Original: Y = y, X = x ) , which is the probability of the rationales{
Revised: Y = y, X = x ) , which is the probability of the rationales {

Original: X i I
Revised: X i I

Original: ( Z i = 1) } 1 ≤ i ≤ d being a necessary cause of the label
Revised: ( Z i = 1) } 1 ≤ i ≤ d being a necessary cause of the label

Original: I { Y = y } .
Revised: I { Y = y } .

Original: Non-spurious rationales shallhave a high PN .
Revised: Non-spurious rationales shall have a high PN .

Original: To achieve both non-spuriousness and efficiency as the desired causal rationales, wefurther define the probability of sufficiency and necessity as follows.
Revised: To achieve both non-spuriousness and efficiency as the desired causal rationales, we further define the probability of sufficiency and necessity as follows.

Original: Definition 2.1.
Revised: Definition 2.1.

Original: Probability of necessity and sufficiency ( PNS ) for rationales:
Revised: Probability of necessity and sufficiency ( PNS ) for rationales:

Original: [Equation]
Revised: [Equation]

Original: When both the selection Z and the label Y are univariate binary and X is removed, then our defined
Revised: When both the selection Z and the label Y are univariate binary and X is removed, then our defined

Original: PN , PS , and PNS boil-down into the classical definitions of POC in Definition 9.2.1 to Definition9.2.3 of Pearl (2000) [28]), respectively.
Revised: PN , PS , and PNS boil-down into the classical definitions of POC in Definition 9.2.1 to Definition 9.2.3 of Pearl (2000) [28]), respectively.

Original: In addition, our definitions imply that the input texts are fixedand the probabilities of causation are mainly detected through interventions on selected rationales.
Revised: In addition, our definitions imply that the input texts are fixed [Figure] and the probabilities of causation are mainly detected through interventions on selected rationales.

Original: This not only reflects the data generation process we proposed in Model (1), but also distinguishes oursettings with existing causal frameworks [see e.g., 28, 35].
Revised: This not only reflects the data generation process we proposed in Model (1), but also distinguishes our settings with existing causal frameworks [see e.g., 28, 36].

Original: Since most times we are more interestedin efficiency and non-spuriousness of each single rationale, we extend the work of Wang and Jordan(2021)
Revised: Since most times we are more interested in efficiency and non-spuriousness of each single rationale, we extend the work of Wang and Jordan (2021)

Original: [35] to further generalize PNS to a conditional version with respect to the selections.
Revised: [36] to further generalize PNS to a conditional version with respect to the selections.

Original: Definition 2.2.
Revised: Definition 2.2.

Original: Conditional probability of necessity and sufficiency ( CPNS ) for j th selection:
Revised: Conditional probability of necessity and sufficiency ( CPNS ) for j th selection:

Original: [Equation]
Revised: [Equation]

Original: Duo to the unobserved counterfactual events in the observational study, we need to identify thosecounterfactuals as statistically estimated quantities.
Revised: Duo to the unobserved counterfactual events in the observational study, we need to identify those counterfactuals as statistically estimated quantities.

Original: To this end, we extend the results in Theorem9.2.14 of Pearl (2000) [28] for rationalization as follows, proved in Appendix A.3.
Revised: To this end, we extend the results in Theorem 9.2.14 of Pearl (2000) [28] for rationalization as follows, proved in Appendix A.3.

Original: Theorem 2.1.
Revised: Theorem 2.1.

Original: Suppose the causal graph in Figure 2 and assumptions in (4) , (5) , and (6) hold, then
Revised: Suppose the causal graph in Figure 2 and assumptions in (4) , (5) , and (6) hold, then

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: [Figure]

Original: The above definition and calculation mainly focus on a single data point.
Revised: The above definition and calculation mainly focus on a single data point.

Original: In practice, we are ofteninterested in selecting necessary and sufficient rationales for the whole dataset.
Revised: In practice, we are often interested in selecting necessary and sufficient rationales for the whole dataset.

Original: Hence, we define anoverall CPNS with a log transformation over the selected rationales as follows.
Revised: Hence, we define an overall CPNS with a log transformation over the selected rationales as follows.

Original: Definition 2.3.
Revised: Definition 2.3.

Original: The log overall CPNS over rationales of the whole dataset is defined as:
Revised: The log overall CPNS over rationales of the whole dataset is defined as:

Original: [Equation]
Revised: [Equation]

Original: where n is sample size, r i is the index set for the selected rationales of the i th data point, and | r i | isthe number of selected rationales.
Revised: where n is sample size, r i is the index set for the selected rationales of the i th data point, and | r i | is the number of selected rationales.

Original: The above definition is utilized as one component of our objective function in our main algorithm.
Revised: The above definition is utilized as one component of our objective function in our main algorithm.

Original: Toy Example of Using POC.
Revised: Toy Example of Using POC.

Original: We further use a toy example to demonstrate why CPNS is usefulfor out-of-distribution (OOD) generalization.
Revised: We further use a toy example to demonstrate why CPNS is useful for out-of-distribution (OOD) generalization.

Original: Suppose there is a dataset of sequences, where eachsequence can be represented as X
Revised: Suppose there is a dataset of sequences, where each sequence can be represented as X

Original: =
Revised: =

Original: ( X 1 , . . .
Revised: ( X 1 , . . .

Original: , X l ) with a equal length as l = 4 and a binary label
Revised: , X l ) with a equal length as l = 4 and a binary label

Original: Y .
Revised: Y .

Original: Here, we set { X 1 , X 2 } are true rationale features and { X 3 , X 4 } are irrelevant/spurious features.
Revised: Here, we set { X 1 , X 2 } are true rationale features and { X 3 , X 4 } are irrelevant/spurious features.

Original: Details of simulation design are in Appendix A.4.
Revised: Details of simulation design are in Appendix A.4.

Original: The results of CPNS and accuracy measured in
Revised: The results of CPNS and accuracy measured in

Original: OOD and in-distribution (IND) test datasets are shown in Figure 3 over 10 replications.
Revised: OOD and in-distribution (IND) test datasets are shown in Figure 3 over 10 replications.

Original: It can be seenthat true rationales ( { X 1 , X 2 } ) yield the highest scores of CPNS and accuracy inthe OOD setting,and the highest log CPNS with high accuracy inthe IND setting.
Revised: It can be seen that true rationales ( { X 1 , X 2 } ) yield the highest scores of CPNS and accuracy in OOD setting, and the highest log CPNS with high accuracy in IND setting.

Original: This motivates us to identify truerationales by maxmizing the score of CPNS , as detailed in the next section.
Revised: This motivates us to identify true rationales by maximizing the score of CPNS , as detailed in the next section.

Original:  Learning Necessary and Sufficient Rationale
Revised:  Learning Necessary and Sufficient Rationale

Original: In this section, we propose to learn necessary and sufficient rationales through the proposedlog OCPNS .
Revised: In this section, we propose to learn necessary and sufficient rationales through the proposed log OCPNS .

Original: Our model framework consists of a selector g θ ( · ) and a predictor h ϕ ( · ) as standardin the traditional rationalization approach, where θ and ϕ denote their parameters.
Revised: Our model framework consists of a selector g θ ( · ) and a predictor h ϕ ( · ) as standard in the traditional rationalization approach, where θ and ϕ denote their parameters.

Original: We can get theselection Z
Revised: We can get the selection Z

Original: = g θ ( X ) and fed it into predictor to get Y =h ϕ
Revised: = g θ ( X ) and fed it into predictor to get Y =

Original:  ( Z ⊙ X ) as shown in Figure 2 and
Revised: h ϕ ( Z ⊙ X ) as shown in Figure 2 and

Original: Figure 4.
Revised: Figure 4.

Original: One main difference between causal rationale and original rationale is that, we generate a series of counterfactual selections by flipping each dimension of the selection Z we obtained fromthe selector.
Revised: One main difference between causal rationale and original rationale is that, we generate a series of counterfactual selections by flipping each dimension of the selection Z we obtained from the selector.

Original: Then we fed raw rationale with new counterfactual rationale into our predictor to makepredictions.
Revised: Then we fed raw rationale with new counterfactual rationale into our predictor to make predictions.

Original: Considering the considerable cost of obtaining reliable rationale annotations from human,we only focus on unsupervised settings.
Revised: Considering the considerable cost of obtaining reliable rationale annotations from human, we only focus on unsupervised settings.

Original: Our goal is to make the selector create selection and rationalewith the property of necessity and sufficiency and our predictor can simultaneously provide accuratepredictions given such rationale.
Revised: Our goal is to make the selector create selection and rationale with the property of necessity and sufficiency and our predictor can simultaneously provide accurate predictions given such rationale.

Original: Hence we expect P ( Y =
Revised: Hence we expect P ( Y =

Original: y
Revised: y

Original: i
Revised: i

Original: | Z j = z i,j , Z − j
Revised: | Z j = z i,j , Z − j

Original: =
Revised: =

Original: z
Revised: z

Original: i, − j , X = x i )to be large while P( Y = y i | Z i,j ̸
Revised: i, − j , X = x i ) to be large while P( Y = y i | Z i,j ̸

Original: =
Revised: =

Original: z i,j , Z i, − j
Revised: z i,j , Z i, − j

Original: =
Revised: =

Original: z
Revised: z

Original: i, − j , X = x i ) to be small.
Revised: i, − j , X = x i ) to be small.

Original: Hence their difference, i.e., the empirical estimation of OCPNS, will be large to reflect the necessary and sufficientcomponents.
Revised: Hence their difference, i.e., the empirical estimation of OCPNS, will be large to reflect the necessary and sufficient components.

Original: Notice that the conditional probabilities here are all estimated by the same predictor.
Revised: Notice that the conditional probabilities here are all estimated by the same predictor.

Original: The reason is that our causal constraints encourage necessary and sufficient rationales, which can bereflected through the prediction.
Revised: The reason is that our causal constraints encourage necessary and sufficient rationales, which can be reflected through the prediction.

Original: For simplicity, we only consider the case where Y is a binary label.
Revised: For simplicity, we only consider the case where Y is a binary label.

Original: Challenge of Directly Optimizing OCPNS.
Revised: Challenge of Directly Optimizing OCPNS.

Original: The proposed OCPNS can be seen as a weighted sumof the difference of two estimated probabilities; there are two potential challenges if we want todirectly optimize it.
Revised: The proposed OCPNS can be seen as a weighted sum of the difference of two estimated probabilities; there are two potential challenges if we want to directly optimize it.

Original: Firstly, many current deep learning classification networks are poorly calibratedmeaning that the estimated probability of the predicted label is overestimated hence those scoresare not reliable probability estimation [13, 25, 19].
Revised: Firstly, many current deep learning classification networks are poorly calibrated meaning that the estimated probability of the predicted label is overestimated hence those scores are not reliable probability estimation [13, 25, 19].

Original: Optimizing OCPNS during training adds morerandomness and doesn’t achieve our goal.
Revised: Optimizing OCPNS during training adds more randomness and doesn’t achieve our goal.

Original: Secondly, the difference of two probabilities sometimescan be negative when our monotonicity assumption is violated during initial training steps.
Revised: Secondly, the difference of two probabilities sometimes can be negative when our monotonicity assumption is violated during initial training steps.

Original: In thiscase, we can’t take the logarithm of OCPNS.
Revised: In this case, we can’t take the logarithm of OCPNS.

Original: To address these computational issues, we consider thefollowing objective function to simultaneously estimate POC from observed data and train the causalrational model that yields high prediction accuracy,
Revised: To address these computational issues, we consider the following objective function to simultaneously estimate POC from observed data and train the causal rational model that yields high prediction accuracy,

Original: [Equation]
Revised: [Equation]

Original: i =1 L ( y i ,h ϕ ( z  i  ⊙ x i ))
Revised: ,h ϕ ( z ′ i ( j ) ⊙ x i ))

Original: + µ   n (cid:88)

Original: i =

Original: 1 L ( y i ,h ϕ ( z ′ i ⊙ x i ))

Original: − n (cid:88) i =

Original: r i (cid:88)

Original: j =

Original: L ( y i ,h ϕ ( z ′ i ( j ) ⊙ x i ))

Original: r i   (cid:124) (cid:123)(cid:122) (cid:125)
Revised: r i   (cid:124) (cid:123)(cid:122) (cid:125)

Original: CausalityConstraint + λ n (cid:88) i =
Revised: CausalityConstraint + λ n (cid:88) i =

Original: 1 δ ( z i ) , (3) where L ( · , · ) defined as the cross-entropy loss, δ ( · ) is the sparsity penalty to control sparsenessof rationales, λ and µ are the tuning parameters, z i and z ′
Revised: 1 δ ( z i ) , (3) where L ( · , · ) defined as the cross-entropy loss, δ ( · ) is the sparsity penalty to control sparseness of rationales, λ and µ are the tuning parameters, z i and z ′

Original: i are obtained through g θ ( x i ) but underdifferent modes.
Revised: i are obtained through g θ ( x i ) but under different modes.

Original: More specifically, z i is obtained under usual training mode (Gumble-Softmaxsampling) and z ′ i is obtained under test mode (top k % selection).
Revised: More specifically, z i is obtained under usual training mode (Gumble-Softmax sampling) and z ′ i is obtained under test mode (top k % selection).

Original: And z ′ i ( j ) is obtained by flippingthe j th dimension of z ′ i .
Revised: And z ′ i ( j ) is obtained by flipping the j th dimension of z ′ i .

Revised: Considering CPNS is the difference of two probability, to maximize it, we would like to increase the first probability and decrease the second one.

Revised: Hence we can transform the difference of two probabilities of two log negative loss likelihood.

Revised: It can be seen that this objective function aims to achieve the same goal as OCPNS.

Original: The algorithm we developed to solve (3) is summarized in Algorithm 1.
Revised: The algorithm we developed to solve (3) is summarized in Algorithm 1.

Original: Algorithm 1 Causal Rationale
Revised: Beer Review Data. The Beer review dataset [26] is a multi-aspect sentiment analysis dataset with token-level rationales annotated by domain experts. We use its appearance, aroma, and palate aspects.

Original: Require:
Revised: Require:

Original: Training dataset D
Revised: Training dataset D

Original: = { ( x i , y i ) } Ni =1 , where x i ∈ R d , learning rate α
Revised: = { ( x i , y i ) } Ni =1 , where x i ∈ R d , learning rate α

Original: Initialize the parameters of selector g θ ( · ) , predictor h ϕ ( · ) , where θ and ϕ denote the parameters while not converge do
Revised: Initialize the parameters of selector g θ ( · ) , predictor h ϕ ( · ) , where θ and ϕ denote the parameters while not converge do

Original: Sample a batch { ( x i , y i ) } ni =1 from D
Revised: Sample a batch { ( x i , y i ) } ni =1 from D

Original: Generate selections S =
Revised: Generate selections S =

Original: { z i } ni =1 through Gumble-Softmax sampling
Revised: { z i } ni =1 through Gumbel-Softmax sampling

Original: Generate selections ˆ S =
Revised: Generate selections ˆ S =

Original: { z ′ i } ni =1 through top k % selection for i = 1 , . . .
Revised: { z ′ i } ni =1 through top k % selection for i = 1 , . . .

Original: , n do for j = 1 , . . .
Revised: , n do for j = 1 , . . .

Original: , r i do
Revised: , r i do

Original: Generate counterfactual selections z
Revised: Generate counterfactual selections z

Original: i
Revised: i

Original: ( j ) by flipping the rationale at the j th dimension of the initial selection
Revised: ( j ) by flipping the rationale at the j th dimension of the initial selection

Original: z ′
Revised: z ′

Original: i end for end for
Revised: i end for end for

Original: Get a new batch of selections ˜ S
Revised: Get a new batch of selections ˜ S

Original: = { z ′ i ( j ) } i = n,j = d i =1 ,j = r i and set S all = S (cid:83) ˆ S (cid:83) ˜ S
Revised: = { z ′ i ( j ) } i = n,j = d i =1 ,j = r i and set S all = S (cid:83) ˆ S (cid:83) ˜ S

Original: Compute L via Eq(3) by using S all and D
Revised: Compute L via Eq(3) by using S all and D

Original: Update parameters θ = θ − α ∇ θ L ; ϕ
Revised: Update parameters θ = θ − α ∇ θ L ; ϕ

Original: = ϕ − α ∇ ϕ L end while
Revised: = ϕ − α ∇ ϕ L end while

Original: 4.1 Datasets Beer Review Data. The Beer review dataset [26] is a multi-aspect sentiment analysis dataset withtoken-level rationales annotated by domain experts.
Revised: [Table]

Original: We use its appearance, aroma, and palate aspects.
Revised: [Table]

Original: Geographic Atrophy (GA) Data.
Revised: Geographic Atrophy (GA) Data.

Original: The GA data is the advanced form of age-related maculardegeneration which affects more than 5 million people worldwide and can lead to irreversible loss of[Table] [Table] vision if left untreated.
Revised: The GA data is the advanced form of age-related macular degeneration which affects more than 5 million people worldwide and can lead to irreversible loss of vision if left untreated.

Original: Early identification of patients who are potentially developing into GA is thuscritical.
Revised: Early identification of patients who are potentially developing into GA is thus critical.

Original: More details about datasets and pre-processing procedures can be found in Appendix A.5.1.
Revised: More details about datasets and pre-processing procedures can be found in Appendix A.5.1.

Original: 4.2 Synthetic Experiments
Revised: 4.2 Synthetic Experiments

Original: We compare our method with VIB [27] (benchmark) and the full-text model (i [Table] e., using the full textto predict the label) by two synthetic experiments using the Beer review data.
Revised: We compare our method with VIB [27] (benchmark) and the full-text model (i.e., using the full text to predict the label) by two synthetic experiments using the Beer review data.

Original: We utilize the samesparse constraint in VIB, and thus the comparison is also an ablation study to verify the usefulness ofour causal module.
Revised: We utilize the same sparse constraint in VIB, and thus the comparison is also an ablation study to verify the usefulness of our causal module.

Original: Implementation details are provided in Appendix A.6.
Revised: Implementation details are provided in Appendix A.6.

Original: Beer-Spurious .
Revised: Beer-Spurious .

Original: We include strong spurious correlation into the dataset by randomly appendingspurious punctuation at the beginning of the sentence.
Revised: We include strong spurious correlation into the dataset by randomly appending spurious punctuation at the beginning of the sentence.

Original: See details in Appendix A.5.2.
Revised: See details in Appendix A.5.2.

Original: Table 1summarizes the results.
Revised: Tablesummarizes the results.

Original: It can be seen that our causal rationalization achieves a comparably goodperformance compared to the full-text model, and more importantly, performs consistently betterthan VIB w.r.t.
Revised: It can be seen that our causal rationalization achieves a comparably good performance compared to the full-text model, and more importantly, performs consistently better than VIB w.r.t.

Original: prediction accuracy and selecting rationales by human annotations.
Revised: prediction accuracy and selecting rationales by human annotations.

Original: Thus, our methodis more robust when handling spurious correlation and shows better generalization performance.
Revised: Thus, our method is more robust when handling spurious correlation and shows better generalization performance.

Original: Beer-Causal .
Revised: Beer-Causal .

Original: In the second setting, we test whether our method can find true causal rationales, byrandomly appending causal punctuation at the beginning of the sentence.
Revised: In the second setting, we test whether our method can find true causal rationales, by randomly appending causal punctuation at the beginning of the sentence.

Original: See details in Appendix
Revised: See details in Appendix

Original: A.5.2.
Revised: A.5.2.

Original: As summarized in Table 2, our causal rationalization obtains both higher accuracy and F1score than VIB, similar to results in the Beer-Spurious setting.
Revised: As summarized in Table 2, our causal rationalization obtains both higher accuracy and F score than VIB, similar to results in the Beer-Spurious setting.

Original: This indicates our approach can bettercapture human annotated rationales and appended causal rationales. 
Revised: This indicates our approach can better capture human annotated rationales and appended causal rationales. [Table] [Table]

Original: 4.3 Real Data Experiments
Revised: 4.3 Real Data Experiments

Original: We evaluate our method on two real datasets, Beer review and GA data.
Revised: We evaluate our method on two real datasets, Beer review and GA data.

Original: Our method achieves aconsistently better performance than VIB on all aspects of Beer review and GA data.
Revised: Our method achieves a consistently better performance than VIB on all aspects of Beer review and GA data.

Revised: For Beer review dataset, especially in causal and noise scenarios, our method can not only identify causal tokens and is also robust to avoid spurious selections which matches our expectation that causal rationalization would be more robust.

Original: In addition,for GA, it can be seen that rationalization approaches could provide better prediction performancethan using  the full-text model.
Revised: In addition, for GA, it can be seen that rationalization approaches could provide better prediction performance than using either the baseline model or the full-text model.

Original: We further examine the generated rationales on GA patients withrationalization methods.
Revised: We further examine the generated rationales on GA patients with rationalization methods.

Original: We observe that our causal rationalization could provide better clinicalmeaningful explanations, with detailed examples provided in Appendix A.5.3.
Revised: We observe that our causal rationalization could provide better clinical meaningful explanations, with detailed examples provided in Appendix A.5.3.

Original: In summary, our method performs uniformly better than the benchmark on the Beer review data in allcases.
Revised: In summary, our method performs uniformly better than the benchmark on the Beer review data in all cases.

Original: For the GA data, our method shows a comparably better performance as VIB and providesmore clinically meaningful interpretations.
Revised: For the GA data, our method shows a comparably better performance as VIB and provides more clinically meaningful interpretations.

Original: These justify the value of our causal rationalization.
Revised: These justify the value of our causal rationalization.

Revised:  Conclusion

Revised: In this work

Original: [Table]
Revised: [Table]

Revised: we propose a novel rationalization approach to find causal interpretations for sentiment analysis tasks.

Original: [Table]
Revised: We formally define non-spuriousness and efficiency of rationales from causal inference perspective and propose a practically useful algorithm. Moreover we show superior performance of our causality-based rationalization method real datasets with extensive experiments compared to state-of-the-art methods. One limitation of our method is that we make the monotonicity assumption, which is strong and can always be violated in real data. For future work, we would try to estimate the lower bound of CPNS to remove this assumption.

Revised: Another promising direction is extending causal rationales to free-text rationale, which may require adjusting the events of probability of causation to align with the generation task.

Original: A AppendixA.1 Related Work
Revised: A Appendix A.1 Related Work

Original: Selective Rationalization.
Revised: Selective Rationalization.

Original: Selective rationalization was firstly introduced in [23] and now becomesan important model for interpretability, especially in the NLP domain [3, 4, 38, 8, 14, 9, 27, 21, 33,33, 10, 7].
Revised: Selective rationalization was firstly introduced in [23] and now becomes an important model for interpretability, especially in the NLP domain [3, 4, 39, 8, 14, 9, 27, 21, 34, 34, 10, 7].

Original: Yu et al.
Revised: Yu et al.

Original: [38] proposed an introspective model consists of 3-players to addresschallenges under the co-operative setting with only the selector and the predictor.
Revised: [39] proposed an introspective model consists of 3-players to address challenges under the co-operative setting with only the selector and the predictor.

Original: Chang et al.
Revised: Chang et al.

Original: (2019)[8] extended the model to extract class-wise explanations.
Revised: [8] extended the model to extract class-wise explanations.

Original: During the training phase, the initialdesign of the framework was not end-to-end because the sampling from these selectors were notdifferentiable.
Revised: During the training phase, the initial design of the framework was not end-to-end because the sampling from these selectors were not differentiable.

Original: To address this issue, some later works adopted differentiable sampling, like gumblesoftmax, other rumble-softmax, or other reparameterization tricks [see e.g., 4, 32, 18].
Revised: To address this issue, some later works adopted differentiable sampling, like gumblesoftmax, other rumble-softmax, or other reparameterization tricks [see e.g., 4, 33, 18].

Original: Paranjapeal.
Revised: Paranjape al.

Original: (2020) [27] derived a sparsity-inducing objective by using Information Bottleneck and thus canexplicit control sparsity of the selected rationales.
Revised: (2020) [27] derived a sparsity-inducing objective by using Information Bottleneck and thus can explicit control sparsity of the selected rationales.

Original: One recent work [9] aimed to find the invariantrationales across multiple environments.
Revised: One recent work [9] aimed to find the invariant rationales across multiple environments.

Original: One thing to notice is that  sufficiency and necessity ofrationales were mentioned in some previous work [14, 7].
Revised: One thing to notice is that causal definitions, including sufficiency and necessity of features were mentioned in some previous work [32, 14, 7].

Original: However, those definitions were mainlyproposed from an empirical perspective which might lack theoretical guarantees.
Revised: However, those definitions were mainly proposed from an empirical perspective which might lack theoretical guarantees.

Original: Instead, we focuson more rigorous definitions of sufficiency and necessity from the angle of causal inference.
Revised: Instead, we focus on more rigorous definitions of sufficiency and necessity from the angle of causal inference.

Original: Sufficiency and Necessity with Interpretable Machine Learning (XAI).
Revised: Sufficiency and Necessity with Interpretable Machine Learning (XAI).

Original: Sufficiency and necessitycan be regarded as the fundamentals of XAI since they are the building blocks of all successfulexplanations [36].
Revised: Sufficiency and necessity can be regarded as the fundamentals of XAI since they are the building blocks of all successful explanations [37].

Original: Recently, many researchers have started to incorporate these properties into theirmodels.
Revised: Recently, many researchers have started to incorporate these properties into their models.

Original: Ribeiro et al.
Revised: Ribeiro et al.

Original: [30] proposed Anchors to find features that are sufficient to preserve currentclassification results.
Revised: [30] proposed Anchors to find features that are sufficient to preserve current classification results.

Original: Dhurandhar et al.
Revised: Dhurandhar et al.

Original: [15] developed an autoencoder framework to find pertinentpositive (sufficient) and pertinent negative (non-necessary) features which can preserve the currentresults.
Revised: [15] developed an autoencoder framework to find pertinent positive (sufficient) and pertinent negative (non-necessary) features which can preserve the current results.

Original: Zhang et al.
Revised: Zhang et al.

Original: [40] considered an approach to explain a neural network by generating minimal,stable, and symbolic corrections to change model outputs.
Revised: [41] considered an approach to explain a neural network by generating minimal, stable, and symbolic corrections to change model outputs.

Original: Yet, sufficiency and necessity shown inabove methods are not defined from a causal perspective.
Revised: Yet, sufficiency and necessity shown in above methods are not defined from a causal perspective.

Original: There are a few works [2, 17, 36, 5] definethe two properties with causal notations by using Pearl’s POC to provide causal interpretations.
Revised: There are a few works [2, 17, 37, 5] define the two properties with causal notations by using Pearl’s POC to provide causal interpretations.

Original: Butthey are only designed for post-hoc analysis rather than developing a new model.
Revised: But they are only designed for post-hoc analysis rather than developing a new model.

Original: Wang and Jordan(2021) [35] is the most relevant work to us to the best of knowledge.
Revised: Wang and Jordan (2021) [36] is the most relevant work to us to the best of knowledge.

Original: In their work, they formallydefined sufficiency and necessity for learned representation with POC and showed identification ofthose counterfactual quantities.
Revised: In their work, they formally defined sufficiency and necessity for learned representation with POC and showed identification of those counterfactual quantities.

Original: Finally, they incorporated those causal inspired constraints into thetraining process to obtain non-spurious and efficient representations.
Revised: Finally, they incorporated those causal inspired constraints into the training process to obtain non-spurious and efficient representations.

Original: In our work, we generalize theseconcepts to rationalization and enable our selected rationales to satisfy the desiderata.
Revised: In our work, we generalize these concepts to rationalization and enable our selected rationales to satisfy the desiderata.

Original: A.2 Assumptions
Revised: A.2 Assumptions

Original: Assumption A.1.
Revised: Assumption A.1.

Original: • Consistency:
Revised: • Consistency:

Original: [Equation]
Revised: [Equation]

Original: • Ignorability:
Revised: • Ignorability:

Original: [Equation]
Revised: [Equation]

Original: • Monotonicity (with X , Y is monotonic relative to Z i ):
Revised: • Monotonicity (with X , Y is monotonic relative to Z i ):

Original: [Equation]
Revised: [Equation]

Original: where ∧ is the logical operation AND.
Revised: where ∧ is the logical operation AND.

Original: For Two events A and B , A ∧ B =
Revised: For Two events A and B , A ∧ B =

Original: True if A = B=
Revised: True if A = B

Original:  True, A ∧ B =
Revised: = True, A ∧ B =

Original: False otherwise.
Revised: False otherwise.

Original: Here, the first assumption implies that the potential label of text under the given selection is the labelthat would be observed for that text.
Revised: Here, the first assumption implies that the potential label of text under the given selection is the label that would be observed for that text.

Original: The second assumption in causal inference usually means nounmeasured confounders, which is automatically satisfied under randomized trials.
Revised: The second assumption in causal inference usually means no unmeasured confounders, which is automatically satisfied under randomized trials.

Original: For observationalstudies, we rely on domain experts to include as many features as possible to guarantee this assumption.
Revised: For observational studies, we rely on domain experts to include as many features as possible to guarantee this assumption.

Original: In rationalization, it means 
Revised: In rationalization, it means our text already contains all information. For monotonicity assumption, it indicates that a change on the wrong selection will not, under any circumstance, make Y change to the true label. In other words, that true selection will increase the

Original: A.3 Proof of Theorem 2.1 
Revised: A.3 Proof of Theorem 2.

Original: Proof: Let’s denote the logical operators and , or as ∧ , ∨ , respectively.
Revised: Proof: Let’s denote the logical operators and , or as ∧ , ∨ , respectively.

Original: Firstly we know
Revised: Firstly we know

Original: [Equation]
Revised: [Equation]

Original: Then we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: where we use the monotonicity assumption in (6).
Revised: where we use the monotonicity assumption in (6) [Equation]

Original: Also, we know
Revised: Also [Equation] we know

Original: [Equation]
Revised: [Equation]

Original: Then we can get
Revised: Then we can get

Original: [Equation]
Revised: [Equation]

Original: Based on the consistency assumption in (4), we either have { Y ( Z j ̸ =
Revised: Based on the consistency assumption in (4), we either have { Y ( Z j ̸ =

Original: z j , Z − j =
Revised: z j , Z − j =

Original: z − j , X = x ) =
Revised: z − j , X = x ) =

Original: y }or { Y ( Z j ̸ =
Revised: y } or { Y ( Z j ̸ =

Original: z j , Z − j =
Revised: z j , Z − j =

Original: z − j , X = x ) ̸ = y } holds.
Revised: z − j , X = x ) ̸ = y } holds.

Original: Therefore, we know the two events in the lastline of (10) are disjoint and further take the probability on both sides to get:
Revised: Therefore, we know the two events in the last line of (10) are disjoint and further take the probability on both sides to get:

Original: [Equation]
Revised: [Equation]

Original: where the last term is exactly CPNS { Z j
Revised: where the last term is exactly CPNS { Z j

Original: = z j , X = x } ,Y =
Revised: = z j , X = x } ,Y =

Original: y | Z − j =
Revised: y | Z − j =

Original: z − j which we want to identify.
Revised: z − j which we want to identify.

Original: Finally with our ignorability assumption (5) we get:
Revised: Finally with our ignorability assumption (5) we get:

Original: [Equation]
Revised: [Equation]

Original: A.4 Toy Example
Revised: A.4 Toy Example

Original: The process of generating such dataset is described below.
Revised: The process of generating such dataset is described below.

Original: Firstly, we generate rationale features{ X 1 , X 2 } following a bivariate normal distribution with positive correlations.
Revised: Firstly, we generate rationale features { X 1 , X 2 } following a bivariate normal distribution with positive correlations.

Original: To create spurious correlation, we generate irrelevant features { X 3 , X 4 } by using mapping functions which map { X 1 , X 2 }to { X 3 , X 4 } , thus irrelevant features are highly correlated with rationale features.
Revised: To create spurious correlation, we generate irrelevant features { X 3 , X 4 } by using mapping functions which map { X 1 , X 2 } to { X 3 , X 4 } , thus irrelevant features are highly correlated with rationale features.

Original: There are datasets: the training dataset, the in-distribution test data and the out-of-distribution test data.
Revised: There are datasets: the training dataset, the in-distribution test data and the out-of-distribution test data.

Original: The trainingdata (cid:8) x train i (cid:9) n train i =1 and the in-distribution test data (cid:8) x test-in i (cid:9) n test-in i =1 follows the same generation process,but for out-of-distribution test data { x test-out i } n test-out i =1 , we modify the mapping function to create adifferent distribution of the features.
Revised: The training data (cid:8) x train i (cid:9) n train i =1 and the in-distribution test data (cid:8) x test-in i (cid:9) n test-in i =1 follows the same generation process, but for out-of-distribution test data { x test-out i } n test-out i =1 , we modify the mapping function to create a different distribution of the features.

Original: Then we make the label Y only depends on ( X 1 , X 2 ) .
Revised: Then we make the label Y only depends on ( X 1 , X 2 ) .

Original: This isequivalent to assume that all the rationales are on the same position and the purpose is to simplify theexplanations.
Revised: This is equivalent to assume that all the rationales are on the same position and the purpose is to simplify the explanations.

Original: For a single x i , we simulate P( y i =
Revised: For a single x i , we simulate P( y i =

Original: | x i )
Revised: | x i )

Original: =
Revised: =

Original: π ( x i ) below:
Revised: π ( x i ) below:

Original: [Equation]
Revised: [Equation]

Original: Then we use threshold value 0 .
Revised: Then we use threshold value 0 .

Original: 5 to categorize the data into one of two classes: y i = 1 if π ≥ 0 .
Revised: 5 to categorize the data into one of two classes: y i = 1 if π ≥ 0 .

Original: 5 andy i = 

Revised: 5 and y i =

Original: if π ≤ 0 .
Revised: if π ≤ 0 .

Original: Since the dataset includes 4 features, there are 15 combinations of the rationale(we ignore the rationale containing no features.
Revised: Since the dataset includes 4 features, there are 15 combinations of the rationale (we ignore the rationale containing no features.

Original: For each rationale, we would fit a logistic regressionmodel by using only selected features and refit a new logistic regression model with subset features tocalculate CPNS .
Revised: For each rationale, we would fit a logistic regression model by using only selected features and refit a new logistic regression model with subset features to calculate CPNS .

Original: Considering in practice monotonicity can be violated (removing one of the selectedrational increases the predicted probability) and it can cause problems when we use log transform.
Revised: In practice monotonicity can be violated (removing one of the selected rational increases the predicted probability) and it can cause problems when we use log transform.

Original: Hence we calculate a surrogate of CPNS:
Revised: Hence we calculate a surrogate of CPNS:

Original: [Equation]
Revised: [Equation]

Original: In our simulation, we set n train = n test-in = n test-out = 20000 , β 0 = 1 , β 1 = 0 .
Revised: In our simulation, we set n train = n test-in = n test-out = 20000 , β 0 = 1 , β 1 = 0 .

Original: 5 and β 2 = 1 .
Revised: 5 and β 2 = 1 .

Original: A.5 ExperimentsA.5.1 Datasets 
Revised: A.5 Experiments A.5.1 Datasets Beer Review Data

Original: Beer Review Data.

Original: Following the same evaluation protocol of some previous works [see e.g.,3, 38, 9, 10], we convert the their original scores which are in the scale of [0 , 1] into binary labels.
Revised: Following the same evaluation protocol of some previous works [see e.g., 3, 39, 9, 10], we convert the their original scores which are in the scale of [0 , 1] into binary labels.

Original: Specifically, reviews with ratings ≤ 0 .
Revised: Specifically, reviews with ratings ≤ 0 .

Original: 4 are labeled as negative and those with ≥ 0 .
Revised: 4 are labeled as negative and those with ≥ 0 .

Original: 6 are labeled aspositive.
Revised: 6 are labeled as positive.

Original: We follow the same train/dev/test split as [10] and it is summarized in Tabel 5.
Revised: We follow the same train/dev/test split as [10] and it is summarized in Table 5.

Original: To makecomputation more feasible, except the raw dataset, we create a short-text version of the dataset byfiltering the texts over length 120.
Revised: To make computation more feasible, except the raw dataset, we create a short-text version of the dataset by filtering the texts over length 120.

Original: Table 6 summarizes the statistics of the Beer review dataset.
Revised: Table 6 summarizes the statistics of the Beer review dataset.

Original: Thisdataset contains Token-level annotations from humans.
Revised: This dataset contains Token-level annotations from humans.

Original: Hence we use accuracy and token F1 toevaluate task performance and human annotated rationales agreement.
Revised: Hence we use accuracy and token F1 to evaluate task performance and human annotated rationales agreement.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: GA Data.
Revised: GA Data.

Original: The proprietary GA dataset used in this study include the medical claim records (diagnosis,prescriptions, and procedures) of 329,023 patients who are diagnosed as GA during 2018 to 2021 in
Revised: The proprietary GA dataset used in this study include the medical claim records (diagnosis, prescriptions, and procedures) of 329,023 patients who are diagnosed as GA during 2018 to 2021 in

Original: US, as well as the records of additional 991,946 patients who have at least one of GA risk factors.
Revised: US, as well as the records of additional 991,946 patients who have at least one of GA risk factors.

Original: Weare tasked to utilize the medical claim data to find high-risk GA patients and reveal important clinicalindications using the causal rationale framework.
Revised: We are tasked to utilize the medical claim data to find high-risk GA patients and reveal important clinical indications using the causal rationale framework.

Original: For this dataset, we use the area under the ROCcurve (AUC) to evaluate task performance.
Revised: For this dataset, we use the area under the ROC curve (AUC) to evaluate task performance.

Original: This dataset doesn’t provide human annotations becauseit requires a huge amount of time and money to hire domain experts to annotate such a big dataset.
Revised: This dataset doesn’t provide human annotations because it requires a huge amount of time and money to hire domain experts to annotate such a big dataset.

Original: A.5.2 Synthetic Dataset
Revised: A.5.2 Synthetic Dataset

Original: Beer-Spurious .
Revised: Beer-Spurious .

Original: The first setting is considered to include strong spurious correlation into the dataset.
Revised: The first setting is considered to include strong spurious correlation into the dataset.

Original: We follow a similar setup in [9, 39] to append punctuation “,” and “.”
Revised: We follow a similar setup in [9, 40] to append punctuation “,” and “.”

Original: at the beginning of the first
Revised: at the beginning of the first

Original: [Table]
Revised: [Table]

Original: sentence with the following distributions:
Revised: sentence with the following distributions:

Original: [Equation]
Revised: [Equation]

Original: Here we set α 1 = 0 . 8 .
Revised: Here we set α 1 = 0 . 8 .

Original: Intuitively, since the first sentence now contains the appended punctuationwith strong spurious correlation, we expect the usual rationalization approach to capture such a clueand our method can identify the spurious feature and avoid selecting it.
Revised: Intuitively, since the first sentence now contains the appended punctuation with strong spurious correlation, we expect the usual rationalization approach to capture such a clue and our method can identify the spurious feature and avoid selecting it.

Original: Beer-Causal .
Revised: Beer-Causal .

Original: Firstly we randomly append either punctuation “,” or “.”
Revised: Firstly we randomly append either punctuation “,” or “.”

Original: at the beginning of the firstsentence with probability P( append ” , ”)
Revised: at the beginning of the first sentence with probability P( append ” , ”)

Original: =
Revised: =

Original: P( append ” . ”)
Revised: P( append ” . ”)

Original: = 0 .
Revised: = 0 .

Original: Then we regard the punctuationas the causal rational which can influence the label:
Revised: Then we regard the punctuation as the causal rational which can influence the label:

Original: [Equation]
Revised: [Equation]

Original: Here we set α 2 = 0 .
Revised: Here we set α 2 = 0 .

Original: Intuitively, since the first sentence now contains the appended punctuationwith mild causal correlation, we expect the usual rationalization approach to miss such a clue and ourmethod can identify the causal feature and select it.
Revised: Intuitively, since the first sentence now contains the appended punctuation with mild causal correlation, we expect the usual rationalization approach to miss such a clue and our method can identify the causal feature and select it.

Original: A.5.3 Extracted Interpretations for GA Patients
Revised: A.5.3 Extracted Interpretations for GA Patients

Original: We visualize the rationales of one GA patient from the claim data extracted using both our methodand the baseline in Figure 5.
Revised: We visualize the rationales of one GA patient from the claim data extracted using both our method and the baseline in Figure 5.

Original: Overall the two methods extracted similar rationales: they all identified“AGE-RELATED MACULAR DEGENERATION” which are the diagnoses of early stage GA, aswell as “OBESITY, UNSPECIFIED” which is a known GA risk factor.
Revised: Overall the two methods extracted similar rationales: they all identified “AGE-RELATED MACULAR DEGENERATION” which are the diagnoses of early stage GA, as well as “OBESITY, UNSPECIFIED” which is a known GA risk factor.

Original: What our method excels is 1) it identified “PRESENCE OF INTRAOCULAR LENS” which indicateshistory of eye surgery for treating myopia or cataracts and is highly pertinent to GA progression,and 2) it avoids “PERSONS ENCOUNTERING HEALTH SERVICES IN OTHER SPECIFIED
Revised: What our method excels is 1) it identified “PRESENCE OF INTRAOCULAR LENS” which indicates history of eye surgery for treating myopia or cataracts and is highly pertinent to GA progression, and 2) it avoids “PERSONS ENCOUNTERING HEALTH SERVICES IN OTHER SPECIFIED

Original: CIRCUMSTANCES”, which is a generic billing code and clinically irrelevant.
Revised: CIRCUMSTANCES”, which is a generic billing code and clinically irrelevant.

Original: A.6 Implementation Details
Revised: A.6 Implementation Details

Original: For the Beer review data, we use two BERT-base-uncased as the selector and the predictor componentsfor rationalization approaches, and one BERT-base for the full-text model.
Revised: For the Beer review data, we use two BERT-base-uncased as the selector and the predictor components for rationalization approaches, and one BERT-base for the full-text model.

Original: Those modules areinitialized with pre-trained Bert.
Revised: Those modules are initialized with pre-trained Bert.

Original: For the GA data, we use the same architecture and the only difference is we replace the word embedding matrix with a randomly initialized health diagnosiscode embedding and the embedding is trained jointly with other modules.
Revised: For the GA data, we use the same architecture and the only difference is we replace the word embedding matrix with a randomly initialized health diagnosis code embedding and the embedding is trained jointly with other modules.

Original: In our experiments, we utilize the Sparse IB constraints for rationalization approaches from [27].
Revised: In our experiments, we utilize the Sparse IB constraints for rationalization approaches from [27].

Original: Wedidn’t include a continuity constraint for two reasons.
Revised: We didn’t include a continuity constraint for two reasons.

Original: Firstly, start with less tuning parameters is moreefficient to verify the usefulness of our causal module.
Revised: Firstly, starting with less tuning parameters is more efficient to verify the usefulness of our causal module.

Original: Secondly, for the GA data, we don’t expectthe rationales of medical claim records to be continuous spans because continuous explanations don’thelp and are even unrealistic.
Revised: Secondly, for the GA data, we don’t expect the rationales of medical claim records to be continuous spans because continuous explanations don’t help and are even unrealistic.

Original: For all experiments, we utilize a batch size of 256 and choose the learning rate α ∈ { 1 e − 5 , 5 e −4 , 1 e − 4 } .
Revised: For all experiments, we utilize a batch size of 256 and choose the learning rate α ∈ { 1 e − 5 , 5 e − 4 , 1 e − 4 } .

Revised: For two branches in our framework, differentiable Gumbel–Softmax and hard top- k % are utilized to compute task loss and causal component loss respectively.

Original: We train for 20 epochs for the Beer review data and 10 epochs for the GA data where thefirst 5 epochs are warm-up (trained without causal component).
Revised: We train for 20 epochs for the Beer review data and 10 epochs for the GA data where the firstepochs are warm-up (trained without causal component).

Original: For training the causal component,we tune the values of the Lagrangian multiplier µ ∈ { 0 . 01 , 0 .
Revised: For training the causal component, we tune the values of the Lagrangian multiplier µ ∈ { 0 . 01 , 0 .

Original: 1 , 1 } and set k = 5 .
Revised: 1 , 1 } and set k = 5 .

Revised: We set the temperature of Gumbel-softmax to be 0 .

Original: For our finalevaluation, we choose top- 10% tokens as the rationales.
Revised: For our final evaluation, we choose top- 10% tokens as the rationales.

Original: We conduct our experiments over 5 randomseeds and calculate the mean and standard deviation of metrics.
Revised: We conduct our experiments over 5 random seeds and calculate the mean and standard deviation of metrics.