VIA
N ONLINEAR
V ARIABLE
D ECORRELATION FOR H EALTHCARE A PPLICATIONS A BSTRACT
Causal inference and model interpretability research are gaining increasing attention, especially in the domains of healthcare and bioinformatics.
Despite recent successes in this field, decorrelating features under nonlinear environments with human interpretable representations has not been adequately investigated.
To address this issue, we introduce a novel method with a variable decorrelation regularizer to handle both linear and nonlinear confounding.
Moreover, we employ association rules as new representations using association rule mining based on the original features to further proximate human decision patterns to increase model interpretability.
Extensive experiments are conducted on four healthcare datasets (one synthetically generated and three real-world collections on different diseases).
Quantitative results in comparison to baseline approaches on parameter estimation and causality computation indicate the model’s superior performance.
Furthermore, expert evaluation given by healthcare professionals validates the effectiveness and interpretability of the proposed model.
Code will be publicly available after acceptance.
I NTRODUCTION
With the rapid growth of Machine Learning (ML), healthcare ML research is becoming popular in the community.
Such ML methods have shown encouraging capability for solving medically related problems, such as disease understanding, diagnosis, and treatment planning, by leveraging a large number of Electric Health Records (EHR).
Although these methods bring benefits to both patients and healthcare professionals (Herpertz et al., 2017; Li et al., 2020), increasing concerns on judgment errors (Royce et al., 2019; Gandhi et al., 2006) as well as deficiency of understanding the workflow of ML systems (Croskerry, 2013) have become major road-blockers for future development and deployment of ML-based healthcare systems.
An important factor behind this difficulty is that the designed black-box ML models are often associated with a limited capacity for performance analysis (Ahmad et al., 2018).
Therefore, building interpretable ML models for healthcare becomes an imperative research direction.
To improve the interpretability of black-box models, more and more methods to enhance model interpretability are emerging (Du et al., 2019; Zafar & Khan, 2019).
However, explanations of black-box models often cannot be perfectly faithful to the original models and leave out much information which cannot be made sense of (Rudin, 2019).
In addition, traditional ML models might be influenced by the data they are trained on.
In order to enhance the interpretability of the model and adapt it to human decision patterns, we introduce association rules in place of the original features.
Recently, most of the existing diagnostic algorithms focus on associative inference and are often not compatible with the situation caused by the incomplete distribution of datasets.
Machine learning methods recognize diseases based on correlations and probability among patients’ symptoms and medical history (Zhang et al., 2021; Kuang et al., 2020a), while doctors diagnose according to the best causal explanations corresponding to the symptoms (Imbens & Rubin, 2015).
Recently, several methods have been proposed to address the agnostic distribution, including domain generalization which is becoming one of the most prominent learning paradigms (Muandet et al., 2013).
Another school of research examines the distribution shift issue from a causal perspective, such as causal transfer learning (Rojas-Carulla et al., 2018) and Structural Causal Model (SCM) (Pearl, 2009) to identify causal variables based on the conditional independence test.
In spite of their advantageous analytical qualities, these approaches are rarely employed in high-dimensional real-world applica- tions due to the complex causal graph and strict assumptions.
More recently, some researchers focus on more general methods under the stability guarantee by variable decorrelation through sample reweighting (Kuang et al., 2020b; Zhang et al., 2021; Kuang et al., 2018; 2021).
They leveraged covariate balancing to eliminate the impact of confounding, assessing the effect of the target feature by reweighting the data so that the distribution of covariates is equalized across different target feature values.
However, their model are limited to linear environments or binary datasets .
In this paper, we attempt to address the aforementioned difficulties by developing a novel method that is inherently more interpretable and can be applied to nonlinear environments for stable prediction.
We utilize an association rule mining algorithm to extract rules as model features, thereby enhancing our model’s interpretability.
To enable our model to operate in nonlinear environments, we model the relationships between features with a F ( x ) function, and perform the Taylor expansion on the F ( x ) function.
The second norm of the parameters from the first derivative to the last derivative are considered as our regularizer.
Experiments conducted on both synthetic and real-world datasets demonstrate the efficacy of our approach.
Promising results in improving the estimation of model parameters, and the stability of prediction over varying distributions in a nonlinear environment demonstrate the superior performance of the proposed method to previous methods.
The main contributions of our work are as follows:
(1) We expand the stable learning problem to a nonlinear environment under model misspecification and agnostic distribution so that stable learning can be widely applied in the real world; (2) We combine machine learning with association rules to help domain specialists understand the model and enhance the interpretability of the model; and (3) We demonstrate the superiority of our methods on synthetic and real-world datasets by calculating traditional metrics and causality.
For medical datasets, we further invite specialized doctors to validate whether our model can produce the correct rules.
R ELATED W ORK 2. M ACHINE L EARNING I NTERPRETABILITY IN H EALTHCARE
Increasing efforts have been devoted to Machine Learning (ML) interpretability research to facilitate ML research and development of real-world applications, especially in healthcare.
Among them, Generalized Additive Models (GAM) (Hastie & Tibshirani, 2017) are a set of classic methods with univariate terms providing straightforward interpretabilities.
GA 2 M-model (Lou et al., 2013) brings additional capability for real-world datasets with the selected interacting pairs based on GAMs.
On the other hand, researchers focus on applying essentially interpretable models in healthcare domain.
For example, Lee & Siau (2001) apply association rules to extract knowledge as complementary information for physicians’ diagnosis.
They also provide some strategies for patients based on the interpretation of association rules.
Lately, Ahmed et al.
(2021) apply association rules to detect major body organs in healthcare system.
Sornalakshmi et al.
(2021) reduce overhead communication when frequent data are extracted to improve association rules mining algorithm on healthcare datasets.
However, the above models are still black-box models based on joint probability distribution without causal inference.
2. A SSOCIATION R ULE M INING
Association rule mining is an important research direction that tries to identify interesting associations, frequent patterns, or causal structures (Perc¸ın et al., 2019; Ordonez et al., 2006).
In particular, association rules are able to discover predictive rules with numeric and categorical attributes.
In diagnosis system, X = x 1 , x 2 , ..., x n represents the set of all symptoms.
An association rule, noted as X ⇒ Y , indicates the disease Y is related to the symptoms X .
Three metrics were proposed to evaluate the significance of rules: support ( X ) =
P ( X ) is the probability that the set appears in the total item set; confidence ( X ⇒ Y )
= support ( X ∪ Y ) / support ( X ) is a measure of reliability; lift ( X ⇒ Y )
=
confidence ( X ⇒ Y ) / support ( Y ) reflects the correlation between X and Y in the association rules (Bayardo Jr & Agrawal, 1999).
In each rule, X is antecedent and Y is the consequent.
The rules that satisfied the minimum support and confidence are called strong association rules.
Strong association rules, can also be divided into effective strong association rules and invalid strong association rules.
How to extract strong association rules is an essential challenge.
Apriori algorithm explores candidate-generation-and-test to obtain strong association rules (Borgelt & Kruse, 2002).
Han proposed an effective method, the FP-Growth algorithm, to efficiently identity frequent patterns on large databases based on tree structures (Han et al., 2000).
Yuan (2017) proposed an improved method based on the inherent defects of the Apriori algorithm by using a new mapping way and pruning frequent itemsets to improve efficiency.
Association rules are interpretable models, whereas these methods always only consider extracting association rules based on connection instead of causality, and they do not combine rules and machine learning.
In a diagnosis system, association rules are often inconsistent with the rules of doctors’ diagnosis.
Therefore, how to extract causal association rules that are consistent with doctors’ diagnostic rules has become an important challenge.
2. C AUSAL I NFERENCE
One key challenge in healthcare is the existence of both observed and unobserved confounders under different environments (Cui & Athey, 2022).
Therefore, causal inference methods become popular for their natural fit to these problems.
For example, causal inference methods with network and hierarchy structure allow researchers to ascribe causal explanations to data (Pearl, 2018; 2009).
A completely constructed causal graph among various features based on an unconfoundedness assumption that helps to reduce the influence of confounders (Ma et al., 2021).
In addition, a
Differentiated Variable Decorrelation (DVD) algorithm is proposed to eliminate the correlations of various variables in different environments (Shen et al., 2020).
Moreover, Xu et al.
(2021) prove the effectiveness of stable learning and demonstrates the necessary of the stable prediction.
Stable Learning
Given various environments e ∈ E within datasets D e =
( X e , Y e ) , the task is to train a predictive model under the environment e i which can achieve uniformly small error under the another environment e j by learning the causality between features
X e i and targets Y e i .
Researchers propose the Deep Global Balancing Regression (DGBR) algorithm and Decorrelated
Weighting Regression (DWR) algorithm for stable prediction across unknown environments.
They successively regard each variable as a treatment variable by using a balancing regularizer with theoretical guarantee (Kuang et al., 2018; 2020b; Cui & Athey, 2022).
In Equation 1, W is sample weight, X · ,j is the j th variable in X , and X · ,
− j = X/ { X · ,j } .
With the global balancing regularizer in Equation 1, a Global Balancing Regression algorithm is proposed to optimize global sample weights and causality for classification task.
[Equation]
However, the above methods have some defects, making it difficult to deploy them on real world datasets.
The regularizer of DGBR or DWR focuses on eliminating the linear confounding under various linear environments.
In addition, such methods that forcibly delete mutual connections ignore information in the intersecting area.
For example, as shown in Figure 1, forcibly eliminating the correlation may leave only three areas: A, B, C and ignore the other areas, where A, B, C can represent three kinds of features.
In fact, the causal effect of the three features should be the union of these areas, whereas DGBR may result in the loss of mutual information.
M ETHOD
To achieve stable learning and estimation with unbiased treatment effect, we make three assumptions for our model:
(i) The set of strong causality rules is a subset of the set of rules with strong correlations; (ii) There do not exist massive observed confounders that cause the diagnostic rules dependent on another unobserved confounding; and (iii) Causal rules tend to include smaller antecedents.
The first and second assumptions imply that we can extract all the diagnostic rules with strong correlation, and then assign weights to these rules based on the causal effect.
The third assumption implies that diagnosis rules with high confidence often result in overfitting due to redundant antecedents.
Therefore, we prefer to choose more robust rules with shorter antecedents.
In this paper, we propose an interpretable model based on association rules and causal inference for EHR datasets to obtain the causality between features through a three-stage process:
A. Mining Association Rules and Transformation Rules: First, we adopt the Apriori algorithm (Agrawal et al., 1994) to obtain association rules and construct a rule matrix for positive and negative samples respectively to avoid asymmetrically distributed data.
Rule representations <
A i , C i
| θ i > are then constructed, where A i is the antecedent of the rule R
i , C
i is the consequent of the rule and θ i is the confidence of the rule, frequent function is to calculate the confidence of each rule:
[Equation]
According to the rules generated, rule sets ∪ i { X i : <
A i , C i | θ i > } are built for dataset D where each rule is considered as a feature.
Rule sets are then transformed into one-zero matrix X leveraging one-hot encoding.
B. Selecting Rules: Massive rules could be generated during the mining process, causing redundancy or even negative effects.
To extract rules with strong correlations between features, we introduce an integer programming objective function:
[Equation]
where ⊙ refers to the Hadamard product and I ( R > 0) is the indicator function converting R , a set of rules, to a one-zero vector with 1 ∗ r dimension.
The value of the indicator function equals to one when the frequency of the rule is more than zero, otherwise it equals to zero.
The estimated parameters of rep ( I ( R > 0) , n ) function are W , and b is the estimated bias.
The function is defined to expand the vector
I ( R 1 ∗ r > 0) to a matrix with dimension n ∗ r where all rows are the same as the first row.
λ 1 and λ 2 represent the bonds for the number of the selected rules.
Since we only consider a binary-classification problem here, which is a common setting for most healthcare diagnosis problems, we take the inverse of the confidence of the negative class rule as the score.
However, the number of rules mined by the association rule algorithm, e.g. Apriori, could be large, resulting an extremely high dimension of R to be able to fit in Equation 2.
Therefore, we propose to delete one redundant rule at a time during each n-fold cross-validation run based on a feature ranking criteria w 2 i .
Details of rule selection can be found in Algorithm 1 in the Appendix under RulesSelection .
Although redundant rules are removed accordingly, redundant items in rules could still impact the performance of the model.
In addition, redundant items in different rules could be easily replaced with other rules.
To solve this problem, we perform an iterative process to delete one item of each rule at a time which brings an updated R with reduced dimension.
Then we can reconstruct crossvalidation sets and feed data into SVM models to get an average accuracy.
The item that improves model’s average accuracy the most will be deleted at every iteration.
This step is summarized in
Algorithm 1 in the Appendix under ItemReduce .
C. Computing Causality Relationship: To better handle real-world nonlinear relationships, we model nonlinear relationships under Taylor expansion with a function F ( x ) as is shown in Equation 3.
Each fixed point’s derivatives can be considered as parameters to be solved by converting into a polynomial fitting problem due to the condition that two polynomials are equal only when both their degree and coefficients are the same.
[Equation]
where x p 1 (0) and x p 2 (0) are two different features which are expanded at 0 by using Taylor expansion.
The elimination of the impact of intersecting areas is achieved by balancing the weight W as is represented in Equation 4.
If x p 1 and x p 2 are independent and nonlinearly uncorrelated, the derivatives of their relation functions are all 0: ∥{ F p 2 → p 1 } / { f p 1 p 2 ( x p 2 (0)) }
∥ = 0 where F p 2 → pare the relationship function between x p 1 (0) and x p 2 (0) can be calculated using Equation
[Equation]
(5) where w i is the weight for each sample and n is the number of datasets.
Combined with Figure 1, physical meaning can be given to the above variables: f ( θ ) is the correlation between each feature and target; C is the factor to expand the influence of the intersection area to get the real causality comparing with W applied to eliminate the influence of the public area:
[Equation]
When we have a smaller γ value, the difference between β and the true correlation coefficient (disjoint region and the target) will become smaller, resulting greater mutual information loss.
Lemma 1.
If the number of features in the datasets and the terms in the Taylor expansion are fixed, when n → ∞ there exists W ⪰ 0 such that
[Equation]
[Figure]
a.Linear Relationship
[Figure] .Linear Relationship(non-linear) [Figure]
c.Square Relationship
[Figure]
d.
Cubic Relationship
[Figure]
f.Beta Error
Proof.
See the appendix A.
E XPERIMENT 4. V ALIDATION ON A S YNTHETIC D ATASET
To examine the proposed constraints’ effect on eliminating linear and nonlinear connotation relationships, we follow previous work (Kuang et al., 2020b) to conduct evaluations on synthetically generated datasets.
The details of experiment settings and baseline methods can be found in the
Appendix ( A.2).
Notice that a different objective function 7 is built for regression task, where W i is the sample weight and the variable ζ is slack variable.
In this experiment, we only expand two terms by the Taylor expansion.
[Equation]
subject to
[Equation]
4.1. R ESULTS
To compare two kinds of regularizers, we apply Pearson Correlation to calculate the relationship strength among features.
Since Pearson Correlation can only describe linear relationship, we construct nonlinear pairs
WV i with ( WV ) 2 j , ( WV ) 3 j and exp ( WV j ) in addition to WV i with WV j .
The result can be found in Figure 2.
Both DWR and the proposed regularizer can handle pure linear relationships (experimental environment(A)) but some improvement is achieved from the proposed regularizer.
As we add nonlinear relationships to the linear experimental environment,
DWR start to have difficulty with the linear relationship part while the proposed method is still able to reduce a large amount of the relationships.
For nonlinear environments, compared with the original unweighted dataset, DWR unexpectedly increases nonlinear relationships where there are no existing nonlinear relationships (square, cubic and exponential).
Instead, our model can deal with nonlinear relationships and reduce nonlinear relationships.
To test our algorithm perform the best on various sample sizes and the number of features, we calcualte the β S and beta V errors: error ( β ) =
(cid:80) i | β true − β | , based on 9 kinds of datasets 1.
We conduct a series of ablative studies to evaluate the stability of our model.
Table 2 summarizes the results of the experiment with various values of C and Lagrange penalty operators γ , and λ .
For each sell, we fix the Lagrange penalty operators and increase C to calculate the β errors and RMSE errors.
The higher C indicates higher integrated mutual information is fed into the model and magnifies the impact of confounding.
Higher γ values will reduce more confounding effects and diminish mutual information.
Here we choose the best parameters ( γ = 600 , λ = 0 .
0005 , C = 0 . 5 )
based on the smallest RMSE also with a smaller β error comparing to ( γ = 1000 , λ = 0 .
0005 , C = 0 .
5 ) which has the same RMSE.
To further confirm that the coefficients estimated by our model are based on causality, we repeat experiments 50 times to calculate (cid:80) ∥ β − ˆ β ∥ , where β and ˆ β represent the true value and estimated parameters, respectively.
In Figure 2, we find that the difference between the estimated parameters and the true values is smaller with our model, compared to other models in the nonlinear environment.
Notice that our model achieves much smaller distribution variance as well as much smaller average values of β errors comparing to baselines.
Although the regularizer of DWR can solve the stable problem in linear environments, it retains or expands nonlinear confounding in the nonlinear environments.
From the above results, we find that our model is able to reduce correlations among all predictors and avoid being affected by nonlinear confounding, resulting a reduced estimation bias in more general environments.
4. V ALIDATION ON T HREE R EAL -W ORLD D ATASETS
To further validate the effectiveness of our model in real-world scenarios, we perform experiments on three different EHR datasets.
All data are prepossessed to ensure no sensitive information is exposed.
4.2. D ATASETS AND S ETTINGS
Heart Disease is retrieved from the repository of the University of California, Irvine (Asuncion &
Newman, 2007).
We follow previous work to use 13 of 76 attributes: Age , Sex , cp , threstbps , chol , fbs , restecg , thalach , exang , oldpeak , slope , cam and thal .
Esophageal Cancer consists of data from 261 patients who underwent esophagectomy for esophageal cancer between 2009 and 2018.
The collected characteristics include patient demographics, medical and surgical history, clinical tumor staging, adjuvant chemoradiotherapy, esophagectomy procedure type, postoperative pathologic tumor staging, adjuvant chemoradiotherapy, postoperative complications, cancer recurrence, and mortality.
Cauda Equina Syndrome (CES) is extracted from the Statewide Planning and Research Cooperative System (SPARCS) (of Health & Bureau, 1984), a comprehensive database of all payers for all hospitalizations in New York State Joo et al.
Based on diagnostic and procedure codes, patients with CES who underwent surgery betweenand 2015 were selected.
Patient demographics (age, gender, race, comorbidities, and insurance status) and hospital characteristics (measured by hospital bed number quartiles).
Pre-Processing: We convert the continuous variables into categorical variables before feeding them to the model.
To handle missing data in the datasets, we adopted MICE (Multiple imputations by chained equations) by transforming imputation problems into estimation problems where each variable will be regressed on the other variables.
This method provides promising flexibility since [Table] [Table] every variable can be assigned a suitable distribution (Wulff & Jeppesen, 2017).
Then we apply the
SMOTE algorithm (Fern´andez et al., 2018) to address the class imbalance issue in our datasets.
Feature Selection: Redundant information in EHR datasets may cause noise and irrelevant information during feature extraction.
A feature selection method (Guyon et al., 2002) is adopted.
To improve the robustness of the model, we divide the dataset randomly into five groups for crossvalidation.
Each time we extract one group as the test set to analyze and measure the average performance in the feature selection process.
Due to the high complexity of our model, we apply and compare the four baseline models: XGboost, SVM, Logistic Regression, and Random Forest [Table] to extract important features in feature selection and input the set of the features with the highest average AUROC scores into our model.
In the end, we extract 13, 47, and 45 features for Heart
Disease , Esophageal Cancer , and Cauda Equina Syndrome , respectively.
Comparison to Baseline Models: After the feature selection process, we transform continuous features into categorical variables before inputting data into our model and then applying one-hot encoding to convert categorical attributes into numeric, since the association rule mining algorithm in our paper cannot accept continuous features.
However, the data set, filtered by feature selection, is directly fed into baseline models since forcing the continuous features to be discretized may lead to worse performance of the model.
We assign 20% data into test datasets and compare our model with five baseline models: Logistic Regression, Random Forest, XGboost, SVM and MLP as shown in Appendix A.
4.2. R ESULTS
To measure the performance of models, we calculate accuracy, precision, recall and F1 scores.
The result is shown in Appendix A.3, Table 3.
In addition to calculating the metrics of the traditional models, we input the filtered rules as one-zero matrix X into the baselines rather than the original datasets.
In Heart Disease and Esophageal Cancer datasets, rules do help XGBoost, Random Forest and MLP to improve the performance, while in Cauda Equina Syndrome datasets rules can improve the performance of all models.
For SVM and Logistic Regression, the effect of the model can be greatly improved after combining the rules.
Our model generally performs the best on all three datasets, similar to the SVM performance, while achieving high interpretability as discussed below.
Combining the experiments in the previous section, better performance is not equivalent to obtaining the real rules.
To compare the causality calculated by our model and baselines, we ask three groups of doctors of the corresponding domains to score each rule.
Three groups of doctors are from
Cardiology, ENT and Neurosurgery departments, and each group consists of three doctors.
we apply the models to calculate the importance of features to score each rule.
To verify rating consistency between our model and doctors leveraging Spearman Coefficients.
Results can be found in Table 3.
As can be observed, causality rankings of the baseline models vary greatly, indicating unstable performance.
In these datasets, the causal value of our model is higher than other baselines, implying that the scoring of our model is more consistent with the standard of doctors.
C ONCLUSION
In this paper, we present a causal inference approach focusing on interpretability and nonlinear environments for healthcare applications.
The proposed method extracts association rules from the raw features as new representations to be used by our model.
A novel regularizer that is capable for handling both linear and nonlinear confoundings is constructed to enable our model’s adaption to real-world applications.
The superior performances on a synthetic dataset and three real-world
EHR datasets from different domains compared to baseline methods validate both the effectiveness and generalizability of the proposed method.
Consistent ratings with healthcare professionals on the extracted rules further validate the model’s interpretability, while not sacrificing accuracy.
A PPENDIX A. A LGORITHM
We combine algorithm 1 with object function 2 to select the robust rules and prune the redundant items.
In the RulesSelection function, we delete one rule each time with lowest ∥ w ∥ 22 and save the rule sets with the highest accuracy.
In the ItemReduce function, we apply cross-validation to train
SVM model and save the item sets with best accuracy.
Input: Rules { X i } are the association rules obtained by Apriori algorithm with training datasets.
data is EHR datasets.
Output: Bestrules 1: function R ULES S ELECTION
( Rules, data ) 2:
Bestrules ← Rules 3:
Objfunction is objective function 4:
Select ← Bestrules 5:
Bestaccuracy ← Select 6:
Lastrules ← ∅ 7: while Select ̸ =
Lastrules do 8:
Lastrules ← Select 9: w ← argmin Objectfunction ( Select, data ) 10:
Selected ← argmin w
i 11:
Temprules ← { Bestrules } / { Selected } 12:
Tempaccuracy ← Temprules 13: if Tempaccuracy > Bestaccuracy then 14:
Bestaccuracy ← Tempaccuracy 15:
Select ← Temprules 16: end if 17: end while 18: return Bestrules 19: end function 20: function I TEM R
EDUCE
Bestrules, data ) 21:
Bestauc ← SV M ( Bestrules, data ) 22:
Lastrules ← ∅ 23: while Bestrules ̸ =
Lastrules do 24:
Item ← argmax SV M ( { Bestrules } / { Item } ) 25:
Accuracy ← SV M ( { Bestrules } / { Item } ) 26: if Accuracy ≥ Bestauc then 27:
Bestauc ← Accuracy 28:
Bestrules ← { Bestrules } / { Item } 29: end if 30: end while 31: return Bestrules 32: end function
A. D ATASETS AND S ETTINGS
In addition to the linear settings, we propose to include nonlinear evaluations under a nonlinear environment:
Linear Environment:
For this setting, we construct features S that causes unstable V by auxiliary variables z with linear relationship among features only.
[Equation]
Nonlinear Environment:
In this setting, we combined square relationship and exponential relationship to generate various environment including potential nonlinear confounding to test our reweighted regularizer.
[Equation]
To further test the robustness of our algorithm, we assume that there are unobserved nonlinear terms, and construct the label Y as shown in Equation 8.
Combined with weighed SVM loss function, we train our model to estimate the regression coefficient β .
In this experiment, we set β s = (cid:8) 13 , − 23 , 1 , − 13 , 23 , − 1 , · · · (cid:9) , β v = −→ 0 , and ε = N (0 , 0 . 3) .
In the experiment, we will set different dimension of β , hence if the dimension of β S is higher than 6, we will set the element of which index is larger than 6 as the i %
-th of β V
[Equation]
Baselines:
We compare our model with five baseline methods:
• Ordinary Least Square (OLS) (Hutcheson, 2011): min ∥
Y − X β ∥• Lasso (Tibshirani, 1996): min ∥
Y − X β ∥ 22 + λ 1 ∥ β ∥• Ridge (Hoerl & Kennard, 1970): min ∥
Y − X β ∥ 22 + λ 1 ∥ β ∥• Decorrelated Weighting Regression (DWR) (Kuang et al., 2020b):
min W,β (cid:80) ni
=1 W i · ( Y i − X i, β )s.t (cid:80) pj
=1 (cid:13)(cid:13)
X T,j Σ W X , − j /n − X T,j W/n · X T, − j W/n (cid:13)(cid:13) 2 2 < λ• Support Vector Machines (SVM) (Suykens & Vandewalle, 1999): min w,b,ζ,ζ ∗
[Equation]
=
[Equation]
ζ
[Equation]
∗ i ) • SVM combined with DWR(DWR SVM): min w,b,ζ,ζ ∗ 12 w T w
+
(cid:80) ni
=1 W i ( ζ i + ζ ∗ i ) s.t (cid:80) pj =1 (cid:13)(cid:13)
X T,j Σ W X , − j /n − X T,j W/n · X T, − j W/n (cid:13)(cid:13) 2 2 < λ
Generating Various Environments To test the stability of the algorithms we generate a set of environment e with a distinct distribution distribution P XY .
Following the Kuang’s experiment Kuang et al.
(2020b) we generate different environments based on various P ( S | V ) .
To simplify the problem, we simulate P ( S b | V on a subset S b ∈ S , where the dimension of S b is 0 .
2 ∗ p .
We applied the bias rate equation Pr = (cid:81) S
i ∈ S b | r | − 5 ∗ D i to tune the P ( S b | V , where
D i = | f ( S ) − sign( r ) ∗ V i | , r ∈ [ − 3 , − 1) ∪ (1 , 3] .
r > 1 indicates that Y and S b have positive unstable relationships, while r < − 1 corresponds to the negative unstable relationships.
The higher absolute value of r the stronger connection between S b and Y , leading to generate different environments.
The result is shown in Figure 3.
A. C AUSAL E XPERIMENT R ESULTS AND E XPLANATION
We compare our model with five traditional methods:
• Logistic Regression We leverage the logistic regression classifier with L-BFGS solver for classification (Bollapragada et al., 2018).
• Random Forest
We apply standard Random Forest classifier to solve the classification problem (Pal, 2005).
XGboost
We adopt XGBoost, an extreme gradient boosting methods, to compare with other models (Chen & Guestrin, 2016).
SVM
We apply supervised learning models, SVM, with linear kernel to analyze data for classification (Suykens & Vandewalle, 1999).
• MLP
We use the traditional neural network multi-layer perceptron to solve this classification task (Agatonovic-Kustrin & Beresford, 2000).
We sort the rules in descending order by calculating the importance and show the top five rules compared with the doctor’s score 4.
The details of the description for each feature in the rules are shown in the Table 5.
The scoring criteria are as follows:
• Score 4: Strongly agree that the rule contains causality.
• Score 3: Agree that the rule contains causality.
• Score 2: Disagree with this rule.
• Score 1: Strongly disagree with this rule.
[Table]
A. I NTRODUCTION FOR F EATURES
[Table]
dangerous if aa
[Table]
A. PROOF
Lemma 2.
If the number of features in the datasets and the terms in the Taylor expansion are fixed, when n → ∞ there exists W ⪰ 0 such that
[Equation]
Proof.
Based on our regularizer
[Equation]
we know that
[Equation]
 
We assume that the covariance is 0:
[Equation]
Combine with the following equation, we can get
[Equation]
[Table]
ˆ
[Table]
ip 2 is influenced by the w i which can be adjusted, and the determinant of matrix is not equal to 0, hence the equation has only the trivial solution.
We can get
[Equation]
If we can prove under our regularizer
[Table]
we can prove our method can work:
[Equation]
We set (cid:0) ˆ x ip 2 , ˆ x 2 ip 2 , . . .
, ˆ x kip 2 (cid:1) is kernel density estimators: g ( x ip 2 .
We set the weight w i is:
[Equation]
We can get:
[Equation]