Paper:
Neural networks provide a powerful framework for learning the association between input and response variables and making accurate predictions and offer promise in using the rapidly growing volume of health care data to surface causal relationships that cannot necessarily be tested in randomized clinical trials. In pursuit of models whose predictive power comes maximally from causal variables, we propose a novel causal regularizer based on the independence of mechanisms assumption. We use the causal regularizer to steer deep neural network architectures towards causally-interpretable solutions. We perform a large-scale analysis of electronic health records. Our causally-regularized algorithm outperforms its L1-regularized counterpart both in predictive performance as well as causal relevance. Finally, we show that the proposed causal regularizer can be used together with representation learning algorithms to yield up to 20% improvement in the causality score of the generated hypotheses.
1 INTRODUCTION
In domains such as healthcare, genomics or social science there is high demand for data analysis that reveals causal relationships between independent and target variables. For example, doctors not only want models that accurately predict the status of patients, but also want to identify the factors that can change the status. The distinction between prediction and causation has at times been subject to controversy in statistics and machine learning (Breiman et al., 2001; Shmueli, 2010; Donoho, 2015). On one hand, machine learning has been focusing almost exclusively on pure prediction tasks, enjoying great commercial success. On the other hand, in many scientific domains pure prediction without consideration of the underlying causal mechanisms is considered unscientific (Shmueli, 2010). In this work, we propose a neural causal regularizer that balances causal interpretability and high predictive power.
Causal Inference: Our notion of causality follows the counterfactual framework of Pearl (2000). Thus, we will say that one random variable X causes another variable Y (which relationship we denote as X → Y ) if intervening or experimenting on X changes the distribution of Y . Consider the problem of identifying the causal relationship between drinking red wine and heart disease (Spirtes, 2010). Regular consumption of red wine correlates with healthy heart. That might mean that drinking red wine decreases heart attack rates. But it might be, for example, that people of high socio-economic status tend to drink more wine, while at the same time tend to suffer fewer heart problems due to better living conditions. To distinguish between these two possibilities, one could implement a controlled trial in which the subjects are told to drink (or not drink) red wine, independently of any other factors —including their socio-economic status.
Such controlled trials are often undesirable or even impossible. In healthcare, it can be due to moral and regulatory reasons; in climate science for example, due to technological limitations (we don’t know how to change climate). In such settings, we would like to still establish causality without resorting to experiment. Even in applications where controlled trials are possible, the large number of causal hypotheses can make it impossible to experimentally test all of them. Furthermore, in domains such as healthcare, many causal factors need to occur simultaneously to have an effect on the target variable, a scenario that we call multivariate causation. Given the exponential number of combinations of the independent variables and different transformations, it is even more difficult to explore all of these multivariate causation scenarios.
Analyzing causation without resorting to experiment is challenging due to unobserved confounders (such as the possible influence of socio-economic status on heart health and wine drinking). Many methods have been proposed for discovering causal relationships among multiple variables from observational data only (Chickering, 2002; Kalisch & Bühlmann, 2007; Colombo et al., 2012), demonstrating various degrees of success. These methods are based on the idea that any given set of causal relationships among multiple variables will leave in the joint distribution well-defined markers in the form of independence relationships among subsets of the variables. These methods, however, are often very sensitive to small changes in the joint distribution.
Causal Regularization: Our main idea is to design a causal regularizer to control the complexity of the statistical models and at the same time favor causal explanations. Compared to the two step procedure of (i) causal variable selection and (ii) multivariate regression, the proposed approach performs joint causal variable selection and prediction, thus avoiding the statistically sensitive hardthresholding of the causality scores in the causal variable selection step. It allows dependencies that cannot be explained via causation to be included in the model. Our contributions are four-fold:
1. We propose a customized causality detector neural network that can accurately discriminate causal and non-causal variables in our healthcare datasets. To this end, we propose new synthetic dataset generation to train the causal structure detectors in (Chalupka et al., 2016; Lopez-Paz et al., 2016) with additional prior knowledge from the healthcare domain. 2. We use the causality detector to construct a causal regularizer that can guide predictive models towards learning causal relationships between the independent and target variables. 3. Given the fact that the causal regularizer seamlessly integrates with non-linear predictive models such as neural networks, we propose a new non-linear predictive model regularized by our causal regularizer, which allows neural causally predictive modeling. 4. Finally, we demonstrate that the proposed causal regularizer can be combined with neural representation learning techniques to efficiently generate multivariate causal hypotheses.
The proposed framework scales linearly with the number of variables, as opposed to many previous causal methods. Combined with a predictive model, it efficiently screens a high-dimensional hypothesis space and proposes plausible hypotheses.
We applied the proposed algorithm to two electronic health records (EHR) datasets: Sutter Health’s heart failure study data and the publicly available MIMIC III (Johnson et al., 2016) dataset. Altogether, we analyzed the influence of 17,081 independent variables on heart failure. To validate our claims, we use expert judgment as the causal ground-truth to compare our causal-predictive solutions with purely predictive solutions that do not take causality into account. As shown in Figure 1, a causallyregularized algorithm outperforms its L1-regularized equivalent both in predictive performance as well as causal performance.
2 METHODOLOGY
In order to “inject causality” into predictive models, we use the “independence of cause and mechanisms” (ICM) assumption, which allows us to construct a neural network causality detector, as described in Section 2.1. We present a causal regularizer for linear models in Section 2.2. Using this regularizer, in Section 2.3 we propose non-linear deep neural networks to learn non-linear causal relationships between the independent and target variables. Finally, we show that the causal regularizer can efficiently explore the space of multivariate causal hypotheses and extract meaningful candidates for causality analysis.
Under review as a conference paper at ICLR 2017
!
"#′ !
"#′
!
"#′ !
"#′
!
"#′ !
"#′
! " ! " ! " ! # " ! # "
! " ! " ! " ! # " ! # "
2.1 CAUSALITY DETECTION BASED ON INDEPENDENCE OF MECHANISMS
As we discussed in the introduction, the task of analysis of causal effect of multiple independent variables on a target variable is difficult. Our approach in this paper is to reduce the problem to analysis of the causal effect of a single independent variable X on the target variable Y , which is known as pairwise causal analysis. In the next subsections, we will describe how to use a pairwise causality detector to perform multivariate causality analysis.
In particular, we are interested in finding causal models where X causes Y , or Y causes X , or the two are confounded based on joint distribution of P (X,Y ). However, even the pairwise causality analysis is infeasible for arbitrary joint distributions. Thus, we need to resort to additional assumptions on the nature of the causal relationships. Recently several algorithms have been proposed that distinguish between the cause and effect based on the natural assumption that steps in the process that generates the data are independent from each other, see (Lemeire & Dirkx, 2006; Janzing et al., 2012; Daniusis et al., 2010; Lopez-Paz, 2016; Chalupka et al., 2016) and the references therein. In this work, we follow (Lopez-Paz et al., 2016; Chalupka et al., 2016) to describe this causality detection approach. In the next subsections, we describe our novel causal regularizer designed based on this causality detection approach and its application in non-linear causality analysis and multivariate causal hypothesis generation.
Conceptual description of the independence between the cause and the mechanism. Algorithms based on the ICM, such as (Chalupka et al., 2016; Lopez-Paz et al., 2016) do not put assumptions on the functional form of the causal relationships between the variables of interest. Instead, they are based on the following assumption on how causal mechanisms come to be. ICM states that the two processes of generation of the cause and mapping from cause to effect are in some sense independent. In our case, we assume that when X → Y (X causes Y ), the probabilities P(Y | X) and P(X) are generated by independent higher-level distributions. This conforms to the scientific idea of Uniformitarianism (Gould, 1965) which, putting roughly, states that the laws of nature apply to all objects similarly. ICM can be described in both deterministic (Janzing & Scholkopf, 2010) and probabilistic sense (Daniusis et al., 2010); this work mainly uses the probabilistic interpretation.
ICM can be used to generate all of the possible graphical models including two observed variables X and Y and an unobserved variable H shown in Figure 2, by requiring that the probabilities in the factorization are independent from each other. The hidden variables can represent the other observed variables such as Z, critical in design of the regularizer in the next subsection.
Following the ICM, we assume that each cause-effect link in the world is probabilistic and can be described by a joint distribution P (cause, effect). In addition, the link itself is sampled from a probabilistic hyperprior. The key assumption is on the structure of this hyperprior, namely that it decomposes into two parts Πc and Πm that have the following properties:
1. For each cause, effect pair, Nature samples the cause’s distribution Pcause from a hyperprior Πc[Pcause].
Given the data, perform the following steps: 1. Generate data samples Si for i = 1, . . . , ntrain from pX,Y according to the ten cases in
Figure 2. 2. Assign label y = 0 to the cases in Figures 2b, 2d, 2g and 2i and y = 1 to the rest. 3. Train a classifier f : S → [0, 1] to classify them as causation (label=1) or not-causation
(label=0). Given the fact that this is a synthetic dataset, we know these labels and we can use supervised learning.
4. On the test set, construct the test sample sets and use the classifier in step 3 to classify the example.
Algorithm 1: The algorithm for constructing the causality detector. The structure of neural network classifier is given in Appendix B.1.
2. At the same time, Nature samples the causal mechanism (the distribution of the effect conditioned on the cause) Peffect|cause from a hyperprior Πm[Peffect|cause]. 3. The hyperpriors are flat —for discrete cause and effect, they are Dirichlet distributions with α uniformly equal to 1.
The last assumption is not crucial and can easily be changed if knowledge about hyperpriors in a specific domain is available. In fact, we tailor the hyperpriors to our task below. These three assumptions give us a full generative model of causal links in the world, a model under which the likelihood ratio test can be used to differentiate between the data generated from each of the ten cases shown in Fig. 2. Chalupka et al. (2016) developed an analytical likelihood ratio test that decides between the causal and anticausal cases (Figures 2b and 2c). Taking into account the confounded cases is, however, difficult or impossible to compute analytically. Nevertheless, it is possible to generate samples from the generative model defined by the ICM and train a neural network to learn to choose the max likelihood causal structure given samples from the joint P (cause, effect). This is the key idea of the causality detectors in (Lopez-Paz et al., 2016; Chalupka et al., 2016).
Mathematical description of the causality detection algorithm. Formally, suppose we have m variables Xi, each with dimensionality di. For each variable we observe a sample of size ni denoted by Si = {(xi,j , yj)}nij=1, where yj are observations of a common target variable Y . Let S denote the set of all such samples. For each sample Si, we are interested in determining the binary label `i ∈ {0, 1} which determines whether Xi causes Y or not. In fact, we are interested in the function approximation problem of learning the mapping f : S 7→ {0, 1}. Several approaches can learn such a mapping function. When X and Y are both discrete and finite, Chalupka et al. (2016) construct the empirical joint distribution p̂i = p̂(Xi, Y ) and train a supervised neural network mapping function f(p̂i)→ `i. Lopez-Paz et al. (2016) learn the mapping 1 ni ∑ni j=1 φ(xi,j , yj) and a neural network f ( 1 ni ∑ni j=1 φ(xi,j , yj) ) → `i. They train both the representation leaning function φ(·, ·) and the classification network in a joint and supervised way. However, it is rare to have the true causal labels ` for training a causal detector. The key idea is to generate a synthetic dataset composed of the cases in Figure 2 based on the ICM assumption. As shown in Algorithm 1, the overall procedure is to generate samples from distributions pX,Y that are one of the ten possible cases in Figure 2. We need to select the distributions such that they impose minimum number of restriction on the data and the synthetically-generated distributions have statistics as similar as possible to those of our true data of interest. For example, in our dataset, the independent variables X are counts of the number of disease codes in patients’ records (cf. Section 3). Thus, we sample X from a mixture of appropriate distributions for count data: the Zipf, Poisson, Uniform, and Bernoulli distributions. The hidden variableH and the response variable Y are sampled from the Dirichlet and Bernoulli distributions. Details of our sampling procedure are provided in Appendix A.
2.2 THE CAUSAL REGULARIZER
As an instructive alternative to our approach, consider the two-step analysis method of first finding the variables Xi that are most likely causes of Y and then performing a sparse multivariate regression to select the important variables. Ideally, if the ICM holds and if we had access to the true joint distributions and could discriminate between causal and non-causal variables with perfect accuracy, the two-step procedure would be sufficient. But real-world datasets always contain noise and selection bias, which can perturb the causality scores generated by the neural network confounder detector.
The problem arises from the fact that our causality detection algorithm might give soft scores such as 0.5 + ε or 0.5 − ε to two variables X1 and X2, respectively. These soft-scores can be interpreted as the probability that each variable is the cause of Y . If we use the two-step procedure, we will include X1 in the regression model but not X2. However, X2 could possibly contribute more to the predictive performance in presence of other variables in the multivariate regression. In other words, any hard cut-off for the purpose of two-step causal variable selection and regression will pose the question of “what should be the best cut-off threshold?” Note that any hard cut-off will be always statistically unstable in presence of noise and selection bias.
Instead, we propose a causally regularized regression approach, where this trade-off is performed naturally via a regularization parameter. We select variables that are both potentially causal with high probability and also significantly predictive.
Causal Regularizer. Now that we have a classifier that outputs ci = P[Xi and Y are not-causal], we can design the following regularizer to encourage learning a causal predictive model:
ŵ = argmin w  1n n∑
j=1 L(xj , yj |w) + λ m∑ i=1 ci|wi|  , (1) where L(X1, . . . , Xn, Y |w) is the loss function of logistic regression for X1, . . . , Xn and Y . The first term in Eq. (1) is a multivariate analysis term, whereas the regularizer might look like a bivariate operation between each independent variable Xi and the target variable Y for i = 1, . . . , p. However, we should note that in the design of the causal regularizer, we have implicitly included the other variables as hidden variables in the analysis. Thus we are allowed to use the regularizer together with multivariate regression. Note that the proposed causal regularizer is also a decomposable regularizer which makes analysis of its theoretical properties easier (Negahban et al., 2012).
The two-step analysis can be cast as a special case of causally predictive modeling where we use hard scores instead of soft scores. Consider the following setting:
ŵ = argmin w  1n n∑
j=1 L(xj , yj |w) + γ m∑ i=1 c′i|wi|  , Where c′i is defined as follows:
c′i = { 1− ε if ci > 1/2 ε if ci ≤ 1/2
Now, consider the limiting case of ε → 0 and γε → λ. This case corresponds to the two-step procedure with L1 regularized logistic regression.
Note that the possibility of having a causal regularizer has been proposed in (Lopez-Paz, 2016, Page 181) and (Lopez-Paz et al., 2016), however a specific causal regularizer has never been developed and evaluated. Furthermore, note that using the score of a “causal-anticausal”-only classifier, as e.g. in (Lopez-Paz et al., 2016), cannot properly regularize a multivariate model such as logistic regression. In our proposal, the rest of the observed independent variables can be considered as hidden variables in our bivariate causality analysis which allows proper regularization. Moreover, a major novelty of our proposed causal regularizer is to do joint causal variable selection (the L1 regularization) and prediction, but the idea in (Lopez-Paz et al., 2016) cannot.
2.3 CAUSAL REGULARIZERS IN NEURAL NETWORKS
The key advantages of causal regularizer can be seen when it is used for regularizing neural networks. We demonstrate two use cases of causal regularizer as shown in Figure 3.
Non-linear Modeling. The objective is to design a non-linear neural network in a way that we can still identify causality. We propose the following non-linear generalized linear model:
σ−1(E[Y ]) = w>x + β>(α(Ex) (Ex)) + b, (2)
where the embedding matrix E ∈ Rq×m maps the input x ∈ Rm to a lower dimensional representation space and the symbol denotes the element-wise product. The logistic sigmoid function σ−1 maps the real values to the [0, 1] interval. The term w>x acts as the skip connection and initialized by the result of logistic regression. The embedding allows dealing with very large set of discrete concepts and can be initialized via techniques such as skip-gram (Mikolov et al., 2013) or GloVe (Pennington et al., 2014). The vector α(Ex) can be computed using a multi-layer preceptron.
The model in Eq. (2) is a particular non-linear extension of logistic regression. We can reorder the equations to write the right hand side of Eq. (2) as ω(x)>x + b, where the new regression coefficient ω can change with every input. Each coordinate of the new regression coefficient can be calculated as ωi(x) = wi + (β α(Ex))>Ei, where Ei denotes the ith column of the embedding matrix E. The variability of ωi(x) for each input x enables us to perform individual causality analysis. For training, we can penalize the ω coefficients and minimize the following loss function
1
n n∑ j=1
{ L̃(xj , yj) + λ
m∑ i=1 ci|ωi(xj)|
} , (3)
where L̃ denotes the negative log-likelihood of the model described by Eq. (2). The change of the prediction vector with each sample x can be related to the probabilistic definition of causation (Pearl, 2000) in the sense that the strength of causality may change from a subject to another one. The fact that in Eq. (2) the impact of each independent variable on the target is measured by ωi(x) allows us to penalize it with our regularizer and push the model to learn more causal relationships.
Multivariate Causal Hypothesis Generation. A key application of our proposed causal regularizer in conjunction with deep representation learning is to efficiently extract multivariate causal hypotheses from the data. Figure 3b shows an example of causal hypothesis generation where the hypotheses are generated via a Multilayer Perceptron (MLP). We assume that there is a representation learning network with K-dimensional output h(x) ∈ IK , where I denotes the range of the output, for example I = (0, 1) for sigmoid and I = [0,∞) for ReLU activation functions. Our goal is to force each dimension of h to be causal, thus h can be used as multivariate causal hypotheses. In particular, we aim at minimizing the following objective function:
1
n n∑ j=1
{ L(w>hj + b) + λ
K∑ i=1 |gi(hj,i)wi|
} (4)
Our approach is to train a causality detector based on (Lopez-Paz et al., 2016) and design the regularizer g(h(x)) based on its score. Then, as shown in Figure 3b, we can combine it with the neural network to regularize the coefficients of the last layer of the multilayer Perceptron which predicts the labels from h. The weights of the lower layers in h(x) are regularized using L1 regularizer to make the generated causal variables simple. To train the network, we select batches with fixed-size of 200 examples. This number is selected to be large enough such that error rate of the causality detector in (Lopez-Paz et al., 2016) becomes lower than 2%. We select the non-linearity for h to be the logistic sigmoid function, thus we use Beta distribution for generating synthetic data for training of the causality classifier.
3 EXPERIMENTS
We evaluate the proposed causal regularizer in Section 2.2 both in terms of their predictive and causal performance. Next, we compare the quality of the codes identified as causes of heart failure identified by different approaches. Finally, we evaluate performance of multivariate causal hypothesis generation by qualitatively analyzing the extracted hypotheses. We defer evaluation of the causality detection algorithms to Appendix A, as they are not the main contributions of this work.
3.1 DATA
The Sutter Health heart failure dataset consists of Electronic Health Records of middle-aged adults collected by Sutter Health for a heart failure study. From the encounter records, medication orders, procedure orders and problem lists, we extracted visit records consisting of diagnosis, medication and procedure codes. We denote the set of such codes by C. Given a visit sequence v1, . . . ,vT , we try to predict if the patient will be diagnosed with heart failure (HF) and identify the key causes of increase heart failure risk. To this end, 3,884 cases are selected and approximately 10 controls are selected for each case (28,903 controls). The case/control selection criteria are fully described in the supplementary section. Cases have index dates to denote the date they are diagnosed with HF. Controls have the same index dates as their corresponding cases. We extract diagnosis codes, medication codes and procedure codes from the 18-month window before the index date. There are in total 17,081 number of unique medical codes in this dataset.
The MIMIC III dataset (Johnson et al., 2016) is a publicly available dataset consisting of medical records of intensive care unit (ICU) patients over 11 years. We use a public query1 to extract the binary mortality labels for the patients. Our goal is to use the codes in the patients’ last visit to the ICU and predict their mortality outcome. Our dataset includes 46,520 patients out of whom 5810 have deceased (mortality=1). A totoal of 14,587 different medical codes are used in this dataset.
Feature construction. Given the sequence of visits v(i)1 , . . . ,v (i) T for patients i = 1, . . . , n, we create a feature vector xi ∈ N|C|0 by counting the number of codes observed in the records of the ith patient. Given the large variations in the number of codes, we logarithmically bin the count data into 16 bins. The final data is in the form of (xi, yi) where yi is ith patient’s label; heart failure and mortality outcome in the Sutter and MIMIC III datasets, respectively.
Training details. Given the fact that we generate synthetic datasets for training the causality detector neural networks, we can generate as many new batches of data for training and parameter tuning purposes as required. We report the test results on a dataset of size 10,000 data points. For training and parameter tuning of the neural network model in Section 2.2, we perform the common 75%/10%/15% training/validation/test splits. Details of training the latter neural network are given in Appendix B.2.
3.2 EVALUATING THE PREDICTIVE PREFORMANCE OF CAUSAL REGULARIZER
In order to characterize the performance of the proposed causal regularizer, we perform penalized logistic regression with the proposed regularizer and the commonly used L1 regularizer. Table 1 shows the test accuracy of heart failure and mortality prediction in Sutter and MIMIC datasets, respectively. We have run each algorithm ten times and report the mean and standard deviation of the performance measures. As we can see, the proposed causal regularizer does not significantly hurt the predictive performance, whereas the two-step procedure significantly reduces the accuracy.
1 https://github.com/MIT-LCP/mimic-code/blob/master/concepts/cookbook/mortality.sql
An interesting phenomenon, shown in Figure 4, is the relative robustness of the performance with respect to the value of the penalization parameter compared to the L1 regularization case. This robustness comes at no surprise, because the causal regularizer assigns very small penalization coefficients to the causal variables and as we discussed in Section 2.2, only with very high values of penalization we can force all coefficients to become zero, see Figure 7 in Appendix A.1. Moreover, this robustness can be attributed to the fact that the causal regularizer might match the true generative process of the dataset better than the flat L1 regularizer and puts the model under less pressure as we increase the penalization parameter. We demonstrate the predictive gain by the non-linear causal model in Figure 5a. Furthermore, the impact of changing the regularization parameter on the number of selected variables is visualized in Figure 8 in Appendix A.1.
3.3 EVALUATING THE CAUSAL PERFORMANCE OF CAUSAL REGULARIZER
In order to evaluate the performance of the algorithms in their ability to identify causal factors, we generate the top 100 influential factors by different methods. We ask a clinical expert to label each factor as “causal”, “not-causal”, and “potentially causal” and assign scores 1, 0, and 0.5 to them, respectively. Table 2 shows the average causality score by each algorithm based on the labels provided by the medical expert. As expected, L1 regularized logistic regression performs poorly, as it is susceptible to the impact of confounded variables. Performance of the causally regularized logistic regression is superior to the two step procedure, which suggests that picking factors that are both causal and highly predictive leads to better causality score. This result together with the predictive results in Table 1 confirms that the causal regularizer can be efficiently used for finding few causal variables that are highly predictive of the target quantity.
The advantages of the regularized approach can also be seen by the results in Table 4. We have marked many disease codes that can potentially increase the risk of heart failure. However, the predicted causality score for them is lower than 0.5 and the two-step procedure would have eliminated from the predictors set (as shown in Table 10 in Appendix C). The causal regularizer approach is able to establish a balance between the prediction and causation and produce more plausible results.
3.4 EVALUATING THE MULTIVARIATE CAUSAL HYPOTHESES
We evaluate the performance of the proposed causal hypothesis generation against the case when we do not use any causal regularization. We generate two lists of top 50 hypotheses using two algorithms and ask our medical expert to label each hypothesis as causal, non-causal or possibly causal with corresponding scores of 1, 0, and 0.5. The results in Figure 5b shows that the causal regularizer can increase the causality score of the hypotheses by up to 20%. We also provide a qualitative analysis of
the causal hypotheses generated by our algorithm. To this end, we pick several hypotheses and show that clinically they are meaningful. Three examples of multivariate causal hypotheses generated via causal regularizer are shown in Table 3.
4 CONCLUSION
We addressed the problem of exploring the high-dimensional causal hypothesis space in applications such as healthcare. We designed a causal regularizer that steers predictive algorithms towards explanations “as causal as possible”. The proposed causal regularizer, based on our causality detector, does not increase the computational complexity of the L1 regularizer and can be seamlessly integrated with a neural network to perform non-linear causality analysis. We also demonstrated the application of the proposed causal regularizer in generating multivariate causal hypotheses. Finally, we demonstrated the usefulness of the causal regularizer in detecting the causes of heart failure using an electronic health records dataset.
ACKNOWLEDGMENT
The authors would like to thank Frederick Eberhardt for helpful discussions. Mohammad Taha Bahadori acknowledges the previous discussions with David C. Kale and Micheal E. Hankin on the concept of causal regularizer.
A THE SAMPLING PROCEDURE FOR COUNT INDEPENDENT VARIABLES
As described in Section 3, our independent variables have count data type. Thus, we need to generate data from distributions for count data, such as Poisson or Zipf distributions with fixed support size of 16. Looking at the histogram of maximum number of code occurrences in Figure 6, we observe that many codes only occur at most once or twice. Thus, we also generate binary and trinary distributions from flat Dirichlet distributions. Finally, to make sure that the space is fully spanned, we also generate samples from Dirichlet distribution with 16 categories. In summary, the dist(s,K) is the mixture of these five distributions. The parameters of Poisson and Zipf are sampled from χ2(1) distribution.
Let dist(s,K) denotes a discrete distribution with parameter s and given support size K.
Direct X → Y : 1. Sample s ∼ χ2(2). Generate PX = dist(s,K). 2. Sample PY |X ∼ Unif(0, 1) for K times. 3. Compute the 2K-dimensional vector
PX,Y (x, y) = [p(1, 0), . . . , p(K, 0), p(1, 1), . . . , p(K, 1)].
Algorithm 2: Another example of generating the synthetic dataset.
Sampling from the graphical models in Figure 2 is done writing the factorization and sampling from directed edges and finally marginalization with respect to hidden variables (Wainwright & Jordan, 2008). The hidden variables are selected to be categorical variables with cardinality selected uniformly from the integers in the interval [2, 100]. The conditional distribution of the hidden variables is selected to be Dirichlet distribution with all-ones parameter vector.
A.1 EVALUATING THE CAUSALITY DETECTOR
Table 5 show two advantages of the proposed sampling procedure for count data in Appendix A in comparison to the binary case proposed by Chalupka et al. (2016). First, in the synthetic dataset, the test error is significantly lower. This is because the size of input to the neural causality detector is 32 compared to 4 for the binary case. Applying the causality detectors to our data, we observe that the causality scores generated by our sampling scheme has significantly higher correlation with the mutual information between independent variables and the target label. Figure 10 in Appendix C
highlights another advantage of the sampling procedure for count data as it is able to identify a larger portion of the variables as non-causal, which is more inline the expectations. Table 7 in Appendix C shows that the mutual information identifies V70.0 (Routine general medical examination at a health care facility) as highly correlated, but the causality detector correctly identifies it as non-causal with causality score 0.0000.
In particular, in Figure 9, the Spearman’s rank correlation is ρ = 0.6689 which indicates a strong correlation. This is intuitive as we expect on average the causal connections to create stronger correlations. Another consequence of the large correlation makes regularization by the non-causality scores safer and guarantees that it will not significantly hurt the predictive performance. In Figure 9, we have marked four codes in the four corner of the figure. An example of highly correlated and causal code we can point out 250.00 (Diabetes mellitus without mention of complication) which is a known cause of heart failure. Code 362.01 (Background diabetic retinopathy) is an effect of diabetes —a common cause of heart failure. Code V06.5 (Need for TD vaccination) is an example of neither causal nor correlated code. Finally, code 365.00 (Preglaucoma) is known for increasing the risk of heart failure, despite the fact that it is not very correlated with heart failure.
B DETAILS OF THE NEURAL NETWORKS
B.1 NEURAL CAUSALITY DETECTOR ARCHITECTURE
We used a multilayer perceptron with seven layers of size 1024 with rectified linear units as activation functions. We use batch normalization for each layer. We used adamax for training and early stopping based on the validation accuracy. Implementation is done in Theano 0.8.
B.2 THE NEURAL NETWORK IN SECTION 2.2
After tuning the parameters of the neural network, we ended up using a multilayer perceptron with three layers with rectified linear units as activation functions. The embedding dimension was 128 obtained by training GloVe on the entire dataset. We used dropout with rate p = 0.8 and adadelta for optimization and early stopping based on the validation accuracy. Implementation is done in Theano 0.8.
C QUALITATIVE RESULTS
0 0.2 0.4 0.6 0.8 1 Causality Score
-5.5
-5
-4.5
-4
-3.5
-3
-2.5
-2
lo g 1
0( M
I)
362.01 250.00
V06.5
365.00
Figure 9: The scatter plot of causation score vs. mutual information.
Ta bl
e 6:
Pe rf
or m
an ce
of si
m pl
e ca
us al
ity de
te ct
or us
in g
co un
td at
a. T
he ca
us al
ity sc
or e
is th
e ou
tp ut
of ou
rn eu
ra ln
et w
or k
in Se
ct io
n 2.
1 P[ X
i → Y
]. M I
de no
te s
th e
m ut
ua li
nf or
m at
io n
be tw
ee n X
i an
d Y
.
C od
e D
es cr
ip tio
n lo
g (M
I )
C au
sa lit
y Sc
or e
V 58
.8 3
E nc
ou nt
er fo
rt he
ra pe
ut ic
dr ug
m on
ito ri
ng -2
.1 05
0 1.
00 00
25 0.
00 D
ia be
te s
m el
lit us
w ith
ou tm
en tio
n of
co m
pl ic
at io
n, ty
pe II
or un
sp ec
ifi ed
ty pe
,n ot
st at
ed as
un co
nt ro
lle d
-2 .0
96 2
1. 00 00 40 1. 9 U ns pe ci fie d es se nt ia lh yp er te ns io n -2 .1 02 3 1. 00 00 V 58 .6 1 L on gte rm (c ur re nt )u se of an tic oa gu la nt s -1 .8 37 1 1. 00 00 42 7. 31 A tr ia lfi br ill at io n -1 .5 79 5 1. 00 00 78 0. 52 In so m ni a, un sp ec ifi ed -3 .3 10 1 1. 00 00 27 2. 4 O th er an d un sp ec ifi ed hy pe rl ip id em ia -2 .6 50 0 1. 00 00 40 1. 1 B en ig n es se nt ia lh yp er te ns io n -2 .4 45 5 1. 00 00 70 2. 0 A ct in ic ke ra to si s -3 .1 49 3 1. 00 00 53 0. 81 E so ph ag ea lr efl ux -3 .3 65 8 1. 00 00 V 04 .8 1 N ee d fo rp ro ph yl ac tic va cc in at io n an d in oc ul at io n ag ai ns ti nfl ue nz a -3 .6 71 2 1. 00 00 V 10 .8 3 Pe rs on al hi st or y of ot he rm al ig na nt ne op la sm of sk in -3 .4 21 4 1. 00 00 25 0. 60 D ia be te s w ith ne ur ol og ic al m an if es ta tio ns ,t yp e II or un sp ec ifi ed ty pe ,n ot st at ed as un co nt ro lle d -2 .3 71 1 0. 99 99 41 4. 00 C or on ar y at he ro sc le ro si s of un sp ec ifi ed ty pe of ve ss el ,n at iv e or gr af t -1 .8 89 6 0. 99 99 25 0. 02 D ia be te s m el lit us w ith ou tm en tio n of co m pl ic at io n, ty pe II or un sp ec ifi ed ty pe ,u nc on tr ol le d -2 .4 81 9 0. 99 99 73 3. 00 O st eo po ro si s, un sp ec ifi ed -3 .4 93 9 0. 99 99 69 2. 74 O th er ch ro ni c de rm at iti s du e to so la rr ad ia tio n -2 .9 42 8 0. 99 99 24 4. 9 U ns pe ci fie d ac qu ir ed hy po th yr oi di sm -3 .1 73 6 0. 99 99 31 1 D ep re ss iv e di so rd er ,n ot el se w he re cl as si fie d -2 .8 48 2 0. 99 99 71 4. 0 R he um at oi d ar th ri tis -3 .0 01 0 0. 99 95 72 4. 2 L um ba go -3 .4 55 4 0. 99 95 49 3. 90 A st hm a, un sp ec ifi ed ty pe ,u ns pe ci fie d -2 .8 82 4 0. 99 94 V 43 .1 L en s re pl ac ed by ot he rm ea ns -2 .8 66 7 0. 99 94 25 0. 40 D ia be te s w ith re na lm an if es ta tio ns ,t yp e II or un sp ec ifi ed ty pe ,n ot st at ed as un co nt ro lle d -2 .3 95 2 0. 99 90 49 6 C hr on ic ai rw ay ob st ru ct io n, no te ls ew he re cl as si fie d -2 .2 99 3 0. 99 90 78 6. 2 C ou gh -2 .3 53 3 0. 99 87 17 3. 3 — - -4 .1 20 4 0. 99 64 73 3. 90 D is or de ro fb on e an d ca rt ila ge ,u ns pe ci fie d -3 .4 00 1 0. 99 19 78 6. 50 C he st pa in ,u ns pe ci fie d -2 .5 65 1 0. 99 15 28 5. 9 A ne m ia ,u ns pe ci fie d -2 .1 52 6 0. 99 05
Ta bl
e 7:
To p
20 co
de s
w ith
hi gh
es tm
ut ua
li nf
or m
at io
n w
ith th
e he
ar tf
ai lu
re ou
tc om
e. T
he ca
us al
ity sc
or e
is th
e ou
tp ut
of ou
r ne
ur al
ne tw
or k
in Se
ct io
n 2. 1 P[ X i → Y ]. M I de no te s th e m ut ua li nf or m at io n be tw ee n X i an d Y .
C od
e D
es cr
ip tio
n lo
g (M
I )
C au
sa lit
y Sc
or e
78 2.
3 E
de m
a -1
.7 35
5 0.
00 27
42 4.
1 A
or tic
va lv
e di
so rd
er s
-2 .2
02 1
0. 00 12 42 5. 4 O th er pr im ar y ca rd io m yo pa th ie s -2 .2 33 0 0. 13 51 V 70 .0 R ou tin e ge ne ra lm ed ic al ex am in at io n at a he al th ca re fa ci lit y -2 .2 42 0 0. 00 00 44 3. 9 Pe ri ph er al va sc ul ar di se as e, un sp ec ifi ed -2 .2 66 8 0. 11 45 42 4. 0 M itr al va lv e di so rd er s -2 .3 08 8 0. 00 03 25 0. 50 D ia be te s w ith op ht ha lm ic m an if es ta tio ns ,t yp e II or un sp ec ifi ed ty pe ,n ot st at ed as un co nt ro lle d -2 .3 28 8 0. 22 71 51 1. 9 U ns pe ci fie d pl eu ra le ff us io n -2 .3 32 0 0. 08 39 42 7. 32 A tr ia lfl ut te r -2 .3 50 8 0. 07 67 27 8. 01 M or bi d ob es ity -2 .3 92 4 0. 03 45 42 5. 8 C ar di om yo pa th y in ot he rd is ea se s cl as si fie d el se w he re -2 .4 09 0 0. 23 22 36 2. 01 B ac kg ro un d di ab et ic re tin op at hy -2 .4 53 6 0. 08 15 58 4. 9 A cu te ki dn ey fa ilu re ,u ns pe ci fie d -2 .4 66 1 0. 28 82 41 2 O ld m yo ca rd ia li nf ar ct io n -2 .4 75 0 0. 07 80 42 8. 0 C on ge st iv e he ar tf ai lu re ,u ns pe ci fie d -2 .4 94 6 0. 00 39 79 1. 0 Pr ot ei nu ri a -2 .4 98 4 0. 18 83 35 7. 2 Po ly ne ur op at hy in di ab et es -2 .5 04 0 0. 21 20 40 2. 90 U ns pe ci fie d hy pe rt en si ve he ar td is ea se w ith ou th ea rt fa ilu re -2 .5 10 3 0. 20 34 25 0. 42 D ia be te s w ith re na lm an if es ta tio ns ,t yp e II or un sp ec ifi ed ty pe ,u nc on tr ol le d -2 .5 31 0 0. 23 78 28 0. 9 Ir on de fic ie nc y an em ia ,u ns pe ci fie d -2 .5 66 1 0. 02 83 42 7. 1 Pa ro xy sm al ve nt ri cu la rt ac hy ca rd ia -2 .5 88 4 0. 03 67 V 53 .3 1 Fi tt in g an d ad ju st m en to fc ar di ac pa ce m ak er -2 .6 01 4 0. 28 94 45 9. 81 V en ou s (p er ip he ra l) in su ffi ci en cy ,u ns pe ci fie d -2 .6 09 3 0. 07 48 41 0. 90 A cu te m yo ca rd ia li nf ar ct io n of un sp ec ifi ed si te ,e pi so de of ca re un sp ec ifi ed -2 .6 15 4 0. 25 10 58 8. 81 Se co nd ar y hy pe rp ar at hy ro id is m (o fr en al or ig in ) -2 .6 19 9 0. 04 78 25 0. 62 D ia be te s w ith ne ur ol og ic al m an if es ta tio ns ,t yp e II or un sp ec ifi ed ty pe ,u nc on tr ol le d -2 .6 36 7 0. 20 51 41 4. 8 O th er sp ec ifi ed fo rm s of ch ro ni c is ch em ic he ar td is ea se -2 .6 40 3 0. 09 23 36 2. 02 Pr ol if er at iv e di ab et ic re tin op at hy -2 .6 49 0 0. 26 29 58 6 R en al fa ilu re ,u ns pe ci fie d -2 .7 32 4 0. 23 88 25 0. 52 D ia be te s w ith op ht ha lm ic m an if es ta tio ns ,t yp e II or un sp ec ifi ed ty pe ,u nc on tr ol le d -2 .7 37 5 0. 21 54
Ta bl
e 8:
To p
20 ca
us al
co de
s (c
i ≥
0 .5
)w ith
hi gh
es tm
ut ua
li nf
or m
at io
n. T
he ca
us al
ity sc
or e
is th
e ou
tp ut
of ou
rn eu
ra ln
et w
or k
in Se
ct io
n 2.
1 P[ X
i → Y
]. M I
de no
te s
th e
m ut
ua li
nf or
m at
io n
be tw
ee n X
i an
d Y
.
C od
e D
es cr
ip tio
n lo
g (M
I )
C au
sa lit
y Sc
or e
42 7.
31 A
tr ia
lfi br
ill at
io n
-1 .5
79 5
1. 00 00 V 58 .6 1 L on gte rm (c ur re nt )u se of an tic oa gu la nt s -1 .8 37 1 1. 00 00 41 4. 00 C or on ar y at he ro sc le ro si s of un sp ec ifi ed ty pe of ve ss el ,n at iv e or gr af t -1 .8 89 6 0. 99 99 25 0. 00 D ia be te s m el lit us w ith ou tm en tio n of co m pl ic at io n, ty pe II or un sp ec ifi ed ty pe ,n ot st at ed as un co nt ro lle d -2 .0 96 2 1. 00 00 40 1. 9 U ns pe ci fie d es se nt ia lh yp er te ns io n -2 .1 02 3 1. 00 00 V 58 .8 3 E nc ou nt er fo rt he ra pe ut ic dr ug m on ito ri ng -2 .1 05 0 1. 00 00 28 5. 9 A ne m ia ,u ns pe ci fie d -2 .1 52 6 0. 99 05 49 6 C hr on ic ai rw ay ob st ru ct io n, no te ls ew he re cl as si fie d -2 .2 99 3 0. 99 90 42 7. 9 C ar di ac dy sr hy th m ia ,u ns pe ci fie d -2 .3 10 0 0. 98 64 58 5. 3 C hr on ic ki dn ey di se as e, St ag e II I( m od er at e) -2 .3 40 6 0. 94 47 78 6. 2 C ou gh -2 .3 53 3 0. 99 87 25 0. 60 D ia be te s w ith ne ur ol og ic al m an if es ta tio ns ,t yp e II or un sp ec ifi ed ty pe ,n ot st at ed as un co nt ro lle d -2 .3 71 1 0. 99 99 25 0. 40 D ia be te s w ith re na lm an if es ta tio ns ,t yp e II or un sp ec ifi ed ty pe ,n ot st at ed as un co nt ro lle d -2 .3 95 2 0. 99 90 58 5. 4 C hr on ic ki dn ey di se as e, St ag e IV (s ev er e) -2 .4 41 9 0. 80 91 40 1. 1 B en ig n es se nt ia lh yp er te ns io n -2 .4 45 5 1. 00 00 28 5. 21 A ne m ia in ch ro ni c ki dn ey di se as e -2 .4 73 4 0. 83 84 25 0. 02 D ia be te s m el lit us w ith ou tm en tio n of co m pl ic at io n, ty pe II or un sp ec ifi ed ty pe ,u nc on tr ol le d -2 .4 81 9 0. 99 99 78 6. 50 C he st pa in ,u ns pe ci fie d -2 .5 65 1 0. 99 15 70 3. 8 O th er sp ec ifi ed di se as es of na il -2 .5 65 9 0. 95 83 42 7. 89 O th er sp ec ifi ed ca rd ia c dy sr hy th m ia s -2 .6 15 3 0. 94 16 78 0. 79 O th er m al ai se an d fa tig ue -2 .6 17 4 0. 91 69 27 2. 4 O th er an d un sp ec ifi ed hy pe rl ip id em ia -2 .6 50 0 1. 00 00 25 0. 01 D ia be te s m el lit us w ith ou tm en tio n of co m pl ic at io n, ty pe I[ ju ve ni le ty pe ], no ts ta te d as un co nt ro lle d -2 .7 04 5 0. 80 49 79 4. 31 N on sp ec ifi c ab no rm al el ec tr oc ar di og ra m [E C G ][ E K G ] -2 .7 60 2 0. 93 51 78 5. 9 O th er sy m pt om s in vo lv in g ca rd io va sc ul ar sy st em -2 .7 71 9 0. 80 24 72 4. 02 Sp in al st en os is ,l um ba rr eg io n, w ith ou tn eu ro ge ni c cl au di ca tio n -2 .7 87 4 0. 89 32 42 5. 9 Se co nd ar y ca rd io m yo pa th y, un sp ec ifi ed -2 .8 07 1 0. 80 24 27 4. 9 G ou t, un sp ec ifi ed -2 .8 38 0 0. 95 39 31 1 D ep re ss iv e di so rd er ,n ot el se w he re cl as si fie d -2 .8 48 2 0. 99 99 10 00 0 — - -2 .8 50 4 0. 97 21
Ta bl
e 9:
To p
30 co
de s
w ith
ca us
al re
gu la
ri za
tio n.
T he
co ef
fic ie
nt is w
i fr
om E
q. (1
).
C od
e D
es cr
ip tio
n C
oe ffi
ci en
t C
au sa
lit y
Sc or e 79 4. 31 N on sp ec ifi c ab no rm al el ec tr oc ar di og ra m [E C G ][ E K G ] 0. 34 22 0. 93 51 42 5. 8 C ar di om yo pa th y in ot he rd is ea se s cl as si fie d el se w he re 0. 32 72 0. 23 22 78 6. 05 Sh or tn es s of br ea th 0. 31 24 0. 55 36 42 4. 90 E nd oc ar di tis ,v al ve un sp ec ifi ed ,u ns pe ci fie d ca us e 0. 30 86 0. 39 08 42 5. 4 O th er pr im ar y ca rd io m yo pa th ie s 0. 28 80 0. 13 51 42 7. 9 C ar di ac dy sr hy th m ia ,u ns pe ci fie d 0. 25 31 0. 98 64 78 5. 9 O th er sy m pt om s in vo lv in g ca rd io va sc ul ar sy st em 0. 23 77 0. 80 24 58 5. 6 E nd st ag e re na ld is ea se 0. 22 25 0. 39 48 51 1. 9 U ns pe ci fie d pl eu ra le ff us io n 0. 22 18 0. 08 39 42 5. 9 Se co nd ar y ca rd io m yo pa th y, un sp ec ifi ed 0. 22 03 0. 80 24 78 2. 3 E de m a 0. 20 65 0. 00 27 27 8. 01 M or bi d ob es ity 0. 19 55 0. 03 45 42 4. 0 M itr al va lv e di so rd er s 0. 19 48 0. 00 03 42 7. 31 A tr ia lfi br ill at io n 0. 17 62 1. 00 00 41 0. 90 A cu te m yo ca rd ia li nf ar ct io n of un sp ec ifi ed si te ,e pi so de of ca re un sp ec ifi ed 0. 17 56 0. 25 10 42 6. 3 O th er le ft bu nd le br an ch bl oc k 0. 16 90 0. 48 90 42 4. 1 A or tic va lv e di so rd er s 0. 16 49 0. 00 12 87 9. 8 O pe n w ou nd (s )( m ul tip le )o fu ns pe ci fie d si te (s ), w ith ou tm en tio n of co m pl ic at io n 0. 16 45 0. 63 99 42 9. 3 C ar di om eg al y 0. 16 19 0. 50 22 78 0. 60 Fe ve r, un sp ec ifi ed 0. 16 02 0. 77 47 48 2. 9 B ac te ri al pn eu m on ia ,u ns pe ci fie d 0. 15 14 0. 74 82 78 6. 09 O th er re sp ir at or y ab no rm al iti es 0. 14 54 0. 73 05 49 6 C hr on ic ai rw ay ob st ru ct io n, no te ls ew he re cl as si fie d 0. 14 03 0. 99 90 V 42 .0 K id ne y re pl ac ed by tr an sp la nt 0. 13 98 0. 43 51 25 0. 03 D ia be te s m el lit us w ith ou tm en tio n of co m pl ic at io n, ty pe I[ ju ve ni le ty pe ], un co nt ro lle d 0. 13 88 0. 47 27 27 6. 51 D eh yd ra tio n 0. 13 47 0. 67 38 40 3. 10 H yp er te ns iv e ch ro ni c ki dn ey di se as e, be ni gn ,w ith ch ro ni c ki dn ey di se as e st ag e It hr ou gh st ag e IV ,o ru ns pe ci fie d 0. 13 16 0. 74 88 25 0. 50 D ia be te s w ith op ht ha lm ic m an if es ta tio ns ,t yp e II or un sp ec ifi ed ty pe ,n ot st at ed as un co nt ro lle d 0. 12 83 0. 22 71 42 7. 89 O th er sp ec ifi ed ca rd ia c dy sr hy th m ia s 0. 12 82 0. 94 16 25 0. 51 D ia be te s w ith op ht ha lm ic m an if es ta tio ns ,t yp e I[ ju ve ni le ty pe ], no ts ta te d as un co nt ro lle d 0. 12 34 0. 54 73
Ta bl
e 10
:T op
30 co
de s
w ith
th e
tw o-
st ep
pr oc
ed ur
e. T
he co
ef fic
ie nt
is th
e lo
gi st
ic re
gr es
si on
co ef
fic ie
nt w
ith L 1
re gu
la ri
za tio
n af
te rc
au sa
lv ar
ia bl
e se
le ct
io n.
C od
e D
es cr
ip tio
n C
oe ffi
ci en
t C
au sa
lit y
Sc or e 78 6. 05 Sh or tn es s of br ea th 0. 36 20 0. 55 36 42 7. 9 C ar di ac dy sr hy th m ia ,u ns pe ci fie d 0. 25 01 0. 98 64 79 4. 31 N on sp ec ifi c ab no rm al el ec tr oc ar di og ra m [E C G ][ E K G ] 0. 21 00 0. 93 51 42 7. 31 A tr ia lfi br ill at io n 0. 20 17 1. 00 00 78 5. 9 O th er sy m pt om s in vo lv in g ca rd io va sc ul ar sy st em 0. 17 81 0. 80 24 78 6. 09 O th er re sp ir at or y ab no rm al iti es 0. 17 72 0. 73 05 41 4. 00 C or on ar y at he ro sc le ro si s of un sp ec ifi ed ty pe of ve ss el ,n at iv e or gr af t 0. 14 78 0. 99 99 49 6 C hr on ic ai rw ay ob st ru ct io n, no te ls ew he re cl as si fie d 0. 14 65 0. 99 90 70 7. 15 U lc er of ot he rp ar to ff oo t 0. 14 07 0. 55 19 70 7. 14 U lc er of he el an d m id fo ot 0. 13 83 0. 51 13 87 9. 8 O pe n w ou nd (s )( m ul tip le )o fu ns pe ci fie d si te (s ), w ith ou tm en tio n of co m pl ic at io n 0. 13 34 0. 63 99 68 2. 9 C el lu lit is an d ab sc es s of un sp ec ifi ed si te s 0. 13 28 0. 59 89 58 5. 4 C hr on ic ki dn ey di se as e, St ag e IV (s ev er e) 0. 13 22 0. 80 91 41 4. 01 C or on ar y at he ro sc le ro si s of na tiv e co ro na ry ar te ry 0. 12 53 0. 69 30 25 0. 51 D ia be te s w ith op ht ha lm ic m an if es ta tio ns ,t yp e I[ ju ve ni le ty pe ], no ts ta te d as un co nt ro lle d 0. 11 80 0. 54 73 28 5. 9 A ne m ia ,u ns pe ci fie d 0. 11 72 0. 99 05 25 0. 02 D ia be te s m el lit us w ith ou tm en tio n of co m pl ic at io n, ty pe II or un sp ec ifi ed ty pe ,u nc on tr ol le d 0. 11 72 0. 99 99 42 5. 9 Se co nd ar y ca rd io m yo pa th y, un sp ec ifi ed 0. 11 32 0. 80 24 25 0. 40 D ia be te s w ith re na lm an if es ta tio ns ,t yp e II or un sp ec ifi ed ty pe ,n ot st at ed as un co nt ro lle d 0. 10 40 0. 99 90 42 7. 89 O th er sp ec ifi ed ca rd ia c dy sr hy th m ia s 0. 10 24 0. 94 16 42 9. 3 C ar di om eg al y 0. 09 68 0. 50 22 78 5. 0 Ta ch yc ar di a, un sp ec ifi ed 0. 08 28 0. 53 67 72 4. 02 Sp in al st en os is ,l um ba rr eg io n, w ith ou tn eu ro ge ni c cl au di ca tio n 0. 08 14 0. 89 32 48 2. 9 B ac te ri al pn eu m on ia ,u ns pe ci fie d 0. 07 96 0. 74 82 78 6. 50 C he st pa in ,u ns pe ci fie d 0. 07 89 0. 99 15 25 0. 00 D ia be te s m el lit us w ith ou tm en tio n of co m pl ic at io n, ty pe II or un sp ec ifi ed ty pe ,n ot st at ed as un co nt ro lle d 0. 07 60 1. 00 00 V 43 .3 H ea rt va lv e re pl ac ed by ot he rm ea ns 0. 07 49 0. 85 58 43 4. 91 C er eb ra la rt er y oc cl us io n, un sp ec ifi ed w ith ce re br al in fa rc tio n 0. 06 88 0. 62 49 28 5. 21 A ne m ia in ch ro ni c ki dn ey di se as e 0. 06 84 0. 83 84 V 72 .8 4 Pr eop er at iv e ex am in at io n, un sp ec ifi ed 0. 06 29 0. 98 66

Review:
The present submission discusses a "causal regularizer", which promotes the use of causal dependencies (X -> Y, where X is a feature of the learning problem, and Y is the target variable) in predictive models. Similarly, such causal regularizer penalizes the use of non-causal dependencies, which can arise due to reverse causation (Y -> X) or confounding (X <- Z -> Y, where Z is a hidden confounder).
+ Overall, this submission tackles one of the most important problems in machine learning, which is to build causal models. The paper discusses and addresses this issue effectively when applied to a dataset in heart disease. In their experiments, the authors correctly identify some of the common causes of heart disease by virtue of their causal regularizer.
- The authors do not discuss the robustness of their approach with respect to choice of hyper-parameters (both describing the neural network architecture and the generative model that synthesizes artificial causal data). This seems like a crucial issue, in particular when dealing with medical data.
- The conclusions of the experimental evaluation should be discussed in greater length. On the one hand, Figure 4.a shows that there are no differences between L1 and causal regularization in terms of predictive performance, but it is difficult to conclude if this result is statistically significant without access to error-bars. On the other hand, Table 3 describes the qualitative differences between L1 and causal regularization. However, this table is hard to read: How were the 30 rows selected? What does the red highlighting mean? Are these red rows some true causal features that were missed? If so, this is related to precision. What about recall? Did the causal regularization pick up many non-causal features as causal?
- Regarding causal classifiers, this paper should do a much better job at reviewing previous work. For instance, the paper "Towards a Learning Theory of Cause-Effect Inference" from Lopez-Paz et al. is missing from the references. However, this prior work studies many of the aspects that are hinted as novel in this submission. In particular, the prior work of Lopez-Paz 1) introduces the concept of Mother distribution (referred as Nature hyper-prior in this submission) which explicitly factorizes the distribution over causes and mechanisms, 2) circumvented intractable likelihoods by synthesizing and training on causal data, 3) tackled the confounding case (compare Figure 1 of this submission and Appendix C of Lopez-Paz), and 4) dealt with discrete data seamlessly (such as the ChaLearn data from Section 5.3 in Lopez-Paz).
On a positive note, this is a well-written paper that addresses the important, under-appreciated problem of incorporating causal reasoning into machine learning. On a negative note, the novelty of the technical contributions is modest and the qualitative evaluation of the results could be greatly extended. In short, I am leaning slightly towards acceptance.