D ECONFOUNDING R EINFORCEMENT L EARNING IN O BSERVATIONAL S ETTINGS
Anonymous authors
Paper under double-blind review
A BSTRACT
In this paper, we propose a general formulation to cope with a family of reinforcement learning tasks in observational settings, that is, learning good policies solely from the historical data produced by real environments with confounders (i.e., the factors affecting both actions and rewards).
Based on the proposed approach, we extend one representative of reinforcement learning algorithms: the ActorCritic method, to its deconfounding variant, which is also straightforward to be applied to other algorithms.
In addition, due to lack of datasets in this direction, a benchmark is developed for deconfounding reinforcement learning algorithms by revising OpenAI Gym and MNIST.
We demonstrate that the proposed algorithms are superior to traditional reinforcement learning algorithms in confounded environments.
To the best of our knowledge, this is the ﬁrst time that confounders are taken into consideration for addressing full reinforcement learning problems.
I NTRODUCTION
In recent years, reinforcement learning (RL) has made great progress, spawning a large number of successful applications especially in terms of games (Silver et al., 2016; Mnih et al., 2013; OpenAI, 2018). Within this background, much attention has been devoted to the development of RL algorithms with the goal of improving treatment policies in healthcare (Gottesman et al., 2018).
In fact, various RL algorithms have been proposed to infer better decision-making strategies for mechanical ventilation (Prasad et al., 2017), sepsis management (Raghu et al., 2017a;b), and treatment of schizophrenia (Shortreed et al., 2011). In healthcare, a common practice is to focus on the observational setting, because ones do not wish to experiment with patients’ lives without evidence that the proposed treatment strategy is better than the current practice (Gottesman et al., 2018) 1 . As pointed out in (Raghu et al., 2017a), even if in the observational setting, RL also has advantages over other machine learning algorithms especially in two situations: when the ground truth of a “good” treatment strategy is unclear in medical literature (Marik, 2015), and when training examples do not represent optimal behavior. On the other hand, although causal inference (Pearl, 2009) has been extensively explored and used in healthcare and medicine (Liu et al.;
Soleimani et al., 2017; Schulam & Saria, 2017; Alaa et al., 2017; Alaa & van der Schaar, 2018;
Atan et al., 2016), the efﬁcient approach to dealing with time-varying data is still unclear (Peters et al., 2017; Hernn & Robins, 2018).
On the basis of the discussion above, in this paper we attempt to combine advantages on both sides to cope with an important family of RL problems in the observational setting, that is, learning good policies solely from the historical data produced by real environments with confounding bias .
This type of problems are inevitable in the future RL research with the burgeoning development in healthcare and medicine.
To the best of our knowledge, however, little work has been done in the direction.
Confounding is a causal concept that is described in the language of causality instead of probability and statistics (Pearl, 2009).
Confounding bias occurs when a variable inﬂuences both who is selected for the treatment and the outcome of the experiment (Pearl & Mackenzie, 2018), which naturally corresponds to the action and the reward in RL, respectively.
As a matter of fact, confounders have been extensively studied in epidemiology, sociology, and economics.
Take for example the widespread kidney stones in which the size of the kidney stone is a confounding factor affecting both the treatment and the recovery (Peters et al., 2017; Pearl, 2009), whether deconfounding the size of the kidney stone or not entirely determines how to choose a more effective treatment.
Similarly, in RL, if unobserved potential confounders exist, they would affect both ac- tions and rewards when an agent interacts with environments and eventually inﬂuence the policy to be optimized.
It is widely acknowledged that one should draw a causal graph before one can achieve any causal conclusion (Pearl, 2009; Pearl & Mackenzie, 2018).
Throughout the paper, we assume that, given causal assumptions, we ﬁrst estimate a model from the observational data we collected from real environments or simulators, and then optimize a policy on the basis of the learned model.
As mentioned previously, this assumption is quite useful in real-world RL applications (e.g., healthcare, medicine, economics, etc), because in most circumstances, except the data we observed, we neither know anything about real environments nor are allowed to do anything in real environments probably for the sake of ethics, laws or cost.
In order to adjust for confounders, we present a general formulation for addressing this class of RL problems, namely deconfounding reinforcement learning (DRL).
More speciﬁcally, given several common confounding assumptions, under some conditions for identiﬁcation we ﬁrst estimate a latent-variable model from observational data in which we simultaneously discover the latent confounders and infer how they affect action and reward, then deconfound the confounders using the causal language developed by (Pearl, 2009), and ﬁnally optimize the policy based on the deconfounding model we calculated.
On the basis of the proposed formulation, we extend one popular RL algorithm, the Actor-Critic method, to its corresponding deconfounding variant, which is straightforward to be applied to other RL algorithms.
Due to lack of datasets in this respect, we revise the classic control toolkit in OpenAI Gym (Brockman et al., 2016), making it a benchmark for comparison of DRL algorithms.
In addition, we also devise a confounding version of the MNIST dataset (LeCun et al., 1998) to verify the performance of our causal model.
Finally, we conduct extensive experiments to demonstrate the superiority of the proposed formulation in confounded environments, in comparison to traditional RL algorithms.
To sum up, our contributions in this paper are as follows:
We propose a general formulation to address a family of RL problems in confounded environments, namely deconfounding reinforcement learning (DRL); 2.
We present the deconfounding variant of Actor-Critic methods, which is obvious to be applied to other RL methods; 3.
We develop a benchmark for DRL by revising the toolkit for classic control in OpenAI Gym (Brockman et al., 2016) and by devising a confounding version of the MNIST dataset (LeCun et al., 1998); 4.
We perform a comprehensive comparison of DRL algorithms with their vanilla versions, showing that the proposed approach has an advantage in confounded environments.
To the best of our knowledge, this is the ﬁrst attempt to build a bridge between confounding and the full RL problem.
This is one of few research papers aiming at understanding the connections between causal inference and the full RL.
B ACKGROUND
In this section, we brieﬂy review confounding in causal inference.
We recommend Pearls excellent monograph for further reading (Pearl, 2009; Pearl & Mackenzie, 2018).
2. S IMPSON ’ S P ARADOX
Let us begin with one of the most famous paradoxes in statistics: Simpson’s Paradox.
Consider the previously mentioned kidney stones, a classic example of Simpson’s paradox (Peters et al., 2017).
We collect electronic patient records to investigate the effectiveness of two treatments against kidney stones, where although the overall probability of recovery is higher for patients who took treatment b , treatment a performs better than treatment b on both patients with small kidney stones and with large kidney stones.
More precisely, we have
[Equation]
but [Equation] where Z is the size of the stone, T the treatment, and R the recovery (all binary).
How do we cope with this inversion of conclusion?
Which treatment do you prefer if you had kidney stones?
Does treatment b cause recovery?
The answers to these questions depend on the causal relationship between treatment, recovery, and the size of the kidney stone.
2. C ONFOUNDING
An intuitive explanation for this kidney stone example of Simpson’s paradox is that larger stones are more severe than small stones and are much more likely to be treated with treatment a , resulting in that treatment a looks worse than treatment b .
Therefore, it is straightforward to assume that the true underlying causal diagram of the kidney stone example is shown in Figure 1(a), where confounding occurs because the size of kidney stones inﬂuences both treatment and recovery.
Here, the size of kidney stones is called confounder.
The term “confounding” originally meant “mixing” in English, which describes that the true causal effect T !
R is “mixed” with the spurious correlation between
T and R induced by the fork T
Z !
R (Pearl & Mackenzie, 2018).
In other words, we will not be able to disentangle the true effect of T on R from the spurious effect if we do not have data on Z .
Conversely, if we have measurements of Z , it is easy to deconfound the true and spurious effects by adjusting for Z that averages the effect of T on R in each subgroup of Z (i.e., different size groups in the case of kidney stones).
2. D O -O PERATOR AND A DJUSTMENT C RITERION
From the viewpoint of causal inference, we can also use the language of intervention, namely dooperator , to formulate confounding.
In fact, in the example of kidney stones, what we are interested in is how these two treatments compare when we force all patients to take treatment a or b , rather than which treatment has a higher recovery rate given only the observational patient records.
Mathematically, we focus on the true effect p ( R = 1 j do ( T = a ))
(i.e., intervention distribution where patients are forced to take treatment a ) instead of the spurious effect p ( R = 1 j T = a ) (i.e., observational distribution where patients are observed to take treatment a ).
Therefore, as described previously, confounding can be naturally formulated by the discrepancy between p ( R j T ) and p ( R j do ( T )) .
Generally speaking, do-operator can be executed in two common ways: by Randomized Controlled Trials (RCTs) (Fisher, 1935) and by adjustment formulas (i.e., Back-door criterion and Front-door criterion) (Pearl, 2009).
RCTs is the so-called golden standard but rather limited due to many impractical factors (e.g., safety, laws, ethics, physically infeasibility, etc.).
Back-door and Front-door criterions require a known causal diagram in which causal assumptions are provided in advance.
According to the Back-door criterion, in the kidney stone example, we can immediately attain
[Equation]
In fact, apart from the two adjustment formulas, we are provided with a syntactic method of deriving claims about interventions, namely do -calculus, developed by Pearl (Pearl, 2009).
The do -calculus consists of three rules which can be repeatedly applied to simplify the expression for an interventional distribution.
It offers a powerful tool to convert intervention probabilities to observation probabilities so that we can estimate the former from observational data alone.
2. P ROXY V ARIABLES FOR C ONFOUNDING
If confounders can be measured, then they can be adjusted for through the methods we discussed in Section 2.3.
However, in most cases where confounders are hidden or unmeasured, without further assumptions, it is impossible to estimate the effect of the intervention on the outcome.
A common practice is to leverage observed proxy variables of the confounders (Angrist & Pischke, 2008; Maddala & Lahiri, 1992; Montgomery et al., 2000), which is one of the promising directions of exploiting big-data to conduct causal inference in the presence of multiple proxy variables for unmeasured confounders (Louizos et al., 2017).
However, using proxy variables to correctly recover causal effects should meet strict mathematical assumptions (Louizos et al., 2017; Edwards et al., 2015; Kuroki & Pearl, 2014;
Miao et al., 2018; Pearl, 2012; Wooldridge, 2009).
Unfortunately, in practice, we do not know whether or not the complicated data from the real world meet those assumptions.
Hence, following the same spirit from (Louizos et al., 2017), with multiple proxy variables available, under relaxed assumptions we estimate a latent-variable model in which we simultaneously discover the latent confounders and infer how they affect treatment and outcome.
D ECONFOUNDING R EINFORCEMENT L EARNING
In this section, we will formally introduce deconfounding reinforcement learning (DRL).
Generally speaking, DRL consists of two steps: learning a deconfounding model shown in Figure 1(b) and optimizing a policy based on the learned deconfounding model.
The main idea in step 1 is to simultaneously discover the hidden confounder and infer the causal effect through an estimation of a latent-variable model. More speciﬁcally, we ﬁrst discuss the time-independent confounding assumption in Section 3.1, and then, based on the assumption, formalize the deconfounding model in Section 3.2. Section 3.3 talks about the problem of identiﬁcation in our model, which is a central issue in causal inference.
After that, we present the details about how to learn the model via variational inference in Section 3.4.
Step 2 is provided in Section 3.5 where we describe how to design the deconfounding actor-critic method.
It is straightforward to apply this to other RL algorithms in the same manner.
3. C AUSAL A SSUMPTIONS
Without loss of generality, as shown in Figure 1(b) 2 , we assume there exists a common confounder in the sequential model, which is time-independent for each individual or for each procedure.
This assumption is so general that it would apply to various RL tasks across domains.
For example, in personalized medicine or precision medicine, socio-economic status can affect both the medication strategy a patient has access to, and the patients general health (Louizos et al., 2017).
Therefore socio-economic status acts as confounder between the medication and health outcomes, in which case socio-economic status is time-independent for each patient during the course of treatment.
In agriculture, soil fertility may serve as one of confounders affecting both the application of fertilizer and the yield of each plot (Pearl & Mackenzie, 2018).
In this circumstance, soil fertility is stable and thought of as a time-independent factor within a period of time (e.g., several months, the growth circle of crops, etc.).
In the example of stock markets, apart from socio-economic status as mentioned above, government policy may also act as one of confounders, all of which can be seen time-independent during a reasonable period of time.
3. T HE M ODEL
Given the causal assumption, we ﬁrst ﬁt a generative model to a sequence of observational data: observations, actions, and rewards, where actions and rewards are confounded by one or several unknown factors.
Formally, Let ⃗x =
( x 1 ; : : : ; x T ) , ⃗a = ( a 1 ; : : : ; a T (cid:0) 1 ) , ⃗r = ( r 2 ; : : : ; r T +1 ) , ⃗z = ( z 1 ; : : : ; z T ) be the sequence of observations, actions, rewards, and corresponding latent states, respectively.
The confounder is denoted by u , and it is worth noting that here u may stand for more than one confounder in which multiple confounders are seen as a whole represented by u .
We assume that x t 2 R D x , a t 2 R D a , r t 2 R D r , z t 2 R D z , and u 2 R D u , where D z ≪ D x .
The generative model for DRL is then given by:
[Equation]
2), the arrow from z t to a t is not necessary when learning the model so that z t can be viewed as not a confounder of a t and r t +1 .
This also provides another reason why we do not need to adjust for z t .
However, in some other cases such as when applied to medical data, the strategies of treatment from physicians deﬁnitely contain valuable information about z t and a t , and therefore the arrow between them is necessary when learning the model.
[Equation]
Note that we parametrize each probability distribution as a Gaussian with its mean and variance modeled by nonlinear functions f k and each f k is parametrized by a neural network with its own parameters (cid:18) k for k = 1 ; : : : ; 8 .
Note that Equation (4) is not necessary in our model, but it is potentially useful acting as a prior policy because it is learned from the observational data containing, for example, the real treatment strategies by doctors. 3. I DENTIFICATION OF C AUSAL E FFECT In our deconfounding model shown in Figure 1(b), the key difference from traditional RL lies in the reward function 3 .
To be more precise, assuming that an agent standing at state z t performs an action a t = a , unlike the traditional reward p ( r t j z t ; a t = a ) , our deconfounding version based on do-operator as depicted in Section 2.3 is given by
[Equation]
where Equation 8 is by the rules of do -calculus applied to the causal graph in Figure 1(b) (Pearl, 2009).
In fact, we can utilize the back-door criterion to directly obtain Equation (8). Furthermore, through Equation (8) we proved that p ( r t j z t ; do ( a t = a )) can be identiﬁed from the joint distribution p ( u;⃗z; ⃗x;⃗a;⃗r ) . Then, motivated by the idea that one is supposed to use the knowledge inferred from the joint distribution between proxy variables and confounders to adjust for the hidden confounders (Louizos et al., 2017; Edwards et al., 2015; Kuroki & Pearl, 2014; Miao et al., 2018; Pearl, 2012; Wooldridge, 2009), in this paper we also assume that the joint distribution p ( u;⃗z; ⃗x;⃗a;⃗r ) can be approximately recovered solely from the observations ( ⃗x;⃗a;⃗r ) .
The details about identiﬁcation can be found in Appendix A. 3. L EARNING Since the nonlinear functions parametrized by neural networks make inference intractable, we will learn the parameters of the model (cid:18) k by employing variational inference along with an inference model, a neural network which approximates the intractable posterior (Rezende et al., 2014; Kingma & Welling, 2013; Krishnan et al., 2015).
More speciﬁcally, using the variational principle, we posit an approximate posterior distribution q ϕ (
z j x ) to obtain the following lower bound on the marginal likelihood:
[Equation]
where the inequality is by Jensen’s inequality and ϕ is the parameter of the inference model q ( z j x ) .
Note that in this general case x stands for observational variables and z for latent variables.
3.4. V ARIATIONAL L OWER B OUND
Directly applying the lower bound in Inequality (9) to our model [Equation] we obtain
[Equation]
Using the Markov property of our model
[Equation]
the full distribution can be factorized in the following way:
[Equation]
In addition
[Equation]
for simplicity’s sake
[Equation]
we also have the factorization assumption for the posterior approximation:
[Equation]
Combining Equation (10), (11) and (12) yields:
[Equation]
where we omit subscripts (cid:18) and ϕ , and a more detailed derivation can be found in Appendix B.
Obviously, Equation (13) is differentiable with respect to the parameters of the model (cid:18) and ϕ .
Using the reparametrization trick (Kingma & Welling, 2013), we can directly apply backpropagation to update the parameters.
3.4. I NFERENCE M ODEL
From the factorization form in Equation (12), we can see that there are two types of inference models: q ( u j ⃗x;⃗a;⃗r ) and q ( ⃗z j ⃗x;⃗a;⃗r ) .
Similar to the generative model in Section 3.2, we also parametrize both of them as Gaussian:
[Equation]
In fact, as shown in Equation (12), q ( ⃗z j ⃗x;⃗a;⃗r ) can be further factorized as the product of q ( z t j z t (cid:0) 1 ; ⃗x;⃗a;⃗r ) for t = 1 ; : : : ; T .
Taking a closer look at this term, based on the Markov property of our model, we have z t j = x 1 ; : : : ; x t (cid:0) 1 ; a 1 ; : : : ; a t (cid:0) 2 ; r 2 ; : : : ; r t j z t (cid:0) 1 , and then the term can be simpliﬁed as follows,
[Equation]
Equation (16) tells us that z t depends on z t (cid:0) 1 and all the current and future observed data ( ⃗x;⃗a;⃗r ) .
Meanwhile, the conditional independence above means that z t (cid:0) 1 contains all the historical data.
Therefore, it is natural to calculate z t based on the whole sequence of data, which is exactly what recurrent neural networks (RNNs) do.
Inspired by (Krishnan et al., 2015; 2017), we similarly choose a bi-directional LSTM (Zaremba & Sutskever, 2014) to parameterize f 11 and f 12 in Equation (15).
Considering Equation (14) has the same structure as Equation (15),
f 9 and f 10 are parameterized by a bi-directional LSTM as well.
More details about the architecture can be found in Appendix D.
Note that for the task of sample predictions, that is, at any time step t , given a new x t , we require to know a t and r t +1 before inferring the distribution over z t .
Hence, we need to introduce two auxiliary distributions, denoted by red and blue dashed lines in Figure 1(b), to help conduct counterfactual reasoning (i.e., sample prediction on unseen x t ).
To be more precise, we have
[Equation]
where f 13 , f 14 , f 15 , and f 16 are also parameterized by neural networks.
To estimate the parameters, we will add these two extra terms in the variational lower bound (Equation (13)):
[Equation] [Equation] 3. D ECONFOUNDING RL A LGORITHMS
Now we have all the building blocks for DRL algorithms.
Once our model is learned from the observational data, it can be directly used as a dynamic environment like those in OpenAI Gym (Brockman et al., 2016).
We can exploit the learned model to generate rollouts for policy optimization.
In practice, Equation (8) is approximated using the Monte Carlo method as follows:
[Equation]
where N is the number of samples from the prior p ( u ) .
In the presence of observational data, a better estimate could be given on the basis of the samples drawn from the approximate posterior q ( u j ⃗x;⃗a;⃗r ) which we compute through the inference network presented in Section 3.4.2.
On the basis of our deconfounding reward function, it is straightforward to extend traditional RL algorithms to their corresponding deconfounding version.
In this paper, we select one representative of them: the Actor-Critic method (Sutton et al., 1998), but it is straightforward to be applied to other algorithms.
Deconfounding Actor-Critic Methods
The actor-critic method is a policy-based method directly parameterizing the policy (cid:25) ( a j z ; (cid:18) ) , which aims to reduce the variance of the estimate of the policy gradient by subtracting a learned function of the state b ( z ) , known as a baseline, from the return.
The learned value function V ( z ; ϕ ) is commonly used as the baseline.
Taking into consideration that the return is the estimate of Q ( z; a ; ϕ Q ) and b ( z ) is the estimate of V ( z ; ϕ V ) , the gradient of the actor-critic loss function at step time t is given by
[Equation]
(21) where Q ( z t ; a t ; ϕ Q ) (cid:0) V ( z t ; ϕ V ) used to be seen as an estimate of the advantage of action a t in state z t .
In practice, Q ( z t ; a t ; ϕ Q ) is usually replaced with one-step return, that is, r t +1 + V
( z t +1 ; ϕ V ) .
Importantly, in deconfounding actor-critic methods, we use r t +1 (cid:24) p ( r t +1 j z t ; do ( a t )) (Equation (20)), as opposed to r t +1 (cid:24) p ( r t +1 j z t ; a t ) in vanilla actor-critic methods. The details about training can be found in Appendix E.
E XPERIMENTAL R ESULTS
It is widely acknowledged that evaluating approaches dealing with confounding is always challenging due to lack of groundtruth and benchmark datasets.
Besides, little work has been done before in DRL, which renders evaluating such algorithms in this respect much harder.
Therefore, we ﬁrst develop benchmark datasets for evaluation of our algorithms, primarily by revising the MNIST dataset (LeCun et al., 1998) and two environments in OpenAI Gym (Brockman et al., 2016): CartPole and Pendulum.
We then evaluate our model as well as compare two proposed deconfounding algorithms to their corresponding vanilla versions on the benchmark datasets we developed.
4. I MPLEMENTATION DETAILS
We used Tensorﬂow (Abadi et al., 2016) for the implementation of our model and DRL algorithms.
Optimization was done with Adam (Kingma & Ba, 2014).
Unless stated otherwise, the setting of all the hyperparameters and architectures of the neural networks we adopted in this paper can be found in Appendix J.
To verify how good the learned model is, we performed two types of tasks: reconstruction and counterfactual reasoning.
The reconstructions were performed by feeding the input sequence into the learned inference network, and then sampling from the resulting posterior distribution according to Equation (15), and ﬁnally feeding those samples into the generative network described in Equation (3).
The counterfactual reasoning, that is, predicting x t +1 given x t we have not seen in the training set and a model trained in the training set, were executed through four steps: 1) Given an x t unseen in the training set, we estimate a t and r t +1 based on Equation (17) and (18); 2) Once we have x t , a t , and r t +1 , it is easy to estimate z t from Equation (15); 3) Using the estimated z t and a t , we can directly compute z t +1 from Equation (6); (4) The ﬁnal step is to reconstruct x t +1 from z t [-+
-]+1 and u according to Equation (3).
Repeating the four steps, we can counterfactually reason out a sequence of data.
To evaluate the confounder u , there are also two scenarios.
The easy one is that, given a sequence of observational data ( ⃗x;⃗a;⃗r ) , it is obvious to estimate u from Equation (14).
The more challenging one is to calculate u given only x t at any time step.
Following the same steps used in the task of counterfactual reasoning, we ﬁrst compute a t and r t +1 based on Equation (17) and (18), and then estimate u through Equation (14).
4. C ONFOUNDING D ATASETS
Due to the space limit, we only introduce the Confounding Pendulum dataset in this section. Confounding CartPole and Confounding MNIST are devised in the same way and can be found in Appendix H.1 and Appendix H.2, respectively. Confounding Pendulum Motivated by (Krishnan et al., 2015) where the authors synthesized a dataset mimicking the healthcare data under harsh conditions (e.g., noisy laboratory measurements, surgeries and drugs affected by patient age and gender, etc.), as well as considering our focus on RL tasks, we revised the original Pendulum in OpenAI Gym
(Brockman et al., 2016) to develop a confounding Pendulum dataset in which a binary confounder is introduced.
More speciﬁcally, we select 100 different screen images of Pendulum to create a synthetic dataset where actions are joint effort with the value of between (cid:0) 2 and 2 4 .
First, 20% bit-ﬂip noise are added to each image , and then, based on a random policy but confounded by a binary factor (will be described soon), actions are performed on each noisy image for ﬁve time steps, which produces a large number of 5-step sequences of noisy images.
To each generated sequence, exactly one sequence of three consecutive squares ( 2 (cid:2) 2 in pixel) is superimposed with the top-left corner of the images in a random starting location. The squares within the sequences are intended to be analogous to seasonal ﬂu or other ailments that a patient could exhibit that are independent of the actions and which last several time steps (Krishnan et al., 2015).
We aim to show that our model can learn long-range patterns, which plays an important role in medical applications.
We treat such generated sequences of images as the observations ⃗x .
The training/validation/test set respectively comprises 140000 = 28000 =sequences of length ﬁve.
Now we are arriving at the key stage: how to deﬁne a reasonable relationship between the confounder, action, and reward.
For simplicity of notation, we denote the confounder by u , action by a , and reward by r .
u is a binary variable mimicking the socio-economic status (i.e., the rich and the poor) . The range of actions a 2 [ (cid:0) 2 ; 2] is grouped into two categories T 1 (i.e., 1 (cid:20) j T 1 j (cid:20) 2 ) and T 2 (i.e., 0 (cid:20) j T 2 j (cid:20) 1 ) representing different treatments (i.e., T 1 is more valid and T 2 ).
How to choose the treatment depends on u .
The reward is deﬁned as follows, [Equation] where r o is the original reward in Pendulum, which is a function of a 5 , and r c is the extra reward caused by both u and a , where r c comes from a two Gaussian mixture
[Equation]
with (cid:27) and (cid:22) r c ﬁxed and (cid:25) r c determined by both u and a (see more in Appendix H.3). Obviously, in the deﬁnition above, r is a function of a and u . It is worth noting that u has an inﬂuence on x t through a t (cid:0) 1 and z t , meaning that x t contains some piece of information about u and therefore can be viewed as its proxy variable.
In our case, for the sake of generality and practice, we assume the inﬂuence between the confounder, action and reward is stochastic. Take the case of kidney stones for example, even though treatment a is more valid than treatment b , there are still some patients choosing treatment b in each category, but with different probabilities.
All the details can be found in Appendix H.3, where a straightforward analogy is provided as well.
4. P ERFORMANCE ANALYSIS OF THE DECONFOUNDING MODEL
In this section, to demonstrate the validity of our deconfounding model, denoted by M decon , we compare with the original model (i.e., the model similar to that shown in Figure 1(b) but without the confounder u ), denoted by M orin .
We train M decon by optimizing Equation (19) but train M orin by a little different loss function excluding the confounder u whose full derivation can be found in Appendix C. Both models are separately trained in a batch manner on the training set (i.e., 140 K sequences of length ﬁve of images) of the confounding dataset. Afterwards, following the steps depicted in Section 4.1, we use each trained model to perform the reconstruction task on the training set, and both reconstruction and counterfactual reasoning tasks on the testing set (i.e., 28 K sequences of length ﬁve of images). Figure 2 presents a comparison of M decon and M orin , in terms of reconstruction and counterfactual reasoning on the confounding Pendulum dataset.
The second row is based on M decon (Figure 1(b)), whilst the top row comes from M orin .
It is evident that the results generated by the deconfounding model is superior to those produced by the model not taking into account the confounder.
To be more speciﬁc, as shown in the zoom of samples on the bottom row, M orin generates more blurry images than M decon , because, without modelling the confounder u , M orin is forced to average over its multiple latent states resulting in more blurry samples. Likewise, we can attain the same conclusion from the samples respectively produced by M orin and M decon on the confounding MNIST dataset, as shown in Figure 7 of Appendix I.
In addition, looking closely at the squares on the generated digit samples (colored in yellow box), it is obvious to observe that M orin will generate non-consecutive squares in the task of counterfactual reasoning, which does not really make sense because only consecutive patterns appear in the training set.
In contrast, this case does not take place on the samples from M decon , showing that our deconfounding model is able to cope with long-range patterns.
It is worth noting that, as shown in Figure 8 of Appendix I, by visualizing the 2 -dimensional confounder u , we can discern that although the prior distribution of the confounder is assumed to be a unit Gaussian distribution, the model still can learn two obvious clusters from the data because it is originally a binary variable.
It demonstrates that our model has an advantage in learning confounders even if the assumed prior over them were not that accurate.
4. C OMPARISON OF RL ALGORITHMS
In this section, we will evaluate the proposed deconfounding actor-critic (AC) method by comparing with its vanilla version on the confounding Pendulum dataset. In the vanilla AC method, given a learned M orin , we optimize the policy by calculating the gradient presented in Equation (21) on the basis of the trajectories/rollouts generated through M orin . Equation (21) involves two functions:
V ( z t ; ϕ V ) and (cid:25) ( a t j z t ; (cid:18) ) , whose parameters can be found in Appendix J. It is worth noting that, in this vanilla case, each reward r t +1 is produced from the conditional distribution p ( r t +1 j z t ; a t ) .
In contrast, the proposed deconfounding AC method is built on M decon . Although the same gradient method (Equation (21)) is utilized to optimize the policy, we base the deconfounding AC approach on the different trajectories/rollouts generated by M decon in which each reward r t +1 relies on the interventional distribution p ( r t +1 j z t ; do ( a t )) computed using Equation (20). In the training phase, for both vanilla AC and deconfounding AC, we run a respective experiment over 1500 episodes with 200 time steps each. In order to reduce non-stationarity and to decorrelate updates, the generated data is stored in an experience replay memory and then randomly sampled in a batch manner (Mnih et al., 2013; Riedmiller, 2005; Schulman et al., 2015; Van Hasselt et al., 2016).
In each episode, we summarize all the rewards and further average the sums over a window of 100 episodes to obtain a more smooth curve.
As shown in Figure 3(a), obviously our deconfounding AC algorithm performs signiﬁcantly better than the vanilla AC algorithm in the confounded environment. In the testing phase, we ﬁrst randomly select 100 samples from the testing set, each starting a new episode, and then use the learned policies to perform reasoning over 200 time steps as we did during the training time. From the resulting 100 episodes, we plot the total reward for each, shown in Figure 3(b), and compute the percentage of the optimal action T 1 in each episode, presented in Figure 3(c). It is worth noting that Figure 3(c) tells us that in each episode our deconfounding AC almost chooses the optimal action at each time step, whilst the vanilla AC makes a wrong decision for more than half time.
R ELATED W ORK
Krishnan et al.
(2015; 2017) used deep neural networks to model nonlinear state space models and leveraged a structured variational approximation parameterized by recurrent neural networks to mimic the posterior distribution.
Levine (2018) reformulated RL and control problems to probabilistic inference, which allows us to bear a large pool of approximate inference methods, and ﬂexibly extend the model.
Raghu et al.
(2017a;b) exploited continuous state-space models and deep RL to deduce treatment policies for septic patients from observational data.
Gottesman et al.
(2018) discussed some issues of evaluating RL algorithms in observational health setting.
However, all the work mentioned above did not take into account confounders in their models.
Louizos et al.
(2017) attempted to learn individual-level causal effects from observational data using variational auto-encoder to estimate the unknown confounder given a causal graph in then nontemporal setting.
Paxton et al.
(2013) developed predictive models based on electronic medical records without using causal inference.
Saria et al.
(2010) proposed a nonparametric Bayesian method to analyze clinical temporal data.
Soleimani et al.
(2017) represented the treatment response curves using linear time-invariant dynamical systems which provides a ﬂexible approach to modeling response over time.
Although the latter two work modeled the sequential data, they both do not exploit RL or causal inference.
Bareinboim et al.
(2015) considered the problem of bandits with unobserved confounders, which is one quite simple RL setting without state transitions.
Sen et al.
(2016) and Ramoly et al.
(2017) further studied contextual bandits with latent confounders.
Forney et al.
(2017) circumvented some problems caused by unobserved confounders in Multi-Armed Bandit by counterfactual-based decision-making.
Zhang & Bareinboim (2017) leveraged causal inference to tackle the problem of transferring knowledge across bandit agents.
However, all these methods are based on the bandit problem, a simpliﬁed version of RL, instead of the full RL problem.
In fact, as far as we are concerned, this is the ﬁrst attempt to build a bridge between confounding and the full RL problem, and this is also one of few research papers aiming at understanding the connections between causal inference and full RL.
C ONCLUSION AND F UTURE W ORK
To address the confounding issue in RL, we introduced a general formulation, namely deconfounding reinforcement learning.
On the basis of the proposed formulation, we presented deconfounding variants of actor-critic methods and showed their superior performance on confounding datasets that we created by revising OpenAI Gym and MNIST.
In the future, we will collaborate with hospitals and apply our approach to real-world medical datasets.
We also hope that our work will stimulate further investigation of connections between causal inference and RL. Appendices A I DENTIFICATION OF C AUSAL E FFECT
As pointed out in (Louizos et al., 2017; Edwards et al., 2015; Kuroki & Pearl, 2014; Miao et al., 2018; Pearl, 2012; Wooldridge, 2009), it is impossible to recover the joint distribution from the observational data if the hidden confounder has no connection to the observed variables. Fortunately, there still exist a large number of possible circumstances, and we refer readers to (Louizos et al., 2017; Kuroki & Pearl, 2014; Miao et al., 2018; Pearl, 2012; Allman et al., 2009) for more details. Here, we only focus on one possible case presented in Figure 1 of (Louizos et al., 2017)
(see Appendix F.1), because the result can be directly used to identify the causal effect in our model. More precisely, in our model, at each time step, the 4-tuple ( z t ; x t ; a t ; r t +1 ) is exactly the same as that case in Figure 1 of (Louizos et al., 2017), and therefore p ( z t ; x t ; a t ; r t +1 ) can be approximately recovered only from the observations ( x t ; a t ; r t +1 ) . Likewise, the joint distribution over the other 4-tuple ( u; x t ; a t ; r t +1 ) can be recovered from ( x t ; a t ; r t +1 ) in the same manner (see Appendix G). Applying this rule repeatedly to the sequential data, we are ﬁnally able to approximately recover p ( u;⃗z; ⃗x;⃗a;⃗r ) solely from the observations ( ⃗x;⃗a;⃗r ) .
We recover the joint distribution using Variational Auto-Encoders (VAEs) (Kingma & Welling, 2013), which is a powerful tool to recover a very large class of latent-variable models as a minimizer of an optimization problem (Tran et al., 2015; Louizos et al., 2017). Generally speaking, however, VAEs currently have little theory available to guarantee that the true model can be identiﬁed through learning (Louizos et al., 2017). The main reason behind is that, in such latent variable models, confounders and causal parameters, both of which are unknown and might inﬂuence each other, will inevitably result in the case that different causal parameters could generate the same observable data (D’Amour, June 9, 2018).
Fortunately, identiﬁcation is possible when the latent confounder and their relationship to the outcome are themselves identiﬁed. For instance, in the presence of proxy variables of the latent confounder, as described in the previous paragraph, both our model and the one in (Louizos et al., 2017) are identiﬁed under some conditions outlined in (Louizos et al., 2017; Kuroki & Pearl, 2014; Miao et al., 2018; Pearl, 2012; Allman et al., 2009). In addition, even if only considering our model, because of the existence of many proxy variables of the confounders (e.g., each of z t and u has at least T proxy variables f x t g t =1 ;:::;T , where z t can be viewed as hidden states in HMMs (Allman et al., 2009)), identiﬁcation can be also justiﬁed if the latent confounders z t and u are categorical (Allman et al., 2009).
In practice, however, we are not able to know the exact nature of the confounders, e.g., what kinds of distributions they follow, if categorical how many categories they have, etc. Hence, we take advantage of VAEs, which allows us to make substantially weaker assumptions about the data generating process and the structure of the hidden confounders (Louizos et al., 2017). Nevertheless, we empirically show that our approach is more beneﬁcial to learning a better policy in the existence of confounders. Note that, in our model we have to differentiate the two types of confounders: the time-independent confounder u and time-dependent confounders f
z t g t =1 ;:::;T , each playing a respective role in the model. The former, as a global confounder, will affect the whole course of treatment, and therefore should be adjusted for. In the example of kidney stones, the existence of the confounder (i.e., the size of stones) will lead to a wrong treatment if not adjusting for it. In contrast, the time-varying confounders f
z t g act as states in RL, which, in principle, should not be adjusted for, because the goal in RL is to learn a good policy in which any action is indeed supposed to be conditional on a speciﬁc state.
On the other hand, in terms of rewards, what an agent expects at each time step is exactly the immediate reward when taking a speciﬁc action at a speciﬁc state, without the need of adjusting for states.
It is worth noting that the case with time-varying confounders f z t g can be thought of to meet a pseudo or weak causal sufﬁciency assumption under which the causal effects of actions on rewards will not be inﬂuenced by states at each time step (Zhang et al., 2017; 2015). This key difference motivates us to only adjust for the time-independent confounder u in this paper. In addition, as shown in Figure 1(b), in our case where a policy depending only on the confounder is applied to generating the data (Section 4.2), the arrow from z t to a t is not necessary so that z t can be viewed as not a confounder of a t and r t +1 . This also provides another reason why we do not need to adjust for z t .
B V ARIATIONAL L OWER B OUND FOR M DECON
[Equation]
( u j ⃗x
[Equation]
⃗a;⃗r ) q ( z 1 j ⃗x
[Equation]
⃗a;⃗r ) (cid:1) (cid:1) (cid:1) q
( z T j z T (cid:0) 1 ; ⃗x;⃗a;⃗r ) d ⃗z
[Equation]
u =
T ∑ t = ∫ u ∫ z(cid:1) (cid:1) (cid:1) ∫ z T q ( u j ⃗x
[Equation]
⃗a;⃗r ) q ( z 1 j ⃗x
[Equation]
⃗a;⃗r ) (cid:1) (cid:1) (cid:1) q ( z
[Equation]
j z T (cid:0) 1 ; ⃗x;⃗a;⃗r ) log (
[Equation] [Equation]
j z t [Equation] u ) p ( a
[Equation]
j z t
[Equation] u ) p ( r [Equation]
1 j z
[Equation]
a t
[Equation]
u ))
[Equation]
u + ∫ u ∫ z(cid:1) (cid:1) (cid:1) ∫ z T q ( u j ⃗x
[Equation]
⃗a
[Equation]
⃗r )
q ( z 1 j ⃗x;⃗a;⃗r ) (cid:1) (cid:1) (cid:1) q
( z T j z T (cid:0) 1 ; ⃗x;⃗a;⃗r ) log p ( u ) q ( u j ⃗x;⃗a;⃗r )
d ⃗z d u + ∫ u ∫ z(cid:1) (cid:1) (cid:1) ∫ z T q ( u j ⃗x
[Equation]
⃗a;⃗r )
[Equation]
( z 1 j ⃗x;⃗a;⃗r ) (cid:1) (cid:1) (cid:1) q
( z T j z T (cid:0) 1 ; ⃗x;⃗a;⃗r ) log p ( z 1 )
[Equation]
( z 1 j ⃗x
[Equation]
⃗a
[Equation]
⃗r )
d ⃗z d u +
T ∑ t = ∫ u ∫ z(cid:1) (cid:1) (cid:1) ∫ z T q ( u j ⃗x;⃗a;⃗r ) q
( z 1 j ⃗x;⃗a;⃗r ) (cid:1) (cid:1) (cid:1) q
( z T j z T (cid:0)
[Equation]
C V ARIATIONAL L OWER B OUND FOR M ORIN
[Equation]
a
[Equation]
[Equation]
D B I - DIRECTIONAL LSTM
In our inference model, we use a similar architecture of bi-directional LSTM to that in (Krishnan et al., 2017).
Apart from different inputs, the main difference is to introduce a t (cid:0) 1 to computing the combined hidden feature as shown in the following formula.
[Equation]
LSTM h righth righth right
[Equation]
LSTM
[Equation]
left
[Table] [Equation]
left
[Table] [Equation]
As mentioned in the main body, DRL consists of two steps: learning a deconfounding model and optimizing a policy based on the learned deconfounding model. To be more speciﬁc, in step 1, given the observational data ( ⃗x;⃗a;⃗r ) , we learn the deconfounding model by optimizing the variational lower bound as presented in Equation (19).
Once the deconfounding model is learned, we know the state transition function p ( z t j z t (cid:0) 1 ; a t (cid:0) 1 ) and can also calculate the deconfounding reward function p ( r t j z t ; do ( a t ))
according to Equation (20). In step 2, we treat the learned deconfounding model as a RL environment like CartPole in OpenAI Gym, and directly exploit it to generate trajectories/rollouts through the state transition function and the deconfounding reward function. On the basis of the generated trajectories/rollouts, we can train the policy network using Equation (21). F E XAMPLE OF K IDNEY S TONES F. E XAMPLE OF A PROXY VARIABLE
For convenience, we directly move Figure 1 of (Louizos et al., 2017) here, as shown in Figure 4.
F. I NTUITION
Take for example the case of kidney stones, in the existence of the confounder (i.e., the size of stones), one prefers treatment b when considering the overall probability of recovery, whilst one chooses with more probabilities treatment a when investigating the recovery rate in each size.
The correct answer is treatment a in this example.
The do -operator in Equation (2) of Section 2.3 provides a formal approach to seeking the correct solution to such confounded problems, which can not be addressed within the traditional RL framework, because the traditional framework only involves the conditional probability (e.g., the overall probability of recovery) instead of the interventional probability as do -operator does. [Figure] G A NALYSIS OF THE D ECONFOUNDING M ODEL At each time step t , from the deconfounding model presented in Figure 1(b), we can extract two 4tuple components, centering at z t (Figure 5) and u (Figure 6), respectively.
It is apparent to observe that both Figure 5 and Figure 6 share the exact same structure with Figure 4.
[Figure] [Equation] [Equation] U u
H C ONFOUNDING D ATASETS H. C ONFOUNDING C ART P OLE
The confounding CartPole dataset can be implemented in the exact same manner as Confounding
Pendulum, except for the action which is originally binary and can be naturally divided into two categories.
H. C ONFOUNDING MNIST
We follow the same setting to develop a confounding MNIST dataset, but the deﬁnitions of action and the original reward term are different.
Similar to the Healing MNIST dataset (Krishnan et al., 2015), rotations of digit images are encoded as the actions ⃗a ( (cid:0) 45 (cid:20) a (cid:20) 45) which, according to the binary confounder u , is divided into two categories (22 : 5 (cid:20) j a j (cid:20) 45) and (0 (cid:20) j a j < 22 : 5) .
The original reward term r o is deﬁned as the minus degree between the upright position and the position the digit rotates to.
For example, if the digit rotates to the position of 3 o’clock or 9 o’clock, then both rewards are (cid:0) 90 .
H. D ETAILS ABOUT C ONFOUNDING S IMULATIONS
The probabilities between the confounder, action and reward are presented as follows, p ( u =
[Equation]
[Equation]
[Equation]
8 ; p (
[Equation] [Equation] [Equation] [Equation]
(24) p ( a = T
[Equation]
j u = 0)
= 0 : 24 ; p ( a
[Equation]
j u
[Equation] = 0 : 76; (25) p ( a = T [Equation]
j u = 1) = 0 : 77 ; p (
[Equation]
u
[Equation]
1) = 0 : 23; (26) p ( r c = R
[Equation]
j a = T 1 ; u = 0)
= 0 : 93 ; p ( r c = R 2 j a = T 1 ; u = 0)
= 0 : 07; (27) p ( r c = R 1 j a = T 2 ; u = 0)
= 0 : 87 ; p (
[Equation]
Note that, in our model r c is a function of u , a t , and z t .
We omit the notation of the state in the conditional probability of r c because z t is ﬁxed at each time step, which does not affect the probability.
To help readers understand this setting, we can make a straightforward analogy.
More speciﬁcally, the confounder u stands for the socio-economic status where u = 0 represents the poor and u = 1 the rich.
Treatment T 1 is more expensive but more valid than Treatment T 2 .
R 1 means a healthier feedback/recovery than R 2 .
Equation (24) says that the poor are much more than the rich (it is indeed in reality).
Equation (25) and Equation (26) tell us the fact that the poor tend to choose the cheaper but less valid treatment T 2 whilst the rich prefer the more expensive but more valid treatment T 1 , which also makes sense in the real world.
Finally, Equations (27)-(30) reveal that T 1 is a better treatment than T 2 within both poor and rich population.
I A DDITIONAL E XPERIMENTAL R ESULTS
As shown in Figure 7 and Figure 8.
J E XPERIMENTAL S ETTINGS
As shown in Table 1 and Table 2.
[Figure]
[Figure]
F un c t i on
A r c h it ec t u r e
D ec o n f o und i n g M o d e l f 1 ; f
F C !
C onv 7 ; !
C onv 14 ;!
C onv 28 ;!
F C !
f s i
g m o i d ; s o f t p l u s g f 3 ; f f F C 100 ; F
Cg !
F C !
F C !
F C!
f t a nh ; s o f t
p l
u s g f 5 ; f f F C 100 ; F
C; F C 100 g !
F C ! F C ! F C! f s i
g m o i d ; s o f t p l u s g f 7 ; f f F C 100 ; F
Cg !
F C !
F C !
F C !
f N on e ; s o f t p l u s g f 9 ; f f [ C onv 5 ; !
C onv 5 ; !
C onv 5 ; !
F C 100 ] ;
F C 100 ; F
Cg !
F C !
F C !
L S T
M 100 ;!
F C!
f N on e ; s o f t p l u s g f 11 ; ff [ C onv 5 ; !
C onv 5 ; !
C onv 5 ; !
F C 100 ] ;
F C 100 ; F
Cg !
F C !
F C !
L S T
M 100 ;!
F C !
f N on e ; s o f t p l u s g f 13 ; f
C onv 5 ; !
C onv 5 ; !
C onv 5 ; !
F C!
f t a nh ; s o f t
p
l
u s g f 15 ; ff [ C onv 5 ; !
C onv 5 ; !
C onv 5 ; !
F C 100 ] ;
F C 100 g !
F C !
F C!
f s i
g m o i d ; s o f t p l u s g
D ec o n f o und i n g
AC
V ( z t ; ϕ V )
F C !
F C !
f N on e ; s o f t p l u s g (cid:25) ( a t j z t ; (cid:18) )
F C !
F C !
f t a nh ; s o f t p l u s g
T a b l e 2 :
A r c h it ec t u r e s f o r d ec on f ound
i ng r e i n f o r ce m e n t l ea r n i ng . H e r e F C k s t a nd s f o r a f u ll y - c
onn ec t e d l a y e
r w it h k un it s , C onv k ; n f o r a c onvo l u ti on l a y e r w it h n
ﬁ lt e r s
o f s i ze k (cid:2) k , L S T
M n ; t f o r a L S T M l a y e r r o lli ng ou t f o r t s t e p s w it
h l a t e n t s i
ze o f n , f (cid:1) g f o r t h e p a r a ll e l op e r a t o r s , a nd [ (cid:1) ]
f o r t h e s e qu e n ti a l op e r a t o r s .
I n d e f a u lt , F C k a n d C onv k ; n a r e f o ll o w e d by a s o f t p l u s ac ti v a ti on l a y e r
a nd a b a t c hno r m l a y e r , w h
i c h a r e o m itt e d h e r e f o r s i m p li c it y .
N o t e t h a t , i n ou r s e tti ng , t h e t w o f un c ti on s i n eac h p
{+a i r f f 
 i (cid:0) 1 ; f 2 i g i = 1 ;:::; 8 s h a r e t h e s a m+}
e p a r a m e t e r s e x ce p t
f
o r t h e l a s t l a y e r .