Paper:
In this paper, we propose a general formulation to cope with a family of reinforcement learning tasks in observational settings, that is, learning good policies solely from the historical data produced by real environments with confounders (i.e., the factors affecting both actions and rewards). Based on the proposed approach, we extend one representative of reinforcement learning algorithms: the ActorCritic method, to its deconfounding variant, which is also straightforward to be applied to other algorithms. In addition, due to lack of datasets in this direction, a benchmark is developed for deconfounding reinforcement learning algorithms by revising OpenAI Gym and MNIST. We demonstrate that the proposed algorithms are superior to traditional reinforcement learning algorithms in confounded environments. To the best of our knowledge, this is the first time that confounders are taken into consideration for addressing full reinforcement learning problems.

Review:
The paper addresses an important and often overlooked issue in off-policy reinforcement learning - the possibility of confounding between the agent's actions and the rewards. This is a subject which has been exhaustively explored in the causal inference literature, and the authors are very correct in suggesting that it should be incorporated into the world of reinforcement learning.  Specifically they propose a generative model with a global latent confounder that is inferred using a variational autoencoder architecture.  
The paper is generally well-written, though some points could be made clearer in my opinion, as detailed below. The experiments are constructed by introducing confounding into existing datasets; performance seems to be good, but I am not entirely sure whether the given architecture is necessary, see comments below. 
High-level comments:
(1) Classic RL deals with confounders all the time. The state is a confounder between the action and the reward. The issue of confounding becomes less trivial when one is performing off-policy RL when the original policy is *unknown*. This is exactly the case that the authors mention when they cite the recent work by Gottesman et al. (2018) who deal with using RL to learn from the actions of physicians in a hospital.  While I am sure the authors are aware of these distinctions, I think the paper would be better if this is spelled out very explicitly. This includes explaining why this issue doesn't come up in classic RL.
(2) Assuming the case above - off-policy RL with unknown confounders - one would usually assume "no unmeasured confounding", i.e. that the observed actions are an unknown but learnable function of the observed states. That is basically the scenario of most off-policy RL.
(3) However, the authors strive to go one step beyond the case (2), to a situation where there is an *unmeasured* confounder affecting both observed actions and rewards. If nothing is known about this unmeasured confounder, then it is generally impossible to learn effective policies, as the causal effects of actions are not identifiable from the observed data. In this paper, the authors make an implicit assumption that while the confounder is unmeasured, it can still be inferred from the data. This is an intermediate step between "no unmeasured confounding" and "complete unmeasured confounding". This is related to work on using proxy variables e.g. Kuroki & Pearl (2014) and even more closely related to the work cited by Louizos et al. (2017).
Again, I think the paper would be much improved if all this is addressed explicitly. 
(4) An important consequence of point (3) above is that in fact adding the single global latent-confounder U is not, in itself, very important from a causal perspective. The sequence of variables Z_1... Z_T are already latent confounders that are assumed to be inferrable from data. It is true that the addition of the global U might change the statistical and optimization properties of the model. This leads to a very important conclusion: the authors should test their model with and without U. I think this specific ablation experiment is crucial. In many cases I am sure that the assumption of a global latent confounder is a good one and is especially useful in the VAE case where it will make optimization more stable. However, in principle, all of U's roles could be taken within the sequence of Z's, and I am curious to see in practice how big of an effect it has.
(5) I wish to add that even if the U variable turns out to not add much empirically, this work is still valid since the sequence of Z's can themselves be considered inferred latent confounders.
Specific comments:
(1) 2.3: there are more than 2 ways of computing the do-operator. RCTs and backdoor are the best known approaches, but not the only ones, e.g. there is frontdoor adjustment. 
(2) I think the paper would be easier to follow if there was one concrete example used throughout. This will make it easier to understand and possibly verify/criticize the assumptions of the generative model.
(3) Related to "higher-level point (4)" above, in eqs. 17 & 18 note that Z_t is unknown, same as U. Both are inferred. This also leads to the question which Z_t is actually used in practice? Is it the mean, or is it also sampled from the approximate posterior q?
(4) Below eq. 19, it would be very useful for the readers if you could explain exactly when would there be a difference between the two versions p(r_{t+1}|z_t,a_t) and p(r_{t+1}|z_t, do(a_t=a))
(5) In the description of all the experiments I was missing a crucial point: how does the introduced confounder affect the reward? Is it only through the different actions? The way it is currently explained, it seems like the added variable introduces lack of *overlap*, but not strictly confounding.
(6) The description of the experiment in 4.3 could be more detailed. What exactly was the training and test? What RL method was used? What did the baseline optimize for? I would like to see an ablation experiment where U is not included in the model. 
(7) In 4.5, what is the "vanilla" method? And as mentioned above, I would like to see an ablation experiment where U is not included in the model.