Paper:
Confounding-Robust Policy Improvement
We study the problem of learning personalized decision policies from observational data while accounting for possible unobserved confounding in the data-generating process. Unlike previous approaches that assume unconfoundedness, i.e., no unobserved confounders affected both treatment assignment and outcomes, we calibrate policy learning for realistic violations of this unverifiable assumption with uncertainty sets motivated by sensitivity analysis in causal inference. Our framework for confounding-robust policy improvement optimizes the minimax regret of a candidate policy against a baseline or reference "status quo" policy, over an uncertainty set around nominal propensity weights. We prove that if the uncertainty set is well-specified, robust policy learning can do no worse than the baseline, and only improve if the data supports it. We characterize the adversarial subproblem and use efficient algorithmic solutions to optimize over parametrized spaces of decision policies such as logistic treatment assignment. We assess our methods on synthetic data and a large clinical trial, demonstrating that confounded selection can hinder policy learning and lead to unwarranted harm, while our robust approach guarantees safety and focuses on well-evidenced improvement.

Review:
This paper studies the conditions under which IPW-based policy may fail due to the existence of latent confounders, and then provides methods for obtaining a robust policy through optimization of the worst-case regret (which is motivated by the sensitivity analysis literature). Given a user-set \Gamma, the bound of the odds ratio (as in the sensitivity-analysis), and the assumption of e(x,u) (propensity) as a logistic model, the authors derive bounds over the true inverse propensity. This leads to the bound on an empirical regret.   Overall, I enjoyed the paper; the writing is clear and well organized. It successfully conveys the idea of the authors. However, as the paper currently stands, there are still important issues that are somewhat unclear and/or not present, which I list below.   Q1. Setting parameters  Given `properly specified` bounds — \Gamma and \Lambda, one can see improvements on the policy. Then, it is natural to ask how the user should think about and decide the values of \Gamma and-or \Lambda. In sensitivity analysis, researchers can try different values of \Gamma to ‘investigate’ how the results of a study would vary. However, the purpose of this paper is to ‘replace’ the given policy w/ the resulting robust-policy w.r.t. \Gamma, so it's not entirely clear to me how to perform such a thought experiment. Please, elaborate on this issue.   Q2. Data-driven Confounding Robust Policy Improvement  The sensitivity parameter is set by the user, leading to a bound on the odds ratio, which eventually will constrain the policy value, or the causal effect of the treatment. There is, however, an entirely data-driven approach for constraining the experimental distribution, which is the final target of the analysis.  As I understand the paper, it tries to bound the PS since the same is not identifiable when unobserved confounding is present. Still, the real goal of learning the PS, in the first place, is to estimate the Y_T distribution.    If one can bound Y_T directly from P(X, T, Y), it seems that asking for the user input may be unnecessary (since one could immediately get a "data-driven" gamma). The idea of bounding E[Y_T] directly from data (under confounding) seems to be the route taken by (ZB, 2017) (see also Ch. 8, Causality, 2000). Given that both approaches appear concerned with optimizing online learning, how are they related?    We further note that it's possible that the user parameters may lead to bounds that are less informative (larger) than the ones coming directly from the observational (logged) data. Please, can you elaborate on this point?   Q3. Empirical Evaluation  a. If the data is obtained without confounding, what would be the performance of the robust policy returned by the algorithm? Even though the paper demonstrated how robust policy could be worsened with unnecessarily large \Gamma, still there is no natural way, as far as I can see, to tell what \Gamma should be.  b. How about the robustness against the violation of the logistic model assumption of e(x,u)? In such a case, is there any natural interpretation of \Gamma?  c. Why is the IPW worse than RIC family when \Gamma=1? (Figure 2 — referenced as Fig 1a) Is it because the used IPW is a vanilla implementation?  It would be crucial to see results of weight-clipped IPW or weight-normalized IPW.  d. If the data is generated from a different distribution, is it possible for IPW to perform better than RIC* w/ \Gamma=1?  e. Similarly, the performance gain due to the use of \Gamma should be compared to \Gamma=1, not to a vanilla IPW. Please, justify and clarify the meaning of RIC performance at \Gamma=1. Isn’t it assuming that there is no confounder?  The above questions are similarly applied to the experiments with IST data. RIC*s appear to be improved as \Gamma increases (Fig 2a, 2b). However, considering \Gamma=1, we only see a degradation of the performance with smaller \Gammas. It is somewhat hard to assess where the performance gain is coming from.  Summary:  - Clarity: The paper is overall well-written with a right balance between text and math. - Originality: It is original in that it provides a new efficient method to yield a robust policy by optimizing worst-case regret, under the assumptions that the user can provide bounds. This judgment is contingent on how this approach relates (generalizes? subsumes?) the data-driven approach discussed above.  - Significance: Without knowing how to set \Gamma, the applicability of the method may be limited, and the improvement guarantee may be somewhat illusive. The baseline should be replaced to accurately assess how the proposed methods work w/ different \Gammas.  - Quality: Overall okay, with good mathematical details but without strong empirical evidence and not very accurate literature review and comparison.  Minor comments:  for for (Line 173)  an uncertainty (Line 9)  There are two figure 2s without figure 1.  Figure 2 (a), (b), (c) not marked.  Reference:  Zhang, J. and Bareinboim, E. Transfer Learning in Multi-Armed Bandits: A Causal Approach. In Proceedings of the 26th International Joint Conference on Artificial Intelligence (IJCAI), 1340-1346, 2017.    Pearl, J. Causality: Models, Reasoning and Inference. Cambridge University Press, 2000.  --- Post-rebuttal   We believe the authors addressed most of the issues and updated my grades accordingly. We would like to highlight that we expect the authors doing a better job in the final version of the paper regarding the two central points:   1. Gamma is the critical parameter that the whole analysis hinges on, so it should have more emphasis an explanation on how, in any cognitively meaningful way, users can conceive setting these parameters to some value.   2. I don’t think the answer regarding Q2 is accurate. The paper should acknowledge that there’s a tradeoff between knowledge an inferential power, and they are not proposing anything magical here. If the user wants to be entirely data-driven, they should go with the work of ZB'17 (or some obvious/natural generalization), while if they do have the substantive knowledge needed to set \Gamma (whatever this could be), they can consider the approach proposed in the paper.   These points should be front and center in the paper since it currently conveys the image that nothing was done before and that \Gamma is a somewhat "magical" parameter. Further, it seems the results are obtained without any effort and there isn't a more data-driven alternatively to their method.  Furthermore, the work on Manski and other folks are great for bounding in the offline setting, but the desirable property about the setting discussed is coupling the knowledge from one policy on another and proving that it can indeed help in practice.   Still, I think the work proposed here is nice and should, therefore, be accepted.