From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Accept: poster
metareview:_summary,_strengths_and_weaknesses: The paper provides a relevant approach for estimating causal effects under particular covariates (as in the individual treatment effect estimand) by using instrumental variables - binary variables in particular.

Strengths: clearly written, solves a problem of relevance.

Weaknesses: the solution itself is relatively derivative, playing safe on how much it differs from the methods which provides a basis to this problem solution.
justification_for_why_not_higher_score: The solution itself is relatively derivative, playing safe on how much it differs from the (nontrivial) methods which provide a basis to this problem solution, so in some sense being a bit predictable. From the point of view of ICLR, it feels somewhat specialised (representation learning plays a very modest and modular role) and perhaps better suited to a more targeted venue.
justification_for_why_not_lower_score: It is well thought through and presented, and to the best of my knowledge I'm not aware of prior work that frames the problem in the way that is framed here.
note_from_PC: if the above contains the word "oral" or "spotlight" please see: "oral" presentation means -> notable-top-5% and "spotlight" means -> notable-top-25%. As stated in our emails, we are disassociating presentation type from AC recommendations

From: ICLR.cc/2023/Conference/Paper904/Authors
title: Response to Reviewer BewB
comment: Thank you for your positive review and your helpful comments. We improved our paper in the following ways:

## Responses to “Abstract”

1. We corrected our abstract, changing the IV terminology from “compliance” to “treatment assignment”.
2. We elaborated on the wording “asymptotically outperforms”.

## Responses to “Section 1”

1. We followed your suggestion and changed the naming from “individual treatment effect” (ITE) to “conditional average treatment effect” (CATE).
2. We fixed the typos you pointed out.
3. See 2.
4. We removed race and gender as examples of information not stored in medical health records.
5. We extended the paragraph where we introduce our framework, providing explanations rather than just stating the properties.
6. In the context of accepting Medicaid, we refrained from using the wording “compliance”.
7. We extended the discussion of Assumption 2 and included your remarks regarding the moderation of treatment effects by unobserved confounders. Of note, all our theoretical results (Theorem 1-3) do still hold for the quantity on the right-hand side of Eq. (2), _even if Assumption 2 does not hold_. In some cases, the estimand can then still be interpreted as the conditional ETT or LATE (see our discussion at the end of Section 2).

## Responses to “Section 2”

1. The conditions in Assumption 4 need to hold for all $x \in \mathcal{X}$ as stated in the paper. They could be relaxed to hold almost everywhere, but then our bound in Theorem 2 would “only” hold almost everywhere.
2.  $r_\tau(x)$ denotes the rate of convergence fo the initial estimator $\hat{\tau}_{\mathrm{init}}$. We added a rigorous definition to the Appendix of our revised paper (new Appendix D).

## Responses to “Section 5” 

1. We removed this sentence from our revised paper.

From: ICLR.cc/2023/Conference/Paper904/Authors
title: Response to Reviewer ozYZ
comment: Thank you for your positive evaluation of our paper! We took all your comments at heart and improved our paper accordingly.

## Responses to “Weaknesses” and “Clarity, Quality, Novelty and Reproducibility”

- We have carefully proofread our paper and fixed all grammar mistakes. 
- Theorem 2 holds under Assumptions 1-4. Of note, the smoothness assumption 3 can be replaced by different assumptions on the “hardness” of the underlying estimation problem, such as sparsity (see Appendix C). We added this information to Theorem 2 in our revised paper. _Regarding Assumption 2_: even if Assumption 2 does not hold, all our theoretical results (Theorem 1-3) do still hold for the quantity on the right-hand side of Eq. (2). However, this quantity cannot be interpreted anymore as the CATE because identifiability is not ensured without Assumption 2. As a remedy, this quantity can still be interpreted as conditional ETT or LATE (see our discussion at the end of Section 2). We added this information to the end of Section 2 of our revised paper.
- We are happy to clarify our contributions as compared to Wang & Tchetgen Tchetgen (2018) and Kennedy (2022). In their paper, Wang & Tchetgen Tchetgen derive a multiply robust parametrization of the efficient influence function (EIF) for the _ATE_ under IV assumptions. However, their estimator and theory are _not applicable_ in the _ITE_ setting. Hence, we propose a pseudo-outcome regression on the EIF as it is done by Kennedy (2022) in the _non-IV setting_. Because we consider the IV-setting, our pseudo outcome is different from Kennedy’s, which means that the estimation and theory cannot simply be adopted. In particular, we need to estimate additional nuisance parameters and eventually obtain a _multiple robust_ ITE estimator. Furthermore, Kennedy provides a convergence rate for the doubly robust CATE estimator in the non-IV setting. In our proof (Appendix B), we rely on certain mathematical results from Kennedy, but use these to derive a _multiply robust_ rate in the _IV setting_ is non-trivial and part of our contribution. In particular, we came up with a clever expression for the bias term $\hat{r}(x)$ (see Appendix B, Lemma 1 and B2).  Hence, both our convergence rate and our proof are novel. We expanded our related work section on doubly robust CATE estimation in our revised paper, clarifying our contribution. In particular, we added a new paragraph labelled *“Doubly robust rates for CATE estimation”*, wherein we reiterate our novelty.

From: ICLR.cc/2023/Conference/Paper904/Authors
title: Response to Reviewer r18k
comment: Thank you for your constructive review! We improved our paper, and addressed your comments in the following ways:

## Responses to “Weaknesses”

1. We followed your suggestion and changed to naming from “individual treatment effect” (ITE) to “conditional average treatment effect” (CATE).
2. The robustness result of Theorem 1 is of algebraic nature, and, hence, all inequalities are meant to hold almost surely. This is in line with other robustness results from the treatment effect estimation literature (e.g., van der Laan and Gruber 2012, Theorem 2). Consistency is a direct consequence: our estimator will converge to the true CATE if the nuisance estimators in one of the three model specifications converge to their oracle estimands. Even though equalities rarely hold in practice due to estimation error, Theorem 1 ensures that our estimator behaves reasonably in large sample regimes if the nuisance models are correctly specified. We clarified this in the revised version of our paper.
3. We are happy to clarify our contribution as compared to Kennedy (2022). In his paper, Kennedy provides a convergence rate for the doubly robust CATE estimator in the non-IV setting. In our proof (Appendix B), we rely on certain mathematical results from Kennedy, but using these to derive a *multiply robust* rate in the *IV setting* is non-trivial and part of our contribution. In particular, we came up with a clever expression for the residual term $\hat{r}(x)$ (see Appendix B, Lemma 1 and B2).  Hence, both our convergence rate and our proof are novel. We expanded our related work section on doubly robust CATE estimation in our revised paper to clarify our contribution. Therein, we added a new paragraph called *„Doubly robust rates for CATE estimation“*.
4. Yes, we can indeed provide a pointwise result. In Theorem 2, we leverage the smoothness assumptions which imply that our bound does not depend on $x$. However, in the proof of Theorem 2, we actually prove a more general bound that depends _pointwise_ on the nuisance estimation errors at $x$.  Furthermore, a result for the mean integrated squared error (MISE) can be obtained by integrating both sides. In our revised paper, we reworked Appendix B and now clearly state this general result (Lemma 2). We also added clarification and reference to Section 4.2 in the main paper.

From: ICLR.cc/2023/Conference/Paper904/Authors
title: Response to Reviewer B9xa
comment: Thank you for your positive review!


## Response to “Weaknesses”



- We took your feedback at heart and added additional experiments based on real-world data (see new Appendix L). In particular, we now enriched our existing real-world experiments to provide more insight into the treatment effect heterogeneity. Here, we analyze the treatment effect heterogeneity with respect to other covariates such as the spoken language or the number of hospital visits before signing in for the lottery. Overall, our experiments using real-world data should provide practical insights and illustrate an application.
- Besides that, we would like to reiterate our previous benchmarking efforts. To strengthen the empirical evaluation of our method, we also performed further experiments using semi-synthetic data where counterfactual outcomes are observable (see Appendix J). Thereby, we follow best practices in the causal inference literature (e.g., Curth and van der Schaar 2021).

From: ICLR.cc/2023/Conference/Paper904/Authors
title: Response to all reviewers
comment: Thank you very much for your positive evaluation of our paper and your helpful comments! We addressed all of them in our revised paper and highlighted changes in red colour. In particular:

- We added additional experiments using real-world data, thereby providing new insights (see Appendix L).
- We clarified our contribution as compared to Kennedy (2022) (see the new section *„Doubly robust rates for CATE estimation“* in the Related Work).
- We included all other suggestions, such as changing the terminology (ITE -> CATE), an extended discussion of Assumption 2, and fixing typos.

Given the improvements above, we are confident that our paper is now a valuable contribution to the literature and a good fit for ICLR 2023.

From: ICLR.cc/2023/Conference/Paper904/Reviewer_B9xa
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper introduces a method of estimating individualized treatment effect (of binary treatments) with a multiple robust estimator. To this end, the authors build on previous work on ITE estimation as well as multiple robust models and extend the latter which results in a two-stage algorithm  (MRIV) which outputs a consistent multiple robust estimator of the ITE. In the first step of MRIV the nuisance parameters and the propensity score are estimated; in the second step the ITE estimator is computed by regressing the pseudo outcome on the confounders. The authors, building on existing approaches, show what the convergence rate of MRIV is, and compare it to the Wald estimator. Subsequently, a TARnet-like architecture of a deep network (but with two distinct representations) is proposed in order to estimate the nuisance parameters (this can be plugged in to step one of the MRIV procedure). The paper concludes with experiments on simulated and real data.
strength_and_weaknesses: The method presented in the paper is novel and relevant. The paper is intuitive and easy to follow.

The experiment on real data is limited and somewhat inconclusive.
clarity,_quality,_novelty_and_reproducibility: The paper is well written, clear and, while building on previous results, contains original material.
summary_of_the_review: This paper fills a small but relevant gap in causal inference by proposing a multiply robust estimator for the ITE. While the authors rely extensively on existing results, the proposed method is novel and and constitutes a valuable contribution to the field.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper904/Reviewer_r18k
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper proposes an estimator for the conditional average treatment effect (CATE) when there are unobserved confounders. The estimator uses binary IV. This estimator is multiply robust (a more general property than the double robustness property), implying that the mean squared error of the estimated CATE decays with the sample size $n$ when one of the three conditions holds true. 


strength_and_weaknesses: Strength: 

This paper provides a nice solution to the important problem of estimating CATE in the presence of unobserved confounders. The proposed method is easy-to-implement, enjoys the nice multiple robustness property, and is demonstrated to have low estimation error in the estimation of CATE as compared to a number of benchmarks. 

Weakness:

1. The target estimand in this paper, $\tau(x) = E[Y(1) - Y(0) \mid X = x]$, is generally referred to as the conditional average treatment effect (CATE) in the literature. Calling it ITE is a bit confusing. 

2. In Theorem 1, the three conditions do not look correct to me. If $\hat{\mu}^Y_0$ refers to an estimator of $\mu^Y_0$, $\hat{\mu}^Y_0$ should have some estimation error, and therefore it is impossible that $\hat{\mu}^Y_0= \mu^Y_0$. Similarly for $\hat{\tau}$, $\hat{\tau}_{init}$, etc. For these three conditions, do the authors refer to the consistent or unbiased estimation of $\mu^Y_0$ (and other parameters)?

3. Given that the technical results are largely built on Kennedy (2022), it is useful to clarify the technical and other contributions as compared to Kennedy (2022).

4. Theoretical results are about the MSE of $\hat\tau(x)$. Can the authors say something about whether $\tau(x)$ can be consistently estimated for every $x$ (i.e., pointwise result)?
clarity,_quality,_novelty_and_reproducibility: The problem this paper studies is well-motivated. The proposed estimator is supported by an upper bound of MSE. The paper is well-written, and I enjoy reading the paper.
summary_of_the_review: This paper provides a multiply robust estimator for CATE, and shows an upper bound for the MSE of the estimated CATE. The proposed estimator can be broadly applicable. I think the paper can be strengthened by addressing the questions listed in "weaknesses".
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper904/Reviewer_ozYZ
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The paper studies the problem of estimating individual treatment effects (ITE) under unobserved confounding using binary instruments. The paper starts with discussing conditions under which the ITE is identifiable. It then proceeds with proposing a two-stage meta learner (MRIV) that estimates the ITE. In the first stage, nuisance components are estimated; and in the second stage, a pseudo outcome is regressed on covariates. The authors then establish theoretical properties of the proposed MRIV estimator and develop a tailored deep neural network using this framework. Finally, the authors demonstrate the performance of the proposed estimator through simulations and real data analysis. 
strength_and_weaknesses: Strengths

-	The paper is very well-written. The proposed method is motivated and explained in a very nice way. 

-	The paper addresses an interesting and important question: estimating individual treatment effects under unobserved confounding.

-	The proposed method is very elegant conceptually. 

-	The proposed method appears to have good performance both theoretically and empirically. 

Weaknesses

-	There are a few grammar mistakes in the paper. 

-	The assumptions are not stated clearly in the theorems. 

-	The paper appears like a nice combination of the Wang & Tchetgen Tchetgen, 2018 paper and the Kennedy, 2022 paper, but there aren’t too many new ideas beyond what’s already in the two papers. It will be extremely helpful if the authors can emphasize a bit more on the new ideas. 
clarity,_quality,_novelty_and_reproducibility: -	There are a few grammar mistakes in the paper. It will be great if the authors can proofread the paper more carefully. Here are a few examples:

Page 3 Example: “assumption 1 holds is when” -> “assumption 1 holds when”

Page 4 line 4: “is a special case of when our Assumption 2 holds” -> “is a special case of our Assumption 2”

Page 4 line 7: ”As result” -> “As a result”

Page 9 Results (line 8): “individual in this age group” -> “individuals in this age group”

-	What are the assumptions made in the theorems? It will be helpful to state them explicitly. Related to this, if Assumption 2 does not hold, is the estimator still a good estimator for the right hand side of equation (2)? Would Theorems 1, 2 and 3 hold for estimating the right hand side of equation (2)?

-	To me, the paper appears like a nice combination of the Wang & Tchetgen Tchetgen, 2018 paper and the Kennedy, 2022 paper. The combination enables solving a harder problem in a nice way: estimating individual treatment effects under unobserved confounding. In terms of technical tools used, are there new ideas beyond what’s already in the two papers? It will be extremely helpful if the authors can emphasize a bit more on the new ideas/tools/method.

summary_of_the_review: Overall, I find this paper a good contribution to the literature. See the above sections for more detailed comments.  
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper904/Reviewer_BewB
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The authors present a method for estimating the CATE in the presence of unobserved confounding when a binary IV is available.
strength_and_weaknesses: See below.
clarity,_quality,_novelty_and_reproducibility: See below.
summary_of_the_review: Overall I found this work to be a reasonable, if straightforward extension of CATE estimation methods to the IV model. I have the following concerns, but most are relatively minor and I recommend acceptance.

Abstract:

1. In partial compliance settings, treatment assignment (not compliance) is used as the IV for actual drug usage.

2. I recommend being more specific "asymptotically outperforms."

Section 1:

1. I recommend replacing the term "Individual treatment effect" with "Conditional average treatment effect since "individual treatment effect" is used to mean something different in much of the causal inference literature.

2. help in potentially --> helps in making potentially

3. , --> .

4. Race and gender are standard parts of intake forms and are both typically stored in medical records.

5. "Our framework...": I found these two sentences hard to follow. I would focus on giving intuition here rather than jargon. For example, the meaning of "multiply robust" depends on the context. You should give intuition for what it is robust to why that is important.

6. Registering for Medicaid should not be referred to as "compliance" even if the statistical model is the same as a partial compliance model. There is no mandate that someone needs to register if eligible. 

7. Assumption 2 is a *major* assumption that requires more discussion than it is given here. For example, if $U$ includes socioeconomic status, as suggested by the authors then this assumption implies that SES cannot moderate the treatment effect. 

Section 2:

1. In Assumption 4: do these conditions need to hold everywhere on $\mathcal{X}$ or is it sufficient for them to hold almost everywhere w.r.t. $P(X)$?

2. Need to explicitly define $r_\tau(n)$ or at least give intuition.

Section 5:

1. "One explanation...": I would refrain from making this hypothesis without including explicit citations to back up the underlying theoretical model.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold