Paper: When Treatment Effect Estimation Meets Collider Bias: A Dual Counterfactual Generative Approach (ICLR-2024)

Reviewer comment (Theory):
Lack of identifiability guarantee. The heuristic generator/discriminator approach does not allow the derivation of identifiability results, meaning that there is no guarantee the estimated treatment effets will be unbiased even if there are abundance of samples and appropriate assumptions.

Author response:
**Response to Weakness 2**

Thank you for raising the issue. However, our motivation is to recover the target population from the collider biased observational data with the help of a representive experimental dataset without collider bias. GAN models is just a way to implement the generation process of the missing $S$ labels in $D_{exp}$ and missing $S=0$ samples in $D_{obs}$. **As long as the generative models are well trained, the $S=0$ samples can be learned well with the information of the target population contained in $D_{exp}$.**

Reviewer comment (Theory):
The proposed approach is only applicable with a binary collider variable.

Author response:
**Response to Weakness 3**

Thank you for the comment. However, we have to emphasize that the definition of $S$ is that, the collected observational data is from $S=1$ and the unobserved data is $S=0$. By the definition, **we believe $S$ can only be binary since it is just an indicator of whether a unit is selected into the observed sample**. Therefore, such binary setting is reasonable common in real-world scenarios.

Reviewer comment (Evaluation):
How is the CATE estimations evaluated on Twins dataset? The Twins dataset is usually used only for evaluating ATE estimations as there is no ground truth counterfactual outcomes.

Author response:
**Response to Question 1**

The Twins dataset is used in previous CATE estimation works [2,3] because **the factual outcomes of the lighter/heavier twin are regarded as the counterfactual outcomes of the heavier/lighter twin.** Therefore, **we can calculate the ground truth CATE** by using the outcome of the heavier twin minus the outcome of the lighter twin.

Reviewer comment (Experiments):
Is there any reason for selecting the datasets used in the manuscript? IHDP itself is a semi-synthetic dataset with only one data generation process, which only covers rather limited real-world scenarios. Twins is commonly used for ATE estimation instead of CATE estimation. Datasets with more comprehensive DGPs would be desirable, such as ACIC2016.

Author response:
**Response to Question 3**

Thank you for the comment. As stated in **Response to Weakness 1** and **Response to Question 1**, we choose the IHDP and Twins datasets as we follow previous CATE estimation works [1,2,3]. We also adopt your advice and conduct additional experiments on ACIC 2016. Due to time limitation, only some of the results have been obtained, but based on the current results, our method still achieves an improvement in performance over baselines on ACIC 2016. Further results will be updated in time.

We hope the above answers can solve your problems. Thank you again for your comments and suggestions. Best wishes!

Reviewer comment (Experiments):
As the authors stated in the Conclusion, they use a GAN-based model to generate unbiased samples, which is hard to train when the sample size is small.

Author response:
Thank you for spending much time reading the paper and giving detailed comments. Responses to some of your questions follow.

**Response to Weaknesses**

We agree with your comments and have conducted experiments with different sizes of experimental datasets to explore the impact of the sample size on the performance of our method. You can find the results in Table 2 on Page 8.

We hope the above answers can solve your problems. Thank you again for your comments and suggestions. Best wishes!

Reviewer comment (Experiments):
Authors claim that the sample from the experimental data faithfully represents "the entire population" P_X, which is incorrect and misleading. External validity is arguably the biggest drawback of experimental data, as factors influencing both selection into the experiment (S=1) and modifying the treatment effect lead to differences in the average causal effects between the experimental population and target populations.

Author response:
Thank you for spending much time reading the paper and giving detailed comments. Responses to some of your questions follow.

**Response to Weaknesses**

We agree that in some scenarios, experimental data may not be representive for the target population, and the proposed method is not applicable. However, **we think there are also many scenarios that the experimental population is the same as the target population, and such experimental design is feasible.** For example, when studying the effect of pricing on restaurant evaluation on the online ordering platform, only those who extremely like the restaurant's dishes or have a lot of complaints will take the time to evaluate the restaurant on the platform, which is the observation data. However, the target population is actually all users who use the platform, which leads to selection bias. The platform can randomly select a small number of users from among all users, and encourage them to evaluate the restaurant by discounts or coupons, which is the experimental data that can represent the target population.

We have to emphasize our problem formulation that throughout the paper, **we use $S=1$ to denote whether a unit is selected into the observational data**, which is the observational population. The experimental data is from another population that we assume is the same as the target population. Therefore, **units in the experimental data are not all from the $S=1$ distribution** (the observational population).

We hope the above answers can solve your problems. Thank you again for your comments and suggestions. Best wishes!

Reviewer comment (Theory):
In healthcare, for example, people who are eligible and join the trials (i.e., the experimental data in this case) are not representative of the entire population. While it is common practice to compare experimental and observational data or attempt to combine them, this is only possible—without further assumptions—to the extent of "overlap" between the entire population (as captured by the observational data) and the experimental population. The reviewer refers to literature on the generalizability of experiments, including works by Rothwell et al. (2005), Mullinick et al. (2015), and Dahabreh et al. (2019).

Author response:
Thank you for spending much time reading the paper and giving detailed comments. Responses to some of your questions follow.

**Response to Weaknesses**

We agree that in some scenarios, experimental data may not be representive for the target population, and the proposed method is not applicable. However, **we think there are also many scenarios that the experimental population is the same as the target population, and such experimental design is feasible.** For example, when studying the effect of pricing on restaurant evaluation on the online ordering platform, only those who extremely like the restaurant's dishes or have a lot of complaints will take the time to evaluate the restaurant on the platform, which is the observation data. However, the target population is actually all users who use the platform, which leads to selection bias. The platform can randomly select a small number of users from among all users, and encourage them to evaluate the restaurant by discounts or coupons, which is the experimental data that can represent the target population.

We have to emphasize our problem formulation that throughout the paper, **we use $S=1$ to denote whether a unit is selected into the observational data**, which is the observational population. The experimental data is from another population that we assume is the same as the target population. Therefore, **units in the experimental data are not all from the $S=1$ distribution** (the observational population).

We hope the above answers can solve your problems. Thank you again for your comments and suggestions. Best wishes!

Reviewer comment (Theory):
The assumptions about how the covariate distributions in experimental and observational datasets differ, with explicit reference to the role of selection/collider bias, should be made more clear, and should remain grounded in the reality that the experimental data does not represent the entire population.

Author response:
Thank you for spending much time reading the paper and giving detailed comments. Responses to some of your questions follow.

**Response to Weaknesses**

We agree that in some scenarios, experimental data may not be representive for the target population, and the proposed method is not applicable. However, **we think there are also many scenarios that the experimental population is the same as the target population, and such experimental design is feasible.** For example, when studying the effect of pricing on restaurant evaluation on the online ordering platform, only those who extremely like the restaurant's dishes or have a lot of complaints will take the time to evaluate the restaurant on the platform, which is the observation data. However, the target population is actually all users who use the platform, which leads to selection bias. The platform can randomly select a small number of users from among all users, and encourage them to evaluate the restaurant by discounts or coupons, which is the experimental data that can represent the target population.

We have to emphasize our problem formulation that throughout the paper, **we use $S=1$ to denote whether a unit is selected into the observational data**, which is the observational population. The experimental data is from another population that we assume is the same as the target population. Therefore, **units in the experimental data are not all from the $S=1$ distribution** (the observational population).

We hope the above answers can solve your problems. Thank you again for your comments and suggestions. Best wishes!