Paper: Empirical Analysis of Model Selection for Heterogeneous Causal Effect Estimation (ICLR-2024)

Reviewer comment (Experiments):
The general framework proposed in the paper seems to have high requirements for the amount of data.

Author response:
**Data requirements in the proposed framework**

The proposed framework does not in principle need large amounts of data. Our goal was to conduct a comprehensive large-scale evaluation study to make a fair comparison between the various evaluation metrics used in the literature. This helps to create an informative prior for practitioners regarding which evaluation metrics maybe useful for them. 

Further, the suggestions of a two-level model selection strategy and causal ensembling do not necessarily need lots of data. It is upto the practitioner to train as many CATE estimators as required for their application and then incorporate the two-level model selection and causal ensembling strategy. Also, with regards to the training of the nuisance models (outcome & propensity models), we are in the typical regime as used by the other works in this area, with the standard train/validation split for the various datasets consider in our work.

Reviewer comment (Novelty):
Some relevant papers were missing, e.g. Nguyen et al., OpportunityFinder: A Framework for Automated Causal Inference.

Author response:
**Missing relevant works**

We tried to be as thorough as we could about the prior works and did cover the most relevant ones. We do appreciate the suggestion by the reviewer and are happy to talk about other related works concerning our study. 

Regarding the work by Nguyen et al. pointed out by the reviewer, their focus is more on sensitivity analysis, which deals with measuring robustness to unobserved confounders, randomized treatments, etc. While sensitivity analysis is an important practical aspect of causal inference, our goal with model selection is more general since sensitivity analysis alone might not be able to result in a clear distinction between several CATE estimators.

Reviewer comment (Experiments):
Due to this is an empirical study, there may be some concerns about the results of the experiments, such as whether all experimental conditions have been taken into account, whether the way the training and validation datasets are partitioned will affect the final experimental conclusions, and etc.

Author response:
**Validity of conclusions in our experimental analysis**

We agree with the reviewer and would like to state that we were extremely careful about such issues. We trained the various nuisance models for CATE estimators and metrics using cross-validation with AutoML to be robust to issues like different partitioning of the training and validation set. Further, we have used a large number of random seeds (10 per experiment) to be robust to discrepancy from initialization of model parameters. Finally, since the statistics of ITE estimates, covariate dimension, etc. can change across datasets (Table 4, 5, 6), to determine robust trends we look at a large collection of datasets (78 in total) and train a wide variety of CATE estimators per dataset.

Reviewer comment (Experiments):
The general framework proposed in the paper seems to have high requirements for the amount of data.

Author response:
**Data requirements in the proposed framework**

The proposed framework does not in principle need large amounts of data. Our goal was to conduct a comprehensive large-scale evaluation study to make a fair comparison between the various evaluation metrics used in the literature. This helps to create an informative prior for practitioners regarding which evaluation metrics maybe useful for them. 

Further, the suggestions of a two-level model selection strategy and causal ensembling do not necessarily need lots of data. It is upto the practitioner to train as many CATE estimators as required for their application and then incorporate the two-level model selection and causal ensembling strategy. Also, with regards to the training of the nuisance models (outcome & propensity models), we are in the typical regime as used by the other works in this area, with the standard train/validation split for the various datasets consider in our work.

Reviewer comment (Novelty):
Some relevant papers were missing, e.g. Nguyen et al., OpportunityFinder: A Framework for Automated Causal Inference.

Author response:
**Missing relevant works**

We tried to be as thorough as we could about the prior works and did cover the most relevant ones. We do appreciate the suggestion by the reviewer and are happy to talk about other related works concerning our study. 

Regarding the work by Nguyen et al. pointed out by the reviewer, their focus is more on sensitivity analysis, which deals with measuring robustness to unobserved confounders, randomized treatments, etc. While sensitivity analysis is an important practical aspect of causal inference, our goal with model selection is more general since sensitivity analysis alone might not be able to result in a clear distinction between several CATE estimators.

Reviewer comment (Experiments):
The deep learning or representation-based CATE methods are excluded. The adopted estimators are numerous indeed, while the underlying machine learning models (Appendix A.2) are rather basic.

Author response:
**Lack of Deep Learning based methods**

We agree with the reviewer that it might be interesting to include deep learning-based CATE methods as well. However, since the scale of the study was already quite large and our primary goal was model selection rather than learning state-of-the-art CATE estimators we decided to focus on basic machine learning models for the nuisance models in the CATE meta-learners. 
Further, the datasets that we work with are mostly tabular datasets, hence basic machine learning techniques with appropriate hyperparameter tuning should have good performance as well. We use state-of-the-art hyperparameter tuning techniques like AutoML to learn well-performing nuisance models.

Reviewer comment (Novelty):
The author propose some new metrics (e.g., the "Switch" approach) which seem reasonable but show marginal performance gaps in the evaluation. I am wondering the motivation and if there is other advantage of using these new metrics?

Author response:
**Motivation behind new metrics**
 
Although our results show that we do not gain much with these proposed metrics, Table 3 still shows that scores like TMLE, Qini, and Calibration (DR) are among the set of dominating metrics across all datasets. Hence, they could be considered as a good prior option for a new study and might be even more suitable for some applications that are tied to their specific use case, like learning under extreme propensity regime, etc. 

The motivation for including adaptive propensity clipping and targeted learning metrics is the extreme propensity regime, where the metrics that depend on inverse propensity scores (IPW, DR) can become biased. Hence, these proposed metrics can handle the extreme propensity region (very small propensity for the observed treatment) by using estimates from the regression-based learners (S/T Learner) or adding inverse propensity correction (TMLE). 

For the Calibration score, the intuition is to match the group ATE across subgroups denoted by the different percentiles of the CATE estimates. We compute the group ATE using DR/TMLE, etc. which is viewed as an unbiased sample of the group ATE, and then we compute the weighted error of the group ATE using the CATE estimator against the unbiased group ATE. For the Qini score, rather than computing the group ATE with the CATE estimator as in Calibration, here we compare the improvement with unbiased group ATE estimates versus the uniform sampling estimate. Intuitively, if there is heterogeneity in the data, then we should expect the policy of uniformly treating an individual to be worse as compared to the policy that assigns treatment to the top-k percentile of the CATE estimates. Hence, the Qini score is qualitatively different than most of the other metrics as it does not directly compare the CATE estimates, rather it evaluates whether the groups most likely to be treated would benefit from the treatment.

Reviewer comment (Experiments):
Although the typical cross validation method could be non-ideal in CATE estimation in general, I am wondering if reporting those numbers can make this work more complete and highlight the effectiveness of the proposed metrics?

Author response:
**Cross-validation results with the ideal metric**

We want to highlight that we compute results with the ideal cross-validation metric (PEHE) with ground-truth ITE estimates, as mentioned at the start of section 5. The ideal PEHE metric is used to normalize the reported scores so they can be made consistent across datasets, and the reported results in Tables 1, 2, and 3 demonstrate the percentage difference in the quality of the chosen estimator by a particular metric compared to the ideal metric. As expected, we can see that none of these metrics are better than the ideal PEHE metric as all the reported scores are greater than zero.

Also, if by the typical cross-validation method, the reviewer meant computing the cross-validation score of the nuisance models (like Mu Score in Schuler et al.), then such an approach is not feasible in our study since we already select over the nuisance models using AutoML and these are shared across the different CATE meta-learners in our experiment. Hence, the cross-validation score of the various nuisance models across CATE meta-learners would be similar.

Reviewer comment (Experiments):
The authors only used 10 replicates (random seeds) to estimate the mean (standard error) Normalized-PEHE, which is probably not large enough.

Author response:
**Number of random seeds used in the study**

As the reviewer stated the scale of the study is quite large, hence 10 random seeds for 78 datasets and 415 CATE estimators per dataset implies a total of $323,700$ CATE estimators trained and evaluated. Therefore, we think that 10 random seeds are enough to determine reliable conclusions given the computational cost of carrying out the study. 

However, we will generate results for another set of 10 random seeds (making it a total of 20 random seeds), and report them soon in a few days.

Reviewer comment (Evaluation):
In Tables 1 and 3, most of the metrics are "dominating" on LaLonde CPS, LaLonde PSID and TWINS while the metrics that are not "dominating" are in the minority. This suggests that most metrics perform similarly on those datasets, which means the conclusions the authors drew are not well supported.

Author response:
**Conclusions from experimental results**

We would like to point that we also report results on the multiple ACIC 2016 datasets (75 datasets) in Table 1, 2, and 3 in the main paper, where it does not happen that the set of dominating metrics is bigger than the set of non-dominating metrics. Note that while there is a single column for the ACIC 2016 dataset, it is a collection of 75 datasets, therefore, the trends reported for them are more informative than the Twins and LaLonde datasets combined. Hence, the conclusions that we draw are valid since we do not rely only on the Twin and LaLonde datasets, but we also include the multiple ACIC 2016 datasets. 

We agree with the reviewer that we may overestimate the set of dominating metrics by only looking at the Twins and LaLonde datasets. However, to determine the set of globally dominating metrics we look at the metrics that are dominating across all the ACIC 2016 datasets as well as the Twins/LaLonde datasets. It is indeed possible for some metrics to be dominating on a few datasets like Twins and LaLonde, but they are not dominating when evaluated across more datasets like ACIC 2016. Specifically, metrics like Match Score and IPW (and its variants) are dominating on the Twins and LaLonde datasets but they are not dominating on the multiple ACIC 2016 datasets. Hence, for a more useful recommendation to practitioners, we suggest metrics that are dominating across a wide range of datasets. Further, the conclusions we draw about the two-level model selection strategy and causal ensembling improving over the single-level strategy are valid since they hold across all the datasets.

Reviewer comment (Theory):
In Section 3, the proposed model selection metrics seem ad hoc to the reviewer, who requests some intuition behind these metrics.

Author response:
**Intuition behind new metrics**

The motivation for including adaptive propensity clipping and targeted learning metrics is the extreme propensity regime, where the metrics that depend on inverse propensity scores (IPW, DR) can become biased. Hence, these proposed metrics can handle the extreme propensity region (very small propensity for the observed treatment) by using estimates from the regression-based learners (S/T Learner) or adding inverse propensity correction (TMLE). 

For the Calibration score, the intuition is to match the group ATE across subgroups denoted by the different percentiles of the CATE estimates. We compute the group ATE using DR/TMLE, etc. which is viewed as an unbiased sample of the group ATE, and then we compute the weighted error of the group ATE using the CATE estimator against the unbiased group ATE. For the Qini score, rather than computing the group ATE with the CATE estimator as in Calibration, here we compare the improvement with unbiased group ATE estimates versus the uniform sampling estimate. Intuitively, if there is heterogeneity in the data, then we should expect the policy of uniformly treating an individual to be worse as compared to the policy that assigns treatment to the top-k percentile of the CATE estimates. Hence, the Qini score is qualitatively different than most of the other metrics as it does not directly compare the CATE estimates, rather it evaluates whether the groups most likely to be treated would benefit from the treatment.

Although our results show that we do not gain much with these proposed metrics, Table 3 still shows that scores like TMLE, Qini, and Calibration (DR) are among the set of dominating metrics across all datasets. Hence, they could be considered as a good prior option for a new study, and might be even more suitable for some applications that are tied to their specific use case, like learning under extreme propensity regime, etc.

Reviewer comment (Experiments):
As mentioned in the Weaknesses, 10 random seeds are not enough.

Author response:
**Number of random seeds used in the study**

As the reviewer stated the scale of the study is quite large, hence 10 random seeds for 78 datasets and 415 CATE estimators per dataset implies a total of $323,700$ CATE estimators trained and evaluated. Therefore, we think that 10 random seeds are enough to determine reliable conclusions given the computational cost of carrying out the study. 

However, we will generate results for another set of 10 random seeds (making it a total of 20 random seeds), and report them soon in a few days.

W5 -> R2: We have updated the draft with results over another set of 10 random seeds, hence a total of 20 random seeds in Tables 10 and 11 in section E (Supplementary). Our conclusions with 20 random seeds are still the same as before with 10 random seeds, hence the conclusions are robust to different random parameter initializations.

Reviewer comment (Writing):
Some terminology clarification could be helpful. There are a few terms which, while present in the literature, are not necessarily universally used. For example, 'nuisance models' and the task being 'model selection' rather than causal modeling/discovery/inference. Also, the first sentence of Section 2 says that X "represents the controls." However, 'control' is a heavily overloaded term in causality (generally referring to treatment = 0 in a binary treatment setting), and since only binary treatment is considered, calling the X's controls is unnecessarily confusing. 'Covariates' or 'confounders' would be clearer. The term 'metric' also feels overloaded and could benefit from clarification of the definition used. In evaluation literature, 'metric' usually refers to a performance measure; for example, in Table 1, 'PEHE' would be the evaluation metric and the items under the 'Metric' column would be 'Estimators'. A sentence or two early on to clarify these definitions would improve clarity.

Author response:
**Regarding the terminology and writing in the paper**

We thank the reviewer for their detailed feedback regarding the presentation and writing! We agree and would make the writing better with clear definitions and clarifications. We have already updated the paper with these suggestions and the changes can be viewed in the new draft. 

Specifically, we refer to X as covariates now as opposed to controls and have added a sentence in section 2 to specify exactly what we mean by metrics for model selection. Further, other suggested changes regarding the writing have been made in the updated draft.

Reviewer comment (Writing):
In Section 2, tau(x) is defined as CATE and tau-hat as a CATE estimate, but tau\~ is not defined before its first appearance under 'CATE Model Selection Metrics' in the equation. The next sentence mentions learning tau\~, but there is no description of what it represents. It appears that a tilde may be missing in the sentence before the equation (where it reads: "includes approximating the ground-truth effect (tau(X)) on the validation set").

Author response:
**Typo in CATE model selection metrics**

We thank the reviewer for pointing this out, indeed there was a typo in the sentence before the equation and it should have been ($\tilde{\tau}$). The same has been corrected in the updated draft.

Reviewer comment (Evaluation):
While the inclusion of novel metrics is appreciated, there is little discussion as to the motivation for including each new one given the large number of metrics already considered. It is unclear whether existing metrics were lacking in some aspect or whether the new metrics capture something not currently captured. Performance-wise, none of the IPTW modifications (IPTW Switch T Score, IPTW CAB T Score) perform better than the IPTW score, and Qini and Calibration scores rarely perform better either. Without strong justification for their inclusion and given their limited performance, they seem to be included for the sake of it and add little to the paper. IPTW and Targeted Learning have at least a sentence each about why they are included (due to handling extreme propensities), but Calibration Score is only justified as having been studied in RCTs, and Qini is described only as originating from uplift modeling literature.

Author response:
**Motivation behind new metrics**

Our motivation was to connect widely used ideas from related problems like policy learning, uplift modeling, etc. to have more good candidates for model selection. Although our results show that we do not gain much with these proposed metrics, Table 3 still shows that scores like TMLE, Qini, and Calibration (DR) are among the set of dominating metrics across all datasets. Hence, they could be considered as a good prior option for a new study.

As the reviewer stated, the motivation for including adaptive propensity clipping and targeted learning metrics is the extreme propensity regime. For the Calibration score, the intuition is to match the group ATE across subgroups denoted by the different percentiles of the CATE estimates. We compute the group ATE using DR/TMLE, etc. which is viewed as an unbiased sample of the group ATE. Subsequently, we compute the weighted error of the group ATE using the CATE estimator against the unbiased group ATE. 

For the Qini score, rather than computing the group ATE with the CATE estimator as in Calibration, here we compare the improvement with unbiased group ATE estimates versus the uniform sampling estimate. Intuitively, if there is heterogeneity in the data, then we should expect the policy of uniformly treating an individual to be worse as compared to the policy that assigns treatment to the top-k percentile of the CATE estimates. Hence, the Qini score is qualitatively different than most of the other metrics as it does not directly compare the CATE estimates, rather it evaluates whether the groups most likely to be treated would benefit from the treatment.

Reviewer comment (Novelty):
Given that the two-level model selection strategy is described as 'novel', more detail would be helpful. It states "we first select its hyperparameters using a metric that is designed with similar inductive bias" — but which metric? Does it vary by estimator? It is also unclear whether this is considered a contribution, since it’s described as novel but not listed under 'Contributions' in the introduction.

Author response:
**Details regarding the two-level model selection strategy**

The details regarding the two-level model selection strategy are present in Section 4, and the same has been mentioned under the contributions section (bullet 2). But we agree the same could be presented in a better manner, and we have done so in the updated draft. 

Indeed, the choice of metric for selecting hyperparameters in the two-level strategy depends on the CATE estimator. Specifically, the hyperparameters for CATE estimators in our study are the final regression models, hence we need to do this for only DR-Learner, R-Learner, X-Learner, and Projected S-Learner, as the others do not have hyperparameters. Now for each of the four CATE estimators mentioned above, we select the hyperparameters using a metric with similar inductive bias, i.e., DR Score for the DR-Learner; R Score for the R-Learner, X Score for the X-Learner, and S Score for the Projected S-Learner. 

Essentially, the two-level model selection strategy reduces the population of CATE estimators to different CATE meta-learners, and the choice of metric for reducing the population of a specific CATE meta-learner (e.g. DR-Learner) is designed to maximize the similarity with the inductive bias of the CATE meta-learner, to do reliable hyperparameter optimization.

Reviewer comment (Writing):
In Section 3.1, the IPTW Switch equation shows using the IPW approach if the propensity score is >= epsilon, and the regression learner when it's < epsilon. Presumably, epsilon is a small number. However, the sentence after that equation says that IPW is used "if the propensity of the observed treatment for that sample is large." This seems inconsistent — if regression learner is used when the propensity score is small (< epsilon), then IPW is used whenever the propensity score is 'not small' (>= epsilon), which is different from it being 'large'. Clarification is needed.

Author response:
**IPTW Switch equation and related description**

We agree with the reviewer and have updated the related description. Indeed, the switch score would rely on IPW if the propensity is not very small. We set epsilon to very small values and hence IPW is used when the propensity for the observed treatment is not very small.

Reviewer comment (Writing):
In Section 4, under "CATE estimators with well-tuned nuisance models", the 3rd paragraph starts with "Further for the CATE estimators with final models (f)" — clarification is needed on what it means for an estimator to have a 'final model' versus not.

Author response:
**Final models in CATE estimators**

By final model, we mean the regression model used for predicting CATE in direct meta-learners like DR Learner (eq 15), R Learner (eq 16). For example in DR Learner, we construct the doubly robust potential outcomes ($Y_{0}^{DR}, Y_{1}^{DR}$) and then predict the CATE by learning the regression model ( $f_{\theta} : X \rightarrow  Y_{1}^{DR} - Y_{0}^{DR}$ ) that maps the covariates to doubly robust potential outcomes. This is referred to as the final (regression) model.

 Details regarding the CATE estimators were pushed in section B in the Appendix due to space constraints, but we will try to add a few lines about them in the main text, ideally in section 2. For now, we have referred to the specific equation when mentioning the final model of CATE estimators in Section 4.