Paper: Counterfactual Generative Models for Time-Varying Treatments (ICLR-2024)

Reviewer comment (Novelty):
Lack of comparison of technological innovations from previous approaches. It might be helpful to understand the contribution of this paper by adding a paragraph discussing this.

Author response:
> Lack of comparison of technological innovations from previous approaches. It might be helpful to understand the contribution of this paper by adding a paragraph discussing this.

We are grateful to the reviewer for emphasizing the significance of comparing our approach with previous methodologies. In our manuscript, we have compared our methods with several state-of-the-art algorithms such as the G-Net and KDE based methods. We have also acknowledged the connection of our work to the broader literature in the Related Work section (1.1). Additionally, we delve into the detailed methodological connections to related methods in Appendix E. Following the valuable feedback received from reviewers, we have further refined and expanded Appendix E to clarify and enhance the comparison with existing approaches.

Reviewer comment (Reproducibility):
How should readers use your methods to estimate ITE?

Author response:
> How do readers use this model to solve causal issues, such as ITE estimation?

We appreciate the reviewer's reference to Individual Treatment Effects (ITE). We would like to clarify that our approach to estimating the counterfactual distribution across a population, rather than individual-focused estimates (ITE), is driven by the practicality of our research's applications, such as in public health policy. Implementing policies on an individual basis can be costly and sometimes unfeasible. By focusing on the population level (or a sub-population), we capture individual differences, which is essential for effective policy-making. In contrast, the variation within individual counterfactuals often stems from observational noise and might need substantial data for precise estimation. Consider, for instance, the COVID-19 data: our objective is to analyze the counterfactual outcome of a state-wide policy (mask mandate) and provide policymakers with a range of possible counterfactual outcomes that reflect variability in county-level confirmed cases. In such contexts, policymakers might be less concerned with the estimation noise at the individual level and more focused on the broader variability across different individuals.

 In the meantime, a related extension of our work might be to infer conditional counterfactual outcomes (related to the conditional average treatment effect, CATE). This corresponds to looking at the distribution of the outcome under a specific subpopulation. Since our covariates are assumed to be time-varying, a common approach is to introduce a set of static baseline covariates, $V$ (Robins et al, 1999). The baseline covariate $V$ denotes the static feature (such as a patient's gender or age) that will influence both the time-varying covariates $X$ and the outcome $Y$. We can then draw counterfactual samples from a specific sub-population by conditioning on the values of the $V$. In this framework, we observe ($Y_t^i$, $\overline{A}_t^i$, $\overline{X}_t^i$, $V^i$), where $V \in \mathbb{R}^{
    \nu}$ is a static baseline variable that varies by individual.  Accordingly, the generator will have an additional input: 
$ g\_{\theta}(z, \overline{a},v): \mathbb{R}^r \times \mathcal{A}^d \times \mathbb{R}^{\nu} \rightarrow \mathcal{Y}$ , and the IPTW weights will become $ w\_\phi(\overline{a},\overline{x},v) = \frac{1}{\prod\_{\tau=t-d+1}^{t} f\_\phi(a\_{\tau}|\overline{a}\_{\tau-1},\overline{x}\_{\tau},v)}$.   Correspondingly, the objective in Proposition 1 will become $\mathbb{E}\_{v} ~\mathbb{E}\_{\overline{a}} ~\left[\mathbb{E}\_{y \sim f\_{\overline{a},v}} ~\log f\_\theta(y|\overline{a},v)\right] \approx \frac{1}{N}\sum\_{(y,\overline{a},\overline{x},v) \in \mathcal{D}}w\_\phi(\overline{a},\overline{x},v) \log f\_\theta(y|\overline{a},v).$ 

 We have conducted additional experiments using the fully synthetic dataset, where the baseline variable, $V$, was divided into two groups. The $V$ was uniformly drawn from $[-1,0]$ in the first group and from $[0,1]$ in the second group. We then looked at the performance of the generative samples corresponding to two sub-groups. We included the results in **Appendix I** (Table 4, Fig.11, and Fig.12) of the revised manuscript.  Our method (MSCVAE) still consistently outperforms other baseline methods in the simplified results below (Wasserstein distance, smaller the better):

- $V\in[-1,0]$

|         | d=1  | d=3  |  d=5  |
| ----------- | ----------- | ----------- |----------- |
| MSM+NN      |   0.408    |  0.449   |  0.368 |
| KDE               |      0.201 |  0.562 |  0.564  |
| Plugin+KDE   |        0.117    |      0.134      |   0.196      |
| G-Net             |        0.431        |      0.823       |   0.843    |
| CVAE             |       0.240          |     0.571       |   0.585   |
| MSCVAE        |        0.051         |      0.083      |    0.186   |

- $V\in[0,1]$

|         | d=1  | d=3  |  d=5  |
| ----------- | ----------- | ----------- |----------- |
| MSM+NN      |   0.407    |  0.466  |  0.388 |
| KDE               |      0.222 |  0.548 |  0.561  |
| Plugin+KDE   |        0.121   |      0.109      |   0.182      |
| G-Net             |        0.452       |      0.739       |   0.768    |
| CVAE             |       0.258          |     0.565      |   0.591   |
| MSCVAE        |        0.047         |      0.068      |    0.172   |

Reviewer comment (Experiments):
What do treatments represent in the datasets used in experiments?

Author response:
> What do treatments represent in the datasets used in experiments?

We appreciate the reviewer's attention to the meaning of the treatment variable in our datasets. As stated in the manuscript, for the semi-synthetic Pennsylvania COVID-19 dataset, the treatment refers to the implementation of state-level mask mandate policies. Similarly, in the real nationwide COVID-19 dataset, it denotes county-level mask mandate policies. Regarding the fully-synthetic data and the TV-MNIST datasets, these treatment variables represent abstract quantities, primarily used for validating the performance of our methods under different data settings. In these cases, the treatment could be conceptualized as hypothetical decision variables. For instance, in the context of the TV-MNIST dataset, it might represent actions like closing or opening one’s eye in relation to handwritten digits.

Reviewer comment (Theory):
The authors may not give detailed explanations about the causal assumptions in the main paper, but at least mention the names of the causal assumptions in the paper.

Author response:
Thanks for the suggestion. In the revised manuscript, we mentioned the three causal assumptions in the methodology section.

Reviewer comment (Theory):
In Algorithm 1, the authors suggest that we should draw a sample epsilon from $N(0,I)$, where $I$ is the total number of individuals. What is the point of setting epsilon as a realization with large variance? Usually, $I$ can be very large. Indeed, if there are a large number of individuals, say $I=10000$, a realization can be very large. Further, why do you model epsilon as normally distributed?

Author response:
We would like to clarify that $I$ should stand for the identity matrix, and $N$ represents the total number of samples. We have fixed it in the revised paper.

>  Why do you model epsilon as normally distributed?

The Gaussian assumption for the VAE is a common assumption which enables a closed form expression for the ELBO  (Kingma and Welling, 2013). To elaborate, our generative framework essentially involves identifying a transformation that converts random noise into the desired target distribution, which in this case is the counterfactual distribution. It's a standard practice in the field of generative models to use the normal distribution as the basis for our source distribution (Kingma and Welling, 2013; Sohn et al, 2015; Higgins et al, 2016).

Reviewer comment (Reproducibility):
In the paper, the authors state that t=1, …, T in the section of PROBLEM SETUP. However, when the authors present Algorithm 1, t = d, …, T. It is strange that t=d, …, T in Algorithm 1. Is it a typo mistake? If not, from my realization about Algorithm 1, d should be determined. How to determine the value of d?

Author response:
Following the previous point, we ranged $t$ from $d$ instead of $1$ because the first $d-1$ time points do not have full-length ($=d$) history.

> How to determine the value of d?

As a standard in longitudinal causal inference  (Robins et al, 1994; Lim et al, 2018; Bica et al, 2020; Li et al, 2021; Bica et al 2021), $d$ is pre-determined as prior knowledge of the data generating process.  This approach is validated by our numerical findings, proving to be an effective method. For instance, in our COVID-19 studies, we set $d=3$ following insights from epidemiological research, which indicates that a mask mandate typically requires two to three weeks to become effective in a population.

Intuitively, $d$ can be considered as a causal structure that determines how many arrows should point to the current variable of interest from history. Therefore, if one is interested in estimating $d$ instead of pre-determining it, a potential method is causal discovery which automatically infer the DAG (Figure 8) from data.

Reviewer comment (Theory):
IPTW values can be notably small and, as highlighted by the author, require precise definition. This circumstance can exacerbate in sequential treatment scenarios.

Author response:
Thank you for your comments regarding our use IPTW. We agree that IPTW can be unstable and might lead to less reliable results in certain contexts.  Nevertheless, it's important to highlight that integrating IPTW with our novel generative framework represents a significant and novel methodological advancement. This combination is particularly effective in capturing complex, high-dimensional patterns in counterfactual outcomes, an area our study uniquely addresses to the best of our knowledge.

To address the stability concerns in a practical context, our experiments incorporated established stabilization strategies, such as quantile truncation and standardization, as extensively discussed in the literature (Xiao et al, 2010; Chesnaye et al, 2022). The results from our experiments also show that incorporating these techniques into the design of our marginal structure model greatly results in promising and dependable numerical outcomes.

Reviewer comment (Experiments):
I believe the following two papers could also serve as baseline references:  
1. "Disentangled Counterfactual Recurrent Networks for Treatment Effect Inference Over Time"  
2. "Estimating Counterfactual Treatment Outcomes Over Time Through Adversarially Balanced Representations"

Author response:
We thank the reviewer for mentioning these baseline methodologies. However, it's important to note that these methods are tailored for mean prediction, which differs from our objective of distribution estimation. Additionally, these methods primarily focus on predicting outcomes at the individual level, whereas our research is geared towards assessing the outcomes at the population level. These fundamental differences might make it challenging for a fair comparison between their methods and ours.  In this regard, we could incorporate these methods to compare with our method in terms of mean estimate, should the reviewer consider it necessary.

Reviewer comment (Theory):
If x isn't utilized in the generator, what's the rationale for calculating weights based on x? Why not solely estimate treatment probability based on the treatment sequence?

Author response:
We would like to clarify that, estimating treatment probability based solely on the treatment sequence will introduce bias. This is because the propensity score, $f(A_t|\overline{A}\_{t-1},\overline{X}\_{t})$, depends the history of both treatments and covariates. Therefore, IPTW can vary for the same treatment sequence, $\overline{a}$, owing to the variability in $X$. Importantly, the covariates $X$ play a crucial role in the standard approaches for IPTW estimation (Robins et al, 1999; Lim et al, 2018). As a result,  while we do not have to directly use  $X$ during training our conditional generative model, we still access them through the IPTW weights in Proposition 1.

Reviewer comment (Theory):
Can you provide a proof for equation 2? What would be the difference in objective if we were to define the distance in terms of Maximum Mean Discrepancy (MMD) or Wasserstein distance?

Author response:
We have included a proof for Eq 2 (Eq 3 in the revised manuscript) in Appendix C (pasted below). The objective function for training the counterfactual generator minimizes the difference between $f\_\theta(\cdot|\overline{a})$ and the true counterfactual distribution $f\_{\overline{a}}$ with respect to a distributional difference $D_f(\cdot,\cdot)$ over all treatment combinations, $\overline{a}$. When the distance measure is the KL-divergence, this  equals maximizing the log-likelihood of the conditional distribution when data is sampled from the counterfactual distribution:
$$\hat{\theta} = argmin\_{\theta \in \Theta} \mathbb{E}\_{\overline{a}} [{\rm KL}(f\_{\overline{a}}(\cdot) || f\_\theta(\cdot|\overline{a})) ]  $$
$$= argmin\_{\theta \in \Theta} \mathbb{E}\_{\overline{a}} [\int \log \left(\frac{f\_{\overline{a}}(y)}{f\_\theta(y|\overline{a})}\right)f\_{\overline{a}}(y) dy]$$
$$= argmax\_{\theta \in \Theta} \mathbb{E}\_{\overline{a}} [ \int \log \left(f\_\theta(y|\overline{a})\right)f\_{\overline{a}}(y) dy ] $$
$$= argmax\_{\theta \in \Theta} \mathbb{E}\_{\overline{a}} [\mathbb{E}_{y \sim f\_{\overline{a}}} ~\log f\_\theta(\cdot|\overline{a}) ]$$

We specifically study KL-divergence because it is well formulated for majority of generative models, including CVAE and many others, and has been focused in related work on counterfactual density estimation (Kennedy 2021, Melnychuk, 2023). Replacing KL with (kernel) MMD or Wasserstein-1 distance may lead to technical difficulties, due to the min-max formulation in MMD and Wasserstein distance. In particular, we have the following unified objective function for kernel MMD and Wasserstein-1 distance:
$$
    \hat{\theta} = argmin\_{\theta \in \Theta}~ \mathbb{E}\_{\overline{a}} [\sup\_{\phi \in \Phi} | \mathbb{E}\_{y \sim f\_\theta(\cdot | \overline{a})}[\phi(y)] -\mathbb{E}\_{y \sim f\_{\overline{a}}(\cdot)}[\phi(y)] \| \].
$$
Here $\Phi$ is a class of functions. For kernel MMD, $\Phi$ is class of kernel mappings and for Wasserstein-1 distance, $\Phi$ consists of all $1$-Lipschitz functions. As can be seen, the objective function now involves an inner maximization operation, which often leads to numerical instability. This phenomenon is well recognized in generative adversarial networks' training (Mescheder et al., 2018; Goodfellow et al., 2020). Therefore, we would like to emphasize the computational efficiency brought by Proposition 1 for the KL case. This proposition allows us to bypass the direct estimation of $f_{\overline{a}}$, significantly enhancing computational scalability in high-dimensional cases.

Reviewer comment (Experiments):
Considering your framework, it appears straightforward to expand it to the individual outcome level. What led to the decision to overlook that possibility?

Author response:
We appreciate the reviewer's reference to Individual Treatment Effects (ITE). We would like to clarify that our approach to estimating the counterfactual distribution across a population, rather than individual-focused estimates (ITE), is driven by the practicality of our research's applications, such as in public health policy. Implementing policies on an individual basis can be costly and sometimes unfeasible. By focusing on the population level (or a sub-population), we capture individual differences, which is essential for effective policy-making. In contrast, the variation within individual counterfactuals often stems from observational noise and might need substantial data for precise estimation. Consider, for instance, the COVID-19 data: our objective is to analyze the counterfactual outcome of a state-wide policy (mask mandate) and provide policymakers with a range of possible counterfactual outcomes that reflect variability in county-level confirmed cases. In such contexts, policymakers might be less concerned with the estimation noise at the individual level and more focused on the broader variability across different individuals.

 In the meantime, a related extension of our work might be to infer conditional counterfactual outcomes (related to the conditional average treatment effect, CATE). This corresponds to looking at the distribution of the outcome under a specific subpopulation. Since our covariates are assumed to be time-varying, a common approach is to introduce a set of static baseline covariates, $V$ (Robins et al, 1999). The baseline covariate $V$ denotes the static feature (such as a patient's gender or age) that will influence both the time-varying covariates $X$ and the outcome $Y$. We can then draw counterfactual samples from a specific sub-population by conditioning on the values of the $V$. In this framework, we observe ($Y_t^i$, $\overline{A}_t^i$, $\overline{X}_t^i$, $V^i$), where $V \in \mathbb{R}^{
    \nu}$ is a static baseline variable that varies by individual.  Accordingly, the generator will have an additional input: 
$ g\_{\theta}(z, \overline{a},v): \mathbb{R}^r \times \mathcal{A}^d \times \mathbb{R}^{\nu} \rightarrow \mathcal{Y}$ , and the IPTW weights will become $ w\_\phi(\overline{a},\overline{x},v) = \frac{1}{\prod\_{\tau=t-d+1}^{t} f\_\phi(a\_{\tau}|\overline{a}\_{\tau-1},\overline{x}\_{\tau},v)}$.   Correspondingly, the objective in Proposition 1 will become $\mathbb{E}\_{v} ~\mathbb{E}\_{\overline{a}} ~\left[\mathbb{E}\_{y \sim f\_{\overline{a},v}} ~\log f\_\theta(y|\overline{a},v)\right] \approx \frac{1}{N}\sum\_{(y,\overline{a},\overline{x},v) \in \mathcal{D}}w\_\phi(\overline{a},\overline{x},v) \log f\_\theta(y|\overline{a},v).$ 

 We have conducted additional experiments using the fully synthetic dataset, where the baseline variable, $V$, was divided into two groups. The $V$ was uniformly drawn from $[-1,0]$ in the first group and from $[0,1]$ in the second group. We then looked at the performance of the generative samples corresponding to two sub-groups. We included the results in **Appendix I** (Table 4, Fig.11, and Fig.12) of the revised manuscript.  Our method (MSCVAE) still consistently outperforms other baseline methods in the simplified results below (Wasserstein distance, smaller the better):

- $V\in[-1,0]$

|         | d=1  | d=3  |  d=5  |
| ----------- | ----------- | ----------- |----------- |
| MSM+NN      |   0.408    |  0.449   |  0.368 |
| KDE               |      0.201 |  0.562 |  0.564  |
| Plugin+KDE   |        0.117    |      0.134      |   0.196      |
| G-Net             |        0.431        |      0.823       |   0.843    |
| CVAE             |       0.240          |     0.571       |   0.585   |
| MSCVAE        |        0.051         |      0.083      |    0.186   |

- $V\in[0,1]$

|         | d=1  | d=3  |  d=5  |
| ----------- | ----------- | ----------- |----------- |
| MSM+NN      |   0.407    |  0.466  |  0.388 |
| KDE               |      0.222 |  0.548 |  0.561  |
| Plugin+KDE   |        0.121   |      0.109      |   0.182      |
| G-Net             |        0.452       |      0.739       |   0.768    |
| CVAE             |       0.258          |     0.565      |   0.591   |
| MSCVAE        |        0.047         |      0.068      |    0.172   |

Reviewer comment (Theory):
Proposition 1 doesn’t make sense. How could $\bar{a}$, a fixed value, be drawn from $\mathcal{D}$? Why is it a sum instead of an average? Wouldn’t the RHS of (4) be the same for all $\bar{a}$ while the LHS is supposed to be different? And why is the index in (5) from $t-d$ instead of from $t-d+1$? In proof of proposition 1, where does the expectation over $\bar{a}$ come from? I would be concerned if the authors actually used this formula in their experiments.

Author response:
> How could $\overline{a}$, a fixed value, be drawn from $\mathcal{D}$? Wouldn’t the RHS of (4) be the same for all $\overline{a}$
 while the LHS is supposed to be different?

We thank the reviewer for pointing it out.  Our generator approximates the underlying counterfactual for any $\overline{a}$,, so there should be an expectation over $\overline{a}$, for the objective function (Equation 2):
    $$\hat{\theta} 
    = argmin_{\theta \in \Theta}~\mathbb{E}_{\overline{a}} ~\left[D_f(f\_{\overline{a}}, f\_\theta(\cdot|\overline{a})) \right],$$

where $D$ represents the distributional distance between $f\_{\overline{a}}$ (true counterfactual distribution) and $f\_\theta$ (the proxy conditional distribution represented by our proposed counterfactual generator $g\_\theta$), such as KL divergence. The resulting weighted loss function in Proposition 1 becomes:

$$
\mathbb{E}\_{\overline{a}} [\mathbb{E}\_{y \sim f\_{\overline{a}}} ~\log f\_\theta(y|\overline{a})] \approx
        \frac{1}{N}\sum\_{(y,\overline{a},\overline{x}) \in \mathcal{D}}w\_\phi(\overline{a},\overline{x}) \log f\_\theta(y|\overline{a})
$$

> Why is it a sum instead of an average? 

 We note that, in Proposition 1, we used the sum for simplicity, which is numerically equivalent to the average in practice as we fixed the batch size in our training.

> And why is the index in (5) from $t-d$ instead of from $t-d-1$? 

  The index in (5) should be from $t-d+1$ instead of $t-d$, which is a typo.  We have fixed in the revised paper.

>  Where does the expectation over $\overline{a}$ come from? 

The$\overline{a}$in the proof for Proposition 1 is due to that we were taking an expectation over $\overline{a}$  (as noted above). We have modified our notations in the revised manuscript.

> Proposition 1 now looks a lot better with all those typos fixed. Nevertheless, proof of proposition 1 still seems to be wrong...

We do acknowledge the possible confusion throughout the proof. Therefore, we have thoroughly updated the proof of Proposition 1 (Appendix D, page 16 of the latest version of the manuscript).  We generally followed the definition and derivation in the slides and chapters of (Robins and Hernán, 2009; Robins, 2018). For clarity, we have also added precise definitions of the notations before the proof and also in-line comments hopefully to make the proof easier to read.

   We would also like to clarify that in the original proof of Proposition 1, the inner integration over $\overline{A}$ is due to the definition of the counterfactual density, which involves integrating the IPW-weighted density of the observation over $\overline{A}$  and $\overline{X}$ and only picking $\overline{A}=\overline{a}$. Therefore this inner integration is not evaluated over different $\overline{a}$. By presenting the proof in a more succinct and clear format in the updated manuscript, we hope such confusion is addressed.

Reviewer comment (Theory):
I don’t seem to understand the comment in Remark 1 that doubly robust methods are less robust to model misspecification than IPW methods, and I cannot find relevant discussions in Appendix D as claimed. I'm curious about why the authors would think so.

Author response:
> I don’t seem to understand the comment in Remark 1 ...

We would like to clarify that we did not suggest that the DR method lacks robustness under model misspecification than IPTW methods. In Remark 1, we intended to convey that it was challenging to extend our IPTW framework to a DR method. This is because in doing so, we need to augment the IPTW estimator with an outcome model for approximating the counterfactual distribution.  Such an extension would require accurately approximating the conditional outcome distribution $f(Y_t|\overline{X}_t,\overline{A}_t)$ and the conditional covariate distribution $f(X_t|\overline{X}\_{t-1},\overline{A}\_{t-1})$. This becomes particularly challenging when $Y$ is high-dimensional, a difficulty that is further exacerbated in the time-varying case with the need to accurately approximate $f(X_t|\overline{X}\_{t-1},\overline{A}\_{t-1})$. The G-Net is the only outcome-based method known to us that addresses such problem, and we have therefore included it as a baseline in our study. Our analysis revealed that G-Net's performance is suboptimal due to the aforementioned challenges, leading us to opt for the IPTW-only framework. We have clarified these points in the Appendix E of the revised paper.

Furthermore, we would like to emphasize our main contribution is to use a generative model to draw high-quality counterfactual samples that follow the underlying counterfactual distribution. We opted for IPTW due to its seamless integration with our generative framework (the weighted loss in Proposition 1) and suitability for the high-d time-varying density approximation (it's easier to estimate the binary propensity model).

>  I never said the authors didn't attempt to estimate the distribution. Although the authors aim to estimate the distribution, they only ... 

We thank the reviewer for the clarification. We agree that in theory, we can extend our IPW framework to the DR setting, and the main reason why we did not opt for DR is due to the **practical** challenges in estimating the outcome model under the high-d outcome settings (as explained in our previous responses). We have further updated the manuscript (Remark 1 and Appendix E) to make it clear that our framework can be potentially extended to the DR framework, and stated the practical reasons why we did not.

Reviewer comment (Theory):
One pivotal assumption is that d, the length of history dependence, is finite and known, in which case the IPW methods in a temporal experiment are only a trivial extension to IPW methods in a static experiment. Also, as d gets large, the variance of those IPW-style methods can easily blow up.

Author response:
> One pivotal assumption is that d, the length of history dependence, is finite and known...


   In line with established practices in longitudinal causal inference, as referenced in works like (Robins et al, 1994; Lim et al, 2018; Bica et al, 2020; Li et al, 2021; Bica et al 2021), we selected $d$ based on our pre-existing understanding of the data generation process. This approach is validated by our numerical findings, proving to be an effective method. For instance, in our COVID-19 studies, we set $d=3$ following insights from epidemiological research, which indicates that a mask mandate typically requires two to three weeks to become effective in a population.

> Also, as d gets large, the variance of those IPW-style methods can easily blow up.

We agree that IPTW can be unstable and might lead to less reliable results in certain contexts.  Nevertheless, it's important to highlight that integrating IPTW with our novel generative framework represents a significant and novel methodological advancement. This combination is particularly effective in capturing complex, high-dimensional patterns in counterfactual outcomes, an area our study uniquely addresses to the best of our knowledge. To address the stability concerns in a practical context, our experiments incorporated established stabilization strategies, such as quantile truncation and standardization, as extensively discussed in the literature (Xiao et al, 2010; Chesnaye et al, 2022). The results from our experiments also show that incorporating these techniques into the design of our marginal structure model greatly results in promising and dependable numerical outcomes.

Reviewer comment (Theory):
The only theoretical guarantee provided in the paper is that the weighted log likelihood is unbiased. In that sense, this is closer to treatment effect estimations where the estimand of interest is a single value, and it is very different from density estimations. Kennedy et al. (2023) (and some of the other papers mentioned by the authors) require stronger conditions simply because their goals are a lot harder. Since this paper is purely applied, I don’t see how they are comparable.

Author response:
> The only theoretical guarantee provided in the paper is that the weighted log likelihood is unbiased... 

We would like to clarify a potential misunderstanding: our technique should be seen as an implicit counterfactual distribution estimator. This is in contrast to standard density estimation methods and extends beyond merely estimating the mean, a subject thoroughly explored in existing research. A key strength of our generative modeling approach is its efficiency in sampling from the estimated counterfactual distribution. The generated samples enable a comprehensive understanding of the time-varying treatment effect and improve decision-making support for policymakers. Specifically, our method involves feeding random noise $z$ into our conditional generator to draw realistic random samples, as detailed in Algorithm 1. With high-dimensional outcome, our approach is far more feasible than what would be required with a traditional density estimation method, where estimating and sampling the density can become computationally prohibitive.
To elucidate how our method estimates the counterfactual distribution, we highlight the use of a conditional generator, $ g_\theta(z, \overline{a})$, to approximate this distribution. This approach is further detailed in Equations 2 of our revised manuscript, where the primary goal is to minimize distributional discrepancies between $f\_\theta(\cdot| \overline{a})$, the proxy conditional distribution  (approximated by $ g\_\theta(z,  \overline{a})$), and $f_{ \overline{a}}$, the actual counterfactual distribution: 
$$\hat{\theta} 
    = argmin_{\theta \in \Theta}~\mathbb{E}_{\overline{a}} ~\left[D_f(f\_{\overline{a}}, f\_\theta(\cdot|\overline{a})) \right].$$

While similar objectives are explored in the works of Kennedy et al. (2021, 2023) and Melnychuk (2023), our method diverges in that we employ a conditional generator rather than a density estimator. In fact, this generative framework has been widely applied to approximate high dimensional distributions in fields such as computer vision. For example, generative models such as the Variational Auto-Encoder (VAE) and Generative Adversarial Networks (GAN) are capable of producing a variety of images that reflect the underlying distribution of the training dataset, without explicitly estimating the density of the high-dimensional image distribution.

Reviewer comment (Theory):
Related to the last point, I fail to see how the absence of a unified theory for doubly robust density approximation in longitudinal settings serves as a reason for not using DR estimators, given that the authors never estimated the density. When the goal is not density estimation, there have been a plethora of studies on variations of IPW and AIPW estimators in longitudinal settings, especially from the dynamic treatment regimes and reinforcement learning literature.

Author response:
> Related to the last point... when the goal is not density estimation...

We agree with the reviewer that if our goal was to estimate a single value such as the mean, there have been a plethora of studies in longitudinal settings. However, as stated in the previous point, our method indeed requires the generated samples to match the counterfactual distribution by using the generative model to approximate such a distribution. Therefore, the absence of a unified theory for doubly robust density approximation in longitudinal settings serves as a major reason for not using DR estimators. As previously mentioned, we explored the outcome-based method, G-Net, which can be potentially combined with IPTW for a DR framework. However, we encountered significant challenges in effectively combining it with our IPTW method due to the sub-optimal empirical performance of G-Net, a result of the methodological challenges discussed above. We remain open and enthusiastic about the possibility of adopting the DR approach, especially if future insights emerge regarding the development of an outcome-based method capable of accurately approximating high-dimensional, time-varying counterfactual distributions.

Reviewer comment (Writing):
I find the notations occasionally confusing, and the authors are somewhat vague regarding the assumptions.

Author response:
> Notations occasionally confusing... vague regarding the assumptions
 
We thank the reviewer for pointing this out. We would like to mention that the assumptions of our methodology are detailed in Appendix A. Furthermore, we have updated the manuscript to enhance the clarity of the notations. If there are still any notations that the reviewer finds unclear, we would highly value specific suggestions for further improvement.