Paper: A Linear Algebraic Framework for Counterfactual Generation (ICLR-2024)

Reviewer comment (Theory):
My main concern is the identification of parameters in the log-likelihood function, i.e., how to ensure the unique solution of target parameters by the EM algorithm. Because this is a key issue in generating the true counterfactual data.

Author response:
EM algorithm.  When the number of mixture components is $K=1$, we can ensure the unique solution of target parameters under rotational and scaling ambiguity (i.e., $\mathbf{WS} = (\mathbf{WR})(\mathbf{R}^{-1}\mathbf{S}) = \mathbf{W'S'}$) that gives a global maximization of the likelihood function. However, a Gaussian mixture model with $K > 1$ is not free of local maxima issues, and EM algorithms only find local optimal solutions of the likelihood function. Our model also does not give the same unique parameters every time it is trained. Normally, however, as we are not too concerned about non-unique solutions from the conventional use of a Gaussian mixture model for density estimation, the authors do not think it is a concerning issue for the generative task. Empirically, we observe that the probability of getting some worse solutions is very low.

Reviewer comment (Theory):
Assumption 4 assumes that the soft intervertion on "s" is linearity and additive. Some limitations or reasonableness to it also require to be discussed.

Author response:
Linear and additive intervention. This paper emphasizes the restriction on the counterfactual generation that receiving treatment at a time cannot cause any difference reversely to pretreatment observation data. The restriction is obeyed in this paper by assuming such linear and additive intervention. Because the linear and additive intervention is made only in the null space to pretreatment observers (as shown in Assumption 4 and Definition 2.2), the intervention does not cause any effect on pretreatment observation data. If we decided to use nonlinear observers, we would have to assume a nonlinear intervention to keep the restriction. In a bigger frame, we can consider two equally qualified options: one is that we assume a linear observer system and employ a Gaussian mixture model and the other is that we assume a nonlinear observer system and use a single Gaussian model. This paper takes the first option and, in the first option, the assumption of linear and additive intervention is reasonable.

Reviewer comment (Novelty):
The motivation of this paper is unclear. The contribution also seems limited. The restriction on counterfactual generation that receiving treatment at a time $T_0$ cannot cause any difference reversely to counterfactual generation at $t < T_0$, comes across as self-evident. It remains ambiguous whether prior research infringed upon this constraint. The comparative advantages of the introduced method with the existing work are not distinctly highlighted, leading to a gap in understanding its unique value proposition.

Author response:
Thanks for the suggestions on clarifying the assumptions, comparative advantages, and more rigorous theoretical treatment. We provide brief explanations, results, and plans for rigorous treatment.

(1)  In the original synthetic control method and its variant models (e.g., robust synthetic control, SyncTwin, etc.), a synthetic control is defined as a weighted average of the units in the donor pool (control group). The post-treatment values of the weighted average are regarded as the counterfactual prediction of post-treatment outcomes. However, the pre-treatment values of the weighted average are not the same as the observed pretreatment data because the problem of finding the optimal (nonnegative) weights is basically an over-determined linear system of equations or a linear system of equations that has no unique solution due to nonnegativity regularization, and it only finds an approximated solution.

Empirically, Figure 3 of our paper shows that there are gaps between the black line (factual data) and the blue line (SCM prediction) in the pretreatment periods, which suggests the infringement that the pretreatment predictions (blue line) are also affected reversely by the policy in 1988.
Furthermore, Figure 4 of the SyncTwin paper also shows the infringement. The authors say:

*“The individual shown in Figure 4 (top) has a sensible ITE estimate because the synthetic twin matches its pretreatment outcomes closely. In addition to visualization, we can calculate the individualized error bound based on $d_i^y$ (Equation 7).”*

They say the synthetic twin matches its pretreatment outcomes "closely", but our SSA-GMM results “exactly” match any pretreatment outcomes. Figure 2 of the CGP paper (Schulam \& Saria, 2017) also shows the gaps between the observed pretreatment measurements (green points) and the model prediction (blue curves). Even a very small difference in the pretreatment periods can make a big difference in the post-treatment prediction, and the restriction cannot be ignored. Our contribution to the self-evident constraint is clear and not limited.

Reviewer comment (Experiments):
To evaluate how true synthetic counterfactuals are, more experiments on synthetic datasets including ground-truth counterfactual data are required. It would be insightful to demonstrate the method's robustness under diverse treatment times and over prolonged post-treatment durations.

Author response:
(2) Table 1 in our paper is a reproduced one of Table 2 of the SyncTwin paper. Table 1 itself shows our state-of-the-art results across different biases and data sizes. The training/testing dataset used was added by independent white noise $\sim\mathcal{N}(0, 0.1)$. Our results of a mean absolute error are at the level of 0.07, which means the ideal performance that cannot be algorithmically further improved when we notice

      (numpy.abs(numpy.random.randn(10000000) * 0.1)).mean() = 0.07979.

Tables 2 and 3 of our paper also support that our method is robust to data missingness and noises. As requested by the Reviewer, we performed more experiments under diverse treatment times and over prolonged post-treatment durations. Our new tables in **Table A** and **Table B** show the state-of-the-art performance of our method.

Reviewer comment (Theory):
The paper could benefit from a rigorous theoretical exploration addressing the method's resilience against noise and confounding bias. Would this be challenging with such a linear framework?

Author response:
(3) This paper could benefit more from a rigorous theoretical exploration against noise, bias, and data missingness. This is a linear framework that enables it to be analyzed theoretically; we can include theoretical analyses in the Appendix and refer to it in the main body of the paper. In the appendix, we will also provide a literature review on those methods, such as MC-NNM and CGP.

Reviewer comment (Writing):
There is a lack of literature review or an introduction to some of the methods appearing in the experimental part, such as MC-NNM (Athey et al., 2021) and CGP (Schulam and Saria, 2017).

Author response:
(3) This paper could benefit more from a rigorous theoretical exploration against noise, bias, and data missingness. This is a linear framework that enables it to be analyzed theoretically; we can include theoretical analyses in the Appendix and refer to it in the main body of the paper. In the appendix, we will also provide a literature review on those methods, such as MC-NNM and CGP.

Reviewer comment (Experiments):
Regarding Fig.4, what methodology did the authors employ to randomly produce counterfactual pre-treatment data?

Author response:
(4) To produce ground truth counterfactual pretreatment and post-treatment data in Figure 4, we used the same Pharmacology model [(link)](https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1472-8206.2006.00404.x) and codes [(link)](https://github.com/ZhaozhiQIAN/SyncTwin-NeurIPS-2021/blob/main/sim/pkpd.py) that SyncTwin used.

Reviewer comment (Writing):
In some instances, the description is not sufficiently clear. For example, the model's explanation here lacks clarity, particularly in the description of assumption 2, which necessitates further comprehension through reading the instructions in the appendix.

Author response:
We appreciate the suggestion to improve our manuscript to bring out the contribution clearly. The main body of the paper presents a probabilistic model and Appendix D discusses its deterministic model as a matrix factorization problem without a prior distribution of Gaussian mixture. We will organize Appendix D so that readers can understand them clearly. It is page-limited to include Appendix D in the main text, and we will refer to it properly in the main text.

Reviewer comment (Writing):
Can you provide an intuitive explanation for Assumption 2? Typically, people use a linear dynamical system model to describe the generative mechanism. It would be beneficial to include some descriptions in the main text rather than placing them in the appendix.

Author response:
In the real world, objects change with time, and so does the data. Transforming such time-varying data into a static time-invariant representation leads to a problem that is easier to solve, which is the motivation and foundation of our approach. In that light, Assumptions 1 and 2 provide a new perspective in which the objects’ new representation, denoted by $\mathbf{S}$, does not change, and the observer's viewpoint, denoted by $\mathbf{W}$, is changing. Such static representations of $\mathbf{S}$ that are invariant to time can be learned well with simpler models (such as in this paper, GMM). Assumption 2 describes the relationship between time-invariant data $\mathbf{S}$ and time-varying data $\mathbf{X}$. This observation is our main insight, which is then embellished by creating a more realistic model where one can include some additive measurement noise.  We further highlight that the validity of our insight and assumption is established by experimental results that offer a definite improvement over the state of the art.