Paper: Efficient Identification of Direct Causal Parents via Invariance and Minimum Error Testing (ICLR-2024)

Reviewer comment (Theory):
My main concern is how to obtain the minimized MSE in practical application, which is a key issue to the proposed theoretical results and it should be properly proofed if possible.

Author response:
We acknowledge that defining an effective strategy to obtain the MSE is important to ensure the proposed methods return valid answers. One could employ more sophisticated methods such as cross-validation or doubly-robust machine learning to obtain more accurate MSE estimates. For example, CORTH (Soleymani et al. 2022) leverages doubly-robust machine learning to devise a more accurate test for causal parents. However, CORTH is also a more computationally costly procedure which makes applying ICP_CORTH to the gene expression problem not tractable. The MSE estimation scheme that we adopted seems to give useful results in practice and further exploration of the MSE estimate is left for future work.

- Soleymani et al. "Causal Feature Selection via Orthogonal Search." 2022

Reviewer comment (Theory):
I note that the simulation data is only generated by a linear model. It should be clearly defined if the proposed theoretical results only focus on the linear model.

Author response:
We are sorry that this is not made clear in the paper. The proposed theoretical results are not dependent on linear data generation. For problems where data generation is non-linear, a non-linear invariance test can be swapped out for the linear invariance test used in the experiment. For example, see Heinze-Demel et al. 2018.

- Heinze-Deml et al. "Invariant causal prediction for nonlinear models." 2018

Reviewer comment (Theory):
If the non-parents node has no noise term, e.g., X_2=2E and X_1=X_2, it seems that the MMSE(X_1 X_2) is equal to (X_1), which may not ensure the correct ICP set output by the proposed method. In other words, the author should claim the causal model used in this paper, such as SCM or ANM. If my understanding is not correct, can you clarify what generation process assumptions you are making?

Author response:
We thank the reviewer for pointing out this case.
In the noiseless case, it is true that MMSE(X_1, X_2) == MMSE (X_1). MMSE-ICP will not pick {X_1, X_2} over {X_1}. This is because MMSE-ICP will first test for {X_1} then test for {X_1, X_2} (line 3, Algorithm 1). When MMSE-ICP finds that {X_1} is invariant, it will skip over {X_1, X_2} (line 4, Algorithm 1).

However, it may be possible that MMSE(X_1) == MMSE(X_2) if X_1 is a deterministic function of X_2. Thus, our approach may output {X_2} instead of {X_1}. Thus, it is true that our approach depends on the assumption that there is noise in the system. However, it does not assume that the noise is additive (ANM) or the generation follows some specific structural causal model (SCM).

Reviewer comment (Reproducibility):
My question is how to test the constraint "isInvariant(Y,S)" (refer to the algorithm). Can you give some illustrations to its implementations?

Author response:
The 'IsInvariance' operation is the same as used in prior work (Peters et al., 2016) and (Mogensen et al., 2022). Specifically, it tests whether the means and variances of the average prediction errors (residuals) across all environments are the same. The pseudo code is given below.

```
p_vals = []
for E_label in groups:
		in_group, out_group = residuals[E == E_label], residuals[E != E_label]
		p_val1 = t_test(in_group, out_group) // returns p-value 
		p_val2 = levene_test(in_group, out_group) // returns p-value based on (Levene 1960)
		p_vals.append(2*min(p_val1, p_val2))
return min(p_vals) < threshold
```

- Levene. "Robust tests for equality of variances". 1960

Reviewer comment (Reproducibility):
The specific procedures for the 'IsInvariance' operation (located at line 1 in Algorithm 1) are not provided, although it appears to involve a test for conditional independence.

Author response:
The 'IsInvariance' operation is the same as used in prior work (Peters et al., 2016) and (Mogensen et al., 2022). Specifically, it tests whether the means and variances of the average prediction errors (residuals) across all environments are the same.

- Peters et al. "Causal inference by using invariant prediction: identification and confidence intervals." 2016
- Mogensen et al. "Invariant Ancestry Search." 2022

Reviewer comment (Experiments):
The running time should be provided on synthetic dataset.

Author response:
We recorded the time each method took when the number of nodes and the number of interventions are equal to 6 (this corresponds to the scenario in Fig 2). This scenario is chosen because all the methods can complete exhaustive search for a problem of this size, thus allowing us to conduct a numerical comparison between all methods. For problems with 100 variables, ICP and IAS will not be able to search through all subsets.

The table below reports the numbers of seconds elapsed on average when executing on an Intel Xeon 6126 CPU core (@ 2.60GHz).
N denotes the sample size.
The numbers in brackets are the standard deviation across different simulations.

Since the official implementation of FGES_MB, ICP, and ICP_CORTH are not written in Python so direct comparison with fastICP should be viewed with caution.
However, we can see that:

1. Among ICP-like methods with exponential worst-case complexity, fastICP is faster than MMSE-ICP, ICP, ICP_CORTH, and IAS.
2. fastICP is also slightly faster than UT-IGSP for large sample size (1E5 samples).

| Method      | Language  |     N=1E2     |     N=1E3    |     N=1E4   |      N=1E5  |
|-------------|-----------|---------------|---------------|---------------|----------------|
| FGES_MB     | Java      | 2.342 (0.028) | 0.246 (0.004) | 0.364 (0.006) |  1.340 (0.019) |
| UT-IGSP     | Python    | 0.026 (0.002) | 0.040 (0.008) | 0.059 (0.017) |  0.607 (0.223) |
| ICP         | R         | 0.190 (0.006) | 0.325 (0.004) | 1.436 (0.026) | 20.296 (0.421) |
| ICP_CORTH   | R         | 1.662 (0.104) | 1.857 (0.122) | 3.759 (0.543) | 28.312 (7.061) |
| IAS         | Python    | 0.016 (0.007) | 0.056 (0.010) | 0.112 (0.022) |  0.659 (0.132) |
| MMSE-ICP    | Python    | 0.035 (0.010) | 0.067 (0.015) | 0.138 (0.027) |  0.932 (0.187) |
| fastICP     | Python    | 0.028 (0.002) | 0.036 (0.002) | 0.070 (0.004) |  0.516 (0.030) |

This table has been added to Appendix C.

Reviewer comment (Evaluation):
Testing invariance may not be easy for general problems.

Author response:
It is true that leveraging invariance testing to build generalizable ML models in general problems has not been widely explored. However, this is partly due to the limitations of ICP since ICP (1) cannot tackle problems with many variables and (2) does not give informative outputs when there are insufficient interventions. These limitations have been explored in (Arjovsky et al. 2019); (Rosenfeld et al. 2021); and (Mogensen et al., 2022). The proposed fastICP addresses both of these limitations and therefore promises to expand the applicability of invariance testing. 

In addition, our proposed approach does not require knowing which variables are intervened upon (unknown intervention targets) so it can be easily applied to problems where the data come from multiple domains/sites.

- Arjovsky et al. "Invariant risk minimization." 2019
- Rosenfeld et al. "The Risks of Invariant Risk Minimization." 2021
- Mogensen et al. "Invariant Ancestry Search." 2022

Reviewer comment (Writing):
It is claimed that the number of interventions required can be much less than the number of variables in the graph, which improves ICP. In my understanding, this is because both observed information (utilized by traditional causal discovery method like GES and PC) and interventional information are used. Is this correct? Please highlight and discuss this point more in the paper.

Author response:
ICP, MMSE-ICP, and fastICP all use observational and interventional data.
However, our methods differ from ICP in the way they pick out the set of causal parents from multiple detected invariant sets.

- ICP takes the intersection of the invariant sets. When there are insufficient interventions, there can be multiple non-overlapping invariant sets so taking the intersection results in a non-informative output.
- Our methods select the best set from the given invariant sets by leveraging the error inequality (Theorem 1). As our experiments demonstrate, this usually yields more informative outputs than ICP.

Reviewer comment (Evaluation):
In the experiment, fig. 5, ICP and the proposed methods have close F1, but ICP's recall is much lower. Does ICP have a higher precision in this case?

Author response:
For the result in Fig. 5, ICP only has higher precision when the number of samples is 1E5.