Paper: Discovering Mixtures of Structural Causal Models from Time Series Data (ICLR-2024)

Reviewer comment (Writing):
Certain notations and explanations remain unclear, and these will be described in the upcoming Questions section.

Author response:
We presented equation 1 for preliminaries, however we assume additive noise in our work due to its structural identifiability. We have clarified this in the updated draft, in Section 3.

Reviewer comment (Experiments):
Some details concerning the comparison between the proposed method and baseline algorithms in the experiments are perplexing, and these questions will also be raised in the later section.

Author response:
Each sample is a time-series of length $T$, while PCMCI+ performs conditional independence testing over time-windows of length $L$, where $T > L$. Thus, CI tests are conducted on a set of time windows per time series.

Reviewer comment (Experiments):
The baselines used in the paper are not algorithms designed for multiple DAGs and mixture SCMs. Even though the authors mentioned other algorithms designed for multiple DAGs, none has been applied in the comparison.

Author response:
We are not aware of any baselines dealing with time-series data that are designed for multiple DAGs. All the algorithms mentioned in the Related Work section are designed for independent data, and hence not directly applicable to our setting. We would appreciate any concrete pointers to comparable algorithms.

Reviewer comment (Theory):
The SCM described in equation (1) differs from equation (6) and also the equation below equation (6); which one is correct? Do you assume additive noise?

Author response:
We presented equation 1 for preliminaries, however we assume additive noise in our work due to its structural identifiability. We have clarified this in the updated draft, in Section 3.

Reviewer comment (Writing):
In Theorem 1 equation (*), what is the definition of $a_i$? Does it mean one sample from $i$th SCM?

Author response:
$a_i$ refers to any point in the support of the mixture distribution such that the condition (*) is satisfied (as specified in mathematical notation in the theorem statement). It is not a ‘sample’ from the ith SCM in the sense that it is not randomly sampled from the SCM, but it can be potentially explicitly chosen for the condition to hold. We have added this clarification to the draft in Appendix A.3.

Reviewer comment (Writing):
In Theorem 1, what are $g_1$ and $g_2$? Should they be $h_1$ and $h_2$?

Author response:
Thank you for pointing out the typo. Yes, they should be $h_1$ and $h_2$. We have corrected this in the new draft in the theorem statement.

Reviewer comment (Theory):
As mentioned in the main paper, "This highlights the idea that learning a mixture model is only beneficial when the underlying SCMs differ from one another significantly." Could you clarify what the significant difference is here?

Author response:
By “significant difference”, we mean that the causal graphs in the mixture distribution differ from one another significantly in terms of their edges and resulting data likelihood.  This is also exemplified by the following table of pair-wise SHDs between the constituent graphs of the mixture models.

| Dataset         | D  | K* | Mean SHD | Std Dev SHD | Min SHD | Max SHD |
|-----------------|----|----|----------|-------------|---------|---------|
| Netsim          |  5 | 14 |     2.59 |        1.17 |       1 |       5 |
| Netsim-permuted | 15 |  3 |    34.00 |        1.63 |      32 |      36 |

We note a notably higher clustering accuracy and improved performance in causal discovery when evaluating the Netsim-permuted dataset. This is in contrast to the Netsim dataset, where the pairwise Structural Hamming Distance (SHD) is smaller, indicating that the mixture graphs in this dataset are more similar to each other.

Reviewer comment (Experiments):
As far as I know, the Rhino algorithm is not designed for heterogeneous time series data. In the experimental section, would it be feasible to compare the proposed algorithm with "Rhino (grouped)" as well? This would involve applying the Rhino algorithm to the grouped data based on the true underlying causal graph. Considering this, the results regarding Rhino in Figures 3 and 6 could be more comprehensive if grouped data were utilized.

Author response:
You are correct that Rhino is not designed for heterogeneous time series data. Based on your suggestion, we have added Rhino (grouped) results on the synthetic datasets in Figures 3, 6 and 7. We observe that Rhino (grouped) on average performs at the same level or worse than MCD on the synthetic datasets (especially for higher number of graphs). This is due to the reduced number of samples per graph, which MCD is more robust to due to weight sharing (as implemented in Equation 10).

Reviewer comment (Evaluation):
Could the post-processing of PCMCI$^{+}$ outputs potentially affect the AUROC/F1 scores of PCMCI$^{+}$? If it does, would this influence tend to favor the proposed method in the comparison results?

Author response:
Post-processing of PCMCI+ outputs actually favors PCMCI+ instead of the proposed method. We chose this for a more fair comparison.  PCMCI+ can only output a Markov Equivalence Class (MEC) for the instantaneous matrix, and hence the orientation of many edges are undetermined. 

We also compared this post-processing scheme to the one presented in [1], where upto 3000 possible DAGs are enumerated from the MEC. The results on the synthetic datasets with $D=5, 10$ are presented below:

Orientation F1 scores with enumeration: 
| **D** | **K** | **PCMCI+ (per sample)** | **PCMCI+ (grouped)** | **PCMCI+ (single graph)** |
|:-----:|:-----:|:-----------------------:|:--------------------:|:-------------------------:|
|   5   |   1   |          0.094          |         0.454        |           0.454           |
|   5   |   5   |          0.095          |         0.494        |           0.242           |
|   5   |   10  |          0.082          |         0.521        |           0.146           |
|   5   |   20  |          0.086          |         0.383        |           0.182           |
|   10  |   1   |          0.065          |         0.621        |           0.621           |
|   10  |   5   |          0.074          |         0.583        |           0.176           |
|   10  |   10  |          0.083          |         0.407        |           0.096           |
|   10  |   20  |          0.063          |         0.336        |           0.108           |

Orientation F1 scores with our post-processing scheme:
| **D** | **K** | **PCMCI+ (per sample)** | **PCMCI+ (grouped)** | **PCMCI+ (single graph)** |
|:-----:|:-----:|:-----------------------:|:--------------------:|:-------------------------:|
|   5   |   1   |          0.098          |         0.600        |           0.600           |
|   5   |   5   |          0.100          |         0.534        |           0.310           |
|   5   |   10  |          0.086          |         0.574        |           0.266           |
|   5   |   20  |          0.102          |         0.477        |           0.210           |
|   10  |   1   |          0.070          |         0.685        |           0.685           |
|   10  |   5   |          0.092          |         0.638        |           0.205           |
|   10  |   10  |          0.093          |         0.484        |           0.151           |
|   10  |   20  |          0.073          |         0.407        |           0.104           |

We observe that PCMCI+ seems to evaluate better with our post-processing scheme compared to the one presented in [1].

Reviewer comment (Experiments):
In the synthetic data experiment, the metrics are averaged across 3 runs. Does each run require a newly generated synthetic dataset? Personally, 3 runs seem to be a limited number for comprehensive evaluation.

Author response:
Based on your suggestion, we have re-run all experiments on the synthetic dataset from the main paper for 5 runs instead of 3, and updated Figure 3 in the draft to reflect this change. We do not observe significant changes when we use 5 runs vs 3. We run 5 runs on the same dataset, i.e. we do not use a newly generated synthetic dataset for each run.

Reviewer comment (Evaluation):
In Figure 3, it seems a little strange that the performance of PCMCI$^{+}$ (grouped) is better with $D=10$ than with $D=5$. The same phenomenon happened in other algorithms. Do you have any clue on why this happened?

Author response:
Since the synthetic datasets are randomly generated, some settings (e.g. D=10) could be inherently more difficult than the others (e.g. D=5) simply due to the stochasticity of the generation process. Our hypothesis seems to be supported by the observation that all methods have similar trends across the two datasets.

Reviewer comment (Evaluation):
Could you explain more about "The clustering accuracy and performance metrics show high standard deviation when K is set to the true number of mixture components K* = 10." which is stated in the appendix and explain more about the "buffers" when $K>K^{*}$?

Author response:
We have added a new section Appendix B.3 with additional experiments for the $D=10, K*=10$ synthetic dataset with $K=10$ and $K=20$ to illustrate our assertion. We plot both the number of unique graphs and the clustering accuracy as a function of number of training steps. We observe that when $K=20$, as training progresses, the algorithm groups together points from different causal graphs until they more or less converge to the “true” number of causal graphs and clustering accuracy converges to (approximately) 100%; however, when $K=10$, we observe that for some runs, the number of unique graphs can sometimes fall below $K^\ast=10$, and thus the clustering accuracy can suffer.

Reviewer comment (Theory):
Why is maximizing ELBO equivalent to maximizing or minimizing each term separately in the log-likelihood in terms of $\theta, \phi$, and $\psi$ as $\phi$ and $\psi$ appear together in the second term?

Author response:
We did not mean that each term in the log-likelihood is maximized or minimized separately; indeed we mean the joint optimization of the ELBO in terms of $\theta, \phi$ and $\psi$ is equivalent to optimizing the log-likelihood. We have updated Section 4 in the draft to clarify this.

Reviewer comment (Writing):
Is it better to mention the Rhino algorithm in the related work section and briefly explain how it works as the proposed algorithm is built on top of Rhino?

Author response:
Thank you for the suggestion! we have already included a self-sufficient description of the Rhino algorithm in Appendix C due to page limit.

Reviewer comment (Theory):
Is it essential to verify whether the data satisfies condition (*) to ensure the reliability of the results? If so, how to verify this?

Author response:
Yes, it is essential to ensure the identifiability of the underlying mixture model in order to ensure that the problem of causal discovery is a statistically feasible problem. Condition (\*) gives one sufficient condition to verify this. While condition (\*) can be difficult to verify exactly in most scenarios since the log-likelihood is rarely explicitly known, in practice one can “approximately” check it given good enough approximations to the log-likelihood functions of the underlying causal models, by explicitly checking if any of the samples satisfy the condition.

Reviewer comment (Reproducibility):
Has any code been provided?

Author response:
We will release our code after the peer review process.

Reviewer comment (Experiments):
If we turn to the experimental section of the paper, we get two examples: one for NetSim (which I’m very familiar with) and another for the DREAM3 gene network. Both of these have problems with respect to the goal of this paper, suggesting that the choice of experimental datasets could be improved.

Author response:
We found it quite challenging to find large-scale real-world experimental datasets for time series causal discovery. If you  have any suggestions or pointers for datasets, we would be glad to run experiments on them and incorporate them into our draft.

Reviewer comment (Experiments):
The problem with the NetSim data is that it’s not an extremely convincing time series, as the records in the simulation are spaced far enough apart in time to render the data nearly i.i.d. In fact, analyzing it as i.i.d. often yields better results than analyzing it as time series, frustratingly, as in this paper:  

Multi-subject search correctly identifies causal connections and most causal directions in the DCM models of the Smith et al. simulation study. NeuroImage, 58(3), 838-848.  

This paper also treats the distributions as a mixture, though doesn't assume non-i.i.d.

Author response:
We respectfully disagree. We note that several contemporary papers on time series causal discovery use this dataset to benchmark their methods [1, 2, 3]. 
Furthermore, we examined the paper cited by the reviewer and discovered that the paper demonstrated that the IMaGES algorithm performed better than other contemporary i.i.d causal discovery algorithms; however, *they did not make a claim that analyzing it as i.i.d yields better results than analyzing it as time series*. Indeed, they did not compare against any temporal causal discovery baselines.

Reviewer comment (Experiments):
As a result, it seems that any study proposing a time series analysis of this data should do a comparison of this result to one obtained by treating the data as i.i.d. instead, since this is a known phenomenon for this particular dataset. This is an issue because the proposed method is specifically designed to deal with time series.

Author response:
Following your suggestion, we benchmarked well-known i.i.d causal discovery algorithms against simulation 3 data from the NetSim dataset. We were unable to run the IMaGES algorithm from [4] since we could not find a publicly available implementation. We compare the performance against reported numbers from temporal causal discovery methods in literature. The results are summarized below:

| **Method** | **Orientation F1** | **Orientation AUROC** |
|:----------:|:------------------:|:---------------------:|
|     PC     |        0.433       |         0.689         |
|   LiNGAM   |        0.050       |         0.413         |
|     GES    |        0.414       |         0.679         |
|  VARLiNGAM |        0.603       |         0.781         |
|  DYNOTEARS |        0.615       |         0.946         |
|   PCMCI+   |        0.583       |         0.771         |

We observe that the temporal causal discovery algorithms handily outperform the i.i.d causal discovery baselines.

Reviewer comment (Evaluation):
The problem with the DREAM3 examples is that none of the reported results have any lift with AUC; they’re all stuck around 0.5, which is more or less random. The differences between the methods are slight. This result doesn’t give a case where the proposed method is particularly helpful.

Author response:
DREAM3 is a challenging dataset due to the low number of samples (46 samples per graph) in the dataset. Nevertheless, we quite significantly outperform the baselines. More importantly, the clustering accuracy is high (94.83%) as reported in Section 5.3, which means that despite being unable to infer the causal relationships themselves, our method can distinguish among data coming from different causal models despite the limited samples. We believe that this is most definitely a significant outcome.

Reviewer comment (Novelty):
The proposed objective function can be seen as a straightforward extension of the standard variational inference optimisation framework. Therefore, the overall novelty and significance of the work may be somewhat limited in this regard.

Author response:
We politely disagree. The problem of inferring multiple causal graphs is quite challenging and has received scant attention in the literature despite its widespread applicability. Our method makes significant contributions with (1) designing an effective deep latent variable model and deriving an ELBO objective to infer the relevant parameters (2) proving the identifiability conditions and (3) solid empirical results.  Even though the objective function is an instance of variational inference, it is not the novelty that we claim in this paper.  Despite its relative “simplicity”, it achieves competitive empirical results.

Reviewer comment (Evaluation):
A major drawback of the work is that the reported results are based on training data, as the proposed method depends on learnable sample-specific parameters. It raises questions about why an encoder producing a K-way categorical random variable given a sample was not considered by the authors.

Author response:
There seems to be a misunderstanding of the problem. Causal discovery is an “unsupervised” problem, in the sense that the ground truth causal graph(s) are assumed to be unknown. Reporting causal discovery results on training data is a standard practice in the literature. We point out that many similar works on optimization-based causal discovery methods work on similar principles, e.g. [1,2,3] Our proposed method reframes the task to an optimization problem, which we solve by training neural networks. In other words, our inference procedure itself involves training on the input data $X$. We have clarified this in the Problem Setting in Section 3 of the paper.

We learn sample-specific parameters, as the task involves inferring membership information solely for the samples within the input data $X$. There is no need for generalization to unseen points. Utilizing an "encoder producing a K-way categorical random variable" would essentially achieve the same objective of learning sample-specific parameters. Using a K-way encoder is a good suggestion that would work especially well for a large sample size. However, since all our experiments are run in the low-sample regime, sample-specific weights work sufficiently well with low memory footprint.

Reviewer comment (Evaluation):
The lack of reported results on generalisation limits insight into the proposed method's ability to perform beyond the training data.

Author response:
As noted earlier, the causal discovery problem only involves inferring causal graphs for the training data $X$, and hence the notion of “generalization beyond the training data” is not applicable in this context.

Reviewer comment (Experiments):
Although the method claims flexibility in terms of core causal structure learning algorithms, performance is only demonstrated based on one causal discovery method.

Author response:
We focus on one “core” causal structure learning algorithm (i.e. Rhino [1]) in order to thoroughly evaluate its capabilities and performance. While other algorithms could potentially be more suitable for specific applications, we find that MCD with Rhino seems to work reasonably well for the tasks we selected and to demonstrate the utility of our framework.

Reviewer comment (Evaluation):
Can you please clarify how the AUROC metric is computed for the different methods being compared?

Author response:
For Rhino and MCD, which parameterize the causal graphs as Bernoulli distributions over each edge, we use the inferred edge probability matrix as the “score”, and evaluate the AUROC metric between the score matrix and the true adjacency matrix. For DYNOTEARS, we use the absolute value of the output scores and evaluate the AUROC. For PCMCI+ and VARLiNGAM, since they only output adjacency matrices, we directly evaluate the AUROC between the predicted and true adjacency matrices. We have updated Appendix D.1 with this clarification.

Reviewer comment (Experiments):
Regarding the experiment on Netsim-permuted data, wouldn’t one expect the results of the proposed method to be inherently favorable compared to non-mixture alternatives? This is because by permuting the variables, we intentionally manipulate the data to explicitly align with the underlying assumption of a mixture model.

Author response:
We performed random permutation of the nodes to demonstrate that even though methods that predict a single graph can perform reasonably well when the different causal models are similar, they are susceptible to poor performance under very mild “perturbations” i.e. permutation of the nodes.  Through node permutation, we generate a mixture distribution where the constituent causal graphs differ significantly. This process validates our hypothesis that the proposed method (MCD) should be effective in such scenarios.

Reviewer comment (Reproducibility):
What specific hyper-parameter value K was chosen for the DREAM3 Gene Network experiment?

Author response:
We used $K=10$. We have updated the draft to specify this choice.

Reviewer comment (Evaluation):
my main skepticism is due to the fact that all the results are for the training set, which I am surprised (and had a stronger positive impression until that point). Trivial enough: For an unquestionable positive score, I would rather need result on test data, on unseen graphs.

Author response:
As pointed out in the reply to Reviewer vYSG, we reiterate that this is standard practice in causal discovery, where the causal graph is inferred without any supervision. We do not assume access to a training corpus of ground-truth graphs or functional relationships; as such, all results are on “unseen graphs”. We have clarified this in the Problem Setting in Section 3 of the paper.

Reviewer comment (Evaluation):
How different the causal graphs in these mixture models (and also the data i.e., average SHD within the cluster is missing. This is really important to really understand what is going on behaviour of the method. (hence, my question). No surprise to see its effect on highly-imbalanced one.

Author response:
Within the cluster, the average SHD would be 0, since we predict the same graph for each cluster. We plot the average pair-wise SHD (over all pairs of causal graphs) for each dataset here:
| Dataset         | D   | K* | Mean SHD | Std Dev SHD | Min SHD | Max SHD |
|-----------------|-----|----|----------|-------------|---------|---------|
| Synthetic       |   5 |  5 |    24.00 |        1.95 |      20 |      27 |
| Synthetic       |   5 | 10 |    23.24 |        3.37 |      14 |      30 |
| Synthetic       |   5 | 20 |    22.61 |        3.29 |      13 |      31 |
| Synthetic       |  10 |  5 |    53.40 |        3.83 |      48 |      59 |
| Synthetic       |  10 | 10 |    54.09 |        3.26 |      45 |      61 |
| Synthetic       |  10 | 20 |    54.27 |        3.73 |      44 |      64 |
| Synthetic       |  20 |  5 |   113.80 |        5.62 |     101 |     120 |
| Synthetic       |  20 | 10 |   111.91 |        5.42 |      99 |     123 |
| Synthetic       |  20 | 20 |   113.59 |        5.06 |      98 |     124 |
| DREAM3          | 100 |  5 |   517.60 |      202.13 |     234 |     896 |
| Netsim          |   5 | 14 |     2.59 |        1.17 |       1 |       5 |
| Netsim-permuted |  15 |  3 |    34.00 |        1.63 |      32 |      36 |
We have also added this table to the paper in Appendix D.4

Reviewer comment (Experiments):
on minor point, an illustrative toy example is lacking would be very useful.

Author response:
Thank you for the suggestion. We have added an illustrative toy example with two causal graphs in Appendix E to explain the importance of using a mixture model for heterogeneous causal discovery.

Reviewer comment (Experiments):
What is the effect of the distance between causal graphs to the performance?

Author response:
We believe that our model is better able to cluster points when the SHD between causal graphs is larger. This is especially prominently seen with the Netsim and Netsim-permuted datasets.

Reviewer comment (Writing):
g_1 and g_2 are not defined in Theorem 1, is it a typo: should be h? or a particular graph?

Author response:
Yes, it is a typo. They should be $h_1$ and $h_2$. This has been fixed in the new draft.