Paper: CausalTime: Realistically Generated Time-series for Benchmarking of Causal Discovery (ICLR-2024)

Reviewer comment (Experiments):
The focus on neural models in simulation and estimation is limiting. It would be best to show how other models are benchmark.

Author response:
Many thanks for the valuable suggestion. In addition to the PCMCI (which is a PC algorithm modified to work with time series) algorithm in the original manuscript, we will include more non-neural-network-based algorithms, including SVAR, Granger Causality, and NTS-NOTEARS. We have added the results to Section 4.3.

Reviewer comment (Theory):
Could you please clarify the notation for the ACG? Your A is represented as a $2N\times 2N$ matrix. The $2N\times N$ portion seemingly represents the "causal" and residual term mixing in the previous time step. However, the actual causal graph likely has an adjacency matrix of a $2N \tau_{max}\times N$ dimension to model edges from parents up to a lag of $\tau_{max}$. Does this notation imply that in this paper you were only considering Markov order 1 models?

Author response:
Thank you for the question. Our ACG $A$ ($2N\times 2N$) is actually the **causal summary graph**, which represents causal relations between time series without referring to time lags, and is common in TSCD [1], i.e., the causal effects may exist on multiple time lags $\tau, 0<\tau\leq \tau_{\text{max}}$. As a result, we are not only considering Markov order 1 models.

[1] J. Runge, A. Gerhardus, G. Varando, V. Eyring, and G. Camps-Valls, “Causal inference for time series,” Nature Reviews Earth & Environment.

Reviewer comment (Experiments):
I would appreciate if you could use more classical approaches, in addition to purely neural models, for benchmarking time-series causation. I suspect that the experiments may favor neural models over others (as noted in the weaknesses). It would be beneficial to include comparisons with the SVAR model, Granger Causality, the PC algorithm modified to work with time series, and similar models.

Author response:
Many thanks for the valuable suggestion. In addition to the PCMCI (which is a PC algorithm modified to work with time series) algorithm in the original manuscript, we will include more non-neural-network-based algorithms, including SVAR, Granger Causality, and NTS-NOTEARS. We have added the results to Section 4.3.

Reviewer comment (Experiments):
Could you characterize the variability of your ground truth ACGs as a function of training your simulator model starting with different seeds?

Author response:
Thanks for the helpful comments! We added experiments with 5 different seeds and measured the variability of ACGs with average standard deviation, shown in Supp. Section A.3.4. We can observe that the variability is relatively small, i.e., 5 graphs are similar.

Reviewer comment (Presentation):
I would appreciate an explanation of how the ground truth causal graphs in Figure 2 are combined.

Author response:
Fig. 2 (Fig. 3 in the revised manuscript) is the visualization of causal graphs extracted from AQI, Traffic, and Medical. For AQI and Traffic, we overlay the ground truth causal graphs onto the map, since each node represents certain locations, and this figure shows that relationships between two variables are highly relevant to their geometric distances. However, for the Medical dataset, we simply plot the causal graph with the networkx package in python. 

We are not sure if this explains your question. If not, we would be happy to answer more questions!

Reviewer comment (Theory):
this paper is only "causal" in a very weak sense. Indeed, given the "No instantaneous effects" assumption, it is possible to write down a dense causal graph in which all prior variables are parents of $\mathbf{x}_t$. Arguably, there is no true causality here, only the problem of learning a time series with *sparse* relationships, because the DAG constraint is automatically satisfied when instantaneous effects in the original time series are discounted.

Author response:
Thank you for your insightful comment. We would like to clarify our contribution in the following two aspects:

1. **Why include "no instantaneous effects" assumption:** We intend to develop a realistic benchmark to test various TSCD algorithms, some of which support instantaneous effects while a variety do not (e.g. CUTS, NGC, PCMCI). As a result, we choose to include the most common assumptions, e.g., No instantaneous effects, Causal Sufficiency, Causal Stationarity. Including instantaneous effects in the generation process is indeed a good idea, but it's beyond the scope of this work, so we include this direction in future works in Section 6.
2. **Why dense causal graph:** Actually, in CausalTime a dense prediction model **does not imply dense causal graphs** (also see Q6). Moreover, "True Causality" is usually inaccessible and we alternatively present a solution to get around this difficulty by splitting the "remaining part" (which **may or may not** contain causal effects) into the residual term. Mathematically, the sparse ACG is indeed the ground truth causal graph for the generated time-series (not the original input time-series). We have added illustrative Fig. 2 to clarify the misunderstanding.

Reviewer comment (Writing):
this key idea is not explained particularly well in Sec 3.4.

Author response:
Thank you. We have updated Section 3.4 to include more explanation. Specifically, we made the following revisions:

1. Alter the subtitle of Section 3.4 to "Splitting the NAR Model to Acquire Actual Causal Graph and Realistic Time-series".
2. Add explanations of why introduce independent residual terms, along with illustrative Fig. 2.
3. Add a more intuitive explanation for Eqn. 5-7.

Reviewer comment (Evaluation):
it is unknown how realistic the causal relationships used in CausalTime are. Indeed, the authors make no claim of doing causal discovery. Hence, the causal relationships used in CausalTime may be very different to those found in nature, implying that CausalTime datasets are not a good surrogate for causal discovery on real time series. Indeed, when using Shapley values, a correspondence between feature importance and causality is suggested which may be incorrect.

Author response:
Thank you for the question. For many fields, including weather, traffic, or medicine, true causality is extremely hard to acquire (or maybe even impossible). If we try to discover causal relationships with another TSCD algorithm, we are faced with the dilemma of "using a TSCD algorithm to build a synthetic dataset to test TSCD algorithms". Therefore, we explicitly bypass doing causal discovery.

As for the rationality of the used causal relationships, ​

1. In AQI and Traffic datasets, causal relationships are highly relevant to geometry distances, since nearby stations have mutual influences, so the extracted HCG and the subsequent ACG are directly from distance graphs, which align with common sense and are widely used [1][2].
 2. In medical datasets, the HCG is extracted with DeepSHAP since it is hard to build reliable graphs from only prior knowledge. Although Shapley values might not exactly match with causality, Shapley-value-based approaches are widely used in the field of medicine and are shown to capture features with actual important relationships [3][4]. 

Consequently, the extracted graphs in our three subsets are built on existing extensive studies and are expected to be **close to those found in nature**.

[1] "Filling the G_ap_s: multivariate time series imputation by graph neural networks," ICLR 2022.

[2] "Graph WaveNet for Deep Spatial-Temporal Graph Modeling." arXiv, May 31, 2019.

[3] S. L. Hyland et al., "Early prediction of circulatory failure in the intensive care unit using machine learning," Nature Medicine.

[4] H.-C. Thorsen-Meyer et al., "Dynamic and explainable machine learning prediction of mortality in patients in the intensive care unit: a retrospective study of high-frequency data in electronic patient records," The Lancet Digital Health.

Reviewer comment (Experiments):
The methods in the paper suggest that including the residual term is important. This, in turn, implies that "natural" causal graphs may be dense. E.g. all components of $\mathbf{x}_{t-1}$ affect $\mathbf{x}_t$. However, in CausalTime, the causal graph is forcibly sparsened to produce $H$, so as to present a more interesting problem to causal discovery algorithms. In Table 2, the inclusion of the residual term seems to reduce discriminative scores by a factor of at least 10, implying it is very necessary to get good reconstruction. Thus- is the forced sparsification used in CausalTime actually contrary to natural time series?

Author response:
Thanks for the interesting question. Indeed, the residual term implies that all components of $x_{t-1}$ are included to predict $x_{t}$. However, **not every variable in the prediction model necessarily means causal effect**, e.g., zero-value weights in a linear prediction model. The residual terms in CausalTime represent causal parents that are not major parts that **may or may not** affect ${x}_{t}$, so the "dense" prediction models in our pipeline do not imply that "natural" causal graphs are dense. 

Anyway, this is a very important aspect of the CausalTime pipeline, and we deeply thank the reviewer for this insightful question. We have added these discussions to Sec. 3.4 and changed "fully connection graph" to "fully connection prediction model" to avoid confusion.