Paper: Tensor Time-Series Forecasting and Anomaly Detection with Augmented Causality (ICLR-2024)

Reviewer comment (Experiments):
The authors ignore a large number of recent excellent graph convolution or graph attention models for time series prediction. It is worth mentioning that the classical baseline DCRNN is the result of 2018 years of reporting and is almost outperformed by current solutions. Obviously, it is difficult for the authors to complete the comparison of SOTA algorithm in such a short rebuttal period. Therefore, the experimental results of the manuscript cannot fully illustrate the effectiveness of the proposed method.

Author response:
Thanks for the review and paying attention to our forecasting module. Your concerns are listed and addressed below.

* First of all, **all the suggested literature [1-6] is added and discussed in the revised paper and colored blue**. Furthermore, paper [2] is a very new paper accepted by NeurIPS 2023 and has not been released to the public yet. Then, we discussed its content based on the previous version on arXiv.

- Second, we add state-of-the-art baselines since one of our baselines, DCRNN, is 2018. On the one hand, we would like to mention our other baseline GTS is in ICLR 2021, which is newer than the suggested baselines [3] in IJCAI 2018, [4] in AAAI 2020, and [5] in AAAI 2021. On the other hand, in the suggested SOTA algorithm [6] in AAAI 2023, baselines [3-5] are included and compared. **Therefore, we did the new experiments for the SOTA baseline ST-SSL [6].** The new table is shown below and has also been included in the revised paper (colored blue). In general, the SOTA baseline ST-SSL achieves a very competitive forecasting performance through contrastive learning, which is superior to one of our proposed methods TacSas, and slightly lower than our TacSas++, which is newly proposed by not only discovering the fine-grained temporal structures but also getting calibrated by contrasting the nearest time window for persistence forecasting. 

|            | ERA5-2017 $(\downarrow)$ | ERA5-2018 $(\downarrow)$ | ERA5-2019 $(\downarrow)$ | ERA5-2020 $(\downarrow)$ |
|------------|--------------------------|--------------------------|--------------------------|--------------------------|
| GRU        | 1.8834 $\pm$ 0.0126       | 1.9764 $\pm$ 0.1466       | 1.6194 $\pm$ 0.2645       | 1.7859 $\pm$ 0.2324       |
| DCRNN      | 0.0819 $\pm$ 0.0025       | 0.0797 $\pm$ 0.0049       | 0.0799 $\pm$ 0.0035       | 0.0826 $\pm$ 0.0033       |
| GTS        | 0.0777 $\pm$ 0.0054       | 0.0766 $\pm$ 0.0029       | 0.0760 $\pm$ 0.0031       | 0.0742 $\pm$ 0.0021       |
| TacSas     | 0.0496 $\pm$ 0.0017       | 0.0499 $\pm$ 0.0017       | 0.0502 $\pm$ 0.0016       | 0.0488 $\pm$ 0.0019       |
| ST-SSL     | 0.0345 $\pm$ 0.0051      | 0.0330 $\pm$ 0.0018       | 0.0361 $\pm$ 0.0021       | 0.0348 $\pm$ 0.0020       |
| TacSas++    | $\textbf{0.0271}$ $\pm$ $\textbf{0.0004}$ | $\textbf{0.0276}$ $\pm$ $\textbf{0.0004}$ | $\textbf{0.0282}$ $\pm$ $\textbf{0.0003}$ | $\textbf{0.0265}$ $\pm$ $\textbf{0.0004}$ |


ST-SSL relies on the off-the-shelf input structure like what DCRNN does, but its input is usually time-evolving and hardly acquired in most real-world settings (and usually gets replaced by random ones). It is this very problem that motivates our paper. 

* Third, thanks again for your raised concern on our forecasting module. **They are very actionable and improve our paper**. **Please also do not overlook our theoretical and empirical efforts** in terms of **fine-grained temporal-spatial structures discovery**, **its leverage for anomaly detection**, **the end-to-end generative model for realizing them all**, and **the new data source**.

Reviewer comment (Experiments):
From the perspective of graph convolutional networks for Spatio-Temporal representation learning, several baselines with currently excellent performance — such as STGCN (Yu et al., 2018), STSGCN (Song et al., 2020), STFGNN (Li & Zhu, 2021), and ST-SSL (Jiahao Ji et al., 2023) — are similar in nature to meteorological information prediction across different geographical locations but are not compared against in the paper. These works, although focused on traffic flow or smart city studies, are relevant from the perspective of geospatial information mining.

Author response:
Thanks for the review and paying attention to our forecasting module. Your concerns are listed and addressed below.

* First of all, **all the suggested literature [1-6] is added and discussed in the revised paper and colored blue**. Furthermore, paper [2] is a very new paper accepted by NeurIPS 2023 and has not been released to the public yet. Then, we discussed its content based on the previous version on arXiv.

- Second, we add state-of-the-art baselines since one of our baselines, DCRNN, is 2018. On the one hand, we would like to mention our other baseline GTS is in ICLR 2021, which is newer than the suggested baselines [3] in IJCAI 2018, [4] in AAAI 2020, and [5] in AAAI 2021. On the other hand, in the suggested SOTA algorithm [6] in AAAI 2023, baselines [3-5] are included and compared. **Therefore, we did the new experiments for the SOTA baseline ST-SSL [6].** The new table is shown below and has also been included in the revised paper (colored blue). In general, the SOTA baseline ST-SSL achieves a very competitive forecasting performance through contrastive learning, which is superior to one of our proposed methods TacSas, and slightly lower than our TacSas++, which is newly proposed by not only discovering the fine-grained temporal structures but also getting calibrated by contrasting the nearest time window for persistence forecasting. 

|            | ERA5-2017 $(\downarrow)$ | ERA5-2018 $(\downarrow)$ | ERA5-2019 $(\downarrow)$ | ERA5-2020 $(\downarrow)$ |
|------------|--------------------------|--------------------------|--------------------------|--------------------------|
| GRU        | 1.8834 $\pm$ 0.0126       | 1.9764 $\pm$ 0.1466       | 1.6194 $\pm$ 0.2645       | 1.7859 $\pm$ 0.2324       |
| DCRNN      | 0.0819 $\pm$ 0.0025       | 0.0797 $\pm$ 0.0049       | 0.0799 $\pm$ 0.0035       | 0.0826 $\pm$ 0.0033       |
| GTS        | 0.0777 $\pm$ 0.0054       | 0.0766 $\pm$ 0.0029       | 0.0760 $\pm$ 0.0031       | 0.0742 $\pm$ 0.0021       |
| TacSas     | 0.0496 $\pm$ 0.0017       | 0.0499 $\pm$ 0.0017       | 0.0502 $\pm$ 0.0016       | 0.0488 $\pm$ 0.0019       |
| ST-SSL     | 0.0345 $\pm$ 0.0051      | 0.0330 $\pm$ 0.0018       | 0.0361 $\pm$ 0.0021       | 0.0348 $\pm$ 0.0020       |
| TacSas++    | $\textbf{0.0271}$ $\pm$ $\textbf{0.0004}$ | $\textbf{0.0276}$ $\pm$ $\textbf{0.0004}$ | $\textbf{0.0282}$ $\pm$ $\textbf{0.0003}$ | $\textbf{0.0265}$ $\pm$ $\textbf{0.0004}$ |


ST-SSL relies on the off-the-shelf input structure like what DCRNN does, but its input is usually time-evolving and hardly acquired in most real-world settings (and usually gets replaced by random ones). It is this very problem that motivates our paper. 

* Third, thanks again for your raised concern on our forecasting module. **They are very actionable and improve our paper**. **Please also do not overlook our theoretical and empirical efforts** in terms of **fine-grained temporal-spatial structures discovery**, **its leverage for anomaly detection**, **the end-to-end generative model for realizing them all**, and **the new data source**.

Reviewer comment (Novelty):
Literature survey misses many SOTA works in spatio-temporal forecasting, for example: Li, Peiyuan, Yin Yu, Daning Huang, Zhi‐Hua Wang, and Ashish Sharma. "Regional heatwave prediction using Graph Neural Network and weather station data." Geophysical Research Letters 50, no. 7 (2023): e2023GL103405.

Author response:
* First, **all your suggested related work is added and discussed in the updated paper and colored blue**.

Reviewer comment (Novelty):
Literature survey misses many SOTA works in anomaly detection, for example: Lira, Hernan, Luis Martí, and Nayat Sanchez-Pi. "A graph neural network with spatio-temporal attention for multi-sources time series data: An application to frost forecast." Sensors 22, no. 4 (2022): 1486.

Author response:
* First, **all your suggested related work is added and discussed in the updated paper and colored blue**.

Reviewer comment (Novelty):
The author should include more recent literature in the survey.

Author response:
* First, **all your suggested related work is added and discussed in the updated paper and colored blue**.

Reviewer comment (Experiments):
According to my understanding, this approach only detects if some anomaly has a very high value, i.e., it detects only extreme values. However, temporal anomalies (anomalies containing rare temporal patterns) and inter-variable anomalies (anomalies where the inter-variable relationship does not match the common pattern) are of main concern and were not detected by the proposed method. The paper needs to show the performance of the proposed model on various data in terms of temporal, inter-variable, or inter-location anomalies as well. Please check: Zhang, C., Song, D., Chen, Y., Feng, X., Lumezanu, C., Cheng, W., Ni, J., Zong, B., Chen, H. and Chawla, N.V., 2019, July. A deep neural network for unsupervised anomaly detection and diagnosis in multivariate time series data. In Proceedings of the AAAI conference on artificial intelligence (Vol. 33, No. 01, pp. 1409-1416).

Author response:
* Third, "_why did not consider other kinds of anomalies, like inter-variable relationship anomalies_"? Thanks for the question. To the best of our knowledge, facing various types of anomalies, different methods are proposed for different anomaly patterns. On the one hand, in our paper, the proposed model did the value anomaly detection, and the selected baselines and discussed related work are also the methods that only did the value anomaly detection. On the other hand, the structural anomaly detection method you suggested also targets one specific type of anomaly among various anomaly patterns, which is also unable to do value anomaly detection. Each aspect has its own value, like value anomaly detection can help with extreme weather detection, and structural anomaly detection can help with high-tech equipment deployment. **Proposing a universal anomaly detection method for all possible anomaly patterns is very interesting, we would like to explore it as a future direction. But for now, it is beyond the scope of our submission.**

Reviewer comment (Experiments):
Question — Why did the author not compare with SOTA methods in the case of forecasting? For example: Li, Peiyuan, Yin Yu, Daning Huang, Zhi‐Hua Wang, and Ashish Sharma. "Regional heatwave prediction using Graph Neural Network and weather station data." Geophysical Research Letters 50, no. 7 (2023): e2023GL103405.

Author response:
- Second, the reason why we did not compare with your suggested SOTA baseline in the case of forecasting "Li et al., Geophysical Research Letters 50" and "Lira et al, Sensors 22" is that we searched online but **did find the published code** for those methods. But, on the one hand, they are now added to the revised paper and discussed. On the other hand, **we added a new sota baseline** suggested by Reviewer TMqL for forecasting, which is a new paper this year "Ji et al., AAAI 2023. Spatio-Temporal Self-Supervised Learning for Traffic Flow Prediction.". In short, our proposed method can still outperform the baselines as follows. More details are added in the revised paper.

|            | ERA5-2017 $(\downarrow)$ | ERA5-2018 $(\downarrow)$ | ERA5-2019 $(\downarrow)$ | ERA5-2020 $(\downarrow)$ |
|------------|--------------------------|--------------------------|--------------------------|--------------------------|
| GRU        | 1.8834 $\pm$ 0.0126       | 1.9764 $\pm$ 0.1466       | 1.6194 $\pm$ 0.2645       | 1.7859 $\pm$ 0.2324       |
| DCRNN      | 0.0819 $\pm$ 0.0025       | 0.0797 $\pm$ 0.0049       | 0.0799 $\pm$ 0.0035       | 0.0826 $\pm$ 0.0033       |
| GTS        | 0.0777 $\pm$ 0.0054       | 0.0766 $\pm$ 0.0029       | 0.0760 $\pm$ 0.0031       | 0.0742 $\pm$ 0.0021       |
| TacSas     | 0.0496 $\pm$ 0.0017       | 0.0499 $\pm$ 0.0017       | 0.0502 $\pm$ 0.0016       | 0.0488 $\pm$ 0.0019       |
| ST-SSL     | 0.0345 $\pm$ 0.0051      | 0.0330 $\pm$ 0.0018       | 0.0361 $\pm$ 0.0021       | 0.0348 $\pm$ 0.0020       |
| TacSas++    | $\textbf{0.0271}$ $\pm$ $\textbf{0.0004}$ | $\textbf{0.0276}$ $\pm$ $\textbf{0.0004}$ | $\textbf{0.0282}$ $\pm$ $\textbf{0.0003}$ | $\textbf{0.0265}$ $\pm$ $\textbf{0.0004}$ |

Reviewer comment (Experiments):
Question — Why did the author not compare with SOTA methods in the case of anomaly detection? For example: Lira, Hernan, Luis Martí, and Nayat Sanchez-Pi. "A graph neural network with spatio-temporal attention for multi-sources time series data: An application to frost forecast." Sensors 22, no. 4 (2022): 1486.

Author response:
- Second, the reason why we did not compare with your suggested SOTA baseline in the case of forecasting "Li et al., Geophysical Research Letters 50" and "Lira et al, Sensors 22" is that we searched online but **did find the published code** for those methods. But, on the one hand, they are now added to the revised paper and discussed. On the other hand, **we added a new sota baseline** suggested by Reviewer TMqL for forecasting, which is a new paper this year "Ji et al., AAAI 2023. Spatio-Temporal Self-Supervised Learning for Traffic Flow Prediction.". In short, our proposed method can still outperform the baselines as follows. More details are added in the revised paper.

|            | ERA5-2017 $(\downarrow)$ | ERA5-2018 $(\downarrow)$ | ERA5-2019 $(\downarrow)$ | ERA5-2020 $(\downarrow)$ |
|------------|--------------------------|--------------------------|--------------------------|--------------------------|
| GRU        | 1.8834 $\pm$ 0.0126       | 1.9764 $\pm$ 0.1466       | 1.6194 $\pm$ 0.2645       | 1.7859 $\pm$ 0.2324       |
| DCRNN      | 0.0819 $\pm$ 0.0025       | 0.0797 $\pm$ 0.0049       | 0.0799 $\pm$ 0.0035       | 0.0826 $\pm$ 0.0033       |
| GTS        | 0.0777 $\pm$ 0.0054       | 0.0766 $\pm$ 0.0029       | 0.0760 $\pm$ 0.0031       | 0.0742 $\pm$ 0.0021       |
| TacSas     | 0.0496 $\pm$ 0.0017       | 0.0499 $\pm$ 0.0017       | 0.0502 $\pm$ 0.0016       | 0.0488 $\pm$ 0.0019       |
| ST-SSL     | 0.0345 $\pm$ 0.0051      | 0.0330 $\pm$ 0.0018       | 0.0361 $\pm$ 0.0021       | 0.0348 $\pm$ 0.0020       |
| TacSas++    | $\textbf{0.0271}$ $\pm$ $\textbf{0.0004}$ | $\textbf{0.0276}$ $\pm$ $\textbf{0.0004}$ | $\textbf{0.0282}$ $\pm$ $\textbf{0.0003}$ | $\textbf{0.0265}$ $\pm$ $\textbf{0.0004}$ |

Reviewer comment (Experiments):
Question — Why did the author not address other anomalies beyond extreme value anomalies?

Author response:
* Third, "_why did not consider other kinds of anomalies, like inter-variable relationship anomalies_"? Thanks for the question. To the best of our knowledge, facing various types of anomalies, different methods are proposed for different anomaly patterns. On the one hand, in our paper, the proposed model did the value anomaly detection, and the selected baselines and discussed related work are also the methods that only did the value anomaly detection. On the other hand, the structural anomaly detection method you suggested also targets one specific type of anomaly among various anomaly patterns, which is also unable to do value anomaly detection. Each aspect has its own value, like value anomaly detection can help with extreme weather detection, and structural anomaly detection can help with high-tech equipment deployment. **Proposing a universal anomaly detection method for all possible anomaly patterns is very interesting, we would like to explore it as a future direction. But for now, it is beyond the scope of our submission.**

Reviewer comment (Writing):
The paper is poorly written. It contains many spelling, grammar and other use-of-language mistakes that make it very difficult to discern the authors' intent and extremely difficult to follow the paper.

Author response:
* First, we further formalized and polished the language of the paper, and uploaded the revised paper.

Reviewer comment (Theory):
Many important issues are handled with an off-hand approach. For example, see the citation of EVT for 'inspiration' or the expression of Thm 3.1. The theorem posits that "under standard causal discovery assumptions" the full causal graph can be recovered. Depending on what these assumptions are, this theorem may be a groundbreaking discovery. However the "proof sketch" of Thm 3.1 simply cites the source paper and does not construct any formal argument.

Author response:
- Second, the reason why we use the term "proof sketch" is that the original paper gives the proof, that's why we included the citation in our Thm 3.1.

Reviewer comment (Experiments):
The experiment setup contains no simple / naive baselines for forecasting or for anomaly detection. For such a complex architecture proposed, there is only a very limited ablation study to show which parts of TacSas are critical to its success.

Author response:
- Finally, we add the SOTA time series forecasting baseline ST-SSL (Ji et al., 2023) in the below table, which outperformed many simple forecasting baselines like ARIMA (Kumar and Vanaiakshi 2015) and SVR (Castro-Neto et al. 2009) in their paper.

|            | ERA5-2017 $(\downarrow)$ | ERA5-2018 $(\downarrow)$ | ERA5-2019 $(\downarrow)$ | ERA5-2020 $(\downarrow)$ |
|------------|--------------------------|--------------------------|--------------------------|--------------------------|
| GRU        | 1.8834 $\pm$ 0.0126       | 1.9764 $\pm$ 0.1466       | 1.6194 $\pm$ 0.2645       | 1.7859 $\pm$ 0.2324       |
| DCRNN      | 0.0819 $\pm$ 0.0025       | 0.0797 $\pm$ 0.0049       | 0.0799 $\pm$ 0.0035       | 0.0826 $\pm$ 0.0033       |
| GTS        | 0.0777 $\pm$ 0.0054       | 0.0766 $\pm$ 0.0029       | 0.0760 $\pm$ 0.0031       | 0.0742 $\pm$ 0.0021       |
| TacSas     | 0.0496 $\pm$ 0.0017       | 0.0499 $\pm$ 0.0017       | 0.0502 $\pm$ 0.0016       | 0.0488 $\pm$ 0.0019       |
| ST-SSL     | 0.0345 $\pm$ 0.0051      | 0.0330 $\pm$ 0.0018       | 0.0361 $\pm$ 0.0021       | 0.0348 $\pm$ 0.0020       |
| TacSas++    | $\textbf{0.0271}$ $\pm$ $\textbf{0.0004}$ | $\textbf{0.0276}$ $\pm$ $\textbf{0.0004}$ | $\textbf{0.0282}$ $\pm$ $\textbf{0.0003}$ | $\textbf{0.0265}$ $\pm$ $\textbf{0.0004}$ |

Reviewer comment (Experiments):
The proposed method is overall interesting - However, could you provide a dedicated quantitative summary and micro case study in the Appendix regarding how TacSas can capture the ground-truth causality but also can be applied when the ground-truth causal structures are hardly available?

Author response:
* First, **regarding how TacSas can capture the ground-truth causality**, we prepared the ground-truth benchmark Lorenz96 and compared TacSas with the SOTA baseline GVAR (Marcinke-vics and Vogt, 2021). More details can be found in Appendix B.1. In short, our TacSas can discover the ground-truth causal structure defined by Lonrenz96 with competitive accuracy compared with GVAR. 

- Second, **for the scenario that causal structures are hardly available**, in the climate datasets we contributed as a new time series source (where the time series items are rich but ground causal structures are not available), the visualization plotted in Figure 3 quantitatively shows the causal structures captured by TacSas, which is the potential reason why TacSas can outperform DCRNN (Li et al., 2018) and GTS (Shang et al., 2021) for capturing fine-grained temporal-spatial structures in the hardly-available scenario.