Paper: Structural Knowledge Informed Continual Multivariate Time Series Forecasting (ICLR-2024)

Reviewer comment (Writing):
This paper is written in a way that takes time to understand the term "regime", so it is not easy to follow.

Author response:
We sincerely appreciate the reviewer for recognizing our contributions and raising concerns in terms of the definition of regimes. In the abstract and introduction, we referred “regime” to “stage” and indicated the underlying temporal dynamics and dependency structures of multivariate time series data for each regime is different (**abstract, the end of the first paragraph and the beginning of the second paragraph in introduction, page 1**). We also discussed this term multiple times under the context of continual learning across the introduction. As shown in Figure 1, we defined different regimes with raw multivariate time series and the corresponding structural knowledge, which are marked and annotated using different colors. Besides, in the beginning of the methodology, we gave the mathematical formulation of our continual forecasting task with a clear notion of regime. Again, we appreciate the reviewer for raising this concern and would emphasize the definition of regime at the beginning of our final manuscript and add another real-world example for demonstration.

Reviewer comment (Novelty):
This paper omits related work on continual learning in time series and graph domains.

Author response:
We explicitly discussed the methods of continual learning in time series in our related work, **section 2.2**, where we also explicitly mentioned methods of continual learning on graphs.

Reviewer comment (Experiments):
There is no comparison of model complexity or training time with other baselines.

Author response:
We compare the number of parameters for each backbone models as following table:
| Model | Parameter Number | Rank |
| :--- | :---: | :---: |
| LSTNet | 53253| 12 |
| STGCN| 96606 | 11 |
| MTGNN| 139990 | 10 |
| AGCRN| 252130 | 8 |
| GTS | 14647763 | 2 |
| ESG | 5999516 | 4 |
| TCN | 170886 | 9 |
| StemGNN | 1060802 | 6 |
| Autoformer | 10612758 | 3 |
| PatchTST | 3226124 | 5 |
| Dlinear | 49168 | 13 |
| TimesNet | 36849590 | 1 |
| SKI-CL | 614731 | 7 |

The number of our models' parameters is significantly smaller than the SOTA baseline models, including Autoformer, PatchTST, TimesNet and can achieve the best performance for continual learning tasks.

Reviewer comment (Novelty):
This paper can be read as an incremental work that brings the existing graph continual learning problem to the MTS domain. While the authors define the catastrophic forgetting problem in different regime settings in this paper, they overshadow all other similar problems in the time series domain. For example, this paper almost entirely ignores concept/temporal drift, dynamic graph learning, and continual learning on traditional time series. In this paper, it is necessary to clarify the similarities and differences with these fields and show the relative position in these research lines.

Author response:
We appreciate the reviewer for raising this concern. Firstly, we would like to clarify that the scope of our paper is different from that of graph continual learning. Graph continual learning aims to maintain the model performance of all previous tasks on a sequence of explicit topological structures that is potentially evolving. However, in our work, we resort to modeling the dependency structure of MTS data that is coupled with the temporal dynamics, and leverage it for downstream forecasting tasks. This means that the dependency structure is not fixed and implicit within each regime.
In addition to that, several terms are mentioned in this comment: concept/temporal drift, dynamic graph learning, and continual learning on traditional time series. For continual learning on traditional time series, we have discussed the continual learning methods for MTS data in the **section 2.2**, and we are not sure which reference the reviewer refers to here. It would be helpful to facilitate the discussion if specific literature can be pointed out.
**Dynamic graph learning:** Next, we discuss the main differences between dynamic graph learning and our method. While both dynamic graph learning and our method deal with the evolving data, dynamic graph learning focuses on adapting to the changes in the explicit graph structure and possibly features over time. Like graph continual learning, dynamic graph learning uses the explicitly given graph structure for message passing and performs downstream tasks like node classification, link prediction, or community detection. While our proposed continual multivariate time series forecasting method needs to discover underlying dependencies structure of variables over different regimes, by capturing the temporal dynamics of MTS data. This also leads to the other key difference between dynamic graph learning and our method: the former always adapts its model to cope with the change (perform well on the new data), while the primary goal of the latter is the model's capability to continually learn without forgetting previous knowledge (i.e., addressing catastrophic forgetting). The focus of methodology can be very different due to the different learning objectives. We have similar discussion regarding the differences between online learning and our setting, in the literature review, **the second paragraph of section 2.2 in page 3**.  
**Concept drift/temporal drift:** Then we would like to discuss the relationship between our research scope and concept drift/temporal drift, and explain why these methods cannot serve as baselines for comparison(as the reviewer mentioned in the other question). Concept/temporal drift of MTS represents the phenomenon where the data distribution of the target MTS changes over time in unforeseen ways. Based on this notion, it is important to emphasize the continual adaptation nature of continual learning setting regarding the concept/temporal drifts, where the catastrophic forgetting is introduced due to the shifts over multiple regimes. That being said, continual MTS forecasters need to handle a variety of regimes, not just adapt to changes in data distribution for a single one, which is to some degree more realistic and challenging. Concept/temporal drift represents a specific challenge within this broader goal. 
Moreover, the learning objectives between concept/temporal drift methods and continual learning methods are very different. For the concept/temporal drift methods, the objective is to adapt the forecaster to these new MTS patterns by effectively detecting, responding to, and learning from these changes. In contrast, as we emphasized before, the main focus of continual learning methods is to prevent the model from forgetting previously learned knowledge (coupled temporal dynamics and variable dependencies in our scope) when adapting to new MTS distribution. Due to the aforementioned differences in adaptation nature and learning objectives, it is very difficult to make fair experimental comparisons under the same setting. 
Finally, regarding the reviewer’s concern, we would like to restate the position/scope of our research compared to the fields mentioned above. We start from the continual learning perspective, explore and address the challenges of MTS forecasting problem under this setting, where we resort to variable dependency modeling and characterization aided by structural knowledge. As we discussed in **section 2.2**, continual learning on MTS data and especially continual forecasting is still an underexplored problem, and we proposed a solution connecting both research areas. Even if the mentioned research fields and our work all tackle the evolving data, the problem formulation, challenges and learning objectives are significantly different as we discussed above.

Reviewer comment (Writing):
The definition of regime written by the authors in the introduction is unclear, so readers may have difficulty understanding it. The motivation to solve problems caused by different regimes is good, but if readers do not understand regimes, the motivation of this paper may be meaningless. Readers may want to look at Figure 1 for simple examples of problems caused by different regimes instead of SKI-CL's framework.

Author response:
We sincerely appreciate the reviewer for recognizing our contributions and raising concerns in terms of the definition of regimes. In the abstract and introduction, we referred “regime” to “stage” and indicated the underlying temporal dynamics and dependency structures of multivariate time series data for each regime is different (**abstract, the end of the first paragraph and the beginning of the second paragraph in introduction, page 1**). We also discussed this term multiple times under the context of continual learning across the introduction. As shown in Figure 1, we defined different regimes with raw multivariate time series and the corresponding structural knowledge, which are marked and annotated using different colors. Besides, in the beginning of the methodology, we gave the mathematical formulation of our continual forecasting task with a clear notion of regime. Again, we appreciate the reviewer for raising this concern and would emphasize the definition of regime at the beginning of our final manuscript and add another real-world example for demonstration.

Reviewer comment (Theory):
The introduction on page 2 of this paper explains that the catastrophic forgetting problem in the MTS task is "the model performance will deteriorate over existing regimes as their associated structural knowledge cannot be maintained." Compared to the definitions of catastrophic forgetting in other existing domains, this is a setting in which no tasks or classes are added/incremented. Is this description appropriate compared to definitions in other domains?

Author response:
We appreciate the reviewer for raising this question. Yes, our description of catastrophic forgetting is appropriate. We would like to emphasize the **full context** of catastrophic forgetting that the reviewer refers to, in the first paragraph of page 2, is about modeling variable dependencies of MTS data for forecasting tasks (which is our research focus, and this problem is still underexplored as we discussed in the literature review, **the first paragraph of section 2.2 in page 3**).  
We refer to the definition of catastrophic forgetting in one of the most cited paper in continual learning [1],  *“Continual learning poses particular challenges for artificial neural networks due to the tendency for knowledge of the previously learned task(s) (e.g., task A) to be abruptly lost as information relevant to the current task (e.g., task B) is incorporated. This phenomenon, termed catastrophic forgetting, occurs specifically when the network is trained sequentially on multiple tasks because the weights in the network that are important for task A are changed to meet the objectives of task B.”*. Our description fits this definition, as the so-called “task” means regime in our case, and the “knowledge” within catastrophic forgetting involves the structural knowledge of MTS data learned by dependency modeling methods. 
The reviewer mentions that this is a setting in which no tasks or classes are added/incremented, which is under the context of classification tasks. In essence, our continual forecasting is a regime incremental setting in regression tasks, which is analogous to class-incremental tasks in classification tasks.  
Therefore, the description demonstrates the catastrophic forgetting problem under our multivariate time series forecasting task and is appropriate compared with continual learning in other domains. 

[1] Kirkpatrick, James, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan et al. "Overcoming catastrophic forgetting in neural networks." Proceedings of the national academy of sciences 114, no. 13 (2017): 3521-3526.

Reviewer comment (Novelty):
This paper needs to add related research to papers that solve concept drift and temporal drift, where the distribution changes with time in a time series. In this paper, there is a need to add related research to the paper that addresses concept drift and temporal drift, where distributions change over time in time-series data. The authors also need to compare the models and experiments in these papers.

Author response:
As we discussed in our response to weakness 5 and question 4, while both concept drift methods and continual learning strategies deal with changing data environments, concept/temporal drift approaches are often more reactive and focused on adapting to changes in data distribution for a specific task. In contrast, continual learning involves a broader spectrum of adaptation, focusing on learning from new data and preventing forgetting over existing data. The continual adaptation nature of continual learning and very different objectives between both fields make it very difficult to include the concept/temporal drift methods as baselines of continual learning and make fair comparisons under the same setting.  
We agree with the reviewer to some extent that the concept/temporal drift and continual learning in MTS both tackle the evolving environment, and some ideas from one field can be shared to facilitate the advancement of each field. **Regarding the reviewer’s concern, we have discussed the aforementioned connections in our updated manuscript.** We would like to thank the reviewer for these references of temporal drifts and will try to involve more recent methods.

Reviewer comment (Novelty):
This paper needs to describe the position in the research of continuous learning, incremental learning, concept/temporal drift, and dynamic graph learning in each domain. Can the authors explain what the similarities and differences are with these research topics?

Author response:
We appreciate the reviewer for raising this concern. Firstly, we would like to clarify that the scope of our paper is different from that of graph continual learning. Graph continual learning aims to maintain the model performance of all previous tasks on a sequence of explicit topological structures that is potentially evolving. However, in our work, we resort to modeling the dependency structure of MTS data that is coupled with the temporal dynamics, and leverage it for downstream forecasting tasks. This means that the dependency structure is not fixed and implicit within each regime.
In addition to that, several terms are mentioned in this comment: concept/temporal drift, dynamic graph learning, and continual learning on traditional time series. For continual learning on traditional time series, we have discussed the continual learning methods for MTS data in the **section 2.2**, and we are not sure which reference the reviewer refers to here. It would be helpful to facilitate the discussion if specific literature can be pointed out.
**Dynamic graph learning:** Next, we discuss the main differences between dynamic graph learning and our method. While both dynamic graph learning and our method deal with the evolving data, dynamic graph learning focuses on adapting to the changes in the explicit graph structure and possibly features over time. Like graph continual learning, dynamic graph learning uses the explicitly given graph structure for message passing and performs downstream tasks like node classification, link prediction, or community detection. While our proposed continual multivariate time series forecasting method needs to discover underlying dependencies structure of variables over different regimes, by capturing the temporal dynamics of MTS data. This also leads to the other key difference between dynamic graph learning and our method: the former always adapts its model to cope with the change (perform well on the new data), while the primary goal of the latter is the model's capability to continually learn without forgetting previous knowledge (i.e., addressing catastrophic forgetting). The focus of methodology can be very different due to the different learning objectives. We have similar discussion regarding the differences between online learning and our setting, in the literature review, **the second paragraph of section 2.2 in page 3**.  
**Concept drift/temporal drift:** Then we would like to discuss the relationship between our research scope and concept drift/temporal drift, and explain why these methods cannot serve as baselines for comparison(as the reviewer mentioned in the other question). Concept/temporal drift of MTS represents the phenomenon where the data distribution of the target MTS changes over time in unforeseen ways. Based on this notion, it is important to emphasize the continual adaptation nature of continual learning setting regarding the concept/temporal drifts, where the catastrophic forgetting is introduced due to the shifts over multiple regimes. That being said, continual MTS forecasters need to handle a variety of regimes, not just adapt to changes in data distribution for a single one, which is to some degree more realistic and challenging. Concept/temporal drift represents a specific challenge within this broader goal. 
Moreover, the learning objectives between concept/temporal drift methods and continual learning methods are very different. For the concept/temporal drift methods, the objective is to adapt the forecaster to these new MTS patterns by effectively detecting, responding to, and learning from these changes. In contrast, as we emphasized before, the main focus of continual learning methods is to prevent the model from forgetting previously learned knowledge (coupled temporal dynamics and variable dependencies in our scope) when adapting to new MTS distribution. Due to the aforementioned differences in adaptation nature and learning objectives, it is very difficult to make fair experimental comparisons under the same setting. 
Finally, regarding the reviewer’s concern, we would like to restate the position/scope of our research compared to the fields mentioned above. We start from the continual learning perspective, explore and address the challenges of MTS forecasting problem under this setting, where we resort to variable dependency modeling and characterization aided by structural knowledge. As we discussed in **section 2.2**, continual learning on MTS data and especially continual forecasting is still an underexplored problem, and we proposed a solution connecting both research areas. Even if the mentioned research fields and our work all tackle the evolving data, the problem formulation, challenges and learning objectives are significantly different as we discussed above.

Reviewer comment (Experiments):
As written in the last paragraph of Section 2.2, the authors are aware that FSNet [1] uses MTS data, but do not consider it in the baseline due to the online learning setting. However, the experimental baseline of FSNet also performs including **DER++** and **ER**. This appears to be possible in the experimental settings of this paper, and comparative experiments are needed. If you can compare FSNet, compare the performance difference compared to the model proposed by the author.

Author response:
We sincerely appreciate the reviewer’s advice. However, FSNet is designed for online forecasting, which focuses on fast adaptation to newly incoming data and uses accumulated error on the training data as evaluation protocols (there is no notion of testing data to evaluate model performance at each regime). The objective of continual learning is to maintain the learned knowledge and model performance over all seen data when adapting to new data (as we discussed in **the second paragraph of section 2,2, page 3**). As such, the problem formulation, learning objective and evaluation protocols are very different to the continual learning setting, making it not very meaningful for comparison. DER++ and ER are commonly used experience-replayed methods for continual learning, so it is natural for us to include them as our baselines. 
The MIR method is also originally designed for online continual learning setting, which uses an in-batch virtual updated model to select samples that cause the highest forgetting to perform experience replay. To enable the fair comparison, we use the aforementioned MIR criteria to select a number of samples from the fixed budget memory per regime.  We report the other SOTA model results below and our proposed method consistently achieves the best performance.     

Traffic-CL:
| Model | |  |  |  |
| :---: | :---: | :---: | :---: | :---: |
|  | AP_MAE | AP_RMSE | AF_MAE | AF_RMSE |
| PatchTST_seq |19.11 | 32.50  | 2.34  | 2.97 |
| PatchTST_mir | 19.04 | 32.23  | 2.28 | 2.79 |
| PatchTST_herd| 18.96 | 32.10  | 2.21  | 2.67 |
| PatchTST_er|   18.77 |  31.50 | 1.98 | 2.01 |
| PatchTST_der++| 18.53 |  31.34  | 1.75  | 1.98 |
| DLinear_seq| 19.69 | 32.75  |2.91  |2.83  |
| DLinear_mir| 19.37 | 32.25  |2.17  |2.59 |
| DLinear_herd| 19.53|  32.40 | 2.25 | 2.68 |
| DLinear_er| 19.19 |  32.30  |1.73  |2.14 |
| DLinear_der++| 19.02|  31.97  | 1.75  | 1.93 |
| TimesNet_seq| 17.77 | 29.91 | 3.13 | 6.93 |
| TimesNet_mir| 17.53 | 29.61 | 2.44 | 5.32 |
| TimesNet_herd| 17.38 | 29.53 | 2.56 | 5.83 |
| TimesNet_er| 17.25 | 29.33 | 1.97 | 4.19 |
| TimesNet_der++| 17.13 | 29.28 | 1.56 | 4.02 |
| SKI-CL| **15.24** | **25.33** | 1.50 | 2.71|



Solar-CL:
| Model | |  |  |  |
| :---: | :---: | :---: | :---: | :---: |
|  | AP_MAE | AP_RMSE | AF_MAE | AF_RMSE |
| PatchTST_seq   | 2.64 | 5.32 | 0.72 | 0.43 |
| PatchTST_mir   | 2.61 | 5.30 | 0.70 | 0.40 |
| PatchTST_herd  | 2.60 | 5.30 | 0.68 | 0.35 |
| PatchTST_er   | 2.57 | 5.27 | 0.47 | 0.30 |
| PatchTST_der++ | 2.53 | 5.17 | 0.43 | 0.28 |
| DLinear_seq    | 3.47 | 6.56 | 1.17 | 1.12 |
| DLinear_mir    | 3.45 | 6.51 | 1.02 | 1.01 |
| DLinear_herd   | 3.41 | 6.50 | 1.03 | 1.00 |
| DLinear_er     | 3.37 | 6.43 | 0.93 | 0.98 |
| DLinear_der++  | 3.25 | 6.37 | 0.83 | 0.79 |
| TimesNet_seq   | 3.92 | 7.18 | 1.46 | 2.51 |
| TimesNet_mir   | 3.77 | 7.15 | 1.22 | 1.57 |
| TimesNet_herd  | 3.83 | 7.10 | 1.03 | 1.44 |
| TimesNet_er    | 3.55 | 7.02 | 0.42 | 0.91 |
| TimesNet_der++ | 3.45 | 6.55 | 0.37 | 0.90 |
| SKI-CL                   | **1.75** | **4.46** | 0.09 | 0.06 |

Reviewer comment (Novelty):
The authors need to mention that papers [3-5] dealing with temporal drift are either experimental baselines or related work.

Author response:
We agree with the reviewer to some extent that the concept/temporal drift and continual learning in MTS both tackle the evolving environment, and some ideas from one field can be shared to facilitate the advancement of each field. **Regarding the reviewer’s concern, we have discussed the aforementioned connections in our updated manuscript.** We would like to thank the reviewer for these references of temporal drifts and will try to involve more recent methods.

Reviewer comment (Experiments):
Section 4.4 of the paper explains as follows: “We perform experiments on the Traffic-CL dataset to validate the effectiveness and sensitivity of two key hyperparameters in SKI-CL, the weight of structure regularizer $\lambda$ (1 by default) and the memory budget (sampling ratio) at each regime (0.01 by default).” However, the results for $\lambda$ of 1 are not shown in Table 3. And the results in Table 3 and Table 4 do not show any settings that match the results in Table 2, so the reliability of the experimental results is reduced.

Author response:
Main experiment results and ablation study for memory budgets are average experiment results that run by using different random seeds. The results have small differences due to randomness but similar trends are observed. We will release our code and models in the final version.

Reviewer comment (Evaluation):
From the experimental results in Table 2, the AF of SKI-CL of the proposed model does not result in the lowest error compared to all baselines. For example, $\text{Autoformer}_{\text{der++}}$ has the lowest MAE for AF in Traffic-CL. There seems to be a lack of explanation as to why other baselines have lower errors.

Author response:
As we discussed in **Appendix B.3**, we emphasized the superiority of average performance over average forgetting in continual learning (**page 18**):
*“Even if we provide both metrics for performance evaluation, we need to **emphasize the superiority of average performance over average forgetting** in continual learning. Average performance provides a direct measure of how well a learning system is performing on a task or set of tasks. It reflects the system’s ability to retain previously learned knowledge over past regimes while adapting to new information. While average forgetting is a relevant metric in assessing the memory capabilities of a learning system, it does not provide a complete picture of the learning system’s retention abilities. The average performance takes into account both the retention of old knowledge and the acquisition of new knowledge, providing a more comprehensive evaluation of the learning system’s performance. Therefore, we **use average performance as the main evaluation metric** and **average forgetting as an auxiliary metric** to measure knowledge retention and model adaptivity.”*
Therefore, we are supposed to firstly look at the average performance. If the average performance is very close, then we pay attention to the average forgetting to further compare the model superiority. In the aforementioned example, autoformer-der++ does not yield satisfactory average performance over past regimes, thus it is less meaningful to directly evaluate the model solely via average forgetting.

Reviewer comment (Writing):
There is a grammatical error in the sentence "given the a collection" above Eq.(1) on page 5.

Author response:
We sincerely appreciate the reviewer for pointing out the typo here, we corrected it in our updated manuscript.

Reviewer comment (Experiments):
This paper studies MTS forecasting under continual learning setting, however, some datasets used for evaluation is standard benchmark. How do you define different regimes in this case?

Author response:
**Weakness 1: How do you define different regimes for some benchmark data?**
Thanks for raising this question! Two datasets are related to the standard benchmarks (PEMS and Solar), but extended by fetching more data under identical settings from the same databases. To be specific, the Traffic-CL dataset is based on an existing PEMSD3-Stream dataset [1] which uses the traffic data (Performance Measurement System in California, District 3) from the year 2011 to 2017. The standard PEMS benchmarks for general multivariate time series forecasting is a subset from this database. Based on the PEMSD3-Stream dataset, we further simulate distinct regimes represented/defined by a different portion of a temporally expanding traffic network (sensors with different geo-locations) from different years.  
For Solar-CL, we use the database of NREL’s Solar Power Data for Integration Studies, which contains solar power data in the United States for the year 2006. The standard benchmark uses the Alabama subset of this database for general multivariate time series forecasting. We construct different regimes by states (spatial locations) with different average annual sunlight levels based on the statistics. 
The detailed description of how we generate datasets with different regimes is provided in **appendix B.1 (pages 14-16)**.
[1] Chen, Xu, Junshan Wang, and Kunqing Xie. "TrafficStream: A Streaming Traffic Flow Forecasting Framework Based on Graph Neural Networks and Continual Learning." Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence (IJCAI-21)

Reviewer comment (Experiments):
I notice that Autoformer which focuses on long term forecasting has been compared here. I wonder what’s the performance of PatchTST here, although it is also originally designed for long term forecasting.

Author response:
**Weakness 2: Comparison with PatchTST baseline**

We agree with the reviewer that some long-term forecasting methods are beyond the scope of our research focus. The performance of PatchTST is shown as follows. (We also provide the evaluation results for Dlinear and TimesNet). Our proposed method consistently achieves the best performance. 

Traffic-CL:
| Model | |  |  |  |
| :---: | :---: | :---: | :---: | :---: |
|  | AP_MAE | AP_RMSE | AF_MAE | AF_RMSE |
| PatchTST_seq |19.11 | 32.50  | 2.34  | 2.97 |
| PatchTST_mir | 19.04 | 32.23  | 2.28 | 2.79 |
| PatchTST_herd| 18.96 | 32.10  | 2.21  | 2.67 |
| PatchTST_er|   18.77 |  31.50 | 1.98 | 2.01 |
| PatchTST_der++| 18.53 |  31.34  | 1.75  | 1.98 |
| DLinear_seq| 19.69 | 32.75  |2.91  |2.83  |
| DLinear_mir| 19.37 | 32.25  |2.17  |2.59 |
| DLinear_herd| 19.53|  32.40 | 2.25 | 2.68 |
| DLinear_er| 19.19 |  32.30  |1.73  |2.14 |
| DLinear_der++| 19.02|  31.97  | 1.75  | 1.93 |
| TimesNet_seq| 17.77 | 29.91 | 3.13 | 6.93 |
| TimesNet_mir| 17.53 | 29.61 | 2.44 | 5.32 |
| TimesNet_herd| 17.38 | 29.53 | 2.56 | 5.83 |
| TimesNet_er| 17.25 | 29.33 | 1.97 | 4.19 |
| TimesNet_der++| 17.13 | 29.28 | 1.56 | 4.02 |
| SKI-CL| **15.24** | **25.33** | 1.50 | 2.71|

Solar-CL:
| Model | |  |  |  |
| :---: | :---: | :---: | :---: | :---: |
|  | AP_MAE | AP_RMSE | AF_MAE | AF_RMSE |
| PatchTST_seq   | 2.64 | 5.32 | 0.72 | 0.43 |
| PatchTST_mir   | 2.61 | 5.30 | 0.70 | 0.40 |
| PatchTST_herd  | 2.60 | 5.30 | 0.68 | 0.35 |
| PatchTST_er   | 2.57 | 5.27 | 0.47 | 0.30 |
| PatchTST_der++ | 2.53 | 5.17 | 0.43 | 0.28 |
| DLinear_seq    | 3.47 | 6.56 | 1.17 | 1.12 |
| DLinear_mir    | 3.45 | 6.51 | 1.02 | 1.01 |
| DLinear_herd   | 3.41 | 6.50 | 1.03 | 1.00 |
| DLinear_er     | 3.37 | 6.43 | 0.93 | 0.98 |
| DLinear_der++  | 3.25 | 6.37 | 0.83 | 0.79 |
| TimesNet_seq   | 3.92 | 7.18 | 1.46 | 2.51 |
| TimesNet_mir   | 3.77 | 7.15 | 1.22 | 1.57 |
| TimesNet_herd  | 3.83 | 7.10 | 1.03 | 1.44 |
| TimesNet_er    | 3.55 | 7.02 | 0.42 | 0.91 |
| TimesNet_der++ | 3.45 | 6.55 | 0.37 | 0.90 |
| SKI-CL                   | **1.75** | **4.46** | 0.09 | 0.06 |


HAR-CL:

| Model | |  |  |  |
| :---: | :---: | :---: | :---: | :---: |
|  | AP_MAE | AP_RMSE | AF_MAE | AF_RMSE |
| PatchTST_seq   | 17.91 | 27.13 | 7.18 | 6.88 |
| PatchTST_mir  | 17.82 | 26.89 | 6.82 | 4.81 |
| PatchTST_herd>  | 17.73 | 26.84 | 6.62 | 4.79 |
| PatchTST_er   | 17.57 | 26.40 | 6.02 | 4.69 |
| PatchTST_der++ | 17.12 | 26.13 | 5.79 | 4.32 |
| DLinear_seq    | 17.32 | 26.31 | 2.71 | 3.43 |
| DLinear_mir    | 16.87 | 26.12 | 2.67 | 3.01 |
| DLinear_herd   | 16.83 | 25.81 | 2.57 | 2.91 |
| DLinear_er    | 16.71 | 25.75 | 2.13 | 2.85 |
| DLinear_der++  | 16.58 | 25.47 | 1.92 | 2.77 |
| TimesNet_seq   | 18.38 | 27.61 | 4.33 | 5.15 |
| TimesNet_mir   | 18.27 | 27.59 | 4.01 | 5.08 |
| TimesNet_herd  | 18.01 | 27.53 | 3.46 | 5.03 |
| TimesNet_er | 17.84 | 27.07 | 3.28 | 4.01 |
| TimesNet_der++ | 17.73 | 26.86 | 3.11 | 3.87 |
| SKI-CL                   | **13.41** | **21.30** | 1.64 | 2.08 |

Please see the next cell for experiment results for Synthetic-CL dataset

Reviewer comment (Novelty):
The dynamic structure learning is not novel.

Author response:
**Weakness2: The dynamic structure learning is not novel.**

The existing literature either learns the structure via training parameters or representations. Our novelty lies in the role of dynamic structure learning for regime characterization, rather than an individual structure learning component. It is important that we impose the consistency regularization in the dynamic structure learning process, which aligns the learned structure with the universal and task-irrelevant structural knowledge to characterize a specific regime. The joint structure modeling based on both components leads to an capability that the existing backbones cannot achieve, i.e., **during the inference stage, our model can automatically infer a consistent structure solely based on the time series data, without knowing the regime and accessing the memory buffer**, as shown in **Figure 1: SKI-CL: Testing (page 2) and validated in Figure 4 (page 9)**. Furthermore, we are the first to explore and test different structure learning scenarios regarding the formulation of edge variables (discrete or continuous) and availability of structural knowledge (completed or partial) for multivariate time series forecasting, with visualizations in **Figures 5-8 in appendix (page 19-21)**.

Reviewer comment (Writing):
What is structural knowledge? prior knowledge? or learned knowledge?

Author response:
**Question 1: Questions about structure knowledge**

Thanks for raising this question. As we discussed in the introduction (**the last three lines in page 1 and the first six lines in page 2**) and literature review(**section 2.1 lines 10-17**), the structural knowledge can be prior knowledge (e.g., physical constraints like traffic network and power grids, domain knowledge of application scenarios like Mel-frequency cepstral coefficients) or the so-called ‘learned’ knowledge if the structure is directly inferred from data by traditional statistical methods(e.g., correlation, k-NN), or learned and inferred individually (e.g., transfer entropy/granger causality).