Paper: Capturing The Channel Dependency Completely Via Knowledge-Episodic Memory For Time Series Forecasting (ICLR-2024)

Reviewer comment (Experiments):
Lack of the performance comparison of PatchTST and TimeNet, which are two SOTA baselines for LT-MTS forecasting from ICLR2023.

Author response:
We would like to thank you for your comment and suggestion to compare with these two models. We provide the following table to compare the performance of the two models on multivariate setting, which is the same as 'response to all reviewers'. It can be seen that our model outperforms these models in most settings. The full efficiency comparison is provided in Table 7 of Appendix B.7.

Reviewer comment (Writing):
In the Recall strategy section, we use m to denote the aggregated knowledge patterns but never used again in other paper’s equations. Does it correspond to the output of Recall(M)? If yes, better to clear it in the equation 2.

Author response:
Thank you for your advice. It corresponds to the output of Recall(M), so we have corrected it as you pointed.

Reviewer comment (Reproducibility):
Could you provide more detailed implementation details such as normalization protocols, splitting ratio, optimizer/scheduler etc.

Author response:
We use Z-score Normalization to normalize the dataset. The splitting ration is set 7:1:2 for train, val and test set on Illness, weather, exchange and Electricity dataset. The splitting ration is set 3:1:2 for train, val and test set on ETTh1, ETTh2, ETTm1 and ETTm2 dataset. We use Adam as the optimizer. The scheduler is used the same as Linear model in their official code. All of these settings are the same as the Linear model and previous methods.

Reviewer comment (Writing):
While the paper does a good job of introducing the model architecture and memory modules, more detailed explanations of certain components, such as the initialization of knowledge patterns and the selection process for hard patterns, could further enhance the reader's understanding.

Author response:
We introduce the initialization of knowledge patterns in the 'storage strategy' of section 3.4.  We typically select the top K channels with the highest prediction loss in each batch as hard  patterns.  For specific details on the update operations, please refer to the 'Update strategy' part in section 3.5. Due to the limited length of the main text, the selection process and update procsee for hard patterns are introduced in Appendix B.1.

Reviewer comment (Evaluation):
It would be valuable to provide a comparative analysis of the computational complexity of the proposed approach compared to existing methods, as it could impact the practicality of the model.

Author response:
Thank you for your advice. We have added computational complexity of the proposed approach compared to existing methods as shown below:
The original Transformer has $O(L^2)$ complexity on both time and space, where L is the number of input length. PatchTST use patch to reduce the L and achieve $O(N^2) (N<L)$ complexity on both time and space, where N is the number of input tokens. Dlinear is a Linear based model, so the complexity on both time and space is $O(L)$. The complexity of our model is caused by the calculation of attention score, which is $O(DM)$(D is the number of channels, M is the memory size). In most cases, D, M is much smaller than L, so the complexity of our model is smaller than the transformer based model.

Reviewer comment (Writing):
The terminology used in the paper appears to be inappropriate, e.g., 'student-like pattern memory,' 'knowledge pattern memory,' and 'episodic pattern memory.'

Author response:
A1: We have made the modifications in the article according to your suggestions and highlighted them in blue.

Reviewer comment (Presentation):
All references are cited incorrectly. Most of them should be cited using \citep{}.

Author response:
A3: We have made the modifications in the article according to your suggestions.

Reviewer comment (Reproducibility):
Details of the combination part before outputting the final prediction results are missing.

Author response:
A1: We have revised the combination and prediction in equation 2 mentioned in section 3.3. The 'Concat' means the concatenation between different tensors, which is usually implemented by torch.cat() in pytorch.

Reviewer comment (Experiments):
In your released source code, I noticed that in the test set dataloader, you set 'drop_last' to True (batch size=8). However, the Linear (Zeng et al. 2023) paper uses 'drop_last=False' and batch size=32. As you directly use their reported results of Linear for comparisons, there may be some inconsistencies in the experimental setups.

Author response:
A1: We set batch size=32 in the scripts when we train our model. The default setting in the code is not the final setting for our model. About 'drop_last problem': The results in the paper of Linear model are obtained from 'drop_last=True'. This issue can be found in their github issue(https://github.com/cure-lab/LTSF-Linear/issues/76). For fair comoparison, the previous methods and our model both use the 'drop_last=True' setting. Due to Jiang et al. (2023) focus on traffic forecasting, their model is based on RNN module which takes too much time to forecast long time series. Another reason is that the RNN based model is not good at long-term series forecasting. Considering that, we do not compare it with our model. About spatio time series: We have added a traffic dataset which records the road occupancy rates from different sensors on San Francisco freeways. The result is shown in 'response to all Reviewers'.

Reviewer comment (Experiments):
The training objective (5) of the memory module (knowledge memory) is basically from the paper by Jiang et al. (2023). Thus, it is recommended to include this spatio-temporal baseline in the experiments. Additionally, it would be beneficial to include more commonly used spatio-temporal datasets in the experiments, such as METR-LA and PEMS-BAY, as suggested by Jiang et al. (2023).

Author response:
A1: We set batch size=32 in the scripts when we train our model. The default setting in the code is not the final setting for our model. About 'drop_last problem': The results in the paper of Linear model are obtained from 'drop_last=True'. This issue can be found in their github issue(https://github.com/cure-lab/LTSF-Linear/issues/76). For fair comoparison, the previous methods and our model both use the 'drop_last=True' setting. Due to Jiang et al. (2023) focus on traffic forecasting, their model is based on RNN module which takes too much time to forecast long time series. Another reason is that the RNN based model is not good at long-term series forecasting. Considering that, we do not compare it with our model. About spatio time series: We have added a traffic dataset which records the road occupancy rates from different sensors on San Francisco freeways. The result is shown in 'response to all Reviewers'.

Reviewer comment (Experiments):
Why choose Linear for comparison, not NLinear or DLinear (Zeng et al. 2023)?

Author response:
A2: Because we do not use 'revin normalization' or 'trend-seasonal decompose' operation which are proved to be effective to improve the performance of forecasting model, we think use Linear model is a more fair baseline. However, considering the issue you mentioned, we add DLinear model as a baseline in 'response to all reviewers'.

Reviewer comment (Presentation):
Figure 2 is somewhat challenging to read. It would be better to display the correlation matrix found by the memories to demonstrate the channel dependencies.

Author response:
A3: We have made the modifications in the article according to your suggestions and highlighted them in blue.

Reviewer comment (Evaluation):
For some of the datasets in the paper where the units of variables differ, it is worth considering whether dependency modeling is necessary because PatchTST's performance seems to be good on them by channel independence.

Author response:
A2: We visualize different channels on different datasets and find that most of channels have similar trends as shown in section 4.4. Thus, we consider that the effective way to capture the channel dependency is necessary. Our viewpoint is to use channel independence to eliminate redundant correlation between channels and then effectively blend information between relevant channels using suitable methods, which may be more effective.
By combining our method with PatchTST and conducting tests on various datasets, we observed significant improvements. Therefore, we believe that the assumption of channel independence may not be entirely reasonable. 
Our future work will be dedicated to address this issue completely.