Paper: WinNet:time series forecasting with a window-enhanced period extracting and interacting (ICLR-2024)

Reviewer comment (Writing):
**Inconsistent Mathematical Notation:** The authors' presentation of mathematical symbols lacks consistency. As an example, function names should conventionally be displayed in regular typeface instead of italic. Furthermore, vectors and matrices should be represented in bold. Adhering to proper notation is pivotal for ensuring clarity and averting potential misunderstandings.

Author response:
Thank you for your suggestions. We have updated our manuscript about the presentation of mathematical symbols and matrices.

Reviewer comment (Presentation):
**Graphics and Formatting Issues:** The image quality in Figure 4 is noticeably poor. Additionally, the authors did not adhere to ICLR's guidelines, as they combined the appendix and main text, which could hamper structured reading and comprehension.

Author response:
Thank you for your suggestions. In this revision, we have combined the main text and appendix into a single PDF to adhere to ICLR’s guidelines.

Reviewer comment (Experiments):
**On CNN-based MTS:** Given the inherent characteristics of time series data, it's a general understanding that Transformer-based methods often underperform compared to Linear and CNN approaches in various scenarios. However, one of the advantages of Transformer methods is their ability to capture interrelationships amongst multi-variables. Evaluating CNN-based methods purely on accuracy might not provide a fair comparison. Moreover, several existing methods, such as Seq-VAE, leverage CNN for MTS tasks. A comparison of WinNet with such methods would have added depth to the evaluation.

Author response:
Thank you for your comments. As shown in Tables 5, 14 and 15, we have compared not only the accuracy with the transformer model, but also the efficiency of the model including training time, inference time and number of parameters. While the Transformer models have the ability to capture the relationships between multiple variables, WinNet uses a channel-independence strategy (following to DLinear [1], RLinear [2]) to eliminate interactions between multiple channels and focuses on capturing seasonality in a single channel, mostly achieving excellent results.

We have searched for Seq-VAE [3] and find that it is a method applied to sentence generation tasks and is not related to time series forecasting tasks. In addition, the method is a combination of VAE and GAN, and neither the network model nor the CNN is relevant. Would you please provide the above paper in the reference format that can help us find it precisely?

In the original version, we have conducted comparisons with the CNN methods TimesNet [4] and MICN [5], and provided a comprehensive comparison against the MLP methods. In this revision, we have also added a comparison with RLinear and RMLP, as shown in Tables 1 and 2. It is believed that the performance of WinNet can be validated by the comparisons mentioned above.

Reviewer comment (Reproducibility):
**Normalization in MTS:** Normalization plays a pivotal role in predicting MTS. What kind of normalization technique has been employed within the WinNet framework?

Author response:
We adopt the regularization method of RevIN, which is widely used in the SOTA methods, such as DLinear, PatchTST [6], and RLinear.

Reviewer comment (Writing):
**Handling Multivariate Time Series in WinNet:** How does WinNet approach and manage the relationships between variables in a multivariate time series (MTS)? A more in-depth discussion on this aspect could enhance the paper's clarity.

Author response:
We adopt a channel-independence strategy that eliminates interactions between multiple variables. This strategy enhances the robustness of the model for multivariate time series with different periodic channels, which demonstrates in the modern methods, including DLinear, PatchTST, and RLinear.

Reviewer comment (Experiments):
**Hyperparameter Settings in WinNet:** Could you elaborate on how the hyperparameters for WinNet were determined? Moreover, how do these hyperparameter choices influence the final results? Insights on this could help understand the model's sensitivity and robustness.

Author response:
In the WinNet experiments, only three parameters need to be determined, including window_len for MLP, kernel_size for Avgpool2D(·), and kernel_size for Conv2D(·). In the original version, we have supplemented the ablation experiments for the window_len as shown in Appendix Table 9. For the Avgpool2D(·) kernel_size, we combine the periodicity of the sequence to set it as w (window size). Furthermore, we have also conducted the ablation studies for kernel_size of Conv2D(·) and the results are presented in Table 10.

Reviewer comment (Reproducibility):
For the I2PE block: How to train the MLP; how to calculate the periodic window when using two periods approximately as one period (11/12 or 23/24) in Table 1; why use n = w; should we do padding for X_1D before performing Equation (1); what is the detailed process of I2PE in Equation (1) (the details can be guessed but they should be described clearly).

Author response:
MLP is a linear transformation at the dimension of the sequence length, and we set d_model to 576 (the square of periodic window size). We are referring to the period obtained by Fast Fourier Transformation (FFT), and for the odd number of 11 or 23, we simply add 1 to find the common multiple easily.  

In the experiments, the number and size of periodic windows are both configured to be the same for generating the square matrix, reinforcing the Avgpool2D(·) operation in the TDPD block. Specifically, the square matrix can balance feature selection between intra-periodic and inter-periodic features during the Avgpool2D(·) operation, which makes the TDPD block more rational.  

We perform a linear transformation in the I2PE block to change the length of the sequence to the square of the periodic window size. Then our model performs the padding and Avgpool2D(·) operations to decompose the 2D tensor into the period-trend and oscillation terms. Therefore, it is unable to perform padding before performing Eq. (1).

Reviewer comment (Writing):
For the TDPD block: what is the meaning of w×w in Equation (2), is it the kernel_size for AvgPool2d; what are the two inputs in "According to the equation 2, the two inputs can be decomposed into the period-trend and oscillation terms".

Author response:
Yes, wxw is kernel_size for Avgpool2D(·), and we change it to k to differentiate the window size. After the I2PE block, we get the two inputs (features), inter-period and intra-period, which are fed into the TDPD and DCB blocks, respectively. In the series decoder, we perform direct additive fusion of the two features.

Reviewer comment (Reproducibility):
For the DCB block: What is the process of CI (channel independence strategy) in Equation (3).

Author response:
We adopt a channel-independence strategy that eliminates interactions between multiple variables. Specifically, we split both intra-periodic and inter-periodic features at the channel dimension and feed the single-channel matrix into the Concat(·) operation to compose a two-channel matrix.

Reviewer comment (Presentation):
In Figure 1, what is the process of CA.

Author response:
In the process of CA, we concatenate the single-channel output after the DCB block at the channel dimension for fusion.

Reviewer comment (Experiments):
The proposal WinNet does not compare with SoTA methods and the performance improvement is not significant.

Author response:
Thank you for your comments. In the experimental results, we have compared ICLR2023 SOTA models (PatchTST [1], MICN [2], Crossformer [3], TimesNet [4]) and AAAI2023 SOTA model (DLinear [5]). According to your recommendations, we have also added 2 MLP models (RLinear [6] and RMLP), which can be seen in Tables 1 and 2. According to ICLR's REVIEWER GUIDE, papers published on or after 28 May 2023 are not required to compare their work to that paper, so we do not compare with PETformer. In addition, the authors of PETformer [7] do not release the source code and there are no results from the 512 length setting in the paper. Therefore, we have only compared 336 input lengths with PETformer and the results are listed as follows:  

|   Methods   | Methods |   WinNet  |   WinNet  |  RLinear  |  RLinear  |    RMLP   |  RMLP | PETformer | PETformer |
|:-----------:|:-------:|:---------:|:---------:|:---------:|:---------:|:---------:|:-----:|:---------:|:---------:|
|    Metric   |  Metric |    MSE    |    MAE    |    MSE    |    MAE    |    MSE    |  MAE  |    MSE    |    MAE    |
|    ETTm1    |    96   | **0.279** |   0.333   |   0.301   |   0.342   |   0.298   | 0.345 |   0.281   | **0.324** |
|    ETTm1    |   192   | **0.319** |   0.357   |   0.335   |   0.363   |   0.344   | 0.375 |   0.321   | **0.351** |
|    ETTm1    |   336   | **0.354** |   0.381   |   0.370   |   0.383   |   0.390   | 0.410 |   0.356   | **0.372** |
|    ETTm1    |   720   | **0.409** |   0.414   |   0.425   |   0.414   |   0.445   | 0.441 |   0.416   | **0.407** |
|    ETTm2    |    96   | **0.159** |   0.248   |   0.164   |   0.253   |   0.174   | 0.259 |   0.160   | **0.245** |
|    ETTm2    |   192   | **0.214** | **0.289** |   0.219   |   0.290   |   0.236   | 0.303 |   0.220   | **0.289** |
|    ETTm2    |   336   | **0.266** |   0.323   |   0.273   |   0.326   |   0.291   | 0.338 |   0.271   | **0.320** |
|    ETTm2    |   720   | **0.355** | **0.377** |   0.366   |   0.385   |   0.371   | 0.391 |   0.357   |   0.379   |
|    ETTh1    |    96   |   0.366   |   0.391   |   0.366   |   0.391   |   0.390   | 0.410 | **0.358** | **0.381** |
|    ETTh1    |   192   |   0.401   |   0.411   |   0.404   |   0.412   |   0.430   | 0.432 | **0.397** | **0.404** |
|    ETTh1    |   336   |   0.424   |   0.426   |   0.420   |   0.423   |   0.431   | 0.441 | **0.419** | **0.417** |
|    ETTh1    |   720   | **0.431** | **0.450** |   0.442   |   0.456   |   0.450   | 0.495 |   0.443   |   0.453   |
|    ETTh2    |    96   |   0.271   |   0.333   | **0.262** | **0.331** |   0.288   | 0.352 |   0.281   |   0.333   |
|    ETTh2    |   192   |   0.330   | **0.374** | **0.319** | **0.374** |   0.343   | 0.387 |   0.345   |   0.376   |
|    ETTh2    |   336   |   0.362   |   0.400   | **0.325** |   0.386   |   0.353   | 0.402 |   0.336   | **0.380** |
|    ETTh2    |   720   |   0.393   |   0.433   | **0.372** | **0.421** |   0.410   | 0.440 |   0.385   |   0.422   |
|   Weather   |    96   | **0.145** |   0.200   |   0.175   |   0.225   |   0.149   | 0.202 |   0.150   | **0.190** |
|   Weather   |   192   | **0.188** |   0.246   |   0.218   |   0.260   |   0.194   | 0.242 |   0.194   | **0.232** |
|   Weather   |   336   | **0.239** |   0.282   |   0.265   |   0.294   |   0.243   | 0.282 |   0.246   | **0.273** |
|   Weather   |   720   |   0.318   |   0.341   |   0.329   |   0.339   | **0.316** | 0.333 |   0.320   | **0.326** |
| Electricity |    96   |   0.131   |   0.227   |   0.140   |   0.235   | **0.129** | 0.224 |   0.131   | **0.223** |
| Electricity |   192   |   0.148   |   0.241   |   0.154   |   0.248   | **0.147** | 0.240 | **0.147** | **0.238** |
| Electricity |   336   |   0.164   |   0.258   |   0.171   |   0.264   |   0.164   | 0.257 | **0.163** | **0.255** |
| Electricity |   720   | **0.201** |   0.293   |   0.209   |   0.297   |   0.203   | 0.291 |   0.203   | **0.289** |

Reviewer comment (Experiments):
It is said in the Appendix that some experimental results are taken from the PatchTST and PETformer, but there is no comparison with PETformer. There are also other MLP-based models (RLinear and RMLP), which outperform PatchTST on some datasets and should be compared with.

Author response:
Thank you for your comments. In the experimental results, we have compared ICLR2023 SOTA models (PatchTST [1], MICN [2], Crossformer [3], TimesNet [4]) and AAAI2023 SOTA model (DLinear [5]). According to your recommendations, we have also added 2 MLP models (RLinear [6] and RMLP), which can be seen in Tables 1 and 2. According to ICLR's REVIEWER GUIDE, papers published on or after 28 May 2023 are not required to compare their work to that paper, so we do not compare with PETformer. In addition, the authors of PETformer [7] do not release the source code and there are no results from the 512 length setting in the paper. Therefore, we have only compared 336 input lengths with PETformer and the results are listed as follows:  

|   Methods   | ... | (table continues as above) |

Reviewer comment (Evaluation):
WinNet cannot beat existing models in many cases even based on the results shown in the manuscript.

Author response:
Thank you for your insightful comments. As shown in Tables 16, 17 and 18, the proposed WinNet achieves superior performance with longer input lengths (e.g., 336 and 512), while it obtains inferior results with shorter input lengths (e.g., 96). The possible reasons can be explained as follows. In general, the periodicity of the data can be easily extracted as the input lengths become longer. Specifically, we decompose the period-trend and oscillation terms for both intra-periodic and inter-periodic features by Avgpool2D(·) operation. However, the input length of 96 is not sufficient for this purpose. Therefore, WinNet achieves inferior performance for some datasets with 96 input lengths.

Reviewer comment (Presentation):
Why are the results of the exchange dataset not given in Table 2?

Author response:
We have added experimental results about the exchange dataset in Table 2 **(now is 1)**.

Reviewer comment (Experiments):
The architectures for the Ablation Studies are not described clearly. In addition, how about the results by using only inter-period or intra-period branch in Figure 1?

Author response:
We have updated the specific details of the ablation experiments in the Appendix A.2 **Model architecture**. Moreover, we have also conducted the ablation studies about only intra-period or inter-period branch. The experimental results of ablation studies show that the method using both intra-period and inter-period achieves the best results. The experimental results can be seen in the Appendix Table 12.

Reviewer comment (Evaluation):
Which kinds of Time and Memory are considered in Table 6?

Author response:
The times in Table 6 (now is 5) are the training times calculated for one iteration, and the memory overhead is obtained by the torch.cuda.memory_allocated function. We have also added the inference times, which presents in Tables 5, 14 and 15. These experimental results demonstrate the efficiency of our model against other SOTA models in univariate prediction, as well as multivariate prediction tasks with different number of channels.

Reviewer comment (Theory):
Some claims are not clear, e.g., it is not clear why "The correlation between period-trend and oscillation terms can provide the local periodicity in time series".

Author response:
As for long-term time series forecasting tasks, we conclude that there is an extremely strong lag correlation between the trend and seasonal terms, as shown in Figure 1 (page 2), and a network model should be designed to extract the correlations, rather than modelling these two terms separately, such as DLinear and MICN.  

In this work, we concatenate the period-trend and oscillation terms in the DCB block at the channel dimension to form a two-channel matrix and then convolve it into a one-channel matrix. The CNN kernel can exactly extract the variation of the two terms within adjacent periods, and the learned parameters are able to perform a proportional aggregation of them, rather than simply adding. Moreover, the aggregation can extract the lag correlation between the two terms. Therefore, a certain local periodicity can be extracted by a single convolution process.

Reviewer comment (Reproducibility):
The code is not available for reproducibility.

Author response:
We will release the source code after the manuscript is accepted.

Reviewer comment (Presentation):
In figure 1, through I2PE block, you can get inter-period and intra-period features. The inter-period features represent the long-period features and the intra-period features represent the short-period features. However, in figure 1, you wrote that the short-period features are inter-period features, and the long-period features are intra-period features.

Author response:
Thank you for your correction. We have updated the descriptions in the Figure 1 (now is 2) accordingly. **(page 4)**

Reviewer comment (Presentation):
In section 3.1, the framework of I2PE is needed.

Author response:
Thank you for your suggestions. We have updated our manuscript and shown the detailed description about the I2PE framework in Eq. (1). **(page 4)**

Reviewer comment (Evaluation):
In the experimental part, I noticed that for some data sets, WINNET's performance is not the best, not even the second best. Some explanation is needed.

Author response:
Thank you for your insightful comments. As shown in Tables 16, 17 and 18, the proposed WinNet achieves superior performance with longer input lengths (e.g., 336 and 512), while it obtains inferior results with shorter input lengths (e.g., 96). The possible reasons can be explained as follows. In general, the periodicity of the data can be easily extracted as the input lengths become longer. Specifically, we decompose the period-trend and oscillation terms for both intra-periodic and inter-periodic features by the Avgpool2D(·) operation. However, the input length of 96 is not sufficient for this purpose. Therefore, WinNet achieves inferior performance for some datasets with 96 input lengths.

Reviewer comment (Presentation):
Please pay attention to typography issues, such as the size of Table 3. Table 4.

Author response:
We have updated our manuscript to unify the form of the tables.

Reviewer comment (Experiments):
In section 3.1, you set the number of periodic windows is the same as the periodic window size. What is the reason for this setting?

Author response:
The number and size of periodic windows are both configured to be the same for generating the square matrix, reinforcing the Avgpool2D(·) operation in the TDPD module. Specifically, the square matrix can balance feature selection between intra-periodic and inter-periodic features during the Avgpool2D(·) operation. This approach avoids the over-extraction of periodicity from either feature.

Reviewer comment (Evaluation):
From figure 4, we can see that in some cases, the performances of other baselines are better than WINNET, which cannot support the conclusion that WINNET outperforms other baselines. Does WINNET outperform only under some certain T settings?

Author response:
Thank you for your insightful comments. The proposed WinNet achieves inferior results with shorter input lengths. The WinNet model is more capable of multivariate time series forecasting tasks with longer input lengths (e.g., 336 and 512). The possible reasons can be explained as follows. In general, the periodicity of the sequence can be easily extracted as the input lengths become longer. Specifically, we decompose the period-trend and oscillation terms for intra-periodic and inter-periodic features by the Avgpool2D(·) operation. However, the input length of 96 is not sufficient for this purpose. Therefore, WinNet achieves inferior performance for some datasets with 96 input lengths.

As shown in Appendix Figure 9, we have conducted an experimental validation on the ETT datasets, and the experimental results further support the above analyses. **(page 19)**