Paper: The Power of Minimalism in Long Sequence Time-series Forecasting (ICLR-2024)

Reviewer comment (Experiments):
As also mentioned in the paper, time series with multiple periodic intervals could not be captured by a single convolutional layer. I think it might make the paper stronger by generating synthetic data with various periodic behaviors and testing various models on it.

Author response:
Many thanks for your careful reading and suggestion. We are sorry for the late submission due to the addition of two new datasets and some new baselines. We supplemented additional experiments to further verify the validity of LTSF-Conv models in LTSF tasks. 

* Solar power prediction is a pivotal concern within the realm of renewable energy, wielding significant influence across diverse domains. We add two supplementary datasets from real-world applications, namely the Solar-Jinta and Solar-Alabama benchmarks. We think public datasets are more convincing. Meanwhile, we also add several superior baselines on Solar-Energy datasets. Solar-Jinta records seven key meteorological factors of solar radiation, collected by the hour. Solar-Alabama contains the solar power production of 137 PV plants in the USA, with a data granularity of 10 minutes. The solar power of different PV plants is influenced by varying geographical and weather conditions. You can download the datasets at https://github.com/laiguokun/multivariate-time-series-data. Table 7 provides a summary of prediction results from several baselines on the Solar-Energy datasets. It can be observed that DConv outperforms the other baselines for most horizons by a large margin. Moreover, compared to other datasets, the Solar-Jinta benchmark has a smaller size (num 8761). The experimental results also indicate that our model performs equally well on small datasets, ensuring its generalizability and robustness.  We firmly believe that LTSF-Conv models as competitive basic units, exhibiting great potential for further expansion of complex network structures.





|  |  |  |  | MSE |  |  |  |  | MAE |  |  |
|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
| Benchmarks | Methods | 96 | 192 | 336 | 720 | Avg | 96 | 192 | 336 | 720 | Avg |
| Solar-Jinta | DConv | **0.480** | **0.521** | **0.539** | **0.648** | **0.547** | 0.481 | **0.496** | **0.522** | **0.573** | **0.518** |
|  | PatchTST | 0.491  | 0.602  | 0.617  | 0.710  | 0.605  | **0.464** | 0.523  | 0.533  | 0.601  | 0.530  |
|  | Flowformer | 0.646  | 0.792  | 0.748  | 1.065  | 0.812  | 0.565  | 0.624  | 0.611  | 0.783  | 0.645  |
|  | Reformer | 0.954  | 0.934  | 0.993  | 0.995  | 0.969  | 0.773  | 0.757  | 0.742  | 0.730  | 0.751  |
|  | Informer | 0.718  | 0.821  | 0.847  | 1.102  | 0.872  | 0.616  | 0.656  | 0.737  | 0.855  | 0.716  |
|  | LogTrans | 0.704  | 0.817  | 0.754  | 1.012  | 0.821  | 0.594  | 0.651  | 0.642  | 0.762  | 0.662  |
|  | DLinear | 0.523  | 0.588  | 0.638  | 0.734  | 0.620  | 0.496  | 0.531  | 0.562  | 0.616  | 0.551  |
| Solar-Energy | DConv | **0.173** | **0.189** | **0.201** | **0.206** | **0.192** | **0.228** | **0.237** | **0.244** | **0.245** | **0.239** |
|  | PatchTST | 0.178  | 0.192  | 0.203  | 0.218  | 0.197  | 0.248  | 0.265  | 0.267  | 0.274  | 0.263  |
|  | Flowformer | 0.190  | 0.267  | 0.279  | 0.243  | 0.245  | 0.229  | 0.263  | 0.268  | 0.252  | 0.253  |
|  | Reformer | 0.195  | 0.223  | 0.253  | 0.281  | 0.238  | 0.234  | 0.252  | 0.286  | 0.286  | 0.265  |
|  | Informer | 0.195  | 0.220  | 0.259  | 0.246  | 0.230  | 0.241  | 0.241  | 0.274  | 0.266  | 0.256  |
|  | LogTrans | 0.219  | 0.217  | 0.224  | 0.241  | 0.225  | 0.240  | 0.244  | 0.258  | 0.269  | 0.252  |
|  | DLinear | 0.221  | 0.231  | 0.247  | 0.255  | 0.239  | 0.294  | 0.301  | 0.317  | 0.314  | 0.307  |

**Table 7**: Multivariate long-term time series forecasting results on Solar benchmarks.

**We look forward to your further feedback**.

Reviewer comment (Experiments):
Every univariate is now processed independently in the current convolutional network. Is there a specific reason for doing so except for the efficiency concerns? How would the performance change if we also include the feature dimension in the convolutional filter?

Author response:
Many thanks for your careful reading and suggestion. In LTSF tasks, maintaining channel independence has been observed to improve prediction performance as compared to channel mixing, as reported in prior research [2]. **Our model utilizes group-wise convolution to cleverly achieve channel independence, while concurrently reducing model complexity.** However, standard CNN uses the idea of channel mixing, which suffers from noise disturbance among the channels and reduces performance. We have conducted additional ablation experiments about depth-wise CNN and general CNN. The results are in Section C.2, Page 13 of the original manuscript. Channel independence helps reduce information confusion. When each channel focuses on capturing specific time series long-term patterns, the model is more capable of distinguishing and understanding these patterns without the noise that channel mixing might introduce. If we also include the feature dimension in the convolutional filter, the performance will degrade.
* Moreover, we also apply channel independence and channel dependence to the MLPs-based models respectively. The results can be found in Table 5 of the attached PDF file. Overall, the accuracy of the CI-based models was higher than the CD-based models.

Reviewer comment (Experiments):
Why just one layer of convolution? How does the performance change if multiple layers are applied? This may help to capture longer periodic patterns or even help with the multi-period issue.

Author response:
We are grateful for your constructive comments. **We want to emphasize that this manuscript does not only aim to showcase the contributions of SOTA results. Simple yet valuable basic units can serve as foundational building blocks, laying the groundwork for scalable and complex networks.**  We firmly believe that LTSF-Conv models serve as straightforward yet competitive basic units, exhibiting great potential for further expansion of complex structures. As the reviewers pointed out, based on our extended experiments, when applying complex network structures, the performance will be further improved. In the future, there is the potential for more valuable research to be explored in this new track.

Reviewer comment (Experiments):
How do we determine the kernel size of the convolution, which should be critical for the forecasting task, especially for the cases where we don't know the periodic interval of the data streams?

Author response:
Thank you for pointing out this. In essence, finding the most suitable convolutional kernel size is a dynamic process, combining experimentation and consideration of specific data intricacies for optimal model performance. It's important to iterate and evaluate the model's performance to find the optimal kernel size for your specific forecasting task. In the absence of known periodic intervals, we usually employ a multi-scale analysis strategy, which enables the model to capture features at different time scales without relying on prior knowledge of periodicity. We can also use cross-validation to assess the performance of different kernel sizes.

Reviewer comment (Novelty):
The model used in the paper consists of only two layers of convolutional networks, along with a decomposition of trend and periodic components, and uses the classical MSE loss function. There is a lack of innovation in the model design.

Author response:
Many thanks for your careful reading and suggestion. We appreciate that you found us some advantages. 

Meanwhile, we would like to provide some insights into the contribution of this paper. When the DLinear model [1] was initially introduced, its simple linear structure sparked considerable controversy. However, it ultimately won the Best Paper Award, defying complex Transformer models in the LTSF field. Subsequently, more works have been dedicated to refining and advancing linear models, including architectural modifications and innovative training methodologies. **We want to emphasize that our manuscript does not only aim to showcase the contributions of SOTA results. Our model structure is as simple as DLinear, while extensive experiments have consistently demonstrated that depth-wise convolutional units possess inherent advantages over linear units (such as DLinear, and RLinear) in LTSF.** Linear mappings often struggle to capture intricate dependencies when handling data with multiple periods among channels. Although modeling each channel independently can alleviate this issue, it will significantly increase the computational cost. More experimental analyses are described in Section 7, Page 9.  **Based on our basic unit, more complex networks can be proposed to further improve the prediction performance of such tasks. In the future, there is the potential for more valuable research to be explored in this new track.**

Reviewer comment (Experiments):
The paper lacks several important baseline models based on CNN architectures, such as SCINet and TimesNet, in the Conv-based model category. This is particularly severe, as all models in this category are proposed in this paper.

Author response:
Q1 : *The paper lacks several important baseline models based on CNN architectures, such as SCINet and TimesNet, in the Conv-based model category (this is particularly severe, as all models in this category are proposed in this paper). MLP-based models lack models like N-Hits, and there is also a lack of references to the aforementioned models.*
* Thank you for your comments. Due to space limitations, we compared representative SOTA models from recent years in original our manuscripts. Following the reviewer’s suggestion, **we have added SCINet, TimesNet, MICN, and N-Hits baseline models.** We adopt their official codes and only change the length of input sequences. The results are summarized in Table 2. In Table 2, we conducted an experiment within a broader window size range of \{96, 336, 512, 720, 1600\} for a fair comparison. **It can be found in the attached supplementary material file.**  The results demonstrate that LTSF-Conv models consistently surpass all SCINet, TimesNet, MICN, and N-Hits on seven LTSF benchmarks. The symbol * denotes the experimental results we conducted after increasing the input length (re-implementation). For N-Hits, authors have adopted an extended step size hyperparameter search, and we directly adopt the results. We will add the related references in the revised paper.

Reviewer comment (Experiments):
MLP-based models lack important baselines like N-Hits, and there is also a lack of references to the aforementioned models.

Author response:
Q1 : *The paper lacks several important baseline models based on CNN architectures, such as SCINet and TimesNet, in the Conv-based model category (this is particularly severe, as all models in this category are proposed in this paper). MLP-based models lack models like N-Hits, and there is also a lack of references to the aforementioned models.*
* Thank you for your comments. Due to space limitations, we compared representative SOTA models from recent years in original our manuscripts. Following the reviewer’s suggestion, **we have added SCINet, TimesNet, MICN, and N-Hits baseline models.** We adopt their official codes and only change the length of input sequences. The results are summarized in Table 2. In Table 2, we conducted an experiment within a broader window size range of \{96, 336, 512, 720, 1600\} for a fair comparison. **It can be found in the attached supplementary material file.**  The results demonstrate that LTSF-Conv models consistently surpass all SCINet, TimesNet, MICN, and N-Hits on seven LTSF benchmarks. The symbol * denotes the experimental results we conducted after increasing the input length (re-implementation). For N-Hits, authors have adopted an extended step size hyperparameter search, and we directly adopt the results. We will add the related references in the revised paper.

Reviewer comment (Experiments):
The discussion of "model performance with respect to lookback" in Section C.1 lacks the inclusion of DConv. Considering Table 1 and Table 2, which compare the model results, the "best" model used in the tables is actually the model with a lookback of 1600, which naturally performs better than other baseline models with smaller lookback values that have not reached their optimal states. Moreover, even the "best" model is outperformed by PatchTST, which does not have data with lookback values of 720, 1000, and 1600.

Author response:
Q2 : *Considering Table 1 and Table 2, which compare the model results, the "best" model used in the tables is actually the model with a lookback of 1600, which naturally performs better than other baseline models with smaller lookback values that have not reached their optimal states. Moreover, even the "best" model is outperformed by PatchTST, which does not have data with lookback values of 720, 1000, and 1600.*
* Thank you for pointing out this.  Previous research [1] has also shown that Transformer-based baselines have not benefited from a longer look-back window. Their performance fluctuates or gets worse as the input lengths increase. We have conducted experiments to analyze it. **The reason why most transformer models do not increase the lookback window length is because the performance will become worse.** More analysis of the look-back windows is described in Section C.1, Page 12. in the original Appendix. 
* We have confidence in the fairness of our previous comparison. **The experimental results not only include Conv-Best and DConv-Best but also the default look-back window length is set at 512, such as Conv and Dconv.** This is using the same lookback window length as PatchTST.
* Different from other Transformer-based baselines, PatchTST can extend the lookback window to 512, but does not benefit from longer windows. It's important to note that this extension comes at the cost of a substantial increase in the model’s computational resource requirements. On our server resources, PatchTST ran out of GPU memory for a look-back window size greater than 720.
* To alleviate your concerns, we switched to a higher-performance server to validate PatchTST with lookback-window size $\in$ \{720, 1600\}. Since our model only experiments from the extended window {336, 512, 720, 1600}, we ignore the 1000 step size. **The results are in Table 3 of the attached supplementary material file.** As expected, **PatchTST did not benefit from longer lookback windows. As the input length increases, the significant memory overhead poses limitations for its practical application.**

Reviewer comment (Presentation):
The quality of the figures illustrating the model is low and overly simplified.

Author response:
Q4 : *There are formatting issues with the caption of Figure 6. The quality of the figures illustrating the model is low and overly simplified.*
* Thank you so much for your careful check. We are sorry that we ignored the issues of font size on the vertical axis in Figure 6. We have revised it. Following the reviewer’s suggestion, we have made modifications to Section 4.1 of the original manuscript to provide a more detailed explanation of the model.

Reviewer comment (Presentation):
There are formatting issues with the caption of Figure 6.

Author response:
Q4 : *There are formatting issues with the caption of Figure 6. The quality of the figures illustrating the model is low and overly simplified.*
* Thank you so much for your careful check. We are sorry that we ignored the issues of font size on the vertical axis in Figure 6. We have revised it. Following the reviewer’s suggestion, we have made modifications to Section 4.1 of the original manuscript to provide a more detailed explanation of the model.

Reviewer comment (Experiments):
Whether the Conv-LTSF still works for time series that does not exhibit strong periodicity?

Author response:
*  Many thanks for your careful reading and suggestion. Following the reviewer’s suggestion, we compared the proposed model (Conv) with the DLinear and strongest Transformer baseline (PatchTST) on the illness benchmark. The illness benchmark describes the ratio of patients seen with illness and the number of patients from 2002 to 2021. It includes weekly data from the Centers for Disease Control and Prevention of the United States (https://gis.cdc.gov/grasp/fluview/fluportaldashboard.html).  Table 4 (in the attached PDF file) provides the corresponding forecasting results. It can be found that Conv outperforms DLinear and PatchTST for all horizons. 

| Methods   |Conv|      | PatchTST|      | DLinear|      |
|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
| Metric    | MSE|MAE| MSE| MAE  | MSE     | MAE  |
| 24        | 0.451 | 0.552| 0.637| 0.552| 0.725   | 0.681|
| 36        | 0.508 | 0.574| 0.765| 0.634| 0.792   | 0.744|
| 48        | 0.597 | 0.636| 0.756 | 0.692| 0.886   | 0.815|
| 60        | 0.675 | 0.680| 0.776 | 0.741| 0.960   | 0.859|
| Avg       | **0.558** | **0.611** | 0.734 | 0.655 | 0.841 | 0.775 |

**Table 4**: The forecasting results on the ILI benchmark under the setting, where sl = 104.

Reviewer comment (Experiments):
I wonder whether explicitly considering the channel dependencies can help further improve the forecasting performance.

Author response:
* In time series forecasting tasks, we intuitively think that channel-mixing techniques are often a superior approach, which allows the model to effectively capture the interdependencies and interactions between different channels. In channel dependencies, information from various channels is integrated efficiently, facilitating a more comprehensive representation of the complex relationships within the data. However, previous studies [2] have found that channel independence can improve performance compared to channel mixing in LTSF.
* CI: so-called channel independence; CD: so-called channel dependencie. Following the reviewer’s suggestion, we apply channel independence and channel dependence to the different models respectively. Table 5 (in the attached PDF file) summarizes the results of different channel strategies on the Weather and ETTm1 datasets. **Most of the CI-based models have a higher testing accuracy than the CD-based models.** For MLP-based models, the overall accuracy of the CI-based models was approximately 1\%–8\% higher than the CD-based models.
* For our models, we have conducted additional ablation experiments about depth-wise CNN (channel independent) and general CNN (channel dependent). The results are in Section C.2, Page 13 of the original manuscript. It can be found that CI-based models also outperform CD-based models.


|  |  | |  | Weather  |  |  | |  | ETTm1  |  |  |
|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
|  |  | 96 | 192 | 336 | 720 | Avg | 96 | 192 | 336 | 720 | Avg |
| RLinear_CD | MSE | 0.175 | 0.217 | 0.265 | 0.328 | 0.246 | 0.301 | 0.34 | 0.373 | 0.43 | 0.361 |
|  | MAE | 0.225 | 0.259 | 0.293 | 0.339 | 0.279 | 0.342 | 0.366 | 0.385 | 0.417 | 0.377 |
| RLinear_CI | MSE | 0.146 | 0.189 | 0.241 | 0.313 | **0.222** | 0.289 | 0.332 | 0.368 | 0.426 | **0.353** |
|  | MAE | 0.194 | 0.234 | 0.274 | 0.327 | **0.257** | 0.335 | 0.361 | 0.38 | 0.413 | **0.372** |
| DLinear_CD | MSE | 0.175 | 0.219 | 0.265 | 0.323 | 0.245 | 0.299 | 0.335 | 0.369 | 0.424 | 0.356 |
|  | MAE | 0.237 | 0.282 | 0.318 | 0.361 | 0.299 | 0.343 | 0.365 | 0.386 | 0.42 | 0.378 |
| DLinear_CI | MSE | 0.146 | 0.19 | 0.243 | 0.317 | **0.224** | 0.285 | 0.327 | 0.367 | 0.428 | **0.351** |
|  | MAE | 0.212 | 0.257 | 0.301 | 0.358 | **0.282** | 0.334 | 0.358 | 0.383 | 0.417 | **0.373** |
| NLinear_CD | MSE | 0.181 | 0.225 | 0.27 | 0.339 | 0.253 | 0.305 | 0.348 | 0.375 | 0.433 | 0.365 |
|  | MAE | 0.232 | 0.268 | 0.3 | 0.348 | 0.287 | 0.347 | 0.375 | 0.388 | 0.421 | 0.382 |
| NLinear_CI | MSE | 0.146 | 0.189 | 0.242 | 0.321 | **0.224** | 0.293 | 0.337 | 0.379 | 0.435 | **0.361** |
|  | MAE | 0.196 | 0.238 | 0.28 | 0.335 | **0.262** | 0.341 | 0.367 | 0.39 | 0.422 | **0.38** |

**Table 5**: Multivariate prediction results on two benchmarks with an input length of 336. CI denotes channel-independence, and CD represents channel-dependence.