Paper: Long-term Time Series Forecasting with Vision Transformer (ICLR-2024)

Reviewer comment (Presentation):
The paper is somewhat not well-organized. The figures (1, 2, and 4) are not rough to convey enough insights and are unportable. The tables (2, 3, and 4) for many experiments are incomplete on the dataset. Some notations can be not so rigorous (\hat_x may be better to denote the model predictions). Thus, this paper essentially needs further polishing.

Author response:
### **For the 1st concern of weaknesses**
  The reviewer mentioned that Figures 1, 2, and 4 do not clearly convey insights. We do not quite understand. Do you have any clearer suggestions? For Tables 2, 3, and 4. Table 2 shows the results of univariate prediction, for which **we only evaluated on the 4 ETT datasets. This is because in the field of time series prediction, univariate prediction is not the main challenge, and conventionally it is evaluated only on these 4 datasets or less**. Anyway, we evaluated Swin4TS on the other four datasets as shown below (evaluated by MSE), and the results showed that Swin4TS still outperformed other baselines.
|             | Swin4TS/CI | PatchTST/64 |  MICN  | TimesNet | FEDformer | Autoformer |
|:-----------:|:----------:|:-----------:|:------:|:--------:|:---------:|------------|
|   Weather   |    0.002   |    0.002    | 0.013  |  0.002   |   0.014   |   0.058    |
|   Traffic   |    0.134   |    0.193    | 0.285  |  0.167   |   0.236   |   0.271    |
| Electricity |    0.277   |    0.709    | 0.364  |  0.422   |   0.373   |   0.421    |
|     ILI     |    0.756   |    0.749    | 2.658  |  0.818   |   1.079   |   1.079    |

However, this comparison is unfair to other baselines because they have not conducted the same experiments in their original work, and they may require appropriate adjustments in hyperparameters for these additional datasets to achieve optimal performance. Table 3 is the ablation analysis for ETTm1 and ETTm2. We also conducted ablation experiments on Swin4TS/CI on four other datasets, and in most cases, the effectiveness was confirmed. Please understand that due to limited time, we were unable to complete all the ablation experiments.
|           | Weather | Traffic | Electricity |  ILI  |
|:---------:|:-------:|:-------:|:-----------:|:-----:|
| with all  |  0.220  |  0.356  |    0.157    | 1.740 |
| w/o shift |  0.222  |  0.355  |    0.157    | 1.751 |
| w/o scale |  0.228  |  0.358  |    0.165    | 1.812 |
|  w/o both |  0.225  |  0.359  |    0.162    | 1.787 |

**It should be noted that presenting obvious ablation effects for all datasets is demanding. At least for time series prediction, it is customary to present ablation results only on some datasets**. Table 4 is about the experiment on hierarchy design. We do not take the 4-stage experiment for the ILI dataset. This is because the input length of ILI is 108 which is unreasonable to design a 4-stage architecture, as we have mentioned in the main text. **Overall, results of this work seem incomplete, primarily due to some customary practices rather than our work being incomplete. Hope the reviewer can consider again.**

Reviewer comment (Novelty):
The contributions of the paper can be not enough, which is the most fatal point. Swin-transformer modules are adapted for the time series. It's like lowering the dimension of attention directly on 1D time series representations. Is this more motivating than design models considering time series properties? How does inductive bias like shift invariance help time series forecasting? Without considering the gap between the two modalities, I think it has not contributed much to the research field.

Author response:
### **For the 2nd concern**
   We would like to reiterate the motivation behind our work. Swin Transformer is essentially a Transformer that takes into account information at multiple scales (i.e., hierarchical design) and combines window-based attention to reduce complexity to linear. Although it was originally proposed to solve CV problems, it is essentially designed for handling sequential data. Time series is naturally sequential data, while image is partitioned into patch series. Therefore, from this perspective, Swin Transformer is even more suitable for handling time series.

   **We want to emphasize that the core architecture of Swin Transformer is independent of CV tasks**. It can be applied to a wide range of tasks for modeling sequential data. A good example of this is the Video Swin Transformer (VST, https://arxiv.org/abs/2106.13230), which actually deals with image-based time series. In fact, VST has not made specific designs for video data, but simply extends Swin Transformer from 2D to 3D, and has achieved SOTA performance on different tasks. **If each frame of video data is reduced to a point or vector, then video data becomes time series. This means that VST already implies an attempt to process time series data with Swin Transformer. The main contribution of our work is showing the structural similarity of time series and image, making these two modalities be modeled within a unified framework**.

  Indeed, time series and images have their own characteristics. Or, as you mentioned, there is a gap between these two modalities. **However, the characteristics of these two modalities can be integrated  to their own underlying patterns. As long as Swin Transformer can learn these underlying patterns, there is no need to design too much specifically for their own features. As you know, ViTs do not have strong inductive bias like CNNs. But they still can get superior performance by training on large datasets**. For example the shift invariance in image you mentioned. Due to that the partition of image in ViT destroys the space structure, ViT uses a positional embedding to learn the space structure. For time series analysis, at least for forecasting task, the shift invariance is not obvious. Thus the positional embedding and the relative bias in Swin Transformer do not facilitate the performance in practice. While for time series classification task, we expect to see the shift invariance may help the model performance.

   **Anyway, thank you for pointing out this. Your concerns made us realize that we did not clearly articulate our motivation and contribution in the previous manuscript. In the Conclusion section of the revised one, we added a short discussion to comprehensively address our views. Please kindly consider again.**

Reviewer comment (Experiments):
The performance incremental is marginal. The forecasting result of Swin4TS over PatchTST can be small. And I think the ablations should be conducted on more extensive datasets. Also, I suggest that experiments such as hyperparameter effects are more suitable to be placed in the appendix, and more experiments exploring the helpful inductive bias should be considered.

Author response:
### **For the 3rd concern** 
   Regarding the performance incremental. **The SOTA for time series prediction has reached a bottleneck**. As far as we know, very recent works nearly all have limited improvements for PatchTST. Regarding the effectiveness of Swin4TS, we have additionally provided more experimental results to validate the feasibility and effectiveness of Swin4TS, such as TNT4TS and hierarchical analysis. We hope that these results will meet your requirements. Regarding the necessity of using backbones from other fields. As stated in our main text, the significance of our work lies in the recognition of the structural similarity between the time series and image modalities, and modeling them in a unified framework. **Therefore, time series modeling can draw on advanced models from the image modality to solve its own problems, instead of designing various sophisticated models as before. This will promote the development of time series analysis**. Finally, the reviewer suggested more helpful inductive bias to prove the rationality of Swin4TS. This is back to the 2nd concern mentioned above, which we have stated.

Reviewer comment (Experiments):
Unsolved contribution: "We successfully apply techniques from ViT to LTSF", which does not "indicate the potential of connecting time series with other domains within the Transformer architecture". What does the 'connection' mean? The author provides no experiments to validate the results of cross-domain/modality transfer.

Author response:
###  **For the 4th concern**
  Our wording may have been misleading. **Our intention was to use a unified model to model time series and image patch series based on their structural similarities, rather than providing contributions for multi-modality task**. In the revised manuscript, we expressed it in a different way.

Reviewer comment (Theory):
SwinTransformer in computer vision are commonly leveraged for high-level feature extraction, which is widely applied on tasks such as image classification. The time series forecasting task, on the contrary, belongs to a typical generative task and needs to learn low-level feature representation for time point reconstruction. How does the proposed model cope with the gap between tasks and representation granularity?

Author response:
###  **For the 1st question**
  Regarding the gap. When Swin Transformer is applied for image classification, it first obtains a representation of the image, and then uses this representation for classification by a linear layer. Similarly, Swin4TS obtains a representation of the input time series and then directly connects a linear layer to map the future time series. **The main difference is just that the former is 2-dimensional and the latter is 1-dimensional**.

    Regarding the representation granularity. Since existing time series datasets are far less complete than image datasets, large models may overfit. The size of Swin4TS is smaller than Swin4CV, which is mainly reflected in the smaller numbers of attention heads, layers, and stages and dimension of hidden vectors. However, **this does not mean that the underlying pattern of time series is essentially low-level. The patterns of time series data, such as meteorological data, may be very complex**, especially considering the interrelationships between multiple time series.

Reviewer comment (Novelty):
Is it an advantage that being able to combine CI/CD mechanisms? Previous forecasting models are commonly implemented by CD, but they can also be easily incorporated with CI, no matter Transformers or non-Transformers.

Author response:
### **For the 2nd question** 
  Previous prediction models, such as Transformer, Autoformer, and Fedformer, can indeed be compatible with CI/CD. However, these models just fuse multiple channels with an embedding layer and do not explicitly consider the correlations between them, and thus the complexity of these models is not relevant to the number of channels. **Swin4TS, on the other hand, can be compatible with both CI/CD strategies and can explicitly consider the correlations between multiple variables under CD strategy**.

Reviewer comment (Experiments):
The current time series forecasting datasets are pretty small, and performance may be satuated or over-fitting. Could the method be used for larger datasets?

Author response:
&emsp;&emsp; Regarding the issue of dataset size, **most existing baselines for time-series prediction use the 8 datasets mentioned in the text as benchmarks. We selected them in order to have a fair comparison with other baselines**. However, as you mentioned, these datasets are relatively small and there are possible distributional shifts between the training and validation sets, which is a problem faced by all existing baseline models.

&emsp;&emsp; In our collaboration project with a wind farm, we applied Swin4TS to predict wind power. The training dataset contained wind power data for 14 wind turbines, recorded every 15 minutes over a span of 5 years, comprising approximately 175k timesteps, far larger than the aforementioned 8 datasets. The goal was to predict wind power for the next 24 hours (96 points) and 72 hours (288 points), with an input size of 288. We compared three models, PatchTST, DLinear, and LightGBM. The prediction results are shown below (the results are in terms of normalized MSE).

|            |     Swin4TS/CI    |     Swin4TS/CD    |     PatchTST    |     DLinear    |     LightGBM    |
|:------------:|:-------------------:|:-------------------:|:-----------------:|:----------------:|:-----------------:|
|     96     |     0.472         |     0.468         |     0.512       |     0.587      |     0.502       |
|     288    |     0.632         |     0.652         |     0.633       |     0.781      |     0.649       |

&emsp;&emsp; Swin4TS has performed well in predicting 96-point and 288-point. Unfortunately, due to commercial cooperation agreements, we cannot open this dataset. **We believe that Swin4TS's superior architecture is the reason for its better performance rather than the accidental fit to specific datasets, and we expect it to perform well on other large datasets as well**.

Reviewer comment (Novelty):
Scalformer [1] also uses hierarchical design and the scales of time series data, this paper didn't mention and compare the similarities and differences with Scalformer. [1] Shabani, Amin, et al. "Scaleformer: iterative multi-scale refining transformers for time series forecasting." ICLR (2023).

Author response:
&emsp;&emsp; Thanks for reminding us of the Scaleformer. Scaleformer and Swin4TS both use a hierarchy design, but their architectures are very different. Firstly, Scaleformer uses an external hierarchy design to obtain predictions at multiple scales, while Swin4TS uses an internal hierarchy design to concentrate information of different scales in the final representation and obtain the final prediction. Secondly, the prediction module in Scaleformer is totally independent, and Scaleformer is compatible with various transformer-based models. However, Swin4TS is a complete model, making it difficult to directly compare the performance with Scaleformer. Due to length limitations, we have added a short comment on Scaleformer in “Related Work” of the revised manuscript.

Reviewer comment (Experiments):
The current time series forecasting datasets are pretty small, and performance may be satuated or over-fitting. Could the method be used for larger datasets?

Author response:
&emsp;&emsp; Regarding the issue of dataset size, **most existing baselines for time-series prediction use the 8 datasets mentioned in the text as benchmarks. We selected them in order to have a fair comparison with other baselines**. However, as you mentioned, these datasets are relatively small and there are possible distributional shifts between the training and validation sets, which is a problem faced by all existing baseline models.

&emsp;&emsp; In our collaboration project with a wind farm, we applied Swin4TS to predict wind power. The training dataset contained wind power data for 14 wind turbines, recorded every 15 minutes over a span of 5 years, comprising approximately 175k timesteps, far larger than the aforementioned 8 datasets. The goal was to predict wind power for the next 24 hours (96 points) and 72 hours (288 points), with an input size of 288. We compared three models, PatchTST, DLinear, and LightGBM. The prediction results are shown below (the results are in terms of normalized MSE).

|            |     Swin4TS/CI    |     Swin4TS/CD    |     PatchTST    |     DLinear    |     LightGBM    |
|:------------:|:-------------------:|:-------------------:|:-----------------:|:----------------:|:-----------------:|
|     96     |     0.472         |     0.468         |     0.512       |     0.587      |     0.502       |
|     288    |     0.632         |     0.652         |     0.633       |     0.781      |     0.649       |

&emsp;&emsp; Swin4TS has performed well in predicting 96-point and 288-point. Unfortunately, due to commercial cooperation agreements, we cannot open this dataset. **We believe that Swin4TS's superior architecture is the reason for its better performance rather than the accidental fit to specific datasets, and we expect it to perform well on other large datasets as well**.

Reviewer comment (Novelty):
Scalformer [1] also uses hierarchical design and the scales of time series data, Could this paper compare the similarities and differences with Scalformer. [1] Shabani, Amin, et al. "Scaleformer: iterative multi-scale refining transformers for time series forecasting." ICLR (2023).

Author response:
&emsp;&emsp; Thanks for reminding us of the Scaleformer. Scaleformer and Swin4TS both use a hierarchy design, but their architectures are very different. Firstly, Scaleformer uses an external hierarchy design to obtain predictions at multiple scales, while Swin4TS uses an internal hierarchy design to concentrate information of different scales in the final representation and obtain the final prediction. Secondly, the prediction module in Scaleformer is totally independent, and Scaleformer is compatible with various transformer-based models. However, Swin4TS is a complete model, making it difficult to directly compare the performance with Scaleformer. Due to length limitations, we have added a short comment on Scaleformer in “Related Work” of the revised manuscript.

Reviewer comment (Experiments):
The paper does not compare to non-transformer based baselines which are computationally more efficient.

Author response:
&emsp;&emsp;**In fact, we have compared three non-transformer methods:** DLinear (MLP-based), TimesNet (CNN-based), and MICN. Please see the description in the “Compare baseline” section.

Reviewer comment (Theory):
The ordering of the series should matter for the CD model which is not a desirable property.

Author response:
&emsp;&emsp;Indeed, under the CD strategy, multiple time series can not be treated as an image, as the order of the time series can vary. In our code implementation, the order of the time series is shuffled in each training batch, and we found that this trick yielded better results than fixed initial order. We provided additional clarification on this point in the main text, and relevant experiments presented in the Appendix C.3.

Reviewer comment (Writing):
The last layer of the CD model is not fully clear.

Author response:
&emsp;&emsp; The final layer of Swin4TS/CD is directly mapped to MT (M is the number of series, and T is the prediction length) after learning a representation using Swin Transformer. **As you might know, this is not a good choice because for the 720-prediction task on the Traffic dataset, the size of final layer would be 862x720. This is also why the prediction performance on Traffic (M=862) and Electricity (M=321) is relatively poor under the CD strategy.** A reasonable way to address this would be to design a U-net architecture and gradually obtain MT through upsampling on the representation. **We added a part in Appendix D to address the effectiveness of this modified framework**. However, to ensure the algorithm’s simplicity and consistency with the CI strategy, we still chose the simpler one as mentioned above.  We did not want to complicate the main architecture because based on the motivation of this work, we hoped to demonstrate the structural similarity between the two modalities of time series and image and that they can be modeled using the same framework.

Reviewer comment (Evaluation):
The model performance is not monotonically increasing with size of context in some corner cases which is not the case with other SOTA models.

Author response:
&emsp;&emsp;  Our results show that prediction performance varies non-monotonically with input length. **In fact, PatchTST has also reported similar phenomena, as shown in Table 9.** For example, for the 720-prediction task on ETTh2, 336-input leads to 0.379-MSE, 720-input leads to 0.394-MSE. **Autoformer and FEDformer exhibit similar phenomena, as shown in Figure 2 of the PatchTST‘s paper**. One possible explanation is that the input data contains both underlying patterns about the time series as well as noise. Once the input length is sufficient to capture the underlying patterns, longer input will introduce more noise and harm prediction performance.

Reviewer comment (Experiments):
Is the ablation of varying hierarchical design done in the correct manner? As I understand more hierarchies also mean more layers? Is it not better to keep the number of layers the same and just have hierarchy in one model while the scales are fixed in all layers in the ablation one?

Author response:
&emsp;&emsp; In our work, we gave the analysis of the hierarchy design in two parts. The first part is the Ablation Study in section 4.2 Results, where **the hierarchy design was ablated in the manner just suggested by the reviewer**, as described in the main text. The second part is the varying hierarchical design in section 4.3 Effect of Hyperparameter (we moved this part in Appendix), which is not ablation of the hierarchy design, but rather an examination of the impact of different hierarchy architectures on the model. Please note the distinction.

Reviewer comment (Reproducibility):
What would be the strategy to handle static and dynamic covariates?

Author response:
&emsp;&emsp; Thanks for your suggestion. We have attempted to integrate dynamic covariates (the timestamp of the samples) into the input via embedding. **Surprisingly, there was a significant improvement in predictive performance on the Traffic dataset!** However, the effect was not observed on other datasets, which may be due to the Traffic dataset’s natural periodicity. We have incorporated this trick into the code, and added some descriptions of it in the Appendix C.6, and update the prediction results in Table 1.

Reviewer comment (Experiments):
For vision models contiguous patches in the window are visually close in the image. This is not the case for the channel dimension in the CD time-series case. In other words does the order of the time-series matter? It should not matter in the ideal case. Can the authors show this experimentally?

Author response:
&emsp;&emsp;Indeed, under the CD strategy, multiple time series can not be treated as an image, as the order of the time series can vary. In our code implementation, the order of the time series is shuffled in each training batch, and we found that this trick yielded better results than fixed initial order. We provided additional clarification on this point in the main text, and relevant experiments presented in the Appendix C.3.

Reviewer comment (Evaluation):
It seems that for some horizon lengths in Figure 6 the performance decreases a little bit with increasing context. This does not seem to be the case for PatchTST and TiDE. Any intuition on why this might be happening?

Author response:
&emsp;&emsp;  Our results show that prediction performance varies non-monotonically with input length. **In fact, PatchTST has also reported similar phenomena, as shown in Table 9.** For example, for the 720-prediction task on ETTh2, 336-input leads to 0.379-MSE, 720-input leads to 0.394-MSE. **Autoformer and FEDformer exhibit similar phenomena, as shown in Figure 2 of the PatchTST‘s paper**. One possible explanation is that the input data contains both underlying patterns about the time series as well as noise. Once the input length is sufficient to capture the underlying patterns, longer input will introduce more noise and harm prediction performance.

Reviewer comment (Novelty):
Novelty: The main idea of the paper is derived from the existing Swin transformer idea, which slightly weakens the novelty of the paper.

Author response:
**For the novelty**. We borrowed the idea of Swin Transformer for modeling time series, and indeed, the novelty is not particularly strong if only considering this. **However, our contribution is not only proposing an effective method for modeling time series. More importantly, we consider the structural similarity between image (partitioned into patch series) and time series, which indicates the possibilities of modeling time series modality using architectures from image modality. We additionally supplemented a TNT4TS framework to support our view.** The reviewer can refer the detailed response to Reviewer 9ZZq and SUuV, and we hope the reviewer can reconsider the contribution of our work.

Reviewer comment (Evaluation):
The confidence intervals are missing from the MSE and MAE values in the results tables.

Author response:
Regarding the issue of the confidence interval for MSE and MAE that you mentioned, our main experiment setup uses a fixed random number, which is conventionally set to the number of current year. Therefore, MSE and MAE are the results of a single experiment, and there is no need to calculate the confidence interval. Of course, we have studied the impact of random seed (i.e. different initialization) on the model's performance, and the relevant results are in Appendix C.1. Since the results are presented in histogram form, there is no calculation of the confidence interval, please understand.

Reviewer comment (Experiments):
What is the strategy for selecting the hyper-parameters of the architecture? Were the hyper-parameters selected for each dataset individually using the validation set? Or were the hyper-parameters selected globally for all the problems simultaneously?

Author response:
For the selection of hyperparameters. We make similar datasets adopt the same hyperparameters, such as the 4 ETT datasets. Furthermore, different datasets may require different hierarchical designs, and the setting of hyper-parameters should ensure that the input length can be divisible during the downsampling process. In addition, other hyperparameters will reference the settings of other models, such as learning rate or early stop. Different datasets, due to their own properties, usually have different parameter sets. For example, in Traffic and Weather, the former’s periodicity is more obvious than the latter, and the interaction between multiple variables is also different.