Paper: MGTST: Multi-scale and Cross-channel Gated Transformer for Multivariate long-term time-series forecasting (ICLR-2024)

Reviewer comment (Theory):
The motivation and some claims are not convincing enough. The advantages and relations among the proposed modules are not clear.

Author response:
The primary objective of our work is to develop a long-term time-series forecasting model that takes into account a broader range of information. The multi-scale architecture primarily focuses on incorporating multi-scale information, while the gated mechanism specifically addresses cross-channel information. 

The motivation for multi-scale architecture is that the optimal segment length varies with different forecasting lengths. We apply the Crossformer model to the ETTh1 dataset and present our findings below. For this analysis, we disable the Multi-Scale architecture to focus solely on the effect of segment length.

**Forecasting length**&nbsp; &nbsp; **Segment length 6** &emsp; **Segment length 12** &emsp; **Segment length 24** 
***

96 &emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;0.398 &emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;0.397&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;0.470

192&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&ensp;0.649&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&ensp;0.502&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;0.423

336&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&ensp;0.941&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&ensp;0.598&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;0.473

720&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&ensp;0.787&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&ensp;0.765&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;0.509

Based on the results, it is worthwhile to consider incorporating patches with different segment lengths into a unified model and subsequently generating the final prediction by evaluating the performance of each segment length. 

On the other hand, our further study posits that the gated mechanism is more robust with the penetration of noise regardless of forecasting length, however, other attention mechanisms such as Cross-channel attention and Cross-channel embedding adopted in Crossformer and Autoformer will suffer from noise. To commence our investigation, we initially train each model on the ETTh1 dataset. Subsequently, during the inference stage, we introduce Gaussian noise to six out of the seven channels and proceed to evaluate the forecasting accuracy of the final channels. The ensuing outcomes are presented as follows:

**Forecasting Length**&nbsp; &nbsp;  **MGTST** **Crossformer** **Autoformer**
***

96 w/o noise &emsp; &emsp; &emsp; &ensp; 0.05215  0.06761 0.08748

96 w noise&emsp; &emsp; &emsp; &emsp; &emsp;0.05215 0.07246 0.10981

192 w/o noise &emsp; &emsp; &emsp; &nbsp; 0.06789 0.09117 0.13042

192 w noise&emsp; &emsp; &emsp; &emsp; &nbsp; 0.06789 0.10189 0.12685

336 w/o noise &emsp; &emsp; &emsp; &nbsp; 0.07802 0.18523 0.13585

336 w noise &emsp; &emsp; &emsp; &emsp; &nbsp;0.07801 0.10782 0.14064

720 w/o noise &emsp; &emsp; &emsp; &nbsp; 0.09770 0.53096 0.19737

720 w noise&emsp; &emsp; &emsp; &emsp; &nbsp; 0.09769 0.29152 0.11747
***

The results show rapid performance degradation of Crossformer and Autoformer in long-term forecasting (336,720) with noise. On the other hand, the performance of the MGTST model exhibits only minimal variations across all forecasting horizons, indicating the robustness of the gated mechanism.

Reviewer comment (Experiments):
Hyperparameter settings. The paper conducted a hyperparameter analysis in the ablation study and provided additional hyperparameter configurations in the appendix. How do the authors determine the hyperparameters? For instance, the group size varies significantly across different datasets. Was it selected using a validation set?

Author response:
We determine the optimal hyperparameter configurations based on the model performance on the validation set. The determination of the group size was influenced by the limitations imposed by our experimental setup. Specifically, the group size needed to be a divisor of the total number of channels in the dataset in order to ensure compatibility. In the case of the Electricity dataset, the channel number could only be evenly divided by 107 and 3, resulting in the observed variation in the group size.

Reviewer comment (Presentation):
The main conceptual diagram in the article is too hastily drawn and occupies too much space.

Author response:
To optimize the utilization of space and enhance efficiency, we have resized the main conceptual diagram and integrated the gate mechanism diagram within it.