Paper: CARD: Channel Aligned Robust Blend Transformer for Time Series Forecasting (ICLR-2024)

Reviewer comment (Experiments):
The paper analyzed the complexity of the proposed model, which is an advantage. However, this paper did not specifically outline the complexities of the baseline models and there were no direct comparisons with the baselines on the running time or efficiency.

Author response:
## Q1:Model complexities/Running time
As we use the patching trick, the order of the total complexity would be the same as PatchTST and Crossformer, which is $\mathcal{O}(L^2/S^2)$. Some Transformer type models (e.g., FEDformer and Autoformer) may even break the quadratic dependent in $L$ and reach linear or nearly linear complexity in $L$. Other CNN and RNN type models (e.g., TimesNet and FilM) by nature maintain the $O(L)$ complexity.  When $S$ is set as a not-very-small number (e.g. $S\approx O(\sqrt{L})$), our model's complexity can also be nearly linear. The condition $S\approx O(\sqrt{L})$ is not very restrictive. Take $L = 900$ as an example, the length of $S=30$ would be enough. For the case $L = 96$, the corresponding $S$ would be around $10$.

The results of the detailed experiment on running time are reported in the following table. We use the TimesNet's codebase. The input/forecasting lengths are set as 96/96 and we keep the batch size the same for all benchmarks and run the experiments on a single A100/80G GPU. Our proposed model yields comparable running time to transformer baselines as well as linear complexity baselines except Dlinear, which implies in practice model could also behave like a linear time model and won't introduce overhead computational cost. 

|                      |           | Card   | autoformer | patchTST | crossformer | fedformer | timesnet | micn   | Dlinear | FilM   | ETSformer |
| -------------------- | --------- | ------ | ---------- | -------- | ----------- | --------- | -------- | ------ | ------- | ------ | --------- |
| hidden=16,batch=128  | train     | 0.0197 | 0.1091     | 0.0164   | 0.2512      | 0.2107    | 0.0672   | 0.0423 | 0.0074  | 0.0747 | 0.0714    |
| etth1                | inference | 0.0046 | 0.0102     | 0.0021   | 0.0127      | 0.0178    | 0.0132   | 0.0030 | 0.0009  | 0.0123 | 0.0061    |
| hidden=128,batch=128 | train     | 0.0779 | 0.1525     | 0.0785   | 0.1186      | 0.2189    | 0.2457   | 0.0613 | 0.0330  | oom    | 0.1354    |
| Weather              | inference | 0.0048 | 0.0139     | 0.0036   | 0.0123      | 0.0378    | 0.0224   | 0.0038 | 0.0014  | oom    | 0.0092    |
| hidden=128,batch=32  | train     | 0.2156 | 0.0835     | 0.3280   | oom         | 0.1903    | oom      | 0.0405 | 0.0163  | oom    | 0.1174    |
| ECL                  | inference | 0.0064 | 0.0160     | 0.0052   | oom         | 0.0349    | oom      | 0.0045 | 0.0021  | oom    | 0.0103    |
| hidden=128,batch=12  | train     | 0.2271 | 0.0960     | 0.1329   | oom         | 0.1649    | 0.6048   | 0.0322 | 0.0139  | oom    | 0.0607    |
| Traffic              | inference | 0.0101 | 0.0153     | 0.0052   | oom         | 0.0369    | 0.0418   | 0.0074 | 0.0058  | oom    | 0.0223    |

Reviewer comment (Experiments):
In the ablation study of the signal decay-based loss function, i.e. Table 12 and Table 13, I observe that the method proposed in this paper does not exhibit a significant advantage compared to the SOTA baseline without the robust loss. Furthermore, it would be better if the authors could provide test results on a broader range of datasets.

Author response:
## Q3:Model performance without Robust loss 
As depicted in the subsequent table, the model itself is capable of achieving state-of-the-art (SOTA) performance using the original MSE loss. The introduction of the robust loss proposed in this paper further enhances the performance. Moreover, it is noteworthy that for the traffic dataset, the original MSE loss yields significantly better results. Therefore, it is important to highlight that the primary driver behind the observed improvements is the proposed model itself, rather than relying solely on the choice of the loss function.


|          | ETTm2 |       | ETTh2 |       | ECL   |       | Traffic |       |
| ------ | ----- | ----- | ----- | ----- | ----- | ----- | ------- | ----- |
|       | MSE       | MAE   | MSE   | MAE   | MSE   | MAE   |MSE   | MAE   |
| 96           | 0.171 | 0.256 | 0.284 | 0.338 | 0.146 | 0.244 | 0.400   | 0.275 |
| 192          | 0.241 | 0.302 | 0.373 | 0.391 | 0.164 | 0.260 | 0.426   | 0.288 |
| 336          | 0.302 | 0.340 | 0.421 | 0.430 | 0.181 | 0.278 | 0.439   | 0.290 |
| 720          | 0.398 | 0.397 | 0.423 | 0.442 | 0.215 | 0.305 | 0.475   | 0.312 |
| avg          | 0.278 | 0.324 | 0.375 | 0.400 | 0.176 | 0.272 | 0.435   | 0.291 |
| avg_with_robustloss | 0.271 | 0.316 | 0.367 | 0.390 | 0.169 | 0.258 | 0.450    | 0.278 |
| sec_best_avg | 0.283 | 0.327 | 0.483 | 0.406 | 0.187 | 0.295 | 0.488   | 0.316 |

Reviewer comment (Theory):
The paper claims that the proposed method can effectively mitigate overfitting issues. Although some empirical analysis has been provided, supplementing with appropriate theoretical analysis would be more convincing.

Author response:
## Q4:Theoretical analysis for mitigating overfitting issues

Thanks for your suggestion. In this paper, we mainly introduce the exponential smoothing (EMA) module and weighted robust loss to mitigate the overfitting. 

We believe the EMA part would use a weighted mean to average out some noise. We may use the following simple example to illustrate it. Let $\{x_1,....,x_{t}\}$ be sequences of i.i.d $\sigma^2$ sub-Gaussian variables and Let $f(t) = (1-\beta)\sum_{i=1}^{t}\beta^{t-i}x_i$ with some $\beta\in (0,1)$. One may verify $f(t) = (1-\beta)x_{t} + \beta f(t-1)$, which implies $\{f(t)\}$ is the sequence of $\{x_t\}$ after EMA. Via the Hoeffiding's inequality, there exist some constant $C>0$ such that with $1-\delta$, we have:

\begin{align}
    |f(t)-f(t-1)| \le \sigma\sqrt{(1-\beta)^2\left(1+\beta^2\right)C^{-1}\log\left(\frac{1}{\delta}\right)}.
\end{align}

If we set $\beta \to 0$, $f(t)\to x_t$ and above inequality shows the divergence between $x_t$ and $x_{t-1}$ would be $O(\sigma)$. When change $\beta$ toward $1$, the divergence decreased on the order of $O(\sigma^2\sqrt{1-\beta)^2(1+\beta^2)})$, which is faster than $O(\sigma)$. It implies that if the sequence $\{x_t\}$ contains high-frequency noise, the smoothed sequence $f(t)$ will have a higher chance of wiping it out and it eventually stabilizes the attention module. We will add those discussions into the final version.

Another design to tackle the overfitting is the weighted loss.  In Appendix H of the paper, we use toy case with input length 2 to show the proposed loss may give better estimation accuracy. 


However,we have to admit that performing end-to-end theoretical analysis for the proposed deep model can be very complicated and beyond the scope of this paper.

Reviewer comment (Experiments):
The article directly applies a fixed-parameter Exponential Moving Average (EMA) without providing specific explanations regarding the potential drawbacks of using a learnable-parameter EMA. The advantages or disadvantages of employing a learnable-parameter EMA are not thoroughly explored.

Author response:
## Q1: Learnabe decay weight VS Fixed decay weight
The results of the learnable decay weight experiment are presented in the following table. The numerical outcomes are found to be very similar to those obtained with the fixed decay weight. However, it is important to note that the training speed could be significantly slower when using the learnable decay weight requiring up to 50\% extra training time. Due to this time-consuming aspect, we have chosen to utilize a fixed decay weight in our approach.

| ETTm1 | Learnable |       | Fixed |       | Diff  |        |
| ----- | --------- | ----- | ----- | ----- | ----- | ------ |
|       | MSE       | MAE   | MSE   | MAE   | MSE   | MAE    |
| 96    | 0.321     | 0.345 | 0.316 | 0.347 | 0.005 | -0.002 |
| 192   | 0.363     | 0.368 | 0.363 | 0.370 | 0.000 | -0.002 |
| 336   | 0.394     | 0.390 | 0.392 | 0.390 | 0.002 | 0.000  |
| 720   | 0.464     | 0.428 | 0.458 | 0.425 | 0.006 | 0.003  |
| avg   | 0.386     | 0.383 | 0.382 | 0.383 | 0.003 | 0.000  |

| ETTm2 | Learnable |       | Fixed |       | Diff   |        |
| ----- | --------- | ----- | ----- | ----- | ------ | ------ |
|       | MSE       | MAE   | MSE   | MAE   | MSE    | MAE    |
| 96    | 0.169     | 0.247 | 0.169 | 0.248 | 0.000  | -0.001 |
| 192   | 0.232     | 0.289 | 0.234 | 0.292 | -0.002 | -0.003 |
| 336   | 0.292     | 0.328 | 0.294 | 0.339 | -0.002 | -0.011 |
| 720   | 0.390     | 0.388 | 0.390 | 0.388 | 0.000  | 0.000  |
| avg   | 0.271     | 0.313 | 0.272 | 0.317 | -0.001 | -0.004 |

| ETTh1 | Learnable |       | Fixed |       | Diff  |       |
| ----- | --------- | ----- | ----- | ----- | ----- | ----- |
|       | MSE       | MAE   | MSE   | MAE   | MSE   | MAE   |
| 96    | 0.384     | 0.391 | 0.383 | 0.391 | 0.001 | 0.000 |
| 192   | 0.438     | 0.422 | 0.435 | 0.420 | 0.003 | 0.002 |
| 336   | 0.482     | 0.444 | 0.479 | 0.442 | 0.003 | 0.002 |
| 720   | 0.475     | 0.463 | 0.471 | 0.461 | 0.004 | 0.002 |
| avg   | 0.445     | 0.430 | 0.442 | 0.429 | 0.003 | 0.002 |

| ETTh2 | Learnable |       | Fixed |       | Diff   |        |
| ----- | --------- | ----- | ----- | ----- | ------ | ------ |
|       | MSE       | MAE   | MSE   | MAE   | MSE    | MAE    |
| 96    | 0.285     | 0.331 | 0.281 | 0.330 | 0.004  | 0.001  |
| 192   | 0.362     | 0.381 | 0.363 | 0.381 | -0.001 | 0.000  |
| 336   | 0.411     | 0.417 | 0.411 | 0.418 | 0.000  | -0.001 |
| 720   | 0.414     | 0.432 | 0.416 | 0.431 | -0.002 | 0.001  |
| avg   | 0.368     | 0.390 | 0.368 | 0.390 | 0.000  | 0.000  |

| Weather | Learnable |       | Fixed |       | Diff   |        |
| ------- | --------- | ----- | ----- | ----- | ------ | ------ |
|         | MSE       | MAE   | MSE   | MAE   | MSE    | MAE    |
| 96      | 0.153     | 0.206 | 0.150 | 0.188 | 0.003  | 0.018  |
| 192     | 0.207     | 0.249 | 0.202 | 0.238 | 0.005  | 0.011  |
| 336     | 0.258     | 0.287 | 0.260 | 0.282 | -0.002 | 0.005  |
| 720     | 0.345     | 0.338 | 0.343 | 0.353 | 0.002  | -0.015 |
| avg     | 0.241     | 0.270 | 0.239 | 0.265 | 0.002  | 0.005  |

Reviewer comment (Evaluation):
While the article highlights the third point of short-term forecasting, the visual results presented in the figures do not seem to align well with the data shown in the tables. Additionally, the evaluation of short-term forecasting appears to be limited to only the M4 dataset, which may not be sufficient to fully demonstrate the model's performance.

Author response:
## Q2: About the short-term forecasting M4 benchmark
Following the discussion with reviewer ajLB on short-term forecasting using the M4 dataset, we have decided to relocate this section to the supplementary information as an initial exploration. In exchange, we will incorporate relevant ablation study findings from the supplementary information into the main draft. We concur with reviewer eTmJ's assessment that relying solely on the M4 dataset for short-term forecasting is inadequate, especially considering the issue of different input lengths for baseline models. We acknowledge that our incorporation of the settings proposed in the recent Timesnet paper lacked a deeper analysis. It is crucial to recognize that as the M4 dataset is a competition dataset, most participants leverage ensemble methods to enhance their performance. However, introducing ensemble methods would significantly complicate the problem and deviate from our main research focus. Additionally, we acknowledge that certain baseline models in the M4 dataset employ different input lengths due to the competition's lack of requirement for uniformity. To strengthen the soundness of our findings, we intend to explore alternative datasets for the short-term forecasting task, an aspect we plan to address in our future work.

Reviewer comment (Novelty):
The proposed approach needs more novelty. Some of the main design ideas are presented and verified by the previous work, such as channel independence, patchify, EMA, and dynamic projection technique.

Author response:
In our paper, we did not utilize channel independence. Instead, our primary objective is to develop a transformer model that effectively harnesses channel-dependent (CD) information. This focus is stated in the abstract, where we introduce the CARD model's channel-aligned attention structure that aims to capture dependencies among multiple variables over time, and other parts of the paper also echo this statement (e.g., summary of contribution, Figure 1, and Section 3.3). 

The core innovation of our paper lies in addressing an important research question: how to construct an efficient transformer for CD settings. In existing literature, CD transformers, such as Autoformer and FEDformer, generally underperform compared to channel-independent (CI) transformers like PatchTST. Various experiments by Han et al., 2023 suggest that enhancing the robustness of CD transformers is crucial. We therefore introduced the EMA module to improve the model's stability and considered a robust loss design. Additionally, to maintain reasonable computational cost and align the information within patch, dynamic projection and dual attention are introduced respectively. 
While we agree with the reviewer that similar structures may have been explored in deep learning literature, we believe such a combination is new in CD transformer design in the time series forecasting domain. Our focus is not on claiming the invention of these structures but on their effective combination to improve CD transformer performance, which we consider a significant novelty of our research.

Moreover, we want to highlight our robust loss. In recent literature (e.g., Autoformer, FEDformer, and PatchTST), the major point is trying to design new model structures but not focusing on training procedures. In this paper, we analyze the variance-varying issue in multi-step forecasting and propose a weighted loss. This relatively simple modification not only enhances the performance of our model but also improves other models, as demonstrated in Table 4. To our best knowledge, this loss design is also new in recent transformer time series forecasting literature.

Reviewer comment (Experiments):
The main experimental setting needs to be consistent with the SOTA method to be convincing, and the experimental results need more justification.

Author response:
Perhaps we didn't clarify it enough, which may have led to confusion for reviewer gMyA. We want to highlight that there are two distinct experimental settings for long-term forecasting benchmarks in our study. The first approach, used by Autoformer, FEDformer, Timesnet, etc., involves a fixed input length of 96 and forecasting for all horizons. These models compare accuracy across different algorithms, and we refer to this approach as "fixed short input". The complete table of results for this particular setting can be found in Appendix E, specifically in Table 8.

On the other hand, models like PatchTST, Dlinear, Crossformer, and some other baselines focus more on the final result by using a much longer input length. Some of them even tune the input length to achieve better performance. As the reviewer correctly noted, PatchTST reports results for input lengths of both 336 and 512 in their main table. However, it is worth mentioning that PatchTST also presents results for an input length of 720 in its paper appendix table 9, although it is not as good as the 336 and 512 cases. 

We refer to them as the "better performance" approach. The complete table of results for this particular setting can be found in Appendix F, specifically in Table 10.

We believe that both approaches are valid and contribute to testing different aspects of the algorithms. One approach evaluates the forecasting ability with a limited input length, while the other tests the ability to achieve better performance using longer inputs. Therefore, we have included results for both settings. The reason for using an input length of 720 is that we achieved better performance in this hard setting length, whereas PatchTST's performance slightly deteriorated. However, we didn't want to lower the reported performance of the baseline models, so we included their best performance setting which was for an input length of 512, to align with the "better performance" approach. 

We hope that this clarification makes our approach clearer.  If reviewer gMyA still prefers the results on 336/512 input length, we are also more than happy to include them in the final version.

Reviewer comment (Experiments):
Channel independence introduces a significant computational overhead. To mitigate this issue, the authors employ dynamic projection (DP) to project head dimensions from d_head to a fixed \( r \) with \( r \ll C \). They demonstrate through ablation experiments involving projected dimensions of 1, 8, and 16 on ETT datasets that DP does not compromise forecasting accuracy. However, ETT datasets only consist of 7 channels, which may not be sufficient to draw a definitive conclusion. It remains unclear how this technique would perform in datasets with a larger number of variables, such as Traffic (862) or ECL (321).

Author response:
Thanks for your suggestion. We've added the results of Weather, ECL, and Traffic datasets under the same setting in Appendix N.2. The projecteddimension varies in 1,8,16. The results are summarized in the following table. We have observed similar performance behavior, suggesting that altering the dynamic projection dimension does not significantly impact the forecasting accuracy. 

| Weather | MSE   | MAE   | MSE   | MAE   | MSE   | MAE   | MSE   | MAE   |
| ------- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- |
|Proj Dim| 1     | 1     | 8     | 8     | 16    | 16    | 32    | 32    |
| 96      | 0.152 | 0.189 | 0.150 | 0.188 | 0.147 | 0.186 | 0.151 | 0.189 |
| 192     | 0.204 | 0.240 | 0.202 | 0.238 | 0.201 | 0.237 | 0.201 | 0.238 |
| 336     | 0.263 | 0.284 | 0.260 | 0.282 | 0.262 | 0.283 | 0.259 | 0.282 |
| 720     | 0.343 | 0.353 | 0.343 | 0.353 | 0.341 | 0.352 | 0.339 | 0.350 |
| avg     | 0.241 | 0.266 | 0.239 | 0.265 | 0.238 | 0.264 | 0.238 | 0.265 |

| ECL      | MSE   | MAE   | MSE   | MAE   | MSE   | MAE   | MSE   | MAE   |
| -------- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- |
| Proj Dim | 1     | 1     | 8     | 8     | 16    | 16    | 32    | 32    |
| 96       | 0.142 | 0.234 | 0.141 | 0.233 | 0.140 | 0.232 | 0.141 | 0.233 |
| 192      | 0.165 | 0.256 | 0.160 | 0.250 | 0.157 | 0.247 | 0.158 | 0.248 |
| 336      | 0.180 | 0.272 | 0.173 | 0.263 | 0.168 | 0.256 | 0.171 | 0.261 |
| 720      | 0.209 | 0.298 | 0.197 | 0.284 | 0.187 | 0.273 | 0.196 | 0.281 |
| avg      | 0.174 | 0.265 | 0.168 | 0.258 | 0.163 | 0.252 | 0.166 | 0.255 |

| Traffic  | MSE   | MAE   | MSE   | MAE   | MSE   | MAE   |
| -------- | ----- | ----- | ----- | ----- | ----- | ----- |
| Proj Dim | 1     | 1     | 8     | 8     | 16    | 16    |
| 96       | 0.413 | 0.274 | 0.419 | 0.269 | 0.416 | 0.270 |
| 192      | 0.447 | 0.273 | 0.443 | 0.276 | 0.435 | 0.268 |
| 336      | 0.462 | 0.278 | 0.460 | 0.283 | 0.447 | 0.275 |
| 720      | 0.490 | 0.299 | 0.490 | 0.299 | 0.482 | 0.294 |
| avg      | 0.453 | 0.281 | 0.453 | 0.282 | 0.445 | 0.277 |

Reviewer comment (Experiments):
In ablation experiments, the primary focus is on evaluating the effectiveness of the new design between CARD and PatchTST. Instead of solely changing parameters, it is important to demonstrate the method's rationality and validity with greater certainty through more detailed ablation studies.

Author response:
From the model structure perspective, our model has two major differences: attention over hidden and channel-aligned modules, which are trying to address the PatchTST potential two drawbacks: ignorance of the cross-channel information and lack of usage within patch information. In Table 18, we examine the performance changes when removing those two structures and verify the effectiveness of those two parts. Moreover, we have also included our ablation analysis for architecture variants in the appendix, specifically in Figure N.1, which visually demonstrates the differences. If they are more engaging for the readers, we can certainly move this analysis to the main draft. The only reason we did not include it here initially was due to limited space constraints.

Reviewer comment (Experiments):
In experiments with less training data, using only 70% of the training data samples compared to the standard training setting for long-term forecasting may not be sufficient.

Author response:
The purpose of this experiment is to challenge the claim made in Dlinear that the transformer model cannot benefit from more data. We believe this claim to be an overstatement, as we demonstrate that even a slight increase in dataset size, from 70\% to 100\%, can result in performance improvement. Although we could show even larger improvements with an even smaller dataset size, we deliberately avoid getting into a debate about whether a 10\% data size is too small and leads to underfitting the model. We acknowledge that these findings are preliminary. To truly test whether the transformer model follows a power law accuracy curve, we would need a much larger dataset. Our goal is simply to highlight that the claim made in Dlinear is underexplored at the very least, and we hope to encourage further investigation within the research community.

Reviewer comment (Experiments):
The paper makes same methodological mistake as the previous line of this research. We really need to compare against basic baselines! Please include results of Zeng et al. in Table 1. Today, this is perhaps the most important baseline to include in the Table, so I'd rather not see some of the other entries in the table than to see this one missing.

Author response:
We highly concur with the reviewer's suggestion that certain early baselines have not been thoroughly examined. Notably, algorithms like nbeats and even persistent models exhibit excellent performance across certain datasets. The reason for excluding them in this paper lies solely in the limitation of available space, as we were apprehensive that some reviewers may raise concerns regarding the omission of recent works. We have run the experiment on Nlinear, Linear, and Repeat models in our experimental settings and summarized the results in the following table. Our model consistently outperforms those baselines.  We assure you that we will incorporate these baselines in a comprehensive table, presented in smaller font size, in the refined version of our manuscript.

 | Dataset  | ETTm1   | ETTm2   | ETTh1   | ETTh2   | Wea.   | ECL   | Traffic | 
 | -------- | ----- | ----- | ----- | ----- | ----- | ----- | ---|
 | -------- | MSE | MSE | MSE | MSE | MSE | MSE | MSE|
 | Nlinear      | 0.429    | 0.291     | 0.602     | 0.413  |0.281   | 0.234    | 0.641    |
 | Linear       | 0.432    |0.374     | 0.625     | 0.649   |0.271  | 0.224    | 0.633 |
 | Repeat     | 1.269    | 0.385    |1.321      | 0.536      | 0.353 | 1.612 |  2.770| 
 | CARD      | 0.383 | 0.271 | 0.443 | 0.367 | 0.242 | 0.169 |0.450|

Reviewer comment (Experiments):
The results in Table 2 are not as impressive as in Table 1. The model is not performing well compared to the first entry in the M4 competition and is barely performing better than the second entry, which is not reflected in Table 2. Again, same methodological mistake of not comparing comprehensively against basic baselines. Please include them in the table if you want to demonstrate methodological rigour and promote consistency across studies. The N-BEATS results are probably presented in the table without ensembling, but even the basic ensemble of 18 models is way better than the proposed model as follows from Figure 3 in https://arxiv.org/pdf/1905.10437.pdf. I suggest that the authors follow well accepted protocol for M4 benchmark evaluation and present the ensemble-based results, or remove Table 2 from the paper and replace it with the results of their ablation studies.

Author response:
After careful consideration, we align with the reviewer's evaluation. In order to test the M4 dataset, we have incorporated the setting proposed in the recent Timesnet paper. However, it is worth noting that as M4 is a competition dataset, most participants employ ensemble methods to enhance performance. Notably, the Nbeats algorithm has exhibited exceptional performance on this dataset, because of its 180 models ensembling trick. The original paper introduces structural and parameter variations to diversify the base model, optimizing it for ensembling. The success of ensembling depends on two crucial factors: the ensembling method itself and the choice of baseline algorithms. It is important to note that different ensembling methods can be better suited for different datasets and may facilitate varying baseline algorithms. Although incorporating such a comparison would undoubtedly introduce complexity, we acknowledge the need to rerun and construct an ensemble framework for all baseline models, including the card method proposed in our study. However, it is important to note that even with this comparison, ensuring absolute fairness may not be guaranteed, as a particular ensemble method might inherently favor a specific baseline model. We recognize that this may deviate from our original narrative, as a model that excels in ensembling may not necessarily outperform others on its own. Ensembling relies heavily on the diversity among baseline models. Additionally, as raised by another reviewer, eTmJ, it is important to acknowledge that certain baseline models in the M4 dataset utilize different input lengths due to the competition's lack of requirement for uniform input lengths. Consequently, we believe your suggestion is highly fitting and thoughtful. We propose relocating this portion to the supplementary information, treating it as a preliminary exploration, and highlighting the significantly improved performance of algorithms like Nbeats when utilized within an ensemble framework.

Reviewer comment (Experiments):
Please add results on ILI and Exchange to Table 1.

Author response:
We have included their results in the following table. Although we did run these two datasets and found that the card's performance was good, we were hesitant to consider them as suitable benchmarks. The ILI dataset is both small and highly diverse, with normalized predicted MSE values close to 2. We are uncertain whether the relatively good performance on such a dataset can be extrapolated to larger and more practical datasets in real-world scenarios. Similarly, when it comes to the Exchange dataset, its nature as a financial dataset governed by free-market principles makes it extremely challenging to make accurate predictions. The signal-to-noise ratio in this dataset is significantly low, posing challenges for short-term forecasting tasks, and rendering long-term forecasting even more arduous, if not seemingly impossible. For example, all baselines in TimesNet paper fail to beat even the simple repeat forecasting reported in Zeng et al.  2023. In the final version, we will include them in the main table. 


|     | exchange |       |     | illness |       |
| --- | -------- | ----- | --- | ------- | ----- |
| Output length | mse | mae | Output length | mse | mae |
| 96  | 0.084    | 0.202 | 24  | 1.665   | 0.803 |
| 192 | 0.174    | 0.295 | 36  | 2.200   | 0.890 |
| 336 | 0.323    | 0.295 | 48  | 1.875   | 0.821 |
| 720 | 0.841    | 0.689 | 60  | 1.923   | 0.853 |
| avg | 0.356    | 0.371 | avg | 1.916   | 0.842 |