Paper: ModernTCN: A Modern Pure Convolution Structure for General Time Series Analysis (ICLR-2024)

Reviewer comment (Theory):
Although the primary claim for performance improvement centers on enlarging effective receptive fields, the paper lacks theoretical analysis as the authors acknowledge.

Author response:
> **Weaknesses1 :** The paper lacks theoretical analysis on effective receptive fields (ERFs)

+ Thanks for pointing out this issue. In current manuscript, we can only provide the theoretical analysis on ERF of our ModernTCN, but not on ERFs of other baselines.  As mentioned in $\underline{\text{section 5.2}}$, the classic theory of ERFs [1] is designed for pure convolution networks. But most of the convolution-based models in time series community are not pure convolution structures. They only use convolution as part of their designs. Therefore, the classic theory of ERFs may not suitable for those baselines.

+ Under such condition, we adopt the visualization of ERFs as an alternative to theoretical analysis for it can provide intuitive comparision of both pure and non-pure convolution-based models.

+ It will be our future work to further study the ERFs of the non-pure convolution-based models and find better theory to explain the ERFs of those models.

Reviewer comment (Presentation):
Some description details are not sufficiently clear. For example, in Table 4, it would be good to clarify which module the authors used as the ‘undecoupled ConvFFN’.

Author response:
> **Weaknesses2 :** Some description details are not sufficiently clear. For example, in Table 4, it would be good to clarify which module the authors used as the ‘undecoupled ConvFFN’.

+ Thanks for your valuable suggestion. We provide more details about our ablation study in $\underline{\text{Appendix H in revised paper}}$. And in $\underline{\text{Appendix H}}$, we introduce all modules we used in our ablation study with $\underline{\text{Figure 10}}$. Please refer to it for more details.

Reviewer comment (Experiments):
In the experiments on regression tasks, it is concerned whether the RevIN (2021) normalization (as detailed in Appendix B.2) has a significant impact on the performance improvement. Therefore, an additional ablation study about RevIN is requested to enhance clarity. This request is based on the observation that RevIN normalization alone led to improvements in both qualitative and quantitative performance in some of the baseline models.

Author response:
> **Q1 :** Ablation study about RevIN on regression tasks

+ Thanks for your valuable suggestion. We conduct ablation study about RevIN on four regression tasks with 6 models to study the impact of RevIN. Results are provided in $\underline{\text{Appendix L in revised paper}}$. The 6 models are ModernTCN, PatchTST, TimesNet, SCINet and two new baselines: RLinear and RMLP, which all include RevIN as part of their structure.

+ Some conclusions are summarized briefly as follows:
    + Although there is a slight degradation in performance without RevIN, our ModernTCN still achieves competitive performance in four regression tasks, indicating that our designs in ModernTCN also make great contribution to the performance improvement.
    + We also find that **RevIN doesn't provide consistent improvement under some forecasting settings**, which means our ModernTCN can directly predict from the non-stationary time series to some degree. This finding further confirms that our designs in ModernTCN can bring great performance improvement on their own.
    + RevIN's impact on different models: Although removing RevIN will cause performance degradation on all 6 models in our experiments, **our ModernTCN is one of the less influenced models**, indicating that our ModernTCN is robust to the usage of RevIN. And we suppose that the extent of RevIN's influence on the model is related to the model's mechanisms. 
    + RevIN's impact on different tasks: We find that short-term forecasting tasks are more sensitive to RevIN.

Reviewer comment (Experiments):
While the paper asserts the claim of enlarging the effective receptive field (ERF) as the key to using convolution in time series analysis in section 5.2, it would be beneficial to include an additional visualization of the ERF of TimesNet (2022) in Figure 1 to further substantiate the claim. TimesNet is another convolution-based model that utilizes shorter kernel sizes and more layers within a block, and it exhibits lower performance than ModernTCN.

Author response:
> **Q2 a:** it would be beneficial to include an additional visualization of the ERF of TimesNet in Figure 1.

+ Thanks for your valuable suggestion. As we introduced in $\underline{\text{Section 2.1}}$ and $\underline{\text{Section 5.1 'Compared with TimesNet'}}$, TimesNet is special in the family of convolution-based models.
It transforms 1D time series into 2D-variations and applies 2D convolution on it, which is quite different from other convolution-based models that mainly use 1D convolution.
Since TimesNet takes a very different path to use convolution in time series analysis, we don't compare the ERF of TimesNet with other 1D convolution-based models in the first version of our manuscript.

+ Following your valuable suggestion, we visualize the ERF of TimesNet and we find that the property of TimesNet' ERF is quite different from other 1D convolution-based models'.
Therefore, we put the visualization result of TimesNet in $\underline{\text{Appendix I in revised paper}}$, along with a detailed comparision of the ERFs between TimesNet and ModernTCN.
One of the conclusions is that the additional data transformation in TimesNet can also influence the ERF, which leads to the special property of TimesNet’s ERF. Please refer to $\underline{\text{Appendix I}}$ for more details.

> **Q2 b:** Compare two kinds of convolution setting: ‘smaller kernels + more layers’ or ‘larger kernel + less layers’

+ Since the ERF of TimesNet is also influenced by the 2D data transformation, comparing the ERFs between TimesNet and ModernTCN cannot directly tell which kind of convolution setting can provide larger ERF.

+ Instead, we provide $\underline{\text{Figure 1 (a) and (b)}}$ for this comparision. These two visualization results are all based on the same ModernTCN structure but with the two different convolution settings, therefore can providing a fairer comparision.
As shown in $\underline{\text{Figure 1 (a) and (b)}}$, ‘larger kernel + less layers’ can provide larger ERF than ‘smaller kernels + more layers’, which is consistent with the theoretical analysis in $\underline{\text{section 5.2}}$.

Reviewer comment (Writing):
Simple errata: Enlaring -> Enlarging (in section 5.2.)

Author response:
> **Q3 :** Simple errata: Enlaring -> Enlarging (in section 5.2.)

* Thanks for your reminder. We have fixed it.

Reviewer comment (Writing):
The details of the blocks (DWConv and CovnFFN) should be introduced clearly (Some readers may not have related background Knowledge).

Author response:
+ Thanks for your valuable suggestion. We provide more background knowledge and details of ModernTCN block in $\underline{\text{Appendix G in revised paper}}$.

Reviewer comment (Experiments):
The performance improvement is marginal. In addition, there is another MLP-based work beating PatchTST in same cases, which should be compared with. Li, Zhe, et al. "Revisiting Long-term Time Series Forecasting: An Investigation on Linear Mapping." arXiv preprint arXiv:2305.10721 (2023).

Author response:
We would like to illustrate our performance improvement from following points:
1. ModernTCN achieves the consistent state-of-the-art performance on five mainstream time series analysis tasks:
    + ModernTCN surpasses lots of advanced time series models and task-specific methods.    
    + ModernTCN shows great generality. In details, ModernTCN ranks top-1 in all five tasks while other competitors will fall short in some tasks ($\underline{\text{Figure 3 left}}$).

2. ModernTCN provides a better balance of efficiency and performance:
    + Apart from achieving state-of-the-art performance, ModernTCN also maintains great efficiency. ($\underline{\text{Figure 3 right}}$).

3. Compared with popular Transformer-based models:     
    + ModernTCN can compete favorably with the best Transformer-based model (PatchTST) in forecasting tasks and surpass it in imputation, classification and anomaly detection tasks. In details, ModernTCN gains 40.9% relative improvement in imputation tasks. And ModernTCN also improves the accuracy in classification from 72.5% to 74.2% and improves the F1-score in anomaly detection from 84.82% to 86.62%.   
    + When compared with other Transformer-based models like FEDformer, the improvement is more significant. ModernTCN can outperform it even in forecasting tasks (with 32.4% relative improvement in long-term forecasting and 8.9% relative improvement in short-term forecasting).   
    + Meanwhile, ModernTCN has more advantage in efficiency than Transformer-based models.

4. Compared with popular MLP-based models:  
    + ModernTCN outperforms all MLP-based baselines (including newly added RMLP and RLinear) in all five tasks.

5. Compared with recent convolution-based models:   
    + ModernTCN outperforms SCINet and MICN in all five tasks. For example, ModernTCN gains 27.4% relative improvement in long-term forecasting and 32.8% relative improvement in imputation. And ModernTCN also improves the accuracy in classification from 71.7% to 74.2%.   
    + TimesNet also demonstrates excellent generality in five mainstream tasks. But ModernTCN still outperforms TimesNet in all five tasks with better efficiency, especially gaining great improvements in long-term forecasting and imputation tasks.

* Thanks for your valuable suggestion. We add RLinear and RMLP as our baselines. The new results are updated in $\underline{\text{Figure 3,4}}$ and $\underline{\text{Table 1,2,3,27,28,29,30,31}}$.
+ ModernTCN achieves better performance than these two MLP-based models in all five tasks, especially achieving 45.1% relative improvement in imputation and improving the accuracy in classification from 70.6% to 74.2%.

Reviewer comment (Experiments):
Although it is said that "we also include the state-of-the-art models in each specific task as additional baselines for a comprehensive comparison", some SoTA models are missed in the paper, e.g., Short-term forecasting: Wang Xue, et al. "Make Transformer Great Again for Time Series Forecasting: Channel Aligned Robust Dual Transformer." arXiv preprint arXiv:2305.12095 (2023). Anomaly Detection: Yang, Yiyuan, et al. "DCdetector: Dual Attention Contrastive Representation Learning for Time Series Anomaly Detection." KDD 2023.

Author response:
+ Thanks for your valuable suggestion. We re-implement CARD based on the pseudo-code in the original paper and add it as a short-term forecasting baseline.
+ In original paper, CARD is trained with cosine learning rate decay and linear warm-up. For fair comparison, we remove this additional training scheme.
+ The new results are updated in $\underline{\text{Table 2, 28}}$. ModernTCN can outperform CARD in short-term forecasting.

+ Thanks for your valuable suggestion. We provide following table to compare DCdetector with ModernTCN in anomaly detection tasks. We calculate F1-score as metric. 
+ We align the experiment settings as follows:
    + We set the series length as 105.
    + We set the anomaly ratio as 0.5 in SMD dataset and 1 in others.

|Dataset|ModernTCN|DCdetector|
|:---:|:---:|:---:|
|MSL|84.92|95.21|
|PSM|97.23|97.16|
|SMAP|71.26|96.20|
|SMD|85.81|83.15|
|SWAT|93.86|96.41|

+ As shown in above Table, DCdetector outperforms ModernTCN in 3 of 5 cases and gains great improvement in SMAP and MSL datasets. Some analysis are provided as follows:
    + The anomaly detection tasks in our paper are based on the **classical reconstruction loss**, while DCdetector proposes a novel detection method based on **contrastive loss** and without reconstruction loss.
    + As mentioned in the origin paper of DCdetector, **reconstruction-based methods will encounter great challenge in time series anomaly detection and may not be the best solution for time series anomaly detection.**
    + In contrast, DCdetector provides a better framework for time series anomaly detection which can alleviate the problem faced by reconstruction-based methods.
 In our paper, **our purpose is to compare the generality of time series backbones and thus only choose the classical reconstruction loss for a fair comparision.**
 Given the great improvement of contrastive learning in time series anomaly tasks, **it is our future work to study how to combine ModernTCN with this excellent contrastive-based detection framework.**

Reviewer comment (Experiments):
How about the short-term forecasting results on the multivariate time series datasets (e.g., the datasets used for long-term forecasting in the paper)?

Author response:
+ Thanks for your valuable suggestion. We conduct the experiments and results are reporeted in $\underline{\text{Appendix K in revised paper}}$. 
+ Experiment details are as follows:
    + We choose the prediction lengths as {$6, 12, 18$}, which meets the prediction length in M4 datasets. And following M4 short-term forecasting tasks, we set input length to be 2 times of prediction length.
    + We choose models that perform well in multivariate datasets (DLinear, RLinear, RMLP and PatchTST) or perform well in short-term forecasting tasks (PatchTST, CARD and TimesNet) as strong baselines. 
    + All models follow their official configurations on above datasets, we only change the input lengths and prediction lengths. 
    + We calculate MSE and MAE of the multivariate prediction results as metric.
+ As shown in $\underline{\text{Appendix K}}$, ModernTCN can outperform above competitive baselines in most cases, ranking top-1 in 49 of 54 cases.

Reviewer comment (Reproducibility):
The code is not available for reproducibility.

Author response:
+ We have already provided experimental details and model settings in $\underline{\text{Appendix A,B,C in original paper}}$. And details about tensor shape and model structure are also included in $\underline{\text{main text in our paper}}$. We also add a **Reproducibility Statement** in $\underline{\text{revised paper}}$. **We guarantee to make the code public upon paper acceptance.**

Reviewer comment (Experiments):
It remains unclear how the proposed ModernTCN block would connect to existing literature. While pure convolution architecture is a sweet spot of efficiency and performance, based on what is claimed in the paper, it would be expected that the way to incorporate cross-variable information with ConvFFN would be useful on top of other variable-independent approaches such as PatchTST and DLinear. But this is missing in the current manuscript.

Author response:
> **Weaknesses1 & Q2 :** Apply the cross-variable component to variable-independent models like DLinear and PatchTST.

+ Thanks for your valuable suggestion. Using cross-variable components to exploit cross-variable information is orthogonal to the designs that focus on capturing temporal information.
So we can apply our cross-variable component to DLinear and PatchTST. 

+ We conduct the experiments in imputation tasks and provide results in $\underline{\text{Appendix J in revised paper}}$.
In conclusion, applying the cross-variable component can bring averaged 28.7\% promotion on PatchTST and 19.7\% promotion on DLinear in imputation tasks,
which validates that our cross-variable component can be used on top of other variable-independent models and improve their performance.

Reviewer comment (Experiments):
What does *Discard Variable Dimension* refer to in Table 4? In Table 4, it is shown that discarding variable dimensions leads to a significant performance degradation. However, in Appendix E, it is shown that univariate forecasting performance is still quite competitive. So what is the difference between the two?

Author response:
> **Q1a :** What does Discard Variable Dimension refer to in Table 4?

+ In $\underline{\text{section 3.2}}$, we design the variable-independent embedding to maintain the variable dimension, replacing the common variable-mixing embedding which will lead to the **discard of variable dimension** in the embedded series. 

+ The essential difference behind these two embedding methods is the opinion of whether we should maintain the variable dimension when analysing multivariate time series.

+ **Since applying the variable-mixing embedding to the multivariate time series will discard its variable dimension, we denote it as the setting 'Discard variable dimension'.**

+ To verifies our opinion to maintain the variable dimension, we use the setting 'Discard variable dimension' for comparision.
 The setting 'Discard variable dimension' leads to significant performance degradation, which verifies the necessity and effectiveness of our design to maintain the variable dimension.

+ We provide $\underline{\text{Appendix H in revised paper}}\$ to introduce the details of all settings in the ablation study. Please refer to it for more details of setting 'Discard variable dimension'.

> **Q1b:** What is the difference between Discard Variable Dimension in Table 4 and univariate forecasting in Appendix E?

+ The former is a multivariate forecasting task. The input is multivariate time series which owes to have a variable dimension to store the information of each variable. And we also need to capture cross-variable information in this multivariate task.
 But in the setting of 'Discard variable dimension', we apply variable-mixing embedding to the multivariate time series and discard its variable dimension, **leading to the loss of variable information and making it unable to further capture the cross-variable dependency**. As a result, the setting 'Discard variable dimension' leads to a significant performance degradation.
 
+ The latter is a univariate forecasting task. The input is univariate time series which doesn't need a variable dimension.
 Since it's a univariate tasks, we mainly focus on capturing cross-time information and **don't need to capture the cross-variable information**.
 Thanks to the larger ERF and better temporal modeling ability in DWConv, our ModernTCN can achieve competitive performance in univariate forecasting tasks.

+ Thanks for pointing out this issue, we rephrase the sentence in $\underline{\text{Appendix E in revised paper}}\$ to avoid confusion.

Reviewer comment (Experiments):
From Table 4, one important piece of ModernTCN is to exploit cross-variable information. However, notice that some of the other models, e.g. DLinear are variable independent. So is the proposed modern TCN block orthogonal to other approaches and is it possible to apply it on top of variable-independent models like DLinear?

Author response:
> **Weaknesses1 & Q2 :** Apply the cross-variable component to variable-independent models like DLinear and PatchTST.

+ Thanks for your valuable suggestion. Using cross-variable components to exploit cross-variable information is orthogonal to the designs that focus on capturing temporal information.
So we can apply our cross-variable component to DLinear and PatchTST. 

+ We conduct the experiments in imputation tasks and provide results in $\underline{\text{Appendix J in revised paper}}$.
In conclusion, applying the cross-variable component can bring averaged 28.7\% promotion on PatchTST and 19.7\% promotion on DLinear in imputation tasks,
which validates that our cross-variable component can be used on top of other variable-independent models and improve their performance.

Reviewer comment (Reproducibility):
Based on the appendix, it seems that very different hyperparameters are used for different tasks, e.g. channel number, two kernel sizes, patching size and strides, number of TCN blocks, etc. What is your hyperparameter search space?

Author response:
> **Q3 a:**  What is the hyperparameter search space?

The search space is:
+ FFN ratio from {$1,2,4,8$}, number of layers from {$1,2,3$}, large kernel size from {$13,31,51,71$} (which are classic large kernel sizes in CV), and small kernel size is fixed as 5.
+ For patch size and stride, we search patch stride from {$4,8$} under two patching modes ($P=S$ and $P=2 \times S$). But in short-term forecasting tasks, some input lengths are very short, so we reduce the patch size and stride. And in imputation tasks, we don't use patching so as to avoid mixing the masked and un-maskded tokens.
+ For channel number, we search it based on different datasets:
    + For long-term forecasting and imputation datasets, the search space is {$32,64,128,256,512$}, which are the common values in these datasets.
    + For short-term M4 datasets, the search space is {$128,256,512,1024,2048$}.
    + For classification UEA datasets and anomaly detection datasets, we decide the channel number following the criterion used in TimesNet [1].

> **Q3 b:**  Different hyperparameters are used for different tasks.

We use different hyperparameters for different tasks for following reasons:
1. The datasets in different tasks may have different property. Thus the suitable hyperparameters like channel number and layer number may vary from different tasks. One evidence is that TimesNet also have different hyperparameters in different tasks according to the original paper [1].
2. The patching operation should be different in different tasks. For example, in short-term forecasting tasks, some input lengths are very short, so we need to reduce the patch size and stride. And in imputation tasks, we don't use patching so as to avoid mixing the masked and un-maskded tokens.

Reviewer comment (Novelty):
How is it different to use a projection layer (as in PatchTST) vs convolution as the stem layer?

Author response:
> **Q4 :** How is it different to use a projection layer (as in PatchTST) vs convolution as the stem layer?

+ These two methods are equivalent. Thus, we merge the segmenting and a Linear projection into a single convolution, which can make the code more concise. And the idea of using convolution to conduct the patching and embedding process is also adopted by some CV backbones like ViT [2] and MLP-mixer [3].
 
+ We conduct an ablation study on both patch-style embedding methods with ModernTCN, PatchTST and Crossformer. As the results shown in following table, these two patch-style embedding methods achieve almost the same performance, which is more evidence that the two methods are equivalent.

|ETTh1(mse/mae)|96|192|336|720|
|:---:|:---:|:---:|:---:|:---:|
|ModernTCN(conv)|0.368/0.394|0.405/0.413|0.391/0.412|0.450/0.461|
|ModernTCN(proj)|0.368/0.394|0.405/0.413|0.391/0.412|0.450/0.461|
|PatchTST(conv)|0.370/0.399|0.413/0.421|0.422/0.436|0.447/0.466|
|PatchTST(proj)|0.370/0.399|0.413/0.421|0.422/0.436|0.447/0.466|
|Crossformer(conv)|0.386/0.429|0.419/0.444|0.440/0.461|0.519/0.524|
|Crossformer(proj)|0.386/0.429|0.419/0.444|0.440/0.461|0.519/0.524|

|ETTm1(mse/mae)|96|192|336|720|
|:---:|:---:|:---:|:---:|:---:|
|ModernTCN(conv)|0.292/0.346|0.332/0.368|0.365/0.391|0.416/0.417|
|ModernTCN(proj)|0.292/0.346|0.332/0.368|0.365/0.391|0.416/0.417|
|PatchTST(conv)|0.292/0.343|0.332/0.369|0.364/0.392|0.418/0.421|
|PatchTST(proj)|0.290/0.342|0.332/0.369|0.366/0.392|0.416/0.420|
|Crossformer(conv)|0.316/0.373|0.375/0.410|0.431/0.442|0.598/0.545|
|Crossformer(proj)|0.316/0.373|0.377/0.411|0.431/0.442|0.600/0.547|

|Weather(mse/mae)|96|192|336|720|
|:---:|:---:|:---:|:---:|:---:|
|ModernTCN(conv)|0.149/0.200|0.196/0.245|0.238/0.277|0.314/0.334|
|ModernTCN(proj)|0.148/0.200|0.196/0.245|0.238/0.277|0.314/0.334|
|PatchTST(conv)|0.149/0.198|0.195/0.241|0.245/0.282|0.314/0.334|
|PatchTST(proj)|0.149/0.198|0.194/0.241|0.245/0.282|0.314/0.334|
|Crossformer(conv)|0.153/0.217| 0.197/0.269|0.252/0.311|0.318/0.363|
|Crossformer(proj)|0.153/0.217| 0.197/0.269|0.252/0.311|0.318/0.363|

> Reference

[1] Haixu Wu, et al. "Timesnet: Temporal 2d-variation modeling for general time series analysis."

[2] Alexey Dosovitskiy, et al. "An image is worth 16x16 words: Transformers for image recognition at scale."

[3] Ilya Tolstikhin, et al. "MLP-Mixer: An all-MLP Architecture for Vision."