Paper: TimeMixer: Decomposable Multiscale Mixing for Time Series Forecasting (ICLR-2024)

Reviewer comment (Experiments):
Absence of some important baselines — Because this paper is based on decomposition, I think the authors have to include methods based on decompositions [1] into baselines. Furthermore, (although they are not accepted to any conference,) it is beneficial to include methods based on MLP-Mixer [2,3]. I think that this absence makes the second and fourth strengths fade.

Author response:
Many thanks to Reviewer ixvB for providing the insightful review and comments.

> **Q1:** Add Scaleformer, MTS-Mixers, TSMixer in comparison.

In the original submission, we compared TimeMixer with 15 baselines. Following the reviewer's request, we have included Scaleformer, MTS-Mixers, and TSMixer into comparison.

As we stated in the $\underline{\text{Section 4 of main text}}$, we experiment with these baselines with their official code but align the hyperparameters to ensure a fair comparison. All the results have been included in the $\underline{\text{Appendix H of revised paper}}$. Here are some key experiments.

In addition to the quantitive results, showcase comparisons have also been added in $\underline{\text{Appendix J of revised paper}}$ for an intuitive comparison.

| Averaged from 4 prediction lengths (MSE \| MAE) | Scaleformer    | MTS-Mixer      | TSMixer        | TimeMixer              |
| ----------------------------------------------- | -------------- | -------------- | -------------- | ---------------------- |
| Weather                                         | 0.416 \| 0.423 | 0.258 \| 0.286 | 0.253 \| 0.304 | **0.240** \| **0.271** |
| Solar-Energy                                    | 0.323 \| 0.378 | 0.261 \| 0.300 | 0.280 \| 0.348 | **0.216** \| **0.280** |
| Electricity                                     | 0.203 \| 0.315 | 0.201 \| 0.293 | 0.220 \| 0.327 | **0.182** \| **0.272** |
| Traffic                                         | 0.578 \| 0.352 | 0.558 \| 0.375 | 0.546 \| 0.372 | **0.484** \| **0.297** |
| ETTh1                                           | 0.495 \| 0.488 | 0.482 \| 0.481 | 0.466 \| 0.467 | **0.447** \| **0.440** |
| ETTh2                                           | 0.451 \| 0.460 | 0.426 \| 0.435 | 0.394 \| 0.412 | **0.364** \| **0.395** |
| ETTm1                                           | 0.438 \| 0.440 | 0.415 \| 0.419 | 0.408 \| 0.416 | **0.381** \| **0.395** |
| ETTh2                                           | 0.303 \| 0.351 | 0.296 \| 0.342 | 0.290 \| 0.344 | **0.275** \| **0.323** |

Reviewer comment (Experiments):
Insufficient experiment in ablation studies — I think the biggest contribution of this paper, or the main reason for acceptance, is different-way processing (coarse-to-fine and fine-to-coarse). Although the thought leading to this asymmetric design is not quite intuitive, Table 5 provides empirical results. However, the used datasets are insufficient. Considering its importance, more datasets should be used so this design has to be tested more rigorously. Furthermore, because small \( M=1 \) is employed for short-term forecasting such as M4, M4 cannot prove the efficacy of the asymmetric design sufficiently. Finally, there isn't an ablation setting where seasonal is coarse-to-fine and the trend is fine-to-coarse. These insufficient ablation studies make the third strength vanish.

Author response:
> **Q2:** Complete the ablation studies to prove the efficacy of asymmetric design.

In the original submission, we conducted experiments on M4 and ETTh1, which are two representative datasets in long- and short-term forecasting. Thanks for the reviewer's suggestion, we have completed the ablations in all the benchmarks. Besides, we also add a new ablation setting, where we adopt the bottom-up mixing for trend and top-down mixing for the seasonal part.

**Overall, we have conducted 10 types of ablations in all 18 benchmarks.** Here are part of the main results. The complete results have been included in $\underline{\text{Table 5 and Table 15,16,17 of revised paper}}$. In all benchmarks, our official design performs best consistently, which can verify the efficacy of our asymmetric design.

It is also notable that completely reversing mixing directions for seasonal and trend parts leads to a serious performance drop. This may come from that the essential microscopic information in finer-scale seasons and macroscopic information in coarser-scale trends are ruined by unsuitable mixing approaches.

| Ablations                                                    | ETTm1 (Predict-336) MSE \|MAE | M4 SMAPE \|MASE \|OWA             | PEMS04 MAE \|MAPE \|RMSE        |
| ------------------------------------------------------------ | -------------------------------- | ------------------------------------ | ----------------------------------- |
| Bottom-up mixing for trend and top-down mixing for seasonal part | 0.412 \| 0.429                   | 13.012 \| 1.657 \| 0.954             | 22.27 \| 15.14 \| 34.67             |
| TimeMixer                                                    | **0.390** \| **0.404**           | **11.723** \| **1.559** \| **0.840** | **19.21** \| **12.53** \| **30.92** |

Reviewer comment (Theory):
While the empirical experiments, inclusive of the ablation study, provide evidence of the effectiveness of TimeMixer, the underlying reasons for its superior performance remain somewhat opaque. Specifically, when juxtaposed against competing models, it's not lucidly expounded how TimeMixer excels in capturing temporal variations. A deeper dive into this comparative analysis would have been enlightening. Introducing spectral analysis or similar methodologies might offer theoretical insights that bridge this understanding gap.

Author response:
> **Q1:** "A deeper dive into this comparative analysis would have been enlightening. Introducing spectral analysis or similar methodologies might offer theoretical insights that bridge this understanding gap."

Thanks for the reviewer's valuable suggestion. We have provided the spectral analysis in the $\underline{\text{Appendix I of revised paper}}$. It is observed that TimeMixer outperforms other baselines significantly in capturing different frequencies.

Reviewer comment (Novelty):
The paper seems to sidestep the exploration of alternative decomposition techniques that are prevalent in the domain. For instance, methods such as the Discrete Fourier Transformer (DFT) have been widely recognized in time series analysis. How does the TimeMixer's Past Decomposable Mixing distinguish itself fundamentally from these frequency-based techniques? A comparative discourse delving into the nuances between the proposed method and established frequency-based approaches would enrich the paper's content and address potential queries from the readership.

Author response:
> **Q2-part 1:** "How does TimeMixer's Past Decomposable Mixing fundamentally differentiate itself from frequency-based methods like the Discrete Fourier Transformer (DFT)?"

**(1) Ablations on decomposition methods**

In this paper, we adopt the moving-average-based season-trend decomposition, which is widely used in previous work, such as Autoformer, FEDformer and DLinear. Following the reviewer's suggestion, we also try the DFT-based decomposition as a substitute. Here we present two types of experiments.

- DFT-based high- and low-frequency decomposition: We treat the high-frequency part like the seasonal part in TimeMixer and the low-frequency part like the trend part.
- DFT-based season-trend decomposition: We replace the moving average with DFT-based seasonal part extraction, which the most significant frequencies are extracted as season. Then the rest is trend.

The results are shown as follows, which has also been included in $\underline{\text{Appendix G.2 of revised paper}}$.

| Ablation on decomposition methods               | ETTm1 (Predict-336) MSE \| MAE | M4 SMAPE \| MASE \| OWA | PEMS04 MAE \| MAPE \| RMSE |
| ----------------------------------------------- | ---------------------------------- | -------------------------- | ------------------------------- |
| DFT-based high- and low-frequency decomposition | 0.392 \| 0.404                     | 12.054 \| 1.632 \| 0.862   | 19.83 \| 12.74 \| 31.48         |
| DFT-based season-trend decomposition            | 0.383 \| 0.399                     | 11.673 \| 1.536 \| 0.824   | 18.91 \| 12.27 \| 29.47         |
| moving-average-based Season-trend decomposition | 0.390 \| 0.404                     | 11.723 \| 1.559 \| 0.840   | 19.21 \| 12.53 \| 30.92         |

We can have the following observations:

- Replacing the season-trend decomposition with high-low-frequency decomposition does not bring performance gain. Since we only explore the proper mixing approach for the former decomposition in the paper, the bottom-up and top-down mixing strategies may be not suitable for high- and low-frequency parts. New visualizations like $\underline{\text{Figure 3 and 4 of original submission}}$ are expected to provide insights to the model design. Thus, we would like to leave the exploration as the future work.

- Enhancing season-trend decomposition with DFT perfroms better than moving average. However, since moving average is quite simple and easy to implement with PyTorch, we eventually chose the moving-average-based season-trend decomposition in TimeMixer, which can also achieve a favorable balance between performance and efficiency.

**(2) Discussion about season-trend decomposition and DFT-based methods**

Firstly, we would like to highlight that **we propose TimeMixer towards a simple but effective method**, which we have already emphasized in the Introduction section. **In the spirit of building a practical model, proving that seasonal-trend decomposition is optimal or fundamentally different from other decomposition methods is out of the scope of our paper.** 

But thanks for the reviewer's suggestion. We would like to leave this discussion of the optimality and completeness of our model as future work, which has been added to the $\underline{\text{Appendix F of revised paper}}$.

Reviewer comment (Experiments):
Average pooling is only one of the pooling methods one could try. Is it possible that any alternatives could be better, or how could the authors prove that average pooling is the optimal one?

Author response:
> **Q2-part 2:** "Could there be alternatives to average pooling that might perform better? How can the authors demonstrate that average pooling is indeed the optimal choice?"

As per the reviewer's request, we replaced the average pooling with 1D convolutions with stride as 2. Here are the results. we can find that the complicated 1D-convolution-based outperforms average pooling slightly. But considering both performance and efficiency, we eventually use average pooling in TimeMixer.

| Ablation on downsampling methods | ETTm1 (Predict-336) MSE \|MAE | M4 SMAPE \|MASE \|OWA | PEMS04  MAE \|MAPE \|RMSE |
| -------------------------------- | -------------------------------- | ------------------------ | ---------------------------- |
| Moving average                   | 0.390 \|  0.404                  | 11.723 \| 1.559 \| 0.840 | 19.21 \| 12.53 \| 30.92      |
| 1D convolutions with stride as 2 | 0.387 \|  0.401                  | 11.682 \| 1.542 \| 0.831 | 19.04 \| 12.17 \| 29.88      |

Again, we just want to build a practical model. Although average pooling may be not optimal, it obviously an effective and easy-to-implement choice. The above ablation has also been included in $\underline{\text{Appendix G.3 of revised paper}}$.

Reviewer comment (Presentation):
In general, upsampling results in more data points, while downsampling results in fewer data points (as illustrated in Figure 1, leftmost). Based on this, I believe the descriptions 'Up-mixing' and 'Down-Mixing' in Figure 2 by the authors may not be appropriate and should perhaps be reversed.

Author response:
We would like to sincerely thank Reviewer P2SD for providing a detailed review and insightful suggestions. 

> **Q1:** "The descriptions 'Up-Mixing' and 'Down-Mixing' in Figure 2 may not be appropriate and should perhaps be reversed."

Sorry for these two inaccurate formalizations. Following your suggestion, we have rephrased 'Up-Mixing' and 'Down-Mixing' to 'Bottom-Up-Mixing' and 'Top-Down-Mixing'  in both figures and text of the $\underline{\text{reviserd paper}}$.

Reviewer comment (Novelty):
The paper lacks significant innovation. (1) Decoupling of multiscale [1], seasonal-trend disentanglement [2] are common modules that have already been proposed. (2) The so-called FMM module appears to be essentially a linear layer mapping the concatenated multiscale features back to the prediction length in the time dimension. In my view, the author's main contribution seems to be the integration of these two [1,2] modules.

Author response:
> **Q2-part1:** "The paper lacks significant innovation. In my view, the author's main contribution seems to be the integration of these two modules from MICN and Autoformer."

**(1) TimeMixer is clearly distinct from MICN and Autoformer.**

Firstly, TimeMixer is a **MLP-based** model, while MICN is **convolution-based** and Autoformer is **Transformer-based**.

Secondly, although multiscale analysis and decomposition have been used in previous models, we would like to highlight that **their usage in TimeMixer is distinct from MICN and Autoformer in both motivation and technical design,** which is summarized in the following tables:

| Motivation comparison | Why use decomposition?                                       | Why use multiscale analysis?                                 |
| --------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ |
| Autoformer            | Disentangle intricate variations                             | **/**                                                        |
| MICN                  | Disentangle intricate variations                             | Aggregate **local information** in different size regions with multiple convolutional kernels |
| **TimeMixer (Ours)**  | We find that **mixing directions should be different for seasonal and trend parts**. | **1. Disentangle intricate variations** **2. Utilize complementary forecasting capabilities in multiscale series** |

| Design comparison    | How to use decomposition?                                    | How to use multiscale analysis?                              |
| -------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ |
| Autoformer           | Decompose **one-scale input series and one-scale intermediate results** | **/**                                                        |
| MICN                 | Decompose **one-scale input series**                         | **Only for the seasonal part**                               |
| **TimeMixer (Ours)** | Decompose **multiscale deep features**                       | **1. For both seasonal and trend features** **2. Propose to use different mixing directions for multiscale seasonal and trend features** |

We do appreciate that MICN and Autoformer introduce multiscale analysis and decomposition into deep time series forecasting. But obviously, these three models are quite different in both model architecture and usage of the aforementioned two modules.

**(2) The design in adopting different mixing directions for decomposed parts should not be overlooked.**

In the paper, **we propose to utilize top-down and bottom-up mixing directions for seasonal and trend parts respectively**, which is a key contribution in our model. This design is well supported by:

- Motivation analysis in $\underline{\text{3rd paragraph in Introduction}}$ and $\underline{\text{Section 3.2}}$.
- Ablations in $\underline{\text{Table 5}}$, which verifies that our proposed mixing approach performs best.
- Visualizations in $\underline{\text{Figure 3}}$, which demonstrates that seasonal and trend parts should utilize different mixing directions.

Thus, TimeMixer is far beyond simple integration of multiscale analysis and decomposition. To highlight this point, we have rephrased the listed contributions in the $\underline{\text{Introduction of revised paper}}$.

> **Q2-part 2:** "The so-called FMM module appears to be essentially a linear layer mapping the concatenated multiscale features back to the prediction length in the time dimension."

**(1) There might be some misunderstandings of FMM (Future-Multipredictor-Mixing).** 

As presented in $\underline{\text{Equation 6 of main text}}$, we adopt **different predictors** to regress the future from past features in different scales separately.

Thus, we think the review might be misconceived in the following two points:

- We do not concatenate multiscale features, but keep the past information in their original length and directly regress the future separately. *(This is also different from the multiscale design in MICN.)*
- FMM is not a single linear layer mapping, but an ensemble of multiple predictors.

For clearness, we have rephrased $\underline{\text{Section 3.3 in the revised paper}}$ to clarify the misleading parts.

**(2) FMM can utilize complementary forecasting capabilities in multiscale series, not just align the time dimension.** 

As we presented in $\underline{\text{Figure 4 of original submission}}$, different scales present complementary forecasting capabilities, which is why we adopt multiple different predictors in FMM. To our best knowledge, this point is firstly explored by TimeMixer, which is also an important design in our paper.

Reviewer comment (Evaluation):
The improvement in experimental performance is marginal. Table 10 (the best results among all baselines) reveals that the actual enhancement by TimeMixer is not substantial, especially in three larger datasets, 'traffic,' 'weather,' and 'Electricity,' where it only holds a slight advantage at the third decimal place.

Author response:
> **Q3:** "The improvement in experimental performance is marginal, especially in three larger datasets, 'Traffic', 'Weather,' and 'Electricity'."

We believe that to measure the value of a deep model, we should consider $\underline{\text{performance, hyperparameter tuning cost and efficiency}}$ simultaneously.

**(1) In this unified hyperparameter setting, TimeMixer surpasses other baselines significantly.**

As shown in $\underline{\text{Table 2,3,4 of original submission}}$, TimeMixer is clearly better than the previous state-of-the-art PatchTST, SCINet, and TimesNet in both performance and efficiency. This advantage is meaningful since sometimes we do not have enough time or sources for hyperparameter searching in real-world applications.

**(2) In the hyperparameter-search setting, TimeMixer is much more efficient than the second-best model PatchTST.**

As pointed out by the reviewer, after a comprehensive hyperparameter-search, the relative improvement of TimeMixer against PatchTST is smaller than the unified hyperparameter setting. But we have to highlight that, TimeMixer still holds a significant advantage in efficiency, making it a valuable model to this community and applications.

To make this clearer, we list some results from $\underline{\text{Table 8 and Table 14 of original submission}}$ as follows. As presented in $\underline{\text{Table 6 of original submission}}$, Solar-Energy is the largest dataset and Weather is the second. We can find that TimeMixer surpasses PatchTST with 25% and 7.8% relative promotion in these two large benchmarks even after the hyperparameter searching. Besides, TimeMixer is over 2 times faster than PatchTST. In addition, we also present the results of DLinear, which is comparable to TimeMixer in efficiency. But TimeMixer beats it with more than 10% performance gain.

These analyses have also been included in the $\underline{\text{Appendix E of revised paper}}$.

| Model     | Solar-Energy (averaged MSE) Largest dataset | Weather (averaged MSE) Second Large dataset | Electricity (averaged MSE) | Traffic (averaged MSE) | GPU memory (predict-384) | Running Time (predict-384) |
| --------- | ----------------------------------------------- | ---------------------------------------------- | -------------------------- | ---------------------- | ------------------------ | -------------------------- |
| PatchTST  | 0.256                                           | 0.241                                          | 0.159                      | 0.391                  | 2097 MiB                 | 0.019 s/iter               |
| DLinear   | 0.329                                           | 0.246                                          | 0.166                      | 0.434                  | 1021 MiB                 | 0.003 s/iter               |
| TimeMixer | **0.192**                                       | **0.222**                                      | **0.156**                  | **0.387**              | 1043 MiB                 | 0.007 s/iter               |