Paper: ENHANCING MULTIVARIATE TIME SERIES FORECAST- ING WITH MUTUAL INFORMATION-DRIVEN CROSS- VARIABLE AND TEMPORAL MODELING (ICLR-2024)

Reviewer comment (Experiments):
The experiments are not extensive. It seems there are some cherry-pickings in particular configurations. How do the performance of ETTh2 and PEMS in Table 1? Some experiments (Table 2,3,4 and Figure 4) only use the results on partial datasets and baseline models. The prediction visualization of Figure 1 (comparing Informer and PatchTST) seems quite different from the reported results in the PatchTST paper. I hope the authors can elaborate more on it.

Author response:
**A1:** Thanks you for the detailed comments, we will answer the question in three parts:

**1. How do the performance of ETTh2 and PEMS in Table 1?** 

Great suggestions. We have made revisions to the manuscript accordingly. First, we conducted extensive experiments to evaluate the effectiveness of InfoTime on the ETTh2 dataset. The results, shown in the updated Table below (also Table 2 and Table 6 of our manuscript), demonstrate that InfoTime can clearly improve the performance of base models on the ETTh2 dataset. 

Moreover, we have made further improvements to Table 3 in order to better illustrate the performance improvement of InfoTime on datasets with clear correlations between channels. Following the advice from the reviewer, we have included the experimental results of Stationary and Crossformer in the revised manuscript. 
With these additions, the revised Table 3 allows for a more comprehensive evaluation of the performance of the models on the PEMS datasets, which are known to have clear correlations between channels.

Furthermore, we have added the ablation results of the aforementioned base models in Table 9. This provides insights into the contribution of InfoTime in improving the performance of both Channel-Independence models and Channel-mixing models.


Based on these experimental results, it can be observed that in datasets such as the PEMS datasets, where there is a clear correlation between variables, Channel-Independence models generally perform worse than Channel-mixing models. However, InfoTime is able to enhance the performance of both types of models.



|Length | 96| | 192 | | 336 | | 720 | |
|:----: | :----:| :----:| :----:|:----:|:----:| :----:| :----:|:---:|
|Metric|MSE|MAE|MSE|MAE|MSE|MAE|MSE|MAE|
|PatchTST| 0.274| 0.335| 0.342| 0.382| 0.365| 0.404| 0.391| 0.428 |
|+InfoTime| **0.271**|**0.332**| **0.334**| **0.373**| **0.357**| **0.395**| **0.385**| **0.421**| 
|RMLP| 0.290| 0.348| 0.350| 0.388| 0.374| 0.410| 0.410| 0.439 |
|+InfoTime| **0.271**| **0.333**| **0.335**| **0.374**| **0.358**| **0.395**| **0.398**| **0.432** |
|Informer| 3.755| 1.525| 5.602| 1.931| 4.721| 1.835| 3.647| 1.625 |
|+InfoTime| **0.336**| **0.390**| **0.468**| **0.470**| **0.582**| **0.534**| **0.749**| **0.620** |
|Stationary| 0.362| 0.393| 0.481| 0.453| 0.524| 0.487| 0.512| 0.494 |
|+InfoTime| **0.286**| **0.335**| **0.371**| **0.388**| **0.414**| **0.425**| **0.418**| **0.437** |
|Crossformer| 0.728| 0.615| 0.898| 0.705| 1.132| 0.807| 4.390| 1.795 |
|+InforTime| **0.333**| **0.386**| **0.455**| **0.453**| **0.554**| **0.513**| **0.757**| **0.619** |

**2. Some experiments only use the results on partial datasets and baseline models** We add the full experimental results of the Channel-Independence models in **Table 6**, and we also add the ablation results for all base models used in this paper in **Table 7 and 8**. We believe that these extensive experimental results are more convincing. We hope that the added experiments can help you to better understand InfoTime.

**3. The prediction visualization of Figure 1 (comparing Informer and PatchTST) seems quite different from the reported results in the PatchTST paper**. 

We believe there may be some misunderstanding here. The dataset used in Figure 1 for prediction visualization is the PEMS08 dataset, which is adopted from [SCINet](https://drive.google.com/drive/folders/17fwxGyQ3Qb0TLOalI-Y9wfgTPuXSYgiI). However, we do not find an evidence that the PatchTST paper has reported results on the PEMS08 dataset. In fact, the PatchTST paper reported results on the traffic dataset which is linked to [http://pems.dot.ca.gov](http://pems.dot.ca.gov). Although PEMS08 and the traffic dataset may share some correlation, they are indeed different datasets. We hope the reviewer provide more details on this issue.

Reviewer comment (Writing):
This paper essentially needs further polishing. There are many screenshots as figures, uncompiled citations (in related works), and confusing notations (e.g. unstated $I^i$, confused use of $X_i$ and $X^i$, unreasonable usage of the superscript: $X^o$).

Author response:
**A2:** Thank you for the valuable feedback on our paper. We appreciate the time and effort you have put into reviewing our work. We have carefully considered your comments and made the necessary revisions to address these issues raises. **Firstly**, we have replaced all the pictures with vector graphics to enhance the quality and clarity of the figures. This improvement ensures that the visual representation aligns with the content of the paper. **Secondly**, we have reviewed and modified the uncompiled citations in the related works section to ensure their accuracy and completeness. We understand the importance of proper citation and have taken the necessary steps to rectify any errors or omissions. Regarding the notation issue, we apologize for the confusion caused by the unstated $I^{i}$. After careful consideration, we agree that $I_{i}$ is a more suitable notation to represent the information constraint. To eliminate any ambiguity, we have modified $I_{i}$ to $I_{c}$ throughout the paper. Additionally, we have revised the notation for $X_{i}$ to $X^{i}$ to ensure consistency. We have also clarified that $X^{o}=X^{\{1,2,...,C\}}/i$ represents the historical data of all channels except the i-th channel. Once again, we sincerely appreciate your corrections on the expression problems in our paper. We have taken your feedback seriously and have made the necessary revisions to improve the clarity and accuracy of our work. We hope that the revised version of our paper will effectively convey our research and assist you in understanding our contributions.

Reviewer comment (Experiments):
The model benchmark tested in the proposed framework can be too few to be representative. The latest models in different categories are encouraged to be included, such as Transformers (Autoformer), linear models (TiDE, TSMixer), and TCN-based models (TimesNet). Datasets with more variables (such as Solar-Energy) are also recommended since the method essentially addresses the cross-variable issue.

Author response:
**A3:** Thank you for your valuable suggestions regarding the models and datasets used in our proposed framework. We appreciate your insight and agree that including a wider range of models and datasets would enhance the representativeness of our study. Regarding the inclusion of Autoformer, we acknowledge its popularity as a recent model in the transformer category. However, it is important to note that the base models used in our paper, such as Crossformer and PatchTST, have demonstrated better performance compared to Autoformer. Therefore, we believe that these base models effectively showcase the effectiveness of our proposed framework on transformer-based models. 

For non-Transformer-based models, we have chosen two linear models (TSMixer, DLinear) and a CNN-based model (TimesNet) to verify the effectiveness of InfoTime. Since the official code of TiDE is based on Tensorflow, and our framework uses pytorch, which is hard to transplant. However, we will add the result of TiDE in the future. Due to the constraints of time in our disscution, we have only tested the effectiveness of InfoTime on ETTh1, ETTh2, Weather, and we also add Solar-Energy and we will provide the full results in the future. We set the input length to 96 for TSMixer and TimesNet, and 336 for DLinear. We believe that the experimental results presented in the table blew demonstrate the effectiveness of InfoTime, as it compares favorably to the base models and showcases the improvement achieved by our InfoTime.

Thank you again for your insightful suggestions, and we appreciate your understanding of the limitations in our current implementation.

| Dataset|Length |TimesNet | |+TAM | | +InfoTime| | TSMixer| |+TAM | | +InfoTime||DLinear ||+TAM || +InfoTime||

Reviewer comment (Theory):
The term "single-step forecaster" can be confusing. I think it in fact outcomes the "multi-step" prediction. And it seems $\lambda=1$ is favored in most cases (even though experiments in Section 4.3 are not extensive). How to support the motivation of Equation 12?

Author response:
**A4:** Thanks for the valuable question. We will answer this question in two parts:

**1. The term "single-step forecaster" can be confusing** We appreciate your comment regarding the term "single-step forecaster." To clarify, a single-step forecaster refers to a non-autoregressive model that generates multi-step predictions in a single step.  In contrast, autoregressive forecasters predict one time step at a time, which can lead to error accumulation issues. While single-step forecasters alleviate this problem, they lack the ability to model the correlation between future time steps effectively.

**2. And it seems $\lambda=1$ is favored in most cases (even though experiments in Section 4.3 are not extensive). How to support the motivation of Equation 12?** We acknowledge the reviewer's observation that $\lambda=1$ appears to be favored in most cases, even though the experiments in Section 4.3 were not extensive. In response to this concern, we have conducted additional experiments and revised Figure 4 (modify a mistake in the orginal version) accordingly to provide a more comprehensive analysis of the impact of different $\lambda$.

The updated results presented in Figure 4 clearly demonstrate that the models' performance shows minimal improvement when $\lambda \geq 0.8$. In fact, we observed that when $\lambda$ was set to 0.8, the model's performance was slightly better than with $\lambda=1.0$. Similar trends were observed in other datasets as well, further reinforcing the notion that deviating from $\lambda=1$ does not significantly affect the model's performance. 

 We appreciate the valuable feedback provided by the reviewer and have incorporated it into our revised submission. Thank you for your question.

Reviewer comment (Evaluation):
About the model analysis. Are there some explainable results showing that the model takes full advantage of the multivariable correlation while avoiding unnecessary noise?

Author response:
**A5:** Thank you for your meaningful question. In order to verify the effectiveness of InfoTime in utilizing cross-variable correlation and avoiding unnecessary noise, we conducted comprehensive experiments on both Channel-mixing models (Informer, Stationary, and Crossformer) and Channel-Independence models (PatchTST and RMLP). By comparing the performance of these models with and without InfoTime, we can gain insights into its impact on utilizing multivariable correlation and avoiding unnecessary noise.

In the case of Channel-mixing models, which directly extract cross-variable information, InfoTime improves their performance. This indicates that InfoTime can enhance the models by effectively utilizing cross-variable information while avoiding unnecessary noise.

On the other hand, InfoTime also improves the performance of Channel-Independence models, which initially operate under the assumption of independent variables. This demonstrates that InfoTime can effectively exploit cross-variable information and leverage it to enhance the performance of these models.

To further illustrate the effectiveness of InfoTime in utilizing cross-variable features while eliminating irrelevant features, we focused on the ETTh1 dataset and compared three models: Informer, Informer+InfoTime, and Informer+CI (Channel-Independence strategy).

The results, presented in the table below, highlight several key findings. Firstly, the Channel-Independence strategy alone improves the model's performance, suggesting that the Channel-mixing model (Informer) learns irrelevant cross-variable information. However, compared to the Channel-Independence strategy, InfoTime achieves even better performance. This demonstrates InfoTime's ability to effectively utilize cross-variable information while eliminating irrelevant information (unnecessary noise).

||Informer|| Informer+InfoTime| | Informer+CI | |
|:----:|:----:|:----:|:----:|:----:|:----:|:----:|
||MSE|MAE|MSE|MAE|MSE|MAE|
|96|0.865|0.713|**0.381**|**0.394**|0.590|0.517|
|192|1.008|0.792|**0.435**|**0.430**|0.677|0.566|
|336|1.107|0.809|**0.485**|**0.461**|0.710|0.600|
|720|1.181|0.865|**0.534**|**0.524**|0.777|0.660|

Furthermore, we explored the ability of InfoTime to utilize cross-variable information and remove noise in a noisy synthetic dataset. The results of this exploration can be found in Appendix A.3 of the revision.

By conducting these experiments and analyzing the results, we provide explainable evidence that InfoTime successfully utilizes multivariable correlation while avoiding unnecessary noise.



Thank you for your question, and I hope this response addresses your concerns effectively.

Reviewer comment (Experiments):
More comparison with statistical forecasting methods or non-transformer methods, either empirically or theoretically, could help introduce the benefits of InfoTime in a more thorough manner. While Zeng et. al (2022) and other methods are briefly discussed for their direct forecasting strategy; it could be helpful to compare InfoTime on the DLinear model to see how CAM varies from the DMS approach. Perhaps this is why RMLP was introduced, but it’s not clear what motivated the construction of RMLP and if it differs at all from previous approaches which could be more suitable baseline models.

Author response:
**A7:** Thank you for your valuable feedback and suggestions. 
The DMS method introduced by Zeng et al. is consistant with the 'single-step forecaster' in our paper which means that predict multi-steps in a single step. AS disscussed in our work, the single-step forecaster or DMS assumes that the future time steps are independent of each other, therefore, they have the following formulation: $p(y^{i}|z^{i},x^{i})=\prod_{j=1}^{P}p(y^{i}_{j}|z^{i},x^{i})$. However, it is important to note that future time steps are not truly independent, and although the single-step forecaster can effectively alleviate the error accumulation problem, it cannot model the correlation between future sequences, which is why we propose the TAM module to address this limitation.

In our paper, we propose the linear model RMLP, which outperforms DLinear. As a result, we chose to use RMLP as the base linear model for comparison. Additionally, we also evaluated the effectiveness of InfoTime on the DLinear (linear model), TSMixer (linear model), and TimesNet (linear model) and included the Solar-energy dataset in our analysis. The results, as shown in the table below, demonstrate that InfoTime consistently improves the performance of DLinear, TSMixer, and TimesNet across different datasets. This highlights the efficacy of InfoTime in enhancing the forecasting capabilities of the Linear models and CNN-based models. Moreover, we appreciate your suggestion to include more benchmarks such as M3 and M4. We found that the M3 and M4 datasets are very different from the datasets used in our work. It is very meaningful to verify the effect of our model on these datasets in our future revision.

Reviewer comment (Experiments):
The experimentation work is not detailed. There are a number of factors that are unclear: the reported informer MSE and MAE on ETTh1, ETTm1, ETTm2 for example are quite a bit higher than in the original Informer, Non-stationary Transformer, and Crossformer papers (>1 for O=720 Informer by authors vs <0.3 in original paper). Perhaps more discussion of the training setup and how it might differ from past benchmarks and why it shows a much higher error could help separate out the actual performance gains given by InfoTime.

Author response:
**A1:** Thank you for your question. We would like to provide further clarification on our experimental work. Firstly, we ensured the reliability and consistency of our results by repeating each experiment three times and reporting the average as the final result. Additionally, we have included the experimental settings of the base models in Appendix A.1, which provides transparency and allows for a better understanding of our methodology. To maintain a fair comparison, we used the same parameter settings as the Informer, Non-stationary Transformer, and Crossformer papers. The experimental scripts we used were obtained from the respective repositories [DLinear](https://github.com/cure-lab/LTSF-Linear/blob/main/scripts/EXP-LongForecasting/Formers_Long.sh), [Non-stationary Transformer](https://github.com/thuml/Nonstationary_Transformers/tree/main/scripts), and [Crossformer](https://github.com/Thinklab-SJTU/Crossformer/tree/master/scripts),. However, we did make a modification by setting the input length $I$ to 96, which may differ from the experimental settings in the original papers. Additionally, for PatchTST and RMLP, we set the input length to 336, as longer input lengths tend to yield better performance for Channel-Independence models. For PEMS03, PEMS04, and PEMS08 datasets, we set $I=336$ for all of these models since all of them perform better in a longer input length.

Regarding your concern about the Informer results, we would appreciate it if you could specify the dataset you are referring to. We have thoroughly checked the results of Informer and have not identified any issues. It is worth noting that our experiments focus on multivariate forecasting rather than univariate forecasting, which may impact the comparison with the original Informer results.

I hope this revised response addresses your concerns more clearly. If you have any further questions or require additional clarification, please feel free to ask.

Reviewer comment (Presentation):
Given the large tables of evaluations, it would be helpful to introduce some visual aides to help with understanding the comparisons, perhaps showing the % improvement over other methods for an aggregation of all benchmarks.

Author response:
**A2:** Thanks for your meaningful suggestion. We appreciate your feedback, and we have taken it into consideration. In response, we have made updates to the evaluation by including the average Mean Squared Error (MSE) and Mean Absolute Error (MAE) for four different prediction lengths, as well as the relative reduction of MSE and MAE. You can find this information in **Table 1** and **Table 6**. We hope that these additions will provide a clearer understanding of the effectiveness of InfoTime. If you have any further questions or suggestions, please feel free to let us know. We value your input and strive to continuously improve our work.

Reviewer comment (Writing):
sp. ‘varibales’ (pg. 5)

Author response:
**A8:** Thank you for pointing out the typo, and we have modified it. We apologize for the oversight and appreciate you bringing it to our attention.

Reviewer comment (Evaluation):
There is a lot of variation in results of benchmarks (ETTh1, ETTm1, ETTm2, Weather, Traffic, Electricity) across various papers. Have the authors considered running 5-fold cross validation on these benchmarks to better understand the error bounds around performance claims and then running tests to evaluate the statistical significance of the improvements?

Author response:
**A1:** Thank you for your question. We would like to provide further clarification on our experimental work. Firstly, we ensured the reliability and consistency of our results by repeating each experiment three times and reporting the average as the final result. Additionally, we have included the experimental settings of the base models in Appendix A.1, which provides transparency and allows for a better understanding of our methodology. To maintain a fair comparison, we used the same parameter settings as the Informer, Non-stationary Transformer, and Crossformer papers. The experimental scripts we used were obtained from the respective repositories [DLinear](https://github.com/cure-lab/LTSF-Linear/blob/main/scripts/EXP-LongForecasting/Formers_Long.sh), [Non-stationary Transformer](https://github.com/thuml/Nonstationary_Transformers/tree/main/scripts), and [Crossformer](https://github.com/Thinklab-SJTU/Crossformer/tree/master/scripts),. However, we did make a modification by setting the input length $I$ to 96, which may differ from the experimental settings in the original papers. Additionally, for PatchTST and RMLP, we set the input length to 336, as longer input lengths tend to yield better performance for Channel-Independence models. For PEMS03, PEMS04, and PEMS08 datasets, we set $I=336$ for all of these models since all of them perform better in a longer input length.

Regarding your concern about the Informer results, we would appreciate it if you could specify the dataset you are referring to. We have thoroughly checked the results of Informer and have not identified any issues. It is worth noting that our experiments focus on multivariate forecasting rather than univariate forecasting, which may impact the comparison with the original Informer results.

I hope this revised response addresses your concerns more clearly. If you have any further questions or require additional clarification, please feel free to ask.

Reviewer comment (Evaluation):
How does the time complexity of CDAM, TAM, and InfoTime compare with the original base models - particularly as the time series get longer?

Author response:
**A3:** Thank you for the insightful question. In our experiments on the Informer model and ETTh1 dataset, we tested the running time (in seconds) of each epoch for both the base models and InfoTime. By default, we set N=4. Upon analyzing the results, we found that TAM is relatively time-consuming, while CDAM is less consuming. Additionally, we observed that InfoTime does not introduce additional time consumption as the length increases. Although InfoTime requires more time to train, we firmly believe that it is valuable in enhancing the performance of the base models. It is worth noting that future research on how to make InfoTime more lightweight is a meaningful and interesting area of exploration. By optimizing its time complexity, InfoTime has the potential to become even more efficient and practical for forecasting longer time series. Thank you again for bringing up this important point.

|     Predicted Length      |   96   | 192 |
|:-------------------------:|:------:| :----: |
|         Original          |  8.5   |   10.5| 
|          TAM N=1          |  9.3   |  11.4 |
|          TAM N=2          |  11.2  | 13.4  |
|          TAM N=3          |  14.2  | 15.5  |
|          TAM N=4          |  20.0  | 22.5  |
|           CDAM            |  10.8  | 12.7  |
|       InfoTime            |  23.5  | 24.5 |

Reviewer comment (Experiments):
The paper introduces a trade-off parameter B in CDAM to control the balance between retaining important information and eliminating irrelevant information from the latent representation. How could this parameter be determined or optimized in practice? Are there any insights into choosing an appropriate value for B based on experimental results or theoretical considerations?

Author response:
**A4:** Thanks for your question. The determination and optimization of the trade-off parameter B in CDAM is an important aspect of leveraging the balance between retaining important information and eliminating irrelevant information from the latent representation. In fact, we consider $\beta$ as a hyper-parameter that needs to be adjusted manually and remains constant during training. While it would be exciting to have the model automatically adjust $\beta$, which is not explored in our work. However, there have been other research works that propose dynamically adjusting $\beta$ during training, such as [Generating Sentences from a Continuous Space](https://aclanthology.org/K16-1002/), [Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing](https://aclanthology.org/N19-1021/), [ControlVAE: Controllable Variational Autoencoder](https://proceedings.mlr.press/v119/shao20b/shao20b.pdf). 

In our experiments, we find that a large value of $\beta$ ($\beta \geq 1e3$ usually) is needed to prevent overfitting and improve performance, as shown in Figure 4 of our paper. This observation suggests that deep-learning-based models have sufficient capacity to capture historical features, and it is crucial to focus on preventing the model from learning irrelevant features.

Reviewer comment (Theory):
Additionally on the upper bound, the authors state that they choose to adopt the sampled vCLUB and minimize I_{vCLUB-S}(Xo; Zi). Could they expand on why this sampling strategy was chosen and what the actual derived upper bound might be?

Author response:
**A6:** Thank you for your question. In comparison to the [VUB](https://arxiv.org/pdf/1612.00410.pdf) and [L1OUT](https://proceedings.mlr.press/v97/poole19a/poole19a.pdf) methods, vCLUB demonstrates a superior ability to approximate the upper bound. Consequently, we have opted to utilize vCLUB as the variational upper bound in our paper. Additionally, we have found that vCLUBSample offers greater efficiency than vCLUB. Hence, we have chosen to employ vCLUBSample in our study.