Paper: Connecting the Patches: Multivariate Long-term Forecasting using Graph and Recurrent Neural Network (ICLR-2024)

Reviewer comment (Writing):
The authors' use of mathematical notation appears inconsistent. For instance, function names should ideally be presented in regular typeface rather than italic. Proper notation ensures clarity and avoids potential confusion.

Author response:
We appreciate your comment for helping to optimize our paper. We check the main text and appendix of our paper and make corrections to address this issue. We modify the font of the function and variable letters, and correct the italicized font of the function to normal, such as covariance function, argtopk, softmax function.

Reviewer comment (Experiments):
While the authors opted for the Pearson correlation coefficient for graph construction, which subsequently serves as the foundational structure for the GNN, one might question the exclusion of making GNN parameters learnable. This adaptability could potentially offer more flexibility to the model.

Author response:
We appreciate your suggestion. Parameterizing the graph structure to achieve greater flexibility is a worthwhile try. We parameterize the graph structure to see the performance changes of GRformer, and the results show that parameterizing the graph structure achieves better performance on some of the datasets(e.g. Weather), but the effect is not significant on most datasets, and even inferior to simple correlation coefficient algorithms. We have updated the results to **appendix (A.8)**.

Reviewer comment (Evaluation):
The Pearson coefficient assumes homoscedasticity in the data. It's unclear if the authors verified this assumption across their datasets. Such checks are crucial to ensure the validity of the chosen coefficient.

Author response:
Thank you for your suggestion. The Pearson correlation coefficient is calculated based on the standardized data. We use the Bartlett function in the Python 'scipy' library to verify the homoscedasticity of the data we use, and obtaine test statistics and test p on 8 datasets, respectively. The results indicate that the statistical values of these data are very small, and p_value is close to 1.0, which makes us confident that the data we use satisfies homoscedasticity. We provide additional clarification on this issue in **appendix (A.5)**.

Reviewer comment (Evaluation):
The paper doesn't seem to explore or discuss other potentially beneficial correlation coefficients like Time-Lagged Cross-Correlation (TLCC) or Dynamic Time Warping (DTW). An exploration or justification of the chosen metric over others could have added depth to their methodology.

Author response:
Thank you for your suggestion! We agree with you with the insight that more attempts on the Graph construction module is beneficial. Based on your suggestion, we try some other graph structure construction algorithms, such as the DTW algorithm, and update the results in **appendix (A.8)**. From the results, although the DTW algorithm has a more complex and refined calculation process compared to the Pearson coefficient, there is not significant performance improvement. The default graph construction method in our proposed GRformer is still relatively simple and widely effective. TLCC generates a correlation matrix similar to Pearson coefficient method, so we do not conduct experiments with this setting.
We also clearly recognize that in the face of more complex scenarios, using different algorithms may also result in better performance. Our graph construction module is independent of the training process, making it easy to be replaced with other graph construction algorithms to generate reasonable graph structures.

Reviewer comment (Experiments):
The methodology introduced by the authors involves several hyperparameters, which seemingly have a significant impact on the model's outcomes. Specifically, when constructing the graph structure: How was the threshold value of 0.8 determined? Regarding the 'topk' selection, how was the value of \( k \) chosen, and does it correlate with the number of variables?

Author response:
Thank you for your question. The threshold (we use $\lambda$ in the following text) 0.8 and top-k value are chosen according to experience in the earlier version. Based on your comment, we set these two parameters as hyperparameters and conduct parameter sensitivity analysis on different datasets. We find that the value settings in the early version do not guarantee optimal results, in the revised version, we update the parameter sensitivity analysis results to **appendix A.5.2** and **A.5.3**. Overall, using different thresholds and topk value under different datasets can achieve better results, e.g. for traffic, the best $\lambda$ is 0.6 and the best top-k is 16.
Top-k value does not have direct correlation with the number of variables, however, based on the multi-hop depth $gdep$, we tend to choose top-k values that meet the following condition: (top-k)$^{gdep}\le M$, where $M$ is the number of variables.

Reviewer comment (Experiments):
How was the value for the EMA parameter \( \alpha \) in the mix-hop propagation process determined?

Author response:
Thank you for your question. We also conduct parameter sensitivity analysis for the value of $\alpha$ in the revised version, and choose the optimal results. We have updated these results to **appendix (A.5.4)**.

Reviewer comment (Novelty):
The idea of enhancing Transformer with RNN is not new [1].

Author response:
We appreciate for your comment. 
- There are indeed some previous studies using RNN to enhance Transformer, and we have noted the work you mentioned. But the way we use RNN is different. In paper [1], Attention machenism is used to generate an enhanced input sequence to be fed in an RNN structure, with its attention score generated with the help of the previous RNN hidden states. The encoder is essentially an RNN that encodes the input sequences into a feature representation, which makes the model still suffer from gradient explosion and vanishing when facing long-term sequence.  In GRformer, we do not directly pass input data through the RNN layer, we only use RNN to explicitly introduce temporal order information in position encodings that are injected into sequence tokens, driven by the features. The encoder of GRformer is essentially a self-attention layer. To our knowledge, there are currently no similar position encoding methods like this to solve MLTSF tasks, and our work also explores the effectiveness of this approach.
- As for the model efficiency, we conduct experiments to visualize the training speed and memory consumption of using RNN-based position encoding. In small dataset like ETTh1, a one-layer RNN-based position encoding module costs almost the same training time and 3.8% more memory compared with learnable position encoding (i.e. PatchTST), while the gap comes to 5.9% more training time and 5.3% more memory in Traffic. The results are added to **appendix (A.6)**.
- We evaluate the effectiveness of using RNN-based position encoding with varying depth on different datasets and add the results in **appendix (A.5.1)*, overall, RNN-based position encoding can achieve 2.51% and 1.83% reduction in MSE and MAE. We also put results of ablation studies on all datasets in **section 4.2**.

Reviewer comment (Experiments):
Ablation study in Table 3 shows that the improvement of RNN against previous learnable position embedding is not significant.

Author response:
We appreciate for your comment. 
- There are indeed some previous studies using RNN to enhance Transformer, and we have noted the work you mentioned. But the way we use RNN is different. In paper [1], Attention machenism is used to generate an enhanced input sequence to be fed in an RNN structure, with its attention score generated with the help of the previous RNN hidden states. The encoder is essentially an RNN that encodes the input sequences into a feature representation, which makes the model still suffer from gradient explosion and vanishing when facing long-term sequence.  In GRformer, we do not directly pass input data through the RNN layer, we only use RNN to explicitly introduce temporal order information in position encodings that are injected into sequence tokens, driven by the features. The encoder of GRformer is essentially a self-attention layer. To our knowledge, there are currently no similar position encoding methods like this to solve MLTSF tasks, and our work also explores the effectiveness of this approach.
- As for the model efficiency, we conduct experiments to visualize the training speed and memory consumption of using RNN-based position encoding. In small dataset like ETTh1, a one-layer RNN-based position encoding module costs almost the same training time and 3.8% more memory compared with learnable position encoding (i.e. PatchTST), while the gap comes to 5.9% more training time and 5.3% more memory in Traffic. The results are added to **appendix (A.6)**.
- We evaluate the effectiveness of using RNN-based position encoding with varying depth on different datasets and add the results in **appendix (A.5.1)*, overall, RNN-based position encoding can achieve 2.51% and 1.83% reduction in MSE and MAE. We also put results of ablation studies on all datasets in **section 4.2**.

Reviewer comment (Novelty):
The mix-hop propagation layer is exactly the same as that in [2] and there is no explicit reference to it in Section 3.2.3.

Author response:
Thank you for your comment. 
- Yes, we do draw inspiration from MTGNN[1] for using multi-hop propagation layers. We explicit reference to it in **section 3.2.3** in the revised version. However, our multi-hop propagation layer has some differences in implementation details compared to MTGNN. In MTGNN, embeddings from each hop are aggregated through a Conv2d layer with a convolutional kernel size of (1,1), which means that the aggregated embeddings of each channel in each hop are all viewed as separate individuals (i.e. they will be assigned with different weights separately). We choose to use a learnable matrix to aggregate the embeddings of different hop neighbors, so that embeddings of each channel from the same hop neighbors will be assigned with the same weights, this makes it easier for the model to identify the importance of neighborhoods’ information of different hops.
- One of our motivation is that, the current channel-mixing and channel-independent models can’t effectively capture the cross-channel dependencies between time series, while graph neural network provides a chance to make better solutions. We explored the effectiveness of graph neural networks in MLTSF tasks, and from the results in **Table 3**, it improves the MSE and MAE by 0.65% and 3.63% respectively.

Reviewer comment (Novelty):
Besides the graph construction via Pearson correlation, the approach is a direct combination of PatchTST and "Connecting the dots".

Author response:
Thank you for your comment. 
- Yes, we do draw inspiration from MTGNN[1] for using multi-hop propagation layers. We explicit reference to it in **section 3.2.3** in the revised version. However, our multi-hop propagation layer has some differences in implementation details compared to MTGNN. In MTGNN, embeddings from each hop are aggregated through a Conv2d layer with a convolutional kernel size of (1,1), which means that the aggregated embeddings of each channel in each hop are all viewed as separate individuals (i.e. they will be assigned with different weights separately). We choose to use a learnable matrix to aggregate the embeddings of different hop neighbors, so that embeddings of each channel from the same hop neighbors will be assigned with the same weights, this makes it easier for the model to identify the importance of neighborhoods’ information of different hops.
- One of our motivation is that, the current channel-mixing and channel-independent models can’t effectively capture the cross-channel dependencies between time series, while graph neural network provides a chance to make better solutions. We explored the effectiveness of graph neural networks in MLTSF tasks, and from the results in **Table 3**, it improves the MSE and MAE by 0.65% and 3.63% respectively.

Reviewer comment (Presentation):
What is the authors' primary objective in visualizing the weights of the MLP in Figure 1(b), given that it only reflects the correlation among hidden states?

Author response:
Thanks for asking the question. We modify the corresponding figures and explanations of **Figure 1(b)** to make it clear. Values of each channel of a multi-variant time series will be passed through an embedding mapping projector, a series of feedforward networks of Encoders and Decoders, and a final mapping projector. These neural networks are usually some MLP layers, such as Linear layer and Conv1d layer. The weight matrix plays an important part in promoting interactions across different channels. We save the weight matrix parameters (we choose parameters from Informer) and calculate their matrix products, and get an equivalent correlation matrix. Compared with the Pearson correlation coefficient matrix (or the correlation matrix obtained by any other algorithms), the result does not effectively promote the interaction between strong correlated sequences and prevent weak correlated sequences from interacting, either.

Reviewer comment (Experiments):
How were the hyperparameters (0.8 and $k$) in Equations (2) and (3) chosen, and what impact do these specific values have on the model's performance and behavior?

Author response:
We appreciate you for this detailed question. The choice of 0.8 and top-k value in the previous version of our paper are based on experience. Based on your comment, in order to provide a more rigorous explaination of our hyperparameter choosing, we conduct hyperparameter sensitivity analysis on the filtering threshold (we use $\lambda$ to represent it) and top-k values of the Pearson coefficient correlation matrix, and update the results to **appendix A.5.2, A.5.3**, and **A.5.4**. A smaller threshold and larger top-k value will allow for more interactions between channels. Directly choosing 0.8 based on experience does have some problems and changing this threshold to smaller values can achieve better results on different datasets(e.g. GRformer gets the optimal results using $\lambda=0.6$ and top-k $=16$ in Traffic). We choose the best results and update **Table 2** in **section 4.1**.

Reviewer comment (Presentation):
I am not sure what I am seeing in Figure 1(b), and I don’t understand how to interpret the authors' claim that cross-channel interaction is chaotic based on simply visualizing the weight matrices of the Transformer's dense layer (internal MLP).

Author response:
We are thankful for your suggestion. We modify the corresponding explanations of **Figure 1(b)** to make it clear. In the revised version, we save the weight matrix parameters and calculated their matrix products, and get an equivalent correlation matrix. Values of each channel of a multi-variant time series will be passed through an embedding mapping projector, a series of feedforward networks of Encoders and Decoders, and a final mapping projector. These neural networks are usually some MLP layers, such as Linear layer and Conv1d layer. The weight matrix plays an important part in promoting interactions across different channels. 
Compared with the Pearson correlation coefficient matrix (or the correlation matrix obtained by any other algorithms), this result does not effectively promote the interaction between strong correlated sequences and prevent weak correlated sequences from interacting, either.

Reviewer comment (Theory):
I am not sure I understand the authors' point about positional encoding not being able to represent temporal orders well. RNNs have their own problems, such as vanishing gradients when modeling long-term temporal dependencies. Are you suggesting that RNNs outperform Transformers in multivariate long-term time series forecasting (MLTSF)?

Author response:
Thank you for this thoughtful question. We agree with you that RNN has problems such as vanishing gradients facing long-term sequences, however, our motivation is to make use of the sequence modeling capability of RNN only to generate position encodings and we express this point in our **contributions** and **section 3.2.1**. The output results of the RNN layer are not directly used as the final feature embeddings, and this could alleviate the vanishing gradient problems.
We make more detailed parameter adjustments to the experimental setup of the ablation study, such as changing the number of RNN layers $C$ to observe performance changes. We choose the best results and update **Table 3** in **section 4.2**. overall, RNN-based position encoding achieves 2.51% and 1.93% improvement in MSE and MAE metrics, respectively. We present the optimal results using RNN-based position encoding with layers 1, 2, and 3 on all the dataset in **appendix (A 5.1)**.

Reviewer comment (Experiments):
Are the ablation results in Table 3 the experiments to back this claim about positional encoding? If that's the case, the performance difference between an RNN-based positional encoding (?) vs a learned positional embedding is almost 0.

Author response:
Thank you for this thoughtful question. We agree with you that RNN has problems such as vanishing gradients facing long-term sequences, however, our motivation is to make use of the sequence modeling capability of RNN only to generate position encodings and we express this point in our **contributions** and **section 3.2.1**. The output results of the RNN layer are not directly used as the final feature embeddings, and this could alleviate the vanishing gradient problems.
We make more detailed parameter adjustments to the experimental setup of the ablation study, such as changing the number of RNN layers $C$ to observe performance changes. We choose the best results and update **Table 3** in **section 4.2**. overall, RNN-based position encoding achieves 2.51% and 1.93% improvement in MSE and MAE metrics, respectively. We present the optimal results using RNN-based position encoding with layers 1, 2, and 3 on all the dataset in **appendix (A 5.1)**.

Reviewer comment (Writing):
What exactly is the RNN-based position encoding method? In the caption for Figure 2, it says "The multi-layer RNN injects temporal order information." However, RNNs are not just injecting temporal order information as some sort of advanced positional encoding method; they can actually learn temporal dependencies. I am not sure if you are distinguishing between positional encoding and learning temporal representation.

Author response:
Thank you for asking this question. Passing data into RNN can indeed extract temporal dependencies, however, this operation may cause information loss because of gradient explosion and vanishing when facing long-term sequence. In our model, RNN-based position encoding refers to the use of RNN structures for only generating position encodings with a feature-driven scheme. We hope to use its sequence modeling characteristics to generate position encoding with strict temporal order information, rather than directly output its results as temporal dependent features. By doing so, we can obtain enhanced position encodings compared to position encodings used by vanilla Transformer-based models and help the model to comprehend the strict temporal order of time series data.

Reviewer comment (Presentation):
Figure 2(b) is hard to understand; at least explain the operator signs in the caption, as arrows are not clear.

Author response:
We apologize for the confusion. **Figure 2(b)** shows the module design based on multi-layer RNN position encoding, all the operator signs are in **Eq(1)**. We add explainations for **Figure 2(b)** in the revised version. The hidden state of the $j$-th RNN unit in $(c-1)$-th layer $r_{(i, j)}^{(c-1)}$ is used as inputs for RNN units at the same position in the next layer (the $j$-th RNN unit in the $c$-th layer) and the previous time step hidden state of RNN units at the next position in the same layer (the $(j+1)$-th RNN unit in the $(c-1)$-th layer). We update the arrows in **Figure 2(b)** and provide a more detailed explanation.

Reviewer comment (Evaluation):
What is the main evidence that Transformer-based models are ineffective at capturing cross-channel dependencies and temporal orders? If Transformers were bad at capturing temporal orders, they would not have become as popular as they are today. I am curious why the authors make such claims, as I do not see any plausible supporting evidence in the manuscript.

Author response:
We appreciate you for this thoughtful question, we agree with you that Transformer is popular now and good at capturing temporal dependencies. We dive deep into the characteristics of time series data and believe that Transformer-based models can be further improved in extracting cross-channel and temporal dependencies.
1. **Cross-channel dependencies.** Values of each channel of a multi-variant time series data will be passed through an embedding mapping projector, a series of feedforward networks of Encoders and Decoders, and a final mapping projector. These modules are usually implemented with MLP layers, and the weight matrices implicitly promotes the interaction of different channels. We modify **Figure 1(b)** to the weight matrix product of MLP layers in previous Transformer models to intuitively compare with the adjacency matrix constructed using Pearson correlation coefficient. The result of weight matrix product leads to a correlation matrix with chaotic data distribution, which may result in strongly correlated sequences not interacting effectively and weakly correlated sequences introducing noise to each other.
2. **Temporal dependencies.** Transformer is good at capturing the temporal dependencies of sequence data because of the use of self-attention mechanism with position encoding. However, classic position encoding methods such as fixed and learnable position encoding are first introduced in NLP tasks. These methods focus more on positional variance, not the strict temporal order, which is a key characteristic of time series data. This motivate us to find a better position encoding method to inject strict temporal order information to sequence tokens, and RNN is a good choice because of its sequence modeling capability. The ablation study results shown in **Table 3** can be the supporting evidence for the above design. The use of RNN-based position encoding reduces the MSE and MAE by 2.51% and 1.93% on average.

Reviewer comment (Reproducibility):
The authors mentioned that they used multi-layered RNNs, however in the appendix, it is said that a 1-layer RNN was used. Can you clarify the details of the RNN architecture?

Author response:
We thank the reviewer for the detailed question. In the earlier version of the paper, we lack the sensitivity analysis on the number of RNN layers ($C$). Based on your comment, in the revised version, we conduct experiments on this hyperparameter and add the results in **appendix (A 5.1)**. We sum the result in the following table, for weather, electricity, ETTh1, ETTm1 and ILI, the model reaches optimal performance with $C=1$; for traffic, ETTh2 and ETTm2, the best RNN layer number is $C=2$.
| Datasets | Weather | Traffic | Electricity | ETTh1 | ETTh2 | ETTm1 | ETTm2 | ILI |
| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| Optimal $C$ | 1 | 2 | 1 | 1 | 1 | 2 | 2 | 1 |

Reviewer comment (Theory):
“To properly capture temporal dependencies, we consider using a multilayer RNN to encode the positions in the time series.” Why can deep RNNs properly capture temporal dependencies while Transformers can’t?

Author response:
Thank you for the comment, the description in the earlier version may cause confusions, so we have modified it to “In order for the model to make use of the strict temporal order of the sequence, we consider using a multi-layer RNN to inject some positional contextual information into the sequence tokens” in **section 3.2.1**. Our viewpoint is that vanilla Transformer can indeed capture temporal dependencies, however, the position encoding methods (e.g. fixed or learnable position encoding) are initially designed for NLP tasks, where the sequence data used in nature language do not emphasize strict temporal order as much as time series data does. We hope to use the sequence modeling capability of RNN to supplement this temporal order information, so that the Transformer-based models can learn this characteristic of time series and obtain more reasonable temporal dependencies.