Paper: ARM: Refining Multivariate Forecasting with Adaptive Temporal-Contextual Learning (ICLR-2024)

Reviewer comment (Writing):
The MKLS block in section 3.3 is difficult to follow due to notations. The authors introduced many subscripts (and subscripts of X, for example); it would be nice to refer to somewhere for clarification.

Author response:
#### Q5. About the Computational Details in MKLS
We apologize for the confusion caused. MKLS employs multiple 1D CNNs with different kernel sizes to obtain various local views $\widetilde{X}_j$ of the input \(X\), and uses channel attention to calculate the weight of these local views on each output channel for a weighted average, resulting in a smoothed local view. Therefore, the output of the \(j\)th CNN should actually be computed as \(\widetilde{X}_j = \mathtt{Conv1d}_j \left(X \right)\). Our previous notation of \(X\) as \(X_j\) was incorrect and caused misunderstanding. This has been corrected in the revision. Please refer to the updated Figure 4 for more details on MKLS computations.

Reviewer comment (Presentation):
Table 2 contains a large amount of results which are replicates of Table 1. The authors may consider eliminating some and finding a better way to present them.

Author response:
We greatly value your suggestions regarding the presentation of our paper and have made the following adjustments in our revision:
- We revised the layout of Table 2 to reduce redundancy and added a comparison of average percentage performance improvements with the original model for each sub-experiment to demonstrate the specific enhancements brought by the A/R/M modules.
- In line with the notations in our paper, we added relevant markings to the Figures of the overall architecture and MKLS block (Figures 2 and 4) to aid in understanding the computational details.
- We bolded the reference in Section 4 that leads to the discussion on computational costs in Appendix A.7, making it more prominent.

Reviewer comment (Theory):
How does MoE better capture temporal patterns compared to autoregressive and exponential smoothing models?

Author response:
### Q2. How does MoE Better Capture Temporal Patterns Compared to Autoregressive (AR) and Exponential Smoothing (ES) Models?

- Firstly, the AR and ES methods mentioned in our paper, such as LSTNet and ES-RNN, do not incorporate the AR and ES parameters in the neural network training. In this context, the pre-fixed AR and ES model structures may not suit time series with significant characteristic differences. We added a brief predictability analysis in Appendix A.8 to show the substantial performance variation of these basic learners when dealing with different time series data. In our AUEL, we use an adaptive EMA for each series to adjust the lookback view and then employ MoE to assign the most suitable univariate MLP predictor to each input series, achieving the best adaptation to characteristic differences.
- Secondly, AR and ES align with the individual univariate model with no parameter sharing described in Figure 3 (a). We consider both this model and the full parameter sharing model in Figure 3 (b) to be suboptimal. If two series in the data have strong relationship (like in the Multi dataset, where one series is a shift of another), parameter sharing between these two series is a good choice. However, if a series in the dataset has no relationship with any other series, sharing its parameters with other series is not a good option. Therefore, we apply the MoE shown in Figure 3 (c) to address this issue, dynamically deciding which predictor from the predictor pool to use for univariate forecasting based on series characteristics.

Reviewer comment (Writing):
What is the inverse processing stage?

Author response:
#### Q3. What is the Inverse Processing Stage?
- In previous methods addressing the input-output distribution shift problem, such as RevIN and NLinear, they estimate each series' output mean and standard deviation based on the input. These values are used to normalize the input, and then applied during the inverse processing stage after the main model processing to ensure that the final output conforms to this distribution. Our AUEL of distribution, positioned between RevIN (full-length lookback) and NLinear (last-step lookback), adaptively adjusts the lookback view for the output distribution estimation, thus following the same computation process above.
- In the AUEL of temporal patterns, the inverse processing stage helps the model balance between learning intra-series and inter-series dependencies. Note the preprocessing MoE in equation (2) as \( \mathtt{MoE}\left( \left[ \widetilde{X}^{\*i}_I   \ \ \ \ \widetilde{X}^{\*i}_P \right] \right) \) and the inverse processing MoE in equation (3) as \( \mathtt{MoE}\left( \left[ \widetilde{X}^{\*i}_I   \ \ \ \ \widehat{X}_{ED}^{i} \right] \right) \), which only differ in the \(L_P\) forecasting horizon, changing the preprocessed values to the main predictor output \(\widehat{X}_{ED}^{i}\). If a series \(i\) is strongly related to other series and the relationship is effectively captured by the main predictor output, we hope the inverse processing \( \mathtt{MoE}\left( \left[ \widetilde{X}^{\*i}_I   \ \ \ \ \widehat{X}_{ED}^{i} \right] \right) \) maintains this output value or makes minor corrections. However, if a series \(i\) has no relation to other series, the inter-series dependencies learned by the main predictor in \(\widehat{X}_{ED}^{i}\) are likely incorrect. In this case, we expect the inverse processing \( \mathtt{MoE}\left( \left[ \widetilde{X}^{\*i}_I   \ \ \ \ \widehat{X}_{ED}^{i} \right] \right) \) to make significant corrections to the \(\widehat{X}_{ED}^{i}\) portion, favoring univariate modelling contribution based on \(\widetilde{X}^{\*i}_I\). On datasets with a large number of series or series from different sources, this correction significantly improves model performance.

Reviewer comment (Theory):
The paper mentions causal dependency/relationship among different series. Could the authors explain in what sense the notion of causality is embedded?

Author response:
#### Q4. About the Causality Mentioned in the Paper
- Thank you for this valuable question. To avoid any further misunderstanding, we have deleted all the words related to "causality" in the revision. In our previous paper version, we tried to use the term 'causality' to convey the notion that data with strong causal relationships, like those generated through simple shifting operations in the "Multi" dataset, are not effectively modeled by existing models.
- Indeed, in earlier versions of our paper, we discussed about whether to include a section in the appendix about the connection between ARM and other time series causality studies. You might have noticed, ARM shares some similarities with previous studies like Granger Causality and Vector Autoregressive: firstly control each time series' own forecasting contribution (A) and then explore/test whether other time series provide additional forecasting contributions (Main Predictor + RM). However, since ARM does not perform any statistical tests and differs significantly in practice from these statistical methods, we chose not to include these discussions to avoid misleading.

Reviewer comment (Writing):
There is confusion about how Multi-kernel local smoothing helps capture temporal dependencies among different series due to notations. For example, what is X_j? Could the authors elaborate more?

Author response:
#### Q5. About the Computational Details in MKLS
We apologize for the confusion caused. MKLS employs multiple 1D CNNs with different kernel sizes to obtain various local views $\widetilde{X}_j$ of the input \(X\), and uses channel attention to calculate the weight of these local views on each output channel for a weighted average, resulting in a smoothed local view. Therefore, the output of the \(j\)th CNN should actually be computed as \(\widetilde{X}_j = \mathtt{Conv1d}_j \left(X \right)\). Our previous notation of \(X\) as \(X_j\) was incorrect and caused misunderstanding. This has been corrected in the revision. Please refer to the updated Figure 4 for more details on MKLS computations.

Reviewer comment (Experiments):
The paper demonstrates superior empirical performance for the forecasting task, but it is hard to see how significant the improvement is compared to other recent models such as PatchTST and DLinear. Do the authors have insights on the sensitivity of hyperparameters in the ARM module during the tuning/training process, since many learnable hyperparameters are introduced?

Author response:
### Q6. Insights on the Sensitivity of Hyperparameters in ARM Module

Thank you for this question. Indeed, the ARM effectively enhances the robustness of LTSF models towards hyperparameter selection. You can refer to section A.5 for our hyperparameter settings. We kept most of the hyperparameters fixed. The only adjustment was that we categorized datasets into small datasets (number of series \(C < 20\)) and large datasets (number of series \(C \geq 20\)). For small datasets, we used a model dimension of the main predictor \(d=16\) and 2 experts in MoE; for large datasets, we used \(d=64\) and 4 experts in MoE. Below, we discuss the improvements in hyperparameter selection brought by ARM.

**a. Lookback Length \(L_I\)**
Previous LTSF models attempted multiple \(L_I \in \{96, 192, 336, 720\}\) during the training phase and chose the optimal one, significantly increasing training costs and making it difficult to select the best \(L_I\) in practical applications. Due to the presence of adaptive EMA, MoE in AUEL, and MKLS, the model can effectively adjust the lookback and local views, making ARM less sensitive to the choice of \(L_I\). In all our experiments, we used only \(L_I=720\) and consistently achieved results surpassing previous SOTA models.

**b. Model Dimension \(d\) of Main Predictor**
With ARM, the optimal dimension \(d\) required for the main predictor is reduced (b.1), and it can adapt to different datasets with substantial variations in inter-series relationship intensity without adjusting \(d\) (b.2).

- **b.1 Reduction of Optimal \(d\)**
   After AUEL controls for univariate forecasting effects, the parameters of the main predictor are primarily used for modeling inter-series dependencies. As mentioned in the response to Q3, MoE distinguishes which series rely mainly on self-forecasting contribution. Therefore, the MKLS-enhanced main predictor only needs to model the remaining inter-series effects, effectively reducing the required \(d\). The following table shows the impact of using AUEL of temporal patterns on the optimal \(d\) in the Electricity dataset (\(C=321 > 20\)) and ETTm1 (\(C=7 < 20\)). For detailed analysis, please refer to section A.4.5 and Table 5.



| Model        | MoE (d=16)      | MoE (d=64)        | MoE (d=256)     | w/o MoE (d=16)   | w/o MoE (d=64)   | w/o MoE (d=256)  |
|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
| Metric       | MSE \| MAE      | MSE \| MAE        | MSE \| MAE      | MSE \| MAE      | MSE \| MAE      | MSE \| MAE      |
| Electricity (96) | 0.128 \| 0.228 | **0.126** \| **0.225** | 0.129 \| 0.229 | 0.313 \| 0.392  | 0.261 \| 0.348  | **0.256** \| **0.341** |
| Electricity (192) | 0.146 \| 0.242 | **0.142** \| **0.239** | 0.145 \| 0.245 | 0.299 \| 0.378  | 0.255 \| 0.349  | **0.251** \| **0.352** |
| ETTm1 (96)    | **0.287** \| **0.340** | 0.295 \| 0.348   | 0.300 \| 0.352 | 0.488 \| 0.460  | **0.467** \| **0.446** | 0.503 \| 0.468 |
| ETTm1 (192)   | **0.325** \| **0.371** | 0.342 \| 0.376   | 0.346 \| 0.385 | 0.531 \| 0.499  | **0.498** \| **0.482** | 0.574 \| 0.512 |

Reviewer comment (Experiments):
What are the complexity/runtime compute/memory requirements to incorporate these modules in the vanilla model? How does that compare to PatchTST and DLinear?

Author response:
### Q7. About the Complexity/Runtime Compute/Memory of ARM

We presented a comparison of the computational costs of Vanilla+ARM with previous models in **section A.7** and **Table 7** of the original paper. In the revision, we have added a comparison with DLinear. The results are also presented as follows. It can be observed that under the optimal hyperparameter settings of each model on the ETTm1 dataset, Vanilla+ARM has competetive computational costs compared to previous models.


|              | Vanilla+ARM (FLOPs) | Vanilla+ARM (Params) | Vanilla (FLOPs) | Vanilla (Params) | Autoformer (FLOPs) | Autoformer (Params) |
|:--:|:--:|:--:|:--:|:--:|:--:|:--:|
| \(X_P=96\)   | 426M                | 7.89M                | 244M            | 14.9M            | 10.9G               | 15.5M               |
| \(X_P=192\)  | 515M                | 10.4M                | 273M            | 17.5M            | 11.6G               | 15.5M               |
| \(X_P=336\)  | 664M                | 14.9M                | 316M            | 21.9M            | 12.7G               | 15.5M               |
| \(X_P=720\)  | 1.15G               | 30.6M                | 431M            | 37.8M            | 15.5G               | 15.5M               |

|              | Informer (FLOPs) | Informer (Params) | PatchTST (FLOPs) | PatchTST (Params) | DLinear (FLOPs) | DLinear (Params) |
|:--:|:--:|:--:|:--:|:--:|:--:|:--:|
| \(X_P=96\)   | 9.41G            | 11.3M             | 5.26G            | 4.87M             | 3.06M           | 138K             |
| \(X_P=192\)  | 10.1G            | 11.3M             | 5.31G            | 7.08M             | 6.10M           | 277K             |
| \(X_P=336\)  | 11.2G            | 11.3M             | 5.38G            | 10.4M             | 10.7M           | 485K             |
| \(X_P=720\)  | 14.0G            | 11.3M             | 5.58G            | 19.2M             | 22.8M           | 1.04M            |

Reviewer comment (Evaluation):
There are discrepancies regarding Table 1. In the original PatchTST paper on traffic datasets Table 3, for MAE: 0.249 for L_P = 96 and 0.256 for L_P = 192, which are different from 0.239 and 0.246 reported in this paper. Could the authors double check if they are consistent?

Author response:
### Q8. Discrepancies in Table 1

We are very grateful for your observation regarding the data entry errors in our paper. We have now corrected these two data points, which also makes our model appear more competitive. We have rechecked the data in Table 1 to ensure its correctness.

Reviewer comment (Presentation):
The figures and tables in the manuscript are very hard to read.

Author response:
We sincerely appreciate your suggestions regarding the formatting of our paper. Following your advice, we have made modifications to significantly enhance its readability.

a. We have increased the font size of Tables 1 and 2 and adjusted the layout of Table 2 to make it more readable.

b. All captions have been resized to a standard font size and have been succinctly revised for clarity.

c. The positions of Figures 2 and 3 have been swapped for a more appropriate presentation.

Reviewer comment (Presentation):
Font size of the figures and tables is very small.

Author response:
We sincerely appreciate your suggestions regarding the formatting of our paper. Following your advice, we have made modifications to significantly enhance its readability.

a. We have increased the font size of Tables 1 and 2 and adjusted the layout of Table 2 to make it more readable.

b. All captions have been resized to a standard font size and have been succinctly revised for clarity.

c. The positions of Figures 2 and 3 have been swapped for a more appropriate presentation.

Reviewer comment (Presentation):
The captions are super long and have a very small font size, which may violate the submission guideline.

Author response:
We sincerely appreciate your suggestions regarding the formatting of our paper. Following your advice, we have made modifications to significantly enhance its readability.

a. We have increased the font size of Tables 1 and 2 and adjusted the layout of Table 2 to make it more readable.

b. All captions have been resized to a standard font size and have been succinctly revised for clarity.

c. The positions of Figures 2 and 3 have been swapped for a more appropriate presentation.

Reviewer comment (Presentation):
The figures are arranged in an order different from how the text refers to them, requiring readers to jump back and forth to find the corresponding (sub-)figure.

Author response:
We sincerely appreciate your suggestions regarding the formatting of our paper. Following your advice, we have made modifications to significantly enhance its readability.

a. We have increased the font size of Tables 1 and 2 and adjusted the layout of Table 2 to make it more readable.

b. All captions have been resized to a standard font size and have been succinctly revised for clarity.

c. The positions of Figures 2 and 3 have been swapped for a more appropriate presentation.

Reviewer comment (Reproducibility):
It is not clear from the manuscript how MoE is operated.

Author response:
In AUEL, the MoE is responsible for building univariate temporal patterns. It adaptively allocates the most suitable univariate MLP predictor for each series. Here are the benefits of using MoE:
- Between the individual univariate models in Figure 3 (a) and the fully-shared-parameter models in Figure 3 (b), MoE in Figure 3 (c) can adapt to datasets with both similar and independent series. 
- The presence of MoE allows the main predictor to focus solely on the contributions of inter-series dependencies, thereby reducing the required number of parameters for the main predictor. 
- In inverse processing, MoE can modify the output of the main predictor, choosing whether to retain the forecasting with inter-series information modeled by the main predictor or to correct it to rely solely on the univariate forecasting of the current series.

We illustrate the input and output of MoE in Equations (2) and (3). In the preprocessing stage, for each series $i$, MoE builds temporal patterns for the $L_P$ part containing only constant values, based on the processed $\\widetilde{X}^{\*i}$ from the AUEL of distribution. This preprocessing stage of AUEL of temporal patterns is computed as $ \\widetilde{X}^{i} = \\left\[ \\widetilde{X}^{\*i}\_I  \\ \\ \\ \\ \\mathtt{MoE}\\left( \\left[ \\widetilde{X}^{\*i}\_I  \\ \\ \\ \\ \\widetilde{X}^{\*i}\_P \\right] \\right) \\right\] $. In the inverse processing stage, MoE determines whether to modify the output based on the input $ \\widetilde{X}^{i}\_I = \\widetilde{X}^{\*i}\_I $ of the main predictor and its output $ \\widehat{X}\_{ED}^{i} $, to balance learning intra-series and inter-series dependencies. This inverse processing stage of AUEL of temporal patterns is calculated as $\\widehat{X}\_M^{i} = \\mathtt{MoE}\\left( \\left[ \\widetilde{X}^{\*i}\_I  \\ \\ \\ \\ \\widehat{X}\_{ED}^{i} \\right] \\right)$.

For the specific computations within the MoE block, we employed the model structure from the Switch Transformer (Fedus et al., 2022), which uses routing operations to match each input series to the corresponding MLP predictor for computation. We also used residual connections on $L_P$ to ensure stable training. Details about these computation methods are described in section **A.6.1** of our paper, which we have also emphasized in the revision by bolding the reference in section 3.1. The description is presented as follows:

> For adaptive learning of temporal patterns, we utilize the MoE predictor structure as described in Switch Transformer (Fedus et al., 2022). We adjusted the output dimension of its final layer to cater to the input length of $L_I+L_P$, an output length of $L_P$, with a hidden dimension of $4(L_I+L_P)$. Given the discrepancy in input and output lengths, the MoE's residual shortcut needs specific modifications. We establish a residual connection using the \\(L_P\\) section of the input before the AUEL preprocessing and before the AUEL inverse processing. Specifically, in the calculation of MoE in AUEL preprocessing, where $\\widetilde{X}^{i} = \\left[ \\widetilde{X}^{\*i}\_I \\ \\ \\mathtt{MoE}\\left( \\left[ \\widetilde{X}^{*i}\_I \\ \\ \\widetilde{X}^{\*i}\_P \\right] \\right) \\right]$, we actually compute \\(\\mathtt{MoE}\\left( \\left[ \\widetilde{X}^{\*i}\_I \\ \\ \\widetilde{X}^{\*i}\_P \\right] \\right)=\\mathtt{MLP}\_{\\mathtt{selected}}\\left( \\left[ \\widetilde{X}^{\*i}\_I \\ \\ \\widetilde{X}^{\*i}\_P \\right] \\right) + \\mathbf{0}^i\_P\\). Here, \\(\\mathtt{MLP}\_{\\mathtt{selected}}\\) denotes the MLP predictor selected by the routing part of MoE based on the input, and \\(\\mathbf{0}^i\_P\\) is the \\(L\_P\\) part before AUEL preprocessing, i.e., the default 0 values before AUEL. During inverse processing, we compute \\(\\widehat{X}\_M^{i}=\\mathtt{MoE}\\left( \\left[ \\widetilde{X}^{\*i}\_I \\ \\ \\widehat{X}\_{ED}^{i} \\right] \\right)=\\mathtt{MLP}\_{\\mathtt{selected}}\\left( \\left[ \\widetilde{X}^{\*i}\_I \\ \\ \\widehat{X}\_{ED}^{i} \\right] \\right)+\\widehat{X}\_{ED}^{i}\\), where the added \\(\\widehat{X}\_{ED}^{i}\\) is the \\(L_P\\) part before the AUEL block.

Reviewer comment (Evaluation):
It is unclear why Vanilla+ARM performs better than applying ARM to other Transformer-based models.

Author response:
Thank you for your insightful question. Indeed, this is a scenario we anticipated during our model design process. Once ARM is applied, the primary role of the main predictor shifts to modeling inter-series dependencies. With the assistance of MKLS, a structurally simple yet effective predictor is required to process the reasonable multivariate representation constructed by MKLS. In cases where token representation is well-designed, the vanilla Transformer has been proven adaptable to a wide range of tasks. Autoformer, on the other hand, introduces an auto-correlation mechanism in the temporal dimension, an improvement that does not align with our goal of modeling inter-series relationships. Informer introduces sparse attention to increase computational efficiency, but this actually compromises performance compared to full attention. Nevertheless, it is worthy to note that, as seen in Table 2, Autoformer+ARM and Informer+ARM still demonstrate stable improvements over past SOTA models like PatchTST and DLinear without ARM.

---
We thank you again for your valuable question, which greatly contributes to improving the quality of our paper. If you have any further questions, we are keen to engage in more discussions.