Paper: Dozerformer: Sequence Adaptive Sparse Transformer for Multivariate Time Series Forecasting (ICLR-2024)

Reviewer comment (Theory):
In the second paragraph of the first page, the authors argue "predicting the value at horizon 1 using the entire historical sequence is suboptimal and inefficient.". This is the main motivation of vary attention in cross-attention modules. However, the reason for suboptimality is not provided in this paragraph. Why do we have to use varying-sequence attention?

Author response:
In Figure 1(a), the heatmap illustrates the correlation among time steps. Notably, distant time steps exhibit a generally low correlation, while those in close proximity display higher correlation. Existing transformer-based methods, generate predictions for all future time steps in a single forward operation. This involves utilizing the entire historical sequence even for a forecasting horizon of 1, resulting in the inclusion of many historical time steps that are not correlated with the target. This practice not only hinders the model's ability to effectively learn useful patterns but also involves unnecessary computations. To address this, we introduced the Vary component, which adaptively increases the historical time steps used in cross-attention as the forecasting horizon grows. This adaptive approach avoids unnecessary computations from time steps not correlated with the targets, thereby enhancing the learning of temporal dependencies.

Reviewer comment (Theory):
The target task is multivariate time series forecasting. In this task, considering inter-feature connections is also important as well as temporal connections [1]. However, there is a discrepancy between the characteristics of the main task and your method. In other words, Dozerformer doesn't incorporate any parts to consider inter-feature dependencies. Can you explain the reason for this design or Do I understand wrong?

Author response:
Yes, we adhere to the channel-independent design introduced by patchTST. Extensive ablation studies conducted by patchTST demonstrated the effectiveness of channel independence in the context of MTS forecasting. While we experimented with inter-dependency learning for Dozerformer, the results indicated a degradation in performance.

Reviewer comment (Theory):
In the second paragraph of the first page, the authors argue "They (might include Transformers with full attention) also ignored the characteristics of MTS data, like locality and seasonality.". This is the reason for local and stride attention. However, when observing Figure 1 (a), Transformers with full attention already capture locality and seasonality automatically. At this point, why do we have to make constraints on self-attention?

Author response:
The conventional full attention mechanism encompasses the entire historical sequence, resulting in the calculation of attentions between all time steps. However, this design presents two limitations: (1) Time steps that lack correlation with the forecasting targets are included in the attention calculation, leading to suboptimal model effectiveness and unnecessary computations. (2) The full attention mechanism is burdened by quadratic computation and memory usage. In response, we introduce the Dozer attention, which eliminates time steps lacking correlation with the targets in the attention mechanism. This removal allows the model to focus exclusively on learning data patterns from time steps correlated with the targets. Consequently, this sparse attention design not only alleviates constraints but also enhances the model's ability for temporal dependency learning and overall efficiency.

Reviewer comment (Reproducibility):
When decomposing time series into seasonal and trend parts, you might use average pooling and residual techniques. Although the same techniques are used in other papers, I recommend mentioning average pooling in your manuscript. Also, I'm curious about what kernel size is used for this average pooling.

Author response:
Yes, we adhered to the decomposition procedure outlined in MICN. In the appendix, we have provided a description indicating the use of average pooling. The kernel sizes employed were 13 and 17, consistent with the MICN setting. Further details about the decomposition procedure can be found in Appendix A.2.

Reviewer comment (Theory):
In Eq. (4), I think something is wrong because when i = 1 and v = 3, $t-v+i-t<0$. Indices cannot be negative. Can you fix it?

Author response:
$t$ represents the current time, and the index of historical time steps starts from $t-1$. The historical time steps utilized by the Vary component gradually increase as the forecasting horizon increases but stop at the maximum input length. We have included a sentence in the manuscript to explicitly state that all the time steps utilized by the Vary component fall within the historical window.

Reviewer comment (Evaluation):
In Section 4.3, you just give us theoretical complexity. Can you give a cost comparison in an empirical way such as flops or wall-clock time?

Author response:
As requested, we conducted experiments to compare the FLOPs of Dozerformer with baseline methods. Additionally, we evaluated efficiency in terms of Params (total number of learnable parameters) and Memory (maximum GPU memory consumption). The results are presented in Table 7 in the Appendix. Dlinear stands out as the only method more efficient than Dozerformer, thanks to its simple linear layer-only architecture. However, the proposed Dozerformer exhibits significantly greater efficiency than both transformer-based and CNN-based methods. For instance, its FLOPs value is only 5.8% of that of patchTST.

Reviewer comment (Experiments):
In Table 3. it seems that local, stride, and vary attention modules are not quite helpful for forecasting in ETTh1 and ETTm1. Can you give more experimental results to prove the efficacy of the three modules?

Author response:
In Table 4, we presented accuracy and efficiency metrics. The lower query-key pairs reduction rates for ETTh1 and ETTm1 compared to the other two datasets are due to specific hyperparameter settings. A larger stride value, smaller local window size, and smaller vary starting size result in a more sparse attention matrix, thus enhancing efficiency. Given the clear daily seasonality in the ETTh1 and ETTm1 datasets, we opted for a less sparse attention matrix.

Reviewer comment (Presentation):
Figure 1b-stride is a bit unclear. Why is it that t+1 and t+O are attending to different number of past steps. It is assumed that each of the figures in 1b refer to the corresponding specific component (such as 'full', 'local' etc.). Furthermore, the stride value in 1b-stride is not provided.

Author response:
Yes, each subfigure in Figure 1b represents a distinct component. Specifically, the subfigure titled "Stride" denotes the Stride component within the Dozer attention mechanism. The Stride component selectively attends queries to keys located at fixed intervals.  
In Figure 1b, the stride value is set to 3. The variance in historical time steps utilized by *t+1* and *t+O* stems from the fact that the historical time steps '*I*' in the figure amount to 8, with indexes ranging from *t-8* to *t-1*. Consequently, the selected time steps could be 2 or 3 for a sample rate of 3, determined by the starting index. To illustrate, when the forecasting target is *t+1*, the historical time steps selected for a stride of 3 are *t-3* and *t-6*. For a forecasting target at *t+2*, the historical time steps selected are *t-2*, *t-5*, and *t-8*.

Reviewer comment (Writing):
Section 3.1 is a bit unclear. How are the outputs of 1x1 convolution and the linear layers combined? It is assumed that the linear layer produces an output $\in \mathbb{R}^{I \times D_1}$ which is then projected to $X_{pred}$ after combining with the output of 1x1 convolution. However, it is not clear how the I rows of the $X_{t}$ matrix get converted to O rows so that they can be added to the output of 1x1 convolution.

Author response:
The Linear model serves as the trend model, tasked with directly inferring predictions for O future time steps of the trend component from I historical records. The outputs’ dimension of both the Linear model and the 1x1 convolution is OxD, after which they undergo elementwise summation to produce the final forecasting values. We express gratitude to the reviewer for highlighting that certain computation procedures were omitted in the manuscript due to page limits. Consequently, we have included a step-by-step detailed computation description of the framework in Appendix A.2.

Reviewer comment (Experiments):
The explorations are performed on datasets that contain signals that perhaps do not change at a very fast rate. It would be beneficial to have experiments on fast changing data, as the effectiveness of local, stride, and vary can be more clearly seen on such datasets.

Author response:
The nine benchmark datasets employed in the experiments vary in sample rate from 10 minutes to 1 week. Some datasets exhibit rapid changes. For instance, Figure 4 and Figure 5 visualize the ETTh1 dataset, illustrating that its values fluctuate between -4 and 1 in less than 24 time steps. If additional datasets with faster-changing characteristics are available and accessible via a provided download link, we are open to incorporating them. We believe that the current selection of nine widely used benchmark datasets already effectively demonstrates the efficacy of Dozerformer in terms of both accuracy and efficiency.

Reviewer comment (Evaluation):
It is not clear what the practical significance is of the difference in forecasted numbers from the different models. Ideally, the forecasted numbers would be used in another appropriate downstream task to show how the different forecasting mechanisms perform. MSE/MAE numbers may not be enough; for example, in Figures 4 and 5, several forecasting approaches produce plots that are very close to each other, and therefore might perform equivalently when the forecasted data is used for a downstream task.

Author response:
I concur with the reviewer. While the improvements in MSE and MAE values are notable, their impact on downstream tasks may not be substantial. As depicted in Figure 4 and Figure 5, the forecasting results of all methods exhibit close proximity. I acknowledge that their performance might be similar for downstream tasks. This is an aspect often overlooked in the field. Notably, recent state-of-the-art papers [1][2][3] have enhanced quantitative metrics, yet their predictions, as visualized, do not exhibit significant differences. It raises a pertinent issue regarding the potential lack of discernible performance distinctions in downstream tasks. I agree that further investigation into this matter is warranted, representing a gap in the field, particularly concerning the application of deep learning models in downstream tasks and understanding how forecasting accuracy influences these tasks.

Reviewer comment (Evaluation):
As efficiency is one of the focus areas of this work, it would be useful to have actual computation numbers (such as memory usage and wall-clock time) for the proposed approach. A comparison of such numbers with other baselines (if available) would also be valuable.

Author response:
As requested, we present experimental results comparing the proposed Dozerformer with baseline methods, illustrating the parameters (total number of learnable parameters), FLOPs (floating-point operations), and memory (maximum GPU memory consumption) of all methods in Table 7 in the appendix. Dlinear stands out as the only method more efficient than Dozerformer, owing to its simple linear layer-only architecture. However, the proposed Dozerformer exhibits significantly greater efficiency than both transformer-based and CNN-based methods. For instance, its FLOPs value is only 5.8% of that of patchTST.

Reviewer comment (Novelty):
It would be nice to have a structural comparison or analysis with Pyraformer, Informer, and PatchTST, which propose sparse attention. Furthermore, I would like to know which aspect of Dozerformer shows a better contribution.

Author response:
The Informer introduced a ProbSparse attention mechanism, selecting only the top-u queries based on query sparsity measurements. In contrast, the Dozer attention, informed by dataset characteristics (locality, seasonality, global correlations), avoids additional computations when choosing a subset of keys. As a comprehensive framework, the Dozerformer significantly outperformed the Informer in terms of accuracy, reducing MSE and MAE by 66.2% and 51.9%, respectively.
The Pyraformer proposed a pyramidal attention module that calculates attention within a C-ary tree. This mechanism selectively attends each query to a limited set of keys, categorized into three types: adjacent keys at the same level (scale) in the graph, keys of parent nodes connected to the query, and keys of child nodes connected to the query. Consequently, the Pyraformer models temporal dependencies at multiple resolutions. However, its notable drawback is the absence of modeling global temporal dependencies at a specific resolution; instead, it attempts to capture local temporal dependencies at a coarse scale. Consequently, its performance is less accurate compared to our proposed Dozerformer, which reduced MSE and MAE values by 64.8% and 51.6%, respectively.
We present a computational comparison of all methods in Table 7 in the manuscript's appendix. Notably, the Pyraformer and Informer stand out as two of the most efficient transformer-based methods. However, the Dozerformer achieved lower FLOPs than both methods, amounting to 1/10 of the Pyraformer and 1/15 of the Informer.
In conclusion, while both the Informer and Pyraformer have proposed sparse attention mechanisms, our proposed Dozerformer outperforms them significantly in terms of accuracy. Additionally, the Dozerformer exhibits greater efficiency, as evidenced by the FLOPs comparison

Reviewer comment (Evaluation):
Performance is not very good compared to PatchTST and Dlinear (LTSF-Linear). Is there any reason or basis for this?

Author response:
As outlined in the paper, the Dozerformer demonstrated a 0.4% reduction compared to PatchTST and an 8.3% reduction compared to Dlinear in terms of Mean Squared Error (MSE). We consider the accuracy improvements to be substantial compared to Dlinear. Notably, PatchTST employs the canonical full attention mechanism, while our method incorporates the proposed Dozer Attention, where each query selectively attends to a subset of keys. Consequently, this approach reduces computational complexity to linear. While the Dozerformer's accuracy improvement over PatchTST may be modest, it achieves a significant enhancement in computation and memory efficiency. Consider ETTh1 with a forecasting horizon of 96 and a historical sequence length of 720 as an example. In this case, the Floating Point Operations Per Second (FLOPS) for Dozerformer amount to 14.91, which is markedly lower than patchTST, which registers at 256.39. Additionally, the parameter count for Dozerformer is 1.61 million, substantially less than patchTST, which has 10.73 million parameters.

Reviewer comment (Novelty):
Although the motivation is clear, I think this paper lacks a point of differentiation from many existing similar studies.

Author response:
The primary contribution of this paper lies in the introduction of the Dozer attention mechanism, comprising three components: local, stride, and vary. The Vary component is particularly novel compared to existing forecasting methods, as it adaptively adjusts the utilization of historical time steps with an increasing forecasting horizon, illustrated in Figure 1 (b). Unlike other transformer-based methods that generate predictions for all future time steps in a single forward operation, employing the same number of historical time steps, the Dozer attention recognizes that time steps further from the forecasting target are less correlated. Thus, the optimal historical sequence length varies with different forecasting horizons, especially for datasets lacking periodicity, as depicted in Figure 1 (a). To the best of our knowledge, the proposed Dozer attention is the only time series forecasting method that dynamically adapts to varying historical records.

Reviewer comment (Novelty):
It would be nice to have a structural comparison or analysis with Pyraformer and Informer, which proposed sparse attention. Furthermore, I would like to know which aspect of Dozerformer shows better contribution.

Author response:
The Informer introduced a ProbSparse attention mechanism, selecting only the top-u queries based on query sparsity measurements. In contrast, the Dozer attention, informed by dataset characteristics (locality, seasonality, global correlations), avoids additional computations when choosing a subset of keys. As a comprehensive framework, the Dozerformer significantly outperformed the Informer in terms of accuracy, reducing MSE and MAE by 66.2% and 51.9%, respectively.
The Pyraformer proposed a pyramidal attention module that calculates attention within a C-ary tree. This mechanism selectively attends each query to a limited set of keys, categorized into three types: adjacent keys at the same level (scale) in the graph, keys of parent nodes connected to the query, and keys of child nodes connected to the query. Consequently, the Pyraformer models temporal dependencies at multiple resolutions. However, its notable drawback is the absence of modeling global temporal dependencies at a specific resolution; instead, it attempts to capture local temporal dependencies at a coarse scale. Consequently, its performance is less accurate compared to our proposed Dozerformer, which reduced MSE and MAE values by 64.8% and 51.6%, respectively.
We present a computational comparison of all methods in Table 7 in the manuscript's appendix. Notably, the Pyraformer and Informer stand out as two of the most efficient transformer-based methods. However, the Dozerformer achieved lower FLOPs than both methods, amounting to 1/10 of the Pyraformer and 1/15 of the Informer.
In conclusion, while both the Informer and Pyraformer have proposed sparse attention mechanisms, our proposed Dozerformer outperforms them significantly in terms of accuracy. Additionally, the Dozerformer exhibits greater efficiency, as evidenced by the FLOPs comparison

Reviewer comment (Evaluation):
Reported results over different benchmarks look promising; however, none of the experiments include any significance tests. So, it is hard to evaluate if the results are statistically significant or not. I would also suggest running some statistical significance tests when comparing results.

Author response:
**Author's response:** The main results presented in Table 1 represent the mean values obtained from six runs with distinct seeds. In compliance with the request, we computed the standard deviation (STD) across these six runs. Notably, for the ILI dataset, the results exhibit a comparatively large average STD in MSE and MAE, amounting to 0.164 and 0.053, respectively, across four forecasting horizons. The average range for these four horizons is 0.419 and 0.137. This discrepancy is attributed to the substantially smaller forecasting horizon in comparison to the other datasets. Nevertheless, the results consistently outperform baseline methods, except for patchTST, even for seeds generating higher MSE and MAE. Across the remaining eight datasets, the average STD for MSE and MAE stands at 0.002 and 0.001, with an average range of 0.007 and 0.003. This underscores the robustness of the results against randomness and underscores the consistently superior performance of the proposed Dozerformer.

Reviewer comment (Experiments):
One question: Did you only try grid search as your HP optimization technique (section 4.1)?

Author response:
**Author's response:** Yes, I only employed the grid search technique to select hyperparameters. For those parameters not chosen through grid search, we followed the settings of recent SOTA papers.