Paper: Correlated Attention in Transformers for Multivariate Time Series (ICLR-2024)

Reviewer comment (Experiments):
Lack of evaluation on prediction tasks: For prediction tasks, such as the MLTSF dataset, the paper does not provide an evaluation of the impact of the Correlated Attention Block (CAB) or compare it with other models that utilize inter-variable correlations. Including such evaluations and comparisons would provide a more comprehensive understanding of the effectiveness of CAB in prediction tasks.

Author response:
$\textbf{Q1:}$ "Can you provide a more detailed evaluation of the Correlated Attention Block (CAB) on prediction tasks, such as the MLTSF dataset? It would be interesting to see how CAB performs compared to other models that utilize inter-variable correlations in prediction tasks. This analysis would shed light on the effectiveness of CAB in different scenarios and provide a better understanding of its potential advantages."
 
 

$\textbf{Response:}$ As mentioned above, we have included additional experiments for Multivariate Long-term Time Series Forecasting in Appendix C in the revised manuscript. As shown therein, when paired with a strong baseline for long-term forecasting, CAB results in  competitive performance. Specifically, in the dataset like Exchange in our test, CAB gives significant performance boost for longer series sequence. With the flexibility of CAB, we look forward to its deployment on top of many existing Transformers for efficiency improvement.

Reviewer comment (Presentation):
Non-compliance with ICLR submission requirements: The paper does not follow the submission requirements of ICLR by placing the appendix together with the main text. It would be better to separate the appendix from the main text, following the formatting guidelines specified by the conference.

Author response:
$\textbf{Q4:}$ "Non-compliance with ICLR submission requirements: The paper does not follow the submission requirements of ICLR by placing the appendix together with the main text. It would be better to separate the appendix from the main text, following the formatting guidelines specified by the conference."


$\textbf{Response:}$ We thank you for that, and will separate the main text from the supplementaries in later versions including our revised manuscript now.

Reviewer comment (Experiments):
Can you provide a more detailed evaluation of the Correlated Attention Block (CAB) on prediction tasks, such as the MLTSF dataset? It would be interesting to see how CAB performs compared to other models that utilize inter-variable correlations in prediction tasks. This analysis would shed light on the effectiveness of CAB in different scenarios and provide a better understanding of its potential advantages.

Author response:
$\textbf{Q1:}$ "Can you provide a more detailed evaluation of the Correlated Attention Block (CAB) on prediction tasks, such as the MLTSF dataset? It would be interesting to see how CAB performs compared to other models that utilize inter-variable correlations in prediction tasks. This analysis would shed light on the effectiveness of CAB in different scenarios and provide a better understanding of its potential advantages."
 
 

$\textbf{Response:}$ As mentioned above, we have included additional experiments for Multivariate Long-term Time Series Forecasting in Appendix C in the revised manuscript. As shown therein, when paired with a strong baseline for long-term forecasting, CAB results in  competitive performance. Specifically, in the dataset like Exchange in our test, CAB gives significant performance boost for longer series sequence. With the flexibility of CAB, we look forward to its deployment on top of many existing Transformers for efficiency improvement.

Reviewer comment (Evaluation):
In Table 2 and Table 3, it is observed that in some cases, the performance of CAB+Transformer is not as good as Nonstationary, and in some cases, Nonstationary+CAB even leads to worse results. Can you provide an explanation for these observations? What factors contribute to the varying performance of the proposed method in different settings? Understanding the limitations and potential trade-offs of the proposed method would provide valuable insights for future improvements.

Author response:
$\textbf{Q2:}$ "In Table 2 and Table 3, it is observed that in some cases, the performance of CAB+Transformer is not as good as Nonstationary, and in some cases, Nonstationary+CAB even leads to worse results. Can you provide an explanation for these observations? What factors contribute to the varying performance of the proposed method in different settings? Understanding the limitations and potential trade-offs of the proposed method would provide valuable insights for future improvements."


$\textbf{Response:}$ For the anomaly detection task in Table 3, we note that the main metrics within interest is usually the F1 score, which is the harmonic mean of precision (P) and recall (R), thereby giving balance between the two. In terms of F1, CAB actually provides consistent improvement when plugged into both Transformer and Non-stationary Transformer, where decrease in F1 score is only in one dataset for each base model. For the imputation task in Table 2, this task differs from other tasks in that it involves missing data. This can make the representation learning in CAB harder, as the lagged cross-correlation filtering step is sensitive to time step data. For example, if a time lag with high correlation is yet missing, CAB would not have data to detect such potentiall lagged cross-correlation.

Reviewer comment (Writing):
Some parts are a bit confusing, for instance, “CrossFormer deploys a convoluted architecture, which is isolated from other prevalent Transformers with their own established merits in temporal modeling and specifically designed for only MTS forecasting, thereby lacking flexibility.” The reviewer is a bit confused and requests more detailed explanation.

Author response:
$\textbf{Q1:}$ " “CrossFormer deploys a convoluted architecture, which is isolated from other prevalent Transformers with their own established merits in temporal modeling and specifically designed for only MTS forecasting, thereby lacking flexibility” I am a bit confused, could you explain more in detail this?"

$\textbf{Response:}$ We just want to discuss that CrossFormer's architecture is very unconventional from other prevalent Transformer with simple multi-head attention that yet has been  shown to be very good in temporal modeling. By deviating from the very base architecture of vanilla Transformer, Crossformer may not utilize such advantages; for your information, Crossformer deploys a Hierarchical Encoder-Decoder architecture with Dimension-Segment-Wise embedding, which devides time into segmants,  Two-Stage Attention and  segment merging, which merges back the time. We thank you for the suggestion, and will re-write it for more clarity in the updated versions in the future.

Reviewer comment (Writing):
It is confusing how the value of k in equation 6 is defined — whether you get the value of c first and then calculate k with the topK operation. It is not very clear why not directly take top k (for instance, top 5 lagging values) and use it instead of getting a value k by using the topK operator. Any motivation behind this choice is requested.

Author response:
$\textbf{Q2:}$ "It is confusing how the value of k in equation 6 is defined, do you get the value of c first and then calculate k with the topK operation? It is not very clear to me why not directly take top k, for instance, top 5, lagging value, and use it instead of getting a value k by using the topK operator? Any motivation behind?"

$\textbf{Response:}$ We would like to clarify that such $k = c \log(T)$ is indeed the input $k$ to TopK operation, instead of getting a value $k$ out of topK operator. So yes, you would choose $c$ first, which is a hyperparameter, and then use $k = c \log(T)$ as the input to TopK. Also, motivation for doing $k = c \log(T)$ is that even if one uses FFT  of $O(T \log(T))$ for fast computation, your final computational cost will still depend on $O(d^2 T \cdot k  )$ for summing up $k$ terms in Eq. (6), so setting $k = c \log(T) = O(\log(T))$ would allow you to maintain such $O(T \log(T))$ complexity.

Reviewer comment (Theory):
The reviewer is not sure how to go from equation 7 to the result presented in the section below, and suggests possibly providing some proof.

Author response:
$\textbf{Q3:}$ "I am not sure how to go from equation 7 to the result they got in the section below, maye some proof?"

$\textbf{Response:}$ We would like to explain it in more details here. First, please note that Eq (7) 
$S_{X, Y}(f) = F(X_t) F^*(Y_t)$ should be interpreted as $S_{X, Y}(f) = F(X_t)(f) \cdot F^*(Y_t)(f)$. Note that in the two equations in Eq Eq (7), the variable $f$ corresponds to frequency domain and $l$ corresponds to time domain. While we write $S_{X, Y}(f)$  and $(X \star Y)(l)$ in point-wise form as in the paper, which demonstrates the Cross-correlation Theorem and how you can compute  $(X \star Y)(l)=  \sum_{t=1}^{T} X_{t-l} Y_t$ via FFT and inverse FFT, the actual implementation using Pytorch's FFT library would let you input, for example in case of $d=1$ feature, the vector $X_t$  of size $T$ in time domain (i.e. a vector of all $l$ stacked together), and perform FFT in $O(T \log(T))$ to return you the vector $F(X_t)$ of size $T/2+1$ in frequency domain (i.e. a vector of all $f$ stacked together). From there, you can compute $S_{X, Y}(f) = F(X_t) F^*(Y_t)$, which is again a vector  of all $f$ stacked together.  From the second equation in Eq (7), you can now  feed such vector into the inverse FFT to get the vector  $(X \star Y)(l)= F^{-1}( S_{X, Y}(f) )$ of all $l = 0\to T-1$ stacked together. We hope that this can clarify your concern.

Reviewer comment (Theory):
It seems that even with FFT, the computational complexity is still quite high for a time series with a large feature dimension.

Author response:
$\textbf{Q4:}$ "It seems even with FFT, the computational complexity is still quite high for a time series with a large feature dimension."

$\textbf{Response:}$ We kindly refer the reviewer to our new experiments on run-time of the baselines in Appendix D of the manuscript. It shows that the overhead incurred by our CAB to the base models is quite minimal, so the resulting models still maintain around the same order of magnitude in computation as other SOTA Transformers. We do agree with you CAB will have another multiplicative factor of $d$ in its complexity, yet CAB has advantage in $T$, which has been usually the real computational bottleneck to be addressed in the literature of Transformers' computational complexities. Further acceleration in $d$ would be interesting future direction.

Reviewer comment (Experiments):
When evaluating a specific task, it is crucial to compare its performance with a model that has been explicitly designed and optimized for that particular task. For instance, both non-stationary transformers, dlinear and FEDformer are designed for forecasting tasks. The reviewer is not 100% sure whether it is a fair comparison when applying those to classification and anomaly detection tasks.

Author response:
$\textbf{Q5:}$ "When evaluating a specific task, it is crucial to compare its performance with a model that has been explicitly designed and optimized for that particular task. For instance, both non-stationary transformers, dlinear and FEDformer are designed for forecasting tasks. The reviewer are not 100% sure that whether it is a fair comparison when applying those to classification, and anomaly detection tasks."

$\textbf{Response:}$ We kindly note that, as mentioned in our original submission, Nonstationary Transformer and FEDformer are in fact the state-of-the-art (SOTA) Transformer-models for respectively imputation and anomaly detection in the recent benchmarks (Wu et al., 2023). 
Yes, while they were designed for forecasting, their architectures were quite robust and good enought to achieve competitive results on even encoder-only tasks. For the overall SOTA, it has been TimesNet for all of these tasks. 
In our revised manuscript, for classficiation task, we have further added Flowformer, which is the SOTA Transformer for classification on these 10 datasets, for completeness. So our experiments for all the 3 encoder-only tasks now include an overall SOTA baseline and a SOTA Transformer for each of the task.

Reviewer comment (Evaluation):
In anomaly detection and classification, transformer + CBA shows significant improvement compared to transformer, but this was not observed in the imputation task. The reviewer requests any insights into this difference.

Author response:
$\textbf{Q7:}$ "In anomaly detection and classification, it shows that transformer +CBA has significant improvement compared to transformer, this was not observed in the imputation task, any insights into that?"

$\textbf{Response:}$ We believe that this may come from two reasons. First, unlike the other tasks, imputation involves missing data, which makes the learning of \emph{lagged} cross-correlation, which is sensitive to time steps, harder. Second, the Transformer baseline itself is not good or compatible for imputation, where you can see on some datasets like ETTm2, ETTh1, ETTh2 and Electricity, its error is order of magnitude larger than the SOTA baselines.

Reviewer comment (Experiments):
The reviewer suggests that it would be interesting to see the performance on the forecasting task as well, since there is nothing in the design that specifically restricts it to non-predictive tasks.

Author response:
$\textbf{Q8:}$ "It would be interesting to see the performance on the forecasting task as well I think since there is nothing in the design that specifically restricts it to the non-predictive task.";
"I am missing the discussion what is the difference when you learning the feature-wise correlation for forecasting or for non-predictive task and what is in this model that makes it more fit for the non-predictive task"


$\textbf{Response:}$ Please find our new experiments on long-term forecasting on three datasets spanning different domains in Appendix C of the paper. CAB demonstrates consistent performance boost when plugged into Non-stationary Transformer as the base model. It in fact outperforms  recent SOTA Transformer models designed for long-term forecasting. 

For forecasting, we believe one challenge to fully utilize the potential of CAB is to have a proper design of the masking mechanism. First, we recall the main difference between the decoder (used in forecasting) and the encoder is that the decoder has two multi-head attention (MHA) blocks: one with masking and one without masking (i.e. conventional MHA). The masking is meant to prevent information to preserve the auto-regressive property, so that the forecasting of the next time step should depend only on the time steps strictly before it. In typical self-attention mechanism, this is done by  masking out (setting to $-\infty$) all values in the lower triangular of the scoring matrix $Q K^T\in \mathbb{R}^{T \times T}$ that is input to softmax. As you can see, $Q K^T$ can be thought of as the attention/correlation matrix in time (with dimension $T\times T$), so masking out the triangular part corresponds to "blinding" the history in making predictions; one the other hand, in CAB, the $Roll(K, l)^T Q \in \mathbb{R}^{d_k \times d_k}$, which is input to softmax, is correlation matrix across feature with dimension $d_k \times d_k$ instead of time, so the same masking strategy, even if naively applicable, would not have intuitive or physical phenomena as in vanilla self-attention, thereby being more of a heuristic approach. In the experiments in Appendix C, we only replace the non-masked MHA with our mixture-of-head attentions comprised of temporal attentions and our CAB, while keeping the masked MHA of the base Transformer. As you can see from the results, the CAB consistently improves performance of Non-stationary Transformer across all the tested datasets and resulting in the model that outperforms the previous SOTA Transformers for long-term forecasting.

Reviewer comment (Theory):
The paper’s main motivation was addressing the feature-wise correlation specifically for non-predictive tasks, but the reviewer is missing a discussion about the difference when learning feature-wise correlation for forecasting versus for non-predictive tasks, and what in this model makes it more fit for non-predictive tasks.

Author response:
$\textbf{Q8:}$ "It would be interesting to see the performance on the forecasting task as well I think since there is nothing in the design that specifically restricts it to the non-predictive task.";
"I am missing the discussion what is the difference when you learning the feature-wise correlation for forecasting or for non-predictive task and what is in this model that makes it more fit for the non-predictive task"


$\textbf{Response:}$ Please find our new experiments on long-term forecasting on three datasets spanning different domains in Appendix C of the paper. CAB demonstrates consistent performance boost when plugged into Non-stationary Transformer as the base model. It in fact outperforms  recent SOTA Transformer models designed for long-term forecasting. 

For forecasting, we believe one challenge to fully utilize the potential of CAB is to have a proper design of the masking mechanism. First, we recall the main difference between the decoder (used in forecasting) and the encoder is that the decoder has two multi-head attention (MHA) blocks: one with masking and one without masking (i.e. conventional MHA). The masking is meant to prevent information to preserve the auto-regressive property, so that the forecasting of the next time step should depend only on the time steps strictly before it. In typical self-attention mechanism, this is done by  masking out (setting to $-\infty$) all values in the lower triangular of the scoring matrix $Q K^T\in \mathbb{R}^{T \times T}$ that is input to softmax. As you can see, $Q K^T$ can be thought of as the attention/correlation matrix in time (with dimension $T\times T$), so masking out the triangular part corresponds to "blinding" the history in making predictions; one the other hand, in CAB, the $Roll(K, l)^T Q \in \mathbb{R}^{d_k \times d_k}$, which is input to softmax, is correlation matrix across feature with dimension $d_k \times d_k$ instead of time, so the same masking strategy, even if naively applicable, would not have intuitive or physical phenomena as in vanilla self-attention, thereby being more of a heuristic approach. In the experiments in Appendix C, we only replace the non-masked MHA with our mixture-of-head attentions comprised of temporal attentions and our CAB, while keeping the masked MHA of the base Transformer. As you can see from the results, the CAB consistently improves performance of Non-stationary Transformer across all the tested datasets and resulting in the model that outperforms the previous SOTA Transformers for long-term forecasting.

Reviewer comment (Novelty):
The novelty of the paper is limited. The proposed correlated attention is basically an extension for Autoformer, which only captures auto-correlation. However, this method neither proposes a good method to reduce the computational complexity caused by calculating cross-correlation, which is almost unacceptable in actual scenarios, nor does the author conduct a comparative experiment with Autoformer to prove that the introduction of cross-correlation can bring practical improvements.

Author response:
$\textbf{Q1:}$ "The novelty of the paper is limited. The proposed correlated attention is basically a extension for Autoformer, which only captures auto-correlation. ";
"The design of CAB is based on autoformer. Why is there no comparison between the effects of CAB and autoformer?"


$\textbf{Response:}$ We believe that our proposed CAB (correlated attention block) has many differences from the Autoformer, and have provided a paragraph (highlighted in blue) at the end of Section 3.2.2 in the revised manuscript to emphasize the differences. We would like to next re-iterate our discussion in the paper here in our response for your convenience. 

Since the CAB aims to capture the lagged cross-correlation,which is relevant to yet more generalized than the auto-correlation module in Autoformer, we believe it is crucial to emphasize the main differences. First, Autoformer overall is a decomposed encoder-decoder architecture proposed for long-term forecasting, so its auto-correlation module is specifically designed to work with series seasonalities extracted from various series decomposition steps of Autoformer. On the other hand, CAB ensures flexibility with any input series representation by deploying normalization step and learnable temperature coefficient $\tau$ reweighting the correlation matrices. Second, while Autoformer computes purely auto-correlation scores and aggregates their exact values for TopK , CAB computes cross-correlation matrices and aggregates the absolute values of such entries for TopK in Equation 5 (as correlation can stem from either positive or negative correlation). Finally, to facilitate robustness to different input series representation, CAB adopts learnable weights $\lambda$ in TopK operation, which balances between auto-correlation and cross-correlation, and $\beta$ in sub-series aggregation, which balances between instantaneous and lagged cross-correlation.

Reviewer comment (Novelty):
The integration of correlated attention to the existing transformer structure is conducted with a mixture-of-head attention structure, which is a concatenation of CAB and transformer outputs. CAB acts as a rather independent component and is not truly integrated into existing transformer structures.

Author response:
$\textbf{Q3:}$ "The integration of correlated attention to existing transformer structure is conducted with a mixture-of-head attention structure, which is a concatenation of CAB and transformer outputs. CAB acts as a rather independent component and does not truly integrated into existing transformer structures."


$\textbf{Response:}$ In fact, we believe that this modularity of CAB is its crucial benefit, since it can be seamessly integrated with many existing architectures. In order to facilitate such flexibility and robustness to any input series representation (by the base Transformer models), we leverage many sub-components for representation learning in CAB design: normalization step and learnable parameters at the core steps like $\tau$ for normalizing correlation matrices, $\lambda$ for balancing auto-correlation and cross-correlation, and $\beta$ for balancing instantaneous and lagged cross-correlation.

Reviewer comment (Experiments):
Judging from the design of CAB, it can predict independently without requiring additional transformer deconstruction. Why didn't you test the independent CAB?

Author response:
$\textbf{Q4:}$ "Judging from the design of CAB, it can predict independently without requiring additional transformer deconstruction. Why didn't you test the independent CAB?" 

$\textbf{Response:}$ In fact, we did test the independent CAB in our original submission's Appendix. We kindly refer the reviewer to our ablation studies in Appendix B. It is the pure-CAB-Transformer baseline, which purely uses CAB (i.e. no temporal attention from any base Transformer) and even disables lagged cross-correlation filtering. pure-CAB-Transformer itself also resulted improvement over Autoformer.

Reviewer comment (Writing):
Page 9, Line 1 of **Conclusion And Future Work**: There's a minor typo that needs correction - "bloc" should be changed to "block."

Author response:
$\textbf{Q4:}$ Typos and reference format 

$\textbf{Response:}$ We thank the reviewer very much for all the suggestions in presentation. We have fixed the typos into the revised manuscript, and will unify the reference format right in the next update. (As our reference list is actually quite long and of 3.5 pages, it will take some time for us to properly take care of all of the references.)

Reviewer comment (Presentation):
Citation Style: The reference list shows some inconsistency in the citation style. To enhance clarity and uniformity, consider standardizing the format across all references. For example, you could list all NeurIPS papers with consistent formatting, and for papers from other conferences or sources, ensure that their respective publication details are included appropriately. For instance:  
- NeurIPS papers should consistently include the conference name and URL. For example, "Vaswani et al. (2017, NeurIPS, URL) and Shen et al. (2020, NeurIPS, URL)."  
- Papers from other conferences or sources should similarly follow a consistent format, such as including the conference name and URL as needed. For example, "Cao et al. (2020, Conference Name, URL)" and "Li et al. (2019, Conference Name, URL)."

Author response:
$\textbf{Q4:}$ Typos and reference format 

$\textbf{Response:}$ We thank the reviewer very much for all the suggestions in presentation. We have fixed the typos into the revised manuscript, and will unify the reference format right in the next update. (As our reference list is actually quite long and of 3.5 pages, it will take some time for us to properly take care of all of the references.)

Reviewer comment (Writing):
Motivation for Correlated Attention Block: While the paper mentions that CAB efficiently learns feature-wise dependencies, it would be beneficial to provide more clarity on the specific aspects of CAB's design that contribute to this efficiency. Clearly articulating which components within the CAB block are the key drivers of this efficiency could help readers better understand the innovation.

Author response:
$\textbf{Q1:}$ "It would be beneficial to provide more clarity on the specific aspects of CAB's design that contribute to this efficiency.";
"Could you provide a detailed breakdown of the distinct contributions of each component within CAB, both in terms of performance enhancement and efficiency gains?"

$\textbf{Response:}$ We kindly refer the reviewer to the ablation studies in Appendix B for performance analysis of specific aspects of CAB. In the revised manuscript, we added blue text further highligting the most crucial components of the CAB. For a summary, from the empirical findings, the lagged cross-correlation filtering component (even when we hard-fix the learnable parameters as constants) resulted in highest increase in the accuracy of 1.39\% from the basic CAB model without lagged cross-correlation. Then, given that the lagged-cross correlation filtering is used, letting the parameter $\beta$, which is in Eq. (6) of our paper and balances the effect of instantaneous and lagged cross-correlation, be learnable would result in the highest increase in accuracy of 1.81\%.

Reviewer comment (Writing):
Limitation of Encoder-Only Models: It's important to acknowledge that the design of CAB is limited to encoder-only models and does not support time series forecasting. While this limitation is briefly mentioned, expanding on the reasons behind this constraint and discussing potential avenues for future work or extensions to address this limitation would add depth to the paper.

Author response:
$\textbf{Q2:}$  "Could you elaborate on the primary challenges or obstacles preventing the integration of CAB into encoder-decoder models for conducting multivariate time series forecasting?";
"It's important to acknowledge that the design of CAB is limited to encoder-only models and does not support time series forecasting. While this limitation is briefly mentioned, expanding on the reasons behind this constraint and discussing potential avenues for future work or extensions to address this limitation would add depth to the paper."

$\textbf{Response:}$ Before providing detailed explanations, we would like to note that:
-  We provide new experiments in Appendix C of the revised manuscript demonstrating that the CAB, even when naively plugged into the decoder architecture of base architectures, indeed consistently improves the performance of such Transformer-models over three long-term forecasting datasets spanning different domains!   
- We believe that the integration of CAB into the decoder can be done even better in the future with proper design of masking mechanism, as discussed in details below. 

First, we recall the main difference between the decoder and the encoder is that the decoder has two multi-head attention (MHA) blocks: one with masking and one without masking (i.e. conventional MHA). The masking is meant to prevent information to preserve the auto-regressive property, so that the forecasting of the next time step should depend only on the time steps strictly before it. In typical self-attention mechanism, this is done by  masking out (setting to $-\infty$) all values in the lower triangular of the scoring matrix $Q K^T\in \mathbb{R}^{T \times T}$ that is input to softmax. As you can see, $Q K^T$ can be thought of as the attention/correlation matrix in time (with dimension $T\times T$), so masking out the triangular part corresponds to "blinding" the history in making predictions; one the other hand, in CAB, the $Roll(K, l)^T Q \in \mathbb{R}^{d_k \times d_k}$, which is input to softmax, is correlation matrix across feature with dimension $d_k \times d_k$ instead of time, so the same masking strategy, even if naively applicable, would not have intuitive or physical phenomena as in vanilla self-attention, thereby being more of a heuristic approach. In the experiments in Appendix C, we only replace the non-masked MHA with our mixture-of-head attetions comprised of temporal attentions and our CAB, while keeping the masked MHA of the base Transformer. As you can see from the results, the CAB consistently improves performance of Non-stationary Transformer across all the tested datasets and resulting in the model that outperforms the previous SOTA Transformers for long-term forecasting.

Reviewer comment (Evaluation):
In Table 2, it is evident that for the first three datasets, the TimesNet baseline consistently outperforms CAB when the mask ratio exceeds 25%. Could you provide insights into this performance discrepancy?

Author response:
$\textbf{Q5:}$ "In Table 2, it is evident that for the first three datasets, the TimesNet baseline consistently outperforms CAB when the mask ratio exceeds 25%. Could you provide insights into this performance discrepancy?"

$\textbf{Response:}$ First, we note that the TimesNet baseline mostly outpeforms CAB on those ETTm1, ETTm2, and ETTh1 datasets. As these datasets do not have very large number of features (7 features), there can be less benefit from CAB. Second, unlike other task, imputation involves missing data, which may affect the learning of \emph{lagged} cross-correlation in view of the missing lags or time steps, so CAB may perform less effectively when the mask ratio is too large. Nevertheless, we would like to note that the  overall performance (averaged over all mask ratio) of Non-stationary+CAB is still better than TimesNet.

Reviewer comment (Experiments):
Could you elaborate on the primary challenges or obstacles preventing the integration of CAB into encoder-decoder models for conducting multivariate time series forecasting?

Author response:
$\textbf{Q2:}$  "Could you elaborate on the primary challenges or obstacles preventing the integration of CAB into encoder-decoder models for conducting multivariate time series forecasting?";
"It's important to acknowledge that the design of CAB is limited to encoder-only models and does not support time series forecasting. While this limitation is briefly mentioned, expanding on the reasons behind this constraint and discussing potential avenues for future work or extensions to address this limitation would add depth to the paper."

$\textbf{Response:}$ Before providing detailed explanations, we would like to note that:
-  We provide new experiments in Appendix C of the revised manuscript demonstrating that the CAB, even when naively plugged into the decoder architecture of base architectures, indeed consistently improves the performance of such Transformer-models over three long-term forecasting datasets spanning different domains!   
- We believe that the integration of CAB into the decoder can be done even better in the future with proper design of masking mechanism, as discussed in details below. 

First, we recall the main difference between the decoder and the encoder is that the decoder has two multi-head attention (MHA) blocks: one with masking and one without masking (i.e. conventional MHA). The masking is meant to prevent information to preserve the auto-regressive property, so that the forecasting of the next time step should depend only on the time steps strictly before it. In typical self-attention mechanism, this is done by  masking out (setting to $-\infty$) all values in the lower triangular of the scoring matrix $Q K^T\in \mathbb{R}^{T \times T}$ that is input to softmax. As you can see, $Q K^T$ can be thought of as the attention/correlation matrix in time (with dimension $T\times T$), so masking out the triangular part corresponds to "blinding" the history in making predictions; one the other hand, in CAB, the $Roll(K, l)^T Q \in \mathbb{R}^{d_k \times d_k}$, which is input to softmax, is correlation matrix across feature with dimension $d_k \times d_k$ instead of time, so the same masking strategy, even if naively applicable, would not have intuitive or physical phenomena as in vanilla self-attention, thereby being more of a heuristic approach. In the experiments in Appendix C, we only replace the non-masked MHA with our mixture-of-head attetions comprised of temporal attentions and our CAB, while keeping the masked MHA of the base Transformer. As you can see from the results, the CAB consistently improves performance of Non-stationary Transformer across all the tested datasets and resulting in the model that outperforms the previous SOTA Transformers for long-term forecasting.

Reviewer comment (Experiments):
Could you provide a detailed breakdown of the distinct contributions of each component within CAB, both in terms of performance enhancement and efficiency gains?

Author response:
$\textbf{Q1:}$ "It would be beneficial to provide more clarity on the specific aspects of CAB's design that contribute to this efficiency.";
"Could you provide a detailed breakdown of the distinct contributions of each component within CAB, both in terms of performance enhancement and efficiency gains?"

$\textbf{Response:}$ We kindly refer the reviewer to the ablation studies in Appendix B for performance analysis of specific aspects of CAB. In the revised manuscript, we added blue text further highligting the most crucial components of the CAB. For a summary, from the empirical findings, the lagged cross-correlation filtering component (even when we hard-fix the learnable parameters as constants) resulted in highest increase in the accuracy of 1.39\% from the basic CAB model without lagged cross-correlation. Then, given that the lagged-cross correlation filtering is used, letting the parameter $\beta$, which is in Eq. (6) of our paper and balances the effect of instantaneous and lagged cross-correlation, be learnable would result in the highest increase in accuracy of 1.81\%.