Paper: Divide-and-Conquer Time Series Forecasting with Auto-Frequency-Correlation via Cross-Channel Attention (ICLR-2024)

Reviewer comment (Novelty):
Design of the framework. The divide-and-conquer framework seems to be a multiple path/view framework that has been extensively studied in the previous literature and it is not surprising that FreCoformer obtains competitive performance as it utilizes the information from both the time domain and frequency domain.

Author response:
(1) Thank you again for the valuable comments. 
In this paper, we would like to provide a "divided-and-conquer" modeling strategy for time series forecasting studies. 
This strategy is the spirit of the proposed framework.
That is, we argue a successful forecasting approach might be designed on top of scenarios-specialized modules to capture eventful characteristics, rather than a unified framework.
Please kindly refer to the Reviewer-cikd's (the first reviewer) reply.

Reviewer comment (Evaluation):
Training/inference efficiency. The following question from point 1 is how is the training/inference efficiency of FreCoformer compared to other competing methods as it computes on different domains. The authors have only compared the GPU memory between FreCoformer and other methods, which is not the most important metric to evaluate the efficiency. For example, training time/number of parameters/FLOPs/GPU latency are commonly used metrics for efficiency evaluation.

Author response:
(2) Thank you for your question regarding our choice to use GPU memory usage as a metric in our study. Our decision is informed by related works in the field, especially in works focusing on transformer-based works [1].
In recent years, a significant challenge has been their high demand for computational memory. The core context of the Transformer is the attention calculation that involves a number of matrix multiplications and matrix storages.
This is essentially relevant when processing long input sequences. That is, low memory cost allows us to use longer time series observations for improving forecasting accuracy [4].

Reviewer comment (Novelty):
Core contribution. The authors have proposed lots of modules and designs in this work that may deviate from the core of 'divide-and-conquer'. For example, NYSTROM-FRECOFORMER just simply adopts the idea from NystromFormer and can hardly be claimed as the contribution of this paper. It is advised the authors rethink what in essence they are trying to present.

Author response:
(3) We would summarize our contributions here: 

1) FeCoformer for enhancement of capturing frequency features;

2) Divide-and-conquer modeling strategy;

3) As we mentioned in (2), memory cost is one of the main focuses in time series forecasting. 
Existing methods address this challenge using architecturally complicated Transformer-based model designs [1, 2].
This direction often decreases the forecasting performance.
Namely, recent work [4] uses a very simple linear model that can outperform all of the previous models on a variety of common benchmarks, and it challenges the usefulness of Transformer for time series forecasting.
Our work does not follow the line in this direction, and we draw attention to the computation mechanism of the Transformer, i.e., the matrix issues in memory cost. We hence introduce matrix approximation methods to address this challenge without any architectural care.
This attempt opens a new possibility for time series forecasting.

Reviewer comment (Reproducibility):
In Section 3.3, the authors say that "To compute the attention matrix A, we first select m landmark columns from the input Qn and Kn matrices in each channel.". Can you give the detailed procedure to select $m$ landmark columns from entire ones?

Author response:
(3) In our implementation of the Nystromformer model [6], the selection of $m$ landmarks is determined through the process described in the following:

The process involves dividing the input sequence into equal-sized segments and then summarizing each segment to form a landmark. Specifically, for a sequence of length $n$, we calculate the stride length $l$ as the ceiling of $n/m$ and then perform a sum reduction over each segment of length $l$ in the query and key matrices. 
This results in $m$ landmarks that represent the original sequence.

Reviewer comment (Presentation):
In Figure 3 (a), the authors say that "Compared to the best-performing PatchTST, our model exhibits an advantage in identifying short-term variations, resulting in detailed fluctuations in periodicity variation." However, I don't know what parts show these results. Can you mark what parts provide this information?

Author response:
(4) This result can be found in Fig.3(a) of the main body.

Among these comparisons, PatchTST (blue line) and ours (red line) have satisfied data fit (dodgerblue discrete points).
Compared to PatchTST, our method can further capture some local fluctuations upon the periodic changes.

Reviewer comment (Writing):
In 'Abstract', the authors argue that "The patching process refines the frequency information, enhancing the locality.". What kind of locality is enhanced by the patching process? Locality on the time domain is quite intuitive but one on the frequency domain is not. I think that you have to elaborate on locality on the frequency domain and why we have to utilize the frequency locality.

Author response:
Question 1: 

As we know, low-frequency oscillation represents the long-term variation or periodicity, and the frequency locality usually refers to the short-term variations, that is, high-frequency components.
As we described in section 3.2, real-world time series exhibit variability across different scenarios. 
Sometimes, areas like banking transactions, electricity consumption, and hospital foot traffic [7, 8, 9] require a focus on short-term variations. 
In our ablation study, Table 4(Right, "Non-FP"), and in Figure 5 of the uploaded PDF, we compare the outcomes of processing the entire frequency domain as a whole versus segmenting it into independent patches. 
The results, both in terms of predictive accuracy and the visualization of frequency information, clearly show the effectiveness of locality enhancement.

Reviewer comment (Writing):
In 'Abstract', there is a sentence saying "The subsequent observations extract the consistent representation within different channels by attention computation and summarize the relevant sub-frequencies to identify eventful frequency correlations for short-term variations.". What does 'Consistent representation within different channels' mean? I think that attention is capable of discovering relationships between different channels, not consistent representations within them.

Author response:
Question 2: 

In the multi-channel time series, some of the channels may exhibit consistent correlations within specific frequency bands(seen in Fig.4 in the new updated PDF). 
We would use channel-wise attention to learn and understand these correlations and key frequency components, which appear in both observation and forecasting.

Reviewer comment (Presentation):
In 'Introduction', there is a sentence saying "This approach relies on heuristic and empirical strategies, i.e., random or top-K frequency selection, often capturing spurious correlations for forecasting (seen in Figure 1(c)).". I think this is a too strong argument because Figure 1 doesn't directly provide evidence for spurious correlations.

Author response:
Question 3: 

Except for ignoring some high frequencies, in Fig.1, there are some spurious components, where many components (red bar) weigh significantly higher than the ground truth and input data.
A similar result can be found in Figure 4 of our uploaded PDF. 
We consider the representation performance of such random or Top-K selection methods (FedFormer [3], ETSformer [4], and JTFT [10]) more dependent on the selected offers to the model. 
This cannot treat the frequency components equally and often needs to be more balanced with some selection.

Reviewer comment (Theory):
"These independent attentions share model parameters across all sub-frequency learning, preventing winner-take-all of redundancy low-frequency components.": About this sentence, Can you explain further? By just sharing the same Transformer's parameters across all sub-frequency learning, why is the phenomenon prevented that low-frequency components dominate against high-frequency ones?

Author response:
Question 4:

Through frequency band independence, our model (i)independently normalize inputs across different frequency bands and (ii)learns features within each band by learning the consistency of different channels. 
This approach ensures that the model allocates equal attention to all frequency bands, rather than predominantly focusing on the low-frequency components.

Reviewer comment (Writing):
"To demonstrate the efficacy of the core design—channel-wise attention in FreCoformer, we visualized the heatmaps of the input and output DFT matrices of the Transformer encoder in FreCoformer in Figure 3(b)." in Section 4.2.2: As far as I know [1], the DFT matrix is like a weight matrix to transform time domain into frequency domain. If the input length $N$ is the same, the DFT matrix always has the same value. What does 'DFT matrices' in the above sentence mean?

Author response:
Question 5:

It is true that the left column is the original DFT transformation matrix.
But the right column shows the resulting DFT transformation matrics by Transformer, that is, the output of the Transformer.

Reviewer comment (Evaluation):
"This balance likely enables our method to efficiently extract pivotal frequency features across the entire frequency spectrum and various temporal variations, enhancing prediction outcomes." in Section 4.2.2: Why is balanced energy across entire frequencies helpful for your method? Can you give me more direct evidence for this sentence?

Author response:
Question 6: 

We show some experiments with/without frequency patching in Figure 5 of the uploaded PDF.
We empirically consider the patch, and independent learning does not draw any attention to any other frequency component. 
This prevents winner-take-all of redundancy low-frequency components.
The purpose of this paper is to provide such a strategy and to begin the investigation into its properties of frequency enhancement, knowing that it will take several papers to properly investigate this line of work.

Reviewer comment (Novelty):
Why the “divide and conquer” framework is named “divide and conquer”? Does it have any correlation with the “divide and conquer” algorithm? Isn’t it just a two-layer linear operation? How similar it is to the linear aggregation operation as proposed in [1]?

Author response:
Thank you again for the valuable comments. 
In this paper, we would like to provide a "divided-and-conquer" modeling strategy for time series forecasting studies. 
It consists of the FreCoformer and the design of a linear module.
Real-world time series exhibit variability across different scenarios.
Therefore, using two modules, our "divide and conquer" framework is designed to capture long-term and short-term variations respectively. 
This strategy is the spirit of the proposed framework.
That is, we argue a successful forecasting approach might be designed on top of scenarios-specialized modules to capture eventful characteristics, rather than a unified framework.
Please refer to the Reviewer-CiKd's (the first reviewer) reply for more experimental evidence.

The paper [1] is architecturally somehow related to our design.
However, all the operations are deployed on the time domain. 
This difference is the main focus of this paper, that is, 
models operated on the time domain extract the characteristics of a combination of various frequency components. 
Time series forecasting would know what frequencies are present in our signal and in what proportions.
Frequency analyses can highlight eventful characteristics, such as key neural waveforms or electrical activities in paper [1].

Reviewer comment (Theory):
Since the frequency-domain signal essentially is a one-to-one projection of the time-domain signal, how much does the proposed network differ from PatchTST (theoretically)? Specifically, do the two architectures share the same, or a subset of solution space? The performance gain over PatchTST seems very marginal in Table 3.

Author response:
As mentioned below, although DFT projects an observation of the time domain to the frequency domain, a Fourier transform of a signal tells you the frequency composition.
PatchTST [2] uses patching on the time domain and aims to learn time-time correlations for forecasting. However, it results in information loss of high frequency, as shown in Figure 1 of the main body.

On the other hand, our method focuses on capturing the important frequency feature, by learning features within each band independently. Through this, FreCoformer can capture more apparent middle-to-high-frequency features related to short-term variations in time series data. 
Our forecasting results outperform PatchTST, particularly on datasets with a rich spectrum of frequency features, such as ETTh1. The improvement in our predictive performance is significant.