Paper: Self-Supervised Contrastive Learning for Long-term Forecasting (ICLR-2024)

Reviewer comment (Theory):
For circumstance of nearly stationary input sequence, the autocorrelation will produce stationary result and cause the collapse of Constrastive Learning framework.

Author response:
We thank reviewer hiP7 for the insightful comment regarding the potential challenges posed by nearly stationary input sequences in our contrastive learning framework.
We understand your concern that "the nearly stationary sequences" might lead to stationary autocorrelation results, which could, in turn, affect the contrastive learning mechanism.

However, even in the nearly stationary sequences, the autocorrelation does not produce stationary results since stationary results in autocorrelation are theoretically possible only in the case of infinitely long time series.
In practice, we deal with finite samples from time series. In such cases, autocorrelation converges to zero, decreasing from 1 to 0 as the lag $\tau$ increases even in "the nearly stationary time series". In other words, autocorrelation exhibits a non-stationary trend and shows a decreasing pattern. This is because the covariance with lagged series $Cov(S_{t+\tau}, S_t) $ decreases as $\tau$ increases when numerically calculating the autocorrelation.
For example, as shown in M-Figure 5, the autocorrelation results of four datasets converge regardless of their stationarity. 

Additionally, the Weather time series, which is considered the most stationary among the tested time series according to the Augmented Dick-Fuller (ADF) test [[1]](https://www.nber.org/papers/t0130) as shown in statistics [[2]](https://proceedings.neurips.cc/paper_files/paper/2022/hash/4054556fcaa934b0bf76da52cf4f92cb-Abstract-Conference.html), also converges to 0 in a similar manner.
Naturally, such stationarity implies small (or absence of) long-term variation.
Therefore, even if our method is applied, as described in Dataset Analysis of Section 4.1, performance improvement could be marginal (3% reduced MSE).
However, we believe there is an unequivocal difference between gaining only a slight performance improvement and "collapse of contrastive learning".
(If reviewer hiP7 was trying to convey a different meaning than "collapse", we would be happy to further discuss this matter.)

Reviewer comment (Theory):
While the authors are aware of the linear assumption of autocorrelation and suggest the potential use of different techniques for high-order one, this suggestions might be inapplicable for the lengthy global input sequences with densely use frequency. Any approaches investigating these non-linear correlation can consume much more resources compare to the current autocorrelation.

Author response:
As reviewer hiP7 mentioned, thoroughly calculating high-order correlation across a very long time series can be computationally challenging.
What we have claimed and demonstrated in this work, however, is the necessity of leveraging global autocorrelation for long-term predictions.
These two are independent topics, and we find it difficult to understand why the former is pointed out as a "weakness" of the latter.
Even if one decides to combine the two somehow in the future (*e.g.* extend our work to incorporate high-order global correlation), there are many possibilities to explore. For example, one could try to reduce the computational complexity by leveraging domain knowledge of the specific dataset or the task (*e.g.* ignore the correlation between certain variables or certain time durations).
Alternatively, one could propose to capture the global high-order correlation in a completely different manner than how we capture the global autocorrelation.
Therefore, we find it unconvincing that the reviewer would use an independent problem as a weakness for the current submission.

Dear Reviewer hiP7,

Thank you for taking the time to review our responses.  We are pleased to hear that you are satisfied with our responses to the first two concerns.

As for the third concern, we appreciate your insights into the potential limitations of utilizing non-linear and high-order correlations for long-term predictions. We acknowledge the distinction between using non-linear correlations and using linear correlation, especially concerning computational costs. We will carefully consider this aspect in future work to broaden the scope of applications.

Best regards,

Reviewer comment (Writing):
Some grammar and typos (e.g. TiemsNet - page 2)

Author response:
Thank the reviewer for pointing out the typos. We have corrected them in the revised manuscript.

Reviewer comment (Novelty):
Capturing long-term variation via the contrastive loss part is separated from the forecasting mechanism. Good finding but have no further design for architectures based on this finding. The forecasting module is a simple dual-head forecaster and already achieves good performance.

Author response:
In response to the reviewer's concern that "The forecasting module is a simple dual-head forecaster and already achieves good performance", we have already demonstrated through an ablation study that the performance decreases when AutoCon is not used as shown in M-Table 3 and (b) of M-Figure 6. 
Therefore, without our contribution, the AutoCon loss, the dual-head forecaster fails to achieve good performance in long-term forecasting. 

And the reviewer claimed, "Capturing long-term variation via the contrastive loss part is separated from the forecasting mechanism", but our long-term encoder is jointly optimized by both forecasting and the AutoCon loss.
This learning approach aligns with our initial proposal of the two-stream architecture, where the linear structure empowers short-term (motivated by recent linear models [[1]](https://ojs.aaai.org/index.php/AAAI/article/view/26317)), and the contrastive loss to capture the long-term variations.
Empirically, considering the performance decline observed in the ablation studies when excluding the AutoCon loss, we find it unconvincing that reviewer MDze claims the contrastive loss operates independently from the forecasting mechanism.

Finally, our deliberate model design has indeed demonstrated superior long-term prediction performance compared to baseline models.
Therefore, as desired by the reviewer, we can incorporate even more complex architectures based on these findings in future work.

Reviewer comment (Theory):
Most of the ambiguity comes from the global information in the proposed representation. The authors should elaborate on the details of the encoder to highlight how the similarity of representations $v_i, v_j$ differ from the linear autocorrelation $r(i, j)$ in order to justify the contrastive objective (3). See question (1).

Author response:
We sincerely apologize for any ambiguity caused by an insufficient explanation. We mainly focused on experiments and visualization results to enhance understanding, which unfortunately led us to neglect the encoder details that should have been included for clarity. Following the reviewer eBaK's helpful guidance, we first clarify how the timestamp-based features are derived and incorporated into the encoder (Question 1).

* Q1: How are the timestamp-based features derived and incorporated into the encoder?  
The timestamp signifies the moment at which data is observed, and this moment is expressed in units of 'year,' 'month,' 'day,' and 'hour.'
The timestamp provides a forecasting model with periodicity on a yearly, monthly, and daily basis as input, aiding the model in learning various business cycles.
Most models use timestamps for long-term forecasting, commonly adding numerical values of periodic functions aligned with each unit's cycle from the timestamp as features to the input sequence.
Specifically, if we use four units (*i.e.,* month, day, weekday, and hour), we get a timestamp-based feature $f_{time} \in \mathbb{R}^{I \times 4} $, which is indexed by $\mathcal{T}$ and is aligned with input sequence $\mathcal{X} \in \mathbb{R}^{I \times c}$ where $I$ denote the input sequence length and $c$ is the number of variables.
Then, with a projection layer $W_{time}$ and $b_{time}$, the time embedding $e_{time} = W_{time} \cdot f_{time} + b_{time} \in \mathbb{R}^{I \times d}$ is added to value embedding $e_{value} = W_{value}\cdot \mathcal{X}+b_{value}$.
Finally, the temporal encoder (*e.g.,* RNN, TCN, and Attention)  takes the embedding sequence $e = e_{value} + e_{time} \in \mathbb{R}^{I \times d}$ to learn temporal dependency.
This approach of utilizing the timestamp information was initially introduced in Informer [[1]](https://ojs.aaai.org/index.php/AAAI/article/view/17325), and our model, along with recent models [[2](https://proceedings.neurips.cc/paper/2021/hash/bcc0d400288793e8bdcd7c19a8ac0c2b-Abstract.html), [3](https://proceedings.mlr.press/v162/zhou22g.html), [4](https://arxiv.org/abs/2210.02186)], adopts the same approach.

Now, we clarify the rationale for our AutoCon objective in relation to this timestamp information. In the baselines used in M-Figure 2, TimesNet, FEDformer, and our model all obtained representations using these timestamps, incorporating them into the input sequences, while PatchTST does not utilize the timestamps.
An important observation is that both TimesNet and FEDformer do not effectively capture the annual cyclic patterns, despite utilizing timestamps as the same as our model.
These failures are noteworthy, particularly considering the Electricity time series, which displays a yearly-long periodicity.
These results show that it is challenging for the model to learn yearly patterns even when given input sequences and timestamps, solely relying on the existing forecasting loss.
Therefore, this result demonstrates the necessity of our AutoCon loss.
Furthermore, to better demonstrate the importance of our loss in capturing long-term periodicities, we conducted an additional experiment where we compared two models: Ours with and without AutoCon loss.
As shown in [R-Figure1](https://tinyurl.com/wo-autocon-repr-sim), compared to the one with AutoCon loss, the one without AutoCon loss demonstrates only a hint of periodicity (not too different from other baselines).

Reviewer comment (Reproducibility):
How is the timestamp-based features derived and incorporated into the encoder?

Author response:
We sincerely apologize for any ambiguity caused by an insufficient explanation. We mainly focused on experiments and visualization results to enhance understanding, which unfortunately led us to neglect the encoder details that should have been included for clarity. Following the reviewer eBaK's helpful guidance, we first clarify how the timestamp-based features are derived and incorporated into the encoder (Question 1).

* Q1: How are the timestamp-based features derived and incorporated into the encoder?  
The timestamp signifies the moment at which data is observed, and this moment is expressed in units of 'year,' 'month,' 'day,' and 'hour.'
The timestamp provides a forecasting model with periodicity on a yearly, monthly, and daily basis as input, aiding the model in learning various business cycles.
Most models use timestamps for long-term forecasting, commonly adding numerical values of periodic functions aligned with each unit's cycle from the timestamp as features to the input sequence.
Specifically, if we use four units (*i.e.,* month, day, weekday, and hour), we get a timestamp-based feature $f_{time} \in \mathbb{R}^{I \times 4} $, which is indexed by $\mathcal{T}$ and is aligned with input sequence $\mathcal{X} \in \mathbb{R}^{I \times c}$ where $I$ denote the input sequence length and $c$ is the number of variables.
Then, with a projection layer $W_{time}$ and $b_{time}$, the time embedding $e_{time} = W_{time} \cdot f_{time} + b_{time} \in \mathbb{R}^{I \times d}$ is added to value embedding $e_{value} = W_{value}\cdot \mathcal{X}+b_{value}$.
Finally, the temporal encoder (*e.g.,* RNN, TCN, and Attention)  takes the embedding sequence $e = e_{value} + e_{time} \in \mathbb{R}^{I \times d}$ to learn temporal dependency.
This approach of utilizing the timestamp information was initially introduced in Informer [[1]](https://ojs.aaai.org/index.php/AAAI/article/view/17325), and our model, along with recent models [[2](https://proceedings.neurips.cc/paper/2021/hash/bcc0d400288793e8bdcd7c19a8ac0c2b-Abstract.html), [3](https://proceedings.mlr.press/v162/zhou22g.html), [4](https://arxiv.org/abs/2210.02186)], adopts the same approach.

Reviewer comment (Experiments):
Which dataset is used in Figure 6? Do you observe the same result on other datasets?

Author response:
We thank reviewer eBaK for pointing out and addressing our oversight. M-Figure 6 presents the experimental results for the ETTh2 dataset. We explicitly identified it in the description of M-Figure 6 of our revised paper. Additionally, we provide results for ETTh1 ([R-Figure3](https://tinyurl.com/figure6-ETTh1)) and Electricity ([R-Figure4](https://tinyurl.com/figure6-ecl)), which show the long-term variations. Although there are some differences in magnitude, the overall trends are similar across the three datasets. We have also added these additional results to the Appendix B.2. We sincerely appreciate the reviewer`s constructive and helpful comments once again.

Reviewer comment (Novelty):
The novelty of the work is incemental. While the authors focus on an interesting problem, the proposed methodology is combination of existing components.

Author response:
We appreciate reviewer pLfj's thoughtful feedback regarding the novelty. We would like to respectfully clarify our perspective on this matter.

Our primary novelty lies in the introduction of global autocorrelation as a foundational element for long-term prediction. Using global autocorrelation in long-term forecasting, to the best of our knowledge, has not been previously explored. While we acknowledge the potential for future refinement and the introduction of more intricate mathematical models, we believe the core novelty of our work is in emphasizing the significance of autocorrelation-based contrastive learning, AutoCon, within the sliding window-based long-term forecasting. 

To incorporate AutoCon, we have renovated a decomposition architecture that is suited for representation learning. While our model can be viewed as a combination of existing components, its architecture is specifically designed to emphasize the significance of the long-term branch. As we mentioned in Section 3.2, this aspect has often been overlooked by even complex existing models, yet our approach maintains simplicity while being effectively structured for long-term representation.

We understand that perspectives on novelty may vary, and we respect alternative opinions. However, it is our strong belief that the incorporation of autocorrelation-based contrastive learning into the long-term forecasting framework, as demonstrated in our work, constitutes a meaningful advancement in the field.

Reviewer comment (Experiments):
It would be useful to add a table with the dataset statistics summarized (in Appendix if it does not fit in the main paper). How does the proposed methodology performs in imbalanced data?

Author response:
* 1: Dataset Statistics  
For reviewer pLfj's valuable suggestions, we added M-Table 5 summarizing dataset statistics in Appendix A.3. The dataset statistics will provide readers with a quick reference to understand the diverse nature of the datasets used in our experiments.

* 2: Extension to imbalanced data  
Regarding reviewer pLfj's query on the performance of our methodology in imbalanced data scenarios, we appreciate the opportunity to clarify this aspect. As our focus is on time series forecasting, which is typically treated as a regression task, the concept of data imbalance manifests differently compared to classification tasks. In time series forecasting, data imbalance might refer to scenarios where certain patterns or trends are underrepresented. While our current study does not explicitly address this form of imbalance, our methodology's ability to learn long-term dependencies can be indirectly beneficial in scenarios where certain long-term trends and patterns are less frequent compared to repeated short-term fluctuations within the window. Specifically, AutoCon captures the overall tendency of the time series data across its entire length using the global autocorrelation.
By considering the global structure of the data, it helps the model to identify and learn from long-term patterns, including those that may not occur frequently.
Thus, we believe that in imbalanced scenarios, the model can recognize and adapt to these infrequent but potentially significant patterns, leading to more accurate forecasting. We acknowledge the importance of this aspect and will consider investigating this in future work to further enhance the robustness of our forecasting approach. Once again, we thank the reviewer for the constructive feedback.

Reviewer comment (Evaluation):
It would be interesting and more convincing on the performance if the authors would add results on more evaluation metrics, e.g., Accuracy, AUROC, AUPRC, etc.

Author response:
We concur with reviewer pLfj's recommendation to incorporate additional evaluation metrics, as this will further substantiate the superior performance of our method in comparison to the baselines.
While existing metrics (*i.e.,* MSE and MAE) are standard metrics for long-term forecasting evaluation, they have limitations. Specifically, they may not adequately capture aspects such as the shape and temporal alignment of the time series, which are crucial for a comprehensive evaluation of a forecasting model's performance.

To address these limitations, we introduced two additional metrics based on Dynamic Time Warping (DTW) [[1]](https://ieeexplore.ieee.org/abstract/document/1163055?casa_token=2XD8Dcxt2fIAAAAA:qR84AD50aWQv7paoubI5peCAiDRousaLO8n6egWNLkc66OF6IbbH9rx1cpvxs5HtveC3ZaB9NQ): Shape DTW and Temporal DTW [[2]](https://arxiv.org/abs/1909.09020). Shape DTW focuses on the similarity of the pattern or shape of the predicted sequence to the actual sequence, providing insight into the model's ability to capture the underlying pattern of the time series. Temporal DTW evaluates the alignment of the predicted sequence with the actual sequence, highlighting the model's accuracy in forecasting the timing of events.

These additional metrics offer a more nuanced assessment of our model's performance, particularly in areas where MSE and MAE may fall short. Lower values in both Shape DTW and Temporal DTW indicate better performance, signifying lesser distortion between the predicted and actual sequences. As shown in R-Table 1, our method demonstrates superior performance for three datasets, which show obvious long-term variations, not only in MSE and MAE but also in these shape and temporal alignment-focused metrics. 
Although it is difficult to measure Accuracy or AUROC due to the absence of labels in the forecasting task, our comprehensive evaluation of the newly used metrics underscores the superiority of our method. Also, these experiments were added to Appendix B.6  in the revised version of the paper.

**R-Table 1: Univariate forecasting results, evaluated using Shape and Temporal DTW in a 720-output setting across three datasets**     

| Dataset            | ETTh1            |                 | ETTh2        |                 | Electricity |  |
|--------------------|------------------|-----------------|--------------|-----------------|----------------------|--------------------------|
| **Model \ Metric** | Shape DTW        | Temporal DTW    | Shape DTW    | Temporal DTW    | Shape DTW | Temporal DTW |
| **Ours**           | **17.14±1.653** | **59.66±1.739** | **42.38±0.630** | **13.47±1.793** | **80.73±7.7982**    | **0.09±0.014**           |
| TimesNet           | 25.80±4.349      | 86.86±15.509    | 62.58±5.390  | 51.19±21.834    | 139.83±16.516      | 0.49±0.826               |
| PatchTST           | 22.21±1.226      | 72.23±4.411     | 65.45±2.976  | 15.23±0.882     | 116.50±29.878      | 0.75±1.674               |
| MICN               | 37.08±12.393     | 65.70±3.588     | 67.69±8.796  | 22.67±3.168     | 123.93±17.543      | 1.90±1.176               |
| DLinear            | 58.32±1.955      | 155.21±8.587    | 82.53±3.627  | 24.88±2.403     | 88.70±3.550        | 0.18±0.145

Reviewer comment (Evaluation):
It would be interesting to show what is the runtime for each method/dataset and time complexities.

Author response:
In Section 4.4, we have briefly provided an analysis of the computational costs for each model. However, due to space limits, detailed information can be found in Appendix B.4. 
We measured the computational cost of various baselines for four different output lengths, ranging from 96 to 2160. In summary, our model recorded significantly lower inference times compared to the baselines except for linear models in a single inference process. Additionally, it demonstrated much faster inference speed compared to CNN-based, TimesNet, and Transformer-based models even as the prediction length increased.
We would sincerely appreciate it if the reviewer could refer to it once more.

Reviewer comment (Reproducibility):
There is no discussion about making the code available publicly.

Author response:
We have already attached our code as supplementary material in the open review system. If this is not made public after the review process, we are also willing to publicly open our code on Github and include the GitHub repository URL in the main paper as either an abstract or a footnote.