Paper: Pushing the Limits of Pre-training for Time Series Forecasting in the CloudOps Domain (ICLR-2024)

Reviewer comment (Experiments):
It is unclear whether the datasets have missing values.

Author response:
__Q: What is the main challenge that these datasets pose?__

A: The main challenge of these datasets is the size. Due to the large number of time series in these dataset, we cannot make sweeping claims about the characteristics of time series in each dataset. One example is that these time series contain seasonality, as observed in Figures 10, 11, and 12 in Appendix F. However, the time series in one dataset can have differing seasonality periods, and some may not even have seasonality, or have cyclical patterns instead (repeating patterns but no fixed period). Unfortunately, these datasets are collected over a relatively short period of time (longest being 30 days), and for some datasets, there has been some obfuscation, whereby the actual date/time information is not given, and thus cannot be correlated back to real world events. These datasets indeed have missing values -- highlighted in Appendix A.2, we performed data cleaning, filtering from the raw data the time series with too many missing values.

Reviewer comment (Experiments):
It is unclear how strong the seasonality is in these datasets.

Author response:
__Q: What is the main challenge that these datasets pose?__

A: The main challenge of these datasets is the size. Due to the large number of time series in these dataset, we cannot make sweeping claims about the characteristics of time series in each dataset. One example is that these time series contain seasonality, as observed in Figures 10, 11, and 12 in Appendix F. However, the time series in one dataset can have differing seasonality periods, and some may not even have seasonality, or have cyclical patterns instead (repeating patterns but no fixed period). Unfortunately, these datasets are collected over a relatively short period of time (longest being 30 days), and for some datasets, there has been some obfuscation, whereby the actual date/time information is not given, and thus cannot be correlated back to real world events. These datasets indeed have missing values -- highlighted in Appendix A.2, we performed data cleaning, filtering from the raw data the time series with too many missing values.

Reviewer comment (Experiments):
Is there any distribution shift captured in these datasets?

Author response:
__Q: Is there a distribution shift captured in these datasets?__

A: Thank you for raising this valuable question. We have performed further analysis to measure distribution shift across (1) pre-train and train-test sets, and (2) across time. In our updated Appendix A.3.1, we observe in Figure 7 that there is little temporal distribution shift across the train to test regions. This is expected, since the collection period of this dataset is over the period of 1 month, where there will not be significant shift over time. However, we do observe that there is a significant distribution shift across different time series in the pre-train set to train-test set.

Reviewer comment (Evaluation):
How do these datasets look in the test windows used for evaluation? How do the forecasts per model look? For example, for ali2018 the forecasts may be similar as Naive performs very well, but what about azure 2017 where the proposed baselines perform exceptionally well?

Author response:
__Q: How do these datasets look in the test windows used for evaluation? And how do the forecasts per model look? I understand that for ali2018 the forecasts are potentially similar as Naive performs very good, but what about azure 2017 where the proposed baselines are performing exceptionally well?__

A: Please see Appendix F for forecast visualizations across all three datasets in the test region. We observe that ali2018 is more challenging dataset, since based on qualitative observations, it has more noise and has long range seasonality patterns, compared to azure2017 which have fairly simple seasonality patterns.

Reviewer comment (Experiments):
The pre-training and zero-shot testing appear to be conducted within the same dataset. If this is correct, it raises concerns about the true generality of the zero-shot performance, particularly regarding its effectiveness in diverse or new datasets and domains.

Author response:
__Q: The pre-training and zero-shot testing appear to be conducted within the same dataset. If this is correct, it raises concerns about the true generality of the zero-shot performance, particularly regarding its effectiveness in diverse or new datasets and domains.__

A: The pre-training and zero-shot testing follows the "in-collection pre-training" setting as explained in Section 1 and Figure 2. This means that the pre-training set and test set comes from the same __collection__ of time series, but consist of different, non-overlapping time series. Detailed in Appendix A.2, we ensure that the data split between the pre-training and train-test sets is performed based on a *top level attribute*, ensuring that there is no overlap between them. Thus, we expect the time series patterns/distribution across pre-train and train-test sets to be different, making the in-collection pre-training setting to be meaningful and challenging.

We verify this hypothesis in Appendix A.3 of our updated manuscript, where we perform further data analysis, qualitatively, and quantitatively, demonstrating the distribution shift between pre-train and train-test sets.

1. Qualitatively, we show in Figure 7, based on histogram plots comparing the (a) train region vs pre-train set, and (b) train region vs train-test region, that the gap in (a) is larger than the gap in (b).
2. Quantitatively, we obtain a measure of distribution shift between the pre-train set and train-test set. We do so by first encoding the time series into a feature vector, comprising of 7 representative features as suggested by [1] -- spectral entropy, strength of trend (hourly and daily), strength of seasonality (hourly and daily), first-order autocorrelation, and the optimal box-cox transformation parameter. Then, we use the Wasserstein distance to measure the distribution difference between pre-train and train-test distributions. Table 7 demonstrates, by comparing with a a baseline of the Wasserstein distance between two random subsets of the pre-train set, that there is indeed a difference in distribution between the pre-train and train-test sets.

|                         | azure2017     | borg2011      | ali2018       |
| ----------------------- | ------------- | ------------- | ------------- |
| Pre-train vs Train-test | 5.26          | 1.69          | 9.81          |
| Pre-train               | 0.160 (0.013) | 0.077 (0.024) | 0.361 (0.024) |
| Train-test              | 0.802 (0.27)  | 0.296 (0.098) | 0.545 (0.268) |

Reviewer comment (Novelty):
The paper mainly focuses on benchmarking existing pre-training model architectures without significant novel adaptations or designs tailored to the specific requirements of the cloud operation domain.

Author response:
__Q: The paper mainly focuses on benchmarking existing pre-training model architectures without significant novel adaptations or designs tailored to the specific requirements of the cloud operation domain.__

A: We address this in our response to all reviewers since this is a recurring issue amongst reviewers.

Reviewer comment (Novelty):
Lack of Novelty: The paper's approach is primarily centered on training a large transformer model on an extensive time series dataset, which may not offer a novel contribution to the field.

Author response:
__Q: Lack of novelty.__

A: We address this in our response to all reviewers since this is a recurring issue amongst reviewers.

Reviewer comment (Experiments):
Limited Analysis of Transfer Learning: The study raises concerns regarding the assessment of transfer learning, as the model is trained on one dataset and tested on a similar one without sufficient information about their differences. This makes it challenging to interpret the extent of transfer learning within an in-collection setting. Additionally, the diversity of time series within the CloudOps datasets remains unclear, which could lead to potential overfitting issues, particularly for large models.

Author response:
__Q: Limited analysis of transfer learning.__

A: Thank you for this valuable suggestion -- we have updated in our manuscript Appendix A.3, where we perform further data analysis to demonstrate the distribution shift between pre-train and train-test sets.

As outlined in Appendix A.2, we ensure that the data split is done based on a *top level attribute*, ensuring that there is no overlap between the pre-train and train-test sets. Thus, we expect the time series patterns/distribution across pre-train and train-test sets to be different, making the in-collection pre-training setting to be meaningful and challenging. We verify this hypothesis in the new Appendix A.3, where we perform a (1) qualitative analysis, and a (2) quantitative analysis. 

1. Qualitatively, we show in Figure 7, based on histogram plots comparing the (a) train region vs pre-train set, and (b) train region vs train-test region, that the gap in (a) is larger than the gap in (b).
2. Quantitatively, we obtain a measure of distribution shift between the pre-train set and train-test set. We do so by first encoding the time series into a feature vector, comprising of 7 representative features as suggested by [1] -- spectral entropy, strength of trend (hourly and daily), strength of seasonality (hourly and daily), first-order autocorrelation, and the optimal box-cox transformation parameter. Then, we use the Wasserstein distance to measure the distribution difference between pre-train and train-test distributions. Table 7 demonstrates, by comparing with a a baseline of the Wasserstein distance between two random subsets of the pre-train set, that there is indeed a difference in distribution between the pre-train and train-test sets.

|                         | azure2017     | borg2011      | ali2018       |
| ----------------------- | ------------- | ------------- | ------------- |
| Pre-train vs Train-test | 5.26          | 1.69          | 9.81          |
| Pre-train               | 0.160 (0.013) | 0.077 (0.024) | 0.361 (0.024) |
| Train-test              | 0.802 (0.27)  | 0.296 (0.098) | 0.545 (0.268) |

Regarding dataset diversity, there are several aspects to discuss. On the level of data attributes, as seen in Table 6 in Appendix A.1, azure2017 measures 1 target variable (CPU utilization) whereas borg2011 and ali2018 measures 2 target variables (CPU utilization and memory usage). Furthermore, each dataset measures unique auxiliary covariates, for example, azure2017 has unique static real features such as virtual core count, memory size, and deployment size. 

Next, we can do a qualitative comparison based on time series visualizations in Appendix F. azure2017 has high frequency cyclical patterns, whereas borg2011 has slightly lower frequency patterns. On the other hand, ali2018 has very low frequency seasonality, and contains significantly more trend information. 

Finally, similar to the analysis on distribution shift, we can perform a quantitative evaluation of the diversity of the datasets. Using the same approach as before, we measure the distance between the distribution of datasets using the Wasserstein distance (we have included this as Appendix A.3.2, including a scatter plot of the features in low-dimensional feature space generated by PCA). 

|               | azure2017    | borg2011      | ali2018       |
| ------------- | ------------ | ------------- | ------------- |
| azure2017 | 0.802 (0.27) | 42.86         | 20.85         |
| borg2011      | 42.86        | 0.296 (0.098) | 10.77         |
| ali2018       | 20.85        | 10.77         | 0.545 (0.268) |

[1] Godahewa, Rakshitha, et al. "Monash time series forecasting archive." arXiv preprint arXiv:2105.06643 (2021).

Reviewer comment (Evaluation):
Limited Improvements in Larger Models: The paper reveals that the gains achieved with "Large" and "xLarge" models are not significant when compared to the base model size. Particularly, the "xLarge" model, despite being over eight times larger than the base model, exhibits only marginal improvements. This suggests that the base model may already be overfitting the datasets.

Author response:
__Q: Limited improvements in larger models.__

A: The reviewer has indeed pointed out an accurate observation. Instead of a weakness, this is one of the findings presented in our empirical analysis of scaling up time series models. Motivated by these results, we performed a more in-depth analysis. In Section 4.4, Figure 5, we observe that as we increase training iterations/number of training samples, the azure2017 dataset reveals a clear trend which starts to plateau. However, there is no such relationship for the borg2011 and ali2018 dataset, which critically, are 0.5x and 0.1x the size of the azure2017 dataset. We indeed come to the same conclusion as the reviewer, highlighting this fact as a key finding in Section 4.4, and providing further evidence in Appendix G by plotting even more fine-grained plots of validation error across different training iterations.