Paper: Large Pre-trained time series models for cross-domain Time series analysis tasks (ICLR-2024)

Reviewer comment (Novelty):
The overall technical novelty is limited. The primary contribution of this work lies in the concept of patch scoring, as presented in Eq. 2, and the subsequent two paragraphs. The overarching design can be seen as an extension of PatchTST. In the realm of time series pre-training, several recent studies, such as SimMTM, have delved into the concept of masked patch reconstruction.

Author response:
**Regarding technical novelty**

The segmentation module is indeed the main novelty of the paper. However, we would like to emphasize that learning domain-specific adaptive segmentation strategies is vital for multi-domain pre-training. Therefore, we cannot simply use PatchTST or other variants of time-series input for all domains. Indeed we have compared LPTM with PatchTST in the new experiments (here)[https://openreview.net/forum?id=KJ1w6MzVZw&noteId=LqihMFfTsp] where LPTM clearly outperforms PatchTST is all benchmarks. While we used simple pre-training methods that have been previously studied, more sophisticated ones can indeed be designed on top of LPTM to improve its performance further.

Reviewer comment (Experiments):
The experimental settings are ill-defined. While this work emphasizes cross-domain adaptation, the evaluation datasets (& domains) substantially overlap with the source datasets (& domains) used in pre-training. I do not think this is a valid evaluation protocol for cross-domain adaptation.

Author response:
**the evaluation datasets (& domains) substantially overlap with the source datasets (& domains) used in pre-training. I do not think this is a valid evaluation protocol for cross-domain adaptation.**

LPTM segmentation module learns segmentation strategies for each domain and can enable evaluation on unseen datasets and tasks on unseen datasets but on domains observed during pre-traning. Therefore, LPTM doesn;t support adaptation to novel unseen domains. However, it is the first of its kind to learn to adapt to unseen datasets across all the pre-train domains leads to the development of a general pre-trained model across multiple domains.

We also provided additional clarification on Fig 1 in previous response.

Reviewer comment (Experiments):
Regarding the construction of the pre-training set, how is the optimal combination of data samples from different domains determined? I found no discussion on this in the experiment section.

Author response:
** how is the optimal combination of data samples from different domains determined?**

Our main aim is to show that LPTM can adapt to pre-train and learn from wide range of domains. Therefore, we do not optimize on the right combination of datasets to optimize downstream performance. We chose widely used datasets from time-series literature that covers diverse range of applications.

However, finding or optimizing for the right combination of datasets is an interesting research direction which actively explored even in LLM pre-training and can be explored for time-series as well using LPTMs.

Reviewer comment (Theory):
How can Eq.2 effectively handle "out-of-distribution" samples from domains that were not encountered during pre-training?

Author response:
**How can Eq.2 effectively handle "out-of-distribution" samples from domains that were not encountered during pre-training?**

The segmentation module identifies good segmentation strategies for each of the domains encountered in the pre-training. Therefore, they cannot be directly used on downstream tasks of new domains. However, they can handle unseen datasets and tasks of domains trained pre-training.

Reviewer comment (Presentation):
Fig1 is confusing. After reviewing Fig1 and its caption, I have several related inquiries: Q1: Why choose GRU over a linear projection as the patch embedder? Q2: How do patch scoring and pruning operate? Q3: What do you intend by h(1) to h(R)? Q4: Where is the self-supervised optimization highlighted?

Author response:
**Regarding Fig 1**

To model the score of segments we need to capture the temporal dynamics of the segment which is done by GRU. This is then also used to embed the segments for the same reason. Moreover, a linear projection cannot captured segments of varying length and their dynamics. The patch scoring is done using the scoring function (Eq 2) and the high scoring subset of segments is chosen and pruned by heuristic mentioned in Sec 3.2 after Eq. 2. We also define h(i) in the paragraph as the the maximum scoring segment that start from time-step i. The full model in Figure 1 is pre-trained on the SSL tasks (Sec 3.3) to learn the parameters.

Reviewer comment (Experiments):
Several critical baselines, such as PatchTST, SimMTM, and TimesNet, are absent. Additionally, it would be advantageous to evaluate using datasets from domains not encountered during pre-training.

Author response:
**Question 7: Missing baselines**

We have added additional baselines of more state-of-art forecasting time-series models including TimesNet and PatchTST at (here)[https://openreview.net/forum?id=KJ1w6MzVZw&noteId=LqihMFfTsp] and show that the performance of LPTM is superior in electricity, traffic, and M3 benchmarks and similar to SOTA in other benchmarks.

Reviewer comment (Writing):
Writing can be greatly improved. Abstract should be 1 paragraph. Mathematical notation is not clear -- many variables are not defined.

Author response:
**Regarding Writing**

We thank the reviewer for their comments. We would be grateful if the reviewer can point to specific instances where the writing is unclear.

Out abstract has multiple paragraphs to first introduce the novel challenges of pre-trained time-series models that are not yet well explored and then introduce the technical novelties and advantages of LPTM in subsequent paragraphs. The abstract is less than 200 words and clearly introduces the scope, importance and novelty of our work.

To the best of our knowledge we have clearly defined all the mathematical notation in the paper. The only source of notational confusion, as other reviewers pointed out is that we have one type in Eq 4 where $g(i,j)$ should be $s(i,j)$.

Reviewer comment (Experiments):
The usefulness of the model is diminished with the dataset specific segmentation module. The model is unable to perform zero-shot forecasting or prediction tasks.

Author response:
**Regarding zero-shot forecasting**

Pre-trained LPTM models are observed to perform similarly or better than SOTA baselines using significantly smaller training data and training time. However, we need some amount of fine-tuning to specific task since each task is different to pre-training SSL tasks and require the model to adapt to it.

However, designing zero-shot learning on LPTM pre-trained models is indeed a very interesting future research problem with non-trivial challenges we do not tackle in our current work.

Reviewer comment (Reproducibility):
Codebase in given link is incomplete. No script for training / predictions. README is empty, without instructions.

Author response:
**Codebase**

We thank the reviewer for the comments. We will improve the documentation of the code when we release it publicly.

Reviewer comment (Theory):
I do not understand the rationale behind Eq (4).

Author response:
Empirically we found that adding the log to SSL losses provided for much stabler training, without which pre-training was not possible. While ideally, the SSL losses converge towards zero, in practice we found that they were generally much larger throughout the pre-training process. Using adaptive loss functions across different ranges of SSL loss values could further lead to improved stability.

Yes, Equation 4 does indeed have a type where $g(i,j)$ should be replaced by $s(i,j)$. We thank the reviewer for pointing this out and hope this should clarify the confusion.

Reviewer comment (Theory):
What is \(g(i, j)\) in Eq. 4? Do you mean \(s(i, j)\)? If so, why summing \(\log(\mathcal{L}_{SSL})\) with the sum of scores?

Author response:
Yes, Equation 4 does indeed have a type where $g(i,j)$ should be replaced by $s(i,j)$. We thank the reviewer for pointing this out and hope this should clarify the confusion.

Reviewer comment (Theory):
Given that the loss that is optimized operates on the aggregation of all scores, it is not clear how it could enforce large scores for selected segments.

Author response:
The score function $s(i,j)$ is trained to predict the logarithm of SSL losses (Eq 4) which enables the model to in turn use it to find better segmentation strategies. This is because a higher $s(i,j)$ using the trained segmentation module implies the segment is useful for the model to undestand important patterns and properties of the time-series (via the SSL tasks).

Reviewer comment (Reproducibility):
The text refers to "aggregation" but not much is said (in Sec 3.2 at least) on which aggregation function is used, why?

Author response:
The aggregation of the time-step representations $z_i, \dots, z_j$ refers to simple summation after passing through a self-attention layer. We will clarify that. We used summation since it is a standard aggregation function used to aggregate along the temporal dimension to represent a time-series, or a segment (in our case) using a fixed length vector embedding.

Reviewer comment (Theory):
While retrieving the optimal \(S(y(1...t))\) is an interesting combinatorial optimization problem, could you elaborate a bit more on this problem? Does it have known solutions? Do you have a way to assess if your approximation is a reasonable one or not?

Author response:
The combinatorial problem referred to is to find the optimal set of segments that cover the entire time-series that have the maximum sum of scores. The set of satifiable soultions is exponential in the number of time-steps. We used a simpler heuristic that guarantees the coverage of the segments but doesn't guarantee optimality. A more thorough theoretical analysis of the problem would be an interesting problem to improve the performance of LPTM.

Reviewer comment (Reproducibility):
**Reproducibility:** While the code is available, many hyper-parameters important to reproduce the results are not mentioned in the manuscript. (1) As an example, the authors do not mention the proportion of time segments that they mask during self-supervised learning ($\gamma$). (2) The number, length, ranges, number of channels etc. of time-series used for pre-training and evaluation are not mentioned either. (3) Furthermore, the authors compare "training time (minutes) till convergence" but fail to mention the compute infrastructure, what kind of time (wall clock?) are they measuring.

Author response:
We provide additional details on important hyperparameters in Appendix Section B.

>  For RANDMASK, we found the optimal gamma = 0.4, and for LASTMASK
gamma = 0.2 was optimal

For Flu-US we pre-trained and trained on the dataset from 2004-2012 and evaluated for the 2013-2020 seasons. We used the standard recommended train-val-test split for ETT2, PEM-Bays, NY-Bike, Nasdaq, and M3 as done in other works. The ratio is usually around 70-20-10. The test and training data splits are also provided for classification datasets from the source UCI repository.

We also mention the computing environment in Appendix Section B as using Nvidia Tesla v100 GPU with 32 GB VRAM. The CPU used is an Intel Xeon with 64 cores and 128 GB of memory. The code running time is benchmarked using wall time or execution of the entire pipeline of training.

Reviewer comment (Writing):
**Clarity:** The paper is unclear and many statements are not rigorously or scientifically defined. For example, (1) the authors claim in Section 3.2 that their segment score function measures "how good the given segment is for the dataset", but do not clarify what the notion of goodness is. The notion of goodness is also not immediately clear as the authors use a hyperbolic Tangent function as the scoring function. (2) The authors invoke $g(i, j)$ for the first time in Equation 4. Consequently, it appears that the paper was put together in a hurry, without careful proofreading.

Author response:
The score function $s(i,j)$ as defined in Equation 2 provides a score for each of the time-series segments that is used to pick the appropriate segments of the model. This is optimized in Equation 4 by making the score function reflect the negative of SSL training loss. Therefore, set of segments with higher scores will provide lower SSL loss.

Equation 4 does indeed have a type where $g(i,j)$ should be replaced by $s(i,j)$. We thank the reviewer for pointing this out and hope this should clarify the confusion.

Reviewer comment (Experiments):
**Experimentation:** (1) A pre-trained model should be able to solve tasks without any fine-tuning, especially since all the training parts of the datasets are observed during pre-training. (2) For smaller datasets, a large model trained from scratch is destined to underfit. Since the authors have not mentioned the size of the model, beating LPTM trained from scratch on a small dataset can be attributed to the model being too big for a small dataset. A smaller model might very well learn from scratch.

Author response:
**On requirement of fine-tuning**

Even pre-trained language models or vision models required some fine-tuning or prompting to adapt to a downstream task. Therefore, it is not clear why a pre-trained model does not need fine-tuning. Indeed, the SSL tasks differ significantly from the downstream tasks such as classification and forecasting. While LASTMASK has similarities with forecasting, the horizon length required for forecasting differs for each downstream task the the last segment length of the segmentation module.

**Smaller models may outperform LPTM**

We do not believe that smaller models trained from scratch can outperform LPTM. Indeed some of the baselines like EpiFNP, MICN, and ColaGNN which are not transformer-based have a smaller number of parameters but in most cases underperform LPTM significantly and only slightly outperform LPTM is specific benchmarks in few instances.

Reviewer comment (Writing):
**Question:** What is $g(i, j)$ in Equation 4?

Author response:
The score function $s(i,j)$ as defined in Equation 2 provides a score for each of the time-series segments that is used to pick the appropriate segments of the model. This is optimized in Equation 4 by making the score function reflect the negative of SSL training loss. Therefore, set of segments with higher scores will provide lower SSL loss.

Equation 4 does indeed have a type where $g(i,j)$ should be replaced by $s(i,j)$. We thank the reviewer for pointing this out and hope this should clarify the confusion.

Reviewer comment (Reproducibility):
**Question:** What is $\gamma$? How are the segments sampled?

Author response:
We provide additional details on important hyperparameters in Appendix Section B.

>  For RANDMASK, we found the optimal gamma = 0.4, and for LASTMASK
gamma = 0.2 was optimal

For Flu-US we pre-trained and trained on the dataset from 2004-2012 and evaluated for the 2013-2020 seasons. We used the standard recommended train-val-test split for ETT2, PEM-Bays, NY-Bike, Nasdaq, and M3 as done in other works. The ratio is usually around 70-20-10. The test and training data splits are also provided for classification datasets from the source UCI repository.

We also mention the computing environment in Appendix Section B as using Nvidia Tesla v100 GPU with 32 GB VRAM. The CPU used is an Intel Xeon with 64 cores and 128 GB of memory. The code running time is benchmarked using wall time or execution of the entire pipeline of training.

Reviewer comment (Novelty):
The model architecture used is quite straightforward - just a transformer encoder. Exploring more sophisticated temporal modeling architectures could be beneficial.

Author response:
Our important contribution is the importance of learning good segmentation strategies for enabling effective representation for multi-domain time-series pre-training.
The main challenge for performing pre-training on heterogeneous datasets
is not the model architecture but the segmentation module.
We show that by using the vanilla transformer architecture we generally outperform state-of-art models designed for specific downstream tasks.
Our segmentation module can be applied to any transformer-based architecture to improve and enable multi-domain pre-training

Reviewer comment (Experiments):
Ablations only evaluate the removal of components, could also analyze additions like other SSL tasks.

Author response:
Many SSL tasks are usually designed for specific applications or for pre-training on a single data source. Moreover, they may not automatically
be retrofitted with our framework of using segments of time series.
Moreover, we showed significant performance improvement using simple SSL tasks of random masking and last token prediction.
Identifying better SSL tasks that can be performed on segment tokens is an interesting future research direction.

Reviewer comment (Experiments):
Though diverse, the pre-training datasets are still limited to a few domains. Expanding the data diversity could help.

Author response:
We leverage datasets from seven very diverse domains ranging from traffic and finance to epidemiology and behavior tracking. Moreover, these datasets are popular benchmarks widely used in literature.
Therefore, we showed the efficacy of LPTM in pre-training on diverse datasets. If the reviewer can suggest any other specific domains we would gladly consider adding them as well.

Reviewer comment (Theory):
Theoretical analysis on how pre-training and adaptive segmentation provide benefits is lacking.

Author response:
While we empirically show the efficacy of the segmentation module in handling diverse datasets, generating a convincing theoretical justification is non-trivial.
This is due to the complexity of the model and dataset distributions from different domains. Our main motivation is based on various patterns of smoothness and local dynamics of the time series that cannot be captured by tokenizing single time stamps or uniform length segments.
This is also analogous to various tokenization strategies used in large language models where the strategies are not theoretically justified but are motivated by empirical and linguistic arguments.

Reviewer comment (Experiments):
Comparisons to more sophisticated domain-specific models like those using additional covariates would be informative.

Author response:
This is an interesting question. LPTM architecture is designed to be a general time-series forecasting model without additional sources of features similar to other top-performing state-of-art baselines. Adapting our methods to multiple covariates is an important challenge for future work.