Paper: TEST: Text Prototype Aligned Embedding to Activate LLM's Ability for Time Series (ICLR-2024)

Reviewer comment (Writing):
Given that the proposed approach relies on contrastive learning, I would suggest the authors to provide a more self-contained and better written text when it comes to contrastive learning. Currently the paper assumes a non-trivial amount of familiarity to the topic and makes it difficult to follow. For instance, what does the following term mean: _jitter-and scale strategy_?

Author response:
Response: We present more related work in Appendix 1.

Reviewer comment (Experiments):
There are no evaluations for zero-shot evaluations. According to the motivation of the paper, it might be relevant as well to consider the case where the time series embedding is developed in a particular dataset, and then tested in another one. Although this might not be necessarily too far from the few-shot evaluation already presented.

Author response:
Response: Zero-shot forecasting experiments are in Appendix 2.2.5. Compared to TS transformers, our model has better results.

Reviewer comment (Evaluation):
There is no analysis on time execution. The authors do not report how long it takes to obtain the time series embeddings.

Author response:
Response: The training of encoders for time series embeddings only needs to use the text embedding obtained from LLM’s embedding layer and does not require training LLM. But LLMs are needed for training their head and inference. The encoder is a causal TCN (Blue font in paragraph 1 of page 7) and its training is fast, usually requiring only learn whole dataset once. Its detailed structure is in Appendix 2.1.1.

Reviewer comment (Evaluation):
The results are presented in an aggregated way across datasets. It would be of interest for researchers and practitioners how the proposed approach performs per dataset.

Author response:
Response: You can check them in Appendix 2.

Reviewer comment (Writing):
What is jitter-andscale strategy?

Author response:
Response: Sorry, given space limitations, some details have not been elaborated (Blue font in paragraph 1 of page 4). Here, the strategy is used for time series data augmentation: Weak augmentation is a jitter-and-scale strategy. We add random variations to the signal and scale up its magnitude; Strong augmentation is a permutation-and-jitter strategy, where permutation includes splitting the signal into a random number of segments with a maximum of M and randomly shuffling them. Please refer to: Eldele et al. Time-Series Representation Learning via Temporal and Contextual Contrasting. IJCAI 2021.

Reviewer comment (Writing):
Can the authors please expand a bit on what the following line means? _In this work, we choose P representative text embedding tp as pivots/prototypes, and map TS embedding to them_

Author response:
Response: The prototype is selected randomly. The prototypes (value, shape, frequency) shown in Figure 1 is an example. As in high dimensional space, almost all vectors are pairwise orthogonal, the number of prototypes rather than the type does matter (Blue font in paragraph 6 of page 5). Thus, by randomly selecting the same number of vectors, the represented space size and expressed number of features are almost the same (Appendix 2.5.2). 

Further, time series does not have text description. How to align objective data (time series, collected from complex systems) with subjective data (text, created and can be perceived by humans) is a topic worth studying but currently unresolved. This work aligns embedding spaces of time series and text through similarity, although there is no semantics and somewhat rigid, it is an enlightening attempt.

Reviewer comment (Writing):
Typo: Figure 3.g: spesific → specific

Author response:
Response: Thanks for your careful checks. We have corrected typos.

Reviewer comment (Writing):
Typo: Figure 2: Fine-turn → Fine-tune.

Author response:
Response: Thanks for your careful checks. We have corrected typos.

Reviewer comment (Presentation):
Why is LLM-QA not showing any variance? Or why is it depicted in this way?

Author response:
Response: As you have considered, different prompts can lead to different results. In this work, we used the prompt templates by referring to Promptcast (Blue font in paragraph 3 of page 7).

Xue et al. Promptcast: A new prompt-based learning paradigm for time series
forecasting. 2023.

Reviewer comment (Experiments):
Why is one-fits-all not used in *all* forecasting experiments? If I understand correctly, one can use it for comparison.

Author response:
Response: In the original version, we compared with one-fits-all named GPT4TS as shown in Figure 3f (blue font in paragraph 3 of page 7). And when we implemented our method, one-fits-all only had a preprinted version and the code was not available. Thus, we directly used results they reported without rerunning code. In this revision, we added detailed results in Appendix.

Reviewer comment (Experiments):
Why are not other models considered here? For instance DeepAR, MQCNN, Wavenet, NHiTS, NBeats, etc. for these experiments? Even models like ARIMA, ETS, seasonal naive would be interesting to consider.

Author response:
Response: Thanks for your reminder. In this revision, we have added more compared models in Appendix. And we’d like to demonstrate that during our tests on the listed datasets, the methods you mentioned did not perform as well as the methods we compared. You can refer to: Wu et al. TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis. ICLR 2023.

Reviewer comment (Evaluation):
From 3.d it seems that the prompt length should be restricted to not be particularly large, but surprisingly the smaller the number of parameters (GPT-117M) the better the performance. How does this fit with the previous observation that larger models perform better?

Author response:
Response: Sorry, it’s a typo. The y axis should represent classification accuracy, we have corrected it.

Reviewer comment (Writing):
When it comes to forecasting, is the proposed approach a point-wise or a probabilistic forecaster?

Author response:
Response: The forecasting mode is the same as LLM, with tokens output sequentially, but our time series token is a subsequence, like subsequence-wise.

Reviewer comment (Experiments):
What is the performance if the LLM taken is the one of the first checkpoint, i.e., when the weights of the model are random? What is the gap between these cases and those taken for the current experiments?

Author response:
Response: This is an interesting thought, but it is not applicable to our approach as the random model does not have a trained embedding dictionary to use as prototypes.

Reviewer comment (Evaluation):
What is the time execution/complexity of the proposed approach? Does this depend on the pretrained model in consideration? For instance, would it take longer for LLaMa-13B than for BERT-110M?

Author response:
Response: The training of encoders for time series embeddings only needs to use the text embedding obtained from LLM’s embedding layer and does not require training LLM. But LLMs are needed for training their head and inference. The encoder is a causal TCN (Blue font in paragraph 1 of page 7) and its training is fast, usually requiring only learn whole dataset once. Its detailed structure is in Appendix 2.1.1.

Reviewer comment (Evaluation):
What is the performance per dataset when it comes to time series forecasting? The current experimental setting only reports results aggregated across different datasets.

Author response:
Response: You can check them in Appendix 2.

Reviewer comment (Novelty):
Based on the experience with LLM and multimodal fusion, it is possible to integrate LLM with image/video/speech representations using techniques such as adapters once good representations are obtained. Similarly, LLM4TS and TS4LLM can be fused with each other, for instance, when we obtain a good TS token embedding using TS4LLM, we can directly fine-tune the large model using LLM4TS, thereby achieving good performance. Therefore, is it appropriate to isolate LLM+TS into LLM4TS and TS4LLM as two distinct categories?

Author response:
We agree with the situation you mentioned. From a technical perspective, these two ideas can be integrated and may work better together. But we give these two categories to highlight different focuses: changing the model more or changing the data more. We want to present our views that fine-tuning LLM to TS data is unnecessary and not worth the loss. The method of changing data rather than models is more practical. The categories are more like purpose-wise instead of technology-wise.
But your suggestions will greatly help to improve the academic rigor of our paper. In this revision, we have provided footnotes for categories (The footnote in blue on page 1). From a technical perspective, LLM+TS can be achieved by pre-training, fine-tuning, tool-augmented methods, external encoders, and their ensemble.

Reviewer comment (Evaluation):
Time series prediction is a crucial task, and as observed from the experiments in this paper, there is a significant gap between TS4LLM, LLM4TS, and traditional methods when performing this task. Have the authors considered analyzing the reasons for this discrepancy?

Author response:
We have added more models, please refer to the Appendix for more detailed experiments.

Different from the subjective text data, which can be perceived by humans, time series is more objective. The data from different scenarios varies greatly. Thus, it’s hard to build a fundamental model for all types of time series. This means models for medical time series data may be difficult to use in meteorological time series data. For example, RNN-based models are suitable for modeling medium and low-frequency data (e.g. vital signs, having sequence dependency), CNN-based models are suitable for modeling high-frequency data (e.g. electrocardiogram, having patterns), Transformer-based models are suitable for modeling long-term medium frequency data (e.g. electricity, weather, having periodic and trend characteristics). In this work, we are exploring the ability of existing LLM to solve time series problems. We do not recommend fine-tuning LLM to time series data as it will result in a loss of language ability. We recommend using additional adapters, such as an external encoder in this work, to enhance its time series analysis capabilities. Of course, we still believe that the current mainstream approach for implementing time series tasks is the traditional model, and the general fundamental model for is the crucial future work.

Reviewer comment (Writing):
In the statement "...we choose P representative text embedding tp as pivots/prototypes...", what specific type of text does P represent?

Author response:
The prototype is selected randomly. The prototypes (value, shape, frequency) shown in the figure is an example. As in high dimensional space, almost all vectors are pairwise orthogonal, the number of prototypes rather than the type does matter (Blue font in paragraph 6 of page 5). Thus, by randomly selecting the same number of vectors, the represented space size and expressed number of features are almost the same (Appendix 2.5.2). 

Further, time series data does not have textual description. How to align objective data (e.g. time series, collected from objective complex systems) with subjective data (e.g. text data, created by humans and can be perceived by humans) is a topic worth studying but currently unresolved. This work aligns the time series embedding space and the text embedding space through similarity, although there is no semantic relationship and it is somewhat rigid, it is also an enlightening attempt.

Reviewer comment (Experiments):
The paper lacks clarity regarding how the representative prototypes are chosen. There are three groups of prototypes (value, shape, frequency), and it's unclear how these are determined. Does their selection depend on the dataset, and what criteria are used to assess their effectiveness?

Author response:
The prototype is selected randomly. The prototypes (value, shape, frequency) shown in the figure is an example. As in high dimensional space, almost all vectors are pairwise orthogonal, the number of prototypes rather than the type does matter (Blue font in paragraph 6 of page 5). Thus, by randomly selecting the same number of vectors, the represented space size and expressed number of features are almost the same (Appendix section 2.5.2). 

Further, time series data does not have textual description. How to align objective data (e.g. time series, collected from objective complex systems) with subjective data (e.g. text data, created by humans and can be perceived by humans) is a topic worth studying but currently unresolved. This work aligns the time series embedding space and the text embedding space through similarity, although there is no semantic relationship and it is somewhat rigid, it is also an enlightening attempt.

Reviewer comment (Experiments):
I would like to see an examination of the performance impact of removing the decoder component. It looks like the decoder is not a necessary component in the proposed method.

Author response:
The decoder is the head of LLM. We use it as a pre-training trick. Of course, it is also possible to not train it during contrastive learning.

Reviewer comment (Experiments):
Have you considered the potential for parameter-efficient fine-tuning methods, such as LoRA, to outperform the use of soft prompts? Similar to using soft-prompts, LoRA does not require updating the model's entire parameter set. Reference: LoRA: Low-Rank Adaptation of Large Language Models, Hu et. al.

Author response:
We agree with your comment. However, in our previous experiments, LoRA was not as good as p-tuning in the embedding layer. Personally, I think LoRA is more suitable for self-attention blocks.

And your suggestion is also what we are considering. We suggest combining LLM and TS in a more effective way, rather than directly fine-tuning. In this way, it’s more possible to enable LLM to have both language and TS analysis capabilities. Thus, beyond this work, we are exploring a plugin-based paradigm of LLM+TS. Plugins are likely to be the additional adapters or external encoders, implemented by using the PEFT methods, like LoRA. The current experiment results show that the PEFT-based plugins can also produce nice results.