Paper: TEMPO: Prompt-based Generative Pre-trained Transformer for Time Series Forecasting (ICLR-2024)

Reviewer comment (Theory):
Theorem 3.1 -- I do not fully understand the implications of Theorem 3.1 and how that affects the design choices of the authors.

Author response:
[Claims W2& Q1] Design choice behind Theorem 3.1:

We have expanded the discussion of Theorem 3.1 to provide more context on how it influenced our design choices in the main paper and here is a more detailed explanation:

Theorem 3.1 gives the insights that if the trend and seasonal components of a time series are not orthogonal (i.e. their inner product is non-zero), then there does not exist a set of orthogonal bases that can completely disentangle and separate the two components.
As analyzed in [1]’s Theorem1, the self-attention layer inherently learns an orthogonal transform that projects inputs into a disentangled latent space, analogous to how PCA decomposes signals into orthogonal principal components. So directly applying attention to a raw time series would fail to disentangle non-orthogonal trend and seasonal components.
Therefore, we manually perform this decomposition using statistical methods like STL, and provide the separated components as inputs to the transformer model. This allows us to provide attention with the separated trend, seasonal, and residual components as inputs. Attention can then focus on modeling each individual component in its disentangled latent space without entanglement. Without explicit decomposition, the trend and seasonal signals would be entangled in the attention space, which will hurt the performance as shown in the Ablation study section and in Table 9.


In summary, Theorem 3.1 formally proves the limitation of standard self-attention for time series decomposition. This motivates the design choice of using statistical decomposition techniques and providing the separated components as input to the transformer.

[1] Zhou, Tian, et al. "One Fits All: Power Time Series Forecasting by Pretrained LM." Advances in Neural Information Processing Systems 36 (2023)

Reviewer comment (Experiments):
Prompt pool captures seasonal and trend components: The authors provide an example of 3 time-series from 1 dataset to demonstrate that the prompts capture season and trend components, but I am not sure this is sufficient evidence. It would be interesting to look at the distribution of prompts for multiple (or all) time-series in one or more datasets, as time-series are clustered based on their trend and/or seasonality components. I believe this would give a more, dataset level evidence for the authors' claims.

Author response:
[W3] Distribution of Prompt Pool:

Thank you for raising this important point for further improve the representation of prompt pool. You are absolutely right that showing prompt selection trends across multiple full datasets would provide more robust evidence that the prompts are capturing seasonal and trend components. To clarify, the examples in Figures 8, 9, 10, 11, and 12 are actually from different datasets (ETTm1, ETTh1, ETTh2, Weather, Electricity), not a single dataset. However, we agree that a dataset-wide analysis could provide more robust evidence that this holds more broadly. 

In addition, we should analyze prompt selection distributions at the dataset level to support our claims. As an additional analysis, Figure 7 (new added in Appendix) plots the prompt selection distribution histograms of the selected prompts for the trend, seasonal, and residual components on ETTm1 and ETTm2 datasets. In our setting, each data can select multiple prompts (in this case, 3 prompts for each component), the frequency here is calculated by the count of each prompt selected by one data.  We can see clear differences in the prompts preferred for periodic vs seasonal components. For example, in ETTm1 prompts 11, 15, and 28 are most frequently chosen for trends while prompts 5, 7, and 18 dominate for residuals. This provides further evidence that the prompt pool can retrieve specialized prompts for distinct time series characteristics after the training stage.

Reviewer comment (Evaluation):
Interpretability: I am not sure how the GAM and SHAP provide interpretability, beyond confirming what is expected from these models, i.e. the residuals do not have any pattern.

Author response:
[Claim W4] The Role of GAM and SHAP:

In response to your concern about the interpretability provided by Generalized Additive Models (GAM) and SHapley Additive exPlanations (SHAP), we have clarified their roles in our Appendix C in terms of beyond merely confirming the absence of patterns in the residuals. On the one hand, GAM explicitly models the different feature effects as additive components, where the interpretability is inherently incorporated into TEMPO; On the other hand, SHAP attributes feature effects after the fact to explain opaque model predictions. Thus, GAM and SHAP are powerful tools not only for confirming expected findings but also for providing deeper insights and explanations into the workings of complex models. The utility of GAM and SHAP in our analysis can be detailed as follows:

First of all, the analyses quantitatively confirm assumptions about model behavior with data-driven evidence, rather than just intuition. This substantiation increases the trust and transparency in the model's predictions. 

Second, the component attribution could reveal unexpected behaviors if present. For example, residual impact being higher than expected could indicate overfitting noise.

Third, SHAP provides nuance beyond high-level expectations, like showing the increasing error of seasonal components in longer forecasts.

Reviewer comment (Experiments):
"Large-scale experiments/benchmarks": The authors omit several benchmarks, and therefore I would argue that the experiments are not large-scale. For e.g., for long horizon datasets, the authors do not use the Influenza-like Illnesses and Exchange-rate datasets which PatchTST and TimesNet, and other recent studies. Secondly, there are multiple short-horizon benchmarks, like M3 or the M4 datasets, and the much larger and comprehensive Monash Forecasting archive, yet the authors do not confirm their methods on these datasets.

Author response:
[Experimentation W1] "Large-scale experiments/benchmarks"

We appreciate your insightful feedback regarding our experiment scope and the need for more comprehensive benchmarks. In our submission, we did not claim to conduct "large-scale experiments/benchmarks". However, to avoid any misunderstanding, we will substitute the term "broad" for "large-scale" in describing our evaluations. For the long-term forecasting setting, we intentionally excluded the Influenza-like Illnesses and Exchange-rate datasets due to their relatively limited time steps compared to the other seven widely used datasets. Specifically, the Exchange-Rate dataset has 8 variants with 7,588 timesteps (granularity: 1 day) and the Influenza-like Illnesses dataset has 7 variants with 966 timesteps (granularity: 1 week). These limited samples potentially constrain the activation of knowledge in a pre-trained model. Nevertheless, we emphasize the significant improvements on the datasets we did evaluate, attesting to our model's strong adaptability as shown in Section 4.3. For short-term forecasting, our goal extends beyond simply comparing results. We aim to illustrate the potential of multi-modal input in both in-domain and cross-domain sectors. Our model, TEMPO, reduces the average SMAPE error by 32.4% and 39.1% in these sectors, respectively. We focused on commonly used forecasting datasets from recent literature, but acknowledge the value of incorporating larger benchmarks, such as the M3, M4, and Monash Forecasting Archive datasets, as you've suggested. We will prioritize incorporating tests on the larger benchmark collections you mentioned in the final version. We welcome any further suggestions you may have for enhancing the experimental scales.

Reviewer comment (Experiments):
Multiple methods are omitted from the evaluation, for e.g. statistical methods such as Auto-ARIMA, Auto-THETA, Auto-ETS etc., and deep learning methods such as N-HITS and N-BEATS. Also the authors cite PromptCast but do not compare their method to this particular baseline.

Author response:
[W2] Multiple Baselines

We appreciate your insights regarding our model evaluation and the suggestion to include statistical methods like Auto-ARIMA, Auto-THETA, Auto-ETS, and deep learning methods such as N-HITS and N-BEATS.
It is important to note that ARIMA, N-HITS, and N-BEATS are primarily designed for univariate time series forecasting, which is not the focus of our study on multivariate long-term forecasting. The results presented in the Autoformer paper reveal that Autoformer outperforms both N-BEATS and ARIMA in univariate time series forecasting settings. Given the outperformance of our model compared to the Autoformer approach, we feel confident that our model could also outperform ARIMA, N-HITS, and N-BEATS.

Regarding the omission of PromptCast, we acknowledge that as a prompt-based approach, it would have been a relevant comparison. However, there are key differences in our methodologies:
- PromptCast requires manual creation and curation of a dataset of prompts and time series for pre-training. In contrast, our method utilizes a prompt tuning strategy that automatically constructs relevant prompts by retrieving from a shared pool. This offers more flexible prompt learning, specifically tailored to the downstream forecasting task.
- Our method integrates decomposition with prompt tuning, which effectively incorporates numerical time series patterns into the pre-trained language model. Conversely, PromptCast applies prompts at a more superficial level without such grounded integration.

Despite these differences, we concur that a direct comparison with PromptCast and other models would enrich our evaluation process. In future work, we plan to expand our evaluation to include classical statistical methods, established deep-learning approaches, and prompt-based competitors like PromptCast. This will provide a more comprehensive comparison and help identify our approach's unique strengths and potential limitations.
Thank you for bringing our attention to these additional models. We appreciate your constructive feedback and will incorporate these insights into our future research.

Reviewer comment (Evaluation):
The value of prompt pool-- The authors demonstrate in Table 9 that the prompt pool helps model prediction. How would they explain the methods without prompt pooling doing better on some datasets for some forecasting horizons?

Author response:
[W3] The value of Prompt Pool

In the era of large models, the notion of learning from scratch has started to give way to more efficient approaches. One of these is the utilization of prompts to guide pre-trained models, which permits focused learning from relevant knowledge, a concept derived from memory consolidation theory. Time series data are often non-stationary, meaning its properties can change over time. This presents a significant challenge for traditional models, which often struggle to adapt accordingly. The goal of the prompt pool is to direct the reusing of learnable representations at the instance level, which embodies temporal knowledge of trends and seasonality. Essentially, this allows the model to adaptively accumulate knowledge across fluctuating time distributions, thereby ensuring the model's ability to forecast is maintained as generative processes progress. 

While the improvement from the prompt pool in TEMPO might seem minimal at first glance, it becomes evident that the model's performance declines when the prompt pool is absent. This is particularly noticeable in the zero-shot setting, as highlighted in Tables 10 and 11 in the Appendix, where the absence of the prompt pool leads to a clear drop in TEMPO's performance. A case study provided in Appendix D.1 further illustrates the utility of the prompt pool, demonstrating its ability to retrieve similar patterns, an attribute we believe contributes to the performance enhancement. Despite these benefits, as discussed in section D.2 on 'leave-one-out' analysis, the prompt pool introduces additional parameters that require tuning. Determining optimal settings for these parameters is a complex issue that we plan to address in future research.

[Minor]: We will polish it accordingly. Thanks for the suggestion!

Reviewer comment (Writing):
Typos: inclduing, outpemforms ... etc.

Author response:
[Minor]: We will polish it accordingly. Thanks for the suggestion!

Reviewer comment (Theory):
Theorem 3.1 -- I do not fully understand the implications of Theorem 3.1 and how that affects the design choices of the authors. What do we learn from the theorem?

Author response:
[Claims W2& Q1] Design choice behind Theorem 3.1:

We have expanded the discussion of Theorem 3.1 to provide more context on how it influenced our design choices in the main paper and here is a more detailed explanation:

Theorem 3.1 gives the insights that if the trend and seasonal components of a time series are not orthogonal (i.e. their inner product is non-zero), then there does not exist a set of orthogonal bases that can completely disentangle and separate the two components.
As analyzed in [1]’s Theorem1, the self-attention layer inherently learns an orthogonal transform that projects inputs into a disentangled latent space, analogous to how PCA decomposes signals into orthogonal principal components. So directly applying attention to a raw time series would fail to disentangle non-orthogonal trend and seasonal components.
Therefore, we manually perform this decomposition using statistical methods like STL, and provide the separated components as inputs to the transformer model. This allows us to provide attention with the separated trend, seasonal, and residual components as inputs. Attention can then focus on modeling each individual component in its disentangled latent space without entanglement. Without explicit decomposition, the trend and seasonal signals would be entangled in the attention space, which will hurt the performance as shown in the Ablation study section and in Table 9.


In summary, Theorem 3.1 formally proves the limitation of standard self-attention for time series decomposition. This motivates the design choice of using statistical decomposition techniques and providing the separated components as input to the transformer.

[1] Zhou, Tian, et al. "One Fits All: Power Time Series Forecasting by Pretrained LM." Advances in Neural Information Processing Systems 36 (2023)

Reviewer comment (Writing):
Model design: The authors mention they use temporal encoding from PatchTST but do not provide any definition or references for it. As far as I am aware, PatchTST does not use temporal encoding. That is, if my understanding of temporal encoding is accurate. Moreover, work prior to PatchTST, either Informer or Autoformer use temporal encoding instead.

Author response:
[Q2] Model Design

We appreciate your keen observation and the opportunity to clarify our use of the term "temporal encoding". In our work, when we refer to "temporal encoding", we mean the process of transforming temporal numerical input data into a format suitable for pre-trained language models, while preserving its temporal information. This process is inspired by the patch operation used in PatchTST.

We recognize, however, that the term "temporal encoding" can easily be conflated with "temporal embedding", a concept used in transformer-based models such as Informer and Autoformer. The latter refers to the practice of embedding time information into the model to capture temporal dynamics, which is different from our use of the term.
To avoid any future misunderstanding, we will provide a clearer contextualization of this term in the revised version of our paper. We're grateful for your feedback, as it helps us enhance the clarity and precision of our work.

Reviewer comment (Experiments):
Experimentation: How are LLM-based models like BERT, T5, LLaMA used for forecasting? What is the input to these models? What is the output?

Author response:
[Q3] How are LLM-based models like BERT, T5, LLaMA used for forecasting? 

You raise an excellent question - we have provided more implementation details for the LLM baselines in our polished version in blue color. To forecast with models like BERT, we use the same overall approach as TEMPO:

- Apply patching to convert the numeric time series into numerical vectors with temporal order and the shape of LLM’s input (Refer to the last paragraph of Section 3.2 and [1])

- Pass the numerical vectors into the LLM to obtain a hidden representation from each LLM.

- Project the representation through a linear layer to predict the target future values (Refer to Section 3.4).

So the LLM takes the numerical past context as input, and outputs forecasted numeric values for the future timesteps. The key difference from TEMPO is the lack of decomposition and prompt tuning.

Reviewer comment (Experiments):
Experimentation: What is the context length for the long-horizon forecasting task?

Author response:
[Q4] Experimentation: What is the context length for the long-horizon forecasting task?

Thanks for raising the question about the context length details for long-horizon forecasting. We have added these details in Section D.1. Following the [1], we use a context length of 336 timesteps for all the long-term forecasting experiments.
As you noted, typical context lengths from prior work are in the range of 168-336. Informer uses 336 as one of the baseline models, so 336 is a relevant reference point.
By aligning with the context length in [1] and Informer, we aim for as fair a comparison as possible with those state-of-the-art baselines.


[1] Zhou, Tian, et al. "One Fits All: Power Time Series Forecasting by Pretrained LM." Advances in Neural Information Processing Systems 36 (2023).

Reviewer comment (Experiments):
Experimentation: How are 5/10% of the data points used? How do you define a data point in a long-horizon forecasting time-series? Also 5-10% seems like a lot of data, given that most of the time-series have multiple channels, and the model processes them independently.

Author response:
[Q5] Experimentation: 5%/10% data:

We value your keen observations and agree that the terminology used in our paper requires clarification. In our research, when we refer to a "data point," we mean each individual timestep in a time series. This is in line with the standard definition used in time series analysis.
Our methodology for producing the training set employs the same technology as the One-fits-all approach. This involves taking a portion (5%/10%) of the time series data for training and using test data for testing. However, we acknowledge your concern that using 5-10% of data might seem excessive, especially considering that our model processes multiple channels independently. In our original submission, we acknowledge that the term "few-shot" might lead to potential misunderstandings in this context. As such, we have consciously avoided using "few-shot" in the main text of our paper. Instead, we have chosen "Data-efficient adaptation" as a section title, which we believe more accurately encapsulates our approach. This choice of wording more accurately reflects our approach, which aims to achieve superior performance with less data compared to traditional methods.

In our revised paper, we will further clarify our definition of a "data point" and our methodology for determining the training set. We appreciate this opportunity to improve the clarity and accuracy of our paper.

Reviewer comment (Novelty):
The idea of decomposition of trend, seasonality, and residuals is not that novel and has been used for time-series forecasting.

Author response:
[W1] Novelty for the decomposition:

While decomposing time series into trend, seasonality, and residuals has been explored before, the innovation in this work lies in the unique integration into a pre-trained transformer model paired with a prompt tuning strategy. Theoretical analysis reveals standard attention may fail to disentangle non-orthogonal trend and seasonal components, common in real data. By leveraging classical STL decomposition and mapping components into separate hidden spaces, the model capitalizes on the semantic reasoning of pre-trained models like GPT to understand each individually.

This principled input representation allows the transformer to capture intricate time series dynamics across diverse applications. The prompt pool further enhances adaptability, enabling instance-level tuning to consolidate shifting temporal knowledge. The model can recall relevant learned representations encoded in prompts based on input similarity.
In essence, while decomposition and prompting have been separately studied, their novel combination here to unlock the capabilities of pre-trained transformers is a key innovation. The approach is grounded in both theory and empirical results. While the individual pieces may not be wholly novel in isolation, their synergistic integration to advance time series modeling with modern techniques represents a significant contribution. The improvements in accuracy and data efficiency validate the benefits of unifying classical decomposition with prompt-based transfer learning in pre-trained transformers.

Reviewer comment (Theory):
The theorem 3.1 does not directly prove the point “more importance in current transformer-based methods as the attention mechanism, in theory, may not disentangle the disorthogonal trend and season signals automatically".

Author response:
[W2] To clarify the theorem 3.1:

We clarified the theorem 3.1 in the main paper in orange color.

Theorem 3.1 establishes that when the trend T(t) and the seasonality S(t) are not orthogonal, there cannot exist a set of orthogonal bases that can disentangle S(t) and T(t) into two disjoint sets of bases. It's important to note that it's quite common that the trend and seasonal components are not independent of each other in many real-world time series, and this interdependence needs to be accounted for when decomposing the time series for analysis and forecasting.

The reason for this lies in the nature of their spectrums: the spectrum of a periodic signal is discrete, while the spectrum of a non-periodic signal is continuous. Consequently, overlaps on the non-zero frequencies of the periodic signal are quite likely.
Principal Component Analysis (PCA), which also aims to learn a set of orthogonal bases on the data, thus, cannot disentangle these two signals into two separate sets of bases. Drawing from [1] Theorem 1, it becomes evident that the self-attention mechanism in pre-trained large models executes a function bearing a strong resemblance to PCA. Hence, unless this operation is manually carried out, the self-attention mechanism cannot automatically decompose the time series into its trend and seasonal components when there exists inherent non-orthogonality of these components. 

Note that, this theoretical understanding can provide insights on why manual decomposition can boost the model's performance. To make better use of the semantic capabilities of the pre-trained model, we propose conducting a decomposition operation on the time series. This operation aims to separate the orthogonal/non-orthogonal trend and seasonal components, thereby enhancing the model's ability to understand and predict the time series data.

[1] Zhou, Tian, et al. "One Fits All: Power Time Series Forecasting by Pretrained LM." Advances in Neural Information Processing Systems 36 (2023)

Reviewer comment (Experiments):
The result shown in Figure 2 seems to be obvious since the trend is easier to learn and may take a large portion of the data.

Author response:
[W3] SHAP value

We have addressed the role of SHapley Additive exPlanations (SHAP) value in Appendix C in our revised version. Here, we briefly summarize the answer as follows:

SHAP values provide a comprehensive quantification of the importance of each feature. They measure the average contribution of each feature to the prediction output across all potential feature combinations. In time series analysis, it's intuitive to assume that the trend significantly influences the prediction process. However, we need a more nuanced understanding of how various types of component errors increase, leading to a decrease in the final prediction accuracy, particularly as the prediction length expands.

For instance, in datasets exhibiting strong seasonality, the seasonal component may display much larger variations than the residual component. Conversely, in datasets with minimal seasonality, the variations between these two components should be more comparable. We can calculate the strength of seasonality via: $\mathrm{S}=\max \left(0,1-\frac{\operatorname{Var}\left(R_t\right)}{\operatorname{Var}\left(S_t\right)+\operatorname{Var}\left(R_t\right)}\right)$. When we compare the seasonality strengths of different datasets, we find that ETTm1 (as shown in Figure 2, with a seasonality strength of 0.99) constitutes strongly seasonal data, whereas the weather dataset (depicted in Figure 13 with a seasonality strength of 0.476) exhibits less seasonality and a more pronounced trend.
These findings align with the conclusions drawn from the SHAP values. The performance degradation of ETTm1, when the prediction length is increased, can be primarily attributed to inaccuracies in the prediction of seasonal terms. In summary, SHAP provides insights into how the model arrives at the predictions and how it can be further improved. The ability to discern how much and where components contribute enables targeted improvements. These insights can guide us in better leveraging inductive bias to enhance both efficiency and effectiveness in the era of pre-training models. One of the interesting future works is that we can adaptively and selectively optimize specific components based on the SHAP scores during the training process. This approach would allow us to focus our computational resources and efforts on the most influential components, thereby improving the overall effectiveness of the model.

Reviewer comment (Experiments):
In the few-shot learning setting, do you need to finetune the model on other time-series datasets first?

Author response:
[Q1] Experiment setting:

You raise an excellent question about the experimental setup. To clarify, in the data-efficient adaptation experiments in Section 4.2, we do not perform any fine-tuning of the model on other time series datasets. For each experiment, only a small fraction of the target dataset's training data (e.g. 5%, 10%) is utilized, without additional data from other domains.
In addition, we evaluate our model's zero-shot learning capabilities in two different settings without any fine-tuning:
- In Section 4.3, the model is trained on 5 time series datasets, then directly tested on ETTm2 and Traffic without ever seeing those datasets.
- In Section 4.4, the model trained on in-domain data is zero-shot tested on the unseen cross-domain sectors of the proposed TETS dataset.

In both cases, the model is not fine-tuned or adapted using any examples from the test datasets. The strong performance demonstrated highlights the model's ability to generalize and transfer knowledge to entirely new distributions without dataset-specific fine-tuning.

Reviewer comment (Theory):
What is the computation complexity of the proposed method, and how it is comparable to other methods?

Author response:
[Q2]  Computation Complexity:
The computational complexity of the proposed TEMPO model can be analyzed as follows:
- Decomposition into trend, seasonal, residual via STL has complexity O(n), where n is the length of the input time series.
- Mapping components to hidden spaces is just linear transformation, also O(n).
- Prompt pool selection is O(nm), where m is the number of prompts, based on similarity scoring.
- The self-attention layers in the GPT transformer backbone dominate complexity. Each layer is O(n^2 * d) for sequence of length n and hidden dimension d. With L layers, the overall complexity is O(Ln^2*d).

So the total complexity is O(Ln^2*d) as the transformer layers dominate. This is comparable to state-of-the-art transformers like Informer, and PatchTST, which also have quadratic self-attention complexity. As shown in Figure 15, TEMPO also has lower training time than models like LLAMA, TimeNET and FEDformer. So, while TEMPO has a comparable order of complexity to related transformers, it provides accuracy, capability and efficiency benefits from the unique proposed components. The overhead of decomposition and prompting is minor compared to the self-attention layers.

Reviewer comment (Evaluation):
Based on the ablation study in Table 9, it seems that w/o prompts have less effect on the final performance. Sometimes without prompts, the performance can be even better. Could you explain this?

Author response:
[Q3] The improvement of Prompt Pool:

As also mentioned by Reviewer 2NZn, the marginal gains from adding the prompt pool in some cases, seem small given its intended benefits. We want to address that dataset characteristics may favor the specialized modeling of other methods over TEMPO's more generalizable approach in some cases. For example, highly linear or simple patterns may benefit from simpler baselines. Here are a few potential reasons the prompt pool improvements may be weaker than expected:

- The pre-trained model itself already encodes substantial temporal knowledge, limiting the additional value prompts can provide.
- The decomposition and input representation techniques may already equip the model to handle trends, seasonality, etc well - reducing reliance on prompts.
- The decomposition may not be optimal for all time series if there are more complex nonstationary patterns. Our current prompt design focuses on trends and seasonalities.
- The prompt tuning method may not be optimizing and selecting prompts from the pool as effectively as possible. Suboptimal prompts could limit gains.
- Prompts help most for complex cases like sparse data or domain shifts. Many evaluation datasets may be too dense/similar for prompts to shine.
- The power of prompts manifests more in their scalability and generalization ability rather than raw performance gains. Their value may be more in transferability.

In summary, the intuitive benefits of the prompt pool may be dampened by the model's existing capability, dataset characteristics, prompt optimization difficulties, and other factors. But your observation highlights opportunities to improve prompt integration to unlock their full potential.

Reviewer comment (Reproducibility):
How the prompts are initialized and trained.

Author response:
[Q4] Initialization and Training of Prompt Pool:

We added more information on the training and initialization of the prompt pool in Appendix E.4 in green color according to your suggestion. Here is a summary of how the prompts are initialized, trained, and designed in our work:
- Initialization: The prompt embeddings in the pool are randomly initialized from a normal distribution, as is standard practice for trainable parameters in neural networks.
- Training: The prompts are trained in an end-to-end manner along with all other model parameters to optimize the forecasting objective. This allows the prompts to be continuously updated to encode relevant temporal knowledge.
- Design: The time series retrieval-based prompt pool length is selected via grid search based on performance. We analyze different design choices for prompt type on TETS dataset: prompt pool, hard prompt and soft prompt, etc. in Appendix D.3 (Table 12, 13). 
- Analysis: Appendix D.4 provides analysis on prompt effectiveness by visualizing prompts selections and similarity. We find prompts successfully retrieve related representations for time series with similar patterns.

In summary, the prompts are standard randomly initialized embeddings that are trained end-to-end. The number of prompts and embedding dimensions are treated as hyperparameters and tuned for good performance. Detailed design analysis provides insights into prompt similarity and selection. Our work offers an initial exploration into prompt-based tuning for time series forecasting, but substantial room remains for advancing prompt pool design. The number and structure of prompts can be investigated, along with initialization schemes, similarity scoring functions, and prompt training strategies. Establishing more rigorous design principles for prompt pools tailored to temporal data would be a valuable direction for future investigation, and could uncover techniques for maximizing the effectiveness of prompt-based knowledge transfer.

Reviewer comment (Experiments):
In Table 1, why don’t you include the traffic forecasting results?

Author response:
[Q5] Traffic dataset:

Due to constraints on space in the current version of the paper, we have not included the traffic forecasting results in Table 1. However, we appreciate the importance of these results for a comprehensive understanding of the method's performance. Therefore, we intend to incorporate them (in Table 5 in Appendix) into the final version of the paper. This addition will ensure a more thorough evaluation and discussion of our proposed method's performance across various applications.

Reviewer comment (Experiments):
How generalizable can the model be for the unseen domain transfer? For example, what if you choose weather and traffic as the training domain then apply the model to EETm1 and EETm2.

Author response:
[Q6] Unseen Domain Transfer

The strong performance shown in Sections 4.3 and 4.4 illustrates the model's capability for generalizing to entirely new and unseen domains without any fine-tuning. For example, in Section 4.3 the model trained on datasets like ETT and Weather generalizes well to diverse unseen datasets like ETTm2 and Traffic. While we did not explicitly test weather/traffic to ETT, based on the zero-shot transfer results, we can reasonably expect competitive performance in this setting as well. In our work, the decomposition and prompting techniques equip the model to extract meaningful representations even for out-of-domain data.
More rigorously evaluating the model's zero-shot learning capacity across diverse domain pairs is an interesting future direction. However, the current results provide promising indications that the model can generalize broadly to unseen distributions. The prompt tuning in particular allows flexibly retrieving relevant knowledge from past experiences.

However, it is worth noting that this transfer is likely to be more effective if there are shared patterns across the datasets. If the training domains (e.g., weather and traffic) share underlying temporal or structural similarities with the target domain (e.g., ETTm1 and ETTm2), the model is better positioned to apply learned insights to the new context. In other words, the strength of the transfer can be influenced by the degree of commonality in patterns across the different domains. Further, adjustments and fine-tuning may be necessary to optimize performance in some cases. Despite these potential challenges, our findings in Section 3.3 and Section 3.4 provide promising evidence of the model's adaptability and robustness in domain transfer scenarios.

Reviewer comment (Experiments):
It seems only decoder-based pretrain model is considered in this paper. The encoder based backbone (like Bert) and encoder-decoder based backbone (like T5) is also recommend in this stage.

Author response:
[W3 & Q3]: More results.

Thank you for raising this important point. You are absolutely right that we should have included results for encoder-only and encoder-decoder models in addition to the decoder-only GPT architecture. Here are a few key reasons we focused solely on GPT initially:

- GPT has shown strong performance on auto-regressive forecasting tasks in prior work, so we wanted to validate its efficacy with our approach.

- The decoder-only design fits naturally with the auto-regressive forecasting formulation.

However, please refer to the Tables 5, 6, and 7 in the appendix for the encoder-based (Bert) model’s results and the encoder-decoder-based (T5) model’s results. T5 and Bert perform better than GPT4TS on the Weather, Traffic and ECL datasets, but worse on ETT* dataset.

Reviewer comment (Evaluation):
Do you have any insights on the prompt pool’s marginal improvement, which is somehow against the intuition;

Author response:
1. [W1 & Q1] Insights on the prompt pool:

You raise an excellent point - the marginal gains from adding the prompt pool in some cases, seem small given its intended benefits. We want to address that dataset characteristics may favor the specialized modeling of other methods over TEMPO's more generalizable approach in some cases. For example, highly linear or simple patterns may benefit from simpler baselines.
Here are a few potential reasons the prompt pool improvements may be weaker than expected:
- The pre-trained model itself already encodes substantial temporal knowledge, limiting the additional value prompts can provide.
- The decomposition and input representation techniques may already equip the model to handle trends, seasonality, etc well - reducing reliance on prompts.
- The decomposition may not be optimal for all time series if there are more complex nonstationary patterns. Our current prompt design focuses on trends and seasonalities.
- The prompt tuning method may not be optimizing and selecting prompts from the pool as effectively as possible. Suboptimal prompts could limit gains.
- Prompts help most for complex cases like sparse data or domain shifts. Many evaluation datasets may be too dense/similar for prompts to shine.
- The power of prompts manifests more in their scalability and generalization ability rather than raw performance gains. Their value may be more in transferability.

Reviewer comment (Experiments):
The description of TETS dataset is suggested to include into the main paper;

Author response:
2. [W2 & Q2] Adding TETS dataset's description:

Thank you for the suggestions on improving the presentation of our paper. We added the summary of our proposed TETS dataset in Appendix H  (in red color) and will add it in the main paper once we have extra space:

The TETS dataset is a financial dataset primarily sourced from the quarterly financial statements of the 500 largest U.S. companies that make up the S\&P 500 index. It spans 11 sectors, including Basic Materials, Communication Services, Energy, Financial Services, Healthcare, Technology, Utilities, Consumer Cyclical, Consumer Defensive, Industrials, and Real Estate. To create a diverse task setting, the dataset is divided into training and evaluation sectors (the first seven sectors), and unseen sectors (the last four) for zero-shot forecasting. Sliding Window[1] is used to build the time series samples, where we can get 19,513 samples from 7 sectors as seen data for training and testing and the other unseen data are only used for zero-shot testing.

The process of collecting and summarizing this contextual information is facilitated by the ChatGPT API from OpenAI. It effectively generates the desired contextual information based on consolidated details such as company information, query, quarter, and yearly context. Neutral contextual information is denoted by 'None' when the API fails to generate any context. The TETS dataset thus offers a comprehensive view of both time series data and their corresponding contextual texts for each company.


[1] Antony Papadimitriou, Urjitkumar Patel, Lisa Kim, Grace Bang, Azadeh Nematzadeh, and Xiaomo Liu. 2020. A multi-faceted approach to large-scale financial forecasting. In Proceedings of the First ACM International Conference on AI in Finance. 1–8.