Paper: Time-LLM: Time Series Forecasting by Reprogramming Large Language Models (ICLR-2024)

Reviewer comment (Presentation):
While this paper is generally well-written, there are areas that could benefit from further refinement. For instance, Fig. 5 lacks clarity and might benefit from being displayed at a larger scale to improve visibility. In patch reprogramming (Sec. 3.1), it would be beneficial to illustrate how the linear projection aligns the hidden dimensions in Fig. 2. Within the Prompt-as-Prefix discussion, reference is made to the inclusion of statistics within the prompt. Elaborating on the specific content and the calculation methods employed would enhance clarity.

Author response:
+ We agree with the reviewer's suggestion to highlight Fig. 5 in a larger size. However, due to space constraints in the main text, scaling up Fig. 5 poses challenges as it may compromise the presentation of other results and discussions.
+ In response to the suggestion of depicting the transformation from $\mathbf{Z}^{(i)}$ to $\mathbf{O}^{(i)}$ in Fig. 2, **we have slightly revised this figure in the part of "Patch Reprogram"**.
+ To construct the statistical prompting block shown in Fig. 4, we compute the following representative statistics: (1) the minimum, maximum, and median values of the given time series; (2) the overall trend of the time series, determined by calculating the sum of differences between consecutive time steps. A sum greater than 0 indicates an upward trend, while a lesser sum denotes a downward trend; (3) the top-5 lags of the time series, identified by computing the autocorrelation using fast Fourier transformation and selecting the five lags with the highest correlation values. Please refer to our code for more details, which has been made available to AC and all reviewers. **We have also revised the paper accordingly and included this technical detail in the appendix.**

Reviewer comment (Novelty):
The paper's foundational concept hinges on the idea of reprogramming. However, it omits references to several pivotal reprogramming studies from recent years (as listed below). Integrating these pertinent studies in the introductory or related works section could provide a more robust understanding of the concept.

Author response:
Thank you for the suggestion. To enhance comprehension of model reprogramming and more related works, **we have revised and expanded the discussion on cross-modality adaptation in Appendix A**.

Reviewer comment (Writing):
There are some writing issues. For instance, there should be uniformity in the references' formatting, with a preference for citing formally published works from conferences or journals over preprints.

Author response:
Thank you for the suggestion. **We have thoroughly reviewed and updated the reference list, including both the format and venues of the references**. Additionally, considering the rapid growth in LLMs and related studies, many references in our work are recent preprints from 2023. These references are crucial in offering the necessary background knowledge for a better understanding of the proposed method.

Reviewer comment (Writing):
Could you elucidate the mechanism behind "Output Projection" (Sec. 3.1), particularly the aspects related to flattening and the linear projection? A formulaic representation would greatly aid in understanding.

Author response:
Upon packing and feedforwarding the prompt and patch embeddings $\mathbf{O}^{i}$ through the frozen LLM, we discard the prefixal part and obtain the output representations, denoted as $\tilde{\mathbf{O}}^{i}$. Subsequently, we follow PatchTST [1] and flatten $\tilde{\mathbf{O}}^{i}$ into a 1D tensor with the length $P \times D$, which is then linear projected as $\hat{\mathbf{Y}}^{i} \in \mathbb{R}^H$. Due to space constraints in the main text, **we have revised the paper accordingly and included this technical detail in Appendix B.1**.

Reviewer comment (Reproducibility):
In the "Prompt-as-Prefix" paragraph, there is a reference to computing the trend and lag in relation to time series, with several potential implementations hinted at. Could the authors detail the methodology used to determine the trend and lag information within the prompts?

Author response:
See our response above to W1.

Reviewer comment (Experiments):
The authors chose a limited set of time series data to benchmark this model on, while many of the models they compare against have been benchmarked against other datasets such as weather, traffic, electricity consumption, and the spread of influenza. This confuses the objective of the paper.

Author response:
We extend our sincere thanks to the reviewer for their constructive feedback on our paper. We are deeply grateful for the recognition of our contributions and the significant potential of this work. Below, we have addressed the main concern raised by the reviewer.

> W1. The authors chose a limited set of time series data to benchmark this model on. Many of the models they compare against have been benchmarked against other data sets: weather, traffic, electricity consumption, and the spread of influenza. a) If this work is meant to show that the model is state-of-the-art across many potential time series tasks then comparisons across other datasets (mentioned above) would be much more convincing. This work only uses ETT and M4. b) If this work is meant to benchmark this model's capabilities on a single data set then please make that clear. The state-of-the-art claims should only be made with regard to the data sets the model was benchmarked against.

+ Our primary objective in this work is extending the capabilities of large language models to practical time series forecasting within a model reprogramming space. We also mean to evidence that Time-LLM consistently outperforms or matches the state-of-the-art performance in mainstream forecasting tasks, especially in few-shot and zero-shot scenarios. We believe this falls into the category (a) mentioned by the reviewer.

+ We have performed additional experiments on four datasets: (1) Weather, (2) Electricity, (3) Traffic, and (4) Influenza-like Illness (ILI). **Our detailed results can be found in the main text (i.e., Tables 1,3, and 4) and appendices (i.e., Tables 10,11,14, and 15) of the revised paper**. For the reviewer's convenience, *a very brief summary* is provided below, highlighting the overall performance on a set of most representative methods on additionaly added four datasets.

  + A concise summary of additional long-term forecasting results in average on Weather, Electricity, Traffic, and Influenza-like Illness (ILI):

  | Method      | Time-LLM  |                     | GPT4TS |       | DLinear |       | PatchTST            |                     | TimesNet |       | FEDformer |       |
  | ----------- | --------- | ------------------- | ------ | ----- | ------- | ----- | ------------------- | ------------------- | -------- | ----- | --------- | ----- |
  | Metric      | MSE       | MAE                 | MSE    | MAE   | MSE     | MAE   | MSE                 | MAE                 | MSE      | MAE   | MSE       | MAE   |
  | Weather     | **0.225** | **0.257**           | 0.237  | 0.270 | 0.248   | 0.300 | **0.225**           | $\underline{0.264}$ | 0.259    | 0.287 | 0.309     | 0.360 |
  | Electricity | **0.158** | **0.252**           | 0.167  | 0.263 | 0.166   | 0.263 | $\underline{0.161}$ | **0.252**           | 0.192    | 0.295 | 0.214     | 0.327 |
  | Traffic     | **0.388** | $\underline{0.264}$ | 0.414  | 0.294 | 0.433   | 0.295 | $\underline{0.390}$ | **0.263**           | 0.620    | 0.336 | 0.610     | 0.376 |
  | ILI         | **1.435** | $\underline{0.801}$ | 1.925  | 0.903 | 2.169   | 1.041 | $\underline{1.443}$ | **0.797**           | 2.139    | 0.931 | 2.847     | 1.144 |

Reviewer comment (Experiments):
If the objective is to show this model gives the state-of-the-art results for the two datasets used, then that is clear. However, if the authors wish to claim this model outperforms others across many domains then more evidence is needed.

Author response:
Dear Reviewer,

Thank you for your valuable and constructive feedback, which has inspired further improvements to our paper. As a gentle reminder, it has been more than 2 days since we submitted our rebuttal. We would like to know whether our response addressed your concerns.

In accordance with your comments, we have answered your concerns and made the following revisions:

+ We have made it clear that **our primary objective** is extending the capabilities of large language models to practical time series forecasting within a model reprogramming space. We also mean to evidence that **Time-LLM consistently outperforms or matches the state-of-the-art performance in mainstream forecasting tasks**, especially in few-shot and zero-shot scenarios.

+ **We have conducted additional experiments (amounting to over 1,000 new results in total) and revised the paper accordingly**, along with the relevant discussions. You may find our responses to Reviewer Z9NA helpful. In a nutshell, the following additional experiments are incorporated:
  + We performed additional experiments on (1) Weather, (2) Electricity, (3) Traffic, and (4) Influenza-like Illness (ILI). Our detailed results can be found in the main text (Tabs. 1,3, and 4) and appendices (Tabs. 10,11,14, and 15) of the revised paper.
  + We included AutoARIMA, AutoTheta, and AutoETS as baseline methods. The detailed results can be found in Tab. 11 of the revised paper.
  + We compared to N-BEATS and N-HITS in long-term forecasting tasks. The detailed results are in Tab. 11 of the revised paper.
  + We also included the comparison with LLMTime, even though this is completely optional according to the author guide this year. The results we have incorporated can be found in Tabs. 5 and 16.
  + We have included additional experiments on M3-Quarterly for short-term forecasting due to rebuttal time constraints. Our results can be found in Table 13 of the revised paper.

We again thank you for your insightful review. We eagerly await your feedback and are ready to respond to any further questions you may have.

Reviewer comment (Evaluation):
The evidence provided, the stated intentions, and the claims of this work do not fully align. Clarification is needed on the primary objective and claims of this work, as more evidence might be required depending on this answer.

Author response:
Dear Reviewer,

Thank you for your valuable and constructive feedback, which has inspired further improvements to our paper. As a gentle reminder, it has been more than 2 days since we submitted our rebuttal. We would like to know whether our response addressed your concerns.

In accordance with your comments, we have answered your concerns and made the following revisions:

+ We have made it clear that **our primary objective** is extending the capabilities of large language models to practical time series forecasting within a model reprogramming space. We also mean to evidence that **Time-LLM consistently outperforms or matches the state-of-the-art performance in mainstream forecasting tasks**, especially in few-shot and zero-shot scenarios.

+ **We have conducted additional experiments (amounting to over 1,000 new results in total) and revised the paper accordingly**, along with the relevant discussions. You may find our responses to Reviewer Z9NA helpful. In a nutshell, the following additional experiments are incorporated:
  + We performed additional experiments on (1) Weather, (2) Electricity, (3) Traffic, and (4) Influenza-like Illness (ILI). Our detailed results can be found in the main text (Tabs. 1,3, and 4) and appendices (Tabs. 10,11,14, and 15) of the revised paper.
  + We included AutoARIMA, AutoTheta, and AutoETS as baseline methods. The detailed results can be found in Tab. 11 of the revised paper.
  + We compared to N-BEATS and N-HITS in long-term forecasting tasks. The detailed results are in Tab. 11 of the revised paper.
  + We also included the comparison with LLMTime, even though this is completely optional according to the author guide this year. The results we have incorporated can be found in Tabs. 5 and 16.
  + We have included additional experiments on M3-Quarterly for short-term forecasting due to rebuttal time constraints. Our results can be found in Table 13 of the revised paper.

We again thank you for your insightful review. We eagerly await your feedback and are ready to respond to any further questions you may have.

Reviewer comment (Experiments):
If this work is meant to show that the model is state-of-the-art across many potential time series tasks, then comparisons across other datasets (mentioned above) would be much more convincing. This work only uses ETT and M4.

Author response:
+ A concise summary of additional 10% few-shot learning results in average on Weather, Electricity, Traffic, and Influenza-like Illness (ILI):

  | Method      | Time-LLM  |                     | GPT4TS              |                     | DLinear |       | PatchTST            |           | TimesNet |       | FEDformer |       |
  | ----------- | --------- | ------------------- | ------------------- | ------------------- | ------- | ----- | ------------------- | --------- | -------- | ----- | --------- | ----- |
  | Metric      | MSE       | MAE                 | MSE                 | MAE                 | MSE     | MAE   | MSE                 | MAE       | MSE      | MAE   | MSE       | MAE   |
  | Weather     | **0.234** | **0.273**           | $\underline{0.238}$ | $\underline{0.275}$ | 0.241   | 0.283 | 0.242               | 0.279     | 0.279    | 0.301 | 0.284     | 0.324 |
  | Electricity | **0.175** | $\underline{0.270}$ | $\underline{0.176}$ | **0.269**           | 0.180   | 0.280 | 0.180               | 0.273     | 0.323    | 0.392 | 0.346     | 0.427 |
  | Traffic     | **0.429** | $\underline{0.306}$ | 0.440               | 0.310               | 0.447   | 0.313 | $\underline{0.430}$ | **0.305** | 0.951    | 0.535 | 0.663     | 0.425 |

  + A concise summary of additional 5% few-shot learning results in average on Weather, Electricity, Traffic, and Influenza-like Illness (ILI):

  | Method      | Time-LLM            |                     | GPT4TS              |                     | DLinear |       | PatchTST  |           | TimesNet |       | FEDformer |       |
  | ----------- | ------------------- | ------------------- | ------------------- | ------------------- | ------- | ----- | --------- | --------- | -------- | ----- | --------- | ----- |
  | Metric      | MSE                 | MAE                 | MSE                 | MAE                 | MSE     | MAE   | MSE       | MAE       | MSE      | MAE   | MSE       | MAE   |
  | Weather     | **0.260**           | 0.309               | $\underline{0.263}$ | **0.301**           | 0.263   | 0.308 | 0.269     | 0.303     | 0.298    | 0.318 | 0.309     | 0.353 |
  | Electricity | $\underline{0.179}$ | **0.268**           | **0.178**           | $\underline{0.273}$ | 0.176   | 0.275 | 0.181     | 0.277     | 0.402    | 0.453 | 0.266     | 0.353 |
  | Traffic     | $\underline{0.423}$ | $\underline{0.298}$ | 0.434               | 0.305               | 0.450   | 0.317 | **0.418** | **0.296** | 0.867    | 0.493 | 0.676     | 0.423 |

+ For more experimental results, **see our revised Appendix D and E**. We have also carried out a new set of short-term forecasting experiments on the M3-Quarterly benchmark, **with the results presented in Table 13 of the revised paper**. Furthermore, comparisons with extra baseline methods were performed, and **these results are detailed in Tables 11 and 16 of the revised paper**.

**We have made our best effort to include the experiments suggested by the reviewer and have revised the paper accordingly**. We trust that these modifications address the concern raised by the reviewer.

Reviewer comment (Evaluation):
If this work is meant to benchmark this model's capabilities on a single dataset then that should be made clear, and state-of-the-art claims should only be made with regard to the datasets the model was benchmarked against.

Author response:
We extend our sincere thanks to the reviewer for their constructive feedback on our paper. We are deeply grateful for the recognition of our contributions and the significant potential of this work. Below, we have addressed the main concern raised by the reviewer.

> W1. The authors chose a limited set of time series data to benchmark this model on. Many of the models they compare against have been benchmarked against other data sets: weather, traffic, electricity consumption, and the spread of influenza. a) If this work is meant to show that the model is state-of-the-art across many potential time series tasks then comparisons across other datasets (mentioned above) would be much more convincing. This work only uses ETT and M4. b) If this work is meant to benchmark this model's capabilities on a single data set then please make that clear. The state-of-the-art claims should only be made with regard to the data sets the model was benchmarked against.

+ Our primary objective in this work is extending the capabilities of large language models to practical time series forecasting within a model reprogramming space. We also mean to evidence that Time-LLM consistently outperforms or matches the state-of-the-art performance in mainstream forecasting tasks, especially in few-shot and zero-shot scenarios. We believe this falls into the category (a) mentioned by the reviewer.

+ We have performed additional experiments on four datasets: (1) Weather, (2) Electricity, (3) Traffic, and (4) Influenza-like Illness (ILI). **Our detailed results can be found in the main text (i.e., Tables 1,3, and 4) and appendices (i.e., Tables 10,11,14, and 15) of the revised paper**. For the reviewer's convenience, *a very brief summary* is provided below, highlighting the overall performance on a set of most representative methods on additionaly added four datasets.

  + A concise summary of additional long-term forecasting results in average on Weather, Electricity, Traffic, and Influenza-like Illness (ILI):

  | Method      | Time-LLM  |                     | GPT4TS |       | DLinear |       | PatchTST            |                     | TimesNet |       | FEDformer |       |
  | ----------- | --------- | ------------------- | ------ | ----- | ------- | ----- | ------------------- | ------------------- | -------- | ----- | --------- | ----- |
  | Metric      | MSE       | MAE                 | MSE    | MAE   | MSE     | MAE   | MSE                 | MAE                 | MSE      | MAE   | MSE       | MAE   |
  | Weather     | **0.225** | **0.257**           | 0.237  | 0.270 | 0.248   | 0.300 | **0.225**           | $\underline{0.264}$ | 0.259    | 0.287 | 0.309     | 0.360 |
  | Electricity | **0.158** | **0.252**           | 0.167  | 0.263 | 0.166   | 0.263 | $\underline{0.161}$ | **0.252**           | 0.192    | 0.295 | 0.214     | 0.327 |
  | Traffic     | **0.388** | $\underline{0.264}$ | 0.414  | 0.294 | 0.433   | 0.295 | $\underline{0.390}$ | **0.263**           | 0.620    | 0.336 | 0.610     | 0.376 |
  | ILI         | **1.435** | $\underline{0.801}$ | 1.925  | 0.903 | 2.169   | 1.041 | $\underline{1.443}$ | **0.797**           | 2.139    | 0.931 | 2.847     | 1.144 |

Reviewer comment (Writing):
The mechanism used to produce the next H steps remains unclear, warranting a more detailed explanation.

Author response:
> W2. The mechanism used to produce the next $H$ steps remains unclear, warranting a more detailed explanation.

+ Time-LLM does not operate in an autoregressive manner.
+ Upon packing and feedforwarding the prompt and patch embeddings $\mathbf{O}^{i}$ through the frozen LLM, we discard the prefixal part and obtain the output representations, denoted as $\tilde{\mathbf{O}}^{i}$. Subsequently, we follow PatchTST [1] and flatten $\tilde{\mathbf{O}}^{i}$ into a 1D tensor with the length $P \times D$, which is then linear projected as $\hat{\mathbf{Y}}^{i} \in \mathbb{R}^{H}$. **We have revised the paper accordingly and included this technical detail in Appendix B.1**, owing to space constraints in the main text.

+ Note that the left figure in Fig. 3(b), i.e., Patch-as-Prefix, is not in our method. This is only for a clearer illustration of our discussion related to Prompt-as-Prefix in Sec. 3.1.

[1] Nie, Y., Nguyen, N. H., Sinthong, P., & Kalagnanam, J. (2022, September). A Time Series is Worth 64 Words: Long-term Forecasting with Transformers. In The Eleventh International Conference on Learning Representations.

Reviewer comment (Presentation):
Fig 3(a): Initially, the patches seem to represent input patches $X_P$. To eliminate any confusion, clearly labeling associated variables like $E, Z$ would be helpful.

Author response:
> W3.1. Fig 3(a): Initially, the patches seem to represent input patches $X_P$. To eliminate any confusion, clearly labeling associated variables like $E$, $Z$ would be helpful.

Thank you for the suggestion. **We have appropriately revised Fig. 3(a)** and updated it with the corresponding notations and remarks.

> W3.2. Fig 3(b): The figure raises questions about whether the model outputs only the subsequent step or the next $H$ steps. Additionally, the function of the intermediate layer remains undefined.

+ Apologies for any confusion caused. As previously stated, the left figure in Fig. 3(b), representing Patch-as-Prefix, **is not part of our method**. It is included solely for a clearer explanation of our discussion on Prompt-as-Prefix in Sec. 3.1. 
+ We believe that our discussion under the part of "Prompt-as-Prefix" in Sec. 3.1 effectively illustrates: (1) the connections and differences between the two paradigms; (2) the main challenge of the Patch-as-Prefix approach in time series forecasting; and (3) the advantages of our proposed Prompt-as-Prefix method.
+ For the definition of the projection head depicted in the right side of Fig. 3(b), please refer to our response above addressing W2.

> W3.3. Error in Fig 3(b): There seems to be a need for a one-step left shift in the output of Patch-as-Prefix to render it an auto-regressive model.

This is correct, but it is important to note that: (1) the left figure in Fig. 3(b) illustrates the training process, and (2) this figure serves only for high-level illustrative purposes and is not the primary focus of our study as mentioned above. Here we adopted a widely recognized style used in [1] (as cited in our paper) to facilitate a clearer understanding of the differences between the two paradigms.

[1] Tsimpoukelli, M., Menick, J. L., Cabi, S., Eslami, S. M., Vinyals, O., & Hill, F. (2021). Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems, 34, 200-212.

Reviewer comment (Presentation):
Fig 3(b): The figure raises questions about whether the model outputs only the subsequent step or the next $H$ steps. Additionally, the function of the intermediate layer remains undefined.

Author response:
> W3.2. Fig 3(b): The figure raises questions about whether the model outputs only the subsequent step or the next $H$ steps. Additionally, the function of the intermediate layer remains undefined.

+ Apologies for any confusion caused. As previously stated, the left figure in Fig. 3(b), representing Patch-as-Prefix, **is not part of our method**. It is included solely for a clearer explanation of our discussion on Prompt-as-Prefix in Sec. 3.1. 
+ We believe that our discussion under the part of "Prompt-as-Prefix" in Sec. 3.1 effectively illustrates: (1) the connections and differences between the two paradigms; (2) the main challenge of the Patch-as-Prefix approach in time series forecasting; and (3) the advantages of our proposed Prompt-as-Prefix method.
+ For the definition of the projection head depicted in the right side of Fig. 3(b), please refer to our response above addressing W2.

Reviewer comment (Presentation):
Error in Fig 3(b): There seems to be a need for a one-step left shift in the output of Patch-as-Prefix to render it an auto-regressive model.

Author response:
> W3.3. Error in Fig 3(b): There seems to be a need for a one-step left shift in the output of Patch-as-Prefix to render it an auto-regressive model.

This is correct, but it is important to note that: (1) the left figure in Fig. 3(b) illustrates the training process, and (2) this figure serves only for high-level illustrative purposes and is not the primary focus of our study as mentioned above. Here we adopted a widely recognized style used in [1] (as cited in our paper) to facilitate a clearer understanding of the differences between the two paradigms.

[1] Tsimpoukelli, M., Menick, J. L., Cabi, S., Eslami, S. M., Vinyals, O., & Hill, F. (2021). Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems, 34, 200-212.

Reviewer comment (Experiments):
**Baselines**: I understand that the authors are following the experiment protocol followed by TimesNet, but there are several limitations: (1) Statistical methods such as AutoARIMA, AutoTHETA, AutoETS, Naive and Seasonal Naive, etc. were not compared with. These methods are important and very performant in practice, (2) N-BEATS and N-HITS were only compared during short-horizon forecasting, (3) Recent papers on using LLMs for time-series forecasting were not compared against, for e.g. LLM4TS [1] and PromptCast [2], (4) I am aware that the paper "LLMs are zero-shot forecasters" [3] only got recently published, but it would improve the experiments if the authors were able to compare with it. I should emphasize that this is completely optional.

Author response:
We express our gratitude to the reviewer for providing constructive feedback on our paper, and we greatly appreciate the acknowledgement of our contributions. We have addressed the specific concerns raised by the reviewer as detailed below.

> W1.1. Statistical methods, such as AutoARIMA, AutoTHETA, and AutoETS, were not compared with.

We have included AutoARIMA, AutoTheta, and AutoETS as baseline methods. **The detailed results can be found in Table 11 of the revised paper**. For the reviewer's convenience, we provide a summary of the relevant (averaged) results below:

| Method      | Time-LLM  |           | AutoARIMA |       | AutoTheta |       | AutoETS |       |
| ----------- | --------- | --------- | --------- | ----- | --------- | ----- | ------- | ----- |
| Metric      | MSE       | MAE       | MSE       | MAE   | MSE       | MAE   | MSE     | MAE   |
| ETTh1       | **0.408** | **0.423** | 0.952     | 0.656 | 1.319     | 0.797 | 1.286   | 0.784 |
| ETTh2       | **0.334** | **0.383** | 0.635     | 0.532 | 1.635     | 0.678 | 1.981   | 0.505 |
| ETTm1       | **0.329** | **0.372** | 1.145     | 0.697 | 1.268     | 0.741 | 1.529   | 0.790 |
| ETTm2       | **0.251** | **0.313** | 3.205     | 0.634 | 0.975     | 0.523 | 2.430   | 0.459 |
| Weather     | **0.225** | **0.257** | 1.080     | 0.448 | 0.477     | 0.373 | 1.137   | 0.406 |
| Electricity | **0.158** | **0.252** | 0.597     | 0.510 | 0.797     | 0.596 | 0.755   | 0.570 |
| Traffic     | **0.388** | **0.264** | 1.344     | 0.765 | 3.510     | 1.284 | 4.395   | 1.285 |
| ILI         | **1.435** | **0.801** | 4.530     | 1.346 | 5.232     | 1.431 | 4.323   | 1.317 |

> W1.2. N-BEATS and N-HITS were only compared during short-horizon forecasting.

We have compared to N-BEATS and N-HITS in long-term forecasting tasks, following the reviewer's suggestion. **The detailed results are in Table 11 of the revised paper**. We have provided a summary of the relevant (averaged) results below:

| Method      | Time-LLM  |           | N-HiTS |       | N-BEATS |       |
| ----------- | --------- | --------- | ------ | ----- | ------- | ----- |
| Metric      | MSE       | MAE       | MSE    | MAE   | MSE     | MAE   |
| ETTh1       | **0.408** | **0.423** | 0.473  | 0.462 | 0.568   | 0.525 |
| ETTh2       | **0.334** | **0.383** | 0.487  | 0.464 | 0.564   | 0.529 |
| ETTm1       | **0.329** | **0.372** | 0.395  | 0.412 | 0.451   | 0.445 |
| ETTm2       | **0.251** | **0.313** | 0.337  | 0.376 | 0.355   | 0.402 |
| Weather     | **0.225** | **0.257** | 0.235  | 0.286 | 0.256   | 0.306 |
| Electricity | **0.158** | **0.252** | 0.205  | 0.296 | 0.259   | 0.351 |
| Traffic     | **0.388** | **0.264** | 0.427  | 0.338 | 0.620   | 0.454 |
| ILI         | **1.435** | **0.801** | 2.997  | 1.171 | 6.839   | 1.882 |

> W1.3. Recent papers on using LLMs for time-series forecasting were not compared against for, e.g. LLM4TS and PromptCast. I am aware that the paper "LLMs are zero-shot forecasters" only got recently published, but it would improve the experiments if the authors were able to compare with it.

This is a good suggestion. Following the reviewer's recommendation, we have also included a comparison with LLMTime, even though this is completely optional according to the author guide this year. Our considerations are twofold: (1) LLM4TS is not yet open-sourced, and (2) LLMTime has shown better performance compared to PromptCast and operates directly on time series data. **The additional results we have incorporated can be found in Tables 5 and 16**. For ease of reference for the reviewer, we have summarized the relevent (averged) results below:

| Method         | Time-LLM  |           | LLMTime |       |
| -------------- | --------- | --------- | ------- | ----- |
| Metric         | MSE       | MAE       | MSE     | MAE   |
| ETTh1 to ETTh2 | **0.353** | **0.387** | 0.992   | 0.708 |
| ETTh1 to ETTm2 | **0.273** | **0.340** | 1.867   | 0.869 |
| ETTh2 to ETTh1 | **0.479** | **0.474** | 1.961   | 0.981 |
| ETTh2 to ETTm2 | **0.272** | **0.341** | 1.867   | 0.869 |
| ETTm1 to ETTh2 | **0.381** | **0.412** | 0.992   | 0.708 |
| ETTm1 to ETTm2 | **0.268** | **0.320** | 1.867   | 0.869 |
| ETTm2 to ETTh2 | **0.354** | **0.400** | 0.435   | 0.443 |
| ETTm2 to ETTm1 | **0.414** | **0.438** | 0.769   | 0.567 |

Note that to ensure a fair comparison, we have aligned the backbone LLM used in LLMTime with that of our method in terms of model size, specifically employing Llama2-7B. It should also be noted that LLMTime does not explicitly involve a transfer step but directly inferences on target datasets, resulting in some entries sharing same results. In a nutshell, Time-LLM demonstrates a substantial improvement over 75% and 53% in terms of the averged MSE and MAE.

Reviewer comment (Experiments):
**Datasets:** Increasing the amount of datasets for experimentation will improve the results. The current set of datasets is pretty limited, even for long horizon forecasting datasets, where datasets such as Influenza-like Illnesses, Exchange Rate, Tourism, and Weather etc. (see PatchTST) were missing. For short-horizon datasets, M3 at the very least, and the Monash time-series forecasting archive can be added to improve results.

Author response:
> W2.1. The current set of datasets is pretty limited, even for long horizon forecasting datasets, where datasets such as Influenza-like Illnesses, Exchange Rate, Tourism, and Weather etc. (see PatchTST) were missing.

We have performed additional experiments on four datasets: (1) Weather, (2) Electricity, (3) Traffic, and (4) Influenza-like Illness (ILI). **Our detailed results can be found in the main text (i.e., Tables 1,3, and 4) and appendices (i.e., Tables 10,11,14, and 15) of the revised paper**. For the reviewer's convenience, *a very brief summary* is provided below, highlighting the overall performance on a set of most representative methods on additionaly added four datasets.

+ A concise summary of additional long-term forecasting results in average on Weather, Electricity, Traffic, and Influenza-like Illness (ILI):

  | Method      | Time-LLM  |                     | GPT4TS |       | DLinear |       | PatchTST            |                     | TimesNet |       | FEDformer |       |
  | ----------- | --------- | ------------------- | ------ | ----- | ------- | ----- | ------------------- | ------------------- | -------- | ----- | --------- | ----- |
  | Metric      | MSE       | MAE                 | MSE    | MAE   | MSE     | MAE   | MSE                 | MAE                 | MSE      | MAE   | MSE       | MAE   |
  | Weather     | **0.225** | **0.257**           | 0.237  | 0.270 | 0.248   | 0.300 | **0.225**           | $\underline{0.264}$ | 0.259    | 0.287 | 0.309     | 0.360 |
  | Electricity | **0.158** | **0.252**           | 0.167  | 0.263 | 0.166   | 0.263 | $\underline{0.161}$ | **0.252**           | 0.192    | 0.295 | 0.214     | 0.327 |
  | Traffic     | **0.388** | $\underline{0.264}$ | 0.414  | 0.294 | 0.433   | 0.295 | $\underline{0.390}$ | **0.263**           | 0.620    | 0.336 | 0.610     | 0.376 |
  | ILI         | **1.435** | $\underline{0.801}$ | 1.925  | 0.903 | 2.169   | 1.041 | $\underline{1.443}$ | **0.797**           | 2.139    | 0.931 | 2.847     | 1.144 |

+ A concise summary of additional 10% few-shot learning results in average on Weather, Electricity, Traffic, and Influenza-like Illness (ILI):

  | Method      | Time-LLM  |                     | GPT4TS              |                     | DLinear |       | PatchTST            |           | TimesNet |       | FEDformer |       |
  | ----------- | --------- | ------------------- | ------------------- | ------------------- | ------- | ----- | ------------------- | --------- | -------- | ----- | --------- | ----- |
  | Metric      | MSE       | MAE                 | MSE                 | MAE                 | MSE     | MAE   | MSE                 | MAE       | MSE      | MAE   | MSE       | MAE   |
  | Weather     | **0.234** | **0.273**           | $\underline{0.238}$ | $\underline{0.275}$ | 0.241   | 0.283 | 0.242               | 0.279     | 0.279    | 0.301 | 0.284     | 0.324 |
  | Electricity | **0.175** | $\underline{0.270}$ | $\underline{0.176}$ | **0.269**           | 0.180   | 0.280 | 0.180               | 0.273     | 0.323    | 0.392 | 0.346     | 0.427 |
  | Traffic     | **0.429** | $\underline{0.306}$ | 0.440               | 0.310               | 0.447   | 0.313 | $\underline{0.430}$ | **0.305** | 0.951    | 0.535 | 0.663     | 0.425 |

+ A concise summary of additional 5% few-shot learning results in average on Weather, Electricity, Traffic, and Influenza-like Illness (ILI):

  | Method      | Time-LLM            |                     | GPT4TS              |                     | DLinear |       | PatchTST  |           | TimesNet |       | FEDformer |       |
  | ----------- | ------------------- | ------------------- | ------------------- | ------------------- | ------- | ----- | --------- | --------- | -------- | ----- | --------- | ----- |
  | Metric      | MSE                 | MAE                 | MSE                 | MAE                 | MSE     | MAE   | MSE       | MAE       | MSE      | MAE   | MSE       | MAE   |
  | Weather     | **0.260**           | 0.309               | $\underline{0.263}$ | **0.301**           | 0.263   | 0.308 | 0.269     | 0.303     | 0.298    | 0.318 | 0.309     | 0.353 |
  | Electricity | $\underline{0.179}$ | **0.268**           | **0.178**           | $\underline{0.273}$ | 0.176   | 0.275 | 0.181     | 0.277     | 0.402    | 0.453 | 0.266     | 0.353 |
  | Traffic     | $\underline{0.423}$ | $\underline{0.298}$ | 0.434               | 0.305               | 0.450   | 0.317 | **0.418** | **0.296** | 0.867    | 0.493 | 0.676     | 0.423 |

Reviewer comment (Theory):
While LLM is a hot topic in the deep learning community, it is still unconvincing to directly transfer the knowledge of natural language in LLMs to time series tasks. Note that i) text and time series are distinct data modalities, and ii) the pre-trained LLMs are not pre-trained with text-time-series pairs.

Author response:
> W1. It is unconvincing to directly transfer the knowledge of natural language in LLMs to time series tasks. Note that 1) text and time series are distinct data modalities, and 2) the pre-trained LLMs are not pre-trained with text-time-series pairs.

+ While LLMs are typically pre-trained on extensive text corpora, they have been proven to be effective in pattern recognition and reasoning over complex sequences of tokens in approximating flexible distributions over numbers [1,3]. This capability can be well extended to time series data, as demonstrated in this work and in recent concurrent and follow-up studies [2-6]. As an emerging and promising field of study, a recent survey [7] also offers a comprehensive overview of research that utilizes LLMs not only for time series but also for more intricate spatio-temporal data mining. This further underscores the proficiency of LLMs in understanding and reasoning on temporal data.
+ In addition to the evidence presented above, LLMs are known to encapsulate rich information and rules about the physical world through natural language modality, leading us to hypothesize that some inherent knowledge may also be relevant to understanding time series. Notable examples, such as those in references [2] and [3], have achieved a certain level of precision in zero-shot forecasting through direct prompting. Our contention is that to fully activate the LLMs' ability in time series understanding and reasoning, it is crucial to effectively align the modalities of time series and natural language, as discussed in the third paragraph of the introduction. This concept is further elaborated in the fourth paragraph of the introduction in two aspects: (1) the central idea is to reprogram the input time series into text prototype representations that are more naturally suited to language models’ capabilities; (2) to further augment the model’s reasoning about time series concepts, we introduce Prompt-as-Prefix (PaP), an innovative approach that enriches the input time series with additional context and provides task instructions in the natural language modality.
+ Our established and newly added experimental results clearly demonstrate the fact that off-the-shelf LLMs are can be very effective in general time series forecasting, whether compared to SOTA time series models or the best available (open-sourced) related models like GPT4TS and LLMTime [3]. Although LLMs are often considered black-boxes, our analysis in Fig. 5 illustrates how the proposed LLM reprogramming framework bridges the gap between natural language and time series.

[1] Mirchandani, S., Xia, F., Florence, P., Driess, D., Arenas, M. G., Rao, K., ... & Zeng, A. (2023, August). Large Language Models as General Pattern Machines. In 7th Annual Conference on Robot Learning.

[2] Xue, H., & Salim, F. D. (2022). Prompt-Based Time Series Forecasting: A New Task and Dataset. arXiv preprint arXiv:2210.08964.

[3] Gruver, N., Finzi, M. A., Qiu, S., & Wilson, A. G. (2023, November). Large Language Models Are Zero-Shot Time Series Forecasters. In Thirty-seventh Conference on Neural Information Processing Systems.

[4] Chang, C., Peng, W. C., & Chen, T. F. (2023, August). Llm4ts: Two-stage fine-tuning for time-series forecasting with pre-trained llms. arXiv preprint arXiv:2308.08469.

[5] Spathis, D., & Kawsar, F. (2023). The first step is the hardest: Pitfalls of Representing and Tokenizing Temporal Data for Large Language Models. arXiv preprint arXiv:2309.06236.

[6] Cao, D., Jia, F., Arik, S. O., Pfister, T., Zheng, Y., Ye, W., & Liu, Y. (2023, October). TEMPO: Prompt-based Generative Pre-trained Transformer for Time Series Forecasting. arXiv preprint arXiv:2310.04948.

[7] Jin, M., Wen, Q., Liang, Y., Zhang, C., Xue, S., Wang, X., ... & Xiong, H. (2023, October). Large models for time series and spatio-temporal data: A survey and outlook. arXiv preprint arXiv:2310.10196.

Reviewer comment (Novelty):
The first contribution *“introducing a novel concept of reprogramming large language models for time series forecasting without altering the pre-trained backbone model“* is not new, as previous works such as GPT4TS [1] have also explored this reprogramming approach, regardless of which parts of the LLMs are fine-tuned. Additionally, the proposed method requires training the input and output layers for adaptation, which means that it does involve some alteration of the pre-trained backbone model.

Author response:
> W2. The first contribution *“introducing a novel concept of reprogramming large language models for time series forecasting without altering the pre-trained backbone model“* is not new, as previous works such as GPT4TS have also explored this reprogramming approach, regardless of which parts of the LLMs are fine-tuned. Additionally, the proposed method requires training the input and output layers for adaptation, which means that it does involve some alteration of the pre-trained backbone model.

+ **GPT4TS does not align with the concept of model reprogramming** [1], as it involves certain levels of fine-tuning in the underlying language model. For a comprehensive understanding of model reprogramming, refer to [1,2] and Sec. 2 & Appendix A of our paper. Additionally, the specifics of GPT4TS can be found in Fig. 2 of [3]. **There is a clear factual error in the statement "...regardless of which parts of the LLMs are fine-tuned"**. To the best of our knowledge, our work is the first propose to reprogram LLMs for bridging the modality between time series and natural language, achieving high-precision in time series forecasting.
+ **We respectfully disagree with the statement that "...the proposed method requires training the input and output layers for adaptation, which means that it does involve some alteration of the pre-trained backbone model".**  To clarify, model reprogramming is characterized by **no internal modifications** to the backbone model, such as altering or fine-tuning its internal layers, as outlined in [1,2]. Please note that the input and output layers mentioned are not components of the LLMs. Hence, in the paper, we specifically mentioned "without altering the pre-trained backbone model".

[1] Chen, P. Y. (2022, Feb). Model reprogramming: Resource-efficient cross-domain machine learning. arXiv preprint arXiv:2202.10629.

[2] Yang, C. H. H., Tsai, Y. Y., & Chen, P. Y. (2021, July). Voice2series: Reprogramming acoustic models for time series classification. In International conference on machine learning (pp. 11808-11819). PMLR.

[3] Zhou, T., Niu, P., Wang, X., Sun, L., & Jin, R. (2023). One Fits All: Power General Time Series Analysis by Pretrained LM. In Thirty-seventh Conference on Neural Information Processing Systems.

Reviewer comment (Experiments):
Compared to related work such as [1], which has applied pre-trained LLMs to various time series tasks, the experiments in this paper are relatively limited. For instance, only prediction tasks are considered, and even for long-term prediction tasks, only the ETT datasets are included. This narrow scope of experiments limits the evaluation of the proposed approach to other time series tasks and datasets.

Author response:
> W4. Compared to related work such as GPT4TS, which has applied pre-trained LLMs to various time series tasks, the experiments in this paper are relatively limited. For instance, only prediction tasks are considered, and even for long-term prediction tasks, only the ETT datasets are included.

+ To clarify, the central theme and primary objective of our research is **time series forecasting** through reprogramming large language models. Although Time-LLM shows considerable promise for general time series analysis, including forecasting, classification, imputation, and more, **we have not made any claims regarding this breadth of application** in our submission. Exploring these additional applications could be a direction for future work. Accordingly, **we have updated Sec. 5 of our paper to prevent any over-interpretation of our current scope**.
+ **We have not** limited our evaluation only to the ETT datasets; we have also conducted assessments using the M4 benchmarks. In response to the reviewer's concerns regarding evaluation, we have performed the experiments on four additional datasets: (1) Weather, (2) Electricity, (3) Traffic, and (4) Influenza-like Illness (ILI). **Our detailed results can be found in the main text (i.e., Tables 1,3, and 4) and appendices (i.e., Tables 10,11,14, and 15) of the revised paper**. For the reviewer's convenience, *a very brief summary* is provided below, highlighting the overall performance on a set of most representative methods on additionaly added four datasets.
  
  + Example 1. A concise summary of additional long-term forecasting results in average on Weather, Electricity, Traffic, and Influenza-like Illness (ILI):

  | Method      | Time-LLM  |                     | GPT4TS |       | DLinear |       | PatchTST            |                     | TimesNet |       | FEDformer |       |
  | ----------- | --------- | ------------------- | ------ | ----- | ------- | ----- | ------------------- | ------------------- | -------- | ----- | --------- | ----- |
  | Metric      | MSE       | MAE                 | MSE    | MAE   | MSE     | MAE   | MSE                 | MAE                 | MSE      | MAE   | MSE       | MAE   |
  | Weather     | **0.225** | **0.257**           | 0.237  | 0.270 | 0.248   | 0.300 | **0.225**           | $\underline{0.264}$ | 0.259    | 0.287 | 0.309     | 0.360 |
  | Electricity | **0.158** | **0.252**           | 0.167  | 0.263 | 0.166   | 0.263 | $\underline{0.161}$ | **0.252**           | 0.192    | 0.295 | 0.214     | 0.327 |
  | Traffic     | **0.388** | $\underline{0.264}$ | 0.414  | 0.294 | 0.433   | 0.295 | $\underline{0.390}$ | **0.263**           | 0.620    | 0.336 | 0.610     | 0.376 |
  | ILI         | **1.435** | $\underline{0.801}$ | 1.925  | 0.903 | 2.169   | 1.041 | $\underline{1.443}$ | **0.797**           | 2.139    | 0.931 | 2.847     | 1.144 |

  + Example 2. A concise summary of additional 10% few-shot learning results in average on Weather, Electricity, Traffic, and Influenza-like Illness (ILI):

  | Method      | Time-LLM  |                     | GPT4TS              |                     | DLinear |       | PatchTST            |           | TimesNet |       | FEDformer |       |
  | ----------- | --------- | ------------------- | ------------------- | ------------------- | ------- | ----- | ------------------- | --------- | -------- | ----- | --------- | ----- |
  | Metric      | MSE       | MAE                 | MSE                 | MAE                 | MSE     | MAE   | MSE                 | MAE       | MSE      | MAE   | MSE       | MAE   |
  | Weather     | **0.234** | **0.273**           | $\underline{0.238}$ | $\underline{0.275}$ | 0.241   | 0.283 | 0.242               | 0.279     | 0.279    | 0.301 | 0.284     | 0.324 |
  | Electricity | **0.175** | $\underline{0.270}$ | $\underline{0.176}$ | **0.269**           | 0.180   | 0.280 | 0.180               | 0.273     | 0.323    | 0.392 | 0.346     | 0.427 |
  | Traffic     | **0.429** | $\underline{0.306}$ | 0.440               | 0.310               | 0.447   | 0.313 | $\underline{0.430}$ | **0.305** | 0.951    | 0.535 | 0.663     | 0.425 |
  
For more experimental results, **see our revised Appendix D and E**. We have also included an additional set of short-term forecasting experiments on M3-Quarterly benchmark, **the results can be found in Table 13 of the revised paper**.

Reviewer comment (Reproducibility):
The link for the source code provided in the paper is empty. I cannot check for more details regarding the experiments.

Author response:
> W5. The link for the source code provided in the paper is empty. I cannot check for more details regarding the experiments.

+ We explicitly stated on our anonymous GitHub page in the paper to prevent any misunderstanding: *"According to the [official author guideline](https://iclr.cc/Conferences/2024/AuthorGuide), we will make our code visible only to the reviewers and area chairs for our submission to protect the intellectual property. Specifically, after the discussion forum is opened, we will make a comment directed to the reviewers and area chairs and put a link to an anonymous repository containing our code."*
+ **We ensured that our code was accessible to the Area Chair and all reviewers by posting an anonymous GitHub link in a comment on OpenReview.**