From: ICLR.cc/2023/Conference/Paper1474/Authors
title: Submission Withdrawn by the Authors

From: ICLR.cc/2023/Conference/Paper1474/Reviewer_XVfc
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper uses pre-trained language models in time series forecasting, such as temperature prediction for the next day. It created three time-series forecasting tasks: City Temperature, Electricity Consumption Load, and SafeGraph Human Mobility Data. The real-valued time series data are converted to text with a fixed pattern in natural language, then used as the input and output for training/testing the language models. Its experiments measure the squared error of the prediction, showing that the language models can achieve similar accuracy to baselines.
strength_and_weaknesses: - Weaknesses: 
1. Using pre-trained language models for time series forecasting is not a sound research setting. The combination of setting used in this paper will not happen in a real-world use case. 
2. The three cases are too simple for time series prediction. They have reoccurring patterns based on the day of the week or the month of the year.
clarity,_quality,_novelty_and_reproducibility: - Clarity: ok
- Quality: ok
- Technical novelty: None
- Reproducibility: It looks reproducible.
summary_of_the_review: The paper's finding is "good to know". The paper is well-written and may be worth sharing in workshops with its datasets, but its scientific and technical contributions do not fit a conference paper.

(Post-rebuttal) All rebuttals were read, and the evaluation was updated.
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 1: The contributions are neither significant nor novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper1474/Reviewer_AUpk
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper studies the time series forecasting problem with pre-trained language models in NLP, e.g. BERT and T5. It first transforms time series into texts, and cast time series forecasting as a question-answering task. Extensive results across multiple datasets show that it can obtain on-par results compared with recent numerical state-of-the-art results.
strength_and_weaknesses: Strength

* The idea of casting time series forecasting task into question-answering task in NLP is quite innovative and simple. 
* Extensive experiments across three datasets show that it can be comparable to recent state-of-the-art numerical time series forecasting methods.

Weaknesses
* Experiments are mainly focusing on univariate single-step forecasting, lacking of extensive comparisons on multiple-step and multiple variable settings. Numerical time series forecasting can forecast hundreds of steps ahead and deal with high-dimensional data. Language model based models cannot deal with this issue trivially.
* Hyper-parameters of numerical time series forecasting models are not detailed in this paper. This is important as hyper-parameters often have huge impacts on model performance.
* Three datasets discusses in this paper lack of diversity. What if numerical values are very small/large/long/even negative? I guess these factors can play an important role on language model based methods while numerical models can handle it easily by preprocessing. 
* Another concerns of language model performance is that language models have been pre-trained on these datasets since these datasets are online publicly available. 
* Have authors considered how to add features for real production usage, e.g. discount event?

Missing reference:

[1] Li et al. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. NeurIPS 2019

[2] Liu et al. Pyraformer: Low-Complexity Pyramidal Attention for Long-Range Time Series Modeling and Forecasting. ICLR 2022


clarity,_quality,_novelty_and_reproducibility: The idea is innovative although its experiments have flaws (discussed in Weakness). Authors provide source code for their methods, which I expect to make reproduction pretty easy.
summary_of_the_review: This paper studies the time series forecasting problem with pre-trained language models in NLP. Extensive results across multiple datasets show that it can obtain on-par results compared with recent numerical state-of-the-art results in univariate single step forecasting case. However, its experimental designs have some flaws, possible data leakage in pre-training stage and lack of comparisons in multiple variate and multiple step forecasting scenarios, making this method less applicable is real production usage.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper1474/Reviewer_ax1T
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The authors reframe the problem of time series forecasting in terms of prompted generation from a large language model (LLM). A short context window $x_{t-k+1:t}$ is parsed as a natural language sentence providing context for the recent history of the series and appended to a question prompting the LLM for the next value $x_{t+1}$. Results for step-ahead forecasting of scalar time series are shown for three datasets and compared both to traditional forecasting methods and across a variety of language models for the prompt-based approach. 
strength_and_weaknesses: Strengths:
- This paper explores a creative idea for forecasting based on a somewhat unexpected connection to conditional generation in LLMs. This idea at least implicitly poses some bold hypotheses that could lead to new insights or research directions. 
- The results in terms of forecasting accuracy are surprisingly competitive given how unusual the approach is from a time series perspective, and they seem good enough to warrant further investigation.

Weaknesses:
- There is relatively little effort made to understand whether these results are robust or to explain how such a surprising result might be possible. The prediction setting is very simple: the size of the context window is fixed to $k=15$, far short of the long contexts often considered in prediction tasks. The form of the prompt is not justified or investigated beyond a contrast with a minimal string sequence input in the supplement. Most importantly, there is almost no attempt to explain why this unusual strategy for time series prediction is reasonably successful. This is a missed opportunity, especially when some hypotheses (such as the one raised in the discussion) could be investigated with some relatively simple experiments.

- Some key methodological details are missing. What algorithm is used for training or fine-tuning? What decoding method is used for generation from the LLMs? What modifications (if any) to the loss are made to account for the numeric value of the tokens and the magnitude of the errors during fine-tuning?

- The authors claim their dataset ("PISA", though this acronym is never defined) as a contribution, but this seems to be minor overall. The size of the data (~300k total observations) is not large for modern time series problems. All three tasks come from previously-available data sources with minor additional curation or parsing.


clarity,_quality,_novelty_and_reproducibility: The idea of the paper is relatively clear, though as mentioned above some methodological details are missing. The datasets are described clearly.

The paper's idea is novel in that it proposes an unexpected (at least from the time series perspective) methodology for short-term forecasting via conditional generation from language models. The paper does meet its full potential in terms of quality as it does not fully establish the robustness of this result and it makes almost no effort to propose or investigate hypotheses as to why the result is observed.

Based on the paper, supplementary material, and code provided I feel reasonably confident that a motivated reader could reproduce the reported experiments.
summary_of_the_review: Overall, the lack of experimentation and investigation surrounding a surprising, potentially quite interesting result on time series prediction with LLMs limits this paper's impact.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper1474/Reviewer_hRSz
confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
summary_of_the_paper: This paper provides a new perspective of time series forecasting by transforming numerical time series into textual prompts in a question answering manner. Large-scale pre-trained language models are then applied to the QA task where the future values to be predicted is asked in the context of past observed values. Compared to state-of-the-art forecasting models based on transformer architectures, language models pre-trained on massive corpus are expected to benefit from even larger training datasets and capability of easy adaptation by fine-tuning. Comparison of numerical time series models and prompt-based language models shows the new framework can produce competitive results on a new dataset combining several benchmark datasets and better generalizability in the zero-shot setting.
strength_and_weaknesses: Strengths:
1. This work provides a novel and interesting perspective for time series forecasting tasks, where numerical sequences are turned into text-based Q&A by template filling so that pre-trained language models that have been proven effective in question answering tasks can be introduced. The idea is quite inspiring, and great potential can be expected in this direction.
2. This work provides a large-scale datasets with prompts on time series to facilitate further research, which makes the idea reproducible and extendable.
3. Extensive experiments are carried out to show the proposed method is competitive with or without pre-training and show generalizablity under the zero-shot setting, making a comprehensive and convincing demonstration.

Weakness :
1. The details of numerical tokens in the prompt is not clear. In table 2, all numerical values in original time series are all integers, which is generally not the case for most time series datasets. When it comes to floating point numbers, it is unclear how vocabularies of pre-trained language models can accommodate all possible values.
2. The considered baselines consider long time series forecasting as a major challenge. The setting in the paper fixes $t_{obs}$ to be 15, and yet the performance and efficiency of PromptCast on long sequences is unclear. In addition, how the performance changes with the time series lengths is an important problem to investigate.
3. One question I have is that is there any regularizations on the model output so that the slots for predicted values will always be numerical? In multi-step forecasting, is the number of prediction determined by the output prompt template in model outputs?
4. (Minor) Results of table 3-7 should be somehow aggregated for easier comparison in reading.
clarity,_quality,_novelty_and_reproducibility: Clarity: the presentation of the methodology and empirical studies is clear.
Quality/novelty: the idea is normal and the investigation is thorough and meaningful.
Reproducibility: reasonable.
summary_of_the_review: Overall I recommend acceptance of this paper due to the novel idea and a high-quality setup for a new task and framework.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
empirical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper