P ROMPT C AST : A N EW P ROMPT - BASED L EARNING P ARADIGM FOR T IME S ERIES F ORECASTING
Anonymous authors
Paper under double-blind review
A BSTRACT
This paper studies the time series forecasting problem from a whole new perspective.
In the existing SOTA time-series representation learning methods, the forecasting models take a sequence of numerical values as input and yield numerical values as output.
The existing SOTA models are largely based on Transformer architecture, modified with multiple encoding mechanisms to incorporate the context and semantics around the historical data.
In this paper, we approach representation learning of time-series from the paradigm of prompt-based natural language modeling.
Inspired by the successes of pre-trained language foundation models, we pose a question about whether these models can also be adapted to solve timeseries forecasting.
Thus, we propose a new forecasting paradigm: prompt-based time series forecasting (PromptCast).
In this novel task, the numerical input and output are transformed into prompts.
We frame the forecasting task in a sentenceto-sentence manner which makes it possible to directly apply language models for forecasting purposes.
To support and facilitate the research of this task, we also present a large-scale dataset (PISA) that includes three real-world forecasting scenarios.
We evaluate different SOTA numerical-based forecasting methods and language generation models such as Bart.
The benchmark results with singleand multi-step forecasting settings demonstrate that the proposed prompt-based time series forecasting with language generation models is a promising research direction.
In addition, in comparison to conventional numerical-based forecasting,
PromptCast shows a much better generalization ability under the zero-shot setting.
We believe that the proposed PromptCast task as well as our PISA dataset could provide novel insights and further lead to new research directions in the domain of time-series representation learning and forecasting.
I NTRODUCTION
Time series forecasting is a research-intensive field, especially with the increasing of applying various deep learning frameworks for prediction such as models based on LSTM (Hochreiter & Schmidhuber, 1997), Temporal Convolutional Network (TCN) (
Lea et al., 2017), and Transformer (Vaswani et al., 2017).
More recently, we are witnessing a fast growth of large-scale pre-trained models in Natural
Language Processing (NLP) field.
These models, also known as foundation models (Bommasani et al., 2021), are often pre-trained with an extremely large amount of data and have demonstrated good performance across various downstream tasks.
For example, BERT (Devlin et al., 2019) can be adapted for multiple NLP tasks, CLIP (Radford et al., 2021) and GLIP (Li et al., 2021) are good at CV tasks.
However, we also notice that this evolution seems mostly limited to the NLP and CV fields.
Hence, we are particularly interested in exploring the research question of whether we can take the advantages of large-scale pre-trained foundation models and adapt these models for predicting time series.
To investigate this question, in this paper, we formally introduce a novel task: prompt-based time series forecasting (PromptCast).
The existing forecasting methods including the state-of-the-art Transformer-based forecasting models (Li et al., 2019; Zhou et al., 2021; Liu et al., 2021; Xu et al., 2021;
Zhou et al., 2022; Drouin et al., 2022) can be simplified as a numerical forecasting paradigm as shown in Figure 1 (a).
Numerical forecasting methods always take numerical values as input and generate numerical values as the prediction for the next time step.
Instead, the input and output of the proposed prompt-based forecasting (Figure 1 (b)) are natural language sentences.
This paradigm change enables the utilization of language generation models for forecasting.
76 degrees tomorrow
The temperature were 77, 68, 66, 73 degrees in past 4 days.
Prompting
Language Model
Fine-tuning
Testing/Inference
The temperature were 68, 66, 73, 76 degrees in past 4 days.
76 Prompting
Thank you for the information
No worries!
How many people will visit our shop tomorrow?
There will be 88 vistors.
(c) Potential application: Forecasting Chatbot (
[Figure]
) PromptCast Framework (a) Numerical Forecasting (e.g., Transformer-based)
Encoder
N x a tt en t i on
Decoder
M x 76 . . .
66  time-of-day . . .
a tt en t i on
This new forecasting paradigm is beneficial in multiple aspects.
PromptCast presents a novel “code less” solution for time series forecasting, which could provide a new perspective rather than purely focusing on designing more and more complicated deep learning forecasting models ( e.g. ,
Transformer-based Informer, Autoformer, and FEDformer).
It also becomes a relatively easyaccessible and user-friendly method for non-researcher users, compared to existing forecasting models that require many tedious parameter searching and training processes, especially under a new forecasting scenario.
As pointed out in a recent research (Reed et al., 2022), the benefits of using a single neural model across different tasks are significant.
The PromptCast task explores the potential of using language foundation models for the forecasting task which could make it possible to broaden the language models beyond the realm of typical text-based tasks.
In addition, it could inspire new research directions and new applications to better serve society.
For example, as illustrated in Figure(c), chatbot with forecasting ability is one of the prospective future applications driven by the research of PromptCast task.
Currently, although AI-powered intelligent assistants or chatbots like Siri and
Alexa can answer queries about general topics, they still fail to answer specific time series forecasting questions.
With the help of PromptCast related research, they would be able to yield predictions based on the given contexts.
To the best of our knowledge, this is the first effort in approaching representation learning of general time-series using language-based paradigm, resulting also in the first large-scale dataset, PISA ( P rompt based t I me S eries forec A sting), tailored for the task of prompt-based time series forecasting.
It covers three real-world forecasting scenarios: weather temperature forecasting, energy consumption forecasting, and customer flow forecasting.
We believe that the release of this dataset will not only support the research of the PromptCast task but also have a great potential to stimulate the related research in the time series analysis domain.
We further develop a benchmark in which we report the forecasting performance of multiple methods including both numerical-based forecasting methods and language generation models.
To evaluate the generalization ability of PromptCast, this benchmark also explores various forecasting settings such as train-from-scratch, zero-shot prediction, multi-step forecasting and multivariate forecasting 1 .
In summary, our contributions are three-fold: (1) We propose a novel prompt-based forecasting paradigm, which differs from the existing forecasting methods.
This is the first time that the general time series forecasting problem is addressed in a natural language generation manner.
(2) We release a large-scale dataset (PISA) with 311,932 data instances in total for the newly introduced task.
The dataset covers diverse time series forecasting scenarios.
(3) We develop a benchmark 2 on the proposed PISA datasets.
It evaluates the state-ofthe-art numerical-based forecasting methods, and popular language generation models under the PromptCast setting.
P ROMPT -B ASED T IME S ERIES F ORECASTING
The prompt-based time series forecasting task is developed from the general time series task.
Here we first describe the general numerical-based forecasting and then formulate the proposed PromptCast task.
Let U =
{ U 1 , U 2 , · · · , U M } denotes a set of M objects-of-interest .
Depending on different specific forecasting scenarios, the objects-of-interest could stand for different objects.
For example, the objects could be places-of-interest (POI) such as bars and parks in human mobility forecasting (Xue et al., 2022) or cities in weather forecasting.
Under the general numerical time series forecasting task setting, the input is a history records of interested numerical data points collected on n continuous time steps ( e.g. , daily data): x mt 1 : t obs =
[ x mt 1 , x mt 2 , · · · , x mt obs | x mt ] , where x mt represents the value of object-of-interest U m observed on time step t .
The forecasting target (output) is the numerical data value x mt obs
+1 of the next time step t obs +1 3 .
The overarching goal of the PromptCast task is to leverage language foundation models to forecast time series in a sentence-to-sentence fashion.
In order to achieve this goal, based on the above formulated problem of the numerical time series forecasting, the numerical values need to be transferred and described as natural language sentences.
This data-to-text transformation is referred as a prompting process in this work (the details of prompting are presented in the next section).
Specifically, the input numerical sequence x m t 1 : t obs is turned into input prompts and the forecasting target value x mt obs +1 is transformed as the output prompt.
Consequently, the time series forecasting can be addressed through a natural language generation paradigm, and language foundation models can be adopted as the core forecasting models in PromptCast task.
D ATASET D ESIGN AND D ESCRIPTION
In this section, we demonstrate the design and construction of the proposed PISA dataset.
The overall designing guideline is: (1) to preprocess original data given in the numerical format (raw data) for the forecasting task setting (Sec. 3.1) and (2) to transform the numerical data to natural language input/output formats with prompts (Sec. 3.2).
We also describe the features and statistics (Sec. 3.3).
3. D ATA S OURCES AND P ROCESSING
To establish a diverse dataset, we consider three real-world time series forecasting scenarios ( sub-sets of our PISA dataset) from different domains: weather forecasting, energy consumption forecasting, and human mobility forecasting.
The data sources of these scenarios are:
• City Temperature (CT) 4 : This data source provides the daily average temperature (in Fahrenheit degrees) of multiple cities globally.
110 international cities are randomly selected to form the dataset.
• Electricity Consumption Load (ECL) :
The original dataset 5 includes the electricity consumption values (in Kwh) of 321 users.
We filtered users with missing values and randomly selected 50 users with full records of the entire data collection period.
In addition, the hourly consumption values of each selected user are aggregated into daily consumption data.
• SafeGraph Human Mobility Data (SG) :
This real-world human mobility data from SafeGraph Weekly Patterns 6 contains the daily raw counts of visitors to POIs.
We expanded the data collection from 5 months in Xue et al.
to almost 15 months and then randomly selected 324 POIs with full records.
The exact data collection periods are reported in Table 1.
Following the standard protocol (Xu et al., 2021; Xue et al., 2022), each sub-set is divided into train/val/test at the ratio of 7:1:2 by the chronological order (Table 1).
The numerical sequence of each object-of-interest in each sub-set is then split into multiple instances (for training/validation/test) by applying sliding windows.
The window size equals to t obs + 1 (including t obs time steps as input historical data and 1 step as the forecasting target) and the step size of the sliding window is 1 day.
Specifically, following previous
[Table]
work (Xue et al., 2022), the observation length of the input sequence is set as 15 ( i
[Table]
e. , t obs = 15 ).
To distinguish the numerical data used for numerical methods and the language-based dataset processed for language models, the numerical sequences processed by the above sliding window is referred as
PISA-numerical whereas the other is named as PISA-prompt (details given in next subsection).
Ethical Considerations.
The only possible sensitive information is the identifier of the object-ofinterest ( i
[Table]
e. , the city name in CT and the POI id in SG).
To remove this, we randomly assigned object-of-interest index U m starting from 1 to M .
Since the original data source of each sub-set is an aggregate statistics with no personal identifiable information, no private information can be decoded from the generated PISA dataset.
3. T EMPLATE -B ASED P ROMPTING
The core of the proposed PromptCast task is shaping the time series forecasting in a language generation manner.
To serve this purpose, a key step in building a dataset for PromptCast is to describe and transform the numerical sequential data ( i.e. , PISA-numerical) to natural language sentences.
As demonstrated in Xue et al.
(2022), using template-based description is an effective and efficient approach to achieve the data-to-text transformation.
In this work, we explicitly introduce three templates for the three sub-sets and Table 2 lists the templates and the corresponding examples.
In a nutshell, the template consists of two main parts: input prompt and output prompt.
The input prompt covers the description of the historical observation and the indicators of the prediction target time step ( i.e. , t obs +1 ).
The output prompt handles the desired prediction value ( x m t obs +1 ) which is used as the ground truth label for training or evaluation.
This input/output prompt setting is similar to the source/target sentence in machine translation.
For researchers who are more familiar with the open question answering setting, our PISA dataset can also be interpreted as a question answering task setting.
The input prompt can be broken into the context part and the question part.
The context provides the historical information for forecasting and the question part can be seen as the input query about the future.
Naturally, the output prompt is the ground truth answer that responds the question.
Based on the templates and the processed numerical sequences, PISA-prompt is then generated.
Note that the instances in PISA-numerical map the instances in PISA-prompt.
For example, the first instance in PISA-prompt is transferred from the first instance in PISA-numerical.
This is to ensure that they can be used to compare the performance of numerical forecasting methods and the language-based forecasting methods in the benchmark.
Our PISA-prompt provides the input prompts and the corresponding output prompts in separate files ( e.g. , val x prompt.txt and val y prompt.txt).
3. S TATISTICS O VERVIEW
To highlight the diversity of PISA, we analyze its key statistics as given in Table 1.
PISA dataset contains 311,932 instances in total obtained from three different forecasting application domains.
[Table]
Each sub-set has its own statistical characteristics.
The last row of Table 1 lists the value range distributions and the average value of each sub-set.
It is noticeable that CT involves negative values in the dataset and ECL includes large numbers with a large range.
This diverse data ensures the representativeness of our PISA dataset.
B ENCHMARK
In this section, we present the benchmarking study and analysis for the proposed PromptCast task.
Through the experiments on the established PISA dataset, we aim to answer two main research questions: RQ1 : Can we use language generation models to predict time series under the PromptCast task setting?
Compared to the conventional numerical-based time-series forecasting methods, what is the performance of language generation-based forecasting models?
RQ2 : Can forecasting time series with prompts as well as using language generation models achieve better generalization ability?
4. E VALUATION M ETRICS
Although the proposed PromptCast task is a language generation task aiming at generating the target output prompts, we are particularly interested in the time series forecasting performance.
To this end, the first step of the evaluation protocol is to decode the numerical predicted values from the generated sentences.
Given that the output prompts follow the same template for each sub-set ( e.g. , “There will be ...”
in the SG sub-set), the numerical value can be easily extracted by simple string parsing.
However, in practice, due to the uncertainty of the inference process, it cannot guarantee that the numerical value can be decoded from the generated output for every testing instance.
To reflect this in the evaluation, we explicitly introduce Missing Rate as one evaluation metric.
It is defined as ( n test − n decoded ) /n test × 100% where n test and n decoded are the total number of instances in the test set and the number of generated instances that can successfully decode the predicted value, respectively.
A smaller Missing Rate means a better performance.
[Table]
After decoding the numerical predicted value
[Table]
the evaluation of PromptCast task will be the same as the evaluation of traditional numerical-based forecasting methods.
As a result, two widely used metrics are also selected to evaluate the prediction performance in our benchmark: the Root Mean
Square Error (RMSE) and the Mean Absolute Error (MAE).
For each deep learning method, we report the average performance and the standard deviation of 5 runnings with different random seeds.
Note that the Missing Rate is naturally ignored when evaluating the numerical-based methods.
4. B ASELINES
In order to provide a useful benchmark of the proposed PromptCast task for other researchers, we select 10 popular natural language generation models and test their performance on our PISA dataset ( i.e. , PISA-prompt).
These language models are T5 (Raffel et al., 2020), Bart (Lewis et al., 2020),
BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), Electra (Clark et al., 2020), Bigbird (Zaheer et al., 2020), ProphetNet (Qi et al., 2020), LED (Beltagy et al., 2020), Blenderbot (Roller et al., 2021), and Pegasus (Zhang et al., 2020).
Furthermore, for the comparison purpose (RQ1) and providing strong baselines for forecasting with prompts methods, we also include the performance of conventional numerical paradigm forecasting methods on the PISA-numerical.
We consider 3 naive forecasting methods: Copy Yesterday (CY), Historical Average (HA), and Copy Last
Week (CLW).
Other three basic numerical forecasting methods are included: AutoARIMA, LSTM, temporal convolutional network (TCN).
Transformer-based time-series forecasting methods including the vanilla Transformer (Vaswani et al., 2017), the state-of-the-art Informer (Zhou et al., 2021),
Autoformer (Xu et al., 2021), and FEDformer (Zhou et al., 2022) are also considered in this part.
Overall, 20 different methods (10 numerical methods and 10 language models) are considered in the current benchmark and we will also expand the benchmark continuously in the future.
4. E XPERIMENTAL P ERFORMANCE 4.3. N UMERICAL -B ASED M ETHODS
This part focuses on evaluating the typical numerical-based forecasting methods with our PISA dataset.
Normally, the position embeddings used in the Transformer architecture (and its variants) only contain the limited position information ( e.g. , the first time step in each input sequence).
This kind of position information remains the same for all different input data instances.
However, for time series data, temporal information ( e.g. , day-of-week and month-of-year ) is an important cue for predicting future data and reflects the global position relations.
For example, the first time step of instance A could correspond to Monday whereas the first time step (same position) of instance B could be Friday.
Thus, appending the temporal embeddings to the basic position position embedding becomes popular
[Table]
[Table]
in Transformer-based time series forecasting methods.
This is equivalent to providing temporal information in the input prompt context ( i.e. , From { t 1 } to { t obs } in Table 2).
Specifically, based on the implementations of Informer (Zhou et al., 2021) 7 and Autoformer (Xu et al., 2021) 8 , we fully investigate and benchmark three different temporal embedding approaches, namely, timeF , fixed , and learned .
The details of these three embeddings are given in Supplementary Materials .
Table 3 presents the performance of different methods with different temporal embedding policies.
In general, FEDformer, Informer and Autoformer achieve the best performance across different sub-sets.
On most of cases, these advanced time series forecasting frameworks outperform the vanilla
Transformer, naive methods, and non-Transformer methods.
Naive methods demonstrate worse forecasting performance compared to other methods, which is as expected.
As for the comparison of different embeddings, the fixed embedding demonstrates an overall good performance.
This embedding leads to good predictions on 5 out of 6 metrics and the timeF is the best performer of the rest metric (MAE on SG).
The learned embedding has the worst performance on CT and SG, whereas it beats the timeF on the ECL sub-set.
The above results show that the fixed embedding is a favorable embedding approach for incorporating the temporal cues.
4.3. P RE - TRAINED L ANGUAGE M ODELS
For language models investigated in the benchmark, the ready-to-use pre-trained weights provided by
HuggingFace (Wolf et al., 2020) 9 are used for initialization.
It is worth noting that the pre-trained weights are trained with general English-language corpora datasets such as BookCorpus (Zhu et al., 2015), CC-News (Liu et al., 2019), and OpenWebText (Radford et al., 2019).
These common language datasets are about general articles and do not include specific time series sequences orientated data.
Although three data sources in PISA are open available, only the original numerical data (in the csv format) can be acquired online instead of the text format with our prompts.
This also guarantees that the data from these three datasets are not used in pre-training and prevents the potential data leakage.
In the experiments, each language model is fine-tuned with the training set of each sub-set in PISA.
The prediction results (RMSE and MAE) of using different language generation models on PISA dataset are listed in Table 4.
According to the table, the top performers (shown in bold) include
Bigbird, Bart, and RoBERTa.
Bigbird achieves the best performance on 4 out of 6 metrics.
When we jointly consider Table 3 and Table 4, it can be seen that using language models perform reasonably well on the CT and ECL sub-sets.
For ECL, although the MAE of using language models is slightly worse than the best performer in Table 3, the RMSE has a relatively large improvement.
Compared against numerical methods, using language models can also yield comparable results on SG. This
The links of pre-trained weights are reported in Table 7 in Supplementary Materials .
benchmark answers RQ1 and indicates that prompt-based forecasting with language models is a promising direction for the time series forecasting research.
Table 5 reports the Missing Rate metric.
It is clearly noticeable that only three methods (ProphetNet,
Electra
[Table]
and BERT) have a tiny amount (less than 0.5%) of missing cases and all cases appear on the CT sub-set.
For other methods that are not listed in Table 5 , the missing rates are all 0.
We further investigate the output sentences that cannot be decoded and find out that the failure cases are related and potentially caused by negative values.
For example, the failure generated sentences are like the temperature will be - - - - where the models fail to generate the tokens after “-”.
Our PISA dataset is valuable in supporting the research directions to address this limitation in the future.
For more qualitative examples, we provide all generated examples of language models in the anonymized
GitHub repository 10 including all the three test sets with various settings such as with pre-trained weights, train-from-scratch, zero-shot, prompt ablations, and multi-step forecasting. [Table]
Zero-shot Performance.
To further explore the language models for prompt-based forecasting [Table] we conduct an experiment under the zero-shot setting (see Table 6). Specifically
[Table] we fine-tune each method on two sub-sets and test the fine-tuned model on the test set of the left sub-set ( e [Table] g. , fine-tune with the training sets of CT and ECL, test on the test set of SG).
For the comparison purpose, we also evaluate the Transformer-based numerical methods under the same zero-shot setting and results are given in Table 6 (the first two parts).
Similar to the results given in Table 3 (the normal training setting) [Table] the fixed embedding has better performance (5 best performance out of 6 metrics) than the other two embeddings under this challenging zero-shot setting.
Since the date range of three sub-sets are different (Table 1), learnable learned and timeF temporal embeddings would result in worse performance when transferred to unseen scenarios.
Except for the Autoformer and FEDformer on
ECL, the numerical-based methods fail to yield reasonable predictions under the zero-shot setting.
Considering the different characteristics of three sub-sets, such a poor performance is as expected for numerical methods.
However, for the language models, although the performance is worse than the normal setting and the train-from-scratch setting, prompt-based forecasting can still generate sensible predictions.
It shows a good generalization ability when time series forecasting is addressed with prompts (RQ2).
This good zero-shot ability could bring benefits in real-world forecasting applications such as fast deployment for new forecasting scenarios and cold start forecasting for scenarios without any existing historical data.
In the future, prompt-based forecasting could also support the exploration of more complicated forecasting scenarios such as can we predict energy consumption based on the weather temperature or can we forecast the customer visiting flow with the temperature trend.
Train-from-Scratch.
Moreover, we disable the pre-trained weights and train language models from scratch with the training set of PISA dataset.
We select three language models that demonstrate good forecasting performance on all three sub-sets for investigation in this part of experiments: Bart,
Pegasus, and Bigbird.
The results are reported in the last part of Table 6.
We observe that there is a performance reduction for each method without loading pre-trained weights.
Another finding is that even training from scratch, the language generation models (especially Bart) still can yield comparable prediction results compared to numerical-based methods.
The above results show that using pre-trained weights could contribute to the forecasting performance gain.
This also reveals one of the advantages of prompt-based forecasting: pre-trained language models can be easily leveraged.
D ISCUSSION AND C ONCLUSION
In this paper, we present a novel task, PromptCast, which predicts time series using language models in a language generation manner.
Given that this is the first work towards PromptCast task and no prior datasets are suitable, we build the first dataset aimed at investigating prompt-based forecasting.
This large-scale dataset (PISA) contains three real-world time series forecasting scenarios.
To advance the research of PromptCast, we also develop a benchmark on the released dataset and provides a set of solid baselines including both state-of-the-art numerical forecasting methods and language generation models.
The experimental results show that using language models under the PromptCast setting has good forecasting performance and generalization ability.
Why Language Model Works for Forecasting.
Using auxiliary information such as time-of-day , day-of-week (Zhou et al., 2021) and semantic information (Xue et al., 2021) has been demonstrated beneficial for boosting the forecasting performance.
It often requires extra efforts to explicitly design special layers or modules in a heuristic way for the auxiliary inputs Although some approaches are proposed to incorporate auxiliary information, it remains challenging for a forecasting model to seamlessly model the contexts and the main temporal information.
While for PromptCast, after prompting, both auxiliary and the core temporal inputs are considered together as tokens.
The intrarelation of numerical value tokens at different time steps and the inter-relation between numerical value tokens and contextual information tokens ( e.g. , date information) could be better learned simultaneously by language foundation models ( e.g. , through the self-attention mechanisms in
Transformers).
The modeling of such relations would then result in good forecasting performance 11 .
Broader Impact.
We believe that this study’s findings would offer forward-thinking concepts and fresh insights for other researchers.
The research of time series forecasting could contribute in solving problems such as climate change and resources allocation for social good.
We also think that the proposed PromptCast task, as well as the PISA dataset, could open new related research directions and provide visionary ideas about downstream applications empowered by this work.
Some potential directions are discussed below.
(1) Automatic Prompting :
In this paper, the transformation for numerical data to text is achieved by templates.
Although template-based prompting is efficient, it still is difficult to produce diverse prompts.
The fixed templates might also cause biases towards templates.
To this end, one research direction is the automatic time series prompting or time series captioning (similar to image captioning (Herdade et al., 2019)) that aims at using generative models to describe the time series data.
(2) Explainable PromptCast : Another research question yet to be fully investigated is why models designed for language modeling tasks can predict time series?
A newly research has started to explore whether language models can be used for non-language downstream tasks (Dinh et al., 2022).
However, future studies in the interpretability and explainability of PromptCast models would be an interesting and valuable research direction.
(3) PromptCast
QA and Chatbots : The research of PromptCast would trigger and promote time series forecasting question answering tasks and building chatbots applications with forecasting ability.
Note that the
PromptCast QA task differs from the recent TimeQA (Chen et al., 2021) that is proposed to answer general time related questions based on Wikipedia text and ForecastQA (Jin et al., 2021) which is also based on text articles.
The core of PromptCast QA task is about the question-answering ability about forecasting upon the given sequential numerical value contexts.