A UTOMATIC F ORECASTING VIA M ETA -L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
In this work, we develop techniques for fast automatic selection of the best forecasting model for a new unseen time-series dataset, without having to ﬁrst train (or evaluate) all the models on the new time-series data to select the best one.
In particular, we develop a forecasting meta-learning approach called
A UTO F ORECAST that allows for the quick inference of the best time-series forecasting model for an unseen dataset.
Our approach learns both forecasting models performances over time horizon of same dataset and task similarity across different datasets.
The experiments demonstrate the effectiveness of the approach over state-of-the-art (SOTA) single and ensemble methods and several SOTA meta-learners (adapted to our problem) in terms of selecting better forecasting models (i.e., 2X gain) for unseen tasks for univariate and multivariate testbeds.
I NTRODUCTION
Accurate time-series forecasting at scale is critical for several industrial domains such as cloud computing Poghosyan et al.
supply chain Abbasimehr et al.
(2020), energy Abdallah et al.
(2021), and ﬁnance Oreshkin et al.
Most of the current time-series forecasting solutions are built by experts and require signiﬁcant manual effort in model construction, feature engineering, and hyper-parameter tuning Bergstra et al.
Hence, they do not scale while generating high-quality forecasts for a wide variety of applications.
Moreover, there is no learning scheme that is uniformly better than all other learning schemes for all problem instances Wolpert (1996) (see Appendix A).
A na ¨ ıve approach would be, given a new dataset, we search over thousands of models to select the best forecasting model for the problem at hand.
However, this approach is practically infeasible due to the untenable time burden for every new problem.
In this work, we formulate the problem of automatic and fast selection of the best time-series forecasting model as a meta-learning problem.
Our solution avoids the infeasible burden of ﬁrst training each of the models and then evaluating each one to select the best model for a new unseen time-series dataset, or even a new time window within a non-stationary dataset.
A practically important desideratum for any solution to this problem is that once the meta-learner L is trained in an ofﬂine manner using a large corpus of time-series data, then we can use it to quickly infer the best forecasting model.
The quick inference requirement of this new problem, makes it challenging to solve, yet practically important.
Our meta-learner L is trained on the models’ performances on historical datasets and the time-series meta-features of these datasets.
We emphasize that the new time-series forecasting model selection meta-learning problem introduced in this paper has several unique characteristics and challenges compared to previous related meta-learning problems, e.g., Finn et al.
(2017); Rusu et al.
; Wistuba et al.
First, existing time-series forecasting models have different designs and different assumptions around the characteristics of time-series (e.g., probabilistic, seasonal, traditional, etc.).
Therefore, different models perform differently depending on the characteristics that each dataset exhibits.
Thus, capturing the similarity among different datasets needs careful selection of representative time-series meta-features.
Second, the new meta-learning approach should capture the temporal variations of the models’ performances over different time windows of the dataset, i.e., the best time-series forecasting model for time window w t is not necessarily the best model for a subsequent time window w t
+ k
(see Figures 5-6 in Appendix B).
Third, the number of available time-series forecasting models is large and thus training each forecasting model and then evaluating the suitability of each in inference leads to an unacceptable time burden for most real-world scenarios.
These challenges motivate the need for our proposed approach.
To solve the problem of automatic time-series forecasting model selection, we propose a temporal meta-learning approach, called A UTO F ORECAST that makes time-series forecasting model selection.
The intuition of our approach is learning both models’ performance evolution over time horizon for same dataset (via our “temporal meta-learner”) and task similarity across different datasets (via our “general meta-learner” and proposing novel time-series meta-features that capture their characteristics).
In particular, we train our meta-learner using a large model space which has over 320 forecasting models (Section 5.1).
We also generate more than 800 meta-features that represent ﬁve different types of meta-features which are (simple, statistical, information theoretic, spectral-based, and landmarker), which reﬂect various characteristics of the time-series dataset (Section 3).
We also consider diverse datasets so our meta-learning model can be generalized on new tasks (Table 1) and release the corpus of datasets, along with their meta-features and the performances across hundreds of models.
Given a new (unseen) dataset, A UTO F ORECAST automatically determines the best forecasting model among a large space of models, without the need to train and evaluate any of the different forecasting models on this new dataset.
The experiments demonstrate the effectiveness of our proposed approach where we validate our meta-learning approach on both univariate and multivariate testbeds.
In particular, we show the superiority of our approach over the state-of-the-art (SOTA) time series forecasting models Salinas et al.
; Wang et al.
; Taylor & Letham (2018); Montero-Manso et al.
Liaw et al.
(2002) and different meta-learning approaches Kadioglu et al.
; Nikoli ´ c et al. (2013); Zhao et al.
Across all datasets, A UTO F ORECAST achieves 2X gain in selecting the best forecasting model.
Moreover, A UTO F ORECAST yields a signiﬁcant reduction in inference time over the na ¨ ıve approach, i.e., across all datasets, A UTO F ORECAST has a 42X median inference time reduction.
We ﬁnd empirically that no single forecasting model triumphs in more than 0.7% of the datasets, thus motivating the need for automatic model selection for new time-series datasets.
Summary of Main Contributions.
The key contributions of this work are as follows:
Problem Formulation : We propose a novel meta-learning approach that predicts the best model for new (unseen) forecasting tasks where each model is composed of time-series speciﬁc parameters along with traditional meta-learning model components.
Temporal Learning of Performances : We propose a time-series meta-learner that learns the models’ performances evolution over time windows of the datasets.
Our meta-learner has two main sub-learners — the time-series (temporal) meta-learner and the general meta-learner that are designed for different data types with different time dependencies.
Specialized Meta-features for Time-series Forecasting : We design novel time-series landmarker meta-features to capture the unique characteristics of a time-series dataset toward effectively quantifying task similarity.
Efﬁciency and Effectiveness :
Given a new time-series dataset, A UTO F ORECAST selects the best performing forecasting algorithm and its associated hyperparameters without requiring any model evaluations, while incurring negligible run-time overhead.
Through extensive experiments on our benchmark testbeds, we show that selecting a model by A UTO F ORECAST outperforms SOTA meta-learners and popular forecasting models.
Benchmark Data: We release our meta-learning database corpus (348 datasets), performances of the 322 forecasting models, and meta-features for the community to access it for forecasting model selection and to build on it with new datasets and models.
R ELATED W ORK
Meta-learning in Time-series Forecasting: There are few works that considered meta-learning for time-series analysis Hooshmand & Sharma (2019);
Ribeiro et al.
(2018); Pan et al.
The works Hooshmand & Sharma (2019); Ribeiro et al.
(2018) applied a neural network time-series forecasting model trained on a source (energy) dataset and ﬁne-tuned it on the target (energy) dataset.
However, these works did not consider using meta-learning for the general problem of forecasting model selection that we consider in our current work.
There also exist few works that have explored model selection problem using ensemble learning Ye & Dai (2018);
Talagala et al.
; Vaiciukynas et al.
However, their problem domain of ensemble learning is different from our problem of model selection.
That is due to the fact that ensemble learning constitutes building multiple models for the same task and does not in itself involve learning from prior experience on other tasks.
In contrast to those works, A UTO F ORECAST can select among any (heterogeneous) set of methods.
Finally, there is a line of work that considered empirical analysis for performance estimation Bergmeir & Ben´ıtez (2012);
Cerqueira et al.
and
model selection Cerqueira et al.
(2021) in time-series forecasting.
However, these works have several distinctions from our work which are the need of evaluating all forecasting models in inference and providing an analysis of the ranking ability of performance estimators without having a meta-learner that learns how to automatically select the best model.
Few-shot Learning & Transfer Learning: Few-shot learning has been recently leveraged for automating machine learning pipeline Ravi & Larochelle (2016);
Snell et al.
(2017); Oreshkin et al.
In particular, the works Ravi & Larochelle (2016); Snell et al.
(2017) investigated different problems outside the domain of time-series forecasting.
The work Oreshkin et al.
(2020) applied meta-learning for zero-shot univariate time series forecasting.
However, that work has the limitations of focusing on solving the cold start problem (i.e., learning model parameter initialization that generalizes better to similar tasks) which is different from our forecasting model selection problem, considering different models from the same N-BEATS architecture Oreshkin et al.
and tackling only univariate time-series datasets.
We emphasize that our framework can use N-BEATS as one forecasting algorithm in our model space.
Finally, there exist few works that applied transfer learning for time series classiﬁcation (TSC)
Fawaz et al.
; Abdallah et al.
; Wen & Keyes (2019); Narwariya et al. (2020).
These works however have two distinctions from our work.
First, they transfer the learned network’s weights to another network that is also trained on a target dataset.
Second, the TSC problem is different from our forecasting problem that we consider here.
P ROBLEM F ORMULATION
Meta-learning Components: We address the problem of model selection for time-series forecasting via the meta-learning approach.
Our proposed meta-learner A UTO F ORECAST depends on:
• A collection of historical time-series forecasting datasets D train
=
{ D 1 , D 2 , · · · , D n } , namely, a meta-train database, where n is the number of the historical datasets in D train .
Note that D i ∈ R n i × v i , where n i is the number of observations of the dataset D i and v i is the number of variables in D i .
• The forecasting models that deﬁne the model space, denoted as M = { M 1 , M 2 , · · · , M m } , where m is the size of the model space M .
• For each dataset D
i ∈ D train , we sample T windows from D i , where each sample window w t from dataset D i has length | w t | ( smaller than the dataset length); See Appenidx C.1. Model Design :
For our forecasting model selection problem, we deﬁne our model as follows.
Deﬁnition 1.
A model M i ∈ M is given by the tuple M i = ( a i , h i , g i ( · )) , where a i is the forecasting algorithm, h i is the hyperparameter vector for the forecasting algorithm a i , and g i ( · ) : R n i × v i → R n i × v
i is the time-series data representation.
We emphasize that h i consists of hyper-parameters of the forecasting algorithm
(e.g., number of RNN layers in DeepAR Salinas et al.
and that g i ( · ) represents time-series different representations (e.g., the smoothing transformation Kalekar et al.
; see Table 9 in Appendix F).
Using the deﬁned meta-train database, the model space, and the sampling windows, we now introduce the performance tensor.
Deﬁnition 2.
Given a meta-train database D train and a model space M , we deﬁne the performance tensor P ∈ R T × n × m as
[Equation] where P k = ( p i,jk ) ∈ R n × m and the element p i,jk = M j ( w k ( D i )) denotes the j th model M j ’s performance on the time window w k of the i th meta-train dataset D i .
We thus denote p ik = (cid:2) p i, 1 k · · · p i,mk (cid:3) as the performance vector of all models in M on time window w k of D i .
The performance tensor represents the prior experience that the meta-learner will build upon to perform efﬁciently on the new unseen task (time-series).
This motivates us to deﬁne our problem.
Deﬁnition 3.
Time-series forecasting model selection problem .
Given a new input task (dataset)
D test (i.e., unseen time-series forecasting task), the time-series forecasting model selection problem is then stated as follows: for each time window w t in D test , select the best model ˆ M t ∈ M to employ on that window of D test .
Formally, such selection problem is given by
[Equation]
Time-series Meta-Features:
A key component of A UTO F ORECAST is the extraction of meta-features that aims to capture the important characteristics of a time-series dataset.
To achieve such a goal, we extract meta-features for each time-series dataset which we deﬁne formally next.
Deﬁnition 4.
Given a time-series dataset D i , we deﬁne the meta-features tensor F i =
{ F i 1 , · · · , F iT } ∈ R T ×
d × v i , where the meta-features matrix
F ik ∈ R d × v i denotes the set of the meta features for the time window w k of the dataset D i , given by
[Equation]
where ψ ( · ) : R | w i |× v i → R d × v i deﬁnes feature extraction module in A UTO F ORECAST and d denotes the number of the meta-features.
We summarize our notations in Table 7 (Appendix C).
Meta-Features Categories:
The set of meta features in our work that capture the main characteristics of a dataset can be organized into ﬁve categories Vanschoren (2018): simple (general task properties), statistical (properties of the underlying dataset distributions), information-theoretic (entropy measures), spectral (frequency domain properties), and landmarker (forecasting models’ attributes on the task) features.
The idea of the proposed landmarker features is to apply a few of the fast, easy-to-construct time-series forecasting models on a dataset and extract features from (i) the structure of the estimated forecasting model, and (ii) its output performance scores.
The complete meta-features list in A UTO F ORECAST are explained in Appendix E (Table 8).
A UTO F ORECAST S OLUTION
A UTO F ORECAST consists of two-phases: ofﬂine training of the meta-learner and online inference that aims at selecting the appropriate model at test time.
We argue that running time of the ofﬂine training phase is not critical since it is done only once.
On the contrary, forecasting model selection for a new time-series dataset should incur small run-time overhead since it is critical for quick selection of the forecasting model.
We now explain our meta-learning approach and its components.
4. M ETA -L EARNING O BJECTIVE AND (O FFLINE ) T RAINING
We show the overview of the major components of A UTO F ORECAST in Figure 1.
We highlight the components transferred from ofﬂine to online stage (model selection) in blue; namely, meta-feature extractors ψ , feature embedding, time-series meta-learner Θ , and general meta-learner Φ .
The meta-learner L has three main inputs; the performance tensor P , the meta-features tensor F , and the loss function.
In the ofﬂine training of the meta-learner L , it learns two components Θ and Φ , where the former captures the temporal relationship between the meta-features of the consecutive time windows within the same dataset and the evolution of the performances of the models on these windows (i.e., the best candidate models on those time-windows).
On the other hand, the general meta-learner Φ predicts the best model for each task (window) without taking into account the temporal relationship among different time windows within the same dataset.
Rationale for Having both General and Time-series Meta-learners: The rationale of having both meta-learners is the fact that the temporal dependency among different time windows depends on the dataset type.
Some datasets have strong temporal dependency which would be predicted efﬁciently by the time-series meta-learner Θ while others datasets would have weak temporal dependence among different windows performances in which the general meta-learner Φ is expected to perform better.
We show the results of such different datasets for our two testbeds in Table 12-13.
[Figure]
General Meta-learner Φ : We propose multi-output regression model for training our general meta-learner Φ in Figure 1.
From running all the models in M on different time windows w t with t ∈ { 1 , . . .
, T } for all datasets in the meta-train database D train , we collect a set of N = T × n distinct training samples
( F it , p it ) , with t ∈ [1 , T ] and i ∈ [1 , n ] .
Recall that F it ∈
R
d × v i is the meta-feature matrix of the window w t of dataset D
i ∈ D train and that p it
∈
R |M| (performance vector on window w t of D i ).
Thus, the multi-output regression model is given by
[Equation]
where Φ denotes the regression function (e.g., linear, NN) and β are the unknown regression parameters.
Thus, the general meta-learner’s objective, denoted by the loss function L Φ , is given by
[Equation]
where L is the loss metric (e.g., MSE, MAPE, etc).
Therefore, Φ learns the mapping between the meta-features of a time window in a dataset and the corresponding best model in the model space.
LSTM-based Time-series Meta-learner Θ : The goal of the time-series meta-learner Θ (in
Figure 1) is to learn how the models’ performances evolve with the time-series meta-feature matrices over time.
For such purpose, we propose time-series multi-regression model to learn such performance evolution.
For any dataset D i , given the time-series meta-feature matrices
F i 1 , . . .
, F it − 1 , F it and the history of the performance vectors p i 1 , . . .
, p it − 1 , we aim to predict performance vector p it of current time window w t .
The time-series regression equation would be
[Equation]
where Θ denotes the time-series regression function, and ˆ p it is the predicted performance vector.
We adapt long-short term memory (LSTM) inputs for our time-series meta-learner Θ .
We denote X t as the input at the time window w t which is given by X t = (cid:2)
F i 1 , p i 1 , F i 2 , p i 2 , · · · , F it − 1 , p it − 1 , F it (cid:3) .
The predicted LSTM’s output denoted by ˆ p it is a function of X t ; detailed equations of such relation between ˆ p it and X t in the LSTM are provided in Appendix D. During training, Θ learns the parameters W f , b f , W l , b l , W o , b o , W u , b u , W v , b v which are the weights and biases of the forget, input, and output layers and cell updates, respectively.
Thus, the objective of the LSTM time-series meta-learner Θ , denoted by L Θ , would be given by
[Equation]
We emphasize that Θ learns temporal relationships between the history of the meta-features and performance vectors over time windows and the corresponding current best model in model space.
Meta-learner Objective : Having established the two main components — the general meta-learner Φ and the time-series meta-learner Θ ) — we now deﬁne the objective of our meta-learner L , which is given by a linear combination of the two components as follows:
[Equation]
The meta-learner L learns jointly general meta-learner Φ (equation 4) and time-series meta-learner Θ (equation 5) given meta-learner inputs, the performance tensor P and the meta-features tensor F .
By deﬁnition, this meta-learner L optimizes the loss over all datasets and all time windows.
4. O NLINE I NFERENCE A ND M ODEL S ELECTION
In the online mode of A UTO F ORECAST , we aim to make use of the trained meta-leaner L to quickly infer the best model for the current task.
Given a new time-series dataset D test ,
A UTO F ORECAST ﬁrst computes the corresponding meta-features tensor by ˆ F test =
ψ ( D test ) .
Those time-series meta-features are then embedded (using PCA) to obtain the ﬁnal meta-features tensor F test .
Then, in the model inference, as shown in Figure 1, the model set performances are predicted for each available model in M .
The model ˆ M t with the lowest predicted error score by L on the time window w t of D test is chosen as the selected model for that window w t .
Now, we explain such model selection process for each time window across the time windows w 0 , w 1 , . .
. , w T of D test as follows.
For the ﬁrst window ( w 0 ), the inference is given by ˆ M 0 ∈ arg min ¯ M ∈M L ( F test) .
For any other window w t (with t > 0 ), the time-series meta-learner Θ inference depends on the history of the meta-features and the history of the models’ performances as follows ˆ M Θ t ∈ arg min ¯ M ∈M Θ ( F test, . .
. , F testt − 1 , F testt , ˆ p
[Equation]
, ˆ p testt − 1 ) .
On the other hand, the general meta-learner Φ inference depends on the predicted (regression) output on the meta-features of current time window where ˆ M Φ t ∈ arg min ¯ M ∈M Φ ( F testt ) .
Thus, the ﬁnal selected model is given by
[Equation]
We emphasize that tie between models can happen in online inference (i.e., two or more models can have an identical predicted performance).
We built upon the several tie breaking techniques that have been examined in the literature Kalousis (2002); Brazdil et al.
(2008), but usually such a choice does not have a strong inﬂuence on the performance of a meta-learning system.
For making the decision between the model selected by the general meta-learner Φ and that selected by the time-series meta-learner Θ , we choose the model with the least predicted error score (illustrated in equation 7).
We present the time complexity analysis of A UTO F ORECAST ’s inference in Appendix H.
E XPERIMENTS
We evaluate A UTO F ORECAST by designing experiments to answer the following research questions:
• Does employing A UTO F ORECAST for time-series forecasting model selection yield improved performance, as compared to no model selection, as well as other selection techniques (such as meta-learners adapted from the AutoML domain)?
• How much reduction in inference time does A UTO F ORECAST give over the na¨ıve method?
• How does performance change with different datasets with different temporal dependence?
5. E XPERIMENTAL S ETUP
Models and Performance Collection:
By pairing seven SOTA time-series forecasting algorithms (which are DeepAR Salinas et al.
(2020), Deep Factors Wang et al.
(2019), Prophet Taylor & Letham (2018), Seasonal Naive Montero-Manso et al.
(2020), Gaussian Process Yan et al.
Vector
Auto Regression Lewis & Reinsel (1985), and Random Forest Regressor Liaw et al.
and their corresponding hyperparameters, and using different data representation methods, we compose a model set M with 322 unique models (see Table 9 for the complete list).
For our testbeds, we ﬁrst generate the performance tensor P , by evaluating the models from M against the benchmark datasets in each testbed.
For consistency, all models are built using the GluonTS Alexandrov et al.
[Table]
(2020), Scikit-learn Pedregosa et al.
(2011), and Statsmodels Seabold & Perktold (2010)
Python libraries on an Intel i7 @2.60 GHz, 16GB RAM, 8-core workstation.
Meta-Train Testbed: Meta-learning works if the new task can leverage prior knowledge.
We thus create two testbeds with 348 public forecasting datasets.
In particular, we collect 308 univariate time-series datasets for the ﬁrst testbed (Table 10 in Appendix G) and 40 multivariate datasets with 317 time-series for the second testbed (Table 11 in Appendix G).
In total, we have 625 time-series in our testbeds.
We refer to Table 1 for a summary of the statistics of the corpus of time-series data we used.
For each dataset in the testbeds, we use different time windows selected randomly from the dataset, where each time window has a length of 16 (i.e., | w t | = 16 ∀ t ∈ { 1 , . .
, T } ).
We emphasize that our approach has no restriction on the length of the time window.
Evaluation: For evaluating A UTO F ORECAST , we split each testbed into 5 folds for cross-validation.
For each fold, after training the meta-learning approach, we use it to infer the best forecasting model for the new unseen test datasets.
Finally, we take the average performance of these ﬁve folds.
We mainly compare the Hit-at- k accuracy (which indicates whether the selected model is within the actual top- k models), of A UTO F ORECAST against different meta-learners baselines.
We also compare the performance of selected model by A UTO F ORECAST against the performance of the selected model by other meta-learners.
This is measured by the mean square error (MSE) and the average rank (for each testbed, the are ranked by the corresponding forecasting
MSE under the selected model for each dataset and then the rank is averaged across all datasets).
Time-series Meta-learner Setup:
For our time-series meta-learner Θ explained in Section 4, we used LSTM with 4 layers where each layer has 50 units.
The training was with 50 epochs with the Adam optimizer with a batch size of 25 and dropout rate of 0.2 to prevent over-ﬁtting.
The evaluation of the effect of such parameters on Θ ’s performance is shown in Appendix J.3.
5. B ASELINES
We adapt leading methods from algorithm selection, and include additional baselines by creating a variant of the proposed A UTO F ORECAST (marked with TSL).
All the baselines can be organized into the following categories (the detailed description of each baseline is given in Appendix I).
No model selection : This category always employs either the same single model or the ensemble of all the models, e.g., Random Forest (RF) Liaw et al.
Simple meta-learners :
Meta-learners in this category pick the generally well-performing forecasting model, globally or locally (via clustering or KNN): Global Best (GB) , ISAC Kadioglu et al.
(2010), and ARGOSMART (AS) Nikoli ´
c et al.
Optimization-based meta-learners :
These
Meta-learners learn task similarities via optimizing performance estimates on meta-features:
Multi-layer Perceptron (MLP) , and
AUTOFORECAST-TSL ; a variant in which meta-learner L consists only of time-series learner Θ .
5. R ESULTS
Now, we show our results to answer the aforementioned evaluation questions.
5.3. U NIVARIATE T ESTBED R ESULTS
To investigate the impact of the train/test similarity on meta-learning performance, we build the univariate testbed that consists of 308 datasets (Appendix G Table 10) with diverse datastes.
Superiority of A UTO F ORECAST compared to all baseline methods w.r.t.
the Hit-at- k , average rank, and MSE:
The different results are provided in Table 2-4 where the best result for every testbed is highlighted in bold.
We observe that A UTO F ORECAST outperforms previous SOTA meta-learning methods adapted to our problem.
For example, A UTO F ORECAST has 79.20%, 171.86%, 423.17%, 375.61%, and 51.59% higher Hit-at- 10 accuracy than GB, AS , ISAC, MLP, and A UTO F ORECAST -TSL, respectively.
Full evaluation of univariate testbed is in Appendix J.
[Table] [Table]
lower MSE)
compared to baseline meta-learners.
[Table]
Statistical Signiﬁcance of A UTO F ORECAST : To compare two methods statistically, we use the pairwise Wilcoxon rank test on performances (i.e., MSE of selected models) across datasets (signiﬁcance level p < 0 . 05 ).
Table 5 shows that A UTO F ORECAST is signiﬁcantly better than most of the baseline meta-learners, i.e., including GB ( 9 . 07 × 10 − 5 ), AS {+( 1 . 
+}
× 10 − 37 ) and A UTO F ORECAST -TSL ( 8 . 16 × 10 − 15 ). Appendix J.2 has full statistical signiﬁcance tests.
Meta-learners perform better than methods without model selection:
As Table 14 (Appendix J) shows, the meta-learners outperforms almost all models with no model selection.
In particular, three meta-learners (Global Best, ISAC, A UTO F ORECAST ) signiﬁcantly outperform the baseline time-series forecasting models.
For instance, A UTO F ORECAST respectively has 92.58%, 84.39%, 88.20%, 87.14%, 83.48%, 98.45%, and 95.75% lower MSE over Seasonal Naive, DeepAR, Deep
Factors, Random Forest, Prophet, Gaussian Process, and VAR, respectively.
These results signify the beneﬁts of using meta-learning for model selection, speciﬁcally using A UTO F ORECAST .
Optimization-based meta learners generally perform better than simple meta learners:
Two of the top-3 meta learners by average rank and MSE (A UTO F ORECAST and A UTO F ORECAST -TSL) are all optimization-based and signiﬁcantly outperform simple meta-learners such as ISAC and AS as shown in Table 2 and Table 4.
The interpretation is that simple meta-learners weigh meta-features equally for task similarity, whereas optimization-based methods learn which meta-features matter (e.g., time-series regression on meta-features in A UTO F ORECAST -TSL), leading to better results.
Dataset-wise Performance:
We present the detailed performances for each dataset by comparing
A UTO F ORECAST with all baseline methods in Table 12.
It is noted these results are averaged across the different time windows for each dataset.
The results shows that A UTO F ORECAST achieves the best MSE and average rank among all meta-learners.
We note that A UTO F ORECAST and
A UTO F ORECAST -TSL have same performance for datasets with higher temporal dependency.
5.3. M ULTIVARIATE T ESTBED R ESULTS
In this testbed, we simulate the case when there are similar meta-train tasks to the test task, where we choose variables from the multivariate for training and the rest in testing.
We build the multivariate testbed that consists of 40 datasets (Table 11 in Appendix G).
For the Multivariate testbed, A UTO F ORECAST still outperforms all baseline methods w.r.t.
average rank, MSE, and Hit-at- k accuracy as shown in Tables 2-4.
Moreover, Figure(Appendix J) shows that for the pool of multivariate datasets (across all time windows),
A UTO F ORECAST gives a gain of 2X in selecting better models as compared to other meta-learning baselines.
Dataset-wise performance for the datasets in the multivariate testbed is shown in Tablein Appendix J. A UTO F ORECAST has the lowest average MSE on most of the multivariate datasets.
[Figure]
A UTO F ORECAST takes less than 1.7 sec for most datasets (median = 1.65).
[Figure]
A UTO F ORECAST gives a median
[Figure]
[Table]
Statistical Signiﬁcance of A UTO F ORECAST :
Table 5 shows that for the Multivariate testbed,
A UTO F ORECAST is also signiﬁcantly better than most of the baseline meta-learners, i.e., including
AS {+( 3 . 
 × 10 − 7 ),+} MLP ( 0 . 0004 ) and A UTO F ORECAST -TSL ( 0 . 0025 )
and has no signiﬁcance from GB and ISAC.
For the full statistical signiﬁcance results for all pairs see Table 15 in Appendix J.2.
5.3. R UNTIME A NALYSIS
Inference run time statistics of A UTO F ORECAST :
Figure 2 shows that A UTO F ORECAST (meta-feature generation and model selection) takes less than 1.7 seconds on most time series datasets.
Moreover, Figure 3 shows that A UTO F ORECAST has signiﬁcant reduction in inference time compared to the na¨ıve approach, median is 42X on both testbeds (see also Appendix J.5).
Comparing A UTO F ORECAST with baselines:
In terms of inference, Table 6 shows that most of the meta-learners are fast taking only a few seconds to infer the best forecasting model.
Finally, we compare the training cost of A UTO F ORECAST against baseline meta-learners.
Table 16 shows that
A UTO F ORECAST has comparable computational training cost.
While the training process is ofﬂine and done only once and hence, not as important as the inference time cost, this experiment reassures us that our better model selection performance does not entail a prohibitive training cost.
C ONCLUSION
We introduced a meta-learning approach to automate the process of time-series forecasting by automatically inferring the best time-series model on an unseen dataset, without needing exhaustive evaluation of all existing models on this dataset.
The problem arises because there are many possible forecasting models with their associated hyperparameters, and different choices are optimal for different datasets.
Further, even within one non-stationary dataset, different models are appropriate for different time windows.
Our proposed solution A UTO F ORECAST is a meta-learner, trained on an extensive pool of historical time-series forecasting datasets and models.
To effectively capture task similarity, we designed novel problem-speciﬁc meta-features.
Extensive experiments on two large testbeds showed that A UTO F ORECAST signiﬁcantly improves time-series forecasting model selection over directly using some of the most popular models as well as several SOTA meta-learners.
We showed that A UTO F ORECAST gives a signiﬁcant improvement in the inference time compared to na¨ıve approaches.
We release the benchmark data for the community to contribute new datasets and models to stimulate further advances on automating time-series forecasting.
Reproducibility Statement: To further research into the important problem introduced in our work, we have publicly released our benchmark data to to enable others reproduce our work.
In particular, we are publicly releasing, with this submission, our meta-learning database corpus of 348 datasets, containing 625 time series in all, performances of the 322 forecasting models, and meta-features for the datasets.
This resource will hopefully encourage the community to standardize efforts at benchmarking time series forecasting model selection.
We also encourage the community to expand this resource by contributing their new datasets and models.
The anonymized website with our database is: https://drive.google.com/drive/ folders/1K1w1Ida5Cr15b5Fhidax-i-fNpWZjvet .
The details of each dataset in the two testbeds, univariate (308 datasets) and multivariate (40 datasets) are explained in Appendix G (Table 10 and Table 11) and the complete list of meta-features is presented in Appendix E (Table 8).
The performance collection is difﬁcult and time-consuming to obtain, yet fundamental for researchers to begin studying the important problem formulated in our work, and developing better approaches to solve it.
Here, we give the result of that time consuming process of collecting the performance of each of the 322 forecasting models on all the datasets (in the above link).
In particular, we collect this for the performance metrics of MSE, training time, and inference time.
This serves as the training data and ground truth evaluation data for A UTO F ORECAST and all other competing protocols.
We refer the reader to Appendix F in which we provide complete description of the algorithms, libraries, and hyper-parameter values used for this collection process.
A PPENDIX A V ALIDATION OF NO - SINGLE - MODEL H YPOTHESIS ON OUR TESTBEDS
Figure 4 shows a validation of the hypothesis that there is no unique single model that works well on all datasets, where we show a histogram of the best models probability distribution across the datasets of the training testbed where different datasets have different (best) models.
That motivates the need for our framework, A UTO F ORECAST , for automating model selection via meta-learning.
[Figure]
B V ARIATION OF BEST MODEL ACROSS TIME WINDOWS
Figure 5 validate the no-free lunch theorem across different datasets (vertically) and across different time windows within the same dataset (Horizontally).
Figure 6 shows the aggregate statistics on all datasets of the univariate testbed.
This contradicts the claims that one forecasting algorithm can work best for different datasets and motivates the need for an effective approach for learning such both dimensions for selecting the best model for new tasks, which we propose in our current work
[Figure]
C S UMMARY OF N OTATION
We summarize the notation used in this paper in Table 7.
C. W INDOW N OTATION IN A UTO F ORECAST
We now reemphasize the notation of time window we used in Section 4.
In A UTO F ORECAST , the time window represents a sequence of time observations in the time series.
In particular, w t denotes the time window index, and | w t | is the length of that time window (e.g., | w 10 | = 16 means that the 10th time window of the dataset has a length of 16 observations).
Notice that for each time window w t , we have the pair ( x t −| w t | : t − 1 , y t : t + H ) , where the different models are trained on x t −| w t | : t − 1 and
[Figure] [Figure] evaluated on y t : t + H , where H is the forecasting horizon. For example, for single-step forecasting on a window w t with length 16, we would train the forecasting model on the ﬁrst 15 observation points of the window and do forecasting on the 16th observation.
Second, we emphasize that we can train a forecasting model on windows with any length
[Figure]
but the forecasting horizon may be limited by the forecasting model parameters.
D A DDITIONAL D ETAILS OF T IME -S ERIES M ETA - LEARNER
We now detail the equations of our LSTM-based time-series meta-learner Θ .
The LSTM cell at time t has two recurrent features, denoted by h it and
c it , called the hidden state and the cell state, respectively.
The LSTM cell consists of three layers (forget gate layer, input gate layer, and output gate layer).
The activation of those layers is given by
[Equation]
[Equation]
where W f , W l , W o and b f , b l , b o ∈ R m denote the weights matrices and the biases of the three layers, respectively.
These are the parameters to be learned during the training of the time-series meta-learner.
The forget gate f it controls how much of the current cell state we should forget, the input gate l it controls how much of the cell update is added to the cell state, and the output gate o i controls how much of the modiﬁed cell state should leave the cell and become the next hidden state.
Moreover, the cell update u it is constructed with a tanh activation function as follows.
[Equation]
where W u and b u ∈ R m are further weight parameters to be learned.
Thus, the new cell and hidden states at time t are given by
[Equation]
Finally, the output equations of the LSTM cell are given by
[Equation]
) where W v and b v ∈ R m are weight parameters to be learned.
This gives the relationship between input X t and predicted performances output ˆ p it .
For ease of notation, we considered a single layer in the above analysis .
However, we have used multiple layers with several units (see Section 5).
E T IME - SERIES M ETA F EATURES
There are prior works that generated standard time-series features Franceschi et al.
(2019), tsfresh Christ et al.
(that we used for generating part of our meta-features).
We now provide details of our meta-features and their different categories.
E. M ETA - FEATURES C ATEGORIES :
For each dataset, we generate a meta-feature vector that consists of more than 800 meta-features.
This meta-features vector includes different components that capture various properties of the time-series dataset, e.g. , the statistical features (number of crossings, count of observations above/below the mean value, quantiles, etc.), the data trend (linear, non-linear, dynamics, etc.), data interdependence (lag autocorrelation, difference features, etc.), information-theoretic features that are typically based on entropy measures, and the frequency (FFT, wavelet, etc.).
Our meta-features vector also includes landmarker features, which are problem-speciﬁc, and aim to capture the unique characteristics of a dataset.
The idea is to apply a few of the fast, easy-to-construct time-series forecasting models on a dataset and extract features from (i) the structure of the estimated forecasting model, and (ii) its output performance scores.
Multivariate meta-features: For this time-series dataset type, we have two types (explained in
Appendix G).
For both types, we generate a single meta-features vector of the multivariate dataset by averaging the meta-features vectors of all variables within the dataset.
E. C OMPLETE L IST OF F EATURES
We summarize the meta-features used by A UTO F ORECAST in Table 8.
When applicable, we provide the formula for computing the meta-feature(s) and corresponding variants we used for generating the meta-features, and the corresponding number of features.
Some are based on Vanschoren (2018).
Speciﬁcally, our meta-features can be categorized into (1) simple features, (2) statistical features, (3) information-theoretic features, (4) Spectral features, and (5) landmarker features.
Broadly speaking, the statistical features captures statistical properties of the underlying data distributions; e.g., min, max, variance, skewness, covariance, etc. of the features and feature combinations.
The information-theoretic features capture information-theoretic underlying characteristics in the time-series; e.g., entropy, trend, non-linearity, change statistics, etc.
Most of those meta-features have been commonly used in the AutoML literature Vanschoren (2018).
To the best of our knowledge, we emphasize that our landmarker meta-features (detailed below) are novel and that some components of the spectral meta-features have not been used in any related work.
E. L ANDMARKER M ETA -F EATURE G ENERATION
In addition to simple, statistical, and information-theoretic meta-features, we use three time-series forecasting landmarker algorithms for computing forecasting-speciﬁc landmarker meta-features,
Auto Regression Lewis & Reinsel (1985), Random Forest Liaw et al.
and Bayesian Ridge
Regression Shi et al.
(2016) to capture landmarker characteristics of a time-series dataset.
We now provide a quick overview of each algorithm and then discuss how we are using them for building meta-features.
The algorithms are executed with the default parameter.
Auto Regression Lewis & Reinsel (1985) :
In this landamarker feature, we ﬁts the unconditional maximum likelihood of an autoregressive process.
The k parameter represents the maximum lag of the process
[Equation]
Then, we extract the coefﬁcients ϕ i whose index i ∈ { 0 , · · · , k } .
Random Forest Liaw et al.
(2002) : is a tree-based ensemble method that builds a collection of base trees using the subsampled unlabeled data, splitting on (randomly selected) features as nodes.
Random Forest grows internal nodes until the terminal leaves contain only one sample or the predeﬁned max depth is reached.
For Random Forest, we use the balance of base trees (i.e., depth of trees and number of leaves per tree) and additional information (e.g., feature importance of each base tree).
It is noted that feature importance information is available for each base tree—we therefore analyze the statistic of mean and max of base tree feature importance. The following information of base trees are used: • Tree depth: min, max, mean, std, skewness, and kurtosis • Number of leaves: min, max, mean, std, skewness, and kurtosis • Mean of base tree feature importance: min, max, mean, std, skewness, and kurtosis • Max of base tree feature importance: min, max, mean, std, skewness, and kurtosis • Out-of-bag estimate score. Bayesian Ridge Regression Shi et al.
(2016) : it estimates a Gaussian probabilistic model of the regression problem using Bayesian regression.
The prior for the coefﬁcient is given by a spherical
Gaussian distribution.
For Bayesian Ridge Regression, we use the following information of the probabilistic model (ﬁtted using the dataset) as landmarker features:
• Mean of the distribution • Log marginal Likelihood score • The precision of the estimated weights • The noise precision • The actual number of iterations to achieve the stopping criterion
F M ODEL S PACE
We now provide details on the model space used to study the meta-learning problem formulated in our work.
Recall that a model in the context of our problem is a time-series forecasting algorithm and the hyperparameters used.
DeepAR Salinas et al.
(2020): DeepAR experiments are using the model implementation provided by GluonTS version 1.7.
We did grid search on different values of number of cells and the number of RNN layers hyperparameters of DeepAR since the defaults provided in GluonTS would often lead to apparently suboptimal performance on many of the datasets.
The training parameters for each dataset are described in Table 9.
All other parameters are defaults of gluonts.model.deepar.
DeepAREstimator.
Deep Factors Wang et al.
Deep Factors experiments are using the model implementation provided by GluonTS version 1.7.
We did grid search over the number of units per hidden layer for the global RNN model and the number of global factors hyperparameters of Deep Factors.
The training parameters for each dataset are described in Table 9.
All other parameters are defaults of gluonts.model.deep factor.
DeepFactorEstimator.
[Table]
== val ]
Statistics zero cross,one cross,minus cross,root hyp test 5 Kurtosis µ 4 σ
Featurenormality sample kurt 2 Skewness
Skw
Featurenormality sample skew 2 Symm looking | µ X − median X | <r ∗ ( max X − min X )
Symmetry r ∈ [0 . 05 , 0 .
95] 12 Reversal Asymmetry
E [ L 2 ( X ) 2 · L ( X ) − L ( X ) · X 2 ]
ReversalAsymmetry lag L 12 AbsoluteSumofChange (cid:80) n i =
| x i +1 − x
i |
Difference mean chg,mean abs chg,cid 5 ChangeQuantiles
CorridorQuantiles
Difference q l ∈ [0 , 1] ,q h ∈ [ q l , 1]
[Figure]
Entropy
[Figure]
Regularity approx entropy,sample entropy 21 LinearTrend
Linear reg chunks
Trend { “pvalue”,“rvalue”,“intercept”,“slope”,“stderr” }
[Figure]
Non-Linearity c
Non-linearity (cid:96) ∈{ 1 , 2 , 3 } ,matrix proﬁle feat 9 FriedrichCoefﬁcients x (cid:48)
=
h ( x )
Dynamics
Langevin Coeffs
[Figure]
WaveletTransformCoefﬁcients
Mexicanhatwavelet
Time-Freq “widths”,“coeffs”
[Figure]
FastFourierTransformCoeeﬁcients
FFT(X)
Frequency { “real”,“imag”,’agg metrics’ }
[Figure]
PolarFastFourierTransformCoeeﬁcients
FFT(X)
[Figure]
{ “abs”,“angle”,’agg metrics’ }
[Figure]
SpectralRegularity
[Figure]
1)AutoRegeressionRegressionCoefﬁcients
SeeAppendixE
Landmarker (cid:96) ∈{ 1 , ··· [Figure] RandomForestNumberofLeaves
SeeAppendixE
Landmarker
[Figure]
[Figure]
[Figure] std [Figure] skew
[Figure]
kurtosis
[Figure]
Treedepth
SeeAppendixE
Landmarker max,min, [Figure] ,std,skew,andkurtosis [Figure]
MeanofBaseTreeFeatureImportance
SeeAppendixE
Landmarker max,min,mean,std,skew,andkurtosis 6 MaxofBaseTreeFeatureImportance
SeeAppendixE
Landmarker max,min,mean,std,skew,andkurtosis 6 Out [Figure] of-bagestimatescore
SeeAppendixE
Landmarker
N/A ( [Figure] BayesianRidgeRegressionMeanofDistribution
SeeAppendixE
Landmarker
N/A [Figure] LogMarginalLikelihood
SeeAppendixE
Landmarker
N/A 1 EstimatedWeights’Precision
SeeAppendixE
Landmarker
N/A 1 EstimatedNoisePrecision
SeeAppendixE
Landmarker
N/A 1 StoppingNumberofIterations
SeeAppendixE
Landmarker
N/A
See hyperparameter deﬁnitions for various algorithms from GluonTS (Alexandrov et al., 2020) and statsmodels (Seabold & Perktold, 2010).
[Figure]
Prophet Taylor & Letham (2018):
Prophet experiments are using the model python implementation provided by Facebook (fbprophet) version 0.7.1.
We did grid search over the change point prior scale and the seasonality prior scale hyperparameters of Prophet.
The training parameters for each dataset are described in Table 9.
All other parameters are defaults of fbprophet.
Prophet.
Seasonal Naive Montero-Manso et al.
Seasonal Naive experiments are using the model implementation provided by GluonTS version 1.7.
We did grid search over the length of seasonality pattern, since it is different unknown for each, dataset hyperparameter of Seasonal Naive.
The training parameters for each dataset are described in Table 9.
All other parameters are defaults of gluonts.model.seasonal naive.
SeasonalNaivePredictor.
Gaussian Process
Yan et al.
(2009): Gaussian Process experiments are using the model implementation provided by GluonTS version 1.7.
We did grid search over the cardinality of the time-series and the maximum number of iterations for jitter to iteratively make the matrix positive deﬁnite hyperparameter of Gaussian Proces.
The training parameters for each dataset are described in Table 9.
All
Vector Auto Regression Lewis & Reinsel (1985):
Vector Auto Regression experiments are using the model implementation provided by statsmodels python library version 0.12.2.
We did grid search over the loss covariance type and the trend hyperparameter of Vector Auto Regression.
The training parameters for each dataset are described in Table 9.
All other parameters are defaults of statsmodels.tsa.var model.
Random Forest Liaw et al.
(2002): Random Forest models’ experiments are using the model implementation provided by sklearn python library version 0.24.2.
We did grid search over the the number of estimators (trees) and the max depth (i.e., the longest path between the root node and the leaf node in a tree) hyperparameter of Random Forest.
The training parameters for each dataset are described in Table 9.
All other parameters are defaults of sklearn.ensemble.
RandomForestRegressor.
Exponential Smoothing Data Representation Kalekar et al.
We use the exponential smoothing from the statsmodels python library version 0.12.2.
We optimized the smoothing level for the exponential smoothing (which controls the weights given for the history samples).
All other parameters are defaults of statsmodels.tsa.api.
ExponentialSmoothing.
G D ATASET D ESCRIPTION G. D ATASET T YPES IN A UTO F ORECAST
In our work, we consider two general types of time-series datasets depending on the number of the variables v i in the time-series dataset.
Now, we detail the different dataset types.
Univariate Time-series Datasets : This kind of datasets is the traditional single time-series which usually consists of single variable that need to be predicted (this variable is likely numerical, but can also be categorical in rare cases).
Formally, a time-series dataset D i is single-variate if v i = 1 .
Multivariate Time-series Datasets : In our work, we consider two main types (subcategories) of the Multivariate datasets that we deﬁne below.
We deﬁne the Multivariate Homogeneous dataset as the multivariate dataset that consists of multiple time-series in which each time-series represents the same metric ( e.g. , collection of r time-series representing CPU usage of r different machines).
In this type, v i =
r (the number of different time-series for that same measurement).
Also, we deﬁne the Multivariate Heterogeneous dataset as multivariate time-series where each time-series column represents a different measurement ( e.g. , wind speed, humidity, temperature).
In this type, v i = | columns ( D i ) | (the number of the different variables for the time-series dataset).
G. D ATASET T ESTBEDS IN A UTO F ORECAST
Dataset Sources: Our testbeds are built to simulate the testbed when meta-train comes from many different distributions.
This diversity makes the meta-learning model learn from such diversity.
Model selection on test data can thus beneﬁt from the prior experience on the train set.
For this purpose, we use benchmark datasets from Kaggle Kaggle (2021), Adobe real traces, and other open source repositories.
In particular, the Adobe trace datasets records CPU and Memory usage fordifferent services running in Adobe production clusters collected for 15 days from May 1 to Mayin 2021.
Such traces are shared for the ﬁrst time in our current work.
Testbeds Summary: In short, we have collected 308 univariate datasets from such different sources.
The details of the datasets (i.e., the dataset name, and the number of points in the dataset) are shown in Table 10.
We also collected 40 multivariate datasets, where each dataset can have different number of variables and different type.
For instance, Adobe service CPU and memory 15 days utilization is a Homogeneous multivariate dataset that has r = 100 .
Most of the datasets are from different application domains (e.g., ﬁnance, IoT, energy, storage, etc.).
The details of the datasets (i.e., the dataset name, the variate name, and the number of points for each variate the dataset) are detailed in Table 11.
For robustness, for each testbed, we split its datasets into 5 folds for cross-validation.
We build the train/test testbed by each time selecting four folds from the datasets for training and the remaining ﬁfth fold for testing.
We re-emphasize that we released the datastes for the community for future usage (link is in Section 1).
[Table]
228 FRNCHB.
MAD.
UN.
JAMES.
CPI.
288 LYNDPIN.
OZONE.
GRANT.
SUNSPOTS.
KIEWA.
72 NEUMUNAS.
YD.
CHICKNYC.
NILEJJ.
robocalls 52 AMERICAN.
RHINE.
USM1.
PORKL.
ENGINES.
188 NIAGARA.
YULE1.
SERIESA.
ASKEW4.
homeruns 118 SKUNK.
stocks price
PRGNP.
OLDMANP.
OOSTANAU.
816 DANUBE.
LACSTJSN.
EMERGING.
CAFFEINE.
us population 816 PLSUPER.
CIG.
SIMAR4.
SUNSPTMO.
ENGLISH.
660 SERIESF.
BOISE.
FREEDMAN.
ASKEW3.
global co 104 WHEAT.
EGGS.
VATNSD.
SERIESJX.
QBIRTHS.
5117 MARTEN.
FOOD.
CONSUM.
FEED.
EXSHAW.
506 WHITEMTN.
centralia
TSEOIL.
BRYCE.
MADISON.
456 apple
GOLDH.
MUSKRAT.
BAYDU.
jfk passengers 468 SFSKYKOM.
SERIESD.
LOGISTIC.
VELMON.
PEAS.
768 OTTER L.
METALS.
BIRTHS
[Table]
[Table] [Table]
NIGERIA
[Table] [Table]
CIGB
[Table] [Table]
[Table] [Table]
66 whin temp
[Table]
shanghai license
[Table] [Table]
180 SERIESL.
[Table]
SOY
[Table]
FORTALEZ
[Table]
NMAGNET
[Table]
AROSA.
[Table]
480 FURNAS.DAT
LAKEMICH
[Table]
MCKENZIE
[Table]
ROCKY
[Table]
SNOW
[Table] [Table]
[Table] [Table]
uk
[Table] [Table]
ASKEW14.
TRINITY.
OGDEN.
97 TPMON.
I.
DEATHS.
SSASK.
SOYH.
99 SAUGEENT.
SKIRTS.
M.
OKAK.
EAGLECOL.
858 ASKEW10.
WOLVEREN.
SCHOLES.
KINGS.
STJOHNS.
600 SERIESJ.
NILEMON.
BOXHU1.
YEAR.
BOXHUN.
217 CORN.
GNP.
SUMMER.
BLUME.
NARAMATA.
515 HURON.
USM2.
CRYER.
usd isk
MSTOUIS.
96 MEASLNYC.
seatbelts
ASKEW12.
NILE.
ALIGN.
55 USL.
ELECUS.
SERIESB2.
brent spot measles 991 RWG.
SNAKE.
REDDEER.
MUMPS.
CANFIRE.
71 CMINER.
CLEARWAT.
bitcoin
NYWATER.
ASKEW15.
432 ASKEW.
NECHES.
RACOON.
ASKEW8.
CD.
44 RAPPAHAN.
AARIVINT.
gdp japan
WBDELAWA.
TPYR.
248 VEL.
TRANEQ.
NINEMILE.
HEBRON.
NAMAKAN.
648 DELL.
PLMICH.
SERIESM.
[Table]
H D ISCUSSION ON I NFERENCE T IME C OMPLEXITY
Recall that the number of meta-features is d , the number of models is m .
The time complexity for the inference part of the general meta-learner Φ is O ( d ) .
On the other hand, the time complexity of the time-series LSTM meta-learner Θ is given by O ( | X t | ) , where | X t | is the length of the input sequence.
Thus, A UTO F ORECAST ’s inference time is given by max {O ( d ) , O ( | X t | ) } .
We emphasize that the inference time of the na¨ıve method is much larger since it is given by O ( (cid:80) mi
{+=
+}
I i ) , where I i is the inference time of forecasting algorithm a i .
The aggregate statistics for reduction in inference time across both testbeds is shown in Appendix J.5.
Moreover, Figure 7 shows that
A UTO F ORECAST yields signiﬁcant reduction in inference time (68X reduction) over the na¨ıve approach on the Adobe trace CPU and Memory 15 days usage dataset.
[Table]
[Figure]
[Table]
fast
[Table]
Memorycapacityprovisioned
[Table]
Memoryusage
[Table]
fast
[Table]
CPUcapacityprovisioned
[Table]
CPUcores
[Table]
Diskreadthroughput
[Table]
Networktransmittedthroughput
[Table]
fast
[Table]
CPUusage
[Table]
I B ASELINES D ETAILS
Meta-learning has recently received a signiﬁcant attention for automating ML pipelines for a variety of different problems outside the domain of time-series forecasting including supervised learning Feurer et al.
; Wistuba et al.
(2018), classiﬁcation and regression Finn et al.
Rusu et al.
(2019), unsupervised learning Abdulrahman et al.
(2018), outlier detection Zhao et al.
(2020) and other applications Mittal et al.
(2020); Vinyals et al.
In particular, meta-learning has been leveraged for such automation by designing models for new tasks based on prior experience Vanschoren (2018); Yao et al.
; Raghu et al.
I. B ASELINE C ATEGORIES
We now detail each such baseline.
Most of the methods proposed in these works can be categorized into the following three main categories.
No model selection : This category always employs either the same single model or the ensemble of all the models:
• Random Forest (RF) Liaw et al.
(2002) : is a SOTA tree ensemble that combines the predictions made by many decision trees into a single model.
In prediction, the random forest regression model takes the average of all the individual decision tree estimates.
Simple meta-learners :
Meta-learners in this category pick the generally well-performing forecasting model, globally or locally:
• Global Best (GB) : is the simplest meta-learner that selects the forecasting model with the largest average performance across all train datasets (across all time windows), without using any meta-features.
• ISAC Kadioglu et al.
(2010) : clusters the meta-train datasets based on meta-features.
Given a new test time-series dataset
it identiﬁes its closest cluster and selects the best model with largest average performance on the cluster’s datasets.
• ARGOSMART (AS) Nikoli´c et al.
(2013) : ﬁnds the closest meta-train time-series dataset (1NN) to a given test time-series dataset, based on meta-feature similarity, and selects the model with the best performance on the 1NN dataset.
[-Occtrain txtHumidity 
-]
[-Occtrain txtOccupancy 
-]
Optimization-based meta-learners : Meta-learners in this category learn meta-feature by task similarities toward optimizing performance estimates: • Multi-layer Perceptron (MLP) :
Given the meta-train datasets and selected time window, the MLP regressor directly maps the meta-features onto model performances by regression.
However, such baseline does not learn temporal dependence within datasets.
• AUTOFORECAST-TSL : is a variant in which the meta-learner L consists only of the time-series learner Θ .
I. U SAGE OF META - LEARNING IN A UTO F ORECAST
We emphasize that we use the term “meta-learning” in the context of traditional principle of meta-learning which is building upon prior experience on a set of historical tasks to “do better” on a new task.
We build the experience across different datasets using our “general meta-learner” and build experience on the sequential dependence among the same dataset using the “LSTM time-series meta-learner”.
We also capture task similarity between a new input task (dataset) and historical datasets using the [-na¨ıve method is much larger since it is given by O ( (cid:80) mi =
-]“meta-features”.
We also emphasize that our proposed method is faster compared to gradient-descent-based bi-level meta-learners, e.g., on our univariate tesbed N-Beats has signiﬁcantly slower training (average = 3600 seconds) and inference time (average = 101 seconds) compared to A UTO F ORECAST (average = 670 seconds for training and 1.13 seconds for inference).
We emphasize that enhancing gradient-descent-based bi-level meta-learners for our new problem is an avenue for future work.
Inference Time  Example: models 
Our approach yields significant reduction in inference time (68X reduction) 
Naïve approach: Doing inference on all models and select the best candidate
[Figure]
[Figure]
J E XTENDED R ESULTS J. F ULL E VALUATION ON D ATASETS PER T ESTBED
Univariate Testbed: We now present the full evaluation of univariate testbed.
Such evaluation is shown in Table 12 (we show one fold in that table in the interest of space).
We note similar performances for the rest of the folds (as reﬂected in the average MSE and average rank for all datasets shown in Table 3-4).
A UTO F ORECAST outperforms the meta-learner baselines for most datasets.
Moreover, A UTO F ORECAST has the lowest average MSE, and the lowest average rank.
Multivariate Testbed: We here present the full evaluation of multivariate testbed.
Such evaluation is shown in Table 13.
It is clear that A UTO F ORECAST ouperforms the meta-learner baselines for most datasets (best for 28 out of the 40 datasets).
Again, we note that A UTO F ORECAST has the lowest average MSE, and the lowest average rank.
In particular, A UTO F ORECAST (MSE = 0.12) gives a gain of 50% over the best baseline (MSE = 0.26) on the Adobe CPU Mem 15d dataset.
Meta-learners perform better than methods without model selection:
Table 14 shows that meta-learners outperforms almost all models with no model selection.
In particular, three meta-learners (Global Best, ISAC, A UTO F ORECAST ) signiﬁcantly outperform the baseline time-series forecasting models.
For instance, Global Best respectively has 79.42%, 56.71%, 67.28%, 64.32%, 54.19%,95.73%, and 88.21% lower MSE over Seasonal Naive, DeepAR, Deep Factors,
Random Forest, Prophet, Gaussian Process, and VAR.
These results signify the beneﬁts of model selection (speciﬁcally using A UTO F ORECAST ).
Prophet and DeepAR have the best performance across the baseline forecasting algorithms:
Table 14 shows that Prophet model has the best average MSE across the baseline forecasting algorithms.
Notably, DeepAR has the second best average MSE across them.
Gain in Selecting better models using A UTO F ORECAST : Figure 8 shows that across the pool of the multivariate testbed (across all time windows), A UTO F ORECAST has superiority in selecting better models compared to the baseline meta-learners (2X gain in selecting better forecasting models).
[Table]
J. S TATISTICAL S IGNIFICANCE OF A UTO F ORECAST
We now show the pairwise statistical test results between every pair of methods by Wilcoxon signed rank test in Table 15.
Statistically better method shown in bold (both marked bold if no signiﬁcance).
We re-emphasize that we use the pairwise Wilcoxon signed rank test on performances (MSE) across datasets (signiﬁcance level p < 0 . 05 ).
In the left, univariate testbed is shown where A UTO F ORECAST is statistically signiﬁcantly better most baselines GB, ORECAST the right multivariate A UTO
ORECAST is statistically signiﬁcantly better We next turn our attention to the tuning of the time-series meta-learner and the timing of A UTO F ORECAST , respectively. In particular, we show both the computational cost and the inference time for A UTO F ORECAST and compare it with our different baselines.
[Table]
The seven have worse performance chose
Moreover,
ORECAST has the
[Table]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
6 Number of LSTM Layers
[Figure]
[Figure]
M S E
70 Number of Units per Layer
[Figure]
M S E
[Figure]
[Figure]
[Figure]
[Figure]
M S E J. T UNING OF T IME - SERIES M ETA - LEARNER Θ
We show the effect of different hyper-parameters used in the training of the time-series meta-learner Θ on the performance (in terms of MSE under the selected model by Θ ).
For searching on each parameter, we ﬁx the other parameters on their best values.
In the interest of space, we show full details for the univariate testbed.
Figure 9 shows the effect of the number of the LSTM layers of the time-series meta-learner.
We observe that LSTM with 4 layers gives the best performance (lowest MSE).
Second, Figure 10 shows the effect of the number of the units in the LSTM layer of the time-series meta-learner.
We observe that LSTM with 50 units per layer gives the best performance (lowest MSE).
Third, Figure 11 shows the effect of the training batch size of the time-series meta-learner.
A batch with size = 25 instances gives the best performance (lowest
MSE).
Finally, Figure 12 illustrates the effect of the number of training epochs of the time-series meta-learner.
We note that training with 50 epochs gives the best performance (lowest MSE).
We have also used dropout rate of 0.2 to prevent over-ﬁtting.
J. C OMPARING C OMPUTATIONAL COST OF A UTO F ORECAST
Table 16 shows that A UTO F ORECAST has comparable computational training cost compared to the other meta-learners baselines.
We re-emphasize that the running time of such ofﬂine training phase is less critical since it is done only once.
However, this experiment shows that our better time series model selection performance does not entail a prohibitive training cost.
A UTO F ORECAST has comparable computational cost compared to other meta-learners baselines.
[Table]
J. I NFERENCE R UN T IME S TATISTICS
Comparison between A UTO F ORECAST and na¨ıve approach: We compare the inference time that A UTO F ORECAST versus the inference time of the na¨ıve approach, i.e., doing inference over all possible models.
We ﬁrst show the aggregate statistics for the gain across all datasets for each testbed.
Figure 3 illustrates the reduction in inference time for using A UTO F ORECAST over the na¨ıve approach for both univariate and multivariate testbeds.
In particular, A UTO F ORECAST gives a median gain of at least 42X over the na¨ıve approach for both testbeds.
Dataset-wise inference time comparison: Next, we pick several random groups of 10 datasets each and show the time (in seconds) that A UTO F ORECAST takes versus the time the na¨ıve approach takes for forecasting model selection.
Figures 13-14 shows such comparison for the univariate testbed.
It is noted the higher reduction of inference time for larger datasets (i.e., with more data points).
[Figure]
[Figure]
Multivariate testbed: We also observe that A UTO F ORECAST (meta-feature generation and model selection) takes less than 1.2 second on most time series datasets for multivariate testbed.
Time overhead of A UTO F ORECAST relative to training of selected model: Figure 15 shows that it incurs only negligible overhead relative to actual training of the selected model (with median = 0.1%).
Similarly, A UTO F ORECAST incurs only negligible overhead relative to actual training of the selected model for the multivariate testbed (with median = 0.3%)
(Figure is omitted for multivariate testbed).
This shows that A UTO F ORECAST is lightweight, incurring small selection time overhead.
K I NTERPRETABLE F EATURES OF D ATASETS
Now, we show some of the interpretable input feature values for all datasets in the univariate testbed.
In particular, we show mean, median, variance, skeweness, kutosis, absolute energy, and benford correlation for each of the individual time series in that testbed.
Such features are shown in Figures 16-23.
Note that time-series datasets have standard normalization before feature extraction.
Such features values show the diversity in the datasets, where each dataset has different characteristics.
We emphasize that in the interest of space, we show sample of these interpretable features.
[Figure]
M e a n
300 Dataset Index 0.
M e d i a n
[Figure]
i
[Figure]
[Figure]
[Figure]
K
[Figure]
r t o
s i s
[Figure]
[Figure]
o
[Figure]
l a t
[Figure]
o
[Figure]
[Figure]