Original: M ULTI -O BJECTIVE M ODEL S ELECTION FOR T IME S ERIES F ORECASTING
Revised: M ULTI -O BJECTIVE M ODEL S ELECTION FOR T IME S ERIES F ORECASTING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Research on time series forecasting has predominantly focused on developing methods that improve accuracy.
Revised: Research on time series forecasting has predominantly focused on developing methods that improve accuracy.

Original: However, other criteria such as training time or latency are critical in many real-world applications.
Revised: However, other criteria such as training time or latency are critical in many real-world applications.

Original: We therefore address the question of how to choose an appropriate forecasting model for a given dataset among the plethora of available forecasting methods when accuracy is only one of many criteria.
Revised: We therefore address the question of how to choose an appropriate forecasting model for a given dataset among the plethora of available forecasting methods when accuracy is only one of many criteria.

Original: For this, our contributions are two-fold.
Revised: For this, our contributions are two-fold.

Original: First, we present a comprehensive benchmark, evaluating 7 classical and 6 deep learning forecasting methods on 44 heterogeneous, publicly available datasets.
Revised: First, we present a comprehensive benchmark, evaluating 7 classical and 6 deep learning forecasting methods on 44 heterogeneous, publicly available datasets.

Original: The benchmark code is open-sourced along with evaluations and forecasts for all methods.
Revised: The benchmark code is open-sourced along with evaluations and forecasts for all methods.

Original: These evaluations enable us to answer open questions such as the amount of data required for deep learning models to outperform classical ones.
Revised: These evaluations enable us to answer open questions such as the amount of data required for deep learning models to outperform classical ones.

Original: Second, we leverage the benchmark evaluations to learn good defaults that consider multiple objectives such as accuracy and latency.
Revised: Second, we leverage the benchmark evaluations to learn good defaults that consider multiple objectives such as accuracy and latency.

Original: By learning a mapping from forecasting models to performance metrics, we show that our method P ARETO S ELECT is able to accurately select models from the Pareto front — alleviating the need to train or evaluate many forecasting models for model selection.
Revised: By learning a mapping from forecasting models to performance metrics, we show that our method P ARETO S ELECT is able to accurately select models from the Pareto front — alleviating the need to train or evaluate many forecasting models for model selection.

Original: To the best of our knowledge, P ARETO S ELECT constitutes the ﬁrst method to learn default models in a multi-objective setting.
Revised: To the best of our knowledge, P ARETO S ELECT constitutes the ﬁrst method to learn default models in a multi-objective setting.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: For decades, businesses have been using time series forecasting to drive strategic decision-making (Simchi-Levi et al., 2013; Hyndman & Athanasopoulos, 2018; Petropoulos et al., 2020).
Revised: For decades, businesses have been using time series forecasting to drive strategic decision-making (Simchi-Levi et al., 2013; Hyndman & Athanasopoulos, 2018; Petropoulos et al., 2020).

Original: Analysts leverage forecasts to gauge resource requirements, retailers forecast future product demand to optimize their supply chains, cloud providers predict future web trafﬁc to scale server ﬂeets, and in the energy sector, forecasting plays a crucial role e.g. to predict load and energy prices.
Revised: Analysts leverage forecasts to gauge resource requirements, retailers forecast future product demand to optimize their supply chains, cloud providers predict future web trafﬁc to scale server ﬂeets, and in the energy sector, forecasting plays a crucial role e.g. to predict load and energy prices.

Original: In domains like these and many other, more precise predictions directly translate into an increase in proﬁt.
Revised: In domains like these and many other, more precise predictions directly translate into an increase in proﬁt.

Original: Thus, it is no surprise that research on forecasting methods has historically focused on improving accuracy.
Revised: Thus, it is no surprise that research on forecasting methods has historically focused on improving accuracy.

Original: In addition to more classical local forecasting methods which ﬁt a model per time series, global forecasting models such as deep learning and tree-based models have demonstrated state-of-the-art forecasting accuracy (Wen et al., 2017; Oreshkin et al., 2019; Salinas et al., 2020a; Smyl, 2020) when sufﬁcient training data is available (Makridakis et al., 2018; Januschowski et al., 2020).
Revised: In addition to more classical local forecasting methods which ﬁt a model per time series, global forecasting models such as deep learning and tree-based models have demonstrated state-of-the-art forecasting accuracy (Wen et al., 2017; Oreshkin et al., 2019; Salinas et al., 2020a; Smyl, 2020) when sufﬁcient training data is available (Makridakis et al., 2018; Januschowski et al., 2020).

Original: This research has led to a large variety of different forecasting models and hyperparameter choices (Hutter et al., 2019), where different models exhibit vastly different characteristics, including accuracy, training time, model size, and inference latency.
Revised: This research has led to a large variety of different forecasting models and hyperparameter choices (Hutter et al., 2019), where different models exhibit vastly different characteristics, including accuracy, training time, model size, and inference latency.

Original: While this variety of forecasting models is a great resource, it introduces challenging questions.
Revised: While this variety of forecasting models is a great resource, it introduces challenging questions.

Original: On the one hand, researchers would like to understand patterns in the performance of different models and to benchmark new models against existing methods.
Revised: On the one hand, researchers would like to understand patterns in the performance of different models and to benchmark new models against existing methods.

Original: On the other hand, practitioners are interested in understanding which model performs best for a particular dataset or application.
Revised: On the other hand, practitioners are interested in understanding which model performs best for a particular dataset or application.

Original: In this work, we address both of these problems: we release one of the most comprehensive publicly available evaluations of forecasting models across 44 datasets.
Revised: In this work, we address both of these problems: we release one of the most comprehensive publicly available evaluations of forecasting models across 44 datasets.

Original: Using this benchmark dataset, we develop a novel method for learning good defaults for forecasting models on previously unseen datasets.
Revised: Using this benchmark dataset, we develop a novel method for learning good defaults for forecasting models on previously unseen datasets.

Original: Importantly, we adopt a multi-objective perspective that allows us to select forecasting models that are simultaneously accurate and satisfy constraints such as inference latency, training time, or model size.
Revised: Importantly, we adopt a multi-objective perspective that allows us to select forecasting models that are simultaneously accurate and satisfy constraints such as inference latency, training time, or model size.

Original: The main contributions of this work can be summarized as follows:
Revised: The main contributions of this work can be summarized as follows:

Original: • We release the evaluations of 13 forecasting methods on 44 public datasets with respect to multiple performance criteria (different forecast accuracy metrics, inference latency, training time, and model size).
Revised: • We release the evaluations of 13 forecasting methods on 44 public datasets with respect to multiple performance criteria (different forecast accuracy metrics, inference latency, training time, and model size).

Original: This constitutes, by far, the most comprehensive publicly available evaluation of forecasting methods.
Revised: This constitutes, by far, the most comprehensive publicly available evaluation of forecasting methods.

Original: Those evaluations can be leveraged, for example, to assess the relative performance of future forecasting methods with little effort.
Revised: Those evaluations can be leveraged, for example, to assess the relative performance of future forecasting methods with little effort.

Original: • As an example application of this benchmark, we use the data to perform a statistical analysis which shows that only a few thousands observations are required for deep learning methods to outperform classical methods.
Revised: • As an example application of this benchmark, we use the data to perform a statistical analysis which shows that only a few thousands observations are required for deep learning methods to outperform classical methods.

Original: In addition, we investigate the beneﬁt of ensembling forecasting models.
Revised: In addition, we investigate the beneﬁt of ensembling forecasting models.

Original: • We propose a novel method that can leverage ofﬂine evaluations to learn good default models for unseen datasets.
Revised: • We propose a novel method that can leverage ofﬂine evaluations to learn good default models for unseen datasets.

Original: Default models are selected to optimize for multiple objectives.
Revised: Default models are selected to optimize for multiple objectives.

Original: • We introduce a technique for ensembling models in a multi-objective setting where the resulting ensemble is not only highly accurate but also optimized for other objectives, such as a low inference latency.
Revised: • We introduce a technique for ensembling models in a multi-objective setting where the resulting ensemble is not only highly accurate but also optimized for other objectives, such as a low inference latency.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Time Series Forecasting.
Revised: Time Series Forecasting.

Original: Time series forecasting has seen a recent surge in attention by the academic community that has started to reﬂect its relevance in business applications.
Revised: Time series forecasting has seen a recent surge in attention by the academic community that has started to reﬂect its relevance in business applications.

Original: Traditionally, univariate, so-called local models that consider time series individually have dominated (Hyndman & Athanasopoulos, 2018).
Revised: Traditionally, univariate, so-called local models that consider time series individually have dominated (Hyndman & Athanasopoulos, 2018).

Original: However, in modern applications, methods that learn globally across a set of time series can be more accurate (Oreshkin et al., 2019; Salinas et al., 2020a;
Revised: However, in modern applications, methods that learn globally across a set of time series can be more accurate (Oreshkin et al., 2019; Salinas et al., 2020a;

Original: Lim et al., 2021;
Revised: Lim et al., 2021;

Original: Montero-Manso & Hyndman, 2021) — in particular, methods that rely on deep learning.
Revised: Montero-Manso & Hyndman, 2021) — in particular, methods that rely on deep learning.

Original: With a considerable choice of models available, it is unclear which methods should perform best on which forecasting dataset, unlike in other machine learning domains where dominant approaches exist.
Revised: With a considerable choice of models available, it is unclear which methods should perform best on which forecasting dataset, unlike in other machine learning domains where dominant approaches exist.

Original: In domains such as computer vision or neural architecture search (NAS), ofﬂine computations have been harvested and leveraged successfully to perform extensive model comparisons or compare different NAS strategies (Ying et al., 2019; Dong & Yang, 2020; Pﬁsterer et al., 2021).
Revised: In domains such as computer vision or neural architecture search (NAS), ofﬂine computations have been harvested and leveraged successfully to perform extensive model comparisons or compare different NAS strategies (Ying et al., 2019; Dong & Yang, 2020; Pﬁsterer et al., 2021).

Original: However, to the best of our knowledge, no comprehensive set of model evaluations has been released in the realm of forecasting.
Revised: However, to the best of our knowledge, no comprehensive set of model evaluations has been released in the realm of forecasting.

Original: Consequently, some questions remain open: how much data is required for global deep learning methods to outperform classical local methods such as ARIMA or ETS (Hyndman & Athanasopoulos, 2018)?
Revised: Consequently, some questions remain open: how much data is required for global deep learning methods to outperform classical local methods such as ARIMA or ETS (Hyndman & Athanasopoulos, 2018)?

Original: What is the impact on accuracy when ensembling different forecasting models?
Revised: What is the impact on accuracy when ensembling different forecasting models?

Original: While some recent methods incorporate ensembling (Oreshkin et al., 2019; Jeon & Seong, 2021), comparisons are often made against individual models which may cloud the beneﬁt of the method proposed versus the sole beneﬁt of ensembling.
Revised: While some recent methods incorporate ensembling (Oreshkin et al., 2019; Jeon & Seong, 2021), comparisons are often made against individual models which may cloud the beneﬁt of the method proposed versus the sole beneﬁt of ensembling.

Original: Learning Default Models.
Revised: Learning Default Models.

Original: Finding the best model or set of hyperparameters is often performed via Bayesian optimization given its theoretical regret guarantees (Srinivas et al., 2012).
Revised: Finding the best model or set of hyperparameters is often performed via Bayesian optimization given its theoretical regret guarantees (Srinivas et al., 2012).

Original: However, even with early-stopping techniques (Golovin et al., 2017; Li et al., 2017), practitioners often restrict the search to a single model due to the large cost of training many models.
Revised: However, even with early-stopping techniques (Golovin et al., 2017; Li et al., 2017), practitioners often restrict the search to a single model due to the large cost of training many models.

Original: One technique to drastically speed up the model/hyperparameter search is to reuse ofﬂine evaluations of related datasets via transfer learning.
Revised: One technique to drastically speed up the model/hyperparameter search is to reuse ofﬂine evaluations of related datasets via transfer learning.

Original: For instance, Wistuba et al.
Revised: For instance, Wistuba et al.

Original: ; Winkelmolen et al.
Revised: ; Winkelmolen et al.

Original: Pﬁsterer et al.
Revised: Pﬁsterer et al.

Original: (2021) leverage ofﬂine evaluations to alleviate the need for training many models by learning a small list of n good defaults that provide a small joint error when evaluated on all datasets.
Revised: (2021) leverage ofﬂine evaluations to alleviate the need for training many models by learning a small list of n good defaults that provide a small joint error when evaluated on all datasets.

Original: This approach bears a lot of similarity with ours.
Revised: This approach bears a lot of similarity with ours.

Original: Key differences are that we consider multiple objectives and model ensembles.
Revised: Key differences are that we consider multiple objectives and model ensembles.

Original: In the time series domain, Shah et al.
Revised: In the time series domain, Shah et al.

Original: (2021) also considers the task of automatically choosing from a large pool of forecasting models the one performing best on a particular dataset but also only optimizes for a single objective and relies on potentially expensive model training for model selection.
Revised: (2021) also considers the task of automatically choosing from a large pool of forecasting models the one performing best on a particular dataset but also only optimizes for a single objective and relies on potentially expensive model training for model selection.

Original:  B ENCHMARKING F ORECASTING M ETHODS
Revised:  B ENCHMARKING F ORECASTING M ETHODS

Original: In this section, we formally introduce the problem of time series forecasting and subsequently provide an overview of the benchmark evaluations that we release.
Revised: In this section, we formally introduce the problem of time series forecasting and subsequently provide an overview of the benchmark evaluations that we release.

Original: By evaluating 13 forecasting meth- ods along with different hyperparameter choices on all 44 benchmark datasets and for two random seeds, the benchmark comprises 4,708 training runs and amasses over 1 TiB of forecast data.
Revised: By evaluating 13 forecasting meth- ods along with different hyperparameter choices on all 44 benchmark datasets and for two random seeds, the benchmark comprises 4,708 training runs and amasses over 1 TiB of forecast data.

Original: At the end of this section, we demonstrate how this benchmark data can be used to perform an extensive comparison of contemporary forecasting methods.
Revised: At the end of this section, we demonstrate how this benchmark data can be used to perform an extensive comparison of contemporary forecasting methods.

Original: 3. T IME S ERIES F ORECASTING
Revised: 3. T IME S ERIES F ORECASTING

Original: The goal of time series forecasting is to predict the future values of one or more time series based on historical observations.
Revised: The goal of time series forecasting is to predict the future values of one or more time series based on historical observations.

Original: Formally, we consider a set Z
Revised: Formally, we consider a set Z

Original: = { z ( i ) 1: T i } Ki =1 of K univariate, equally-spaced time series of lengths
Revised: = { z ( i ) 1: T i } Ki =1 of K univariate, equally-spaced time series of lengths

Original: T i , where z ( i ) a : b = ( z ( i ) a , z ( i ) a +1 , . . .
Revised: T i , where z ( i ) a : b = ( z ( i ) a , z ( i ) a +1 , . . .

Original: , z ( i ) b ) denotes the vector of observations for the i -th time series in the time interval a ≤ t ≤ b .
Revised: , z ( i ) b ) denotes the vector of observations for the i -th time series in the time interval a ≤ t ≤ b .

Original: In probabilistic time series forecasting, a model then estimates the probability distribution across the forecast horizon τ , i.e. the distribution over the τ future values, from the historical observations:
Revised: In probabilistic time series forecasting, a model then estimates the probability distribution across the forecast horizon τ , i.e. the distribution over the τ future values, from the historical observations:

Original: [Equation]
Revised: [Equation]

Original: Models typically approximate the joint distribution of Eq.
Revised: Models typically approximate the joint distribution of Eq.

Original: (1) via Monte Carlo sampling (Salinas et al., 2020a) or learn to directly predict a set of distribution quantiles for multiple time steps using quantile regression (Wen et al., 2017; Lim et al., 2021).Model Evaluation.
Revised: (1) via Monte Carlo sampling (Salinas et al., 2020a) or learn to directly predict a set of distribution quantiles for multiple time steps using quantile regression (Wen et al., 2017; Lim et al., 2021).

Original: To measure the accuracy of forecasting models, we employ the continuous ranked probability score (CRPS) (Matheson & Winkler, 1976; Gneiting & Raftery, 2007).

Original: Given the quantile function ( F ( i ) t ) − 1 of the forecast probability distribution in Eq.

Original: (1) for a time series z

Original: ( i ) at time step t , the CRPS is deﬁned as

Original: [Equation]

Original: where Λ α is the quantile loss (or pinball loss) deﬁned over a quantile level 0 < α < 1 :

Original: [Equation]

Original: Here, I is the indicator function.

Original: To compute an aggregated score for a model forecasting multiple time series and time steps, we compute a normalized CRPS (nCRPS):

Original: [Equation]

Original: We approximate the nCRPS using the 10-quantiles α ∈ { 0 . 1 , 0 . 2 , 0 . 3 , . . .

Original: While we focus on nCRPS for the evaluations in this paper, the benchmark dataset that we release contains four additional forecast accuracy metrics (MAPE, sMAPE, NRMSE, ND).

Original: 3. M ETHODS
Revised: 3. M ETHODS

Original: We consider 13 models in total that can be distinguished into 7 local methods (estimating parameters individually from each time series) and 6 global methods (learning from all available time series jointly).
Revised: We consider 13 models in total that can be distinguished into 7 local methods (estimating parameters individually from each time series) and 6 global methods (learning from all available time series jointly).

Original: Details about models can be found in Appendix A.
Revised: Details about models can be found in Appendix A.

Original: Local Methods.
Revised: Local Methods.

Original: We use Seasonal Naïve (Hyndman & Athanasopoulos, 2018) and NPTS (Rangapuram et al., 2021) as simple, non-parametric baselines.
Revised: We use Seasonal Naïve (Hyndman & Athanasopoulos, 2018) and NPTS (Rangapuram et al., 2021) as simple, non-parametric baselines.

Original: Further, we integrate ARIMA and ETS (Hyndman & Athanasopoulos, 2018) as well-known statistical methods and STL-AR (Talagala, 2021) and Theta (Assimakopoulos & Nikolopoulos, 2000) as inexpensive alternatives.
Revised: Further, we consider ARIMA and ETS (Hyndman & Athanasopoulos, 2018) as well-known statistical methods and STL-AR (Talagala, 2021) and Theta (Assimakopoulos & Nikolopoulos, 2000) as inexpensive alternatives.

Original: In addition, we consider
Revised: Lastly, we include

Original: Prophet (Taylor & Letham, 2018), a well-interpretable time series model which has received plenty of attention by practitioners.
Revised: Prophet (Taylor & Letham, 2018), an interpretable model that has received plenty of attention.

Original: Global Methods.
Revised: Global Methods.

Original: All global methods that we use are deep learning models, namely: Simple Feedforward (Alexandrov et al., 2020), MQ-CNN and MQ-RNN (Wen et al., 2017), DeepAR (Salinas et al., 2020a), N-BEATS (Oreshkin et al., 2019), and TFT (Lim et al., 2021).
Revised: All global methods that we use are deep learning models, namely: Simple Feedforward (Alexandrov et al., 2020), MQ-CNN and MQ-RNN (Wen et al., 2017), DeepAR (Salinas et al., 2020a), N-BEATS (Oreshkin et al., 2019), and TFT (Lim et al., 2021).

Original: For each model (except MQ-RNN), we consider three hyperparameter settings: this includes the default set provided by their implementation as well as hyperparameter sets that roughly halve and double the default model capacity.
Revised: For each model (except MQ-RNN), we consider three hyperparameter settings: this includes the default set provided by their implementation as well as hyperparameter sets that roughly halve and double the default model capacity (see Table 2 in the appendix).

Original: Additionally, we consider three different context lengths that governs the length of the time series that predictions are conditioned on.
Revised: Additionally, we consider three different context lengths that govern the length of the time series that predictions are conditioned on.

Original: [Table]
Revised: Model Training.

Original:  learning models since parametric local methods estimate parameters at prediction time.
Revised: Model training is only required for deep learning models since parametric local methods estimate parameters at prediction time.

Original: Deep learning models are trained for a ﬁxed duration depending on the size of the dataset.
Revised: Deep learning models are trained for a ﬁxed duration depending on the size of the dataset.

Original: In addition, they are checkpointed multiple times during training and forecasts are generated for all checkpoints.

Original: Further details can be found in Appendix C.
Revised: Further details can be found in Appendix C.

Original: 3. D ATASETS
Revised: 3. D ATASETS

Original: Our benchmark provides 44 heterogeneous public datasets in total.
Revised: Our benchmark provides 44 heterogeneous public datasets in total.

Original: The datasets greatly differ in the number of time series (from 8 to ≈ 170,000), their mean length (from ≈ 22 to ≈ 500,000), their frequency (minutely, hourly, daily, weekly, monthly, quarterly, yearly), and the forecast horizon (from 6 to 60).
Revised: The datasets greatly differ in the number of time series (from 8 to ≈ 170,000), their mean length (from ≈ 22 to ≈ 500,000), their frequency (minutely, hourly, daily, weekly, monthly, quarterly, yearly), and the forecast horizon (from 6 to 60).

Original: Thus, we expect them to cover a wide range of datasets encountered in practice.
Revised: Thus, we expect them to cover a wide range of datasets encountered in practice.

Original: Datasets areprimarily taken from various forecasting competitions, the UCI (Dua & Graff, 2017), and the Monash time series forecasting repository (Godahewa et al., 2021).
Revised: Datasets are taken from various forecasting competitions, the UCI (Dua & Graff, 2017), and the Monash time series forecasting repository (Godahewa et al., 2021).

Original: All individual datasets, their sources,  basic statistics and descriptions, as well as the detailed description of the data preparation procedure can be found in Appendix B.
Revised: Dataset sources, descriptions, basic statistics, and an explanation of the data preparation procedure can be found in Appendix B.

Original: 3. M ETRICS
Revised: 3. M ETRICS

Revised: To measure the accuracy of forecasting models, we employ the normalized continuous ranked probability score (nCRPS) (Matheson & Winkler, 1976; Gneiting & Raftery, 2007) whose deﬁnition

Revised: [Table]

Original:  Besidesthe forecasting accuracy metrics outlined above, we store inference latency, model size (i.e. number of parameters) and training time for deep learning models.
Revised: is given in Appendix A. The benchmark dataset that we release contains four additional forecast accuracy metrics (MAPE, sMAPE, NRMSE, ND). Besides forecasting accuracy metrics, we store inference latency, model size (i.e. number of parameters) and training time for deep learning models.

Original: Latency is measured by dividing the time taken to generate predictions for all test time series in the dataset by their number.
Revised: Latency is measured by dividing the time taken to generate predictions for all test time series in the dataset by their number.

Original: 3. B ENCHMARK A NALYSIS
Revised: 3. B ENCHMARK A NALYSIS

Original: Having outlined the benchmark, we now want to demonstrate the usefulness of the evaluations for research.
Revised: Having outlined the benchmark, we now want to demonstrate the usefulness of the evaluations for research.

Original: For this, we want to analyze how local (classical) and global (deep learning) forecasting methods compare against each other.
Revised: For this, we want to analyze how local (classical) and global (deep learning) forecasting methods compare against each other.

Original: In fact, this comparison has been the object of heated discussions in the forecasting community (Makridakis et al., 2018; Januschowski et al., 2020).
Revised: In fact, this comparison has been the object of heated discussions in the forecasting community (Makridakis et al., 2018; Januschowski et al., 2020).

Original: Method Comparison.
Revised: Method Comparison.

Original: In order to compare methods, we consider their latency and accuracy (in terms of nCRPS) across all datasets.
Revised: In order to compare methods, we consider their latency and accuracy (in terms of nCRPS) across all datasets.

Original: Table 1 shows the median latency as well as the the meth- [Figure] [Figure]
Revised: Table 1 shows each method’s relative latency compared to

Original: ods’ average nCRPS rank and the rank’s standard deviation.
Revised: Seasonal Naïve as well as the the methods’ nCRPS rank 1 .

Original: As far as latency is concerned, global methods, once trained, allow to generate forecasts considerably faster than local methods (except for Seasonal Naïve).
Revised: As far as latency is concerned, global methods, once trained, allow to generate forecasts considerably faster than local methods (except for Seasonal Naïve).

Original: The reason is simple: once a deep learning model is trained it simply needs to run a forward pass.
Revised: The reason is simple: once trained, they simply need to run a forward pass.

Original: In contrast, thereference implementations for the local methods chosen here do not differentiate between training and inference, but rather estimate their parameters at prediction time.
Revised: In contrast, the implementations for the local methods chosen here do not differentiate between training and inference, but rather estimate their parameters at prediction time.

Original: Across datasets, the median latency of all considered deep learning models improves between 50% and 95% upon the latency of the fastest local method (excluding Seasonal Naïve).
Revised: Across datasets, the relative latency of all considered deep learning models improves between 15% and 95% upon the latency of the fastest local method (excluding Seasonal Naïve).

Original: As far as accuracy is concerned, complex deep learning models — namely DeepAR and TFT — generally perform best across the benchmark datasets.
Revised: As far as accuracy is concerned, complex deep learning models — namely DeepAR and TFT — generally perform best across the benchmark datasets.

Original: Except for MQ-RNN, global methodsalso compare favorably against local methods.
Revised: Except for MQ-RNN, global methods compare favorably against local methods.

Original: As a measure of relative model stability, Table 1 additionally shows the standard deviation of the nCRPS rank across all benchmark datasets.
Revised: As a measure of relative model stability, Table 1 additionally shows the standard deviation of the nCRPS rank across all benchmark datasets.

Original: Notably, it shows that TFT is  the method which performs best on average,and, at the same time, its rank is  comparatively consistent compared to other methods.
Revised: Not only is TFT the method which performs best on average, its rank is also comparatively consistent compared to other methods.

Original: A relatively high standard deviation for the nCRPS rank of DeepAR, however, indicates that it generates relatively inaccurate forecasts on some datasets.
Revised: Especially,

Original:  Table 1 also indicates the performance of ensembles obtained by averaging the predictions of n = 9 ( n = 3 for MQ-RNN) deep learning models trained with different hyperparameters ( hyperensembles ) as done for N-BEATS (Oreshkin et al., 2019) or in M5 competition’s third-ranking method (Jeon & Seong, 2021).
Revised: [Figure] [Figure] Table 1 also indicates the performance of ensembles obtained by averaging the predictions of n = 9 ( n = 3 for MQ-RNN) deep learning models trained with different hyperparameters ( hyperensembles ) as done for N-BEATS (Oreshkin et al., 2019) or in M5 competition’s third-ranking method (Jeon & Seong, 2021).

Original: While generally bearing a signiﬁcant cost in latency, the beneﬁt of ensembling is signiﬁcant.
Revised: While generally bearing a signiﬁcant cost in latency, the beneﬁt of ensembling is signiﬁcant.

Original: For instance, the Simple Feedforward hyper-ensemble yields a competitive model that outperforms DeepAR in terms of both accuracy and latency.
Revised: For instance, the Simple Feedforward hyper-ensemble yields a competitive model that outperforms DeepAR in terms of both accuracy and latency.

Original: Lastly, DeepAR and
Revised: Lastly, DeepAR and

Original: TFT hyper-ensembles result in the most competitive ensembles.
Revised: TFT hyper-ensembles result in the most competitive ensembles.

Original: Comparison of Classical and Deep Learning Methods.
Revised: Comparison of Classical and Deep Learning Methods.

Original: We further conduct a statistical analysis to compare the classical, local time series models with theglobal, deep learning models listed in Section 3.2.
Revised: We further conduct a statistical analysis to compare the classical time series models with the deep learning models listed in Section 3.2.

Original: For this, we construct the null hypothesis that the accuracy of classical models is equal to or better than the accuracy of deep learning methods.
Revised: For this, we construct the null hypothesis that the accuracy of classical models is equal to or better than the accuracy of deep learning methods.

Original: This is a claim put forward, for example, by Makridakis et al.
Revised: This is a claim put forward, for example, by Makridakis et al.

Original: More formally, we write H 0 : Q class ≤ Q deep where Q class and Q deep describe the distribution of nCRPS values of classical and deep learning models, respectively.
Revised: More formally, we write H 0 : Q class ≤ Q deep where Q class and Q deep describe the distribution of nCRPS values of classical and deep learning models, respectively.

Original: The evaluations ofour benchmark constitute the samples of these distributions.
Revised: Samples of the distributions are derived from the respective single-model evaluations in our benchmark.

Original: Q class and Q deep are continuous and belong to unknown families of distributions with unequal variances.
Revised: Q class and Q deep are continuous and belong to unknown families of distributions with unequal variances.

Original: Hence, we choose the nonparametric two-sample Kolmogorov-Smirnov test (Gibbons & Chakraborti, 2003) to compute the p -value, i.e. the probability of H 0 holding true, on each individual dataset.
Revised: Hence, we choose the nonparametric two-sample Kolmogorov-Smirnov test (Gibbons & Chakraborti, 2003) to compute the p -value, i.e. the probability of H 0 holding true, on each individual dataset.

Original: We evaluate the null hypothesis H 0 on all 44 benchmark datasets and plot the p -value for the different dataset sizes in Figure 1.
Revised: We evaluate the null hypothesis H 0 on all 44 benchmark datasets and plot the p -value for the different dataset sizes in Figure 1.

Original: At a signiﬁcance level of α = 5% , H 0 can be rejected for 30 out of 44 datasets.
Revised: At a signiﬁcance level of α = 5% , H 0 can be rejected for 30 out of 44 datasets.

Original: In fact, deep learning models exhibit competitive performance for almost all datasets and regularly outperform classical methods.
Revised: In fact, deep learning models exhibit competitive performance for almost all datasets and regularly outperform classical methods.

Original: Further, they tend to perform comparatively better with increasing dataset size  and, nonetheless, show competitive performanceeven for small datasets.
Revised: Further, they tend to perform comparatively better with increasing dataset size — even if for some large datasets, classical models are indistinguishable — and, nonetheless, show competitive performance for small datasets.

Original: The latter is of particular interest since it contradicts tribal knowledge thatvery large datasets are needed to outperform local/classical methods and disputes the claim presented by Makridakis et al.
Revised: The latter is of particular interest since it contradicts tribal knowledge that large datasets are needed to outperform local/classical methods and disputes the claim presented by Makridakis et al.

Original: (2018) since only a few thousands observations seem sufﬁcient to outperform the classical models considered.
Revised: (2018) since only a few thousands observations seem sufﬁcient to outperform the classical models considered.

Original: These statements are further supported by Figure 2 where we compare the best deep learning model  against the best classical model  on each dataset.
Revised: These statements are further supported by Figure 2 where we compare the best deep learning model x deep against the best classical model x class on each dataset.

Original: We compute the relative improvement given by deep learning models as Q deep Q class  − 1 .
Revised: We compute the relative improvement given by deep learning models as nCRPS ( x deep ) nCRPS ( x class ) − 1 .

Original: Except for few outliers (“KDD2018”, “M3 Yearly”, “M4 Quarterly”, “Taxi”), the best deep learning model outperforms all classical models on 40 out of 44 datasets.
Revised: Except for few outliers, the best deep learning model outperforms all classical models on 40 out of 44 datasets (see Appendix F for a further analysis).

Original: Fitting a linear regression model on the relative improvements shows that the improvement by using deep learning models is only slightly ampliﬁed as the datasets grow in size.
Revised: Fitting a linear regression model on the relative improvements shows that the improvement by using deep learning models is only slightly ampliﬁed as the datasets grow in size.

Original: We note, that we conﬂate deep learning and global models in our above discussion.
Revised: We note, that we conﬂate deep learning and global models in our above discussion.

Original: MonteroManso & Hyndman (2021) show the general superiority of global models over local models both theoretically and empirically and our results conﬁrm their ﬁndings further.
Revised: MonteroManso & Hyndman (2021) show the general superiority of global models over local models both theoretically and empirically and our results conﬁrm their ﬁndings further.

Original:  L EARNING M ULTI -O BJECTIVE D EFAULTS
Revised:  L EARNING M ULTI -O BJECTIVE D EFAULTS

Original: The benchmark introduced in the previous section outlined how to acquire ofﬂine evaluations of forecasting methods on various datasets.
Revised: The benchmark introduced in the previous section outlined how to acquire ofﬂine evaluations of forecasting methods on various datasets.

Revised: However, in the presence of multiple conﬂicting objectives such as accuracy and latency, it is not clear how one could choose the best models.

Original: In this section, we ﬁrst formalize the problem of learning good defaults from these evaluations for a single objective.
Revised: In this section, we ﬁrst formalize the problem of learning good defaults from these evaluations for a single objective.

Original: Afterwards, we formally introduce multi-objective optimization and show how learning defaults can be extended to account for multiple objectives.
Revised: Afterwards, we formally introduce multi-objective optimization and show how learning defaults can be extended to account for multiple objectives.

Original: In the end, we report experimental results obtained by using our method.
Revised: In the end, we report experimental results obtained by using our method.

Original: 4. P REREQUISITES
Revised: 4. P REREQUISITES

Original: In our setting, we consider an objective function f : X → R m for a given task (in our case, a task is a dataset).
Revised: In our setting, we consider an objective function f : X → R m for a given task (in our case, a task is a dataset).

Original: The objective function maps any time series model x ∈ X to m objectives (such as nCRPS, latency, etc.)
Revised: The objective function maps any time series model x ∈ X to m objectives (such as nCRPS, latency, etc.)

Original: that ought to be minimized.
Revised: that ought to be minimized.

Original: In addition, we assume that model evaluations on M different but related tasks with objective functions f (1) , . ..
Revised: In addition, we assume that model evaluations on T different but related tasks with objective functions f (1) , . .

Original:  , f ( M ) are available.
Revised: . , f ( T ) are available.

Original: The set of  model evaluations is then given as
Revised: The set of ofﬂine model evaluations is then given as

Original: D = M [ j = n(cid:16) x ( j ) i , y ( j ) i (cid:17)o N j i = with y ( j ) i = f ( j ) ( x ( j ) i )
Revised: [Equation]

Original: ∈

Original: R m (5) where N j evaluations are available for task j and y ( j ) i denotes the evaluation of x ( j ) i on task j .
Revised: where N j evaluations are available for task j and y ( j ) i denotes the evaluation of x ( j ) i on task j .

Original: Learning Defaults.
Revised: Learning Defaults.

Original: To learn a set of n good default models for unseen datasets in the singleobjective setting, Pﬁsterer et al.
Revised: To learn a set of n good default models for unseen datasets in the singleobjective setting, Pﬁsterer et al.

Original: (2021) propose to pick the models that minimize the joint error obtained when evaluating them on all datasets.
Revised: (2021) propose to pick the models that minimize the joint error obtained when evaluating them on all datasets.

Original: This amount to pick the set of models { x 1 , . . .
Revised: This amount to pick the set of models { x 1 , . . .

Original: , x n } ⊂ X which minimize the following objective:
Revised: , x n } ⊂ X which minimize the following objective:

Original: [Equation]
Revised: [Equation]

Original: where k denotes a single ﬁxed objective (for instance, the classiﬁcation error).
Revised: where k denotes a single ﬁxed objective (for instance, the classiﬁcation error).

Original: Since the problem is NP-complete, a greedy heuristic with a provable approximation guarantee is used: it iteratively adds the model minimizing Eq.
Revised: Since the problem is NP-complete, a greedy heuristic with a provable approximation guarantee is used: it iteratively adds the model minimizing Eq.

Original: 6 to the current selection.
Revised: 3 to the current selection.

Original: However, this approach only works for a single objective of choice f k rather than taking all objectives f (such as accuracy and latency) into account.
Revised: However, this approach only works for a single objective of choice f k rather than taking all objectives f (such as accuracy and latency) into account.

Original: In addition, it does not easily support the selection of ensembles since objectives can interact in differently when models are ensembled: while accuracy will likely increase by a small amount, the latencies of ensemble members need to be added up.
Revised: In addition, it does not easily support the selection of ensembles since objectives can interact in differently when models are ensembled: while accuracy will likely increase by a small amount, the latencies of ensemble members need to be added up.

Original: Taking into account all possible ensembles of a given size n would also blow up the combinatoric search when optimizing Eq. 6.
Revised: Taking into account all possible ensembles of a given size n would also blow up the combinatoric search when optimizing Eq. 3.

Original: Multi-Objective Optimization.
Revised: Multi-Objective Optimization.

Original: In multi-objective optimization, there generally exists no singular solution.
Revised: In multi-objective optimization, there generally exists no singular solution.

Original: Thus, when considering the objective function f , there exists no single optimal model, but rather a set of optimal models (Emmerich & Deutz, 2018).
Revised: Thus, when considering the objective function f , there exists no single optimal model, but rather a set of optimal models (Emmerich & Deutz, 2018).

Original: A model x ∈ X dominates another model x ′ ∈ X ( x ≺ f x ′ )
Revised: A model x ∈ X dominates another model x ′ ∈ X ( x ≺ f x ′ )

Original: if f i ( x ) ≤ f i ( x ′ ) for all i and there exists some i such that f i ( x ) <
Revised: if f i ( x ) ≤ f i ( x ′ ) for all i and there exists some i such that f i ( x ) <

Original: f i ( x ′ ) .
Revised: f i ( x ′ ) .

Original: The set of all non-dominated solutions (i.e. models) is denoted as the Pareto front P f ( X ) whose members are commonly referred to as Pareto-optimal solutions:[Equation]
Revised: The set of all non-dominated solutions (i.e. models) is denoted as the Pareto front P f ( X ) whose members are commonly referred to as Pareto-optimal solutions:

Original: [Equation]
Revised: [Equation]

Original: To quantify the quality of a set S = { x 1 , . .
Revised: To quantify the quality of a set S = { x 1 , . .

Original: , x n } ⊂ X of selected models, we use the hypervolume error (Zitzler & Thiele, 1998; Li & Yao, 2019).
Revised: , x n } ⊂ X of selected models, we use the hypervolume error (Zitzler & Thiele, 1998; Li & Yao, 2019).

Original: Given a set of points Y ⊂ R m and a reference point r ∈ R m , we ﬁrst deﬁne the hypervolume as the Lebesgue measure Λ of the dominated space:
Revised: Given a set of points Y ⊂ R m and a reference point r ∈ R m , we ﬁrst deﬁne the hypervolume as the Lebesgue measure Λ of the dominated space:

Original: [Equation]
Revised: [Equation]

Original: In turn, the hypervolume error ε of the set S is deﬁned as the difference in hypervolume compared to the true Pareto front P f ( X ) :
Revised: In turn, the hypervolume error ε of the set S is deﬁned as the difference in hypervolume compared to the true Pareto front P f ( X ) :

Original: [Equation]
Revised: [Equation]

Original: Thus, the hypervolume error ε ( S ) reaches its minimum when S ⊇ P f ( X ) .
Revised: Thus, the hypervolume error ε ( S ) reaches its minimum when S ⊇ P f ( X ) .

Original: To account for different scales in the optimization objectives, we further standardize all objectives by quantile normalization (Bolstad et al., 2003)
Revised: To account for different scales in the optimization objectives, we further standardize all objectives by quantile normalization (Bolstad et al., 2003)

Original: such that they follow a uniform distribution U (0 , 1) — this is feasible as X is ﬁnite.
Revised: such that they follow a uniform distribution U (0 , 1) — this is feasible as X is ﬁnite.

Original: While the normalization makes our comparisons robust to monotonic transformations of the objectives (Binois et al., 2020), it also allows us to choose the reference point for Eq. 8 as r = 1 m .
Revised: While the normalization makes our comparisons robust to monotonic transformations of the objectives (Binois et al., 2020), it also allows us to choose the reference point for Eq. 5 as r = 1 m .

Original: Learning Multi-Objective Defaults.
Revised: Learning Multi-Objective Defaults.

Original: Using the hypervolume error, we can extend the minimization problem of Eq.
Revised: Using the hypervolume error, we can extend the minimization problem of Eq.

Original: 6 to learn defaults in the multi-objective setting.
Revised: 3 to learn defaults in the multi-objective setting.

Original: For this, we propose the following minimization problem:
Revised: For this, we propose the following minimization problem:

Original: [Equation]
Revised: [Equation]

Original: In words, we seek to ﬁnd a set of complementary model conﬁgurations { x 1 , . . .
Revised: In words, we seek to ﬁnd a set of complementary model conﬁgurations { x 1 , . . .

Original: , x n } ⊂ X that provide a good approximation of the Pareto front P f ( j ) ( X ) across tasks j ∈ { 1 , . .. , M } .
Revised: , x n } ⊂ X that provide a good approximation of the Pareto front P f ( j ) ( X ) across tasks j ∈ { 1 , . . , T } .

Original: 4. P ARETO S ELECT
Revised: 4. P ARETO S ELECT

Original: We now introduce P ARETO S ELECT which tackles the minimization of Eq. 10.
Revised: We now introduce P ARETO S ELECT which tackles the minimization of Eq. 7.

Original: On a high-level,
Revised: On a high-level,

Original: P ARETO S ELECT works as follows:
Revised: P ARETO S ELECT works as follows:

Original: ﬁrst, it ﬁts a parametric surrogate model ˜ f
Revised: ﬁrst, it ﬁts a parametric surrogate model ˜ f

Original: θ that predicts the performances of model conﬁgurations by leveraging the evaluations of D .
Revised: θ that predicts the performances of model conﬁgurations by leveraging the evaluations of D .

Original: Then, it uses the surrogate ˜ f
Revised: Then, it uses the surrogate ˜ f

Original: θ to estimate the objectives for all models in X and applies the non-dominated sorting algorithm on the objectives to select a list of default models.
Revised: θ to estimate the objectives for all models in X and applies the non-dominated sorting algorithm on the objectives to select a list of default models.

Original: The surrogate model ˜ f θ is trained by attempting to correctly rank model conﬁgurations within each task in the available ofﬂine evaluations D .
Revised: The surrogate model ˜ f θ is trained by attempting to correctly rank model conﬁgurations within each task in the available ofﬂine evaluations D .

Original: For each model x ∈ X , ˜ f θ outputs a vector ˜
Revised: For each model x ∈ X , ˜ f θ outputs a vector ˜

Original: y ∈
Revised: y ∈

Original: R m for m optimization objectives.
Revised: R m for m optimization objectives.

Original: Models can then be ranked for each objective independently.
Revised: Models can then be ranked for each objective independently.

Original: In order to ﬁt ˜ f θ on the data D , we use listwise ranking (Xia et al., 2008) and apply linear discounting to focus on correctly identifying the top conﬁgurations similar to Zügner et al.
Revised: In order to ﬁt ˜ f θ on the data D , we use listwise ranking (Xia et al., 2008) and apply linear discounting to focus on correctly identifying the top conﬁgurations similar to Zügner et al.

Original: The corresponding loss function can be found in Appendix D.
Revised: The corresponding loss function can be found in Appendix D.

Original: For the parametric surrogate model ˜ f
Revised: For the parametric surrogate model ˜ f

Original: θ , we choose a simple MLP similar to Salinas et al.
Revised: θ , we choose a simple MLP similar to Salinas et al.

Original: (2020b).
Revised: (2020b).

Original: We use two hidden layers of size 32 with LeakyReLU activations after each hidden layer and apply a weight decay of 0.01 as regularization.
Revised: We use two hidden layers of size 32 with LeakyReLU activations after each hidden layer and apply a weight decay of 0.01 as regularization.

Original: Importantly, we do not tune these hyperparameters.
Revised: Importantly, we do not tune these hyperparameters.

Original: Predictive performances for every model can eventually be computed as
Revised: Predictive performances for every model can eventually be computed as

Original: [Equation]
Revised: [Equation]

Original: We note that for minimizing Eq.
Revised: We note that for minimizing Eq.

Original: 10, there is no difference in performing ranking or regression: since we apply quantile normalization to compute the hypervolume, the magnitude of the values predicted by ˜ f θ is irrelevant.
Revised: 7, either regression or ranking objectives can be used: since we apply quantile normalization to compute the hypervolume, the magnitude of the values predicted by ˜ f θ is irrelevant.

Original: However, we ﬁnd the ranking setting to be superior to regression for ﬁnding good default models (see Appendix D for more details).
Revised: However, we ﬁnd the ranking setting to be superior to regression for ﬁnding good default models (see Appendix D for more details).

Original: Multi-Objective Sorting.
Revised: Multi-Objective Sorting.

Original: The “best” conﬁgurations are then determined by applying the nondominated sorting algorithm (NDSA) — an extension of sorting to the multi-dimensional setting (Srinivas & Deb, 1994; Emmerich & Deutz, 2018) — to the predictive performances ˜ Y .
Revised: The “best” conﬁgurations are then determined by applying the nondominated sorting algorithm (NDSA) — an extension of sorting to the multi-dimensional setting (Srinivas & Deb, 1994; Emmerich & Deutz, 2018) — to the predictive performances ˜ Y .

Original: Figureprovides an illustration of the sorting procedure.
Revised: Figureprovides an illustration of the sorting procedure.

Original: NDSA can be described as a two-level procedure.
Revised: NDSA can be described as a two-level procedure.

Original: First, it partitions the conﬁgurations to be sorted in multiple layers by iteratively computing the
Revised: First, it partitions the conﬁgurations to be sorted in multiple layers by iteratively computing the

Original: Pareto fronts, then it applies a sorting function sort to rank the conﬁgurations in each layer:
Revised: Pareto fronts, then it applies a sorting function sort to rank the conﬁgurations in each layer:

Original: [Equation]
Revised: [Equation]

Revised: We choose sort to compute an ϵ -net

Revised: (Clarkson, 2006) for which there exists a simple greedy algorithm.

Revised: When sorting a set X , the ϵ -net sorting operation yields an order sort ( X ) =

Original: [Figure]
Revised: [ x 1 , . . . , x | X | ] . The ﬁrst element x 1 is chosen randomly from X and x i +1 is deﬁned iteratively as the item which is farthest from the current selection in terms of Euclidean distance, formally: [Equation]

Revised: The ﬁnal ordering is eventually obtained by concatenating the sorted layers.

Revised: While previous work followed different approaches for the sort operation in the NDSA algorithm (Srinivas & Deb, 1994; Deb et al., 2002), we leverage the ϵ -net since Salinas et al.

Revised: (2021) highlighted its theoretical guarantees and Schmucker et al.

Revised: (2021) provided empirical evidence for its good performance.

Revised: The pseudo-code of P ARETO S ELECT is given in Algorithm 1 in the appendix.

Revised: In the case where predictions of the surrogate are error-free, the recommendations of the algorithm can be guaranteed to be perfect with zero hypervolume error.

Original: [Figure]
Revised: [Figure]

Original: Layer
Revised: Layer

Original: [Figure] [Figure] 
Revised: [Figure] [Figure] [Figure]

Original: Layer  

Revised: Layer

Original: Objective #
Revised: Objective #

Original: Layer
Revised: Layer

Original: Objective #
Revised: Objective #

Original: Layer
Revised: Layer

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: choose to compute sort ( X 
Revised: Proposition 1. Assume that n ≥ |P f ( X ) | and for all x ∈ X , ˜ f

Original: = [ x 
Revised: θ ( x ) = f

Original: X | ] . {+Then, ε ( { x 1 , . . . , x n } ) = 
+}
Revised: ( x ) . {+Then, ε ( { x 1 , . . . , x n } ) = 
+}

Original: The ﬁrst element x 1 is chosen randomly from X and x i +1 is deﬁned iteratively as the item which is farthest from the current selection in terms of Euclidean distance, formally:
Revised: where x 1 , . . , x n are the ﬁrst n models selected by our method. Proof. By Eq.

Original: [Equation] The ﬁnal ordering is eventually obtained by concatenating the sorted layers.
Revised: 9, the ﬁrst k recommendations of Pareto Select are such that { x 1 , . . . , x k } = P ˜ f θ ( X ) where k = |P

Original: While previous work followed different approaches for the sort operation in the NDSA algorithm (Srinivas & Deb, 1994; Deb et al., 2002), we leverage the ϵ -net since Salinas et al.
Revised: ˜ f θ ( X ) | . Since for all x ∈ X , ˜ f θ ( x ) = f

Original: (2021) highlighted its theoretical guarantees and Schmucker et al.
Revised: ( x ) , we get that P ˜ f θ ( X ) = P f ( X ) . It follows that for any k ≤ n , P f ( X ) = { x 1 , . . .

Original: (2021) provided empirical evidence for its good performance in practice.{+, x k } ⊂ { x 1 , . . . , x n } and consequently ε ( { x 1 , . . . , x n } ) = 
+}
Revised: {+, x k } ⊂ { x 1 , . . . , x n } and consequently ε ( { x 1 , . . . , x n } ) = 
+}

Original: The pseudocode of P ARETO S ELECT is given in Algorithm 1 in the appendix.
Revised: since the hypervolume error of any set of points containing the Pareto front is zero.

Original: 4. R ESULTS
Revised: 4. R ESULTS

Original: In order to evaluate our approach, we leverage the data collected with the benchmark presented in
Revised: In order to evaluate our approach, we leverage the data collected with the benchmark presented in

Original: Section 3.
Revised: Section 3.

Original: For evaluation, we perform leave-one-out-cross-validation where we use each dataset one after the other as the test dataset and estimate the parameters θ of our parametric surrogate model ˜ f
Revised: For evaluation, we perform leave-one-out-cross-validation where we use each dataset one after the other as the test dataset and estimate the parameters θ of our parametric surrogate model ˜ f

Original: θ on the remaining 43 datasets.
Revised: θ on the remaining 43 datasets.

Original: The multi-objective setting in the following experiments aims to minimize latency as well as nCRPS.
Revised: The multi-objective setting in the following experiments aims to minimize latency as well as nCRPS.

Original: In Figure 4, we report the average hypervolume obtained for P ARETO S ELECT and several baselines, averaged over all 44 test datasets.
Revised: In Figure 4, we report the average hypervolume obtained for P ARETO S ELECT and several baselines, averaged over all 44 test datasets.

Original: P ARETO S ELECT picks the top elements of the non-dominated sort on the surrogate’s predictive performances.
Revised: P ARETO S ELECT picks the top elements of the non-dominated sort on the surrogate’s predictive performances.

Original: As baselines, we consider a variant of P ARETO S ELECT which considers the nCRPS as the single objective ( Single-Objective ), the approach of Pﬁsterer et al.
Revised: As baselines, we consider a variant of P ARETO S ELECT which considers the nCRPS as the single objective ( Single-Objective ), the approach of Pﬁsterer et al.

Original: (2021) which greedily picks models to minimize the joint nCRPS on the training data ( Greedy
Revised: (2021) which greedily picks models to minimize the joint nCRPS on the training data ( Greedy

Original: Model-Free ), and random search over the entire model space of size |X| =
Revised: Model-Free ), and random search over the entire model space of size |X| =

Original: 247 ( Random ).
Revised: 247 ( Random ).

Original: The hypervolume obtained by P ARETO S ELECT with 10 default models is small ( ≈ 0 . 06 ) compared to the best baseline that reaches ≈ 0 . 20 .
Revised: The hypervolume obtained by P ARETO S ELECT with 10 default models is small ( ≈ 0 . 06 ) compared to the best baseline that reaches ≈ 0 . 20 .

Original: Further, random search requires a total of 27 models to be on par with the hypervolume 
Revised: Further, random search requires a total of 27 models to be on par with the hypervolume of only 10 models selected via our method.

Original: [Figure]
Revised: [Figure]

Original: Figure 5 depicts the top n = 10 recommendations produced by P ARETO S ELECT when predicting default models on a single dataset.
Revised: Figure 5 depicts the top n = 10 recommendations produced by P ARETO S ELECT when predicting default models on a single dataset.

Original: The default models cover the true Pareto front well: our method is able to pick models with low latency, low nCRPS, and a good trade-off between these two objectives.
Revised: The default models cover the true Pareto front well: our method is able to pick models with low latency, low nCRPS, and a good trade-off between these two objectives.

Original: 4. M ULTI -O BJECTIVE E NSEMBLING
Revised: 4. M ULTI -O BJECTIVE E NSEMBLING

Original: As shown in Section 3.5, ensembling models yields signiﬁcant gains in accuracy but comes at the cost of considerable latency.
Revised: As shown in Section 3.5, ensembling models yields signiﬁcant gains in accuracy but comes at the cost of considerable latency.

Original: Since a massively large ensemble model is of no practical use, it is critical to be able to select an ensemble with a latency that is acceptable for an application.
Revised: Since a massively large ensemble model is of no practical use, it is critical to be able to select an ensemble with a latency that is acceptable for an application.

Original: Thus, we consider ensembles under different latency constraints.
Revised: Thus, we consider ensembles under different latency constraints.

Original: For a latency constraint c , we build an ensemble with n ≤ 10 members x 1 , . . .
Revised: For a latency constraint c , we build an ensemble with n ≤ 10 members x 1 , . . .

Original: , x n by iteratively picking the model with best nCRPS such that the ensemble latency constrain c is still met.
Revised: , x n by iteratively picking the model with best nCRPS such that the ensemble latency constrain c is still met.

Original: To generate ensemble predictions, we combine the forecasts of the base models x 1 , . . .
Revised: To generate ensemble predictions, we combine the forecasts of the base models x 1 , . . .

Original: , x n by averaging all 10-quantiles { 0 . 1 , . . . , 0 . 9 } for each time series and time step independently.
Revised: , x n by averaging all 10-quantiles { 0 . 1 , . . . , 0 . 9 } for each time series and time step independently.

Original: Albeitit can occur that “good” models with low nCRPS are combined with “bad” models having a high nCRPS due to the greedy selection of the ensemble members, ensembles did not beneﬁt from weighing the members’ forecasts non-uniformly in our experiments.
Revised: Albeit the performances of the base models might differ signiﬁcantly, our experiments showed that more complicated, non-uniform weighting schemes yielded less accurate forecasts on average. This aligns with current literature showing that simple averaging is often most effective (Petropoulos et al., 2020).

Original: Figure 6 shows the performance of different latency-constrained ensembles, relative to the performance of an unconstrained ensemble that picks the n = 10 models with lowest predicted nCRPS.
Revised: Figure 6 shows the performance of different latency-constrained ensembles, relative to the performance of an unconstrained ensemble that picks the n = 10 models with lowest predicted nCRPS.

Original: As expected, higher latency constraints c allows the ensemble to perform better.
Revised: As expected, higher latency constraints c allows the ensemble to perform better.

Original: For c ≥ 100 ms, the constrained ensemble recover the Pareto front and outperforms all individual models x ∈ X as well as ensembles.
Revised: For c ≥ 100 ms, the constrained ensemble recovers the Pareto front and outperforms all individual models x ∈ X as well as ensembles.

Original: The small gap to the Pareto front for constraints c ≤ 50 ms can be attributed to multiple effects.
Revised: The small gap to the Pareto front for constraints c ≤ 50 ms can be attributed to multiple effects.

Original: First, the nCRPS values predicted by the surrogate ˜ f θ are not optimal.
Revised: First, the nCRPS values predicted by the surrogate ˜ f θ are not optimal.

Original: Second, models may not satisfy c on every dataset although they have a lower latency on average (e.g. the default TFT conﬁguration violates c = 2.5 ms on 17/44 datasets).
Revised: Second, models may not satisfy c on every dataset although they have a lower latency on average (e.g. the default TFT conﬁguration violates c = 2.5 ms on 17/44 datasets).

Original: Table 1 additionally reports performance metrics of latency-constrained ensembles. When considering the average nCRPS rank across datasets, the ensemble for c = 10 ms outperforms all individual base models  while the ensemble for c = 100 ms outperforms all ensembles as well.
Revised: Table 1 additionally reports performance metrics of latency-constrained ensembles: for instance, the ensemble for c = 10 ms outperforms all individual base models in terms of average nCRPS rank while the ensemble for c = 100 ms outperforms all hyper-ensembles as well.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we have presented (i) a new benchmark for evaluating forecasting methods in terms of multiple metrics and (ii) P ARETO S ELECT , a new algorithm that can learn default model conﬁgurations in a multi-objective setting.
Revised: In this work, we have presented (i) a new benchmark for evaluating forecasting methods in terms of multiple metrics and (ii) P ARETO S ELECT , a new algorithm that can learn default model conﬁgurations in a multi-objective setting.

Original: In future work, we will consider applying those techniques to other domains such as computer vision where ofﬂine evaluations of multiple datasets were recently made available by Dong & Yang (2020).
Revised: In future work, we will consider applying those techniques to other domains such as computer vision where ofﬂine evaluations of multiple datasets were recently made available by Dong & Yang (2020).

Original: Additionally, future research may extend the presented predictive surrogate model to the case of ensembles: this would enable P ARETO S ELECT to be used directly to select members for ensembles of any latency.
Revised: Additionally, future research may extend the presented predictive surrogate model to the case of ensembles: this would enable P ARETO S ELECT to be used directly to select members for ensembles of any latency.

Original:  R EPRODUCIBILITY
Revised:  R EPRODUCIBILITY

Original: Public datasets and method implementations were used to ensure the reproducibility of our experiments.
Revised: Public datasets and method implementations were used to ensure the reproducibility of our experiments.

Original: In Appendix A, we provide all hyperparameters used for training the benchmark forecasting models via GluonTS.
Revised: In Appendix A, we provide all hyperparameters used for training the benchmark forecasting models via GluonTS.

Original: The source and processing of each dataset as well as the procedure for generating data splits is detailed in Appendix B. We also provide details on the surrogate model used and a comparison with other choices in Appendix D.
Revised: The source and processing of each dataset as well as the procedure for generating data splits is detailed in Appendix B. We also provide details on the surrogate model used and a comparison with other choices in Appendix D.

Original: The pseudocode of P ARETO S ELECT is given in Appendix E.
Revised: The pseudocode of P ARETO S ELECT is given in Appendix E.

Original: As a source of truth, we share the entire code used to run the benchmark and its evaluation with the submission.
Revised: As a source of truth, we share the entire code used to run the benchmark and its evaluation with the submission.

Original: This code as well as the generated evaluations will be released with the paper.
Revised: This code as well as the generated evaluations will be released with the paper.

Original: A M ODELS
Revised: A M ODELS

Original: Table 2 provides an overview over all benchmark models along with hyperparameters considered.
Revised: Table 2 provides an overview over all benchmark models along with hyperparameters considered.

Original: In addition to three hyperparameter settings per deep learning model (except for MQ-RNN), three different context lengths l are considered.
Revised: In addition to three hyperparameter settings per deep learning model (except for MQ-RNN), three different context lengths l are considered.

Original: These represent multiples of the forecast horizon τ and are thus dataset-dependent.
Revised: These represent multiples of the forecast horizon τ and are thus dataset-dependent.

Original: Eventually, this results in 9 model conﬁgurations per deep learning model (and 3 for MQ-RNN).
Revised: Eventually, this results in 9 model conﬁgurations per deep learning model (and 3 for MQ-RNN).

Original: The implementation of all models are taken from GluonTS (Alexandrov et al., 2020).
Revised: The implementation of all models are taken from GluonTS (Alexandrov et al., 2020).

Original: All deep learning models are implemented in MXNet (Chen et al., 2015).
Revised: All deep learning models are implemented in MXNet (Chen et al., 2015).

Original: ARIMA, ETS, STL-AR, and Theta are available in GluonTS as well but forward computations to the R forecast package (Hyndman et al., 2021). 
Revised: ARIMA, ETS, STL-AR, and Theta are available in GluonTS as well but forward computations to the R forecast package (Hyndman et al., 2021). Model Evaluation.

Revised: To measure the accuracy of forecasting models, we employ the continuous ranked probability score (CRPS) (Matheson & Winkler, 1976; Gneiting & Raftery, 2007).

Original: B D ATASETS
Revised: Given the quantile function ( F ( i ) t ) − 1 of the forecast probability distribution in Eq. (1) for a time series z ( i ) at time step t , the CRPS is deﬁned as [Equation]

Revised: where Λ α is the quantile loss (or pinball loss) deﬁned over a quantile level 0 < α < 1 :

Revised: [Equation]

Revised: Here, I is the indicator function.

Revised: To compute an aggregated score for a model forecasting multiple time series and time steps, we compute a normalized CRPS (nCRPS):

Revised: [Equation]

Revised: We approximate the nCRPS using the 10-quantiles α ∈ { 0 . 1 , 0 . 2 , 0 . 3 , . . .

Revised: While we focus on nCRPS for the evaluations in this paper, the benchmark dataset that we release contains four additional forecast accuracy metrics (MAPE, sMAPE, NRMSE, ND).

Revised: B D ATASETS

Original: Our benchmark provides 44 datasets which were obtained from multiple sources.
Revised: Our benchmark provides 44 datasets which were obtained from multiple sources.

Original: 16 datasets were obtained though GluonTS (Alexandrov et al., 2020), 24 datasets are taken from the Monash Time
Revised: 16 datasets were obtained though GluonTS (Alexandrov et al., 2020), 24 datasets are taken from the Monash Time

Original: [Table]

Original: Series Forecasting Repository (Godahewa et al., 2021), and the remaining 4 datasets were originally published as part of Kaggle 1 forecasting competitions.
Revised: Series Forecasting Repository (Godahewa et al., 2021), and the remaining 4 datasets were originally published as part of Kaggle 2 forecasting competitions.

Original: Table 3 provides basic statistics about all datasets.
Revised: Table 3 provides basic statistics about all datasets.

Original: B. D ATASET D ESCRIPTIONS
Revised: B. D ATASET D ESCRIPTIONS

Original: In the following [Table] we provide brief descriptions of all datasets and provide their sources.
Revised: In the following, we provide brief descriptions of all datasets and provide their sources.

Original: We link the original source (if possible) along with any work which pre-processed the data (if applicable)
Revised: We link the original source (if possible) along with any work which pre-processed the data (if applicable).

Original: [Table] The datasets “Corporación  by ourselves
Revised: The datasets “Corporación Favorita”, “Restaurant”, “Rossmann”, and “Walmart” — which were obtained from Kaggle — are processed by ourselves.

Original: Australian Electricity Demand (O’Hara-Wild et al., 2021; Godahewa et al., 2021) provides the 30minute electricity demand of ﬁve Australian states (New South Wales, Queensland, South Australia,
Revised: Australian Electricity Demand (O’Hara-Wild et al., 2021; Godahewa et al., 2021) provides the 30minute electricity demand of ﬁve Australian states (New South Wales, Queensland, South Australia,

Original: Tasmania, Victoria).
Revised: Tasmania, Victoria).

Original: Time series range from January 2002 to April 2015.
Revised: Time series range from January 2002 to April 2015.

Original: Bitcoin (Godahewa et al., 2021) provides over a dozen of potential inﬂuencers of the price of Bitcoin .
Revised: Bitcoin (Godahewa et al., 2021) provides over a dozen of potential inﬂuencers of the price of Bitcoin .

Original: Among others, these inﬂuencers are daily hash rate, block size, or mining difﬁculty.
Revised: Among others, these inﬂuencers are daily hash rate, block size, or mining difﬁculty.

Original: We exclude one time series from the original dataset as its absolute values are ≥ 10 18 .
Revised: We exclude one time series from the original dataset as its absolute values are ≥ 10 18 .

Original: Data is provided from January 2009 to July 2021.
Revised: Data is provided from January 2009 to July 2021.

Original: CIF 2016 (Štˇepniˇcka & Burda, 2017; Godahewa et al., 2021) is the dataset from the “Computational
Revised: CIF 2016 (Štˇepniˇcka & Burda, 2017; Godahewa et al., 2021) is the dataset from the “Computational

Original: Intelligence in Forecasting” competition in 2016.
Revised: Intelligence in Forecasting” competition in 2016.

Original: One third of the time series originate from the banking sector while the remaining two thirds are generated artiﬁcially.
Revised: One third of the time series originate from the banking sector while the remaining two thirds are generated artiﬁcially.

Original: In the competition, the time series have two different prediction horizons, however, GluonTS forces us to adopt only a single horizon (for which we choose the most common one).
Revised: In the competition, the time series have two different prediction horizons, however, GluonTS forces us to adopt only a single horizon (for which we choose the most common one).

Original: COVID Deaths (Dong et al., 2020; Godahewa et al., 2021) provides the daily COVID-19 death counts of various countries between January 22 and August 21 in the year 2020.
Revised: COVID Deaths (Dong et al., 2020; Godahewa et al., 2021) provides the daily COVID-19 death counts of various countries between January 22 and August 21 in the year 2020.

Original: Car Parts (Hyndman, 2015; Godahewa
Revised: Car Parts (Hyndman, 2015; Godahewa

Original: et al., 2021) provides intermittent time series of car parts sold monthly by a US car company between January 1998 and April 2002.
Revised: et al., 2021) provides intermittent time series of car parts sold monthly by a US car company between January 1998 and April 2002.

Original: Periods in which no parts are sold have a value of zero.
Revised: Periods in which no parts are sold have a value of zero.

Original: Corporación Favorita (Favorita, 2017) contains the daily unit sales of over 4,000 items at 54 different stores of the supermarket company Corporación Favorita in Ecuador.
Revised: Corporación Favorita (Favorita, 2017) contains the daily unit sales of over 4,000 items at 54 different stores of the supermarket company Corporación Favorita in Ecuador

Original: Unit sales were manually set to 0 on days where there was no data available. Data ranges from January 2013 to August 2017. All covariates that are provided in the Kaggle competition are discarded.
Revised: [Table]

Original: Dominick (James M. Kilts Center, 2020; Godahewa et al., 2021) contains time series with the weekly proﬁts of numerous stock keeping units (SKUs).
Revised: Dominick (James M. Kilts Center, 2020; Godahewa et al., 2021) contains time series with the weekly proﬁts of numerous stock keeping units (SKUs).

Original: The data is obtained from the grocery store company Dominick’s over a period of seven years.
Revised: The data is obtained from the grocery store company Dominick’s over a period of seven years

Original:  Electricity (Dua & Graff, 2017; Alexandrov et al., 2020) is comprised of the hourly electricity consumption (in kWh) of hundreds of households between January 2012 and June 2014.
Revised: [Table] Electricity (Dua & Graff, 2017; Alexandrov et al., 2020) is comprised of the hourly electricity consumption (in kWh) of hundreds of households between January 2012 and June 2014.

Original: Exchange Rate (Lai et al., 2018; Alexandrov et al., 2020) provides the daily exchange rates (on weekdays) between the US dollar and the currencies of eight countries (Australia, Great Britain,
Revised: Exchange Rate (Lai et al., 2018; Alexandrov et al., 2020) provides the daily exchange rates (on weekdays) between the US dollar and the currencies of eight countries (Australia, Great Britain,

Original: Canada, Switzerland, China, Japan, New Zealand, Singapore) in the period from 1990 to 2013.
Revised: Canada, Switzerland, China, Japan, New Zealand, Singapore) in the period from 1990 to 2013.

Original: Fred-MD (McCracken & Ng, 2016; Godahewa et al., 2021) contains monthly time series of macroeconomic indicators (e.g. interest rates, employment rates, ...)
Revised: Fred-MD (McCracken & Ng, 2016; Godahewa et al., 2021) contains monthly time series of macroeconomic indicators (e.g. interest rates, employment rates, ...)

Original: from the Federal Reserve Bank since January 1959 (until September 2019).
Revised: from the Federal Reserve Bank since January 1959 (until September 2019).

Original: Hospital (Hyndman, 2015; Godahewa et al., 2021) provides the monthly patient counts for various products that are related to medical problems between 2000 and 2007.
Revised: Hospital (Hyndman, 2015; Godahewa et al., 2021) provides the monthly patient counts for various products that are related to medical problems between 2000 and 2007.

Original: KDD 2018 (Bekkerman et al., 2018; Godahewa et al., 2021) is the dataset used by the KDD Cup 2018.
Revised: KDD 2018 (Bekkerman et al., 2018; Godahewa et al., 2021) is the dataset used by the KDD Cup 2018.

Original: It contains hourly time series with air quality levels (represented by various chemical compounds) measured by stations in Beijing and London between January 2017 and April 2018.
Revised: It contains hourly time series with air quality levels (represented by various chemical compounds) measured by stations in Beijing and London between January 2017 and April 2018.

Original: London Smart Meters (Jean-Michael, 2019; Godahewa et al., 2021) was originally published on Kaggle and contains the half-hourly (electrical) energy consumption (in kWh) of thousands of households in the period from November 2011 to February 2014.
Revised: London Smart Meters (Jean-Michael, 2019; Godahewa et al., 2021) was originally published on Kaggle and contains the half-hourly (electrical) energy consumption (in kWh) of thousands of households in the period from November 2011 to February 2014.

Original: M1 (Makridakis et al., 1982; Godahewa et al., 2021) datasets are taken from the M1 competition in 1982.
Revised: M1 (Makridakis et al., 1982; Godahewa et al., 2021) datasets are taken from the M1 competition in 1982.

Original: Time series have varying start- and end dates and cover a wide semantic spectrum of domains (demographics, micro, macro, industry).
Revised: Time series have varying start- and end dates and cover a wide semantic spectrum of domains (demographics, micro, macro, industry).

Original: M3 (Makridakis & Hibon, 2000; Alexandrov et al., 2020) datasets are obtained from the M3 competition.
Revised: M3 (Makridakis & Hibon, 2000; Alexandrov et al., 2020) datasets are obtained from the M3 competition.

Original: Across frequencies, time series were collected from six different domains: demographics, micro, macro, industry, ﬁnance, and other.
Revised: Across frequencies, time series were collected from six different domains: demographics, micro, macro, industry, ﬁnance, and other.

Original: For the time series with no speciﬁed frequency, we assume a quarterly frequency (as this is the default 
Revised: For the time series with no speciﬁed frequency, we assume a quarterly frequency (as this is the default in GluonTS).

Original:  M5 (Makridakis et al., 2020a; Alexandrov et al., 2020) is the dataset from the M5 competition and contains daily unit sales of 3,049 products across 10 Walmart stores located in California, Texas, and Wisconsin.
Revised: M4 (Makridakis et al., 2020b; Alexandrov et al., 2020) datasets are taken from the prestigious M competition. The time series are collected from the same domains as in the M3 competition. M5 (Makridakis et al., 2020a; Alexandrov et al., 2020) is the dataset from the M5 competition and contains daily unit sales of 3,049 products across 10 Walmart stores located in California, Texas, and Wisconsin.

Original: While covariates are available (e.g. product prices or special events), none of the forecasting methods we consider makes use of them.
Revised: While covariates are available (e.g. product prices or special events), none of the forecasting methods we consider makes use of them.

Original: Sales data is provided from January 2011 to
Revised: Sales data is provided from January 2011 to

Original: April 2016.
Revised: April 2016.

Original: NN5 (Taieb et al., 2012; Godahewa et al., 2021) data was used in the NN5 competition run in 2008.
Revised: NN5 (Taieb et al., 2012; Godahewa et al., 2021) data was used in the NN5 competition run in 2008.

Original: The time series provide daily amounts of cash withdrawals at dozens of ATMs in the UK from March 1996 to May 1998.
Revised: The time series provide daily amounts of cash withdrawals at dozens of ATMs in the UK from March 1996 to May 1998.

Original: Pedestrian Count (City of Melbourne, 2017; Godahewa et al., 2021) provides hourly pedestrian counts in Melbourne from May 2009 to May 2020.
Revised: Pedestrian Count (City of Melbourne, 2017; Godahewa et al., 2021) provides hourly pedestrian counts in Melbourne from May 2009 to May 2020.

Original: The counts are captured by sensors scattered in the city.
Revised: The counts are captured by sensors scattered in the city.

Original: Restaurant (Holdings, 2017) provides the number of daily visitors of hundreds of restaurants in
Revised: Restaurant (Holdings, 2017) provides the number of daily visitors of hundreds of restaurants in

Original: Japan between January 2016 and April 2017, as tracked by the AirREGI system.
Revised: Japan between January 2016 and April 2017, as tracked by the AirREGI system.

Original: As dates for which restaurants are closed do not provide visitor numbers, we impute zeros for these missing values.
Revised: As dates for which restaurants are closed do not provide visitor numbers, we impute zeros for these missing values.

Original: The Kaggle competition where this dataset is obtained from also provides data about reservations but this data remains unused for our purposes.
Revised: The Kaggle competition where this dataset is obtained from also provides data about reservations but this data remains unused for our purposes.

Original: Rideshare (Godahewa et al., 2021) is comprised of hourly time series representing various attributes related to services offered by Uber in Lyft in New York.
Revised: Rideshare (Godahewa et al., 2021) is comprised of hourly time series representing various attributes related to services offered by Uber in Lyft in New York.

Original: Time series include e.g. minimum, maximum, and average price, or the maximum distance traveled — grouped by starting location and provider.
Revised: Time series include e.g. minimum, maximum, and average price, or the maximum distance traveled — grouped by starting location and provider.

Original: Data is available for the period of a month, starting in late November 2018.
Revised: Data is available for the period of a month, starting in late November 2018.

Original: Rossmann (Rossmann, 2015) provides daily sales counts at hundreds of different stores of the Rossmann drug store chain.
Revised: Rossmann (Rossmann, 2015) provides daily sales counts at hundreds of different stores of the Rossmann drug store chain.

Original: Just like for the M5 dataset, covariates are available (e.g. distance to competition, special events, or discounts) but not used by any method.
Revised: Just like for the M5 dataset, covariates are available (e.g. distance to competition, special events, or discounts) but not used by any method.

Original: Data is available from January 2013 until August 2015.
Revised: Data is available from January 2013 until August 2015.

Original: San Francisco Trafﬁc (Caltrans, 2020; Godahewa et al., 2021) contains hourly occupancy rates of freeways in the San Francisco Bay area.
Revised: San Francisco Trafﬁc (Caltrans, 2020; Godahewa et al., 2021) contains hourly occupancy rates of freeways in the San Francisco Bay area.

Original: Data is available for two full years, starting from January 1, 2015.
Revised: Data is available for two full years, starting from January 1, 2015.

Original: Solar (Zhang, 2006; Alexandrov et al., 2020) is comprised of the hourly power consumption of dozens of photovoltaic power stations in the state of Alabama in the year 2006.
Revised: Solar (Zhang, 2006; Alexandrov et al., 2020) is comprised of the hourly power consumption of dozens of photovoltaic power stations in the state of Alabama in the year 2006.

Original: Taxi (NYC Taxi and Limousine Commission, 2015; Salinas et al., 2019; Alexandrov et al., 2020) contains the number of taxi rides in hundreds of locations around New York in 30 minute windows.
Revised: Taxi (NYC Taxi and Limousine Commission, 2015; Salinas et al., 2019; Alexandrov et al., 2020) contains the number of taxi rides in hundreds of locations around New York in 30 minute windows.

Original: Training data contains data from January 2015 while test data contains data from January in the subsequent year.
Revised: [Table]

Original: Temperature Rain (Godahewa et al., 2021) provides the daily temperature observations and rain forecasts gather from 422 weather stations across Australia.
Revised: Temperature Rain (Godahewa et al., 2021) provides the daily temperature observations and rain forecasts gather from 422 weather stations across Australia.

Original: Data ranges from May 2015 through
Revised: Data ranges from May 2015 through

Original: April 2017.
Revised: April 2017.

Original: Tourism (Athanasopoulos et al., 2011; Godahewa et al., 2021)
Revised: Tourism (Athanasopoulos et al., 2011; Godahewa et al., 2021)

Original: datasets come from a Kaggle forecasting competition where time series were supplied by tourism bodies from Australia, Hong Kong, and New Zealand as well as academics.
Revised: datasets come from a Kaggle forecasting competition where time series were supplied by tourism bodies from Australia, Hong Kong, and New Zealand as well as academics.

Original: Monthly data ranges from 1979 through 2007, quarterly data from from 1975 through 2007, and yearly data from 1960 through 2008.
Revised: Monthly data ranges from 1979 through 2007, quarterly data from from 1975 through 2007, and yearly data from 1960 through 2008.

Original: Vehicle Trips (Flowers & Fischer-Baum, 2015; Godahewa et al., 2021) contains the daily number of trips served by hundreds of for-hire vehicle (FHV) companies (such as Uber ).
Revised: Vehicle Trips (Flowers & Fischer-Baum, 2015; Godahewa et al., 2021) contains the daily number of trips served by hundreds of for-hire vehicle (FHV) companies (such as Uber ).

Original: Data ranges from
Revised: Data ranges from

Original: January to September 2015.
Revised: January to September 2015.

Original: Walmart (Walmart, 2014) contains data about Walmart store sales similarly to the M5 competition.
Revised: Walmart (Walmart, 2014) contains data about Walmart store sales similarly to the M5 competition.

Original: Data is provided as the weekly sales (in USD) across 45 stores and 81 departments in different regions and ranges from February 2010 to November 2012.
Revised: Data is provided as the weekly sales (in USD) across 45 stores and 81 departments in different regions and ranges from February 2010 to November 2012.

Original: Just like for many of the other Kaggle competitions, covariates are available, yet unused.
Revised: Just like for many of the other Kaggle competitions, covariates are available, yet unused.

Original: Weather (Sparks, 2021; Godahewa et al., 2021) is comprised of daily data collected from hundreds of weather stations in Australia.
Revised: Weather (Sparks, 2021; Godahewa et al., 2021) is comprised of daily data collected from hundreds of weather stations in Australia.

Original: Collected data is the amount of rain, the minimum and maximum temperature, and the amount of solar radiation.
Revised: Collected data is the amount of rain, the minimum and maximum temperature, and the amount of solar radiation.

Original: Wiki (Gasthaus et al., 2019; Alexandrov et al., 2020) similarly provides daily pages views for several thousand Wikipedia pages in the period from January 2012 to July 2014.
Revised: Wiki (Gasthaus et al., 2019; Alexandrov et al., 2020) similarly provides daily pages views for several thousand Wikipedia pages in the period from January 2012 to July 2014.

Original: Wind Farms (AEMO, 2020; Godahewa et al., 2021) contains time series with the minutely power production of hundreds of wind farms in Australia.
Revised: Wind Farms (AEMO, 2020; Godahewa et al., 2021) contains time series with the minutely power production of hundreds of wind farms in Australia.

Original: Data is available for a period of one year, starting in August 2019.
Revised: Data is available for a period of one year, starting in August 2019.

Original: B. D ATA P REPARATION
Revised: B. D ATA P REPARATION

Original: For all of the datasets in Table 3 except for “Taxi”, we perform similar preprocessing.
Revised: For all of the datasets in Table 3 except for “Taxi”, we perform similar preprocessing.

Original: Let
Revised: Let

Original: Z = { z ( i ) 1: T i } Ki =1 be the set of K univariate time series of lengths T i .
Revised: Z = { z ( i ) 1: T i } Ki =1 be the set of K univariate time series of lengths T i .

Original: Then, we run the following preprocessing steps:
Revised: Then, we run the following preprocessing steps:

Original: Remove all time series which are constant to always be able to compute the MASE metric.
Revised: Remove all time series which are constant to always be able to compute the MASE metric.

Original: [Equation]
Revised: [Equation]

Original: Remove all time series which are too short.
Revised: Remove all time series which are too short.

Original: We exclude all time series of length
Revised: We exclude all time series of length

Original: T ≤ ( p + 1) τ where τ is the prediction horizon and p is a dataset-speciﬁc integer which governs over how many prediction lengths we want to perform testing 2 . 
Revised: T ≤ ( p + 1) τ where τ is the prediction horizon and p is a dataset-speciﬁc integer which governs over how many prediction lengths we want to perform testing 3 . We are using

Original: We are using p = 1 for all datasets but “Electricity” ( p = 7 ), “Exchange Rate” ( p = 3 ), “Solar” ( p =4 ), “Trafﬁc” ( p = 7 ) and “Wiki” ( p = 3 ).
Revised: p = 1 for all datasets but “Electricity” ( p = 7 ), “Exchange Rate” ( p = 3 ), “Solar” ( p =

Revised: 4 ), “Trafﬁc”

Revised: p

Original:  [Equation]
Revised: = 7 ) and “Wiki” ( p = [Equation]

Original: Split time series into training validation and test data.
Revised: Split time series into training, validation and test data.

Original: For

Original:  each time series, we split off the prediction horizon p − 1 times to obtain time series for the test data.
Revised: For each time series, we split off the prediction horizon p − 1 times to obtain time series for the test data.

Original: Then, we split off the prediction horizon another time to obtain a validation time series per training time series
Revised: Then, we split off the prediction horizon another time to obtain a validation time series per training time series.

Original: Eventually, we construct the following sets:
Revised: Eventually, we construct the following sets:

Original: [Equation]
Revised: [Equation]

Original: For the “Taxi” dataset, we need to slightly augment the preprocessing to allow for discontinuities in the available time series.
Revised: For the “Taxi” dataset, we need to slightly augment the preprocessing to allow for discontinuities in the available time series.

Original: Essentially, each time series z ∈ Z consists of an old part z old and a new part z new (data throughout January of two successive years).
Revised: Essentially, each time series z ∈ Z consists of an old part z old and a new part z new (data throughout January of two successive years).

Original: We then obtain the training data from z old and construct the validation data by cutting the prediction horizon from z old .
Revised: We then obtain the training data from z old and construct the validation data by cutting the prediction horizon from z old .

Original: The test dataset is constructed from z new as in bullet point 3 with p = 56 .
Revised: The test dataset is constructed from z new as in bullet point 3 with p = 56 .

Original: C T RAINING D ETAILS
Revised: C T RAINING D ETAILS

Original: This section describes in detail how the deep learning models from Table 2 are trained on the datasets outlined in the previous section.
Revised: This section describes in detail how the deep learning models from Table 2 are trained on the datasets outlined in the previous section.

Original: As outlined in Section 3.2, parametric local methods directly estimate their parameters at training time.
Revised: As outlined in Section 3.2, parametric local methods directly estimate their parameters at training time.

Original: While deep learning models are trained on Z train such that models at different checkpoints can be compared using Z val , parameters of parametric local methods are estimated using Z val .
Revised: While deep learning models are trained on Z train such that models at different checkpoints can be compared using Z val , parameters of parametric local methods are estimated using Z val .

Original: C. T RAINING T IME
Revised: C. T RAINING T IME

Original: For all deep learning models, we run training for a predeﬁned duration depending on the dataset size.
Revised: For all deep learning models, we run training for a predeﬁned duration depending on the dataset size.

Original: For n total observations in the training data, we run training for d hours whered = 2 η with η
Revised: For n total observations in the training data, we run training for d hours where

Original: [Equation]
Revised: [Equation]

Original: Training therefore always runs between one and eight hours (refer to Table 3 for training times on individual datasets).
Revised: Training therefore always runs between one and eight hours (refer to Table 3 for training times on individual datasets).

Original: Even on small datasets, we train for an hour to ensure that the trained model receives sufﬁciently many gradient updates.
Revised: Even on small datasets, we train for an hour to ensure that the trained model receives sufﬁciently many gradient updates.

Original: C. P ROBABILISTIC F ORECASTS
Revised: C. P ROBABILISTIC F ORECASTS

Original: Forecasts of all methods are required to be probabilistic to store the 10-quantiles q 0 . 1 , q 0 . 2 , . . .
Revised: Forecasts of all methods are required to be probabilistic to store the 10-quantiles q 0 . 1 , q 0 . 2 , . . .

Original: , q 0 .
Revised: , q 0 .

Original: While some methods forecast the quantiles directly (e.g. MQ-CNN, TFT), other methods provide samples of the output distribution (e.g. DeepAR) or point forecasts (e.g. N-BEATS).
Revised: While some methods forecast the quantiles directly (e.g. MQ-CNN, TFT), other methods provide samples of the output distribution (e.g. DeepAR) or point forecasts (e.g. N-BEATS).

Original: For samplingbased models, we simply compute the empirical 10-quantiles while for the point forecasts, we set each quantile  to theforecasted value 3 .
Revised: For samplingbased models, we simply compute the empirical 10-quantiles. For the point forecasts, we interpret forecasts as Dirac distributions centered around the forecasted value y . Thus, each quantile can be set to the value y .

Original: C. L EARNING R ATE S CHEDULING
Revised: C. L EARNING R ATE S CHEDULING

Original: All deep learning models are trained using the Adam optimizer (Kingma & Ba, 2014) with an initial learning rate of ρ = 10 − 3 .
Revised: All deep learning models are trained using the Adam optimizer (Kingma & Ba, 2014) with an initial learning rate of ρ = 10 − 3 .

Original: During training, we decrease the learning rate three times by a factor of λ = 12 using a linear schedule.
Revised: During training, we decrease the learning rate three times by a factor of λ = 12 using a linear schedule.

Original: When training for a duration d , we decrease the learning rate after durations D = { d 4 , d 2 , 3 d 4 } .
Revised: When training for a duration d , we decrease the learning rate after durations D = { d 4 , d 2 , 3 d 4 } .

Original: This results in a ﬁnal learning rate of λ 3 ρ = 1 . 25 · 10 − 4 .
Revised: This results in a ﬁnal learning rate of λ 3 ρ = 1 . 25 · 10 − 4 .

Original: The decay is always applied after the ﬁrst batch exceeding one of the durations in D .
Revised: The decay is always applied after the ﬁrst batch exceeding one of the durations in D .

Original: C. V ALIDATION AND T ESTING
Revised: C. V ALIDATION AND T ESTING

Original: During training, we save the model and compute its loss L as well as the nCRPS Q on the validation data for a total of 11 times.
Revised: During training, we save the model and compute its loss L as well as the nCRPS Q on the validation data for a total of 11 times.

Original: For a model being trained for a duration t , we perform these computations after durations D
Revised: For a model being trained for a duration t , we perform these computations after durations D

Original: =
Revised: =

Original: H ∪ { k t 9 | k ∈ { 1 , . .
Revised: H ∪ { k t 9 | k ∈ { 1 , . .

Original: , 9 }} with intervals
Revised: , 9 }} with intervals

Original: I = { t · 3 − k | k ∈ { 0 , . . .
Revised: I = { t · 3 − k | k ∈ { 0 , . . .

Original: For each d ∈ D , we run validation on Z val after the ﬁrst batch where the total training time exceeds d and compute the losses L d and Q d .
Revised: For each d ∈ D , we run validation on Z val after the ﬁrst batch where the total training time exceeds d and compute the losses L d and Q d .

Original: Note that we do not add the time taken to run validation to the training time to evaluate whether it exceeds d .
Revised: Note that we do not add the time taken to run validation to the training time to evaluate whether it exceeds d .

Original: We argue that validation can be run swiftly.
Revised: We argue that validation can be run swiftly.

Original: If necessary, it could also be run on a subset of all available time series for further speedup.
Revised: If necessary, it could also be run on a subset of all available time series for further speedup.

Original: For testing, we choose 5 of the 11 models that we saved during training.
Revised: For testing, we choose 5 of the 11 models that we saved during training.

Original: For each d ∈ I , we choose the model with the lowest nCRPS that was encountered up to d .
Revised: For each d ∈ I , we choose the model with the lowest nCRPS that was encountered up to d .

Original: Note that this may result in choosing the same model multiple times if continued training did not yield any improvements on the validation data.
Revised: Note that this may result in choosing the same model multiple times if continued training did not yield any improvements on the validation data.

Original: C. I NFRASTRUCTURE AND T RAINING S TATISTICS
Revised: C. I NFRASTRUCTURE AND T RAINING S TATISTICS

Original: All training jobs were scheduled on AWS Sagemaker 4 using CPU-only ml.c5.2xlarge instances (8 CPUs, 16 GiB memory).
Revised: All training jobs were scheduled on AWS Sagemaker 4 using CPU-only ml.c5.2xlarge instances (8 CPUs, 16 GiB memory).

Original: In few cases, we ran jobs on other instance types:
Revised: In few cases, we ran jobs on other instance types:

Original: • ml.m5.2xlarge instances (8 CPUs, 32 GiB memory) to counteract out-of-memoryissues.
Revised: • ml.m5.2xlarge instances (8 CPUs, 32 GiB memory) to counteract out-of-memoryissues.

Original: Used for N-BEATS on “Corporación Favorita” and “Weather”, DeepAR on “Corporación Favorita”, and Prophet on “Wind Farms”.
Revised: Used for N-BEATS on “Corporación Favorita” and “Weather”, DeepAR on “Corporación Favorita”, and Prophet on “Wind Farms”.

Original: • ml.m5.4xlarge instances (16 CPUs, 64 GiB memory) to allow for even more memoryintensive models.
Revised: • ml.m5.4xlarge instances (16 CPUs, 64 GiB memory) to allow for even more memoryintensive models.

Original: Used for N-BEATS on “London Smart Meters”.
Revised: Used for N-BEATS on “London Smart Meters”.

Original: • ml.c5.18xlarge instances (72 CPUs, 144 GiB memory) to speed up predictions by parallelization.
Revised: • ml.c5.18xlarge instances (72 CPUs, 144 GiB memory) to speed up predictions by parallelization.

Original: Used for ARIMA on “Taxi”.
Revised: Used for ARIMA on “Taxi”.

Original: In total, we ran 4,708 training jobs.
Revised: In total, we ran 4,708 training jobs.

Original: Including validation and testing, this amounted to roughly 684 days of total training time ( ∼ 3 hours and 30 minutes per training job).
Revised: Including validation and testing, this amounted to roughly 684 days of total training time ( ∼ 3 hours and 30 minutes per training job).

Original: Parallelized overinstances, actual training time for our benchmark was roughly 4 days and amounts to approximately $7,300 of compute costs on AWS.https: com/sagemaker/
Revised: Parallelized overinstances, actual training time for our benchmark was roughly 4 days and amounts to approximately $7,300 of compute costs on AWS.

Original: D S URROGATE M ODEL
Revised: D S URROGATE M ODEL

Original: In the following, we discuss the choice of the surrogate model.
Revised: In the following, we discuss the choice of the surrogate model.

Original: First, we outline why a parametric surrogate model is desirable.
Revised: First, we outline why a parametric surrogate model is desirable.

Original: Then, we describe how model conﬁgurations are vectorized to be used by parametric surrogate models.
Revised: Then, we describe how model conﬁgurations are vectorized to be used by parametric surrogate models.

Original: Afterwards, we list the ranking loss with linear discounting that we mention in Section 4.2 and eventually compare the ranking performance of different surrogate models.
Revised: Afterwards, we list the ranking loss with linear discounting that we mention in Section 4.2 and eventually compare the ranking performance of different surrogate models.

Original: D. C HOICE OF S URROGATE M ODEL
Revised: D. C HOICE OF S URROGATE M ODEL

Original: Intuitively, a nonparametric surrogate model might appear to be optimal for ranking models.
Revised: Intuitively, a nonparametric surrogate model might appear to be optimal for ranking models.

Original: However, a parametric surrogate has several advantages.
Revised: However, a parametric surrogate has several advantages.

Original: First, it can interpolate between neighboring conﬁgurations, which is essential when the conﬁgurations differ between different datasets or when the objective is noisy.
Revised: First, it can interpolate between neighboring conﬁgurations, which is essential when the conﬁgurations differ between different datasets or when the objective is noisy.

Original: Second, a parametric model allows for suggesting new conﬁgurations that e.g. maximize capacity under a given constraint.
Revised: Second, a parametric model allows for suggesting new conﬁgurations that e.g. maximize capacity under a given constraint.

Original: Third, a parametric surrogate model ˜ f
Revised: Third, a parametric surrogate model ˜ f

Original: θ may also be conditioned on the dataset by taking as input dataset features (such as number of time series or observations in the task at hand) by concatenating features to the input vector of ˜ f
Revised: θ may also be conditioned on the dataset by taking as input dataset features (such as number of time series or observations in the task at hand) by concatenating features to the input vector of ˜ f

Original: θ — although we do not consider this possibility here.
Revised: θ — although we do not consider this possibility here.

Original: D. I NPUT V ECTORIZATION
Revised: D. I NPUT V ECTORIZATION

Original: When using XGBoost or an MLP as surrogate model for predicting model performance, models must be represented as feature vectors.
Revised: When using XGBoost or an MLP as surrogate model for predicting model performance, models must be represented as feature vectors.

Original: For this, we proceed as follows to encode model conﬁgurations:
Revised: For this, we proceed as follows to encode model conﬁgurations:

Original: • The model type (i.e. ARIMA, DeepAR, ...) is encoded via a one-hot encoding, yielding a 13-dimensional vector.
Revised: • The model type (i e. ARIMA, DeepAR, ...) is encoded via a one-hot encoding, yielding a 13-dimensional vector.

Original: • Every model hyperparameter (across models) deﬁnes a new real-valued feature.
Revised: • Every model hyperparameter (across models) deﬁnes a new real-valued feature.

Original: Hyperparameters that are shared among deep learning models (context length multiple, training time, learning rate 5 ) are re-used across models.
Revised: Hyperparameters that are shared among deep learning models (context length multiple, training time, learning rate 5 ) are re-used across models

Original: This results in 15 additional features which are all standardized to have zero mean and unit variance.
Revised: This results in 15 additional features which are all standardized to have zero mean and unit variance.

Original: Inputs to the MLP are, thus, 28-dimensional.
Revised: Inputs to the MLP are, thus, 28-dimensional.

Original: Features that are missing (e.g. all classical models do not provide a context length multiple) are imputed by the features’ means (resulting in zeros due to the standardization).
Revised: Features that are missing (e.g. all classical models do not provide a context length multiple) are imputed by the features’ means (resulting in zeros due to the standardization).

Original: D. R ANKING L OSS WITH L INEAR D ISCOUNTING
Revised: D. R ANKING L OSS WITH L INEAR D ISCOUNTING

Original: Section 4.2 outlines that the MLP surrogate model ˜ f
Revised: Section 4.2 outlines that the MLP surrogate model ˜ f

Original: θ is trained via listwise ranking using linear discounting.
Revised: θ is trained via listwise ranking using linear discounting.

Original: For training on the ofﬂine evaluations D , ˜ f
Revised: For training on the ofﬂine evaluations D , ˜ f

Original: θ minimizes the loss L via SGD.
Revised: θ minimizes the loss L via SGD.

Original: As ranking is performed for each objective k ∈ { 1 , . . .
Revised: As ranking is performed for each objective k ∈ { 1 , . . .

Original: , m } and task j ∈ { 1 , . ..
Revised: , m } and task j ∈ { 1 , . .

Original: , M } independently, L can be written as follows:
Revised: , T } independently, L can be written as follows:

Original: [Equation]
Revised: [Equation]

Original: where ˜ f θ ; k yields the surrogate model’s outputs for objective k and L kj ( ˜ f θ ; k ) is deﬁned as follows (where we ignore indices k and j in the formula for notational convenience):
Revised: where ˜ f θ ; k yields the surrogate model’s outputs for objective k and L kj ( ˜ f θ ; k ) is deﬁned as follows (where we ignore indices k and j in the formula for notational convenience):

Original: [Equation]
Revised: [Equation]

Original: Here, N provides the number of available evaluations for task j and r ( i ) deﬁnes the index of the conﬁguration x that is ranked at position i with respect to objective k among all conﬁgurations for task j .
Revised: Here, N provides the number of available evaluations for task j and r ( i ) deﬁnes the index of the conﬁguration x that is ranked at position i with respect to objective k among all conﬁgurations for task j .

Original: The ranks of the conﬁgurations x 1 , . . .
Revised: The ranks of the conﬁgurations x 1 , . . .

Original: , x N with respect to an objective k can be computed from the corresponding evaluations y 1; k , . . .
Revised: , x N with respect to an objective k can be computed from the corresponding evaluations y 1; k , . . .

Original: , y N ; k .
Revised: , y N ; k .

Original: Note that rank i = 1 provides the index for the “best” conﬁguration and ˜ f
Revised: Note that rank i = 1 provides the index for the “best” conﬁguration and ˜ f

Original: θ outputs a smaller value for conﬁgurations that have a smaller rank (i.e. are “better”).[Table]
Revised: θ outputs a smaller value for conﬁgurations that have a smaller rank (i.e. are “better”).

Original: D. C OMPARISON OF D IFFERENT M ODELS
Revised: D. C OMPARISON OF D IFFERENT M ODELS

Original: The following compares the MLP surrogate trained via listwise ranking introduced in Section 4 to other possible choices for the surrogate model.
Revised: The following compares the MLP surrogate trained via listwise ranking introduced in Section 4 to other possible choices for the surrogate model.

Original: Much like the approach described in Section 4.3, we evaluate surrogate models by performing LOOCV: surrogates are evaluated on each of thebenchmark datasets one after the other, being trained on the remaining 43 datasets.
Revised: Much like the approach described in Section 4.3, we evaluate surrogate models by performing LOOCV: surrogates are evaluated on each of thebenchmark datasets one after the other, being trained on the remaining 43 datasets.

Original: Surrogates are evaluated with respect to different ranking metrics that give an overview of the models’ ability to rank the nCRPS values of model conﬁgurations on different datasets.
Revised: Surrogates are evaluated with respect to different ranking metrics that give an overview of the models’ ability to rank the nCRPS values of model conﬁgurations on different datasets.

Original: As metrics for measuring the ranking performance on a single dataset, we consider the following which yield metrics betweenand 1:
Revised: As metrics for measuring the ranking performance on a single dataset, we consider the following which yield metrics betweenand 1:

Original: • Mean Reciprocal Rank (MRR) (Liu, 2009):
Revised: • Mean Reciprocal Rank (MRR) (Liu, 2009):

Original: Computed as one divided by the predicted rank of the model with the lowest true nCRPS.
Revised: Computed as one divided by the predicted rank of the model with the lowest true nCRPS.

Original: • Precision@k
Revised: • Precision@k

Original: (Liu, 2009):
Revised: (Liu, 2009):

Original: The fraction of the k models with the lowest true nCRPS captured in the top k predictions.
Revised: The fraction of the k models with the lowest true nCRPS captured in the top k predictions.

Original: Normalized Discounted Cumulative Gain (NDCG) (Liu, 2009):
Revised: Normalized Discounted Cumulative Gain (NDCG) (Liu, 2009):

Original: A measure for the overall ranking performance.
Revised: A measure for the overall ranking performance.

Original: It ﬁrst computes the discounted cumulative gain (DCG) by summing the discounted relevance scores 1 , 0 . 9 , . . .
Revised: It ﬁrst computes the discounted cumulative gain (DCG) by summing the discounted relevance scores 1 , 0 . 9 , . . .

Original: 1 of the 10 models with the lowest true nCRPS scores — more speciﬁcally, a relevance score π ( i ) is discounted by log 2 k +where k is the predicted rank of the model with true rank i .
Revised: 1 of the 10 models with the lowest true nCRPS scores — more speciﬁcally, a relevance score π ( i ) is discounted by log 2 k +where k is the predicted rank of the model with true rank i .

Original: Then, it normalizes the DCG by using the ideal DCG (iDCG) to obtain the NDCG.
Revised: Then, it normalizes the DCG by using the ideal DCG (iDCG) to obtain the NDCG.

Original: As a comparison to the ranking MLP surrogate with linear discount ( MLP Ranking + Discounting ), we ﬁrst consider a random surrogate which predicts nCRPS by sampling from U (0 , 1) as a baseline ( Random ).
Revised: As a comparison to the ranking MLP surrogate with linear discount ( MLP Ranking + Discounting ), we ﬁrst consider a random surrogate which predicts nCRPS by sampling from U (0 , 1) as a baseline ( Random ).

Original: Then, we use a nonparametric model which predicts the nCRPS as the average among all training datasets ( Nonparametric Regression ) and one that predicts the average rank of the nCRPS across training datasets ( Nonparametric Ranking ).
Revised: Then, we use a nonparametric model which predicts the nCRPS as the average among all training datasets ( Nonparametric Regression ) and one that predicts the average rank of the nCRPS across training datasets ( Nonparametric Ranking ).

Original: Additionally, we consider XGBoost surrogates which are trained via regression ( XGBoost Regression ) and pairwise ranking 6 ( XGBoost
Revised: Additionally, we consider XGBoost surrogates which are trained via regression ( XGBoost Regression ) and pairwise ranking 6 ( XGBoost

Original: Ranking ), respectively.
Revised: Ranking ), respectively.

Original: Lastly, we consider an MLP with the same architecture but trained via regression instead of listwise ranking ( MLP Regression ) and one without any discounting ( MLP
Revised: Lastly, we consider an MLP with the same architecture but trained via regression instead of listwise ranking ( MLP Regression ) and one without any discounting ( MLP

Original: Ranking ).
Revised: Ranking ).

Original: Table 4 clearly shows that the MLP with listwise ranking and linear discounting markedly outperforms all other choices for the surrogate model.
Revised: Table 4 clearly shows that the MLP with listwise ranking and linear discounting markedly outperforms all other choices for the surrogate model.

Original: Especially, it is much more capable of identifying  the top conﬁgurations, stressed by the very high values of MRR and Precision@5 compared to the other surrogate models.
Revised: Especially, it is much more capable of identifying [Table] the top conﬁgurations, stressed by the very high values of MRR and Precision@5 compared to the other surrogate models.

Original: This can likely be attributed to the linear discounting that encourages the model to focus on ranking the top conﬁgurations correctly.
Revised: This can likely be attributed to the linear discounting that encourages the model to focus on ranking the top conﬁgurations correctly.

Revised: Thus, the loss formulation is more stable in the presence of outliers

Revised: [Table]

Original: E P ARETO S ELECT PSEUDO CODE
Revised: E P ARETO S ELECT PSEUDO CODE

Original:  Data: Time series models X , ofﬂine evaluations D , number of default models n
Revised: Algorithm Data: Time series models X , ofﬂine evaluations D , number of default models n

Original: Result:
Revised: Result:

Original: A set { x 1 , . . . , x n } ⊂ X of model defaults function non_dominated_sort
Revised: A set { x 1 , . . . , x n } ⊂ X of model defaults function non_dominated_sort

Original: ( X , ˜ f
Revised: ( X , ˜ f

Original: θ ) : Λ = [] while X ̸ = ∅ do P
Revised: θ ) : Λ = [] while X ̸ = ∅ do P

Original: X , ˜ f θ = { x ∈ X | ¬∃ x ′ ∈
Revised: X , ˜ f θ = { x ∈ X | ¬∃ x ′ ∈

Original: X : x ′ ≺ f x } // Compute the Pareto front
Revised: X : x ′ ≺ f x } // Compute the Pareto front

Original: X = X \ P X , ˜ f
Revised: X = X \ P X , ˜ f

Original: θ // Remove Pareto front extend ( Λ , compute_epsilon_net ( P X , ˜ f
Revised: θ // Remove Pareto front extend ( Λ , compute_epsilon_net ( P X , ˜ f

Original: θ , ˜ f θ )) return Λ function compute_epsilon_net( X , ˜ f θ ) :
Revised: θ , ˜ f θ )) return Λ function compute_epsilon_net( X , ˜ f θ ) :

Original: λ =
Revised: λ =

Original: [ pop ( X )] // Choose and remove a random element while X ̸ = ∅ do
Revised: [ pop ( X )] // Choose and remove a random element while X ̸ = ∅ do

Original: D = {} // Mapping from remaining elements to minimum distances for x ∈ X do // Compute minimum distance to all chosen elements
Revised: D = {} // Mapping from remaining elements to minimum distances for x ∈ X do // Compute minimum distance to all chosen elements

Original: D [ x ] = min x ′ ∈ λ
Revised: D [ x ] = min x ′ ∈ λ

Original: ∥
Revised: ∥

Original: ˜
Revised: ˜

Original: f θ ( x ) − ˜ f
Revised: f θ ( x ) − ˜ f

Original: θ ( x ′ ) ∥//
Revised: θ ( x ′ ) ∥//

Original: Append the element furthest away from the current selection append ( λ , pop ( X , argmax x ∈X D [ x ] )) return λ ˜ f
Revised: Append the element furthest away from the current selection append ( λ , pop ( X , argmax x ∈X D [ x ] )) return λ ˜ f

Original: θ = train_surrogate_model ( D ) //
Revised: θ = train_surrogate_model ( D ) //

Original: Multi-objective sorting according to predictions x 1 , . .
Revised: Multi-objective sorting according to predictions x 1 , . .

Original: , x | X | =
Revised: , x | X | =

Original: non_dominated_sort ( X , ˜ f
Revised: non_dominated_sort ( X , ˜ f

Original: θ ) return x 1 , . . .
Revised: θ ) return x 1 , . . .

Original: , x min( n, | X | )
Revised: , x min( n, | X | )

Revised: F A NALYSIS OF DATASETS CHARACTERISTICS

Revised: In this section, we try to answer the following question: are there some simple dataset characteristics that can help to know which model is going to perform best?

Revised: In particular, we analyze in details the results of Section 3.5 where we showed that, on some datasets, (1) statistical methods outperformed deep learning methods and that (2) the performance of statistical and deep learning methods could not be distinguished.

Revised: We start by analyzing more in depth the four out 44 datasets where deep learning models do not outperform classical methods — Figure 2 showed that the best deep learning method outperforms the best classical methods on all but four of our benchmark datasets.

Revised: On these four datasets (“KDD2018”, “M3 Yearly”, “M4 Quarterly”, “Taxi”), the summary statistics differ wildly as can be seen from Table 3 both in terms of the number of observations, the average length, and other characteristics.

Revised: Further, the classical method that outperforms the best deep learning method is different for each of these datasets (“KDD2018”: NPTS, “M3 Yearly”:

Revised: Theta, “M4 Quarterly”: ETS, “Taxi”: STL-AR).

Revised: Notably, this does not include the local method that performs best on average (i.e.

Revised: ARIMA, see Table 1).

Revised: Figure 1 showed that for 14 out of the 44 benchmark datasets, the performance of classical methods is indistinguishable from the performance of deep learning methods.

Revised: To further study these datasets, we plot in Figure 7 the correlation matrix of all k benchmark methods’ nCRPS ranks across the datasets (using only the default conﬁguration for the deep learning models).

Revised: Notably, for twelve datasets (“Exchange Rate” through “M1 Yearly” on the y-axis), we observe a very low correlation with a large number of datasets.

Revised: On these datasets, all models perform similarly due to strong stochasticity (e.g. “Exchange Rate”, “Bitcoin”, “M1 Quarterly”, ...) or seasonality (e.g. “Tourism”, “Ride share”, ...) and predicting their ranking is therefore hard.

Revised: Notably, 11 of these datasets overlap with the 14 datasets where classical methods are indistinguishable from deep learning methods.

Revised: In line with these observations, we found that adding simple dataset grouping features such as domain type (electricity, retail, ﬁnance) as an input for a surrogate model predicting the performance of forecasting methods does not have much effect.

Revised: Our conclusion is that the ranks of the best performing models seem to be relatively hard to identify only from simple dataset grouping rules.

Revised: Nonetheless, we hope that by releasing the benchmark data, we help future research to identify dominant methods from dataset characteristics.

Revised: [Table]

Revised: [Figure]