  Evaluating predictive models is a crucial task in predictive analytics. This
process is especially challenging with time series data where the observations
show temporal dependencies. Several studies have analysed how different
performance estimation methods compare with each other for approximating the
true loss incurred by a given forecasting model. However, these studies do not
address how the estimators behave for model selection: the ability to select
the best solution among a set of alternatives. We address this issue and
compare a set of estimation methods for model selection in time series
forecasting tasks. We attempt to answer two main questions: (i) how often is
the best possible model selected by the estimators; and (ii) what is the
performance loss when it does not. We empirically found that the accuracy of
the estimators for selecting the best solution is low, and the overall
forecasting performance loss associated with the model selection process ranges
from 1.2% to 2.3%. We also discovered that some factors, such as the sample
size, are important in the relative performance of the estimators.
