Paper:
Efficient and Robust Automated Machine Learning
The success of machine learning in a broad range of applications has led to an ever-growing demand for machine learning systems that can be used off the shelf by non-experts. To be effective in practice, such systems need to automatically choose a good algorithm and feature preprocessing steps for a new dataset at hand, and also set their respective hyperparameters. Recent work has started to tackle this automated machine learning (AutoML) problem with the help of efficient Bayesian optimization methods. Building on this, we introduce a robust new AutoML system based on scikit-learn (using 15 classifiers, 14 feature preprocessing methods, and 4 data preprocessing methods, giving rise to a structured hypothesis space with 110 hyperparameters). This system, which we dub AUTO-SKLEARN, improves on existing AutoML methods by automatically taking into account past performance on similar datasets, and by constructing ensembles from the models evaluated during the optimization. Our system won the first phase of the ongoing ChaLearn AutoML challenge, and our comprehensive analysis on over 100 diverse datasets shows that it substantially outperforms the previous state of the art in AutoML. We also demonstrate the performance gains due to each of our contributions and derive insights into the effectiveness of the individual components of AUTO-SKLEARN.
In practice, the budget b would comprise computational resources, such as CPU and/or wallclock time and memory usage. This problem definition reflects the setting of the ongoing ChaLearn AutoML challenge [1]. The AutoML system we describe here won the first phase of that challenge.
Here, we follow and extend the AutoML approach first introduced by AUTO-WEKA [2] (see http://automl.org). At its core, this approach combines a highly parametric machine learning framework F with a Bayesian optimization [3] method for instantiating F well for a given dataset.
The contribution of this paper is to extend this AutoML approach in various ways that considerably improve its efficiency and robustness, based on principles that apply to a wide range of machine learning frameworks (such as those used by the machine learning service providers mentioned above). First, following successful previous work for low dimensional optimization problems [4,5,6], we reason across datasets to identify instantiations of machine learning frameworks that perform well on a new dataset and warmstart Bayesian optimization with them (Section 3.1). Second, we automatically construct ensembles of the models considered by Bayesian optimization (Section 3.2). Third, we carefully design a highly parameterized machine learning framework from high-performing classifiers and preprocessors implemented in the popular machine learning framework scikit-learn [7] (Section 4). Finally, we perform an extensive empirical analysis using a diverse collection of datasets to demonstrate that the resulting AUTO-SKLEARN system outperforms previous state-of-the-art AutoML methods (Section 5), to show that each of our contributions leads to substantial performance improvements (Section 6), and to gain insights into the performance of the individual classifiers and preprocessors used in AUTO-SKLEARN (Section 7).

Review:
SUMMARYThe authors introduce a new automated machine learning (autoML) system for classification, as well as associated technical advances that improve upon its baseline performance.The new system is referred to as auto-sklearn, since it is built upon the scikit-learn library of machine learning algorithms.The system uses 16 classification algorithms, 11 preprocessing methods, and 3 data preprocessing methods.The baseline system involves optimizing over algorithms and hyperparameters for each algorithm, to find a choice that minimizes the cross-validation error.This is accompanied by two further contributions. First, the authors experiment with meta-learning to assist the optimization; this involves extracting meta-features from a large collection of data sets (140, in this case), and characterizing the types of data sets on which each algorithm performs well.Second, the authors experiment with automated construction of ensembles to further improve the classification performance; this is done via an efficient ensemble selection method combining intermediate points evaluated during the optimization.The performance of the baseline system, as well as the augmented system with and without meta-learning and ensembling, is evaluated on 140 diverse data sets. The baseline system tends to provide better performance compared to two existing autoML systems: Auto-WEKA and hyperopt-sklearn. On top of this, employing meta-learning significantly improves performance beyond the baseline, and using ensembling in addition improves it still further.The authors also provide a detailed analysis of the performance of the 16 individual algorithms, providing insight into the relative merits of various standard classifiers. COMMENTSThis work represents a significant effort in terms of engineering and computation, for which authors are to be commended.The article is clearly written and easy to read. The motivation, formulation of the problem, description of the proposed method, description of experiments, and presentation of results are all excellent. The proposed system demonstrates an impressive advance in autoML performance. The use of meta-learning, in particular, is very interesting.The observation that ensemble selection can be used to efficiently combine intermediate points from the optimization is also nice.My criticisms are relatively minor.1. A possible criticism is that this work largely represents an engineering and computational effort, with few significant advances in terms of novel techniques or theory. However, I believe this is outweighed by the scale of the effort, the strength of the performance demonstrated, and the potential utility of this tool.2. Line 92-93: If the number of algorithms m is large, could this lead to overfitting, even though we are optimizing with respect to cross-validation error?3. Do the authors have any insights into how best to choose the number and type of algorithms to include in the system? The current system seems to be based simply on what is available in scikit-learn. Perhaps in future work, it would be interesting to know if there is a principled way to choose the composition of algorithms to be included, beyond the general suggestion that it helps to have a diversity of algorithms.4. The meta-learning approach is appealing. However, no explanation is given for why the authors chose the particular approach that they did. The approach chosen is roughly analogous to k-nearest neighbors, but one can imagine many other possibilities involving other classifiers or regression algorithms. Are there reasons to prefer the chosen approach over other approaches? 5. Perhaps in future work, it would be interesting if the meta-learning aspect of the system could be formalized via a mathematical objective function that could be optimized in order to improve the performance of the overall system.6. Line 150: Minor point --- I don't get how this is Bayesian... I would think a Bayesian approach would involve something like putting a prior on the algorithms/hyperparameters, and sampling from the posterior.7. Tables 4 and 5: Minor point --- it might be helpful to move the auto-sklearn column to the front or back, in order to highlight it. Or maybe to sort the columns alphabetically or by aggregate performance.