Replacing the Ex-Def Baseline in AutoML by Naive AutoML
Abstract
Automated Machine Learning (AutoML) is the problem of automatically ﬁnding the pipeline with the best generalization performance on some given dataset.
AutoML has received enormous attention in the last decade and has been addressed with sophisticated black-box optimization techniques like Bayesian Optimization, Genetic Algorithms, or Tree Search.
These approaches are almost never compared to simple baselines to see how much they improve over simple but easy to implement approaches.
We present Naive AutoML, a very simple baseline for AutoML that exploits meta-knowledge about machine learning problems and makes simplifying, yet, eﬀective assumptions to quickly come to high-quality solutions.
In 1h experiments, state of the art approaches can hardly improve over Naive AutoML, which in turn comes along with advantages such as interpretability and ﬂexibility.
1. Introduction
AutoML is the problem of automatically ﬁnding the data transformation and learning algorithms with the best generalization performance on a given dataset.
The combination of such algorithms is typically called machine learning pipeline , because several algorithms for data manipulation and analysis are concatenated sequentially.
To optimize such a machine learning pipeline, important decisions do not only include the learning algorithm itself but also its parameters as well as a deﬁnition of which features should be used by the learner.
The baselines used to assess the performance of AutoML tools are often other AutoML tools or random search.
A simple but perhaps more sensible baseline than random search would be to imitate the steps a human data scientist would take.
Without such baselines, we do not learn how AutoML tools improve upon ad-hoc techniques but only how they compare relatively to each other.
To our knowledge, the only work accounting for such baselines is (Thornton et al., 2013), using the Exhaustive-Default (“Ex-def”) baseline which is to take the default parametrized model that is best in a cross-validation.
They also discuss a grid search, which is however not applicable in practice.
In this paper we make up for such a baseline, which imitates the process a human expert might go through to ﬁnd the best pipeline.
Instead of crawling an enormous search space and to pick the best one seen within some timeout, the idea is to conduct an analytical process and derive the appropriate pipeline from it.
In this, we implicitly assume a form of independence of the optimization decisions and hence act a bit naive w.r.t.
potential interactions between them; this is why we call the approach Naive AutoML.
In this sense, we complement the “Ex-def” baseline (Thornton et al., 2013) by what a somewhat more experienced data scientist would maybe do.
The surprising result of our experimental evaluation is that, for runtimes of 1h, the black box optimizers are hardly ever able to improve upon Naive AutoML.
In fact, the experiments even show that the “Ex-def” baseline itself is already quite strong in this time frame.
This is not a contradiction to the results in (Thornton et al., 2013), where a timeout of 30 hours was used.
While this observations calls for more exhaustive experiments, it is already evident that simple baselines are stronger than perhaps believed.
Even from these results, we can learn two things.
First, it seems sensible to adopt
Naive AutoML as a standard baseline for the evaluation of AutoML tools in order to demonstrate their superiority over a very simple yet often eﬀective approach.
Second, the success of Naive AutoML challenges the view of black box optimization for AutoML in general and invites to consider more speciﬁc approaches taylored for AutoML.
2. Problem Deﬁnition
In this paper, we are focused on AutoML for supervised learning.
Formally, in the supervised learning context, we assume some instance space X ⊆ R d and a label space Y .
A dataset
D ⊂ { ( x, y ) | x ∈ X , y ∈ Y} is a ﬁnite relation between the instance space and the label space, and we denote as D the set of all possible datasets.
We consider two types of operations over instance and label spaces:
Transformers.
A transformer is a function t : X A → X B , converting an instance x of instance space
X A into an instance of another instance space X B .
Predictors.
A predictor is a function p :
X p
→ Y , assigning an instance of its instance space
X p a label of the original label space Y .
In this paper, a pipeline P = t 1 ◦ .. ◦ t k ◦ p is a functional concatenation in which t i : X i − 1 → X i are transformers with X
= X being the original instance space, and p :
X k
→ Y is a predictor.
Hence, a pipeline is a function P : X → Y that assigns a label to each object of the instance space.
We denote as P the space of all pipelines of this kind.
In general, the ﬁrst part of a pipeline could not only be a sequence but also a transformation tree with several parallel transformations that are then merged (Olson and Moore, 2016), but we do not consider such structures in this paper since they are not necessary for our key argument.
In addition to the sequential structure, many AutoML approaches restrict the search space still a bit further.
This is by putting an order on the pre-processing steps and allowing only on transformer per type.
So P will only contain pipelines compatible with this order.
We can then express every element of P as a concatenation of k + 1 functions, where k is the number of considered transformation algorithm types.
In this space of possible pipelines, the goal is to ﬁnd the one that optimizes a pre-deﬁned performance measure φ : D × P → R .
Typical measures are the error rate, log-loss or, for the case of binary classiﬁcation, AUC/ROC or the F1-measure on a given dataset.
3. Naive AutoML 3.1 Naivety Assumptions
Naive AutoML makes, among others, the assumption that the optimal pipeline is the one that is locally best for each of its transformers and the ﬁnal predictor.
In other words, taking into account pipelines with (up to) k transformers and a predictor, we assume that for all datasets D and all 1 ≤ i ≤ k +
[Equation]
is invariant to the choices of c 1 , ..
c i − 1 , c i +1 , ..,
c k +1 , which are supposed to be ﬁxed in the above equation.
Note that we here use the letter c instead of t for transformers or p for the predictor, because c may be any of the two types.
We dub the approach Naive AutoML, because there is a direct link to the assumption made by the Naive Bayes classiﬁer.
Consider P an urn and denote as Y the event to observe an optimal pipeline in the urn.
Then
[Equation]
in which we consider c j to be ﬁxed components for j (cid:54)
=
i , and only c i being subject to optimization.
Applying Bayes theorem again to P ( c i | Y ) and observing that the remaining product is a constant regardless the choices of c
i (cid:54) =
j , it gets clear that the optimal solution is the one that maximizes the probability of being locally optimal, and that this choice is independent of the choice of the other components.
The typical approach to optimize the c i is not to directly construct those functions but to adopt parametrized model building processes that create these functions.
For example, c 1 could be a projection obtained by determining some features which we want to stay with, or c k +1 could be a trained neural network.
These induction processes for the components can be described by an algorithm a i and a hyper-parametrization θ i of the algorithm.
The component c i is obtained by running a i under parameters θ i with some training data.
So to optimize c i , we need to optimally choose a i and θ i .
Within this regime, Naive AutoML makes the additional (maybe not always correct) assumption that even each component c i can be optimized by local optimization techniques.
More precisely, it is assumed that the algorithm that yields the best component when using the default parametrization is also the algorithm that yields the best component if all algorithms are run with the best parametrization possible.
3.2 A Prototype for Naive AutoML
In this paper, we work with a speciﬁc exemplary realization scheme of Naive AutoML.
However, this scheme is not what we refer to as Naive AutoML but only one prototypical instantiation .
We certainly do not claim this prototype to be the last answer in Naive AutoML; better schemes can be found in future work.
Our prototype scheme consists of six stages and is sketched in Fig.
stage simply cross-validates every learning algorithm once without setting any of its hyperparameters.
This stage corresponds to the “Ex-def” baseline used in (Thornton et al., 2013).
The second stage simply pairs all feature scalers with all learners and observes their performance.
The third stage is independent of the ﬁrst two stages and adopts ﬁltering techniques to identify a subset of the features that are expected to bring the best performance (on average).
The fourth stage combines all previous candidates with each homogeneous [Figure] meta learner, and the ﬁfth stage tunes the learner parameters of the incoming candidates (with a given time budget).
In the ﬁnal stage, the most promising candidates are evaluated against a validation dataset.
The stages are described in more detail in the Appendix.
This stage scheme produces pipelines with up to three components.
A candidate is encoded by a tuple ( s , F , a , θ ), where s encodes the feature transformations (applied to all columns), F is a set of feature indices to consider, a is the prediction algorithm, and θ the parametrization for a .
The reason why s is a function but F is a set is simply that the feature selection is a projection that is entirely described by the features it shall retrain, but the scaling implies a functional transformation of data that cannot be captured so easily.
Despite being only an example scheme, one advantage of Naive AutoML that already becomes clear here is that it directly generates important insights that can signiﬁcantly support the data scientist working with it.
For example, a question like “what is the potential of feature selection on the given data?” can be answered by black-box approaches only after some post-processing, if at all.
In our scheme, the ﬁltering stage (cf.
Sec. A.3) is a very good basis to give an initial answer to this question.
In this sense, Naive AutoML presents itself as more amenable to the growing demand for meaningful interaction between the tool and the human (Wang et al., 2019; Crisan and Fiore-Gartland, 2021; Drozdal et al., 2020; Wang et al., 2021) compared to the currently adopted black-box approaches.
4. Evaluation
In this section, we want to address the following question: How much can state-of-the-art tools for Python and Java improve over the baseline imposed by Naive AutoML?
We stress our point of view that Naive AutoML is in fact the baseline here and not the competing technique.
Naive AutoML updates the previous “Ex-def” baseline by proposing a more sophisticated data scientist (acknowledging that an expert data scientist would be even more ﬂexible and hence might even be stronger than our Naive AutoML).
4.1 Compared Algorithms
We ran Naive AutoML in two conﬁgurations.
We refer to “primitive” as the version of
Naive AutoML that only adopts the probing (ﬁrst) stage and nothing else.
Once again, this corresponds to the “Ex-def” baseline proposed in (Thornton et al., 2013).
The proﬁle containing all the stages (including validation), is referred to as “full”.
On the state-of-the-art side, we compare solutions with competitive performance for both WEKA (Hall et al., 2009) and scikit-learn (Pedregosa et al., 2011).
To our knowledge,
ML-Plan is the best performing AutoML tool for WEKA to date, but we also consider
Auto-WEKA to contrast its performance against the “Ex-def” baseline.
On the scikit-learn side, we consider auto-sklearn as a competitor (without warm-starting and without ﬁnal ensemble building).
We consider version 0.12.0, which underwent substantial changes and improvements compared to the original version proposed by (Feurer et al., 2015).
4.2 Experiment Setup
The evaluation is based on the dataset portfolio proposed in (Gijsbers et al., 2019).
This is a collection of datasets available on openml.org (Vanschoren et al., 2013).
To complement these datasets, we have added some of the datasets that were used in the Auto-WEKA paper (Thornton et al., 2013) and have been used frequently for comparison in publications on AutoML.
Five datasets of (Gijsbers et al., 2019) (23, 31, 188, 40996, 41161, 42734) were removed due to technical issues in the data loading process.
For each dataset, 10 random train-test splits were created, and all algorithms were run once on each of these splits, using the train data for optimizing, and the test data to assessing the performance.
Needless to say, the splits were the same for all approaches.
For all algorithms, we allowed a total runtime of 1h, and the runtime for a single pipeline execution was conﬁgured to take up to 1 minute.
In Auto-WEKA, it is not possible to conﬁgure the maximum runtime for single executions; this variable is interenally controlled.
In Naive AutoML, we imposed stage time bounds for the meta and the parameter tuning stages of 5 minutes respectively (the other stages were not equipped with a dedicated timeout).
Of course, on hitting the time bound of 1h, Naive AutoML was stopped regardless the phase in which it was, and the best seen solution was returned.
Computations were run on Linux machines, each of them equipped with 2.6Ghz Intel
Xeon E5-2670 processors and 32GB memory.
In spite of the technical possibilities, we did not parallelize evaluations, i.e. all the tools were conﬁgured to run with a single CPU core.
4.3 Results
Due to space limitations, Fig. 2 only summarizes the results very superﬁcially through ﬁnal ranks.
More detailed results can be found in Table 2 and Fig. 4 in Sec.
C of the appendix.
The code (both Java and Python) to reproduce these results is publicly available 1 .
The results show that the cases in which any of the advanced AutoML tools achieves a statistically signiﬁcant improvement form a small minority.
In fact, there are even quite some datasets on which Naive AutoML performs better .
Auto-WEKA is never better than the “primitive” Naive AutoML approach (“Ex-def”) and even worse in 6 cases.
The performance of ML-Plan is only marginally better in the comparison with Naive AutoML but also does not show a substantial improvement over it and is outperformed even 11 times by “Ex-def”.
The comparison between auto-sklearn and the “full” Naive AutoML proﬁle, we [Figure] can see that it substantially improves upon Naive AutoML in only ﬁve out of 67 cases and is outperformed 8 times.
Among the ﬁve cases of improvement, auto-sklearn uses in four of the cases algorithms not supported by Naive AutoML in the current version.
The observations on the “primitive” proﬁle can be put into words by saying that if we run a simple for loop over the possible base learners (using default parameters and ignoring meta-learners or any kind of feature transformation), then we obtain an equal or better performance than Auto-WEKA, ML-Plan, or auto-sklearn in more than 90% of the cases; no pre-processors are required.
The results for the “primitive” solution arrive often in the range of some few minutes.
So they are cheap to obtain, and other approaches rarely can improve upon it even when running for an hour.
In our view, there are two possible explanations for the missing superiority of the black-box optimization techniques.
The ﬁrst is simply that the global optimum is in fact obtained by choosing the best algorithm without any further optimizations.
That is, there is simply no potential for optimization.
On some datasets (with almost perfect performance) this is clearly the case.
The second is that there maybe is potential for optimization, but the resource limitations impede that the black-box approaches can develop their full potential.
Theoretically, all of the black-box approaches converge to the globally optimal solution.
However, only a tiny fraction of the search space can be examined in any reasonable timeout, and the number of evaluations that can be made in that time are not enough to learn enough about the performance landscape to steer the search process in a meaningful way.
It is hence worthwhile to run the experiments with higher timeouts to see whether the
AutoML tools can eventually outperform Naive AutoML and after which time.
5. Conclusion
This paper proposes Naive AutoML, a slightly more sophisticated baseline for AutoML than proposed with “Ex-def” in (Thornton et al., 2013).
Instead of searching a complex pipeline search space with a black-box optimizer, Naive AutoML imitates the sequential workﬂow conducted by a data scientist, which implicitly deﬁnes a solution pipeline.
We empirically demonstrate that state-of-the-art tools are rarely able to outperform this baseline (sometimes the contrary is true).
Producing competitive results to the state of the art, Naive AutoML is not only more transparent and understandable to the expert but also more ﬂexible, because all modiﬁcations can be directly realized in the stage implementations instead of having it to be injected into the solver through the problem encoding.
Appendix Appendix A. Details on the Stages of the Naive AutoML Prototype
A candidate is encoded by a tuple ( s , F , a , θ ), where s encodes the feature transformations (applied to all columns), F is a set of feature indices to consider, a is the prediction algorithm, and θ the parametrization for a .
The symbol ⊥ is used to denote that a part of the description remains empty.
In the following sub-sections, we assume that candidates are evaluated with some standard evaluation scheme.
A typical choice to evaluate candidates is to use standard cross validation techniques.
For example, in the evaluation in Sec. 4, we conduct a Monte-Carlo
Cross Validation (MCCV) with a train fold size of 70% and 5 repetitions.
This means that we build 5 random splits of 70% training data and 30% validation data each, train and test over the 5 folds using the desired metric, and then average these observations to get to a score.
However, Naive AutoML is not committed to a particular type of scoring function and could, for example, also be run with a 10-fold cross validation.
We hence just shall assume that every pipeline has some score , and we leave it to the concrete implementation to implement one or another method.
A.1 The Probing Stage
This stage corresponds to the “Ex-def” suggested in the evaluation of (Thornton et al., 2013).
Formally, we create and evaluate all candidates of the form ( ⊥ , ⊥ , a, ⊥ ), where a is one of the available base prediction algorithms.
Here, we do not consider ensemble learners like Boosting or Bagging but only those that are already implemented with a speciﬁc base learner, such as Random Forests.
A.2 The Feature-Scaling Stage
In this step, we examine the beneﬁt of diﬀerent feature scaling operations for the base learners.
To this end, two or three cheap distance-sensitive pilot classiﬁers like kNN or
SVMs are used to assess the impact of diﬀerent feature scaling techniques such as (mean)- normalization, standardization (mean 0 and std 1), or quantile-based re-scaling.
In addition, it can make sense to add as a pilot the one or two best candidates resulting from the probing stage.
Formally, for each scaler s and each pilot algorithm a p , we evaluate the candidate ( s, ⊥ , a p , ⊥ ).
For each scaling technique s , if at least one pilot classiﬁer improves upon its result of the original data, we evaluate all other candidates ( s, ⊥ , a, ⊥ ) as well, where a is any non-pilot base learner.
A.3 The Filtering Stage (Classiﬁer-Independent Attribute Selection).
This stage consists of two steps.
In the ﬁrst step, we compute the set F of features that are considered relevant.
In the second step, we examine how the previously created candidates behave when using only the features of F .
That is, for every candidate ( s, ⊥ , a, ⊥ ), we create and evaluate the candidate ( s, F, a, ⊥ ).
Here s may be ⊥ as well, and the candidates are created in the order of the performance of the pipeline without feature selection.
Typ-
[Figure]
[Figure]
E rr o r R a t e
CfsSubsetEvalCorrelationAttributeEvalGainRatioAttributeEvalInfoGainAttributeEvalOneRAttributeEvalReliefFAttributeEvalSymmetricalUncertAttributeEval [Figure] Features [Figure]
[Figure]
E rr o r R a t e
CfsSubsetEvalCorrelationAttributeEvalGainRatioAttributeEvalInfoGainAttributeEvalOneRAttributeEvalReliefFAttributeEvalSymmetricalUncertAttributeEval ically, one may want to deﬁne a timeout for this stage to not evaluate overly unpromising candidates if a lot of time has already been used.
The feature set F is computed based on the results of diﬀerent ﬁltering techniques (Hall, 1999).
Diﬀerent techniques to rank the relevance of features exist.
In contrast to wrapping (Kohavi and John, 1997), ﬁltering techniques do not adopt (costly) learners to judge feature relevance and hence are typically cheap to compute.
We execute all such available ﬁlters, which gives us a set of rankings.
Then, for each of these rankings, we compute the performance when using the ﬁrst l features where l is increased until the performance starts to decrease.
To assess the performance, a pilot classiﬁer is evaluated on the respective feature set in some kind of cheap cross-validation.
In this sense, Naive AutoML uses existing pre-processors only as a source of suggestions for candidate feature sets.
To motivate this procedure, it is worth to have a look at the performance curves one obtains for the diﬀerent ﬁlters.
For each number x of features, we can plot the performance of some pilot learner if using only the ﬁrst x features in the ranking created by some ﬁlter.
Fig.
3 illustrates these curves for two datasets and using diﬀerent ﬁlters from WEKA.
In the case of the cnae-9 dataset (left), we can see that there is only marginal improvement after x = 100, and for the madelon dataset (right) we can even observe that performance decreases quickly.
In both cases, it is evident that only a relatively small portion of the features is needed to obtain the same or even better results compared to when all features are used (notice that the plot curves to not even show the full range of features for both datasets).
Note that the ﬁltering stage optimizes the feature set F independently of the outcomes of earlier stages.
Neither the results of the probing stage nor those of the scaling stage are used in the determination of F .
Those outcomes are only relevant in that they deﬁne the candidates to which the feature selection F should be added, and their order for the evaluation.
One important additional use case for the analysis of such curves is the reduction of evaluation times during the AutoML process.
Even if we do not need to reduce the dimensionality to obtain good results as in the case of madelon (right), we can often still approximate the prediction performance on the full feature set suﬃciently well.
For example, in the case of cnae-9 (left), we only need to use 100 of the 856 features to obtain comparable results, which should be good enough to steer the search process.
Since we know that the runtime of many learners often increases super-linearly (sometimes quadratically) in the number of attributes (Mohr et al., 2021), there is a huge potential in runtime reduction.
For example, the time to train a random forest on cnae-9 on some reference machine is 2.2s on average but only 0.5s on 100 attributes with almost the same outcome.
A.4 Meta-Learner Stage.
In this stage, each candidate of the input pool is taken and used as a template to derive new pipelines in which the base learner is wrapped into a meta-learner.
The feature transformation algorithms, if present, are not touched in this stage.
The algorithms we consider here as meta-learners are also sometimes called ( homogeneous ) ensembles.
The idea of those learners is to take several copies of a base learner and somehow combine them into a new augmented learner.
Typical examples are Bagging (Breiman, 1996) and AdaBoost (Freund and Schapire, 1998).
This is opposed to heterogeneous ensembles like Stacking (Wolpert, 1992) or majority vote ensembles, in which diﬀerent learners of diﬀerent types are combined.
Since we augment existing pipelines (with one learner), we only work with homogeneous ensembles here.
Forming a heterogeneous ensemble is a reasonable ﬁnal step.
This would be identical to the strategy pursued in (Feurer et al., 2015) to eventually take the best k learners seen so far and merge them into a voting ensemble.
Alternatively, more sophisticated approaches could try to optimize such an ensemble using previous observations.
A.5 Parameter Tuning.
This stage simply tries to ﬁnd better hyperparameters for the predictor in one or more candidate pipelines.
Given a total timeout for this phase, the candidates are optimized in the order of their performance with a (local) timeout and a maximum number of evaluations.
For algorithms with a small parameter space, all candidates can be enumerated, e.g. knearest neighbors with some reasonable candidate set for the number of neighbors.
For all other algorithms, standard techniques for ﬁnding good conﬁgurations such as SMAC (Hutter et al., 2011), Hyperband (Li et al., 2017), etc. can be employed.
In this paper, we even only adopt a simple random search.
A.6 (Validation-Based) Model Selection.
The default decision of Naive AutoML to select the best seen pipeline might not always be the best.
Despite the simplicity and naivity of Naive AutoML, there is some signiﬁcant optimization going on, potentially leading to over-ﬁtting.
The potential need of some validation has been recognized earlier (Mohr et al., 2018), and we adopt a similar strategy here.
More precisely, the idea is to keep back a certain portion of the original data that is not shown to the optimization process and only used in this ﬁnal stage.
The main diﬀerence to (Mohr et al., 2018) is that we do not add the validation data to the pool and then run a cross-validation on the augmented dataset, but we here simply conduct a “classical” single-fold validation on this hold-out set.
The data portion used for validation can be chosen dynamically based on the desired guarantees on the generalization performance.
For a set of m remaining competitive candidates, the Hoeﬀding bound allows us to estimate the out-of-sample error with
P ( | µ − ν | > ε ) < 2 me − 2 εn , where µ is the true out-of-sample error, ν is the error on the validation fold, and n is the size of the validation fold.
Given suﬃcient data and for a moderate number m , say m = 10, this can be quite a good bound and impose an important remedy against over-ﬁtting after an exhaustive optimization eﬀort.
Unfortunately, the number of validation samples available is often not suﬃcient to make the Hoeﬀding bound meaningful.
In most cases, we want to choose ε ≈ 0 .
01 and have the bound relatively small, say, 0 . 1.
To assure such a bound for even only one candidate (eﬀectively testing its performance), the validation fold must already contain roughlyexamples.
For m = 10, we would need 35,000 validation instances, which are often not available.
We hence propose to use both the “internal” score observed during the optimization process and the validation score for model selection and weight the two scores based on the validation set size.
To this end, we introduce a parameter ¯ n that quantiﬁes the number of validation instances required to exclusively use validation instances to compute the score.
Intuitively, ¯ n is the number of instances needed to get the desired certainty in the Hoeﬀding bound, e.g., 35,000 instances.
The satisfaction of the Hoeﬀding bound can then be expressed as
[Equation]
where n is the actual size of the validation fold.
However, a close-to-0 satisfaction does not necessarily mean a close-to-0 weight of the validation score.
For example, if we haveinstances in total and use 100 of them for validation, then 100 is probably far away from ¯ n , and the satisfaction is almost 0.
Still, the 100 instances are a valuable complement to the 400 instances used for training.
In this case, it makes more sense to weight the validation score based on the ratio between the number of validation instances and totally available instances; here, this would be 0.2.
Hence, instead of using the satisfaction of the validation fold size directly as a weight, we use it to determine the point on a linear scale between the above ratio and 1.
Formally, we deﬁne the pipeline score in the validation phase then to be
[Equation]
where φ int is the internal score obtained using only the data available for optimization, φ validate is the performance obtained using only the validation data (the pipeline has still been deﬁned not using these data points), n is the number of instances in the validation fold, and
[Equation]
In the last term, N is the total number of instances available.
Appendix B. Datasets
All datasets are available via the openml.org platform (Vanschoren et al., 2013).
Appendix C. Result Tables for Naive AutoML
The results of the experiments are shown in Table 2.
The reported metric is the error rate .
For each dataset and each approach, we report the trimmed mean (10% trimmed) with standard deviation.
Nan entries are caused by memory overﬂows.
We refrain from the now somewhat common practice of reporting average ranks, because, in our view, those obscure a lot of important details in the comparison, e.g. on which (and how many ) datasets which algorithm is better than another and by which margin .
The formatting semantics of the table is as follows.
The best entries (w.r.t to the trimmed mean) per machine learning library are formatted in bold, and those whose result distribution cannot be said to be statistically diﬀerent to the best one (according to a
Wilcoxcon signed rank test with conﬁdence 0.05) or where the performance diﬀerence is “irrelevant” ( < 0 . 01) are underlined.
With respect to the latter one we of course understand that in some cases such marginal diﬀerence can be relevant, but here we still treat them as equally good to give more signiﬁcance to the symbols used to denote improvements.
For each of the state-of-the-art techniques, we use the • or ◦ symbol to indicate a score that is “substantially” better or worse than the one obtained with Naive AutoML.
The symbols are used twice, once to compare against the “primitive” proﬁle and once comparing against the “full” proﬁle.
As above, substantial here means that it is not only statistically signiﬁcant but also that the absolute improvement is at least 0.01.
The symbol † is used instead of a • in cases in which auto-sklearn constructed pipelines with feature transformation algorithms not supported in the algorithm scheme of Naive AutoML; examples are the PCA or feature encodings based on trees.
Based on these results, we can now answer the three research questions above.
openmlid name
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
[Table] [Table]
4538 GesturePhaseSegmentationProcessed  5 4541 Diabetes130US  3 23512 higgs
[Table]
[Table]
white  7 40668 connect-
[Table]
3 40670 dna  3 40685 shuttle  7 40701 churn
[Table]
Satellite
[Table] [Table]
Internet-Advertisements  2
Australian  2 40982 steel
[Table]
plates
[Table]
fault  7 40983 wilt  2 40984 segment  7 41027 jungle chess 2pcs
raw endgame complete  3 41138 APSFailure  2 41142 christine  2 41143 jasmine  2 41144 madeline  2 41145 philippine  2 41146 sylvine  2 41147 albert  2 41150 MiniBooNE  2 41156 ada  2 41157 arcene  2 41158 gina  2 41159 guillermo  2 41162 kick  2 41163 dilbert  5 41164 fabert  7 41165 robert  10 41166 volkert  10 41167 dionis  355 41168 jannis  4 41169 helena  100 42732 sf-police-incidents  2
Click prediction small
[Figure]
[Figure]
[Figure]
[Figure]
[Table]