Automatic Termination for Hyperparameter Optimization
Abstract Bayesian optimization (BO) is a widely popular approach for the hyperparameter optimization (HPO) in machine learning.
At its core, BO iteratively evaluates promising configurations until a user-defined budget, such as wall-clock time or number of iterations, is exhausted.
While the final performance after tuning heavily depends on the provided budget, it is hard to pre-specify an optimal value in advance.
In this work, we propose an effective and intuitive termination criterion for BO that automatically stops the procedure if it is sufficiently close to the global optimum.
Our key insight is that the discrepancy between the true objective (predictive performance on test data) and the computable target (validation performance) suggests stopping once the suboptimality in optimizing the target is dominated by the statistical estimation error.
Across an extensive range of real-world HPO problems and baselines, we show that our termination criterion achieves a better trade-off between the test performance and optimization time.
Additionally, we find that overfitting may occur in the context of HPO, which is arguably an overlooked problem in the literature, and show how our termination criterion helps to mitigate this phenomenon on both small and large datasets.
1 Introduction
While the performance of machine learning algorithms crucially depends on their hyperparameters, setting them correctly is typically a tedious and expensive task.
Hyperparameter optimization (HPO) emerged as a new sub-field in machine learning that tries to automatically determine how to configure a machine learning model.
One of the most successful strategies for HPO is
Bayesian optimization (BO; Močkus, 1975; Chen et al., 2018; Snoek et al., 2012; Melis et al., 2018) - a powerful framework for sequentially optimizing a costly blackbox objective, that is the predictive performance of a model configured with certain hyperparameters.
BO iteratively searches for a better predictive performance via (i) training a probabilistic model on the evaluations of the models performance and (ii) selecting the most promising next hyperparameter candidate.
In practice, the quality of the solution found by BO heavily depends on a pre-defined budget, such as the number of BO iterations or wall-clock time.
If this budget is too small, BO might result into hyperparameters of poor predictive performance.
If the budget is too large, compute resources will be wasted.
The latter can be especially fragile in HPO when one cannot fully reduce the discrepancy between the validation and test errors, thus resulting in overfitting as we show in our experiments.
A naive approach suggests terminating BO if the best-found solution remains unchanged for some subsequent BO iterations.
Though the idea is sensible, it might be challenging to define a suitable number, since it is a fixed, predetermined choice, that does not take the observed data into account.
Another approach is to track the probability of improvement (Lorenz et al., 2016) or the expected improvement (Nguyen et al., 2017), and stop the optimization process once it falls below a given threshold.
However, determining this threshold may in practice be less intuitive than setting the number of iterations or the wall-clock time.
Instead of stopping BO completely, in McLeod et al.
(2018), it is proposed to switch to local optimization when the global regret is smaller than a pre-defined target.
This condition can also be used to terminate BO early, but it comes with additional complexity such as identifying a (convex) region for local optimization and again a predefined budget.
Automatically terminating the sequential procedure of BO is a rather under-explored topic, in contrast to the more widely considered orthogonal direction of speeding up HPO via stopping the model training.
A seminal idea there is to avoid the computation of low performing hyperparameters, e.g., by learning curves (Swersky et al., 2014), multi-fidelity approach Hyperband (Li et al., 2017), its combination (Klein et al., 2017), and further modifications like Hyperband BOHB (Falkner et al., 2018), asynchronous Hyperband (Li et al., 2020) and its model based version (Klein et al., 2020).
The profound distinction of our method with this line of works is in the problem setup: instead of stopping the training, our method aims to terminate the whole HPO process.
This orthogonality allows for combination of both ways of to achieve overall larger speed ups.
In this work, we propose a simple and interpretable automatic termination criterion for BO.
The criterion consists of two main ingredients: (i) high-probability confidence bound on the regret (i.e., the difference of our current solution to the global optimum) and (ii) the termination threshold.
The first already allows a user to specify a desired tolerance that defines how accurate should the final solution be compared to the global optimum.
For the case when cross-validation is used, we recommend a threshold based on the statistical properties of the cross-validation estimator.
This threshold takes into account the irreducible discrepancy between the actual HPO objective (i.e., performance on new data) and the target function optimized via BO (i.e., the validation error).
Our extensive empirical evaluation on a variety of HPO and neural architecture search (NAS) benchmarks suggests that our method is more robust and effective in maintaining the final solution quality than common baselines.
We also surface overfitting effects in HPO on both small and large datasets, arguably an overlooked problem, and demonstrate that our termination criterion helps to mitigate it.
2 Background
Bayesian optimization (BO) refers to a class of methods for gradient-free optimization of an objective 𝑓 : Γ → R in an iterative manner.
At every step 𝑡 , a learner selects an input 𝛾 𝑡 ∈ Γ and observes a noisy output
[Equation]
where 𝜀 𝑡 is typically assumed to be i.i.d.
(sub)-Gaussian noise with some variance 𝜎 2 𝜀 .
The decision of the next input to evaluate depends on a probabilistic model, used to approximate the objective 𝑓 , and an acquisition function, which determines the decision rule.
A popular choice for the probabilistic model is a Gaussian process (GP) : 𝑓 ∼ 𝐺𝑃 ( 𝜇,𝜅 ) (Rasmussen and Williams, 2006), specified by some mean function 𝜇 : Γ → R and some kernel 𝜅 : Γ × Γ → R .
As observations 𝑦 1: 𝑡 = [ 𝑦 1 , . . .
,𝑦 𝑡 ] ⊤ for the selected inputs 𝐺 𝑡 = { 𝛾 1 , . . .
,𝛾 𝑡 } are being collected, they are used to update the posterior belief of the model defined by the posterior mean 𝜇 𝑡 ( 𝛾 ) and variance 𝜎 2 𝑡 ( 𝛾 ) :
[Equation]
where ( 𝐾 𝑡 ) 𝑖,𝑗
=
𝜅 ( 𝛾 𝑖 ,𝛾 𝑗 ) and 𝜅 𝑡 ( 𝛾 ) 𝑇 =
[ 𝜅 ( 𝛾 1 ,𝛾 ) , . . . ,
𝜅 ( 𝛾 𝑡 ,𝛾 )] 𝑇 .
The next input to query is determined by an acquisition function that aims to trade off exploration and exploitation.
Common choices include probability of improvement (Kushner, 1963), entropy search (Hennig and Schuler, 2012) and GP upper-confidence bound (Srinivas et al., 2010) to name a few.
The convergence of BO can be quantified via (simple) regret :
[Equation]
where 𝛾 ∗ is the global optimizer of 𝑓 and 𝛾 ∗ 𝑡 = arg min
𝛾 ∈ 𝐺 𝑡 𝑓 ( 𝛾 ) .
Specifying adequate tolerance that defines how small the regret should be to terminate BO is of high importance as it determines both the quality and the cost of the solution.
However, this criterion cannot be directly evaluated in practice, as the input 𝛾 ∗ and the optimum 𝑓 ( 𝛾 ∗ ) are not known.
Hyperparameter optimization (HPO) is a classical application for BO.
Consider a supervised learning problem that requires to train a machine learning model (e.g., a neural network)
M on some feature-response data points
[Equation] x [Equation]
sampled i.i.d.
from some unknown data distribution 𝑃 .
The model is obtained by running a training algorithm
(e.g., optimizing the weights of the neural network via SGD) on D , both of which depend on hyperparameters
𝛾 (e.g., learning rates used, batch size, etc.).
We use the notation M 𝛾 ( x; D ) to refer to the prediction that the model produced by M makes for an input x, when trained with hyperparameters 𝛾 on data D .
Given some loss function ℓ (· , ·) , the population risk of the model on unseen data points is given by the expected loss E 𝑃 [ ℓ ( y , M 𝛾 ( x , D ))] .
The main objective of HPO is to identify hyperparameters 𝛾 , such that the resulting model minimizes the population risk:
[Equation]
In practice, however, the population risk cannot be evaluated since 𝑃 is unknown.
Thus, typically, it is estimated on a separate finite validation set D 𝑉 drawn from the same distribution 𝑃 .
Practical
HPO focuses on minimizing the empirical estimator
ˆ 𝑓 ( 𝛾 ) of the expected loss 𝑓 ( 𝛾 )
leading to the optimizer 𝛾 ∗ D :
[Equation]
At its core, BO-based HPO sequentially evaluates the empirical estimator ˆ 𝑓 ( 𝛾 𝑡 ) for promising hyperparameters 𝛾 𝑡 and terminates after some specified number of BO rounds, reporting the solution 𝛾 ∗ 𝑡 = arg min
[Equation] ˆ 𝑓 ( 𝛾 ) , where 𝐺 𝑡 = { 𝛾 1 , . . . ,𝛾 𝑡 } are the solutions considered so far.
We can define the simple regret for the reported solution w.r.t.
the validation loss by
[Equation]
Inconsistency in the optimization objective.
Importantly, the true HPO objective 𝑓 ( 𝛾 ) in Eq.
(3) and the empirical surrogate ˆ 𝑓 ( 𝛾 ) in Eq.
(4) used for tuning by BO generally do not coincide.
Therefore, existing BO approaches may yield sub-optimal solutions to the population risk minimization, even if they succeed in globally optimizing ˆ 𝑓 ( 𝛾 ) .
This issue, however, is typically neglected in practical HPO as well as a potential overfitting to the validation error.
In contrast, we propose a termination condition for BO motivated by the discrepancy in the objectives.
3 Termination criterion for Hyperparameter Optimization
This section firstly motivates why early termination of HPO can be beneficial and then addresses the following two questions: (1) How to estimate the unknown simple regret and (2)
What threshold of the simple regret can be used to stop HPO.
3.1 Motivation for the termination criterion
We start by analysing the effect of optimizing ˆ 𝑓 in lieu of 𝑓 .
We observe that challenges in optimizing 𝑓 are both due to the statistical error of the empirical BO objective ˆ 𝑓 ( 𝛾 ) and the sub-optimality of the BO candidates, encoded in the simple regret ˆ 𝑟 𝑡 .
The key insight of the following proposition is that iteratively reducing ˆ 𝑟 𝑡 to 0 may not bring any benefits if the statistical error dominates.
Proposition 1.
Consider the expected loss 𝑓 and its estimator ˆ 𝑓 defined, respectively, in Eqs.
(3) and (4) , and assume the statistical error of the estimator is bounded as ∥ ˆ 𝑓 − 𝑓
Let 𝛾 ∗ and 𝛾 ∗ D be their optimizers: 𝛾 ∗ = arg min 𝛾 ∈ Γ 𝑓 ( 𝛾 ) and
𝛾 ∗ D = arg min 𝛾 ∈ Γ
ˆ 𝑓 ( 𝛾 ) .
Let 𝛾 ∗ 𝑡 be some candidate solution to min 𝛾 ∈ Γ ˆ 𝑓 ( 𝛾 ) with sub-optimality in function value ˆ 𝑟 𝑡 : = ˆ 𝑓 ( 𝛾 ∗ 𝑡 ) − ˆ 𝑓 ( 𝛾 ∗ D ) .
Then the gap in generalization performance 𝑓 ( 𝛾 ∗ 𝑡 ) − 𝑓
( 𝛾 ∗ ) can be bounded as follows:
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] Moreover, without further restrictions on 𝑓 , ˆ 𝑓 , 𝛾 ∗ 𝑡 and 𝛾 ∗ , the upper bound is tight.
Proof: The equality in Eq.
(6) is due to adding and subtracting the same values.
The inequality in Eq.
(7) results from the following bounds:
[Equation]
The proposition bounds the sub-optimality of the target objective 𝑓 in terms of the statistical error 𝜖 𝑠𝑡 and the simple regret ˆ 𝑟 𝑡 .
This naturally suggests terminating HPO at a candidate 𝛾 ∗ 𝑡 for which the simple regret ˆ 𝑟 𝑡 is of the same magnitude as the statistical error 𝜖 𝑠𝑡 , since further reduction in ˆ 𝑟 𝑡 may not improve notably the true objective.
However, neither of the quantities 𝜖 𝑠𝑡 and ˆ 𝑟 𝑡 are known.
Below, we propose a termination criterion that relies on estimates of both quantities.
Firstly, we show how to use confidence bounds on ˆ 𝑓 ( 𝛾 ) to obtain high probability upper bounds on the simple regret ˆ 𝑟 𝑡 (Srinivas et al., 2010; Ha et al., 2019).
Secondly, we estimate the statistical error 𝜖 𝑠𝑡 in the case of cross-validation (Stone, 1974; Geisser, 1975) where the model performance is defined as an average over several training-validation runs.
To this end, we rely on the statistical characteristics (i.e., variance or bias) of such cross-validation-based estimator that are theoretically studied by
Nadeau and Bengio (2003) and Bayle et al.
3.2 Building blocks of the termination criterion
Upper bound for the simple regret ˆ 𝑟 𝑡 .
The key idea behind bounding ˆ 𝑟 𝑡 is that, as long as the
GP-based approximation of ˆ 𝑓 (·) is well-calibrated, we can use it to construct high-probability confidence bounds for ˆ 𝑓 (·) .
In particular, Srinivas et al.
(2010) show that, as long as ˆ 𝑓 has a bounded norm in the reproducing kernel Hilbert space (RKHS) associated with the covariance function 𝜅 used in the GP, ˆ 𝑓 ( 𝛾 ) is bounded (with high probability) by lower and upper confidence bounds lcb 𝑡 ( 𝛾 )
=
𝜇 𝑡 ( 𝛾 ) − √︁ 𝛽 𝑡 𝜎 𝑡 ( 𝛾 ) and ucb 𝑡 ( 𝛾 ) =
𝜇 𝑡 ( 𝛾 ) + √︁ 𝛽 𝑡 𝜎 𝑡 ( 𝛾 ) .
Hereby, 𝛽 𝑡 is a parameter that ensures validity of the confidence bounds (see Appendix A.2.3 for practical discussion and ablation study).
Consequently, we can bound the unknown ˆ 𝑓 ( 𝛾 ∗ 𝑡 ) and ˆ 𝑓 ( 𝛾 ∗ D ) that define the sub-optimality ˆ 𝑟 𝑡 :
[Equation]
where the inequality for ˆ 𝑓 ( 𝛾 ∗ 𝑡 ) is due to the definition of the reporting rule 𝛾 ∗ 𝑡
= arg min 𝛾 ∈ 𝐺 𝑡
ˆ 𝑓 ( 𝛾 ) over the evaluated points 𝐺 𝑡 = { 𝛾 1 , . . .
,𝛾 𝑡 } .
We illustrate the idea with an example in Fig. 1.
Termination threshold.
We showed how to control the optimization error via the (computable) regret upper bound ¯ 𝑟 𝑡 above.
We now explain when to stop BO, i.e., how to choose some threshold 𝜖 𝐵𝑂 and an iteration 𝑇 : ¯ 𝑟 𝑇 ≤ 𝜖 𝐵𝑂 .
Following Proposition 1, we suggest setting 𝜖 𝐵𝑂 to be of similar magnitude as the statistical error 𝜖 𝑠𝑡 of the empirical estimator (since smaller regret ˆ 𝑟 𝑡 is not beneficial when 𝜖 𝑠𝑡 dominates).
In case of cross-validation being used for HPO, one can estimate this statistical error 𝜖 𝑠𝑡 and we further discuss how it can be done. [Figure] Threshold = [Figure]
Stop iteration Cross-validation is the standard approach to compute an estimator ˆ 𝑓 ( 𝛾 ) of the population risk.
The data D is partitioned into 𝑘 equal-sized sets D 1 , . . .
, D 𝑘 used for (a) training the model
M 𝛾 (· ; D − 𝑖 ) , where D − 𝑖 = ∪ 𝑗 ≠ 𝑖 D 𝑖 (i.e., training on all but the 𝑖 -th fold), and (b) validating M 𝛾 (· ; D 𝑖 ) on the 𝑖 -th fold of the data.
These two steps are repeated in a loop 𝑘 times, and then the average over 𝑘 validation results is computed.
The statistical error 𝜖 𝑠𝑡 of an estimator can be characterised in terms of its variance
Var ˆ 𝑓 ( 𝛾 ) =
E [( ˆ 𝑓 ( 𝛾 ) − E ˆ 𝑓 ( 𝛾 )) 2 ] and bias 𝐵 ( 𝛾 ) =
E [ ˆ 𝑓 ( 𝛾 )] − 𝑓
( 𝛾 ) , where the latter can be neglected in case of cross-validation (Bayle et al., 2020).
Though the variance Var ˆ 𝑓 ( 𝛾 ) of the cross-validation estimate is generally unknown, Nadeau and Bengio (2003) propose an unbiased estimate for it.
Specifically, for the sample variance (denoted as 𝑠 2cv ) of 𝑘 -fold cross-validation, a simple post-correction technique to estimate the variance Var ˆ 𝑓 ( 𝛾 ) is Var
[Equation]
where | D 𝑖 | , | D − 𝑖 | are the set sizes.
For example, in the case of 10-fold cross-validation we have
Var ˆ 𝑓 ( 𝛾 ) ≈ 0 .
21 𝑠 2cv ( 𝛾 ) .
We are now ready to propose our termination condition in the following.
Termination condition for BO. Consider the setup of Proposition 1 where ˆ 𝑓 (·) is a cross-validationbased estimator being iteratively minimized by BO.
Let 𝛾 ∗ 𝑡 the solution reported in round 𝑡 , and ¯ 𝑟 𝑡 defined in Eq.
(8) be the simple regret bound computed at each iteration 𝑡 .
Let the variance Var ˆ 𝑓 ( 𝛾 ∗ 𝑡 ) of the estimator ˆ 𝑓 (·) be approximated according to Eq.
Then, BO is terminated once:
[Equation]
Intuitively, the termination is triggered once the maximum plausible improvement becomes less than the standard deviation of the estimate.
This variance-based termination condition adapts to different algorithms or datasets and its computation comes with negligible computational cost on top of cross-validation.
The pseudo-code for the criterion is summarised in Algorithm 1.
If cross-validation cannot be used or is computationally prohibitive, the user can define the right-hand side of the termination condition.
In this case, the upper bound on the left-hand side still has an intuitive interpretation: the user can set the threshold based on their desired solution accuracy.
This case is demonstrated in Fig.
1, with an example of automatic termination for tuning an MLP.
4 Experiments
The main challenge of any termination criterion for HPO is to balance between reducing runtime and performance degradation.
We thus study in experiments how the speed-up gained from different termination criterions affects the final test performance.
To this end, we define two new metrics that account for the trade-off between resources saved and performance drop and provide a list of reasonable baselines.
The code to reproduce the experimental results is publicly available. ∗
Baselines.
Since automatic BO terminating is a rather under-explored topic, we consider the following baselines that are, to the best of our knowledge, the only ones directly related to our method:
• Näive convergence test controlled by a parameter 𝑖 (referred as Conv- 𝑖 ): stopping BO if the best observed validation metric remains unchanged for 𝑖 consecutive iterations.
It is challenging to define a suitable 𝑖 suitable across different benchmarks since 𝑖 is a fix, predetermined choice, that does not take the observed data into account (in contrast to our method that refines the regret estimation).
We consider common in practice values 𝑖 = { 10 , 30 , 50 } and study other values in Appendix.
• Threshold for Expected improvement (EI; Nguyen et al., 2017): stopping BO once EI drops below a pre-defined threshold.
Choosing a threshold crucially depends on a problem at hand, e
g., values studied by the original paper result in too aggressive stopping across a range of our experiments.
We thus extend it with a more finer grained grid resulting into { 10 − 9 , 10 − 13 , 10 − 17 } .
• Threshold for Probability of improvement (PI; Lorenz et al., 2016): stopping BO once PI drops below a pre-defined threshold.
Similar to EI baseline, we tune the threshold and use { 10 − 5 , 10 − 9 , 10 − 13 } .
Metrics.
We measure the effectiveness of a termination criterion via two metrics quantifying (i) the change in test error on a held-out dataset and (ii) the time saved.
Given a budget of 𝑇 BO iterations, we compare the test error 𝑦 𝑇 after 𝑇 iterations to the test error 𝑦 𝑒𝑠 after early stopping is triggered.
For each experiment, we compute the relative test error change RYC as RYC
[Equation]
−
[Equation] 𝑒𝑠 max ( 𝑦 [Equation] This allows us to aggregate the results over different algorithms and datasets, as RYC ∈ [− 1 , 1 ] and can be interpreted as follows:
A positive RYC represents an improvement in the test error when applying early stopping, while a negative RYC indicates the opposite.
Similarly, let the total training time for a predefined budget 𝑇 be 𝑡 𝑇 and the total training time when early stopping is triggered be 𝑡 𝑒𝑠 .
Then the relative time change RTC is defined as
[Equation]
indicates a reduction in total training time, RTC ∈ [ 0 , 1 ] .
While reducing training time is desirable, it should be noted that this can be achieved through any simple stopping criterion (e.g., consider interrupting HPO with a fixed probability after every iteration).
In other words, the RTC is not a meaningful metric when decoupled from the RYC and, thus, both need to be considered in tandem.
Selecting the data for the bound estimate.
Since we are only interested in the upper bound of the simple regret, we conjecture that using only the top performing hyperparameter evaluations may improve the estimation quality.
To validate this, we use results of BORE (Tiao et al., 2021) on the naval dataset from HPO-Bench (Klein and Hutter, 2019) where we can quantify the true regret (see Section 4.2).
We compute the upper bound by Eq.
(8) using three options: 100%, top 50% or top 20% of the hyperparameters evaluated so far and measure the distance to the true regret (see Fig. 2).
From Fig.
2, fitting a surrogate model with all the hyperparameter evaluations poses a challenge for estimating the upper bound of the regret, which is aligned with recent findings on more efficient
BO with local probabilistic model, especially for high-dimensional problems (Eriksson et al., 2019).
Using the top 20% evaluations gives the best upper bound estimation quality in the median, at the cost of the most under-estimations of the true regret (2553).
Our method would stop too early due to the under-estimation, thus negatively impacting the RYC score, as shown in Fig.
As a result, we use the top 50% hyperparameters evaluations for the upper bound estimation throughout this paper .
[Figure]
4.1 BO for hyperparameter tuning with cross-validation
We tune XGBoost (XGB; 9 hyperparameters) and Random Forest (RF; 3 hyperparameters) onsmall tabular datasets, where we optimize error rate for classification and rooted mean square error for regression, computed via 10-fold cross validation.
Methods setup.
We use a Matérn 5/2 kernel for the GP and its hyperparameters are found based on a type II maximum likelihood estimation (see Appendix A.1 for more details).
The termination is triggered only after the first 20 iterations to ensure a robust fit of GP.
We use Eq.
(9) to compute our stopping threshold, i.e., Var ˆ 𝑓 ( 𝛾 ) ≈ 0 .
21 𝑠 2cv ( 𝛾 ) .
We additionally use empirical scaling 0 .
5 𝑠 2cv ( 𝛾 ) to study the affect of varying the magnitude.
Results.
We present the RYC-RTC results aggregated across the datasets in Fig.
3, Tableand Figs.
8 and 9 (Appendix A.1).
The main take-away from Fig.
3 is as follows: more aggressive early stopping might indeed speed up but it leads to worse test performance, both in terms of average and standard deviation over the datasets.
In contrast, the desirable behavior is the trade-off between RYC and RTC, where lower RYC error bars are prioritized over lower RTC error bars.
In other words, the methods that adaptively stops BO, i.e., stops when it necessary and does not stop when it is not, is preferable.
Fig.
3 indicates that our method successfully maintains a high solution quality across a wide range of scenarios (lower RYC variance) via adapting the termination to the particular propblem (higher variance RTC).
The results also show an anticipated RYC-RTC trade-off for 𝑖 in Conv- 𝑖 and thresholds in EI and PI baselines, where the solution quality improves as the thresholds increases and consequently the speed-up drops.
The average RYC results in
Table 3 however are dominated by our method. [Figure]
4.2 Neural Hyperparameter and Architecture Search
In these experiments we study out termination criterion beyond the BO scope, showing its main advantage of being applicable for any iterative HPO method.
In addition, here we also show how to use our method if cross-validation is unavailable.
To demonstrate this, we apply it to several state-of-the-art methods: TPE (Bergstra et al., 2011), BORE (Tiao et al., 2021), GP-BO (Snoek et al., 2012) as well as random search (RS) (Bergstra and Bengio, 2012).
Benchmarks.
We consider two popular tabular benchmark suites: NAS-HPO-Bench (Klein and Hutter, 2019), which mimics the hyperparameter and neural architecture search of multi-layer perceptrons on tabular regression datasets, and NAS-Bench-201 (Dong and Yang, 2020) for neural architecture search on image classification datasets.
Notice that for NAS-Bench-201, we used validation metrics to compute RYC instead of test metrics, thus, no positive RYC scores are observed.
For a detailed description of these benchmarks we refer to the original paper.
Methods setup.
We consider the following thresholds on the final regret { 0 . 0001 , 0 .
001 , 0 . 01 } , corresponding to a loss of performance of 0 .
1% and 1%, respectively.
Note that, it is easier to set the threshold for our method because it is a threshold on the regret in the metric space that users aim to optimize and it gives more explicit control of this trade-off, thus making it more interpretable.
For each method and dataset, we perform 50 independent runs with a different seed.
Results.
Fig.
4 and Fig.
5 show results for BORE and the rest can be found in the Appendix.
While no method Pareto dominates the others, our termination criterion shows a similar trend as in
Section 4.1 by prioritising accuracy over speed.
Users choose the threshold based on their preference regarding the speed-accuracy trade-off, i.e, a higher threshold saves more wall-clock time but potentially leads to a higher drop in performance.
We further show the distribution of true regrets at the stopping iteration triggered by our method with the considered thresholds on HPO-Bench in Fig.
From Fig.
5, with a high threshold of 0 .
01, all the experiments (4 datasets with 50 replicates) are early stopped by our method and 41 (20%) experiments end up with true regret being higher than the threshold.
With a low threshold of 0.0001, 112 experiments are stopped and 12 (10.7%) experiments end up with true regret above the threshold.
In short, our method achieves 80% to 90% success rate where the true regret is within the user-defined tolerance.
For every method we aggregated the scores over datasets with other HPO optimizers in Fig.(see Appendix A.2).
We can see that the speed up of the convergence check baseline is affected very mildly by the optimizers while the RYC scores largely depend on the optimiser: RYC scores with random search are worse than with BORE.
In contrast, the RYC scores for our termination criterion are similar across optimisers, especially for smaller thresholds.
On the other hand, the speed up [Figure] for a given threshold tends to vary.
This can be explained by the difference of the optimizer’s performance, for examples random search is not as efficient as BORE, and hence the regret is mostly above the stopping threshold.
In summary, while convergence check baselines are by design robust in terms of time saved, our method is more robust in terms of maintaining the solution quality.
4.3 Overfitting in BO for Hyperparameter Optimization
Proposition 1 emphasises an important problem of BO-based HPO: while focusing (and minimizing) the validation error, we cannot fully reduce the discrepancy between the validation and test errors.
Empirically, we show this might happen when correlation between the test and validation errors is low, thus improvement in validation performance does not lead to the better test results.
A particular example of such low correlation in the small error region is presented in Fig.
10 (Appendix A.2) when tuning XGB and Random Forest on tst-census dataset.
In our experiments, a positive RYC score is an indicator of overfitting, showing that the test error at the terminated iteration is lower than the test error in the final round.
We observe positive
RYC scores in both Fig.
3 and Fig.
4, one with cross-validation on small datasets and one with medium-sized datasets.
Hence, we would like raise attention to the possible overfitting issue that occurs in HPO for which our method can be used as a plugin to mitigate overfitting.
5 Conclusion
Despite the usefulness of hyperparameter optimizations (HPO), setting a budget in advance remains a challenging problem.
In this work, we propose an automatic termination criterion that can be plugged into many common HPO methods.
The criterion uses an intuitive and interpretable upper bound of simple regret, allowing users explicitly control the accuracy loss.
In addition, when cross-validation is used in the evaluations of hyperparameters, we propose to use an analytical threshold rooted from the variance of cross validation results.
[Figure] The experimental results suggest that our method can be robustly used across many HPO optimizers.
Depending on the user-defined thresholds, with 80% to 90% chance, our method achieves true regret within that threshold, saving unnecessary computation and reducing energy consumption.
We also observe that overfitting exists in HPO even when cross-validation is used.
We hope our work will draw the attention of the HPO community to the practical questions of how to set budget in advance and how to mitigate overfitting when tuning hyperparameters in machine learning.
6 Limitations and Broader Impact
Further study of better variance estimate might be beneficial and we leave the modeling and estimation of this variance for the future work.
In a broader scope of BO beyond HPO, a similar idea can be used when the point evaluations are not only corrupted by random noise but also adversarial corruptions (resulting again in a discrepancy between the true objective and the computable target).
In a broader context, we highlight that BO can reduce the computational cost required to tune
ML models, mitigating the energy consumption and carbon footprint associated with brute force techniques such as random and grid search.
The automatic termination criterion we presented in this work can have a positive societal impact by further reducing the cost of tuning ML models.
7 Acknowledgements
This research has been gratefully supported by NCCR Automation grant 51NF40 180545.
The authors thank the anonymous reviewers of this paper for their helpful feedback.
A Appendix A.1 Experiments setting
A.1.1 BO setting.
We used an internal BO implementation where expected improvement (EI) together with Mat‘ern-52 kernel in the GP are used.
The hyperparameters of the GP includes output noise, a scalar mean value, bandwidths for every input dimension, 2 input warping parameters and a scalar covariance scale parameter.
The closest open-source implementations are GPyOpt using input warped GP ∗ or AutoGluon BayesOpt searcher † .
We maximize type II likelihood to learn the
GP hyperparameters in our experiments.
We ran 200 iterations sequentially for benchmarks with cross validation, 500 iterations for NAS-HPO-Bench, 200 for NAS-Bench-201.
A.1.2 Algorithm.
We present the pseudo-code for the termination criterion in Algorithm 1. Algorithm
Require: Model M 𝛾 parametrized by 𝛾 ∈ Γ , data { D 1 , . . .
, D 𝑘 } for 𝑘 -fold cross-validation, acquisition function 𝛼 ( 𝛾 ) 1: Initialize 𝑦 ∗ 𝑡 = +∞ and 𝐺 𝑡 = {} 2: for 𝑡 = 1 , 2 , . . .
do 3:
Sample 𝛾 𝑡 ∈ arg max 𝛾 ∈ Γ 𝛼 ( 𝛾 ) 4: for 𝑖 = 1 , 2 , . .
,𝑘 do 5:
Fit the model M 𝛾 𝑡 (· ; D − 𝑖 ) , where D − 𝑖 = ∪ 𝑗 ≠ 𝑖 D 𝑖 6:
Evaluate the fitted model 𝑦 𝑖𝑡 =
| D 𝑖 | (cid:205) x 𝑖 , y 𝑖 ∈ D 𝑖 ℓ ( y 𝑖 , M 𝛾 𝑡 ( x 𝑖 , D − 𝑖 )) 7: end for 8:
Calculate the sample mean 𝑦 𝑡 = 1 𝑘 (cid:205) 𝑘 𝑦 𝑖𝑡 , 9: if 𝑦 𝑡 ≤ 𝑦 ∗ 𝑡 then 10:
Update 𝑦 ∗ 𝑡 = 𝑦 𝑡 and the current best 𝛾 ∗ 𝑡 = 𝛾 𝑡 11:
Calculate the sample variance 𝑠 2cv
= 1 𝑘 (cid:205) 𝑖 ( 𝑦 𝑡 − 𝑦 𝑖𝑡 )12:
Calculate the variance estimate Var ˆ 𝑓 ( 𝛾 ∗ 𝑡 ) ≈ (cid:16) 1 𝑘 + | D 𝑖 | | D − 𝑖 | (cid:17) 𝑠 2cv from Eq.
(9) 13: end if 14:
Update 𝐺 𝑡 = 𝐺 𝑡 − 1 ∪ 𝛾 𝑡 and 𝑦 1: 𝑡 = 𝑦 1: 𝑡 − 1 ∪ 𝑦 𝑡 15:
Update 𝜎 𝑡 , 𝜇 𝑡 with Eq.
Calculate upper bound ¯ 𝑟 𝑡 : = min 𝛾 ∈ 𝐺 𝑡 ucb 𝑡 ( 𝛾 ) − min 𝛾 ∈ Γ lcb 𝑡 ( 𝛾 ) for simple regret from Eq.
(8) 17: if the condition ¯ 𝑟 𝑡 ≤ √︃ Var ˆ 𝑓 ( 𝛾 ∗ 𝑡 ) holds then 18: terminate BO loop 19: end if 20: end for 21: Output: 𝛾 ∗ 𝑡
A.1.3 Search spaces for cross-validation experiments.
XGBoost (XGB) and RandomForest (RF) are based on scikit-learn implementations and their search spaces are listed in Table 1.
A.1.4 Datasets in cross validation experiments.
We list the datasets that are used in our experiments, as well as their characteristics and sources in Table 2.
For each dataset, we first randomly draw 20% as test set and for the rest, we use 10-fold cross validations for regression datasets and 10-fold stratified cross validation for classification datasets.
The actual data splits depend on the seed [Table] hyperparameter search space scale
[Table]
controlled in our experiments. For a given experiment, all the hyperparameters trainings use the same data splits for the whole tuning problem.
For the experiments without cross-validation, we use 20% dataset as validation set and the rest as training set. dataset [Table] classification
[Table]
classification
[Table] crimes classification [Table] openmlclassification
[Table]
openml farmads classification
[Table]
openmlclassification
[Table]
openml sylvine classification
[Table]
openml op100classification   openml openml classification   openml philippine classification
data.gov fabert classification   openml openml classification   openml openml regressionopenml tst
[Table]
census regression
[Table]
A.2 Detailed results
Fig.
8 demonstrates the results of threshold study EI and PI baselines for cross-validation.
Fig.shows results for extended set of parameter 𝑖 for Conv- 𝑖 baseline.
We also show the scatter plots of RTC and RYC scores for different automatic termination methods on HPO-Bench-datasets in Fig.
6 and the results on NAS-Bench-201 in Fig.
[Figure]
A.2.1 Detailed numbers
of RYC and RTC scores.
We report detailed RYC scores and RTC scores of different HPO automatic termination methods for the experiments in the main text in Table 3,
Table 4 and Table 5.
[Table]
[Figure]
[Figure]
A.2.2 Correlation between validation and test metrics.
In Fig.
10, we show the correlation between validation and test metrics of hyperparameters when tuning XGB and RF on tst-census dataset in
Fig.
A.2.3 The choice of 𝛽 𝑡 .
High-probability concentration inequalities (aka confidence bounds) are important to reason about the unknown objective function and are used for theoretically grounded convergence guarantees in some (GP-UCB-based) BO methods (Srinivas et al., 2010; Ha et al., 2019;
Kirschner et al., 2020; Makarova et al., 2021).
There, 𝛽 𝑡 stands for the parameter that balances [Figure]
is a fixed, predetermined choice, that does not take the observed data into account (in contrast to our method that refines the regret estimation as the data is being observed).
[Table]
between exploration vs. exploitation and ensures the validity of the confidence bounds.
The choice of 𝛽 𝑡 is then guided by the assumptions made on the unknown objective, for example, the objective being a sample from a GP or the objective having the bounded norm in RKHS (more agnostic case used in Section 3).
In our experiments, we follow the common practice of scaling down 𝛽 𝑡 which is usually used to improve performance over the (conservative) theoretically grounded values (see e.g., Srinivas et al.
; Kirschner et al.
; Makarova et al.
Particularly, throughout this paper, we
[Figure] [Table] [Figure] set 𝛽 𝑡 = 2 log (| Γ | 𝑡 2 𝜋 2 / 6 𝛿 ) where 𝛿 = 0 .
1 and | Γ | is set to be the number of hyperparameters.
We then further scale it down by a factor of 5 as defined in the experiments in Srinivas et al.
We provide an ablation study on the choice of 𝛽 𝑡 in Fig.
[Figure]
u pp e r b o un dt r u e r e g r e t
BORE on Naval dataset
t =
#HP (#negative=537) t = 0.2 (#negative=6654)
B Reproducibility Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[Yes] We discuss any societal impacts in Section 6.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
https://automl.cc/ethics-accessibility/ [Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results, including all requirements (e.g., requirements.txt with explicit version), an instructive README with installation, and execution commands (either in the supplemental material or as a url)?
[Yes] We provide the link to the code as footnote in Section 4.
(b) Did you include the raw results of running the given instructions on the given code and data?
[Yes] (c) Did you include scripts and commands that can be used to generate the figures and tables in your paper based on the raw results of the code, data, and instructions given?
[Yes] (d) Did you ensure sufficient code quality such that your code can be safely executed and the code is properly documented?
[Yes] (e) Did you specify all the training details (e.g., data splits, pre-processing, search spaces, fixed hyperparameter settings, and how they were chosen)?
[Yes] (f) Did you ensure that you compared different methods (including your own) exactly on the same benchmarks, including the same datasets, search space, code for training and hyperparameters for that code?
[Yes] (g) Did you run ablation studies to assess the impact of different components of your approach?
[Yes] (h)
Did you use the same evaluation protocol for the methods being compared?
[Yes] (i) Did you compare performance over time?
[Yes] (j) Did you perform multiple runs of your experiments and report random seeds?
[Yes] (k) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (l) Did you use tabular or surrogate benchmarks for in-depth evaluations?
[Yes] (m) Did you include the total amount of compute and the type of resources used (e.g., type of gpus, internal cluster, or cloud provider)?
[No] The lower bound of the regret estimation is the most compute intensive step and we have already included them in the open source repository.
(n) Did you report how you tuned hyperparameters, and what time and resources this required (if they were not automatically tuned by your AutoML method, e.g. in a nas approach; and also hyperparameters of your own method)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[No] (c) Did you include any new assets either in the supplemental material or as a url?
[Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[No] These are publicly available assets.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (irb) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]