A UTOMATIC T ERMINATION FOR H YPERPARAMETER
O PTIMIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Bayesian optimization (BO) is a widely popular approach for the hyperparameter optimization (HPO) of machine learning algorithms.
At its core, BO iteratively evaluates promising conﬁgurations until a user-deﬁned budget, such as wall-clock time or number of iterations, is exhausted.
While the ﬁnal performance after tuning heavily depends on the provided budget, it is hard to pre-specify an optimal value in advance.
In this work, we propose an effective and intuitive termination criterion for BO that automatically stops the procedure if it is sufﬁciently close to the global optima.
Across an extensive range of real-world HPO problems, we show that our termination criterion achieves better test performance compared to existing baselines from the literature, such as stopping when the probability of improvement drops below a ﬁxed threshold.
We also provide evidence that these baselines are, compared to our method, highly sensitive to the choices of their own hyperparameters.
Additionally, we ﬁnd that overﬁtting might occur in the context of HPO, which is arguably an overlooked problem in the literature, and show that our termination criterion mitigates this phenomenon on both small and large datasets.
I NTRODUCTION
While the performance of machine learning algorithms crucially depends on their hyperparameters, setting them correctly is typically a tedious and expensive task.
Hyperparameter optimization (HPO) emerged as a new sub-ﬁeld in machine learning that tries to automatically determine how to conﬁgure a machine learning algorithm.
One of the most successful strategies for HPO is Bayesian optimization (BO; Mo ˇ ckus, 1975;
Chen et al., 2018; Snoek et al., 2012; Melis et al., 2018) which iteratively trains a probabilistic model on the evaluations of the tuned algorithm.
to select the most promising next candidate point that trades-off exploration and exploitation.
In practice, the quality of the ﬁnal solution found by BO heavily depends on a user deﬁned budget, such as wall-clock time or the number of iterations, which needs to be deﬁned in advanced.
If this budget is too small, BO might return hyperparameters that result in a poor predictive performance.
If the budget is too large, compute resources will be wasted and, in some cases, it may result in overﬁtting as we will show in our experiments.
Automatically stopping BO is a rather under-explored topic in the literature.
A simple baseline is to stop if BO has not found a better solution than the current best incumbent for some successive iterations, which is in the same vein as early stopping for neural network training.
Another approach is to track probability of improvement (Lorenz et al., 2016) or expected improvement (Nguyen et al., 2017), and stop the optimization process once it falls below a given threshold.
However, determining this threshold may in practice be less intuitive than setting the number of iterations or the wall-clock time.
Instead of stopping BO completely,
McLeod et al.
(2018) propose to switch to local optimization when the global regret is smaller than a pre-deﬁned target.
This condition can also be used to terminate BO early, but it comes with additional complexity such as identifying a (convex) region for local optimization and again a predeﬁned budget.
In this work, we propose a simple and interpretable automatic termination criterion for BO.
In our criterion, we construct high-probability conﬁdence bound on the regret (i.e., the difference of our current solution to the global optimum) exploiting the probabilistic model of the objective.
Thus, users are now asked to specify a desired tolerance that deﬁnes how accurate should the ﬁnal solution be compared to global optimum .
In addition, we propose to determine the threshold via a cross-validation estimate of the generalization error.
This choice takes into account the irreducible discrepancy between the actual objective and the target function optimized via BO, namely the difference between the performance on new data (i.e., the population risk) and the validation error.
Our extensive empirical evaluation on a variety of HPO and NAS benchmarks suggests that our method is more robust and effective in maintaining the ﬁnal solution quality than common baselines (Lorenz et al., 2016;
Nguyen et al., 2017).
We also surface overﬁtting effects in HPO on both small and large datasets, arguably an overlooked problem in the literature, and demonstrate that our termination criterion helps to mitigate it.
B ACKGROUND
Bayesian optimization (BO) refers to methods of optimizing a black-box objective f : Γ → R in an iterative manner.
At every step t , a learner selects an input γ t ∈ Γ and observes a noisy output
[Equation]
where ε t is typically assumed to be i.i.d.
(sub)-Gaussian noise with some variance (proxy) σ 2 ε .
The decision of the next input to evaluate depends on a probabilistic model, used to approximate the objective f , and an acquisition function, which determines the decision rule.
A popular choice for the probabilistic model is a Gaussian process (GP) : f ∼ GP ( µ, κ ) (Rasmussen & Williams, 2006), speciﬁed by some mean function µ : Γ → R and some kernel κ : Γ × Γ → R .
As observations y 1: t =
[ y 1 , . . .
, y t ] (cid:62) for the selected inputs G t = { γ 1 , . . . , γ t } are being collected, they are used to update the posterior belief of the model deﬁned by the posterior mean µ t ( γ ) and variance σ 2 t ( γ ) as:
[Equation]
[Equation]
(2) where ( K t ) i,j = κ
( γ i , γ j ) and κ t ( γ ) T =
[ κ ( γ 1 , γ ) , . .
, κ ( γ t , γ )] T .
The next input to query is determined by an acquisition function that aims to trade off exploration and exploitation.
Common choices include probability of improvement (Kushner, 1963), entropy search (Hennig & Schuler, 2012), GP upper-conﬁdence bound (Srinivas et al., 2010), to name a few.
The convergence of BO can be quantiﬁed via (simple) regret , i.e., the sub-optimality in function value:
[Equation]
where γ ∗ is the global optimizer of f and γ ∗ t = arg min γ ∈ G t f ( γ ) .
Specifying adequate tolerance that deﬁnes how small the regret should be to terminate BO is of high importance as it determine both the quality and the cost of the solution.
However, this criterion cannot be directly evaluated in practice, as the input γ ∗ and the optimum f ( γ ∗ ) are not known.
[-Under certain regularity assumptions on f , they provide bounds for the number of iteration t required to achieve an (cid:15) -near-optimal solution for some given (cid:15) ≥ 
-]
Hyperparameter optimization (HPO) is a widely considered application for BO.
Consider a supervised learning setting training a machine learning model (e.g., a neural network)
M on some feature-response data points D
= { (x i , y i ) } n i =1 sampled i.i.d.
from some unknown data distribution
P .
The model is obtained by running a training algorithm
(e.g., optimizing the weights of the neural network via SGD) on D , and the model returned also depends on hyperparameters γ
(e.g., learning rates used, batch size, etc.).
We use the notation M γ (x; D ) to refer to the prediction that the model produced by M makes for an input x , when trained with hyperparameters γ on data D .
Given some loss function (cid:96) ( · , · ) , the population risk of the model on unseen data points is given by the expected loss E P [ (cid:96) (y , M γ (x , D ))] .
The main objective of HPO is to identify hyperparameters γ , such that the resulting model minimizes the population risk:
[Equation]
In practice, however, the population risk cannot be evaluated since P is unknown.
Thus, typically, it is estimated on a separate ﬁnite validation set D V drawn from the same distribution P .
As the result, practical HPO focuses on minimizing the empirical estimate ˆ f ( γ ) of the expected loss f ( γ ) leading to (probably different) optimizer γ ∗D :
[Equation]
At its core, BO-based HPO evaluates noisy empirical estimate ˆ f ( γ t ) for promising hyperparameters γ t for some ﬁnite number of iterations and the ﬁnal performance after termination heavily depends on that number. Alternatively, one can also terminate BO when sufﬁciently close to the global optima, i.e., using an analogue to the simple regret r t for the validation loss ˆ f ( γ ) and ˆ f ( γ ∗ t ) = min γ ∈ G
[Equation]
f [Equation] γ ) : [Equation]
Inconsistency in the optimization objective.
Importantly, the true HPO objective f ( γ ) in Eq.
(4) and the empirical surrogate ˆ f ( γ ) in Eq.
(5) used for tuning by BO generally do not coincide.
Therefore, existing BO approaches may yield sub-optimal solutions to the population risk minimization, even if they succeed in globally optimizing ˆ f ( γ ) .
This issue, however, is typically neglected in practical HPO.
In contrast, we propose a termination condition for BO motivated by the discrepancy in the objectives.
T ERMINATION CRITERION FOR H YPERPARAMETER O PTIMIZATION
This section ﬁrstly motivates why early termination of HPO can be beneﬁcial and then addresses the following two questions: (1) How to estimate the unknown simple regret and (2) What threshold of the simple regret can be used to stop HPO.
We start by analysing the true discrepancy of interest: between the population risk f ( γ ∗ t ) at some input γ ∗ t and true optimum f ( γ ∗ ) . We then observe that this discrepancy sums from the statistical error of the empirical BO objective ˆ f ( γ ) as well the sub-optimality of the BO candidates (encoded in the simple regret ˆ r t ).
The key insight of the following proposition is that iterative reducing of ˆ r t to 0 may not bring any beneﬁts if the statistical error dominates.
Proposition 1.
Consider the expected loss f and its estimator ˆ f deﬁned in Eqs.
(4) and (5) , respectively, and assume the statistical error of the estimator is bounded as ||
ˆ f − f || ∞ ≤ (cid:15) st for some (cid:15) st ≥ 0 .
Let γ ∗ and γ ∗D be their optimizers: γ ∗ =
arg min
γ ∈ Γ f ( γ ) and γ ∗D = arg min γ ∈ Γ ˆ f ( γ ) . Let γ ∗ t be some candidate solution to min γ ∈ Γ
ˆ f ( γ ) with sub-optimality in function value ˆ r t :
=
ˆ f ( γ ∗ t ) − ˆ f ( γ ∗D ) . Then the gap in generalization performance f ( γ ∗ t ) − f ( γ ∗ ) can be bounded as follows:
[Equation]
Moreover, without further restrictions on f , ˆ f , γ ∗ t and γ ∗ , the upper bound is tight.
Proof: While the second inequality is due to the deﬁnition of γ ∗ t , the others can proved as follows: [Equation] arg min γ
[Equation]
The proposition provides the discrepancy bound in terms of the statistical error (cid:15) st and simple regret ˆ r t . This naturally incites terminating HPO at a candidate γ ∗ t for which the simple regret ˆ r t is of the same magnitude as the statistical error (cid:15) st (as further reduction in ˆ r t may not improve notably the true objective).
However, neither of the quantities (cid:15) st and ˆ r t is known.
Below, we propose a termination criterion that relies on estimates of both quantities.
Firstly, we show how to use conﬁdence bounds on ˆ f ( γ ) to obtain high probability upper bounds on the simple regret ˆ r t Srinivas et al. ; Ha et al.
Secondly, we estimate the statistical error (cid:15) st in the case of cross-validation (Stone, 1974; Geisser, 1975) where the model performance is deﬁned as an average over several training-validation runs.
To this end, we rely on the statistical characteristics (i.e., variance or bias) of such cross-validation-based estimator that are theoretically studied by Nadeau & Bengio (2003); Bayle et al. (2020).
When cross validation is not used, one could deﬁne an intuitive threshold in advance due to the usage of interpretable upper bound of simple regret. [Figure] 3. B UILDING BLOCKS OF THE TERMINATION CRITERION
Upper bound for the simple regret ˆ r t .
The key idea behind bounding ˆ r t is that, as long as the GP-based surrogate model of ˆ f ( · ) is well-calibrated, we can use it to construct high-probability conﬁdence bounds for ˆ f ( · ) .
In particular, Srinivas et al.
(2010) show that as long as ˆ f has bounded norm in the reproducing kernel Hilbert space (RKHS) associated with the covariance function κ used in the GP, ˆ f ( γ ) is bounded (with high probability) by lower and upper conﬁdence bounds lcb t ( γ ) = µ t ( γ ) −√ β t σ t ( γ ) and ucb t ( γ )
= µ t ( γ )+ √ β t σ t ( γ ) .
Hereby, β t is a parameter that ensures validity of the conﬁdence bounds (see Appendix A.2.3 for practical discussion and ablation study).
Consequently, we can bound the unknown ˆ f ( γ ∗ t ) and ˆ f ( γ ∗D ) that deﬁne the sub-optimality ˆ r t :
[Equation]
where the inequality for ˆ f ( γ ∗ t ) is due to the deﬁnition of the reporting rule ˆ f ( γ ∗ t )
=
min
[Equation]
G t
[Equation]
f ( γ ) over the evaluated points G t = { γ 1 , . . .
, γ t } .
We illustrate the idea with an example in Fig.
1a.
Termination threshold.
We showed how to control the optimization error via the (computable) regret upper bound ¯ r t in the above and now we propose to stop the BO when ¯ r t is smaller than some threshold (cid:15) BO , i.e. at an iteration T : ¯ r T ≤ (cid:15) BO .
Following Proposition 1, we suggest setting (cid:15) BO to be of similar magnitude as the statistical error (cid:15) st (since smaller regret ˆ r t is not beneﬁcial when (cid:15) st dominates).
We now discuss how to estimate the statistical error (cid:15) st in case of cross-validation.
Cross-validation is the standard approach to compute an estimator ˆ f ( γ ) of the population risk.
The data D is partitioned into k equal-sized sets D 1 , . . .
, D k used for (a) training the model M γ ( · ; D − i ) , where
D − i = ∪ j (cid:54) =
i D i (i.e., training on all but the i -th fold), and (b) validating M γ ( · ; D i ) on the i -th fold of the data.
These two steps are then repeated in a loop k times, and then the average over k validation results is computed, yielding the noisy BO evaluation y
=
ˆ f ( γ ) + ε where the noise is due to the randomness in the training procedure or to epistemic uncertainty.
The statistical error (cid:15) st of an estimate can be characterised in terms of its variance and bias where the latter can be neglected in case of cross-validation (Bayle et al., 2020).
Though the variance of the cross-validation estimate Var ˆ f ( γ ) =
E [( ˆ f ( [Equation] γ )) 2 ] is generally unknown, Nadeau & Bengio (2003) propose an estimate for it.
Speciﬁcally, for the sample variance (denoted as s 2cv ) of k -fold cross-validation, a simple post-correction technique to estimate the variance Var ˆ f ( γ ) is
[Equation]
where |D i | , |D − i | are the set sizes.
For example, in the case of 10-fold cross-validation we have
Var ˆ f ( γ ) ≈ 0 .
21 s 2cv ( γ ) .
We are now ready to propose our termination condition in the following.
Termination condition for BO. Consider the setup of Proposition 1 where ˆ f ( · ) is a cross-validationbased estimator being iteratively minimized by BO. Let y t = ˆ f ( γ t ) + ε t be a noisy evaluation at the input γ t and γ ∗ t = arg min γ i ∈{ γ 1 ,...,γ t } y
i denote the best observed input by BO iteration t .
Let ¯ r t deﬁned in
Eq.
(7) be the simple regret bound computed at each iteration t .
Let the variance Var ˆ f ( γ ∗ t ) of the estimator ˆ f ( · ) be approximated according to Eq.
Then, BO is terminated once:
[Equation]
Intuitively, the termination is triggered once the maximum plausible improvement becomes less than the standard deviation of the estimate.
This variance-based termination condition adapts to different algorithms or datasets and its computation comes with negligible computational cost on top of cross-validation.
The pseudo code for the criterion is summarised in Appendix A.1.2.
If cross-validation cannot be used or is computationally prohibitive, the user can deﬁne the right-hand side of the termination condition.
In this case, the upper bound on the left-hand side still has an intuitive interpretation: the user can set the threshold based on their desired solution accuracy.
This case is demonstrated in Fig.
1b, with an example of automatic termination for tuning an MLP.
E XPERIMENTS
The main challenge of early stopping HPO is to ﬁnd the right trade-off between reducing runtime and performance degradation.
We thus study in experiments how the speed-up gained from different termination criterions affects the ﬁnal test performance.
4. E XPERIMENTAL S ETUP
We start with describing our experimental setup to validation our approach.
To establish a sensible experimental setup, we deﬁne two new metrics that account for the trade-off between resources saved and drop in ﬁnal performance and provide a list of reasonable baselines.
4.1. B ASELINES
To validate our proposed termination criterion we consider the following baselines:
• The ﬁrst baseline is a näive convergence test controlled by a parameter i : BO is stopped once the best observed validation metric remains unchanged for i consecutive iterations.
This convergence condition heavily relies on i , which needs to be chosen in advance.
We consider values commonly used in practice, namely i = { 10 , 30 , 50 } .
• We also compare to Lorenz et al.
; Nguyen et al.
(2017), which terminates BO once the value of the Probability of Improvement (PI) or Expected Improvement (EI) drops below a pre-deﬁned threshold.
Both acquisition functions do not live on a well-deﬁned scale, rendering it hard to identify a sensible threshold in practice.
We follow the recommendations from Nguyen et al.
; Lorenz et al.
and consider the following values for EI { 10 − 9 , 10 − 13 , 10 − 17 } and for PI { 10 − 5 , 10 − 9 , 10 − 13 } .
Empirically, we observe that all thresholds lead to signiﬁcant speeds up but, at the same time, to a severe degradation of the ﬁnal solution quality.
Because of space constraints, we only show the results for 10 − 17 for EI and 10 − 13 for PI, and report the full results in Appendix A.2.
4.1. M ETRICS
To measure the effectiveness of a termination criterion, we analyze two metrics to quantify the change in test error and the time saved.
Particularly, given a BO budget T, we compare the test error y es when early stopping is triggered to the test error y T .
For each experiment, we compute the relative test error change RYC as
[Equation]
This allows us to aggregate the results over different algorithms and datasets, as RYC ∈ [ − 1 , 1] and can be interpreted as follows:
A positive RYC represents an improvement in the test error when applying early stopping, while a negative RYC indicates the opposite.
Similarly, let the total training time for a predeﬁned budget T be t T and the total training time when early stopping is triggered be t es .
Then the relative time change RTC is deﬁned as
[Equation]
A positive RTC, where RTC ∈ [0 , 1] , indicates a reduction in total training time.
While reducing training time is desirable, it should be noted that this can be achieved through any simple stopping criterion (e.g., consider interrupting HPO with a ﬁxed probability after every iteration).
In other words, the RTC is not a meaningful metric when decoupled from the RYC and we will thus consider the two in tandem in the following experiments.
4. H OW TO SELECT THE DATA TO ESTIMATE THE BOUND ?
Since we are only interested in estimating the upper bound of the regret of the incumbent, we conjecture that using only the top performing hyperparameter evaluations may improve the estimation quality.
To validate this, we use BORE (Tiao et al., 2021) tuning results on the naval dataset from
HPO-Bench Klein & Hutter (2019)
(described in Section 4.4) where we can quantify the true regret.
We compute the upper bound by Eq.
(7) using three options: 100%, top 50% or top 20% of the hyperparameters evaluated so far.
The quality of the bound is measured by the difference to the true regret.
Results are shown in Fig.
2a, where the median of 50 replicates are shown as solid line and the 20’th and 80’th quantiles are shown as dashed and dotted lines, respectively.
We also show the number of negative differences (the upper bound is smaller than the true regret) in the legend next to the different options in the this ﬁgure.
From Fig.
2a, ﬁtting a surrogate model with all the hyperparameter evaluations poses a challenge for estimating the upper bound of the regret, which is aligned with recent ﬁndings on more efﬁcient
BO with local probabilistic model, especially for high-dimensional problems (Eriksson et al., 2019).
Using the top 20% evaluations gives the best upper bound estimation quality in the median, at the cost of the most under-estimations of the true regret (2553).
Our method would stop too early due to the under-estimation, which would negatively impact the quality measured by RYC scores, as shown in Fig.
2b.
As a result, we use top 50% hyperparameters evaluations for the upper bound estimation throughout this paper .
[Figure]
4. S TANDARD BO WITH CROSS - VALIDATION
We use standard BO with matérn 5/2 kernel in the GP surrogate.
The hyperparameters of the GP are estimated with type II maximum likelihood estimation.
We tune two algorithms: XGBoost (XGB) with 9 hyperparameters and Random Forest (RF) with three hyperparameters on 19 small tabular datasets using cross-validation.
The detailed BO setting, hyperparameter search space of the algorithms, as well as the characteristics of the datasets can be found in Appendix A.1.
We optimize classiﬁcation error or rooted mean square error for classiﬁcation and regression datasets, respectively.
For our stopping method, we use Eq.
(8) as our stopping threshold.
We apply automatic termination only after the ﬁrst 20 iterations to ensure a robust ﬁt of the surrogate models both for our method and the baselines.
We present the aggregated results across datasets for XGB and RF in Fig.
3 where the mean (shown in the dots) and standard deviation (shown in the error bars in both RTC and RYC dimensions) of RTC and RYC scores for different termination methods are shown.
From Fig.
3, a general trend on i in the convergence check baseline is visible: as i increases, the speed-up decreases while the solution quality increases.
The EI and PI based stopping criteria behave similarly in terms of both RTC and RYC scores.
The methods tend to stop BO very early, thus leading to signiﬁcant speed up.
However, maybe not surprisingly, such an aggressive early stopping leads to worse test performance on average.
The convergence check baseline and our method could achieve various trade-offs between speed and solution quality effectively by changing patience hyperparameter or the post-correction term, and our method prioritizes quality over speed.
Furthermore, the standard deviations of the RYC and RTC scores also provide us with interesting insights.
The RYC variances of our method are usually smaller than the baselines ones, indicating that we successfully maintain a high solution quality across a wide range of scenarios.
On the other hand, the RTC variances of our method are usually higher than the baselines, which highlights that our method adapts to different scenarios rather than stopping BO at similar iterations.
[Figure]
4. N EURAL H YPERPARAMETER AND A RCHITECTURE S EARCH
A main advantage or our termination criterion is that it is applicable for any iterative HPO method.
To demonstrate this, we apply it to several state-of-the-art Bayesian optimization methods from the literature: TPE (Bergstra et al., 2011), BORE Tiao et al.
GP-BO (Snoek et al., 2012) as well as random search (RS) (Bergstra & Bengio, 2012).
For each method and dataset, we performindependent runs with a different seed.
We consider two popular tabular benchmark suites from the literature: NAS-HPO-Bench (Klein & Hutter, 2019), which mimics the hyperparameter and neural architecture search of multi-layer perceptrons on tabular regression datasets, and NAS-Bench-201 (Dong & Yang, 2020) for neural architecture search on image classiﬁcation datasets.
Notice that for NAS-Bench-201, we used validation metrics to compute RYC instead of test metrics, thus, no positive RYC scores are observed.
For a detailed description of these benchmarks we refer to the original paper.
We consider the following thresholds on the ﬁnal regret { 0 . 0001 , 0 .
001 , 0 . 01 } in our experiments, corresponding to a loss of performance of 0 .
1% and 1% compared to global optimum.
Due to space constraints we will only show results for BORE in Figure Fig.
4 and Fig.
5, respectively, and provide all other results in Fig.
6 in the appendix.
While no method Pareto dominates the others, our termination criterion shows a similar trend as in
Section 4.3 and tends to prioritise accuracy over speed.
Users need to choose the threshold based on their own preference with respect to the speed-accuracy tradeoff, i.e a higher threshold saves more wall-clock time but potentially leads to a higher drop in performance.
However, our criterion beneﬁts from an intuitive and interpretable threshold.
We further show a distribution of true regrets at the stopping iteration triggered by our method with the considered thresholds on HPO-Bench in Fig.
5d.
From Fig.
5d, with a high threshold of 0 .
01 , all the experiments (4 datasets with 50 replicates) are early stopped by our method and 41 (20%) experiments end up with true regret being higher than the threshold.
With a low threshold of 0.0001, 112 experiments are stopped and 12 (10.7%) experiments end up with true regret being higher than the threshold.
In short, our method achieves 80% to 90% success rate of stopping the BO with true regret within the user-deﬁned tolerance.
[Figure]
For every method we aggregated the scores over datasets with other HPO optimizers in Fig.
We can see that the speed up of the convergence check baseline is affected very mildly by the optimizers while the RYC scores largely depend on the optimiser: RYC scores with random search are worse than with BORE.
In contrast, the RYC scores for our termination criterion are similar across optimisers, especially for smaller thresholds.
On the other hand, the speed up for a given threshold tends to vary.
This can be explained by the difference of the optimizer’s performance, for examples random search is not as efﬁcient as BORE, and hence the regret is mostly above the stopping threshold.
In summary, while convergence check baselines are by design robust in terms of time saved, our method is more robust in terms of maintaining the solution quality.
4. O VERFITTING IN BO FOR H YPERPARAMETER O PTIMIZATION
Proposition 1 emphasises an important problem of BO-based HPO: while focusing (and minimizing) the validation error, we cannot fully reduce the discrepancy between the validation and test errors.
Empirically, we show this might happen when correlation between the test and validation errors is low, thus improvement in validation performance does not lead to the better test results. A particular example of such low correlation in the small error region is presented in Fig.
8 (Appendix A.2) when tuning XGB and Random Forest on tst-census dataset. [Figure] d [Figure]
for HPO-Bench.
The number of stopped experiments and the number of “wrong” cases where the true regret is larger than the threshold are shown in the legend in Fig. 5d.
In our experiments, a positive RYC score is an indicator of overﬁtting, showing that the test error at the terminated iteration is lower than the test error in the ﬁnal round.
We observe positive RYC scores in both Fig.
3 and Fig.
4, one with cross-validation on small datasets and one with medium-sized datasets.
Hence, we would like raise attention to the possible overﬁtting issue that occurs in HPO for which our method can be used as a plugin to mitigate overﬁtting.
C ONCLUSION
Despite the usefulness of hyperparameter optimizations (HPO), setting a budget in advance remains a challenging problem.
In this work, we propose an automatic termination criterion that can be plugged into many common HPO methods.
The criterion uses an intuitive and interpretable upper bound of simple regret, allowing users explicitly control the accuracy loss.
In addition, when cross validation is used in the evaluations of hyperparameters, we propose to use an analytical threshold rooted from the variance of cross validation results.
The experimental results suggest that our method can be robustly used across many HPO optimizers.
Depending on the user-deﬁned thresholds, with 80% to 90% chance, our method achieves true regret within that threshold, saving unnecessary computation and reducing energy consumption.
We also observe that overﬁtting exists in HPO even when cross-validation is used.
We hope our work will draw the attention of the HPO community to the practical questions of how to set budget in advance and how to mitigate overﬁtting when tuning hyperparameters in machine learning.
Ethics Statement In a broader context, we highlight that BO can reduce the computational cost required to tune ML models, mitigating the electricity consumption and carbon footprint associated with brute force techniques such as random and grid search.
The automatic termination criterion we presented in this work can have a positive societal impact by further reducing the cost of tuning ML models.
On the other hand, BO is a general methodology to optimize gradient-free functions and is not limited to speciﬁc application domains.
Our early-stopping approach does not decrease the risk for misuse, calling for methods to enforce fairness constraints Perrone et al. (2021) as well as for care at model-deployment time.
Reproducibility Statement To improve the reproducibility, the code to rerun the experiments can be found at https://anonymous.4open.science/r/BO-early-stopping-555E , which will be made publicly available after the paper is published.
Besides, we also detail the experiments setting in relevant aspects: the choices for BO are in Appendix A.1.1, the search spaces of the algorithms are in Appendix A.1.
and the datasets sources and descriptions are in Appendix A.1.4.
A A PPENDIX A. E XPERIMENTS SETTING A.1. BO SETTING
We used an internal BO implementation where expected improvement (EI) together with Mat‘ern-52 kernel in the GP are used.
The hyperparameters of the GP includes output noise, a scalar mean value, bandwidths for every input dimension, 2 input warping parameters and a scalar covariance scale parameter.
The closest open-source implementations are GPyOpt using input warped GP 1 or AutoGluon BayesOpt searcher 2 .
We maximize type II likelihood to learn the GP hyperparameters in our experiments.
A.1. A LGORITHM A.1. S EARCH SPACES FOR CROSS VALIDATION EXPERIMENTS
XGBoost (XGB) and RandomForest (RF) are based on scikit-learn implementations and their search spaces are listed in Table 1.
A.1. D ATASETS IN CROSS VALIDATION EXPERIMENTS
We list the datasets that are used in our experiments, as well as their characteristics and sources in Table 2.
For each dataset, we ﬁrst randomly draw 20% as test set and for the rest, we use 10-fold cross validations for regression datasets and 10-fold stratiﬁed cross validation for classiﬁcation datasets.
The actual data splits depend on the seed controlled in our experiments.
For a given experiment, all the hyperparameters trainings use the same data splits for the whole tuning problem.
For the experiments without cross-validation, we use 20% dataset as validation set and the rest as training set.
A. D ETAILED RESULTS
We ﬁrst show the scatter plots of RTC and RYC scores for different automatic termination methods on HPO- Bench-datasets in Fig.
6 and the results on NAS-Bench-201 in Fig.
A.2. D ETAILED NUMBERS OF RYC AND RTC SCORES
We report detailed RYC scores and RTC scores of different HPO automatic termination methods for the experiments in the main text in Table 3, Table 4 and Table 5.
Require: Model M γ parametrized by γ ∈ Γ , data {D 1 , . . .
, D k } for k -fold cross-validation, acquisition function α ( γ ) 1: Initialize y ∗ t = + ∞ and G t = {} 2: for t = 1 , 2 , . . .
do 3:
Sample γ t ∈ arg max γ ∈ Γ α ( γ ) 4: for i = 1 , 2 , . . .
, k do 5:
Fit the model M γ ( · ; D − i ) , where D − i = ∪ j (cid:54) =
i D i 6:
Evaluate the ﬁtted model y it = 1 |D i | (cid:80) x i , y i ∈D i (cid:96) (y i , M γ (x i , D − i ))
7: end for 8:
Calculate the sample mean y t = 1 k (cid:80) k
y it , 9: if y t ≤ y ∗ t then 10:
Update y ∗ t = y t and the current best γ ∗ t = γ t 11:
Calculate the sample variance s 2cv = 1 k (cid:80) i ( y t − y it )12:
Calculate the variance estimate Var ˆ f ( γ ∗ t ) ≈ (cid:16) 1 k + |D i | |D − i | (cid:17) s 2cv from Eq.
(8) 13: end if 14:
Update G t =
G t − 1 ∪ γ t and y 1: t = y 1: t − 1 ∪ y t 15:
Update σ t , µ t with Eqs.
(1) and (2) 16:
Calculate upper bound ¯ r t :=
min γ ∈ G t ucb t ( γ ) − min γ ∈ Γ lcb t ( γ ) for simple regret from Eq.
if the condition ¯ r t ≤ (cid:113) Var ˆ f ( γ ∗ t ) holds then 18: terminate BO loop 19: end if 20: end for 21: Output: γ ∗ t
[Table]
A.2. C ORRELATION BETWEEN VALIDATION AND TEST METRICS
In Fig.
[Table]
we show the correlation between validation and test metrics of hyperparameters when tuning XGB and RF on tst-census dataset in Fig. 8. A.2. T HE CHOICE OF β t
High-probability concentration inequalities (aka conﬁdence bounds) are important to reason about the unknown objective function and are used for theoretically grounded convergence guarantees in some (GP-UCB-based) BO methods (Srinivas et al., 2010;
Ha et al., 2019; Kirschner et al., 2020; Makarova et al., 2021).
There, β t stands for the parameter that balances between exploration vs. exploitation and ensures the validity of the conﬁdence bounds.
The choice of β t is then guided by the assumptions made on the unknown objective, for example, the
[Table] [Table] [Table]
uci
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Figure]
objective being a sample from a GP or the objective having the bounded norm in RKHS (more agnostic case used in Section 3).
[Figure]
(d) is
[Table]
In our experiments, we follow the common practice of scaling down β t which is usually used to improve performance over the (conservative) theoretically grounded values (see e.g., Srinivas et al. ; Kirschner et al. ; Makarova et al. Particularly, throughout this paper, we set β t = 2 log( | Γ | t 2 π 2 / 6 δ ) where δ = 0 .
1 and | Γ | is set to be the number of hyperparameters.
We then further scale it down by a factor of 5 as deﬁned in the experiments in Srinivas et al.
We provide an
[Table]
[Table]
[Figure]
[Figure]
u pp e r b o un d - t r u e r e g r e t
BORE on Naval dataset
[Figure]