D YNAMIC AND E FFICIENT G RAY -B OX H YPERPARAMETER O PTIMIZATION FOR D EEP L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Gray-box hyperparameter optimization techniques have recently emerged as a promising direction for tuning Deep Learning methods.
However, the multibudget search mechanisms of existing prior works can suffer from the poor correlation among the performances of hyperparameter conﬁgurations at different budgets.
As a remedy, we introduce D Y HPO, a method that learns to dynamically decide which conﬁguration to try next, and for what budget .
Our technique is a modiﬁcation to the classical Bayesian optimization for a gray-box setup.
Concretely, we propose a new surrogate for Gaussian Processes that embeds the learning curve dynamics and a new acquisition function that incorporates multi-budget information.
We demonstrate the signiﬁcant superiority of D Y HPO against stateof-the-art hyperparameter optimization baselines through large-scale experiments comprising 50 datasets (Tabular, Image, NLP) and diverse neural networks (MLP,
CNN/NAS, RNN).
I NTRODUCTION
Hyperparameter Optimization (HPO) is arguably an acute open challenge for Deep Learning (DL), especially considering the crucial impact HPO has on achieving state-of-the-art empirical results.
Unfortunately, HPO for DL is a relatively under-explored ﬁeld and most DL researchers still optimize their hyperparameters via obscure trial-and-error practices.
On the other hand, traditional
Bayesian Optimization HPO methods (Snoek et al., 2012; Bergstra et al., 2011) are not directly applicable to deep networks, due to the infeasibility of evaluating a large number of hyperparameter conﬁgurations.
In order to scale HPO for DL, three main directions of research have been recently explored.
(i) Online HPO methods search for hyperparameters during the optimization process via meta-level controllers (Chen et al., 2017; Parker-Holder et al., 2020), however, this online adaptation can not accommodate all hyperparameters (e.g. related to architectural changes).
(ii)
Gradient-based HPO techniques, on the other hand, compute the derivative of the validation loss w.r.t.
hyperparameters by reversing the training update steps (Maclaurin et al., 2015; Franceschi et al., 2017; Lorraine et al., 2020), however, the reversion is not directly applicable to all cases (e.g. dropout rate).
The last direction, (iii) Gray-box HPO techniques discard sub-optimal conﬁgurations after evaluating them on lower budgets (e.g. after few epochs) (Li et al., 2017; Falkner et al., 2018).
In contrast to the online and gradient-based alternatives, gray-box approaches can be deployed in an off-the-shelf manner to all types of hyperparameters and architectures.
The gray-box concept is based on the intuition that a poorly-performing hyperparameter conﬁguration can be identiﬁed and terminated by inspecting the validation loss of the ﬁrst few epochs, instead of waiting for the full convergence.
The most prominent gray-box algorithm is Hyperband (Li et al., 2017), which runs random conﬁgurations at different budgets (e.g. different number of epochs) and successively halves these conﬁgurations by keeping only the top performers.
Follow-up works, such as BOHB (Falkner et al., 2018) or DEHB (Awad et al., 2021), replace the random sampling of Hyperband with more intelligent sampling based on Bayesian optimization or differentiable evolution.
Despite their great practical potential, existing gray-box methods suffer from a major issue.
The low-budget (few epochs) performances are not always a good indicator for the full-budget (full convergence) performances.
For example, a properly regularized network converges slower in the ﬁrst few epochs, however, typically performs better than a non-regularized variant after the full convergence.
In other words, there is a poor rank correlation of the conﬁgurations’ performances at different budgets.
As a remedy, we introduce D Y HPO, a gray-box method that dynamically decides how many conﬁgurations to try and how much budget to spend on each conﬁguration.
D Y HPO is a Bayesian Optimization (BO) approach based on Gaussian Processes (GP), that extends and adapts BO to the multi-budget (a.k.a. multi-ﬁdelity) case.
In this perspective, we propose a deep kernel GP that captures the learning dynamics (learning curve until the evaluated budget).
As a result, we train a kernel capable of capturing the similarity of a pair of hyperparameter conﬁgurations, even if the pair’s conﬁgurations are evaluated at different budgets (technically, different learning curve lengths).
Furthermore, we extend Expected Improvement (Jones et al., 1998) to the multi-budget case, by introducing a new mechanism for the incumbent conﬁguration of a budget.
The joint effect of modeling a GP kernel across budgets together with a dedicated acquisition function leads to D Y HPO achieving a statistically signiﬁcant empirical gain against state-of-the-art gray-box baselines (Falkner et al., 2018;
Awad et al., 2021), including prior work on multi-budget
GPs (Kandasamy et al., 2017;
Metz et al., 2020).
We demonstrate the performance of D Y HPO in three diverse popular types of deep learning architectures (MLP, CNN/NAS, RNN) and 50 datasets of three diverse modalities (tabular, image, natural language processing).
We believe our method is a step forward towards making HPO for DL practical and feasible.
Overall, our contributions can be summarized as follows:
• We introduce a novel Bayesian surrogate model that is based on a Gaussian Process with a deep kernel.
This surrogate model predicts the validation score of a machine learning model based on the hyperparameter conﬁguration and the learning curve.
• We derive a simple yet robust way to combine this surrogate model with Bayesian optimization, reusing most of the existing components currently used in traditional Bayesian optimization methods.
Finally
, we demonstrate the efﬁciency of our method for hyperparameter optimization and neural architecture search tasks compared to the current state-of-the-art methods.
As an overarching goal, we believe our method is an important step towards scaling HPO for DL.
M OTIVATION
As mentioned earlier, a major source of sub-optimality for the current gray-box techniques (Li et al., 2017; Falkner et al., 2018; Awad et al., 2021) is the poor rank correlation of performances at low and high budgets, which is endemic to the successive halving mechanism.
In essence, prior methods mistakenly discard good hyper-parameter conﬁgurations by myopically relying only on the early performance after few epochs.
In contrast, our method ﬁxes the endemic poor correlation of successive halving, by learning to decide which conﬁguration to continue optimizing for one more time-budget step (e.g. one more epoch).
In that perspective, our strategy resembles Freeze-Thaw (Swersky et al., 2014), however, differs in ﬁtting deep kernel GP surrogates with learning curves and a special multi-budget acquisition.
D Y HPO never discards a conﬁguration as all options remain latent until the acquisition selects the most promising one and allocates one more budget unit of optimization.
We illustrate the differences between our strategy and successive halving with the experiment of
Figure 1, where we showcase the HPO progress of three different methods on the ”Helena” dataset from the LCBench benchmark (Zimmer et al., 2021).
Random search is an example of a black-box approach that trains each candidate until completion without considering the intermediate scores.
Hyperband (Li et al., 2017) is a gray-box approach that statically pre-allocates the budget for a set of candidates (Hyperband bracket) according to a predeﬁned policy.
Finally, D Y HPO dynamically adapts the allocation of budgets for conﬁgurations after every HPO step.
The plots in the top row of Figure 1 show the learning curves of multiple neural networks trained with different hyperparameter conﬁgurations.
The darker the color of a learning curve, the later the model corresponding to the learning curve was trained during the optimization process.
In an optimal scenario, there should be no learning curve of darker color trained for a very long time if there is already a learning curve of lighter color with higher accuracy.
Since black-box functions do not consider intermediate responses, this trend is not observed for random search.
This is also not the case for gray-box methods such as Hyperband, because it is possible that no conﬁguration selected for a Hyperband bracket is outperforming the current best conﬁguration.
Since at least one
[Figure]
of the bracket candidates
must be fully trained according to the predeﬁned policy, we observe this suboptimal behavior in Figure 1.
This motivates our idea of dynamically deciding when we want to continue training a conﬁguration based on the intermediate response.
Clearly, D Y HPO invests only a small budget into conﬁgurations that show little promise as indicated by the intermediate scores.
R ELATED W ORK ON G RAY - BOX HPO
Multi-Fidelity Bayesian Optimization.
Bayesian optimization is a black-box function optimization framework that has been successfully applied in optimizing hyperparameter and neural architectures alike (Snoek et al., 2012; Kandasamy et al., 2018; Bergstra et al., 2011).
To further improve
Bayesian optimization, several works propose low-ﬁdelity data approximations of hyperparameter conﬁgurations.
Low-ﬁdelity approximations of hyperparameter conﬁgurations can be obtained by training on a subset of the data (Swersky et al., 2013; Klein et al., 2017) or terminating training early (Swersky et al., 2014).
Several methods extend Bayesian optimization to multi-ﬁdelity data by engineering new kernels suited for this problem (Swersky et al., 2013; 2014; Poloczek et al., 2017).
Kandasamy et al.
(2016) extends GP-UCB (Srinivas et al., 2010) to the multi-ﬁdelity setting by learning one Gaussian Process (GP) with a standard kernel for each ﬁdelity.
Their later work improves upon this method by learning one GP for all ﬁdelities that enables the use of continuous ﬁdelities (Kandasamy et al., 2017).
The work by Takeno et al.
(2020) follows a similar idea but proposes to use an acquisition function based on information gain instead of UCB.
While most of the works rely on GPs to model the surrogate function, Li et al.
(2020) use a Bayesian neural network that models the complex relationship between ﬁdelities with stacked neural networks, one for each ﬁdelity.
Multi-Fidelity Bandits.
Hyperband (Li et al., 2017) is a bandits-based multi-ﬁdelity method for hyperparameter optimization, that due to its simplicity and strong performance, enjoys great popularity.
The algorithm selects hyperparameter conﬁgurations at random and uses successive halving (Jamieson & Talwalkar, 2016) with different settings to early-stop less promising training runs.
Several improvements have been proposed to Hyperband with the aim to replace the random sampling of hyperparameter conﬁgurations with a more guided approach (Bertrand et al., 2017; Wang et al., 2018; Wistuba, 2017).
BOHB (Falkner et al., 2018) uses TPE (Bergstra et al., 2011) and builds a surrogate model for every ﬁdelity adhering to a ﬁxed-ﬁdelity selection scheme.
DEHB (Awad et al., 2021)
samples candidates using differential evolution which handles discrete and large hyperparameter search spaces better than BOHB.
Deep Kernel Learning with Bayesian Optimization.
We are among the ﬁrst to use deep kernel learning with Bayesian optimization and to the best of our knowledge the ﬁrst to use it for multiﬁdelity Bayesian optimization.
Rai et al.
(2016) consider the use of a deep kernel instead of a manually designed kernel in the context of standard Bayesian optimization, but, limit their experimentation to synthetic data and do not consider its use for hyperparameter optimization.
Perrone et al.
; Wistuba & Grabocka (2021) use a pre-trained deep kernel to warm start Bayesian optimization with meta-data from previous optimizations.
These approaches are multi-task or transfer learning methods that require the availability of meta-data from related tasks.
In contrast to prior work, we propose the ﬁrst deep kernel GP for multi-ﬁdelity HPO that is able to capture the learning dynamics across ﬁdelities/budgets, combined with an acquisition function that is tailored for the gray-box setup.
Furthermore, our work represents an important step towards scaling HPO for Deep Learning (DL), by demonstrating a statistically signiﬁcant reduction in terms of HPO time on a series of DL network architectures and a large set of diverse datasets.
P RELIMINARIES
Black-Box Optimization.
As mentioned earlier, the problem of optimizing hyperparameters can be modeled as a black-box optimization problem.
The objective is to maximize the response function f : X → R that returns the validation score for training a machine learning model with a hyperparameter conﬁguration x ∈ X .
In practice, this observation is noisy such that we observe in fact y i =
f ( x i ) + ε where ε ∼ N (0 , σ 2 n ) .
Gray-Box Optimization.
Since many machine learning algorithms allow to measure at various ﬁdelities, a relaxation of the black-box to the gray-box optimization problem is in many cases logical and allows for signiﬁcantly faster optimization.
The gray-box setting allows us to query conﬁgurations with a budget smaller than the maximum budget B .
Thus, we can query from the response function f : X × N → R where f i,j = f ( x i , j ) is the response after spending a budget of j on conﬁguration x i .
As before, these observations are noisy and we observe y i,j = f
( x i , j ) +
ε j where ε j ∼ N (0 , σ 2 j,n ) .
Please note, we assume that the budget required to query f
i,j + b after querying f i,j is only b .
While this is not necessarily the case for all problems, the models we consider are learned incrementally.
Furthermore, we are able to make use of the learning curve
Y i,j − 1 =
( y i, 1 , . . .
, y i,j − 1 ) when predicting f i,j .
Bayesian Optimization (BO).
BO is a general framework for solving black-box optimization problems and is very popular for optimizing hyperparameters.
It has two ingredients, i.e. a surrogate model and an acquisition function.
The surrogate is a probabilistic model which approximates the black-box function using the available information of function evaluations.
The acquisition function returns the expected utility for a conﬁguration given the surrogate model’s prediction.
The BO framework executes the following steps sequentially.
First, the surrogate model is trained on the available information about the black-box function, denoted as the history of evaluations D .
Then, the conﬁguration with the highest expected utility x i is evaluated and y i is observed.
The tuple ( x i , y i ) is added to D and the process is repeated until the HPO budget is exhausted.
A common choice for the surrogate model are Gaussian Processes (Rasmussen & Williams, 2006), a popular choice for the acquisition function is expected improvement (Jones et al., 1998).
Gaussian Processes (GP).
GPs are probabilistic machine learning models.
Given a training data set D = { ( x i , y i ) | i = 1 , . . . , n } with n data points, the Gaussian Process assumption is that y i is a random variable and the joint distribution of all y i is assumed to be multivariate Gaussian distributed as:
[Equation]
Furthermore, f ∗ for test instances x ∗ are jointly Gaussian with y as:
[Equation]
The mean function m is often set to 0 and its covariance function k depends on parameters θ .
For notational convenience, we use:
[Equation]
to deﬁne the kernel matrix.
We can derive the posterior predictive distribution with mean and covariance as follows:
[Equation]
Often, the kernel function is manually engineered, one popular example is the squared exponential kernel.
However, in this work, we make use of the idea of deep kernel learning (Wilson et al., 2016).
The idea is to model the kernel as a neural network ϕ and learn the best kernel transformation as:
[Equation]
As we will see later, this allows us to use convolutional operations as part of our kernel.
D YNAMIC M ULTI -F IDELITY H YPERPARAMETER O PTIMIZATION
In this section, we will describe D Y HPO, our proposed method for hyperparameter optimization in the gray-box setting.
At ﬁrst, we will describe the surrogate model which is a Gaussian Process with a deep convolutional kernel.
Then, we describe a variation of the popular expected improvement acquisition function (Jones et al., 1998), modiﬁed to consider multiple ﬁdelities, and conclude with the ﬁnal algorithm.
5. M ULTI -F IDELITY S URROGATE WITH D EEP C ONVOLUTIONAL K ERNEL
We propose to use a Gaussian Process surrogate model that infers the value of f i,j based on the hyperparameter conﬁguration x
i , the budget j as well as the past learning curve Y
i,j − 1 .
For this purpose, we use a deep kernel which is deﬁned as
[Equation]
We use a squared exponential kernel for k and the neural network ϕ is composed of linear and convolutional layers as shown in Figure 2.
We normalize the budget j to a range between 0 andby dividing it by the maximum budget B .
Afterward, it is concatenated with the hyperparameter conﬁguration x
i and fed to a linear layer.
[Equation]
[Figure]
The learning curve Y i,j − 1 is transformed by a one-dimensional convolution followed by a global max pooling layer.
Finally, both representations are fed to another linear layer.
Its output will be the input to the kernel function k .
Both, the kernel k and the neural network ϕ consist of trainable parameters θ and w , respectively.
We ﬁnd their optimal values by computing the maximum likelihood estimates as:
[Equation]
In order to solve this optimization problem, we use gradient descent and Adam (Kingma & Ba, 2015) with a learning rate of 0 . 1 .
Given the maximum likelihood estimates, we can approximate the predictive posterior as formalized in Equation 8, and ultimately compute the mean and covariance of this Gaussian using Equation 4.
[Equation]
5. M ULTI -F IDELITY E XPECTED I MPROVEMENT
Expected improvement (Jones et al., 1998) is a commonly used acquisition function for the blackbox setting and is deﬁned as:
[Equation]
where y max is the largest observed value of f .
We propose a multi-ﬁdelity version of it as:
[Equation]
where:
[Equation]
Simply put, y max j is the largest observed value of f for a budget of j if it exists already, otherwise, it is the largest observed value for any budget.
If there is only one possible budget, the multi-ﬁdelity expected improvement is identical to expected improvement.
5. T HE D Y HPO A LGORITHM
Algorithm 1: D Y HPO Algorithm 1: while not converged do 2: x i ← arg max x ∈X EI MF ( x , j ) (Section 5.2) 3:
Observe y i,j .
D ← D ∪ { (( x i , Y i,j − 1 , j ) , y i,j ) } 5:
Update the surrogate model on D .
(Section 5.1) 6: return x i with largest y i,j .
The D Y HPO algorithm looks very similar to many black-box Bayesian optimization algorithms as shown in
Algorithm 1.
The big difference is that at each step we dynamically decide which candidate conﬁguration to train for a small additional budget .
Possible candidates are previously unconsidered conﬁgurations as well as conﬁgurations that did not reach the maximum budget.
In Line 2, the most promising candidate is chosen using the acquisition function introduced in Section 5.2 and the surrogate model’s predictions.
It is important to highlight that we do not maximize the acquisition function along the budget dimensionality.
Instead, we set j such that it is by exactly one higher than the budget used to evaluate x i before.
If x i has not been evaluated for any budget yet, j is set to 1 .
This ensures that we explore conﬁgurations by slowly increasing the budget.
After having selected the candidate and the corresponding budget, the function f is evaluated and we observe y i,j (Line 3).
This additional data point is added to D in Line 4.
Then in Line 5, the surrogate model is updated according to the training scheme described in Section 5.1.
E XPERIMENTS
We evaluate D Y HPO in three different settings on hyperparameter optimization for tabular, text, and image classiﬁcation against several competitor methods.
These include Hyperband (Li et al., 2017),
BOHB (Falkner et al., 2018), DEHB (Awad et al., 2021), and Dragonﬂy (Metz et al., 2020).
We use
Dragonﬂy’s multi-ﬁdelity optimizer (Kandasamy et al., 2017).
For a sanity check, we also compare against random search (Bergstra & Bengio, 2012).
We use the publicly available implementations whenever available and implemented Hyperband and random search ourselves.
We report the mean of ten repetitions and report two common metrics such as the regret and the average rank.
The regret refers to the absolute difference between the score of the solution found by an optimizer compared to the best possible score.
If we report the regret as an aggregate result over multiple datasets, we report the mean over all regrets.
The average rank is a metric we use to aggregate results over different datasets.
For each dataset, the best performing method obtains a rank of 1.
Ties are broken by using the average rank, e.g., if the methods have scores 0.9, 0.8, 0.8, 0.7, the ranks are 1, 2.5, 2.5, and 4.
For both metrics, smaller is better.
6. F EEDFORWARD N EURAL N ETWORKS
In our ﬁrst experiment, we evaluate the various methods on how well they optimize neural networks for tabular datasets.
For this purpose, we use the LCBench learning curve benchmark (Zimmer et al.,
[Figure] [Figure]
DyHPODEHBBOHB
[Figure]
RandomDragonflyHyperband DEHBBOHBDyHPO [Figure] RandomDragonflyHyperband
DEHBBOHBDyHPO This benchmark contains learning curves for 35 different datasets where 2,000 neural networks per dataset are trained for 50 epochs with Auto-PyTorch.
The objective is to optimize seven different hyperparameters of funnel-shaped neural networks, i.e., batch size, learning rate, momentum, weight decay, dropout, number of layers, and maximum number of units per layer.
We show the aggregated results in Figure 3.
Here we aggregate the normalized wallclock time by dividing the actual wallclock time of baselines by the total wallclock time of our method D Y HPO including the overhead incurred by ﬁtting the deep GP.
In that manner, we can aggregate wallclock times across datasets.
However, all the results for each individual dataset are reported in the Appendix (Figure 12 and 13).
Furthermore, we use a critical difference diagram to report the pairwise statistical difference between all methods (Figure 4).
Horizontal lines indicate groups of methods that are not signiﬁcantly different from each other.
As suggested in the work of Demsar (2006), we use the Friedman test to reject the null hypothesis followed by a pairwise posthoc analysis based on the Wilcoxon signed-rank test ( α = 0 . 05 ).
While all gray-box methods have a very similar performance in the very beginning, D Y HPO quickly outperforms its competitors with respect to both evaluation metrics with statistical signiﬁcance.
On this task, both BOHB and DEHB are signiﬁcantly better than Hyperband but there is no clear winner among these two methods.
6. R ECURRENT N EURAL N ETWORKS
We continue with evaluating all methods on NLP tasks using search spaces provided in
TaskSet (Metz et al., 2020).
The objective is to optimize eight hyperparameters for a set of different recurrent neural networks (RNN) that differ in embedding size, RNN cell, and other architectural features.
The set of hyperparameters consists of optimizer-speciﬁc hyperparameters, such as the learning rate, the exponential decay rate of the ﬁrst and second momentum of Adam, β 1 and β 2 , and
Adam’s constant for numerical stability ε .
Furthermore, there are two hyperparameters controlling linear and exponential learning rate decays, as well as L1 and L2 regularization terms.
As before, we provide the aggregated results in the main paper (Figure 5) and provide detailed results in the
Appendix (Figure 11).
We show the critical difference diagram in Figure 6.
As before, all gray-box methods have a very similar performance in the very beginning, but, D Y HPO quickly outperforms its competitors with respect to both evaluation metrics.
The difference is signiﬁcant in the ﬁrst interval measured, in fact, D Y HPO is providing the best results across all different tasks for the majority [Figure] [Figure] of the elapsed time.
Given enough time
[Figure]
BOHB is able to catch up but D Y HPO’s improvement is still statistically signiﬁcant.
6. C ONVOLUTIONAL N EURAL N ETWORKS
[Figure]
Neural Architecture Search (Zoph & Le, 2017) (NAS) raised a lot of interest in the deep learning community in the last few years.
Since it can be reformulated to a hyperparameter optimization problem, we can apply our method as well as many other standard hyperparameter optimization methods to it.
We refrain from comparing against specialized NAS methods, since, this is out-of-scope for our work but refer the interested reader to the comparison of
Dong & Yang (2020).
To evaluate our methods in this scenario, we use NAS-Bench-201 (Dong & Yang, 2020).
This benchmark has precomputed about 15,600 architectures trained for 200 epochs for the image classiﬁcation datasets
CIFAR-10, CIFAR-100, and ImageNet.
The objective is to select for each of the six operations within the cell of the macro architecture one of ﬁve different operations.
All other hyperparameters such as learning rate and batch size are kept ﬁx.
We report the results in Figure 8, the remaining results can be found in the Appendix (Figure 10).
Initially, D Y HPO provides better results, but, given enough time, most methods perform no longer signiﬁcantly different.
We see also no difference between Hyperband and BOHB or DEHB.
A reason for this observation will be discussed in the next section. [Figure] t
ImageNet16DyHPO
DyHPO w/o LC
However, not worsens it
6. A BLATION S TUDY
One of the main differences between D Y HPO and similar methods such as the work by Kandasamy et al.
(2017), is that the learning curve is an input to the kernel function.
For this reason, we investigate the impact of this design choice.
We consider a variation of D Y HPO, D Y HPO w/o LC, which is identical to its counterpart, and the only difference is that the learning curve is not part of the input.
We report the results in Figure 7 and refer to the Appendix for additional results (Figure 9).
One of the striking observations is that the learning curve has only a small impact for LCBench in contrast to NAS-Bench-201 where it is signiﬁcant.
Our hypothesis is that for LCBench the hyperparameter representation is a valuable feature such that the additional use of the learning curve (which in fact is already implicitly considered by the Gaussian Process) is not required.
For NAS-Bench 201, however, the architecture representation is poor and does not allow for good predictions on its own.
This hypothesis aligns well with our previous experiments, where BOHB did better on
LCBench compared to Hyperband whereas we have not seen any difference on NAS-Bench 201.
As a reminder, BOHB is a variation of Hyperband that considers the hyperparameter representation to sample candidates.
We conclude, that the additional use of an explicit learning curve representation might not lead to a strong improvement in every scenario, however, it also does not seem to deteriorate it.
Furthermore, there are cases where the explicit consideration leads to signiﬁcantly better results.
L IMITATIONS OF O UR P APER
Although D Y HPO shows a convincing and statistically signiﬁcant reduction of the HPO time on diverse Deep Learning (DL) experiments, we cautiously characterized our method only as a ”step towards” scaling HPO for DL.
The reason for our restrain is the lack of tabular benchmarks for HPO on very large deep learning models, such as Transformers-based architectures (Devlin et al., 2019).
We hope our promising results will motivate the DL community, that own compute power, to invest in creating pre-computed tabular HPO benchmarks for very deep models, where HPO researchers can demonstrate the empirical performance of their proposed methods.
C ONCLUSIONS
In this work, we present D Y HPO, a new Bayesian optimization (BO) algorithm for the gray-box setting.
We introduced a new surrogate model for BO that uses a learnable deep kernel and takes the learning curve as an explicit input.
Furthermore, we motivated a variation of expected improvement for the multi-ﬁdelity setting.
Finally, we compared our approach on diverse benchmarks on a total of 50 different tasks against the current state-of-the-art methods on gray-box hyperparameter optimization (HPO).
Our method shows signiﬁcant gains and has the potential to become the de facto standard for HPO in Deep Learning.
A CKNOWLEDGMENTS
Double-blind.
E THICS S TATEMENT
In our work, we use only publicly available data with no privacy concerns.
Furthermore, our algorithm reduces the overall time for ﬁtting deep networks, therefore, saving computational resources and yielding a positive impact on the environment.
Moreover, our method can help smaller research organizations with limited access to resources to be competitive in the deep learning domain, which reduces the investment costs on hardware.
Although our method signiﬁcantly reduces the time taken for optimizing a machine learning algorithm that achieves peak performance, we warn against running our method for an extended time only to achieve marginal gains in performance, unless it is mission-critical.
Last but not least, in order to save energy, we invite the community to create sparse benchmarks with surrogates, instead of dense tabular ones.
R EPRODUCIBILITY S TATEMENT
We attempt to facilitate reproduction of our results with the following measures:
• We use only publicly available datasets and provide a detailed description of preprocessing (Section A.2) and the datasets themselves (Section A.1).
• All our baselines are publicly available or trivial to implement.
We provide all required details in Section A.5.
• We clearly describe our method in Section 5 and provide additional details in Section A.4.
Finally, we plan to make the source code of our method publicly available.
A E XPERIMENTAL S ETUP A. B ENCHMARKS
LCBench.
LCBench 1 is a feedforward neural network benchmark on tabular data which consists of 2000 conﬁguration settings for each of the 35 datasets.
The conﬁgurations were evaluated during
HPO runs with AutoPyTorch.
LCBench features a search space of 7 numerical hyperparameters, where every hyperparameter conﬁguration is trained for 50 epochs.
TaskSet.
TaskSet 2 is a benchmark that features over 1162 diverse tasks from different domains and includes 5 search spaces.
In this work, we focus on NLP tasks and we use the Adam8p search space with 8 continuous hyperparameters.
We refer to Figure 11 for the exact task names considered in our experiments.
The learning curves provided in TaskSet report scores after every 200 iterations.
We refer to those as ”steps” in Figure 5.
NAS-Bench-201.
NAS-Bench-201 3 is a benchmark consisting of 15625 hyperparameter conﬁgurations representing different architectures on the CIFAR-10, CIFAR-100 and ImageNet datasets.
NAS-Bench-201 features a search space of 6 categorical hyperparameters and each architecture is trained for 200 epochs.
A. P REPROCESSING
In the following, we describe the preprocessing applied to the hyperparameter representation.
For
LCBench, we apply a log-transform to batch size, learning rate, and weight decay.
For TaskSet, we apply it on the learning rate, L1 and L2 regularization terms, linear and exponential decay of the learning rate.
All continuous hyperparameters are scaled to the range between 0 and 1 using sklearn’s MinMaxScaler.
If not mentioned otherwise, we use one-hot encoding for the categorical hyperparameters.
As detailed in the following, some baselines have a speciﬁc way of dealing with them.
In that case, we use the method recommended by the authors.
A. F RAMEWORK
The framework contains the evaluated hyperparameters and their corresponding validation curves.
The list of candidate hyperparameters is passed to the baseline-speciﬁc interface, which in turn, optimizes and queries the framework for the hyperparameter conﬁguration that maximizes utility.
Our framework in turn responds with the validation curve and the cost of the evaluation.
In case a hyperparameter conﬁguration has been evaluated previously up to a budget b and a baseline requires the response for budget b + 1 , the cost is calculated accordingly only for the extra budget requested.
A. I MPLEMENTATION D ETAILS
We implement the Deep Kernel Gaussian Process using GPyTorch 1.5 (Gardner et al., 2018).
We use an RBF kernel and the dense layers of the transformation function ϕ
(Figure 2) have 128 and 256 units.
We used a convolutional layer with kernel size three and four ﬁlters.
All parameters of the Deep Kernel are estimated by maximizing the marginal likelihood.
We achieve this by using gradient ascent and Adam (Kingma & Ba, 2015) with a learning rate of 0 . 1 and batch size of 64.
We stop training as soon as the training likelihood is not improving for 10 epochs in a row or we completed 1,000 epochs.
For every new data point, we start training the GP with its old parameters to reduce the required effort for training.
A. B ASELINES
Random Search & Hyperband.
Random search and Hyperband sample hyperparameter conﬁgurations at random and therefore the preprocessing is irrelevant.
We have implemented both from scratch and use the recommended hyperparameters for Hyperband, i.e. η = 3 .
BOHB.
For our experiments with BOHB, we use version 0.7.4 of the ofﬁcially-released code 4 .
DEHB.
For our experiments with DEHB, we use the ofﬁcial public implementation 5 .
We developed an interface that communicates between our framework and DEHB.
In addition to the initial preprocessing common for all methods, we encode categorical hyperparameters with a numerical value in the interval [0, 1].
For a categorical hyperparameter x i , we take N equal-sized intervals, where N i represents the number of unique categorical values for hyperparameter x
i and we assign the value for a categorical value n ∈ N i to the middle of the interval [ i, i + 1] as suggested by the authors.
For conﬁguring the DEHB algorithm we used the default values from the library.
Dragonﬂy.
We use the publicly available code of Dragonﬂy 6 .
No special treatment of categorical hyperparameters is required since Dragonﬂy has its own way to deal with them.
We use version 0 . 1 . 6 with default settings.
B A DDITIONAL P LOTS
In this section, we provide additional plots for the performance comparison between all methods for the individual datasets in our benchmarks.
In Figure 12 and 13 we show the performance comparison for all the datasets from LCBench regarding regret over the number of epochs.
As can be seen,
D Y HPO manages to outperform the other competitors in the majority of the datasets, and in the datasets that it does not, it is always close to the top-performing method and the difference between methods is marginal.
[Figure]
Additionally, in Figure 10 we show the performance comparison over time of every method for the
CIFAR-10 and CIFAR-100 datasets in the NAS-Bench-201 benchmark.
As can be seen, D Y HPO converges faster and has a better performance compared to the other methods over the majority of the time, however, towards the end although it is the optimal method or close to the optimal method, the difference in regret is not signiﬁcant anymore.
Furthermore, Figure 11 shows the performance comparison for the datasets chosen from TaskSet over the number of steps.
Looking at the results, D Y HPO is outperforming all methods convincingly on the majority of datasets by converging faster and with signiﬁcant differences in the regret evaluation metric.
[Figure] [Figure]
Lastly
[Figure]
in Figure 9 we ablate the learning curve input in our kernel, to see the effect it has on performance for the CIFAR-10 and CIFAR-100 datasets.
The results indicate that the learning curve plays an important role in achieving better results by allowing faster convergence and a better anytime performance. [Figure] [Figure] [Figure]
[Figure]
C O N THE E FFECTIVENESS OF D Y HPO
D Y HPO effectively explores the search space and identiﬁes promising candidates.
This is visualized in Figure 14 in which we plot the precision of each method for different considered budgets.
The precision at an epoch i is deﬁned as the number of top 1% candidates trained for at least i epochs divided by the number of all candidates trained for at least i epochs.
The higher the precision, the less irrelevant candidates were considered and the less computational resources were wasted.
For small budgets, the precision is low since D Y HPO spends budget to consider some candidates but then promising candidates are successfully identiﬁed and the precision quickly increases. For LCBench and Taskset, all other methods dedicate signiﬁcantly more resources to irrelevant candidates which explains why D Y HPO ﬁnds good candidates faster. For NAS-Bench-201, DEHB can match the precision but only at a later stage. Simply put, the baselines select much more ”poor” conﬁgurations (i.e. outside the top 1% performers) compared to our method D Y HPO.
This argument is further supported by Figure 15 where we visualize the average regret of all the candidates trained for at least the speciﬁed number of epochs in the x-axis.
In contrast to the regret plots in Section 6, here we do not show the regret of the best conﬁguration, but the mean regret of all the selected conﬁgurations.
The analysis deduces a similar ﬁnding as in Figure 14 above.
Our method D Y HPO selects highly more qualitative hyperparameter conﬁgurations than all the baselines.
D P ROMOTION OF P OOR P ERFORMING C ANDIDATES
An interesting property of multi-ﬁdelity HPO is the phenomenon of poor rank correlations among the validation performance of candidates at different budgets.
In other words, a conﬁguration that achieves a poor performance at a small budget, might perform strongly at a larger budget.
For instance, a well-regularized neural network will converge slower than an un-regularized network in the early optimization epochs, but eventually perform better when converged.
We analyze this phenomenon and report the respective results in Figure 16.
In this experiment we measure the percentage of ”good” conﬁgurations at a particular budget, that were ”bad” performers in at least one of the smaller budgets.
We deﬁne a ”good” performance at a budget B, when a conﬁguration achieves a validation accuracy ranked among the top 1/3 compared to the validation accuracies of all the other conﬁgurations that were run until that budget B. Similarly a ”bad” performance at a budget B represents a conﬁguration whose validation accuracy belongs to the bottom 2/3 of all conﬁgurations ran at that budget B.
[Figure]
i
[Figure]
o f P oo r
P e
r
f
o r m e r P r o m o t
i
[Figure]
In Figure 16 we analyze the percentage of ”good” conﬁgurations at each budget denoted by the xaxis, that were ”bad” performers in at least one of the lower budgets.
Such a metric is a proxy for the degree of the promotion of ”bad” conﬁgurations towards higher budget.
We present the analysis for all the competing methods of our experimental protocol from Section 6.
We have additionally included the ground-truth line annotated as ”Baseline”, which represents the fraction of past poor performers among all the feasible conﬁgurations in the search space.
In contrast, the respective methods compute the fraction of promotions only among the conﬁgurations that those methods have considered (i.e. selected within their HPO trials) until the budget indicated by the x-axis.
We see that in all the search spaces LCBench, TaskSet and NASBench-201 there is a high degree of ”good” conﬁgurations that were ”bad” at a previous budget, with fractions of the ground-truth ”Baseline” varying from ca.
40% in LCBench, up to ca.
70% in the NASBench-201 datasets.
On the other hand, the analysis demonstrates that our method D Y HPO has promoted more ”good” conﬁgurations that were ”bad” in a lower budget, compared to all the rival methods.
In particular, ca.
80% of selected conﬁgurations at the datasets from the LCBench benchmark were ”bad” performers at a lower budget, while in the case of NASBench-201 this fraction approaches the level of 95%.