Supervising the Multi-Fidelity Race of Hyperparameter Configurations
Abstract
Multi-fidelity (gray-box) hyperparameter optimization techniques (HPO) have recently emerged as a promising direction for tuning Deep Learning methods.
However, existing methods suffer from a sub-optimal allocation of the HPO budget to the hyperparameter configurations.
In this work, we introduce DyHPO, a
Bayesian Optimization method that learns to decide which hyperparameter configuration to train further in a dynamic race among all feasible configurations.
We propose a new deep kernel for Gaussian Processes that embeds the learning curve dynamics, and an acquisition function that incorporates multi-budget information.
We demonstrate the significant superiority of DyHPO against state-of-the-art hyperparameter optimization methods through large-scale experiments comprisingdatasets (Tabular, Image, NLP) and diverse architectures (MLP, CNN/NAS, RNN).
Introduction
Hyperparameter Optimization (HPO) is arguably an acute open challenge for Deep Learning (DL), especially considering the crucial impact HPO has on achieving state-of-the-art empirical results.
Unfortunately, HPO for DL is a relatively under-explored field and most DL researchers still optimize their hyperparameters via obscure trial-and-error practices.
On the other hand, traditional Bayesian
Optimization HPO methods [Snoek et al., 2012, Bergstra et al., 2011] are not directly applicable to deep networks, due to the infeasibility of evaluating a large number of hyperparameter configurations.
In order to scale HPO for DL, three main directions of research have been recently explored.
(i)
Online HPO methods search for hyperparameters during the optimization process via meta-level controllers [Chen et al., 2017, Parker-Holder et al., 2020], however, this online adaptation can not accommodate all hyperparameters (e.g. related to architectural changes).
(ii) Gradient-based HPO techniques, on the other hand, compute the derivative of the validation loss w.r.t.
hyperparameters by reversing the training update steps [Maclaurin et al., 2015, Franceschi et al., 2017, Lorraine et al., 2020], however, the reversion is not directly applicable to all cases (e.g. dropout rate).
The last direction, (iii) Gray-box HPO techniques discard sub-optimal configurations after evaluating them on lower budgets [Li et al., 2017, Falkner et al., 2018].
In contrast to the online and gradient-based alternatives, gray-box approaches can be deployed in an off-the-shelf manner to all types of hyperparameters and architectures.
The gray-box concept is based on the intuition that a poorly-performing hyperparameter configuration can be identified and terminated by inspecting the validation loss of the first few epochs, instead of waiting for the full convergence.
The most prominent gray-box algorithm is Hyperband [Li et al., 2017], which is based on successive halving.
It runs random configurations at different budgets (e.g. number of epochs) and successively halves these configurations by keeping only the top performers.
Follow-up works, such as BOHB [Falkner et al., 2018] or DEHB [Awad et al., 2021], replace the random sampling of
Hyperband with a sampling based on Bayesian optimization or differentiable evolution.
Despite their great practical potential, gray-box methods suffer from a major issue.
The low-budget (few epochs) performances are not always a good indicator for the full-budget (full convergence) performances.
For example, a properly regularized network converges slower in the first few epochs, however, typically performs better than a non-regularized variant after the full convergence.
In other words, there can be a poor rank correlation of the configurations’ performances at different budgets.
We introduce D Y HPO, a Bayesian Optimization (BO) approach based on Gaussian Processes (GP), that proposes a novel treatment to the multi-budget (a.k.a.
multi-fidelity) setup.
In this perspective, we propose a deep kernel GP that captures the learning dynamics.
As a result, we train a kernel capable of capturing the similarity of a pair of hyperparameter configurations, even if the pair’s configurations are evaluated at different budgets.
Furthermore, we extend Expected
Improvement [Jones et al., 1998] to the multibudget case, by introducing a new mechanism for the incumbent configuration of a budget.
We illustrate the differences between our racing strategy and successive halving with the experiment of Figure 1, where, we showcase the HPO progress of two different methods on the "Helena" dataset from the LCBench benchmark [Zimmer et al., 2021].
Hyperband [Li et al., 2017] is a gray-box approach that statically pre-allocates the budget for a set of candidates (Hyperband bracket) according to a predefined policy.
However, D Y HPO dynamically adapts the allocation of budgets for configurations after every
HPO step (a.k.a.
a dynamic race).
As a result, D Y HPO invests only a small budget on configurations that show little promise as indicated by the intermediate scores.
The joint effect of modeling a GP kernel across budgets together with a dedicated acquisition function leads to D Y HPO achieving a statistically significant empirical gain against state-of-the-art gray-box baselines [Falkner et al., 2018, Awad et al., 2021], including prior work on multi-budget
GPs [Kandasamy et al., 2017, 2020] or neural networks [Li et al., 2020b].
We demonstrate the performance of D Y HPO in three diverse deep learning architectures (MLP, CNN/NAS, RNN) anddatasets of three diverse modalities (tabular, image, natural language processing).
We believe our method is a step forward toward making HPO for DL practical and feasible.
Overall, our contributions can be summarized as follows
: • We introduce a novel Bayesian surrogate for gray-box HPO optimization.
Our novel surrogate model predicts the validation score of a machine learning model based on both the hyperparameter configuration, the budget information, and the learning curve.
• We derive a simple yet robust way to combine this surrogate model with Bayesian optimization, reusing most of the existing components currently used in traditional Bayesian optimization methods.
Finally, we demonstrate the efficiency of our method for HPO and neural architecture search tasks compared to the current state-of-the-art methods in HPO, by outperforming seven strong HPO baselines with a statistically significant margin.
As an overarching goal, we believe our method is an important step toward scaling HPO for DL.
Related Work on Gray-box HPO
Multi-Fidelity Bayesian Optimization and Bandits.
Bayesian optimization is a black-box function optimization framework that has been successfully applied in optimizing hyperparameter and neural architectures alike [Snoek et al., 2012, Kandasamy et al., 2018, Bergstra et al., 2011].
To further improve Bayesian optimization, several works propose low-fidelity data approximations of hyperparameter configurations by training on a subset of the data [Swersky et al., 2013, Klein et al., 2017a], or by terminating training early [Swersky et al., 2014].
Additionally, several methods extend
Bayesian optimization to multi-fidelity data by engineering new kernels suited for this problem [Swersky et al., 2013, 2014, Poloczek et al., 2017].
Kandasamy et al.
[2016] extends GP-UCB [Srinivas et al., 2010] to the multi-fidelity setting by learning one Gaussian Process (GP) with a standard kernel for each fidelity.
Their later work improves upon this method by learning one GP for all fidelities that enables the use of continuous fidelities [Kandasamy et al., 2017].
The work by Takeno et al.
[2020] follows a similar idea but proposes to use an acquisition function based on information gain instead of UCB.
While most of the works rely on GPs to model the surrogate function, Li et al.
[2020b] use a
Bayesian neural network that models the complex relationship between fidelities with stacked neural networks, one for each fidelity.
Hyperband [Li et al., 2017] is a bandits-based multi-fidelity method for hyperparameter optimization that selects hyperparameter configurations at random and uses successive halving [Jamieson and Talwalkar, 2016] with different settings to early-stop less promising training runs.
Several improvements have been proposed to Hyperband with the aim to replace the random sampling of hyperparameter configurations with a more guided approach [Bertrand et al., 2017, Wang et al., 2018, Wistuba, 2017].
BOHB [Falkner et al., 2018] uses TPE [Bergstra et al., 2011] and builds a surrogate model for every fidelity adhering to a fixed-fidelity selection scheme.
DEHB [Awad et al., 2021] samples candidates using differential evolution which handles large and discrete search spaces better than BOHB.
Mendes et al.
[2021] propose a variant of Hyperband which allows to skip stages.
Learning Curve Prediction A variety of methods attempt to extrapolate a partially observed learning curve in order to estimate the probability that a configuration will improve over the current best solution.
Domhan et al.
[2015] propose to ensemble a set of parametric functions to extrapolate a partial learning curve.
While this method is able to extrapolate with a single example, it requires a relatively long learning curve to do so.
The work by Klein et al. [2017b] build upon the idea of using a set of parametric functions. The main difference is that they use a heteroscedastic Bayesian model to learn the ensemble weights. Baker et al.
[2018] propose to use support vector machines (SVM) as an auto-regressive model.
The SVM predicts the next value of a learning curve, the original learning curve is augmented by this value and we keep predicting further values.
The work by
Gargiani et al.
[2019] use a similar idea but makes prediction based on the last K observations only and uses probabilistic models.
Wistuba and Pedapati [2020] propose to learn a prediction model across learning curves from different tasks to avoid the costly learning curve collection.
In contrast to D Y HPO, none of these methods selects configuration but is limited to deciding when to stop a running configuration.
Multi-Fidelity Acquisition Functions Klein et al.
[2017a] propose an acquisition function which allows for selecting hyperparameter configurations and the dataset subset size.
The idea is to reduce training time by considering only a smaller part of the training data.
In contrast to EI MF , this acquisition function is designed to select arbitrary subset sizes whereas EI MF is intended to slowly increase the invested budget over time.
Mendes et al.
[2020] extend the work of Klein et al.
[2017a] to take business constraints into account.
Deep Kernel Learning with Bayesian Optimization. We are among the first to use deep kernel learning with Bayesian optimization and to the best of our knowledge the first to use it for multifidelity Bayesian optimization.
Rai et al.
[2016] consider the use of a deep kernel instead of a manually designed kernel in the context of standard Bayesian optimization, but, limit their experimentation to synthetic data and do not consider its use for hyperparameter optimization.
Perrone et al.
Wistuba and Grabocka [2021] use a pre-trained deep kernel to warm start Bayesian optimization with meta-data from previous optimizations.
The aforementioned approaches are multi-task or transfer learning methods that require the availability of meta-data from related tasks.
In contrast to prior work, we propose a method that introduces deep learning to multi-fidelity HPO with Bayesian Optimization, and captures the learning dynamics across fidelities/budgets, combined with an acquisition function that is tailored for the gray-box setup.
Dynamic Multi-Fidelity HPO 3.1 Preliminaries
Gray-Box Optimization.
The gray-box HPO setting allows querying configurations with a smaller budget compared to the total maximal budget B .
Thus, we can query from the response function f : X × N → R where f i,j = f ( x i , j ) is the response after spending a budget of j on configuration x i .
As before, these observations are noisy and we observe y i,j = f
( x i , j ) +
ε j where ε j ∼ N (0 , σ 2 j,n ) .
Please note, we assume that the budget required to query f
i,j + b after having queried f i,j is only b .
Furthermore, we use the learning curve Y i,j − 1 =
( y i, 1 , . . .
, y i,j − 1 ) when predicting f i,j .
Gaussian Processes (GP).
Given a training data set D = { ( x i , y i ) } ni =1 , the Gaussian Process assumption is that y i is a random variable and the joint distribution of all y i is assumed to be multivariate Gaussian distributed as y ∼ N ( m ( X ) , k
( X , X )) .
Furthermore, f ∗ for test instances x ∗ are jointly Gaussian with y as:
[Equation]
The mean function m is often set to 0 and its covariance function k depends on parameters θ .
For notational convenience, we use K n = k
( X , X | θ ) + σ 2 n I , K ∗
=
k
( X , X ∗ | θ ) and K ∗∗ = k
( X ∗ , X ∗ | θ ) to define the kernel matrices.
We can derive the posterior predictive distribution with mean and covariance as follows:
[Equation]
Often, the kernel function is manually engineered, one popular example is the squared exponential kernel.
However, in this work, we make use of the idea of deep kernel learning [Wilson et al., 2016].
The idea is to model the kernel as a neural network φ and learn the best kernel transformation
K ( θ, w ) :
=
k ( φ ( x , w ) , φ ( x ′ ; w ) | θ ) , which allows us to use convolutional operations in our kernel.
3.2 Deep Multi-Fidelity Surrogate
We propose to use a Gaussian Process surrogate model that infers the value of f i,j based on the hyperparameter configuration x
i , the budget j as well as the past learning curve Y
i,j − 1 .
For this purpose, we use a deep kernel as:
[Equation] [Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
(cid:80) (cid:80)
[Figure]
We use a squared exponential kernel for k and the neural network φ is composed of linear and convolutional layers as shown in Figure 2.
We normalize the budget j to a range betweenand 1 by dividing it by the maximum budget
B .
Afterward, it is concatenated with the hyperparameter configuration x
i and fed to a linear layer.
The learning curve Y i,j − 1 is transformed by a one-dimensional convolution followed by a global max pooling layer.
Finally, both representations are fed to another linear layer.
Its output will be the input to the kernel function k .
Both, the kernel k and the neural network φ consist of trainable parameters θ and w , respectively.
We find their optimal values by computing the maximum likelihood estimates as:
[Equation]
In order to solve this optimization problem, we use gradient descent and Adam [Kingma and Ba, 2015] with a learning rate of 0 . 1 .
Given the maximum likelihood estimates, we can approximate the predictive posterior through p (cid:16) f i,j | x i , Y i,j − 1 , j, D , ˆ θ , ˆ w (cid:17) , and ultimately compute the mean and covariance of this Gaussian using Equation 2.
3.3 Multi-Fidelity Expected Improvement
Expected improvement [Jones et al., 1998] is a commonly used acquisition function and is defined as:
[Equation]
where y max is the largest observed value of f .
We propose a multi-fidelity version of it as:
[Equation]
where:
[Equation]
Simply put, y max j is the largest observed value of f for a budget of j if it exists already, otherwise, it is the largest observed value for any budget.
If there is only one possible budget, the multi-fidelity expected improvement is identical to expected improvement.
3.4 The D Y HPO Algorithm
The D Y HPO algorithm looks very similar to many black-box Bayesian optimization algorithms as shown in Algorithm 1.
The big difference is that at each step we dynamically decide which candidate configuration to train for a small additional budget .
1: b ( x ) =
∀ x ∈ X 2: while not converged do 3: x i ← arg max x ∈X EI MF ( x , b ( x ) + 1) (Sec. 3.3) 4:
Observe y i,b ( x i )+1 .
5: b ( x i )
+
Possible candidates are previously unconsidered configurations as well as configurations that did not reach the maximum budget.
In Line 2, the most promising candidate is chosen using the acquisition function introduced in
Section 3.3 and the surrogate model’s predictions.
It is important to highlight that we do not maximize the acquisition function along the budget dimensionality.
Instead, we set the budget b such that it is by exactly one higher than the budget used to evaluate x i before.
This ensures that we explore configurations by slowly increasing the budget.
After the candidate and the corresponding budget are selected, the function f is evaluated and we observe y i,j (Line 3).
This additional data point is added to D in Line 4.
Then in Line 5, the surrogate model is updated according to the training scheme described in Section 3.2.
Experimental Protocol 4.1 Experimental Setup
We evaluate D Y HPO in three different settings on hyperparameter optimization for tabular, text, and image classification against several competitor methods, the details of which are provided in the following subsections.
We ran all of our experiments on an Amazon EC2 M5 Instance (m5.xlarge).
In our experiments, we report the mean of ten repetitions and we report two common metrics, the regret and the average rank.
The regret refers to the absolute difference between the score of the solution found by an optimizer compared to the best possible score.
If we report the regret as an aggregate result over multiple datasets, we report the mean over all regrets.
The average rank is the metric we use to aggregate rank results over different datasets.
We provide further implementation and training details in Appendix A.4.
Our implementation of D Y HPO is publicly available.
4.2 Benchmarks
In our experiments we use the following benchmarks.
We provide more details in Appendix A.1.
LCBench
: A learning curve benchmark [ Zimmer et al., 2021] that evaluates neural network architec tures for tabular datasets
LCBench contains learning curves for different datasets, where 2, neural networks per dataset are trained for epochs with Auto-
PyTorch TaskSet: A benchmark that features diverse tasks Metz et al.
[2020] from different domains and includes 5 search spaces with different degrees of freedom, where, every search space includeshyperparameter configurations. In this work, we focus on a subset of
NLP
tasks ( tasks and we use the Adam8p search
space with 8 continuous hyperparameters. NAS-Bench-201: A benchmark consisting of 15625 hyperparameter configurations representing different architectures on the CIFAR-10, CIFAR-100 and ImageNet datasets Dong and Yang [2020].
NAS-Bench-201 features a search space of 6 categorical hyperparameters and each architecture is trained for 200 epochs.
Baselines
Random Search: A random/stochastic black-box search method for HPO.
HyperBand:
A multi-arm bandit method that extends successive halving by multiple brackets with different combinations of the initial number of configurations, and their initial budget [Li et al., 2017].
BOHB:
An extension of Hyperband that replaces the random sampling of the initial configurations for each bracket with recommended configurations from a model-based approach [Falkner et al., 2018].
BOHB builds a model for every fidelity that is considered.
DEHB: A method that builds upon Hyperband by exploiting differential evolution to sample the initial candidates of a Hyperband bracket [Awad et al., 2021].
ASHA:
An asynchronous version of successive halving (or an asynchronous version of Hyperband if multiple brackets are run).
ASHA Li et al.
[2020a] does not wait for all configurations to finish inside a successive halving bracket, but, instead promotes configurations to the next successive halving bracket in real-time.
MF-DNN: A multi-fidelity Bayesian optimization method that uses deep neural networks to capture the relationships between different fidelities
Li et al.
[2020b].
Dragonfly: We compare against BOCA [Kandasamy et al., 2017] by using the Dragonfly library Kandasamy et al.
This method suggests the next hyperparameter configuration as well as the budget it should be evaluated for.
4.4 Research Hypotheses and Associated Experiments
Hypothesis 1: D Y HPO achieves state-of-the-art results in multi-fidelity HPO.
Experiment 1: We compare D Y HPO against the baselines of Section 4.3 on the benchmarks of Section 4.2 with the experimental setup of Section 4.1.
For TaskSet we follow the authors’ recommendation and report the number of steps (every 200 iterations).
Hypothesis 2: D Y HPO’s runtime overhead has a negligible impact on the quality of results.
Experiment 2: We compare D Y HPO against the baselines of Section 4.3 over the wallclock time.
The wallclock time includes both (i) the optimizer’s runtime overhead for recommending the next hyperparameter configuration, plus (ii) the time needed to evaluate the recommended configuration.
In this experiment, we consider all datasets where the average training time per epoch is at least 10 seconds, because, for tasks where the training time is short, there is no practical justification for [Figure] complex solutions and their overhead.
In these cases, we recommend using a random search.
We don’t report results for TaskSet because the benchmark lacks training times.
Hypothesis 3: D Y HPO uses the computational budget more efficiently than baselines.
Experiment 3: To further verify that D Y HPO is efficient compared to the baselines, we investigate whether competing methods spend their budgets on qualitative candidates.
Concretely we: i) calculate the precision of the top (w.r.t. ground truth) performing configurations that were selected by each method across different budgets, ii) compute the average regret of the selected configurations across budget, and iii) we compare the fraction of top-performing configurations at a given budget that were not top performers at lower budgets, i.e. measure the ability to handle the poor correlation of performances across budgets.
Results
Experiment 1: D Y HPO achieves state-of-the-art results.
In our first experiment, we evaluate the various methods on the benchmarks listed in Section 4.2.
We show the aggregated results in
Figure 3, the results show that D Y HPO manages to outperform competitor methods over the set of considered benchmarks by achieving a better mean regret across datasets.
Not only does D Y HPO achieve a better final performance, it also achieves strong anytime results by converging faster than the competitor methods.
For the extended results, related to the performance of all methods on a dataset level, we refer the reader to Appendix B.
[Figure]
In Figure 4, we provide further evidence that D Y HPO’s improvement over the baselines is statistically significant.
The critical difference diagram presents the ranks of all methods and provides information on the pairwise statistical difference between all methods for two fractions of the number of HPO steps (50% and 100%).
We included the LCBench and TaskSet benchmarks in our significance plots.
NAS-Bench-201 was omitted because it has only 3 datasets and the statistical test cannot be applied.
Horizontal lines indicate groupings of methods that are not significantly different.
As suggested by the best published practices Demsar [2006], we use the Friedman test to reject the null hypothesis followed by a pairwise post-hoc analysis based on the Wilcoxon signed-rank test ( α = 0 . 05 ).
For LCBench, D Y HPO already outperforms the baselines significantly after 50% of the search budget, with a statistically significant margin.
As the optimization procedure continues, D Y HPO manages to extend its gain in performance and is the only method that has a statistically significant improvement against all the other competitor methods.
Similarly, for TaskSet, D Y HPO manages to outperform all methods with a statistically significant margin only halfway through the optimization procedure and achieves the best rank over all methods.
However, as the optimization procedure continues, BOHB manages to decrease the performance gap with D Y HPO, although, it still achieves a worse rank across all datasets.
Considering the empirical results, we conclude that Hypothesis 1 is validated and that D Y HPO achieves state-of-the-art results on multi-fidelity HPO .
[Figure]
Experiment 2:
On the impact of D Y HPO’s overhead on the results.
We present the results of our second experiment in Figure 5 (left), where, as it can be seen, D Y HPO still outperforms the other methods when its overhead is considered.
For LCBench, D Y HPO manages to get an advantage fairly quickly and it only increases the gap in performance with the other methods as the optimization process progresses.
Similarly, in the case of ImageNet from NAS-Bench-201, D Y HPO manages to gain an advantage earlier than other methods during the optimization procedure.
Although in the end
D Y HPO still performs better than all the other methods, we believe most of the methods converge to a good solution and the differences in the final performance are negligible.
For the extended results, related to the performance of all methods on a dataset level over time, we refer the reader to the plots in Appendix B. Additionally, in Figure 5 (right), we provide the critical difference diagrams for LCBench that present the ranks and the statistical difference of all methods halfway through the optimization procedure, and in the end.
As it can be seen, D Y HPO has a better rank with a significant margin with only half of the budget used and it retains the advantage until the end.
Experiment 3:
On the efficiency of D Y HPO.
In Figure 6 (left), we plot the precision of every method for different budgets during the optimization procedure, which demonstrates that D Y HPO effectively explores the search space and identifies promising candidates.
The precision at an epoch i is defined as the number of top 1% candidates that are trained, divided by the number of all candidates trained, both trained for at least i epochs.
The higher the precision, the more relevant candidates were considered and the less computational resources were wasted.
For small budgets, the precision is low since D Y HPO spends budget to consider various candidates, but then, promising candidates are successfully identified and the precision quickly increases.
This argument is further supported in Figure 6 (middle), where we visualize the average regret of all the candidates trained for at least the specified number of epochs on the x-axis.
In contrast to the regret plots, here we do not show the regret of the best configuration, but the mean regret of all the selected configurations.
The analysis deduces a similar finding, our method D Y HPO selects more qualitative hyperparameter configurations than all the baselines.
An interesting property of multi-fidelity HPO is the phenomenon of poor rank correlations among the validation performance of candidates at different budgets.
In other words, a configuration that achieves a poor performance at a small budget can perform better at a larger budget.
To analyze this phenomenon, we measure the percentage of "good" configurations at a particular budget, that were "bad" performers in at least one of the smaller budgets.
We define a "good" performance at a budget
B when a configuration achieves a validation accuracy ranked among the top 1/3 of the validation accuracies belonging to all the other configurations that were run until that budget B.
In Figure 6 (right), we analyze the percentage of "good" configurations at each budget denoted by the x-axis, that were "bad" performers in at least one of the lower budgets.
Such a metric is a proxy for the degree of the promotion of "bad" configurations towards higher budgets.
We present the analysis for all the competing methods of our experimental protocol from Section 4.
We have additionally included the ground-truth line annotated as "Baseline", which represents the fraction of past poor [Figure] r e c i s i [Figure]
o f T op
[Figure]
i da t e [Figure] performers among all the feasible configurations in the search space. In contrast, the respective methods compute the fraction of promotions only among the configurations that those methods have considered (i.e. selected within their HPO trials) until the budget indicated by the x-axis.
We see that there is a high degree of "good" configurations that were "bad" at a previous budget, with fractions of the ground-truth "Baseline" going up to 40% for the LCBench benchmark and up to 80% for the
NAS-Bench-201 benchmark.
On the other hand, the analysis demonstrates that our method D Y HPO has promoted more "good" configurations that were "bad" in a lower budget, compared to all the rival methods.
In particular, more than 80% of selected configurations from the datasets belonging to either benchmark were "bad" performers at a lower budget.
The empirical evidence validates Hypothesis 3 and demonstrates that D Y HPO efficiently explores qualitative candidates.
We provide the results of our analysis for
D Y HPO’s efficiency on the additional benchmarks (Taskset) in Appendix B.
Ablating the impact of the learning curve
[Figure]
One of the main differences between D Y HPO and similar methods Kandasamy et al.
[2017], is that the learning curve is an input to the kernel function.
For this reason, we investigate the impact of this design choice.
We consider a variation of D Y HPO w/o CNN, which is simply
D Y HPO without the learning curve.
It is worth emphasizing that both variants (with and without the learning curve) are multi-fidelity surrogates and both receive the budget information through the inputted index j in Equation 3.
The only difference is that D Y HPO additionally incorporates the pattern of the learning curve.
We run the ablation on the NAS-Bench-201 benchmark and report the results for ImageNet, the largest dataset in our collection.
The ablation results are shown in Figure 7, while the remaining results on the other datasets are shown in Figure 8 of the appendix.
Based on the results from our learning curve ablation, we conclude that the use of an explicit learning curve representation leads to significantly better results.
Limitations of Our Method
Although D Y HPO shows a convincing and statistically significant reduction of the HPO time on diverse Deep Learning (DL) experiments, we cautiously characterized our method only as a "step towards" scaling HPO for DL.
The reason for our restrain is the lack of tabular benchmarks for HPO on very large deep learning models, such as Transformers-based architectures [Devlin et al., 2019].
Additionally, the pause and resume part of our training procedure can only be applied when tuning the hyperparameters of parametric models, otherwise, the training of a hyperparameter configuration would have to be restarted.
Lastly, for small datasets that can be trained fast, the overhead of model-based techniques would make an approach like random search more appealing.
Conclusions
In this work, we present D Y HPO, a new Bayesian optimization (BO) algorithm for the gray-box setting.
We introduced a new surrogate model for BO that uses a learnable deep kernel and takes the learning curve as an explicit input.
Furthermore, we motivated a variation of expected improvement for the multi-fidelity setting.
Finally, we compared our approach on diverse benchmarks on a total of 50 different tasks against the current state-of-the-art methods on gray-box hyperparameter optimization (HPO).
Our method shows significant gains and has the potential to become the de facto standard for HPO in Deep Learning.
Acknowledgments
Josif Grabocka and Arlind Kadra would like to acknowledge the grant awarded by Eva-MayrStihl Stiftung.
In addition, this research was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under grant number 417962828.
Furthermore, the authors acknowledge support by the state of Baden-Württemberg through bwHPC and the German Research
Foundation (DFG) through grant no INST 39/963-1 FUGG (bwForCluster NEMO).
Societal Implications
In our work, we use only publicly available data with no privacy concerns.
Furthermore, our algorithm reduces the overall time for fitting deep networks, therefore, saving computational resources and yielding a positive impact on the environment.
Moreover, our method can help smaller research organizations with limited access to resources to be competitive in the deep learning domain, which reduces the investment costs on hardware.
Although our method significantly reduces the time taken for optimizing a machine learning algorithm that achieves peak performance, we warn against running our method for an extended time only to achieve marginal gains in performance, unless it is mission-critical.
Last but not least, in order to save energy, we invite the community to create sparse benchmarks with surrogates, instead of dense tabular ones.
A Experimental Setup A.1 Benchmarks
LCBench. LCBench 4 is a feedforward neural network benchmark on tabular data which consists of 2000 configuration settings for each of the 35 datasets. The configurations were evaluated during
HPO runs with AutoPyTorch.
LCBench features a search space of 7 numerical hyperparameters, where every hyperparameter configuration is trained for 50 epochs.
The objective is to optimize seven different hyperparameters of funnel-shaped neural networks, i.e., batch size, learning rate, momentum, weight decay, dropout, number of layers, and maximum number of units per layer.
TaskSet.
TaskSet 5 is a benchmark that features over 1162 diverse tasks from different domains and includes 5 search spaces.
In this work, we focus on NLP tasks and we use the Adam8p search space with 8 continuous hyperparameters.
We refer to Figure 11 for the exact task names considered in our experiments. The learning curves provided in TaskSet report scores after every 200 iterations. We refer to those as "steps". The objective is to optimize eight hyperparameters for a set of different recurrent neural networks (RNN) that differ in embedding size, RNN cell, and other architectural features.
The set of hyperparameters consists of optimizer-specific hyperparameters, such as the learning rate, the exponential decay rate of the first and second momentum of Adam, β 1 and β 2 , and
Adam’s constant for numerical stability ε .
Furthermore, there are two hyperparameters controlling linear and exponential learning rate decays, as well as L1 and L2 regularization terms.
NAS-Bench-201.
NAS-Bench-201 6 is a benchmark that has precomputed about 15,600 architectures trained for 200 epochs for the image classification datasets CIFAR-10, CIFAR-100, and
ImageNet.
The objective is to select for each of the six operations within the cell of the macro architecture one of five different operations.
All other hyperparameters such as learning rate and batch size are kept fixed.
NAS-Bench-201 features a search space of 6 categorical hyperparameters and each architecture is trained for 200 epochs.
A.2 Preprocessing
In the following, we describe the preprocessing applied to the hyperparameter representation.
For
LCBench, we apply a log-transform to batch size, learning rate, and weight decay.
For TaskSet, we apply it on the learning rate, L1 and L2 regularization terms, linear and exponential decay of the learning rate.
All continuous hyperparameters are scaled to the range between 0 and 1 using sklearn’s MinMaxScaler.
If not mentioned otherwise, we use one-hot encoding for the categorical hyperparameters.
As detailed in subsection A.5, some baselines have a specific way of dealing with them.
In that case, we use the method recommended by the authors.
A.3 Framework
The framework contains the evaluated hyperparameters and their corresponding validation curves.
The list of candidate hyperparameters is passed to the baseline-specific interface, which in turn, optimizes and queries the framework for the hyperparameter configuration that maximizes utility.
Our framework in turn responds with the validation curve and the cost of the evaluation.
In case a hyperparameter configuration has been evaluated previously up to a budget b and a baseline requires the response for budget b + 1 , the cost is calculated accordingly only for the extra budget requested.
A.4 Implementation Details
We implement the Deep Kernel Gaussian Process using GPyTorch 1.5 [Gardner et al., 2018].
We use an RBF kernel and the dense layers of the transformation function φ have 128 and 256 units.
We used a convolutional layer with a kernel size of three and four filters.
All parameters of the Deep Kernel are estimated by maximizing the marginal likelihood.
We achieve this by using gradient ascent and
Adam [Kingma and Ba, 2015] with a learning rate of 0 . 1 and batch size of 64.
We stop training as soon as the training likelihood is not improving for 10 epochs in a row or we completed 1,000 epochs.
For every new data point, we start training the GP with its old parameters to reduce the required effort for training.
A.5 Baselines
Random Search & Hyperband.
Random search and Hyperband sample hyperparameter configurations at random and therefore the preprocessing is irrelevant.
We have implemented both from scratch and use the recommended hyperparameters for Hyperband, i.e. η = 3 .
BOHB.
For our experiments with BOHB, we use version 0.7.4 of the officially-released code 7 .
DEHB.
For our experiments with DEHB, we use the official public implementation 8 .
We developed an interface that communicates between our framework and DEHB.
In addition to the initial preprocessing common for all methods, we encode categorical hyperparameters with a numerical value in the interval [0, 1].
For a categorical hyperparameter x i , we take N i equal-sized intervals, where N i represents the number of unique categorical values for hyperparameter x
i and we assign the value for a categorical value n ∈ N i to the middle of the interval [ n, n + 1] as suggested by the authors. For configuring the DEHB algorithm we used the default values from the library. Dragonfly. We use the publicly available code of Dragonfly 9 .
No special treatment of categorical hyperparameters is required since Dragonfly has its own way to deal with them.
We use version 0 . 1 .with default settings. MF-DNN. We use the official implementation of MF-DNN by the authors 10 .
Initially, we tried to use multiple incremental fidelity levels like for D Y HPO, however, the method runtime was too high and it could not achieve competitive results.
For that reason, we use only a few fidelity levels like the authors do in their work Li et al.
[2020b].
We use the same fidelity levels as for Hyperband, DEHB, and BOHB to have a fair comparison between the baselines.
We also use the same number of initial points as for the other methods to have the same maximal resource allocated for every fidelity level.
ASHA-HB.
We use the public implementation from the well-known optuna library (version 2 . 10 . 0 ).
We used the same eta, minimum and maximal budget as for HB, DEHB, and BOHB in our experiments, to have a fair comparison.
B Additional Plots
In Figure 8, we ablate the learning curve input in our kernel, to see the effect it has on performance for the CIFAR-10 and CIFAR-100 datasets from the NAS-Bench-201 benchmark.
The results indicate that the learning curve plays an important role in achieving better results by allowing faster convergence and a better anytime performance.
[Figure]
[Figure]
Additionally, in Figure 9, we show the performance comparison over the number of epochs of every method for the CIFAR-10 and CIFAR-100 datasets in the NAS-Bench-201 benchmark.
While, in
Figure 10, we present the performance comparison over time.
As can be seen, D Y HPO converges faster and has a better performance compared to the other methods over the majority of the time or steps, however, towards the end although it is the optimal method or close to the optimal method, the difference in regret is not significant anymore.
Furthermore
[Figure]
Figure 11 shows the performance comparison for the datasets chosen from TaskSet over the number of steps.
Looking at the results, D Y HPO is outperforming all methods convincingly on the majority of datasets by converging faster and with significant differences in the regret evaluation metric.
In Figure 12 and
[Figure]
we show the performance comparison for all the datasets from LCBench regarding regret over the number of epochs.
Similarly, in Figure 14 and 15, we show the same performance comparison, however, over time.
As can be seen, D Y HPO manages to outperform the other competitors in the majority of the datasets, and in the datasets that it does not, it is always close to the top-performing method, and the difference between methods is marginal.
In Figure 16 we provide the extended results of Experiment 3 for TaskSet.
We show the precision, average regret, and promotion percentage for poor-performing configurations for D Y HPO and the other competitor methods.
Lastly, we explore the behavior of D Y HPO after finding the configuration which is returned at the end of the optimization as the best configuration.
In Figure 17, we show how the budget is distributed on the configurations considered during that part of the optimization.
Clearly, D Y HPO is spending very little budget on most configurations.
Furthermore, we investigated how many new configurations are considered during this phase.
For LCBench, 76.98% of considered configurations are new demonstrating that D Y HPO is investigating most of the budget into exploration.
These are even more extreme for TaskSet (93.16% and NAS-Bench-201 (97.51%).
[Figure]
MF
[Figure]
DNN
DyHPO
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
t e s
[Figure]