Supervising the Multi-Fidelity Race of Hyperparameter Configurations
Abstract
Multi-fidelity (gray-box) hyperparameter optimization techniques (HPO) haverecently emerged as a promising direction for tuning Deep Learning methods.
However, existing methods suffer from a sub-optimal allocation of the HPO budget to the hyperparameter configurations.
In this work, we introduce DyHPO, a
Bayesian Optimization method that learns to decide which hyperparameter configuration to train further in a dynamic race among all feasible configurations.
Wepropose a new deep kernel for Gaussian Processes that embeds the learning curvedynamics, and an acquisition function that incorporates multi-budget information.
We demonstrate the significant superiority of DyHPO against state-of-the-art hyperparameter optimization methods through large-scale experiments comprising 50datasets (Tabular, Image, NLP) and diverse architectures (MLP, CNN/NAS, RNN)
Introduction
Hyperparameter Optimization (HPO) is arguably an acute open challenge for Deep Learning (DL),especially considering the crucial impact HPO has on achieving state-of-the-art empirical results.
Unfortunately, HPO for DL is a relatively under-explored field and most DL researchers still optimizetheir hyperparameters via obscure trial-and-error practices.
On the other hand, traditional Bayesian
Optimization HPO methods [Snoek et al., 2012, Bergstra et al., 2011] are not directly applicable todeep networks, due to the infeasibility of evaluating a large number of hyperparameter configurations.
In order to scale HPO for DL, three main directions of research have been recently explored.
(i)
Online HPO methods search for hyperparameters during the optimization process via meta-levelcontrollers [Chen et al., 2017, Parker-Holder et al., 2020], however, this online adaptation can notaccommodate all hyperparameters (e.g. related to architectural changes).
(ii) Gradient-based HPOtechniques, on the other hand, compute the derivative of the validation loss w.r.t.
hyperparameters byreversing the training update steps [Maclaurin et al., 2015, Franceschi et al., 2017, Lorraine et al.
,2020], however, the reversion is not directly applicable to all cases (e.g. dropout rate).
The lastdirection, (iii) Gray-box HPO techniques discard sub-optimal configurations after evaluating them onlower budgets [Li et al., 2017, Falkner et al., 2018].
In contrast to the online and gradient-based alternatives, gray-box approaches can be deployed inan off-the-shelf manner to all types of hyperparameters and architectures.
The gray-box concept isbased on the intuition that a poorly-performing hyperparameter configuration can be identified andterminated by inspecting the validation loss of the first few epochs, instead of waiting for the fullconvergence.
The most prominent gray-box algorithm is Hyperband [Li et al., 2017], which is basedon successive halving.
It runs random configurations at different budgets (e.g. number of epochs)and successively halves these configurations by keeping only the top performers.
Follow-up works,such as BOHB [Falkner et al., 2018] or DEHB [Awad et al., 2021], replace the random sampling of
Hyperband with a sampling based on Bayesian optimization or differentiable evolution.
Despite their great practical potential, gray-box methods suffer from a major issue.
The low-budget(few epochs) performances are not always a good indicator for the full-budget (full convergence)performances.
For example, a properly regularized network converges slower in the first few epochs,however, typically performs better than a non-regularized variant after the full convergence.
In otherwords, there can be a poor rank correlation of the configurations’ performances at different budgets.
We introduce D Y HPO, a Bayesian Optimization(BO) approach based on Gaussian Processes(GP), that proposes a novel treatment to themulti-budget (a.k.a.
multi-fidelity) setup.
In thisperspective, we propose a deep kernel GP thatcaptures the learning dynamics.
As a result, wetrain a kernel capable of capturing the similarityof a pair of hyperparameter configurations, evenif the pair’s configurations are evaluated at different budgets.
Furthermore, we extend Expected
Improvement [Jones et al., 1998] to the multibudget case, by introducing a new mechanismfor the incumbent configuration of a budget
We illustrate the differences between our racing strategy and successive halving with theexperiment of Figure 1, where, we showcasethe HPO progress of two different methods onthe "Helena" dataset from the LCBench benchmark [Zimmer et al., 2021].
Hyperband [Liet al., 2017] is a gray-box approach that statically pre-allocates the budget for a set of candidates (Hyperband bracket) according to a predefinedpolicy.
However, D Y HPO dynamically adapts the allocation of budgets for configurations after every
HPO step (a.k.a.
a dynamic race).
As a result, D Y HPO invests only a small budget on configurationsthat show little promise as indicated by the intermediate scores.
The joint effect of modeling a GP kernel across budgets together with a dedicated acquisitionfunction leads to D Y HPO achieving a statistically significant empirical gain against state-of-the-artgray-box baselines [Falkner et al., 2018, Awad et al., 2021], including prior work on multi-budget
GPs [Kandasamy et al., 2017, 2020] or neural networks [Li et al., 2020b].
We demonstrate theperformance of D Y HPO in three diverse deep learning architectures (MLP, CNN/NAS, RNN) and 50datasets of three diverse modalities (tabular, image, natural language processing).
We believe ourmethod is a step forward toward making HPO for DL practical and feasible.
Overall, our contributionscan be summarized as follows:
• We introduce a novel Bayesian surrogate for gray-box HPO optimization.
Our novel
surrogate model predicts the validation score of a machine learning model based on both the
hyperparameter configuration, the budget information, and the learning curve.•
We derive a simple yet robust way to combine this surrogate model with Bayesian opti-
mization, reusing most of the existing components currently used in traditional Bayesianoptimization methods.
• Finally, we demonstrate the efficiency of our method for HPO and neural architecture search
tasks compared to the current state-of-the-art methods in HPO, by outperforming sevenstrong HPO baselines with a statistically significant margin.
As an overarching goal, webelieve our method is an important step toward scaling HPO for DL.
Related Work on Gray-box HPO
Multi-Fidelity Bayesian Optimization and Bandits.
Bayesian optimization is a black-box function optimization framework that has been successfully applied in optimizing hyperparameter andneural architectures alike [Snoek et al., 2012, Kandasamy et al., 2018, Bergstra et al., 2011].
Tofurther improve Bayesian optimization, several works propose low-fidelity data approximations ofhyperparameter configurations by training on a subset of the data [Swersky et al., 2013, Klein et al.,
2017a], or by terminating training early [Swersky et al., 2014].
Additionally, several methods extend
Bayesian optimization to multi-fidelity data by engineering new kernels suited for this problem [Swersky et al., 2013, 2014, Poloczek et al., 2017].
Kandasamy et al.
[2016] extends GP-UCB
[Srinivaset al., 2010] to the multi-fidelity setting by learning one Gaussian Process (GP) with a standard kernelfor each fidelity.
Their later work improves upon this method by learning one GP for all fidelities thatenables the use of continuous fidelities [Kandasamy et al., 2017].
The work by Takeno et al.
[2020]follows a similar idea but proposes to use an acquisition function based on information gain insteadof UCB.
While most of the works rely on GPs to model the surrogate function, Li et al.
[2020b] use a
Bayesian neural network that models the complex relationship between fidelities with stacked neuralnetworks, one for each fidelity.
Hyperband [Li et al., 2017] is a bandits-based multi-fidelity method for hyperparameter optimizationthat selects hyperparameter configurations at random and uses successive halving [Jamieson and Talwalkar, 2016] with different settings to early-stop less promising training runs.
Several improvementshave been proposed to Hyperband with the aim to replace the random sampling of hyperparameterconfigurations with a more guided approach [Bertrand et al., 2017, Wang et al., 2018, Wistuba, 2017].
BOHB [Falkner et al., 2018] uses TPE [Bergstra et al., 2011] and builds a surrogate model for everyfidelity adhering to a fixed-fidelity selection scheme.
DEHB [Awad et al., 2021] samples candidatesusing differential evolution which handles large and discrete search spaces better than BOHB.
Mendeset al.
[2021] propose a variant of Hyperband which allows to skip stages.
Deep Kernel Learning with Bayesian Optimization.
We are among the first to use deep kernellearning with Bayesian optimization and to the best of our knowledge the first to use it for multifidelity Bayesian optimization.
Rai et al.
[2016] consider the use of a deep kernel instead of a manuallydesigned kernel in the context of standard Bayesian optimization, but, limit their experimentationto synthetic data and do not consider its use for hyperparameter optimization.
Perrone et al.
Wistuba and Grabocka [2021] use a pre-trained deep kernel to warm start Bayesian optimization withmeta-data from previous optimizations.
The aforementioned approaches are multi-task or transferlearning methods that require the availability of meta-data from related tasks.
In contrast to prior work, we propose a method that introduces deep learning to multi-fidelity HPOwith Bayesian Optimization, and captures the learning dynamics across fidelities/budgets, combinedwith an acquisition function that is tailored for the gray-box setup.
Dynamic Multi-Fidelity HPO
3.1 Preliminaries
Gray-Box Optimization.
The gray-box HPO setting allows querying configurations with a smallerbudget compared to the total maximal budget B .
Thus, we can query from the response functionf : X × N → R where f i,j = f ( x i , j ) is the response after spending a budget of j on configuration x i .
As before, these observations are noisy and we observe y i,j = f
( x i , j ) +
ε j where ε j ∼ N (0 , σ 2 j,n ) .
Please note, we assume that the budget required to query f
i,j + b after having queried f i,j is only b .
Furthermore, we use the learning curve Y i,j − 1 =
( y i, 1 , . . .
, y i,j − 1 ) when predicting f i,j .
Gaussian Processes (GP).
Given a training data set D = { ( x i , y i ) } ni =1 , the Gaussian Processassumption is that y i is a random variable and the joint distribution of all y i is assumed to bemultivariate Gaussian distributed as y ∼ N ( m ( X ) , k
( X , X )) .
Furthermore, f ∗ for test instancesx ∗ are jointly Gaussian with y as:
[Equation]
The mean function m is often set to 0 and its covariance function k depends on parameters θ .
For notational convenience, we use K n = k
( X , X | θ ) + σ 2 n I , K ∗
=
k
( X , X ∗ | θ ) and K ∗∗ =k
( X ∗ , X ∗ | θ ) to define the kernel matrices.
We can derive the posterior predictive distribution withmean and covariance as follows:
[Equation]
Often, the kernel function is manually engineered, one popular example is the squared exponentialkernel.
However, in this work, we make use of the idea of deep kernel learning [Wilson et al.,
The idea is to model the kernel as a neural network φ and learn the best kernel transformation
K ( θ, w ) :
=
k ( φ ( x , w ) , φ ( x ′ ; w ) | θ ) , which allows us to use convolutional operations in our kernel.
3.2 Deep Multi-Fidelity Surrogate
We propose to use a Gaussian Process surrogate model that infers the value of f i,j based on thehyperparameter configuration x
i , the budget j as well as the past learning curve Y
i,j − 1 .
For thispurpose, we use a deep kernel as:
[Equation]
′ ,j ′ − 1 ,
[Equation] [Equation]
[Equation]
) (3) x i j
Y i,j −· 1 B (cid:80) (cid:80) (cid:80)
Convolution max (cid:80) (cid:80) (cid:80)
We use a squared exponential kernel for k andthe neural network φ is composed of linear andconvolutional layers as shown in Figure 2.
Wenormalize the budget j to a range between 0and 1 by dividing it by the maximum budget
B .
Afterward, it is concatenated with the hyperparameter configuration x
i and fed to a linearlayer.
The learning curve Y i,j − 1 is transformedby a one-dimensional convolution followed bya global max pooling layer.
Finally, both representations are fed to another linear layer.
Its output will be the input to the kernel function k .
Both, the kernel k and the neural network φconsist of trainable parameters θ and w , respectively.
We find their optimal values by computing themaximum likelihood estimates as:
[Equation]
In order to solve this optimization problem, we use gradient descent and Adam [Kingma and Ba,2015] with a learning rate of 0 . 1 .
Given the maximum likelihood estimates, we can approximate thepredictive posterior through p (cid:16) f i,j | x i , Y i,j − 1 , j, D , ˆ θ , ˆ w (cid:17) , and ultimately compute the mean andcovariance of this Gaussian using Equation 2.
3.3 Multi-Fidelity Expected Improvement
Expected improvement [Jones et al., 1998] is a commonly used acquisition function and is defined as:
[Equation] [Equation]
where y max is the largest observed value of f .
We propose a multi-fidelity version of it as:
[Equation]
6) where:
[Equation]
Simply put, y max j is the largest observed value of f for a budget of j if it exists already, otherwise, itis the largest observed value for any budget.
If there is only one possible budget, the multi-fidelityexpected improvement is identical to expected improvement.
3.4 The D Y HPO Algorithm
The D Y HPO algorithm looks very similar to many black-box Bayesian optimization algorithms asshown in Algorithm 1.
The big difference is that at each step we dynamically decide which candidateconfiguration to train for a small additional budget .
Algorithm
1: b ( x ) =
∀ x ∈ X 2: while not converged do 3: x i ← arg max x ∈X EI MF ( x , b ( x ) + 1) (Sec. 3.3) 4:
Observe y i,b ( x i )+1 .
5: b ( x i ) ← b ( x i ) +6:
D ← D ∪ (cid:8) (( x i , Y i,b ( x i ) − 1 , b ( x i )) , y i,b ( x i ) ) (cid:9) 7:
Update the surrogate on D .
(Sec. 3.2) return x i with largest y i, · .
Possible candidates are previously unconsidered configurations as well asconfigurations that did not reach themaximum budget.
In Line 2, the mostpromising candidate is chosen usingthe acquisition function introduced in
Section 3.3 and the surrogate model’spredictions.
It is important to highlight that we do not maximize the acquisition function along the budget dimensionality.
Instead, we set the budget b such that it is by exactly onehigher than the budget used to evaluate x i before.
This ensures that we explore configurations by slowly increasing the budget.
After thecandidate and the corresponding budget are selected, the function f is evaluated and we observe y i,j(Line 3).
This additional data point is added to D in Line 4.
Then in Line 5, the surrogate model isupdated according to the training scheme described in Section 3.2.
Experimental Protocol
4.1 Experimental Setup
We evaluate D Y HPO in three different settings on hyperparameter optimization for tabular, text, andimage classification against several competitor methods, the details of which are provided in thefollowing subsections.
We ran all of our experiments on an Amazon EC2 M5 Instance (m5.xlarge).
In our experiments, we report the mean of ten repetitions and we report two common metrics suchas the regret and the average rank.
The regret refers to the absolute difference between the score ofthe solution found by an optimizer compared to the best possible score.
If we report the regret as anaggregate result over multiple datasets, we report the mean over all regrets.
The average rank is themetric we use to aggregate rank results over different datasets.
We provide further implementationand training details in Appendix A.4.
4.2 Benchmarks
In our experiments, we use the following benchmarks.
We provide more details in Appendix A.1.
LCBench:
A learning curve benchmark [Zimmer et al., 2021] that evaluates neural network architectures for tabular datasets.
LCBench contains learning curves for 35 different datasets, where 2,000neural networks per dataset are trained for 50 epochs with Auto-PyTorch.
TaskSet: A benchmark that features diverse tasks Metz et al.
[2020] from different domains andincludes 5 search spaces with different degrees of freedom, where, every search space includes 1000hyperparameter configurations.
In this work, we focus on a subset of NLP tasks (12 tasks) and weuse the Adam8p search space with 8 continuous hyperparameters.
NAS-Bench-201: A benchmark consisting of 15625 hyperparameter configurations representingdifferent architectures on the CIFAR-10, CIFAR-100 and ImageNet datasets Dong and Yang [2020].
NAS-Bench-201 features a search space of 6 categorical hyperparameters and each architecture is
trained for 200 epochs.
4.3 Baselines
Random Search: A random/stochastic black-box search method for HPO.
HyperBand:
A multi-arm bandit method that extends successive halving by multiple brackets with
different combinations of the initial number of configurations, and their initial budget [Li et al., 2017].BOHB: An extension of Hyperband that replaces the random sampling of the initial configurations
for each bracket with recommended configurations from a model-based approach [Falkner et al.,2018].
BOHB builds a model for every fidelity that is considered.
[Figure]
DEHB: A method that builds upon Hyperband by exploiting differential evolution to sample theinitial candidates of a Hyperband bracket [Awad et al., 2021].
ASHA:
An asynchronous version of successive halving (or an asynchronous version of Hyperband ifmultiple brackets are run).
ASHA Li et al.
[2020a] does not wait for all configurations to finish insidea successive halving bracket, but, instead promotes configurations to the next successive halvingbracket in real-time.
MF-DNN: A multi-fidelity Bayesian optimization method that uses deep neural networks to capturethe relationships between different fidelities Li et al.
[2020b].
Dragonfly: We compare against BOCA [Kandasamy et al., 2017] by using the Dragonfly library Kandasamy et al.
This method suggests the next hyperparameter configuration as well as thebudget it should be evaluated for.
4.4 Research Hypotheses and Associated Experiments
Hypothesis 1: D Y HPO achieves state-of-the-art results in multi-fidelity HPO.
Experiment 1: We compare D Y HPO against the baselines of Section 4.3 on the benchmarksof Section 4.2 with the experimental setup of Section 4.1.
For TaskSet we follow the authors’recommendation and report the number of steps (every 200 iterations).
Hypothesis 2: D Y HPO’s runtime overhead has a negligible impact on the quality of results.
Experiment 2: We compare D Y HPO against the baselines of Section 4.3 over the wallclock time.
The wallclock time includes both (i) the optimizer’s runtime overhead for recommending the nexthyperparameter configuration, plus (ii) the time needed to evaluate the recommended configuration.
In this experiment, we consider all datasets where the average training time per epoch is at least10 seconds, because, for tasks where the training time is short, there is no practical justification forcomplex solutions and their overhead.
In these cases, we recommend using a random search.
Wedon’t report results for TaskSet because the benchmark lacks training times.
Hypothesis 3: D Y HPO uses the computational budget more efficiently than baselines.
Experiment 3: To further verify that D Y HPO is efficient compared to the baselines, we investigatewhether competing methods spend their budgets on qualitative candidates.
Concretely we: i) calculatethe precision of the top (w.r.t. ground truth) performing configurations that were selected by eachmethod across different budgets, ii) compute the average regret of the selected configurations acrossbudget, and iii) we compare the fraction of top-performing configurations at a given budget thatwere not top performers at lower budgets, i.e. measure the ability to handle the poor correlation ofperformances across budgets.
Results
Experiment 1: D Y HPO achieves state-of-the-art results.
In our first experiment, we evaluatethe various methods on the benchmarks listed in Section 4.2.
We show the aggregated results in
Figure 3, the results show that D Y HPO manages to outperform competitor methods over the set ofconsidered benchmarks by achieving a better mean regret across datasets.
Not only does D Y HPO achieve a better final performance, it also achieves strong anytime results by converging faster thanthe competitor methods.
For the extended results, related to the performance of all methods on adataset level, we refer the reader to Appendix {+B.
 [Figure] [Figure]+}
RandomDragonflyHyperbandMF-DNN
ASHABOHBDEHBDyHPO
LCBench
[Figure]
The results correspond to results after 500 and 1,000 epochs.
In Figure 4, we provide further evidence that D Y HPO’s improvement over the baselines is statisticallysignificant.
The critical difference diagram presents the ranks of all methods and provides informationon the pairwise statistical difference between all methods for two fractions of the number of HPOsteps (50% and 100%).
We included the LCBench and TaskSet benchmarks in our significance plots.
NAS-Bench-201 was omitted because it has only 3 datasets and the statistical test cannot be applied.
Horizontal lines indicate groupings of methods that are not significantly different.
As suggested bythe best published practices Demsar [2006], we use the Friedman test to reject the null hypothesisfollowed by a pairwise post-hoc analysis based on the Wilcoxon signed-rank test ( α = 0 . 05 ).
For LCBench, D Y HPO already outperforms the baselines significantly after 50% of the search budget,with a statistically significant margin.
As the optimization procedure continues, D Y HPO manages toextend its gain in performance and is the only method that has a statistically significant improvementagainst all the other competitor methods.
Similarly, for TaskSet, D Y HPO manages to outperform allmethods with a statistically significant margin only halfway through the optimization procedure andachieves the best rank over all methods.
However, as the optimization procedure continues, BOHBmanages to decrease the performance gap with D Y HPO, although, it still achieves a worse rankacross all datasets.
Considering the empirical results, we conclude that Hypothesis 1 is validatedand that D Y HPO achieves state-of-the-art results on multi-fidelity HPO .
[Figure]
Experiment 2:
On the impact of D Y HPO’s overhead on the results.
We present the results ofour second experiment in Figure 5 (left), where, as it can be seen, D Y HPO still outperforms theother methods when its overhead is considered.
For LCBench, D Y HPO manages to get an advantagefairly quickly and it only increases the gap in performance with the other methods as the optimizationprocess progresses.
Similarly, in the case of ImageNet from NAS-Bench-201, D Y HPO manages togain an advantage earlier than other methods during the optimization procedure.
Although in the end
D Y HPO still performs better than all the other methods, we believe most of the methods converge toa good solution and the differences in the final performance are negligible.
For the extended results,related to the performance of all methods on a dataset level over time, we refer the reader to theplots in Appendix B. Additionally, in Figure 5 (right), we provide the critical difference diagramsfor LCBench that present the ranks and the statistical difference of all methods halfway through the
[Figure]
r e c i s i
[Figure]
o f T op
[Figure]
i da t e
[Figure]
optimization procedure, and in the end.
As it can be seen, D Y HPO has a better rank with a significantmargin with only half of the budget used and it retains the advantage until the end.
Experiment 3:
On the efficiency of D Y HPO.
In Figure 6 (left), we plot the precision of everymethod for different budgets during the optimization procedure, which demonstrates that D Y HPOeffectively explores the search space and identifies promising candidates.
The precision at an epoch iis defined as the number of top 1% candidates that are trained, divided by the number of all candidatestrained, both trained for at least i epochs.
The higher the precision, the more relevant candidates wereconsidered and the less computational resources were wasted.
For small budgets, the precision islow since D Y HPO spends budget to consider various candidates, but then, promising candidatesare successfully identified and the precision quickly increases.
This argument is further supportedin Figure 6 (middle), where we visualize the average regret of all the candidates trained for at leastthe specified number of epochs on the x-axis.
In contrast to the regret plots, here we do not showthe regret of the best configuration, but the mean regret of all the selected configurations.
Theanalysis deduces a similar finding, our method D Y HPO selects more qualitative hyperparameterconfigurations than all the baselines.
An interesting property of multi-fidelity HPO is the phenomenon of poor rank correlations amongthe validation performance of candidates at different budgets.
In other words, a configuration thatachieves a poor performance at a small budget can perform better at a larger budget.
To analyze thisphenomenon, we measure the percentage of "good" configurations at a particular budget, that were"bad" performers in at least one of the smaller budgets.
We define a "good" performance at a budget
B when a configuration achieves a validation accuracy ranked among the top 1/3 of the validationaccuracies belonging to all the other configurations that were run until that budget B.
In Figure 6 (right), we analyze the percentage of "good" configurations at each budget denoted by thex-axis, that were "bad" performers in at least one of the lower budgets.
Such a metric is a proxy forthe degree of the promotion of "bad" configurations towards higher budgets.
We present the analysisfor all the competing methods of our experimental protocol from Section 4.
We have additionallyincluded the ground-truth line annotated as "Baseline", which represents the fraction of past poorperformers among all the feasible configurations in the search space.
In contrast, the respectivemethods compute the fraction of promotions only among the configurations that those methods haveconsidered (i.e. selected within their HPO trials) until the budget indicated by the x-axis.
We see thatthere is a high degree of "good" configurations that were "bad" at a previous budget, with fractions of the ground-truth "Baseline" going up to 40% for the LCBench benchmark and up to 80% for the
NAS-Bench-201 benchmark.
On the other hand, the analysis demonstrates that our method D Y HPO has promoted more "good"configurations that were "bad" in a lower budget, compared to all the rival methods.
In particular,more than 80% of selected configurations from the datasets belonging to either benchmark were "bad"performers at a lower budget.
The empirical evidence validates Hypothesis 3 and demonstratesthat D Y HPO efficiently explores qualitative candidates.
We provide the results of our analysis for
D Y HPO’s efficiency on the additional benchmarks (Taskset) in Appendix B.
Ablating the impact of the learning curve
[Figure]
One of the main differences between D Y HPOand similar methods Kandasamy et al.
[2017], isthat the learning curve is an input to the kernelfunction.
For this reason, we investigate theimpact of this design choice.
We consider avariation of D Y HPO w/o CNN, which is simply
D Y HPO without the learning curve.
It is worth emphasizing that both variants (withand without the learning curve) are multi-fidelitysurrogates and both receive the budget information through the inputted index j in Equation 3.
The only difference is that D Y HPO additionallyincorporates the pattern of the learning curve.
We run the ablation on the NAS-Bench-201 benchmark and report the results for ImageNet, thelargest dataset in our collection.
The ablation results are shown in Figure 7, while the remainingresults on the other datasets are shown in Figure 8 of the appendix.
Based on the results from ourlearning curve ablation, we conclude that the use of an explicit learning curve representation leads tosignificantly better results.
Limitations of Our Method
Although D Y HPO shows a convincing and statistically significant reduction of the HPO time ondiverse Deep Learning (DL) experiments, we cautiously characterized our method only as a "steptowards" scaling HPO for DL.
The reason for our restrain is the lack of tabular benchmarks for HPOon very large deep learning models, such as Transformers-based architectures [Devlin et al., 2019].
Additionally, the pause and resume part of our training procedure can only be applied when tuningthe hyperparameters of parametric models, otherwise, the training of a hyperparameter configurationwould have to be restarted.
Lastly, for small datasets that can be trained fast, the overhead ofmodel-based techniques would make an approach like random search more appealing.
Conclusions
In this work, we present D Y HPO, a new Bayesian optimization (BO) algorithm for the gray-boxsetting.
We introduced a new surrogate model for BO that uses a learnable deep kernel and takes thelearning curve as an explicit input.
Furthermore, we motivated a variation of expected improvementfor the multi-fidelity setting.
Finally, we compared our approach on diverse benchmarks on atotal of 50 different tasks against the current state-of-the-art methods on gray-box hyperparameteroptimization (HPO).
Our method shows significant gains and has the potential to become the de factostandard for HPO in Deep Learning.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 6.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See
Section “Societal Implications” in the Appendix.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[No] We provide our
main algorithm in Section
and we additionally provide the detailed implementation
details in Appendix A for all methods and benchmarks.
We will release the code for
the camera-
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] We report the statistical significance of the performancedifference between our method and the baselines in Section
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Section 4.1.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 4.2 andSection 4.3.
(b) Did you mention the license of the assets?
[Yes] See Appendix A.1 and A.5 where we
provide references to the assets where the license is included.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
The benchmarks and baselines are open-sourced.
(e) Did you discuss whether the data you are using/curating contains personally identi-
fiable information or offensive content?
[N/A]
The data does not contain personally
identifiable information or offensive content.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]