Bag of Baselines for Multi-objective Joint Neural Architecture Search and Hyperparameter Optimization
Abstract
While both neural architecture search (NAS) and hyperparameter optimization (HPO) have been studied extensively in recent years, NAS methods typically assume ﬁxed hyperparameters and vice versa.
Furthermore, NAS has recently often been framed as a multi-objective optimization problem, in order to take, e.g., resource requirements into account.
In this paper, we propose a set of methods that extend current approaches to jointly optimize neural architectures and hyperparameters with respect to multiple objectives.
We hope that these methods will serve as simple baselines for future research on multi-objective joint NAS + HPO.
1. Introduction
Neural architecture search (NAS) and hyperparameter optimization (HPO) are both important components of AutoML, but there exists little work on joint NAS and HPO.
NAS methods typically assume ﬁxed hyperparameter conﬁgurations and HPO ﬁxed architectures, even though it seems natural that diﬀerent architectures require diﬀerent hyperparameter conﬁgurations to yield optimal performance.
Indeed, there is evidence that this is the case.
For example, Gastaldi (2017) showed that the strongest version of the proposed shake-shake regularization performs best for some architectures, but is too strong for other architectures, resulting in poor performance or even divergence during training.
Thus, a joint optimization of hyperparameter conﬁgurations and architecture can be expected to be beneﬁcial.
Furthermore, while NAS and HPO methods typically optimize for accuracy, in many real-world applications there is more than one objective.
Common objectives next to accuracy are, e.g., memory requirements, energy consumption or latency on the target hardware where the neural network is eventually deployed.
In this paper, we take a ﬁrst step in the direction of multi-objective joint NAS + HPO by proposing and empirically evaluating a set of simple, yet powerful baseline methods.
All our baseline methods essentially extend current NAS or HPO approaches to cover both classical and architectural hyperparameters, optimized under multiple objectives; see Table 1 for an overview of the methods we propose.
2. Related Work and Background
Neural Architecture Search (NAS).
NAS refers to the task of learning neural network architectures from data (Elsken et al., 2019b; Wistuba et al., 2019).
NAS approaches often employ black-box optimization methods, such as evolutionary algorithms (Real et al., 2017, 2019), reinforcement learning (Zoph and Le, 2017), or Bayesian optimization (Mendoza et al., 2016; Kandasamy et al., 2018).
However, due to the large computational costs, researchers have developed methods tailored towards NAS, e.g., (gradient-based) optimization on one-shot models (Bender et al., 2018; Pham et al., 2018; Liu et al., 2019).
Hyperparameter Optimization (HPO).
The ﬁeld of HPO (see, e.g., (Feurer and
Hutter, 2019)) automates the search for well performing hyperparameter conﬁgurations.
Bayesian Optimization (BO) (Brochu et al., 2010; Shahriari et al., 2016) is a popular framework for solving HPO problems by employing cheap-to-evaluate surrogate model for predicting the performance of hyperparameter conﬁgurations and using an acquisition function to trade-oﬀ exploration and exploitation for selecting a new candidates.
Evolutionary methods are also a common choice for HPO (Loshchilov and Hutter, 2016; Jaderberg et al., 2017).
For this, a population of hyperparameter conﬁgurations is evolved over time by mutating or crossing well-performing candidates.
Multi-ﬁdelity methods are also often employed as a tool for HPO to speed up function evaluations.
Successive Halving (SH) (Jamieson and Talwalkar, 2016) and Hyperband (HB) (Li et al., 2018) are two powerful multi-ﬁdelity strategies that allocate more budget on the well-performing hyperparameter conﬁgurations and achieve strong anytime performance.
However, both strategies select new conﬁgurations at random without exploiting the knowledge gained about well-performing regions.
BOHB (Falkner et al., 2018), which combines BO and HB, overcomes this issue by guiding Hyperband via a TPE model.
Joint NAS + HPO.
Few researchers so far have considered the joint optimization of architectures and hyperparameter conﬁgurations.
Domhan et al.
and Mendoza et al.
(2016) use SMAC (Hutter et al., 2011) to jointly optimize both architectures and hyperparameter conﬁgurations, Hundt et al.
(2019) use BO with Gaussian processes.
Zela et al.
(2018), Runge et al.
(2019) and Zimmer et al.
(2021) employed BOHB (Falkner et al., 2018) to achieve the same goal, and Awad et al.
(2021) proposed DEHB for this problem.
Saikia et al.
(2019) optimized architecture and hyperparameters in two stages:
they ﬁrst employ
DARTS (Liu et al., 2019) to search for better architectures for disparity estimation and then optimize the hyperparameters of the resulting architecture with BOHB.
Finally, Dong et al.
(2020) extended NAS methods using one-shot models to also consider hyperparameters.
Multi-objective Optimization.
Multi-objective optimization (e.g., Miettinen (1999)) deals with the problem of minimizing multiple objective functions f 1 ( λ ) , . . .
, f n ( λ ).
In general, there is no single conﬁguration λ that minimizes all objectives since the objectives are typically contradicting.
Rather, there are multiple Pareto-optimal solutions, meaning that one cannot reduce any f i without increasing at least one other f j ( i (cid:54) = j ).
The set of
Pareto-optimal solutions is called the Pareto front .
One class of algorithms for solving multi-objective problems are evolutionary algorithms.
Criteria for selecting candidates being mutated and deﬁning the best current solutions are typically based on non-dominated sorting (NDS) (Srinivas and Deb, 1994; Deb et al., 2002) and the hypervolume indicator (Emmerich et al., 2005; Beume et al., 2007; Bader and
Zitzler, 2011).
NDS extends the ranking of a set of candidates based on a single objective to multiple objectives, please refer to Appendx A.1 for more details.
The hypervolume indicator of a population measures, informally speaking, the space of objective function values covered by the population; thus maximizing the hypervolume indicator corresponds to improving the Pareto front.
Based on the hypervolume indicator, the hypervolume subset selection problem (HSSP) (Bader and Zitzler, 2011) is deﬁned as the problem of ﬁnding a subset of the population so that the hypervolume is maximized for this subset.
The HSSP can also be employed to identify candidates that contribute little to the hypervolume and thus can be considered poor, see Appendx A.2 for a more details.
BO approaches have been extended to multi-objective problems.
For example, expected hypervolume improvement (EHVI) (Emmerich, 2005) extends the work by Mockus et al. (1978) by considering improvements to the Pareto front.
Ozaki et al.
(2020) extended TPE (Bergstra et al., 2011) to multi-objective TPE (MOTPE).
In concurrent work, Salinas et al.
(2021) and Schmucker et al. (2021) extend Hyperband and Asynchronous Successive Halving (Li et al., 2020), respectively, with NDS and a multi-objective candidate selection scheme. Both approaches ares closely related to MO-BOHB (Section 3.2).
Multi-objective optimization also naturally arise in NAS since many real-world applications require efﬁcient architectures w.r.t.
, e.g., energy consumption or latency.
Consequently, a line of research frames NAS either as a constrained (Tan et al., 2019;
Cai et al., 2019) or multiobjective (Elsken et al., 2019a; Lu et al., 2020) optimization problem.
3. Proposed Methods
In the following, we propose ﬁve simple, yet powerful extensions of existing HPO and NAS optimization techniques to multi-objective joint HPO + NAS.
3.1 SH-EMOA : Speeding up Evolutionary Multi-Objective Algorithms (EMOA)
The ﬂexibility and conceptual simplicity of evolutionary algorithms make them directly applicable to multi-objective optimization problems.
For example SMS-EMOA (Beume et al., 2007) evaluates the performance of each candidate based on its contribution to the dominated hypervolume.
Although eﬀective, evolutionary algorithms tend to be very sampleineﬃcient, making them computationally quite expensive.
In order to deal with this problem, we propose SH-EMOA to speed up EMOAs by using a multi-ﬁdelity approach based on successive halving (please see Algorithm 1 in the appendix for full details).
In a nutshell, we iterate EMOA by doubling the training budgets in each iteration, while the number of candidates is halved.
Thus, many candidates are evaluated with a small budget to cover a wide range of solutions, while only well-performing candidates proceed to the next stage, and are evaluated with the next higher budget and used to generate new candidates.
We use NDS and the HSSP to identify poorly performing candidates.
3.2 MO-BOHB : Generalization of BOHB to an Arbitrary Number of Objectives
In order to extend BOHB to multi-objective optimization, we make two modiﬁcations.
Firstly, we replace TPE (Bergstra et al., 2011) originally used in BOHB by MOTPE (Ozaki et al., 2020) for selecting new conﬁgurations w.r.t.
multiple objectives.
Secondly, we extend
HB in a similar fashion as for SH-EMOA and MOTPE to decide with which conﬁguration to proceed in the next stage: we use NDS and the result of the HSSP; see Algorithms 2 and 3.
3.3 MS-EHVI : Mixed Surrogate Expected Hypervolume Improvement
Although EHVI can be directly applied for joint NAS and HPO obtaining competent results, we further enhance the algorithm by a simple observation from Elsken et al. (2019a): while some of the objective functions are expensive to evaluate (e.g., evaluating the accuracy is expensive since it requires training the network ﬁrst), others are cheap to evaluate (e.g., the number of parameters).
Thus, rather than relying on a surrogate model for every objective function as in vanilla EHVI, we solely use surrogate models for the expensive objective function and directly evaluate the cheap objectives.
This way, we avoid ﬁtting surrogate models for objectives which are cheap to evaluate anyway.
We refer to Algorithm 4.
3.4 MO-BANANAS
BANANAS (White et al., 2021) uses neural networks as a performance predictor of candidates within BO.
To extend BANANAS for multi-objective optimization, we employ crowd sorting (Raquel and Jr, 2005) in combination with independent Thompson sampling to select parents used to generate new candidates by mutating the parents.
Based on the performance predictor, the next candidates are chosen - again based on independent Thompson sampling, see Algorithm 5.
We furthermore employ successive halving to quickly discard poorly-performing architectures.
3.5 BULK & CUT
BULK & CUT combines a simple evolutionary strategy with BO.
The name BULK & CUT comes from the fact that the algorithm ﬁrst looks for high accuracy models by successively enlarging them with network morphisms (Chen et al., 2016), then shrinking them using pruning techniques in combination with knowledge distillation (Hinton et al., 2015).
BULK & CUT comprises three sequential phases:(i) initialization: sample random architectures, (ii) bulkup: generate oﬀsprings by applying network morphisms and (iii) cut-down: prune models.
After the initialization phase is completed, parents are selected for the bulk-up phase.
For this, we propose the Paretsilon Greedy criterion, which combines non-dominated sorting and an (cid:15) -greedy exploration strategy, as described in Algorithm 6.
Once a parent is chosen, an oﬀspring is generated by applying a network morphism (a commonly used operation in the NAS literature that avoids costly retraining from scratch (Elsken et al., 2019a;
Cai et al., 2018)).
In the cut-down phase, we use structured pruning (Anwar et al., 2017) to shrink models, i.e., eliminating units from fully-connected layers and ﬁlters from convolutional layers, instead of dropping individual weights.
We employ knowledge distillation (Hinton et al., 2015) for training the shrunken models so that they match their parent’s output (Elsken et al., 2019a; Prakosa et al., 2020; Chen et al., 2021).
During all phases, the non-architectural hyperparameters (e.g., learning rate and weight decay) are optimized via constrained BO.
The term constrained here refers to the fact that the optimization of the acquisition function is performed with constrains on the already chosen architectural hyperparameters.
Algorithm 7 in the appendix summarizes how BULK & CUT works.
4. Experiments
We used the Oxford-Flowers dataset (Nilsback and Zisserman, 2006) with images scaled down to a resolution of 16x16 for computational reasons.
Each proposed method, along with random search as the simplest baseline, was run 10 times for 24 hours each.
A maximum budget of 25 epochs for training is deﬁned, although it is up to the methods to decide if they want to train with a smaller budget to speed up the search.
We search over a variable number of convolutional layers (followed by ReLU and max-pooling), with a variable number of ﬁlters and kernel size each as well as optional batch normalization.
A global average pooling may be applied afterwards, followed by a variable number of fully connected layers.
We use Adam (Kingma and Ba, 2015) to optimize neural network weights, with a searchable learning rate and batch size.
We refer to Table 2 for a summary of our search space.
We furthermore evaluated the proposed methods on Cifar-10 on the NAS-Bench-201 (Dong and Yang, 2020) search space (except BULK & CUT since it is not directly applicable to the benchmark).
NAS-Bench-201 does not contain any optimizable hyperparameters, however comes with a more interesting NAS search space and furthermore allows inspecting the true Pareto front due to the tabular nature of the benchmark.
We target network size (by means of number of parameters) and classiﬁcation accuracy as the objectives of our multi-objective optimization.
We split the datasets into training, validation and test splits.
The splits are used for training model parameters, NAS + HPO and ﬁnal evaluation, respectively.
4.1 Results
Visualizing sampled conﬁgurations.
Figure 1 visualizes the sampled conﬁgurations for each method across all 10 random seeds on Flowers.
All methods (except random search) explore the Pareto front, which is what they were designed for.
However, they also signiﬁcantly diﬀer in the exploration strategy.
SH-EMOA explores both objective functions equally well but still samples poor conﬁgurations in later iterations.
MO-BOHB tends to focus on smaller networks in later iterations, while MS-EHVI very quickly discovers networks close to the Pareto front and mostly samples new conﬁgures there.
After an initial phase, BULK & CUT also mostly samples candidates close to the Pareto front in later iterations.
Performance comparison.
Figure 2 (a,b,d,e) shows the hypervolumes over time on
Flowers and NAS-Bench-201, respectively.
For Flowers, all our methods clearly outperform
[Figure]
[Figure] [Figure] [Figure] [Figure]
[Figure] [Figure] [Figure]
[Figure] [Figure] [Figure]
[Figure]
random search.
MS-EHVI converges very fast but is eventually outperformed by BULK &
CUT , which however performs less strongly in the initial phase.
SH-EMOA , MO-BOHB and
MO-BANANAS perform similarly.
On NAS-Bench-201, SH-EMOA and MO-BOHB outperform the other methods initially, but all methods eventually achieve a similar performance.
Figure 2 (c, f) also shows the ﬁnal Pareto fronts when combining the results from all seeds.
For
Flowers, the proposed methods perform similarly for the range of parameters from 10 3 to 10 5 , but some methods have problems with covering smaller or larger models.
When looking at results for each seed (Figure 3 in the appendix), we however also noticed that the results vary across seeds, indicating that initializing might have a high impact on the performance and that the budget of 24 hours might not be suﬃcient for the methods to converge or that they simply get stuck in a local optimum.
On NAS-Bench-201, each method discovers a Pareto front very close to the true Pareto front, suggesting that the chosen budget is suﬃciently large to explore the whole space.
5. Conclusions
We addressed the problem of joint hyperparameter optimization and neural architecture search under multiple objectives by extending existing methods to this scenario.
We propose several methods serving as baselines for future research in this direction.
To facilitate this, all our code is available at https://github.com/automl/multi-obj-baselines .
Appendix A. Related work - extended
[Table] A.1 Non-dominated sorting
NDS extends the ranking of a set of candidates based on a single objective to multiple objectives
f = ( f 1 , . . .
, f n ) in the following way:
• Compute the Pareto front F 1 = pareto front ( P| f ) of the current population P and assign all members of this Pareto front F 1 the best rank.
Remove the previous Pareto front from the population P and compute the Pareto front for the remaining population: F 2 = pareto front ( P\F 1 | f ).
Members of this new Pareto front F 2 are assigned the second best rank.
• Iterate this process until all members of the population have been assigned a rank.
Thus, a run of NDS partitions the population into sets F 1 , . .
, F k , where a candidate λ ∈ F i outperforms another candidate λ (cid:48) ∈ F j with respect to all objectives if i < j .
A.2 Hypervolumne subset selection problem
The hypervolume indicator I H of a population measures, informally speaking, the space of objective function values covered by the population; maximizing the hypervolume indicator corresponds to improving the Pareto front and ﬁnding better solutions.
Based on the hypervolume indicator, the hypervolume subset selection problem (HSSP) (Bader and Zitzler, 2011) is deﬁned as the problem of ﬁnding a subset P HSPP ⊂
P of a certain size k so that the hypervolume is maximized for this subset: P HSPP ∈ arg max P (cid:48)
⊂P , |P (cid:48)
|
=
k
I H
( P (cid:48) ).
The
HSSP can also be solved to identify a poorly performing candidate by setting k = |P| −and choosing the poor candidate λ poor as the one that gets removed from the population via HSSP: { λ poor } =
P \ P HSPP .
We refer to Bader and Zitzler (2011) for a more formal introduction.
Appendix B. Full Details on the Various Baselines B.1 Implementation details on SH-EMOA
• Population size and total number of samples: In Flowers dataset, we use s pop =and n total fe = 15000.
Algorithm 1:
Input : number of function evaluations n total fe , population size s pop , maximum budget b max , number of SH iterations n , objectives f
Output: Pareto front w.r.t.
f 1 Generate initial population P of size s pop 2 b ← (cid:98) b max / 2 n − 1 (cid:99) /
* initial budget */ 3 n fe ← (cid:98) n total fe / (cid:80) n − 1 i =0 2 − i (cid:99) /
* number of FE for the initial budget */ 4 for i = 1 to n do
Evaluate f ( λ ) for all λ ∈ P with budget b for j = 1 to n fe − s pop /* generate candidates for remaining FEs */ do
Generate new candidate λ new /* parent selection and variation */
Evaluate f ( λ new ) on budget b [ F 1 , ..., F k ] ← NDS( P ∪ { (cid:0) λ new , f ( λ new ) (cid:1) } ) λ poor ← HSSP( F k ,
|F k
| − 1)
P ← ( P ∪ { (cid:0) λ new , f ( λ new ) (cid:1) } ) \ { (cid:0) λ poor , f ( λ poor ) (cid:1) } n fe ← n fe //* half number of FE in next budget */ b ← 2 b /* double budget */ 15 return pareto front ( P| f )
• Parent selection: We use tournament selection by randomly sampling k potential parents from the current population (uniform distribution) and choose the parent with highest ﬁtness.
We use k = 3.
• Variation: On each step we choose either mutation or recombination strategy with equal probability.
Mutation is deﬁned as a uniformly distributed random variation of 5 hyperparameters from the parent conﬁguration.
For recombination, we use two parents and choose each hyperparameter from one of them with equal probability.
As we have a conditional search space, relationships between hyperparameters are taken into account when a new individual is created. For example, if a mutation increases the total number of convolutional layers, the size of the kernel for each new layer is also added. B.2 Implementation details on MO-BOHB
Algorithms 2 and 3 show pseudo code for MO-BOHB and its sampling step, respectively.
Note the close resemblance to the original BOHB (diﬀerences marked in red), as MO-BOHB generalizes BOHB to any number of objectives.
Note however, that there are two minor diﬀerences between the current version of our proposed MO-BOHB and the original BOHB implementation: (i) MO-BOHB uses an hierarchy of one-dimensional KDEs, whereas BOHB use a single multi-dimensional KDE, and (ii) we do not multiply bandwidths by a constant factor b w .
In future versions of MO-BOHB , we suspect that using a single multi-dimensional KDE and multiplication of bandwidths may further improve performance by better handling interaction eﬀects between (architectural) hyperparameters and encouraging more exploration around promising conﬁgurations, respectively.
Algorithm 2:
Input : budgets b min and b max , conﬁgurations discarding factor η ∈ N > 0 , and objectives f
Output: Pareto front w.r.t.
f 1 s max ← (cid:98) log η b
max b min (cid:99) 2 P b ← [ ] ∀ b ∈ { η − s · b max | s = s max , s max − 1 , ..., 0 } 3 while not stopping criterion do for s ∈ { s max , s max − 1 , ..., 0 } do sample n = (cid:100)
s max
+1 s +1 η s (cid:101) conﬁgurations λ 1 , ..., λ n using Algorithm run modiﬁed SH on λ 1 , ..., λ n with initial budget η − s · b max add observations { ( λ i , f
( λ i )) } of each budget b to P b 8 return pareto front ( P b max | f )
Algorithm 3:
Input : observations P , fraction of random runs ρ , quantile γ , number of samples n , and minimum number of points
N min to build a model
Output: next conﬁguration to evaluate 1 if rand () <
ρ then return random conﬁguration 3 b ← arg max {P b :
|P b | ≥ N min + 2 } 4 if b = ∅ then return random conﬁguration 6 greedily split P into good P l or bad
P g observations using NDS & HSSP 7 ﬁt KDEs l and g based on P l or P g , respectively 8 draw n samples according to l ( λ ) 9 return sample with highest ratio l ( λ ) g ( λ )
In all our experiments, we set the meta-parameters of MO-BOHB as follows:
For the HB part of MO-BOHB , we set the conﬁguration discarding factor of to η = 3, use minimum budget b min = 5 and maximum budget b max = 25.
In the BO part of MO-BOHB , we use an random fraction ρ = 1 / 6, set the quantile to γ = 0 .
1, sample n = 24 conﬁgurations, and use a minimum of N min = 2 · | HP s | + 1 points before building a model.
B.3 Details on MS-EHVI
Algorithm 4: MS-EHVI
Input : expensive and cheap objectives f
= ( f exp , f cheap ), surrogate model ˆ f exp , number of function evaluations n fe
Output: Pareto front w.r.t.
f 1 Initialize population P with initial observations 2 for k = 1 to n fe do
Fit surrogate model ˆ f exp on P
Select next candidate: λ new ∈ argmax λ EHVI (cid:0) λ
|P , ˆ f exp , f cheap (cid:1)
Evaluate f exp ( λ new )
Update data: P ← P ∪ (cid:110)(cid:0) λ new , f ( λ new ) (cid:1)(cid:111) 7 return pareto front
( P| f )
B.4 Details on MO-BANANAS
We do not use the path-based encoding from White et al.
(2021) since it is not meaningful for our search space.
Rather, we employ a simple real-valued vector representation, which furthermore also directly allows us to include non-architectural hyperparameters.
We employ Gaussian noise for mutating parents: We assume integer-valued hyperparameters (e.g., number of layers), and normalize each value by dividing by the maximum value to map each hyperparameter to the range [0 , 1]: we then add Gaussian noise.
Before each function evaluation, this continuous representation is discretized by choosing the integer-valued hyperparameter which is closed to the mutation value after normalization.
Algorithm
Input
Output: Pareto
1 Generate initial population P 2 for i = 1 to n do train neural predictor ˆ f on P sort P using NDS( P ) and crowdingDistance( P )
P ← front
B.5 Details on BULK & CUT
do
Algorithm
Input : time budgets T 1 < T 2 < T 3 , exploration probability (cid:15) , objectives f
Output: set of Pareto optimal solutions 1 P ← ∅ //population set 2 t ← elapsed time () 3 while t < T 3 do if t ∈ [0 , T 1 ) then λ α ← random architecture () if t ∈ [ T 1 , T 2 ) then λ α ← paretsilon greedy ( P , (cid:15) ) /
* parent selection (see Alg. 6) */ λ α ← network morphism ( λ α ) if t ∈ [ T 2 , T 3 ] then λ α ← paretsilon greedy ( P , (cid:15) ) /
* parent selection (see Alg. 6) */ λ α ← prune and distill knowledge ( λ α ) λ β ← constrained BO ( λ α ) /
* other hyperparameters */ evaluate f ( λ α , λ β )
Update constrained BO with (cid:0) λ α , λ β , f ( λ α , λ β ) (cid:1)
P ← P ∪ { (cid:0) λ α , λ β , f ( λ α , λ β ) (cid:1) } t ← elapsed time () 17 return pareto front ( P )
Appendix C. Experimental details.
We used the Oxford-Flowers dataset (Nilsback and Zisserman, 2006), a small dataset composed of 17 diﬀerent classes with 80 examples each, to show the performance of the proposed approaches in environments where many, cheap function evaluations are available.
All images are scaled down to 16 x 16 for computational reasons.
We split the datasets as follows: we randomly split the data into 60% for training, 20% for validation and 20% for testing.
Neural network weights are always trained on training data, their performance on validation data is used to guide hyperparameter and architecture optimization, and the test set is only used for evaluation.
Appendix D. Supplemental Results
Figures 3 and 4 show all the Pareto fronts found with diﬀerent seeds.
[Table] [Table]
[Figure] [Figure]