Tackling Neural Architecture Search With Quality Diversity Optimization
Abstract Neural architecture search (NAS) has been studied extensively and has grown to become a research field with substantial impact.
While classical single-objective NAS searches for the architecture with the best performance, multi-objective NAS considers multiple objectives that should be optimized simultaneously, e.g., minimizing resource usage along the validation error.
Although considerable progress has been made in the field of multiobjective NAS, we argue that there is some discrepancy between the actual optimization problem of practical interest and the optimization problem that multi-objective NAS tries to solve.
We resolve this discrepancy by formulating the multi-objective NAS problem as a quality diversity optimization (QDO) problem and introduce three quality diversity NAS optimizers (two of them belonging to the group of multifidelity optimizers), which search for high-performing yet diverse architectures that are optimal for application-specific niches, e.g., hardware constraints.
By comparing these optimizers to their multi-objective counterparts, we demonstrate that quality diversity NAS in general outperforms multiobjective NAS with respect to quality of solutions and efficiency.
We further show how applications and future NAS research can thrive on QDO.
1 Introduction
The goal of neural architecture search (NAS) is to automate the manual process of designing optimal neural network architectures.
Traditionally, NAS is formulated as a single-objective optimization problem with the goal of finding an architecture that has minimal validation error [13, 35, 45, 47, 46, 63].
Considerations for additional objectives such as efficiency have led to the formulation of constraint NAS methods that enforce efficiency thresholds [1] as well as multiobjective NAS methods [10, 12, 37, 53, 36] that yield a Pareto optimal set of architectures. However, [Figure] © 2022 the authors, released under CC BY 4.
in most practical applications, we are not interested in the complete Pareto optimal set.
Instead, we would like to obtain solutions for a discrete set of scenarios (e.g., end-user devices), which we henceforth refer to as niches in this paper.
This is illustrated in Figure 1.
A concrete example is finding neural architectures for microcontrollers [32] and other edge devices [38], e.g., in 𝜇 NAS [32] architectures for “mid-tier” IoT devices are searched.
To evaluate the benefits for larger devices, the search would need to be restarted with adapted constraints, thus wasting computational resources.
Formulating the search as a multi-objective problem would also waste resources; once an architecture satisfies the constraints of a device, we are not interested in additional trade-offs, and we select only based on the validation error.
We therefore argue that the multi-objective NAS problem can and usually should be formulated as a quality diversity optimization (QDO) problem, which directly corresponds to the actual optimization problem of interest.
The main contributions of this paper are: We (1) formulate multi-objective NAS as a QDO problem; (2) show how to adapt black-box optimization algorithms for the QDO setting; (3) modify existing QDO algorithms for the NAS setting; (4) propose novel multifidelity QDO algorithms for NAS; and (5) illustrate that our approach can be used to extend a broad range of NAS methods from conventional to Once-for-All methods.
2 Theoretical Background and Related Work
Let A denote a search space of architectures and Λ the search space of additional hyperparameters controlling the training of an architecture 𝐴 .
Furthermore, let 𝑓 err : A × Λ → R denote the validation error obtained after training an architecture 𝐴 ∈
A with a set of hyperparameters 𝜆 ∈ Λ for a given number of epochs ( 𝜆 epoch ∈ 𝜆 ).
Typically, we consider 𝜆 ∈ Λ to be fixed, except for 𝜆 epoch in multifidelity methods, and we therefore omit 𝜆 in the following.
The goal of single-objective NAS is to find the architecture with the lowest validation error, 𝐴 ∗ (cid:66) arg min 𝐴 ∈
A 𝑓 err ( 𝐴 ) .
NAS methods can be categorized along three dimensions: search space, search strategy, and performance estimation strategy [13].
For chain-structured neural networks (simply a connected sequence of layers), cell-based search spaces have gained popularity [46, 45].
In cell-based search spaces, different kinds of cells – typically, a normal cell preserving dimensionality of the input and a reduction cell reducing spatial dimension – are stacked in a predefined arrangement to form a final architecture.
Regarding search strategy, popular methods utilize Bayesian optimization (BO) [3, 8, 39, 24, 57], evolutionary methods [41, 34, 47, 46, 12], reinforcement learning [63, 64], or gradient-based algorithms [35, 45].
For performance estimation, popular approaches leverage lower fidelity estimates [31, 14, 64] or make use of learning curve extrapolation [8, 27].
Multi-Objective Neural Architecture Search Contrary to the single-objective NAS formulation, multi-objective NAS does not solely aim for minimizing the validation error but simultaneously optimizes multiple objectives.
These objectives typically take resource consumption – such as memory requirements, energy usage or latency – into account [10, 12, 37, 53, 36].
Denote by 𝑓 1 , . .
, 𝑓 𝑘 the 𝑘 ≥ 2 objectives of interest, where typically 𝑓 1 = 𝑓 err and denote by f ( 𝐴 ) the vector of objective function values obtained for architecture 𝐴 ∈ A , f ( 𝐴 ) = ( 𝑓 1 ( 𝐴 ) , . . .
, 𝑓 𝑘 ( 𝐴 )) ′ .
The optimization problem of multi-objective NAS is then formulated as min 𝐴 ∈
A f ( 𝐴 ) .
There is no architecture that minimizes all objectives at the same time since these are typically in competition with each other. Rather, there are multiple Pareto optimal architectures reflecting different trade-offs in objectives approximating the true (unknown) Pareto front.
An architecture 𝐴 is said to dominate another architecture 𝐴 ′
iff ∀ 𝑖 ∈ { 1 , . . .
,𝑘 } : 𝑓 𝑖 ( 𝐴 ) ≤ 𝑓 𝑖 ( 𝐴 ′ )
∧ ∃ 𝑗 ∈ { 1 , . . .
,𝑘 } :
𝑓 𝑗 ( 𝐴 ) <
𝑓 𝑗 ( 𝐴 ′ ) .
Constrained and Hardware-Aware Neural Architecture Search
In contrast, Constrained NAS [62, 15, 55] solves the problem of finding an architecture that optimizes one objective (e.g., validation error) with constraints on secondary objectives (e.g., model size).
Constraints can be naturally given by the target hardware that a model should be deployed on.
Hardware-Aware NAS in turn searches for an architecture that trades off primary objectives [60] against secondary, hardware-specific metrics.
In Once-for-All [5], a large supernet is trained which can be efficiently searched for subnets that, e.g., meet latency constraints of target devices.
For a recent survey, we refer to [1].
Quality Diversity Optimization
The goal of a QDO algorithm is to find a set of high-performing, yet behaviorally diverse, solutions.
Similarly to multi-objective optimization, there is no single best solution.
However, whereas multi-objective optimization aims for the simultaneous minimization of multiple objectives, QDO minimizes a single-objective function with respect to diversity defined on one or more feature functions .
A feature function measures a quality of interest and a combination of feature values points to a niche, i.e., a region in feature space .
QDO could be considered a set of constrained optimisation problems over the same input domain where the niche boundaries are constraints in feature space.
The key difference is that constrained optimisation seeks a single optimal configuration given some constraints, while QDO attempts to identify the optimal configuration for each of a set of constrained regions simultaneously.
In this sense, QDO could be framed as a so-called multi-task optimization problem [43] where each task is to find the best solution belonging to a particular niche.
QDO algorithms maintain an archive of niche-optimal observations, i.e., a best-performing observed solution for each niche.
Observations with similar feature values compete to be selected for the archive, and the solution set gradually improves during the optimization process.
Once the optimization budget has been spent, QDO algorithms typically return this archive as their solution.
QDO is motivated by applications where a group of diverse solutions is beneficial, such as the training of robot movement where a repertoire of behaviours must be learned [7], developing game playing agents with diverse strategies [44], and in automatic design where QDO can be used by human designers to search a large dimensional search space for diverse solutions before the optimization is finished by hand.
Work on automatic design tasks have been varied and include air-foil design [18], computer game level design [16], and architectural design [9].
Recently, QDO algorithms were used for illuminating the interpretability and resource usage of machine learning models while minimizing their generalization error [52].
In the earliest examples, Novelty Search (NS; [29]) asks whether diversity alone can produce a good set of solutions.
Despite not actively pursuing objective performance, NS performed surprisingly well in some settings and was followed by Novelty Search with Local Competition [30], the first true quality diversity (QD) algorithm.
MAP-Elites [42], a standard evolutionary QDO algorithm, partitions the feature space a-priori into niches and attempts to identify the optimal solution in each of these niches.
QDO has seen much work in recent years and a variant based on
BO, BOP-Elites, was proposed recently [25].
BOP-Elites models the objective and feature functions with surrogate models and implements an acquisition function over a structured archive to achieve high sample efficiency even in the case of black-box features.
3 Formulating Neural Architecture Search as a Quality Diversity Optimization Problem
In the example in Figure 1, a quality diversity NAS (subsequently abbreviated as qdNAS) problem is given by the validation error and three behavioral niches (corresponding to different devices) that are defined via resource usage measured by a single feature function.
Let 𝑓 1 : A → R , 𝐴 ↦→ 𝑓 1 ( 𝐴 ) denote the objective function of interest (in our context, 𝑓 err ).
Denote by 𝑓 𝑖 : A → R , 𝐴 ↦→ 𝑓 𝑖 ( 𝐴 ) ,𝑖 ∈ { 2 , . . .
,𝑘 } ,𝑘 ≥ 2 the feature function(s) of interest (e.g., memory usage).
Behavioral niches 𝑁
𝑗 ⊆ A , 𝑗 ∈ { 1 , . . . ,𝑐 } ,𝑐 ≥ 1 are sets of architectures characterized via niche-specific boundaries b 𝑖𝑗 = (cid:2) 𝑙 𝑖𝑗 , 𝑢 𝑖𝑗 (cid:1) ⊆
R on the images of the feature functions.
An architecture 𝐴 belongs to niche 𝑁 𝑗 if its values with respect to the feature functions lie between the respective boundaries, i.e.: 𝐴 ∈
𝑁 𝑗 ⇐⇒ ∀ 𝑖 ∈ { 2 , . . .
,𝑘 } : 𝑓 𝑖 ( 𝐴 ) ∈
b 𝑖𝑗 . The goal of a QDO algorithm is then to find for each behavioral niche 𝑁 𝑗 the architecture that minimizes the objective function 𝑓 1 :
[Equation]
In other words, the goal is to obtain a set of architectures S (cid:66) (cid:8) 𝐴 ∗ 1 , . . .
, 𝐴 ∗ 𝑐 (cid:9) that are diverse with respect to the feature functions and yet high-performing.
A Remark about Niches
In the classical QDO literature, niches are typically constructed to be pairwise disjoint, i.e., a configuration can only belong to a single niche (or none) [42, 25].
However, depending on the concrete application, relaxing this constraint and allowing for overlap can be beneficial.
For example, in our context, an architecture that fits on a mid-tier device should also be considered for deployment on a higher-tier device, i.e., in Figure 1, the boundaries indicated by vertical dashed lines resemble the respective upper bound of a niche whereas the lower bound is unconstrained.
This results in niches being nested within each other, i.e., 𝑁 1 ⊊ 𝑁 2 ⊊ . . .
⊊ 𝑁 𝑐 ⊆ A , with 𝑁 1 being the most restrictive niche, followed by 𝑁 2 .
In Supplement A, we further discuss different ways of constructing niches in the context of NAS.
3.1 Quality Diversity Optimizers for Neural Architecture Search
As the majority of NAS optimizers are iterative, we first demonstrate how any iterative optimizer can in principle be turned into a QD optimizer.
Based on this correspondence, we introduce three novel QD optimizers for NAS: BOP-Elites*, qdHB and BOP-ElitesHB.
Let 𝑓 1 : A → R , 𝐴 ↦→ 𝑓 1 ( 𝑥 ) denote the objective function that should be minimized.
In each iteration, an iterative optimizer proposes a new configuration (e.g., architecture) for evaluation, evaluates this configuration, potentially updates the incumbent (best configuration evaluated so far) if better performance has been observed, and updates its archive.
For generic pseudo code, see Supplement B.
Moving to a QDO problem, there are now feature functions 𝑓 𝑖 :
A → R , 𝐴 ↦→ 𝑓 𝑖 ( 𝐴 ) , 𝑖 ∈ { 2 , . . .
,𝑘 } ,𝑘 ≥ 2, and niches 𝑁 𝑗 , 𝑗 ∈ { 1 , . .
. ,𝑐 } ,𝑐 ≥ 1, defined via their niche-specific boundaries b 𝑖𝑗 = (cid:2) 𝑙 𝑖𝑗 ,𝑢 𝑖𝑗 (cid:1) ⊆
R on the images of the feature functions.
Any iterative single-objective optimizer must then keep track of the best incumbent per niche (often referred to as an elite in the QDO literature) and essentially becomes a QD optimizer (see Algorithm 1).
The challenge
Algorithm 1: Input : 𝑓 1 , 𝑓 𝑖 ,𝑖 ∈ { 2 , . . .
,𝑘 } ,𝑘 ≥ 2, 𝑁 𝑗 , 𝑗 ∈ { 1 , . .
. ,𝑐 } ,𝑐 ≥ 1, D design , 𝑛 total
Result: 𝑆 = { 𝐴 ∗ 1 , . . .
, 𝐴 ∗ 𝑐 } 1 D ← D design 2 for 𝑗 ← 1 to 𝑐 do 𝐴 ∗ 𝑗 ← arg min 𝐴 ∈ D | 𝑁𝑗 𝑓 1 ( 𝐴 )
# initial incumbent of niche 𝑁 𝑗 based on archive 4 end 5 for 𝑛 ← 1 to 𝑛 total do
Propose a new candidate 𝐴 ★ # subroutine
Evaluate 𝑦 ← 𝑓 1 ( 𝐴 ★ ) , ∀ 𝑖 ∈ { 2 , . . .
,𝑘 } : 𝑧 𝑖 ← 𝑓 𝑖 ( 𝐴 ★ ) if 𝐴 ★ ∈ 𝑁 𝑗 ∧ 𝑦 < 𝑓 1 ( 𝐴 ∗ 𝑗 ) then 𝐴 ∗ 𝑗 ← 𝐴 ★ # update incumbent of niche 𝑁 𝑗 end
D ← D ∪ (cid:8)(cid:0)
𝐴 ★ ,𝑦, 𝑧 2 , . . .
, 𝑧 𝑘 (cid:1)(cid:9) 12 end in designing an efficient and well-performing QD optimizer now mostly lies in proposing a new candidate for evaluation that considers improvement over all niches.
Bayesian Optimization A recently proposed model-based QD optimizer, BOP-Elites [25], extends BO [54, 17] to QDO.
BOP-Elites relies on Gaussian process surrogate models for the objective function and all feature functions.
New candidates for evaluation are selected by a novel acquisition function – the expected joint improvement of elites (EJIE), which measures the expected improvement to the ensemble problem of identifying the best solution in every niche:
[Equation]
Here, P ( 𝐴 ∈ 𝑁
𝑗 | D ) is the posterior probability of 𝐴 belonging to niche 𝑁 𝑗 , and E 𝑦 (cid:2) I | 𝑁 𝑗 ( 𝐴 ) (cid:3) is the expected improvement (EI; [23]) with respect to niche 𝑁 𝑗 :
[Equation]
where 𝑓 min 𝑁𝑗 is the best observed objective function value in niche 𝑁 𝑗 so far, and 𝑦 is the surrogate model prediction for 𝐴 .
A new candidate is then proposed by maximizing the EJIE, i.e., Line 6 in Algorithm 1 looks like the following: 𝐴 ★ ← arg max 𝐴 ∈ A 𝛼 EJIE ( 𝐴 ) .
BOP-Elites*
In order to adapt BOP-Elites for NAS, we introduce several modifications.
First, we employ truncated (one-hot) path encoding [56, 57]. In path encoding, architectures are transformed into a set of binary features indicating presence for each path of the directed acyclic graph from the input to the output.
By then truncating the least-likely paths, the encoding scales linearly in the size of the cell [57] allowing for an efficient representation of architectures.
Second, we substitute the Gaussian process surrogate models used in BOP-Elites with random forests [4] allowing us to model non-continuous hyperparameter spaces.
Random forests have been successfully used as surrogates in BO [21, 33], often performing on a par with ensembles of neural networks [57] in the context of NAS [51, 59].
Third, we introduce a local mutation scheme similarly to the one used by the BANANAS algorithm [57] for optimizing the infill criterion EJIE:
Since our aim is to find high quality solutions across all niches, we maintain an archive of the incumbent architecture in each niche and perform local mutations on each incumbent.
We refer to our adjusted version as BOP-Elites* in the remainder of the paper to emphasize the difference from the original algorithm.
For the initial design, we sample architectures based on adjacency matrix encoding [56].
Multifidelity Optimizers For NAS, performance estimation is the computationally most expensive component [13], and almost all NAS optimizers can be made more efficient by allowing access to cheaper, lower fidelity estimates [13, 31, 14, 64].
By evaluating most architectures at lower fidelity and only promoting promising architectures to higher fidelity, many more architectures can be explored given the same total computational budget.
The fidelity parameter is typically the number of epochs over which an architecture is trained.
qdHB One of the most prominent multifidelity optimizers is Hyperband (HB; [31]), a multiarmed bandit strategy that uses repeated Successive Halving (SH; [22]) as a subroutine to identify the best configuration (e.g., architecture) among a set of randomly sampled ones.
Given an initial and maximum fidelity, a scaling parameter 𝜂 , and a set of configurations of size 𝑛 , SH evaluates all configurations on the initial smallest fidelity, then sorts the configurations by performance and only keeps the best 1 / 𝜂 configurations.
These configurations are then trained with fidelity increased by a factor of 𝜂 .
This process is repeated until the maximum fidelity for a single configuration is reached.
HB repeatedly runs SH with different sized sets of initial configurations called brackets.
Only two inputs are required: 𝑅 , the maximum fidelity and 𝜂 , the scaling parameter that controls the proportion of configurations discarded in each round of SH.
Based on these inputs, the number 𝑠 max and size 𝑛 𝑖 of different brackets is determined.
To adapt HB to the QD setting, we must track the incumbent architecture in each niche and promote configurations based on their performance within the respective niche (see Supplement B): To achieve this, we choose the top
⌊ 𝑛 𝑖 / 𝜂 ⌋ configurations to be promoted uniformly over the 𝑐 niches (done in the top k_qdo function), i.e., we iteratively select one of the niches uniformly at random and choose the best configuration observed so far that has yet not been selected for promotion until ⌊ 𝑛 𝑖 / 𝜂 ⌋ configurations have been selected.
Note that during this procedure, it may happen that not enough configurations belonging to a specific niche have been observed yet.
In this case, we choose any configuration uniformly at random over the set of all configurations that have yet to be promoted.
With those modifications, we propose qdHB, as a multifidelity QD optimizer.
BOP-ElitesHB While HB typically shows strong anytime performance [31], it only samples configurations at random and is typically outperformed by BO methods with respect to final performance if optimizer runtime is sufficiently large [14].
BOHB [14] combines the strengths of HB and BO in a single optimizer, resulting in strong anytime performance and fast convergence.
This approach employs a fidelity schedule similar to HB to determine how many configurations to evaluate at which fidelity but replaces the random selection of configurations in each HB iteration by a model-based proposal.
In BOHB, a Tree Parzen Estimator [2] is used to model densities 𝑙 ( 𝐴 ) =
𝑝 ( 𝑦 < 𝛼 | 𝐴, D ) and 𝑔 ( 𝐴 ) =
𝑝 ( 𝑦 > 𝛼 | 𝐴, D ) , and candidates are proposed that maximize the ratio 𝑙 ( 𝐴 )/ 𝑔 ( 𝐴 ) , which is equivalent to maximizing EI [2].
Based on BOP-Elites* and qdHB, we can now derive the QD
Bayesian optimization Hyperband optimizer (BOP-ElitesHB): Instead of selecting configurations at random at the beginning of each qdHB iteration, we propose candidates that maximize the EJIE criterion.
This sampling procedure is described in Supplement B.
4 Main Benchmark Experiments and Results
We are interested in answering the following research questions: ( RQ1 ) Does qdNAS outperform multi-objective NAS if the optimization goal is to find high-performing architectures in pre-defined niches?
( RQ2 ) Do multifidelity qdNAS optimizers improve over full-fidelity qdNAS optimizers?
To answer these questions, we benchmark our three qdNAS optimizers – BOP-Elites*, qdHB, and
BOP-ElitesHB– on the well-known NAS-Bench-101 [61] and NAS-Bench-201 [11] and compare them to three multi-objective optimizers adapted for NAS: ParEGO*, moHB*, and ParEGOHB as well as a simple Random Search 1 .
Experimental Setup It is important to compare optimizers using analogous implementation details.
We therefore use truncated path encoding and random forest surrogates throughout our experiments for all model-based optimizers.
Furthermore, we use local mutations as described in [57] in order to optimize acquisition functions in BOP-Elites*, BOP-ElitesHB, ParEGO*, and ParEGOHB.
To control for differences in implementation, we re-implement all optimizers and take great care in matching the original implementations.
We provide full details regarding implementation in Supplement B and only briefly introduce conceptual differences: ParEGO* is a multi-objective optimizer based on ParEGO [28] and only deviates from BOP-Elites* in that it considers a differently scalarized objective in each iteration, which is optimized using local mutations similar to the acquisition function optimization of BOPElites*.
moHB* is an extension of HB to the multi-objective setting (promoting configurations based on non-dominated sorting with hypervolume contribution for tie breaking, for similar approaches see, e.g., [48, 49, 50, 19]).
ParEGOHB is a model-based extension of moHB* that relies on the ParEGO scalarization [28] and on the same acquisition function optimization as ParEGO*.
All optimizers were evaluated on NAS-Bench-101 (Cifar-10, validation error as the first objective and the number of trainable parameters as the feature function/second objective) and NAS-Bench201 (Cifar-10, Cifar-100, ImageNet16-120, validation error as the first objective and latency as the feature function/second objective).
For multifidelity, we train architectures for 4 , 12 , 36 , 108 epochs on NAS-Bench-101 and for 2 , 7 , 22 , 67 , 200 epochs on NAS-Bench-201 (reflecting 𝜂 = 3 in the HB variants).
As the optimization budget, we consider 200 full architecture evaluations (resulting in a total budget of 21600 epochs for NAS-Bench-101 and 40000 epochs for NAS-Bench-201).
For each of these four settings, we construct three different scenarios by considering different niches of interest with respect to the feature function, resulting in a total of 12 benchmark problems.
In the small/medium/large settings, two, five and ten niches are considered, respectively.
Niches are constructed to be overlapping, and boundaries are defined based on percentiles of the feature function.
For the small setting, the boundary is given by the 50% percentile ( 𝑞 50% ), effectively resulting in two niches with boundaries [ 0 ,𝑞 50% ) and [ 0 , ∞) .
For the medium and large settings, percentiles indicating progressively larger niches were used, ranging from: 1% to 30% and 70% respectively.
More details on the niches can be found in Supplement C. All runs were replicated 100 times.
Results As an anytime performance measure, we are interested in the validation error obtained for each niche, which we aggregate in a single performance measure as (cid:205) 𝑐𝑗 = 1 𝑓 err
( 𝐴 ∗ 𝑗 ) , i.e., we consider the sum of validation errors over the best-performing architecture per niche.
If a niche is empty, we assign a validation error of 100 as a penalty (this is common practice in QDO, i.e., if no solution has been found for a niche, this niche is assigned the worst possible objective function value [25]).
For the final performance, we also consider the analogous test error.
Figureshows the anytime performance of optimizers.
We observe that model-based optimizers (BOPElites* and ParEGO*) in general strongly outperform Random Search, and BO HB optimizers (BOP-ElitesHB and ParEGOHB) generally outperform their full-fidelity counterparts, although this effect diminishes with increasing optimization budget.
In general, HB variants that do not rely on a surrogate model (qdHB and moHB*) show poor performance compared to the model-based optimizers.
Moreover, especially in the small number of niches setting, QD strongly outperforms multi-objective optimization.
Mean ranks of optimizers with respect to final validation and test performance are given in Table 1.
For completeness, we also report critical differences plots of these ranks in Supplement C.
We also conducted two four-way ANOVAs on the average performance after half and all of the optimization budget is used, with the factors problem (benchmark problem), multifidelity (whether an optimizer uses multifidelity), QDO (whether an optimizer is a QD optimizer) and model-based (whether the optimizer relies on a surrogate model) 2 .
For half the budget used, results indicate significant main effects of the factors multifidelity ( 𝐹 ( 1 ) = 19 .
13 , 𝑝 = 0 . 0001)
, QDO ( 𝐹 ( 1 ) = 11 .
08 , 𝑝 = 0 . 0017)
and model-based ( 𝐹 ( 1 ) = 21 . 13 , 𝑝 < 0 . 0001).
For all of the budget used, the significance of multifidelity diminishes, whereas the main effects of QDO ( 𝐹 ( 1 ) = 18 . 31 , 𝑝 = 0 . 0001) and model-based ( 𝐹 ( 43 . 44 ) , 𝑝 < 0 . 0001 ) are still significant.
We can conclude that QDO in general outperforms competitors when the goal is to find high-performing architectures in pre-defined niches. Multi-fidelity optimizers improve over full-fidelity optimizers but this effect diminishes with increasing budget.
Detailed results are reported in Supplement C.
Regarding efficiency, we analyzed the expected running time (ERT) of the QD optimizers given the average performance of the respective multi-objective optimizers after half of the optimization budget: For each benchmark problem, we computed the mean validation performance of each multi-objective optimizer after having spent half of its optimization budget and investigated the ERT of the analogous 3 QD optimizer.
For each benchmark problem, we then computed the ratio of ERTs between multi-objective and QD optimizers and averaged them over the benchmark problems.
For BOP-ElitesHB, we observe an average ERT ratio of 2 .
41, i.e., in expectation, BOP-ElitesHB is a factor of 2 .
41 faster than ParEGOHB in reaching the average performance of ParEGOHB (after half the optimization budget).
For qdHB and BOP-Elites*, the average ERT ratios are 1 .
14 and 1 . 44.
We conclude that all QD optimizers are more efficient than their multi-objective counterparts.
More details can be found in Supplement C. [Figure] [Figure] [Table]
[Figure]
[Table] [Figure] [Table] 5 Additional Experiments and Applications
In this section, we illustrate how qdNAS can be used beyond the scenarios investigated so far and present results of additional experiments ranging from a comparison of qdNAS to multi-objective NAS on the MobileNetV3 search space to an example on how to incorporate QDO in existing frameworks such as Once-for-All [5] or how to use qdNAS for model compression. Benchmarks on the MobileNetV3 Search Space We further investigated how qdNAS compares to multi-objective NAS on a search space that is frequently used in practice [20]. We consider
CNNs divided into a sequence of units with feature map size gradually being reduced and channel numbers being increased. Each unit consists of a sequence of layers where only the first layer has stride 2 if the feature map size decreases and all other layers in the units have stride 1. Units can use an arbitrary number of layers (elastic depth chosen from { 2 , 3 , 4 } ) and for each layer, an arbitrary number of channels (elastic width chosen from { 3 , 4 , 6 } ) and kernel sizes (elastic kernel size chosen from { 3 , 5 , 7 } ) can be used. Additionally, the input image size can be varied (elastic resolution ranging from 128 to 224 with a stride 4).
For more details on the search space, see [5]. To allow for reasonable runtimes we use accuracy predictors (based on architectures trained and evaluated on ImageNet as described in [5]) and resource usage look-up tables of the Once-for-All module [5, 6] and construct a surrogate benchmark. As an objective function we select the validation error and as a feature function/second objective the latency (in ms) when deployed on a Samsung Note(batch size of 1), or the number of FLOPS (M) used by the model.
So far, we have only investigated qdNAS in the context of 𝑘 = 2, i.e., considering one objective and one feature function.
Here, we additionally consider a setting of 𝑘 = 3, by incorporating both latency and the size of the model (in
MB) as feature functions/second and third objective.
We compare BOP-Elites* to ParEGO* and a
Random Search due to the accuracy predictors not supporting evaluations at multiple fidelities.
We again construct three scenarios by considering different niches of interest with respect to the feature functions taking inspiration from latency and FLOPS constraints as used in [5] (details are given in Table 4 in Supplement C).
Optimizers are given a total budget of 100 architecture evaluations.
Figure 3 shows the anytime performance of optimizers with respect to the validation error summed over niches (averaged over 100 replications).
BOP-Elites* strongly outperforms the competitors on all benchmark problems.
More details are provided in Supplement D
[Figure]
Making Once-for-All Even More Efficient In Once-for-All [5], an already trained supernet is searched via regularized evolution [46] for a well performing subnet that meets hardware requirements of a target device relying on an accuracy predictor and resource usage look-up tables. This is sensible if only a single solution is required, however, if various subnets meeting different constraints on the same device are desired, repeated regularized evolution is not efficient. Moreover, look-up tables do not generalize to new target devices in which case using as few as possible architecture evaluations suddenly becomes relevant again.
We notice that the search for multiple architectures within Once-for-All can again be formulated as a QDO problem and therefore compare MAP-Elites [42] to regularized evolution when performing a joint search for architectures meeting different latency constraints on a Samsung Note 10. Results are given in Table 2 with MAP-Elites consistently outperforming regularized evolution, making this novel variant of Once-for-All even more efficient. More details are provided in Supplement E.
[Table] Applying qdNAS to Model Compression We are interested in deploying a MobileNetV2 across different devices that are mainly constrained by memory. For each device, we can therefore only consider models up to a fixed amount of parameters, similarly as depicted in Figure 1.
Given that we have a pretrained model that achieves high performance, we want to compress this model exploiting redundancies in model parameters.
In our application, we use the Stanford Dogs dataset [26] and rely on the neural network intelligence (NNI; [40]) toolkit for model compression.
Pruning consists of several (iterative) steps as well as re-training of the pruned architectures.
Choices for the pruner itself, pruner hyperparameters, and hyperparameters controlling retraining are available and must be carefully selected to obtain optimal models (see Supplement F).
We consider the number of model parameters as a proxy measure for memory requirement, yielding three overlapping niches for different devices.
The pre-trained MobileNetV2 achieves a validation error of 20 .
25 using around 2 . 34 million model parameters.
We define niches with boundaries corresponding to compression rates (number of parameters after pruning) of 40% to 50%, 40% to 60%, and 40% to 70%.
As the
QD optimizer, we use BOP-ElitesHB and specify the number of fine-tuning epochs a pruner can use as a fidelity parameter, since fine-tuning after pruning is costly but also strongly influences final performance.
After evaluating only 69 configurations (a single BOP-ElitesHB run with 𝜂 = 3), we obtain high-performing pruner configurations for each niche, resulting in the performance vs. memory requirement (number of parameters) trade-offs shown in Table 3.
[Table]
6 Conclusion
We demonstrated how multi-objective NAS can be formulated as a QDO problem that, contrary to multi-objective NAS, directly corresponds to the actual optimization problem of interest, i.e., finding high-performing architectures in different niches.
We have shown how any iterative black box optimization algorithm can be adapted to the QDO setting and proposed three QDO algorithms for NAS, with two of which making use of multifidelity evaluations.
In benchmark experiments, we have shown that qdNAS outperforms multi-objective NAS while simultaneously being more efficient.
We furthermore illustrated how qdNAS can be used for model compression and how future NAS research can thrive on QDO.
QDO is orthogonal to the NAS strategy of an algorithm and can be similarly used to extend, e.g., one-shot NAS methods.
Limitations
The framework we describe relies on pre-defined niches, e.g., memory requirements of different devices.
If niches are mis-specified or cannot be specified a priori, multi-objective
NAS may outperform qdNAS.
However, an initial study (see Supplement H) how qdNAS performs in a true multi-objective setting, which would correspond to unknown niches, shows little to no performance degradation depending on the choice of niches.
Moreover, we only investigated the performance of qdNAS in the deterministic setting.
Additionally, our multifidelity optimizers require niche membership to be unaffected by the multifidelity parameter.
Finally, we mainly focused on model-based NAS algorithms that we have extended to the QDO setting.
Broader Impact Our work extends previous research on NAS and therefore inherits its implications on society and individuals such as potential discrimination in resulting models.
Moreover, evaluating a large number of architectures is computationally costly and can introduce serious environmental issues.
We have shown that qdNAS allows for finding better solutions, while simultaneously being more efficient than multi-objective NAS.
As performance estimation is extremely costly in NAS, we believe that this is an important contribution towards reducing resource usage and the CO 2 footprint of NAS.
7 Reproducibility Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Section 6.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Section 6.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results, including all requirements (e.g., requirements.txt with explicit version), an instructive README with installation, and execution commands (either in the supplemental material or as a url)?
[Yes] The full code for experiments, application, figures and tables can be obtained from the following GitHub repository: https://github.com/slds-lmu/qdo_nas .
(b) Did you include the raw results of running the given instructions on the given code and data?
[Yes] Raw results are provided via the same GitHub repository.
(c) Did you include scripts and commands that can be used to generate the figures and tables in your paper based on the raw results of the code, data, and instructions given?
[Yes] Scripts to generate figures and tables based on raw results are provided via the same GitHub repository.
(d) Did you ensure sufficient code quality such that your code can be safely executed and the code is properly documented?
[Yes] (e) Did you specify all the training details (e.g., data splits, pre-processing, search spaces, fixed hyperparameter settings, and how they were chosen)?
[Yes] For our benchmark experiments we used NAS-Bench-101 and NAS-Bench-201.
Regarding the Additional Experiments and Applications Section, all details are reported in Supplement D, Supplement E and Supplement F. (f) Did you ensure that you compared different methods (including your own) exactly on the same benchmarks, including the same datasets, search space, code for training and hyperparameters for that code?
[Yes] As described in Section 4.
(g) Did you run ablation studies to assess the impact of different components of your approach?
[Yes] See Supplement C, Supplement G and Supplement H. (h) Did you use the same evaluation protocol for the methods being compared?
[Yes] As described in Section 4.
(i) Did you compare performance over time?
[Yes] Anytime performance was assessed with respect to the number of epochs as described in Section 4 or the number of architecture evaluations as described in Section 5.
(j) Did you perform multiple runs of your experiments and report random seeds?
[Yes] All runs of main benchmark experiments were replicated 100 times.
Random seeds can be obtained via https://github.com/slds-lmu/qdo_nas .
(k) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] All results include error bars accompanying mean estimates.
(l) Did you use tabular or surrogate benchmarks for in-depth evaluations?
[Yes] We used the tabular NAS-Bench-101 and NAS-Bench-201 benchmarks.
(m) Did you include the total amount of compute and the type of resources used (e.g., type of gpus, internal cluster, or cloud provider)?
[Yes] As described in Supplement I. (n) Did you report how you tuned hyperparameters, and what time and resources this required (if they were not automatically tuned by your AutoML method, e.g. in a nas approach; and also hyperparameters of your own method)?
[N/A] No tuning of hyperparameters was performed.
If
(e.
... (a) If your work uses existing assets, did you cite the creators?
[Yes] NAS-Bench-101, NASBench-201, Naszilla, the Once-for-All module, NNI, and the Stanford Dogs dataset are cited appropriately.
(b) Did you mention the license of the assets?
[Yes] Done in Supplement I. (c) Did you include any new assets either in the supplemental material or as a url?
[Yes] We provide all our code via https://github.com/slds-lmu/qdo_nas .
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] All assets used are either released under the Apache-2.0 License or MIT License.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (irb) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Acknowledgements. The authors of this work take full responsibilities for its content.
Lennart Schneider is supported by the Bavarian Ministry of Economic Affairs, Regional Development and Energy through the Center for Analytics - Data - Applications (ADACenter) within the framework of BAYERN DIGITAL II (20-3410-2-9-8). This work was supported by the German Federal Ministry of Education and Research (BMBF) under Grant No. 01IS18036A.
Paul Kent acknowledges support from EPSRC under grant EP/L015374. This work has been carried out by making use of AI infrastructure hosted and operated by the Leibniz-Rechenzentrum (LRZ) der Bayerischen Akademie der Wissenschaften and funded by the German Federal Ministry of Education and Research (BMBF) under Grant No. 01IS18036A. The authors gratefully acknowledge the computational and data resources provided by the Leibniz Supercomputing Centre ( www.lrz.de ).