C OMP OFA: C OMPOUND O NCE -F OR -A LL N ETWORKS FOR F ASTER M ULTI -P LATFORM D EPLOYMENT
Anonymous authors
Paper under double-blind review
A BSTRACT
The emergence of CNNs in mainstream deployment has necessitated methods to design and train efﬁcient architectures tailored to maximize the accuracy under diverse hardware & latency constraints.
To scale these resource-intensive tasks with an increasing number of deployment targets, Once-For-All (OFA) proposed an approach to jointly train several models at once with a constant training cost.
However, this cost remains as high as 40-50 GPU days and also suffers from a combinatorial explosion of potentially sub-optimal model conﬁgurations.
We ﬁnd that the cost of this one-shot training is dependent on the size of the model design space, and hence seek to speed up the training by constraining the design space to conﬁgurations with better accuracy-latency trade-offs.
We incorporate the insights of compound relationships between model dimensions to build CompOFA , a design space smaller by several orders of magnitude.
Through experiments on ImageNet, we demonstrate that even with simple heuristics we can achieve a 2x reduction in training time 1 and 216x speedup in model search/extraction time compared to the state of the art, without loss of Pareto optimality!
We also show that this smaller design space is dense enough to support equally accurate models for similar diversity of hardware and latency targets, while also reducing the complexity of the training and subsequent extraction algorithms.
Source code is at https: //github.com/compofa-blind-review/compofa-iclr
I NTRODUCTION
CNNs are emerging in mainstream deployment across diverse hardware platforms, latency requirements, and/or workload characteristics.
The available processing power, memory, and latency requirements may vary vastly across deployment platforms – say, from server-grade GPUs to lowpower embedded devices, cycles of high or low workload, etc.
Hence, it becomes vital to design or search models tailored to the deployment scenario, maximizing accuracy constrained by the desired model inference latency.
Building such architectures (either manually or by searching) and then training them are resource-intensive tasks – requiring massive computational resources, expertise of both ML and underlying systems, time, dollar cost, and CO 2 emissions every time they are performed.
Repeating such intensive processes for each deployment target is prohibitively expensive w.r.t.
multiple metrics of cost and doesn’t scale.
Once-For-All (OFA) (Cai et al., 2020) proposed to address this challenge by decoupling the search and training phases through a novel progressive shrinking algorithm.
In this approach, OFA trains a family of models in a single-shot by sharing weights between a family of 10 19 sub-networks of varying depth, width, kernel size, and image resolution.
Subsequently, it carries out a Neural
Architecture Search (NAS) to extract a specialized sub-network for a speciﬁc deployment target – a task performed independently from the former training phase.
This massive search space leads to a training cost that remains prohibitively expensive.
Though this cost can be amortized over a number of deployment targets, it is still signiﬁcant (e.g., reaching 1200 GPU hours for OFA).
This exhaustive approach of training all model conﬁgurations leaves out opportunities for any accuracy or latency guided exploration.
Thus, it suffers a clear inefﬁciency in training a large number of models of a sub-optimal accuracy-latency trade-off, lying below the
Pareto frontier.
These extraneous models not only go un-utilized but also add interference, which necessitates phased training to stabilize their simultaneous optimization.
Finally, searching & extracting an optimal model from this space can only be done via indirect estimations of latency and accuracy lookup tables.
On the other hand, common practices as well as empirical studies (Tan & Le, 2019; Radosavovic et al., 2020) have shown that model dimensions such as depth, width, and resolution are not orthogonal – models that follow some compound couplings between these dimensions produce a better accuracy-latency trade-off than those with unconstrained settings.
In this work, we show that the same or better accuracy-latency trade-offs can be achieved by model families smaller by several orders of magnitude.
We reduce the search space (and, correspondingly, the train time) from O (10 19 ) models to 243 , by leveraging two key insights: (1) architecture dimensions (such as depth and width) can be coupled, eliminating models which are suboptimal in the accuracy-latency tradeoff space; (2) 1ms density granularity on the accuracy/latency Pareto frontier is sufﬁcient for practical systems deployments.
We show that this size reduction allows for simpler one-shot training at merely half the cost in GPU hours, dollars, and CO 2 emissions, while also speeding up model extraction by up to 216x.
This new reduced search space, named CompOFA , is further shown to be still dense enough to produce models covering the same range and density of latency targets for hardware platforms as diverse as OFA, eliminating orders of magnitude of suboptimal model conﬁgurations from training and NAS search.
The generality of CompOFA is ﬁnally validated on another base architecture (Section 5.6), achieving similar gains.
R ELATED W ORK
Efﬁcient neural network design has been an active area of research due to the high computational complexity of CNNs.
NAS is increasingly used to guide or replace previously manual design processes.
Early NAS techniques (Zoph et al., 2018; Zoph & Le, 2016) sampled several architectures and trained them from scratch every time, making them extremely compute-hungry.
The technique of weight-sharing has emerged as one way to address these inefﬁciencies (Berman et al., 2020;
Bender et al., 2018;
Brock et al., 2017; Guo et al., 2019; Liu et al., 2018; Pham et al., 2018).
These methods slice candidate sub
-networks from a larger super-network , thereby sharing weights between them.
Simultaneously, latency-guided NAS methods (Tan et al., 2019;
Cai et al., 2018; Berman et al., 2020; Stamoulis et al., 2019; Wu et al., 2019) have sought to incorporate model complexity into their search to ﬁnd efﬁcient models for a given target latency on a given hardware target.
Nevertheless, these methods yield a single model per run – both search and training must be repeated for new deployment targets.
With compute costs reaching as high as O (10 4 ) GPU hours, this linear scaling is infeasible for an ever-growing need for multi-platform, multi-latency deployment.
OnceFor-All (OFA) (Cai et al., 2020) proposed to reduce this cost by using weight-sharing for a large model family that collectively supports a diverse range of latencies.
They perform one-time training of O (10 19 ) sub-networks, which can then be independently searched to support a given deployment target later, thus amortizing the training cost.
However, this cost is still prohibitively expensive, reaching 1200 GPU hours.
The unnecessarily large search space complicates both the training and the searching, stemming from an uninhibited combinatorial explosion of model conﬁgurations.
On the other hand, empirical studies on neural network design spaces (Tan & Le, 2019; Radosavovic et al., 2020) have recently shown that model dimensions (e.g., depth, width, resolution) are not independent – underlying relations between them can be used to obtain an optimal accuracy-latency trade-off.
In other words the number of degrees of freedom used to induce the architecture search space can be reduced without loss of Pareto optimality.
This insight forms the basis of our work to constrain the design space of OFA networks while maintaining the same quality
(w.r.t. accuracy) as well as diversity of models with reduced train and search costs.
Dynamic Neural Networks with weight-sharing across models of varying latencies have been explored before (Yu et al., 2018; Yu & Huang, 2019) but with much fewer models (e.g. 4 in Yu et al.
and few dimensions (e.g. only width) which make for much sparser and narrower support for diverse latency targets.
We explore a middle ground between these works and Cai et al.
(2020) to build design spaces that are tractable yet sufﬁciently diverse to support many latency targets and varying in multiple model dimensions.
Yu et al.
(2020) proposed replacing OFA’s multi-stage training by a single-stage one.
However, its training cost remains high at over 2300 TPU hours for O (10 12 ) models.
We propose a similar reduction of OFA’s multi-stage training but halve the training cost in this process.
While Yu et al.
(2020) challenges the conventional training practices of weight-shared NAS, our work emphasizes on insights from Tan & Le (2019) and Radosavovic et al.
(2020) to reﬁne the network design space to good models and show that an intractably large cardinality is not necessary.
M OTIVATION 3. D ESIGN S PACE P ARAMETRIZATION
Consider a network architecture N composed of m micro-architectural blocks, B 1 , B 2 , . . .
, B m .
Each block is parametrized by its depth, per-layer width, and per-layer kernel size as B i ( d i , W i , K i ) .
Here d i denotes the number of layers (depth) in the block, and W i and K i are lists denoting the width & kernel sizes of each of these d i layers.
Once-For-All(Cai et al., 2020) builds a family of networks N 1 , N 2 , . .
of varying accuracies and latencies.
The weights of common layers and channels of a block B i are shared across all the networks.
The “block” used in OFA is the Inverted Residual from MobileNetV3 (Howard et al., 2019) and hence “width” here refers to the channel expansion ratio.
d i , w ij , k ij are sampled independently from sets D = [2 , 3 , 4] , W = [3 , 4 , 6] , K = [3 , 5 , 7] respectively.
In OFA, each of these dimensions is treated as orthogonal.
Hence, the resulting number of possible networks is enormous, with O (10 19 ) models for m = 5 blocks(Cai et al., 2020).
3. C OMPOUND R ELATION OF M ODEL D IMENSIONS
The combinatorial explosion from just 3 independent model dimensions of D, W, K yields a model family with an enormous number of models.
While the aim for jointly training “every possible model” in this large design space is to support a diverse range of hardware platforms, we note the following concerns that arise from this:
Model dimensions are not orthogonal Scaling
model dimensions like depth, width, and resolution to higher FLOP regimes is a common
practice
and Tan & Le (2019) systematically
showed
that compound
relations
ex
- ist between these dimensions for achieving
an optimal accuracy-latency trade-off.
In partic ular,
increasing
model
dimensions
in accordance with a compound scaling rule
gives
better results than doing so independently.
Radosavovic et al.
(2020) elevated this
concept
to
the model population level
and found that a quantized
linear relation
between model depth
and
width
yielded design spaces with
a
better
concentration
of well
-performing
models
These works solidiﬁed the common practice of
jointly increasing model depth and width
Yet
, all these other sub-optimal models are still included in the search space when all dimensions are sampled independently.
Large model families complicate training & searching
The interference between OFA’s sub-networks complicates their simultaneous optimization and necessitates techniques like progressive shrinking (Cai et al., 2020), increasing training time.
Further, extracting models for a desired target cannot rely on simple enumeration and instead needs to rely on predicting model accuracy/latency.
This requires additional training for these accuracy predictors speciﬁc to the trained model family and collecting latency look-up tables (Cai et al., 2018).
With a more tractable cardinality, we could rely on more standard, faster approaches during training and achieve “off-the-shelf” usability during search.
Hardware latency differences below a certain granularity are noisy
Finally, the difference between unique architectures’ accuracy or latency needs to be distinguishable during search.
The original OFA design space covers a FLOP range of 120MFLOPs.
Within this range, there are O (10 19 ) architecture choices.
Even if these models were uniformly distributed in buckets of 1 FLOP, we would still have an O (10 11 ) models in each FLOP target.
This is not accounting for the fact that each of these models support variable resolution, which would further multiply
6 width   d e p t h 71.
78 Accuracy Heatmap  6 width   d e p t h 6.
40 Latency Heatmap
On any hardware, inference time below a certain threshold is expected to be indistinguishable from noise.
This threshold can vary depending on application context or hardware (e.g. it can be upto 1-5 ms in ML serving scenarios).
Irrespective of the threshold, having these many models with such ﬁne-grained differences in their compute requirement is well below the minimum error resolution at which they can be meaningfully compared.
This motivates us to consider that a design space that is sparser by several orders of magnitude might still support the same density and range of deployment targets.
Figure 1 shows heatmaps comparing accuracy and latency of OFA sub-networks with uniform depth and width for a ﬁxed kernel size.
A closer look into these heatmaps, depicts a monotonically increasing gradient along the diagonal of each of these two dimensional heatmaps.
This observation showcases that when the conﬁgurations are increased along both the depth and width dimensions we retrieve a class of sub-networks that achieve a better accuracy-latency trade-off as opposed to those formulated through a single dimensional conﬁguration change.
Thereby, emphasizing the existence of a coefﬁcient that dictates the relationship between depth and width conﬁgurations of a model.
This suggests that by quantitatively conforming to this coupling between the model dimensions, we can aim toward reducing the network design space of an architecture without sacriﬁcing its accuracy.
Subsequently, this further implies that OFA is unnecessarily training a lot more models than required and we additionally show that OFA’s model distribution suffers from high variance as well (Appendix A.1).
Conclusively, this leads us toward designing a solution that leverages these insights thereby pruning out the redundant and ineffective sub-networks otherwise generated by the unconstrained OFA architecture.
C OMPOUND OFA 4. C OUPLING H EURISTIC
Motivated by the above observations, we devise a simple heuristic to constrain the network design space by scaling or shrinking the depth and width dimensions together and not independently.
Speciﬁcally, in each block, whenever we sample the i th largest depth d i ∈ D , we correspondingly sample the i th largest width w i ∈ W for all d i layers in the block.
For instance, with D =
[2 , 3 , 4] and W = [3 , 4 , 6] , each block can have either two layers of channel expansion ratio 3, or three layers of ratio 4, or four layers of ratio 6.
Once this modiﬁed search space is deﬁned by the heuristic, the method of extracting a given model conﬁguration remains the same – we slice out a sub-network of the speciﬁed dimensions from the largest network, thus sharing common weights between all sub-networks.
This signiﬁcantly reduces the degrees of freedom in the design space.
An additional consequence of this is that all layers within a block now have the same width, causing an extra reduction in the design space.
Nevertheless, we will show in Section 5 that the design space is still diverse and dense enough to provide models for different deployment requirements.
We ﬁx the kernel size by default, but for fair comparison we also show a variant with Elastic-Kernel.
With a ﬁxed-kernel, we create a simpliﬁed search space with kernel sizes ﬁxed to either 3 or 5 in each block, to match the original setting used in MobileNetV3.
Thereby in this design space, depth (or width) alone can fully specify a block’s conﬁguration and this yields a family of 3 5 = 243 models.
In the elastic-kernel design space, we expand the kernel size, sampled from K =
[3 , 5 , 7] per layer, same as in OFA, and this results in a family of (3 2 + 3 3 + 3 4 ) 5 ≈ 10 10 models.
Unless otherwise speciﬁed, we use “CompOFA” to refer to the ﬁxed-kernel design space.
As we show in the following section, the source of our training speedup is the reduction in cardinality of the search space.
The input resolution to the model affects the model inference time but does not affect the number of unique trainable architectures.
Hence, we keep the resolution elastic, which allows us one architecture to support multiple latencies for “free”, without increasing our search space cardinality (and therefore, training budget).
4. T RAINING S PEEDUP
Cai et al.
(2020) proposed a “progressive shrinking” approach to train the OFA sub-networks in a phased manner, starting with the largest model and then progressively including smaller subnetworks in training.
At each batch, a different sub-network is sampled for forward/backward pass.
If the expected time to complete one epoch in phase p is E ( t p ) and the training is run for e p number of epochs, then the total time to train a model family becomes T family ∝ (cid:80) p ∈
Phases e p × E ( t p ) .
Our goal is to reduce T family while achieving the same level of accuracy-latency trade-off and supporting the same range of deployment scenarios.
Since we do not wish to train bigger or smaller models themselves, we try to keep E ( t p ) unchanged and target a reduction in the number of phases to reduce T family .
While the number of models does not explicitly factor in T family , note that it implicitly affects the count & duration of training phases – progressive shrinking was required in the ﬁrst place due to interference between a large number of models.
The maximum number of simultaneously trainable models in CompOFA’s search space is smaller by 17 orders of magnitude.
With this signiﬁcantly smaller family size, we ﬁnd that the interference between these models is reduced to the extent that we are now able to remove progressive shrinking altogether, and instead achieve the same accuracy with all trainable sub-networks included in training (see Section 5).
In Section 5 we show that this allows us to reduce the training time by 50%.
Additionally, we show an ablation to conﬁrm that this reduction in phases is only possible due to the reduced number of models in CompOFA, and not in the original large design space of OFA.
E XPERIMENTS 5. T RAINING S ETUP
We train the full-sized CompOFA network ( D = [4] , W = [6] ) on ImageNet (Deng et al., 2009) using the same base architecture of MobileNetV3 as OFA.
We use a batch size of 1536 and a learning rate of 1.95 to train on 6 NVIDIA V100 GPUs.
All other training hyperparameters are kept the same as OFA for accurate comparison.
Next, for CompOFA with ﬁxed kernel sizes, we unlock all permissible model conﬁgurations ( D, W =
[(2 , 3) , (3 , 4) , (4 , 6)] ) in one stage, as opposed to progressively adding more conﬁgurations.
We train this for 25 epochs with an initial learning rate of 0.06, followed by 120 epochs with an initial learning rate of 0.18 to obtain our ﬁnal network.
We sample N sample = 4 models in each batch.
The full network serves as a teacher model for knowledge distillation (Hinton et al., 2015).
The comparison of the possible conﬁgurations and epoch durations for each training phase is summarized in Table 1.
5. T RAINING T IME AND C OST
By using just one stage of training after the teacher model (in the default ﬁxed kernel setting), instead of multiple stages of the same duration, we are able to reduce the training cost of CompOFA by 50%.
[Table]
based on hourly price of 1 NVIDIA V100 on Google Cloud.
CO 2 emission estimation is based on
Strubell et al.
[Table]
Our reproduction of OFA’s training scheme on 6 GPUs takes 978 GPU hours while CompOFA-Fixed
Kernel ﬁnishes in 493 GPU hours, as shown in Table 1.
Table 2 shows the translation of this time reduction in terms of dollar cost and CO 2 emissions.
Note that the total duration of an epoch with the same design space conﬁguration is comparable, as we do not target a speedup by training smaller models.
Instead, the speedup stems largely from halving the number of epochs, by way of reducing the phases for training.
This reduction is in turn possible due to a smaller, constrained design space.
5. A CCURACY -L ATENCY T RADE - OFF
After training CompOFA networks, we carried out an evolutionary search (Real et al., 2019) to retrieve specialized sub-networks for diverse hardware deployment scenarios, as done in OFA.
The evolutionary search fetches sub-networks that maximize accuracy subject to a target efﬁciency (latency or FLOPs) constraint.
Similar to OFA, we use a 3-layer accuracy predictor common across all platforms.
For latency, we use a lookup-table based latency estimator for Samsung Note 10 CPU provided by Cai et al.
For other hardware platforms – namely NVIDIA GeForce RTX
Ti GPU and the Intel(R) Xeon(R)
Gold 6226 CPU – we measured actual latency with a batch size of 64 and 1 respectively.
The estimated accuracies of the best models returned by this search were then veriﬁed on the actual ImageNet validation set.
Figure 2 reports the performances of accuracy-latency trade-off for CompOFA and OFA networks.
We observe that the best model in CompOFA for evaluated latency targets are at least as accurate, despite their signiﬁcantly smaller family size and training budget.
This result validates our intuition behind the simple heuristic that aids in creating a smaller design space without losing out on Paretooptimality or density of the generated models.
[Figure] [Figure]
5. D ESIGN S PACE C OMPARISON
Apart from individual models
[Figure]
we further evaluate CompOFA at a population level through a statistical sampling of design space as introduced by Radosavovic et al.
Using Samsung
Note 10 as sample hardware, we divide the supported range of latencies into buckets of 5ms each.
For each latency bucket, we randomly sample 50 models in OFA and CompOFA.
Figure 3 plots the cumulative distribution function (CDF) of classiﬁcation error for each of these latency buckets.
In each bucket, CompOFA’s CDFs lie above those of OFA, showing that CompOFA has a higher fraction of models that exceed a given accuracy.
Additionally in Figures 4, we show 20 of these models in each bucket for each search space, demonstrating that the average accuracy of a randomly picked model from CompOFA is higher than that in OFA.
The takeaway of both these evaluations is that not only does CompOFA yield competitive best models at a given latency, but a better concentration of accurate models with fewer sub-optimal models – i.e. a more accurate overall model population.
5. E FFECT OF NUMBER OF PHASES
Cai et al.
(2020) showed that OFA networks suffer a top-1 accuracy drop of up to 3.7% when progressive shrinking is removed, showing its role in resolving interference.
With O (10 19 ) models interfering with each other for simultaneous optimization of their weights, a phased approach is needed to stabilize training by gradually train model conﬁgurations.
In CompOFA, our primary method of reducing the training time is by reducing the number of phases in training.
We repeat a similar ablation to compare the effect of progressive shrinking in OFA & CompOFA.
For both design spaces, we train with & without progressive shrinking and compare the accuracies of common subnetworks.
Figure 5 shows CompOFA achieves the same accuracy with and without progressive shrinking.
We attribute this to lower interference between sub-networks in a design space smaller by 17 orders of magnitude.
Next, we build a CDF of model conﬁgurations in CompOFA and then extract the same sub-networks from OFA.
Figure 5 shows that CompOFA still maintains a slightly higher accuracy despite comparing the same models in both spaces.
Thus, models discarded in the process of constraining the design space do not add to the accuracy of OFA or CompOFA – the retained models can achieve the same (if not better) accuracy independent of any potential collaboration from these discarded models.
[Figure]
[Figure] [Figure]
OFA suffers upto
5. G ENERALIZING TO OTHER ARCHITECTURES
The guiding principle behind our heuristic – namely, the optimality of coupling depth & width over the unconstrained search space – is expected to apply to other architectures, datasets
[Figure]
and tasks
[Figure]
[Table]
In this section, we demonstrate similar savings in training budget with our heuristic applied to a different base architecture.
We train OFA and CompOFA with the base architecture changed from MobileNetV3 to ProxylessNAS (Cai et al., 2018).
We keep the same depths and widths ( D = [2 , 3 , 4] , W = [3 , 4 , 6]) and build the search spaces of OFA & CompOFA using the cartesian product or the heuristic, respectively.
In
CompOFA, we ﬁx the kernel size to 3 in all layers.
Both networks use a width multiplier of 1.3.
The training hyperparameters, schedule, and search spaces are identical to those described in Section 5.1 and Table 1.
Hence, CompOFA again requires half the training epochs – and thererfore half the training time, cost, and CO 2 emissions.
Figure 6 compares the CDFs of models common to both
OFA & CompOFA under this setting, showing that the CompOFA has the same or marginally higher accuracies for the same model conﬁgurations despite half the training budget.
5. S EARCH TIME
The intractable cardinality of OFA necessitated the use of latency estimators as a proxy to real latency measurement.
With a signiﬁcantly smaller design space, this need is removed – avoiding the time and effort of building these latency tables that eases “off-the-shelf” practical usability of
CompOFA.
We introduce a simple memoization technique during the evolutionary search to cache the measured latencies of model architectures and hence avoid its re-measurement, which is only practical with the smaller search space.
Table 3 reports the average run time of NAS algorithms for a single latency target with this optimization added to both CompOFA and OFA, reducing the search time to just 75 seconds for CompOFA Fixed-Kernel.
We also show that the evolutionary search converges in fewer iterations for CompOFA, in Appendix A.
C ONCLUSION
We build upon Once-For-All (OFA) networks to propose CompOFA – a design space for OFA networks using compound couplings between model dimensions, which speeds up the process of oneshot training & neural architecture search with hardware efﬁciency in the loop.
We leverage the insight of compound coupling between model dimensions to signiﬁcantly reduce the architecture search space induced.
Even with a simple coupling heuristic proposed, we achieve the same Pareto optimality and sufﬁcient Pareto density at a fraction of the state-of-the-art cost.
2x budget reduction and up to 216x search/extraction latency reduction is shown through elimination of unnecessary models from the design space.
We make an observation that intractably large search spaces have latency granularity that is unnecessarily ﬁne for realistic hardware deployments, and that CompOFA’s signiﬁcantly smaller search space is sufﬁciently dense to produce efﬁcient models at the same range, density, and diversity of latency/hardware targets.
This smaller cardinality reduces interference in weight-shared training to the extent that we can afford to train CompOFA to the same accuracy with a 50% reduction in the training budget.
Finally, during the search stage, we demonstrate that this reduced search space eliminates the time and effort needed to build latency lookup tables, thus improving the ease of “off-the-shelf” usability of our method in the real-world settings.
A A PPENDIX A. E XTRA M ODELS IN OFA
Figure 7 depicts the stratiﬁcation of the latency levels into multiple buckets of size 5ms each and uniformly sampled 100 sub-networks of OFA for each bucket.
This box-plot helps in uncovering the considerable variance between the maximum and minimum accuracies of the models of the same latency bucket in OFA.
This variance is mainly caused due to OFA’s enormous search space that contains redundant models that do not improve the overall accuracy of the model distribution belonging to a certain latency bucket.
A. T RAINING S CHEDULE FOR C OMP OFA-E LASTIC K ERNEL
Table 4 shows the training schedule for CompOFA with Elastic Kernel.
Compared to OFA, the training budget is reduced by 31%.
[Table]
A. F ASTER C ONVERGENCE OF NAS
An evolutionary algorithm is set to converge after a certain number of iterations ( N ) beyond which the ﬁtness value of the population ( P ) does not improve signiﬁcantly.
OFA runs NAS with a setting of N = 500 iterations for a population of size | P | = 100 .
Figure 8 demonstrates that the search time of NAS could further be reduced by reducing the number of iterations to N = 300 and N = for CompOFA Elastic-Kernel and CompOFA Fixed-Kernel respectively, without losing out on their
Pareto-optimality.
Coupled with the removal of the latency predictor, these CompOFA-speciﬁc optimizations to the search are effective in reducing the search time and improving direct usability of CompOFA
[Figure] [Figure]