P ROXY - LESS A RCHITECTURE S EARCH VIA B INARIZED P ATH L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Neural architecture search (NAS) has great impact by automatically designing efﬁcient neural network architectures.
However, the computation demand is prohibitive with conventional NAS algorithms (e.g. 10 4 GPU hours), making it difﬁcult to directly search the architecture on large-scale tasks (e.g. ImageNet).
As a result, NAS needs to utilize proxy tasks, such as training on a smaller dataset (e.g. CIFAR-10), or learning with only a few blocks rather than the full depth, or training only for a few epochs rather than till convergence.
These architectures optimized on proxy tasks are not guaranteed to be optimal on target task.
In this paper, we present Proxy-less Architecture Search that can directly learn the architectures on a large-scale target task.
We reduce the computational cost of architecture search by two orders of magnitude (to the same level of normal training).
Additionally, by using the measured hardware latency (rather than FLOPs) as a direct objective, we can specialize neural network architectures for different hardware architectures.
Experiments on CIFAR-10 and ImageNet demonstrate the effectiveness of directness.
On CIFAR-10, our model achieves 2.08% test error with only 5.7M parameters, better than the previous state-of-the-art architecture AmoebaNet-B, while using 6 × fewer parameters.
On ImageNet, our model achieves 3.1% better top-1 accuracy than MobileNetV2, while being 1.2 × faster with measured GPU latency.
Compared with previous architecture search algorithms, we saved the GPU hours by two orders of magnitude, GPU memory by one orders of magnitude, both on ImageNet.
We also analyzed the insights of efﬁcient CNN models specialized for different hardware platforms.
I NTRODUCTION
Recently neural architecture search (NAS) has demonstrated much success in automating neural network architecture design for various deep learning tasks, such as image recognition (Zoph et al., 2018;
Cai et al., 2018a; Liu et al., 2018a; Zhong et al., 2018) and language modeling (Zoph & Le, 2017).
Despite the remarkable results, conventional NAS algorithms are prohibitively computationintensive, requiring thousands of models to be trained on the target task in a single experiment.
Therefore, directly applying NAS to a large-scale task (e.g. ImageNet) is computationally difﬁcult or impossible.
As a trade-off, Zoph et al.
(2018) propose to search for building blocks on proxy tasks, such as training for fewer epochs, starting with a smaller dataset (e.g. CIFAR-10), or learning with fewer blocks.
Then top-performing blocks are stacked and transferred to the target large-scale task.
This paradigm has been widely adopted in subsequent NAS algorithms (Liu et al., 2018a;b;
Real et al., 2018;
Cai et al., 2018b; Liu et al., 2018c; Tan et al., 2018; Luo et al., 2018).
However, these blocks optimized on proxy tasks are not guaranteed to be optimal on target task.
For example, when proxy strategy ”fewer epochs” is adopted, lower weight decay may be preferred because of faster convergence on early training stage.
But it might lead to a worse ﬁnal result due to overﬁtting.
Also, competitive blocks on CIFAR-10 can have less compelling results on ImageNet (Liu et al., 2018c).
Moreover, to enable transferability, such methods typically search for only a few architectural motifs and then repeatedly stack the same pattern, which restricts the block diversity and thereby harms performance (Tan et al., 2018).
In this work, we propose a simple and effective solution to aforementioned limitations, called Proxyless Architecture Search , which directly learns the architectures on the target task rather with proxy (Figure 1) while allowing blocks to have different structures.
To achieve this, we reduce the computation cost (GPU hours and GPU memory) of architecture search to the same level of normal training in the following ways.
[Figure]
GPU hour-wise, inspired by recent works (Bender et al., 2018; Liu et al., 2018c), we directly train an over-parameterized network that contains all candidate paths (Figure 2).
And during training we explicitly introduce architecture parameters to learn which paths are redundant, while these redundant paths are pruned at the end of training to get a compact optimized architecture.
In this way, we simplify architecture search to be a single training process of the over-parameterized network without any meta-controller (or hypernetwork).
However, naively including all the candidate paths leads to GPU memory explosion (Liu et al., 2018c; Bender et al., 2018), as the memory consumption grows linearly w.r.t.
the number of choices.
Thus, GPU memory-wise, we binarize the architecture parameters (1 or 0) and force only one path to be active at run-time, which reduces the required memory to the same level of training a compact model.
These binarized architecture parameters can be optimized via gradient decent, using the BinaryConnect technique (Courbariaux et al., 2015).
Furthermore, to handle non-differentiable objectives (e.g. latency), we propose a REINFORCE-based (Williams, 1992) algorithm as an alternative strategy to update these binary parameters.
In our experiments on CIFAR-10 and ImageNet, beneﬁt from the efﬁciency and directness, our method can achieve strong empirical results.
On CIFAR-10, our model reaches 2.08% test error with only 5.7M parameters.
On ImageNet, our model achieves 75.1% top-1 accuracy which is 3.1% higher than MobileNetV2 (Sandler et al., 2018) while being 1.2 × faster.
Beside strong empirical results, our method also allows specializing neural network architectures for a certain device.
Conventionally, the same network architecture is employed for different hardware architectures.
For example, ResNet-50 is popular among various platforms, CPU, GPU and
TPUs 1 .
However, the specialized network architecture is needed for different hardware architectures to achieve best accuracy-efﬁciency trade-offs.
Our study shows that CPU and GPU have totally different preferences on efﬁcient neural network architectures: CPU prefers slim and deep structures, while GPU prefers wide and shallow ones, which matches the degree of their parallelisms (Figure 4).
To the best of our knowledge, this is the ﬁrst work to study the preferences of different platforms.
R ELATED W ORK
The use of machine learning techniques, such as reinforcement learning or neuro-evolution, to replace human experts in designing neural network architectures, usually referred to as neural architecture search, has drawn an increasing interest (Zoph & Le, 2017;
Liu et al., 2018a;b;c; Cai et al., 2018a;b; Pham et al., 2018; Brock et al., 2018; Bender et al., 2018;
Elsken et al., 2017; 2018b).
In
NAS, architecture search is typically considered as a meta-learning process.
And a meta-controller (e.g. a recurrent neural network (RNN)), is introduced to explore a given architecture space with training a network in the inner loop to get an evaluation for guiding exploration.
Consequently, such methods are computationally expensive to run, especially on large-scale tasks, e.g. ImageNet.
Some recent works (Brock et al., 2018; Pham et al., 2018) try to improve the efﬁciency of this meta-learning process by reducing the cost of getting an evaluation.
In Brock et al.
(2018), a hypernetwork is utilized to generate weights for each sampled network and hence can evaluate the architecture without training it.
Similarly, Pham et al.
(2018) propose to share weights among all sampled networks under the standard NAS framework (Zoph & Le, 2017).
These methods speed up architecture search by orders of magnitude, however require a hypernetwork or an RNN controller, thus remain complicated and hard to optimize.
Moreover, they mainly focus on small-scale tasks (e.g. CIFAR) rather than large-scale tasks (e.g. ImageNet).
Our work is most closely related to One-Shot (Bender et al., 2018) and DARTS (Liu et al., 2018c), both of which get rid of the meta-controller (or hypernetwork) by modeling architecture search as a single training process of an over-parameterized network that comprises all candidate paths.
Specifically, One-Shot trains the over-parameterized network with DropPath (Zoph et al., 2018) that drops out each path with some ﬁxed probability.
Then they use the pre-trained over-parameterized network
[Figure]
Architecture Parameters
Binary Gate (0:prune, 1:keep)
INPUT
OUTPUT α β … δ  … (2) Update architecture parameters
INPUT α β … δ  … update fmap not in memory fmap in memory
CONV 5x
POOL 3x3 ...
Weight Parameters CONV 3x
Identity
CONV 5x
POOL 3x3 ...
Identity
[Figure]
CONV 3x
OUTPUT to evaluate architectures, which are sampled by randomly zeroing out paths.
DARTS additionally introduces a real-valued architecture parameter for each path and jointly train weight parameters and architecture parameters via standard gradient decent.
In this work, we address the large memory issue in these two methods through path binarization.
Another relevant topic is network pruning (Han et al., 2016) that aims to improve the efﬁciency of neural networks by removing insigniﬁcant neurons (Han et al., 2015) or channels (Liu et al., 2017).
Similar to these works, we start with an over-parameterized network and then prune the redundant parts to derive the optimized architecture.
The distinction is that they focus on layer-level pruning that only modiﬁes the ﬁlter (or units) number of a layer but fails to change the topology of the network, while we focus on learning effective network architectures through path-level pruning.
M ETHOD
In this section, we ﬁrst describe the construction of the over-parameterized network with all candidate paths, then introduce how we leverage binarized architecture parameters to reduce the memory consumption of training the over-parameterized network to the same level as normal training.
Finally, we present two algorithms for learning the binarized architecture parameters, based on BinaryConnect (Courbariaux et al., 2015) and REINFORCE (Williams, 1992) respectively.
3. C ONSTRUCTION OF O VER -P ARAMETERIZED N ETWORK
A neural network can be expressed as a directed acyclic graph (DAG), where each edge is associated with an operation o ( · ) that transforms the feature map.
As the lack of connection between two nodes in DAG can be indicated using the zero operation, the task of learning both the connections and operations that are applied at different positions within the network can thereby reduce to choosing operations on its edges (Liu et al., 2018c).
For simplicity, we omit nodes and denote a neural network as N ( e, · · · , e n ) where e i represents a certain edge in DAG in following discussions.
Let O =
{ o i } be the set of N candidate primitive operations (e.g. convolution, pooling, identity, zero, etc).
To construct the over-parameterized network that includes any architecture in the search space, instead of setting each edge to be a deﬁnite primitive operation, we set each edge to be a mixed operation that has N parallel paths (Figure 2), denoted as m O .
As such, the over-parameterized network can be expressed as N ( e = m 1 O , · · · , e n = m n O ) .
Here, we assume that the candidate set keeps the same throughout the network for ease of discussion, while we can actually arbitrarily adjust the candidate set for each certain position.
Given input x , the output of a mixed operation m O is deﬁned based on the outputs of its N paths.
In
One-Shot, m O ( x ) is the sum of { o i ( x ) } , while in DARTS, m O ( x ) is weighted sum of { o i ( x ) } where the weights are calculated by applying softmax to N real-valued architecture parameters { α i } :
[Equation]
To derive the compact optimized architecture from the over-parameterized network, we ﬁrstly train the over-parameterized network, then identify and prune N − 1 redundant paths for each mixed operation.
However, straightforwardly training such an over-parameterized network can be problematic.
As shown in Eq.
(1), the output feature maps of all N paths are calculated and stored in the memory, while training a compact model only involves one path.
Therefore, One-Shot and DARTS roughly need N times GPU memory and GPU hours compared to training a compact model.
On large-scale dataset, this can easily exceed the memory limits of hardware with large design space.
In the following section, we solve this memory issue based on the idea of path binarization.
3. L EARNING B INARIZED P ATH
In this work, inspired by BinaryConnect (Courbariaux et al., 2015), we employ a binarization approach for reducing the cost of training the over-parameterized network.
Speciﬁcally, similar to
DARTS, we also introduce N real-valued architecture parameters { α i } , but instead of directly using the real-valued weights { p i } (referred to as path weights), we additionally incorporate a binarization process that transforms the real-valued path weights to binary gates:
[Equation]
Based on the binary gates g , the output of the mixed operation is given as:
[Equation]
And the expectation of m Binary O
( x
[Equation]
equals to m DARTS O
[Equation]
x
[Equation] [Equation]
As illustrated in Eq.
(3) and Figure 2, by using the binary gates rather than real-valued path weights (Liu et al., 2018c), only one path of activation is active in memory at run-time and the memory requirement of training the over-parameterized network is thus reduced to the same level of training a compact model.
Notice that setting the binary gate value to 0 corresponds to dropping out the path while setting to 1 corresponds to keeping the path.
Therefore, when training the weight parameters of the overparameterized network (Figure 2 left), we are actually training the network with DropPath regularization (Zoph et al., 2018), as is done in One-Shot (Bender et al., 2018).
While in our case, the probability of dropping out the path is not regarded as a hyperparameter but a trainable parameter.
Furthermore, we are also not aiming to make the over-parameterized network to be a good reference for ranking sampled architectures but directly learn the architecture by learning architecture parameters.
Additionally, we allow only one path to be activated during training, while in One-Shot most of the paths are activated, beginning with all paths activated.
3. T RAINING B INARIZED A RCHITECTURE P ARAMETERS
An illustration of the training procedure of the weight parameters and binarized architecture parameters in the over-parameterized network is provided in Figure 2.
When training weight parameters, we freeze the architecture parameters and stochastically sample binary gates according to Eq.
(2) for each batch of input data.
Then the weight parameters of active paths are updated via standard gradient decent on the training set (Figure 2 left).
When training architecture parameters, the weight parameters are ﬁxed, then we reset the binary gates and update the architecture parameters on the validation set (Figure 2 right).
These two update steps are performed in an alternating manner.
Once the training of architecture parameters is ﬁnished, we can then derive the compact architecture by pruning redundant paths.
In this work, we simply keep the path with the highest path weight.
Unlike weight parameters, the architecture parameters are not directly involved in the computation graph and thereby cannot be updated using the standard gradient decent.
In this section, we introduce two algorithms for learning the architecture parameters.
3.3. G RADIENT -B ASED U PDATES
The gradient-based algorithm for learning binarized architecture parameters is built upon BinaryConnect (Courbariaux et al., 2015) that proposes to update the real-valued weight using the gradient w.r.t.
its corresponding binary gate.
In our case, analogously, the gradients w.r.
t. architecture
parameters can be approximately estimated using ∂L/∂g i in replace of ∂L/∂p i : ∂L
[Equation]
∂α
[Equation]
=
[Equation]
(5) where δ ij = 1 if i = j and δ ij =
if i (cid:54) =
j .
Since the binary gates g are involved in the computation graph, as shown in Eq.
(3), ∂L/∂g j can be calculated through backpropagation.
However, computing ∂L/∂g j actually requires to calculate and store o j ( x ) .
Therefore, directly using Eq.
(5) to update the architecture parameters would also require roughly N times GPU memory compared to training a compact model.
To address this issue, we consider to factorize the task of choosing one path out of N candidates into multiple binary selection tasks.
The intuition is that if a path is the best choice at a certain position, it should be the better choice when solely compared to any other path.
Following this idea, within an update step of the architecture parameters, we ﬁrst sample two paths according to the multinomial distribution ( p 1 , · · · , p N ) and mask all the other paths as if they do not exist.
As such the number of candidates temporarily decrease from N to 2, while the path weights { p i } and binary gates { g i } are reset accordingly.
Then we update the architecture parameters of these two sampled paths using the gradients calculated via Eq.
Finally, as path weights are computed by applying softmax to the architecture parameters, we need to rescale the value of these two updated architecture parameters by multiplying a ratio to keep the path weights of unsampled paths unchanged.
As such, in each update step, one of the sampled path is enhanced (path weight increases) and the other sampled path is attenuated (path weight decreases) while all other paths keep unchanged.
In this way, regardless of the value of N , only two paths are involved in each update step of the architecture parameters, and thereby the memory requirement is reduced to the same level of training a compact model.
3.3. REINFORCE-B ASED U PDATES FOR N ON -D IFFERENTIABLE O BJECTIVES
The gradient-based algorithm relies on the gradients of the loss function w.r.t.
binary gates.
But not all objectives are differentiable, such as latency, energy and memory.
In this work, we consider another solution for updating binary weights based on REINFORCE (Williams, 1992), as an alternative to BinaryConnect (Courbariaux et al., 2015).
Consider a network that has both weight parameters and binarized architecture parameters α .
Here we assume the network only has one mixed operation for ease of illustration.
Since only one path will be retained when deriving the ﬁnal optimized architecture, the goal of updating architecture parameters is actually to ﬁnd the optimal binary gates g that maximizes a certain reward, denoted as R ( · ) .
Therefore, according to REINFORCE (Williams, 1992), we have the following updates for architecture parameters:
[Equation]
where g i denotes the i th sampled binary gates, p ( g i ) denotes the probability of sampling g i according to Eq.
and N g i is the compact network according to the binary gates g i .
Since Eq.
(6) does not require R ( N g ) to be differentiable w.r.t.
g , it can thus handle non-differentiable objectives.
An interesting observation is that Eq.
(6) has a similar form to the standard NAS (Zoph & Le, 2017), while it is not a sequential decision making process and no RNN meta-controller is used in our case.
Furthermore, since both gradient-based updates and REINFORCE-based updates are essentially two different update rules to the same binarized architecture parameters, it is possible to combine them to form a new update rule for the architecture parameters.
E XPERIMENTS AND R ESULTS
We demonstrate the effectiveness of our proposed method on two benchmark datasets (CIFARand ImageNet) for the image classiﬁcation task.
Unlike previous NAS works (Zoph et al., 2018; Cai
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
et al., 2018b; Liu et al., 2018c)
that ﬁrst learn CNN blocks on CIFAR-10 under small-scale setting (e.g. fewer blocks), then transfer the learned block to ImageNet or CIFAR-10 under large-scale setting by repeatedly stacking it, we directly learn the architectures on the target task (either Cifar or ImageNet) and allow each block to be different.
4. E XPERIMENTS ON CIFAR-
Architecture Space.
For CIFAR-10 experiments, we use the tree-structured architecture space that is introduced by Cai et al.
(2018b)
with PyramidNet (Han et al., 2017) as the backbone.
Speciﬁcally, we replace all 3 × 3 convolution layers in the residual blocks of a PyramidNet with tree-structured cells, each of which has a depth of 3 and the number of branches is set to be 2 at each node (except the leaf nodes).
The allocation scheme and merge scheme of a non-leaf node are set to be replication and add respectively.
For further details about the tree-structured architecture space, we refer to the original paper (Cai et al., 2018b).
In our experiments, instead of using a meta-controller to predict which operation should be chosen at each edge in a tree-structured cell, we set each edge to be a mixed operation over the candidate set 2 and use the proposed method to learn the cell structures.
Moreover, by using distinct architecture parameters for different cells, they are free to be have different structures.
Additionally, we use two hyperparameters to control the depth and width of a network in this architecture space, i.e. B and F , which respectively represents the number of blocks at each stage (totally 3 stages on CIFAR-10) and the number of output channels of the ﬁnal block.
Training Details.
CIFAR-10 consists of 50,000 training images and 10,000 test images.
We randomly sample 5,000 images from the training set as a validation set for learning architecture parameters which are updated using the Adam optimizer with an initial learning rate of 0.006 for gradientbased algorithm (Section 3.3.1) and 0.01 for REINFORCE-based algorithm (Section 3.3.2).
In the following discussions, we refer to these two algorithms as Proxyless-G (gradient) and Proxyless-R (REINFORCE) respectively.
After the training process of the over-parameterized network completes, a compact network is derived according to the architecture parameters, as discussed in Section 3.3.
Next we train the compact network using the same training settings except that the number of training epochs increases from 200 to 300.
Additionally, when the DropPath regularization (Zoph et al., 2018; Huang et al., 2016) is adopted, we further increase the number of training epochs to 600 (Zoph et al., 2018).
Results.
We ﬁrst apply the proposed method to learn architectures in the tree-structured architecture space with B = 18 and F = 400 .
Since we do not force cells to share the same structure and each cell has 12 learnable edges, totally 12 × 18 ×
3 = 648 decisions are required to fully determine the architecture.
Furthermore, each edge has 7 possible options, so there are 7 648 ≈ 10 547 possible architectures in the entire search space.
The test error rate results of our proposed method and other state-of-the-art architectures on CIFAR10 are summarized in Table 1, where “c/o” indicates the use of Cutout (DeVries & Taylor, 2017).
Compared to these state-of-the-art architectures, our proposed method can achieve not only lower test error rate but also better parameter efﬁciency.
Speciﬁcally, Proxyless-G reaches a test error rate of 2.08% which is slightly better than AmoebaNet-B (Real et al., 2018)
(the previous best architecture on CIFAR-10).
Notably, AmoebaNet-B uses 34.9M parameters while our model only uses 5.7M parameters which is 6 × fewer than
AmoebaNet-B.
Furthermore, compared with PathLevel
EAS (Cai et al., 2018b) that also explores the tree-structured architecture space, both Proxyless-G
[Table] [Table]
and Proxyless-R achieves similar or lower test error rate results with half fewer parameters.
The strong empirical results of our proposed method demonstrate the beneﬁts of removing the constraint of forcing all cells to be the same and directly exploring a large architecture space instead of repeatedly stacking the same cell which is learned on a small architecture space.
It also shows that our proposed method can effectively handle such a huge architecture space.
To further justify our proposed method, we directly compare our method to two most related works, i.e. One-Shot (Bender et al., 2018) and DARTS (Liu et al., 2018c), using the same architecture space under three different settings, i.e. ( B = 4 , F = 256) , ( B = 8 , F = 256) and ( B = 8 , F = 400) .
Additionally, to save time, the DropPath regularization is not used when training all the derived architectures.
So they are trained for 300 rather than 600 epochs under all three settings.
For DARTS, due to its high GPU memory requirement (Section 3.1), we apply it to learn the cell structure on the ﬁrst setting, i.e. ( B = 4 , F = 256) , and then transfer the learned cell to the other two settings.
For One-Shot, we implement a modiﬁed version, denoted as One-Shot ∗ .
We set the probability of keeping a path to be 1 /N , where N is the number of candidate operations, rather than a high value in the original paper.
As such, One-Shot ∗ can be viewed as a reduced version of our proposed method, where the architecture parameter updates are disabled.
Also by setting a low keep probability, which reduces the GPU memory consumption, One-Shot ∗ can be directly applied to all of the three settings.
The results are reported in Figure 3 where we can observe that: (i) Both Proxyless-G and ProxylessR perform better than One-Shot ∗ under all settings, which demonstrates the importance of architecture parameter updates; (ii) DARTS performs better than One-Shot ∗ on the ﬁrst setting where
DARTS is directly learning on.
But on the other two settings, the transferred cell of DARTS performs worse than the directly learned cell of One-Shot ∗ ; (iii) Proxyless-G performs worse than
Proxyless-R on the ﬁrst setting while performs better on the other two settings.
Notice that the architecture space is larger in the last two settings, we suppose it might because that gradient-based algorithm is more efﬁcient in handling large architecture space than REINFORCE-based algorithms.
4. E XPERIMENTS ON I MAGE N ET
Not all rewards are differentiable.
For ImageNet experiments, we aim to learn efﬁcient CNN architectures that not only have high accuracy but also low latency on speciﬁc devices (e.g. GPU,
CPU, mobile phone, etc.).
Therefore, it is a multi-objective NAS task (Hsu et al., 2018; Dong et al., 2018; Elsken et al., 2018a;
He et al., 2018; Tan et al., 2018), where one of the objectives is nondifferentiable (i.e. latency).
Thus, we employ Proxyless-R for ImageNet experiments.
Same as Tan et al.
(2018), we use ACC ( m ) ×
[ LAT ( m ) /T ]
w as the optimization goal, where ACC ( m ) denotes the accuracy of model m , LAT ( m ) denotes the latency of m , T is the target latency and w is a hyperparameter for controlling the trade-off between accuracy and latency.
We use GPU latency and CPU latency in our experiments as reward.
If given mobile farm infrastructure, our algorithm can be easily transferred to the mobile setting 4 .
The GPU latency is measured on V100 GPU with a batch size of 8 (single batch will make GPU severely under-utilized).
The CPU latency is measured on AWS p3.8 instance with a batch size of 1.
Architecture Space.
We use MobileNetV2 (Sandler et al., 2018) as the backbone to build the architecture space.
Speciﬁcally, rather than repeating the same mobile inverted bottleneck convolution (MBConv), we allow a set of MBConv layers with various kernel sizes { 3 , 5 , 7 } and expansion ratios { 3 , 6 } .
To enable a direct trade-off between width and depth, we initiate the over-parameterized network to have more blocks than MobileNetV2 and allow a block with the residual connection to be skipped by adding the zero operation to the candidate set of its mixed operation.
In this way, with a limited latency budget, the network can either choose to be shallower and wider by skipping more blocks and using larger MBConv layers or choose to be deeper and thinner by keeping more blocks and using smaller MBConv layers.
[Table] [Table]
7x
[Figure]
7x
[Figure]
7x
[Figure]
oo
[Figure]
ng
F C
[Figure]
x
[Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure]
192 x 7 x192 x 7 x
[Figure]
7 x
[Figure]
[Figure] [Figure]
5x
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
an expansion ratio of 3 and 6 respectively.
[Table]
Training Details.
We randomly sample 50,000 images from the training set as a validation set during the architecture search.
The settings for updating architecture parameters are the same as
CIFAR-10 experiments except the initial learning rate is 0.001.
The over-parameterized network is trained for 100 epochs on the remaining training images with batch size 256.
ImageNet Classiﬁcation Results.
Table 2 reports the results of our searched architecture with accuracy and GPU latency as objectives.
We use the GPU latency of MobileNetV2 as the target latency.
Compared to MobileNetV2 and ResNet-34, our model improves the top-1 accuracy by 3.1% and 1.8% respectively while maintaining lower GPU latency.
While compared with MnasNet, our model can achieves 1.1% higher top-1 accuracy with 1.2 × fewer GPU latency.
More importantly, we are much more sample efﬁcient: the GPU-hour is 100 × fewer than MnasNet (Table 3).
Discussion on Our Searched Models for Different Platforms.
Beside GPU latency, we also apply our method to learn architectures with CPU latency as an objective.
The results of our GPU model and CPU model are reported in Table 4, while their detailed architectures are provided in Figure 4.
We notice that the architecture is totally different when targeting different platforms: (i) The GPU model is shallower and wider, especially in early stages where the feature map has higher resolution; (ii) The GPU model prefers large MBConv operations (e.g. 7 × 7 MBConv6), while the CPU model would go for smaller MBConv operations.
This is because GPU has much higher parallelism than
CPU so it can take advantage of large MBConv operations.
Beside distinctions, another interesting observation is that both GPU model and CPU model prefer to choose larger MBConv operations in the ﬁrst block within each stage where the feature map is downsampled.
We suppose it might because larger MBConv operations are beneﬁcial for the network to preserve more information when downsampling.
Notably, such kind of patterns can not be captured in previous NAS methods as they force the blocks to share the same structure throughout the network (Liu et al., 2018a) or within each stage (Tan et al., 2018).
C ONCLUSION
In this work, we introduce Proxy-less Architecture Search, which is able to directly learn architectures on the target task without any proxy.
We also reduces the cost (GPU-hours and GPU memory) of NAS to the same level of normal training using path binarization.
Beneﬁt from the direct search, we achieve strong empirical results on CIFAR-10 and ImageNet.
Furthermore, we allow specializing network architectures for different platforms by directly incorporating the measured hardware latency into optimization objectives.
We compared the optimal model on CPU/GPU and raised the awareness that specialized neural network architecture is needed for different hardware architectures.
A T HE L IST OF C ANDIDATE O PERATIONS U SED ON CIFAR-
Similar to previous works (Zoph et al., 2018; Cai et al., 2018b; Liu et al., 2018c), we adopt the following 7 operations in our CIFAR-10 experiments:
• 3 × 3 dilated depthwise-separable convolution • Identity • 3 × 3 depthwise-separable convolution • 5 × 5 depthwise-separable convolution • 7 × 7 depthwise-separable convolution • 3 × 3 average pooling • 3 × 3 max pooling
B D ETAILS ABOUT T ABLE
Mnas trains 8,000 mobile-sized models on ImageNet, each of which is trained for 5 epochs for learning architectures.
And our method trains a single model for 100 epochs whose cost of training is at the same level as training a mobile-size model.
Therefore we use roughly 400 × fewer GPU hours than Mnas, while our GPU memory cost also keeps at the same level of training a mobile-sized model.
The GPU memory of DARTS is inferred using our architecture space on ImageNet, as the original paper does not directly search for architectures on ImageNet.
Additionally, when training a mobile-sized model on ImageNet with a batch size of 64, the GPU memory consumption is roughly 10GB.
C L EARNING S PECIALIZED N EURAL N ETWORK A RCHITECTURES FOR M OBILE P HONE
In this section, we present an economical way for learning specialized neural network architectures for mobile phone.
By incorporating an accurate mobile latency prediction model, we can use the predicted mobile latency rather than measured mobile latency during architecture search.
As such, we remove the need for an expensive mobile farm infrastructure (Tan et al., 2018) that is hard for academic research teams to build.
[Table]
[Figure]
C. M OBILE L ATENCY P REDICTION
Measuring the latency on-device is accurate but not ideal for scalable neural architecture search.
There are two reasons: (i) Slow.
As suggested in TensorFlow-Lite, we need to average hundreds of runs to produce a precise measurement, approximately 20 seconds.
This is far more slower than a single forward / backward execution.
(ii) Expensive.
A lot of mobile devices and engineering work are required to build an automatic pipeline to gather the latency from a mobile farm.
Instead of direct measurement, we build a model to estimate the latency.
We need only 1 phone rather than a farm of phones, which has only 0.75ms latency RMSE.
We use the latency model to search, and we use the measured latency to report the ﬁnal model’s latency.
We sampled 5k architectures from our candidate space, where 4k architectures are used to build the latency model and the rest are used for test.
We measured the latency on Google Pixel 1 phone using TensorFlow-Lite.
The features include (i) type of the operator (ii) input and output feature map size (iii) other attributes like kernel size, stride for convolution.
As illustrated in Figure 5, we
[Table]
x96 x 14 x96 x 14 x192 x 7 x192 x 7 x192 x 7 x192 x 7 x320 x 7 x
M B5x5 80 xx
M B7x
M B7x7 96 xx observe a strong correlation between the predicted latency and real measured latency on the test set, suggesting that the latency prediction model can be used to replace the mobile farm infrastructure with little error introduced.
C. R ESULTS ON M OBILE S ETTING
The mobile latency is measured on Pixel 1 phone with a batch size of 1.
The results on the mobile setting are summarized in Table 5 and the detailed architecture of the searched model is illustrated in Figure 6.
We ﬁnd that our model can still achieve superior performances compared to both human-designed and automatically searched architectures.
Speciﬁcally, our model improves the top-1 accuracy by 2.6% and 0.6% respectively with similar mobile latency on Pixel 1 phone ( < 80ms), compared to MobileNetV2 and MnasNet.
And compared with NASNet-A and AmoebaNetA, our model runs 2.4 × faster with slightly higher top-1 accuracy.
D M AKING L ATENCY D IFFERENTIABLE WITH L ATENCY P REDICTION M ODEL
On resource-constrained platforms such as mobile phone and embedded devices, beside accuracy, latency (not FLOPs) is another very important objective when designing efﬁcient neural network architectures (Tan et al., 2018).
Unfortunately, unlike accuracy that can be optimized using the gradient of the loss function (e.g. cross entropy loss) (Liu et al., 2018c), latency is a non-differentiable objective.
In Section 3.3.2, we have presented a REINFORCE-based approach for handling nondifferentiable objectives.
In this section, we propose another approach to address this issue.
Specifically, we introduce a novel latency regularization loss based on our proposed latency prediction model (see Section C.1).
By adding the latency regularization loss to the normal loss function, we are able to directly optimize the trade-off between accuracy and latency in a differentiable manner.
D. L ATENCY R EGULARIZATION L OSS
The latency regularization loss is based on a key observation that the expected latency of a mixed operation is actually differentiable w.r.t.
architecture parameters.
Consider a mixed operation with a candidate set { o j } and each o j is associated with an architecture parameter p j which represents the probability (or weights) of choosing o j .
As such, we have the expected latency of a mixed operation (i.e. a learnable block) as:
[Equation]
where E [ latency i ] is the expected latency of the i th learnable block, F ( · ) denotes the latency prediction model and F ( o ij ) is the predicted latency of o ij .
The gradient of E [ latency i ] w.r.t.
architecture parameters can thereby be given as:
[Equation]
For the whole network with a sequence of mixed operations (Figure 7 left), since these operations are executed in a sequential manner when doing inference, the expected latency of the network can be expressed with the sum of the expected latencies of these mixed operations:
[Equation]
[Table]
[Table]
[Table]
[Table]
[Table]
[Figure]
where E [ latency ] is the expected latency of the network.
As such, E [ latency ] is also differentiable w.r.t.
architecture parameters.
We incorporate the expected latency of the network into the normal loss function by multiplying a scaling factor λ 2 ( > 0) which controls the trade-off between accuracy and latency.
The ﬁnal loss function is given as (also shown in Figure 7 right):
[Equation]
where Loss CE denotes the cross-entropy loss and λ 1 || w
|| 22 is the weight decay term.
D. E FFECTIVENESS OF L ATENCY R EGULARIZATION L OSS
In this section, we present experiments on ImageNet to justify the effectiveness of the proposed latency regularization loss.
The summarized results are reported in Table 6, where the results of
Proxyless-G without latency regularization loss is also included for comparison.
We observe that Proxyless-G has no incentive to choose computation-cheap operations without latency regularization loss.
The resulting architecture initially has 158ms latency on Pixel 1.
After rescaling the network using depth multiplier (Sandler et al., 2018; Tan et al., 2018), the latency reduces to 83ms.
However, this model can only achieve 71.8% top-1 accuracy on ImageNet, which is 0.2% lower than MobileNetV2.
Therefore, we conclude that it is essential to take latency as a direct objective when designing efﬁcient (low-latency) neural network architectures.
In contrast, with latency regularization loss, Proxyless-G can make a good trade-off between accuracy and latency.
Speciﬁcally, with similar latency, it improves the top-1 accuracy on ImageNet by 2.2% and 2.4% respectively, compared to MobileNetV2 and Proxyless-G (without latency loss).
[Table]
Without it, differentiable NAS has no incentive to choose computation-cheap operations and would fail to reach compelling accuracy results under latency constraints.