A
RCHITECTURE C OMPRESSION
Anonymous authors
Paper under double-blind review
A BSTRACT
In this paper we propose a novel approach to model compression termed Architecture Compression.
Instead of operating on the weight or ﬁlter space of the network like classical model compression methods, our approach operates on the architecture space.
A 1-D CNN encoder/decoder is trained to learn a mapping from discrete architecture space to a continuous embedding and back.
Additionally, this embedding is jointly trained to regress accuracy and parameter count in order to incorporate information about the architecture’s effectiveness on the dataset.
During the compression phase, we ﬁrst encode the network and then perform gradient descent in continuous space to optimize a compression objective function that maximizes accuracy and minimizes parameter count.
The ﬁnal continuous feature is then mapped to a discrete architecture using the decoder.
We demonstrate the merits of this approach on visual recognition tasks such as CIFAR-10/100,
FMNIST and SVHN and achieve a greater than 20x compression on CIFAR-10.
I NTRODUCTION
The recent adoption of CNNs for real-time, on-device applications has fueled a great demand for better model compression.
Conventional methods such as quantization, pruning and distillation have proven to be reliable approaches in reducing redundancies in networks.
However, one avenue where there has been little progress is that of architecture space based compression.
Approaches such as Iandola et al.
, Howard et al.
(2017), Rastegari et al.
(2016), have introduced hand deﬁned heuristics to design networks that are efﬁcient without sacriﬁcing accuracy.
However, designing such networks still remains a laborious task, requiring much human expertise.
In such a context, an efﬁcient dataset-driven approach to determine the optimal architecture is desirable.
Recent methods such as Ashok et al.
; He et al.
; Zhou et al.
Tan et al.
(2018) have proposed dataset-driven architecture based model compression using reinforcement learning.
However, one drawback of such RL based methods is that they are less sample-efﬁcient, often requiring a large number of architecture evaluations and several hours or days of training to ﬁnd a single compressed model.
Furthermore, it is unclear how these approaches compare to random search and whether they only succeed due to large-scale exploration and evaluation of many architectures.
To the best of our knowledge, this is the ﬁrst paper to introduce a novel gradient descent based approach to perform architecture compression.
To facilitate learning, we describe a mapping from discrete architecture space to a continuous space that encodes the structure of architecture for a speciﬁc dataset.
We train a one-dimensional convolutional encoder/decoder neural network to learn the structure of discrete architecture space while using accuracy and parameter regressors to impose the structure inherent to learning the dataset.
For compression, we encode an input architecture and leverage this learned continuous latent space to perform gradient descent on the encoded feature.
The augmented feature is then decoded into a compressed discrete architecture.
We demonstrate the effectiveness of our approach on several visual learning tasks of varying difﬁculty (FMNIST, SVHN, CIFAR-10, CIFAR-100) and show 5-20x compression on architectures.
We also compare this approach to conventional compression methods such as pruning, distillation and reinforcement learning and show that our architectures are smaller and faster than those produced by the baseline methods.
R ELATED WORKS 2. A RCHITECTURE S EARCH
There have been recent works in architecture search such as Zoph & Le (2016), Baker et al.
Miikkulainen et al.
(2017), Real et al.
(2017), that use reinforcement learning or evolutionary algorithms to traverse the space of architectures.
These approaches characterize architectures as discrete entities in a non-differentiable space and use either a constrained state space or heuristics to discover architectures.
These approaches have been subject to criticism from the machine learning community for using large amounts of computation.
Work such as Ashok et al.
(2018), Tan et al.
, Zhou et al.
, He et al.
attempt to search for efﬁcient architectures instead of just the best.
However these approaches also currently rely on reinforcement learning.
In contrast to these approaches, our method shows that we can leverage classical optimization techniques such as gradient descent to effectively search the space of architectures for a constrained objective.
Methods such as Liu et al.
and Luo et al.
(2018) attempt to make the space of architectures differentiable however this work is focused on discovering repeatable convolutional cells instead of optimizing the architecture as a whole.
2. P RUNING , Q UANTIZATION , H AND DESIGNED MODELS
Pruning-based methods preserve the weights that matter most and remove the redundant weights
LeCun et al.
Hassibi et al.
(1993), Srinivas & Babu (2015), Han et al.
(2015b), Han et al.
(2015a), Mariet & Sra (2015), Anwar et al.
(2015), Guo et al.
While pruning-based approaches typically operate on weight space, our approach operates on the the model architecture.
Additionally, our method offers greater ﬂexibility as we can use memory, inference time, power, and other hardware constraints to guide the compression process, resulting in the optimal architecture for a given dataset and constraints.
Hand designed models such as Iandola et al.
(2016) and Howard et al.
(2017) have been shown to work well in practical applications such as self-driving cars and mobile phones.
These are good stepping stones in making progress in designing efﬁcient architectures.
Our work focuses on designing a general method to design such architectures in a fully automated data-driven manner.
S TRUCTURE OF ARCHITECTURE COMPRESSION SPACE
In this section we show how to map certain classes of architectures to a unique vector representation in order to enable learning.
We then analyze the cardinality of architecture space for compression and prove that this space has several desirable properties for learning a mapping.
3. T OPOLOGICAL REPRESENTATION OF ARCHITECTURES
We observe that most modern neural network architectures A ∈ A , with the exception of recurrent neural networks can be expressed as a directed acyclic graph (DAG).
We use the property that every
DAG has a topological ordering (Proof in section 7) to represent an architecture as a partially ordered sequence of layers, l i ∈ L .
Formally,
[Equation]
such that
[Equation]
where the → operator denotes that the activation of l i is the input of l j .
Furthermore, this topological ordering is unique (Proof in section 7.2) for standard convolutional networks and those with simple skip connections (e.g. ResNet).
We will denote this ordered representation of an architecture as
[Equation]
where o is the topological sorting function.
Note also that for unique, one-to-one mappings, there exists an inverse function o − 1 to recover A given l 1: N
[Equation]
3. C ARDINALITY OF ARCHITECTURE COMPRESSION SPACE
Let A ∈
A be an architecture that is parameterized by θ and trained on some dataset D .
Then we can model the accuracy a ∈ [0 , 1] as a continuous random variable stochastically sampled from a distribution P θ ( a | A, D ) , since θ is typically optimized using a variant of stochastic gradient descent.
The expected accuracy E θ [ a | A, D ] , while not exactly computable due to the large dimension of θ , can be approximated by training and evaluating the network under varying intializations θ 0 , i.e.
[Equation]
where g minimizes an objective function over D with respect to θ .
Thus, we are interested in learning a mapping from discrete architecture space to the expected accuracy
[Equation]
However, since we are interested in model compression , we also want to construct the inverse mapping from the expected accuracy to discrete architecture space for a given parameter count p ∈ R +
[Equation]
In the following subsections, we prove that such a mapping, while not unique, is feasible to learn since the range of valid discrete architectures is ﬁnite.
For the following sections, we will denote parametric layers i.e. convolutional layers, batch normalization and fully connected layers as f θ ( x ) , where x is the activation of the previous layer and θ represents the parameters of the layer.
Non-parametric pooling layers as max pooling and average pooling layers are written as q s ( x ) , where s > 1 (no upsampling) is the stride of the layer.
Lastly activations such as sigmoid, ReLU, softmax etc. are represented as σ ( x ) .
We also make the following practical assumptions in the following proofs:
The input X ∈ R H × W × C , where H, W, C ∈ N +
and H, W, C are ﬁxed.
For layers l i ∈ A , if l i ∈ Σ , l i +1 / ∈ Σ , where Σ is the set of possible activation functions.
Pooling layers reduce dimensions of input image by some factor
Lemma 3.1.
For a spatial input of dimension d 0 , the number of pooling layers possible in a valid architecture is ﬁnite.
Proof.
Let x i be the input of the ith layer with spatial dimension d i ∈ N + .
We observe that a pooling operation with stride s ∈ N + > 1 and kernel size k ∈ N + x i +
=
q s ( x i ) results in an output of d i +
= (cid:98) d
i − k s (cid:99) + 1 <
d
i .
Since d 0 is ﬁxed (Assumption 1) and s is discrete, for any d 0 there is a ﬁnite number of pooling layers possible in a valid architecture.
Theorem 3.2.
For a given parameter constraint p , the set of valid architectures that meet the parameter constraint A p =
{ A | numParams ( A ) ≤ p } is ﬁnite.
Proof.
Next, suppose we have a partially constructed network with parameter count p 0 .
Then by assumption 2, we can add at most one activation function.
Furthermore by Lemma 3.2, we have a ﬁnite choice of pooling layers such that the architecture still remains valid.
Suppose we add a parametric layer
l i = f θi , the parameter constraint exceeded if p 0 + | θ i | > p .
Thus, we have the upperbound | θ i | ≤ p − p 0 .
Since | θ i | is discrete, we have a ﬁnite number of choices of layers such that the partially constructed network is still valid.
We then observe that since d i monotonically decreases if we add a pooling layer and the upperbound of |
θ i | monotonically decreases if we add a parametric layer.
Since at least every other layer has to be one of these types, the length of any valid architecture is ﬁnite.
Thus, we arrive at the conclusion that there are a ﬁnite number of valid architectures for a given parameter constraint.
A PPROACH
Having proved that a mapping is feasible, we now describe our approach to train a 1-dimensional convolutional encoder/decoder, an accuracy regressor and a parameter regressor to learn such a mapping (Outline in Fig. 1).
The variable length input vector to the network denoted by x 1: T =
[ x i , ..., x T ] ∈
Z 5 × T is a topologically ordered representation of the architecture as per Section 3.1.
Each layer is represented by the hyperparameters of each layer.
Speciﬁcally, it is a discrete 5 dimensional vector as follows:
[Equation]
where t is the layer type, k the kernel size, o the number of output channels, s the stride and p the padding.
4. E NCODER AND DECODER
The encoder is a one-dimensional convolutional neural network that takes as input the discrete architecture representation and outputs a continuous feature.
Prior approaches use recurrent networks such as LSTMs or bidirectional RNNs where backpropagating through large sequences can result in vanishing gradients and difﬁcult credit assignment Hochreiter et al.
In our approach, we use a 1D-CNN encoder for several reasons.
It can operate on variable length sequences, CNNs with sufﬁcient depth have a large receptive ﬁeld over the input sequence and it does not suffer from vanishing gradients since the length of gradient propagation is no longer a function of the sequence length but a function of the depth of the inference network.
The encoder E , computes the function:
[Equation]
where x 1: T ∈ Z 5 × T is a variable length sequence representing the architecture and y ∈ R N a continuous feature embedding.
Details of the network can be found in the appendix.
The decoder has a similar residual 1D-CNN architecture as the encoder except that convolutions are replaced with transposed convolutions.
The decoder D , computes the function:
[Equation]
where x ∗ 1: T ∈ R 5 × T is a continuous variable length sequence representing the architecture and y ∈ R N a continuous feature embedding.
The continuous variable x ∗ 1: T is discretized to ˆ x 1: T ∈ Z 5 × T .
4. A CCURACY AND PARAMETER REGRESSORS
The accuracy regressor and parameter regressor are two fully connected networks that take the continuous embedding of the architecture y as input and predict the expected accuracy, E θ [ a ] and parameter count, p of the architecture A .
The accuracy regressor learns the function
[Equation]
(10) where a ∈ [0 , 1] ⊂ R is trained to predict a ∗ =
E θ [ a ] (which is approximated as per 3.2 by training the network with multiple intitializations).
This network consists of 3 fc-relu blocks with a sigmoid output activation to convert the output to [0 , 1] .
The parameter regressor learns the function
[Equation]
where p ∈ R + is trained to match the ground truth parameter count p ∗ .
The ground truth parameter count is computed as p
[Equation]
=
[Equation]
where | l i | is the number of parameters of each layer in the network.
The mean parameter count ¯ p is subtracted to center the parameter count around 0 and the standard deviation of the parameter count σ p , is used to scale the parameter count for better conditioning of the range of the output.
This network consists of 3 fc-relu blocks with a ReLU function to predict a non-negative real scalar.
4. O PTIMIZATION
In order to learn to encode/decode the architecture and predict accuracy/parameter count, we jointly optimize a multi-task loss function.
While all four tasks are formulated as regression problems, each has a differing domain.
To account for this, we choose appropriate loss functions to best suit each task.
The decoder is trained to approximate positive integral values of varying scale since we may have small values for padding, kernel size, type and stride and large values for number of outputs.
To enable faster convergence, we use a mean squared error.
[Equation] [Equation]
A standard L-1 loss is used for the accuracy regressor since the output of the sigmoid function is bounded between [0 , 1] .
[Equation]
For training the parameter regressor, we use the Huber loss with δ = 1 .
This loss is less sensitive to outliers in the data and does not suffer from exploding gradients as described in Girshick (2015).
This is important since the domain of the parameter count is R and it is possible that the training set may have outliers such as large networks that do not converge.
[Equation]
The network is then trained to minimize the joint loss function:
[Equation]
4. A RCHITECTURE COMPRESSION USING GRADIENT DESCENT
do 3:
4: p
← ( num params ( A
− ¯ p σ p 5: x ← o
6: y ← encoder θ 1 ( x ) 7: ˆ a ← acc regressor θ
( y ) 8: ˆ p ← param regressor θ y ) 9:
ˆ x ← decoder θ 4 ( y ) 10:
L = L d (ˆ x, x ) + λ 1 L a (ˆ a, a ∗ ) + λ 2 L p (ˆ p, p ∗ ) 11: θ = argmin θ L 12: end for 13: end procedure 14: procedure C OMPRESS ( A ) 15: x
A
16: y
17: ˆ a
L
[Equation] [Equation]
20: y ← y + η dL c dy 21: x comp ← decoder θ 4 ( y ) 22:
A comp ← o − 1 ( x comp ) 23: end procedure
Once the inference network has converged and can predict the accuracy and number of parameters of a network, we use the network for architecture compression.
In order to compress an architecture, we perform gradient descent on the continuous embedding space using the compression objective function.
[Equation]
The compression objective function simultaneously minimizes the number of parameters and maximizes the accuracy of the network to achieve compression.
We can also incorporate known hardware constraints p 0 or accuracy requirements a 0 to relax this objective function by modifying it as follows
[Equation]
The gradient from this objective is then backpropagated to the continuous embedding y with learning rate η to generate a new embedding y (cid:48) .
[Equation]
The decoder converts this embedding into a temporal sequence x comp that is then discretized and converted into an architecture A comp
=
o − 1 ( x comp ) .
E XPERIMENTS
In this section, we empirically evaluate our approach, showing that it is capable of compressing networks by upwards of 10x by benchmarking them on modern classiﬁcation tasks such as CIFAR-10,
CIFAR-100, FMNIST and SVHN.We show that our approach outperforms current state of the art architecture compression techniques by comparing the results to several baselines.
In the following experiments, we used a randomly generated dataset of 1500 architectures for training the encoder/decoder and regressors.
Each architecture was trained for each task for 5 epochs with m=5 different random initializations to obtain a target expected accuracy.
As observed in Ashok et al.
; Zoph & Le (2016); Tan et al.
(2018), 5 epochs of training seems to provide sufﬁcient signal about the convergence characteristics of the network.
All of the following experiments were run on 2x NVIDIA 1080TI GPUs.
Additional training details such as choice of optimizer and hyperparameters are included in Section 10 in the appendix.
5. D ATASETS
FMNIST The Fashion-MNIST Xiao et al.
(2017) dataset consists of 28 × 28 pixel grey-scale images depicting images of fashion products from 10 categories.
We use the standard 60,000 training images and 10,000 test images for experiments.
CIFAR-10 The CIFAR-10 Krizhevsky & Hinton (2009) dataset consists of 10 classes of objects and is divided into 50,000 train and 10,000 test images (32x32 pixels).
This dataset provides an incremental level of difﬁculty over the FMNIST dataset, using multi-channel inputs to perform model compression.
SVHN The Street View House Numbers Netzer et al.
(2011) dataset contains 32x32 colored digit images with 73257 digits for training, 26032 digits for testing.
This dataset is slightly larger that
CIFAR-10 and allows us to observe the performance on a wider breadth of visual tasks.
CIFAR-100 To further test the robustness of our approach, we evaluated it on the CIFAR-100 dataset.
CIFAR-100 is a harder dataset with 100 classes instead of 10, but the same amount of data, 50, train and 10,000 test images (32x32).
Since there is less data per class, there is a steeper size-accuracy tradeoff.
5. C OMPRESSION EXPERIMENTS
[Table]
In this section we evaluate the ability of our approach to compress architectures.
For our experiments, we use a standard convolutional architecture consisting of stacked conv-bn-relu blocks and a fully connected layer.
We use a 7 block network CONV-7 for Fashion-MNIST and 10 block network
CONV-10 for the others.
Table shows original average accuracy of the model on the validation set, followed by the number of parameters in the original model followed by two columns for the improvement in accuracy in the compressed architecture and the compression rate.
We notice a minor improvement in accuracy by the compressed architecture in Fashion-MNIST while observing a minor drop in the other datasets.
Additionally, our method is able to achieve solid compression on all the datasets and up to 20x compression on CIFAR-10 (1.2M parameters for ﬁnal compressed architecture).
5.2. B ASELINES
We compare our approach to various model compression baselines including reinforcement learning, pruning and knowledge distillation on the CIFAR-10 dataset.
The experiments show that our approach is able to ﬁnd a smaller and more accurate model than the other approaches.
[Table]
C ONCLUSION
In conclusion we have described an approach to compress CNNs by ﬁrst training a continuous embedding on a representation of the architecture and then performing gradient descent to determine an optimal architecture for the given task.
We also introduced a novel theoretical analysis of CNNs which we hope will inspire future work.
Finally, we also show that our method performs well over a variety of computer vision datasets
A PPENDIX 7. T OPOLOGICAL ORDERING
Lemma 7.1.
If G is a directed acyclic graph (DAG), then G has a topological ordering
Proof.
We prove this by induction on n , where n = | G | .
Base case:
For n = 1 , the statement is true since the topological ordering is G .
Hypothesis:
Assume that there exists a topological ordering on all DAGs G of size k .
Inductive step:
Given a DAG G (cid:48) with k + 1 nodes, select a node v with no incoming edges.
Then G ∗ =
G (cid:48) − { v } is a DAG since deleting v cannot induce a cycle on G (cid:48) .
Since | G ∗ | = k , the hypothesis implies that G ∗ has a topological ordering T ∗ .
Since v has no incoming edges, no partial order is violated by placing v at the head of the topologically ordered sequence.
Thus T =
[ v, T ∗ ] is a valid topological ordering of G (cid:48) and the lemma is proven.
Lemma 7.2.
If a topological sort l 1: T =
[ l 0 ...
l i ...
l T ] has the property that ∀ i, ∃ edge ( i, i + 1) , then it is unique
Proof.
It follows directly that the above property implies that there exists a Hamiltonian path on the graph since traversing the graph in the topologically sorted order satisﬁes each node being visited once.
We can then show that the existence of a Hamiltonian path in a DAG implies unique ordering.
Suppose a DAG has two Hamiltonian paths, then let x ∈ P 1 ,
y ∈ P 2 be the ﬁrst nodes on the paths ( P 1 , P 2 ) that differ.
This implies that there is a path from x → y that is a subpath of P 1 and a path from y → x that is a subpath of P 2 .
This implies a cycle in the graph, which is a contradiction.
7. V ISUALIZATION OF 1D-CNN FILTERS
D ETAILS ON 1-D CNN ARCHITECTURE
Let x i ∈
Z K × T be the ith activation or in the case of the ﬁrst layer, the discrete representation of an architecture.
Then a 1-d convolution operation applies a ﬁlter w ∈ R L × K to a window of L features in the input vector.
We then add a bias b ∈ R to this output, to produce a feature map h i .
Finally we apply a batch normalization function g , an activation function σ and add the residual to produce a new activation x i +1 as follows.
Appropriate padding is added to the output in order to preserve the dimension and allow element-wise addition of the residual.
[Equation]
In the case of a strided convolution where the temporal dimension of the output is smaller than the original input, we apply a convolution with the same stride s .
The operations then become the following.
[Equation]
where f s ( x, w, b ) performs convolutions on input x with stride s , weights w and bias b .  S AMPLING OF RANDOM ARCHITECTURES
In section 4.4, we describe a set of randomly generated architectures that are trained and then used to learn a continuous architecture search space.
In this section, we detail how this set of architectures is generated.
In order to generate a sensible set of architectures, we randomly sample architectures and then reject degenerate architectures.
First we sample the length of the architecture randomly from a discrete unifrom distribution l ∼
U (2 , 50) .
We set the lower bound to be 2 since we need at least one classiﬁcation layer (fully connected) and one convolution layer for the architecture to be valid.
Furthermore, we set the upperbound to 50 layers as an arbitrary limit that is within experimental constraints.
After this, we sample the 5 layer conﬁguration variables for each of the l layers randomly.
Speciﬁcally, t ∼ U (1 , 8) where:
Convolution 2.
MaxPool 3.
BatchNorm 4. ReLU 5.
Sigmoid 6.
Dropout 7. AveragePool 8.
Linear
Additionally, k ∼ U (1 , 10) , s ∼ U (1 , 10) , o ∼ U (16 , 4096) , p ∼ U (1 , 10) .
For layers that do not have certain conﬁguration variables (e.g. batch normalization has no stride), we set the variable to be 0.
For certain layers like MaxPool, ReLU or BatchNorm that do not change the number of output channels, we set the variable to the number of input channels.
Furthermore, We also enforce that the number of input channels to the ﬁrst is the same as the input of the task example and that the ﬁnal layer is a fully connected layer with the number of outputs corresponding to the number of classes in the task.
Lastly, we ﬁlter out degenerate architectures if they do not contain any convolutional layers, the output dimension is too small to proceed or the memory/computational footprint is too large to be practical.
T RAINING DETAILS 10. R ANDOMLY GENERATED ARCHITECTURES
This section describes the implementation details for the training procedure of the randomly generated architectures.
All the experiments used the Adam optimizer and were run on the PyTorch framework.
The same procedure was used to train the ﬁnal output architecture.
Fashion-MNIST
The architectures for FMNIST were trained for 50 epochs with a starting learning rate of 0.01.
The learning rate is reduced by a factor of 10 in the 30th epoch.
A batch size of 64 was used.
{+CIFAR-10/
+}
The architectures for CIFAR-10/100 were trained for 150 epochs with a starting learning rate of 0.001.
The learning rate is decreased by a factor of 10 in the 80th and 120th epochs.
Standard data augmentation with horizontal mirroring (p=0.5), random cropping with padding ofpixels and mean subtraction of (0.5, 0.5, 0.5).
A batch size of 128 was used.
SVHN The architectures for SVHN were trained for 150 epochs with a starting learning rate of 0.001.
The learning rate is decreased by a factor of 10 in the 80th and 120th epochs.
Mean subtraction of (0.5, 0.5, 0.5) and a batch size of 128 was used.
10. A RCHITECTURE COMPRESSION SEARCH
The encoder, decoder and regressors were optimized using the SGD optimizer with nesterov momentum=0.5 and a learning rate of 0.003.
The CIFAR-10/100 networks were trained jointly for 100000 iterations while the SVHN and FMNIST ones were trained for 80000 iterations.
Furthermore, learning rate decay was used with period 30 and decay multiplier 1/5.