N EURAL E PITOME S EARCH FOR A RCHITECTUREA GNOSTIC N ETWORK C OMPRESSION
Anonymous authors
Paper under double-blind review
A BSTRACT
Traditional compression methods including network pruning, quantization, low rank factorization and knowledge distillation all assume that network architectures and parameters are one-to-one mapped.
In this work, we propose a new perspective on network compression, i.e., network parameters can be disentangled from the architectures.
From this viewpoint, we present the Neural Epitome Search (NES), a new neural network compression approach that learns to ﬁnd compact yet expressive epitomes for weight parameters of a speciﬁed network architecture end-to-end.
The complete network to compress can be generated from the learned epitome via a novel transformation method that adaptively transforms the epitomes to match shapes of the given architecture.
Compared with existing compression methods, NES allows the weight tensors to be independent of the architecture design and hence can achieve a good trade-off between model compression rate and performance given a speciﬁc model size constraint.
Experiments demonstrate that, on ImageNet, when taking MobileNetV2 as backbone, our approach improves the full-model baseline by 1.47% in top-1 accuracy with 25% MAdd reduction, and with the same compression ratio, improves AutoML for Model Compression (AMC) by 2.5% in top-1 accuracy.
Moreover, taking EfﬁcientNet-B0 as baseline, our NES yields an improvement of 1.2% but has 10% less MAdd.
In particular, our method achieves a new state-of-the-art results of 77.5% under mobile settings ( < 350M MAdd).
Code will be made publicly available.
I NTRODUCTION
Despite the remarkable performance achieved in many applications, powerful deep convolutional neural networks (CNNs) typically suffer from high complexity (Han et al., 2015a).
The large model size and computation cost hinders their deployment on resource limited devices, such as mobile phones.
Very recently, huge efforts have been made to compress powerful CNNs.
Existing compression techniques can be generally categorized into four categories: network pruning (Han et al., 2015a; Collins & Kohli, 2014; Han et al., 2015b), low rank factorization (Jaderberg et al., 2014), quantization (Jacob et al., 2018;
Hubara et al., 2017; Rastegari et al., 2016), and knowledge distillation (Hinton et al., 2015; Papernot et al., 2016).
Network pruning targets on removing unimportant connections or weights to reduce the number of parameters and multiply-adds (MAdd).
Low rank factorization decomposes an existing layer into lower-rank and smaller layers to reduce the computation cost.
Weights quantization aims to use less number of bits to store the weights and activation maps.
Knowledge distillation uses a well trained teacher network to train a lightweight student network.
All of those compression methods assume that the model parameters (weight tensors) must have one-to-one correspondence to the architectures.
As a result, they suffer from performance drop since changing architectures will inevitably lead to loss of informative parameters.
In this paper, we consider the network compression problem from a new perspective where the shape of the weight tensors and the architecture are designed independently.
The key insight is that the network parameters can be disentangled from the architecture and can be compactly represented by a small-sized parameter set (called epitome), inspired by success of epitome methods in image/video modeling and data sparse coding (Jojic et al., 2003; Cheung et al., 2008; Aharon & Elad, 2008).
As shown in Figure 1, unlike conventional convolutional layers that use the architecture tied weight tensors to convolve with the input feature map, our proposed neural epitome search (NES) approach ﬁrst learns a compact yet expressive epitome along with an adaptive transformation function to
[Figure]
Weight
[Figure]
expand the epitomes.
The transformation function is able to generate a variety of parameters from epitomes via a novel learnable transformation function, which also guarantees the representation capacity of the resulting weight tensors to be large.
Our transformation function is differentiable and hence enables the NES approach to search for optimal epitome end-to-end, achieving a good trade-off between required model size and performance.
In addition, we propose a novel routing map to record the index mapping used for the transformation between the epitome and the convolution kernel.
During inference, this routing map enables the model to reuse computations when the expanded weight tensors are formed based on the same set of elements in the epitomes and therefore effectively reduces the computation cost.
Beneﬁting from the learned epitomic network parameters and transformation method, compared to existing compression approaches, NES has less performance drop.
To the best of our knowledge, this is the ﬁrst work that automatically learns compact epitomes of network parameters and the corresponding transformation function for network compression.
To sum up, our work offers the following attractive properties:
• Our method is ﬂexible.
It allows the weight tensors to be independent of the architecture design.
We can easily control the model size by deﬁning the size of the epitomes given a speciﬁed network architecture.
This is especially beneﬁcial in the context of edge devices.
• Our method is effective.
The learning-based transformation method empowers the epitomes with highly expressive capability and hence incurs less performance drop even with large compression ratios.
• Our method is easy to use.
It can be encapsulated as a drop in replacement to the current convolutional operator.
There is no dependence on specialized platforms/frameworks for NES to conduct compression.
To demonstrate the efﬁcacy of the proposed approach, We conduct extensive experiments on CIFAR10 (Krizhevsky & Hinton, 2009) and ImageNet (Deng et al., 2009).
On CIFAR-10 dataset, our method outperforms the baseline model by 1.3%.
On ImageNet, our method outperforms MobileNetV full model by 1 . 47% in top-1 accuracy with 25% MAdd reduction, and MobileNetV2-0.35 baseline by 3 . 78% .
Regarding MobileNetV2-0.7 backbone, our method improves AMC (He et al., 2018) by 2.47%.
Additionally, when taking EfﬁcientNet-b0 (Tan & Le, 2019) as baseline, we have an improvement of 1.2% top-1 accuracy with 10% MAdd reduction.
R ELATED WORK
Traditional model compression methods include network pruning (Collins & Kohli, 2014; Han et al., 2015b), low rank factorization (Jaderberg et al., 2014), quantization (Jacob et al., 2018;
Hubara et al., 2017; Rastegari et al., 2016) and knowledge distillation (Hinton et al., 2015; Papernot et al., 2016).
For all of those methods, as mentioned in Section 1 extensive expert knowledge and manual efforts are needed and the process might need to be done iteratively and hence is time consuming.
Recently, AutoML based methods have been proposed to reduce the experts efforts for model compression (He et al., 2018;
Zoph et al., 2018; Noy et al., 2019;
Li et al., 2019) and efﬁcient convolution architecture design (Liu et al., 2018; Wu et al., 2018; Tan et al., 2018).
As proposed in
AutoML for model compression (AMC (He et al., 2018)), reinforcement learning can be used as an agent to remove redundant layers by adding resource constraints into the rewards function which however is highly time consuming.
Later, gradient based search method such as DARTS (Liu et al., 2018) is developed for higher search efﬁciency over basic building blocks.
There are also methods that use AutoML based method to search for efﬁcient architectures directly (Wu et al., 2018;
He et al., 2018).
All of those methods are searching for optimized network architecture with an implicit assumption that the weights and the model architecture have one-to-one correspondence.
Different from all of the above mentioned methods, our method provides a new search space by separating the model weights from the architecture.
The model size can thus be controlled precisely by nature.
Our method is also related to the group-theory based network transformation.
Based on the group theory proposed in Cohen & Welling (2016), recent methods try to design a compact ﬁlter to reduce the convolution layer computation cost such as WSNet (Jin et al., 2017) and CReLU (Shang et al., 2016).
WSNet tries to reduces the model parameters and computations by allowing overlapping between adjacent ﬁlters of 1D convolution kernel.
This can be seen as a simpliﬁed version of our method as the overlapping can be regarded as a ﬁxed rule transformation.
CReLu tries to learn diversiﬁed features by concatenating ReLU output of original and negated inputs.
However, as the rule is ﬁxed, the design of those schemes are application speciﬁc and time consuming.
Besides, the performance typically suffers since the scheme is not optimized during the training.
In contrast, our method requires negligible human efforts and the transformation rule is learned end-to-end.
M ETHOD 3. O VERVIEW
A convolutional layer is composed of a set of learnable weights (or kernels) that are used for feature transformation.
The observation of this paper is that the learnable weights in CNNs can be disentangled from the architecture.
Inspired by this fact, we provide a new perspective on the network compression problem, i.e., ﬁnding a compact yet expressive parameter set, called epitome , along with a proper transformation function to fully represent the whole network, as illustrated in Figure 1.
Formally, consider a convolutional network with a ﬁxed architecture consisting of a stack of convolutional layers, each of which is associated with a weight tensor θ i ( i is layer index).
Further, let
L ( X, Y ; θ ) be the loss function used to train the network, where X and Y are the input data and label respectively and θ = { θ i } denotes the parameter set in the network.
Our goal is to learn epitomes
E =
{ E i } which have smaller sizes than θ and the transformation function τ = { τ i } to represent the network parameter θ with the compact epitome as τ ( E ) .
In this way, network compression is achieved.
The objective function of neural epitome search (NES) for a given architecture is to learn optimal epitomes E ∗ and transformation functions τ ∗ :
[Equation]
where | · | calculates the number of all elements.
The above NES approach provides a ﬂexible way to achieve network compression since the epitomes can be deﬁned to be of any size.
By learning a proper transformation function, the epitomes of predeﬁned sizes can be adaptively transformed to match and instantiate the speciﬁed network architecture.
In the following sections, we will elaborate on how to learn the transformation functions τ ( · ) , the epitomes , E ∈ R W E × H E × C
Ein × C Eout , and how to compress models via NES in an end-to-end manner.
In this paper, we use “a sub-tensor in the epitome” to describe a patch of the epitome that will be selected to construct the convolution weight tensor.
The sub-tensor, E s , is represented with the starting index and the length along each dimension in the epitome as shown below:
[Equation]
where ( p, q, c in , c out ) denote the starting index of the sub-tensor and ( w, h, β 1 , β 2 ) denotes the length of the sub-tensor along each dimension.
3. D IFFERENTIABLE S EARCH FOR E PITOME T RANSFORMATION
As aforementioned, NES generates convolution weight tensors from the epitome E via a transformation function τ .
In this section, we explain how the transformation τ is designed and optimized.
Spatial index
S i g m o i d 3 ×
C onv 1 ×
C onv
Epitome
Weight Tensor ( p 1 , q 1 ) ( p 2 , q 2 ) ( p 3 , q 3 ) ...
Convolution
Indexing learner
Concat
Input
Output
We start with the formulation of a conventional convolution operation.
Then we introduce how the transformed epitome is deployed to conduct the convolution operations, with a reduced number of parameters and calculations.
A conventional 2D convolutional layer transforms an input feature tensor F ∈ R W × H × C in to an output feature tensor G ∈ R W × H × C out through convolutional kernels with weight tensor θ ∈ R w × h × C in × C out .
Here ( W, H, C in , C out ) denote the width, height, input and output channel numbers of the feature tensor; w and h denote width and height of the convolution kernel.
The convolution operation can be formulated as:
[Equation]
Instead of maintaining the full weight tensor θ , NES maintains a much smaller epitome E that can generate the weight tensor and thus achieves model compression.
To make sure the generated weights τ ( E ) can conduct the above convolution operation without incurring performance drop, we carefully design the transformation function τ with following three novel components: (1) a learnable indexing function η to determine starting indices of the sub-tensor within the epitome E to sample the weight tensor; (2) a routing map M that records the location mapping from the sampled sub-tensors epitome to the generated weight tensor; (3) an interpolation-based sampler to perform the sampling from E even the indices are fractional.
We now explain their details.
The indexing function η is used to localize the sub-tensor within the epitome that is used to generate the weight tensor, as illustrated in Figure 2.
Concretely, given the input feature tensor F ∈ R W × H × C in , the function generates indices as follows,
[Equation]
where p , q , c in , c out are the learned starting indices along the spatial, input channel and ﬁlter dimensions respectively to sample the sub-tensor within epitome to generate the model weight tensors.
η is the index learner and it outputs the index ( p (cid:48) , q (cid:48) , c (cid:48) in , c (cid:48) out ) normalized by sigmoid, each ranging from 0 to 1. These outputs are further up-scaled by a scaling function S ( · ) to the corresponding dimension of the epitome by S ( · ) [Equation] ) where W E , H E , C Ein , C Eout are dimensions of the epitome and ⊗ denotes element-wise multiplication.
The learned indices are then fed into the following interpolation based sampler to generate the weight R cin = 2, β 1 = 4, C in =c in = 1. 0.2 × + 0.8 × c in = 2.6 (1 - 0.6) × + 0.6 ×
Concat H E W E C in
C in = 8 Epitome starting index
Weight tensor
E Channel index S i g m o i d 3 × C onv 1 ×
C onv
I nd e x i ng l ea r n e r h w E[:,:, 1:1+4,:] E[:,:, 2:2+4,:] tensor.
We implement the indexing learner by a two-layer convolution module that can be jointly optimized with the backbone network end-to-end. In particular, we use separate indexing learners and epitomes for each layer of the network.
More implementation details are given in Appendix A.3.
Routing map The routing map is constructed to record the position correspondence between the convolution weight tensor and the sub-tensor in the epitome.
It takes a position within the weight tensor as input and returns the corresponding index of the sub-tensor in the epitome.
The mapped starting index of the sub-tensor in the epitome can thus be retrieved from the routing map fast.
More importantly, the indexing learner can be removed during inference with the help of the routing map.
The routing map is built as a look-up table during the training phase by recording the moving average of the output index from η .
For example, the starting index pair as shown in Figure 2 can be fetched via ( p t , q t ) =
M ( i, j, m ) where ( i, j, m ) is the spatial location in the weight tensor and ( p t , q t ) is the starting index of the selected sub-tensor in the epitome at training epoch t .
The routing map M is constructed via Eqn.
(6) as shown below with momentum µ during the training phase.
µ is treated as a hyper-parameter and is decided empirically 1 :
[Equation]
Interpolation based sampler The learned starting index and the pre-deﬁned dimension ( w, h, β 1 , β 2 ) of the sub-tensor in the epitome is then fed into the sampler function. The sampler function samples the sub-tensor within the epitome to generate the weight tensor.
To simplify the illustration on the sampler function, we use the transformation along the spatial dimension as an example as shown in Figure 2.
The weight tensor is generated via the equation as shown below:
[Equation]
where G ( a, b ) = max(0 , 1 − | a − b | )
; n w and n h enumerate over all integral spatial locations within
E .
Eqn.
(7) depicts how to ﬁnd the two nearest integer index sub-tensor around the learned index and conducts a weighted summation over the involved epitome elements to generate θ .
When applying the sampling along the input channel dimension, the starting indices (p,q) in the equation is replaced with c in and iterating along the input channel dimension as shown below:
[Equation]
Where R cin =
(cid:100) C in /β 1 (cid:101) is the number of samplings applied along the input channel dimension.
An example of the generation process with Eqn.
(8) along the channel dimension can be found in Figure 3.
The transformation function can be applied in any dimension of the weight tensor.
Figureillustrates the transformation along the spatial dimension and Figure 3 shows the transformation along the input channel dimension.
The transformation along the ﬁlter dimension is the same as the transformation along the input channel dimension.
However, transformation along the ﬁlter dimension is easier for the computation reuse. We will show this in details in section 3.4.
3. L EARNING TO S EARCH E PITOMES E ND - TO - END
Beneﬁting from the differentiable Eqn.
(7), the elements in E and the transformation learner η can be updated together with the convolutional layers through back propagation in an end-to-end manner.
For the epitome E , as its transformed parameter E ( p,q ) can be used in multiple positions, the gradients of the epitomes are the summation of all the positions where the weight parameters are used.
Here, for clarity, we use { τ − 1 ( p, q ) } to denote the set of the indices in the convolution kernel that are mapped from the same position ( p, q ) in E .
Note that here we abuse the notion of (p,q) to denote the spatial position in the epitome.
The gradients of E ( p,q ) can, thus, be calculated as:
[Equation]
where θ z is the kernel parameters that are transformed from E ( p,q ) , and α z are the fractions that are assigned to E ( p,q ) during the transformation.
The epitome can thus be updated via Eqn.
[Equation]
) where we use η lr to denote the learning rate and subscript t to denote the training epoch.
Eqn.
(9) and (10) use the parameter updating rule along the spatial dimension as an example.
The above equations can be applied on any dimension by replacing the index mapping.
The indexing learner η can be simply updated according to the chain rule.
3. C OMPRESSION E FFICIENCY
Parameter reduction By using the routing map which records location mappings from the sub-tensor in the epitome to the convolution weight tensor, the indexing learner can be removed during the inference phase.
Thus, the total number of parameters during inference is decided by the size of the epitome and the routing map.
Recall that the epitome E is a four dimensional tensor with shape ( W E , H E , C Ein , C Eout ) .
The size of an sub-tensor in the epitome is denoted as ( w, h, β 1 , β 2 ) 2 where β 1 ≤ C Ein and β 2 ≤ C Eout .
The size of the epitome can be calculated as W E × H E × C
Ein × C Eout .
The size of the routing map M is calculated as 3 × R cin + R cout where R cout =
(cid:100) C out /β 2 (cid:101) is the number of starting indexes learned along the output channel dimension, and 3 × R cin =
3 × (cid:100) C in /β 1 (cid:101) is the number of starting index learned along the spatial and input channel dimension.
Note that we can enlarge the size of the sub-tensor in the epitome to reduce the size of the routing map.
Here, the size is referring to the number of parameters.
Detailed explanations of how R cin is calculated can be found in the Figure 3.
The parameter compression ratio r can thus be calculated via Eqn.
[Equation]
From Eqn.
(11), it can be seen that the compression ratio is nearly proportional to
[Equation]
the ratio between the size of the epitome and the generated weight tensor.
Detailed proof can be found in Appendix D.
The above analysis demonstrates that NES provides a precise control of the model size via the proposed transformation function.
Computation reduction As the weight tensor θ is generated from the epitome E , the computation in convolution can be reused when different elements in θ are from the same portion of elements in E .
Concretely, we propose two novel schemes to reuse the computation along the input channel
Learned Channel Wrapping
F
F F
F F
F
W 1 =
E
W 2 =
E
W 3 =
E
W 4 =
E
W 5 =
E
W 6 =
E
F 1 + F
F 2 + F
F 3 + F
E
E E
[Figure]
dimension and the ﬁlter dimension respectively. Channel wrapping .
During the inference, the computation along the input channel dimension is reduced with channel wrapping as illustrated in Figure 4.
For the elements in the input feature map that are multiplied with the same element in the epitome, we group the feature map elements ﬁrst and then multiplied with the weight tensor in the epitome as follows:
[Equation]
where R cin =
(cid:100) C in /C
Ein (cid:101) is the number of samplings (Eqn.
(8)) applied along the input channel dimension and m + c in + c (cid:48)
×
C Ein is the learned position with ( p, q, c in ) =
M ( i, j, m ) and c in ∈ [1 , C Ein ] .
This process is also illustrated in Figure 4.
Product map and integral map .
For the reuse along the ﬁlter dimension, given the routing map M for the transformation, we ﬁrst calculate the convolution results between the epitome and the input feature map once and then save the results as a product map P .
During inference, given P , the multiplication in convolution can be reused in a lookup table manner with O (1) complexity:
[Equation]
The additions in Eqn.
(13) can also be reused via an integral map I (Crow, 1984) as done in Viola et al.
With the product map and the integral map, the MAdd can be calculated as:
[Equation]
Hence, the computation cost reduction ratio can be written as:
MAdd
[Equation]
See more details and analysis in Appendix D.
Discussion.
We make a few remarks on the advantages of our proposed method as follows.
The proposed NES method disentangles the weight tensors from the architecture by using a learnable transformation function.
This provides a new research direction for model compression by bringing in better design ﬂexibility against the traditional compression methods on both sides of software and [Table] [Table] hardware.
On the software side, NES needs no special acceleration algorithms.
All the operations employed by NES are compatible with popular neural network libraries and can be encapsulated as a drop in operator.
On the hardware side, the memory allocation of NES is more ﬂexible.
This is especially helpful for hardware platform design where the off chip memory access is the main power consumption as demonstrated in Han et al.
NES provides a way to balance the computation/memory-access ratio in hardware: a smaller epitome with a complex transformation function results in a computation intense model while a large epitome with simple transformation function results in a memory intensive model.
Such ratio is an important hardware optimization criteria which however is not covered by most previous compression methods.
E XPERIMENTS
We ﬁrst evaluate the efﬁcacy of our method in 1D convolutional model compression on the sound dataset ESC-50 (Piczak, 2015) for the comparison with WSNet.
We then test our method with
MobileNetV2 and EfﬁcientNet as the backbone on 2D convolutions on ImageNet dataset (Deng et al., 2009) and CIFAR-10 dataset (Krizhevsky & Hinton, 2009).
Detailed experiments settings can be found in Appendix A. For all experiments, we do not use additional training tricks including the squeeze-and-excitation module (Hu et al., 2018) and the Swish activation function (Ramachandran et al., 2017) which can further improve the results unless those are used in the bachbone model originally.
The calculation of MAdd is performed for all convolution blocks.
We evaluate our methods in terms of three criteria: model size, multiply-adds(MAdd) and the classiﬁcation performance.
4. 1D CNN COMPRESSION
For 1D convolution compression, we compare with WSNet.
Similar to WSNet (Jin et al., 2017), we use the same 8-layer CNN model for a fair comparison.
The compression ratio in WSNet is decided by the stride ( S ) and the repetition times along the channel dimension ( C ), as shown in Table 1.
From
Table 1, one can see that with the same compression ratio, our method outperforms WSNet by 6.5% in classiﬁcation accuracy.
This is because our method is able to learn proper weights and learn a transformation rules that are adaptive to the dataset of interest and thus overcome the limitation of
WSNet where the sampling stride is ﬁxed.
More results can be found in Appendix B.
[Table]
4. 2D CNN COMPRESSION
Implementation details.
We use both MobilenetV2 (Sandler et al., 2018) and EfﬁcientNet (Tan &
Le, 2019) as our backbones to evaluate our approach on 2D convolutions.
Both models are the most representative mobile networks very recently and have achieved great performance on ImageNet and CIFAR-10 datasets with much fewer parameters and MAdd than ResNet (He et al., 2016) and
VGGNet (Simonyan & Zisserman, 2014).
For a fair comparison, we follow the experiment settings as in the original papers.
Results on ImageNet.
We ﬁrst conduct experiments on the ImageNet dataset to investigate the effectiveness of our method.
We use the same width multiplier as in Sandler et al.
(2018) as our baseline.
We choose four common width multiplier values, i.e. , 0 .
35 and 0 .
18 (Sandler et al., 2018; Zhang et al., 2018; Yu et al., 2018) for a fair comparison with other compression approaches.
The performance of our method and the baseline is summarized in Table 2.
For all the width multiplier values, our method outperforms the baseline by a signiﬁcant margin.
It can be observed that under higher compression ratio the performance gain is also larger.
Moreover, the performance of MobileNetV2 drops signiﬁcantly when the compression ratio is larger than 3 × .
However, our NES method increases the performance by 3.78% at a large compression ratio.
This is because when the compression ratio is high, each layer in the baseline model does not have enough capacity to learn good representations.
Our method is able to generate more expressive weights from the epitome with a learned transformation function.
We also compare our method with the state-of-the-art compression methods in Table 3.
Since an optimized architecture tends to allocate more channels to upper layers (He et al., 2018; Yu & Huang, 2019a), we also run experiments with larger size of the epitome for upper layers.
The results is denoted with sufﬁx ‘-A’ in Table 3.
Comparison with more models are shown in Figure 5.
As shown,
NES outperforms the current SOTA mobile model (less than 400M MAdd model) EfﬁcientNet-b by 1.2% with 40M less MAdd.
Obviously, our method performs even better than some NAS-based methods.
Although our method does not modify the model architecture, the transformation from the epitome to the convolution kernel optimizes the parameter allocation and enriches the model capacity through the learned weight combination and sharing. [Table] Our method [Table]
Results on CIFAR-10.
We also conduct experiments on CIFAR-10 dataset to verify the efﬁciency of our method as shown in Table 4.
Our method achieves 3 .
5 × MAdd reduction and 5.64 × model size reduction with only 1% accuracy drop, outperforming NAS-based AUTO-SLIM (Yu & Huang, 2019a).
More experiments and implementation details are shown in the supplementary material.
Discussion.
From the above results, one can observe signiﬁcant improvements of our method over competitive baselines, even the latest architecture search based methods.
The improvement of our method mainly comes from alleviating the performance degradation due to insufﬁcient model size by learning richer and more reasonable combination of weights parameters and allocating suitable weight parameters sharing among different ﬁlters.
The learned transformation from the epitome to the convolution kernel increases the weight representation capability with less increase on the memory footprint and the computation cost.
This distinguishes our method from previous compression methods and improves the model performance signiﬁcantly under the same compression ratio.
C ONCLUSION
We present a novel neural epitome search method which can reuse the parameters efﬁciently to reduce the model size and MAdd with minimum classiﬁcation accuracy drop or even increased accuracy in certain cases.
Motivated by the observation that the parameters can be disentangled form the architecture, we propose a novel method to learn the transformation rule between the ﬁlters to make the transformation adaptive to the dataset of interest.
We demonstrate the effectiveness of the method on CIFAR-10 and ImageNet dataset with extensive experimental results.
A I MPLEMENTATION DETAILS A. M OBILE N ET V2 S ETTINGS
Epitome dimensions for MobileNetV2 bottleneck.
With our NES method, the shape of the feature map produced by each layer can be kept the same as the ones from the original model before compression.
However, the number of channels in the feature map is reduced using the width multiplier method for MobileNetV2.
Hence, for a fair comparison, we only apply the width multiplier on the output dimension of the ﬁrst 1 × 1 convolutional layer and the input channel dimension of the second 1 × 1 convolutional layer within the bottleneck blocks of MobileNetV2 for obtaining the same feature map shape between blocks as our method.
Based on this principle, we generate weight tensor based on the epitome along the ﬁlter dimension for the ﬁrst 1 × 1 convolutional layers within the bottleneck and along the input channel dimension for the second 1 × 1 convolutional layers.
Speciﬁcally, we set the epitome shape per layer as (#in channels , #out channels × expansion multiplier , 1 , k ) for the ﬁrst 1 × 1 convolution layer and ( #in channels × expansion multiplier , #out channels , 1 , k ) for the second 1 × 1 layer as shown in Table 5.
Here, expansion is referring to the ratio between the input size of the bottleneck and the inner size as detailed in Figure 2 of (Sandler et al., 2018).
The shape represents the number of input channels, the number of output channels and the kernel size, respectively.
The compression ratio for each layer, c , can thus be calculated as c = 1 multiplier .
(Howard et al., 2017) t is the expansion ratio as deﬁned in MobileNetV2.
Input
Operators
Output
Epitome
Comp.
ratio h × w × k 1 × 1 , conv2d, ReLU h × w × tk w c × h c
× k × ctk w c
×
h c c hs × ws × tk 3 × 3 , depth-wise separable, ReLU hs × ws × tk – hs × ws × tk 1 × 1 , conv2d, linear hs × ws × k (cid:48)
w c × h c × ctk × k (cid:48)
w c × h c c
A. E PITOME D IMENSION D ESIGN
The size of the epitome can be calculated precisely by the original model and the desired compression ratio r .
For a CNN with C n -dimensional convolutional layers and K fully-connected layers, its number of parameters can be calculated as (cid:80) Ci =1 (cid:81)
d L id + (cid:80)
Kk =1 N kin N kout , where L id denotes the length of the convolution weight tensor along the d th dimension of the i th convolutional layer.
N kin and N kout denote the input and output dimension of the k th fully-connected layer.
We assign an epitome E j ∈ R W Ej × H E j for each layer j , and a routing map M : ( x 1 , x 2 , . . .
, x n ) → ( p, q ) , i.e. , the weight value of an n -d ﬁlter at location ( x 1 , x 2 , . . .
, x n ) being equal to E ( p, q ) .
We deﬁne the dimension of the epitome to be 2D here to illustrate the general case and later, we will show that in practice, the dimension of the epitome can be increased to save the computation memory.
The size of the total epitome is, thus, (cid:80) C +
K j =1 W
Ej
×
H
Ej .
After learning the routing map for layer j , we store the location mapping as a lookup table of size M j .
The size of all the mapping tables is (cid:80) C
+ K j =1 M j .
Hence, the compression ratio can be calculated as
[Equation]
[Equation]
In our analysis, we use a uniform compression ratio for all the layers.
Therefore, given a compression ratio r , the size of the epitome for each layer can be calculated accordingly.
This deterministic design of the epitome size is hardware friendly and can be used to control the memory allocation.
Epitome patch design We set the patch size along each dimension to be w, h, β 1 and β 2 .
Note that the transformation along each dimension are independent and hence can be conducted separately.
The starting index of the transformation along the spatial dimension and the input channel dimension are learned in pairs.
This is because the transformation along the spatial dimension will also increase the input channel dimension.
A. I NDEX SEARCH SPACE IN E PITOME
The selection space of the starting index for the transformation is not all the indices available along the channel dimension.
We partition the channels into groups to build a super-index with a group length l g .
For example, for an epitome that has C channels along the input channel dimension, the potential channel index ranges from 0 to C - 1.
With our super-index scheme, adjacent l g channels are grouped as a single index C g and thus, the selection space of the index ranges from 0 to C/l g − 1 .
The range of the super-index is used to scale the output from the transformation learner.
B M ORE R ESULTS ON 1D C ONVOLUTION C OMPRESSION
We also conduct experiments to examine the highest compression ratio that our method can achieve without performance drop compared to WSNet (Jin et al., 2017).
For a fair comparison, we choose the same 8-layer
CNN model backbone as used in WSNet.
Conﬁguration details have been demonstrated in Table 1 in the formal paper.
The results are shown in Table 6.
[Table]
C NES FOR F ULLY -C ONNECTED L AYER
Let D ∈ R N in × N out denotes the parameter matrix for a fully-connected (FC) layer.
We could use Eqn.
(18) to sampling along the input dimension and Eqn.
(23) to sample along the output dimension.
We also conduct experiments on CIFAR-10 dataset to verify the efﬁcacy of our method on FC layers as shown in Table
[Table]
Our method
[Table]
D P ROOF ON P ARAMETER AND C OMPUTATION R EDUCTION
With the epitome E as deﬁned in the main text, our method introduces a novel transformation function where the convolution ﬁlter weights are transformed from E with τ ( · ) .
In this section, we start with the most general situation where the epitome is two-dimensional, E ∈ R W E × H E , and we will show that increasing the dimension of epitome can reduce the computation memory cost aggressively.
Our method can be extended to n -dimension convolution and fully-connected layers straightforwardly.
The transformation process along the input channel dimension is illustrated in Figure {+
+}
In our method, all the weight parameters θ i,j,m,c are transformed from the compact epitome E .
The convolution with NES is shown as below:
[Equation]
Proof on computational cost reduction. Since the weight elements per convolutional layer are formed based on the same E , there is computational redundancy when two convolution kernels are selected from the Learned Channel Wrapping
F F
F F
F F
W 1 =
E
W 2 =
E
W 3 =
E
W 4 =
E
W 5 =
E
W 6 =
E
F 1 + F
F 2 + F
F 3 + F
E
E E
[Figure]
same portion of E as shown in Figure 2.
To reuse the multiplication in convolution, we ﬁrst compute the multiplication between each element in the epitome E and the input feature map.
The results are saved as a product map P such that the computations are done only once.
However, given the epitome E ∈ R W E × H E , the product map size is W × H × C in × W E × H E × C out which consumes large computational memory.
To reduce its size, we propose to increase the dimensions of the epitome to E ∈ R W E
× H E × C
Ein × C Eout in order to group the computation results in the product map.
Each entry in the product map P is calculated as the dot product between the channel dimension along the input feature map and the third dimension along the compact weight matrix.
We set C Ein to be smaller than C in to further boost the compression and β 1 equal to C Ein .
As illustrated in Figure 4, the input channels of the feature map is ﬁrst grouped by
[Equation]
where R cin =
(cid:100) C in /C
Ein (cid:101) is the compression ratio along the input channel dimension and m + c in + c
× C Ein is the learned position with ( p, q, c in , n ) =
M ( i, j, c, m ) , where c in ∈ [1 , C Ein ] .
The transformed input feature map ˜ F has the same number of channels as C Ein .
Let ( i, j, m ) index the transformed feature map location.
The product map is then calculated as
[Equation]
where · denotes the dot product operation.
Now, the multiplications can be reused by replacing the convolution kernels with the product map P .
Based on Eqns.
(17), (18), (19), the convolution can be reduced by
[Equation]
To reuse the additions, we adopt an integral image I which is proposed in (Crow, 1984) but differently we extend the integral image dimension to make it suitable for 2D convolution based on P .
Our integral image can be constructed by
[Equation]
 
[Equation]
else
[Equation]
From Eqn.
(21), the 2D convolution results can be retrieved in a similar way to (Jin et al., 2017) as follows:
G t w ,t h ,p,q,n = I ( t w + w − 1 , t h
+ h − 1 , p + w − 1 , q + h − 1 , n ) − I ( t w − 1 , t h − 1 , p − 1 , q − 1 , n − 1) .
As we set C Eout to be smaller than the output channel dimension of the convolution kernel, we reuse the computation results from the epitome via
[Equation]
where r out ∈ { 0 , 1 , ..., R cout − 1 } , R cout = (cid:100) C out /β 2 (cid:101) is the number of samplings conducted along the output channel dimension, and n = M ( r out × β 2 ) is the learned mapping along the ﬁlter dimension.
The ﬁlter length β 2 ∈ { 1 , 2 , ..., C Eout } is a hyper-parameter and is decided empirically.
In our experiments, we choose β 2 = C out .
Thus, the MAdds can be calculated as
[Equation]
Suppose we use sliding window with a stride of 1 and no bias term, the MAdds of the conventional convolution can be calculated based on Eqn.
(3) as shown below:
[Equation]
Therefore, the total MAdd reduction ratio is
MAdd Reduction Ratio =
[Equation]
Proof on parameter reduction.
The parameter compression ratio for a 2D convolution layer can be calculated as follows:
[Equation]
where w and h denote the width and height of the kernel of the layer.
W c and H c denote the spatial size of the corresponding epitome.
From Eqn.
(26) and Eqn.
(27), it can be observed that the compression ratio is mainly decided by C out C in [Equation]