F ALCON : F AST AND L IGHTWEIGHT C ONVOLUTION FOR C OMPRESSING AND A CCELERATING CNN
Anonymous authors
Paper under double-blind review
A BSTRACT
How can we efﬁciently compress Convolutional Neural Networks (CNN) while retaining their accuracy on classiﬁcation tasks?
A promising direction is based on depthwise separable convolution which replaces a standard convolution with a depthwise convolution and a pointwise convolution.
However, previous works based on depthwise separable convolution are limited since 1) they are mostly heuristic approaches without a precise understanding of their relations to standard convolution, and 2) their accuracies do not match that of the standard convolution.
In this paper, we propose F ALCON , an accurate and lightweight method for compressing CNN.
F ALCON is derived by interpreting existing convolution methods based on depthwise separable convolution using EHP, our proposed mathematical formulation to approximate the standard convolution kernel.
Such interpretation leads to developing a generalized version rank- k F ALCON which further improves the accuracy while sacriﬁcing a bit of compression and computation reduction rates.
In addition, we propose F ALCON -branch by ﬁtting F ALCON into the previous state-of-the-art convolution unit ShufﬂeUnitV2 which gives even better accuracy.
Experiments show that F ALCON and F ALCON -branch outperform 1) existing methods based on depthwise separable convolution and 2) standard CNN models by up to 8 × compression and 8 × computation reduction while ensuring similar accuracy.
We also demonstrate that rank- k F ALCON provides even better accuracy than standard convolution in many cases, while using a smaller number of parameters and ﬂoating-point operations.
I NTRODUCTION
How can we efﬁciently reduce size and energy consumption of Convolutional Neural Networks (CNN) while maintaining their accuracy on classiﬁcation tasks?
Nowadays, CNN is widely used in various areas including computer vision (Krizhevsky et al.
; Simonyan & Zisserman (2014);
Szegedy et al.
(2017)), natural language processing (Yin et al.
recommendation system (Kim et al.
(2016a)), etc.
In addition, model compression has become an important technique due to an increase in the model capacity and the number of parameters in CNN.
One recent and promising direction for compressing CNN is depthwise separable convolution (Sifre (2014)) which replaces standard convolution with depthwise and pointwise convolutions.
The depthwise convolution applies a separate 2D convolution kernel for each input channel, and the pointwise convolution changes the channel size using 1 × 1 convolution (details in Section 2.1).
Several recent methods (Howard et al.
; Sandler et al.
; Zhang et al.
based on depthwise separable convolution show reasonable performances in terms of compression and computation reduction.
However, existing approaches based on depthwise separable convolution have several crucial limitations.
First, they are heuristic methods, and their relation to the standard convolution is not clearly identiﬁed.
Second, due to their heuristic nature, generalizing the methods is difﬁcult.
Third, although they give reasonable compression and computation reduction, their accuracy is not sufﬁcient compared to that of standard-convolution-based models.
In this paper, we propose F ALCON , an accurate and lightweight method for compressing CNN.
F ALCON overcomes the limitations of the previous methods based on the depthwise separable convolution using the following two main ideas.
First, we precisely deﬁne the relationship between the standard convolution and the depthwise separable convolution using EHP (Extended Hadamard
Product), which is our proposed mathematical formulation to correlate the standard convolution kernel with the depthwise convolution kernel and the pointwise convolution kernel.
We then design
F ALCON by ﬁne-tuning and reordering the results of EHP to improve the accuracy of convolution operations.
Second, based on the precise deﬁnition, we generalize the F ALCON to design rank- k
F ALCON , which further improves accuracy while sacriﬁcing a bit of compression and computation reduction rates.
We also propose F ALCON -branch by ﬁtting F ALCON into the state-of-the-art convolution unit ShufﬂeUnitV2 which gives even higher accuracy.
As a result, F ALCON and F ALCON -branch provide a superior accuracy compared to other methods based on depthwise separable convolution, with similar compression and computation reduction rates, and rank- k F ALCON further improves accuracy, outperforming even the original convolution in many cases.
Our contributions are summarized as follows
: • Generalization.
We analyze and generalize depthwise separable convolution to our proposed EHP (Extended Hadamard Product) operation.
This generalization enables a precise understanding of the relationship between depthwise separable convolution and standard convolution.
Furthermore, with ﬁne-tuning operations, it leads to our proposed method F ALCON .
• Algorithm.
We propose F ALCON , a CNN compression method based on depthwise separable convolution.
F ALCON is carefully designed to compress CNN with little accuracy loss.
We also propose rank- k F ALCON to further improve the accuracy with a little sacriﬁce in compression and computation reduction rates.
F ALCON can be easily integrated into other architectures, and we propose F ALCON -branch which combines F ALCON with a branch architecture for a better performance.
We theoretically analyze the compression and computation reduction rates of F ALCON and other competitors.
• Experiments.
We perform extensive experiments and show that F ALCON 1) outperforms other state-of-the-art methods based on depthwise separable convolution for compressing CNN, and 2) provides up to 8 × compression and computation reduction compared to the standard convolution while giving similar accuracies.
Furthermore, we show that rank- k F ALCON provides even better accuracy than the standard convolution in many cases while using a smaller number of parameters and ﬂoating-point operations
The rest of this paper is organized as follows.
Section 2 explains preliminaries.
Section 3 describes our proposed method F ALCON .
Section 4 presents experimental results.
After discussing related works in Section 5, we conclude in Section 6.
P RELIMINARY
We describe preliminaries on depthwise separable convolution and methods based on depthwise separable convolution.
Symbols used in this paper are described in Table 4 of Appendix.
2. D EPTHWISE S EPARABLE C ONVOLUTION
Depthwise Separable Convolution (DSConv) consists of two sub-layers: depthwise convolution and pointwise convolution.
The architecture of each convolution layer in DSConv is illustrated in Figure 5(a).
Depthwise convolution (DWConv) kernel consists of several D × D 2-dimensional ﬁlters.
The number of 2-dimension ﬁlters is the same as that of input feature maps.
Each ﬁlter is applied on the corresponding input feature map, and produces an output feature map.
Pointwise convolution (PWConv), known as 1 × 1 convolution, is a standard convolution with kernel size 1.
DSConv is deﬁned as follows:
[Equation]
where D i,j,m and P m,n are depthwise convolution kernel and pointwise convolution kernel denotes intermediate feature maps
DSConv performs on
[Figure]
Input
[Figure]
Output
[Figure]
N Channels
N Channels BN
[Figure] [Figure]
Input
Standard Convolution
Output
Concat
M/2 Channels
Channel Shuffle
Channel Split
M/2Channels
M/2 Channels BN+Relu
M Channels M Channels
M Channels (
[Figure]
Pointwise Convolution
Input
Depthwise Convolution
Output
Concat
M/2 Channels
Channel Shuffle
Channel Split
M/2 Channels BN+Relu
M Channels
M Channels M Channels
M/2 Channels
M/2Channels
[Figure]
input feature maps I h
Then, DSConv performs PWConv on O (cid:48) h (cid:48) ,w (cid:48) ,m using equation 2, and generates output feature maps
2. M ETHODS B ASED ON D EPTHWISE S EPARABLE C ONVOLUTION
Several CNN methods have been proposed based on Depthwise Separable Convolution (DSConv) recently.
DSConv was ﬁrst introduced by Sifre (2014).
Chollet & Franois (2016) built Xception module using DSConv in a few layers.
Howard et al.
(2017) built Mobilenet with all convolution layers replaced by DSConv.
Sandler et al.
(2018) built MobileNetV2 with inverted bottleneck block, denoted as MobileConvV2 in this paper.
Zhang et al.
(2017) built a CNN model with Shufﬂenet
Unit, denoted as ShufﬂeUnit in this paper.
Ma et al.
(2018) improved Shufﬂenet by designing ShufﬂenetV2 Unit, denoted as ShufﬂeUnitV2 in this paper.
The architecture and detailed descriptions are in Figure 5 and Appendix D.
P ROPOSED M ETHOD
We describe F ALCON , our proposed method for compressing CNN.
We ﬁrst deﬁne Extended
Hadamard Product (EHP), a key mathematical formulation to generalize depthwise separable convolution, in Section 3.1.
We interpret depthwise separable convolution used in Mobilenet using EHP in Section 3.2.
We propose F ALCON in Section 3.3 and explain why F ALCON can replace standard convolution.
Then, we propose rank-k F ALCON , which extends the basic F ALCON , in Section 3.4.
We show that F ALCON can be easily integrated into a branch architecture to compress it with little sacriﬁce of accuracy in Section 3.5.
Finally, we theoretically analyze the performance of F ALCON in Appendix C.
3. E XTENDED H ADAMARD P RODUCT (EHP)
We deﬁne Extended Hadamard Product (EHP), a generalized elementwise product for two operands of different shapes, to generalize the formulation of the relation between standard convolution and depthwise separable convolution.
Before generalizing the formulation, we give an example of formulating the relation between standard convolution and depthwise separable convolution.
Suppose we have a 4-order standard convolution kernel K ∈ R
I × J × M × N , a 3-order depthwise convolution kernel D ∈ R I × J × M , and a pointwise convolution kernel P ∈ R M × N .
Let K i,j,m,n be ( i, j, m, n ) -th element of K , D i,j,
m be ( i, j, m ) -th element of D , and P m,n be ( m, n ) -th element of P .
Then, it can be shown that applying depthwise convolution with D and pointwise convolution with P is equivalent to applying standard convolution kernel K where K i,j,m,n = D
i,j,m · P m,n (see Section 3.2 for detailed proof).
⋯ 𝑁 𝐷 𝐷 𝑀 =
𝐷 𝐷 𝑀 ⊙ ’ 𝑁 𝑀
StandardConvolution
Depthwise Separable Convolution used in Mobilenet 𝓚 ∈ ℝ 𝑫×𝑫×𝑴×𝑵 𝓓 ∈ ℝ 𝑫×𝑫×𝑴 𝐏 ∈ ℝ 𝑴×𝑵 (a) ⋯ 𝑁 𝐷 𝐷 𝑀 =
Standard Convolution
Convolution of FALCON 𝐷 𝐷 𝑁 ⊙ ’ 𝑀 𝑁 𝑇𝑇 ),+,,,- 𝓚 ∈ ℝ 𝑫×𝑫×𝑴×𝑵 𝓓 ∈ ℝ 𝑫×𝑫×𝑵 𝐏 ∈ ℝ 𝑵×𝑴 (
[Figure]
Standard Convolution
Convolution of rank- FALCON (c)
To formally express this relation, we deﬁne Extended Hadamard Product (EHP) as follows.
Deﬁnition 1 (Extended Hadamard Product) .
Given p -order tensor D ∈ R I 1 ×···× I p − 1 × M and qorder tensor P ∈ R M × J 1 ×···× J q − 1 , the Extended Hadamard Product D (cid:12) E P of D and P is deﬁned to be the tensor K ∈ R I 1 × ...
× I p − 1 × M × J 1 × ...
× J q − 1 where the last axis of D and the ﬁrst axis of P are the common axes such that
[Equation]
for all elements of K .
[Equation]
Contrary to Hadamard Product which is deﬁned only if the shapes of the two operands are the same,
Extended Hadamard Product (EHP) deals with tensors of different shapes.
Now, we deﬁne a special case of Extended Hadamard Product (EHP) for a third-order tensor and a matrix.
Deﬁnition 2 (Extended Hadamard Product for a third order tensor and a matrix) .
Given a thirdorder tensor D ∈ R
I × J × M and a matrix P ∈ R M × N , the Extended Hadamard Product D (cid:12) E P is deﬁned to be the tensor K ∈ R
I × J × M × N where the third axis of the tensor D and the ﬁrst axis of the matrix P are the common axes such that
[Equation]
for all elements of K .
[Equation]
We will see that the depthwise separable convolution in Mobilenet can be easily expressed with EHP in Section 3.2; we also propose a new architecture F ALCON based on EHP in Section 3.3.
EHP is also a core operation that helps us understand other convolution architectures including MobilenetV and Shufﬂenet (see Appendix E).
3. D EPTHWISE S EPARABLE C ONVOLUTION AND EHP
In this section, we discuss how to represent the convolution layer of Mobilenet as Extended
Hadamard Product (EHP) described in Section 3.1.
We interpret the depthwise separable convolution, which is the convolution of Mobilenet, as an application of EHP.
This interpretation leads to designing a better convolution architecture F ALCON in Section 3.3.
We represent the relationship between standard convolution kernel K ∈ R D × D × M × N and depthwise separable convolution consisting of depthwise convolution kernel D ∈ R D × D × M and pointwise convolution kernel P ∈ R M × N using one EHP operation.
Figure 2(a) illustrates the relationship between standard convolution and depthwise separable convolution used in Mobilenet.
We show that applying depthwise separable convolution with D and P is equivalent to applying standard convolution with a kernel K which is constructed from D and P .
Theorem 1.
Applying depthwise separable convolution with depthwise convolution kernel D ∈ R D × D × M and pointwise convolution kernel P ∈ R M
×
N is equivalent to applying standard convolution with kernel K = D (cid:12) E P .
Proof.
See Appendix B.1.
3. FA ST AND L IGHTWEIGHT CON VOLUTION (F ALCON )
We propose F ALCON (FAst and Lightweight CONvolution), a novel lightweight convolution that replaces standard convolution.
F ALCON is an efﬁcient method with fewer parameters and computations than those that the standard convolution requires.
In addition, F ALCON has better accuracy than competitors while having similar compression and computation reduction rates.
The main idea of
F ALCON is 1) to carefully align depthwise and pointwise convolutions, and 2) initialize kernels using the convolution kernels of the trained standard model.
We observe that a typical convolution has more output channels than input channels.
In such a setting, performing depthwise convolution after pointwise convolution would allow the depthwise convolution to extract more features from richer feature space; on the other hand, performing pointwise convolution after depthwise convolution as in
Mobilenet only combines features extracted from a limited feature space.
Based on the observation,
F ALCON ﬁrst applies pointwise convolution to generate an intermediate tensor O (cid:48) ∈
R H
× W × N
and then applies depthwise convolution.
We represent the relationship between standard convolution kernel K ∈ R D × D × M × N and F ALCON by applying an EHP operation on pointwise convolution kernel P ∈ R N × M and depthwise convolution kernel D ∈ R D × D × N in Figure 2(b).
In F ALCON , the kernel K is represented by EHP of D and P as follows:
[Equation]
m · D i,j,n . where TT (1 , 2 , 4 , 3) indicates tensor transpose operation to permute the third and the fourth dimensions of a tensor.
Note that the common axis is the output channel axis of the standard convolution, unlike
EHP for depthwise separable convolution where the common axis is the input channel axis of the standard convolution.
As in Section 3.2, we show that applying F ALCON is equivalent to applying standard convolution with a specially constructed kernel.
Theorem 2.
F ALCON which applies pointwise convolution with kernel P ∈ R N × M and then depthwise convolution with kernel D ∈ R D
× D × N is equivalent to applying standard convolution with kernel K =
TT (1 , 2 , 4 , 3) ( D (cid:12) E P ) .
Proof.
See Appendix B.2.
Based on the equivalence, we initialize pointwise convolution and depthwise convolution kernels D and P of F ALCON by ﬁtting them to the convolution kernels of the trained standard model; i.e., D , P = arg min D (cid:48) , P (cid:48)
|| K − TT (1 , 2 , 4 , 3) ( D (cid:48) (cid:12) E P (cid:48) )
|| F .
After pointwise convolution and depthwise convolution, we add batch-normalization and ReLU activation function as shown in Figure 1(b).
We note that F ALCON signiﬁcantly reduces the numbers of parameters and FLOPs compared to standard convolution, which we discuss at Appendix C.
3. R ANK - k F ALCON
We propose rank- k F ALCON , an extended version of F ALCON that improves accuracy while sacriﬁcing a bit of compression and computation reduction rates.
The main idea is to perform k independent
F ALCON operations and sum up the result.
Then, we apply batch-normalization (BN) and ReLU activation function to the summed result.
Since each F ALCON operation requires independent parameters for pointwise convolution and depthwise convolution, the number of parameters increases and thus the compression and the computation reduction rates decrease; however, it improves accuracy by enlarging the model capacity.
We formally deﬁne the rank- k F ALCON with EHP as follows.
Deﬁnition 3 (Rank- k F ALCON with Extended Hadamard Product) .
Rank- k F ALCON expresses standard convolution kernel K ∈ R D
× D × M × N as EHP of depthwise convolution kernel D ( i ) ∈
R D × D × N and pointwise convolution kernel P ( i ) ∈
R N × M for i = 1 , 2 , ..., k such that
K = k (cid:88) i =
TT (
[Equation] [Equation]
[Equation]
( D ( i ) (cid:12) E P ( i ) ) s.t.
K i,j,m,n = k (cid:88) i =
P ( i ) n,m · D ( i ) i,j,n (cid:3)
Figure 2(c) illustrates the relation between standard convolution and rank- k F ALCON .
For each i = 1 , 2 , ..., k , we construct the tensor K ( i ) using EHP of the depthwise convolution kernel D ( i ) and the pointwise convolution kernel P ( i ) .
Then, we construct the standard kernel K by the elementwise sum of the tensors
K
( i ) for all i .
3. F ALCON -B RANCH
F ALCON can be easily integrated into a CNN architecture called standard convolution operation with a branch (StConv-branch), which consists of two branches: standard convolution on the left branch and a residual connection on the right branch (see Figure 1(c)).
Ma et al.
(2018) improved the performance of CNN by applying depthwise and pointwise convolutions on the left branch of
StConv-branch.
Since F ALCON replaces standard convolution, we observe that StConv-branch can be easily compressed by applying F ALCON on the left branch.
StConv-branch ﬁrst splits an input in half along the depth dimension.
A standard convolution operation is applied to one half, and no operation to the other half.
The two are concatenated along the depth dimension, and an output is produced by shufﬂing the channels of the concatenated tensor.
F ALCON -branch
(see Figure 1(d)) is constructed by replacing the standard convolution branch (left branch) of StConv-branch with F ALCON .
Advantages of F ALCON -branch are that 1) the branch architecture improves the efﬁciency since convolutions are applied to only half of input feature maps and that 2) F ALCON further compresses the left branch effectively.
F ALCON -branch is initialized by ﬁtting F ALCON to the standard convolution kernel of the left branch of StConv-branch.
E XPERIMENTS
We validate the performance of F ALCON through extensive experiments.
We aim to answer the following questions:
• Q1.
Accuracy vs. Compression (Section 4.3).
What are the accuracy and the compression tradeoffs of F ALCON , F ALCON -branch, and competitors?
Which method gives the best accuracy for a given compression rate?
• Q2.
Accuracy vs. Computation (Section 4.4).
What are the accuracy and the computation tradeoffs of F ALCON , F ALCON -branch, and competitors?
Which method gives the best accuracy for a given amount of computation?
• Q3.
Rank- k F ALCON (Section 4.5).
How do the accuracy, the number of parameters, and the number of FLOPs change as the rank k increases in F ALCON ?
4. E XPERIMENTAL S ETUP
Datasets.
We perform image classiﬁcation task on four famous datasets - CIFAR10, CIFAR100,
SVHN
[Table]
and ImageNet.
Detailed information of these datasets is described in Table
[Table] [Table]
[Table] [Table]
Models.
For CIFAR10, CIFAR100, and SVHN datasets, we choose VGG19 and ResNet34 to evaluate the performance.
We shrink the sizes of both models since the sizes of these three datasets are smaller than that of Imagenet.
In VGG19, we reduce the number of fully connected layers and the number of features in fully connected layers: three large fully connected layers (4096-4096-1000) in VGG19 are replaced with two small fully connected layers (512-10 or 512-100).
In ResNet34, we remove the ﬁrst 7 × 7 convolution layer and max-pooling layer since the input size ( 32 × 32 ) of these datasets is smaller than the input size ( 224 × 224 ) of ImageNet.
On both models, we replace all standard convolution layers (except for the ﬁrst convolution layer) with those of F ALCON or other competitors in order to compress and accelerate the model.
For ImageNet, we choose
VGG16 BN (VGG16 with batch normalization after every convolution layer) and ResNet18.
We use the pretrained model from Pytorch model zoo as the baseline model with standard convolution, and replace the standard convolution with other types of convolutions.
Competitors.
We compare F ALCON and F ALCON -branch with four convolution units consisting of depthwise convolution and pointwise convolution: DSConv, MobileConvV2, ShufﬂeUnit, and
ShufﬂeUnitV2 (see Figure 5, Section 2.1, and Appendix D for more details).
To evaluate the effectiveness of ﬁtting depthwise and pointwise convolution kernels to standard convolution kernel, we build EHP-in which is DSConv where kernels D and P are ﬁtted from the pretrained standard convolution kernel K ; i.e.,
[Equation]
Implementation.
We construct all models using Pytorch framework.
All the models are trained and tested on GeForce GTX 1080 Ti GPU.
4. F ITTING C ONVOLUTION U NIT INTO M ODEL
We evaluate the performance of F ALCON against DSConv, MobileConvV2, ShufﬂeUnit, and ShufﬂeUnitV2.
We take each standard convolution layer (StConv) as a unit, and replace StConv with those from F ALCON or other competitors.
We evaluate the classiﬁcation accuracy, the number of parameters in the model, and the number of FLOPs needed for forwarding one image.
We only explain how to apply F ALCON in this section.
The details of how to ﬁt other convolution units into the models are described in Appendix F.
F ALCON .
When replacing StConv with F ALCON , we use the same setting as that of StConv.
I.e., if there are BN and ReLU after StConv, we add BN and ReLU at the end of F ALCON ; if there is only
ReLU after StConv, we add only ReLU at the end of F ALCON .
This is because F ALCON is initialized by approximating the StConv kernel using EHP.
Using the same setting for BN and ReLU as StConv is more efﬁcient for F ALCON to approximate the StConv.
We initialize the pointwise convolution kernel and the depthwise convolution kernel of F ALCON by approximating the pretrained standard convolution kernel using EHP.
The approximation process is as follows: 1) we ﬁrst initialize the pointwise convolution kernel and the depthwise convolution kernel randomly, and 2) the pointwise convolution kernel and the depthwise convolution kernel are updated using gradient descent such that the mean squared error of their EHP product and the standard convolution kernel is minimized.
Rank- k F ALCON uses the same initialization method. 4. A CCURACY VS . C OMPRESSION
We evaluate the accuracy and the compression rate of F ALCON and competitors.
Table 2 shows the results on four image datasets.
Note that F ALCON or F ALCON -branch provides the highest accuracy in 7 out of 8 cases while using similar or smaller number of parameters than competitors.
Speciﬁcally, F ALCON and F ALCON -branch achieve up to 8 × compression rates with less than 1% accuracy drop compared to that of the standard convolution (StConv).
Figure 3 shows the tradeoff between accuracy and the number of parameters.
Note that F ALCON and F ALCON -branch show the best tradeoff (closest to the “best” point) between accuracy and compression rate, giving the highest accuracy with similar compression rates.
Ablation study. We perform an ablation study on two components: 1) the order of depthwise and pointwise convolutions, and 2) initialization.
We observe that with a similar number of parameters, 1) F ALCON always results in better accuracy than EHP-in, and 2) EHP-in always results
[Table]
in better accuracy than DSC.
Furthermore, F ALCON results in better accuracy than F ALCON without initialization in 6 out of 8 cases.
These observations prove our claims in Section 3.3 that 1) EHP-out (F ALCON ) is more efﬁcient than EHP-in, and 2) the ﬁtting and the initialization of kernels using EHP improves accuracy.
Additionally, we observe that overall performance is more sensitive towards ordering compared to initialization
[Table]
4. A CCURACY VS . C OMPUTATION
We evaluate the accuracy and the amount of computation of F ALCON and competitors.
We use the number of multiply-adds ﬂoating point operations (FLOPs) needed for forwarding one image to a model as the metric of computation.
Table 2 also shows the accuracies and the number of
FLOPs of methods on four image datasets.
Note that F ALCON or F ALCON -branch provide the highest accuracy in 7 out of 8 cases while using similar FLOPs as competitors do.
Compared to
StConv, F ALCON and F ALCON -branch achieve up to 8 × FLOPs reduction across different models on different datasets.
Figure 4 shows the tradeoff between accuracy and the number of FLOPs.
Note that F ALCON and F ALCON -branch show the best tradeoff (closest to the “best” point) between accuracy and computation, giving the highest accuracy with a similar number of FLOPs.
4. R ANK - K F ALCON
We evaluate the performance of rank- k F ALCON by increasing the rank k and monitoring the changes in the numbers of parameters and FLOPs.
In Table 3, we observe three trends as the rank k increases:
1) the accuracy becomes higher than that of rank-1 F ALCON , 2) the number of parameters increases,
[Figure] [Figure]
and 3) the number of ﬂoating point operations (FLOPs) increases.
Although the rank k that gives the best tradeoff of rank and compression/computation reduction varies, rank- k F ALCON improves the accuracy of F ALCON in all cases.
Especially, we note that rank F ALCON often gives even higher accuracy than the standard convolution, while using smaller number of parameters and FLOPs.
For example, rank-3 F ALCON applied to VGG19 on CIFAR100 dataset shows 1.31 percentage point higher accuracy compared to the standard convolution, with 2.8 × smaller number of parameters and 2.8 × smaller number of FLOPs.
Thus, rank- k F ALCON is a versatile method to further improve the accuracy of F ALCON while sacriﬁcing a bit of compression and computation.
R ELATED W ORK
Over the past several years, a lot of studies focused on compressing and accelerating DNN to reduce model size, running time, and energy consumption.
It is believed that DNNs are over-parameterized.
Weight-sharing (Han et al.
; Ullrich et al.
(2017); Chen et al. (2015); Choi et al.
; Agustsson et al.
(2017)) is a common compression
[Table]
method which stores only assignments and centroids of weights.
While using the model, weights are loaded according to assignments and centroids.
Pruning (Han et al.
; Li et al.
(2016)) aims at removing useless weights or setting them to zero.
Although weight-sharing and pruning can signiﬁcantly reduce the model size, they are not efﬁcient in reducing the amount of computation.
Quantizing (Courbariaux et al.
; Hou et al.
; Zhu et al.
the model into binary or ternary weights reduces model size and computation simultaneously: replacing arithmetic operations with bit-wise operations remarkably accelerates the model.
Layer-wise approaches are also employed to efﬁciently compress models.
A typical example of such approaches is low-rank approximation (Lebedev et al.
; Kim et al.
(2016b)
; Novikov et al. (2015)); it treats the weights as a tensor and uses general tensor approximation methods to compress the tensor.
To reduce computation, approximation methods should be carefully chosen, since some of approximation methods may increase computation of the model.
Compressing existing models has limitations since they are originally designed to be deep and large to give high accuracy.
A recent trend is to design a brand new architecture that is small and efﬁcient.
Mobilenet (Howard et al.
(2017)), MobilenetV2 (Sandler et al.
, Shufﬂenet (Zhang et al.
(2017)), and ShufﬂenetV2 (Ma et al.
(2018)) are the most representative approaches, and they use depthwise convolution and pointwise convolution as building blocks for designing convolution layers.
Our proposed F ALCON gives a thorough interpretation of depthwise convolution and pointwise convolution, and applies them into model compression, giving the best accuracies with regard to compression and computation.
C ONCLUSION
We propose F ALCON , an accurate and lightweight convolution method to replace standard convolution.
By interpreting existing convolution methods based on depthwise separable convolution using
EHP operation, F ALCON and its general version
rank- k F ALCON provide accurate and efﬁcient compression on CNN.
We also propose F ALCON -branch, a variant of F ALCON integrated into a branch architecture of CNN for model compression.
Extensive experiments show that F ALCON and its variants give the best accuracy for a given number of parameter or computation, outperforming other convolution models based on depthwise separable convolution.
Compared to the standard convolution, F ALCON and F ALCON -branch give up to 8 × compression and 8 × computation reduction while giving similar accuracy.
We also show that rank- k F ALCON provides better accuracy than the standard convolution, while using smaller numbers of parameters and computations.
A C ONVOLUTIONAL N EURAL N ETWORK
Convolutional Neural Network (CNN) is a type of deep neural network used mainly for structured data.
CNN uses convolution operation in convolution layers.
In the following, we discuss CNN when applied to typical image data with RGB channels.
Each convolution layer has three components: input feature maps, convolution kernel, and output feature maps.
The input feature maps I ∈
R H
× W × M and the output feature maps O ∈ R H (cid:48)
× W (cid:48)
× N are 3-dimensional tensors, and the convolution kernel K ∈ R D × D × M × N is a 4-dimensional tensor.
The convolution operation is deﬁned as:
[Equation]
where the relations between height h i and width w j of input, and height h (cid:48) and width w (cid:48) of output are as follows:
[Equation]
where s is the stride size, and p is the padding size.
The third and the fourth dimensions of the convolution kernel K must match the number M of input channels, and the number N of output channels, respectively.
Convolution kernel K can be seen as N 3-dimensional ﬁlters F n ∈ R D × D × M .
Each ﬁlter F n in kernel K performs convolution operation while sliding over all spatial locations on input feature maps.
Each ﬁlter produces one output feature map.
B P ROOFS OF T HEOREMS B. P ROOF OF T HEOREM
Proof.
From the deﬁnition of EHP, K i,j,m,n = D i,j,m · P m,n .
Based on equation 3, we replace the kernel K i,j,m,n with the depthwise convolution kernel D i,j,m and the pointwise convolution kernel P m,
[Equation]
[Equation]
where I h i ,w j ,m is the ( h i , w j , m ) -th entry of the input.
We split the above equation into the following two equations.
[Equation]
where O (cid:48) h (cid:48) ,w (cid:48) ,m ∈ R H (cid:48) × W (cid:48)
× M is an intermediate tensor.
Note that equation 5 and equationcorrespond to the depthwise convolution and the pointwise convolution, respectively.
Therefore, the output O (cid:48) h (cid:48) ,w (cid:48) ,m is equal to the output applying depthwise separable convolution used in Mobilenet.
B. P ROOF OF T HEOREM
Proof.
From equation 3, we replace the kernel K i,j,m,n with the pointwise convolution kernel P and the depthwise convolution kernel D .
[Equation]
where I h i ,w j ,m is the ( h i , w j , m ) -th entry of the input I .
We split the above equation into the following two equations.
[Equation]
where I , O (cid:48) , and O are the input, the intermediate, and the output tensors of convolution layer, respectively.
Note that equation 7 and equation 8 correspond to pointwise convolution and depthwise convolution, respectively.
Therefore, the output O h (cid:48) ,w (cid:48) ,n is equal to the output applying F ALCON .
C Q UANTITATIVE A NALYSIS
In this section, we evaluate the compression and the computation reduction of F ALCON and rank- k
F ALCON .
All the analysis is based on one convolution layer.
The comparison of the numbers of parameters and FLOPs of F ALCON and other competitors is in Appendix G.
C. F ALCON
We analyze the compression and the computation reduction rates of F ALCON in Theorems 3 and 4.
Theorem 3.
Compression Rate ( CR ) of F ALCON is given by
CR = # of parameters in standard convolution
[Equation]
parameters
[Equation]
ALCON
[Equation]
where D 2 is the size of standard kernel, M is the number of input channels, and N is the number of output channels.
Proof.
Standard convolution kernel has D 2 MN parameters.
F ALCON includes pointwise convolution and depthwise convolution which requires MN and D 2 N parameters, respectively.
Thus, the compression rate of F ALCON is CR
[Equation]
Theorem 4.
Computation Reduction Rate ( CRR ) of F ALCON is described as:
[Equation]
FLOPs in standard convolution
[Equation]
FLOPs
[Equation]
where H (cid:48) and W (cid:48) are the height and the width of output, respectively, and H and W are the height and the width of input, respectively.
Proof.
The standard convolution operation requires H (cid:48) W (cid:48)
D 2 MN FLOPs (Molchanov et al.
F ALCON includes pointwise convolution and depthwise convolution.
Pointwise convolution has kernel size D = 1 with stride s = 1 and no padding, so the intermediate tensor O (cid:48) has the same height and width as those of the input feature maps.
Thus, pointwise convolution needs
HWMN FLOPs.
Depthwise convolution has the number of input channel M = 1 , so it needs
H (cid:48) W (cid:48)
D 2 N FLOPs.
The total FLOPs of F ALCON is HWMN + H (cid:48) W (cid:48)
D 2 N , thus the computation reduction rate of F ALCON is CRR =
H (cid:48) W (cid:48)
D 2 MN
HWMN + H (cid:48) W (cid:48)
D 2 N .
C. R ANK - K F ALCON
We analyze the compression and computation reduction rates of rank- k F ALCON in Theorem 5.
Theorem 5.
Compression Rate ( CR k ) and Computation Reduction Rate ( CRR k ) of rank-k F ALCON are described as:
[Equation]
Proof.
The numbers of parameters and FLOPs increase for k times since rank-k F ALCON duplicates
F ALCON for k times.
Thus, the compression rate and the computation reduction rate are calculated as CR k = CR k and CRR k = CRR k .
D D ESCRIPTION OF R ELATED C ONVOLUTION U NITS
MobileNetV2.
Sandler et al.
(2018) proposed a new convolution architecture which we call as
MobileConvV2, in their MobilenetV2 model.
MobileConvV2 consists of three sub-layers as shown in Figure 5(b).
The ﬁrst and the third sub-layers are pointwise convolution for adjusting the number of channels.
The ﬁrst sub-layer expands the number of channels from M to tM , where t is an
[Figure]
Output
[Figure]
(a) Mobilenet (Howard et al.
[Figure]
Input
DepthwiseConvolution
Output
[Figure]
Add
[Figure] [Figure]
[Figure]
(Sandler et al.
[Figure]
Input
DepthwiseConvolution
Output
[Figure]
Add
[Figure] [Figure]
BN
[Figure]
Shuffle (c) Shufﬂenet (Zhang et al.
M/
[Figure]
Input
DepthwiseConvolution
Output
PointwiseConvolution
Concat
M/2Channels
M/
[Figure]
BN+Relu
Channel Shuffle
Channel Split
M/
[Figure]
Channels
M/2Channels BN+Relu
M Channels M Channels
M Channels (
[Figure] [Figure]
et al
[Figure]
expansion ratio.
The second sub-layer is a D × D depthwise convolution.
Since depthwise convolution cannot change the number of channels, the third sub-layer adjusts the number of channels from tM to N .
There is a shortcut connection between the input, and the output of the third sub-layer to facilitate ﬂow of gradient across multiple layers.
MobileConvV2 needs tM 2 + D 2 tM + tMN parameters and tHWM 2 + tH (cid:48) W (cid:48)
D 2 M + tH (cid:48) W (cid:48)
MN FLOPs.
ShufﬂeNet.
Zhang et al.
(2017) proposed a computation-efﬁcient CNN architecture named Shufﬂenet.
As shown in Figure 5(c), each unit of Shufﬂenet (we call it ShufﬂeUnit) consists of three sublayers, ﬁrst group pointwise convolution, depthwise convolution, and second group pointwise convolution, as well as a shortcut.
The number of depthwise convolution channels is 14 of output channels
N .
ShufﬂeUnit uses group convolution in two pointwise convolution layers to reduce the parameters and FLOPs.
However, it is hard to exchange information among groups when group convolutions are stacked.
To deal with this problem, ShufﬂeUnit adds a channel shufﬂe layer after the ﬁrst pointwise group convolution.
The channel shufﬂe layer rearranges the order of channels.
making it possible to obtain information from different groups.
The number of groups is represented as g .
ShufﬂeUnit needs 14 g MN + 14 D 2 N + 14 g N 2 parameters and 14 g HWMN + 14 H (cid:48) W (cid:48)
D 2 N + 14 g H (cid:48) W (cid:48) N 2 FLOPs.
ShufﬂenetV2.
Ma et al.
(2018) proposed a practically efﬁcient CNN architecture ShufﬂenetV2.
As shown in Figure 5(d), each unit of ShufﬂenetV2 (we call it ShufﬂeUnitV2) consists of two branches.
The left branch consists of two pointwise convolutions and one depthwise convolution like MobileConvV2, and the right branch is an identity operation.
Note that outputs of both branches maintain the number of channels as M/ 2 .
The ﬁnal output is produced by concatenating and shufﬂing the output tensors from both of the branches.
ShufﬂeUnitV2 needs 12 ( M 2 + D 2 M ) parameters and 12 HW ( M 2 + D 2 M ) FLOPs.
E G ENERALITY OF EHP
We show that EHP is a key operation to understand other convolution architectures based on depthwise separable convolution.
MobilenetV2.
As shown in Figure 5(b), MobilenetV2 has an additional pointwise convolution before depthwise convolution in Mobilenet: one layer of MobilenetV2 consists of two pointwise convolutions and one depthwise convolution.
In another point of view, MobilenetV2 can be understood as F ALCON followed by additional pointwise convolution; i.e., MobilenetV2 performs EHP operation as F ALCON does, and performs additional pointwise convolution after that.
Shufﬂenet.
As shown in Figure 5(c), Shufﬂenet consists of depthwise convolution and pointwise group convolution which is a variant of pointwise convolution.
We represent the convolution layer of
Shufﬂenet using EHP as follows.
Let g be the number of groups.
We divide the standard convolution kernel K ∈ R D
× D × M × N into g group standard convolution kernels.
Then, the relation of g -th group standard convolution kernel K g ∈ R D × D ×
Mg × Ng with regard to g -th depthwise convolution kernel D g ∈ R D × D × Mg and g -th pointwise group convolution kernel P g ∈ R
Mg × Ng is
K g
[Equation]
g
[Equation]
where m g = 1 , 2 , ..., Mg and n g = 1 , 2 , ..., Ng .
Each group standard convolution is equivalent to the combination of a depthwise convolution and a pointwise convolution, and thus easily expressed with EHP as in Mobilenet.
Therefore, each layer of Shufﬂenet is equivalent to the layer consisting of one group convolution followed by standard convolution.
ShufﬂenetV2.
As shown in Figure 5(d), the left branch of ShufﬂenetV2 has the same convolutions as in MobilenetV2: it consists of two pointwise convolutions and one depthwise convolution.
Like
MobilenetV2, the left branch of ShufﬂenetV2 can be understood as F ALCON followed by additional pointwise convolution.
F F ITTING O THER C ONVOLUTION U NITS INTO M ODELS
DSConv.
DSConv (shown in Figure 5(a)) has the most similar architecture as F ALCON among competitors, and thus DSConv has nearly the same number of parameters as that of F ALCON .
As in the setting of F ALCON , the existence of BN and ReLU at the end of DSConv depends on that of
StConv.
MobileConvV2.
In MobileConvV2 architecture (shown in Figure 5(b)), we adjust the numbers of parameters and FLOPs by changing the expansion ratio t as described in Appendix D, which is represented as ‘MobileConvV2- t ’.
We choose t = 0 .
5 as the baseline MobileConvV2 to compare with
F ALCON , since two pointwise convolutions bring lots of parameters and FLOPs to MobileConvV2.
ShufﬂeUnit.
In ShufﬂeUnit (shown in Figure 5(c)), we adjust the numbers of parameters and
FLOPs by changing the width multiplier α
(Howard et al.
and the number of groups g , which is represented as ‘ShufﬂeUnit α × (g= g )’.
Note that the width multiplier is used to adjust the number of input channels M and the number of output channels N of a convolution layer; if the width multiplier is α , the numbers of input and output channels become αM and αN , respectively.
While experimenting with ResNet, we ﬁnd that ShufﬂeUnit does not cooperate well with ResNet:
ResNet34 with ShufﬂeUnit does not converge.
We suspect that residual block and ShufﬂeUnit may conﬂict with each other because of redundant residual connections: the gradient may not ﬁnd the right path towards previous layers.
For this reason, we delete the shortcut of all residual blocks in
ResNet34 when using ShufﬂeUnit.
ShufﬂeUnitV2.
In ShufﬂeUnitV2 (shown in Figure 5(d)),we also adjust the number of parameters and FLOPs by changing the width multiplier α , which is represented as ’ShufﬂeUnitV2 α × ’.
Other operations of ShufﬂeUnitV2 stay the same as in Ma et al.
G P ARAMETERS AND FLOP S
We summarize the numbers of parameters and FLOPs for F ALCON and competitors in Table 5.
[Table]