D YNAMIC E ARLY T ERMINATING OF M ULTIPLY A CCUMULATE O PERATIONS FOR S AVING C OMPUTATION C OST IN C ONVOLUTIONAL N EURAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Deep learning has been attracting enormous attention from academia as well as industry due to its great success in many artiﬁcial intelligence applications.
As more applications are developed, the need for implementing a complex neural network model on an energy-limited edge device becomes more critical.
To this end, this paper proposes a new optimization method to reduce the computation efforts of convolutional neural networks.
The method takes advantage of the fact that some convolutional operations are actually wasteful since their outputs are pruned by the following activation or pooling layers.
Basically, a convolutional ﬁlter conducts a series of multiply-accumulate (MAC) operations.
We propose to set a checkpoint in the MAC process to determine whether a ﬁlter could terminate early based on the intermediate result.
Furthermore, a ﬁne-tuning process is conducted to recover the accuracy drop due to the applied checkpoints.
The experimental results show that the proposed method can save approximately 50% MAC operations with less than 1% accuracy drop for CIFAR- 10 example model and Network in Network on the CIFAR- 10 and CIFAR- 100 datasets.
Additionally, compared with the state-ofthe-art method, the proposed method is more effective on the CIFAR- 10 dataset and is competitive on the CIFAR- 100 dataset
I NTRODUCTION
In recent years, deep learning with deep neural network (DNN) has been a breakthrough solution in machine learning.
Its success has been demonstrated in many artiﬁcial intelligence (AI) applications, such as image classiﬁcation (Krizhevsky et al.
(2012)), face recognition (Learned-Miller et al.
(2016)), object detection (Redmon et al.
big data analysis (Gheisari et al.
(2017)), medical science (Litjens et al.
(2017)), etc.
Due to the high computational complexity, a DNN is usually implemented on a powerful hardware device, such as a graphics processing unit workstation.
However, considering the diversity of AI applications, it is desirable to implement a neural network model on an edge device which usually has limited hardware resources for saving power and cost.
To this end, many model compression and acceleration methods were proposed to compress neural network models.
Their common objective is to reduce the computational complexity and/or memory usage without signiﬁcantly affecting the performance and accuracy.
The existing methods have been classiﬁed into four categories (Cheng et al.
): (1) parameter pruning (Han et al.
(2015a)
; Han et al.
(2015b)), (2) low-rank factorization (Jaderberg et al.
(2014); Lebedev et al.
; Tai et al.
(2015)), (3) convolutional ﬁlter transformation (Szegedy et al.
and (4) knowledge distillation (Hinton et al.
The parameter pruning-based methods eliminate noncritical parameters for simplifying models.
The low-rank factorization-based methods exploit low-rank matrix approximation to simplify convolutional ﬁlters.
The convolutional ﬁlter transformation-based methods transform ﬁlters based on equivariance properties to compress convolutional neural network (CNN) models.
The knowledge distillation-based methods train a compact neural network to mimic the function of a large network with the distilled knowledge from the large network.
Most compression methods ﬁxedly modify the structure of a model.
However, the criticality of a parameter or a ﬁlter heavily depends on the input data.
A critical ﬁlter may become non-crucial under different input cases.
These static methods may not be comprehensive enough when con- sidering diverse input data.
Thus, in this paper, we propose a dynamic optimization method for
CNNs.
The proposed method does not modify the structure of a model, but introduces a speciﬁc multiply-accumulate (MAC) unit to reduce the MAC operations dynamically in the inference phase.
In a CNN, a convolutional layer is usually followed by an activation layer and a pooling layer.
The activation layer introduces non-linearity into the system.
Rectiﬁed linear unit (ReLU) is the most popular activation function, which applies f ( x ) = max (0 , x ) to rectify the feature maps from the convolutional layer.
Since negative values are changed to 0 , the computation efforts of the ﬁlters that output negative values in the convolutional layer can be seen as wasteful.
Furthermore, a pooling layer reduces the scale of the feature maps by extracting only signiﬁcant data.
The most common pooling method, max pooling, preserves the data with the largest value.
Similarly, if the output value of a convolutional ﬁlter is not large enough to pass through the max pooling layer, the computation efforts of the ﬁlter are wasteful as well.
Based on the observations, if we are able to detect in advance that a convolutional ﬁlter will output a negative value, we can stop it early for saving computation efforts.
A convolutional ﬁlter basically conducts a series of multiply-accumulate (MAC) operations.
Since the output is an accumulated result, it could be feasible that using an intermediate result in the MAC process to predict the sign of the output.
Thus, we propose to determine whether a convolutional ﬁlter should continue or terminate according to an intermediate result.
In the proposed optimization method, a convolutional ﬁlter is given a checkpoint.
When the checkpoint iteration is reached, we check the intermediate result of the MAC process.
If the result is less than 0 , we stop the MAC process to save the remaining operations and output the intermediate result directly; otherwise, the MAC process ﬁnishes all the MAC operations.
Since saving the MAC operations may affect the accuracy, we further ﬁne-tune the optimized CNN to improve its accuracy with the applied checkpoints.
Unlike the previous static optimization methods, which explicitly compress a CNN before the inference phase, the proposed method is dynamic.
It determines the saved computation efforts during the inference phase, and the saved efforts vary for different input data.
Furthermore, unlike the previous methods (Viola & Jones (2001); Teerapittayanon et al. that terminate the whole network and make a prediction directly at an early layer, the proposed method focuses on early stopping the
MAC processes in the convolutional ﬁlters.
Basically, each input data still needs to go through the whole network for making the prediction.
In the experiments, we apply the proposed method to CIFAR- 10 example model 1 (Jia et al.
and Network in Network (Lin et al.
(2013)), with the CIFAR- 10 and the CIFAR- 100 datasets (Krizhevsky (2009)).
For CIFAR- 10 example model, the results show that our method saves 50 .
MAC operations with only 0 .
8% accuracy drop under the CIFAR- 10 dataset, and saves 43 .
MAC operations with only 0 .
09% accuracy drop under the CIFAR- 100 dataset.
For Network in
Network, our method saves 47 .
43% MAC operations with only 0 .
41% accuracy drop under the
CIFAR- 10 dataset, and saves 47 .
73% MAC operations with only 0 .
58% accuracy drop under the
CIFAR- 100 dataset.
Furthermore, our method is competitive with the existing methods (He et al.
(2017); Li et al.
For the CIFAR- 10 dataset, our method even saves more MAC operations with less accuracy drop.
In summary, we propose a dynamic optimization method for CNNs, which reduces MAC operations in the inference phase.
It effectively saves the computation cost of a CNN with only a small effect on the accuracy.
Additionally, it allows a user to trade off the saved computation cost against the accuracy drop.
The remainder of this paper is organized as follows: Section 2 reviews some background and related works.
Section 3 shows a motivational example and illustrates the key idea behind the proposed method.
Section 4 presents the proposed method.
Section 5 shows the experimental results.
Finally, the conclusion is presented in Section
P RELIMINARIES 2. CNN AND CONVOLUTIONAL FILTER
CNNs have been extensively used in many AI ﬁelds, such as computer vision and natural language processing.
A CNN is a multi-stage model.
In general, a stage is composed of a convolutional layer, an activation layer and a pooling layer.
The convolutional layer applies convolution operations to the input feature maps.
The following activation and pooling layers rectify the feature maps and reduce their scale, and pass the outputs to the next stage.
Additionally, one or multiple fully connected layers are applied to the last few layers of a CNN.
Convolutional layers are the core of a CNN.
A convolutional layer is made up of a set of learnable ﬁlters and a ﬁlter is composed of weights and a bias.
The functionality of a ﬁlter at the l th layer can be mathematically formulated as EQ 1, where, K , W , and H are the depth, width, and height of the ﬁlter, respectively.
z l − 1 p,q,k is an input from the l − 1 th layer and h p,q,k is the corresponding weight in the ﬁlter.
Additionally, b is the bias.
There are a total of H × W × K + 1 MAC operations for the ﬁlter to generate an output value.
For convenience, we call the process of conducting the
H × W × K + 1 MAC operations a MAC process .
Depending on the size of the input feature maps, a ﬁlter usually needs to conduct many MAC processes for generating the outputs.
[Equation]
Basically, the convolutional layers dominate the computation efforts of a CNN.
Thus, we propose to reduce the MAC operations for saving computation efforts.
2. R ELATED WORKS
In recent years, many efforts have been devoted to simplifying the computational complexity of
CNNs.
Most methods aim to simplify a CNN model by removing redundancies or noncritical pa-
rameters without affecting the overall accuracy
The low-rank factorization-based methods approximate convolution operations by decomposing a weight matrix into a product of low-rank matrices (Jaderberg et al.
(2014); Lebedev et al.
Tai et al. (2015); Sainath et al.
; Kim et al.
; Chung & Shin (2016)).
As a result, the convolutional layers are simpliﬁed, but the ﬁlter count does not change.
Pruning parameters was initially to prevent a trained model from overﬁtting to the training dataset (Hassibi & Stork (1993); LeCun et al.
However, researchers then observed that there may exist redundant or noncritical parameters in a model.
Thus, some works propose to eliminate connections between layers by pruning small-magnitude weights (Han et al.
(2015a)
; Han et al.
(2015b);
Han et al.
However, the simpliﬁcation does not necessarily reduce computation time and the irregular sparsity in the convolutional layers requires speciﬁc implementation.
Instead of pruning weights, recent methods prune ﬁlters directly without introducing irregular sparsity (Li et al.
; He et al.
Li et al.
propose to prune ﬁlters that are identiﬁed as noncritical to the output accuracy (Li et al.
The method signiﬁcantly reduces the computation efforts by removing whole ﬁlters and the corresponding feature maps.
Furthermore, instead of analyzing ﬁlter weights, the state-of-the-art method proposed by He et al.
exploits redundancies in feature maps to prune ﬁlters and feature maps, and reconstruct the following feature maps to the next layer
(He et al.
Unlike the previous methods, the proposed method does not explicitly modify the structure of a model.
Since the criticality of a weight/ﬁlter may differ for different input feature maps, the proposed method dynamically identiﬁes the ﬁlters to be optimized based on the input feature maps.
Additionally, a weight ﬁne-tuning process is used to avoid signiﬁcant accuracy drop.
[Table]
M OTIVATIONAL EXAMPLE
In a convolutional layer, a ﬁlter receives inputs from the previous layer and conducts a series of MAC operations to generate outputs to the next layer.
If an output value is not large enough, it could be blocked by the following layers, and thus is not responsible for the CNN output.
The key idea of the proposed method is to use an intermediate result of a MAC process to determine whether the process should stop and output the intermediate result directly, or it should complete the computation.
If the intermediate result is small enough, the output value has a higher probability to be blocked and terminating the process can save MAC operations with little effect on the CNN output.
We conducted a simple experiment on the ﬁrst convolutional layer of CIFAR- 10 example model (Jia et al.
with 1000 input images from the CIFAR- 10 dataset (Krizhevsky (2009)) to explore the feasibility of the idea.
The ﬁrst convolutional layer has 32 ﬁlters and each has a size of 5 × 5 × 3 .
A ﬁlter’s MAC process consists of 76 MAC operations, i.e., 76 iterations.
The extra 1 is for the following bias.
For each ﬁlter, we sort the weights in the decreasing order of magnitudes, so that the MAC operation with a larger-magnitude weight is computed ﬁrst.
Table 1 shows the probability that MAC processes with a negative intermediate result at the i th iteration truly output a negative value ( MAC TN ), and the percentage of saved MAC operations if the processes with a negative intermediate result are terminated accordingly ( MAC SV ).
As expected, MAC TN increases and MAC SV decreases as i increases.
That is, if we decide whether a MAC process should stop or not at an earlier iteration, we can save more MAC operations, but we have a lower probability to correctly terminate the process.
However, from Table 1, it is observed that even we make a decision at the 3 rd iteration, the probability that the decision is correct is more than 60% .
The probability is acceptable, especially when we consider the possibility that the effect of a wrong decision might be blocked by the following layers.
Therefore, the strategy of early terminating a MAC process based on an intermediate result should be promising.
Next, our objective is to develop a method to determine at which iteration the intermediate result should be considered for maximizing the saved MAC operations with minimal accuracy drop.
P ROPOSED METHOD
The proposed method saves MAC operations by taking the advantage of the fact that the computations in some ﬁlters could be wasteful, since the results might be set to 0 or blocked by the following activation or max pooling layers.
In the following subsections, we ﬁrst present the procedure of the proposed two-step method.
Then, we analyze the method and discuss the ideas behind it.
In the ﬁrst step, we determine the checkpoint of each ﬁlter, i.e., the iteration at which we check the intermediate result.
Then, in the second step, we ﬁne-tune the model with the applied checkpoints.
After optimization, each ﬁlter has a checkpoint.
During the inference phase, when a MAC process reaches the checkpoint iteration, we check the intermediate result.
If it is less than 0 , we terminate the process and the ﬁlter directly outputs the intermediate result; otherwise, it completes the MAC process.
4. S TEP 1 : S ORT WEIGHTS AND SET UP CHECKPOINTS
Given a pre-trained CNN model, we ﬁrst sort the weights of each ﬁlter in the decreasing order of magnitudes.
Then, we determine the checkpoints based on a user-deﬁned parameter e t , which speciﬁes the allowed maximum accuracy drop for the training dataset before Step 2 .
To set a checkpoint, we need to test all the training data for estimating the accuracy drop it causes.
It would be very time-expensive to set the checkpoint of each ﬁlter one by one, especially for a large network with numerous ﬁlters.
Thus, we use a compromise method that determines the checkpoints layer by layer.
That is, all the ﬁlters in the same layer share a common checkpoint.
We iteratively select a convolutional layer and determine its checkpoint starting from the center to the outer layers.
Additionally, for two outer layers that have the same distance from the centre, we ﬁrst consider the layer nearer the input.
For each selected layer in which a MAC process of a ﬁlter has n MAC iterations, the checkpoint could be the (cid:98) n ∗ 5% (cid:99) th iteration or the (cid:98) n ∗ 32% (cid:99) th iteration, or there is no checkpoint.
We ﬁrst try the checkpoint of the (cid:98) n ∗ 5% (cid:99) th iteration.
If the resultant accuracy drop for the training data exceeds e t , we then try the (cid:98) n ∗ 32% (cid:99) th iteration.
However, if the resultant accuracy drop still exceeds e t , we do not set up a checkpoint to the layer and then consider the next layer.
The pseudo code of the process is shown in Algorithm 1.
When a ﬁlter applies a checkpoint, it changes the forward propagation step through the ﬁlter.
Let us use an example in Fig.
1 to illustrate the MAC process of a ﬁlter which is applied a checkpoint.
First, the weights and the corresponding inputs are sorted in the decreasing order of weight magnitudes.
Then, the MAC operations with larger weight magnitudes are conducted ﬁrst, until the checkpoint iteration is reached.
If the intermediate result is less than 0 , the MAC process terminates and outputs the intermediate result directly; otherwise, it continues to complete all the MAC operations and outputs the result.
4. S TEP 2 : F INE - TUNE PARAMETERS
filter 𝑧 𝑙− ℎ ( 𝑙−1 ) th layer 𝑧 𝑙− ……… ..
∗ ∗ ℎ
Check41-5𝑧 ′𝑙− ……… ..
∗ ℎ ′
[Figure] [Figure]
The optimization in Step 1 leads to accuracy drop due to the saved MAC operations.
Thus, we further ﬁne-tune the model to ﬁt the applied checkpoints.
The ﬁne-tuning process is similar to the training process.
The main difference is that the ﬁne-tuning process starts with the prelearned weights and applies the checkpoints in the forward propagation.
The overall ﬂow of the proposed method is as follows:
Given a pre-trained CNN model C , the training data, and a userdeﬁned parameter e t , we ﬁrst sort the weights of each ﬁlter in C in the decreasing order of magnitudes.
Then, we determine the checkpoint of each convolutional layer in C .
The accuracy drop of
C due to the applied checkpoints cannot exceed e t .
Finally, we ﬁne-tune C to recover its accuracy.
4. A NALYSIS 4.3. E ARLY TERMINATING OF MAC PROCESSES
As mentioned above, the proposed method takes the advantage of the fact that many computations of the convolutional ﬁlters are unnecessary, because only the positive and large enough outputs can pass through the following activation and max pooling layers.
The functionality of a convolutional ﬁlter has been shown in EQ 1.
By ﬂattening the ﬁlter and inserting a checkpoint into the c th iteration, EQ 1 can be rewritten as EQ 2.
[Equation]
In EQ 2, (cid:80) c − 1 i =
z l − 1 i h i represents the intermediate result we use to determine whether the MAC process should terminate early or not.
Let u t denotes the intermediate result.
Next, let us consider the four possibilities of u t and u and the error matrix.
(1) True positive: both u t and u are positive.
In this situation, the ﬁlter completes the MAC process and does not lead to any error.
(2) True negative: both u t and u are negative.
In this situation, the ﬁlter terminates early and we save ( H ∗ W ∗ K − c ) MAC operations without introducing any error.
(3) False positive: u t is positive, but u is negative.
In this situation, the ﬁlter completes the MAC process without saving any computation and introducing any error, because u t is positive.
(4) False negative: u t is negative, but u is positive.
In this situation, although the ﬁlter saves ( H ∗ W ∗ K − c ) MAC operations, it generates an erroneous output.
After passing through the following activation layer, the error is
[Equation]
[Table]
The four situations with the corresponding MAC operation counts and errors are summarized in
Table 2.
Both the true negative and the false negative lead to MAC operation reduction, while the false negative introduces an error of u .
[Table]
W EIGHT SORTING
In the proposed method, we determine whether a convolutional ﬁlter should terminate early according to an intermediate result.
The best-case scenario is that the intermediate result and the ﬁnal result have the same sign: both positive or both negative.
To this end, we sort the weights in the decreasing order of magnitudes before performing the MAC process.
The MAC operation that results in a larger magnitude would be conducted earlier.
Sorting weights is a straightforward idea, but it is very important in the proposed method.
In the motivational example in Table 1, we have a probability of more than 60% of making a correct decision at the 3 rd iteration.
However, without weight sorting, the probability is approximately 50% .
4.3. C HECKPOINT SETUP
As mentioned above, we layer-wisely determine the checkpoints of the convolutional layers for reducing computation efforts.
Since the convolutional layers near the inputs deal with lower-level features and the convolutional layers near the outputs have larger impacts on the outputs, they usually have a smaller error tolerance.
Thus, we set up checkpoints starting from the center to the outer layers.
Furthermore, for a convolutional layer, it is impractical to test all the possible checkpoints and then determine the most suitable one.
Basically, a checkpoint closer to the beginning of the MAC process leads to more MAC operation reduction, but a larger error.
Thus, we empirically consider only two checkpoints: the (cid:98) n ∗ 5% (cid:99) th iteration and the (cid:98) n ∗ 32% (cid:99) th iteration. Several previous works (Hinton & Van Camp (1993); Giryes et al.
; Salimans & Kingma (2016)) show that the majority of weight distributions in DNNs follow the Gaussian manner.
Thus we use one standard deviation region ( 68% ) and two standard deviation region ( 95% ) to determine the two checkpoint locations, 32% and 5% .
Please note that although we consider only the two checkpoints, the proposed method can be easily extended for more checkpoints.
4.3. P ARAMETER FINE - TUNING
Due to the applied checkpoints, some ﬁlters might generate erroneous outputs, causing accuracy drop.
Parameter ﬁne-tuning is a process to recover the accuracy by further training the model with the applied checkpoints.
Since the difference between the model output and the expected output increases, the learnable weights will be changed in the ﬁne-tuning process to ﬁt the checkpoints.
E XPERIMENTAL RESULTS
We implemented the proposed method within the Caffe (Jia et al.
environment developed by the BVLC (Berkeley Vision and Learning Center).
The experiments were conducted on a Linux workstation that comprises of two Intel Xeon E 5 - 2620 2 .
10 GHz CPUs, 64 GB memory, and four
NVIDIA GeForce GTX 1080 Ti GPUs.
For comparison, we applied the proposed method and two ﬁlter/channel pruning-based previous methods, CP (He et al.
and PFEC (Li et al. (2016)), to two classic CNN models, CIFARexample model (C 10 -Net)
(Jia et al.
and Network in Network (NiN)
(Lin et al. (2013)), with two image classiﬁcation benchmark datasets, CIFAR-
10 and CIFAR- 100 (Krizhevsky (2009)).
Each method was conducted multiple times with different parameters to comprehend the comparison.
For our method, we changed the parameter e t .
Ten different e t , 5% , 10% , ..., 45% , and 50% , were applied.
For CP and PFEC, we modiﬁed the numbers of pruned ﬁlters.
The CIFAR- 10 and the CIFAR- 100 datasets have 10 and 100 classes, respectively.
Each dataset has 60 000 labeled images with the size of 32 × 32 pixels.
In the experiments, we applied zero padding to each image, such that each image has a size of 40 × 40 pixels.
Additionally, several image data augmentation techniques, including shufﬂing, random cropping, mirroring, and image mean, supported by Caffe were applied.
When ﬁne-tuning the weights, we conducted 20 epochs for the CIFAR- 10 dataset and 40 epochs for the CIFAR- 100 dataset.
5. C 10 -N ET 5.1. CIFAR- 10 DATASET
C 10 -Net has three convolutional layers and two fully connected layers.
The experimental results on the inference accuracy and the MAC operation count of each method under the CIFAR- 10 dataset are summarized in Figure 2a.
The Baseline (marked as a blue diamond) is the original model without any optimization, which conducts approximately 12 .
4 millions of MAC operations and achieves an accuracy of 86 .
The red dots denote the results of our method.
Basically, the accuracy decreases as e t increases.
The gray squares and orange triangles denote the results of CP and FPEC, respectively.
The results show that all the three methods can trade off the accuracy against the MAC operation count.
However, for several cases, our method and CP achieve similar accuracies but save more
MAC operations, compared to FPEC.
Furthermore, when we consider the objective of saving more
MAC operations with less accuracy drop, our method with e t = 10% obtains the best/largest tradeoff ratio of the saved MAC operation count to the accuracy drop among all the methods.
It achieves 50 .
22% MAC operation reduction with only an accuracy drop of 0 .
5.1. CIFAR- 100 DATASET
The experimental results on the CIFAR- 100 dataset are summarized in Figure 2b.
Similarly, CP is more effective than FPEC, and our method is competitive with CP.
The best trade-off ratio is achieved by our method with e t = 5% , where 43 .
64% MAC operations are saved with an accuracy drop of 0 .
[Figure]
N I
[Figure]
5.2. CIFAR- 10 DATASET
Unlike C 10 -Net
[Figure]
NiN is composed of nine convolutional layers.
The experimental results on the
CIFAR- 10 dataset are summarized in Figure 3a.
It is observed that our method is more effective than FPEC as well as CP.
The best trade-off ratio is achieved by our method with e t = 5%
[Figure]
where 47 .
43% MAC operations are saved with an accuracy drop of 0 .
5.2. CIFAR- 100 DATASET
The experimental results on the CIFAR- 100 dataset are summarized in Figure 3b.
Our method is also competitive with CP and more effective than FPEC.
The best trade-off ratio of our method is achieved by setting e t = 5% , where 47 .
73% MAC operations are saved with an accuracy drop of 0 .
[Figure]
In summary
[Figure]
our method can save approximately 50% MAC operations with an accuracy drop of less than 1% for each model and dataset.
Like CP and FPEC
[Figure]
our method can trade off the accuracy against the MAC operation count.
The best trade-off ratio of our method is achieved by setting e t = 5% or e t = 10% .
Compared to CP and FPEC
[Figure]
our method achieves a better trade-off ratio for the CIFAR- 10 dataset
[Figure]
and is competitive with CP for the CIFAR- 100 dataset.
C ONCLUSION
We present a dynamic optimization method to CNN models for reducing MAC computations in the convolutional layers.
The method consists of two steps.
The ﬁrst step applies a checkpoint to each convolutional ﬁlter, which is used to determine whether a MAC process of the ﬁlter could terminate early according to an intermediate result during the inference phase.
The second step is to ﬁnetune the weights of the ﬁlters to recover the accuracy drop due to the applied checkpoints.
The experimental results show that the proposed method is effective for saving MAC operations with only slight accuracy drop.
The proposed method could promisingly make more AI applications run on edge devices.