Paper:
This paper aims to reduce test-time computational load of a deep neural network. Unlike previous methods which factorize a weight matrix into multiple real-valued matrices, our method factorizes both weights and activations into integer and noninteger components. In our method, the real-valued weight matrix is approximated by a multiplication of a ternary matrix and a real-valued co-efficient matrix. Since the ternary matrix consists of three integer values, {−1, 0,+1}, it only consumes 2 bits per element. At test-time, an activation vector that passed from a previous layer is also transformed into a weighted sum of binary vectors, {−1,+1}, which enables fast feed-forward propagation based on simple logical operations: AND, XOR, and bit count. This makes it easier to deploy a deep network on low-power CPUs or to design specialized hardware. In our experiments, we tested our method on three different networks: a CNN for handwritten digits, VGG-16 model for ImageNet classification, and VGG-Face for large-scale face recognition. In particular, when we applied our method to three fully connected layers in the VGG-16, 15× acceleration and memory compression up to 5.2% were achieved with only a 1.43% increase in the top-5 error. Our experiments also revealed that compressing convolutional layers can accelerate inference of the entire network in exchange of slight increase in error.
1 INTRODUCTION
It is widely believed that deeper networks tend to achieve better performance than shallow ones in various computer vision tasks. As a trade-off of such impressive improvements, deeper networks impose heavy computational load both in terms of processing time and memory consumption due to an enormous amount of network parameters. For example, VGG-16 model (Simonyan & Zisserman, 2015) requires about 528 MBytes to store the network weights where fully connected layers account for 89% of them. A large number of multiplications and additions must also be processed at each layer which prevent real-time processing, consume vast amounts of electricity, and require a large number of logic gates when implementing a deep network on a FPGA or ASIC.
This article addresses the above issues. Specifically, we aimed to reduce the test-time computational load of a pre-trained network. Since our approach does not depend on a network configuration (e.g. a choice of an activation function, layer structures, and a number of neurons) and acts as a post-processing of network training, pre-trained networks shared in a download site of MatConvNet (Vedaldi & Lenc, 2015) and Model Zoo (BVLC) can be compressed and accelerated. Our method is outlined in Figure 1. The main idea is to factorize both weights and activations into integer and non-integer components. Our method is composed of two building blocks, as shown below.
Ternary weight decomposition for memory compression: We introduce a factored representation where the real-valued weight matrix is approximated by a multiplication of a ternary basis matrix and a real-valued co-efficient matrix. While the ternary basis matrix is sufficiently informative to reconstruct the original weights, it only consumes 2 bits per element. The number of rows of the coefficient matrix is also smaller than that of the original weight matrix. These compact representations result in efficient memory compression.
Binary activation encoding for fast feed-forward propagation: It has been reported that an inner product between a ternary and binary vector can be computed extremely fast by using three logical operations: AND, XOR, and bit count (Ambai & Sato, 2014). To use this technique, we approximate the activation vector by a weighted sum of binary vectors. This binary encoding must be processed as fast as possible at test-time. To overcome this issue, we use a fast binary encoding method based on a small lookup table.
1.1 RELATED WORK
There have been extensive studies on accelerating and compressing deep neural networks, e.g., on an FFT-based method (Mathieu et al., 2014), re-parameterization of a weight matrix (Yang et al., 2015), pruning network connection (Han et al., 2015; 2016), and hardware-specific optimization (Vanhoucke et al., 2011). In the following paragraphs, we only review previous studies that are intimately connected to ours.
It was pointed out by Denil et al. (2013) that network weights have a significant redundancy. Motivated by this fact, researchers have been involved in a series of studies on matrix/tensor factorization (Jaderberg et al., 2014; Zhang et al., 2015). In these studies, a weight matrix (or tensor) was factorized by minimizing an approximation error of original weights or activations. Jaderberg et al. (2014) exploited 1-D separable filter decomposition to accelerate feed-forward propagation. Zhang et al. (2015) proposed low-rank approximation based on generalized SVD to compress an entire deep network. Taking into account the lessons learned from these best practices, we also exploit the redundancy of the weights.
There is an another series of studies, integer decomposition (Hare et al., 2012; Yuji et al., 2014; Ambai & Sato, 2014), which involved accelerating test-time speed of a classifier by using fast logical operations. Although their contributions are limited to a shallow architecture such as a linear SVM, they achieved a noticeable acceleration. In these approaches, a real-valued weight vector is approximated by a weighted sum of a few binary or ternary basis vectors. To use fast logical operations, they extracted binary features from an image. Hare et al. (2012) and Yuji et al. (2014) exploited binary basis vectors, and Ambai & Sato (2014) investigated a case of ternary basis to improve approximation quality.
In a manner of speaking, our method is a unified framework of matrix/tensor factorization and integer decomposition reviewed in the above and inherits both their advantages. While the weight matrix is factorized to exploit low-rank characteristics, the basis matrix is restricted to take only three integer values, {−1, 0,+1}. In contrast to recent binary weighted networks such as XNOR-Net (Rastegari et al., 2016) which quantizes both activations and weights during backpropagation, it is not necessary for our method to change training algorithms at all. We can benefit from recent sophisticated training techniques, e.g. batch normalization (Ioffe & Szegedy, 2015), in combination with our method. Furthermore, our method does not need (iterative) end-to-end retraining which is needed for several previous studies such as network pruning (Han et al., 2015; 2016) and distillation (Hinton et al., 2014).
2 NETWORK COMPRESSION MODEL
In this section, we introduce our compression model and discuss time and space complexity. We consider a convolutional layer with a filter size of wx × wy × c, where wx and wy are the spacial size and c is a number of input channels. If wx = wy = 1, we can regard this layer as a fully connected layer. This three dimensional volume is reshaped to form a DI dimensional vector where DI = wx×wy×c. The filter weights and biases can be formulated by W ∈ RDI×DO and b ∈ RDO , where DO is a number of output channels. Let x ∈ RDI denote an activation vector obtained by
vectorizing the corresponding three dimensional volume. In test-time, we need to compute W>x+b followed by a non-linear activation function.
In our compressed network, W is decomposed into two matrices before test-time as follows:
W ≈MwCw, (1)
where Mw ∈ {−1, 0,+1}DI×kw is a ternary basis matrix, Cw ∈ Rkw×DO is a co-efficient matrix, and kw is the number of basis vectors, respectively. Since Mw only takes the three values, it consumes only 2 bits per element. Setting a sufficiently small value to kw further reduces total memory consumption. From the viewpoint of approximation quality, it should be noted that a large number of elements in W takes close to zero values. To fit them well enough, a zero value must be included in the basis. The ternary basis satisfies this characteristic. In practice, the ternary basis gives better approximation than the binary basis, as we discuss in Section 3.
The activation vector x is also factored to the following form:
x ≈Mxcx + bx1, (2)
where Mx ∈ {−1,+1}DI×kx is a binary basis matrix, cx ∈ Rkx is a real-valued co-efficient vector, bx ∈ R is a bias, and kx is the number of basis vectors, respectively. Since elements of x are often biased, e.g., activations from ReLU take non-negative values and have a non-zero mean, bx is added to this decomposition model. While cx and bx reflect a range of activation values, Mx determines approximated activation values within the defined range. This factorization must be computed at test-time because the intermediate activations depend on an input to the first layer. However, in practice, factorizing x into Mx, cx, and bx requires an iterative optimization, which is very slow. Since a scale of activation values within a layer is almost similar regardless of x, we pre-computed canonical cx and bx in advance and only optimized Mx at test-time. As we discuss in Section 4, an optimal Mx under fixed cx and bx can be selected using a lookup table resulting in fast factorization.
Substituting Eqs.(1) and (2) into W>x+ b, approximated response values are obtained as follows:
W>x+ b ≈ (MwCw)>(Mxcx + bx1) + b = C>wM > wMxcx + bxC > wM > w1+ b. (3)
A new bias bxC>wM > w1+ b in Eq.(3) is pre-computable in advance, because Cw,Mw, and bx are fixed at test-time. It should be noted that M>wMx is a multiplication of the ternary and binary matrix, which is efficiently computable using three logical operations: XOR, AND, and bit count, as previously investigated (Ambai & Sato, 2014). After computing M>wMx, the two co-efficient components, cx and Cw, are multiplied from the right and left in this order. Since cx and Cw are much smaller than W, the total number of floating point computations is drastically reduced.
The time and space complexity are summarized in Tables 1 and 2. As can be seen from Table 1, most of the floating operations are replaced with logical operations. In this table, B means the bit width of a variable used in the logical operations, e.g., B = 64 if a type of unsigned long long is used in C/C++ language. Table 2 suggests that if kw is sufficiently smaller than DI and DO, the total size of Mw and Cw is reduced compared to the original parameterization.
Algorithm 1 Decompose W into Mw and Cw Require: W, kw Ensure: factorized components Mw and Cw.
1: R←W 2: for i← 1 to kw do 3: Initialize m(i)w by three random values {−1, 0,+1}. 4: Minimize ||R−m(i)w c(i)w ||2F by repeating the following two steps until convergence. 5: [Step 1] c(i)w ←m(i)>w R/m(i)>w m(i)w 6: [Step 2] mij ← arg min
α∈{−1,0,+1} ||rj − αc(i)w ||22, for j = 1, · · · , DI
7: R← R−m(i)w c(i)w 8: end for
3 TERNARY WEIGHT DECOMPOSITION
To factorize W, we need to solve the following optimization problem.
Jw = min Mw,Cw
||W −MwCw||2F . (4)
However, the ternary constraint makes this optimization very difficult. Therefore, we take an iterative approach that repeats rank-one approximation one by one, as shown in Algorithm 1. Let m (i) w ∈ {−1, 0,+1}DI×1 denote an i-th column vector of Mw and c(i)w ∈ R1×DO denote an i-th row vector of Cw. Instead of directly minimizing Eq. (4), we iteratively solve the following rank-one approximation,
J (i)w = min m(i)w ,c(i)w ||R−m(i)w c(i)w ||2F , (5)
where R is a residual matrix initialized by W. We applied alternating optimization to obtain m(i)w and c(i)w . If m (i) w is fixed, c (i) w can be updated using a least squares method, as shown in line 5 of Algorithm 1. If c(i)w is fixed, mij , the j-th element of m (i) w , can be independently updated by exhaustively verifying three choices {−1, 0,+1} for each j = 1, · · · , DI , as shown in line 6 of Algorithm 1, where rj is a j-th row vector of R. After the alternating optimization is converged, R is updated by subtracting m(i)w c (i) w and passed to the next (i+ 1)-th iteration. Comparison of binary constraints with ternary constraints can be seen in Appendix A.
4 BINARY ACTIVATION ENCODING
Binary decomposition for a given activation vector x can be performed by minimizing
Jx(Mx, cx, bx;x) = ||x− (Mxcx + bx1)||22. (6)
In contrast to the case of decomposing W, a number of basis vectors kx can be set to a very small value (from 2 to 4 in practice) because x is not a matrix but a vector. This characteristic enables an exhaustive search for updating Mx. Algorithm 2 is an alternating optimization with respect to Mx, cx, and bx. By fixing Mx, we can apply a least squares method to update cx and bx (in lines 3-4 of Algorithm 2). If cx and bx are fixed, m (j) x , the j-th row vector of Mx, is independent of any other m (j′) x , j′ 6= j. We separately solve DI sub-problems formulated as follows:
m(j)x = arg min β∈{−1,+1}1×kx (xj − (βcx + bx))2, j = 1, · · · , DI , (7)
where xj is a j-th element of x. Since kx is sufficiently small, 2kx possible solutions can be exhaustively verified (in line 5 of Algorithm 2).
Our method makes this decomposition faster by pre-computing canonical cx and bx from training data and only optimizing Mx at test-time using lookup table. This compromise is reasonable because of the following two reasons: (1) scale of activation values is similar regardless of vector elements
Algorithm 2 Decompose x into Mx, cx, and bx Require: x, kx Ensure: factorized components Mx, cx, and bx.
1: Initialize Mx by three random values {−1,+1}. 2: Minimize ||x− (Mxcx + bx1)||22 by repeating the following two steps until convergence. 3: [Step 1] Update cx and bx using a least squares method. 4: cx ← (M>xMx)−1M > x (x− bx1), bx ← 1>(x−Mxcx)/DI 5: [Step 2] Update m(j)x for each j = 1, · · ·DI by an exhaustive search that minimizes Eq.(7).
within a layer, and (2) cx and bx reflect a scale of approximated activation values. Knowing these properties, cx and bx are obtained by minimizing Jx(M̂x, cx, bx; x̂) ,where x̂ is constructed as follows. First, NT different activation vectors T ∈ {xi}NTi=1 are collected from randomly chosen NT training data. Second, n elements are randomly sampled from xi. The sampled nNT elements are concatenated to form a vector x̂ ∈ RnNT . We use cx and bx as constants at test-time, and discard M̂x.
At test-time, we only need to solve the optimization of Eq. (7) for each xj . This can be regarded as the nearest neighbour search in one-dimensional space. We call βcx + bx a prototype. There are 2kx possible prototypes because β takes 2kx possible combinations. The nearest prototype to xj and an optimal solution m(j)x can be efficiently found using a lookup table as follows.
Preparing lookup table: We define L bins that evenly divide one-dimensional space in a range from the smallest to largest prototype. Let x̂l denote a representative value of the l-th bin. This is located at the center of the bin. For each x̂l, we solve Eq. (7) and assign the solution to the bin.
Activation encoding: At test-time, xj is quantized into L-levels. In other words, xj is transformed to an index of the lookup table. Let pmax and pmin denote the largest and smallest prototype, respectively. We transform xj as follows:
q = (L− 1)(xj − pmin)/(pmax − pmin) + 1, (8) l̂ = min(max(bq + 1/2c, 1), L). (9)
The range from pmin to pmax is linearly mapped to the range from 1 to L by Eq. (8). The term q is rounded and truncated from 1 to L by the max and min function in Eq. (9). If L is sufficiently large, the solution assigned to the l̂-th bin can be regarded as a nearly optimal solution because the difference between xj and the center of the bin x̂l̂ becomes very small. We found that L = 4096 is sufficient. The time complexity of this encoding is O(DI).
5 EXPERIMENTS
We tested our method on three different convolutional neural networks: CNN for handwritten digits (LeCun et al., 1998), VGG-16 for ImageNet classification (Simonyan & Zisserman, 2015), and VGGFace for large-scale face recognition (Parkhi et al., 2015). To compute memory compression rate, a size of W and a total size of Mw and Cw were compared. To obtain a fair evaluation of computation time, a test-time code of forward propagation was implemented without using any parallelization scheme, e.g., multi-threading or SIMD, and was used for both compressed and uncompressed networks. The computation time includes both binary activation encoding and calculation of Eq. (3). We used an Intel Core i7-5500U 2.40-GHz processor.
5.1 CNN FOR HANDWRITTEN DIGITS
MNIST is a database of handwritten digits which consists of 60000 training and 10000 test sets of 28× 28 gray-scale images with ground-truth labels from 0 to 9. We trained our CNN by using an example code in MatConvNet 1.0-beta18 (Vedaldi & Lenc, 2015). Our architecture is similar to LeNet-5 (LeCun et al., 1998) but has a different number of input and output channels. Each layer’s configuration is shown below:
(conv5-20)(maxpool)(conv5-64)(maxpool)(fc1024-640)(relu)(fc640-10)(softmax), (10)
where the parameters of a convolutional layer are denoted as (conv<receptive field size>-<number of output channels>), and parameters of a fully connected layer are denoted as (fc<number of input channels>-<number of output channels>). The (maxpool) is 2×2 subsampling without overlapping. The error rate of this network is 0.86%.
We applied our method to the first fully connected layer (fc1024-640) and set n = 10 andNT = 1000 to learn cx and bx from randomly chosen nNT activations. The cases of kx = 1, 2, 3, 4 and kw = DO, DO/2, DO/5 were tested. This means that kw was set to 640, 320, and 128.
Figures 2(a) and (b) show the relationships among the increases in error rates, memory compression rates, and acceleration rates. It was observed that error rates basically improved along with increasing kx and saturated at kx = 4. It is interesting that kx = 2, only 2 bits per element for encoding an activation x, still achieved good performance. While the smaller kw achieved better compression and acceleration rate, error rates rapidly increased when kw = DO/5. One of the well balanced parameters was (kx, kw) = (4, DO/2) which resulted in 1.95× faster processing and a 34.4% memory compression rate in exchange of a 0.19% increase in the error rate.
5.2 VGG-16 FOR IMAGENET CLASSIFICATION TASK
A dataset of ILSVRC2012 (Russakovsky et al., 2015) consists of 1.2 million training, 50,000 validation, and 100,000 test sets. Each image represents one of 1000 object categories. In this experiment, we used a network model of VGG-16 (model D in (Simonyan & Zisserman, 2015)) that consists of 13 convolutional layers and 3 fully connected layers followed by a softmax layer. The architecture is shown below:
(input) · · · (fc25088-4096)(relu)(fc4096-4096)(relu)(fc4096-1000)(softmax), (11) where layers before the first fully connected layer are omitted.
First, all three fully connected layers were compressed with our algorithm. We set n = 10 and NT = 1000 to learn cx and bx from randomly chosen nNT activations. The cases of kx = 2, 3, 4 and kw = DO/2, DO/4, DO/8, DO/16 were tested. The case of kx = 1 was omitted because this setting resulted in a very high error rate. Note that each of the fully connected layers has different DO. The kw was independently set for each layer according to its DO. The top-5 error rates were evaluated on the validation dataset. The top-5 error rate of the original network is 13.4%.
The three lines with circles in Figure 3 show these results. It should be noted that much higher acceleration rates and smaller compression rates with small loss of accuracies were achieved than the case of the network for MNIST. Interestingly, the case of kw = DO/4 still performed well due to the low-rank characteristics of weights in the VGG-16 network.
Although the error rates rapidly increased when kw took much smaller values, we found that this could be improved by tuning kw of the third layer. More specifically, we additionally tested the
following cases. While kw was set to DO/2, DO/4, DO/8, and DO/16 for the first and second layers, kw was fixed to DO for the third layer. The kx was set to 4. This is plotted with a red line in Figure 3. In this way, the memory compression rate and acceleration rate noticeably improved. Setting appropriate parameters for each layer is important to improve the total performance. Table 3 shows the details of the best balanced case in which 15× faster processing and 5.2% compression rate were achieved in exchange of a 1.43% increase in error rate.
Next, we also tested to compress convolutional layers. In this experiment, kw and kx were set to DO and 4. This setting accelerates each of the layers averagely 2.5 times faster. Table 4 shows positions of compressed layers, top-5 errors, and acceleration rates of the entire network. Although kw and kx must be larger than those of fully connected layers to avoid error propagation, it is still beneficial for entire acceleration. In summary, while compressing fully connected layers is beneficial for reducing memory, compressing convolutional layers is beneficial for reducing entire computation time.
5.3 VGG-FACE FOR FACE RECOGNITION TASK
The VGG-Face (Parkhi et al., 2015) is a model for extracting a face descriptor. It consists of a similar structure to VGG-16. The difference is that VGG-Face has only two fully connected layers, as shown below. (input) · · · (fc25088-4096)(relu)(fc4096-4096). (12) This network outputs a 4096-dimensional descriptor. We can verify whether two face images are identical, by evaluating the Euclidean distance of two l2-normalized descriptors extracted from
them. In our experiment, we did not apply a descriptor embedding technique based on triplet loss minimization (Parkhi et al., 2015). Following the evaluation protocol introduced in a previous paper (Parkhi et al., 2015), we used Labeled Faces in the Wild dataset (LFW) (Huang et al., 2007), which includes 13,233 face images with 5,749 identities. The LFW defines 1200 positive and 1200 negative pairs for testing. We used the 2400 test pairs to compute ROC curve and equal error rate (EER). The EER is defined as an error rate at the ROC operating point where the false positive and false negative rates are equal. The EER of the original network is 3.8%.
First, the two fully connected layers were compressed using our algorithm. We set n = 10 and NT = 1000 to learn cx and bx from randomly chosen nNT activations. We tested the cases of kx = 1, 2, 3, 4, and kw = DO/2, DO/4, DO/8, DO/16. Figure 4 reveals an interesting fact that even the fastest and smallest network configuration, kx = 1 and kw = DO/16, had less impact on the EER, in contrast to the previous ImageNet classification task in which the recognition results were corrupted when kx = 1. This indicates that the 4096-dimensional feature space is well preserved regardless of such coarse discretization of both weights and activations.
Next, we also tested to compress convolutional layers. In this experiment, kw and kx were set to DO and 4 which are the the same setting used in Table 4. Table 5 shows positions of compressed layers and EERs. The acceleration rates were almost the same as the results shown in Table 4. This is because architecture of VGG-face is the same as VGG-16 and we used the same parameter for kw and kx. Interestingly, compressing multiple layers from 2nd to 10th still preserves the original EER. As can be seen from this table, our method works very well depending on a certain kind of machine learning task.
6 CONCLUSION
We proposed a network compression model that consists of two components: ternary matrix decomposition and binary activation encoding. Our experiments revealed that the proposed compression model is available not only for multi-class recognition but also for feature embedding. Since our approach is post-processing for a pre-trained model, it is promising that recent networks designed for semantic segmentation, describing images, stereo matching, depth estimation, and much more can also be compressed with our method. For future work, we plan to improve approximation error further by investigating the discrete optimization algorithm.
A BINARY VS. TERNARY
Figure 5 illustrates the reconstruction errors of a 4096×1000 weight matrix of the last fully connected layer in VGG-16 model (Simonyan & Zisserman, 2015). We tested both the binary and ternary constraints on Mw for comparison. The reconstruction error Jw monotonically decreased along with an increase in kw. It was clear that the ternary basis provided better reconstruction than the binary basis.

Review:
This paper explores a new quantization method for both the weights and the activations that does not need re-training. In VGG-16 the method reaches compression ratios of 20x and experiences a speed-up of 15x. The paper is very well written and clearly exposes the details of the methodology and the results.
My major criticisms are three-fold: for one, the results are not compared to one of the many other pruning methods that are described in section 1.1, and as such the performance of the method is difficult to judge from the paper alone. Second, there have been several other compression schemes involving pruning, re-training and vector-quantization [e.g. 1, 2, 3] that seem to achieve much higher accuracies, compression ratios and speed-ups. Hence, for the practical application of running such networks on low-power, low-memory devices, other methods seem to be much more suited. The advantage of the given method - other then possibly reducing the time it takes to compress the network - is thus unclear. In particular, taking a pre-trained network as a starting point for a quantized model that is subsequently fine-tuned might not take much longer to process then the method given here (but maybe the authors can quantify this?). Finally, much of the speed-up and memory reduction in the VGG-model seems to arise from the three fully-connected layers, in particular the last one. The speed-up in the convolutional layers is comparably small, making me wonder how well the method would work in all-convolutional networks such as the Inception architecture.
[1] Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding,