C AT :
C O M P R E S S
I O N - A WA R E
T R A
I N I N G
F O R
B A N D W
I D T H R
E D U C T
I O N
Anonymous authors
Paper under double-blind review
A B S T R AC T
Convolutional neural networks (CNNs) have become the dominant neural network architecture for solving visual processing tasks.
One of the major obstacles hindering the ubiquitous use of CNNs for inference is their relatively high memory bandwidth requirements, which can be a main energy consumer and throughput bottleneck in hardware accelerators.
Accordingly, an efﬁcient feature map compression method can result in substantial performance gains.
Inspired by quantization-aware training approaches, we propose a compression-aware training (CAT) method that involves training the model in a way that allows better compression of feature maps during inference.
Our method trains the model to achieve low-entropy feature maps, which enables efﬁcient compression at inference time using classical transform coding methods.
CAT signiﬁcantly improves the state-of-the-art results reported for quantization.
For example, on ResNet-34 we achieve 73.1% accuracy (0.2% degradation from the baseline) with an average representation of only 1.79 bits per value.
Reference implementation accompanies the paper.
I N T RO D U C T I O N
Deep Neural Networks (DNNs) have become a popular choice for a wide range of applications such as computer vision, natural language processing, autonomous cars, etc.
Unfortunately, their vast demands for computational resources often prevents their use on power-challenged platforms.
The desire for reduced bandwidth and compute requirements of deep learning models has driven research into quantization (Hubara et al., 2016;
Yang et al., 2019b; Liu et al., 2019;
Gong et al., 2019), pruning (LeCun et al., 1990; Li et al., 2017; Molchanov et al., 2019), and sparsiﬁcation (Gale et al., 2019;
Dettmers & Zettlemoyer, 2019).
In particular, quantization works usually focus on scalar quantization of the feature maps: mapping the activation values to a discrete set { q i } of size L .
Such a representation, while being less precise, is especially useful in custom hardware, where it allows more efﬁcient computations and reduces the memory bandwidth.
In this work, we focus on the latter, which has been shown to dominate the energy footprint of CNN inference on custom hardware (Yang et al., 2017).
We show that the quantized activation values { q i } can further be coded to reduce memory requirements.
The raw quantized data require (cid:100) log 2 ( L ) (cid:101) bits per value for storage.
This number can be reduced by compressing the feature maps.
In particular, in the case of element-wise compression of independent identically distributed values, the lower bound of amount of bits per element is given by the entropy (Shannon, 1948):
[Equation]
of the quantized values { q i } , where p ( q i ) denotes the probability of q i .
In this work, we take a further step by manipulating the distribution of the quantization values so that the entropy H ( q ) is minimized.
To that end, we formulate the training problem by augmenting the regular task-speciﬁc loss (the cross-entropy classiﬁer loss in our case) with the feature map entropy serving as a proxy to the memory rate.
The strength of the latter penalty is controlled through a
[Figure]
As training progresses, the probability of non-positive pre-activation values increases, zeroing more post-ReLU values.
The sharp peak at zero reduces entropy and thus improves compressibility
Effect of entropy regularization.
Without regularization, the distribution has much heavier tails and thus has higher entropy.
With increasing regularization, the probability of extreme values is signiﬁcantly reduced.
The entropy penalty λ was selected so that the overall accuracy is not affected regularized unregularized parameter λ > 0 .
Fig.
1 demonstrates the effect of the entropy penalty on the compressibility of the intermediate activations.
Contributions.
Our paper makes several contributions.
Firstly, we introduce Compression-Aware
Training (CAT), a technique for memory bandwidth reduction.
The method works by introducing a loss term that penalizes the entropy of the activations at training time and applying entropy encoding (e.g., Huffman coding) on the resulting activations at inference time.
Since the only overhead of the method at inference time is entropy encoding that can be implemented efﬁciently, the improvement is universal for any hardware implementation, being especially efﬁcient on computationally optimized ones, where memory I/
O dominates the energy footprint
(Yang et al., 2017; Jouppi et al., 2017).
We demonstrate a 2 to 4 -fold memory bandwidth reduction for multiple architectures: MobileNetV and ResNet on the ImageNet visual recongition task, and SSD512 on the PASCAL VOC object detection task.
We also investigate several differentiable loss functions which lead to activation entropy minimization and show a few alternatives that lead to the same effect.
Finally, we analyze the rate-distortion tradeoff of the method, achieving even stronger compression at the expense of minor accuracy reduction: for ResNet-18, we manage to achieve entropy inferior to one bit per value, at the expense of losing 2% of the top-1 accuracy.
R E L AT E D W O R K
Recent studies (Yang et al., 2017; Wang et al., 2019) have shown that almost 70% of the energy footprint on custom hardware is due to the data movement to and from the off-chip memory.
Nonetheless, the techniques for memory bandwidth reduction have not yet received signiﬁcant attention in the literature.
One way to improve memory performance is by fusing convolutional layers (Xiao et al., 2017;
Xing et al., 2019), reducing the number of feature maps transfers.
This reduces both runtime and energy consumption of the hardware accelerator.
Another option is to use on-chip cache instead of external memory.
Morcel et al.
(2019) have shown order of magnitude improvement in power consuption using this technique.
Another important system parameter dominated by the memory bandwidth is latency.
Jouppi et al.
(2017) and Wang et al.
(2019) showed that the state-of-the-art
DNN accelerators are memory-bound, implying that increasing computation throughput without reducing the memory bandwidth barely affects the total system latency.
Quantization reduces computation and memory requirements; 16-bit ﬁxed point has become a de facto standard for fast inference.
In most applications, weights and activations can be quantized down to 8 bits without noticeable loss of precision (Lee et al., 2018; Yang et al., 2019a).
Further quantization to lower precision requires non-trivial techniques (Mishra et al., 2018;
Zhang et al., 2018), which are currently capable of reaching around 3 - 4 bits per entry without compromising precision (Choi et al., 2018b;a; Dong et al., 2019).
Another way to reduce memory bandwidth is by compressing the intermediate activations prior to their transfer to memory with some computationally cheap encoding, such
that Huffman (Chandra, 2018; Chmiel et al., 2019) or run-length (RLE) encoding (Cavigelli et al., 2019).
A similar approach of storing only nonzero values was utilized by Lin & Lai (2018).
Chmiel et al.
(2019) used linear dimensionality reduction (PCA) to increase the effectiveness of Huffman coding, while Gudovskiy et al.
(2018) proposed to use nonlinear dimensionality reduction techniques.
Lossless coding was previously utilized in a number of ways for DNN compression: Han et al.
and Zhao et al.
(2019) used Huffman coding to compress weights, while Wijayanto et al.
(2019) used more complicated DEFLATE (LZ77 + Huffman) algorithm for the same purpose.
Aytekin et al.
(2019) proposed to use compressibility loss, which induces sparsity and has been shown (empirically) to reduce entropy of the non-zero part of the activations.
M E T H O D
We consider a feed-forward DNN F composed of L layers; each subsequent layer processing the output of the previous one: x i = F i (cid:0) x
i − 1 (cid:1) , using the parameters w ∈ R N .
We denote by x 0 = x and x L =
y the input and the output of the network, respectively.
The parameters w of the network are learned by minimizing L ( x, y ; w ) + λ R ( w ) , with the former term L being the task loss, and the latter term R being a regularizer (e.g., (cid:107) w (cid:107) 2 ) inducing some properties on the parameters w .
3. E NTROPY ENCODING AND RATE REGULARIZATION
Entropy encoders are a family of lossless data encoders which compress each symbol independently.
In this case, assuming i.i.d.
distribution of the input, it has been show that optimal code length is − log b p , where b is number of symbols and p i is the probability of i th symbol (Shannon, 1948).
Thus, for a discrete random variable X we deﬁne an entropy H ( X ) =
− E log 2 X = − (cid:80)
i p ( x i ) log 2 p ( x i ) , which is a lower bound of the amount of information required for lossless compression of X .
The expected total space required to encode the message is N · H , where N is number of symbols.
Since we encode the activations with entropy encoder before writing them into memory, we would like to minimize the entropy of the activations to improve the compression rate.
One example of entropy encoder is Huffman coding – a preﬁx coding which assigns shorter codes to the more probable symbols.
It is popular because of simplicity along with high compression rate (Szpankowski, 2000) bounded by H ( X ) ≤ R ≤ H ( X )
+ 1 (the comparison of Huffman coding rates to entropy
is shown in Fig. 3).
Another entropy encoder is arithmetic coding – highly efﬁcient encoder which achieves optimal rates for big enough input but requires more computational resources for encoding.
3. D IFFERENTIABLE ENTROPY - REDUCING LOSS
Since the empirical entropy is a discrete function, it is not differentiable and thus cannot be directly minimized with gradient descent.
Nevertheless, there exist a number of differentiable functions which either approximate entropy or have same minimizer.
Thus we optimize
[Equation]
where L p is a target loss function and L H is some regularization which minimizes entropy.
Soft entropy First, we consider the differentiable entropy estimation suggested by Agustsson et al.
We start from deﬁnition of the entropy,
[Equation]
where q is a vector of quantized values.
Let m be an index of the bin the current value is mapped to, and Q a one-hot encoding of this index, i.e.,
[Equation]
where δ im denotes Kroneker’s delta.
To make the latter expression differentiable, we can replace argmax with softmax:
[Equation]
where T is the temperature parameter, and ˜ Q ( x ) → Q ( x ) as T → ∞ .
Finally, the soft entropy ˆ H is deﬁned as
[Equation]
To improve both memory requirements and time complexity of the training, we calculate soft entropy only on part of the batch, reducing the amount of computation and the gradient tensor size.
We empirically conﬁrm that this choice gives a reasonable approximation of the real entropy (Appendix B).
Compressibility loss for entropy reduction An alternative loss promoting entropy reduction was proposed by Aytekin et al.
(2019) under the name of compressibility loss and based on earlier work by Hoyer (2004):
[Equation]
This loss has the advantage of computational simplicity, and has been shown both theoretically and practically to promote sparsity and low entropy in input vectors.
While originally applied to the weights of the network, here we apply the same loss to the activations.
As shown in Section 4.1, both the soft entropy and the compressibility loss lead to similar results.
Our method for reducing memory bandwidth can be described as follows: at training time, we ﬁne-tune (training from scratch should also be possible, but we have not tested it) the pre-trained network F with the regularized loss (2), where we use L H =
(cid:80)
ˆ H ( x i ) in case of differentiable entropy and L H =
(cid:80) L c ( x i ) in case of compressability loss.
At test time, we apply entropy coding on the activations before writing them to memory, thus reducing the amount of memory transactions.
In contrast to Chmiel et al.
(2019), who avoided ﬁne-tuning by using test time transformation in order to reduce entropy, our method does not requires complex transformations at test time by inducing low entropy during training.
E X P E R I M E N TA L R E S U LT S
We evaluate the proposed scheme on common CNN architectures for image classiﬁcation on ImageNet (ResNet-18/34/50, MobileNetV2), and object detection on Pascal VOC dataset (SSD512 1 (Liu et al., 2016)).
The weights were initialized with pre-trained model and quantized with uniform quantization
[Figure]
[Table]
[Table]
[Table]
ResNet-50, CAT   76.
[Table] [Table]
[Table] [Table]
[Table]
SSD512-SqueezeNet (Gudovskiy et al., 2018) [Table] [Table]
[Table]
SSD512-VGG, CAT
[Table] [Table] [Table] [Table]
[Table]
using the shadow weights, i.e. applying updates to a full precision copy of quantized weights (Hubara et al., 2016; Rastegari et al., 2016).
The activation were clipped with a learnable parameter and then uniform quantized as suggested by Baskin et al.
Similarly to previous works (Zhou et al., 2016; Rastegari et al., 2016), we used the straight-through estimator (Bengio et al., 2013) to approximate the gradients.
We quantize all layers in the network, in contrast to the common practice of leaving ﬁrst and last layer in high-precision (Zhou et al., 2016; Baskin et al., 2018).
Since the weights are quantized only to 8 bit, we have noticed small to no difference between the quantized and non-quantized weights.
For optimization, we use SGD with a learning rate of 10 − 4 , momentum 0 . 9 , and weight decay 4 × 10 − 5 for up to 30 epochs (usually, 10 − 15 epochs were sufﬁcient for convergence).
Our initial choice of temperature was T = 10 , which performed well.
We tried, following the common approach (Jang et al., 2016), to apply exponential scheduling to the temperature, but it did not have any noticeable effect on the results. [Figure] [Table] [Figure]
[Table] [Figure] [Table] In Fig.
2 we compare our method with EPBC (Cavigelli et al.
and GF (Gudovskiy et al.
EPBC is based on a lossless compression method that maintains the full precision accuracy while reducing bit rate to approximately 3.5 bits/value in both models.
GF, on the other hand, provides strong compression at the expense of larger accuracy degradations.
In addition, Gudovskiy et al.
(2018) have compressed only part of the layers.
Unlike these two methods, CAT allows more ﬂexible tradeoff between compression and accuracy.
CAT shows better results in ResNet-34 and show either better accuracy or compression for MobileNetV2.
We also run our method on additional architectures:
ResNet-18, ResNet-50, and SSD512 with VGG backbone; the results are listed in Table 1.
Even though we can not directly compare detection results with Gudovskiy et al.
(2018), the drop in accuracy is lower in our case.
4. A BLATION STUDY
Rate-Accuracy Tradeoff The proposed CAT algorithm tries to balance between minimizing the rate and maximizing the accuracy of the network by means of the parameter λ in Eq.
To check this tradeoff, we run the same experiment in ResNet-18 and MobileNetV2 with different values of λ in the range of 0 − 0 .
3 , with results shown in Fig.
We show the results of the theoretical entropy and of the Huffman coding, which was chosen for its simplicity but more efﬁcient encoders can be combined with our method.
For higher rate Huffman coding is close ( ∼ 3% overhead) to the theoretical entropy, while for lower entropy the difference is higher and is bounded by 1 bit per value – in the latter case, different lossless coding schemes such as arithmetic coding can provide better results.
Robustness For checking the robustness of our method, we performed several runs with the same hyper-parameters and a different random seed.
Statistics reported in Table 2 suggest that the method is robust and stable under random intialization.
Soft entropy vs. compressibility loss Replacing the soft entropy with a different loss which minimizes the entropy almost did not affect the results, as shown in Table 3.
We conclude that the desired effect is a result of entropy reduction rather than a particular form of regularization promoting it.
[Table] Batch size We have noticed that training ResNet-50 on a single GPU mandated the use of small batches, leading to performance degradation.
Increasing the batch size from 16 to 64 without changing other hyperparameters increased accuracy by more than 0.5% with entropy increase by less than 0.
bits/value.
D I S C U S S I O N
Quantization of activations reduces memory access costs which are responsible for a signiﬁcant part of energy footprint in NN accelerators.
Conservative quantization approaches, known as post-training quantization, take a model trained for full precision and directly quantize it to 8-bit precision.
These methods are simple to use and allow for quantization with limited data.
Unfortunately, post-training quantization below 8 bits usually incurs signiﬁcant accuracy degradation.
Quantization-aware training approaches involve some sort of training either from scratch (Hubara et al., 2016), or as a ﬁne-tuning step from a pre-trained ﬂoating point model (Han et al., 2016).
Training usually compensates signiﬁcantly for model’s accuracy loss due to quantization.
In this work, we take a further step and propose a compression-aware training method to aggressively compress activations to as low as 2 average bit/value representations without harming accuracy.
Our method optimizes the average-bit per value needed to represent activation values by minimizing the entropy.
We demonstrate the applicability of our approach on classiﬁcation tasks using the models
MobileNetV2, ResNet, and object detection task using the model SSD512.
Due to the low overhead, the method provides universal improvement for any custom hardware, being especially useful for the accelerators with efﬁcient computations, where memory transfers are a signiﬁcant part of the energy budget.
We show that the effect is universal among loss functions and robust to random initialization
A M P L E S I Z E I N S O F T E N T RO P Y C A L C U L AT I O N
To check whether the amount of values used to calculate soft entropy is enough, we ran a soft entropy evaluation on a single tensor and compared it to real values.
Since the tensors are large (hundreds of thousands of elements), even 5% of the values already provides reasonable approximation of the real entropy, as shown in Fig.
B.1.
[Table]
[Figure]
Real Entropy
Chosen ratio
[Figure]