Compact and Computationally Efﬁcient Representation of Deep Neural Networks
Anonymous Author(s)
AfﬁliationAddress email
Abstract
At the core of any inference procedure in deep neural networks are dot productoperations, which are the component that require the highest computational resources.
One common approach to reduce the complexity of these operations isto prune and/or quantize the weight matrices of the neural network and to employdata structures, e.g., sparse matrix data structures, that can efﬁciently exploit thestatistics present in the matrix.
However, most of these common data structuresmake strong statistical assumptions about the distribution of the elements in thematrix, which may not always be satisﬁed, in particular, when we consider theentire set of highly compressible matrices.
We present new efﬁcient data structures for representing matrices that exhibit lowentropy statistics relative to the maximum likelihood estimates of the probabilitymass distribution of their elements.
We show that these formats are especiallysuitable for representing neural networks that have been compressed using state-ofthe-art lossy compression techniques.
We show that the proposed data structurescan not only be regarded as a generalization of sparse formats, but are also moreenergy and time efﬁcient under practically relevant assumptions.
For instance,we experimentally show that we are able to attain up to x16 compression ratios,x1.7 speed ups and x20 energy savings when we lossless convert state-of-the-artnetworks such as AlexNet, VGG-16, ResNet152 and DenseNet into the new datastructures.
Introduction
Deep neural networks [9, 14] nowadays became the state-of-the-art in many ﬁelds of machine learning.
However, most deep neural network models require the computation of many dot product operationsbetween large matrices when performing inference.
This requires a great amount of computationalresources which difﬁcult or even prohibit their deployment into resource constrained devices.
This fact motivated an entire research ﬁeld of model compression [1] that aimed to reduce thecomplexity of inference of deep neural networks.
For instance, some of the most popular techniquesinclude sparsiﬁcation of the network [3, 8, 7, 13] or quantization of the weight elements [17, 11].
These approaches aim to reduce both storage and execution complexity of the dot product oprationsinvolved in the inference procedure, by either leveraging on the sparsity of the weight matrices, e.g.,by representing them in one of the sparse data structures and performing the dot products accordingly,or by converting the element values into low bit-length numerical representations.
However, highestcompression ratios and execution efﬁciencies can be achieved when one employs compressionschemes that either implicitly [6, 10, 12] or explicitly [2, 16, 4] aim to minimize the entropy of theweight elements relative to the maximum likelihood estimate of their probability mass distribution.
[Figure] [Figure]
[Figure]
However, weight matrices that exhibit low entropy statistics are not necessarily sparse, neither thecardinality of the elements has to be low.
For instance, ﬁgure 1 plots the distribution of the weightelements of the last classiﬁcation layer of VGG-16 [15] ( 1000 × 4096 dimensional matrix), afterhaving applied uniform quantization on the weight elements.
We stress that the prediction accuracyand generalization of the network was not affected by this operation.
As we can see, on the one hand,the distribution of the compressed layer does not satisfy the sparsity assumption, i.e., there is notone particular element (such as 0) that appears specially frequent in the matrix.
On the other hand,although the cardinality of the weight elements is in this case smaller than the original size (7 bits asopposed to 32), one can not efﬁciently exploit this property with regards to the dot product operationon most conventional hardware, since they do not support multiplications on mixed-precissionnumerical representations.
Moreover, the entropy of the matrix is about H = 5 . 5 , which is smallerthan 7 bits, thus indicating that even higher efﬁciency gains may be obtained.
Although [6] proposedan optimal entropy coder for representing these type of matrices, specialized hardware is needed inorder to efﬁciently run inference under the resulting representation [5].
Hence, under our knowledge,there is currently no simple representation that can be applied on this type of matrices, that is able tosimultaneously achieve high compression ratios and low computational complexity relative to the dotproduct operation.
The compressed entropy row (CER) and compressed shared weight row(CSER) representations
Notice, that we expect the ratio ¯ k/n between the average number of unique elements ¯ k that appearin a particular row (or column) and the respective number of columns (or rows) n to be low , formatrices that exhibit low entropy statistics.
The intuition behind this is that we can interpret thenumber 2 H as the effective number of unique elements that a random variable with entropy value Houtputs.
Indeed, only 15 distinct values dominate the entries of the weight matrix displayed in ﬁgure1, which is only 1.5% of the number of columns of the matrix.
Hence, in this work, we propose twodata structures that exploit this property, which we named compressed shared weight row (CSER)and compressed entropy row (CER).
Both apply the same principles, but differ only slightly on theirstatistical assumptions.
The CER and CSER representations are able to save signiﬁcant amounts of storage requirementsby leveraging on the fact that we only need to store the unique elements that appear on the matrixrows once .
In contrast, although sparse data structures already implicitly apply this technique for themost frequent element (namely, the 0 value), they store the other values multiple times (the non zeroentries), inducing high redundancies in their ﬁnal representations.
Moreover, we can design efﬁcientdot product algorithm by implicitly encoding the distributive law of multiplications in the CER and
CSER representations.
By incorporating the distributive law we can not only save a great numberof multiplications, but also a great number of expensive read/write operations associated to them.
[Table]
The particular encoding/decoding procedure as well as the associated dot product pseudocodes aredescribed more extensively in our main manuscript [18].
Concretely, we can show that the storage requirement as well as the dot product complexity (asmeasured relative to each element in the matrix) of the CSER and CER representations asymptoticallyfulﬁllcomplexity =
O ((1 − p 0 ) log 2 n ) + O (cid:18)
¯ k n (cid:19)
+ O (cid:18)n (cid:19) (1) where p 0 is the maximum likelihood estimate of the probability mass distribution of the most frequentvalue, n the row size and
¯ k the average number of unique elements that appear in a row.
Hence,equation (1) states that the efﬁciency of the CER and CSER representations depend partially onthe sparsity level of the matrix (the term (1 − p 0 ) ) and the per row cardinalty of it’s elements (theterm ¯ k ).
Consequently, the CER and CSER representations will result in being particularly efﬁcientwhen both terms are simultaneously minimized.
Notice, that both terms decrease as the entropyof the matrix statistics decreases.
Also, as we mentioned in the introduction, most frameworksthat achieve state-of-the-art compression ratios aim to directly minimize these two componentssimultaneously [6, 10, 12, 2, 16, 4].
In addition, (1) also states that the CER and CSER data structurescan efﬁciently exploit a wider set of matrices, in particular those matrices whose element statisticsfail to be efﬁciently executed by the dense and sparse representations.
Finally, we would like to stressonce more that the CER and CSER data structures do not require specialized hardware in order to runefﬁciently, as long as their complexity (1) is sufﬁciently small.
Experiments
We applied our representations on compressed weight matrices of deep neural networks and benchmarked their efﬁciency with regards to 4 metrics: 1) the storage requirement, 2) the total number ofoperations needed to perform a matrix-vector multiplication, 3) the respective time complexity and 4)the respective energy consumption.
We run all our experiments on a conventional CPU hardware.
We were able to attain up to x16 compression ratios, x1.7 speed ups and x20 energy savings (comparedto the original dense representation) after converting the weight matrices of the by [6] compressed
AlexNet model 1 into the CER and CSER representations.
These are signiﬁcant gains comparedto, e.g., the common CSR sparse format which attained x6, x1.7 and x5 gains respectively.
Wealso benchmarked our data structures on networks that were compressed using lossy compression techniques that do not require to retrain the network in the process.
This case is of particular interest, since in many real world scenarios one may have a full-sized model but no access to the training data.
Also, in most cases, the statistics of the compressed weight matrices do not fulﬁll the common prior assumptions of most conventional data structures in such scenario (e.g., ﬁgure 1).
In contrast, the
CER and CSER are able to exploit the statistical properties present, and consequently attain higher gains in efﬁciency throughout all four above mentioned benchmarks (see table 1).