Compact and Computationally Efﬁcient Representation of Deep Neural Networks
Abstract
At the core of any inference procedure in deep neural networks are dot product operations, which are the component that require the highest computational resources.
One common approach to reduce the complexity of these operations is to prune and/or quantize the weight matrices of the neural network. Usually, this results in matrices whose entropy value is low, as measured relative to the maximum likelihood estimate of the probability mass distribution of it’s elements. In order to efﬁciently exploit such matrices one usually relies on, inter alia, sparse matrix representations.
However, most of these common matrix storage formats make strong statistical assumptions about the distribution of the elements in the matrix, and can therefore not efﬁciently represent the entire set of matrices that exhibit low entropy statistics (thus, the entire set of compressed neural network weight matrices).
In this work we address this issue and present new efﬁcient representations for matrices with low entropy statistics.
We show that the proposed formats can not only be regarded as a generalization of sparse formats, but are also more energy and time efﬁcient under practically relevant assumptions.
For instance, we experimentally show that we are able to attain up to x16 compression ratios, x1.7 speed ups and x20 energy savings when we convert the weight matrices of state-of-the-art networks such as AlexNet, VGG-16, ResNet152 and DenseNet into the new representations.
Introduction
Deep neural networks [10, 16] became the state-of-the-art in many ﬁelds of machine learning.
However, most deep neural network models require the computation of many dot product operations between large matrices when performing inference.
This requires a great amount of computational resources which difﬁcult their deployment into resource constrained devices.
This fact motivated an entire research ﬁeld of model compression [1] that aimed to reduce the complexity of inference of deep neural networks.
For instance, some of the most popular techniques include sparsiﬁcation of the network [3, 9, 8, 14] or quantization of the weight elements [19, 12].
These approaches aim to reduce both storage and execution complexity of the dot product oprations involved in the inference procedure, by either leveraging on the sparsity of the weight matrices, e.g., by representing them in one of the sparse storage formats [15, 4] and performing the dot products accordingly, or by converting the element values into low bit-length numerical representations.
[Figure]
However, highest compression ratios and execution efﬁciencies can be achieved when one employs compression schemes that either implicitly [7, 11, 13] or explicitly [2, 18, 5] aim to minimize the entropy of the weight elements relative to the maximum likelihood estimate of their probability mass distribution.
However, weight matrices that exhibit low entropy statistics are not necessarily sparse, neither the cardinality of the elements has to be low.
For instance, ﬁgure 1 plots the distribution of the weight elements of the last classiﬁcation layer of VGG-16 [17] ( 1000 × 4096 dimensional matrix), after having applied uniform quantization on the weight elements.
We stress that the prediction accuracy and generalization of the network was not affected by this operation.
As we can see, on the one hand, the distribution of the compressed layer does not satisfy the sparsity assumption, i.e., there is not one particular element (such as 0) that appears specially frequent in the matrix.
On the other hand, we can also not take advantage of the fact that we could trivially compress each element of the original dense representation down to 7 bits (because we only have at most 2 7 unique elements present in the matrix), since most conventional hardware do not support multiplications on mixed-precission numerical representations (recall, that in this case the activation values would be still represented by single precission ﬂoating point values).
Moreover, the entropy of the matrix is about H = 5 . 5 , which is smaller than 7 bits, thus indicating that even higher efﬁciency gains may be obtained.
Although [7] proposed an optimal entropy coder for representing these type of matrices, specialized hardware is needed in order to efﬁciently run inference under the resulting representation [6].
Hence, under our knowledge, there is currently no simple representation that can be applied on this type of matrices, that is able to simultaneously achieve high compression ratios and low computational complexity relative to the dot product operation.
The compressed entropy row (CER) and compressed shared weight row (CSER) representations
Notice, that we expect the ratio ¯ k/n between the average number of unique elements ¯ k that appear in a particular row (or column) and the respective number of columns (or rows) n to be low , for matrices that exhibit low entropy statistics.
The intuition behind this is that we can interpret the number 2 H as the effective number of unique elements that a random variable with entropy value H outputs.
Indeed, only 15 distinct values dominate the entries of the weight matrix displayed in ﬁgure 1, which is only 1.5% of the number of columns of the matrix.
Hence, in this work, we propose two new matrix representations that exploit this property, which we named compressed shared weight row (CSER) and compressed entropy row (CER).
Both apply the same principles, but differ only slightly on their statistical assumptions.
The CER and CSER representations are able to save signiﬁcant amounts of storage requirements by leveraging on the fact that we only need to store the unique elements that appear on the matrix rows once .
In contrast, sparse storage formats store the same non zero values multiple times, inducing high redundancies in their ﬁnal representations.
Moreover, we can design efﬁcient dot product algorithm by implicitly encoding the distributive law of multiplications in the CER and CSER representations. By
[Table]
incorporating the distributive law we can not only save a great number of multiplications
[Table]
but also a great number of expensive read/write operations associated to them. The particular encoding/decoding procedure as well as the associated dot product algorithms are described more extensively in our main manuscript [20]. In the appendix we show an example and also present the respective pseudocodes.
We can show that the storage requirement as well as the dot product complexity (as measured relative to each element in the matrix) of the CSER and CER representations asymptotically fulﬁll
[Equation]
where p 0 is the maximum likelihood estimate of the probability mass distribution of the most frequent value, n the row size and
¯ k the average number of unique elements that appear in a row.
Hence, equation (1) states that the efﬁciency of the CER and CSER representations depend partially on the sparsity level of the matrix (the term (1 − p 0 ) ) and the per row cardinalty of it’s elements (the term ¯ k ).
Consequently, the CER and CSER representations will result in being particularly efﬁcient when both terms are simultaneously minimized.
Notice, that both terms decrease as the entropy of the matrix statistics decreases.
Also, as we mentioned in the introduction, most frameworks that achieve state-of-the-art compression ratios aim to directly minimize these two components simultaneously [7, 11, 13, 2, 18, 5].
In addition, (1) also states that the CER and CSER representations can efﬁciently exploit a wider set of matrices, in particular those matrices whose element statistics fail to be efﬁciently executed by the dense and sparse representations.
Experiments
We applied our representations on compressed weight matrices of deep neural networks and benchmarked their efﬁciency with regards to 4 metrics: 1) the storage requirement, 2) the total number of operations needed to perform a matrix-vector multiplication, 3) the respective time complexity and 4) the respective energy consumption.
We run all our experiments on a conventional CPU hardware.
We were able to attain up to x16 compression ratios, x1.7 speed ups and x20 energy savings (compared to the original dense representation) after converting the weight matrices of the by [7] compressed
AlexNet model 1 into the CER and CSER representations.
These are signiﬁcant gains compared to, e.g., the common CSR sparse format [15, 4] which attained x6, x1.7 and x5 gains respectively.
We also benchmarked our matrix representations on networks that were compressed using lossy compression techniques that do not require to retrain the network in the process.
This case is of particular interest, since in many real world scenarios one may have a full-sized model but no access to the training data.
Also, in most cases, the statistics of the compressed weight matrices do not fulﬁll the common prior assumptions of most conventional matrix representations in such scenario (e.g., ﬁgure 1). In contrast, the CER and CSER are able to exploit the statistical properties present, and attain higher gains in efﬁciency throughout all four above mentioned benchmarks (see table 1). Acknowledgment
This work was supported by the Fraunhofer Society through the MPI-FhG collaboration project “Theory & Practice for Reduced Learning Machines”. This research was also supported by the German Ministry for Education and Research as Berlin Big Data Center under Grant 01IS14013A and by the Institute for Information & Communications Technology Promotion and funded by the Korea government (MSIT) (No. 2017-0-01779 and No. 2017-0-00451).
A Example of how the CER and CSER formats compare to the dense and CSR matrix representations
Consider the following matrix
[Equation]

Now assume that we want to: 1) store this matrix with the minimum amount of bits and 2) perform the dot product with a vector a ∈ R 12 with the minimum complexity.
A.1 Storage requirements
Dense format : The dense representation would trivially store the elements of M in a 5 × 12 long array (in addition to it’s dimensions m = 5 and n = 12 ).
Sparse format: Sparse formats attempt to attain higher compression ratios by not storing the 0 values in their matrix representation.
In particular, the Compressed Sparse Row (or CSR in short) format stores the values of the matrix in the following way:
• Scans the non-zero elements in row-major order (that is, from left to right, up to down) and stores them in an array (which we denote as W ).
• Simultaneously, it stores the respective column indices in another array (which we call colI ).
• Finally, it stores pointers that signalize when a new row starts (we denote this array as rowPtr ).
Hence, the matrix M would take the form
W :[3 , 2 , 4 , 2 , 3 , 4 , 4 , 4 , 4 , 4 , 4 , 4 , 4 , 4 , 3 , 4 , 4 , 2 , 4 , 4 , 4 , 3 , 4 , 4 , 4 , 4 , 4 , 4] colI :[1 , 3 , 4 ,
[Table]
[Table]
[Table]
[Table]
[Table]
rowPtr
[Table]
[Table]
[Table]
[Table]
If we assume the same bit-size per element for all arrays, then the CSR data structure does not attain higher compression gains in spite of not saving the zero valued elements (62 entries vs. 60 that are being required by the dense data structure).
CER format: Firstly, notice that many elements in M share the same value.
In fact, only the four values Ω = { 0 , 4 , 3 , 2 } appear in the entire matrix.
Hence, it appears reasonable to assume that data structures that repeatedly store these values (such as the dense or CSR structures) induce high redundancies in their representation.
Secondly, notice that different elements appear more frequent than others, and their relative relation does not change throughout the rows of the matrix, and the ordering.
Concretely, we have a set of unique elements
Ω
= { 0 , 4 , 3 , 2 } which appear
P # = { 32 , 21 , 4 , 3 } times respectively in the matrix, and we obtain the same relative order of highest to lowest frequent value throughout the rows of the matrix.
Hence, we can design an efﬁcient data structure which leverages on both properties in the following way:
Store unique elements present in the matrix in an array in frequency-major order (that is, from most to least frequent).
We name this array Ω .
Store respectively the column indices in row-major order, excluding the ﬁrst element (thus excluding the most frequent element).
We denote it as colI . 3.
Store pointers that signalize when the positions of the next new element in Ω starts.
We name it Ω Ptr .
If a particular pointer in Ω Ptr is the same as the previous one, this means that the current element is not present in the matrix and we jump to the next element.
Store pointers that signalize when a new row starts.
We name it rowPtr .
Hence, this new data structure represents matrix M as
Ω :[0 , 4 , 3 , 2] colI :[4 , 9 , 11 , 1 , 8 , 3 , 7 , 0 , 1 , 5 , 8 , 9 , 11 , 0 , 3 , 7 , 2 , 9 , 3 , 4 , 5 , 8 , 9 , 7 , 1 , 2 , 5 , 7] Ω Ptr :[0 , 3 , 5 , 7 , 13 , 16 , 17 , 18 , 23 , 24 , 28] rowPtr :[0 , 3 , 4 , 7 , 9 , 10]
Notice that we can uniquely reconstruct M from this representation.
One can verify that indeed, the CER data structure only requires 49 entries (instead of 60 or 62) attaining as such a compressed representation of the matrix M .
CSER format:
In some cases, it may well be that the probability distribution across rows in a matrix are not similar to each other.
Hence, the second assumption in the CER data structure would not apply and we would only be left with the ﬁrst one.
That is, we only know that not many distinct elements appear per row in the matrix or, in other words, that many elements share the same value.
The compressed shared elements row (or CSER in short) data structure is a slight extension to the previous
CER representation.
Here, we add an element pointer array, which signalizes which element in Ω the positions are referred to (we called it Ω I ).
Hence, the above matrix would then be represented as follows
[Equation]
[Equation]
, 4] colI :[4 , 9 , 11 , 1 , 8 , 3 , 7 , 0 , 1 , 5 , 8 , 9 , 11 , 0 , 3 , 7 , 2 , 9 , 3 , 4 , 5 , 8 , 9 , 7 , 1 , 2 , 5 ,
Ω I :[3 , 2 , 1 , 3 , 3 , 2 , 1 , 3 , 2 ,
Ω Ptr :[0 , 3 , 5 , 7 , 13 , 16 , 17 , 18 , 23 , 24 ,
rowPtr :[0 , 3 , 4 , 7 , 9 ,
Thus, for storing matrix M we require 59 entries, which is still a gain but not a signiﬁcant one.
Notice, that now the ordering of the elements in Ω is not important anymore.
A.2 Dot product complexity
We just saw that we can attain gains with regard to compression if we represent the matrix in the CER and CSER data structures.
However, we can also devise corresponding dot product algorithms that are more efﬁcient than their dense and sparse counterparts.
As an example, consider only the scalar product between the second row of matrix M with an arbitrary input vector a =
[ a 1 a 2 . . .
a 12 ] (cid:62) .
In principle, the difference in the algorithmic complexity arises because each data structure implicitly encodes a different expression of the scalar product, namely
[Equation]
For instance, the dot product algorithm associated to the dense format would calculate the above scalar product by
loading M and a . 2.
calculating 4 a 0 + 4 a 1 + 0 a 2 + 0 a 3 + 0 a 4 + 4 a 5 + 0 a 6 + 0 a 7 + 4 a 8 + 4 a 9 + 0 a 10 + 4 a 11 .
This requires 24 load (12 for the matrix elements and 12 for the input vector elements), 12 multiply,add and 1 write operations (for writing the result into memory).
We purposely omitted the accumulate operation which stores the intermediate values of the multiply-sum operations, since their cost can effectively be associated to the sum operation.
Moreover, we only considered read/write operations from and into memory.
Hence, this makes 48 operations in total.
In contrast, the dot product algorithm associated with the CSR representation would only multiply-add the non-zero entries.
It does so by performing the following steps
Load the subset of rowPtr respective to row 2.
Thus, rowPtr → [7 , 13] . 2.
Then, load the respective subset of non-zero elements and column indices.
Thus, W → [4 , 4 , 4 , 4 , 4 , 4] and colI → [0 , 1 , 5 , 8 , 9 , 11] .
Finally, load the subset of elements of a respective to the loaded subset of column indices and subsequently multiply-add them to the loaded subset of W .
{+Thus, a → [ a 0 , a 1 , a 5 , a 8 , a 10 , a 11 ] and calculate 4 a 
+}
+ 4 a 1 + 4 a 5 + 4 a 8 + 4 a 9 + 4 a 11 .
By executing this algorithm we would require 20 load operations (2 from the rowPtr and 6 for the
W , the colI and the input vector respectively)
6 multiplications
5 additions and 1 write.
In total this dot product algorithm
requires
32 operations.
However, we can still see that the above dot product algorithm is inefﬁcient in this case since we constantly multiply by the same element 4.
Instead, the dot product algorithm associated to, e.g., the
CER data structure, would perform the following steps
Load the subset of rowPtr respective to row 2.
Thus, rowPtr → [3 , 4] . 2.
Load the corresponding subset in Ω Ptr .
Thus, Ω Ptr → [7 , 13] . 3.
For each pair of elements in Ω Ptr , load the respective subset in colI and the element in Ω .
Thus, Ω → [4] and colI → [0 , 1 , 5 , 8 , 9 , 11] .
For each loaded subset of colI , perform the sum of the elements of a respective to the loaded colI .
{+Thus, a → [ a 0 , a 1 , a 5 , a 8 , a 10 , a 11 ] and do a 
+}
{++ a 1 + a 5 + a 8 + a 9 + a 
+}
=
z .
Subsequently, multiply the sum with the respective element in Ω .
Thus, compute 4 z .
A similar algorithm can be devised for the CSER data structure.
One can ﬁnd both pseudocodes in the appendix.
The operations required by this algorithm are 17 load operations (2 from rowPtr ,from Ω Ptr , 1 from Ω , 6 from colI and 6 from a ), 1 multiplication, 5 additions and 1 write.
In total this are 24 operations. Hence, we have observed that for the matrix M , the CER (and CSER) data structure does not only achieve higher compression rates , but it also attains gains in efﬁciency with respect to the dot product operation. B CER and CSER dot product pseudocodes Algorithms 1 and 2 show the pseudocodes of the dot product algorithm of the CER and CSER data structures.
1: procedure DOT cer (M,X) 2: Ω , colI, wPtr, rowPtr ← M 3:
N, L ← dim ( X ) 4:
Y = 0 ∈ R M ×
L 5: for l ≤ L do 6: r start =7: w start =8: for r idx = 1 < len ( rowPtr )
do 9: r end ← rowPtr
[ r idx ] 10: y =11: w count =12: for w idx = r start + 1 < r end + 1 do 13: w end ← wPtr [ w idx ] 14: y (cid:48)
=15: for i =
w start < w end do 16:
I ← colI
[ i ] 17: y (cid:48)
← y (cid:48)
+ X
[ I, l ] 18: if w start + 1 = w end then 19: y ← y + y (cid:48)
∗ Ω[ w count ] 20: w count ← w count +21:
w start ← w end 22: r start ← r end 23:
Y [ m, k ] ← y 24: return Y
1: procedure DOT cser (M,X) 2: Ω , colI, wI, wPtr, rowPtr ← M 3:
N, L ← dim ( X ) 4:
Y = 0 ∈ R M ×
L 5: for l ≤ L do 6: r start =7: w start =8: w count =9: for r idx = 1 < len ( rowPtr ) do 10: r end ← rowPtr
[ r idx ] 11: y =12: for w idx = r start + 1 < r end + 1 do 13: w end ← wPtr [ w idx ] 14: y (cid:48)
=15: for i =
w start < w end do 16:
I ← colI
[ i ] 17: y (cid:48)
← y (cid:48)
+ X
[ I, l ] 18: y ← y + y (cid:48) ∗ Ω[ wI [ w count ]] 19: w count ← w count
+20: w start ← w end 21: r start ← r end 22:
Y [ m, k ] ← y 23: return Y