Succinct Source Coding of Deep Neural Networks
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Deep neural networks have shown incredible performance for inference tasks ina variety of domains.
Unfortunately, most current deep networks are enormouscloud-based structures that require signiﬁcant storage space, which limits scalingof deep learning as a service (DLaaS) and use for on-device augmented intelligence.
This paper ﬁnds algorithms that directly use lossless compressed representations of deep feedforward networks (with synaptic weights drawn from discretesets), to perform inference without full decompression.
The basic insight thatallows less rate than naïve approaches is the recognition that the bipartite graphlayers of feedforward networks have a kind of permutation invariance to the labeling of nodes, in terms of inferential operation and that the inference operationdepends locally on the edges directly connected to it.
We also provide experimental results of our approach on the MNIST dataset.
Introduction
Deep learning has achieved incredible performance for inference tasks such as speech recognition,image recognition and natural language processing.
Most current deep neural networks, however,are enormous cloud-based structures that are too large and too complex to perform fast, energyefﬁcient inference on device or for scaling deep learning as a service (DLaaS).
Compression, withthe capability of providing inference without full decompression, is important.
Universal source coding for feedforward deep networks having synaptic weights drawn from ﬁnite sets that essentiallyachieve the entropy lower bound were introduced in (1).
Here, we provide—for the ﬁrst time—analgorithm that directly uses these compressed representations for inference tasks without completedecompression.
Structures that can represent information near the entropy bound while also allowing efﬁcient operations on them are called succinct structures (2; 3; 4).
Thus, we provide a succinctstructure for feedforward neural networks, which may ﬁt on-device and enable scaling of DLaaS.
Related Work: There has been recent interest in compact representations of neural networks(5; 6; 7; 8; 9; 10; 11; 12; 13; 14).
While most of these algorithms are lossy, we provide an efﬁcientlossless algorithm, which can be used on top of any lossy algorithm that quantizes or prunes network weights; prior work on lossless compression of neural networks either used Huffman coding ina way that did not exploit invariances or was not succinct and required full decompression for inference.
The proposed algorithm builds on the sublinear entropy-achieving representation in (1) but isthe ﬁrst time succinctness—the further ability to perform inference with negligible space needed forpartial decompression—has been attempted or achieved.
Our inference algorithm is similar to arithmetic decoding and so computational performance is also governed by efﬁcient implementations ofarithmetic coding.
Efﬁcient high-throughput implementations of arithmetic coding/decoding havebeen developed for video, e.g. as part of the H.264/AVC and HEVC standards (15; 16).
Feedforward neural network structure
Let us describe the neural network model considered in (1) which will be used here to developsuccinct structures of deep neural networks.
In a feedforward neural network, each node j com- xxa a U V   putes an activation function g ( · ) applied to the weighted sum of its inputs, which we can note isa permutation-invariant function: a j = g ( (cid:80) i w ij a i ) =
g (cid:0)(cid:80) i
w π ( i )
j a π ( i ) (cid:1) , for any permutationπ .
Consider a feedforward neural network with K − 1 hidden layers where each node contains Nnodes (for notational convenience) such that the nodes in all the K − 1 hidden layers are indistinguishable from each other (when edges are ignored) but the nodes in the input and output layersare labeled and can be distinguished.
There is an edge of color i , i = 0 , . . .
, m , between any twonodes from two different layers independently with probability p i , where p 0 is the probability of noedge.
Consider a substructure: partially-labeled bipartite graphs , see Fig. 1, which consists of twosets of vertices containing N vertices each with one of the sets containing labeled vertices and theother set containing unlabeled vertices.
An edge of color i exists between any two nodes taken onefrom each set with probability p i , i = 0 , . . .
, m where p 0 is the probability of no edge.
Refer to(1) for detailed discussion on the structure.
To construct the K -layer neural network, think of it asmade of a partially-labeled bipartite graph for the ﬁrst two layers but then each time the nodes of anunlabeled layer are connected, we treat it as a labeled layer, based on its connection to the previouslabeled layer (i.e. we can label the unlabeled nodes based on the nodes of the previous layer it isconnected to), and iteratively complete the K -layer neural network.
Succinct representation of a partially-labeled bipartite graph
First, we consider the succinct representation of a partially labeled bipartite graph, followed by thatof a K -layered neural network.
Alg. 1 is an inference algorithm for a partially-labeled bipartitegraph with input to the graph X and output Y .
Later we use this algorithm to make inferences ina K -layered neural network where outputs of unlabeled layers correspond to outputs of a hiddenlayer.
The optimally compressed representation of a partially-labeled bipartite graph produced by(1, Alg.1) is taken as an input by Alg. 1, in addition to the input X to the graph, and the output Yof the graph is given out.
If the graph has N nodes in each layer, then only an additional O ( N ) bitsof dynamic space is required by Alg.
1 for the inference task while it takes O ( N 2 ) bits to store therepresentation and hence the structure in succinct as discussed below.
Lemma 1.
Output Y obtained from Alg.
1 is a permutation of ˜ Y , the output from the uncompressedneural network representation.
Proof.
Say, we have an m × 1 vector X to be multiplied with an m × n weight matrix W , to get theoutput ˜ Y , an n × 1 vector.
Then, ˜ Y = W T X , and so the j th element of ˜ Y , ˜ Y
j = (cid:80) m i =1 W Tj,i x i .
In
Alg. 1, while traversing a particular depth i , we multiply all Y j s with X
i W i,j and hence when wereach depth N , we get the Y vector as required.
The change in permutation of ˜ Y with respect to Yis because while compressing W , we do not encode the permutation of the columns, retaining therow permutation.
Theorem 2.
The additional dynamic space requirement of Alg.
1 is O ( N ) , and hence the compressed representation formed in (1, Alg. 1) is succinct.
Proof.
The major dynamic space requirement is for decoding of individual nodes, and the queue, Q .
Clearly, the space required for Q , is much more than the space required for decoding a single node.
We show the expected space complexity corresponding to Q is less than or equal to 2( m + 1) N (1 +2 log 2 ( m +2 m +1 )) using Elias-Gamma integer codes for each entry in Q .
Note that Q has nodes fromat most two consecutive depths, and since only the child nodes of non-zero nodes are encoded, andthe number of non-zero nodes at any depth is less than N , we can have a maximum of 2( m + 1) Nnodes encoded in Q .
Let α 0 , ..., α k be the non-zero tree nodes at some depth d of the tree, wherek = ( m +1) N .
Let S be the total space required to store Q .
Using integer codes, we can encode anypositive number x in 2 log 2 ( x ) + 1 bits, and to allow 0 , we need 2 log 2 ( x + 1) + 1 bits.
Thus, the
1: Input neural compressed representation of the partially-labeled bipartite graph obtained from
Output y y N vector the neural network, compressed
3: Initialize: Y
0] , d neurons the current depth, j = 1 , an empty queue Q , an empty string L return the compressed representation
L once the has executed
Q 4: while Q is not empty and d ≤ do 5: Set i = 0 . Set f
= the ﬁrst element obtained after dequeuing Q .
6: while i ≤ m and f > 0 do 7: decode the child node of f corresponding to color i and store it as c .
Encode c back in L 1 .
Enqueue c in Q .
Add x l × w i to each of y j to y ( j + c ) .
Add c to j .
10: if j = 1 , at least one non-zero node has been processed at the current depth then 11: d = d +12: end if 13: end while 14: end while 15: Update the Y vector using the required activation function.
[Table]
[Table]
[Table]
M = 50 , N = M = 50 , N =
M = 50 , N =    arithmetic-geometric inequality implies S ≤ 2( (cid:80) 2 log 2 ( α i + 1) + 1) ≤ 2 N ( m + 1) + 4 N ( m +1) log 2 ( m +2 m +1 ) .
From Prop. 1 and (1, Theorem 1), the additional dynamic space for Alg. 1 is O ( N ) ,while the entropy of a partially-labeled bipartite graph is O ( N 2 ) .
Thus, the structure is succinct.
Now consider the structure of the K -layered neural network as in Sec. 2 and provide its succinctrepresentation.
The extra dynamic space for K -layers remains the same as for 2-layers as describedin Alg. 1 as inference is done one layer at a time.
Theorem 3.
The compressed structure obtained by the iterative use of (1, Alg. 1) is succinct.
Experiments
We trained a feedforward neural network of dimension 784 × 50 × 50 × 50 × 50 × 10 on the
MNIST dataset using gradient descent algorithm to get 98 .
4% accuracy on the test data.
Networkweights were quantized using a uniform quantizer into 33 steps to get a network with an accuracyof 97 .
5% on the training data and an accuracy of 93 .
48% on the test data.
The weight matricesfrom the second to the last layer were rearranged based on the weight matrices corresponding tothe previous layers as needed for Alg. 1 to work.
These matrices, except the last matrix connectedto the output, were compressed using (1, Alg. 1) to get the compressed network, and arithmeticcoding was implemented by modiﬁcation of an existing implementation.
The compressed networkperforms exactly as the quantized network as it should, since we compress losslessly.
We observethat the extra memory required for inference is negligible compared to the size of the compressednetwork.
Detailed results from the experiment and dynamic space requirements are described in
Table 1, where H ( p ) is the empirical entropy calculated from the weight matrices.