W EIGHTLESS : LOSSY WEIGHT ENCODING FOR DEEP NEURAL NETWORK COMPRESSION
Anonymous authors
Paper under double-blind review
A BSTRACT
The large memory requirements of deep neural networks limit their deployment and adoption on many devices.
Model compression methods effectively reduce the memory requirements of these models, usually through applying transformations such as weight pruning or quantization.
In this paper, we present a novel scheme for lossy weight encoding which complements conventional compression techniques.
The encoding is based on the Bloomier ﬁlter, a probabilistic data structure that can save space at the cost of introducing random errors.
Leveraging the ability of neural networks to tolerate these imperfections and by re-training around the errors, the proposed technique, Weightless, can compress DNN weights by up to 496 × with the same model accuracy.
This results in up to a 1.51 × improvement over the state-of-the-art.
I NTRODUCTION
The continued success of deep neural networks (DNNs) comes with increasing demands on compute, memory, and networking resources.
Moreover, the correlation between model size and accuracy suggests that tomorrow’s networks will only grow larger.
This growth presents a challenge for resource-constrained platforms such as mobile phones and wireless sensors.
As new hardware now enables executing DNN inferences on these devices (Apple, 2017; Qualcomm, 2017), a practical issue that remains is reducing the burden of distributing the latest models especially in regions of the world not using high-bandwidth networks.
For instance, it is estimated that, globally,million users will be using 2G networks by 2020 (GSMA, 2014), which can take up to 30 minutes to download just 20MB of data.
By contrast, today’s DNNs are on the order of tens to hundreds of MBs, making them difﬁcult to distribute.
In addition to network bandwidth, storage capacity on resource-constrained devices is limited, as more applications look to leverage DNNs.
Thus, in order to support state-of-the-art deep learning methods on edge devices, methods to reduce the size of DNN models without sacriﬁcing model accuracy are needed.
Model compression is a popular solution for this problem.
A variety of compression algorithms have been proposed in recent years and many exploit the intrinsic redundancy in model weights.
Broadly speaking, the majority of this work has focused on ways of simplifying or eliminating weight values (e.g., through weight pruning and quantization), while comparatively little effort has been spent on devising techniques for encoding and compressing.
In this paper we propose a novel lossy encoding method, Weightless , based on Bloomier ﬁlters, a probabilistic data structure (Chazelle et al., 2004).
Bloomier ﬁlters inexactly store a function map, and by adjusting the ﬁlter parameters, we can elect to use less storage space at the cost of an increasing chance of erroneous values.
We use this data structure to compactly encode the weights of a neural network, exploiting redundancy in the weights to tolerate some errors.
In conjunction with existing weight simpliﬁcation techniques, namely pruning and clustering, our approach dramatically reduces the memory and bandwidth requirements of DNNs for over the wire transmission and ondevice storage.
Weightless demonstrates compression rates of up to 496 × without loss of accuracy, improving on the state of the art by up to 1.51 × .
Furthermore, we show that Weightless scales better with increasing sparsity, which means more sophisticated pruning methods yield even more beneﬁts.
This work demonstrates the efﬁcacy of compressing DNNs with lossy encoding using probabilistic data structures.
Even after the same aggressive lossy simpliﬁcation steps of weight pruning and clustering (see Section 2), there is still sufﬁcient extraneous information left in model weights to allow an approximate encoding scheme to substantially reduce the memory footprint without loss of model accuracy.
Section 3 reviews Bloomier ﬁlters and details Weightless.
State-of-the-art compression results using Weightless are presented in Section 4.
Finally, in Section 4.3 shows that Weightless scales better as networks become more sparse compared to the previous best solution.
R ELATED W ORK
Our goal is to minimize the static memory footprint of a neural network without compromising accuracy.
Deep neural network weights exhibit ample redundancy, and a wide variety of techniques have been proposed to exploit this attribute.
We group these techniques into two categories: (1) methods that modify the loss function or structure of a network to reduce free parameters and (2) methods that compress a given network by removing unnecessary information.
The ﬁrst class of methods aim to directly train a network with a small memory footprint by introducing specialized structure or loss.
Examples of specialized structure include low-rank, structured matrices of Sindhwani et al.
and
randomly-tied weights of Chen et al.
Examples of specialized loss include teacher-student training for knowledge distillation (Bucila et al., 2006; Hinton et al., 2015) and diversity-density penalties (Wang et al., 2017).
These methods can achieve signiﬁcant space savings, but also typically require modiﬁcation of the network structure and full retraining of the parameters.
An alternative approach, which is the focus of this work, is to compress an existing, trained model.
This exploits the fact that most neural networks contain far more information than is necessary for accurate inference (Denil et al., 2013).
This extraneous information can be removed to save memory.
Much prior work has explored this opportunity, generally by applying a two-step process of ﬁrst simplifying weight matrices and then encoding them in a more compact form.
Simpliﬁcation changes the number or characteristic of weight values to reduce the information needed to represent them.
For example, pruning by selectively zeroing weight values (LeCun et al., 1989;
Guo et al., 2016) can, in some cases, eliminate over 99% of the values without penalty.
Similarly, most models do not need many bits of information to represent each weight.
Quantization collapses weights to a smaller set of unique values, for instance via reduction to ﬁxed-point binary representations (Gupta et al., 2015) or clustering techniques (Gong et al., 2014).
Simplifying weight matrices can further enable the use of more compact encoding schemes, improving compression.
For example, two recent works Han et al.
; Choi et al.
(2017) encode pruned and quantized DNNs with sparse matrix representations.
In both works, however, the encoding step is a lossless transformation, applied on top of lossy simpliﬁcation.
W EIGHTLESS
Weightless is a lossy encoding scheme based around Bloomier ﬁlters.
We begin by describing what a
Bloomier ﬁlter is, how to construct one, and how to retrieve values from it.
We then show how we encode neural network weights using this data structure and propose a set of augmentations to make it an effective compression strategy for deep neural networks.
3. T HE B LOOMIER FILTER
A Bloomier ﬁlter generalizes the idea of a Bloom ﬁlter (Bloom, 1970), which are data structures that answer queries about set membership.
Given a subset S of a universe U , a Bloom ﬁlter answers queries of the form, “Is v ∈ S ?”.
If v is in S , the answer is always yes; if v is not in S , there is some probability of a false positive, which depends on the size of the ﬁlter, as size is proportional to encoding strength.
By allowing false positives, Bloom ﬁlters can dramatically reduce the space needed to represent the set.
A Bloomier ﬁlter (Chazelle et al., 2004) is a similar data structure but instead encodes a function.
For each v in a domain S , the function has an associated value f ( v ) in the range R =
[0 , 2 r ) .
Given an input v , a Bloomier ﬁlter always returns f ( v ) when v is in S .
When v is not in S , the Bloomier ﬁlter returns a null value ⊥ , except that some fraction of the time there is a “false positive”, and the Bloomier ﬁlter returns an incorrect, non-null value in the range R .
X
H 
 H
W ' l
H
W l m 0 0 0 0 0 0 100 0 0 00 0 0 0 0 1 0 0 0 000 1 0 00 00 0 0 0 0 0 100 1 0 00
t M M 0 0 0 0 0
r w' i,j
HM
Decoding Let S be the subset of values in U to store, with | S | = n .
A Bloomier ﬁlter uses a small number of hash functions (typically four), and a hash table X of m = cn cells for some constant c (1.25 in this paper), each holding t > r bits.
For hash functions H 0 , H 1 , H 2 , H M , let
H 0 , 1 , 2 ( v ) → [0 , m ) and H M ( v ) → [0 , 2 r ) , for any v ∈ U .
The table X is set up such that for every v ∈ S ,
[Equation]
Hence, to ﬁnd the value of f ( v ) , hash v four times, perform three table lookups, and exclusive-or together the four values.
Like the Bloom ﬁlter, querying a Bloomier ﬁlter runs in O (1) time.
For u / ∈ S , the result, X H 0 ( u ) ⊕ X H 1 ( u ) ⊕ X H 2 ( u ) ⊕ H M ( u ) , will be uniform over all t -bit values.
If this result is not in [0 , 2 r ) , then ⊥ is returned and if it happens to land in [0 , 2 r ) , a false positive occurs and a result is (incorrectly) returned.
An incorrect value is therefore returned with probability 2 r − t .
Encoding Constructing a Bloomier ﬁlter involves ﬁnding values for X such that the relationship above holds for all values in S .
There is no known efﬁcient way to do so directly.
All published approaches involve searching for conﬁgurations with randomized algorithms.
In their paper introducing Bloomier ﬁlters, Chazelle et al.
(2004) give a greedy algorithm which takes O ( n log n ) time and produces a table of size (cid:100) cn (cid:101) t bits with high probability.
Charles & Chellapilla (2008) provide two slightly better constructions.
First, they give a method with identical space requirements but runs in O ( n ) time.
They also show a separate O ( n log n ) -time algorithm for producing a smaller table with c closer to 1.
Using a more sophisticated algorithm for construction should allow for a more compact table and, by extension, improve the overall compression rate.
However, we leave this for future work and use the method of (Chazelle et al., 2004) for simplicity.
While construction can be expensive, it is a one-time cost.
Moreover, the absolute runtime is small compared to the time it takes to train a deep neural network.
In the case of VGG-16, our unoptimized
Python code built a Bloomier ﬁlter within an hour.
We see this as a small worthwhile overhead given the savings offered and in contrast to the days it can take to fully train a network.
3. A PPROXIMATE WEIGHT ENCODING WITH B LOOMIER FILTERS
We propose using the Bloomier ﬁlter to compactly store weights in a neural network.
The function f encodes the mapping between indices of nonzero weights to their corresponding values.
Given a
[Figure]
weight matrix W , deﬁne the domain S to be the set of indices { i, j | w i,j (cid:54) = 0 } .
Likewise, the range
R is [ − 2 a − 1 , 2 a − 1 ) − { 0 } for a such that all values of W fall within the interval.
Due to weight value clustering (see below) this range is remapped to [0 , 2 r − 1 ) and encodes the cluster indices.
A null response from the ﬁlter means the weight has a value of zero.
Once f is encoded in a ﬁlter, an approximation W (cid:48) of the original weight matrix is reconstructed by querying it with all indices.
The original nonzero elements of W are preserved in the approximation, as are most of the zero elements.
A small subset of zero-valued weights in W (cid:48) will take on nonzero values as a result of random collisions in X , possibly changing the model’s output.
Figure 1 illustrates the operation of this scheme: An original nonzero is correctly recalled from the ﬁlter on the right and a false positive is created by an erroneous match on the left (red).
Complementing Bloomier ﬁlters with simpliﬁcation Because the space used by a Bloomier ﬁlter is O ( nt ) , they are especially useful under two conditions: (1) The stored function is sparse (small n , with respect to | U | ) and (2) It has a restricted range of output values (small r , since t > r ).
To improve overall compression, we pair approximate encoding with weight transformations.
Pruning networks to enforce sparsity (condition 1) has been studied extensively (Hassibi & Stork, 1993; LeCun et al., 1989).
In this paper, we consider two different pruning techniques: (i) magnitude threshold plus retraining and (ii) dynamic network surgery (DNS) (Guo et al., 2016).
Magnitude pruning with retraining as straightforward to use and offers good results.
DNS is a recently proposed technique that prunes the network during training.
We were able to acquire two sets of models,
LeNet-300-100 and LeNet5, that were pruned using DNS and include them in our evaluation; as no reference was published for VGG-16 only magnitude pruning is used.
Regardless of how it is accomplished, improving sparsity will reduce the overall encoding size linearly with the number of non-zeros with no effect on the false positive rate (which depends only on r and t ).
The reason for using two methods is to demonstrate the beneﬁts of Weightless as networks increase in sparsity, the
DNS networks are notably more sparse than the same networks using magnitude pruning.
Reducing r (condition 2) amounts to restricting the range of the stored function or minimizing the number of bits required to represent weight values.
Though many solutions to discretize weights exist (e.g., limited binary precision and advanced quantization techniques Choi et al.
, we use k -means clustering.
After clustering the weight values, the k centroids are saved into an auxiliary table and the elements of W are replaced with indices into this table.
This style of indirect encoding is especially well-suited to Bloomier ﬁlters, as these indices represent a small, contiguous set of integers.
Another beneﬁt of using Bloomier ﬁlters is that k does not have to be a power of 2.
When decoding Bloomier ﬁlters, the result of the XORs can be checked with an inequality, rather than a bitmask.
This allows Bloomier ﬁlters to use k exactly, reducing the false positive rate by a factor of 1 − k 2 r .
In other methods, like that of Han et al.
, there is no beneﬁt, as any k not equal to a power of two strictly wastes space.
Tuning the t hyperparameter The use of Bloomier ﬁlters introduces an additional hyperparameter t , the number of bits per cell in the Bloomier ﬁlter.
t trades off the Bloomier ﬁlter’s size and the false positive rate which, in turn, effects model accuracy.
While t needs to be tuned, we ﬁnd it far easier to
[Figure]
reason about than other DNN hyperparameters.
Because we encode k clusters, t must be greater than (cid:100) log 2 k (cid:101) , and each additional t bit reduces the number of false positives by a factor of 2.
This limits the number of reasonable values for t : when t is too low the networks experience substantial accuracy loss, but also do not beneﬁt from high values of t because they have enough implicit resilience to handle low error rates (see Figure 2).
Experimentally we ﬁnd that t typically falls in the range of 6 to 9 for our models.
Retraining to mitigate the effects of false positives We encode each layer’s weights sequentially.
Because the weights are ﬁxed, the Bloomier ﬁlter’s false positives are deterministic.
This allows for the retraining of deeper network layers to compensate for errors.
It is important to note that encoded layers are not retrained.
The randomness of the Bloomier ﬁlter would sacriﬁce all beneﬁts of mitigating the effects of errors.
If the encoded layer was retrained, a new encoding would have to be constructed (because S changes) and the indices of weights that result in false positives would differ after every iteration of retraining.
Instead, we ﬁnd retraining all subsequent layers to be effective, typically allowing us to reduce t by one or two bits.
The process of retraining around faults is relatively cheap, requiring on the order of tens of epochs to converge.
The entire optimization pipeline is shown in Figure 3.
Compressing Bloomier ﬁlters When sending weight matrices over a network or saving them to disk, it is not necessary to retain the ability to access weight values as they are being sent, so it is advantageous to add another layer of compression for transmission.
We use arithmetic coding, an entropy-optimal stream code which exploits the distribution of values in the table (MacKay, 2005).
Because the nonzero entries in a Bloomier ﬁlter are, by design, uniformly distributed values in [1 , 2 t − 1) , improvements from this stage largely come from the prevalence of zero entries.
E XPERIMENTS
We evaluate Weightless on three deep neural networks commonly used to study compression: LeNet300-100, LeNet5 (LeCun et al., 1998), and VGG-16 (Simonyan & Zisserman, 2015).
The LeNet networks use MNIST (Lecun & Cortes) and VGG-16 uses ImageNet (Russakovsky et al., 2015).
The networks are trained and tested in Keras (Chollet, 2017).
The Bloomier ﬁlter was implemented in house and uses a Mersenne Twister pseudorandom number generator for uniform hash functions.
To reduce the cost of constructing the ﬁlters for VGG-16, we shard the non-zero weights into ten separate ﬁlters, which are built in parallel to reduce construction time.
Sharding does not signiﬁcantly affect compression or false positive rate as long as the number of shards is small (Broder & Mitzenmacher, 2004).
We focus on applying Weightless to the largest layers in each model, as shown in Table 1.
This corresponds to the ﬁrst two fully-connected layers of LeNet-300-100 and VGG-16.
For LeNet5, the second convolutional layer and the ﬁrst fully-connected layer are the largest.
These layers account for 99.6%, 99.7%, and 86% of the weights in LeNet5, LeNet-300-100, and VGG-16, respectively.
[Table]
(The DNS version is slightly different than magnitude pruning, however the trend is the same.)
While other layers could be compressed, they offer diminishing returns.
Compression baseline The results below present both the absolute compression ratio and the improvements Weightless achieves relative to Deep Compression (Han et al., 2016)
[Table]
which represents the current state-of-the-art
The absolute compression ratio is with respect the original standard 32-bit weight size of the models.
This baseline is commonly used to report results in publications and, while larger than many models used in practice, it provides the complete picture for readers to draw their own conclusions.
For comparison to a more aggressive baseline, we reimplemented Deep
Compression in Keras.
Deep Compression implements a lossless optimization pipeline where pruned and clustered weights are encoded using compressed sparse row encoding (CSR) and then compresses
CSR encoding tables with Huffman coding.
The compression achieved by Deep Compression we use as a baseline is notably better than the original publication (e.g., VGG-16 FC-0 went from 91 × to 119 × ).
Error baseline Because Weightless performs lossy compression, it is important to bound the impact of the loss.
We establish this bound as the error of the trained network after the simpliﬁcation steps (i.e., post pruning and clustering).
In doing so, the test errors from compressing with Weightless and
Deep Compression are the same (shown as Baseline Error % in Table 1).
Weightless is sometimes slightly better due to training ﬂuctuations, but never worse.
While Weightless does provide a tradeoff between compression and model accuracy, we do not consider it here.
Instead, we feel the strongest case for this method is to compare against a lossless technique with iso-accuracy and note compression ratio will only improve in any use case where degradation in model accuracy is tolerable.
4. S PARSE W EIGHT E NCODING
Given a simpliﬁed baseline model, we ﬁrst evaluate the how well Bloomier ﬁlters encode sparse weights.
Results for Bloomier encoding are presented in Table 2, and show that the ﬁlters work exceptionally well.
In the extreme case, the large fully connected layer in LeNet5 is compressed by 445 × .
With encoding alone and demonstrates a 1.99 × improvement over CSR, the alternative encoding strategy used in Deep Compression.
Recall that the size of a Bloomier ﬁlter is proportional to mt , and so sparsity and clustering determine how compact they can be.
Our results suggest that sparsity is more important than the number of clusters for reducing the encoding ﬁlter size.
This can be seen by comparing each LeNet models’ magnitude pruning results to those of dynamic network surgery—while DNS needs additional clusters, the increased sparsity results in a substantial size reduction.
We suspect this is due to the ability of
DNNs to tolerate a high false positive rate.
The t value used here is already on the exponential part of the false positive curve (see Figure 2).
At this point, even if k could be reduced, it is unlikely t can be since the additional encoding strength saved by reducing k does little to protect against the
[Table]
This is why we see Bloomier ﬁlters work so well here–most applications cannot function with this level of approximation, nor do they have an analogous retrain mechanism to mitigate the errors’ effects.
[Table]
4. C OMPRESSING W EIGHT E NCODINGS
For sending a model over a network
[Table]
an additional stage of compression can be used to optimize for size.
Han et al.
(2016) propose using Huffman coding for their
[Table]
and we use arithmetic coding, as described in Section 3.2.
The results in Table 3 show that while Deep Compression gets relatively more beneﬁt from a ﬁnal compression stage, Weightless remains a substantially better scheme overall.
Prior work by Mitzenmacher (2002) on regular Bloom ﬁlters has shown that they can be optimized
[Figure]
for better post-compression size.
We believe a similar method could be used on Bloomier ﬁlters, but we leave this for future work.
4. S CALING WITH S PARSITY
Recent work continues to demonstrate better ways to extract sparsity from DNNs (Guo et al., 2016;
Ullrich et al., 2017; Narang et al., 2017), so it is useful to quantify how different encoding techniques scale with increasing sparsity.
As a proxy for improved pruning techniques, we set the threshold for magnitude pruning to produce varying ratios of nonzero values for LeNet5 FC-0.
We then perform retraining and clustering as usual and compare encoding with Weightless and Deep Compression (all without loss of accuracy).
Figure 4 shows that as sparsity increases, Weightless delivers far better compression ratios.
Because the false positive rate of Bloomier ﬁlters is controlled independent of the number of nonzero entries and addresses are hashed not stored, Weightless tends to scale very well with sparsity.
On the other hand, as the total number of entries in CSR decreases, the magnitude of every index grows slightly, offsetting some of the beneﬁts.
C ONCLUSION
This paper demonstrates a novel lossy encoding scheme, called Weightless, for compressing sparse weights in deep neural networks.
The lossy property of Weightless stems from its use of the Bloomier ﬁlter, a probabilistic data structure for approximately encoding functions.
By ﬁrst simplifying a model with weight pruning and clustering, we transform its weights to best align with the properties of the Bloomier ﬁlter to maximize compression.
Combined, Weightless achieves compression of up to 496 × , improving the previous state-of-the-art by 1.51 × .
We also see avenues for continuing this line of research.
First, as better mechanisms for pruning model weights are discovered, end-to-end compression with Weightless will improve commensurately.
Second, the theory community has already developed more advanced—albeit more complicated— construction algorithms for Bloomier ﬁlters, which promise asymptotically better space utilization compared to the method used in this paper.
Finally, by demonstrating the opportunity for using lossy encoding schemes for model compression, we hope we have opened the door for more research on encoding algorithms and novel uses of probabilistic data structures.
A PPENDIX 6. C ONSTRUCTING B LOOMIER FILTERS
Given a set of key value pairs, in this case weight addresses and cluster indexes, a Bloomier ﬁlter is constructed as follows.
The task of construction is to ﬁnd a unique location in the table for each key such that the value can be encoded there.
For each to be encoded key a neighborhood of 3 hash digests are generated, indexes of these 3 are named iotas.
To begin, the neighborhoods of all keys are computed and the unique digests are identiﬁed.
Keys with unique neighbors are removed from the list of keys to be encoded.
When a key is associated with a unique location, its iota value (i.e., the unique neighbor index into the neighborhood of hashs), is saved in an ordered list along with the key itself.
The process is repeated for the remaining keys and continues until either all keys identify a unique location or none can be found.
In the case that this search fails a different hash function can be tried or a larger table is required (increasing m ).
Once the unique locations and ordering is known, the encoding can begin.
Values are encoded into the ﬁlter in the reverse order in which the unique locations are found during the search described above.
This is done such that as non-unique neighbors of keys collide, they can still resolve the correct encoded values.
For each key, the unique neighbor (indexed by the saved iota value) is set as the XOR of all neighbor ﬁlter entries (each a t -bit length vector), a random mask M , and the key’s value.
As the earlier found unique key locations are populated in the ﬁlter, it is likely that the neighbor values will be non-zero.
By XORing together them in reverse order the encoding scheme guarantees that the correct value is retrieved.
(The XORs can be thought of cancelling out all the erroneous information except that of the true value.)
An implementation of the Bloomier ﬁlter will be released along with publication.
6. S UPPLEMENTAL MATERIAL
Epochs 29.
V a li d a t i o n
E rr o r ( % )
t
[Figure]
( % ) k
=k
=k
=k
=
[Figure]