E NCODING W EIGHTS OF I RREGULAR S PARSITY FOR F IXED - TO -F IXED M ODEL C OMPRESSION
Anonymous authors
Paper under double-blind review
A BSTRACT
Even though ﬁne-grained pruning techniques achieve a high compression ratio, conventional sparsity representations (such as CSR) associated with irregular sparsity degrade parallelism signiﬁcantly.
Practical pruning methods, thus, usually lower pruning rates (by structured pruning) to improve parallelism.
In this paper, we study ﬁxed-to-ﬁxed (lossless) encoding architecture/algorithm to support ﬁnegrained pruning methods such that sparse neural networks can be stored in a highly regular structure.
We ﬁrst estimate the maximum compression ratio of encodingbased compression using entropy.
Then, as an effort to push the compression ratio to the theoretical maximum (by entropy), we propose a sequential ﬁxed-to-ﬁxed encoding scheme.
We demonstrate that our proposed compression scheme achieves almost the maximum compression ratio for the Transformer and ResNet-50 pruned by various ﬁne-grained pruning methods.
I NTRODUCTION
As one of the efﬁcient compression methods, pruning reduces the number of parameters by replacing model parameters of low importance with zeros (LeCun et al., 1990).
Since magnitude-based pruning has shown that pruning can be conducted with low computational complexity (Han et al., 2015), various practical pruning methods have been studied to achieve higher compression ratio (Zhu and
Gupta, 2017; Molchanov et al., 2017; Louizos et al., 2018; Gale et al., 2019).
Recently, pruning has been extended to a deeper understanding of weight initialization.
Based on the Lottery Winning Ticket hypothesis (Frankle and Carbin, 2018), (Renda et al., 2020) suggests a weight-rewinding method to explore sub-networks from full-trained models.
Furthermore, pruning methods at initialization steps without pre-trained models have also been proposed (Lee et al., 2019b; Wang et al., 2020).
Despite a high compression ratio, ﬁne-grained pruning that eliminates each parameter individually has practical issues to be employed in parallel computing platforms.
One of the popular formats to represent sparse matrices after pruning is the Compressed Sparse Row (CSR) whose structures are irregular.
For parallel computing, such irregular formats degrade inference performance that is dominated by matrix multiplications (Gale et al., 2020).
Algorithm 1 presents a conventional sparse matrix-vector multiplication (SpMV) algorithm using CSR format which involves irregular and data-dependent memory accesses.
Correspondingly, performance gain using a sparse matrix multiplication (based on CSR) is a lot smaller than the compression ratio of pruning (Yu et al., 2017).
Structured pruning methods (e.g, block-based pruning (Narang et al., 2017; Zhou et al., 2021), ﬁlter-based pruning (Li et al., 2017), and channel-based pruning (He et al., 2017; Liu et al., 2017)) have been suggested to enhance parallelism by restricting the locations of pruned weights.
Those methods, however, induce degraded accuracy and/or lower pruning rate than ﬁne-grained pruning.
In this paper, as an efﬁcient method to compress sparse NNs pruned by ﬁne-grained pruning, we consider weight encoding techniques.
As shown in Algorithm 2, encoded weights are multiplied by a ﬁxed binary matrix M ⊕ to reconstruct the original weights.
We propose an encoding method and
M ⊕ design methodology to compress sparse weights in a regular format.
It should be noted that a sparse matrix multiplication can be even slower than a dense matrix multiplication unless pruning rate is high enough (Yu et al., 2017; Gale et al., 2020) that does not happen with Algorithm 2 for memory-intensive workloads.
We study the maximum compression ratio of such encoding-based compression using entropy and propose a sequential ﬁxed-to-ﬁxed scheme that keeps high parallelism after ﬁne-grained pruning. We show that by our proposed ﬁxed-to-ﬁxed scheme, a compression ratio
[Figure]
can approach the maximum (estimated by entropy) even under the variation of the unpruned weights in a block.
Algorithm 1: SpMV (CSR format)
In: Dense vector x , CSR vectors dat , row , col
Out: Dense vector y for i ← 1 to n do for j ← row i to row i +
do y
i ← y
i + dat
[ j ] × x [ col [ j ]] /*Irregular, data-dependent access*/
In: Dense vector x ∈ R m , Encoded vectors w e 1 ..n ∈ R k Fixed [Table] [Table]
[Table] m ** ( k (cid:28) m )
Out: Dense vector y for i ← 1 to n do
W i ← w ei × M ⊕ over GF(2) y i
= W i · x with Mask (for zero skipping) / * Decoding m elements using w ei (Regular access)*/
F IXED - TO -F IXED S PARSITY E NCODING
Data compression is a process of encoding original data into a smaller size.
If a ﬁxed number of original bits are encoded into a ﬁxed number of (smaller) bits, such a case is categorized into a “ﬁxed-to-ﬁxed” compression scheme.
Similarly, “ﬁxed-to-variable”, “variable-to-ﬁxed”, and “variableto-variable” categories are available while variable lengths of original and/or encoded bits allow higher compression ratio than ﬁxed ones (e.g., Huffman codes (Huffman, 1952) as ﬁxed-to-variable scheme, Lempel-Ziv (LZ)-based coding (Ziv and Lempel, 2006) as variable-to-ﬁxed scheme, and
Golomb codes (Golomb, 1966) as variable-to-variable scheme).
Among those 4 categories, “ﬁxed-to-ﬁxed” compression is the best for NNs that rely on parallel computing systems.
Fixed-to-ﬁxed compression schemes are, however, challenging when ﬁne-grained pruning is employed in NNs because the number of unpruned weights in a ﬁxed-size block varies.
Accordingly, most previous sparsity representations (such as CSR format) translate a ﬁxed-size weight block into a variable-size block while such a translation would demand non-uniform memory accesses that lead to signiﬁcantly degraded memory bandwidth utilization as shown in Figure 1.
Speciﬁcally, in the case of ﬁxed-to-variable sparsity format (e.g., CSR) in Figure 1, we observe that memory bandwidth is low because ﬁne-grained pruning induces a variable number of pruned weights for a certain block or row while memory is designed to access a ﬁxed amount of consecutive data.
Since higher sparsity leads to higher relative standard deviation (i.e., coefﬁcient of variation) on pruned weights in a block, low memory bandwidth is a signiﬁcant issue even though the amount of data to be stored is reduced (see Appendix A).
As a result, for ﬁxed-to-variable sparsity format, it is difﬁcult to implement ﬁne-grained pruning with parallel computing systems that require high memory bandwidth (Yu et al., 2017).
On the other hand, ﬁxed-to-ﬁxed compression schemes in Figure 1 can maintain the same memory bandwidth regardless of sparsity.
[Figure]
Encoding into fixed-length codes
Decoding in a fixed rate x Unpruned
Pruned
[Figure]
[Figure]
[-X 0 X0 X 
-]
In this work, we propose a “ﬁxed-to-ﬁxed” compression scheme as shown in Figure 2 when the number of pruned weights in a block can vary.
A successful ﬁxed-to-ﬁxed compression of sparse
NNs should consider the followings:
• (High compression ratio)
The maximum compression ratio is limited by the minimum entropy (that can be obtained by a ﬁxed-to-variable scheme as we discuss in Appendix D).
Suppose that a block to be encoded contains (ﬁxed) n b bits among which (ﬁxed) n u bits are unpruned.
A ﬁxed-to-ﬁxed encoding scheme is required to support high compression close to ( n b /n u ) (estimated by entropy).
Fixed-to-ﬁxed decoding, then, accepts (ﬁxed) N in bits as an input and produces N out bits as an output while N out /N in = n b /n u . •
(Variation tolerance)
For a ﬁne-grained pruning, n u is given as a random variable whose distribution is affected by pruning rate, n b size, a particular pruning method,
and so on.
Our goal is to maintain a ﬁxed-to-ﬁxed scheme with a high compression ratio even under
Var[ n u ] (cid:54) = 0 .
In Figure 2, for example, 5 blocks of original data have various n u values while the size of an encoded block is ﬁxed to be 4 (= E [ n u ] ).
We will discuss how to design a variation tolerant encoding.
R ANDOM N UMBER G ENERATOR
Before we discuss compression schemes, let us assume that a binary masking matrix is given to represent which weights are pruned or not (note such a binary masking matrix can be compressed signiﬁcantly (Lee et al., 2019a)).
Then, a pruned weight can be described as a don’t care value ( (cid:53) ) that is to be masked.
We also assume that 1) pruning each weight is performed independently with pruning rate S and 2) unpruned weight is assigned to 0 or 1 with equal probability (such assumptions are not necessary when we demonstrate our experimental results in Section 5).
To obtain both “high compression ratio” and “variation tolerance” while a ﬁxed-to-ﬁxed compression scheme is considered, we adopt random number generator schemes that enable encoding/decoding of weights.
A random number generator accepts a ﬁxed number of inputs and produces n b bits so as to implement a ﬁxed-to-ﬁxed compression scheme.
As shown in Figure 3, a weight block is compared with every output of a random number generator.
If there is an output vector matching original (partially masked) weights, then a corresponding input vector of a random number generator can be an encoded input vector.
As an effort to increase the Hamming distance between any two outputs (i.e., the number of bit positions in which two bits are different), 2 n u outputs out of 2 n b
[Table]
[Figure]
possible candidates need to be randomly selected.
Note that random encoding has already been suggested by Claude Shannon to introduce channel capacity that is the fundamental theory in digital communication (Morelos-Zaragoza, 2006).
Since then, practical error correction coding techniques have been proposed to implement random-like coding by taking into account efﬁcient decoding (instead of using a large look-up table).
Similar to error correction coding that usually depends on linear operations over Galois Field with two elements, or GF (2) (Morelos-Zaragoza, 2006), for simple encoding of original data, recently, two compression techniques for sparse NNs have been proposed.
An XOR-gate decoder produces (a large number of) binary outputs using (a relatively much smaller number of) binary inputs while outputs are quantized weights (Kwon et al., 2020).
Another example is to adopt a Viterbi encoding/decoding scheme (Forney, 1973) to generate multiple bits using a single bit as an input (Ahn et al., 2019).
For a block that cannot be encoded into a compressed one by a random number generator, we can attach patch data to ﬁx unmatched bits (Kwon et al., 2020) or re-train the model to improve the accuracy (Ahn et al., 2019).
To compare the random number generation capability of various block-level compression schemes, we introduce ‘encoding efﬁciency’ given as a percentage.
[Equation]
Let S be pruning rate ( 0 ≤ S ≤ 1 ).
To measure encoding efﬁciency ( E ), we assume that the compression ratio of a random number generator (=the number of output bits / the number of input bits) is 1 / (1 − S ) .
We generate numerous randomly pruned (binary) weight blocks, and for each block, we investigate all of the possible outputs that a random number generator can produce.
If there is a block missing a matching output of a generator, then the maximum number of correctly matched bits is recorded for each block.
We repeat such an experiment for all of the blocks.
Note that
E cannot be higher than 100% for any generators.
3. F IXED P RUNING R ATE IN A B LOCK
For simplicity, we assume that n u in a block is a ﬁxed number.
Let us study E when n u is ﬁxed using an XOR-gate decoder introduced in (Kwon et al., 2020).
For an XOR-gate decoder, when N out is the number of output bits and N in is the number of input bits, a matrix M ⊕ ∈ { 0 , 1 } N out × N in presents connectivity information between an input vector w x ( ∈ { 0 , 1 } N in ) and an output vector w y ( ∈ { 0 , 1 } N out ) such that we have w y =
M ⊕ · w x over GF (2) .
For example, if the second row of M ⊕ (with N in = 4 and N out = 8 ) is given as [1 0 1 1] , then w y 2 = w x 1 ⊕ w x 3 ⊕ w x 4 (‘ ⊕ ’ indicates a binary XOR operation or an addition over GF (2) equivalently).
An element of M ⊕ is randomly ﬁlled with 0 or 1 as a simple random number generator design technique (Kwon et al., 2020).
To measure E , let N in /N out ≈ 1 − S such that N out = (cid:98) N in · (1 / (1 − S ))
(cid:99) .
Correspondingly, for a certain S , a block size (= N out ) increases as N in increases.
When n b = N out and n u = N in ,
Figure 4a describes statistics of E when random M ⊕ matrices are associated with random blocks.
From Figure 4a, it is clear that increasing N in is the key to improving encoding efﬁciency.
Note that, however, increasing N in and N out complicates the decoding complexity (due to large M ⊕ ) and the encoding complexity as well (due to an exponentially large search space).
3. V ARIABLE P RUNING R ATE IN A B LOCK
Now we allow n u in a block to ﬂuctuate.
For Figure 4b, we assume that pruning each weight is a Bernoulli event (with S probability) such that n u in a block follows a binomial distribution
B ( N out , 1 − S ) (thus, E [ n u ] = N out (1 − S ) and Var[ n u ] = N out S (1 − S ) ).
N in is given as E [ n u ] in
Figure 4b.
Compared to Figure 4a, E becomes lower mainly because some blocks would have n u larger than N in (i.e., too many unpruned weight bits that a random number generator cannot target).
[Equation]
[Figure]
Note that the coefﬁcient of variation of n u
[Equation]
decreases as N out increases.
Indeed, the gap between E in Figure 4a and E in Figure 4b tends to be slightly reduced when N in (as well as corresponding N out ) increases.
To illustrate, as shown in Figure 5, when there are two blocks of different n u , concatenating two blocks into a block (thus, increasing N out ) can increase the probability of successful encoding due to efﬁcient usage of N in .
Increasing N in and N out by n times, however, requires an XOR-gate decoder to be larger by n 2 times with only a marginal gain
in E . Another way to improve E under the variation on n u is to decrease N out and increase N in .
In this case, however, the compression ratio is reduced consequently.
We need a solution to design a ﬁxed-to-ﬁxed sparsity compression technique that can improve E of a random number generator under the variation on n u without sacriﬁcing compression ratio (i.e., N in /N out
= 1 − S ).
To validate our observations obtained by synthetic random data, we compute E of an XOR-gate decoder using the Transformer model (Vaswani et al., 2017).
The ﬁrst fully-connected layer of the
Transformer is pruned by a magnitude-based pruning method (Han et al., 2015) with pruning rate S .
Interestingly, E described in Figure 4c is similar to that of Figure 4b.
As such, our assumption that pruning a weight is a Bernoulli event is valid for the context of magnitude-based pruning.
P ROPOSED S EQUENTIAL ENCODING S CHEME
If Var[ n u ] is non-zero and N in is ﬁxed, then blocks of small n u ( < N in ) would have many possible input vectors with matching output vectors while other blocks with large n u ( > N in ) may have no any single possible input vector.
Such unbalance among encoding success rates over blocks can be mitigated if a part of input vectors associated with a block of small n u can be reused for the neighboring blocks of large n u .
In other words, by sharing some parts of an input vector for multiple consecutive blocks (of diverse n u values), input search space size of each block can be balanced.
Reusing inputs is fulﬁlled by shift registers that have been also introduced to convolutional codes such as Viterbi codes (Morelos-Zaragoza, 2006).
In this section, we propose sequential encoding techniques to address the limitations of previous ﬁxed-to-ﬁxed compression schemes (for example,
XOR-gate-only decoder (Kwon et al., 2020) lacking tolerance for n u variation, and Viterbi encoders (Ahn et al., 2019) that receive only one bit as an input (i.e., N in is restricted to be 1).
Weight manipulation Since our encoding/decoding techniques process data in a block-level (in the form of a 1-D vector), original sparse weight matrices (or tensors) need to be reshaped through grouping, ﬂattening, and slicing.
Assuming that a number format has the precision of n w bits (e.g., n w =32 for FP32), as the ﬁrst step of weight manipulation, a weight matrix W ∈ R m × n is grouped into n w binary matrices W b 1 ..n w ∈{ 0 , 1 } m × n (otherwise, n w successive bits are pruned or unpruned). Then, each binary matrix (or tensor) W bi is ﬂattened to be a 1-D vector and each vector is sliced into w b i, 1 ..l blocks when l (= (cid:100) mn N out (cid:101) ) indicates the number of blocks in a 1-D (ﬂattened) vector.
[Figure]
Mask
[Figure]
Decoding with an input sequence For an XOR-gate decoder (as a non-sequential decoder) at time index t , an output vector w b (cid:48) t ( ∈ { 0 , 1 } N out ) is a function of an input vector w et ( ∈ { 0 , 1 } N in ) such that w et is utilized only for one time index.
In our proposed sequential encoding scheme, w et is exploited for multiple time indices using shift registers.
Speciﬁcally, we copy w et to a shift register whose outputs are connected to the inputs of the next shift register.
Let N s be the number of shift registers.
In the proposed scheme, an XOR-gate decoder receives inputs from N s shift registers as well as w et .
Then, as shown in Figure 6, a sequential decoder (consisting of an XOR-gate decoder and shift registers) produces w b (cid:48) t as a function of an input sequence of ( w et , w et − 1 , ...) while such a function can be described as
[Equation]
N s ) over GF (2) where A (cid:95) B implies a concatenation of A and B and the sequence length is ( N s + 1 ).
Note that in Figure 6, an input vector w et is reused N s times and an XOR-gate decoder accepts ( N s + 1) · N in bits to yield N out bits (hence, M ⊕ ∈ { 0 , 1 } N out × (( N s +1) · N in ) ).
Increasing N s enables 1) a larger XOR-gate decoder (without increasing N in ) that improves E as demonstrated in Figure 4 and 2) multiple paths from an input vector to multiple output vectors (of various n u ) resulting in balanced encoding. Note that our
XOR-gate decoder (or effectively memory decompressor) is best implemented by ASICs or FPGAs where each XOR requires only 6 transistors (Rabaey et al., 2004).
[Figure]
Balanced encoding Figure 7 illustrates a decoding process when N s =1, N in =3, and
N out =8.
Each weight vector encoded con a different number of unpruned weights
For w b (cid:48) t at time index t (in Figure 7), a less number of unpruned weights tends to enlarge search space for w et .
On the other hand for w b (cid:48) t +1 at time index ( t + 1) , a large number of unpruned weights would highly restrict search space for w et .
Correspondingly, compared to the case when w et is associated with only one w b (cid:48) block (i.e., non-sequential encoding with N s =0), search space for w e t can be balanced as more w b (cid:48) blocks of various n u are correlated with w et .
Note that as Var[ n u ] of one block increases, according to the central limit theorem, N s is required to be larger to maintain the balance of encoding capability of each w et .
As we demonstrate in the next section, under the variation on n u , even small non-zero N s can enhance E substantially while increasing N in (and N out while N s =0) improves E only marginally (as described in Figure 4).
[Figure]
40k [Figure] # o f E n c r y p t
e d B i t s
[Figure]
Encoding algorithm
In the case of a non-sequential encoding scheme, because of one-to-one correspondence between w et and w b (cid:48) t through M ⊕ , encoding can be performed independently for each block (e.g., for a given masked w bt , we select one matching w b (cid:48)
t out of all available w b (cid:48) t sets by a decoder).
Such block-wise encoding, however, is not applicable to a sequential encoding scheme that needs to consider the whole sequence of w b blocks to ﬁnd an input sequence fed into a decoder.
Suppose that we ﬁnd a particular output sequence matching l blocks after exploring all feasible output sequences provided by a generator and the input sequence ( w e 1 , w e 2 , ..., w el ), the time-complexity of encoding would be O (2 N in · l ) .
Fortunately, sequential decoding operations shown in Figure 7 can be models as a hidden Markov model, where each state is represented by concatenating w et , w et − 1 , ..., w et − N s
and there are 2 N in paths for the next state transitions (Viterbi, 1998).
Consequently, the time- and space-complexity can be reduced to be O (2 N in · ( N s +1) · l ) by dynamic programming that computes the maximum-likelihood sequence in a hidden Markov model (Forney, 1973).
For details of our encoding algorithm, the reader is referred to Appendix E.
Lossless compression Any random number generators cannot produce outputs perfectly matching all unpruned weights (i.e., E is always less than 100%).
To correct unmatched outputs of a random number generator (by ﬂipping 0 ↔ 1 ) in order to enable lossless compression, the locations of all unmatched weight bits need to be recorded.
Note that if E ≈ 1 , the number of unmatched weight bits is a lot smaller than the number of encoded weight bits.
If that is the case, such correction information can be stored in a separate on-chip memory that can be independently accessed without disturbing decoding operations.
We propose a format representing such correction information in
Appendix F where each unmatched weight bit requires N c bits in the correction format ( N c is around 10 in Appendix E).
Taking into account a compression ratio of a generator given as N out / N in and additional correction information (using N c bits per one unmatched weight bit), a binary weight matrix W b ∈ { 0 , 1 } m × n can be compressed into ( N in N out mn + N err ) bits when N err =
N c × (# of unmatched bits) =
N c mn (1 − S )(1 − E ) .
Subsequently, memory reduction can be expressed as
[Equation]
when N in /N out is given as (1 − S ) .
Thus, memory reduction approaches S when E approaches 1.
E XPERIMENTAL R ESULTS
In this section, we demonstrate the encoding capability of our proposed sequential encoding techniques using synthetic random data and NNs pruned by various pruning methods.
Even though various heuristic algorithms can be suggested, we adopt a simple dynamic programming technique for weight encoding that minimizes the number of unmatched weight bits.
For our experiments, N in is selected to be 8 such that we feed a decoder on a byte-level.
5. S YNTHETIC R ANDOM D ATA ( N in =8)
Setup We generate a random weight 1-D vector of 1M bits.
We also create a random masking data of 1M bits in which the percentage of zeros equals S .
M ⊕ matrix (formulating the structure of an XOR-gate decoder) basically needs to be designed to maximize the randomness among outputs.
Measuring randomness, however, is challenging and such measurement may not be highly correlated to E .
Alternatively, we try numerous random M ⊕ matrices and choose a particular M ⊕ of the highest E .
Speciﬁcally, for a given set of N in and N out , an element of M ⊕ ∈ R N out × (( N s +1) · N in ) is randomly assigned to 0 or 1 with equal probability.
Then, E of those random M ⊕ matrices is estimated by using given random binary weight vectors and masking data.
The best M ⊕ providing the highest E (for a given set of N in and N out ) is then utilized for our experiments.
Compression capability Figure 8 demonstrates the impact of N s on E and corresponding memory reduction(%) with various N out and 1M random bits when N in =8 and S =0.9.
Regardless of N s , as N out increases (i.e., the compression ratio (= N out /N in = N out / 8 ) of a decoder increases), the number of encoded bits is reduced while the number of unmatched (error) bits increases.
Because of such a trade-off between the number of encoded bits and the number of error bits, there exists a certain N out that maximizes the memory reduction.
Note that compared to a non-sequential encoding (of N s =0), sequential encoding (even with small N s ) signiﬁcantly reduces the number of error bits and maintains high E (of almost 100%) until N out reaches N in × (1 / (1 − S ))
=80.
Indeed, memory reduction becomes highest (=89.32%) when N s is highest and N out is 80 in Figure 8.
As we discussed for lossless compression in Section 5, 89.32% of memory reduction is close to S (=90%) that is the maximum memory reduction obtained when E ≈ 1 .
In the remainder of this paper, N out is given as
N in × (1 / (1 − S )) to maximize the memory reduction.
[Table]
Impact of S on memory reduction
Table 1 presents memory reduction using various S .
For a certain S in Table 1, as N s increases, the difference between S and memory reduction decreases.
In other words, regardless of S , sequential encoding/decoding principles are crucial for memory reduction to approach S which is the maximum.
Increasing S also facilitates memory reduction to approach
S in Table 1 as described in Eq. 2 where increasing S with a constant E enhances memory reduction.
[Figure]
[Figure] [Figure]
[Figure] [Figure]
1.0 Ratio of Zeros( [Figure]
[Figure]
Inverting technique So far, we assumed that a binary weight matrix holds equal amounts of zeros and ones.
A few representations such as binary-codingbased quantization (Rastegari et al., 2016; Xu et al., 2018) and signed INT8 (Jacob et al., 2018) would inherently justify such an assumption.
However, there exist exceptional representations as well (e.g., FP32).
We conduct experiments to ﬁnd a relationship between the ratio of zeros and E using a random weight vector as shown in Figure 9.
E increases if a substantial amount of zeros are employed as unpruned weight bits, because of a higher chance to ﬁnd trivial inputs (of all zeros fed into XOR operations) to produce zero outputs.
Hence, to improve E (especially when N s is low), we propose an inverting technique where an entire binary weight vector is inverted if the ratio of zeros is less than 50%.
5. S PARSE T RANSFORMER AND R ES N ET -50 ( N in =8)
We measure compression capability of our proposed sequential encoding scheme using sparse
Transformer (Vaswani et al., 2017) on WMT’14 en-de dataset and ResNet-50 (He et al., 2016) on
ImageNet.
Those two models 1 in FP32 format are pruned by various methods including magnitudebased one (Han et al., 2015), L0 regularization (Louizos et al., 2018), and random pruning (Gale et al., https
[Table] [Table]
(also variational dropout (Molchanov et al., 2017) in Appendix G).
For the ResNet-50 model (on ImageNet), we also consider signed INT8 format (Jacob et al., 2018).
Table 2 presents E and memory reduction when every layer of the Transformer and ResNet-50 is pruned by the same pruning rate S .
Both E and memory reduction are signiﬁcantly improved by increasing N s .
Even compared to the case when inverting technique is applied to non-sequential encoding ( N s =0 ), we observe that sequential encoding ( N s > 0 ) without inverting yields a lot higher compression capability.
Note that the compression capabilities of random pruning and magnitude-based pruning methods are similar in
Table 2 such that our experiments with synthetic random data are justiﬁed.
Such justiﬁcation is also veriﬁed in Table 3 that is achieved by using two selected layers of the Transformer.
Compared to random pruning, magnitude-based and L0 regularization pruning methods exhibit somewhat lower E that is related to higher coefﬁcients of variation of n u .
See Appendix G for additional results.
All in all, our proposed encoding method designed in the context of random pruning is also effective for other ﬁne-grained pruning methods.
Through various cases including synthetic data and benchmark models, we demonstrated the superiority of the proposed encoding scheme to previous ﬁxed-to-ﬁxed compression methods including a non-sequential XOR-gate decoder of N s =
(Kwon et al., 2020) and a Viterbi-based encoder structure where N in is limited to be 1 (Ahn et al., 2019).
C ONCLUSION
In this paper, we proposed a sequential encoding scheme that is a useful ﬁxed-to-ﬁxed compression for sparse NNs.
We studied the maximum compression ratio using entropy based on the strategy of mapping a weight block into a small number of symbols.
We also investigated random number generators as a practical ﬁxed-to-ﬁxed decoder using an XOR-gate decoder.
Random number generators can improve compression capability if input vectors are reused for multiple time indices through shift registers.
A M EMORY B ANDWIDTH WITH F IXED - TO -V ARIABLE S PARSITY F ORMAT
Memory bandwidth is expressed as the access rate (usually in units of bytes/second) at which the data can be read from or written into memory.
While the number of memory transactions can be reduced a lot by pruning weights, the utilization of memory bandwidth is affected by the variability of the length of data to be accessed during one transaction (Yu et al., 2017).
In the case of ﬁxed-to-ﬁxed sparsity representation, since all encoded blocks have the same size, full memory bandwidth is utilized.
On the other hand, in the case of ﬁxed-to-variable weight representation, encoded blocks have variable sizes such that memory bandwidth can be wasted.
To be more speciﬁc, let n w , n b , and
S be the number of weights to be encoded, the number of unpruned weights in a block, and sparsity, respectively.
Assuming that pruning a weight is a Bernoulli trial, for CSR format, we obtain
[Equation]
Thus, the coefﬁcient of variation (or relative standard deviation) is given as (cid:112)
[Equation]
, which increases as S increases ( 0 < S < 1 ).
In other words, for ﬁxed-to-variable sparsity format, more memory bandwidth is wasted as more weights are pruned (as shown in Figure 1).
Note that for ﬁxed-to-ﬁxed sparsity format, we have Var[ n b ] = 0 , and thus, the memory bandwidth is not wasted at all.
B P ERFORMANCE OF S PARSE M ATRIX M ULTIPLICATION USING CSR F ORMAT
[Figure]
For inference of NNs, it is known that performance (including latency and throughput) is dominated by matrix multiplications.
Figure S.10 presents performance when a ( 2048 × 2048 ) sparse matrix (of
CSR format) is multiplied by a ( 2048 × k ) dense matrix when k is usually small for inference with small batch size.
MKL library (operated by i7-7700 @ 3.6GHz) and CUDA 10.2 library (performed by nVIDIA V100) perform sparse matrix multiplications whose execution times are normalized with respect to corresponding dense matrix multiplications (i.e., using a dense ( 2048 × 2048 ) matrix).
From Figure S.10, it should be noted that even with a high compression ratio (due to high pruning rate), if CSR format is adopted, sparse matrix multiplications can be even slower than dense matrix multiplication.
Thus, proposing a regular format after ﬁne-grained pruning is critical for parallel computing to achieve performance gain by pruning.
C R ELATED W ORKS
This section describes the previous works regarding how to represent sparsity after ﬁne-grained weight pruning.
As mentioned in Section 1, high sparsity and compression ratio can be expected by ﬁne-grained (unstructured) pruning, which prunes individual weights at randomly distributed locations.
To represent sparse weights with a reduced memory footprint, recording unpruned weights associated with (binary) masking information may be the simplest way. However, for such a case, a reconstruction process is required right before matrix multiplications during inference and it is challenging to be parallelizable due to its random locations (i.e., the number of pruned weights to be processed by a computation unit should highly vary as mentioned in Appendix A). As efforts to save memory footprints with ﬁne-grained sparsity, by large, there have been two lines of researches, namely, ﬁxedto-variable and ﬁxed-to-ﬁxed formats. The Compressed Sparse Row (CSR) format is a well-known example of ﬁxed-to-variable formats.
Since DeepCompression (Han et al., 2016b) utilized the CSR format for representing sparse neural networks (NNs), most of the acceleration library kernels and computing systems have supported CSR data formats along with related APIs for CSR format (e.g. cusparse library in CUDA ).
Based on recording unpruned values and relational indices row-wise, the CSR format can lead to increased parallelism for computing sparse neural networks.
However, because the number of unpruned weights in each row still varies, CSR inherently leads to irregular memory access patterns.
SpMM or SpMV operations based on CSR format (even when including block CSR or advanced hardware design for CSR (Han et al., 2016a)) cannot still achieve full (or high) memory bandwidth utilization of computing systems, as numerous studies have raised related issues (Wen et al., 2016; Yu et al., 2017; Ahn et al., 2019;
Zhou et al., 2018; Kwon et al., 2020; Gale et al., 2020)
(we also reported similar concerns in Appendix B).
Despite a highly reduced amount of parameters to be stored, pruned networks in an unstructural manner have not been fully employed by the currently available commercialized computing systems.
Fixed-to-ﬁxed data compression, on the other hand, can achieve fully-parallelizable computations along with memory-saving formats and higher memory bandwidth because the length of compressed data is supposed to be ideally equal among any subsets of compressed data.
By adopting compression approaches that have been widely used in well-established engineering areas (e.g., digital communication, VLSI testing, and so on), Viterbi-based compression (Ahn et al., 2019) and XOR-gates-based compression (Kwon et al., 2020) have been proposed.
Note that despite regular memory access patterns, compression methods of Ahn et al.
(2019) require a heavy re-training process to acquire proper weight bits and masks while presenting the limitation that the compression ratio should be ﬁxed to be integer values.
As for the work of (Kwon et al., 2020), the encoding efﬁciency of previous
XOR-gates-based compression is signiﬁcantly degraded due to combinational encoding algorithm and inefﬁciency of patch systems.
Our proposed work can be regarded as an extended study of XOR-gates-based compression: 1) this paper veriﬁes that typical data representations (e.g., FP32 and INT8) for DNNs can also be compressed by the XOR-gate decoder (additionally, the inverting technique can boost the encoding efﬁciency), 2)
By using the proposed sequential encoding/decoding schemes, encoding efﬁciency
E can closely approach the theoretical upper bound of ﬁxed-to-ﬁxed sparsity formats, and 3) Our encoding algorithm can explore encoded bits according to sequential decoding within the limited size of XOR-gate decoder while Kwon et al.
(2020) proposed a simple heuristic algorithm assuming combinational decoding only (i.e., N s = 0 ).
D F UNDAMENTAL L IMITS OF C OMPRESSION
We are interested in the upper bound of compression ratio that can be analyzed by entropy (while allowing ﬁxed-to-variable compression).
Then, when we suggest a ﬁxed-to-ﬁxed compression scheme, we can estimate how close the compression capability of a ﬁxed-to-ﬁxed scheme is to the maximum.
Entropy presents the minimum average number of bits to represent an event when a probability distribution of those events is provided (Morelos-Zaragoza, 2006).
To investigate the entropy of pruned weight blocks (and the maximum compression ratio correspondingly), a block of bits is assigned to a symbol such that a probability distribution of symbols minimizes the entropy.
In other words, symbol assignment is designed to minimize the average number of bits (i.e., entropy to represent symbols) that is given as
[Equation]
where n is the total number of symbols and p i is the occurrence probability of each symbol.
Symbol assignment We concatenate all of k -th bits of weights into a group and produce n w groups when n w is the number of bits to represent a weight (e.g., n w is 32 for single-precision and 8 for
INT8) and 1 ≤ k ≤ n w .
Symbol assignment is performed in each group independently without referring to other groups.
Suppose that a certain block from one of the groups is given as { 0 (cid:53)(cid:53) 1 } and corresponding mask bits (that are shared by all n w groups) are { 1001 } (0 means masking).
By ﬁlling up (cid:53) with 0 or 1, { 0 (cid:53)(cid:53) 1 } is selected to be assigned to one of 4 symbols (i.e., { 0001 } , { 0011 } , { 0101 } or { 0111 } ) while such a symbol selection decides entropy.
In the following two examples (where n b =4), we illustrate the symbol assignment method that minimizes the entropy.
Entropy ( n u = 1, e.g., { (cid:53)(cid:53)(cid:53) 0 } , { (cid:53) 1 (cid:53)(cid:53) } )
In this case, every block (of n b =4 and n u =1) can be assigned to either one of two symbols ( { 0000 } , { 1111 } ).
Since P (0000) =
P (1111) = 0 .
5 , from Eq.
6, H = − (0 .
5 × ( − 1) + 0 .
5 × ( − 1))
= 1 .
Thus, a block of n b =4 and n u =1 can be compressed into 1 bit that indicates one of two symbols.
There are many other sets of two symbols to meet H = 1 , such as ( { 0010 } , { 1101 } ) and ( { 1010 } , { 0101 } ).
Entropy ( n u = 2, e.g., { (cid:53) 01 (cid:53) } , { 10 (cid:53)(cid:53) } )
A set of symbols should meet the following requirement: if we choose random n 1 and random n 2 ( 1 ≤ n 1 , n 2 ≤ n b , n 1 (cid:54) = n 2 ) and collect { n 1 -th bit, n 2 -th bit} of each symbol, then, each of {00, 01, 10, 11} should appear in the collection.
Under such a constraint, after a careful investigation using a full search, the minimum number of symbols is 5 to represent all blocks of n b =4 and n u =2.
An example set of 5 symbols with corresponding occurrence probability to minimize entropy is as follows: P (0000)
= 6 / 24 , P (1110) = 6 / 24 , P (0101)
= 5 / 24 ,
P (1001)
= 4 / 24 , and P (0011)
= 3 / 24 .
Then, from Eq. 6, H is approximately 2.28 (bits) attainable through ﬁxed-to-variable compression.
On the other hand, for a ﬁxed-to-ﬁxed scheme, a block (of n b =4 and n u =2) is compressed into 3 bits to represent one of 5 symbols.
For n u =3, the minimum number of symbols is 8 and a block can be compressed into 3 bits.
All in all, when n u is ﬁxed across blocks, H can be equal to or slightly higher than n u .
E E NCODING A LGORITHM
In this section, we describe our encoding algorithm based on dynamic programming algorithm that can minimize the number of unmatched bits. Our encoding algorithm presents time-complexity as
O ( l · 2 N in · ( N s +1) ) and space-complexity as O (2 N in · N s ) .
Thus, even though increasing N in and
N s enhances E , ( N in × N s ) is empirically limited to be less than 26 under the constraint of 32GB memory of a single GPU.
For each binary weight W bi (0 ≤ i ≤ n w ) , the function ENCODING generates the encoded vectors w i, (1 ..l + N s ) .
XOR-gate decoder ( M ⊕ ) is pre-determined and ﬁxed for inference.
Note that the number of encoded vectors ( w e 1 .. ( l + N s ) ) is l + N s , not l .
w e 1 and w e 2 are pre-determined as BIN (0) and used for encoding the ﬁrst binary weight vector.
W is a binary weight vector to be
MASK is pruning mask information dp is ( N s +1) -dimensional array. dp t ] [ b ] stores the minimum number of error bits when BIN a and b are ﬁxed as and
− is s + array for history.
Additional Functions SIZE( W ) returns the number of parameters in W
( ar , shape ) returns same data ar returns
Function ERR_NUM( x, y, mask ) :
nErr ←for i ← 0 to N out − 1 do if mask [ i ] (cid:54) = {+
+} and x [ i ] (cid:54) =
y [ i ] then nErr ← nErr +return nErr Function ENCODING( W bi , MASK ) : l ← SIZE( W bi )/ N out data ← RESHAPE( W bi , [ l, N out ] ) (cid:46)
Slicing mask ← RESHAPE( MASK, [ l, N out ] ) (cid:46) Slicing
INIT ( dp, INF ) (cid:46) Initialize for starting point dp [ N s ][0][0] ← 0 and w e 1 , w e 2 ← BIN (0) , BIN (0) (cid:46) Find the minimum number of errors for t ← N s +1 to l + N s do for i t ← 0 to 2 N in − 1 do for i t − 1 ← 0 to 2 N in − 1 do for i t − 2 ← 0 to 2 N in − 1 do out ← M ⊕ ( BIN ( i t − 2 ) (cid:95) BIN ( i t − 1 ) (cid:95) BIN ( i t ) ) n err ← ERR_NUM( out, data [ t ] , mask
[ t ] ) if dp [ t ][ i t ][ i t − 1 ] > n err + dp [ t − 1][ i t − 1 ] [ i t − 2 ] then dp [ t ][ i t ]
[ i t − 1 ] ← n err + dp [ t − 1]
[ i t − 1 ][ i t − 2 ] path [ t ][ i t ][ i t − 1 ] ← BIN ( i t − 2 ) (cid:46) Find vectors min err ← INF .
for i t = 0 to 2 N in − 1 do for i t − 1 = 0 to 2 N in − 1 do if min err > dp [ l + N s ]
[ i t ] [ i t − 1 ] then min err ← dp [ l + N s ][ i t ] [ i t − 1 ] w el +2 , w el +1 ← BIN ( i t − 1 ) , BIN ( i t ) (cid:46) Get encoded following array for t ← l to 2 N s + 1 by − 1 do w et ← path [ t + N s ][ DEC ( w et +2 )]
[ DEC ( w et +1 )] return { w e 1 , w e 2 , ..., w el + N s }
F M EMORY R EDUCTION WITH L OSSLESS C OMPRESSION
Decoding
[Figure]
For lossless compression, the unmatched bits (error bits) should be corrected right after decoding procedures.
Since the random number generator produces 0 or 1, the unmatched bits can be simply corrected by ﬂipping.
To compute the memory reduction, we suggest block-wise correction logic to be conducted by ﬂipping error bits in a p -length vector while each error location is given as a bit position inside a vector.
As depicted in Figure S.11, when the block-wise correction is performed while a block size has p -length, the decoded vectors w b (cid:48) 1 ..l are reshaped to w rb (cid:48)
..k ( k = (cid:100) mnp (cid:101) ) and each reshaped vector w rb (cid:48) is corrected by corresponding error bit locations indicating which bit is to be ﬂipped inside a block (of p -length).
The amount of compressed bits can be computed as
[Equation]
The ﬁrst term is the number of bits of w e 1 ..l as the compression results by sequential encoding.
The second term is the number of ﬂag bits while each ﬂag bit indicates whether a non-zero number of unmatched bits exists in each p -length vector (as E approaches 1, there are many p -length vectors that skip the correction step)).
The third term includes locations of unmatched bits (inside a p -length block) to be ﬂipped and an additional one bit to specify the end of the streaming error bit locations (i.e., ‘1’ means the following (log 2 p ) bits contains the next correction information of the same block).
Thus, each block of p -length involves (log 2 512 + 1) ×
(# of unmatched bits)) for error bit locations.
G A DDITIONAL A NALYSIS ON E XPERIMENTS
In this section
[Figure]
we provide additional analysis and results for experiments in Section 6.2.
[Figure]
inverting technique for the FP32 models.
T a b l e
S . 4 :
C o e f ﬁ c
i e n t o f v a r i a ti on o f n u a nd
E o f s e l ec t e d l a y e r s o f t
h e
T r a n s f o r m e r p r un e d by r a ndo m , m a
gn
it
u
d e - b a s e d , Lr e gu l a r i za ti on , o r v a r i a ti on a l d r opou t p r un i ng m e t hod .
N i n , N o u t S HA P E
S
L AY
E R
N A M
E
P R
UN I NG M ET [Table]
E
[Table]
N s =
[Table]
s =
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
D E C 3 / S
EL F _
A TT /
Q
L
R E G .
D E C 3 / FF
N
L
R E G .
D E C 5 / S
EL F _
A TT /
K
V A R .
D R O
P OU
T 0 .
D E C 1 / FF
N
V A R .
D R O
P OU
T 0 .
E N C 2 / S
EL F _
A TT /
OU T P
U T
R
ANDO
M 0 .
D E C 5 / FF
N
R
ANDO
M 0 .
E N C 2 / S
EL F _
A TT /
OU T P
U T
M
AGN I T
UD E 0 .
D E C 5 / FF
N
M
AGN I T
UD E 0 .
E N C 2 / S
EL F _
A TT /
OU T P
U T
L
R E G .
D E C 5 / FF
N
L
R E G .
D E C 2 / S
EL F _
A TT /
V
V A R .
D R O
P OU
T 0 .
D E C 4 / FF
N
V A R .
D R O
P OU
T 0 .
T a b l e
S . 5 :
C o e f ﬁ c
i e n t o f v a r i a ti on o f n u a nd
E o f s e l ec t e d l a y e r s o f t
h e
R e s
N e t50 p r un e d by r a ndo m , m a
gn it u d e - b a s e d , o r v a r i a ti on a l d r opou t p r un i ng m e t hod .
N i n ,
N o u t
S HA
P E
S
L AY
E R
N A M
E
P R
UN I NG M ET [Table]
N
[Table]
N
[Table]
N
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
N
[Table] M [Table] [Table]
P_
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
_ L
AY E R 3_ B
N
V A R .
D R O
P OU
T 0 .
G R
{+OU P_ L AY E R 
 _ B N+}
V A R .
D R O
P OU
T 0 .
[Table] [Table] [Table] [Figure]