B
IT -
P
RUNING : A S PARSE M ULTIPLICATION -L ESS D OT -P RODUCT
Anonymous authors
Paper under double-blind review
A BSTRACT
Dot-product is a central building block in neural networks.
However, multiplication ( mult ) in dot-product consumes intensive energy and space costs that challenge deployment on resource-constrained edge devices.
In this study, we realize energy-efficient neural networks by exploiting a mult -less, sparse dotproduct.
We first reformulate a dot-product between an integer weight and activation into an equivalent operation comprised of additions followed by bit-shifts ( add-shift-add ).
In this formulation, the number of add operations equals the number of bits of the integer weight in binary format.
Leveraging this observation, we propose Bit-Pruning, which removes unnecessary bits in each weight value during training to reduce the energy consumption of add-shift-add .
Bit-Pruning can be seen as soft Weight-Pruning as it prunes bits, not the whole weight element.
In extensive experiments, we demonstrate that sparse mult -less networks trained with Bit-Pruning show a better accuracy-energy trade-off than sparse mult networks trained with Weight-Pruning.
(Code will be open-sourced)
I NTRODUCTION
Modern deep neural networks (DNNs) contain numerous dot-products between input features and weight matrices.
However, it is well known that multiplication ( mult ) in dot-product consumes intensive energy and space costs, challenging DNNs’ deployment on resource-constrained edge devices.
This drives several attempts for efficient DNNs by reducing the energy of mult .
[Table]
Quantization (Yin et al., 2019; Esser et al., 2019; Li & Baillieul, 2004) discretizes the weight and/or activation into a low-bit representation; low-precision mult requires less energy than the high-precision counterpart.
Power of two networks (Li et al., 2019b; Miyashita et al., 2016; Elhoushi) restricts the mult operation to the power of two (PoT); PoT mult can be realized by a bit-shift ( shift ), which consumes orders of magnitude less energy than mult .
Although these approaches have achieved remarkable success in reducing energy consumption, they define the energyefficient model structure before training .
That is, they limit model capacity and impose training with a precision that is difficult to use gradient-based optimization; e.g., approximate gradient with the straight-through estimator (STE)
(Yin et al., 2019), making it challenging to achieve good accuracy in the low-bit regime (e.g., 4bit, 2bit, or binary).
Aside from low-bit approaches, Weight-Pruning (Frankle & Carbin, 2018; Chen et al., 2022;
Yang et al., 2019; Wortsman et al., 2019) learns efficient model structure during training by removing unimportant weight from high capacity models.
Owing to its data-driven nature, it could remove the unnecessary mult without sacrificing accuracy (e.g., 95% reduction in unstructured pruning).
Empowered by the recent progress of in/near-memory computing architectures (Gholami et al., 2021), several DNN accelerators designed for the unstructured sparsity have emerged (Han et al., 2016;
Bamberg et al., 2021;
Dietrich et al., 2022; Zhang et al., 2021), realizing the efficient inference of these sparse networks.
Furthermore, optimized implementation for general-purpose CPUs also proved to be a good candidate for unstructured sparsity Kurtz et al.
Motivated by the significant energy reduction of sparse mult operations by Weight-Pruning and recent advances in the frameworks supporting unstructured sparsity, we envision new frontiers in the accuracy/energy trade-off by realizing sparse and mult -less dot-product for hardware supporting the unstructured sparsity.
Noting that the dot-product between integer weights and activation can be decomposed into a PoT basis and binary vector, we first reformulate a dot-product between integer weight and activation ( mult-add ) into an equivalent operation comprised of additions followed by bit-shifts and additions ( add-shift-add , Figure 1).
In this formulation, the number of the first add operations equals the bitcount of the weight elements in binary format.
From this observation, we propose Bit-Pruning, which removes unnecessary add (therefore unnecessary bits in weight) during training in a data-driven manner to reduce the energy consumption in the add-shift-add operation.
Because of the difficulty of optimization in binary format, we optimize parameters on a high-precision differentiable mult-add network with bit-sparsity regularization, which promotes weights to be sparse in binary format.
After training, the obtained bit-sparse mult-add network is converted to an equivalent add-shift-add network comprised of sparse add for efficient inference on DNN accelerators supporting unstructured sparsity.
As depicted in Figure 1, Bit-Pruning can be seen as a soft and fine-grained Weight-Pruning; it does not necessarily remove the whole weight elements but removes only some unnecessary bits of the weight values.
This interpretation raises the following question:
Which pruning
offers a better accuracy/energy trade-off?
mult in mult-add network (Weight-Pruning) or add in add-shift-add network
Bit-Pruning
We conducted an extensive evaluation to answer the question (section 4); the results suggest that pruning the add is several times more energy efficient than pruning the mult .
Remark : We do not argue that the add-shift-add representation itself is efficient.
In fact, the estimated energy consumption of (dense) mult-add and add-shift-add without pruning are almost the same, as we see in section 3.3.
Our research interest is to investigate whether the finegrained pruning by removing bits rather than weights can find a more efficient sparse structure; the add-shift-add representation permits this investigation.
P RELIMINARIES 2. D OT - PRODUCT WITH M ULT -A DD
The dot-product is a fundamental building block of neural networks.
In this study, we mainly focus on the dot-product that appears in convolution 1 .
Let the weights and activation be quantized to M and N bits, respectively, where M, N are sufficiently fine quantization levels such as M, N = 8 , 32 .
Convolution with the input I ∈
Z C i × W × H and weight W ∈ Z C o × C i × k × k is equivalent to matrix multiplication of unfolded input X ∈ Z C
i kk × WH and the reshaped weight W ∈ Z C o × C
i kk :
[Equation]
For brevity, we consider a single input pixel u and an output channel j .
Then, the convolution which corresponds to the dot-product of the u -th column vector of the unfolded input X · ,u and j -th row vector of the weight W j is written as follows:
[Equation]
where K = C i kk .
Computing (2) using naive dense mult-add requires K mult followed by ( K − 1) add , which consumes a lot of energy as it contains many mult operations (Table 1).
2. S PARSE M ULT -A DD BY W EIGHT -P RUNING
When a DNN processor supports unstructured weight and dynamic activation sparsity (Han et al., 2016; Bamberg et al., 2021), mult can be skipped either when the activation or weight is zero.
In this case, the number of mult operation become σ mult K where σ mult is the ratio of the nonzero element defined as bitcount ( W j ̸ = 0 | X · ,u ̸ = 0) /K .
Therefore, the dot-product using sparse mult-add consumes less energy than dense mult-add .
The sparse mult-add dot-product is typically realized by Weight-Pruning , which trains a network with the weight-sparsity regularization L wgt 2 (Liu et al., 2015; Han et al., 2015):
[Equation]
where L task is a loss specific to the given task (e.g., cross-entropy for classification), W = { W ( l ) }
Ll =1 is a set of weight matrices from all L layers, X ( l ) is an input of l -th layer, and η is a hyper-parameter to balance the accuracy and computational energy.
In practice, l 1 -norm or Hoyer(-
Square) loss (Yang et al., 2019) is applied to relax l 0 -norm for efficient training.
Note that, when the target hardware only supports unstructured weight sparsity (no activation sparsity); the same argument can be applied by replacing X ( l ) ⊙ with numel( X ( l ) ) .
B IT -P RUNING
For more efficient DNN inference, we propose a sparse and mult -less dot-product by softening the
Weight-Pruning.
Our intuition is that better energy-accuracy tradeoff could be achieved by bit-level fine-grained pruning instead of the weight-level course pruning.
We first reformulate a dot-product between integer weight and activation into an equivalent operation comprised of add-shift-add , which does not contain heavy mult (Section 3.1, Figure 1).
In this formulation, we can see that the number of the first add (which directly translates to energy consumption) equals the bitcount of the weight element in binary format.
Then, we propose Bit-Pruning , which removes unnecessary bits of weight elements during training to achieve efficient inference (Section 3.2).
3. R EFORMULATE D OT - PRODUCT WITH A DD -S HIFT -A DD
An M -bit weight matrix 3 W ∈ Z C o × K is represented in binary format with a tensor B ∈ { 0 , 1 } C o × M × K , where B j, · ,k ∈ { 0 , 1 } M is the binary representation of the weight value W j,k ∈ Z :
[Equation]
k
[Equation]
For and y ∈ Z J × K , x ⊙ y ∈ Z I × J × K is defined as ( x ⊙ y ) i,j,k
= x i,j · y j,k .
For simplicity, we assume the weight element W j,k is positive in the following explanation.
Then, we rewrite the mult-add dot-product (2) into the following add-shift-add formulation:
[Equation]
This is computed first by accumulating the input X · ,u along the binary vector B j,m using add , then multiplying 2 m using shift , and finally aggregating the results of all M bits by add .
When the operation is mapped into the hardware supporting the unstructured sparsity, the first add using the binary vector B j,m requires bitcount ( B j,m ) times add .
The shift and last add are executed for M − 1 times (Table 2).
3. S PARSE A DD -S HIFT -A DD BY B IT -P RUNING
In the add-shift-add formulation of (5), the number of the first add operations equals the bitcount of the binary tensor B , which directly translates to energy consumption.
Therefore, our goal is to learn the sparse binary tensor of the add-shift-add network that also achieves good performance by minimizing the following loss:
[Equation]
where B = { B ( l ) } Ll =1 is a set of binary tensors from all L layers.
However, binary parameters prohibit gradient computation, making it difficult to train with standard backpropagation.
Instead of directly optimizing the binary tensor B , as done by Yang et al.
(2021), we deal with the original weight matrix W while promoting it to be sparse in the binary format B .
Notice that any dot-product with the integer weight can be precisely expressed as the add-shift-add dot product, and the number of add counts for B depends on the corresponding integer weight W .
For example, when W j,k is zero, then B j, · ,k = [0 , 0 , 0 , 0 , 0 , 0 , 0] ⊤ , and there is no add .
When W j,k is
PoT (e.g., 2 3 ), then B j, · ,k = [0 , 0 , 1 , 0 , 0 , 0 , 0] ⊤ , and there is only one add .
In such a way, a number of add in add-shift-add dot-product depends on the value of the corresponding weight W j,k (Figure 3a), and there is favorable value in terms of energy consumption.
We leverage this to propose a framework called Bit-Pruning, which optimizes the network’s energy consumption in a data-driven manner.
The binary tensor B of the add-shift-add network is trained as the high-precision mult-add network having weight W (Figure 2), and a sparsification of B is realized by imposing the bit-sparsity regularization L bit on weight W :
[Equation]
where L bit measures a deviation of the current weight W from bit-sparse proximal weight ˆ W , explained as follows.
Similar to Weight-Pruning (3), we relax the l 0 -norm in (7) with l 1 -norm or
Hoyer(-Square) loss in practice.
Proximal weight Given the current weight W , the proximal weight
ˆ W is computed (element-wise) as a minimizer of the sum of weight proximity cost C mv and add count cost C add :
[Equation] [Equation]
[Figure]
where C add ( w ) is the bitcount of w in binary format (Figure 3a) and C mv ( w, w ′ ) measures the proximity of two weight values w and w ′ .
In this study, we adopted C mv ( w, w ′ ) :
= | sgn( w ) | w | p − sgn( w ′ ) | w ′ | p | .
We choose p = 1 / 2 , to discount the change in C mv for a larger value (assuming accuracy is less affected by the change when it is large).
Intuitively, C add encourages ˆ
W j,k to be sparse in the binary format while C mv keeps it close to the current weight W j,k and λ ( l ) controls its balance (larger λ induce more sparsity).
Owing to the discrete structure of the weight space (e.g., 8 bit), the solution of (8) can be analytically computed for given λ
(Figure 3b-3c).
Efficient sparse inference.
Once the training has been completed, the high-precision mult-add network is converted into a mathematically equivalent add-shift-add network for efficient inference.
It is expected to be comprised of sparse add which consumes less energy.
3. W HICH IS EFFICIENT ? W EIGHT -P RUNING OR B IT -P RUNING
[Table]
An important question in this study is whether pruning the add (Bit-Pruning) is more efficient than pruning the mult (Weight-Pruning).
When considering the dot-product between two K -dimensional, M -bit vectors, the energy consumption of sparse add-shift-add ( E add-shift-add ) over that of sparse mult-add ( E mult-add ) is calculated from Table 2:
[Equation]
where σ mult and σ add are the ratio of nonzero elements in weight ( W j ) and the ratio of nonzero bits in its binary format ( B j ), respectively.
E mult , E add , and E shift are energy consumption of corresponding operations
and we assume K ≫ M in the last approximation.
In the case of ASIC, we
[Table]
roughly get E mult
≈
ME add from Table 1; therefore, the computational efficiency of Bit-Pruning over Weight-Pruning is almost determined by the ratio of obtained connection density σ add /σ mult .
We expect that the σ add in a Bit-Pruned network is much smaller than the σ mult in a WeightPruned network when both networks achieve comparable accuracy.
This is because Bit-Pruning removes connections more finely than Weight-Pruning; in other words, given the same nonzero ratio, i.e., σ add = σ mult , Bit-Pruned weight can have a more nonzero element (at most M times) than Weight-Pruned weight when it learns a weight that is sparse in binary format.
In the extreme case where all the nonzero weights in a network are represented as PoT, the computational cost in add-shift-add representation is M times smaller than that in mult-add representation.
The evaluation on actual DNN training is presented in Section 4.
3. B IT -P RUNING AS A UNIFIED LOW - ENERGY MODEL
(Non-uniform) quantization.
Bit-Pruning effectively applies different non-uniform mixedprecision quantization for each weight element .
As a result of the Bit-Pruning, we will get a sparse
B ; this has a similar effect to the non-uniform quantization of weight.
Existing quantization techniques learn (non-uniform) step size for a group of weights ; contrary,
Bit-Pruning realizes pseudonon-uniform quantization by selecting a bit pattern .
PoT dot-product.
Bit-Pruning can be interpreted as the soft version of the (additive) PoT quantization.
The bit-sparsity regularization prefers sparse binary representation, which corresponds to the PoT; however, it could learn to choose individual non-PoT values for each weight element when necessary (i.e., when it achieves less total energy for the same accuracy).
The learned weight by
Bit-Pruning is dominated by PoT, while some have multiple nonzero bits (See Figure 6).
Unstructured Weight-Pruning.
Bit-Pruning can be seen as a soft, fine-grained version of WeightPruning.
Both bit-sparsity and weight-sparsity regularization loss are minimized when W = 0 ; this corresponds to zero add in add-shift-add and zero mult in mult-add .
However, bitsparsity regularization does not necessarily remove the whole weight elements but only some unnecessary bits in the weight element.
E XPERIMENT 4. E VALUATION OF ACCURACY / ENERGY TRADE - OFF
Which is energy efficient; pruning add (Bit-Pruning) or pruning mult (Weight-Pruning)?
We evaluated the accuracy/energy trade-off of both methods on CIFAR-10 (Krizhevsky et al., a), CIFAR-
(Krizhevsky et al., b), and ImageNet (Deng et al., 2009).
For both bit-sparsity regularization L bit (7) and weight-sparsity regularization L wgt (3) which is used for all the conv layer, we approximate l 0 -norm using Hoyer-Square loss (Yang et al., 2019).
We use the same network architecture, quantization method (we use LSQ Esser et al.
and training strategy except for the loss function for both frameworks.
The primary experimental setup 5 is summarized in Table 3.
To compare the trade-off, we train networks across various sparsity coefficients η of L wgt and L bit (fixed across the entire training sequence).
[Figure]
[Figure]
CIFAR-10/CIFAR-100.
We use ResNet18 (He et al., 2016a) as one of the most popular architectures and train them from scratch.
Figure 4 summarizes the results on CIFAR-10 (Figure 4a,4b,4c) and CIFAR-100 (Figure 4d,4e,4f).
ImageNet.
We conducted an experimednt using pre-trained ConvNeXt (Liu et al., 2022) on ImageNet (Deng et al., 2009).
The
ConvNeXt is designed to be energy efficient by its network architecture; c.f., it heavily utilizes depthwise convolutions to reduce the number of mult .
Results are summarized in Figure 5.
Summary.
In all ranges of accuracy, add-shift-add network trained with Bit-Pruning consumes less energy than that of mult-add network trained with Weight-Pruning.
In particular, huge energy saving is observed in high accuracy regime, demonstrating the superior data-adaptivity of Bit-Pruning over Weight-Pruning.
We see a similar trend for the large-sized, already efficient networks (ConvNeXt) on a large-scale dataset (ImageNet).
The number of add (in add-shift-add network) and mult (in mult-add network) can be compared by multiplying M (number of bits for weight) to the add count in the plots (Because we have
E mult ≈ ME add from Table 1, Section 3.3).
The number of add (from Bit-Pruning) is larger than mult (from Weight-Pruning), however; their ratio is significantly less than M ; therefore, we’ll get the better trade-off.
Notice that the Bit-Pruned network does not show efficiency in mult-add representation (BP(mult)), and the Weight-Pruned network does not show efficiency in add-shift-add representation (WP(add)).
This is because the Bit-Pruned network prefers sparse weights in binary format (Figure 6), which is not necessarily zero, and the Weight-pruned network prefer small weight, which is not necessarily sparse in binary format.
WP(add) is not efficient for the opposite reason.
[Figure]
4. A BLATION ON BASE BIT WIDTH
Can we obtain a better accuracy-energy trade-off by pruning bits from a wider bit tensor?
We conducted the accuracy/energy trade-off comparison between Bit-Pruning with different base bit widths (4bit/8bit) (Figure 7).
The result shows the network with a wide tensor (8bit) shows a better trade-off than one using a narrow tensor (4bit).
We consider it is because a network with a wide bit has more choices for selecting good PoT values, suggesting that a better bit-sparse network is obtained from the larger design space, just like the case of Weight-Pruning (Frankle & Carbin, 2018).
4. L EARNED WEIGHT DISTRIBUTION
Figure 6 visualizes histograms of learned weight.
Starting from the initial uniform distribution, both
Bit-Pruning and Weight-Pruning drive most of the weight to zero; this corresponds to removing mult in the mult-add network and add in the add-shift-add network, respectively.
In
Bit-Pruning, most of the remaining nonzero weights are concentrated on sparse values in a binary format, such as PoT. Whereas, Weight-Pruning did not show such particular preference.
R ELATED W ORKS
ShiftAddNet (You et al., 2020) is mult -less DNNs utilizing shift and add for efficient training and inference.
It realizes a better accuracy/energy trade-off than the former study utilizing only add (AdderNet (Chen et al., 2020)).
Ours and ShiftAddNet have similarities in that both utilize shift and add to realize mult -less DNNs.
However, their computational model is fundamentally different from ours; ShiftAddNet computes the interaction between weight and input by first computing the dot-product with PoT weight followed by the l 1 norm with add weight.
It does not have an equivalent mult-add dot-product.
Its limited expressiveness and the non-differentiability of the PoT operation make training harder (requiring specifically designed gradient computation) and prohibit it from reaching the accuracy of the corresponding mult-add network.
Ours can be learned as an ordinal high-precision differentiable mult-add network; therefore, it is possible to reach the performance of the original network.
Besides, motivation and target hardware is different;
ShiftAddNet aims to realize efficient training and inference on dense vector-type processors, while our target is hardware supporting unstructured sparsity solely for inference 6 .
Instead, using the recent extension, ShiftAddNAS Huihong Shi (2022), ours and ShiftAddNet can be combined, e.g., the best combination of shift-add and add-shift-add are automatically searched.
Lottery ticket hypothesis states that “dense, randomly-initialized, feed-forward networks contain sub-networks that have the equivalent accuracy as the original network (winning tickets): unstructured weight pruning naturally uncovers the winning tickets” (Frankle & Carbin, 2018; Ramanujan et al., 2020).
The comparison of different bit widths in Bit-Pruning (Section 4.2) might indicate the lottery ticket hypothesis in bits ; a network having more connections in terms of add-shift-add representation (i.e., the large binary tensor B ) has more chance of containing the winning ticket .
Bit-level sparsity have been considered in several areas.
BSQ Yang et al.
(2021) proposed a novel technique for learning the layer-wise bit-width for mixed-precision networks by inducing grouped bit-label sparsity.
The novel notion of bit-slice sparsity (sparsity within a sliced subdivision of binary weight) is introduced by Zhang et al.
(2019) to realize efficient inference on emerging ReRAM-based DNN accelerators.
Because their motivation for considering the bit-level sparsity differs from ours, their proposed method for inducing it is also totally different.
Therefore, it can not efficiently sparsify the add in the add-shift-add network 7 .
[Figure]
C ONCLUSION
We propose Bit-Pruning, a novel framework for learning efficient mult -less sparse DNNs.
The intensive evaluation shows pruning add (bit) of the add-shift-add network is more energy efficient than pruning mult (weight) of the mult-add network.
6. L IMITATIONS & F UTURE WORK
Learning weight moving cost.
In Bit-Pruning, sparse add for efficient inference is achieved by guiding the weights closer to the proximal weight (sparse in binary format and close to the current weight) defined in (8).
The distance to current weight needs to be defined as weight moving cost C mv .
In this study, we predefined it (Section 3.2).
We consider this to be sub-optimal, and it might be possible to determine the cost (e.g., by learning p ) from data to achieve a better tradeoff, e.g., some weight can move a lot without affecting the task loss and vice versa.
Evaluation on quantization susceptible module.
In this study, we evaluate Bit-Pruning on simple classification tasks using simple network modules such as conv .
Some DNN modules are known to be more susceptible to quantization noise, such as the object detection head (Li et al., 2019a), transformer module (Bamberg et al., 2021), and recurrent module (He et al., 2016b), making it challenging to quantize them in a lower bit regime.
Bit-Pruning might be an ideal choice for realizing energy-efficient inference on these modules because it could virtually utilize high-precision weight to achieve the accuracy of the original high-precision network.
Benchmark on real hardware We show the estimate of computational gain by Bit-Pruning using the basic statistic known in the literature (Table 1); we have not evaluated the actual energy consumption, wall-clock time, or area cost in the actual hardware.
The proposed add-shift-add or existing sparse mult-add models require specialized hardware that supports unstructured/dynamic sparsity or optimized CPU implementation Kurtz et al.
(2020) to exploit the sparsity to the full extent.
The sparsity-aware accelerators often utilize in/near-memory computing architecture (Gholami et al., 2021).
We believe the Bit-Pruning is more suitable than Weight-Pruning for this architecture because add logic is cheaper in size and energy to place in or near memory than mult logic.
A D ETAILS IN EXPERIMENT A. I MPLEMENTATION
We use Pytorch (Paszke et al., 2019) for all the implementation.
For experiments on CIFAR-10 and
CIFAR-100, we use a single NVidia A6000 GPU.
Training of 200 epochs took a few hours.
For experiments on ImageNet, we use A100 DGX with four GPUs.
Training of 100 epochs took five days.
A. N ETWORK ARCHITECTURE
CIFAR-10/CIFAR-100 We use ResNet18 (He et al., 2016a) for this experiment.
To adapt to the smaller input size of 32 (compared with 224 of ImageNet (Deng et al., 2009)), we follow the protocol of CIFAR-10 example used in Lighting Bolts library (Falcon & Cho, 2020); replace the first 7 × 7 convolution layer (stride 2 ) for 3 × 3 convolution layer (stride 1 ) and remove the following max-pooling layer.
The output dimensions of the final linear layer are changed according to the dataset’s number of classes (10 for CIFAR-10 and 100 for CIFAR-100).
ImageNet We use ConvNeXt-B (Liu et al., 2022).
To utilize the activation sparsity, we replace
GELU (Hendrycks & Gimpel, 2016) activation with ReLU activation.
From their intensive ablation study (figure 2 in their paper), this change of activation function has very little impact on accuracy.
A. D ATA A UGMENTATION
CIFAR-10/CIFAR-100 We follow the convention in the literature for training the datasets; randomly crop of 32 × 32 region after 4 pixel padding then applies random horizontal flip.
ImageNet We use the data augmentation techniques used in ConvNeXt (Liu et al., 2022) (settings for the fine-tuning stage) except for the stochastic depth.
Specifically, we use random erasing of the probability 0 .
25 and label smoothing of factor 0 .
We use PyTorch Image Models library (timm) (Wightman, 2019) for the data augmentation.
A. D ETAIL IN W EIGHT -P RUNING
For Weight-Pruning, we use magnitude-based pruning using Hoyer-Square Yang et al.
(2019) for the magnitude computation combined with the LSQ-based quantization.
In Weight-Pruning, weights elements that are zero after the quantization is pruned.
The algorithm for the Weight-Pruning is similar to the one proposed in the neural-wiring Wortsman et al.
, edge-popup Ramanujan et al.
(2020), and the ones adopted by Hoyer-Square Yang et al.
We use quantization by LSQ instead of thresholding, and the elements of the weight that are zero after the quantization are pruned.
Therefore, pruned weight can be active again, which is key to finding a good pruned network having good accuracy/energy tradeoff.
We adopt the Weight-Pruning setting for a fair comparison with the proposed Bit-Pruning.
In BitPruning, quantized weight is converted into binary format, and elements that become zeros are pruned.
In other words, both methods prune a value smaller than the threshold (e.g., half the quantization scale) where the difference between the target weight ( 0 for Weight-Pruning and ˆ W for
Bit-Pruning) is evaluated using Hoyer-Square.
A. D ETAIL IN B IT -P RUNING A.5. B ALANCING PARAMETER λ ( l ) FOR PROXIMAL WEIGHT
As a preliminary study investigating the effectiveness of Bit-Pruning, we use the same λ for entire layers.
But we can use a different value of λ for each layer.
It can be set proportional to the number of operations for the given layers, or it could take the susceptibility of the weight change to the output layer into account.
Learning the layer-wise λ end-to-end may also be possible using task loss.
However, this requires an additional ingenuity to approximate the gradient w.r.t.
λ (because of the non-differentiability f arg min operation).
In any case, it will introduce an additional hyperparameter to be tuned.
In this study, we focus on comparing the Weight-Pruning and Bit-Pruning in a simple and fair setup as possible.
Therefore we left the exploration in this direction for future work.
A.5. S CHEDULING THE STRENGTH η OF SPARSITY REGULARIZATION
We use constant η across the entire training sequence both for Weight-Pruning and Bit-Pruning.
We adopted this setting because we want to compare the Weight-Pruning and Bit-Pruning in a simple and fair configuration as possible.
Scheduling of the η will improve the accuracy/energy tradeoff; however, we prioritize the comparison on simple and fair configuration, and we left the exploration of the dynamic scheduling of η as future work.
[Figure]
B A DDITIONAL EXPERIMENT RESULT
As a supplemental result of the accuracy/energy comparison experiment (Figure 4) presented in the main paper, the results, including the training accuracy, are shown in Figure 8 for CIFAR-
(Figure 8a,8b,8c) and CIFAR-100 (Figure 8d,8e,8f).
The training loss of Bit-Pruning is lower than that of Weight-Pruning for the same computational energy.
It suggests that the add-shift-add network has more capacity than the mult-add network given the same energy.
Figure 9 - 10 shows the evolution of major statics during training epochs.
The results presented in
Figure 8 in the main paper is a summary of (a) and (e) of Figure 9 - 10.
Using a large weight for the sparsity regularization η (both for Weight-Pruning and Bit-Pruning), both weight and activation become sparse (Figure 9 - 10 (c), (g), (h)).
This is because the sparsity regularization ( L wgt of (3) for Weight-Pruning and L bit of (7) for Bit-Pruning) can be reduced not only by sparsifying the mult or add but also by suppressing the activation by promoting the activation to the minus region of ReLU activation function.
[Figure]
[Figure]
is equivalent to dense connection
C L OSS LANDSCAPE OF PROXIMAL WEIGHT
The loss landscape of the bit sparsity regularization of (7) is visualized in Figure 11.
There will be multiple local minima, depending on the magnitude of λ ( l ) .
When λ ( l ) is very large, the local minima will be PoT and zero; on the other hand, when λ ( l ) is zero, it coincides with the add cost
C add of Figure 3a.
Depending on the current weight value, the weights are guided toward the local minima except in the case when λ ( l ) = 0 .
(In this case ˆ W = W
and we won’t have any gradient from the bit-sparsity regularization of (7))
[Figure]
D M ORE DISCUSSION ON COMPUTATIONAL EFFICIENCY
As discussed in the main paper (Section 3.3), we roughly get E mult
≈ ME add from Table 1 in the case of ASIC; therefore, the computational efficiency of Bit-Pruning over Weight-Pruning is almost determined by the ratio of obtained connection density σ add /σ mult .
When the network weight is initialized with a uniform random variable, then about 50% of the bits in B is one (Figure 6a); in this case, their computational efficiency is comparable (same order).
By the fine-grained pruning, we expect that the average σ add in a Bit-Pruned network is much smaller than the average σ mult in a Weight-Pruned network when both networks achieve comparable accuracy (Figure 12).
In the extreme case where all the nonzero weights in a network are represented as PoT, the computational cost in add-shift-add representation is M times smaller than that in mult-add representation.
The mult -equivalent computation can be executed using roughly E add instead of E mult .
E A BLATION
In this section, we compare Bit-Pruning and Weight-Pruning in a wide range of configurations which we could not cover in the main paper.
The additional results are shown only for CIFAR10 using 8-bit activation.
Note that the code we’ll open source includes the script to run the additional experiments on other settings (same as our main experiment result shown in Figure 4).
E. S MALLER SIZED NETWORK
Wight-Pruning and Bit-Pruning assume some redundancy in weights to be pruned.
To investigate the behavior of both methods in a situation where the number of possible unnecessary weights is scarce, we’ve conducted experiments using the narrow version of ResNet18 (narrow ResNet18).
The narrow
ResNet18 has 8 × less input and output channels w.r.t the original ResNet18 we’ve used in the main experiments, except the input and final layer.
In this situation, we can not expect significant pruning (either weights or bits).
[Figure]
Figure 13 shows the result from using the narrow ResNet18.
As expected, even when there is no sparsity regularization, e.g., η = 0 , we already observe significant degeneration in accuracy compared with the original full-sized ResNet18, both on Weight-Pruning and Bit-Pruning.
Increasing the weight for the regularization η , we observed the reduction in energy at the cost of accuracy degeneration in both methods.
However, we observed a similar trend as in the case of the full-size network; the Bit-Pruning shows a better tradeoff than the Weight-Pruning.
E. D IFFERENT BALANCING PARAMETER λ ( l ) FOR THE P ROXIMAL W EIGHT
The proximal weight ˆ W is computed by finding the best balance between the gain from reducing the bit cost C bit and proximity from the current weight. The proximity is evaluated by the weight moving cost function C mv .
Intuitively, C add encourages ˆ W j,k to be sparse in the binary format while C mv keeps it close to the current weight W j,k .
The best balance is determined by choice of λ and C mv .
The proximal weight ˆ W will be sparser in binary representation when λ is large.
In the main paper, we choose to use λ ( l ) = 1 . 0 for all layers (Tab.3).
We choose this value without careful parameter search (e.g., by evaluating the test accuracy) but by simply looking at the loss landscape of Figure 11.
We consider it will induce sparsity as its local minimum is mainly composed of one nonzero element (except two nonzeroes at 96 and all zero at 0).
To see the results when λ ( l ) is small where there are more local minimums (Figure 11-(b)), we conducted an experiment using a smaller value of λ ( l ) = 0 .
1 for all layers.
Figure 14 shows the result from using smaller value of λ ( l ) ( λ ( l ) = 0 . 1 ).
We observe the opposite results from the main experiment; the accuracy/energy tradeoff of Bit-Pruning is worse than that of
Weight-Pruning.
[Figure]
The degenerated tradeoff may be attributed to the too-conservative proximal weight, or it may come from the difficulty in training using the loss function that has too many local minima (Figure 11-(b)).
E. F INE TUNING VS . T RAINING FROM SCRATCH
In the main paper, we trained the ResNet18 network from scratch for CIFAR10/100 and fine-tuned the ConvNeXt network from a pre-trained network for ImageNet.
We trained the network from scratch because we wanted to compare Weight-Pruning and Bit-Pruning in a fair and simple setting as possible.
We were concerned the results may be affected by the pre-trained network.
(We use a pre-trained network for ConvNeXt because we could not run large-scale training using of larger dataset such as ImageNet-22K.)
Nonetheless, it is important to see which performs better to consider the application scenario of BitPruning (and also Weight-Pruning).
For this, we conducted experiments to compare the fine-tuning and the train-from-scratch.
For the pre-trained network, we used the network trained as η =for both Weight-Pruning and Bit-Pruning.
We employed the training procedure used in train-fromscratch.
[Figure]
Figure 15 compares the results from fine-tuning and train-from-scratch (same results from the Figure 4-(b) in the main paper).
The results are almost the same on the high-accuracy regime but we observe faster convergence in the case of fine-tuning.
We observe the instability in the low energy regime (large η ).
This instability might be attributed to the learning-rate scheduler (e.g., warm-up, etc.) designed for the train-from-scratch scenario.
Bit-Pruning (and Weight-Pruning) could be used for both scenarios (fine-tuning and train-fromscratch).
But given the similar results (at least on the high-accuracy regime) and faster convergence, it may be beneficial to use a pre-trained network when available.
E. U NIFORM WEIGHT MOVING COST C mv WITH p = 1 .
The proximal weight ˆ W is computed by finding the best balance between the gain from reducing the bit cost C bit and proximity from the current weight evaluated by the weight moving cost function
C mv .
Intuitively, C add encourages ˆ W j,k to be sparse in the binary format while C mv keeps it close to the current weight W j,k .
The best balance is determined by choice of λ and C mv .
In section E.2, we ablate the balancing parameter λ ( l ) ; here, we ablate the choice of the weight moving cost C mv .
In the main experiments
[Figure]
we use p = 1 / 2 because we consider the changes in weight value when its norm is small to have more effect on the accuracy than when it is large.
For example, changes of 2 when the weight was 1 (200% change) may have a more significant impact on accuracy than the weight was 200 (1%).
When p = 1 / 2 , C mv changes a lot when its norm is small and does not change much when its norm is large
[Figure]
We conducted the ablation experiments using the moving cost C mv with p = 1 . 0 ; this choice assumes the change in weight will equally affect the accuracy regardless of the norm of the current weight.
Figure 17 visualizes the loss landscape of the bit-sparsity regularization when p = 1 .which is significantly different from the case when p = 1 / 2 shown in Figure 11.
Figure 16 shows the results when p = 1 . 0 .
We observe a degradation in the accuracy/energy tradeoff.
We suspect this is because the L bit with p = 1 . 0 does not reflect the tradeoff of changing the value of w to the accuracy.
This result suggests that we could expect a better tradeoff by using smarter
C mv , e.g., by learning from data (as we discussed in future work in Sec.6).
[Figure]
F C OMPARISON WITH THE B IT -S LICE LOSS
Inducing group-wise bit sparsity (bit-slice sparsity) is explored to realize efficient DNN for emerging resistive random-access memory (ReRAM)-based DNN accelerators Zhang et al.
In the
ReRAM-based accelerators
, each crossbar can not hold the whole 8-bit weight; therefore, a single weight element is distributed into the different crossbars, each having a 2-bit which they call a bitsliced representation.
In this hardware restriction, their goal is to sparsify each slice.
The sparsity within each slice is induced by minimizing the bit-slice l 1 loss, defined as a l 1 norm of each sliced weight value (in their case, each slice has 2bit).
By reducing the l 1 norm for each slice, the weight as a whole is also expected to have fewer nonzero elements in binary representation.
Ours and Zhang et al.
(2019) are different in their motivation; therefore, their technique for inducing bit-sparsity is also different.
Their goal is to induce the sparsity within the slice; on the hand, our goal is to induce the sparsity in add of the proposed add-shift-add formulations.
They induce the sparsity by minimizing the l 1 loss within each slice and do not consider the continuity of weight as a whole; on the hand, we consider the continuity of the whole weight when inducing the sparsity to keep the change of weight value minimum to prevent the accuracy deterioration by the significant change in the weight value.
Although the motivation is different, Bit-Pruning and bit-slice sparsification of Zhang et al.
(2019) share some similarities in the scene in that both aim at realizing the sparse representation of weight in binary format.
Therefore, we conducted additional experiments by comparing them in terms of their ability to induce sparsity.
Figure 18 visualize the loss landscape of their bit-slice regularization L bitslice .
Because the l 1 loss is computed inside the slice, it can not take the state of other bits into account; therefore, it can not take the continuity of the weight into account.
When the number of slices is 1 (Figure 18-(d)), the bit-slice l 1 loss reduces to the l 1 regularization for the weight value; therefore, we can not expect the sparsity as it simply reduces the norm of the weight.
When the number of slices equals the bit width M (Figure 18-(a)), it guides each bit to zero, ignoring the weight value’s continuity.
When the number of slices is 2 (Figure 18-(c), which is their experimental setup), it only considers the continuity in the slice and ignores the continuity as weight as a whole.
For example, consider the case when w = 63 (6 nonzero bit), then our proximal weight is w =(1 nonzero bit). Both weights are guided toward the proximal value, which differs only 1 in the weight value.
Still, nonzero bits are reduced significantly (from 6 to 1).
On the other hand in case of the bit-slice l 1 loss it guides the weight towards 63 10 = (00 | 11 | 11 | 11) 2 (cid:55)→
= (
| 10 | 10 | 10) .
The corresponding weight value change drastically; therefore, we’ll expect a significant accuracy drop. Worse still, the gain for the bit sparsity is small (from 6 to 3).
Our bit-sparsity loss guides the weight that maximizes the gain while preventing the weight from changing a lot by simultaneously considering the proximity of the weight value and the number of bits.
We conducted experiments to compare the result of our bit-sparsity regularization and their bit-slice l 1 regulation.
We modified their loss function calc_l1_and_zero_ratio from their provided code 8 to be used in our Bit-Pruning framework.
For a fair comparison, we also use l 1 loss (instead of Hoyer-Square loss) for our bit-sparsity regularization.
Figure 19 compare their performance.
Our bit-sparsity regularization ( l 1 version) shows a better tradeoff than the bit-slice l 1 regulation.
[Figure]
[Figure]
G W EIGHTED B IT -P RUNING R EGULARIZATION
Our Bit-Pruning loss is defined as the difference (in Hoyer-Square sense) between the proximal weight ˆ W and current weight W .
The proximal weight takes both bit-cost and proximity to the current weight into account (8).
It is a compromised point (found by solving the optimization of (8)) that reduces the bit-cost while keeping the change from the current weight small as possible.
The bit-cost C bit of the computed proximal weight is always less than or equal to the current weight regardless of the choice of the moving cost C mv ; therefore, sparsity always induces.
However, the bit-sparsity regularization does NOT take the possible reduction of the bit-cost C bit into account once the proximal weight is determined.
Let’s consider a case when w = 63 and w = 65 .
They would have the same proximal ˆ w = 64 .
Obviously, w = 63 has more nonzero bits than w = 65 .
Using the bit-sparsity regularization of (7), both w = 63 and w = 65 are guided toward w = 64 ; therefore, both weights are directed toward the sparse in binary representation having only one nonzero bit.
But we’ll expect more gain when w = 63 becomes w = 64 ( 6 (cid:55)→ 1 ) than w = 65 becomes w = 64 ( 2 (cid:55)→ 1 ).
The bit-sparsity regularization of (7) does not take this into consideration for computing the gradient.
Instead, they are treated equally, ignoring their different gain in bit-cost.
To incorporate these gains into account, we consider using weight to reflect the gain; the weighted version of the bit sparsity regularization is defined as follows:
[Equation]
[Figure]
Figure 20 shows the results.
Opposed to our expectation, we could not observe a noticeable improvement in the accuracy-energy tradeoff.
We have not found the reason yet; we may observe some improvement using a different optimizer, model designs [Figure] or datasets.
In this work, we focused on comparing Bit-Pruning and Weight-Pruning in a simple and fair setup; therefore, we left the explanation of more advanced Bit-Pruning regularization for future work.
H A DDITIONAL DISCUSSION WITH RELATED WORKS H. L OW - RANK FACTORIZATION
Low-rank factorization is often utilized to reduce the number of mult (Howard et al., 2019) by using factorized matrix multiplication.
The factorization can be realized as depthwise or 1x1 convolution; they are utilized in a computationally efficient network such as MobileNet (Howard et al., 2017) or ConvNeXt (which we evaluate in our experiment).
Recent research demonstrates that the lowrank form could be learned by optimization (Idelbayev & Carreira-Perpinan, 2020).
Our approach is orthogonal to this approach, and it can be combined, c.f., applying Bit-Pruning for the learnable low-rank network to further reduce the computation cost.
H. BSQ
BSQ Yang et al.
(2021) proposed a novel method for realizing gradient-based optimization of the layer-wise precision, targeting the hardware supporting the mixed-precision.
They formulate the optimization of layer-wise precision as the bit-wise optimization by reformulating the weight using the sum of the PoT basis (same as our binary representation of (4).
Then they directly optimize for each bit using a straight-through estimator (STE).
Ours and BSQ differ in their motivation and technique for inducing bit-level sparsity.
The goal of BSQ is to learn the layer-wise precision targeting the hardware supporting the dense mixed-precision operation (e.g., GPUs); on the other hand, our goal is to realize an efficient mult -free network based on the proposed add-shift-add with sparse add targeting the hardware supporting the unstructured sparsity (e.g., data-flow processors, CPUs, FPGAs, etc.).
Due to the difference in their goal, BSQ applies the sparsification for groups of weights (e.g., each layer), while we apply the sparsification for each individual weight element .
In BSQ, sparsity is induced by optimizing each bit independently using STE, which may suffer from severe quantization error.
In contrast, ours optimize in the quasicontinuous weight space using the
proximal weight by utilizing the equivalent of the add-shift-add and mult-add .