L OW E NTROPY D EEP N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
The movement of data between processes and memory, not arithmetic operations, dominate the energy costs of deep learning inference calculations.
This work focuses on reducing these data movement costs by reducing the number of unique weights and weight-space entropy of a network.
To this end, we investigate the merits of a method, which we call Weight Fixing Networks (WFN).
We design the approach to realise four model outcome objectives: i) very few unique weights, ii) low-entropy weight encodings, iii) unique weight values which are amenable to energy-saving versions of hardware multiplication, and iv) lossless task-performance.
Some of these goals are conﬂicting.
To best balance these conﬂicts, we combine a few novel (and some well-trodden) tricks; a novel regularisation term, (i, ii) a view of clustering cost as relative distance change (i, ii, iv), and a focus on whole-network re-use of weights (i, iii).
Our Imagenet experiments demonstrate lossless compression using 56x fewer unique weights and a 1.9x lower weight-space entropy than SOTA quantisation approaches.
I NTRODUCTION
The Importance of Data Movement Costs.
Although there has been a signiﬁcant amount of attention exploring both algorithmic (Sze et al., 2020) and hardware-based (Chen et al., 2020) approaches to reducing the energy costs of deep learning inference, there is often a noted disconnect between the two (Sze et al., 2017).
The most expensive energy costs lie in memory reads (Horowitz, 2014; Gao et al., 2017).
For every off-chip DRAM data read, you pay the equivalent of over two hundred 32-bit multiplications in energy costs 1 (Horowitz, 2014).
Algorithmic techniques that hope to reduce energy consumption have focussed predominantly on metrics like ﬂoating-point operations (FLOPs); intuitively, fewer FLOPs should translate into smaller energy costs from the reduced number of multiplications and potentially, fewer parameters to read from memory.
However, it has been observed that this is a weak proxy for the energy consumed in running a deep learning model for inference (Sze et al., 2017).
Data movement costs can still be high if we do not consider the re-use of weights, the delay between their re-use, and ﬁlter shape effects.
Simply reducing the FLOPs alone does not guarantee energy cost reductions – particularly those incurred through data read and write activity.
Accelerator Design Considerations.
These data movement costs are difﬁcult to reduce in von
Neumann architectures (Li et al., 2015; Sebastian et al., 2020), and so there is a move towards co-design of algorithm-hardware inference accelerators.
A core consideration of accelerator design lies in exploring dataﬂow mappings.
These mappings determine how data used in computation is distributed across memory components and optimised to take advantage of the re-use of weights, matrix multiplication partial-sums (psums), and data inputs (Han et al., 2016; Chen et al., 2017; 2015; 2020).
Modern deep learning accelerators also make use of hundreds of processing elements (Chen et al., 2020)
(PE’s) for computation.
Each PE contains a small amount of cheap and fast access memory to store a few pieces of information (weights, psums, inputs, etc.).
A dataﬂow mapping cuts down data movement costs by making the best use of each PE and ensuring that the information they contain is recycled as much as possible.
A weight/input/psum that cannot be used quickly after its latest use will be written to storage and require a re-read later.
Less re-use leads to increased data movement costs.
Consider, for example, weight-stationary dataﬂow, commonly
[Figure]
n i
[Figure]
used in SOTA accelerator designs (Farshchi et al., 2019; Jouppi et al., 2017).
Here the items stored statically in the PE are the model weights.
Input data is then fed into the relevant PE’s where multiplication is conducted locally.
The ideal situation for a dataﬂow mapping would be to pay a single data movement cost for each unique weight value and then reference these weight values using indexing.
The indexing costs can be kept low using an appropriate encoding scheme such as
Huffman coding and approximated by the entropy of the weight space.
Thus, the indexing access costs plus unique value access costs can be much smaller than the unquantised network (Mao &
Dally, 2016; Wu et al., 2018).
Objectives.
So we ask ourselves what we could do algorithmically to maximise the beneﬁt of accelerator dataﬂows?
We think an easy win is to reduce the number of unique weights a network uses.
Fewer unique weights whilst ﬁxing the network topology and the total number of parameters will mean that more weights are re-used more often.
This additional re-use gives more opportunity to dataﬂows to maintain often-used weights in PE’s.
To further enhance the compressibility, it is desirable for the distribution of the unique weights to be concentrated around a handful of values.
The high probability density weights would then be used more often and could then be reliably stored inside PE’s, saving both the cost of overwriting these weight values and re-fetching them when needed later.
Finally, we ask what the ideal values of these weights would be.
From a computational perspective, not all multiplications are created equal.
Integer powers-of-two, for example, can be implemented as simple bit-shifts.
Mapping the weights used most to these values offers potential further energy reductions.
Putting these three requirements together: few unique weights; a low-entropy encoding with a distribution of weights highly concentrated around a tiny subset of values; and a focus on powers-of-two values for weights — all motivated to reduce computation costs in accelerator designs — we present our contribution.
Weight Fixing Networks.
Our work’s overarching objective is to transform a network comprising many weights of any value (limited only by value precision) to one with the same number of weights but just a few unique values.
Rather than selecting the unique weights a priori, we let the optimisation guide the process in an iterative cluster-then-train approach.
In each iteration, we cluster an ever-increasing subset of weights to one of a few cluster centroids.
We map the pre-trained network weights to these cluster centroids, which are the pool of unique weights.
The training stage follows standard gradient descent optimisation to minimise performance loss with two key additions.
Firstly, only an ever decreasing subset of the weights are free to be updated.
And secondly, we use a new regularisation term to penalise weights in proportion to their nearest clusters’ relative distance.
We iteratively cluster subsets of weights to their nearest cluster centre, as has been demonstrated successfully previously (Zhou et al., 2017).
The way we determine which subset to move is a core component of our contribution which leads to the superior compression results we achieve.
[Figure]
Small Relative Distance Change.
Rather than selecting subsets with small Euclidean distances to cluster centres, or those that have small magnitude (Zhou et al., 2017), we make the simple observation that the relative – as opposed to absolute – weight change matters.
The distance a weight is moved when quantised is dependent on the distance between the weight w i and its new value w i + δw i .
When the new value is zero — as is the case for pruning methods — then the magnitude of the weight is the distance.
Yet, this is not the case more generally.
We demonstrate the importance of optimising quantisation for small relative changes with simple empirical observations.
Using a pre-trained ResNet-18 model, we test adding relative vs absolute noise to the layers’ weights and measure the accuracy change.
For relative noise we set a noise level β and adjust all weights w i in a layer l as: w li ← w li + N (0 , β | w li | ) .
The standard deviation of noise added is determined by the original value w li .
We contrast this with absolute noise experiments, where we instead set w li ← w li + N (0 , β | w l | ) where | w l | corresponds to the mean absolute weight value in layer l .
We run each layer- β combination experiment multiple times – to account for ﬂuctuation in the randomised noise – and present the results in Figure 1.
Even though the mean variation of noise added is the same, noise relative to the original weight value (multiplicative noise) is much better tolerated than absolute (additive noise).
Since moving weights to quantisation centres is analogous to adding noise, we translate these results into our approach and prioritise weights that have small relative distances to be clustered ﬁrst.
We ﬁnd avoiding signiﬁcant quantisation errors requires ensuring that | δw i | | w i | is small for all weights.
If this is not possible, then performance could suffer.
In this case, we create an additional cluster centroid in the vicinity of an existing cluster to reduce this relative distance.
Our work also challenges the almost universal trend in the literature (Yuhang Li, Xin Dong, 2020; Jung et al., 2019;
Zhang et al., 2018a;
Zhou et al., 2016;
Yamamoto, 2021; Oh et al., 2021) of leaving the ﬁrst and last layers either at full precision or 8-bit; we attempt a full network quantisation.
The cost of not quantising the ﬁrst layer – which typically requires the most re-use of weights due to the larger resolution of input maps – and the ﬁnal linear layer – which often contains the largest number of unique weight values – is too signiﬁcant to ignore.
With multiple stages of training and clustering, we ﬁnish with a signiﬁcantly reduced set of unique weights.
The regularisation term encourages high probability regions in the weight distribution and a lower-entropy weight-space.
The initial choice of cluster centroids as powers-of-two helps us meet our third objective – energy-saving multiplication.
Overall we ﬁnd four distinct advantages over the works reviewed:
• We assign a cluster value to all weights — including the ﬁrst and last layers.
• We emphasise a low entropy encoding with a regularisation term, achieving entropies smaller than even those seen using 3-bit quantisation approaches – over which we report superior performance.
• We require no additional layerwise scaling; the unique weights are shared across all layers.
• WFN substantially reduces the number of unique parameters in a network when compared to existing SOTA quantisation approaches.
R ELATED W ORK
Clip and Scale Quantisation.
Typical quantisation approaches reduce the number of bits used to represent components of the network.
Quantisation has been applied to all parts of the network with varying success; the weights, gradients, and activations have all been attempted (Hubara et al., 2016; Lee et al., 2017;
Jung et al., 2019; Yang et al., 2019;
Shkolnik et al.
; Zhou et al., 2016).
Primarily, these approaches are motivated by the need to reduce the energy costs of the multiplication of 32-bit ﬂoating-point numbers.
This form of quantisation uses a rounding function and scaling factor ( sf ) in making value adjustments where a weight w i is mapped to a new value w (cid:48)
i = sf ∗ round ( w i ) for some predetermined rounding function.
The scaling factor (determined by a clipping range) can be learned channel-wise (Jacob et al., 2018; Zhang et al., 2018c), or more commonly, layerwise with separate formulations.
This results in different channels/layers having a diverse pool of mapping values for the network weights/activations/gradients.
Quantisation can be performed without training — known as post-training quantisation, or with added training steps – called quantisation-aware training (QAT).
Retraining incurs higher initial costs but results in superior performance.
A clipping+scaling quantisation example relevant to our own is the work of (Zhou et al., 2017), where the authors restrict the layerwise rounding of weights to powers-of-two.
The use of powers-of-two has the additional beneﬁt of energy-cheap bit-shift multiplication.
A follow-up piece of work (Yuhang Li, Xin Dong, 2020) suggests additive powers-of-two (APoT) instead to capture the pre-quantised distribution of weights better.
Weight Sharing Quantisation.
Other formulations of quantisation do not use clipping and scaling factors (Stock et al., 2020; Tartaglione et al., 2021; Wu et al., 2018).
Instead, they adopt clustering techniques to cluster the weights and ﬁx the weight values to their assigned group cluster centroid.
These weights are stored as codebook indices, allowing for compressed representation methods such as Huffman encoding to squeeze the network further.
Unlike clipping+scaling quantisation techniques, and like ours, these methods share the pool of weights across the entire network.
The work by (Wu et al., 2018) is of particular interest since both the motivation and approach are related to ours.
Here the authors use a spectrally relaxed k-means regularisation term to encourage the network weights to be more amenable to clustering.
In their case, they focus on a ﬁlter-row codebook inspired by the row-stationary dataﬂow used in some accelerator designs (Chen et al., 2017).
However, their formulation is explored only for convolution, and they restrict clustering to groups of weights (ﬁlter rows) rather than individual weights due to computational limitations as recalibrating the k-means regularisation term is expensive during training.
Similarly (Stock et al., 2020; Fan et al., 2021), focus on quantising groups of weights into single codewords rather than the individual weights themselves.
Weight-sharing approaches similar to ours include (Ullrich et al., 2017).
The authors use the distance from an evolving Gaussian mixture as a regularisation term to prepare the clustering weights.
Although it is successful with small dataset-model combinations, the complex optimisation — particularly the additional requirement for Inverse-Gamma priors to lower-bound the mixture variance to prevent mode collapse— limits the method’s practical applicability due to the high computational costs of training.
In our formulation, the weights already ﬁxed no longer contribute to the regularisation prior, reducing the computational overhead.
We further reduce computation by not starting with a complete set of cluster centres but add them iteratively when needed.
M ETHOD
Quantisation.
Consider a network N parameterised by N weights W = { w 1 , ..., w N } .
Quantising a network is the process of reformulating N (cid:48) ← N where the new network N (cid:48) contains weights which all take values from a reduced pool of k cluster centres C
= { c 1 , ..., c k } where k (cid:28) N .
After quantisation, each of the connection weights in the original network is replaced by one of the cluster centres w i ← c j , W (cid:48)
= { w (cid:48)
i | w (cid:48)
i ∈ C, i = 1 , · · · , N } , | W (cid:48)
| = k , where W (cid:48) is the set of weights of the new network N (cid:48) , which has the same topology as the original N .
Method Outline.
WFN is comprised of T ﬁxing iterations where each iteration t ∈ T has a training and a clustering stage.
The clustering stage is tasked with partitioning the weights into two subsets W = W tfixed ∪ W tfree .
W tfixed is the set of weights ﬁxed at one of the cluster centre values c k ∈ C .
The weights w i ∈ W tf ixed are not updated by gradient decent in this, nor any subsequent training stages, these weights are ﬁxed .
In contrast, the free-weights denoted by W tfree remain trainable during the next training stage.
With each subsequent iteration t we increase the proportion of weights that take on ﬁxed cluster centre values p t such that p 0 < p 1 . . .
< p T , where p t =
| W tfixed | | W | .
By iteration T , all weights will be ﬁxed to one of the cluster centres making p T = 1 .
The training stage combines gradient descent based error correction along with a tight-cluster regularisation term and is tasked with maintaining lossless performance as we ﬁx more of the weights to cluster centres.
Clustering Stage.
In the clustering stage, we work backwards from our goal of minimising the relative distance travelled for each of the weights to determine which values cluster centres c i ∈ C should take.
Given a weight w i ∈ W and cluster centre c j ∈ C we use a distance function
D ( w i , c j ) =
| w i − c
j | and the original weight value w i to deﬁne a relative distance measure
D rel ( w i , c j ) =
D ( w i ,c j ) | w i | .
Normalising by w i ensures that distance is measured proportionally to the original weight value, rather than in absolute terms.
To use this in determining the cluster centres, we enforce a threshold δ on this relative distance, D rel ( w i , c j ) ≤ δ for small δ .
We can then deﬁne the cluster centres c j ∈ C which make this possible using a simple recurrence relation.
Assume we have a starting cluster centre value c j , we want the neighbouring cluster value
c j +1 to be such that if a network weight w i is between these clusters w i ∈ [ c j ,
c j +1 + c j] then D rel ( w i , c j ) ≤ δ .
Plugging in c j +1 + c jand c j into D rel and setting it equal to δ we have:
[Equation]
this leads to a recurrence relation:
[Equation]
that provides the next cluster centre value given the previous one.
With this, we can generate all the cluster centres given some boundary conditions.
Starting with some value c 0 , we generate c 1 which is then used to generate c 2 and so on.
We introduce the lower-bound cluster threshold c
= δ 0 , which we use to generate the cluster centres and any absolute weight value that is less than δ 0 is assigned the cluster value 0.
This lower bound serves two purposes: ﬁrstly, it reduces the number of proposal cluster centres which would otherwise increase exponentially in density around zero, and additionally, the zero-valued weights will allow sparsity-leveraging hardware to avoid operations that use these weights, reducing the computational overhead.
As an upper-bound, we stop the recurrence once a cluster centre is larger than the maximum weight in the network, max
[Equation]
Generating the Proposed Cluster centres.
Putting this together, we have a starting point c
= δ 0 , a recurrence relation to produce cluster centres given c 0 that maintains a relative distance change when weights are moved to their nearest cluster centre, and a centre generation stopping condition c
j ≤ max i ∈ W | w i | , c j ∈ C .
We use the δ 0 value as our ﬁrst proposed cluster centre c 0 with the recurrence relation generating a proposed cluster set of size s .
Since all these values will contain only positive values, we join this set with its negated version along with a zero value to create a proposal cluster set C S
= { a [Table]
[Table]
[Table] [Table]
[Figure]
To account for the zero threshold δ 0 and for ease of notation as we advance, we make a slight amendment to the deﬁnition of the relative distance function D rel ( w i , c j ) :
[Equation]
Reducing k with Additive Powers-of-two Approximations.
Although using all of the values in C S as centres to cluster the network weights would meet the requirement for the relative movement of weights to their closest cluster to be less than δ , it would also require a large number of k = | C S | clusters.
In addition, the values in C S are also of full 16-bit precision, and we would prefer many of the weights to be powers-of-two for ease of multiplication in hardware.
With the motivation of reducing k and encouraging powers-of-two clusters whilst maintaining the relative distance movement where possible, we look to a many-to-one mapping of the values of C S to further cluster the cluster centres.
Building on the work of others (Zhou et al., 2017;
Yuhang Li, Xin Dong, 2020), we map each of the values c i ∈ C S to their nearest power-of-two, round ( c i ) =
sgn ( c i )2 (cid:98) log 2 ( c i ) (cid:101) and, for ﬂexibility, we further allow for additive powers-of-two rounding.
With additive powers-of-two rounding, each cluster value can also come from the sum of powers-of-two values ( b -bit) up to order ω where the order represents the number of powers-of-two that can contribute to the approximation 2 .
As an example, given c k = 0 . 45 , c k ∈ C S , an ω =approximation would round c k to its nearest power-of-two c 1 k = 12 1 = 0 .
whereas an order ω =approximation is c 2 k = 12 1 − 12 4 = 0 .
A higher-order grants a better approximation, but at the cost of more cluster centroids overall and requires ω bit-shifts + additions in hardware.
We give full details of the algorithm to map the proposal set C S to the ω -order approximation (cid:101) C ω in the
Appendix.
Minimalist Clustering.
We are now ready to present the clustering procedure for a particular iteration t , which we give the pseudo-code for in Algorithm 1.
We start the iteration with ω =and a set of weights not yet ﬁxed W tfree .
For the set of cluster centres (cid:101) C ω of order ω , let c ω ∗ ( i ) =
min c ∈ (cid:101) C ω D
+rel ( w i , c ) be the one closest to weight w i .
n ωk = (cid:80) i
I [ c ωk =
c ω ∗ ( i )] counts the number of weights assigned to cluster centre c ωk ∈ (cid:101) C ω , where the indicator function I [ x ] is 1 if x is true and 0 otherwise.
Let k ∗ =
arg max k n ωk so that c ωk ∗ is the modal cluster.
For the cluster k ∗ let permutation π of { 1 , . . .
, N } that maps w i (cid:55)→ w (cid:48)
π ( i ) , be such that the sequence ( w (cid:48) 1 ( k ∗ ) , w (cid:48) 2 ( k ∗ ) , . .
w (cid:48)
N ( k ∗ )) is arranged in ascending order of relative distance from the cluster c ωk ∗ .
In other words, D +rel ( w (cid:48)
i ( k ∗ ) , c ωk ∗ ) ≤ D +rel ( w (cid:48)
i +1 ( k ∗ ) , c ωk ∗ ) , for i = 1 , . . .
, ( N − 1) .
We choose n to be the largest integer such that:
[Equation]
and deﬁne { w (cid:48) 1 , w (cid:48) 2 , . .
w (cid:48) n } to be the set of weights to be ﬁxed at this stage of the iteration.
These are the weights that can be moved to the cluster centre c ωk ∗ without exceeding the average relative distance δ of the weights from the centre.
The corresponding weight indices from the original network N are in { π − 1 (1) , . .
, π − 1 ( n ) } , and called fixed new in the algorithm.
If there are no such weights that can be found, i.e. , for some cluster centre l ∗ , the minimum relative distance D +rel ( w (cid:48)
( l ∗ ) , c l ∗ ) >
δ , the corresponding set fixed new is empty.
In this case, there are no weights that can move to this cluster centre without breaking the δ constraint and we increase order ω ← ω
+ 1 to compute a new c ωk ∗ , repeating the process until | fixed new | > 0 .
Once the condition | fixed new | > 0 is satisﬁed, we ﬁx the identiﬁed weights { w (cid:48) 1 , w (cid:48) 2 , . . .
, w (cid:48) n } to their corresponding cluster centre value c ωk ∗ and move them into W t +1 fixed .
We continue the process of identifying cluster centres and ﬁxing weights to these centres until | W t +1 fix | ≥ Np t , at which point the iteration t is complete and the training stage of iteration t + 1 begins.
Our experiments found that a larger δ has less impact on task performance during early t iterations and so we reduce δ linearly with each increasing t ∈ T (full details in the Appendix).
We will show later that, with a small δ , over 75% of the weights can be ﬁxed with ω = 1 and over 95% of weights with ω ≤ 2 .
Training Stage.
Despite the steps taken to minimise the impact of the clustering stage, without retraining, performance would suffer.
To negate this, we perform gradient descent to adjust the remaining free weights W tfree .
This allows the weights to correct for any loss increase incurred after clustering where training aims to select values W tfree that minimise the usual loss (cross-entropy in our case) whilst W fixed remain unchanged.
Cosying up to Clusters.
Having the remaining W tfree weights closer to the cluster centroids C post-training makes clustering less damaging to performance.
We coerce this situation by adding a regularisation term — given in Equation 5 — to the retraining where p ( c j | w i ) =
[Equation]
The idea is to penalise the free-weights W tfree proportionally to their distance away from the clusters to which they are most likely to be assigned — i.e. the closest.
Clusters that are unlikely to be weight w i ’s nearest — and therefore ﬁnal ﬁxed value — do not contribute much to the penalisation term.
We balance this regularisation term against the cross-entropy training loss with the γ hyper-parameter, which we ﬁx to be a α proportion of the other loss term γ =
α L cross entropy L reg .
We note that the γ term is detached from the computational graph and treated as a constant (otherwise the L reg term would cancel).
[Equation]
E XPERIMENT D ETAILS
We apply WFN to fully converged models trained on the CIFAR-10 and ImageNet datasets.
Our pre-trained models are all publicly available with strong baseline accuracies 3 : Resnet-(18,34,50)
(Wu et al., 2017), MobileNetV2 (Sandler et al., 2018) and, GoogLeNet (Chollet, 2017).
We run ten weight-ﬁxing iterations for three epochs, increasing the percentage of weights ﬁxed until all weights are ﬁxed to a cluster.
In total, we train for 30 epochs per experiment using the Adam optimiser (Kingma & Ba, 2015) with a learning rate 2 × 10 − 5 .
We use grid-search to explore hyper-parameter combinations using ResNet-18 and MobileNetV2 models with the CIFAR-10 dataset and ﬁnd that the regularisation weighting α = 0 .
4 works well across all experiments reducing the need to further hyper-parameter tuning as we advance.
The distance threshold δ gives the practitioner control over the compression-performance trade-off (see Figure 3), and so we report a range of values.
We give full details of the experiments run along with the results of a hyper-parameter ablation study using
CIFAR-10 in the Appendix.
[Figure]
t s F i x e d
R ESULTS
[Figure]
[Table] [Table]
We begin by comparing WFN for a range of δ values against a diverse set of quantisation approaches that have comparable compression ratios (CR) in Table 5.
The 3-bit quantisation methods we compare include: LSQ (Esser et al., 2020), LQ-Net (Zhang et al., 2018b) and APoT (Yuhang Li,
Xin Dong, 2020).
We additionally compare with the clustering-quantisation methods using the
GoogLeNet model:
Deep- k -Means (Wu et al., 2018) whose method is similar to ours, KQ (Yu et al., 2020), and GreBdec (Yu et al., 2017).
Whilst the results demonstrate WFN’s lossless performance with SOTA CR, this is not the main motivation for the method.
Instead, we are interested in how
WFN can reduce the number of unique parameters in a network and corresponding weight-space entropy which together give the potential beneﬁt of future accelerator designs to reduce data movement costs.
We note to the interested reader there are additional theoretical reasons for pursuing low weight-space entropy networks (Hinton & van Camp, 1993; Gr¨unwald, 2000; Hansen & Yu, 2001), but our primary focus is from the perspective of reducing accelerator data movement costs. Capturing the beneﬁt of algorithmic developments to reduce data movement costs in accelerator designs is non-trivial due to the signiﬁcant dependency on architecture memory hierarchy, communication orchestration costs and the dataﬂow mapping used. Some active works attempt to provide robust estimations in bridging the gap between estimation and actual energy costs, through acceler-
[Table] ator mapping estimation (Kwon et al., 2019) or ﬁxing the mapping and estimating the energy costs on a particular accelerator (Yang et al., 2017). However, support is limited to speciﬁc ﬁxed architecture designs and does not account for the referenced weight-sharing.
Instead, we rely on analytical metrics as a proxy such as the one proposed by (Wu et al., 2018) itself a subset of measurements conducted by (Sakr et al., 2017) (Equation 6) which attempts to estimate the weight representation (and correlated data movement) energy costs as the product of N w , B w and | W | , which refer to the number of operations the weights are involved in, the weight-representation bit-width for each weight, and the unique weights in the network, respectively. Due to the low weight-space entropy we achieve, we suggest Huffman encoding to represent the network index-set (as is used by various accelerator designs (Moons & Verhelst, 2016; Han et al., 2016)) and so the weight-representational bit-width will vary for each weight. We amend the original formulation to account for this (Eq 7). Here N w i is the number of times weight w i is used in during an inference computation, and B w i refers to the Huffman-encoded representation bit-width of w i .
[Equation]
The authors of the APoT have released the quantised model weights and code, and we use the released model-saves 4 of this SOTA model to compare the entropy, Rep (cid:48)
( N (cid:48) ) , unique parameter count, model size and accuracy in Table 2.
Our work outperforms APoT in weight-space entropy, unique parameter count and weight representational cost by a large margin.
Taking the ResNetexperiments as an example, the reduction to just 164 weights compared with APoT’sdemonstrates the effectiveness of WFN.
This is in part due to our full-network quantisation (APoT, as aforementioned, does not quantise the ﬁrst, last and batch-norm parameters), but even when we discount these advantages and look at weight subsets ignoring the ﬁrst, last and batch-norm layers WFN uses many times fewer parameters and half the weight-space entropy. We provide a further breakdown of these results in the Appendix.
Finally, let us examine how WFN achieves the substantial reduction in weight-space entropy.
In Figure 4 we see that not only do the resultant WFN networks have very few unique weights, but the weight distribution is such that the vast majority of all of the weights are a small handful of powers-of-two values (order 1) and the other unique weights (outside of the top 4) are of low frequency and added only in the ﬁnal ﬁxing iterations.
C ONCLUSION
We have presented WFN, a compression pipeline that can successfully compress whole neural networks.
A single network codebook, focusing on reducing entropy of the weight-space coupled with the number of unique weights in the network.
The WFN process produces highly compressible and hardware-friendly representations of networks using just a few unique weights without performance degradation.
Although the results demonstrate strong compression ability, few unique parameters, and low weight-space entropy, the true potential of the method is that it will gives accelerator designers more scope for weight re-use, keeping most/all weights close to computation,
A. D ETAILS OF THE P OWERS - OF - TWO A PPROXIMATION A LGORITHM
We map our proposal set C S to a ω -order approximation where each of the clusters c k ∈ C S are written as ω powers-of-two (Eq 8).
We do so using Algorithm 2.
Figure 5 demonstrates how the values of C S are rounded given different orders.
[Equation]
[-k = round ( c k ) for i = 
 → ω do δ c k ← c k − c (cid:48) k if | δ c k | ≥ δc k then c (cid:48)-][Equation]
[Figure]
To account for the observation that smaller relative distances are well tolerated in the early stages but less so as the ﬁnal weights are quantised (high t ), δ linearly.
For any given t ∈ T we use δ t =
δ ( T − t + 1) as our threshold.
A. E XPERIMENT D ETAILS
We give a full breakdown of the parameters used across all experiments ran in Table
[Figure]
[Table]
A. H YPER - PARAMETER E XPLORATION
We conducted a hyper-parameter search on both the ResNet-18 and MobileNet (Figures 6, 7).
We ﬁnd similar results across models when selecting hyper-parameters, and additionally, we observe that as we increase the regularisation term weighting, a weight-space entropy reduction.
Further, we found that negating this term altogether α
=
has the surprising effect of increasing the resultant entropy values and causes an accuracy performance drop, demonstrating the utility of the regularisation term.
A. L AYERWISE B REAKDOWN
In Figure 8 we examine how the parameter count and layer-parameter entropy change with each layer for both the WFN and APoT approaches.
We ﬁnd both gains over the unquantised layers of
APoT, but also that the entropy and parameter count in the convolutional layers (those quantised by
APoT) are similar.
A. A F ULL M ETRIC C OMPARISON
In Table 4 we give the full metric breakdown comparing WFN to the state-of-the-art APoT work. We calculate the unique parameter count and entropy values for subsets of the weights. No BN corresponds to all weights other than those in the batch-norm layers, and No BN-FL is the set of weights not including the ﬁrst-last and batch-norm layers. It’s clear here that WFN outperforms
APoT even when we discount the advantage gained of taking on the challenge of quantising all layers.
[Figure] [Figure] [Table]
A. P RUNING E XPERIMENTS
To explore how WFN interacts with pruning we conduct a simple set of experiments.
Instead of starting the WFN process with all weights un-ﬁxed we randomly select p % of the weights to be pruned in each layer. We then run WFN as before starting with p t = p , reducing the number of T iterations. The results, shown in Figure [Table]
are conducted with a ResNet-18 and Cifarcombination, painting a mixed picture.
On the one hand, WFN and pruning at lower levels ( < 50% ) is well tolerated and provide two beneﬁts, a lower weight-space entropy and few weight-ﬁxing iterations.
On the other hand, full-precision networks can tolerate much higher ranges of pruning so there it would seem that a certain amount of synergy between the two approaches is present but this
[Figure]
It’s important to note that WFN already has a form of pruning built-in with the δ 0 value balancing the emphasis on pruning over quantisation. [Figure] L a y e r U n i q u e P a r a m C o un t
WFNAPoT (
[Figure]
[Figure]
[Figure]
APoT (
[Figure]
[Figure]
) cwbwbbcwbwbbcwbwbbcwbwbbcwbwbbcwbwbbcwbwbbcwbwbbcwbwbbcwbwbbcwbwbbcwbwbbcwbwbbcwbwbbcwbwbbcwbwbbcwbwbb lw lb
L a y e r
En t r o p y
[Figure]
[Figure]