D ECOUPLING W EIGHT R EGULARIZATION FROM B ATCH S IZE FOR M ODEL C OMPRESSION
Anonymous authors
Paper under double-blind review
A BSTRACT
Batch size selection affects the generalization ability of gradient descent, and small batch size is usually preferred.
Conventionally, weight regularization is performed for every mini-batch without considering when would be the right time to regularize weights during optimization steps.
For model compression, which regularizes weights to follow compressed forms, compression-aware training also performs weight compression for every mini-batch to compute the impact of compression on the loss function.
In this paper, we propose a new hyper-parameter called “Non-Regularization period” or NR period during which weights are not regularized (or compressed).
We ﬁrst investigate the inﬂuence of NR period on regularization using weight decay and weight random noise insertion.
Throughout various experiments, we show that stronger weight regularization demands longer NR period (regardless of batch size) to best utilize regularization effects.
From our empirical evidence, we argue that weight regularization for every mini-batch allows small weight decay/noise only and limited regularization effects such that there is a need to search for right NR period and weight regularization strength to enhance model accuracy.
Consequently, NR period becomes especially crucial for model compression where strong weight regularization is necessary to increase compression ratio.
Using various models, we show that simple weight regularization to comply with compression formats along with long NR period is enough to achieve high compression ratio and model accuracy.
I NTRODUCTION
For Deep Neural Networks (DNNs), a common training method updates weights by gradient descent and explicit weight manipulation through regularization.
To overcome some practical issues of gradient descent on non-convex optimization problems, there have been several enhancements such as learning rate scheduling and adaptive update schemes using momentum and update history (Ruder, 2016).
Optimizing batch size is another way to yield efﬁcient gradient descent.
Note that large batch size has the advantage of enhancing parallelism of the training system in order to speed up training, critical for DNN research (Dean et al., 2012).
Despite such advantages, small batch size is preferred because it improves generalization associated with ﬂat minima search (Keskar et al., 2016) and other hyper-parameter explorations are more convenient (Masters & Luschi, 2018).
Small batch size also affects weight regularization if weight updates for gradient descent and weight regularization are supposed to happen for every mini-batch.
For example, for weight decay conducted for every minibatch, if batch size is modiﬁed, then the weight decay factor should also be adjusted accordingly (Loshchilov & Hutter, 2017).
Weight regularization is a process that adds information to the model as a way to avoid overﬁtting (Goodfellow et al., 2016; van Laarhoven, 2017).
In this paper, we explore weight compression as a form of weight regularization as it severely restricts the search space of weights (i.e., regularized by compression forms).
Moreover, model compression shrinks the effective model size, which is an important regularization principle (Goodfellow et al., 2016)
(note that improved model accuracy by model compression is reported (Frankle et al., 2019)).
Weights are regularized in numerous ways by model compression.
For example, each weight can be pruned (e.g., Han et al.
or quantized (e.g., Guo et al.
(2017)) to yield a sparse model representation or to reduce the number of bits to represent each weight.
While weight regularization for model compression can simply be performed once after training, compression-aware training can improve model accuracy by reﬂecting [Figure] the impact of model compression on the loss function for every mini-batch update (Courbariaux et al., 2015; Zhu & Gupta, 2017; Zhu et al., 2017; Guo et al., 2017).
Now, our question in this paper is the following: Should weight regularization follow batch size that is considered for gradient descent?
If the answer is “No,” then we would need to consider a new hyper-parameter search space with the right weight regularization timing for existing weight regularization methods and model compression schemes.
In this paper, we report the following observations: 1) Model compression tends to induce weight decay and random weight noise; 2) For weight decay and weight noise insertion, less frequent weight regularization allows stronger regularization to best utilize regularization effects regardless of batch size; and 3) Similarly, if weight regularization for compression is performed less frequently, training for model compression permits stronger weight regularization with improved model accuracy, and thus, achieves higher compression ratios.
We conﬁrm our above observations through various experiments and propose an occasional regularization method to decouple weight regularization from batch size selection that is optimized for gradient descent.
We verify that our simple model compression techniques (without modifying the underlying training procedures) based on occasional weight regularization (with longer nonregularization period) can achieve higher compression ratio and higher model accuracy compared to previous techniques that demand substantial modiﬁcations to the training process.
N OISE M ODEL ON W EIGHT C OMPRESSION
We ﬁrst study the relationship between model compression ratio and weight regularization strength using quantization and singular-value decomposition (SVD) as model compression techniques.
We assume a popular quantization method based on binary codes for which a weight vector w is approximated to be (cid:80) qi
=
α i b i for q -bit quantization, where α is a scaling factor and b (= {− 1 , +1 } n ) is a binary vector, and n is the vector size.
The quantization error || w − (cid:80) i
α i b
i
|| 2 is minimized by a method proposed by Xu et al.
(2018) to compute α and b .
For SVD, a weight matrix W ∈ R m × n is approximated to be W (cid:
) ∈
R m × n by minimizing || W − W (cid:48)
|| subject to rank ( W (cid:48) )
≤ R , where
R is the target rank.
For our experiments, we use a synthetic (2048 × 2048 ) weight matrix where each element is randomly generated from the Gaussian distribution N ( µ = 0 , σ 2 = 1) .
Then, we are interested in the amount of change of each weight after quantization and SVD.
Assuming that weight noise through compression is expressed as (cid:15) in the form of w (cid:48)
= w (1 + (cid:15) ) , Figure 1 shows the distribution of (cid:15) with various quantization bits or target ranks.
From (cid:15) distributions skewed to be negative, it is clear that weights tend to decay more with higher compression ratio, along with a wider range of random noise.
Reasonable explanations of Figure 1 would include: 1) weights generated from the
Gaussian distribution are uncorrelated such that an approximation step (by compression) using multiple weights would result in noise for each weight, 2) in the case of SVD, elements associated with small eigenvalues are eliminated, 3) averaging effects in quantization reduce the magnitude of large weights.
For weight pruning, (cid:15) becomes − 1 or 0 (i.e., weight decay for selected weights).
Correspondingly, we study on weight decay and weight noise insertion in the next two sections as an effort to gain a part of basic knowledge on improved training for model compression, even though actual model compression would demand much more complicated weight noise models.
[Figure]
N ON -R EGULARIZATION P ERIOD
Since weight regularization cannot precede updates for gradient descent, in order to control the frequency of weight regularization, an available option is to skip a few batches without regularization.
In this paper, we propose a new hyper-parameter, called “Non-Regularization period” or NR period, to enable occasional regularization and to deﬁne the interval of two consecutive regularization events as shown in Figure 2.
NR period is expressed as a multiple of batches.
Strong weight regularization facilitates the chance of escaping a local minimum (depicted as step 2 in Figure 2) or require longer NR period to return to a local minimum (described as step 5 in
Figure 2).
Let us estimate NR period ( pNR ) for a returning case.
Given a parameter set w (that is assumed to be close enough to a local minimum) and a learning rate γ , the loss function of a model
L ( w ) can be approximated as
[Equation]
(1) using a local quadratic approximation where H is the Hessian of L and w 0 is a set of parameters at a local minimum.
After regularization is performed at step t , w can be updated by gradient descent as follows:
[Equation]
Thus, after pNR , we obtain w t + pNR = w
+ ( I − γH ( w 0 )) pNR ( w t − w 0 ) , where I is an identity matrix.
Suppose that H is positive semi-deﬁnite and all elements of I − γH ( w 0 ) are less than 1.0, w t + pNR can converge to w 0 with long pNR which should be longer with larger ( w t − w 0 ) (i.e., stronger weight regularization) or smaller γH ( w 0 ) .
In the next sections, we focus on the relationship between pNR and the strength of weight regularization.
NR P ERIOD S TUDY ON W EIGHT D ECAY AND W EIGHT N OISE I NSERTION
Weight decay is one of the most well-known regularization techniques (Zhang et al., 2018) and different from L 2 regularization in a sense that weight decay is separated from the loss function calculation (Loshchilov & Hutter, 2017).
Weight decay is performed as
[Equation]
where θ is a constant weight decay factor.
Weight noise insertion is another regularization technique aiminig at reaching ﬂat minima (Goodfellow et al., 2016; Hochreiter & Schmidhuber, 1995).
Suppose that random Gaussian noise is added to weights such that w (cid:48)
=
w + (cid:15) when (cid:15) ∼ N (0 , ηI ) .
Then, L ( w (cid:48) )
=
E [ f w + (cid:15) ( x ) − y ] 2 where x , y , f are input, target, and prediction function, respectively.
Using Taylor-series expansion to second-order terms, we obtain f w + (cid:15) ( x ) ≈ f w ( x ) + (cid:15) (cid:62) ∇ f ( x ) + (cid:15) (cid:62) ∇ 2 f ( x ) (cid:15) / 2 .
Correspondingly, the loss function can also be approximated as
[Equation]
where the second term disappears near a local minimum and the third term induces ﬂat minima.
Random noise insertion with other distribution models can be explained in a similar fashion (Goodfellow et al., 2016).
[Figure] [Figure]
We study the impact of NR period on weight decay and weight noise insertion using ResNet-32 on
CIFAR-10 model (He et al., 2016) and a long short-term memory (LSTM) model on PTB dataset (Zaremba et al., 2014).
For LSTM model, we use 2 layers with 200 hidden units and the hyperparameter set introduced by Zaremba et al.
For weight noise model, we plug θ ∼ U ( − a, + a ) (uniform distribution) into Eq.
(3) to simplify the experiments.
Figure 3 shows model accuracy of
ResNet-32 given different NR period and weight decay factors.
For both weight decay and weight noise insertion, the choice of θ (representing the amount of each weight regularization) has a clear correlation with NR period (refer to Appendix for training and test accuracy graphs).
If we wish to apply stronger weight regularization, then such weight regularization should be conducted less frequently (i.e., larger weight decay factor requires longer NR period) to maximize the regularization effect and achieve high model accuracy.
Similar observations are discovered by LSTM model on PTB as shown in Figure 4.
Lower perplexity (indicating better generalization) is obtained when NR period increases as weight regularization becomes stronger for each regularization event.
Note that compared with a conventional weight decay factor selection (i.e., θ in Eq.
(3) when pNR = 1 ), weight decay factor can be approximately 1,000 times larger with pNR ≈ 1 , 000 in Figure 3 and
Figure 4.
If e w is the mean absolute difference given as ( = E [ | w − w (cid:48) | ] ), where w (cid:48) is a weight vector after regularization, then Figure 3 and Figure 4 imply that optimal e w /pNR seems to be constant.
Consequently, optimal e w depends on pNR , and hence, we argue that a wide exploration of various NR period (for occasional regularization) and different weight regularization strength is necessary in order to best utilize regularization effects on a DNN model while previous attempts employ pNR = 1 only.
In addition, the observation that stronger weight regularization is enabled by longer pNR is our basic training principle for model compression.
[Figure]
NR P ERIOD FOR M ODEL C OMPRESSION
As discussed, weight compression incurs a much more complicated weight regularization model than weight decay or uniform weight noise insertion because 1) as shown in Figure 1, diversiﬁed noise models need to be combined to describe weight regularization after model compression and 2) compression-aware training methods would reduce the strength of weight regularization as training is performed with more epochs and weights converge to a compressed form.
Nonetheless, we can conjecture that the best training scheme for model compression may require the condition of pNR (cid:54) = 1 that can be empirically justiﬁed.
We apply weight quantization, low-rank approximation (SVD), and pruning to an LSTM model on
PTB that we selected for the previous section.
We do not modify underlying training principles and use the following simple strategy:
1) Train the model during NR period (as if model compression is not being considered.)
Then, perform weight compression in the form of w (cid:48)
=
h ( w ) .
With new full-precision weight w (cid:48) , repeat the above two steps.
h ( w ) can be a magnitude-based pruning (i.e., h ( w ) =
w if | w | is larger than a certain threshold, or h ( w ) = 0 , otherwise), α b for quantization, SVD function, or even as-yet undiscovered functions.
Figure 5 shows model accuracy associated with a number of different sets of pNR and model compression strength (i.e., target rank for low-rank approximation and the number of quantization bits).
Optimal pNR for the best model accuracy is deﬁnitely much larger than 1 . To explain how Figureis aligned with Section 4, we investigate the relationship between model accuracy and the average of e w /pNR ( e w = E [ | w − w (cid:48) | ] , where w (cid:48) is a weight vector after weight decay, SVD, or pruning) during entire training. We ﬁrst determine optimal e w /pNR (= e opt ) for the best model accuracy. As discussed in Section 4, e opt is assumed to be constant regardless of compression ratio or decay
[Figure] ) using PTB LSTM model when weights are regularized by weight decay, SVD, or pruning factor, and obtained by ﬁnding hyper-parameter sets associated with maximum model accuracy in Figure 4 and Figure 5 and by taking the average of corresponding e w values.
When error is deﬁned to be | e w /pNR − e opt | , Figure 6 shows test perplexity and error of PTB LSTM model with different pNR .
Such deﬁned error is affected by pNR as shown in Figure 6, and indeed, when error approaches to an optimal value we gain improved model accuracy.
Unlike weight decay where e w is directly computed by decay factors, for model compression techniques, e w is not directly related to compression-related hyper-parameters (such as ranks and pruning rates).
As a result, while Figure 4 shows clear correlation between decay factors and pNR for best model accuracy, Figuresuggests that compression ratio and pNR are weakly correlated.
Hence, pNR is a hyper-parameter to be determined empirically for model compression.
Nonetheless, optimal pNR is deﬁnitely much larger than 1, as shown in Figure 6, and decoupled from batch size selection.
That means weight regularization for model compression needs to be conducted much less frequently compared with gradient descent since batch size selection considers generalization ability of gradient descent, not regularization effects.
Note that periodic compression has been introduced in the literature to gradually improve compression ratio or automate hyper-parameter search process.
DropPruning repeats dropping weights randomly and retraining the model while some previously dropped weights are unpruned until pruning rate reaches a target number (Jia et al., 2018).
Weights are incrementally quantized to improve model accuracy (Zhou et al., 2017) or the number of quantization bits can be controlled differently for each layer by a loop based on reinforcement learning (Elthakeb et al., 2018).
Structured pruning and ﬁne-tuning process can be iterated to increase pruning rate (Molchanov et al., 2016; Liu et al., 2017).
All of these previous works assume pNR = 1 (i.e., performing compression for every mini batch) while the goal is increasing compression ratio slowly or ﬁnding a set of hyper-parameters through iterative ﬁne-tuning stages.
Our proposed compression technique can be combined with such periodic compression methods (incremental compression or automatic hyper-parameter selection are also applicable to our proposed method).
In the work by He et al.
(2018), soft ﬁlter pruning is conducted with pNR = 1 epoch without analysis of why such occasional pruning improves model accuracy.
C OMPARISON WITH P REVIOUS M ODEL C OMPRESSION T ECHNIQUES
In this section, we compare some of previous model compression techniques with our compression scheme that introduces pNR and obviates special training algorithm modiﬁcations.
6. F INE -G RAINED W EIGHT P RUNING
The initial attempt of pruning weights was to locate redundant weights by computing the Hessian to calculate the sensitivity of weights to the loss function (LeCun et al., 1990).
However, such [Table] a technique has not been considered to be practical due to signiﬁcant computation overhead for computing the Hessian.
Magnitude-based pruning (Han et al., 2015) has become popular because one can quickly ﬁnd redundant weights by simply measuring the magnitude of weights.
Since then, numerous researchers have realized higher compression ratio largely by introducing Bayesian inference modeling of weights accompanying supplementary hyper-parameters.
For example, dynamic network surgery (DNS) (Guo et al., 2016) permits weight splicing when a separately stored full-precision weight becomes larger than a certain threshold.
Optimizing splicing threshold values, however, necessitates extensive search space exploration, and thus, longer training time.
Variational dropout method (Molchanov et al., 2017) introduces an explicit Bayesian inference model for a prior distribution of weights, which also induces various hyper-parameters and increased computational complexity.
We perform magnitude-based pruning at every pNR step.
As a result, even though weights are pruned and replaced with zero at pNR steps, pruned weights are still updated in full precision during
NR period.
If the amount of updates of a pruned weight grows large enough between two consecutive regularization steps, then the weight pruned at last pNR step may not be pruned at the next pNR step.
Such a feature (i.e., pruning decisions are not ﬁxed) is also utilized for weight splicing in DNS (Guo et al., 2016).
Weight splicing in DNS relies on a hysteresis function (demanding sophisticated ﬁnetuning process with associated hyper-parameters) to switch pruning decisions.
Pruning decisions through our scheme, on the other hand, are newly determined at every pNR step.
We present experimental results with LeNet-5 and LeNet-300-100 models on MNIST dataset which are also reported by Guo et al.
; Molchanov et al.
LeNet-5 consists of 2 convolutional layers and 2 fully connected layers while 3 fully connected layers construct LeNet-300-100.
We train both models for 20000 steps using Adam optimizer where batch size is 50.
All the layers are pruned at the same time and the pruning rate increases gradually following the equation introduced in Zhu & Gupta (2017):
[Equation]
where E is a constant, p f is the target pruning rate, p i is the initial pruning rate, t is the current step, and the pruning starts at training step t i and reaches p f at training step t f .
After t f steps, pruning rate is maintained to be p f .
For LeNet-5 and LeNet-300-100, t i , p i , E are 8000 (step), 25(%), and 7, respectively.
t f is 12000 (step) for LeNet-5 and 13000 (step) for LeNet-300-100.
Note that these choices are not highly sensitive to test accuracy as discussed in Zhu & Gupta (2017).
We exclude dropout to improve the accuracy of LeNet-300-100 and LeNet-5 since pruning already works as a regularizer (Han et al., 2015; Wan et al., 2013).
We keep the original learning schedule and the total number of training steps (no additional training time for model compression).
Table 1 presents the comparison on pruning rates (see Appendix for test accuracy).
Despite the simplicity, our pruning scheme produces higher pruning rate compared with DNS and similar com- pared with variational dropout technique which involves much higher computational complexity.
For Table 1, we use pNR =10 for LeNet-5 and pNR =5 for LeNet-300-100.
L OW -R ANK A PPROXIMATION
We apply our proposed occasional regularization algorithm integrated with Tucker decomposition (Tucker, 1966) to convolutional neural network (CNN) models and demonstrate superiority of pNRbased scheme over conventional training methods.
In CNNs, the convolution operation requires a 4D kernel tensor K
= R d × d × S × T where each kernel has d × d dimension, S is the input feature map size, and T is the output feature map size.
Then, following the Tucker decomposition algorithm, K is decomposed into three components as
[Equation]
where C i,j,r s ,r t is the reduced kernel tensor, R s is the rank for input feature map dimension, R t is the rank for output feature map dimension, and P S and P T are 2D ﬁlter matrices to map C i,j,r s ,r t to K i,j,s,t .
Each component is obtained to minimize the Frobenius norm of ( ˜ K i,j,s,t − K i,j,s,t ).
As a result, one convolution layer is divided into three convolution layers, speciﬁcally, (1 × 1) convolution for P S , ( d × d ) convolution for C i,j,r s ,r t , and (1 × 1) convolution for P T (Kim et al., 2016).
In prior tensor decomposition schemes, model training is performed as a ﬁne-tuning procedure after the model is restructured and ﬁxed (Lebedev et al., 2015; Kim et al., 2016).
On the other hand, our training algorithm is conducted for Tucker decomposition as follows:
Step 1: Perform normal training for pNR (batches) without considering Tucker decomposition Step 2: Calculate C , P S , and P T using Tucker decomposition to obtain ˜ K Step 3: Replace K with ˜ K Step 4: Go to Step 1 with updated K
After repeating a number of the above steps towards convergence, the entire training process should stop at Step 2, and then the ﬁnal decomposed structure is extracted for inference.
Because the model is not restructured except in the last step, Steps 2 and 3 can be regarded as special steps to encourage wide search space exploration so as to ﬁnd a compression-friendly local minimum where weight noise by decomposition does not noticeably degrade the loss function.
Using the pre-trained ResNet-32 model with CIFAR-10 dataset (He et al., 2016; Kossaiﬁ et al., 2019), we compare two training methods for Tucker decomposition: 1) typical training with a decomposed model and 2) pNR -based training, which maintains the original model structure and occasionally injects weight noise through decomposition.
Using an SGD optimizer, both training methods follow the same learning schedule: learning rate is 0.1 for the ﬁrst 100 epochs, 0.01 for the next 50 epochs, and 0.001 for the last 50 epochs.
Except for the ﬁrst layer, which is much smaller than the other layers, all convolution layers are compressed by Tucker decomposition with rank R s and R t selected to be S and T multiplied by a constant number R c ( 0 .
3 ≤ R c ≤ 0 .
7 in this experiment).
Then, the compression ratio of a convolution layer is d 2 ST/ ( SR s + d 2 R s R t + TR t ) =
d 2 ST/ ( S 2 R c + d 2 R 2 c ST + T 2 R c ) , which can be approximated to be 1 /R 2 c if S = T and d (cid:29) R c .
pNR is chosen to be 200.
Figure 7 shows test accuracy after Tucker decomposition 1 by two different training methods.
Note that test accuracy results are evaluated only at Step 3 where the training process can stop to generate a decomposed structure.
In Figure 7, across a wide range of compression ratios (determined by
R c ), the proposed scheme yields higher model accuracy compared to typical training.
Note that even higher model accuracy than that of the pre-trained model can be achieved by our method if the compression ratio is small enough.
In fact, Figure 8 shows that our technique improves training loss and test accuracy throughout the entire training process.
Initially, the gap of training loss and test accuracy between pre-regularization and post-regularization is large.
Such a gap, however, is quickly reduced through training epochs.
Overall, ResNet-32 converges successfully through the [Figure] [Figure]
For ResNet-34 on ImageNet experiments and VGG19 on CIFAR-10 (including additional compression techniques), refer to Appendix [Figure]
C ONCLUSION
In this paper, we show that weight regularization should be decoupled from batch size to enhance regularization effects.
To accomplish such decoupling, we introduce a new hyper-parameter called non-regularization period or NR period during which weights are updated only for gradient computations.
A larger amount of weight decay and weight noise insertion need to be supported by longer
NR period to maximize the regularization effect.
We ﬁnd that model compression with higher compression ratio is also better trained by longer NR period.
We demonstrate that various DNN models can be compressed by pruning, quantization and low-rank approximation successfully with longer
NR period while the underlying training algorithm do not need to be modiﬁed
A A PPENDIX
[Table]
[Figure]
[Table] [Table] [Table]
W
[Table]
g h t
D e c a y
F a c t o r ( θ )
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table] [Table] [Table] [Figure]
[Table] [Table]
[Table]
[Table] [Table]
[Figure] [Figure]
A.
P ARAMETER P RUNING
DC (Deep Compression) and Sparse VD represent a magnitude-based et
[Table]
We apply pNR -based
pruning to an RNN model to verify the effectiveness of pNR .
We choose an
LSTM model (Zaremba et al., 2014) on the PTB dataset (Marcus et al., 1993).
Following the model structure given in Zaremba et al.
(2014), our model consists of an embedding layer, 2 LSTM layers, and a softmax layer.
The number of LSTM units in a layer can be 200, 650, or 1500, depending on the model conﬁgurations (referred as small, medium, and large model, respectively).
The accuracy is measured by Perplexity Per Word (PPW), denoted simply by perplexity in this paper.
We apply gradual pruning with E = 3 , t i = 0 , p i = 0 , t f = 3 rd epoch (for medium) or 5 th epoch (for large) to the pre-trained PTB models.
pNR -based pruning for the PTB models is performed using pNR = 100 and the initial learning rate is 2.0 for the medium model (1.0 for pre-training) and 1.
for the large model (1.0 for pre-training) while the learning policy remains to be the same as in
Zaremba et al.
[Table]
For all of the pruning rates selected, Table 4 shows that our compression scheme improves perplexity better than the technique in Zhu & Gupta (2017) which is based on Han et al. (2015).
The superiority of pNR -based pruning is partly supported by the observation that non-zero weights successfully avoid to be small through retraining while the conventional pruning still keeps near-zero (unmasked) weights as depicted in Figure 15.
[Table]
[ Q u a n ti za ti on ] N o nR e g u l a r i z a t i o n
{+P e r i o d 1 2 3# of Bits 138 .135 .131 .128 .126 .125 .
 [Table] [Table] [Table]+}
[Table] [Table] [Table]
{+[Table] [Table] [ P r un i ng ]                   
 N o nR+}
{+e g u l a r i z a t i o n P e r i o d 0 .0 .0 .0 .0 . Pruning Rate 117 .117 .117 .116 .116 .115 .116 .115 .115 .114 .115 .114 .114 .114 .114 .114 .113 .114 .114 .
+}
i [Figure] [Figure]
[Figure]
[Figure] [Figure]
A. T UCKER D ECOMPOSITION WITH O CCASIONAL R EGULARIZATION
To investigate the effect of NR period on local minima exploration with ResNet-32 on CIFAR-10,
Figure 16 presents the changes of loss function and weight magnitude values incurred by occasional regularization.
In Figure 16(left), ∆ L / L is given as the loss function increase ∆ L (due to weight regularization at pNR steps) divided by L , which is the loss function value right before weight regularization.
In Figure 16(right), ∆ w is deﬁned as || w − ˜ w || 2 F /N ( w ) , where w is the entire set of weights to be compressed, ˜ w is the set of weights regularized by Tucker decomposition, N ( w ) is the number of elements of w , and
|| X
|| 2 F is the Frobenius norm of X .
Initially, w ﬂuctuates with large corresponding ∆ L .
Then, both ∆ L and ∆ w decrease and Figure 16 shows that occasional regularization ﬁnds ﬂatter local minima (in the view of Tucker decomposition) successfully.
When the learning rate is reduced at 100th and 150th epochs, ∆ L and ∆ w decrease signiﬁcantly because of a lot reduced local minima exploration space.
In other words, occasional regularization helps an optimizer to detect a local minimum where Tucker decomposition does not alter the loss function value noticeably.
[Figure]
A. 2-D IMENSIONAL SVD E NABLED BY O CCASIONAL R EGULARIZATION
In this subsection, we discuss why 2D SVD needs to be investigated for CNNs and how occasional regularization enables a training process for 2D SVD
. A.4. I SSUES OF 2D SVD ON C ONVOLUTION L AYERS
Convolution can be performed by matrix multiplication if an input matrix is transformed into a
Toeplitz matrix with redundancy and a weight kernel is reshaped into a T × ( S × d × d ) matrix (i.e., a lowered matrix)
Chetlur et al.
Then, commodity computing systems (such as CPUs and GPUs) can use libraries such as Basic Linear Algebra Subroutines (BLAS) without dedicated hardware resources for convolution Cho & Brand (2017).
Some recently developed DNN accelerators, such as Google’s Tensor Processing Unit (TPU) Jouppi et al.
(2017), are also focused on matrix multiplication acceleration (usually with reduced precision).
For BLAS-based CNN inference, reshaping a 4D tensor K and performing SVD is preferred for low-rank approximation rather than relatively inefﬁcient Tucker decomposition followed by a lowering technique.
However, a critical problem with SVD (with a lowered matrix) for convolution layers is that two decomposed matrices by SVD do not present corresponding (decomposed) convolution layers, because of intermediate lowering steps.
As a result, ﬁne-tuning methods requiring a structurally modiﬁed model for training are not available for convolution layers to be compressed by SVD.
On the other hand, occasional regularization does not alter the model structure for training.
For occasional regularization, SVD can be performed as a way to feed noise into a weight kernel K for every regularization step.
Once training stops at a regularization step, the ﬁnal weight values can be decomposed by SVD and used for inference with reduced memory footprint and computations.
In other words, occasional regularization enables SVD-aware training for CNNs.
[Figure]
A.4. T ILING -B ASED SVD FOR S KEWED W EIGHT M ATRICES
A reshaped kernel matrix K ∈ R T × ( S × d × d ) is usually a skewed matrix where row-wise dimension ( n ) is smaller than column-wise dimension ( m ) as shown in Figure 17 (i.e., n (cid:28) m ).
A range of available rank r for SVD, then, is constrained by small n and the compression ratio is approximated to be n/r .
If such a skewed matrix is divided into four tiles as shown in Figure 17 and the four tiles do not share much common chateracteristics, then tiling-based SVD can be a better approximator and rank r can be further reduced without increasing approximation error.
Moreover, fast matrix multiplication is usually implemented by a tiling technique in hardware to improve the weight reuse rate Fatahalian et al.
Hence, tiling could be a natural choice not only for high-quality SVD but also for high-performance hardware operations.
To investigate the impact of tiling on weight distributions after SVD, we tested a (1024 × 1024) random weight matrix in which elements follow a Gaussian distribution.
A weight matrix is divided by (1 × 1) , (16 × 16) , or (128 × 128) tiles (then, each tile is a submatrix of (1024 × 1024) , (64 × 64) , or (8 × 8) size).
Each tile is compressed by SVD to achieve the same overall compression ratio of 4 × for all of the three cases.
As described in Figure 17 (on the right side), increasing the number of tiles tends to increase the count of near-zero and large weights (i.e., variance of weight values increases).
Figure 17 can be explained by sampling theory where decreasing the number of random samples (of small tile size) increases the variance of sample mean.
In short, tiling affects the variance of weights after SVD (while the impact of such variance on model accuracy should be empirically studied).
[Table]
We applied the tiling technique and SVD to the 9 largest convolution layers of ResNet-32 using the
CIFAR-10 dataset.
Weights of selected layers are reshaped into 64 × (64 × 3 × 3) matrices with the tiling conﬁgurations described in Table 5.
We perform training with the same learning schedule and pNR (=200) used in Section 3.
Compared to the test accuracy of the pre-trained model (=92.63%), all of the compressed models in Table 5 achieves higher model accuracy due to the regularization effect of our compression scheme.
Note that for each target compression ratio, the relationship between tile size and model accuracy is not clear.
Hence, various conﬁgurations of tile size need to be explored to enhance model accuracy, even though variation of model accuracy for different tile size is small.
Compression Ratio
A cc u r a c y ( % )
[Table]
[Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table]
[Table] [Table]
Typical Scheme
[Table]
Tucker Decomp.
[Table]
Tiled
[Table]
with
[Table]
Tiled
[Table]
Tiled
[Table]
[Table]
A. E XPERIMENTAL R ESULTS ON L OW -R ANK A PPROXIMATION FOR CNN S
In this subsection, we apply low-rank approximation trained by occasional regularization to various CNN models.
Figure 18 summarizes the test accuracy values of ResNet-32 (with CIFAR-10 dataset) compressed by various low-rank approximation techniques.
Note that tiled SVD and normal SVD are enabled only by occasional regularization, which obviates model structure modiﬁcation during training.
All conﬁgurations in Figure 18 use the same learning rate scheduling and the number of training epochs as described in Section 3.
Results show that tiled SVD yields the best test accuracy and test accuracy is not highly sensitive to tile conﬁguration.
SVD presents competitive model accuracy for small compression ratios.
As compression ratio increases, however, model accuracy using SVD signiﬁcantly degrades.
From Figure 18, tiled SVD associated with occasional regularization is clearly the best low-rank approximation scheme.
[Table]
=10) 5.93M (3.20 × ) 193.85M (3.34 × ) 92.
64 × 64 ( r =9) 5.55M (3.41 × ) 173.44M (3.73 × ) 92.
64 × 64 ( r =8) 4.74M (4.00 × ) 153.04M (4.33 × ) 92.
We compare Tucker decomposition trained by a typical ﬁne-tuning process and tiled SVD trained by occasional regularization using the VGG19 model 2 with CIFAR-10.
Since this work mainly discusses compression on convolution layers, fully-connected layers of VGG19 are compressed and ﬁxed before compression of convolution layers (refer to Appendix for details on the structure of
VGG19).
Except for small layers with S < 128 (that presents small compression ratio as well), all convolution layers are compressed with the same compression ratio.
During 300 epochs to train convolution layers, learning rate is initially 0.01 and is then halved every 50 epochs.
In the case of tiled SVD, pNR is 300 and tile size is ﬁxed to be 64 × 64 (recall that the choice of pNR and tile size do not affect model accuracy signiﬁcantly).
As described in Table 6, while Tucker decomposition with conventional ﬁne-tuning shows degraded model accuracy through various R c , occasional-regularization-assisted tiled SVD presents noticeably higher model accuracy.
[Figure]
We also test our proposed low-rank approximation training technique with the ResNet-34 model
He et al.
(2016) using the ImageNet dataset Russakovsky et al. (2015).
A pre-trained ResNet-34 is ﬁne-tuned for Tucker decomposition (with conventional training) or tiled SVD (with occasional regularization) using the learning rate of 0.01 for the ﬁrst 20 epochs, 0.001 for the next 30 epochs, and 0.0001 for the remaining 30 epochs.
Similar to our previous experiments, the same compression ratio is applied to all layers except the layers with S <
(such exceptional layers consist of 1.4% of the entire model).
In the case of Tucker decomposition, selected convolution layers are compressed with R c = 0 .
46 to achieve an overall compression of 3 . 1 × .
For tiled SVD, lowered matrices are tiled and each tile of (64 × 64) size is decomposed with r =10 to match an overall compression of 3 . 1 × .
As shown in Figure 19, occasional-regularization-based tiled SVD yields better training loss and test accuracy compared to Tucker decomposition with typical training.
At the end of the training epoch in Figure 19, tiled SVD and Tucker decomposition achieves 73.00% and 72.31% for toptest accuracy, and 91.12% and 90.73% for top-5 test accuracy, while the pre-trained model shows 73.26% (top-1) and 91.24% (top-5).
A. L OWERING T ECHNIQUE FOR CNN S
Figure 20 describes a kernel matrix reshaped from a 4D kernel tensor and an input feature map matrix in the form of a Toeplitz matrix.
At the cost of redundant memory usage to create a Toeplitz matrix, lowering enables matrix multiplication which can be efﬁciently implemented by BLAS libraries.
A kernal matrix can be decomposed by 2D SVD. [Figure] [Table] W
[Figure]
S x d x d (H+1-d) x (W+1-d)
Lowering(ex.im2col) *when stride is 1 with no padding
LRA via SVD (rank= k )
T
S x d x d k k
Kernel x Input Feature Map [-1 2 3 6 7 4 515212223 19 24 
-][Figure] [Figure]
S
A. M ODEL D ESCRIPTIONS FOR L OW -R ANK A PPROXIMATION E XPERIMENTS
In this section, we describe model structures and layers selected for low-rank approximation experiments.
Small layers close to the input are not compressed because both weight size and compression rate are too small.
[Table] [Table]
[Table] [Table]
No    0.21M ( 1.05%)
No    0.07M ( 0.35%)
No    0.98M ( 4.90%)
Yes    0.28M ( 1.40%)
Yes    6.18M (30.77%)
Yes    1.13M ( 5.59%)
Yes    11.25M (55.94%)
Yes
Total 20.11M (100.0%)