C HIP N ET : B UDGET -A WARE P RUNING WITH H EAVISIDE C ONTINUOUS A PPROXIMATIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
Structured pruning methods are among the effective strategies for extracting small resource-efﬁcient convolutional neural networks from their dense counterparts with minimal loss in accuracy
However, most existing methods still suffer from one or more limitations, that include 1) the need for training the dense model from scratch with pruning-related parameters embedded in the architecture, 2) requiring model-speciﬁc hyperparameter settings, 3) inability to include budget-related constraint in the training process and 4) instability under scenarios of extreme pruning.
In this paper, we present ChipNet , a deterministic pruning strategy that employs continuous Heaviside function and a novel crispness loss to identify a highly sparse network out of an existing dense network.
Our choice of continuous Heaviside function is inspired by the ﬁeld of design optimization, where the material distribution task is posed as a continuous optimization problem but only discrete values (0 or 1) are practically feasible and expected as ﬁnal outcomes.
Our approach’s ﬂexible design facilitates its use with different choices of budget constraints while maintaining stability for very low target budgets.
Experimental results show that ChipNet outperforms state-of-the-art structured pruning methods by remarkable margins of up to 16.1% in terms of accuracy.
Further, we show that the masks obtained with ChipNet are transferable across datasets.
For certain cases, it was observed that masks transferred from a model trained on featurerich teacher dataset provide better performance on the student dataset than those obtained by directly pruning on the student data itself
I NTRODUCTION
Convolution Neural Networks (CNNs) have resulted in several breakthroughs across various disciplines of deep learning, especially for their effectiveness in extracting complex features.
However, these models demand signiﬁcantly high computational power, making it hard to use them on lowmemory hardware platforms that require high-inference speed.
Moreover, most of the existing deep networks are heavily over-parameterized resulting in high memory footprint (Denil et al., 2013;
Frankle & Carbin, 2018).
Several strategies have been proposed to tackle this issue, that include network pruning (Liu et al., 2018), neural architecture search using methods such as reinforcement learning (Jaafra et al., 2019) and vector quantization (Gong et al., 2014), among others.
Among the methods outlined above, network pruning has proved to be very effective in designing small resource-efﬁcient architectures that perform at par with their dense counterparts.
Network pruning refers to removal of unnecessary weights or ﬁlters from a given architecture without compromising its accuracy.
It can broadly be classiﬁed into two categories: unstructured pruning and structured pruning .
Unstructured pruning involves removal of neurons or the corresponding connection weights from the network to make it sparse.
While this strategy reduces the number of parameters in the model, computational requirements are still the same (Li et al., 2017).
Structured pruning methods on the other hand remove the entire channels from the network.
This strategy preserves the regular structure, thereby taking advantage of the high degree of parallelism provided by modern hardware (Liu et al., 2017; Gordon et al., 2018).
Several structured pruning approaches have been proposed in the recent literature.
A general consensus is that variational approaches using sparsity prior loss and learnable dropout parameters out- perform the deterministic methods (Lemaire et al., 2019).
Some of these methods learn sparsity as a part of pretraining, and have proved to perform better than the three stage pretrain-prune-ﬁnetune methods.
However, since such approaches need to train the model from scratch with pruning-related variables embedded into the network, they cannot beneﬁt from off-the-shelf pretrained weights (Liu et al., 2017; Alvarez & Salzmann, 2017).
Others require choosing hyperparameters based on the choice of the network, and cannot be easily adapted for new models (Gordon et al., 2018).
Further, with most of these methods, controlled pruning cannot be performed, and a resource-usage constraint can only be satisﬁed through trial-and-error approach.
Recently, Lemaire et al.
(2019) presented a budget-aware pruning method that includes the budget constraint as a part of the training process.
A major drawback of this approach and other recent methods is that they are unstable for very low resource budgets, and require additional tricks to work.
Overall, a robust budget-aware pruning approach that can be coupled with different budget constraints as well as maintains stability for very low target budgets, is still missing in the existing literature.
In this paper, we present ChipNet , a deterministic strategy for structured pruning that employs continuous Heaviside function and crispness loss to identify a highly sparse network out of an existing pretrained dense network.
The abbreviation ‘ChipNet’ stands for Continuous Heaviside Pruning of
Networks.
Our pruning strategy draws inspiration from the ﬁeld of design optimization, where the material distribution task is posed as a continuous optimization problem, but only discrete values (0 or 1) are practically feasible.
Thus, only such values are produced as ﬁnal outcomes through continuous Heaviside projections.
We use a similar strategy to obtain the masks in our sparsity learning approach.
The ﬂexible design of ChipNet facilitates its use with different choices of budget constraints, such as restriction on the maximum number of parameters, FLOPs, channels or the volume of activations in the network.
Through experiments, we show that ChipNet consistently outperforms state-of-the-art pruning methods for different choices of budget constraints.
ChipNet is stable for even very low resource budgets, and we demonstrate this through experiments where network is pruned to as low as 1% of its parameters.
We show that for such extreme cases,
ChipNet outperforms the respective baselines by remarkable margins, with a difference in accuracy of slightly beyond 16% observed for one of the experiments.
The masks learnt by ChipNet are transferable across datasets.
We show that for certain cases, masks transferred from a model trained on feature-rich teacher dataset provide better performance on the student dataset than those obtained by directly pruning on the student data itself.
R ELATED WORK
As has been stated in the hypothesis by Frankle & Carbin (2018), most neural networks are overparameterized with a large portion (as much as 90%) of the weights being of little signiﬁcance to the output of the model.
Clearly, there exists enormous scope to reduce the size of these networks.
Several works have explored the efﬁciency of network pruning strategies for reducing storage requirements of these networks and accelerating inference speed (LeCun et al., 1990; Dong et al., 2017).
Some early works by Han et al.
(2015a;c); Zhu & Gupta (2017)
involve removal of individual neurons from a network to make it sparse.
This reduces the storage requirements of these networks, however, no improvement in inference speed is observed.
Recently, several works have focused on structured network pruning, as it involves pruning the entire channel/ﬁlters or even layers to maintain the regular structure (Luo et al., 2017; Li et al., 2017; Alvarez & Salzmann, 2016).
The focus of this paper is on structured network pruning, thus, we brieﬂy discuss here the recent works related to this approach.
The recent work by Li et al.
(2017) identiﬁes less important channels based on L1-norm.
Luo et al.
; He et al.
(2017) perform channel selection based on their inﬂuence on the activation values of the next layer.
Liu et al.
(2017) perform channel-level pruning by imposing LASSO regularization on the scaling terms in the batchnorm layers, and prune the model based on a global threshold.
He et al.
(2018b) automatically learn the compression ratio of each layer with reinforcement learning.
Louizos et al.
(2017); Alvarez & Salzmann (2017; 2016) train and prune the network in a single stage strategy.
The above mentioned approaches cannot optimize networks for a pre-deﬁned budget constraint.
Adding budget constraint to the pruning process can provide a direct control on the size of the pruned network.
For example, Morphnet imposes this budget by iteratively shrinking and expanding a network through a sparsifying regularizer and uniform layer wise width multiplier, respectively,
− 6 − 4 − 2  60 .0 .0 .0 .
ψ ˜ z (a) β 0.10.51.
˜
z
z (b) γ − 6 − 4 − 2  60 .0 .0 .0 .
β
=2 . 0 ,γ =4 .ψ
p r
o j ec ti on s (c) ˜ zz − 6 − 4 − 2  60 .0 .0 .0 .
ψ
L c (d) β =1 . 0 ,γ =4 β =1 . 1 ,γ =16 β =1 . 1 ,γ =64 β
=2 . 0 ,γ = β
=2 . 0 ,γ =128 β =1 . 0 ,
γ = continuous the and is adaptable to speciﬁc resource constraints (Gordon et al., 2018).
However, it requires a modelspeciﬁc hyperparameter grid search for choosing the regularization factor.
Another approach is BAR (Lemaire et al., 2019) that uses a budget-constrained pruning approach based on variational method.
A limitation of this approach is that for low resource budgets, it needs to explicitly ensure that at least one channel is active in the downsample layer to avoid fatal pruning .
The approach proposed in this paper does not require any such tweaking, and is stable for even very low resource budgets.
P ROPOSED A PPROACH 3. L EARNING S PARSITY M ASKS
Sparsity learning forms the core of our approach.
It refers to learning a set of sparsity masks for a dense convolutional neural network ( parent ).
When designing the smaller pruned network ( child ), these masks identify the parts of the parent that are to be included in the child network.
We ﬁrst describe here the general idea of learning these masks in the context of our method.
The proposed approach falls in the category of structured pruning where masks are designed for channels and not individual neurons.
Let f : R d → R k denote a convolutional neural network with weights W ∈ R m and a set of hidden channels H ∈ R p .
We deﬁne z ∈ R p as a set of sparsity masks, where z i ∈ z refers to the mask associated with the feature map h i ∈ H .
To apply the mask, z i is multiplied with all the entries of h i .
The optimization problem can further be stated as
[Equation]
where (cid:12) denotes elementwise multiplication and { x , y } ∈ D are data samples used to train the network f .
The desired sparsity of the network is deﬁned in terms of the equality constraint, where V ( · ) denotes the budget function and V 0 is the maximum permissible budget.
Our proposed formulation of pruning is independent from the choice of the budget function.
We later show this through experiments with volume budget as in Lemaire et al.
(2019), channel budget similar to Liu et al.
(2017), and budget deﬁned in terms of parameters and FLOPs as well.
Originally, z i ∈ z would be deﬁned such that z i ∈ { 0 , 1 } , and a discrete optimization problem is to be solved.
For the sake of using gradient-based methods, we convert it to a continuous optimization problem, such that z i ∈ [0 , 1] .
Such reformulation would lead to intermediate values of z occurring in the ﬁnal optimized solution.
Any intermediate value of z , for example z = 0 .
4 , would imply that a fraction of the respective channel is to be used, and clearly such a solution is practically infeasible.
We propose to overcome this challenge through the use of simple nonlinear projections and a novel loss term, and these are discussed in detail in the next section.
3. C ONTINUOUS H EAVISIDE A PPROXIMATION AND L OGISTIC C URVES
At the backbone of our pruning strategy lies three important functions: the commonly used logistic curves , continuous Heaviside function and crispness loss term.
Figure 1 presents a graphical representation of these functions.
We further provide below a brief motivation for the choice of these functions as well as their signiﬁcance in our pruning approach.
Logistic curves.
A commonly used function for adding nonlinearity to a neural network (LeCun et al., 1998), logistic curve projects an input from the real space to a range of 0-1 (Figure 1a), and can be mathematically stated as
[Equation]
where ψ denotes the optimization parameter corresponding to the mask z , ψ 0 is the midpoint of the curve, and ˜ z denotes the resultant intermediate projection.
The additional parameter β is used to control the growth rate of the curve, and forms an important ingredient of our approach.
While low values of β can produce an approximately linear curve between -1 and 1, higher values turn it into a step function.
During the initial stages of training, we propose to keep β very low, and increase it to higher values at later stages of the optimization process.
With increased values of β , the values further from 0.5 are made more favorable for ˜ z .
In our experience, the logistic curve alone cannot be used to obtain approximately discrete (0-1) solutions for z in a continuous optimization scheme.
The nonlinearity introduced by this function cannot sufﬁciently penalize the intermediate values between 0 and 1, and optimization algorithm can easily identify values of ψ for which the projected values are far from both.
An example experiment demonstrating this issue is presented in Appendix C.2.
To circumvent this issue, we add another nonlinear projection using a continuous approximation of the Heaviside function.
Continuous Heaviside function.
A continuous approximation to the Heaviside step function, referred as continuous Heaviside function in this paper, is a commonly used projection strategy to solve continuous versions of binary optimization (0-1) problems in the domain of design optimization (Guest et al., 2004; 2011).
The generalized form of this function can be stated as:
[Equation]
where, the parameter γ dictates the curvature of the regularization.
Figure 1b shows the continuous
Heaviside function for several values of γ .
We see that z is linear for γ = 0 and approaches the
Heaviside step function for very large values of γ .
The advantages of our projection function are twofold.
First, during projection, the values close to 0 and 1 are not affected irrespective of the choice of γ .
This implies that the masks identiﬁed with most conﬁdence in the early stage of training are not directly impacted by the continuation applied on the value of γ , thus helping in the convergence of the training process.
Here, ‘continuation’ refers to slowly adapting the value of γ during the course of training.
Second, even the values of ˜ z which are slightly greater than 0 are also nonlinearly projected to close to 1, and this effect is more prominent for larger values of γ .
The projection adds higher penalty on values between 0 and 1, and makes them extremely unfavorable when higher values of γ are chosen.
While the continuous Heaviside function helps to obtain approximately discrete masks, there is still no explicit constraint or penalty function that can regulate this.
To overcome this problem, we tie the outputs of logistic and continuous Heaviside functions to deﬁne a novel loss term, referred as crispness loss.
Crispness Loss.
This novel loss term explicitly penalizes the model performance for intermediate values of z , and drives the convergence towards crisp (0-1) masks.
It is deﬁned as the squared Lnorm of the difference between ˜z and z , stated as L c = (cid:107)
˜z − z (cid:107) 22 , and from Figure 1c, we see that
L c achieves its minima when either ˜ z
= z = 0 or ˜ z
= z = 1 .
Further, the trend of this loss function with respect to ψ for different values of β and γ is shown in Figure 1d.
It can be seen that for lower values of β and γ , the loss value is low, and the crispness function plays little to no role in driving the pruning process.
When the value of γ slowly increases, the peak of the graph shifts upwards as well as towards the left, thereby increasing the penalty associated with values of ψ .
This drives the values of ψ to move farther from the origin.
The left shift in the graph adds higher penalty on the negative values, forcing them to become even more negative, thus forcing the respective z to move closer to 0.
The additional loss function associated with the model generally favors values towards 1.
For example, cross-entropy loss used for classiﬁcation would prefer to set all values in z to 1 to be able to maximize the classiﬁcation accuracy.
With increasing values of γ forcing the masks towards 0, a balance between the two is identiﬁed during the training process.
The term β acts as a regularizer that to some extent counteracts the too abrupt impact of γ and regulates the convergence of the training process.
3. I MPOSING B UDGET C ONSTRAINT
The simplicity of our pruning strategy decouples it from the choice of budget constraint.
In this paper, we show the working with four different choices of budget constraints: channel, activation volume, parameters and FLOPs .
These choices are inspired from some of the recent state-of-the-art methods from the existing literature (Liu et al., 2017; Lemaire et al., 2019).
For budget calculation, values of the masks z should be close to 0 or 1.
However, during the initial iterations of training, masks would contain intermediate values as well.
This makes it difﬁcult to accurately calculate the budget for the constraint speciﬁed in Eq.
Thus, rather than computing it directly over the masks z , these are computed on ¯ z , where ¯ z i ∈ ¯
z is obtained by applying a logistic projection on z with ψ 0 = 0 .
5 (Eq. 2).
Further discussion related to it is provided in Appendix C.3.
The budget constraint is imposed using a loss term L b , referred as budget loss .
We deﬁne the budget loss as L b =
( V ( z ) − V 0 ) 2 , where V ( · ) can be one of the 4 budget functions described below.
Channel budget.
It refers to the maximum number of hidden channels h that can be used across all convolutional layers of the network.
Mathematically, it can be stated as V ( c ) =
( (cid:80) pi
=
¯ z i ) /p , where p denotes the number of hidden channels in the network.
Constraint on the channel budget limits the number of channels, and thus the number of weights in the network.
Volume budget.
This budget controls the size of the activations, thereby imposing an upper limit on the memory requirement for the inference step.
We deﬁne volume budget V ( v ) =
( (cid:80) N ( h ) j =1 (cid:80) p
j i =
A j ¯ z i ) /
( (cid:80) N ( h )
j =
A j · p j ) , where N ( h ) denotes the number of convolutional layers, and A j and p j denote area of the feature maps and their count, respectively, in the j th layer.
Parameter budget.
This budget directly controls the total number of parameters in the network, and can thus be used to impose an upper limit on the size of the model.
For details, see Appendix A.1.
FLOPs budget.
This budget can be directly used to control the computational requirement of the model.
Mathematical formulae to calculate it is stated in Appendix A.1.
3. S OFT AND H ARD P RUNING
Input : pretrained network weights W ; budget constraint function V ( · ) ; budget value V 0 ; training data D ; pruning iterations N
Output: learnt sparsity masks z ψ i ∈
Ψ ← random initialization for k = 1 . . .
N do ( x , y ) ← sample ( D ) ˜
z ← L OGISTIC ( ψ ) z ← C ONTINUOUS H EAVISIDE (˜ z )
ˆ y ← Forward ( x , W , z )
V ← V ( z )
L ← C HIP N ET L OSS ( V , V 0 , ˜ z , z , ˆ y , y ) ( ∇ W , ∇ ψ ) ← Backward ( L ) ( W , ψ ) ←
OptimizationStep ( ∇ W , ∇ ψ ) end
The pruning stage in our approach comprises two steps: soft pruning and hard pruning .
After a deep dense network has been pretrained, masks are added to the network and soft pruning is performed.
The steps involved in soft pruning are stated in Algorithm 1.
During this stage, the network is optimized with the joint loss L =
L ce + α 1 L c + α 2 L b , where α 1 and α 2 are the weights for the crispness and budget loss terms, respectively.
After every epoch of soft pruning, the performance of the network is evaluated in a hard pruned manner.
For this purpose, masks z are used, and a cutoff is chosen using binary search such that the budget constraint is exactly satisﬁed.
Values above this cutoff are converted to 1 and the ones lower turned to 0.
Finally, the model with best performance on the validation set is chosen for ﬁne tuning.
E XPERIMENTS 4. E XPERIMENTAL S ETUP
We test the efﬁcacy of our pruning strategy on several network architectures for four different choices of budget constraint functions.
The architectures chosen in this study include WideResNet-26-
VolumePruningFactor
T e s t
A cc u r ac y
CIFARChipNetBARMorphNetLZR
VolumePruningFactor
T e s t
A cc u r ac y ( % )
CIFARChipNetBARMorphNetLZR
VolumePruningFactor
T e s t
A cc u r ac y ( % )
TinyImageNet
ChipNetBARMorphNetLZR
[Table]
(WRN-26-12) (Zagoruyko & Komodakis, 2016), PreResNet-164 (He et al., 2016b), ResNet-50 and
ResNet-101 (He et al., 2016a).
For datasets, we have chosen CIFAR-10/100 (Krizhevsky, 2009) and
Tiny ImageNet (Wu et al.).
For the combined loss L in Eq. 1, weights α 1 and α 2 are set to 10 and 30, respectively, across all experiments.
Implementation details related to the pretraining, pruning and ﬁnetuning steps, as well details of the hardware are described in Appendix B.
4. R ESULTS
Performance of pruned sparse networks.
We present here results obtained using ChipNet for
WRN-26-12 and PreResNet-164 pruned with volume and channel constraints, respectively.
For the two other constraints, parameters and FLOPs budget, we perform a comparative study later in this paper.
Volume budget.
Figure 2 shows the comparison of performance values for WRN-26-12 for
CIFAR-10, CIFAR-100 and TinyImageNet datasets, when pruned using ChipNet.
We compare our results with BAR (Lemaire et al., 2019), MorphNet (Gordon et al., 2018) and LZR (Louizos et al., 2017) for volume pruning factors of 2, 4, 8 and 16.
Details related to the three baselines are presented in Appendix D.1.
ChipNet consistently outperforms all the baselines across all datasets and for all choices of the budget.
For the case of extreme pruning of 16 folds on CIFAR-100, the performance of BAR is close to ours, while the other two baselines signiﬁcantly underperform.
Channel budget We study here the pruning efﬁcacy of ChipNet coupled with channel constraint on
PreResNet-164 architecture for CIFAR-10 and CIFAR-100 datasets.
Results are compared with the
[Table]
[Figure]
network slimming approach (Liu et al., 2017), implementation details related to which can be found in Appendix D.1.
As constraints, we use channel budgets of 60%, 40%, 20% and 10%.
Table 1 presents the results for different choices of channel budgets.
We also report the number of parameters in the pruned network as well as the associated FLOPs.
It is seen that ChipNet outperforms the baseline method for all the experimental settings.
For CIFAR-10 in particular, we see that for even very low channel budget of 10%, accuracy of the pruned network drops by only 3.1%.
For 10% channel budget, our method outperforms the network slimming strategy on CIFAR10 and CIFAR-100 by remarkable margins of 8.5% and 16.1%, respectively.
Note that lower channel usage does not necessarily imply lower number of parameters or reduced
FLOPS in the pruned network, and we analyze this for the various cases of pruning considered in
Table 1.
We see that ChipNet performs selection of channels in a more optimized way, such that better accuracy is achieved with fewer parameters.
In terms of FLOPS, both methods perform at par.
Although, the FLOPS for ChipNet are slightly higher for the channel budget of 10%, this overhead is insigniﬁcant compared to the gain in accuracy and reduction of parameters.
Overall, we infer that
ChipNet couples well with the channel constraint, and is stable for even extreme pruning cases of as low as 10% channel budget.
Effect of the choice of budget.
Here, we analyze the impact of one budget type over another to understand whether the choice of budget really matters when pruning a network.
As a ﬁrst experiment, we study side-by-side the results for channel and volume constraints when used to prune ResNet-50 on
CIFAR-10 and CIFAR-100 datasets.
Results of this experiment are shown in Table 2.
Note that we do not intend to identify a winner among the two, since both are meant to optimize different aspects of the network.
For baseline comparison, the network is also pruned using the BAR method.
The volume budget variant of ChipNet outperforms BAR by a signiﬁcant margin.
Moreover, we see that for the same amount of volume constraint, the number of parameters used by BAR are lower than our method for most cases.
A reason for signiﬁcant drop in performance of BAR could be that the optimization algorithm does not fully exploit the choice of channels to be dropped, thereby choosing a sub-optimal set and losing too many parameters from the network.
Between the results of volume and channel constraints for ChipNet, at the ﬁrst look, it seems that channel constraint is better throughout.
However, as stated above, a direct comparison between the two is unfair.
For example, volume constraints are meant to reduce the number of activations, and in turn would also reduce the FLOPs.
This is evident from the results as FLOPS reported for volume constraint are always lower than the respective channel constraint.
For a better understanding of the effects of these budgets, we perform another experiment for pairwise analysis of these constraints.
Figure 3 shows the pairwise plots of the budgets used to prune WRN-26-12 on CIFAR-100.
From the ﬁrst two plots, we see that the scores reported are higher for any volume budget when the network is optimized with volume constraint, and similarly higher for a certain channel budget when the network is optimized for it.
Similar observations can also be made between the number of parameters and FLOPs.
In a nutshell, we observe that the pruned network performs best with respect to the constraint for which the masks are trained.
Thus, the choice of constraint type should not be arbitrary but based on the practical applications, such as reducing FLOPs, among others.
Stability and robustness.
Our pruning strategy is also very stable, and this has already been demonstrated earlier for channel and volume pruning at low resource budgets.
Compared to the baselines, networks obtained with ChipNet are found to perform signiﬁcantly better even without the need for any additional tweaking such as explicitly opening certain channels to ensure network connectivity (Liu et al., 2017; Lemaire et al., 2019).
Another example demonstrating the stability is volume pruning (6.25%) of ResNet-50 on CIFAR-100, where ChipNet performs 11.8% better than BAR.
To account for robustness, we have extensively performed hyperparameter grid search on a channel budget of 6.25% for WRN-26-12 to identify the suitable values for α 1 and α 2 .
It has been observed that values in the neighborhood of this point do not affect the performance.
Details related to this grid search are further provided in Appendix C.5.
Further, the same hyperparameter setting has been used for all the experiments.
The consistent results across all datasets shows that ChipNet is robust.
[Table]
Transfer learning of masks.
Inspired by knowledge distillation (Hinton et al., 2015), where reﬁned information obtained from a deeper teacher network is transferred to a shallow student network, we study here the transfer of sparsity masks across datasets.
For teacher and student, we use Tiny ImageNet and CIFAR-100 datasets, respectively, and ResNet-101 is pruned for different choices of channels budgets.
Table 8 reports the performance scores for the pruned network on CIFAR-100 when the masks are learnt on CIFAR-100 as well as when they are learnt on Tiny ImageNet and transferred.
Interestingly, for moderate channel budgets of 40% and 60%, we see that the models using masks transferred from Tiny ImageNet perform better than those obtained directly on CIFAR-100.
This gain in performance from mask transfer could be attributed to the feature-richness of the chosen teacher dataset.
We also see that for the very low budget case of 20%, masks from the student dataset outperform that from the teacher.
For such low budgets, the expressive power of the model is too low to fully exploit the knowledge from the transferred masks.
C ONCLUSION
We have presented ChipNet, a deterministic strategy for structured pruning of CNNs based on continuous Heaviside function and crispness loss.
Our approach provides the ﬂexibility of using it with different budget constraints.
Through several experiments, it has been demonstrated that ChipNet outperforms the other methods on representative benchmark datasets.
We have also shown that
ChipNet can generate well-performing pruned architectures for very low resource budgets as well.
To conclude, with the strongly effective pruning capability that ChipNet exhibits, it can be used by the machine learning community to design efﬁcient neural networks for a variety of applications.
L IMITATIONS AND F UTURE W ORK
In this paper, we have explored the stability and robustness of ChipNet from various perspectives.
Through experiments, we have shown that ChipNet consistently performs well across several CNN architectures and datasets.
We analyzed it with respect to different choices of budget constraints, performed stability tests for even extreme scenarios of as low as 1% parameters, analyzed how the masks get distributed across the network, and even studied the transferability of masks.
For all these experiments, ChipNet has proved to work well.
However, before ChipNet can be considered a fullproof solution for pruning, additional experiments might be needed.
For example, the applicability of ChipNet is not yet explored on large datasets such as ImageNet and we would like to explore it in our future work.
Further, it would be of interest to explore how the pruned architectures obtained using ChipNet perform for tasks beyond classiﬁcation, such as segmentation and object tracking.
Improving inference speed is an important aspect of object tracking, and it has not been explored from the point of view of network pruning.
We would like to see if the recent object tracking algorithms that use backbones such as ResNet-50 and ResNet-101 can be made faster through our pruning method.
A PPENDICES A E XTENSION : P ROPOSED A PPROACH A. B UDGET C ONSTRAINTS
Additional details related to the 4 budget constraints discussed in this paper follow below.
Channel budget.
It refers to the maximum number of hidden channels h that can be used across all convolutional layers of the network.
Mathematically, it can be stated as
[Equation]
where p denotes the number of hidden channels in the network.
Volume budget.
This budget controls the size of the activations, thereby imposing an upper limit on the memory requirement for the inference step.
We deﬁne volume budget as
[Equation]
where N ( h ) denotes the number of convolutional layers in the network, and A j and p j denote area of the feature maps and their count, respectively, in the j th layer.
Parameter budget.
This budget term directly controls the total number of parameters in the network, and can thus be used to impose an upper limit on the size of the model parameters.
It is deﬁned as
[Equation]
where K j denotes area of the kernel.
The two terms in the numerator account for the number of parameters in the convolutional layer and the batchnorm layer.
FLOPs budget.
This budget can be directly used to control the computational requirement of the model.
Assuming that a sliding window is used to achieve convolution and the nonlinear computational overhead is ignored, the FLOPs budget of the convolution neural network can be deﬁned as in Molchanov et al.
[Equation]
B T RAINING P ROCEDURE
Details regarding the pretraining, pruning and ﬁnetuning steps are discussed below:
B. P RE
[Equation]
T RAINING
WRN-26-12, MobileNetV2, ResNet-50, ResNet-101, ResNet-110 were trained with batch size of 128 at initial learning rate of 5 × 10 − 2 using SGD optimizer with momentum 0 .
9 and weight decay 10 − 3 .
We use step learning rate strategy to decay learning rate by 0 .
5 after every 30 epochs.
For CIFAR-10 and CIFAR-100, models were trained for 300 epochs whereas for Tiny ImageNet the number of epochs were reduced to half to have maintain same number of iterations.
Preresnet-164 was trained with batch size of 64 at initial learning rate of × 10 − 1 using SGD optimizer with momentum 0 .
9 and weight decay 10 − 4 .
We use Multi Step Learning rate strategy to decay learning rate by 0 .
1 after 80 th and 120 th epoch.
The model was trained for 160 epochs for all datasets.
This strategy is adopted from Liu et al.
B. P RUNING
A common pruning strategy was applied for all models irrespective of budget type or dataset.
AdamW (Loshchilov & Hutter, 2019) with constant learning rate of 10 − 3 and weight decay of 10 − 3 was used as optimizer.
Weight decay for ψ was kept 0 .
Weight for budget loss and crispness loss is kept constant to 30 and 10 respectively.
β increases by 2 × 10 − 2 after every epoch starting from 1 and γ doubles after every 2 epochs starting from 2.
B. F INE -T UNING
The ﬁnetuning of pruned model is done exactly similar to the pre-training step.
B. H ARDWARE
All experiments were run on a Google Cloud Platform instance with a NVIDIA V100 GPU (16GB), 16 GB RAM and 4 core processor.
C A DDITIONAL E XPERIMENTS C. P RUNING WITH VOLUME AND CHANNEL BUDGET
This section shows results of ChipNet along with different baselines pruned with with channel and volume budget.
Table 4 is an extension to Table 2 presented in section 4.2.
Table 5 shows numerical values corresponding to ﬁgure 2 discussed in section 4.2.
[Table]
ChipNet (V) 93.
BAR (V) 91.
ChipNet (C) 93.
ChipNet (V) 92.
BAR (V) 91.
ChipNet (C) 92.
ChipNet (V) 12.
BAR (V) 88.
ChipNet (C) 92.
ChipNet (V) 6.
BAR (V) 84.
C. P RUNING WITH ONLY L OGISTIC C URVES
As discussed in section 3.2, continuous Heavyside approximation helps to penalize intermediate values of z to attain values closer to 0-1.
Only with logistic curves the distribution of soft masks gets concentrated at one point as shown in Figure 4b.
Although, the budget constraint will be satisﬁed, this kind of distribution hinders effective channel selection as the relative importance of z cannot be determined concretely.
Contrary to this, using heaviside function with crispness loss models z in
[Table]
[Figure]
terms of their relative importance as shown in Figure 4a and hence results in more effective pruning of less important channels.
C. R OLE OF LOGISTIC - ROUNDING IN BUDGET CALCULATION
As discussed in section 3.3 budget calculation is done on ¯ z rather than computing it directly over the masks z where ¯ z i ∈
¯
z is obtained by applying a logistic projection on z with ψ 0 = 0 .
5 (Eq. 2).
Importance of this projection can be seen through ﬁgure 4c.
The distribution of soft masks obtained with the proposed approach (Figure 4a) is clearly much more distinct than the one calculated without logistic round projection (Figure 4c).
Thus a better threshold can be selected to choose the active sparsity mask that satisﬁes the budget constraint.
[Figure]
C. E FFECT OF THE CHOICE OF BUDGET
We further visualize how ChipNet performs pruning across the various convolutional layers of a network for different choices of budget.
Figure 5 shows the number of active channels per convolutional layer for several pruning factors for the 4 budget types.
These results have been obtained on WRN-26-12.
We see that the pruned networks with low resource budgets are aligned with those with the higher budgets in terms of distribution of active channels across layers.
This could mean that the networks pruned for low resource budgets should be achievable hierarchically from those pruned on larger budgets.
Further, we also see that there are layers with almost no channels left.
The performance of our model is still not affected since these dead layers correspond to the skip connections in the network.
ChipNet identiﬁes such extra connections, and eliminates them if the performance of the model is not affected signiﬁcantly.
C. H YPERPARAMETER GRID SEARCH
An extensive grid search is done to search hyperparameters, α 1 , α 2 , b inc and g inc for pruning WRN26-12 on CIFAR-100 at 6.25% volume budget.
Here α 1 and α 2 are the weightage values given to crispness loss and budget loss repectively in the joint loss as shown in section 3.4.
β inc and γ inc refers the the number of epochs after which value of beta increases by 0.1 and value of gamma doubles, effect of these hyperparameters is discussed in section 3.2.
We found a cloud of values for which the pruning accuracy is comparable.
This cloud can be seen in table 6.
We choose the best values from these for all our other experiments as we concluded that model pruning is less senstive to these hyperparameters.
[Table]
C. S ENSTIVITY A NALYSIS
In this section we show the senstivity analysis for WRN-26-12 on Cifar-100 at 16x Volume budget constraint.
We ran 5 experiments of pruning where value of all four hyperparameters α 1 , α 2 , β inc , γ inc were sampled from the uniform distribution with -+10% perturbations from the tuned values.
[Table]
C. T RANSFERABILITY OF M ASK
Here we show the complete results of Table 3 to depict the transferability of mask proposed in section 4.
[Table]
D I MPLEMENTATION DETAILS D. B ASELINE METHODS
BAR, LZR, MorphNet, WM, ID on WRN-26-12 : All results are taken from Lemaire et al.
We reproduced a few results to cross-check and ensure that there are no big deviations.
We found that our reproduced results were very close to the one reported in the paper.
BAR on Resnet-50 : Results are produced from the code given by (Lemaire et al., 2019).
Pruning strategy is adopted from Lemaire et al.
and the number of iterations are adjusted to match ours for fair comparison.
Network Slimming on PreResNet-164 : We have reproduced the results by using the same pretraining, pruning and ﬁnetuning strategy is used by Liu et al.
(2017) and the same pretraining and ﬁnetuning strategy is used for our results in order to do fair comparison of pruning algorithms.
E P SEUDO CODE
Here we present the explanations and pseudo-codes for the various functions used in 1.
E. L OGISTIC FUNCTION
Algorithm
Input : Optimization parameter corresponding to every mask ψ ; Growth rate control parameter β
Output: Resultant intermediate projection ˜ z
˜ z ← 1 1+ e − βψ
E.
C ONTINUOUS H EAVISIDE FUNCTION
Input : Intermediate projection ˜ z ; Curvature regularization parameter γ
Output: Resultant ﬁnal projection z z ← 1 − e − γ ˜ z
+ ˜ ze − γ
E.
C HIP N ET L OSS FUNCTION
Input : Target budget V 0 ; Current model budget V ; Intermediate projection ˜ z ; Final projection z ; Predicted output ˆ y ; Ground truth y ; Crispness loss weight α 1 ; Budget loss weight α 2 Output: Loss L
L ce ← − (cid:80) y
log(ˆ y )
L c ← (cid:107) ˜
z − z (cid:107)
L b ← ( V − V 0 )
L ← L ce + α 1 L c + α 2 L b
E. F ORWARD FUNCTION
Forward function takes three inputs - network weights ( W ), input data batch ( x ) and sparsity masks ( z ).
The forward function is the forward pass of regular CNN; with one change that the respective sparsity masks are multiplied to the activation obtained after every batch normalization layer.
E. B ACKWARD FUNCTION
The backward function is the back propagation pass of a regular CNN to obtain the gradients of the loss with respect to the model parameters ( W and ψ ).
F A DDITIONAL R ESULTS
[Table]
[Table] [Table]