P RUNING COMPACT C ONV N ETS FOR E FFICIENT I NFERENCE
Anonymous authors
Paper under double-blind review
A BSTRACT
Neural network pruning is frequently used to compress over-parameterized networks by large amounts, while incurring only marginal drops in generalization performance.
However, the impact of pruning on networks that have been highly optimized for efﬁcient inference has not received the same level of attention.
In this paper, we analyze the effect of pruning for computer vision, and study state-ofthe-art ConvNets, such as the FBNetV3 family of models.
We show that model pruning approaches can be used to further optimize networks trained through NAS (Neural Architecture Search).
The resulting family of pruned models can consistently obtain better performance than existing FBNetV3 models at the same level of computation, and thus provide state-of-the-art results when trading off between computational complexity and generalization performance on the ImageNet benchmark.
In addition to better generalization performance, we also demonstrate that when limited computation resources are available, pruning FBNetV3 models incur only a fraction of GPU-hours involved in running a full-scale NAS.
I NTRODUCTION
Neural networks frequently suffer from the problem of over-parameterization , such that the model can be compressed by a large factor to drastically reduce memory footprint, computation as well as energy consumption while maintaining similar performance.
This is especially pronounced for models for computer vision (Simonyan & Zisserman, 2014), speech recognition (Pratap et al., 2020) and large text understanding models such as BERT (Devlin et al., 2018).
The improvements obtained from intelligently reducing the number of model parameters has several beneﬁts, such as reduction in datacenter power consumption, faster inference and reduced memory footprint on edge devices such as mobile phones which also enable decentralized techniques ex. federated learning (Kairouz et al., 2019).
There are several techniques to reduce model size while maintaining similar generalization performance, such as model quantization (Polino et al., 2018), NAS (Neural Architecture Search) (Elsken et al., 2019) and model distillation through teacher-student networks (Gou et al., 2021).
For the scope of this paper, we consider pruning as a technique to remove trainable weights in the network, and save on computation costs for the FBNet family of models.
The motivations for this are two-fold.
Firstly, state-of-the-art models such as FBNet (Wu et al., 2019) already adopt the best practices in the area of efﬁcient hardware-aware design of convolutional neural network based models, and are widely used across different vision tasks.
This makes them suitable baselines to understand whether pruning can offer any performance gain over their already optimized behavior.
While there has been limited work on pruning for efﬁcient convolution network models they investigate older architectures such as
EfﬁcientNet and MobileNet (Aﬂalo et al., 2020) or integrate pruning into expensive techniques such as joint prune-and-architecture search (Wang et al., 2020).
For each of the constituent models of the FBNetV3 family (FBNetV3A, FBNetV3B,..., FBNetV3G) we reduce the number of parameters using two pruning based approaches: (1) Global magnitudebased pruning :
Starting with the pre-trained model, we prune all weights whose magnitude is below a threshold chosen in order to achieve a target number of FLOPs for the pruned model; (2) Uniform magnitude-based pruning :
Starting with the pre-trained model, we prune weights in each layer whose magnitude is below a level-speciﬁc threshold in order to yield a pruned model achieving a target number of FLOPs with the same sparsity in each layer.
After either pruning method is applied, we ﬁne-tune the pruned model for a certain number of epochs until convergence is reached.
Within the scope of our study in this paper, we are mostly interested in the following research questions:
• RQ1 : Pruning to improve computation vs. performance tradeoff.
Can a model obtained by pruning a larger FBNetV3 model M1 (optimized using NAS) achieve higher generalization performance than a smaller FBNetV3 model M2 when the pruned model has the same number of FLOPs as M2 ?
• RQ2 :
Pruning as an efﬁcient paradigm.
When a larger FBNetV3 model M1 is available and computational resources are limited, is pruning a faster and less computationally expensive approach to obtain a model with higher accuracy at a desired computation level (FLOPs) than running a full-ﬂedged architecture search?
Pruning to improve computation vs. performance tradeoff ( RQ1 ).
There have been recent research advances in the area of building hardware-aware efﬁcient models (Deng et al., 2020).
These can provide good generalization performance while adhering to constraints on memory, inference latency and battery power, which are often dictated by the hardware environment where inference happens.
Experiments described in existing work on efﬁcient vision models such as ChamNet (Dai et al., 2019),
MobileNet (Howard et al., 2017), EfﬁcientNet (Tan & Le, 2019) and FBNetV2 (Wan et al., 2020) have shown that it is possible to achieve even higher performances on standard image recognition tasks such as ImageNet (Deng et al., 2009) at a certain level of FLOPs.
However the efﬁcient design of these models does not solve the over-parameterization problem completely, and none of these approaches study how model pruning can be performed to obtain even better trade-offs between computation and model accuracy.
This paper is the ﬁrst of its kind to understand how we can improve on the state-of-the-art in this problem space.
Pruning as an efﬁcient paradigm ( RQ2 ).
In addition to achieving state-of-the-art performance with reduced FLOPs, we are also interested in understanding how such pruned models can be obtained inexpensively with limited resources that are generally available to a machine learning practitioner who has access to existing optimized models but limited computing resources.
For example, the
FBNetV3 models are freely available through Facebook’s Mobile Model Zoo 1 , while EfﬁcientNet models can be obtained at GitHub 2 .
While the techniques needed to obtain computation- and latencyfriendly models have been democratized through open-sourcing the source code as well as the models themselves, fully applying these techniques necessitates costly operations such as ﬁnding an optimal network topology through meta-learning approaches (You et al., 2020) and search algorithms such as
Genetic Algorithms (GAs) (Goldberg & Deb, 1991).
Given the high-degree of intractability of this problem, expensive computational resources are often needed in this case, easily exceeding the budget available to a university research laboratory or an angel-stage startup (Zoph & Le, 2016).
When a starting model is already available, for example through open-sourcing, the best option would be to perform a cheap modiﬁcation of the model to ﬁt a certain target FLOPs/latency requirement.
In this paper we have compared the NAS approaches for training FBNetV3 models with our pruning techniques on a computational complexity metric (GPU-hours) to effectively answer RQ2 .
Benchmark results.
In addition to experimental outcomes for answering RQ1 and RQ2 , we also benchmark pruned FBNetV3 models using available open-sourced quantized sparse kernels and conduct ablation studies to obtain additional insights into pruning performance.
These results augment our main observations and demonstrate that with existing hardware support, it is possible to deploy pruned cutting-edge computer vision models with practical latency reductions and improve further beyond the performance vs. FLOPs trade-off.
We conduct our experiments on ImageNet, which is an object-recognition task on a large training dataset of 1.2 million images.
We show that computationally less intensive techniques such as uniform and global magnitude-based pruning of larger FBNetV3 models can yield higher test accuracies than small models while having the same number of FLOPs.
Given a target computation budget for an efﬁcient model, we show that it is more practically advantageous (both in terms of performance and running time) to simply prune the larger model than run a neural architecture search to ﬁnd the target model from scratch.
The technique we have employed for pruning (unstructured sparsity) is already tried and tested, however our novelty lies in studying whether efﬁcient image recognition models such as FBNetV can be optimized further to improve on the FLOPs-accuracy curve, and the contributions are two fold : (1) FBNets are themselves state-of-the-art in efﬁcient vision models and we achieve better accuracy-FLOPs tradeoff over these models and (2) from the standpoint of computational overhead, we signiﬁcantly reduce the amount of GPU hours required to obtain such models.
Pruning a publicly available NAS optimized model incurs ≈ 4x less GPU hours to achieve a target FLOPs level, compared to training a full-ﬂedged NAS to obtain a model which has less accuracy at the same FLOPs level.
Paper organization.
The remainder of this paper is organized as follows.
In Section 2, we describe related work in the area of efﬁcient vision model design and also provide an introduction to different pruning techniques.
In Section 3, we discuss our experimental setup, including a description of the baseline models and the global and uniform pruning approaches we have employed.
Sectiondescribes our main ﬁndings and we conclude the paper in Section 5.
R ELATED W ORK
We discuss related literature in the areas of computationally efﬁcient vision models and model pruning .
Within the scope of our work, we mainly focus on inference efﬁciency of models in contrast to training efﬁciency.
Computationally efﬁcient vision models: Neural networks for computer vision are generally characterized by convolutional layers and fully-connected layers, along with blocks such as residual or skip connections.
This makes such networks resource intensive in terms of FLOPs, which affects the memory storage and power consumed, and also leads to increased latency.
It is of paramount importance to design more efﬁcient networks which can provide higher performance for the same
FLOPs or latency level, or even to optimize them appropriately to provide the same performance at reduced FLOPs/latency.
This can be performed either through the design of new simpliﬁed layers, for example in deep residual learning (He et al., 2016) or though explicit model compression as in weight quantization (Polino et al., 2018).
Extremely deep networks for image recognition often suffer from not only high complexity and inference latency, but also from the issue of vanishing gradients (Pascanu et al., 2013).
This was addressed through deep residual networks which effectively simpliﬁed network design through skip-connections.
MobileNets (Howard et al., 2017) are one of the earlier approaches to building small low-latency networks by using depthwise separable convolutions with two parameters, width and resolution multipliers.
They demonstrate the effectiveness of MobileNets across different vision tasks, such as face embeddings and object detection.
MobileNetV2 (Sandler et al., 2018) extends MobileNets by utilizing inverted residual ﬁlter structures and linear bottlenecks, obtaining improvements on state-of-the-art models both in terms of accuracy and computational complexity.
ShufﬂeNets (Zhang et al., 2018) propose dedicated residual units where 1 × 1 convolutions are replaced with pointwise group convolutions and channel shufﬂing reducing FLOPs computations.
More recently, the focus on building efﬁcient neural network models has shifted to techniques that treat the design of efﬁcient networks as a search problem, falling under the umbrella of Neural
Architecture Search (NAS).
EfﬁcientNets
(Tan & Le, 2019) propose a novel scaling method which adjusts the network’s length, width, and resolution to optimize performance subject to target memory and FLOPs constraints.
They also deﬁne a novel baseline that is optimized by a multi-objective neural architecture search.
The FBNet collections of models—FBNet (Wu et al., 2019), FBNetV2 (Wan et al., 2020) and FBNetV3 (Dai et al., 2021)—employ neural architecture search to obtain highly-optimized models that improve on the state-of-the-art for different visual understanding tasks.
FBNet frames the architecture search as a differentiable meta-learning problem with gradient based techniques, namely
DNAS —Differentiable Neural Architecture Search—by Wu et al.
and avoids selecting the optimized model over a discrete set.
The subsequent entry in this collection, FBNetV2, expands the search space over conventional DNAS, and employs a masking scheme to maintain the same level of computational complexity while searching over this expanded space.
FBNetV3 further improves on the state-of-the-art by employing Neural Architecture Recipe Search (NARS) and searching over the space of not only architectures, but also corresponding recipes (which are generally hyper-parameters).
In this paper, we consider FBNetV3 models as our baselines as they are state-of-the-art.
We are interested in understanding if they are overparameterized and evaluate how much model pruning can improve performance at a certain FLOPs level over the state-of-the-art in this family of models.
Model Pruning: Modern neural networks, particularly those processing complex sensory inputs (such as speech, vision and language) for perception applications, are often over-parameterized.
It is only to be expected that we should be able to compress such networks signiﬁcantly to maintain the same level of performance at decreased level of computation (fewer weights and reduced FLOPs), memory footprint and power consumption.
Foundational efforts in this space include the Optimal Brain
Surgeon (Hassibi & Stork, 1993) and Optimal Brain Damage (LeCun et al., 1990).
Recently the idea of network pruning has been formalized through the lottery ticket hypothesis (Frankle & Carbin, 2018), which claims that randomly initialized, feed-forward networks have winning sub-networks that perform just as well as the original network on an unseen test dataset.
Model pruning is generally of two types: unstructured and structured pruning.
Unstructured pruning, as the name suggests, doesn’t adhere to any structure and prunes neurons based on chosen criteria (such as magnitude).
This has the advantage of providing higher performance, but is difﬁcult to implement in hardware, as it needs dedicated support for efﬁcient sparse matrix multiplications.
Meanwhile, structured pruning is the practice of removing entire groups of neurons (e.g., blocks within the weight matrix, or channels in convolutional neural networks).
This is easy to implement without dedicated hardware support, but has the issue of lower generalization performance than unstructured pruning (Yao et al., 2019).
In the literature, there have also been several studies, for example investigating whether rewinding (training from scratch with a ﬁxed mask) can perform just as well as the ﬁne-tuning on top of the original unpruned network (Renda et al., 2020). Blalock et al.
(2020) provide an overview survey of recent advances and open problems in neural network pruning.
In the research area of designing efﬁcient networks for computer vision, there has not been much focus on understanding how pruning can be applied to the current generation of models.
Most literature on pruning is based on older networks such as VGGNet, ResNet (He et al., 2016), and
MobileNet (Sandler et al., 2018).
Our work improves upon these existing studies by understanding how pruning can improve the FLOPs-accuracy tradeoff over existing state-of-the-art networks.
P RUNING T ECHNIQUES AND S ETUP
In this section, we describe the main components of our techniques and experimental setup, including
Baseline Models , Pruning Techniques , Latency Measurement and Metrics .
We have mainly used standard splits of the ImageNet dataset, further details are in Section A.1 of the appendix.
3. B ASELINE M ODELS
Dai et al.
(2020) address the previous limitations of NAS-based architecture search where these approaches can only search over architectures given a training recipe (set of hyperparameters), and thus cannot optimize over both.
As described in Section 2, the most recent state-of-the-art models are based on NARS (Neural Architecture-Recipe Search), which we select as baseline models.
Tablelists the accuracy of FBNetV3 models (Dai et al., 2021) on the ImageNet classiﬁcation task, along with the number of model parameters and computation complexity in terms of FLOPs.
Each baseline model consists of multiple IRF (Inverted Residual Filter) blocks, which contain convolutional layers of different kernel sizes.
For our experiments, we are mostly interested in 1 ×convolutions as potentially prunable, since within each FBNetV3 model, the 1 × 1 convolution layers constitute >80% of total model FLOPs for all models in the family, and the open-sourced sparsity kernel support we use for latency benchmarking is available only for fully connected layers.
A 1 ×convolution can be transformed into an equivalent fully connected layer with a few tensor reshape operations without any signiﬁcant loss of performance or latency.
For each initial and target FBNetV3 model X and Y , where X is larger than Y , we prune X to a sparsity level of S so that the FLOP count is the same as for Y .
The number of FLOPs consumed by a linear layer of sparsity S is proportional to the number of sparse matrix multiplications performed and is given by S ∗ F , where F is the corresponding dense FLOPs.
Thus if F 1 × 1 ( X ) is the number of
FLOPs consumed by the 1 × 1 convolution layers and F ( x ) is the total number of FLOPs consumed by model X , we have:
[Equation]
Hence, sparsity measures the fraction of 1 × 1 convolution weights removed, and so higher sparsity indicates a smaller model.
For the uniform pruning scnario, Table 1 shows the amount of sparsity required to prune each larger FBNetV3 model to a smaller one based on Eq.
For global pruning, (1) does not hold, and we compute the target sparsities empirically from the layer shapes instead with details provided in Section A.2.
We prune each larger FBNetV3 model to a discrete FLOPs target based on a deﬁned set of smaller models in the family, and not to a continuous range of FLOPs values, as it makes it easier to compare models directly based on a given computation budget.
If we can demonstrate that for the same computation level, the pruned larger FBNetV3 model has higher performance than a smaller model with the same FLOPs, it is sufﬁcient to demonstrate that we can improve on the FLOPs-accuracy curve over the state-of-the-art.
3. P RUNING T ECHNIQUES
In this paper, we utilize a pre-trained FBNetV3 model with higher number of FLOPs without training an image classiﬁcation model from scratch with sparsity, which would be time consuming and computationally intensive.
There are several approaches in the literature such as prune-and-ﬁnetune (Han et al., 2015) and iterative pruning with sparsity scheduling (Frankle & Carbin, 2018).
We have utilized the former for our experiments, as although studies have shown that iterative and incremental pruning approaches lead to better generalization performance, they typically require training for high number of epochs, need tuning and selection of optimal sparsity schedules and are computationally resource intensive.
We have therefore not considered them in our experiments.
For our prune and ﬁne-tune experiments, we have used 8-GPU boxes, with each box having Nvidia V (Volta) 32G GPUs.
As described in Section 1, we perform both global and magnitude-based pruning experiments.
For the latency benchmarking, we also perform magnitude-based uniform pruning with a sparse block size of 1 × 4 as explained in Section 3.3.
We have conducted a hyper-parameter tuning for the learning rate parameter, with LR values in the set {4e-5, 8e-5, 1.6e-4}, as ﬁne-tuning generally admits smaller learning rates than training from scratch.
We have found that using the same learning rate for all models, along with the same hyper-parameter settings used for training the seed model is sufﬁcient to obtain pruned networks which are superior to the baseline FBNetV3 models.
Hence minimal hyper-parameter tuning was required for our experiments and we have used values of settings such as weight decay and momentum to be the same as those used for training the baseline FBNetV3 models.
During ﬁne-tuning after pruning, we have used a smoothed validation loss to stop the process early after a convergence tolerance (0.01%) is reached between two consecutive epochs.
Generally, we have observed ﬁne-tuning to converge around ∼ 250 epochs.
3. LATENCY MEASUREMENTS AND M ETRICS
We are interested not only in the sparsity level of our pruned models and the image recognition performance, but also in metrics which potentially improve due to model sparsity, such as number of parameters, the FLOP count and the model latency.
For reporting model performance under pruning, we use standard image recognition metrics such as Top-1 and Top-5 test accuracies.
We measure overall model sparsity, which is different to the layer sparsity since we only prune 1 × 1 convolution layers, as explained in Section 3.1.
We report the model FLOPs, because this metric captures the computational footprint of the model and its power consumption.
Last, we record the total latency (in ms.) under pruning.
The sparse kernels used in our experiments are already in open-source and released under the PyTorch sparse quantization library 3 .
Prior to using these kernels, we perform uniform layer-wise block-based pruning with block sizes of 1 × 4 .
Magnitude based pruning is implemented at block level, and the model is quantized to 8-bit integers (int8) before latency benchmarking, which is performed on Intel CPUs designed using the Skylake micro-architecture.
While we would expect sparsity to translate to tangible inference speedups, this is highly dependent on the sparse kernel support provided by hardware.
Current hardware is not wellsuited for unstructured randomly sparse matrix multiplications and tend to do better with structured sparsity in models (Anwar et al., 2017).
We have utilized block sparsity within the weight matrix for latency experiments.
However this often tends to come at a decreased level of model performance.
The design of highly performant sparse models under structured sparsity with reasonable inference speedups remains an important research topic outside the scope of this paper. [Table]
R ESULTS 4. P RUNED FBN ET V3 MODEL PERFORMANCE
To answer RQ1 , we consider the family of FBNetV3 models as baselines and seed models for further pruning.
For each pair of models X , Y in the family, we calculate the amount of sparsity required to prune the larger model X to a model that consumes the same number of FLOPs as the target smaller model Y , via Equation 1.
There are 21 potential seed and target model pairs, however we conduct pruning experiments only for a depth of 2 for tractability.
For example, given FBNetV3E as the seed, we only prune it to FLOPs targets corresponding to FBNetV3D and FBNetV3C. Tablepresents the accuracy and number of parameters of the pruned models at each target FLOPs level.
The improvement in performance is apparent even at lower FLOPs targets, where we might expect baseline models such as FBNetV3A to not be over-parameterized.
For example, pruning FBNetV3C to a target of 356.6 MFLOPs obtains a network which is 1.43% better than FBNetV3A. Figure 1 plots the Top-1 ImageNet testing accuracy vs. FLOPs for the best pruned models as seen from Table 1.
This clearly shows that pruning FBNetV3 models with minimal ﬁne-tuning can signiﬁcantly improve on the state-of-the-art for FLOPs vs. Top-1 accuracy trade-off.
This analysis is performed for both uniform layer-wise and global magnitude-based prune with ﬁne-tune settings.
Global pruning ranks the weights of the entire network in contrast to uniform layer-wise pruning, which ranks each layer’s weights to determine the sparsity mask.
It would be expected that global pruning performs better than uniform pruning for the same target sparsity level or number of non-sparse parameters.
However in our experiments we determine the pruning threshold based on FLOPs targets, and ﬁnd global pruning to require higher sparsity levels, which results in uniform pruning outperforming global pruning in
Top-1 ImageNet accuracy in most cases.
4. P RUNING C OMPLEXITY
In addition to demonstrating the improvement over state-of-the-art obtained by pruning FBNetV models, it is also important to quantify the reduction in computational complexity obtained in pruning a larger FBNetV3 model compared to training an FBNetV3 model directly through NAS (Network
Architecture Search).
RQ2 (pruning for efﬁcient model search) asks if the pruning and subsequent ﬁne-tuning approach in Section 4.1 is faster than a full-ﬂedged neural architecture search.
During pruning and subsequent ﬁne-tuning, we train the pruned networks till the validation loss converges to within a pre-speciﬁed tolerance, as described in Section 3.2.
The time needed is generally less than when training the original FBNetV3 models, which runs for 400 epochs.
The number of GPU-hours is computed as (number of training GPU nodes) * (number of GPUs per node) * (training time to A B
C D E F
C->A
D->B
E->C
F->D
F->E
G->F
[Table]
convergence) for each network.
In Table
[Table]
for each of the best performing uniformly-pruned models in Section
[Table]
1 we report the number of GPU-hours consumed by the prune and ﬁne-tune strategy, along with the GPU-hours consumed when obtaining a FBNetV3 model through architecture search using the method described in Dai et al.
The results are quite conclusive—we not only obtain pruned models superior in performance to the original neural search optimized models [Table] but also as described in Section 1, computational cost is signiﬁcantly lower when starting from a pre-trained model with higher FLOPs.
Given the performance improvements obtained with lower computational resources, this approach is beneﬁcial for an experimental setting where researchers have access to open-sourced pre-trained models and limited GPU resources, for example in a small startup or an academic environment.
We observe that the degree of speedup reduces as the network size gets bigger (e.g., in FBNetV3A vs. FBNetV3C) due to higher training time to convergence.
Nevertheless, we still obtain a speedup of 3-5 times compared to a full NAS (Neural Architecture Search).
4. L ATENCY E XPERIMENTS
We also measure the latency-performance tradeoff for the pruned FBNetV3G models.
FBNetV3G is the largest model in the family and so is expected to have the best generalization performance under high sparsity levels.
As described in Section 3.3, we prune the network using block sparsity (where the block size is 1 × 4 ) to sparsity levels in the set {40%, 50%, 60%}.
We have not utilized lower sparsity levels, as we have observed that for the selected kernels we need at least 40% sparsity to yield any signiﬁcant latency beneﬁts.
We have pruned all 1 × 1 convolution layers uniformly here and subsequently converted them to fully-connected layers for compatibility with the quantized sparse kernels.
In Figure 2a, we present the Top-1 ImageNet accuracy vs. latency curve [Figure] after pruning the FBNetV3G network for different sparsity levels.
The pruned FBNetV3G models show marked performance reduction with lower latency as expected, with a sparsity level of 60% translating to around 7% absolute accuracy reduction with a latency reduction of 18 ms (16% relative).
While the 1 × 1 convolution layers account for >80% of FLOPs, they only constitute 25% of overall network latency.
This is consistent with previous literature (Dudziak et al., 2020) which shows that computational complexity (ex. FLOPs) and latency are not well-correlated, and indeed the latter is more dependent on layer shapes.
This result underscores the need to develop more latency-friendly pruning techniques which can potentially improve on the state-of-the-art in this domain.
4. I NSIGHTS INTO PRUNING EXPERIMENTS
Our pruning experiments demonstrate that we can improve on the state-of-the-art
FBNetV3 models in generalization performance for a given FLOPs level.
In this subsection, we obtain an insight into (1) the sparsity pattern under global magnitude-based pruning and (2) the sensitivity of each layer when pruned in isolation under uniform layer-wise magnitude pruning (sparsity level of 95%).
For (1) in Figure 2b, we plot the amount of sparsity obtained per 1 × 1 convolution layer.
The model being considered is an FBNetV3E network pruned to a sparsity level of 43.6%, to the same FLOPs level as FBNetV3C and subsequently ﬁne-tuned.
We note that the sparsity level in lower layers is lower which is potentially required for maintaining the performance .
Higher sparsity can be admitted in upper layers of the network where it has learnt more redundant representations.
SE (Squeeze and
Excitation) 1 × 1 convolution layers generally tend to get pruned more compared to other layers, with the sparsity being >99% for two such SE layers in stage xif 5 _ 0 .
This indicates that we can also consider revisiting SE layer role in FBNetV3 networks, and even remove entire layers in future work to yield additional latency and FLOPs beneﬁts.
For analysis (2) we prune each 1 × 1 convolution layer in isolation at a sparsity target of 95% and record the Top-1 test accuracy obtained on ImageNet dataset.
For each type of layer, PW:expansion,
PWL: bottleneck, SE:
Squeeze-Excitation we plot the distribution of accuracies in Figure 2d.
We observe that the PW and PWL layers are most sensitive to high sparsity, while SE layers are able to retain performance adequately.
We could also avoid pruning the most sensitive layers (appearing as outliers in the ﬁgure) to maintain generalization performance.
This observation corroborates ﬁndings from analysis (1), and motivates us to revisit the role of squeeze-excitation layers in future work.
C ONCLUSIONS
In this paper, we have investigated the problem of improving on the current state-of-the-art FLOPs vs. performance trade-off for FBNets which have been pre-optimized by NAS (Neural Architecture
Search).
We have employed network pruning techniques, and our results demonstrate that we can further improve on performance over FBNetV3 at a given FLOPs target through global as well as uniform magnitude-based pruning.
This happens not only for relatively over-parameterized networks such as FBNetV3G, but also smaller networks such as FBNetV3A which have lower computational complexity.
On average, the GPU-hours incurred during pruning is about ∼ 4 × less than that consumed by a full-scale NAS.
We have also performed latency measurements on the FBNetV3G model and conducted an analysis to understand the sparsity patterns and sensitivity of different
FBNetV3 layers to pruning.
For future work, we plan to investigate squeeze-excitation layers in more detail and explore structured pruning approaches such as channel and layer pruning to further improve on the latency-performance tradeoff for this family of models
A A PPENDIX A. D ATASET
Our pruning experiments are conducted on the ImageNet dataset, which is commonly used in the literature to evaluate performance of image classiﬁcation models.
It is a collection of millions of images, where there is a deﬁned taxonomy based on the WordNet hierarchy.
The taxonomy comprises of approximately 22,000 visual subcategories, making this a large-scale classiﬁcation problem.
ImageNet was ﬁrst introduced by Deng et al.
and has been adopted by the machine learning and computer vision communities to benchmark image classiﬁcation models.
We use the entire dataset consisting of 14 million images for our experiments
[Table]
and we utilize both the training set and the validation set.
We split the training set to also create a smaller validation set (of 50, images evenly distributed across all image categories) for parameter tuning and setting the training convergence criterion.
The ImageNet validation set is used in our experiments as an testing set for reporting model generalization performance
[Table]
A. O BTAINING SPARSITY LEVELS FOR GLOBAL MAGNITUDE - BASED PRUNING
While the sparsity level required to prune the 1 × 1 convolution layers can be obtained from Equationfor the uniform pruning case, for global magnitude-based pruning all weights in such layers are ranked globally and then thresholded to determine the sparsity mask.
There is no closed form expression we can use to determine the sparsity level given the FLOPs target for this scenario.
To obtain the sparsity level, we have used the pre-trained seed model and pruned it to a sparsity level s .
We can calculate the overall FLOPs consumed due to sparsity s by plugging in the layer-wise shapes and sparsity levels.
Extrapolating backwards from target FLOPs, we can easily ﬁnd out which sparsity level corresponds to this.
It is important to note that since we prune and ﬁne-tune with a ﬁxed sparsity mask, the number of FLOPs estimated at sparsity level s does not change even after the network is ﬁne-tuned.