Original: N ETWORK P RUNING THAT M ATTERS : A C ASE S TUDY ON R ETRAINING V ARIANTS
Revised: N ETWORK P RUNING THAT M ATTERS : A C ASE S TUDY ON R ETRAINING V ARIANTS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Network pruningand retraining is an effective method to reduce the computational expense of over-parameterized neural networks for deployment on low-resource systems.
Revised: Network pruning is an effective method to reduce the computational expense of over-parameterized neural networks for deployment on low-resource systems.

Original: Recent state-of-the-art techniques for retraining pruned networks such as weight rewinding and learning rate rewinding have been shown to outperform the traditional ﬁne-tuning technique in recovering the lost accuracy (Renda et al., 2020), but so far it is unclear what accounts for such performance.
Revised: Recent state-of-the-art techniques for retraining pruned networks such as weight rewinding and learning rate rewinding have been shown to outperform the traditional ﬁne-tuning technique in recovering the lost accuracy (Renda et al., 2020) but so far it is unclear what accounts for such performance.

Original: In this work, we conduct extensive experiments to verify and analyze the uncanny effectiveness of learning rate rewinding.
Revised: In this work, we conduct extensive experiments to verify and analyze the uncanny effectiveness of learning rate rewinding.

Original: We ﬁnd that the reason behind the success of learning rate rewinding is the usage of a large learning rate.
Revised: We ﬁnd that the reason behind the success of learning rate rewinding is the usage of a large learning rate.

Original: Similar phenomenon can be observed in other learning rate schedules that involve large learning rates, e g., the 1-cycle learning rate schedule (Smith & Topin, 2019).
Revised: Similar phenomenon can be observed in other learning rate schedules that involve large learning rates, e g., the 1-cycle learning rate schedule (Smith & Topin, 2019).

Original: By leveraging the right learning rate schedule in retraining, we demonstrate a counter-intuitive phenomenon in that randomly pruned networks could even achieve better performance than methodically pruned networks (ﬁne-tuned with the conventional approach).
Revised: By leveraging the right learning rate schedule in retraining, we demonstrate a counter-intuitive phenomenon in that randomly pruned networks could even achieve better performance than methodically pruned networks (ﬁne-tuned with the conventional approach).

Original: Our results emphasize the cruciality of the learning rate schedule in pruned network retraining – a detail often overlooked by practioners during the implementation of network pruning.
Revised: Our results emphasize the cruciality of the learning rate schedule in pruned network retraining – a detail often overlooked by practioners during the implementation of network pruning

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Training neural networks is an everyday task in the era of deep learning and artiﬁcial intelligence.
Revised: Training neural networks is an everyday task in the era of deep learning and artiﬁcial intelligence.

Original: Generally speaking, given data availability, large and cumbersome networks are often preferred as they have more capacity to exhibit good data generalization.
Revised: Generally speaking, given data availability, large and cumbersome networks are often preferred as they have more capacity to exhibit good data generalization.

Original: In the literature, large networks are considered easier to train than small ones (Neyshabur et al., 2018; Arora et al., 2018; Novak et al., 2018; Brutzkus & Globerson, 2019).
Revised: In the literature, large networks are considered easier to train than small ones (Neyshabur et al., 2018; Arora et al., 2018; Novak et al., 2018; Brutzkus & Globerson, 2019).

Original: Thus, many breakthroughs in deep learning are strongly correlated to increasingly complex and over-parameterized networks.
Revised: Thus, many breakthroughs in deep learning are strongly correlated to increasingly complex and over-parameterized networks.

Original: However, the use of large networks exacerbate the gap between research and practice since real-world applications usually require running neural networks in low-resource environments for numerous purposes: reducing memory, latency, energy consumption, etc.
Revised: However, the use of large networks exacerbate the gap between research and practice since real-world applications usually require running neural networks in low-resource environments for numerous purposes: reducing memory, latency, energy consumption, etc.

Original: To adopt those networks to resourceconstrained devices, network pruning (LeCun et al., 1990; Han et al., 2015; Li et al., 2016) is often exploited to remove dispensable weights, ﬁlters and other structures from neural networks.
Revised: To adopt those networks to resourceconstrained devices, network pruning (LeCun et al., 1990; Han et al., 2015; Li et al., 2016) is often exploited to remove dispensable weights, ﬁlters and other structures from neural networks.

Original: The goal of pruning is to reduce overall computational cost and memory footprint without inducing signiﬁcant drop in performance of the network.
Revised: The goal of pruning is to reduce overall computational cost and memory footprint without inducing signiﬁcant drop in performance of the network.

Original: A common approach to mitigating performance drop after pruning is retraining: we continue to train the pruned models for some more epochs.
Revised: A common approach to mitigating performance drop after pruning is retraining: we continue to train the pruned models for some more epochs.

Original: In this paper, we are interested in approaches based on learning rate schedules to control the retraining.
Revised: In this paper, we are interested in approaches based on learning rate schedules to control the retraining.

Original: A well-known practice is ﬁne-tuning , which aims to train the pruned model with a small ﬁxed learning rate.
Revised: A well-known practice is ﬁne-tuning , which aims to train the pruned model with a small ﬁxed learning rate.

Original: More advanced learning rate schedules exist, which we generally refer to as retraining .
Revised: More advanced learning rate schedules exist, which we generally refer to as retraining .

Original: The retraining step is a critical part in implementing network pruning, but it has been largely overlooked and tend to vary in each implementation including differences in learning rate schedules, retraining budget, hyperparameter choices, etc.
Revised: The retraining step is a critical part in implementing network pruning, but it has been largely overlooked and tend to vary in each implementation including differences in learning rate schedules, retraining budget, hyperparameter choices, etc.

Original: Recently, Renda et al.
Revised: Recently, Renda et al.

Original: (2020) proposed a state-of-the-art technique for retraining pruned networks namely learning rate rewinding (LRRW).
Revised: (2020) proposed a state-of-the-art technique for retraining pruned networks namely learning rate rewinding (LRW).

Original: Speciﬁcally, instead of ﬁne-tuning the pruned networks with a ﬁxed learning rate, usually the last learning rate from the original training schedule (Han et al., 2015;
Revised: Speciﬁcally, instead of ﬁne-tuning the pruned networks with a ﬁxed learning rate, usually the last learning rate from the original training schedule (Han et al., 2015;

Original: Liu et al., 2019), the authors suggested using the learning rate schedule from the previous t epochs (i.e. rewinding).
Revised: Liu et al., 2019), the authors suggested using the learning rate schedule from the previous t epochs (i.e. rewinding).

Original: This seemingly subtle change in learning rate schedule led to an important result: LRRW was shown to achieve comparable performance to more complex and computationally expensive pruning algorithms while only utilizing simple norm-based pruning.
Revised: This seemingly subtle change in learning rate schedule led to an important result: LRW was shown to achieve comparable performance to more complex and computationally expensive pruning algorithms while only utilizing simple norm-based pruning.

Original: Unfortunately, the authors did not provide the analysis to justify the improvement.
Revised: Unfortunately, the authors did not provide the analysis to justify the improvement.

Original: In general, it is intriguing to understand the importance of a learning rate schedule and how it affects the ﬁnal performance of a pruned model.
Revised: In general, it is intriguing to understand the importance of a learning rate schedule and how it affects the ﬁnal performance of a pruned model.

Original: In this work, we study the behavior of pruned networks under different retraining settings.
Revised: In this work, we study the behavior of pruned networks under different retraining settings.

Original: We found that the efﬁcacy from retraining with learning rate rewinding is rooted in the use of a large learning rate, which helps pruned networks to escape from the suboptimality due to pruning.
Revised: We found that the efﬁcacy from retraining with learning rate rewinding is rooted in the use of a large learning rate, which helps pruned networks to converge faster after pruning.

Original: We demonstrate that the success of learning rate rewinding over ﬁne-tuning is not exclusive to the learning rate schedule coupling with the original training process.
Revised: We demonstrate that the success of learning rate rewinding over ﬁne-tuning is not exclusive to the learning rate schedule coupling with the original training process.

Original: Retraining with a large learning rate could outperform ﬁne-tuning even with some modest retraining, e.g., for a few epochs, and regardless of network compression ratio.
Revised: Retraining with a large learning rate could outperform ﬁne-tuning even with some modest retraining, e.g., for a few epochs, and regardless of network compression ratio.

Original: We argue that retraining is of paramount importance to regain the performance in network pruning and should not be overlooked when comparing two pruning algorithms.
Revised: We argue that retraining is of paramount importance to regain the performance in network pruning and should not be overlooked when comparing two pruning algorithms.

Original: This is evidenced by our extensive experiments: (1) randomly pruned network can outperform methodically pruned network with only (hyper-parameters free) modiﬁcations of the learning rate schedule in retraining, and (2) a simple baseline such as norm- based pruning can perform as well as as other complex pruning methods by using a large learning rate restarting retraining schedule.
Revised: This is evidenced by our extensive experiments: (1) randomly pruned network can outperform methodically pruned network with only (hyper-parameters free) modiﬁcations of the learning rate schedule in retraining, and (2) a simple baseline such as norm-based pruning can perform as well as as other complex pruning methods by using a large learning rate restarting retraining schedule.

Original: The contributions of our work are as follows.
Revised: The contributions of our work are as follows.

Original: • We document a thorough experiment on learning rate schedule for the retraining step in network pruning with different pruning conﬁgurations; • We show that learning rate matters: pruned models retrained with a large learning rate consistently outperform those trained by conventional ﬁne-tuning regardless of speciﬁc learning rate schedules;
Revised: • We document a thorough experiment on learning rate schedule for the retraining step in network pruning with different pruning conﬁgurations; • We show that learning rate matters: pruned models retrained with a large learning rate consistently outperform those trained by conventional ﬁne-tuning regardless of speciﬁc learning rate schedules;

Original: • We present a novel and counter-intuitive result achieved by solely applying large learning rate retraining: a randomly pruned network and a simple norm-based pruned network can perform as well as networks obtained from more sophisticated pruning algorithms.
Revised: • We present a novel and counter-intuitive result achieved by solely applying large learning rate retraining: a randomly pruned network and a simple norm-based pruned network can perform as well as networks obtained from more sophisticated pruning algorithms.

Original: Given the signiﬁcant impact of learning rate schedule in network pruning , we advocate the following practices: learning rate schedule should be considered as a critical part of retraining when designing pruning algorithms.
Revised: Given the signiﬁcant impact of learning rate schedule in network pruning, we advocate the following practices: learning rate schedule should be considered as a critical part of retraining when designing pruning algorithms.

Original: Rigorous ablation studies with different retraining settings should be made for a fair comparison of pruning algorithms.
Revised: Rigorous ablation studies with different retraining settings should be made for a fair comparison of pruning algorithms.

Original: To facilitate reproducibility, we would release our implementation upon publication.
Revised: To facilitate reproducibility, we would release our implementation upon publication.

Original:  P RELIMINARY AND M ETHODOLOGY
Revised:  P RELIMINARY AND M ETHODOLOGY

Original: Pruning is a common method to produce compact and high performance neural networks from their original large and cumbersome counterparts. 
Revised: Pruning is a common method to produce compact and high performance neural networks from their original large and cumbersome counterparts. We can categorize pruning approaches into three classes:

Original: In most cases, pruning consists of three steps: training the original network to convergence, prune redundant weights based on some criteria, and retrain the pruned model to regain the performance loss due to pruning.
Revised: Pruning after training - which consists of three steps: training the original network to convergence, prune redundant weights based on some criteria, and retrain the pruned model to regain the performance loss due to pruning (Li et al., 2016; Han et al., 2015; Luo et al., 2017; Ye et al., 2018; Wen et al., 2016;

Revised: He et al., 2017); Pruning during training - we update the “pruning mask” while training the network from scratch, thus, allowing pruned neurons to be recovered (Zhu & Gupta, 2017; Kusupati et al., 2020; Wortsman et al., 2019; Lin et al., 2020b; He et al., 2019; 2018);

Revised: Pruning before training - Inspired by the Lottery Ticket Hypothesis (Frankle & Carbin, 2019), some recent works try to ﬁnd the sparsity mask at initialization and train the pruned network from scratch without changing the mask (Lee et al., 2019; Tanaka et al., 2020; Wang et al., 2020).

Original:  Traditionally, the last step is referred to as ﬁne-tuning , i.e., continue to train the pruned model with a small learning rate obtained from the last epoch of the original model.
Revised: In this work, we are mainly concerned with the ﬁrst category i.e. pruning after training which has the largest body of work to our knowledge. Traditionally, the last step is referred to as ﬁne-tuning , i.e., continue to train the pruned model with a small learning rate obtained from the last epoch of the original model.

Original: This seemly subtle step is often overlooked when designing pruning algorithms.
Revised: This seemly subtle step is often overlooked when designing pruning algorithms.

Original: Particularly, we found that the implementation of previous pruning algorithms have many notable differences in their retraining step: some employed a small value of learning rate (e.g. 0 .
Revised: Particularly, we found that the implementation of previous pruning algorithms have many notable differences in their retraining step: some employed a small value of learning rate (e.g. 0 .

Original: 001 on
Revised: 001 on

Original: ImageNet) to ﬁne-tune the network (Molchanov et al., 2016; Li et al., 2016; Han et al., 2015) for a small number of epochs, e.g., 20 epochs in the work by Li et al.
Revised: ImageNet) to ﬁne-tune the network (Molchanov et al., 2016; Li et al., 2016; Han et al., 2015) for a small number of epochs, e.g., 20 epochs in the work by Li et al.

Original: (2016); some used a larger value of learning rate ( 0 . 01 ) with much longer retraining budgets, e.g., 60 , 100 and 120 epochs respectively on ImageNet (Zhuang et al., 2018; Gao et al., 2020; Li et al., 2020);
Revised: (2016); some used a larger value of learning rate ( 0 . 01 ) with much longer retraining budgets, e.g., 60 , 100 and 120 epochs respectively on ImageNet (Zhuang et al., 2018; Gao et al., 2020; Li et al., 2020);

Original: You et al. 
Revised: You et al. ; Li et al.

Original: (2019) utilized 1-cyclelearning rate schedule (Smith & Topin, 2019)  instead of conventional step-wise schedule.
Revised: (2020) respectively utilized 1-cycle (Smith & Topin, 2019) and cosine annealing learning rate schedule [Figure] we instead of conventional step-wise schedule.

Original: Despite such difference, the success of eachpruning algorithm is only attributed to the pruning  itself.
Revised: Despite such difference, the success of each algorithm is only attributed to the pruning algorithm itself.

Original: This motivates us to ask the question: do details like learning rate schedule used for retraining matter?
Revised: This motivates us to ask the question: do details like learning rate schedule used for retraining matter?

Original: In this section, we strive to understand the behavior of pruned models under different retraining conﬁgurations and how they impact the ﬁnal performance.
Revised: In this section, we strive to understand the behavior of pruned models under different retraining conﬁgurations and how they impact the ﬁnal performance.

Original: Speciﬁcally, we conduct experiments with different retraining schedules on simple baselines such as (cid:96) 1 -norm ﬁlters pruning (Li et al., 2016) and magnitude-based weights pruning (Han et al., 2015).
Revised: Speciﬁcally, we conduct experiments with different retraining schedules on simple baselines such as (cid:96) 1 -norm ﬁlters pruning (Li et al., 2016) and magnitude-based weights pruning (Han et al., 2015).

Original: We show that the efﬁcacy of several pruning algorithms can be boosted by simply modifying the learning rate schedule.
Revised: We show that the efﬁcacy of several pruning algorithms can be boosted by simply modifying the learning rate schedule.

Original: More importantly, the[Figure] performance gain by retraining can be remarkable: the accuracy loss can drop to zero and in some cases achieved
Revised: More importantly, the performance gain by retraining can be remarkable: the accuracy loss can drop to zero and in some cases better accuracy than baseline models can be achieved.

Original: To analyze the effect of retraining a pruned network, we based on learning rate rewinding (Renda et al., 2020) and experiment with different retraining settings.
Revised: To analyze the effect of retraining a pruned network, we based on learning rate rewinding (Renda et al., 2020) and experiment with different retraining settings.

Original: Although in the previous work, Renda et al.
Revised: Although in the previous work, Renda et al.

Original: (2020) demonstrated the efﬁcacy of learning rate rewinding across datasets and pruning criteria, there is a lack of understanding of the actual reason behind the success of this technique.
Revised: (2020) demonstrated the efﬁcacy of learning rate rewinding across datasets and pruning criteria, there is a lack of understanding of the actual reason behind the success of this technique.

Original: Here we hypothesize that the initial pruned network is a suboptimal solution, staying in a local minima.
Revised: Here we hypothesize that the initial pruned network is a suboptimal solution, staying in a local minima.

Original: Learning rate rewinding succeeds because it uses a large learning rate to encourage the pruned networks to converge to another, supposingly better, local minima.
Revised: Learning rate rewinding succeeds because it uses a large learning rate to encourage the pruned networks to converge to another, supposingly better, local minima.

Original: Our experiment setups are as follows.
Revised: Our experiment setups are as follows.

Original: Retraining techniques.
Revised: Retraining techniques.

Original: To verify this conjecture empirically, we conduct experiments with different learning rate schedules including learning rate rewinding (Renda et al., 2020) while varying pruning algorithms, network architectures and datasets.
Revised: To verify this conjecture empirically, we conduct experiments with different learning rate schedules including learning rate rewinding (Renda et al., 2020) while varying pruning algorithms, network architectures and datasets.

Original: In this work, we consider the following retraining techniques:
Revised: In this work, we consider the following retraining techniques:

Original: F INE - TUNING (FT) Fine-tuning is the most common retraining techniques (Han et al., 2015; Li et al., 2016; Liu et al., 2019).
Revised: F INE - TUNING (FT) Fine-tuning is the most common retraining techniques (Han et al., 2015; Li et al., 2016; Liu et al., 2019).

Original: In this approach, we continue train the pruned networks for t epochs with the last (smallest) learning rate of original training.
Revised: In this approach, we continue train the pruned networks for t epochs with the last (smallest) learning rate of original training.

Original: L EARNING R ATE R EWINDING (LRW) Renda et al.
Revised: L EARNING R ATE R EWINDING (LRW) Renda et al.

Original: (2020) propose to reuse the learning rate schedule of the original training when retraining pruned networks.
Revised: (2020) propose to reuse the learning rate schedule of the original training when retraining pruned networks.

Original: Speciﬁcally, when retraining for t epochs, we reuse the learning rate schedule from the previous t epochs, i e., rewinding.
Revised: Speciﬁcally, when retraining for t epochs, we reuse the learning rate schedule from the previous t epochs, i.e., rewinding.

Original: S CALED L EARNING R ATE R ESTARTING (SLR):
Revised: S CALED L EARNING R ATE R ESTARTING (SLR):

Original: In this approach, we employ the learning rate schedule that is proportionally identical to the standard training.
Revised: In this approach, we employ the learning rate schedule that is proportionally identical to the standard training.

Original: For example, the learning rate is dropped by a factor of 10 × at 50% and 75% of retraining epochs on CIFAR, which is akin to original training learning rate adjustment.
Revised: For example, the learning rate is dropped by a factor of 10 × at 50% and 75% of retraining epochs on CIFAR, which is akin to original training learning rate adjustment.

Original: The original learning rate schedule can be found in Appendix A. 4.
Revised: The original learning rate schedule can be found in Appendix A. 4.

Original: C YCLIC L EARNING R ATE R ESTARTING
Revised: C YCLIC L EARNING R ATE R ESTARTING

Original: (CLR): Instead of using stepwise learning rate schedule as scaled learning rate restarting , we leverage the 1-cycle (Smith & Topin, 2019), which is shown to give faster convergence speed than conventional approaches. 
Revised: (CLR): Instead of using stepwise learning rate schedule as scaled learning rate restarting , we leverage the 1-cycle (Smith & Topin, 2019), which is shown to give faster convergence speed than conventional approaches. [Figure] [Figure] on ImageNet with (cid:96) 1 -norm ﬁlters pruning (Li et al., 2016) (c).

Original: Note that for the last two strategies (SLR and CLR), we warmup the learning rate for 10% of total retraining budget.
Revised: Note that for the last two strategies (SLR and CLR), we warmup the learning rate for 10% of total retraining budget.

Original: For simplicity, we always use the largest learning rate of the original training for learning rate restarting .
Revised: For simplicity, we always use the largest learning rate of the original training for learning rate restarting .

Original: For CLR and SLR, the learning rate is increased from the smallest learning rate of original training to the largest one according to cosine function.
Revised: Speciﬁcally, the learning rate is increased from the smallest learning rate of original training to the largest one according to cosine function.

Original: The detailed learning rate schedule of each techniques is depicted in Figure 1. 
Revised: The detailed learning rate schedule of each technique is depicted in Figure 1. See also Appendix F for the choice of warmup epochs.

Original: Pruning algorithms.
Revised: Pruning algorithms

Original: We consider the following dimensions in our experiments.
Revised: We consider the following dimensions in our experiments.

Original: For pruning methods, we use (cid:96) 1 -norm ﬁlters pruning (PFEC)
Revised: For pruning methods, we use (cid:96) 1 -norm ﬁlters pruning (PFEC)

Original: (Li et al., 2016) and (global) magnitude-based weights pruning (MWP) (Han et al., 2015) and evaluate them on the CIFAR-10, CIFAR-100 and ImageNet dataset.We examine both variations of pruning [Figure] while varying retraining budgets.
Revised: (Li et al., 2016) and (global) magnitude-based weights pruning (MWP) (Han et al., 2015) and evaluate them on the CIFAR-10, CIFAR-100 and ImageNet dataset.

Original: As can be seen, learning rate schedule matters.

Original: Schedules that employ large learning rates (LLRW, SLR, CLR) are signiﬁcantly better than ﬁne-tuning.

Original: Among them, CLR performs best in most cases.

Original: [Figure] namely one-shot pruning and iterative pruning when comparing the proposed retraining techniques.
Revised: We examine both variations of pruning namely one-shot pruning and iterative pruning when comparing the proposed retraining techniques.

Original: Furthermore, we also experiment the CLR schedule on HRank (Lin et al., 2020), Taylor Pruning (TP) (Molchanov et al., 2019) and Soft Filter Pruning (SFP) (He et al., 2018).
Revised: Furthermore, we also experiment the CLR schedule on HRank (Lin et al., 2020a), Taylor Pruning (TP) (Molchanov et al., 2019) and Soft Filter Pruning (SFP) (He et al., 2018).

Original: Our implementation and hyperparameters of (cid:96) 1 -norm  pruning and magnitude weight pruning are based on the public implementation of Liu et al.
Revised: Our implementation and hyperparameters of (cid:96) 1 -norm ﬁlters pruning and magnitude weight pruning are based on the public implementation of Liu et al.

Original: (2019), which is shown to obtain comparable results with the original works.
Revised: (2019), which is shown to obtain comparable results with the original works.

Original: For remaining algorithms, we use ofﬁcial implementations with hyperparameters speciﬁed according to their papers.
Revised: For remaining algorithms, we use ofﬁcial implementations with hyperparameters speciﬁed according to their papers.

Original: The detailed conﬁgurations of training and ﬁne-tuning is provided in the Appendix A.2 for interested readers.
Revised: The detailed conﬁgurations of training and ﬁne-tuning is provided in the Appendix B for interested readers.

Original: Evaluation.
Revised: Evaluation.

Original: For CIFAR-10 and CIFAR-100, we run each experiment three times and report “ mean ± std ”.
Revised: For CIFAR-10 and CIFAR-100, we run each experiment three times and report “ mean ± std ”.

Original: For ImageNet, we run each experiment once.
Revised: For ImageNet, we run each experiment once.

Original: These settings are kept consistently across architectures, pruning algorithms, retraining techniques, and ablation studies unless otherwise stated.
Revised: These settings are kept consistently across architectures, pruning algorithms, retraining techniques, and ablation studies unless otherwise stated.

Original:  A CASE STUDY ON R ETRAINING IN N ETWORK P RUNING 3. R ETRAINING COST AND P ERFORMANCE T RADE - OFF
Revised:  A CASE STUDY ON R ETRAINING IN N ETWORK P RUNING 3. R ETRAINING COST AND P ERFORMANCE T RADE - OFF

Original: We ﬁrst investigate the performance of retraining techniques while varying the retraining budget.
Revised: We ﬁrst investigate the performance of retraining techniques while varying the retraining budget.

Original: Figure 2 illustrates the results of (cid:96) 1 -norm ﬁlters pruning (PFEC)
Revised: Figure 2 illustrates the results of (cid:96) 1 -norm ﬁlters pruning (PFEC)

Original: (Li et al., 2016) with different retraining techniques on CIFAR-10.
Revised: (Li et al., 2016) with different retraining techniques on CIFAR-10.

Original: We can observe that, larger values of learning rate always attain higher performance than ﬁne-tuning regardless of number of retraining epochs. 
Revised: We can observe that, larger values of learning rate always attain higher performance than ﬁne-tuning regardless of number of retraining epochs. Furthermore, with

Original: Furthermore, with some additional retraining budgets, e.g., 80 to 120 epochs compare to 40 epochs in original works, we can attain much higher compression ratio with almost no accuracy drop (see Figure 6 in Appendix). Figure 3 reports the accuracy of ResNet-56 pruned with magnitude-based weights pruning (Han et al., 2015) in a one-shot manner. We study the impact of different retraining schedules on low and high sparsity i.e. 50% and 90% respectively. It can be observed that under both setting learning rate restarting approaches consistently outperform ﬁne-tuning across number of retraining epochs.
Revised: 6 Compression Ratio −

Original: To verify the effectiveness of CLR on large-scale datasets, we conduct experiment using (cid:96) 1 -norm ﬁlters pruning with ResNet-34 on ImageNet in Figure 3(c). We ﬁnd that CLR achieve lower performance than ﬁne-tuning for low epochs. This is expected phenomenon since (Huang et al., 2017)
Revised: [Figure]

Original: 6 Compression Ratio −− ∆ A cc u r a c y
Revised: − [Figure] ∆ A cc u r a c y

Original: ResNet-56, PFEC FT
Revised: ResNet-56, PFEC FT

Original: LRW
Revised: LRW

Original: SLR
Revised: SLR

Original: CLR (a)  6 Compression Ratio −−  ∆
Revised: CLR (a)  6 Compression Ratio −−  ∆

Original: A cc u r a c y
Revised: A cc u r a c y

Original: ResNet-110, PFEC FT
Revised: ResNet-110, PFEC FT

Original: LRW
Revised: LRW

Original: SLR
Revised: SLR

Original: CLR (b) 0 .0 .0 .0 .
Revised: CLR (b) 0 .0 .0 .0 .

Original: 8 Sparsity − ∆ A cc u r a c y
Revised: 8 Sparsity − ∆ A cc u r a c y

Original: ResNet-50, ImageNet FT
Revised: ResNet-50, ImageNet FT

Revised: CLR (c) some additional retraining budgets, e.g., 80 to 120 epochs compare to 40 epochs in original works, we can attain much higher compression ratio with almost no accuracy drop (see Figure 5 in Appendix).

Revised: Figure 3 reports the accuracy of ResNet-56 pruned with magnitude-based weights pruning (Han et al., 2015) in a one-shot manner.

Revised: We study the impact of different retraining schedules on low and high sparsity i.e. 50% and 90% respectively.

Original:  CLR (c) also found that learning rate restarting on ImageNet usually requires training for 45 epochs while it only needs 40 epochs on CIFAR-10.
Revised: It can be observed that under both setting learning rate restarting consistently outperforms ﬁne-tuning across number of retraining epochs. To verify the effectiveness of CLR on large-scale datasets, we conduct experiment using (cid:96) 1 -norm ﬁlters pruning with ResNet-34 on ImageNet in Figure 3(c). We ﬁnd that CLR achieve lower performance than ﬁne-tuning for low epochs. This is expected phenomenon since (Huang et al., 2017) also found that learning rate restarting on ImageNet usually requires training for 45 epochs while it only needs 40 epochs on CIFAR-10.

Original: When retraining with number of epochs higher than 45 , CLR can reach higher accuracy than ﬁne-tuning.
Revised: When retraining with number of epochs higher than 45 , CLR can reach higher accuracy than ﬁne-tuning.

Original: Thus, the value of LRW can be a good heuristic to choose the learning rate for retraining after pruning.
Revised: Thus, the value of LRW can be a good heuristic to choose the learning rate for retraining after pruning.

Original: A concrete example is that we achieve signiﬁcant gains with Taylor pruning when retraining for only 25 epochs as shown in Table 2.
Revised: A concrete example is that we achieve signiﬁcant gains with Taylor pruning when retraining for only 25 epochs as shown in Table 8.

Original: 3. M ODEL S IZE AND P ERFORMANCE T RADE - OFF
Revised: 3. M ODEL S IZE AND P ERFORMANCE T RADE - OFF

Original: In this section, we investigate the trade-off of between model size and performance of compact models under a ﬁxed retraining budgets.
Revised: In this section, we investigate the trade-off of between model size and performance of compact models under a ﬁxed retraining budgets.

Original: Renda et al.
Revised: Renda et al.

Original: (2020) found that learning rate rewinding usually saturate at half of original training, thus, we perform on retraining for 80 epochs on CIFARand 45 epochs on ImageNet.
Revised: (2020) found that learning rate rewinding usually saturate at half of original training, thus, we perform on retraining for 80 epochs on CIFARand 45 epochs on ImageNet.

Original: We ﬁrst experiment with iterative (cid:96) 1 -norm  pruning on CIFAR-10 and report the results in Figure 4(a,b), we can observe that SLR and CLR also perform comparable or better than LRW in this setting.
Revised: We ﬁrst experiment with iterative (cid:96) 1 -norm ﬁlters pruning on CIFAR-10 and report the results in Figure 4(a,b), we can observe that SLR and CLR also perform comparable or better than LRW in this setting.

Original: We then conduct experiments with iterative unstructured pruning on ImageNet with ResNet-50 to verify the superior of CLR compared to ﬁne-tuning in Figure 4(c), we ﬁnd that MWP with CLR signiﬁcantly outperform ﬁne-tuning and even increase the accuracy of the network until 80% sparsity (of convolutional layers).
Revised: We then conduct experiments with iterative unstructured pruning on ImageNet with ResNet-50 to verify the superior of CLR compared to ﬁne-tuning in Figure 4(c), we ﬁnd that MWP with CLR signiﬁcantly outperform ﬁne-tuning and even increase the accuracy of the network until 80% sparsity (of convolutional layers).

Original: In our experiments, CLR usually reachs slightly higher accuracy than LRW and SLR.
Revised: In our experiments, CLR usually reaches slightly higher accuracy than LRW and SLR.

Original: A possible explanation is that 1-cycle schedule gives better convergence speed than conventional schedules (Smith & Topin, 2019), thus, bringing much better results when employing “low” retraining budgets.Discussion.
Revised: A possible explanation is that 1-cycle schedule gives better convergence speed than conventional schedules (Smith & Topin, 2019), thus, bringing much better results when employing “low” retraining budgets.

Original: Our empirical results suggest that practitioners should employ retraining with large learning rate schedules as an alternative technique for ﬁne-tuning to obtain compact models with better performance after network pruning.

Original: In our results, cyclic learning rate restarting (CLR) is slightly more efﬁcient than scaled learning rate restarting (SLR) and learning rate rewinding (LRW).

Original: 3. E VALUATION WITH OTHER PRUNING ALGORITHMS
Revised: 3. E VALUATION WITH O THER P RUNING A LGORITHMS

Original: We investigate whether these results hold with other pruning algorithms than simple norm-based pruning.
Revised: We investigate whether the effectiveness of learning rate restarting holds with other pruning algorithms than simple norm-based pruning.

Original: As the performance of larger learning rate schedules such as LRW, SLR, and CLR are rather similar, we select CLR for use in this experiment.
Revised: As the performance of larger learning rate schedules such as LRW, SLR, and CLR are rather similar, we select CLR for use in this experiment.

Original: Particularly, we examine CLR withHRank (Lin et al., 2020), Taylor pruning (Molchanov et al., 2019) and SFP (He et al., 2018).
Revised: Particularly, we examine CLR with SFP (He et al., 2018) in this section..

Original: Soft Filters Pruning (He et al., 2018).
Revised: Soft Filters Pruning (He et al., 2018).

Original: Previous sections demonstrate that the success of large learning rate hold for a wide range of pruning algorithms, network architectures and datasets.
Revised: Previous sections demonstrate that the success of large learning rate hold for a wide range of pruning algorithms, network architectures and datasets.

Original: However, we note that all examined pruning algorithms followed the pruning-retraining paradigm, i.e., it is necessary to retrain the pruned model.
Revised: However, we note that all examined pruning algorithms followed the pruning-retraining paradigm, i.e., it is necessary to retrain the pruned model.

Original: In this part, we investigate regularizing where we  gradually, softly prune the network during the course of training.
Revised: In this part, we investigate regularizing where we [Table] gradually, softly prune the network during the course of training.

Original: In contrast to pruning-retraining , the latter approach does not require further training after ﬁnal pruning.
Revised: In contrast to pruning-retraining , the latter approach does not require further training after ﬁnal pruning.

Original: Being that said, we still make a case study to verify if retraining does improve the performance of models pruned with this type of algorithms.
Revised: That being said, we still make a case study to verify if retraining does improve the performance of models pruned with this type of algorithms.

Original: [Table] All columns have same meaning with corresponding columns in Table 6. [Table] To make a fair comparison between ﬁne-tuning and no ﬁne-tuning, we randomly split the conventional training set of CIFAR-10/CIFAR-100 (including 50000 images) to train ( 90% images of total set) and val ( 10% remaining images) set and we then report the result of best-validation models on “standard” test set (including 5000 images).
Revised: To make a fair comparison between ﬁne-tuning and no ﬁne-tuning, we randomly split the conventional training set of CIFAR-10/CIFAR-100 (including 50000 images) to train ( 90% images of total set) and val ( 10% remaining images) set and we then report the result of best-validation models on “standard” test set (including 5000 images).

Original: We report the performance of pruned network without ﬁne-tuning, with ﬁne-tuning and with learning rate restarting (CLR) in Table 1.
Revised: We report the performance of pruned network without ﬁne-tuning, with ﬁne-tuning and with learning rate restarting (CLR) in Table 1.

Original: We ﬁnd that applying ﬁne-tuning on these model usually give very fast convergences speed (roughly in ﬁrst 20 epochs).
Revised: We ﬁnd that applying ﬁne-tuning on these model usually give very fast convergences speed (roughly in ﬁrst 20 epochs).

Original: However, ﬁne-tuned models exhibit negligible gain then pruned models without ﬁne-tuning.
Revised: However, ﬁne-tuned models exhibit negligible gain then pruned models without ﬁne-tuning.

Original: In contrast, CLR signiﬁcantly improves the performance even with small retraining budgets.
Revised: In contrast, CLR signiﬁcantly improves the performance even with small retraining budgets.

Original:  Taylor Pruning (Molchanov et al., 2019). In this method, the authors estimated the contributions of each ﬁlter to the ﬁnal performance by utilizing the ﬁrst- and second-order Taylor expansion and performing iterative pruning.
Revised: Interested readers can ﬁnd our results with other pruning algorithms namely HRank (Lin et al., 2020a) and Taylor pruning (Molchanov et al., 2019) in the Appendix C. 3. S CALING O RIGINAL T RAINING

Original: In the original implementation, the authors ﬁne tuned the network with a small learning rate ( 0 . 001 ) for 25 epochs on ImageNet.
Revised: Liu et al.

Original: In our implementation, we opt to evaluate the accuracy of network retrained with learning rate restarting (CLR) with a relatively small budget. Therefore, we keep the retraining budget the same and only increase the learning rate to 0 .
Revised: present empirical evidence that 3-stage pipeline of structured pruning is inferior to training compact models from scratch if training budgets of compact models are scaled proportion to FLOPs ratio between original and pruned model. Following Evci et al.

Original: Table 2 shows the accuracy of ResNet-50 on ImageNet when pruning 72% of network weights.
Revised: ; Gale et al. , we conduct experiments to compare this strong baseline with (cid:96) 1 -norm ﬁlters pruning while employing CLR on CIFAR-10, CIFAR-100 and ImageNet.

Original: It can be seen that for a small retraining budget, employing learning rate restarting also gives signiﬁcant gain in performance. HRank (Lin et al., 2020).
Revised: Speciﬁcally, we compare pruned networks retrained with ﬁne-tuning and CLR mechanism with Scratch-B(udget) and Scratch-E(pochs) which train pruned networks from scratch with the same computational budget and epochs respectively 1 .

Original: Inspired by the discovery that average rank of feature maps of a ﬁlter is consistent, the authors suggested to iteratively remove low-rank feature maps that contain less information.
Revised: When evaluating with CIFAR-10 and CIFAR-100, we found that the validation accuracies of ScratchB, Scratch-E, and CLR are very similar to each other, and therefore, the comparisons and ranking could be very sensitive to the variance of the accuracy. To reduce variance, we employ the following training scheme.

Original: Particularly, we iteratively remove ﬁlters in each layer and then retrain the network for 30 epochs each time. We examine the performance of pruned modelswith HRank on CIFAR while employing CLR at each ﬁne-tuning step.
Revised: First, we re-run each experiment 5 times on CIFAR-10 and CIFAR-100 (more than previous experiments to reduce noise). Second, we randomly (and independently between each run) split the original training set of CIFAR-10 to training (90% images of total set) and validation set (10% images of total set) and report the performance of best validation models on original test set . Note that for ImageNet, we re-run the experiments 3 times.

Original: Due to the heavily expensive computational cost for retraining, we only run each model once for HRank. In the original version of this method, the authors used the learning rate of 0 . 01 and drop by 10 × at epochs 5 and 10 .
Revised: We found that for ImageNet, the variance is negligible and thus we simply report the best validation accuracy. For retraining, we adopt the same retraining budgets as in (Liu et al., 2019; Li et al., 2016) ( 40 and 20 epochs for CIFAR-10 and ImageNet respectively). Since the total number of retraining epochs is

Original: In our implementation, we restart the learning rate to 0 . 
Revised: [Table] limited, for ImageNet, we restart the learning rate to 0 . 01 (instead of 0 .

Original: 1 and apply the CLR schedule.
Revised: 1 in the previous section) to guarantee the convergence of the models.

Original: Thequantitative results  are shown in Table 6 (Appendix).
Revised: The results of pruned network via PFEC are shown in Table 2.

Original: In our experiments, CLR also attains comparable or better results than proposed retraining methods in original work. [Figure]
Revised: We can observe that for CIFAR-10, the results of ﬁne-tuning and Scratch-B/E are aligned with the prior work of Liu et al. (2019) while CLR achieve slightly better results than Scratch-E and comparable with

Original: Method
Revised: Scratch-B.

Original: [Figure]
Revised: However, PFEC+CLR signiﬁcantly outperform both ﬁne-tuning and Scratch-B by a large margin. Discussion. Our empirical results suggest that practitioners should employ retraining with large learning rate schedules as an alternative technique for ﬁne-tuning to obtain compact models with better performance after network pruning.

Original: [Table]
Revised: In our results, cyclic learning rate restarting (CLR) is slightly more efﬁcient than scaled learning rate restarting (SLR) and learning rate rewinding (LRW).

Original:  R ANDOM P RUNING WITH L EARNING R ATE R ESTARTING
Revised:  I NTERPLAY OF P RUNING A LGORITHMS AND R ETRAINING S CHEMES

Original: Section 3 suggests that learning rate schedule can signiﬁcantly improve the performance of several pruning algorithms.
Revised: Section 3 suggests that learning rate schedule can signiﬁcantly improve the performance of several pruning algorithms.

Original: We notice that there are notable differents between settings of implementation of network pruning beside retraining such as the training scheme of baseline models.
Revised: We notice that there are notable differences between settings of implementation of network pruning especially in retraining phase.

Original: In this section, we show that the difference between implementation could easily lead to misleading results and unfair comparisons between pruning algorithms
Revised: In this section, we show that the difference between implementation could easily lead to misleading results and unfair comparisons between pruning algorithms. 4. A S TRONG B ASELINE : (cid:96) 1 - NORM F ILTERS P RUNING WITH CLR

Original: [Table]
Revised: In this section, we demonstrate that even with same retraining budgets, utilizing simple CLR with (cid:96) 1 -norm ﬁlters pruning can achive comparable or exceed the performance of more sophisticated saliency metrics without meticulous hyperparameters searching. The implementation details are as follows. Our baseline For pruning, we adopt the pretrained of Torchvision and apply (cid:96) 1 -norm Filters Pruning on these models.

Revised: In our implementation, the number of removed ﬁlters in each block are approximately equal so that the ﬁnal pruned models have similar number of parameters with the compared one.

Revised: Thus, if any, this setting should favor other approaches involving laborious sensitive analysis.

Original: Firstly
Revised: For retraining, we apply the CLR learning rate schedule while choosing the maximum value of learning rate according to learning rate rewinding . For pruning algorithms that only retrain for small number of epochs (e.g. 25 epochs in case of Taylor Pruning) we also restart the learning rate value to the slightly higher value of 0 . 01 . Generative Adversarial Learning (GAL) Lin et al.

Revised: (2019) suggest to leverage generative adversarial learning, which learns a sparse soft mask in a label-free and an end-to-end manner, to effectively

Original: [Table] 
Revised: [Table] [Table]

Original: let us investigate the simple (cid:96) 1 -norm ﬁlters pruning on small datasets.
Revised: solve the optimization problem. The original work retrain the pruned network for 30 epochs with batch size of 32 and initial learning rate of 0 . 01 (which is decayed by factor of 10 every 10 epochs). Thus

Original: Figure 5 illustrates the performance of networks pruned with PFEC and random pruning on CIFAR-10 and CIFARwhen retraining with 40 epochs – the  same setting used by Li et al.
Revised: [Table] we use CLR with learning rate of 0 . 01 and batch size of 32 in our experiments when comparing with GAL . Table 3 shows the results of PFEC + CLR when retrained under same settings with GAL.

Original: We can see that randomly pruned networks consistently achieve superior performance than methodically pruned networks (ﬁne-tuned with standard learning rateschedule) in terms of accuracy.
Revised: We can see that PFEC+CLR can create a very strong baseline compare with GAL. HRankPlus An extension of HRank (Lin et al., 2020a) which is better than HRank in both retraining efﬁciency and performance 2 . Inspired by the discovery that average rank of feature maps of a ﬁlter is consistent, the authors suggested to iteratively remove low-rank feature maps that contain less information. In the original implementation, the authors retrain the pruned networks for 90 epochs with initial learning rate of 0 . 1 and also utilize label smoothing.

Original: However, random pruning obtain lower accuracy than PFEC when using identical retraining techniques.
Revised: For simplicity, we do not use label smoothing in our implementation. As shown in Table 12, PFEC+CLR also attain similar results with

Original: To understand if this result generalizes to sophisticated pruning algoirithms and large-scale datasets, we also conduct experiments with HRank and ImageNet in Table 3.
Revised: HRankPlus approach. Additional results on CIFAR-10 and Discrimination-aware Channel Pruning (Zhuang et al., 2018), Taylor Pruning (Molchanov et al., 2019), Provable Filters Pruning (Liebenwein et al., 2020) are reported in Appendix D for interested readers.

Original: Particularly, we retrain these networks with largest learning rate during training, we use retrain budgets of 40 and 45 epochs for
Revised: 4. R ANDOM P RUNING WITH L EARNING R ATE R ESTARTING

Original: ImageNet and CIFAR-10/CIFAR-100 – the smallest number of epochs for networks to converge after restarting (Huang et al., 2017).
Revised: In last subsection, we demonstrated that slight modiﬁcation in retraining of simple (cid:96) 1 -norm Filters

Original: We can observe that random pruning still can outperform methodical pruning on many occasions even with same retraining budgets and only alter the learning rate schedule.
Revised: Pruning can exceed the perfomance of much more sophisticated pruning algorithms. In this subsection, we investigate the interplay between pruning saliency metrics and retraining conﬁgurations by comparing accuracy of randomly pruned networks with the original performance of methodically pruned networks. We directly adopt the original implementation from the authors of Taylor Pruning, HRankPlus for comparisons.

Original: Furthermore, we show that by naively neglecting retraining conﬁgurations (i.e. budget), we can also reach state-of-the-art results on CIFAR-10 with PFEC Li et al. 
Revised: For PFEC and MWP we make use of the implementation of Liu et al. We selected these works because they do not utilize a similar learning rate value as LRW in the original implementation, i.e., the authors applied a smaller value then the heuristic of LRW. The detailed training recipe of the original works can be found in Appendix E.1.

Original: Speciﬁcally, we prune two
Revised: To make the randomly pruned network have the same structure as the methodically pruned networks, i.e., equal number of ﬁlters per layer, we propose to replace the importance score of each neuron estimated by these methods by a uniformly random score value. For retraining randomly pruned networks, we adopt the same principle at described in Section 4.1. Table 5 presents novel results of performance of random pruning + CLR with corresponding results of considered pruning algorithms. It is worth pointing out that we report the original results of all

Original: [Table]
Revised: [Table]

Original: “standard” models namely ResNet-56 and ResNet-110 and retrain with CLR for 152 epochs
Revised: methodical pruning are taken directly from their papers respectively. In our experiments, Random Pruning + CLR could surpass the performance of sophisticated saliency metrics even on large-scale and challenging dataset such as ImageNet with only minimal change in learning rate schedule and initial learning rate value

Original: [Table]
Revised: [Table]

Original: and then we compare with more sophisticated pruning algorithms in Table [Table]
Revised: Comprehensive results of random pruning with different compression ratio for (cid:96) 1 -norm Filters

Original:  These results suggest that retraining techniques, e [Table] g., learning rate schedule and retraining budget, play a pivotal role to ﬁnal performance.
Revised: Pruning and MWP is reported in Appendix E.2. These results suggest that retraining techniques, e.g., learning rate restarting and learning rate schedule, play a pivotal role to ﬁnal performance.

Original: Thus, in order to perform fair comparison of different methods, one should be cautious of this seemingly subtle detail.
Revised: Thus, in order to perform fair comparison of different methods, one should be cautious of this seemingly subtle detail.

Original: While it is unclear to set up a fair comparison between pruning algorithms that belong to different categories, e.g., retraining vs. regularization, iterative vs. oneshot pruning, we advocate standardizing  retraining conﬁgurations for each algorithm group.
Revised: While it is unclear to set up a fair comparison between pruning algorithms that belong to different categories, e.g., pruning before/during/after training, iterative vs. oneshot pruning, we advocate standardizing (a set of) retraining conﬁgurations for each algorithm group or thorough hyperparamters searching to ﬁnd the best conﬁguration of each pruning method when evaluating different algorithms.

Original:  D ISCUSSION AND C ONCLUSION
Revised:  D ISCUSSION AND C ONCLUSION

Original: In this work we conducted extensive experiments to show that learning rates do matter in achieving good performance of pruned neural networks.
Revised: In this work, we conducted extensive experiments to show that learning rates do matter in achieving good performance of pruned neural networks.

Original: We concluded that compared to traditional ﬁne-tuning, learning rate rewinding is an efﬁcient way to retrain pruned networks to recover performance drop due to pruning.
Revised: We concluded that compared to traditional ﬁnetuning, learning rate restarting in general is an efﬁcient way to retrain pruned networks to recover performance drop due to pruning.

Original: The success of learning rate rewinding is accounted by the use of large learning rates in the retraining.
Revised: The success of learning rate rewinding is accounted by the use of large learning rates in the retraining.

Original: We believe that these ﬁndings help raise awareness of proper use of learning rate schedule when desiging pruning algorithms, standardizing empirical experiments and allowing fair comparisons.
Revised: We believe that these ﬁndings help raise awareness of proper use of learning rate schedule when desiging pruning algorithms, standardizing empirical experiments and allowing fair comparisons.

Original: Our takeaway message is:
Revised: Our takeaway message is:

Original: Pruning algorithms should be compared in the same retraining conﬁgurations.
Revised: Pruning algorithms should be compared in the same retraining conﬁgurations.

Original: Our work is not without limitations.
Revised: Our work is not without limitations.

Original: So far we investigated with hyperparameters identical to those of the original implementations of the pruning algorithms and only experiment with different learning rate schedules.
Revised: So far we investigated with hyperparameters identical to those of the original implementations of the pruning algorithms, and only experiment with different learning rate schedules.

Original: We also limited our experiments to the image classiﬁcation task in computer vision.
Revised: We also limited our experiments to the image classiﬁcation task in computer vision.

Original: Considering more datasets and other domains is a great research avenue for future works.
Revised: Considering more datasets and other domains is a great research avenue for future work.

Original: AA PPENDIX A. N ETWORK P RUNING F ORMULATION
Revised: A PPENDIX A N ETWORK P RUNING F ORMULATION

Original: Let us start with a formal deﬁnition of the network pruning problem.
Revised: Let us start with a formal deﬁnition of the network pruning problem.

Original: Let θ ∈ R D be the parameters of a (large) neural network that needs to be pruned, D be the dataset for training.
Revised: Let θ ∈ R D be the parameters of a (large) neural network that needs to be pruned, D be the dataset for training.

Original: The parameters θ is updated to minimize the loss function L ( θ ; D ) .
Revised: The parameters θ is updated to minimize the loss function L ( θ ; D ) .

Original: Deﬁne sparsity mask m ∈ { 0 , 1 } D indicating if a weight should be kept or removed.
Revised: Deﬁne sparsity mask m ∈ { 0 , 1 } D indicating if a weight should be kept or removed.

Original: Similar to Molchanov et al.
Revised: Similar to Molchanov et al.

Original: (2016), we can formulate the pruning algorithms as ﬁnding the optimal sparsity mask such that:
Revised: (2016), we can formulate the pruning algorithms as ﬁnding the optimal sparsity mask such that:

Original: [Equation]
Revised: [Equation]

Original: where (cid:12) denotes the Hadamard (element-wise) product, θ ∗ represents the (locally) optimal solution of θ .
Revised: where (cid:12) denotes the Hadamard (element-wise) product, θ ∗ represents the (locally) optimal solution of θ .

Original: An alternative approach is minimizing the different between ﬁnal output of two networks (Lin et al., 2019):
Revised: An alternative approach is minimizing the different between ﬁnal output of two networks (Lin et al., 2019):

Original: [Equation] 
Revised: [Equation] [Equation]

Original: where F : R D → R D (cid:48) is the function mapping input feature (i.e. images) to the ﬁnal output (before softmax).
Revised: where F : R D → R D (cid:48) is the function mapping input feature (i.e. images) to the ﬁnal output (before softmax).

Original: Theoretically speaking, as the pruning formulation only takes the loss function into account, and there is a caveat that pruning can result in a set of weights that belong to a bad local minima, making the network generalize poorly.
Revised: Theoretically speaking, as the pruning formulation only takes the loss function into account, and there is a caveat that pruning can result in a set of weights that belong to a bad local minima, making the network generalize poorly.

Original: In practice, it is common to observe such issues from the performance drop of the pruned model.
Revised: In practice, it is common to observe such issues from the performance drop of the pruned model.

Original: In such cases, a ﬁne-tuning process can be applied to further optimize the weights, mitigating the accuracy drop.
Revised: In such cases, a ﬁne-tuning process can be applied to further optimize the weights, mitigating the accuracy drop.

Original: However, as what ﬁne-tuning means, the weights might be only slightly adjusted without any guarantee about the ﬁnal network performance.
Revised: However, as what ﬁne-tuning means, the weights might be only slightly adjusted without any guarantee about the ﬁnal network performance.

Original: A. T RAINING CONFIGURATION
Revised: B T RAINING C ONFIGURATION

Original: The implementation of (cid:96) 1 -norm ﬁlters pruning (PFEC)
Revised: The implementation of (cid:96) 1 -norm ﬁlters pruning (PFEC)

Original: Li et al.
Revised: Li et al.

Original: and magnitude-based weights pruning (MWP) Han et al.
Revised: and magnitude-based weights pruning (MWP) Han et al.

Original: (2015) are adopted from the work of Liu et al.
Revised: (2015) are adopted from the work of Liu et al.

Original: The implementations of other algorithms are taken from the authors’ ofﬁcial repositories.
Revised: The implementations of other algorithms are taken from the authors’ ofﬁcial repositories.

Original: For simplicity, we adopt Pytorch’s pretrained models for ImageNet.
Revised: For simplicity [Table] we adopt Pytorch’s pretrained models for ImageNet.

Original: The unpruned models (used for (cid:96) 1 -norm ﬁlters pruning) are trained with below conﬁgurations.
Revised: The unpruned models (used for (cid:96) 1 -norm ﬁlters pruning) are trained with below conﬁgurations.

Original: [Table]
Revised: [Table]

Original: A. A DDITIONAL R ESULTS
Revised: B. A DDITIONAL R ESULTS

Original: In addition results in Figure 2, we conduct experiments to examine impact of different retraining techniques to networks pruned with low and high compression ratios.
Revised: In addition results in Figure [Table] we conduct experiments to examine impact of different retraining techniques to networks pruned with low and high compression ratios.

Original: The results are shown in Figure 
Revised: The results are shown in Figure 5. [Figure]

Revised: ∆ A cc u r a c y

Original: [Table]
Revised: [Figure]

Revised: FT

Original: [Figure]
Revised: [Figure]

Revised: Epochs

Original: [Table]
Revised: [Figure] c y VGG- [Figure]

Revised: 72% Sparsity

Revised: FT

Revised: LRW

Revised: SLR

Revised: CLR

Revised: [Table]

Revised: C E VALUATING CLR WITH O THER P RUNING A LGORITHMS

Revised: Taylor Pruning (Molchanov et al., 2019).

Revised: In this method, the authors estimated the contributions of each ﬁlter to the ﬁnal performance by utilizing the ﬁrst- and second-order Taylor expansion and performing iterative pruning.

Revised: In the original implementation, the authors ﬁne tuned the network with a small learning rate ( 0 . 001 ) for 25 epochs on ImageNet.

Revised: In our implementation, we opt to evaluate the accuracy of network retrained with learning rate restarting (CLR) with a relatively small budget.

Revised: Therefore, we keep the retraining budget the same and only increase the learning rate to 0 .

Revised: Table 8 shows the accuracy of ResNet-50 on ImageNet when pruning 72% of network weights.

Revised: It can be seen that for a small retraining budget, employing learning rate restarting also gives signiﬁcant gain in performance.

Revised: HRank (Lin et al., 2020a).

Revised: Inspired by the discovery that average rank of feature maps of a ﬁlter is consistent, the authors suggested to iteratively remove low-rank feature maps that contain less information.

Revised: Particularly, we iteratively remove ﬁlters in each layer and then retrain the network for 30 epochs each time.

Revised: We examine the performance of pruned models with HRank on CIFAR while employing CLR at each ﬁne-tuning step.

Revised: Due to the heavily expensive computational cost for retraining, we only run each model once for HRank.

Revised: In the original version of this method, the authors used the learning rate of 0 . 01 and drop by 10 × at epochs 5 and 10 .

Revised: In our implementation, we restart the learning rate to 0 .

Revised: 1 and apply the CLR schedule.

Revised: The quantitative results are shown in Table 7.

Revised: In our experiments, CLR also attains comparable or better results than proposed retraining methods in original work.

Revised: D S TRONG B ASELINES WITH (cid:96) 1 - NORM F ILTERS P RUNING D. CIFAR-

Revised: Furthermore, we show that by naively neglecting retraining conﬁgurations (i.e. budget), we can also reach state-of-the-art results on CIFAR-10 with PFEC Li et al.

Revised: Speciﬁcally, we prune two “standard” models namely ResNet-56 and ResNet-110 and retrain with CLR for 152 epochs, and then we compare with more sophisticated algorithms in Table 9.

Revised: [Table]

Revised: [Table]

Revised: D. I MAGE N ET

Revised: Taylor Pruning (Molchanov et al., 2019)

Revised: In this method, the authors estimated the contributions of each ﬁlter to the ﬁnal performance by utilizing the ﬁrst- and second-order Taylor expansion and performing iterative pruning.

Revised: Original work retrains pruned networks for 25 epochs with learning rate of 0 .

Revised: 001 with the exception for Taylor-FO-BN-56% .

Revised: Table 10 demonstrates the performance of PFEC+CLR compare with original works of Molchanov et al.

Revised: In most cases, PFEC+CLR attains higher or comparable accuracy with Taylor Pruning .

Revised: [Table]

Revised: Discrimination-aware Channel Pruning (Zhuang et al., 2018)

Revised: The authors introduce additional discrimination-aware losses into the network to increase the discriminative power of intermediate layers and then select the most discriminative channels for each layer by considering the additional loss and the reconstruction error

Revised: [Table]

Revised: We use the retraining budget of 60 epochs and set initial learning rate to 0 .

Revised: 01 same as Zhuang et al.

Revised: However, we employ the 1-cycle learning rate instead of stepwise learning rate as original work.

Revised: The detailed comparison is presented in Table 11.

Revised: [Table]

Revised: Provable Filters Pruning (Liebenwein et al., 2020)

Revised: This algorithm uses a small batch of input data points to assign a saliency score to each ﬁlter and constructs an importance sampling distribution where ﬁlters that highly affect the output are sampled with correspondingly high probability.

Revised: Original implementation of Liebenwein et al.

Revised: (2020) retrain the pruned network on ImageNet forepochs with standard learning rate schedule (drop learning rate by factor of 10 after 30 epochs) with batch size of 256 and weight decay of 0 . 0001 .

Revised: In our implementation, pruned networks are trained for 90 epochs with maximum learning rate of 0 . 1 while keeping the same conﬁgurations for all other hyperparameters as original work.

Revised: PFEC + CLR and Provable Filters

Revised: Pruning (PFP) on ImageNet.

Revised: The results of PFP are taken directly from original paper (Liebenwein et al., 2020).

Revised: [Table]

Revised: E R ANDOM P RUNING WITH CLR E. R ETRAINING C ONFIGURATION

Revised: We compare the conﬁguration between original implementation of aforementioned pruning methods and corresponding random pruning in Table 13.

Revised: Note that for HRankPlus, the original implementation employs different values of weight decay for each model: 0 . 005 , 0 . 006 , 0 .

Revised: 002 for VGG-16,

Revised: ResNet-56, ResNet-110, DenseNet-40 respectively.

Revised: We found that these value is relatively higher than “conventional” values (0.0001) making the models hard to converge after restarting.

Revised: Thus, we use weight decay of 0 .

Revised: 0005 through out experiments with HRankPlus.

Revised: In all our experiments, other details such as batch size, retraining budget are set to similar value of original implementation.

Revised: E. R ANDOM P RUNING WITH V ARIOUS C OMPRESSION R ATIO

Revised: (cid:96) 1 -norm Filters Pruning Figure 6 illustrates the performance of networks pruned with PFEC and random pruning on CIFAR-10 and CIFAR-100 when retraining with 40 epochs – the same setting used by Li et al.

Revised: We can see that randomly pruned networks consistently achieve superior

Revised: [Table]

Revised: [Figure]

Revised: performance than methodically pruned networks (ﬁne-tuned with standard learning rate schedule) in terms of accuracy.

Revised: However, random pruning obtain lower accuracy than PFEC when using identical retraining techniques.

Revised: Magnitude-based Weights

Revised: Pruning We extend the scope of experiment in Sec 4.2 to unstructured pruning and analyze performance of MWP with CLR.

Revised: Figure 7 represents the results of random pruning with CLR and methodically pruning while varying difference compression ratio in both iterative and oneshot pruning manner.

Revised: Speciﬁcally, we retrain the trimmed network for 40 epochs.

Revised: We can observe that though Random Pruning + CLR can achieve higher accuracy with low sparsity, the performance of randomly pruned network immensely reduced with the increasing of compression ratio.

Revised: F P ERFORMANCE OF CLR WITH D IFFERENT W ARMUP E POCHS

Revised: To avoid the laborious hyperparameters tuning process, we simply warm up the learning rate for

Revised: CLR and SLR for the ﬁrst 10% budget of retraining.

Revised: In this section, we investigate whether the performance of pruned network is sensitive to this hyperparameter.

Revised: Figure 8 presents the results of pruned networks obtained with ﬁne-tuning and CLR while varying number of warmup epochs namely 0 , 5 , 10 , 15% of total retraining.

Revised: We can observe that there is

Original: {+Compression Ratio − 1 .− 1 .− 0 .0 .0 .
+}
Revised: {+Compression Ratio − 1 .− 1 .− 0 .0 .0 .
+}

Revised: .∆

Revised: A cc u r a c y

Revised: ResNet-56, CIFAR-10 (Oneshot)

Revised: FT R-FT

Revised: CLR R-CLR 0 .0 .0 .0 .

Original: {+Compression Ratio − 1 .− 1 .− 0 .0 .0 .
+}
Revised: {+Compression Ratio − 1 .− 1 .− 0 .0 .0 .
+}

Revised: .∆

Revised: A cc u r a c y

Revised: ResNet-56, CIFAR-10 (Iterative)

Revised: FT CLR

Revised: R-CLR

Revised: [Figure]

Revised: -56 PFEC, 89% Sparsity

Revised: [Figure]

Revised: no signiﬁcant diversity in results of CLR with different number of warmup epochs and all of them exceed performance of ﬁne-tuning by a large margin.