Original: L OSSLESS F ILTER P RUNING VIA A DAPTIVE C LUSTERING FOR C ONVOLUTIONAL N EURAL N ETWORKS
Revised: L OSSLESS F ILTER P RUNING VIA A DAPTIVE C LUSTERING FOR C ONVOLUTIONAL N EURAL N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The filter pruning method introduces structural sparsity by removing selected filters and is thus particularly effective for reducing complexity.
Revised: The filter pruning method introduces structural sparsity by removing selected filters and is thus particularly effective for reducing complexity.

Original: However, previous works face two  limitations.
Revised: However, previous works face two common limitations.

Original: The pruned filters are prevented from contributing to the final results, which brings performance degradation especially when it comes to a large pruning rate.
Revised: The pruned filters are prevented from contributing to the final outputs, resulting in performance degradation, especially when it comes to a large pruning rate.

Original: To recover accuracy, the time-consuming finetuning step is necessary.
Revised: To recover accuracy, the time-consuming fine-tuning step is required.

Original: Thetime cost  and the need for training data make it hard to deploy in real-world scenarios.
Revised: The cost in time and the need for training data make it difficult to deploy in real-world scenarios.

Original: To address the aforementioned limitations, we propose a novel filter pruning method called Cluster Pruning (CP).
Revised: To address the aforementioned limitations, we propose a novel filter pruning method called Cluster Pruning (CP).

Original: Our CP reconstructs the redundant filters from the perspective of similarity and removes them equivalently using the proposed channel addition operation in a lossless manner.
Revised: Our CP reconstructs the redundant filters from the perspective of similarity and removes them equivalently using the proposed channel addition operation in a lossless manner.

Original: Pruning in such a way allows CP to get rid of the need for fine-tuning.
Revised: Pruning in such a way allows CP to preserve as many learned features as possible while getting rid of the need for fine-tuning.

Original: Specifically, each filter is first distinguished by clustering and then reconstructed as the centroid  it belongs to.
Revised: Specifically, each filter is first distinguished by clustering and then reconstructed as the centroid to which it belongs.

Original: Filters are further updated to eliminate the effect caused by mistakenly selected.
Revised: Filters are then updated to eliminate the effect caused by mistakenly selected.

Original: After convergence, CP can equivalently remove identical filters through the proposed channel addition operation.
Revised: After convergence, CP can equivalently remove identical filters through the proposed channel addition operation.

Original: The strategies for adjusting the pruning rate and the adaptive coefficient for clusteringfurther make our CP  smoother and more efficient.
Revised: The strategies for adjusting the pruning rate and the adaptive coefficient for clustering make our CP even smoother and more efficient.

Original: Extensive experiments on CIFAR-10 and ImageNet datasets show that our method can achieve the best trade-off  compared with other state-of-the-art algorithms.
Revised: Extensive experiments on CIFAR-10 and ImageNet datasets show that our method achieves the best trade-off between performance and complexity compared with other state-of-the-art algorithms.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The state-of-the-art performance of deep convolutional neural networks (CNNs) (Ren et al., 2015)
Revised: The state-of-the-art performance of deep convolutional neural networks (CNNs) (Ren et al., 2015)

Original: (Chen, 2015) (Lin et al., 2014) is based on deeper and wider architecture, which hinders the deployment in resources-limited devices due to its expensive computation cost and memory footprint.
Revised: (Chen, 2015) (Lin et al., 2014) is based on deeper and wider architecture, which hinders the deployment in resource-limited devices due to its expensive computation cost and memory footprint.

Original: As an effective compression technique, network pruning has attracted numerous attention.
Revised: As an effective compression technique, network pruning has attracted numerous attention.

Original: Unlike fine-grained weight pruning (Guo et al., 2016)
Revised: Unlike fine-grained weight pruning (Guo et al., 2016)

Original: (Han et al., 2015) (LeCun et al., 1989) which causes unstructured sparsity, filter pruning directly deletes the whole filter and is efficient in saving memory usage and computational cost on general platforms.
Revised: (Han et al., 2015) (LeCun et al., 1989) which causes unstructured sparsity, filter pruning directly deletes the whole filter and is efficient in saving memory usage and computational cost on general platforms.

Original: Numerous research efforts have been conducted to study how to define unimportant filters. ”
Revised: Numerous studies have been conducted to study how to define unimportant filters. ”

Original: Moresimilar-less-important” is a promising viewpoint to prune filters.
Revised: More-similarless-important” is a promising viewpoint for filter pruning.

Original: As shown in Fig. 1(a), the observation of these semantically similar feature map pairs,  which has also been pointed out by GhostNet (Han et al., 2020), indicates the prevalent redundancy on the feature level.
Revised: The observation of these semantically similar feature map pairs, as shown in Fig. 1(a), which has also been pointed out by GhostNet (Han et al., 2020), indicates the prevalent redundancy on the feature level.

Original: For further investigation, we visualize filters after PCA in Fig.
Revised: For further investigation, we visualize filters after PCA in Fig.

Original: 1(c), which demonstrates the phenomenon of cluster formation.
Revised: 1(c), which demonstrates the phenomenon of cluster formation.

Original: Considering such natural clustering property, we intuitively seek to prune redundancy based on similarity.
Revised: Given such natural clustering property, we intuitively seek to prune redundancy based on similarity.

Original: Compared with the commonly used ℓ p -norm criteria (Yu et al., 2021) (Lin et al., 2020c) (Li et al., 2020b), similarity-based criteria avoid the sensitivity to the  distributionof filters (He et al., 2019) which can lead to the erroneous removal of those relatively unimportant filters.
Revised: In comparison to the commonly used ℓ p -norm criteria (Yu et al., 2021) (Lin et al., 2020c) (Li et al., 2020b), similarity-based criteria avoid the sensitivity to the filter distribution (He et al., 2019) which can lead to the erroneous removal of those relatively unimportant filters.

Original: However, no matter which criteria the network is pruned under, previous works either directly remove filters or set them to zero (or masking).
Revised: However, no matter which criteria the network is pruned under, previous works either directly remove filters or set them to zero (or masking).

Original: Although these filters are considered trivial under certain criteria, they do not make zero contribution and still contain semantic information helpful for further processing.
Revised: Although these filters are considered trivial under certain criteria, they do not make zero contribution and still contain semantic information helpful for further processing.

Original: Pruning in such way will undoubtedly cause performance degradation, especially when it comes to a large pruning rate.
Revised: Pruning in such manner will undoubtedly degrade performance, especially at

Original: Moreover, even if the accuracy can be regained through

Original: [Figure]
Revised: [Figure]

Original:  fine-tuning, its high computational cost and the need for training data make it inefficient to carry out in realistic scenarios.
Revised: large pruning rates. Furthermore, even if the accuracy can be regained through fine-tuning, the high computational cost and the requirement for training data make it inefficient to carry out in realistic scenarios.

Original: In this paper, we propose  a novel filter pruning method, Cluster Pruning (CP) based on clustering.
Revised: In this paper, we propose Cluster Pruning (CP), a novel filter pruning method based on clustering.

Original: The core of our CP is that instead of removing or zeroing out unimportant filters, we intend to use the equivalence of the convolutional layer to remove filters without impact.
Revised: The core of our CP is that instead of removing or zeroing out unimportant filters, we intend to use the equivalence of the convolutional layer to remove filters without impact.

Original: To achieve this, we gradually replace filters with the value of cluster centroids while training, which still preserves their ability to extract features.
Revised: To accomplish this, we gradually replace filters with the value of cluster centroids while training, which still preserves their ability to extract features.

Original: After convergence, we can obtain a network whose filters are divided into several clusters and exactly identical within each cluster.
Revised: After convergence, we can obtain a network whose filters are divided into several clusters and exactly identical within each cluster.

Original: By taking advantage of the linear and combinational properties of convolutional layer, the proposed channel addition operation can discard all butkeep one filter of each cluster.
Revised: The proposed channel addition operation can discard all but one filter from each cluster by leveraging the linear and combinational properties of the convolutional layer.

Original: Our CP no longer requires the time-consuming finetuning process, since it is equivalent before and after pruning.
Revised: Our CP no longer requires the time-consuming fine-tuning process, since it is equivalent before and after pruning.

Original: As shown in Fig. 1(b), our method can remain comparable performance even at a large pruning rate, demonstrating our advantage in the trade-off between complexity and performance.
Revised: As shown in Fig. 1(b), our method can maintain comparable performance even at a large pruning rate, demonstrating our advantage in the trade-off between complexity and performance.

Original: To summarize, our main contributions are three-fold as follows:
Revised: To summarize, our main contributions are three-fold as follows:

Original: • We investigate the similarity between filters and propose our Cluster Pruning (CP) which prunes the network based on such similarity.
Revised: • We investigate the similarity between filters and propose our Cluster Pruning (CP), which prunes the network based on such similarity.

Original: We propose an adaptive coefficient for clustering which well preserves the extracted features.
Revised: We propose an adaptive coefficient for clustering which well preserves the extracted features.

Original: Various strategies for the pruning rate are designed to smooth the pruning process.
Revised: Various strategies for the pruning rate are designed to smooth the pruning process.

Original: • The channel addition operation is proposed to equivalently remove generated identical filters.
Revised: • The channel addition operation is proposed to equivalently remove generated identical filters.

Original: It allows CP to omit the time-consuming fine-tuning step and prevents from steep accuracy degradation especially for large pruning rates.
Revised: It enables CP to omit the time-consuming fine-tuning step and avoids steep accuracy degradation especially at large pruning rates.

Original: • Extensive experiments on CIFAR-10 and ImageNet datasets demonstrate that the proposed CP can achieve the best trade-off between complexity and performance compared with other state-of-the-art methods.
Revised: • Extensive experiments on CIFAR-10 and ImageNet datasets demonstrate that the proposed CP can achieve the best trade-off between complexity and performance compared with other state-of-the-art methods.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: In this section, we will give a comprehensive overview of different pruning methods, from the aspects of pruning structure, pruning criteria and pruning manner.
Revised: In this section, we will give a comprehensive overview of different pruning methods, from the aspects of pruning structure, pruning criteria and pruning manner.

Original: Pruning Structure.
Revised: Pruning Structure.

Original: Early works of network pruning concentrate on fine-grained granularities: weight-level (LeCun et al., 1989), vector-level (Wen et al., 2016) and kernel level (Li et al., 2017).
Revised: Early works of network pruning concentrate on fine-grained granularities: weight-level (LeCun et al., 1989), vector-level (Wen et al., 2016) and kernel level (Li et al., 2017).

Original: For architecture consistency, this type of non-structured method can only zeroize the unnecessary parameters rather than prune.
Revised: For architecture consistency, this type of non-structured method can only zeroize the unnecessary parameters rather than prune.

Original: The need of saving special coordinates for every weight makes it difficult to satisfy nowadays trillion-level models.
Revised: The need of saving special coordinates for every weight makes it difficult to satisfy nowadays trillion-level models.

Original: During inference, although the model sizes and the number of multiply-accumulate operations are dramatically decreased, the irregular structure of dense matrices requires additional computations and special hardware designs for acceleration.
Revised: During inference, although the model sizes and the number of multiply-accumulate operations are dramatically decreased, the irregular structure of dense matrices requires additional computations and special hardware designs for acceleration.

Original: Pruning Criteria.
Revised: Pruning Criteria.

Original: Selecting a proper criterion to identify filters which need to be pruned is a major point of network pruning.
Revised: Selecting a proper criterion to identify filters which need to be pruned is a major point of network pruning.

Original: (Li et al., 2017) prunes filters with small ℓ 1 -norm in each layer, while (Han et al., 2016)
Revised: (Li et al., 2017) prunes filters with small ℓ 1 -norm in each layer, while (Han et al., 2016)

Original: (Lin et al., 2018) are based on ℓ 2 -norm.
Revised: (Lin et al., 2018) are based on ℓ 2 -norm.

Original: (Molchanov et al., 2017) uses the Taylor series to estimate the loss change after each filter’s removal and prune the filters that cause minimal training loss change.
Revised: (Molchanov et al., 2017) uses the Taylor series to estimate the loss change after each filter’s removal and prune the filters that cause minimal training loss change.

Original: Network slimming (Liu et al., 2017) applies LASSO on the scaling factors of BN, by setting the BN scaling factor to zero, channel-wise pruning is enabled.
Revised: Network slimming (Liu et al., 2017) applies LASSO on the scaling factors of BN, by setting the BN scaling factor to zero, channel-wise pruning is enabled.

Original: FPGM (He et al., 2019) prunes filters nearest to the geometric median of each layer using Euclidian distance, refraining from the limits of norm-based criterion.
Revised: FPGM (He et al., 2019) prunes filters nearest to the geometric median of each layer using Euclidian distance, refraining from the limits of norm-based criterion.

Original: (Hu et al., 2016) indicates activation may also be an indicator, and it introduces Average Percentage Of Zeros (APoZ) to judge if one output activation map is contributing to the result.
Revised: (Hu et al., 2016) indicates activation may also be an indicator, and it introduces Average Percentage Of Zeros (APoZ) to judge if one output activation map is contributing to the result.

Original: Regardless of the criteria used to identify redundancy, previous works either remove or zero out unimportant parameters, which will inevitably result in information loss.
Revised: Regardless of the criteria used to identify redundancy, previous works either remove or zero out unimportant parameters, which will inevitably result in information loss.

Original: Our method, however, solves these drawbacks by an equivalent approach, which will be discussed in Section 3.
Revised: Our method, however, solves these drawbacks by an equivalent approach, which will be discussed in Section 3.

Original: Pruning Manner.
Revised: Pruning Manner.

Original: The traditional filter pruning pipeline will reduce the model capacity of the original models, thus facing the problem of unrecoverable performance loss after incorrect pruning.
Revised: The traditional filter pruning pipeline will reduce the model capacity of the original models, thus facing the problem of unrecoverable performance loss after incorrect pruning.

Original: Besides, the overreliance on pre-trained models and the huge time cost of fine-tuning make it unsuitable to deploy in real-world scenarios.
Revised: Besides, the overreliance on pre-trained models and the huge time cost of fine-tuning make it unsuitable to deploy in real-world scenarios.

Original: To overcome that, SFP (He et al., 2018) zeroizes pruned filters with a binary mask and updates filters in a soft manner to maintain the capacity of the network.
Revised: To overcome that, SFP (He et al., 2018) zeroizes pruned filters with a binary mask and updates filters in a soft manner to maintain the capacity of the network.

Original: Based on SFP, ASFP (He et al., 2020b) gradually increases the pruning rate to alleviate the accuracy drop caused by pruning and stabilize the whole pruning process.
Revised: Based on SFP, ASFP (He et al., 2020b) gradually increases the pruning rate to alleviate the accuracy drop caused by pruning and stabilize the whole pruning process.

Original: STP (Rong et al., 2020) uses the first-order Taylor series to measure the importance of filters while SRFP (Cai et al., 2021a) removes filters smoothly by gradually decaying to zero so that it can better preserve the trained information.
Revised: STP (Rong et al., 2020) uses the first-order Taylor series to measure the importance of filters while SRFP (Cai et al., 2021a) removes filters smoothly by gradually decaying to zero so that it can better preserve the trained information.

Original: (Cai et al., 2021b) analysis the characteristic of hard manner and soft manner, and propose GHFP to smoothly switch from soft to hard to achieve a balance between performance and convergence speed.
Revised: (Cai et al., 2021b) analysis the characteristic of hard manner and soft manner, and propose GHFP to smoothly switch from soft to hard to achieve a balance between performance and convergence speed.

Original: Similarly, our work can be also classified as a SFP-basd method, which well guarantees the model capacity after pruning.
Revised: Similarly, our work can be also classified as a SFP-basd method, which well guarantees the model capacity after pruning.

Original:  M ETHODOLOGY 3. F ORMULATION
Revised:  M ETHODOLOGY 3. F ORMULATION

Original: In this section, we formally introduce the symbol and notation.
Revised: In this section, we formally introduce the symbol and notation.

Original: The deep CNN network can be parameterized by W i ∈ R N iout ∗ N iin ∗ h i ∗ w i , where 1 ⩽ i ⩽ L and L is the total number of convolutional layers in a network, h i and w i represent the height and width of a kernel.
Revised: The deep CNN network can be parameterized by W i ∈ R N iout ∗ N iin ∗ h i ∗ w i , where 1 ⩽ i ⩽ L and L is the total number of convolutional layers in a network, h i and w i represent the height and width of a kernel.

Original: N iout and N iin denote the number of output channels and input channels for the i -th convolution layer, respectively.
Revised: N iout and N iin denote the number of output channels and input channels for the i -th convolution layer, respectively.

Original: The output feature map O i ∈
Revised: The output feature map O i ∈

Original: R N iout ∗ h
Revised: R N iout ∗ h

Original: i +1 ∗
Revised: i +1 ∗

Original: w i +1 is calculated by the convolution operation of input feature map I i ∈
Revised: w i +1 is calculated by the convolution operation of input feature map I i ∈

Original: R N iin ∗ h
Revised: R N iin ∗ h

Original: i ∗ w i and W i , shown as below:
Revised: i ∗ w i and W i , shown as below:

Original: [Equation]
Revised: [Equation]

Original: where O i,j and W i,j denote the j -th output channel of output feature maps and the j -th filter of the i -th layer, respectively.
Revised: where O i,j and W i,j denote the j -th output channel of output feature maps and the j -th filter of the i -th layer, respectively.

Original: Assume that the pruning rate of i -th layer is P
Revised: Assume that the pruning rate of i -th layer is P

Original: i , then the number of filters after pruning will reduce from N iout to N iout ∗ (1 − P i ) .
Revised: i , then the number of filters after pruning will reduce from N iout to N iout ∗ (1 − P i ) .

Original: Accordingly, the size of the pruned output tensor would be N iout ∗ (1 − P i ) ∗ h
Revised: Accordingly, the size of the pruned output tensor would be N iout ∗ (1 − P i ) ∗ h

Original: i +1 ∗
Revised: i +1 ∗

Original: w
Revised: w

Original: i +1 .
Revised: i +1 .

Original: Given a dataset D = { ( x i , y i ) } ni =1 and a desired sparsity level k
Revised: Given a dataset D = { ( x i , y i ) } ni =1 and a desired sparsity level k

Original: (i
Revised: (i

Original: [Equation]
Revised: [Equation]

Original: e., the number of remaining filters), filter pruning can be formulated as:
Revised: e., the number of remaining filters), filter pruning can be formulated as:

Original: [Equation]
Revised: [Equation]

Original: Features × ...
Revised: Features × ...

Original: Filters ...
Revised: Filters ...

Original: Filter Clustering &Selection
Revised: Filter Clustering &Selection

Original: C l u s t er s ...
Revised: C l u s t er s ...

Original: Initial Network
Revised: Initial Network

Original: Filter Recovery
Revised: Filter Recovery

Original: Channel Addition
Revised: Channel Addition

Original: Iteratively × ...
Revised: Iteratively × ...

Original: × ...
Revised: × ...

Original: Layer i
Revised: Layer i

Original: Layer i+ ... ......
Revised: Layer i+ ... ......

Original: Convergence (cid:731)
Revised: Convergence (cid:731)

Original: [Figure]
Revised: [Figure]

Original: where ℓ ( . ) is the loss function (e.g., cross-entropy loss), W ′ is the filter set of the pruned network.
Revised: where ℓ ( . ) is the loss function (e.g., cross-entropy loss), W ′ is the filter set of the pruned network.

Original: Typically, hard filter pruning prunes the model layer by layer and fine-tune iteratively to complement the degradation of the performance, while soft filter pruning (SFP) and its variants prune filters by simply zeroizing them with a Boolean matrix, indicates whether the filter is pruned or not.
Revised: Typically, hard filter pruning prunes the model layer by layer and fine-tune iteratively to complement the degradation of the performance, while soft filter pruning (SFP) and its variants prune filters by simply zeroizing them with a Boolean matrix, indicates whether the filter is pruned or not.

Original: 3. C LUSTER P RUNING
Revised: 3. C LUSTER P RUNING

Original: Based on the observation that filters are naturally clustered in each layer, our CP intends to manually introduce redundancy by generating identical filters.
Revised: Based on the observation that filters are naturally clustered in each layer, our CP intends to manually introduce redundancy by generating identical filters.

Original: As depicted in Fig.
Revised: As depicted in Fig.

Original: 2, our pipeline is composed of four steps, 1) Clustering Step 2)
Revised: 2, our pipeline is composed of four steps, 1) Clustering Step 2)

Original: Selection Step 3)
Revised: Selection Step 3)

Original: Recovery Step and 4) Channel Addition.
Revised: Recovery Step and 4) Channel Addition.

Original: The first three steps are carried out iteratively till convergence, right before those redundant filters are pruned with the proposed channel addition operation.
Revised: The first three steps are carried out iteratively till convergence, right before those redundant filters are pruned with the proposed channel addition operation.

Original: Clustering Step: We first reshape 4D tensor W
Revised: Clustering Step: We first reshape 4D tensor W

Original: i into a 2D matrix of size ( N iin ∗ h
Revised: i into a 2D matrix of size ( N iin ∗ h

Original: i ∗ w i ) ∗ N iout .
Revised: i ∗ w i ) ∗ N iout .

Original: Therefore, each column of this 2D matrix stands for the filter of the weight tensor.
Revised: Therefore, each column of this 2D matrix stands for the filter of the weight tensor.

Original: Then we can regard them as coordinates in high dimensional space and conduct clustering as:
Revised: Then we can regard them as coordinates in high dimensional space and conduct clustering as:

Original: [Equation]
Revised: [Equation]

Original: where c i is the amount of clusters of i -th layer, µ c ( j ) i is the cluster centroid which W ij is assigned to.
Revised: where c i is the amount of clusters of i -th layer, µ c ( j ) i is the cluster centroid which W ij is assigned to.

Original: The optimization can be solved by kmeans (MacQueen, 1967) or other clustering algorithms.
Revised: The optimization can be solved by kmeans (MacQueen, 1967) or other clustering algorithms.

Original: After clustering, filters in the i -th layer can be divided into c i clusters, where their norm are relatively close and may have a certain linear relationship (He et al., 2019).
Revised: After clustering, filters in the i -th layer can be divided into c i clusters, where their norm are relatively close and may have a certain linear relationship (He et al., 2019).

Original: Filters within each cluster can generate similar feature maps of the next layer, therefore, the clusters with more filters can be identified as more redundant.
Revised: Filters within each cluster can generate similar feature maps of the next layer, therefore, the clusters with more filters can be identified as more redundant.

Original: Selection Step: We adopt the Euclidean distance to evaluate the similarity of filters as Eq. 4.
Revised: Selection Step: We adopt the Euclidean distance to evaluate the similarity of filters as Eq. 4.

Original: In general, those filters close to assigned cluster centroids are considered much more redundant.
Revised: In general, those filters close to assigned cluster centroids are considered much more redundant.

Original: Based on this understanding, we can find the “most similar” filters and replace them with the centroids to which they belong.
Revised: Based on this understanding, we can find the “most similar” filters and replace them with the centroids to which they belong.

Original: Since the distance between these filters and corresponding clustering centroids is small, such reconstruction does not have too much negative impact with a small pruning rate.
Revised: Since the distance between these filters and corresponding clustering centroids is small, such reconstruction does not have too much negative impact with a small pruning rate.

Original: [Equation]
Revised: [Equation]

Original: To further avoid the severe impact of such reconstruction when 1) the pruning rate is large 2) pruning the network from scratch, we limit P i to gradually increase from the initial value towards the goal pruning rate P goal .
Revised: To further avoid the severe impact of such reconstruction when 1) the pruning rate is large 2) pruning the network from scratch, we limit P i to gradually increase from the initial value towards the goal pruning rate P goal .

Original: The definition of P i is listed as follows:
Revised: The definition of P i is listed as follows:

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: where P initi represents the initial pruning rate for the i -th layer.
Revised: where P initi represents the initial pruning rate for the i -th layer.

Original: To stabilize the pruning process, we consider two kinds of strategies, which are linear increase and exponential increase respectively.
Revised: To stabilize the pruning process, we consider two kinds of strategies, which are linear increase and exponential increase respectively.

Original: The linear increase strategy can be written as:
Revised: The linear increase strategy can be written as:

Original: [Equation]
Revised: [Equation]

Original: where epoch max is the number of total training epochs.
Revised: where epoch max is the number of total training epochs.

Original: Starting from zero, the pruning rate will linearly increase until it reach the goal pruning rate, as illustrated in Fig. 3(a).
Revised: Starting from zero, the pruning rate will linearly increase until it reach the goal pruning rate, as illustrated in Fig. 3(a).

Original: Similarly, the exponential increase strategy can be given by:
Revised: Similarly, the exponential increase strategy can be given by:

Original: [Equation]
Revised: [Equation]

Original:  where k 1 and k 2 are two hyper-parameters to control the growth speed of the pruning rate.
Revised: ) where k 1 and k 2 are two hyper-parameters to control the growth speed of the pruning rate.

Original: For different setting of hyper-parameters, the trend of pruning rate can be fast to slow, or vice versa, as shown in Fig.
Revised: For different setting of hyper-parameters, the trend of pruning rate can be fast to slow, or vice versa, as shown in Fig.

Original: With the adaptive strategies of pruning rate, the whole pruning process will become smoother, which is intuitively reflected on the final performance.
Revised: With the adaptive strategies of pruning rate, the whole pruning process will become smoother, which is intuitively reflected on the final performance.

Original: Recovery Step: Filters can be wrongly reconstructed since the cluster centroids are changing along the whole iterative pruning process.
Revised: Recovery Step: Filters can be wrongly reconstructed since the cluster centroids are changing along the whole iterative pruning process.

Original: Thus after the selection step, we retrain the network for some epochs so that it can recover from the wrong assignment.
Revised: Thus after the selection step, we retrain the network for some epochs so that it can recover from the wrong assignment.

Original: As the pruning step is integrated into the normal training schema, the model can be trained and pruned synchronously.
Revised: As the pruning step is integrated into the normal training schema, the model can be trained and pruned synchronously.

Original: Channel Addition: Our CP iterates over the clustering, selection and recovery steps till the model converges.
Revised: Channel Addition: Our CP iterates over the clustering, selection and recovery steps till the model converges.

Original: Afterwards, each layer contains lots of identical filters within different clusters, where we can utilize the proposed channel addition operation to obtain a actual compact model.
Revised: Afterwards, each layer contains lots of identical filters within different clusters, where we can utilize the proposed channel addition operation to obtain a actual compact model.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 2, suppose filters W i,m and W i,n are identical, we can safely prune W i,n by adding the n -th channel to the m -th of the filters in the next layer, which can be represented by:
Revised: 2, suppose filters W i,m and W i,n are identical, we can safely prune W i,n by adding the n -th channel to the m -th of the filters in the next layer, which can be represented by:

Original: [Equation]
Revised: [Equation]

Original: where I ′ i
Revised: where I ′ i

Original: +1 denotes the input feature map of i +1-th layer after pruning W i,n .
Revised: +1 denotes the input feature map of i +1-th layer after pruning W i,n .

Original: W i +1 , : ,m and W
Revised: W i +1 , : ,m and W

Original: i +1 , : ,n represent the m -th and n -th channel of W i +1 , respectively.
Revised: i +1 , : ,n represent the m -th and n -th channel of W i +1 , respectively.

Original: W Ci,n is the complementary set of W i,n .
Revised: W Ci,n is the complementary set of W i,n .

Original: It is worth noting that the mainstream CNN structures consist of the common Conv-BN cell.
Revised: It is worth noting that the mainstream CNN structures consist of the common Conv-BN cell.

Original: Thus, in order to omit the time-consuming fine-tuning step after channel addition, inspired by ConvNeXt (Liu et al., 2022) which only preserve part of normalization layers, we remove those BN layers after the target convolutional layers.
Revised: Thus, in order to omit the time-consuming fine-tuning step after channel addition, inspired by ConvNeXt (Liu et al., 2022) which only preserve part of normalization layers, we remove those BN layers after the target convolutional layers.

Original: The details of our CP are illustrated in Algorithm 1.
Revised: The details of our CP are illustrated in Algorithm 1.

Original: 3. ADAPTIVE COEFFICIENT FOR CLUSTERING
Revised: 3. ADAPTIVE COEFFICIENT FOR CLUSTERING

Original: Setting the hyper-parameter c i properly is quite critical since it determines both the performance and complexity of the pruned network.
Revised: Setting the hyper-parameter c i properly is quite critical since it determines both the performance and complexity of the pruned network.

Original: One possible approach is to set c
Revised: One possible approach is to set c

Original: i to a constant across layers, which is simple and intuitive.
Revised: i to a constant across layers, which is simple and intuitive.

Original: However, as deeper layers suppose to have more channels to extract various features, setting it to a constant will do harm to the diversity of features.
Revised: However, as deeper layers suppose to have more channels to extract various features, setting it to a constant will do harm to the diversity of features.

Original: Another possible way is to use metrics such as Silhouette Coefficient, which allows for fine-grained settings for each layer.
Revised: Another possible way is to use metrics such as Silhouette Coefficient, which allows for fine-grained settings for each layer.

Original: Although it can provide better accuracy, the time-consuming process for traversing each layer makes it less efficient.
Revised: Although it can provide better accuracy, the time-consuming process for traversing each layer makes it less efficient.

Original: Besides, it can be hard to control the overall FLOPs reduction, as it is automatically determined.
Revised: Besides, it can be hard to control the overall FLOPs reduction, as it is automatically determined.

Original: For the best trade-off between accuracy and FLOPs reduction, we propose our adaptive coefficient for clustering as follows.
Revised: For the best trade-off between accuracy and FLOPs reduction, we propose our adaptive coefficient for clustering as follows.

Original: Considering that the network logically should be assigned more clusters as it goes deeper, we scale the pruning rate to the worse case and restrict it to satisfy the realistic pruning rate P r as in Eq. 9, which factually treats P r as the lower pruning bound.
Revised: Considering that the network logically should be assigned more clusters as it goes deeper, we scale the pruning rate to the worse case and restrict it to satisfy the realistic pruning rate P r as in Eq. 9, which factually treats P r as the lower pruning bound.

Original: [Equation]
Revised: [Equation]

Original: then
Revised: then

Original: [Equation]
Revised: [Equation]

Original: In our experiment, since the pruning rate is same across layers, we can simply scale the arctangent function to modify its upper bound, as shown in Fig. 3(b).
Revised: In our experiment, since the pruning rate is same across layers, we can simply scale the arctangent function to modify its upper bound, as shown in Fig. 3(b).

Original: The ablation studies validate the advantages of the proposed adaptive coefficient with other methods.Algorithm
Revised: The ablation studies validate the advantages of the proposed adaptive coefficient with other methods.

Original:  E XPERIMENT 4. E XPERIMENTAL S ETTINGS
Revised:  E XPERIMENT 4. E XPERIMENTAL S ETTINGS

Original: Datasets.
Revised: Datasets.

Original: We conduct experiments on two publicly available datasets CIFAR-10 (Krizhevsky, 2009) and ImageNet (Russakovsky et al., 2015) to show the efficiency of our method.
Revised: We conduct experiments on two publicly available datasets CIFAR-10 (Krizhevsky, 2009) and ImageNet (Russakovsky et al., 2015) to show the efficiency of our method.

Original: CIFAR-10 dataset contains 60000 32*32 images with 10 classes, 50000 images for training while the remaining for testing.
Revised: CIFAR-10 dataset contains 60000 32*32 images with 10 classes, 50000 images for training while the remaining for testing.

Original: ImageNet is a large-scale dataset which contains 1.28 million 224*224 training images and 50k validation images drawn from 1000 categories.
Revised: ImageNet is a large-scale dataset which contains 1.28 million 224*224 training images and 50k validation images drawn from 1000 categories.

Original: For both datasets, we first preprocess the data by subtracting the mean and dividing the standard-deviation, then adopt the same data augmentation scheme as (He et al., 2018)
Revised: For both datasets, we first preprocess the data by subtracting the mean and dividing the standard-deviation, then adopt the same data augmentation scheme as (He et al., 2018)

Original: (He et al., 2019).
Revised: (He et al., 2019).

Original: [Table]
Revised: [Table]

Original: Network Architecture.
Revised: Network Architecture.

Original: We study the performance on various mainstream CNN models, including
Revised: We study the performance on various mainstream CNN models, including

Original: VGGNet (Simonyan & Zisserman, 2015) with a plain structure and ResNet series (He et al., 2016) with residual blocks.
Revised: VGGNet (Simonyan & Zisserman, 2015) with a plain structure and ResNet series (He et al., 2016) with residual blocks.

Original: For CIFAR-10 dataset, we test our CP on VGGNet and ResNet-56, while on
Revised: For CIFAR-10 dataset, we test our CP on VGGNet and ResNet-56, while on

Original: ImageNet, we test it on commonly used ResNet-50.
Revised: ImageNet, we test it on commonly used ResNet-50.

Original: Since ResNet is less redundant than VGGNet, we will focus more on it.
Revised: Since ResNet is less redundant than VGGNet, we will focus more on it.

Original: Configurations.
Revised: Configurations.

Original: The models are trained from scratch using SGD with a weight decay of 0.
Revised: The models are trained from scratch using SGD with a weight decay of 0.

Original: and Nesterov momentum of 0.9 (Sutskever et al., 2013).
Revised: and Nesterov momentum of 0.9 (Sutskever et al., 2013).

Original: For CIFAR-10, we train the model with a batchsize of 128 for 200 epochs and use an initial learning rate as 0.1, while on ImageNet, the number of epochs, batchsize and the initial learning rate are set to 100, 256 and 0.1, respectively.
Revised: For CIFAR-10, we train the model with a batchsize of 128 for 200 epochs and use an initial learning rate as 0.1, while on ImageNet, the number of epochs, batchsize and the initial learning rate are set to 100, 256 and 0.1, respectively.

Original: The learning rate is divided by 5 at epoch 60, 120, 160 on CIFAR-10, and is divided by 10 everyepochs on ImageNet.
Revised: The learning rate is divided by 5 at epoch 60, 120, 160 on CIFAR-10, and is divided by 10 everyepochs on ImageNet.

Original: We prune all the convolutional layers of VGGNet for it has a plain structure.
Revised: We prune all the convolutional layers of VGGNet for it has a plain structure.

Original: While for ResNet, due to the existence of shortcut, we prune all the convolutional layers except for the last one of every residual block for simplification.
Revised: While for ResNet, due to the existence of shortcut, we prune all the convolutional layers except for the last one of every residual block for simplification.

Original: We compare our results with other stateof-the-art filter pruning methods and all the results are obtained from the original papers.
Revised: We compare our results with other stateof-the-art filter pruning methods and all the results are obtained from the original papers.

Original: Each experiment is conducted 3 times and we report the mean value for comparison.
Revised: Each experiment is conducted 3 times and we report the mean value for comparison.

Original: 4. P RUNING R ESULTS ON CIFAR- 
Revised: 4. P RUNING R ESULTS ON CIFAR-

Original: ResNet-56.
Revised: ResNet-56.

Original: We summarize the results of ResNet-56 on CIFAR-10 in Table 1.
Revised: We summarize the results of ResNet-56 on CIFAR-10 in Table 1.

Original: Our CP achieves comparable performance on CIFAR-10 compared with other filter pruning methods.
Revised: Our CP achieves comparable performance on CIFAR-10 compared with other filter pruning methods.

Original: For example,
Revised: For example,

Original: GDP accelerates ResNet-56 by 34.3 % with 0.35 % accuracy drop while our CP can prune 58.6 % of total FLOPs with only 0.05 % accuracy drop.
Revised: GDP accelerates ResNet-56 by 34.3 % with 0.35 % accuracy drop while our CP can prune 58.6 % of total FLOPs with only 0.05 % accuracy drop.

Original: Furthermore, CP outperforms FS on accuracy loss (0.05 % v.s. 0.07 % ) and pruned FLOPs (58.6 % v.s. 41.5 % ) on ResNet-56.
Revised: Furthermore, CP outperforms FS on accuracy loss (0.05 % v.s. 0.07 % ) and pruned FLOPs (58.6 % v.s. 41.5 % ) on ResNet-56.

Original: VGG-16.
Revised: VGG-16.

Original: Table 1 shows the performance of various pruning methods on VGG-16.
Revised: Table 1 shows the performance of various pruning methods on VGG-16.

Original: Those numbers in brackets denote the overall pruning rate.
Revised: Those numbers in brackets denote the overall pruning rate.

Original: Our CP provides lower accuracy loss compared to
Revised: Our CP provides lower accuracy loss compared to

Original: AutoPrune and VCNNP (-0.05 % v.s. 0.07 % , 0.9 % ) with similar FLOPs reduction.
Revised: AutoPrune and VCNNP (-0.05 % v.s. 0.07 % , 0.9 % ) with similar FLOPs reduction.

Original: Compared with
Revised: Compared with

Original: GAL and Hrank, CP outperforms each of them in all aspects (45.2 % v.s. 61.9 % and 65.3 % v.s.
Revised: GAL and Hrank, CP outperforms each of them in all aspects (45.2 % v.s. 61.9 % and 65.3 % v.s.

Original: 72.2 % in FLOPs reduction, 0.36 % v.s. 3.23 % and 1.15 % v.s. 1.62 % in accuracy loss), which proves its ability to compress and accelerate a neural network with a plain structure.
Revised: 72.2 % in FLOPs reduction, 0.36 % v.s. 3.23 % and 1.15 % v.s. 1.62 % in accuracy loss), which proves its ability to compress and accelerate a neural network with a plain structure.

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: P RUNING R ESULTS ON I MAGE N ET
Revised: P RUNING R ESULTS ON I MAGE N ET

Original: For the ImageNet dataset, we evaluate our CP on ResNet-50 and compare our results with other state-of-the-art methods.
Revised: For the ImageNet dataset, we evaluate our CP on ResNet-50 and compare our results with other state-of-the-art methods.

Original: Table 2 shows the performance of our CP compared with the previously mentioned methods.
Revised: Table 2 shows the performance of our CP compared with the previously mentioned methods.

Original: As a result, our method can achieve better FLOPs reduction and lower accuracy drop.
Revised: As a result, our method can achieve better FLOPs reduction and lower accuracy drop.

Original: For instance, our CP can prune 61.8 % FLOPs of ResNet-50 with only 1.06 % top-1 and 0.68 % top-5 accuracy drop, while ABCPruner can only prune 56.6 % of total FLOPs with 2.49 % top1 accuracy drop.
Revised: For instance, our CP can prune 61.8 % FLOPs of ResNet-50 with only 1.06 % top-1 and 0.68 % top-5 accuracy drop, while ABCPruner can only prune 56.6 % of total FLOPs with 2.49 % top1 accuracy drop.

Original: Compared with HRank, CP achieves 0.11 % less top-1 accuracy drop and yields 18.0 % more FLOPs reduction.
Revised: Compared with HRank, CP achieves 0.11 % less top-1 accuracy drop and yields 18.0 % more FLOPs reduction.

Original: This proves our CP can obtain a better trade-off between performance and complexity compared with other state-of-the-art pruning methods, which is highly attributed to the preservation of feature extraction when CP reconstructs filters.
Revised: This proves our CP can obtain a better trade-off between performance and complexity compared with other state-of-the-art pruning methods, which is highly attributed to the preservation of feature extraction when CP reconstructs filters.

Original: 4. A BLATION S TUDY
Revised: 4. A BLATION S TUDY

Original: Varying pruning rates.
Revised: Varying pruning rates.

Original: To better shed light on the performance of our CP, we present the test accuracy under different pruning rates for ResNet-20 in Fig. 4(a).
Revised: To better shed light on the performance of our CP, we present the test accuracy under different pruning rates for ResNet-20 in Fig. 4(a).

Original: The network is trained from scratch with the number of clusters
Revised: The network is trained from scratch with the number of clusters

Original: c i is fixed at 25 % .
Revised: c i is fixed at 25 % .

Original: As shown in Fig. 4(a), with the decrease in pruning rate, the test accuracy increases linearly, and the pruned FLOPs decreases gradually as well.
Revised: As shown in Fig. 4(a), with the decrease in pruning rate, the test accuracy increases linearly, and the pruned FLOPs decreases gradually as well.

Original: When the pruning rate is too large, it is difficult to recover even after fine-tuning.
Revised: When the pruning rate is too large, it is difficult to recover even after fine-tuning.

Original: Thus, for a better trade-off between model complexity and performance, the pruning rate should be chosen carefully.
Revised: Thus, for a better trade-off between model complexity and performance, the pruning rate should be chosen carefully.

Original: Adaptive coefficient for clustering.
Revised: Adaptive coefficient for clustering.

Original: We demonstrate the effectiveness of the proposed adaptive coefficient for clustering in Table.
Revised: We demonstrate the effectiveness of the proposed adaptive coefficient for clustering in Table.

Original: Compare with the constant setting, the adaptive coefficient can achieve better performance and more FLOPs reduction.
Revised: Compare with the constant setting, the adaptive coefficient can achieve better performance and more FLOPs reduction.

Original: This is mainly because it allows deeper layers to preserve more various features.
Revised: This is mainly because it allows deeper layers to preserve more various features.

Original: We also compare our method with using Silhouette Coefficient, whose accuracy is slightly higher than ours.
Revised: We also compare our method with using Silhouette Coefficient, whose accuracy is slightly higher than ours.

Original: However, as it automatically determines c i for each layer, the total FLOPs reduction is uncontrollable.
Revised: However, as it automatically determines c i for each layer, the total FLOPs reduction is uncontrollable.

Original: It tends to not prune the network as much as possible.
Revised: It tends to not prune the network as much as possible.

Original: Besides, the time cost of traversing all possible choices is intolerant.
Revised: Besides, the time cost of traversing all possible choices is intolerant.

Original: In summary, our adaptive coefficient can well balance performance and complexity.
Revised: In summary, our adaptive coefficient can well balance performance and complexity.

Original: Influence of the number of clusters.
Revised: Influence of the number of clusters.

Original: In order to explore the influence of c i , we compare the test accuracy of pruned ResNet-20 with different c i on CIFAR-10 dataset.
Revised: In order to explore the influence of c i , we compare the test accuracy of pruned ResNet-20 with different c i on CIFAR-10 dataset.

Original: As shown in Fig. 4(b), the overall trend of accuracy is that the larger c i is, the higher test accuracy becomes.
Revised: As shown in Fig. 4(b), the overall trend of accuracy is that the larger c i is, the higher test accuracy becomes.

Original: However, the
Revised: However, the

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: pruned FLOPs is also downward and there exist some knee points of accuracy, which suggests that c i should not be too large and it needs to be well designed to balance the general performance.
Revised: pruned FLOPs is also downward and there exist some knee points of accuracy, which suggests that c i should not be too large and it needs to be well designed to balance the general performance.

Original: Different pruning strategies.
Revised: Different pruning strategies.

Original: We compare the results of three decay strategies when pruning
Revised: We compare the results of three decay strategies when pruning

Original: ResNet-20 on CIFAR-10 with the training epochs increasing.
Revised: ResNet-20 on CIFAR-10 with the training epochs increasing.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 5, no matter if it is trained from scratch or not, the fast-slow strategy outperforms others although it causes a relatively larger accuracy loss at the initial stage of the training process.
Revised: 5, no matter if it is trained from scratch or not, the fast-slow strategy outperforms others although it causes a relatively larger accuracy loss at the initial stage of the training process.

Original: The linear strategy creates identical filters in a much smoother manner which leads to a disastrous non-convergence issue.
Revised: The linear strategy creates identical filters in a much smoother manner which leads to a disastrous non-convergence issue.

Original: Choice of clustering methods.
Revised: Choice of clustering methods.

Original: We compare the experimental results under different clustering methods, as shown in Table.
Revised: We compare the experimental results under different clustering methods, as shown in Table.

Original: Clustering methods including Kmeans, Spectral, Birch, and Agglomerative are evaluated, respectively.
Revised: Clustering methods including Kmeans, Spectral, Birch, and Agglomerative are evaluated, respectively.

Original: No significant performance differences are observed, which further validates the robustness of our method.
Revised: No significant performance differences are observed, which further validates the robustness of our method.

Original: A logical explanation of such a phenomenon can be that our reconstruction of filters mainly concentrates on locations where density is high and these clustering methods all well identify such locations.
Revised: A logical explanation of such a phenomenon can be that our reconstruction of filters mainly concentrates on locations where density is high and these clustering methods all well identify such locations.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: To conclude, in this paper, we point out the limitations of previous works and propose a novel filter pruning method based on clustering, named Cluster Pruning, to accelerate deep CNNs.
Revised: To conclude, in this paper, we point out the limitations of previous works and propose a novel filter pruning method based on clustering, named Cluster Pruning, to accelerate deep CNNs.

Original: Specifically,
Revised: Specifically,

Original: CP considers the similarity between filters as redundancy and removes them in a softer way without information loss.
Revised: CP considers the similarity between filters as redundancy and removes them in a softer way without information loss.

Original: Thanks to these, CP allows filters to be pruned more stable as the training procedures run and achieves state-of-the-art performance in several benchmarks.
Revised: Thanks to these, CP allows filters to be pruned more stable as the training procedures run and achieves state-of-the-art performance in several benchmarks.

Original: In the future, we plan to work on how to combine CP with other norm-based criteria and more importantly, other acceleration algorithms, e.g., knowledge distillation and matrix decomposition, to push the performance to a higher stage.
Revised: In the future, we plan to work on how to combine CP with other norm-based criteria and more importantly, other acceleration algorithms, e.g., knowledge distillation and matrix decomposition, to push the performance to a higher stage.