R EVISIT K ERNEL P RUNING WITH L OTTERY R EGULATED G ROUPED C ONVOLUTIONS
A BSTRACT
Structured pruning methods which are capable of delivering a densely pruned network are among the most popular techniques in the realm of neural network pruning, where most methods prune the original network at a ﬁlter or layer level.
Although such methods may provide immediate compression and acceleration beneﬁts, we argue that the blanket removal of an entire ﬁlter or layer may result in undesired accuracy loss.
In this paper, we revisit the idea of kernel pruning (to only prune one or several k × k kernels out of a 3D-ﬁlter), a heavily overlooked approach under the context of structured pruning.
This is because kernel pruning will naturally introduce sparsity to ﬁlters within the same convolutional layer — thus, making the remaining network no longer dense.
We address this problem by proposing a versatile grouped pruning framework where we ﬁrst cluster ﬁlters from each convolutional layer into equal-sized groups, prune the grouped kernels we deem unimportant from each ﬁlter group, then permute the remaining ﬁlters to form a densely grouped convolutional architecture (which also enables the parallel computing capability) for ﬁne-tuning.
Speciﬁcally, we consult empirical ﬁndings from a series of literature regarding Lottery Ticket Hypothesis to determine the optimal clustering scheme per layer, and develop a simple yet cost-efﬁcient greedy approximation algorithm to determine which group kernels to keep within each ﬁlter group.
Extensive experiments also demonstrate our method often outperforms comparable SOTA methods with lesser data augmentation needed, smaller ﬁnetuning budget required, and sometimes even much simpler procedure executed (e.g., one-shot v. iterative).
Please refer to our GitHub repository for code.
I NTRODUCTION
The applications of convolutional neural networks (CNNs) have demonstrated proven success in various computer vision tasks (Voulodimos et al., 2018).
However, with modern CNN architectures being increasingly deeper and wider, over-parameterization has become one of the major challenges of deploying such models to devices with limited computational resources and memory capacity (Frankle & Carbin, 2019).
Therefore, the study of network pruning — the technique of removing redundant weights from the originally trained network without signiﬁcantly sacriﬁcing accuracy — has been an important subject both for practical concerns (Mozer & Smolensky, 1989) and better understanding of the properties and mechanisms of neural networks (Arora et al., 2018).
In the realm of CNN pruning, a spectrum of techniques have been studied where the two ends are populated by structured pruning and unstructured pruning methods (Mao et al., 2017).
Methods from the former end often propose to remove redundant weights in groups while following some geometrical constraints — such as removing a certain ﬁlter or layer.
The methods from the latter end, on the other hand, prune the network with a more ﬁne-grained view where they evaluate every weight individually.
Yet, there are many other methods lay in between of the two ends.
Methods that are more “unstructured” are believed to be capable of yielding better accuracy retention with a commensurate amount of parameters pruned, due to having a higher degree of freedom on how and where to introduce sparsity to the originally dense network.
Empirical ﬁndings also support this claim (Liu et al., 2019; Mao et al., 2017).
Despite having advantages on accuracy retention, the resultant networks from methods closer to the unstructured end will be more sparse and less regulated on where to introduce sparsity.
It may not provide actual compression and acceleration without relying on custom-indexing, sparse convolution libraries, or even dedicated hardware devices; thus, limits the deployability of such methods (Mao et al., 2017).
Meanwhile, methods closer to the structured end, by preserving a more regulated resultant network, are more likely to be library/hardware-friendly.
The most deployable structured pruning method may deliver a densely resultant network and therefore gain immediate compression and acceleration beneﬁts.
We denote this kind of pruning method as densely structured .
Naturally, many scholars want to develop new methods within the realm of densely structured pruning but with better accuracy retention.
The majority of densely structured pruning methods focus on pruning the original network at a ﬁlter or layer level.
We argue that the blanket removal of an entire ﬁlter or layer may harm the representation power of the network and result in undesired accuracy loss — as removing a ﬁlter would consequently remove all feature maps generated by such ﬁlter.
Even worse, removing a layer would eliminate more feature maps and even face the danger of layer-collapse , a phenomenon of having an untrainable pruned network due to premature pruning of an entire layer (Tanaka et al., 2020).
In this paper, we revisit the idea of kernel pruning (to only prune one or several k × k kernels from a 3D-ﬁlter, instead of an entire one) as an alternative and less aggressive pruning approach with higher degree of freedom.
We hypothesize that by not removing the entire ﬁlter, the representation power of the original network will be better preserved.
Although the idea of kernel pruning is nothing too novel (as it is simply a special case of individual weights pruning with 100% of weights of a kernel pruned), it is mostly applied under the context of unstructured pruning or structured pruning methods which may not deliver a dense pruned network (Mao et al., 2017; Ma et al., 2020).
This is because a direct implementation of kernel pruning with no constraint would introduce sparsity across the network and therefore make the pruned network no longer dense.
We address this problem by proposing a versatile grouped pruning framework, where we:
Cluster similar ﬁlters from each convolutional layer into a (predeﬁned) number of equalsized ﬁlter groups.
For each ﬁlter group, identify a certain portion of grouped kernels to prune according to the required pruning ratio.
Permute the remaining ﬁlters to form a densely grouped convolutional architecture according to the number of groups used in step 1.
Like most other post-train grouped pruning methods, we face the challenge of determining which clustering schemes and which importance metrics to use in step 1 and 2 of the above procedure.
Upon investigations and experiments, we discovered that a classical clustering scoring system (e.g., the Silhouette score ) might not capture the better clustering scheme in regard to accuracy retention.
Yet many ﬁlter importance metrics require sophisticated procedures, which are not computationally friendly or easy to execute when applied at a kernel level.
We address the ﬁrst challenge by consulting model-generated information — in this case, the empirical ﬁndings on Lottery Ticket Hypothesis (LTH) and related literature on weights shifting — to develop a scoring system that identiﬁes the optimal clustering scheme among options per each convolutional layer (Frankle & Carbin, 2019;
Renda et al., 2020).
For the second challenge, we design a simple and cost-efﬁcient greedy algorithm with multiple restarts to generate multiple candidate kernel selection queues and identify the one queue where the preserved kernels are most “distinctive” from each other yet “similar” to the pruned kernels.
The main contributions and advantages of our method are:
• Bring back an overlooked approach : We brought attention to the heavily overlooked approach of kernel pruning under the context of densely structured pruning.
Simple but effective — a vanilla adaptation of our framework outperforms
sophisticated variants of other comparable frameworks (e.g., ﬁlter pruning)
: Even by just applying well-understood classical mathematical tools, extensive experiments demonstrate our method outperforms comparable SOTA methods across different networks and datasets.
Additionally, our method often needs less data augmentation, a smaller ﬁne-tuning budget, and it executes without requiring any custom retraining, special ﬁne-tuning, or iterative prune-train cycles — which is rare for the approaches relying on LTH-related studies.
• Better longevity : We developed a framework that is compatible with furtherdeveloped/discovered clustering schemes and inductive biases, or more advanced variations upon them.
This overcomes one of the major drawbacks of many ﬁlter pruning methods: as most of them propose different ﬁlter importance metrics that are largely incompatible with each other either in terms of their procedures or computational requirements.
• Improved deployability : The resultant network of our method is structured as a densely grouped convolution, which enables parallel computing capability and greatly increases the practical deployability of our methods: as we can now share the required computation and memory footprint across multiple end-user devices, where most of them have very limited said resources individually (e.g., IoT devices, mobile phones, and wearable technologies).
• General impact on LTH and beyond network pruning :
Please refer to Section 5.
R ELATED W ORK
Many prior arts have explored the possibility of obtaining a smaller model with comparable performance by removing redundant weights (Zhu & Gupta, 2018;
Han et al., 2016), ﬁlters (Molchanov et al., 2017; Yu et al., 2018;
He et al., 2019; Wang et al., 2019a), layers (Wang et al., 2019b; Lin et al., 2019), image input (Howard et al., 2017; Han et al., 2020), or from all three dimensions (Wang et al., 2021).
It is clear that ﬁlter pruning attracts the most attention among all structured pruning approaches.
Our method is inspired by grouped convolution , a widely adopted convolutional architecture which could be implemented efﬁciently on common devices (Iandola et al., 2017).
Although kernel pruning used together with ﬁlter clustering is not a popular trend, we have seen such a combination in work like Yu et al.
However, the proposed method by Yu requires iterative analysis of many different intermediate feature maps per layer, involves a complex knowledge distillation application during the ﬁne-tuning stage, and lacks comparable experiment results to recent pruning literature.
Where our method (and concurrent work like Zhang et al.
(2022)) provides a much cleaner adaptation of the abovementioned combination that delivers beyond-SOTA performance with a straightforward one-shot pruning and standard ﬁne-tuning procedure.
In addition, our method consults empirical ﬁndings on the lottery ticket hypothesis and its derived literature regarding weights shifting (Frankle & Carbin, 2019; Renda et al., 2020;
Zhou et al., 2019), and we propose a novel greedy kernel pruning algorithm that is again simple, efﬁcient, yet effective — more on this in Section 3.
P ROPOSED M ETHOD 3. P RELIMINARIES
Assume a convolutional neural network W has L convolutional layers, we denote the W (cid:96) to be the (cid:96) -th convolutional layer of W (for (cid:96) ∈ { Z + | [1 , L ] } ).
Therefore, we shall have a 4-D tensor
W (cid:96) ∈ R C (cid:96) out × C (cid:96) in × H (cid:96) × W (cid:96) where C (cid:96) out represents the number of ﬁlters in W (cid:96) (also known as the number of output channels in some literature), C (cid:96) in represents the number of kernels per ﬁlter (a.k.a.
number of input channels ), and H (cid:96) × W (cid:96) represents the size of each kernel.
The overall procedure of our method can be mainly divided into four stages: 1) Clustering ﬁlters into n equal-sized groups, where the best clustering scheme for each convolutional layer is determined using the tickets magnitude increase score derived from prior arts on lottery ticket hypothesis and weight-shifting; 2) Evaluating several candidate grouped kernel pruning strategies generated by a greedy approximation algorithm with multiple restarts, where the strategy with preserved grouped kernels that are most distinctive from each other, yet most similar to the pruned grouped kernels gets selected; 3) Permuting the preserved ﬁlters to form a grouped convolutional architecture with n groups; 4) Fine-tuning the pruned and grouped network to recover accuracy lost from pruning.
[Table]
ay
[Table]
[Figure]
3. C LUSTERING FILTERS INTO GROUPS
The ﬁrst step of our method is to cluster ﬁlters from the same convolutional layer into n equal-sized groups.
Known that for a layer W (cid:96) we have C (cid:96) out ﬁlters, there shall be C (cid:96) out /n ﬁlters inside each equal-sized ﬁlter group.
We denote F (cid:96)i to be the i -th ﬁlter in W (cid:96) (namely, F (cid:96)i = W
(cid:96)
[ i, : , : , :] ) for i ∈ { Z + | [1 , C (cid:96) out ] } .
Filter clustering is a maturely adopted technique in network pruning since it is a widely accepted assumption that when similar ﬁlters are clustered together, the representation power of some ﬁlters can be covered by the rest of the ﬁlters in the same group (which therefore enables the potential of pruning).
Additionally, this technique drastically decomposes the scope of the problem, as we may now proceed to evaluate in a group-by-group fashion instead of evaluating all of the ﬁlters from W (cid:96) at the same time.
Many prior arts have developed methods on ﬁlter clustering with linearity assumptions (Guo et al., 2020), via retraining with a custom loss function (Wu et al., 2018), or through an iterative process (Yu et al., 2017).
We argue that since each ﬁlter F (cid:96)i is a tensor of C (cid:96) in × H (cid:96) × W (cid:96) , considerations regarding non-linearity and high-dimensional relationships should be added.
Therefore, we utilize the following three combinations of proven mathematical tools on dimensionality reduction and clustering in order to cluster ﬁlters from each layer into n equal-sized groups.
3.2. A VAILABLE CLUSTERING SCHEMES
To reduce the dimension of ﬁlters in W (cid:96) and cluster them into n equal-sized groups, we utilize the following three sets of methods: K-PCA + k -Means , Spectral Clustering + k -Means , and t -SNE + DBSCAN (with slight modiﬁcations done to obtain the desired cluster size and number).
Since all ﬁve methods here have been maturely studied, we will omit the introduction of these methods; please refer to section A.5 for details.
We denote each set of methods as a clustering scheme .
3.2. D ECIDE OPTIMAL CLUSTERING SCHEME PER LAYER WITH TMI SCORE
With the three proposed and many other unimplemented clustering schemes, one natural question to ask is how are we suppose to decide the optimal clustering scheme?
And at what scale should we make such a decision?
The latter question was relatively easier to answer as we already set on clustering ﬁlters into groups in a layer-by-layer fashion, so it is reasonable enough to ﬁnd out the most suitable clustering scheme for each layer.
But to address the former question, we need to ﬁnd a way to quantify the “quality” of proposed clustering results.
Mathematicians have provided us with classical tools such as the Silhouette score to measure the quality of clusters.
However, experiment results suggest such tools can hardly identify the better cluster method under the context of accuracy retention (A.1.1).
We hypothesize that this is because tools like the Silhouette score focus on properties regarding clusters themselves — such as distances between clusters, or cohesion within each cluster — while accuracy retention relies on ﬁlter importance, or balance between ﬁlter groups under the context of the network.
Therefore, we decide to introduce network-induced information to help us quantify the quality of different clustering results: where we consult empirical ﬁndings from a series of literature regarding Lottery Ticket Hypothesis and weights shifting.
Background on Lottery Ticket Hypothesis, Weights Shifting, and Weights Rewinding For a dense convolutional neural network W initialized with weights of W 0 , the conventional post-train pruning procedure has three steps:
Train the proposed network for t epochs.
Where we denote the network has weights of W k at the k -th epoch for 0 ≤ k ≤ t . 2.
Prune the network with weights W t according to the deﬁned pruning ratio and achieve subnetwork W (cid:48) t , where W (cid:48) t ⊂ W t .
Fine-tune the network with W (cid:48) t and f more epochs to recover accuracy.
The Lottery Ticket Hypothesis claims that instead of ﬁne-tuning upon the subnetwork W (cid:48) t , there exists a subnetwork W (cid:48) k
(deﬁned as W (cid:48) t but replaced with its counterpart weights from W k ) that, when trained in isolation for f more epochs, may match or outperform the test accuracy of the ﬁnetuned W (cid:48)
t achieved from step 3 above.
This W (cid:48)
k is therefore known as the winning ticket (Frankle & Carbin, 2019).
A vast amount of research has been done to demonstrate the existence of winning tickets across different networks and datasets, making the lottery ticket hypothesis one of the most tested inductive biases among neural networks (Renda et al., 2020).
Scholars have additionally investigated the relationship between the winning ticket’s weights and ﬁnal weights in regard to accuracy retention.
Zhou et al.
(2019) did one of the most thorough experiments on exploring such relationship by deploying nine different zero mask criteria related to initial weights, ﬁnal weights, and weights shifting during training.
They concluded that || w t | − | w
|| (where w t , w 0 respectively represent the same weight in W t and W 0 ), the magnitude increase of a weight at its initialization and after training, has demonstrated most signiﬁcant positive correlation with accuracy retention.
Note that the Zhou et al.
(2019) experiments were conducted under the assumption that the winning ticket exists at W 0 .
This is because, for early or even concurrent arts on the lottery ticket hypothesis, scholars have their disagreements on whether to reinitialize the weights of W (cid:48) t to their initial values (namely, k = 0 for k in W (cid:48) k )
(Frankle & Carbin, 2019), to near initialization ( 0 < k (cid:28) t ) (Renda et al., 2020), or just to reset randomly (Liu et al., 2019).
However, with more comprehensive experiments conducted, the ﬁndings reveal there are multiple winning tickets that exist within a range of epochs starting from near initialization: such that for any k
where 0 < k 1 ≤ k ≤ k 2 < t , W (cid:48)
k can be a winning ticket (with k 1 being close to 0 ).
This technique is referred as weights rewinding and we denote the range of [ k 1 , k 2 ] as the tickets window (Renda et al., 2020).
Ticket Magnitude Increase scoring system Based on the fact that a winning ticket may occur at near initialization but not necessary exactly at initialization, we slightly modify the magnitude increase criteria from Zhou et al.
(2019) by replacing the initialization weights as the ticket weights.
Thus, we deﬁne the ticket magnitude increase score (hereinafter “TMI score”) as || w t | − | w k || for 0 < k (cid:28) t (where w t , w k respectively represent the same weight in W t and W k ).
Since we aim to use such scoring system to govern the clustering quality of ﬁlter groups, we further expand this scoring system from individual weight to a ﬁlter-level.
For a ﬁlter F (cid:96)i ∈ R C (cid:96) in × H (cid:96) × W (cid:96) , we denote the weight at ( h, w ) index from the c -th kernel of F (cid:96)i as W (cid:96)
[ i,c,h,w ] ; so the TMI score for ﬁlter F (cid:96)i in regard to a winning ticket at the k -th epoch should be deﬁned as the following.
[Equation]
Similarly, for a ﬁlter group g (cid:96) of { F (cid:96) i , F (cid:96) j , . . . } , its TMI score would be:
[Equation]
Qualifying cluster results with TMI scores With the TMI score for a ﬁlter group deﬁned, the next question to address is what makes a good clustering result in regard to TMI scores?
We already observed that tools like the Silhouette score could not capture the better clustering result in terms of accuracy retention, and we hypothesized that this is due to the lack of attention on ﬁlter importance and balance between ﬁlter groups .
Since Zhou et al.
(2019) has established the relationship between better accuracy retention and weights with larger magnitude increase, we expect such relationship will expand to a ﬁlter level where ﬁlters with large TMI scores may help on accuracy retention and therefore be deemed “more important.”
We conﬁrm such assumption in A.4.1.
In terms of the balance between multiple ﬁlter groups — with each ﬁlter’s TMI score being an indicator of its importance — we would prefer clustering results where all ﬁlter groups have a similar TMI score.
As if one ﬁlter group’s TMI score is signiﬁcantly larger than another ﬁlter group, the former group will presumably contain more “important ﬁlters” than the latter group.
When we proceed to prune the same ratio of grouped kernels out of both groups, the network will lose more “important kernels” and therefore damage the accuracy retention.
In other words, we want to minimize the intervals between the TMI scores of all ﬁlter groups.
So for a clustering result G (cid:96) , assume g (cid:96)i , g (cid:96)j , ..., g (cid:96)n − 1 , g (cid:96)n are the n ﬁlter groups of G (cid:96) sorted in descending order according to their TMI scores.
We denote the intervals of G (cid:96) with respect to winning ticket k as:
[Equation]
where the clustering result G (cid:96) =
arg min G (cid:96) ( Sum [ Interval ( G (cid:96) , k )]) is preferred.
To further identify the clustering result with the most balanced ﬁlter groups in terms of TMI scores, we also want to minimize the variance of intervals between the TMI scores of all ﬁlter groups.
By combining the two heuristics with a balancing parameter α , the overall scoring system for a clustering result G (cid:96) is deﬁned as:
[Equation]
where the function R denotes the ranking of the current G (cid:96) against other proposed clustering results (for ranking in an ascending order).
For example, if clustering result G (cid:96) has the smallest
Sum [ Interval ( G (cid:96) , k )] among all proposed clustering results, then R ( Sum [ Interval ( G (cid:96) , k )]) = 1 .
We choose to use rank instead of the raw value as we are only interested in if one clustering result is better than another on one particular criterion (interval mean or interval variance), but not by how much.
Therefore, by projecting both criteria to rank indices, we avoid the imbalance between the raw values of the two criteria.
The arg min of this Equation 4 yields the best G (cid:96) .
Increase robustness with multiple ticket evaluations Now we deﬁned how to identify the preferred clustering scheme from all proposed G (cid:96) s in layer W (cid:96) , the last challenge is to ﬁnd the winning ticket k for TMI score calculations.
Unfortunately, ﬁnding the winning ticket requires an extremely computationally intensive process called iterative magnitude pruning , in which multiple training and pruning cycles are involved (Frankle et al., 2020).
To make matters worse, since we know that there are often multiple winning tickets available from the training process (Renda et al., 2020), picking one ticket but not the other will potentially yield a different set of preferred clustering schemes for the network.
This makes the k both a hard-to-ﬁnd task and a sensitive parameter to tune.
We address both challenges by relying on the same ﬁnding from Renda et al.
(2020), which demonstrates that for any k
where 0 < k 1 ≤ k ≤ k 2 < t , W (cid:48)
k can be a winning ticket.
This tickets window [ k 1 , k 2 ] is shown to be robust for the similar models across different datasets and pruning methods.
Thus, we can decide which model we will prune on, identify its tickets window by consulting experiment results from Renda et al.
(2020), and truncate some epochs in such window to conduct multiple evaluations.
The optimal clustering scheme for W (cid:96) will be the one that most often yield as the preferred clustering scheme per different k settings 1 .
By doing this, we ﬁrst avoid searching for the winning ticket k , which greatly reduces the computational power needed to implement our method.
Secondly, it increases the robustness of our method, since the optimal clustering scheme per layer is no longer sensitive to the choice of k .
Last, granted the wide range of [ k 1 , k 2 ] observed from most networks, practically we don’t even need to search for the exact ( k 1 , k 2 ) pair; an approximated guess with a reasonable overlap with the ideal ( k 1 , k 2 ) would often sufﬁce.
The kind of tolerance, combined with the proven generalizability of winning tickets across different tasks, models, and datasets (Morcos et al., 2019), with more efﬁcient methods of ﬁnding the winning tickets become available (Tanaka et al., 2020;
You et al., 2020).
Our proposed method can also be applied to experiments outside of Renda et al.
3. E VALUATE CANDIDATE PRESERVED GROUPED KERNEL QUEUES
With ﬁlters of W (cid:96) grouped into ﬁlter groups { g (cid:96) 1 , g (cid:96) 2 , ..., g (cid:96)n } , we will carry out the pruning procedure within each ﬁlter group.
Let K g (cid:96) i be the i -th grouped kernel in ﬁlter group g (cid:96) (for 1 ≤ i ≤ C (cid:96) in ) such that K g (cid:96) i is the collection of the i -th kernels from every ﬁlter in g (cid:96) .
The pruning of a ﬁlter group g (cid:96) can be viewed as the task of identifying a set group kernels in g (cid:96) that needs to be preserved.
To start off such identiﬁcation, we ﬁrst hypothesize a set of grouped kernels that are most “distinctive” from each other may provide the best help on preserving the representation power of the original ﬁlter group.
We deﬁne a distance matrix between grouped kernels from g (cid:96) as D ( g (cid:96) )) where
D ( g (cid:96) )
[ u,v ] =
| K g (cid:96) u − K g (cid:96) v | .
This D ( g (cid:96) )) shall be a symmetric matrix with its diagonal full of 0 s.
With this matrix, we may now translate this problem as a maximum edge-weight connected subgraph problem: Given a ﬁlter group g (cid:96) represented as an undirected complete graph G ( V, E ) with edge weights w ( u, v )
=
D ( g (cid:96) )
[ u,v ] between nodes u and v , ﬁnd a subset V ∗ ⊆
V with | V ∗ | =
s where the total edge sum of G [ V ∗ ] :
= ( V ∗ , E ∩ (cid:0) V ∗ 2 (cid:1) ) is maximal (with s =
(1 − pruning rate ) · C (cid:96) in ).
The brute force solution of this problem requires iterating through every possible subset of grouped kernels with size s and calculate their edge sum.
With (cid:0) | V | s (cid:1) possible subsets available and each subset having (cid:0) s 2 (cid:1) edges, this procedure has a time complexity of O ( s 2 · | V | s ) .
This can be quite a compute-extensive task given each graph G ( V, E ) deduced from g (cid:96) has a | V | equals to C (cid:96) in ; which can be as large as 512 in models like ResNet-101.
To address this, we hereby propose a simple greedy algorithm with multiple restarts that may approximate the V ∗ in question.
Our approximated solution was based on the hypothesis that, assume we already have a set of nodes
U ∗ ⊂ V ∗ representing grouped kernels which will be preserved.
To ﬁnd the next node u next ∈ V ∗ for u next (cid:54)∈ U ∗ , such u next should have a maximal edge sum to all nodes in U ∗ .
Namely, u next = arg max u next ( (cid:80) u
∈ U ∗ w ( u, u next )) .
With this, for a ﬁlter group g (cid:96) we may simply set the ﬁrst grouped kernel u initial ∈ U ∗ to be every grouped kernel K g (cid:96) i ∈
g (cid:96) , ﬁnd the u next until | V ∗ | =
s , and obtain C (cid:96) in pruning strategies.
This trick is known as multiple restarts and is widely adopted in many algorithms such as k -Means clustering.
We formalize the procedure of our approximation algorithm in pseudocode at A.2.1.
The core of our algorithm resides at line 10 , where we use the computed result of previous row in M to reduce time complexity of our algorithm.
With M ∈ R s × C (cid:96) in and ﬁlled C (cid:96) in times, our approximation algorithm therefore has a time complexity of O ( s | V | 2 ) (as | V | =
C (cid:96) in ).
This provides a sensible improvement upon the O ( s 2 · | V | s ) brute force solution.
To further and better identify the best grouped kernel pruning strategy among the C (cid:96) in candidate strategies in PS , we deﬁne the score of a strategy V ∗ on g (cid:96) as the following:
[Equation]
The ﬁrst term calculates the inner distance sum of V ∗ , where a larger value represents greater inner heterogeneity within the kept grouped kernels.
The second term iterates through every pruned grouped kernel p , ﬁnds the γ number of kept grouped kernels that are closest to each p according to the distance matrix D ( g (cid:96) ) , then sums over the distance between each p to its corresponding γ kept grouped kernels.
A pruning strategy with a smaller second term has better homogeneity between kept and pruned grouped kernels.
Since a strategy with greater inner heterogeneity and smaller outer homogeneity is desired, we further introduce a tunable parameter − β to balance two terms and deﬁne the best the pruning strategy of g (cid:96) to be V ∗ best = arg max V ∗ ( Score ( V ∗ , g (cid:96) )) for all V ∗ s available in PS .
We have also implemented a toy experiment to verify if our grouped kernel pruning method may obtain the optimal or a close-to-optimal solution, please refer to A.1.2 for details.
Ablation studies conducted at A.4.5 also conﬁrm the efﬁcacy of our greedy algorithm.
3. L AYER RECONSTRUCTION TO GROUPED CONVOLUTION ARCHITECTURE
The layer reconstruction of our method happens at two places: 1) Given a convolutional layer W (cid:96) , we permute the output channels of such layer according to the ﬁlter clustering result.
This makes ﬁlters within W (cid:96) form n grouped ﬁlters as demonstrated in Figure 1 (a) to (b).
2) After a certain ratio of grouped kernels were pruned from their corresponding ﬁlter groups, we permute the input channels of the remaining grouped kernels to remove sparsity from each ﬁlter group (Figure 1(c) to (d)).
As the result of the above two channel permutations, we will have n dense grouped ﬁlters left in the pruned W (cid:96) .
Since every grouped ﬁlter has its own set of input-output pathways, the pruned
W (cid:96) is essentially a grouped convolution architecture.
This architecture enables parallel computing capabilities, because every grouped ﬁlter can be deployed to a different device so that all group ﬁlters may compute simultaneously.
Please refer to A.2.2 for the general procedure in pseudocode.
E XPERIMENTS AND R ESULTS 4. E XPERIMENT S ETTINGS
Network Architectures and Datasets We evaluate the efﬁcacy of our method on popular networks with various depth and architectures: ResNet-20/32/56/110 with the BasicBlock implementation and ResNet-50/101 with the BottleNeck implementation (He et al., 2016).
For datasets, we choose CIFAR-10 (Krizhevsky, 2009), Tiny-ImageNet (Wu et al., 2017), and ImageNet (ILSVRC-12)
(Deng et al., 2009) as they vary in complexity and Wang et al.
(2021) has carried out a fairly large-scale comparative experiments of many different structured pruning algorithms running under the same setting — which provides us with a rich background to compare against.
Compared Methods and Performance Evaluation We test our method against various existing pruning methods showed in Table 2.
All mentioned pruning methods are evaluated against the following criteria:
∆ Acc, ↓ Params, and ↓ FLOPs which are deﬁned as the difference of top1 accuracy, parameters reduction, and FLOPs reduction between the baseline and pruned model (where x represents x % of parameters/FLOPs were removed/reduced comparing to the original network).
In addition, we investigate the pruning procedure and pruning setting of the compared methods in the three following aspects: 1) Does the method require an iterative pruning procedure? 2) Does the method require a special ﬁne-tuning setup that deviates from Section 3.2.2 or Figure 1(a) of Wang et al.
What is the ﬁne-tuning budget of the method-in-question?
We believe these are crucial information for building a fair understanding of the three aforementioned numerical criteria.
Note, a method with iterative pruning procedure will automatically require special ﬁnetuning, as it will ﬁne-tune/retrain after each pruning operation, which is intrinsically different from the procedure introduced in Section 3.2.2.
Our training and pruning settings are fairly simple given our method follows a one-shot procedure, please refer to A.3.1 for details.
4. R ESULTS AND ANALYSIS
As demonstrated in Table 2, our method yields superior accuracy retention than various
modern
SOTA methods across six tested network-dataset combinations with a commensurate amount of parameters pruned with the exception of ResNet-56 on Tiny-ImageNet, where our performance is reasonably close ( − 0 .
12% behind) to the best offering of 3D by Wang et al.
Speciﬁcally, our method achieves such results with a simple one-shot pruning procedure and a standard ﬁnetuning setup.
Which are usually considered two less advantageous design as many recent SOTA methods utilize iterative prune-train cycles (Wang et al., 2021; Frankle & Carbin, 2019).
Yet some simple tricks like dynamic pruning rate , soft-pruning (keep the pruned components updatable until very end), and weight reinitialization before ﬁne-tuning may often provide most algorithms another performance boost (Li et al., 2017;
He et al., 2018; Renda et al., 2020; Liu et al., 2019).
We opt not to implement such tricks to provide a cleaner delivery of our method.
Last, although not universally observed, our method often requires a smaller ﬁne-tuning budget than other offerings.
Here we include an abbreviated version of our experiment report at Table 1, please refer to Table 2 for the full one.
Please also refer to A.3.2 and A.3.3 for discussions on hyperparameters and speedup.
[Table]
4. A BLATION STUDY
We anatomize the effectiveness of different components of our proposed method.
Speciﬁcally, we ﬁrst demonstrate if the idea of our proposed TMI system and greedy approach are sound at a fundamental level; then we evaluate if they are better than some common alternative approaches with similar functionalities.
With the procedure design evaluated, we also provide some real-word runtime experiments of our proposed pruning procedure.
Last, we evaluate the relationship between the hyperparameters and the pruned network’s accuracy retention.
Please refer to A.4 for details.
C ONCLUSION
We proposed a densely structured pruning framework capable of yielding beyond SOTA performance with a straightforward one-shot procedure.
We believe the power of densely structured kernel pruning may go well beyond our implementation, as a) there will certainly be more — and hopefully better — inductive biases and unsupervised clustering schemes made available, and b) our adaptation of kernel pruning and grouped convolution is rather vanilla, where more sophisticated variants are available to explore; our framework may thrive on these further-discoveries.
In addition, our work serves as a proof of the lottery tickets-induced heuristics can be used to guide a structured pruning strategy.
This is an often overlooked usage despite the popularity of LT-related research, possibly due to the high cost of winning tickets searching.
In such case, we hope the multiple (potential) tickets evaluations trick we introduced in our method may help in levitating such concerns.
This might have an impact beyond the ﬁeld of network pruning, as scholars of other tasks might ﬁnd the ticket-induced heuristics to be useful but are deterred by the high cost of tickets searching.
Where our trick provides a form of approximation by just inspecting the weights shifting log saved during training.
R EPRODUCIBILITY S TATEMENT
As we advocate our proposed framework is able to shine a new light on kernel pruning under the context of densely structured pruning we have prepared a GitHub repository with checkpoints placed on every stage of our method.
We hope this will facilitate the reproduction of our work and invite our fellow scholars to research and optimize on different stages of our pruning framework.
To reproduce the exact training and pruning settings of our experiments in Table please refer to A.
A A PPENDIX A. S UPPLEMENTARY MATERIAL TO P ROPOSED M ETHOD A.1. Silhouette score AND A CCURACY R ETENTION
We experiment ResNet-32 on CIFAR-10 (BA = 92 .
with K-PCA + Equal-sized k -Means and Spectral + Equal-sized k -Means.
The mean Silhouette score of the latter combination is 0 .
= 12 .
49 times higher than the previous combination; indicating that if the Silhouette score may capture the better clustering result with respect to accuracy retention, the latter approach should be signiﬁcantly better.
However, the K-PCA approach has a pruned accuracy of 92 .
68% where the Spectral approach has a pruned accuracy of 92 .
41% , a +0 . 27% in favor of the method which the Silhouette score strongly suggests against.
This demonstrates that tools like the
Silhouette score might not be a good judge of clustering quality in terms of accuracy retention
A.1. E FFECTIVENESS OF OUR GREEDY APPROXIMATION ALGORITHM WITH TOY EXAMPLE
To test out whether our greedy approximation algorithm may yield a solution that is close enough to the optimal one we conducted a trial of TMI-GKP with β = 0 in Equation 5 on ResNet-32.
For the ﬁrst ten convolutional layers (80 ﬁlter groups pruned), all approximated solutions are within 0 .
5% in terms of the score difference (as deﬁned in Equation 5) to the optimal solution — which implies the effectiveness of our approximation algorithm.
A. A DDITIONAL M ETHOD P ROCEDURE I NFORMATION A.2. G REEDY G ROUPED K ERNEL P RUNING P ROCEDURE
Input: Filter group g (cid:96) (cid:46) one of the n ﬁlter group in layer W (cid:96)
Input: s (cid:46) number of grouped kernels to preserve, s =
(1 − pruning rate ) · C (cid:96) in
Initialize: Empty list PS (cid:46) for storing grouped kernels pruning strategies
Initialize: C (cid:96) in , D ( g (cid:96) )
(cid:46) number of grouped kernel in g (cid:96) and distance matrix of g (cid:96) 1: for each K i ∈ g (cid:96) do 2:
Initialize: Empty list U ∗ (cid:46) storing preserved grouped kernels per the current strategy 3:
Initialize: Zero-ﬁlled matrix M ∈ R s × C (cid:96) in (cid:46) the cost matrix 4:
M [0 ,j ] ← w ( K i , K j ) for all K j ∈ g (cid:96) \ K i 5:
U ∗ .
append ( K i ) (cid:46) make K i the ﬁrst grouped kernel to keep 6: for row ← 1 to s do 7:
K last ← U ∗ [ − 1] 8: for each grouped kernel K ∈ g (cid:96) but K (cid:54)∈ U ∗ do 9:
K index ← g (cid:96) .
getindex ( K ) 10:
M [ row ,K index ] ← M [ row − 1 ,K index ] + w ( K last , K ) 11:
U ∗ .
append (arg max K ( M [ row ] )) 12: if | U ∗ | ==
s then 13:
V ∗ ← U ∗ ,
PS. append ( V ∗ ) and break 14: return PS
A.
G ENERAL
Input: W , ( k 1 , k 2 ) (cid:46) trained model, tickets window
Input: CS, PR, n (cid:46) clustering schemes, pruning ratio, number of groups 1: for each convolutional layer W (cid:96)
∈ W do 2:
Initialize: Empty list G (cid:96) candidates (cid:46)
To store candidate clustering results 3: for k i ∈ [ k 1 , k 2 ] do 4:
Cluster ﬁlters from W (cid:96) into n equal-sized groups according to each scheme in CS 5:
Identify the preferred clustering result G (cid:96)k i ← arg min G (cid:96) (
Score ﬁlters clustering ( G (cid:96) , k i ))
[Equation 4] 6:
G (cid:96) candidates .
append ( G (cid:96)k i ) 7:
Identify G (cid:96) best ← the most occurred item in G (cid:96) candidates 8:
Apply G (cid:96) best on W (cid:96) 9: for each ﬁlter group g (cid:96)i ∈ G (cid:96) best do 10:
Get grouped kernel pruning strategies V ∗ ← Algorithm 1 ( g (cid:96)i )
[A.2.1] 11:
Identify V ∗ best ← arg max V ∗ (
Score grouped kernel pruning ( V ∗ , g (cid:96)i ))
[Equation 5] 12:
Apply V ∗ best on g (cid:96)i 13:
W (cid:96) ← pruned W (cid:96) reconstructed in grouped convolution architecture 14: return W
A.2. T IME C OMPLEXITY A NALYSIS OF THE TMI C LUSTERING P ROCEDURE
Our pruning procedure is basically two-stage: ﬁlter clustering and grouped kernel pruning.
Since we have already analyzed the time complexity of the second stage at the end of Section 3.3, we hereby focus on the time complexity of our ﬁlter clustering procedure.
Given a clustering result of convolutional layer W (cid:96) , we ﬁrst calculate the TMI score of each ﬁlter group deﬁned by Equation 2.
The TMI score of a clustering result in regards to an epoch k has a time complexity of O ( W (cid:96) ) for W (cid:96) ∈ R C (cid:96) out × C (cid:96) in × H (cid:96) × W (cid:96) .
This is because TMI scores are essentially achieved by ﬁnding out the magnitude increase of individual weights in W (cid:96) from epoch k to epoch t (the ﬁnal epoch).
TMI scores for ﬁlters and ﬁlters groups are simply sum of the TMI scores of a certain individual weights — where the summing procedure is at most O ( W (cid:96) ) as we only got these many weights to sum.
The TMI score for this clustering result is then determined by Equation 4.
Which is a O ( n ) procedure as we have n groups (thus n − 1 intervals).
Granted O ( n ) (cid:28) O
( W (cid:96) ) , this term is negligible and the overall time complexity is still O ( W (cid:96) )
Note we may have multiple clustering results generated by multiple clustering schemes, where we use their TMI scores and Equation 4 to determine the best clustering result for layer W (cid:96) .
Thus, we need to multiple the time complexity of a single clustering result to the number of clustering schemes available, we denote this number as CS num .
The overall complexity is now CS num ·O ( W (cid:96) ) .
It is hard to analyze the time complexity of each clustering scheme as it involves various dimension-reduction and clustering combinations.
For the ease of expression, we uniformly denote all clustering schemes to have a time complexity of O ( CS ( W (cid:96) )) on layer W (cid:96) (for CS ( W (cid:96) )
> W (cid:96) as a clustering scheme need to read all weights in W (cid:96) to produce a proper clustering result).
Last, as illustrated in Section 3.2.2 - Increase robustness with multiple ticket evaluations , we need to repeat the whole procedure for k num times for k num being the number of potential ticket epochs we evaluated.
Thus, the ﬁnal “rough” big-
O complexity for our ﬁlter clustering procedure is:
[Equation]
In our experiments, such k num is usually set to 35 (unless the network is too large) and CS num isas we have three different clustering schemes available (see Section 3.2.1).
By the “absorption” law of big-
O analysis, the theoretical time complexity is only O ( CS ( W (cid:96) ))
— which is identical to a standard single-shot ﬁlter clustering procedure.
This implies the theoretical lightweight-ness of our proposed method.
For more information, we included a discussion on how to adjust the pruning procedure to meet a time/computation budget at A.2.4 with real-world runtime experiments available at A.4.7.
A.2.
A DJUST THE PRUNING PROCEDURE TO MEET A TIME / COMPUTATION BUDGET
Although the analysis at A.2.3 and experiments in A.4.7 demonstrate our TMI pruning procedure is reasonably fast, we admit that it can be slow if given a wide network to prune or were asked to evaluate many potential ticket epochs — as our algorithm will have to repetitively evaluate the network with different TMI scores over and over again.
We hereby provide several points for adjustability of our method.
Reduce the range of ticket window : If a ticket window is deﬁned to be [ k 1 , k 2 ] , consider using [ k (cid:48) 1 , k (cid:48) 2 ] where k 1 < k (cid:48) 1 and k (cid:48) 2 < k 2 .
So less potential ticket epochs were evaluated.
Add a ticket
step :
For a ticket window [ k 1 , k 2 ] , consider
adding
a k step
so instead of
evaluating k 1 , k
+
1 , k
+
+
.. we now evaluate
k 1 , k
+ k step
k
+ k step + k step
By
doing
this,
less potential ticket
epochs were evaluated while a wide range of potential ticket epochs from different stage of
the network training are still considered.
Relax the granularity of clustering evaluation : The proposed TMI-GKP determines the optimal clustering scheme at a per layer manner.
For CNN models with block-like structure (such as ResNet), one may opt to determine the clustering scheme for one layer of the block, then proceed to use such clustering scheme on the whole block.
Adjust clustering schemes : One may opt to reduce the number of clustering schemes available for the TMI score evaluation.
Or one may opt to use clustering schemes which are less computational demanding.
Granted the TMI system is likely to capture the “better” clustering scheme among the options, the method would still function, but likely not at its full potential.
A. S UPPLEMENTARY MATERIAL TO E XPERIMENTS AND R ESULTS A.3. T RAINING AND P RUNING S ETTINGS
For all experiments done on CIFAR-10 and Tiny-ImageNet, we train the baseline models forepochs with the learning rate starting at 0 .
1 and dividing by 10 per every 100 epochs.
The baseline model is trained using SGD with a weight-decay set to 5e-4 , momentum set to 0 . 9 , and a batch-size of 64.
All data are augmented with random crop and randomly horizontal ﬂip.
For the experiments done on ImageNet, we train the ResNet-50 model for 90 epochs with the weight-decay set to 1eand learning rate dividing by 10 per every 30 epochs (while keeping all other settings the same as CIFAR-10 and Tiny-ImageNet experiments).
Our pruning settings are largely identical to our training settings except for the learning rate, which is set to 0 .
01 at the start.
A.3. C HOICE OF H YPERPARAMETERS
There are three main tunable hyperparameters in the mainframe of TMI-GKP, α from Equationand β, γ from Equation 5.
Note in practice γ is not a ﬁxed value but rather a value that has a ﬁxed proportion to its the convolutional layer’s input channels, i.e., γ = γ ratio · C (cid:96) in for layer W (cid:96) .
This is because different convolutional layer may have a different C in , so it makes better sense to let γ of a layer adjust along with its C in .
For experiments shown in Table 2, we ﬁx α to 0 .
5 and γ ratio to 0 .
2 for convenience.
For experiments of ResNet-56 on CIFAR-10 and ResNet-101 on Tiny-ImageNet, we set β to 2 .
For the experiment of ResNet-32 on CIFAR-10, we set β to 1 .
For the rest of the experiments in Table 2, we set β
=
(cid:0) V ∗ 2 (cid:1) /
( | g (cid:96) \ V ∗ | · γ ) with respect to Equation 5.
This basically implies we want the two terms in Equation 5 to evaluate an equal amount of grouped kernel pairs.
To further ensure/demonstrate the reproducibility of our work, in our codebase we will provide a
Google Colab notebook that replicates all CIFAR-10 pruning experiments conducted in Table 2.
For every completed experiment, our codebase may register a setting.json , a cluster.json (upon extraction of a pruned model), and an experiment.log .
Where the setting.json includes all hyperparameters’ settings, cluster.json includes the clustering scheme per each convolutional layer and its resultant permutation matrix for converting the baseline model from (a) to (b) in Figure 1, and experiment.log contains the experiment printouts.
In addition, we have implemented a set of methods so a fellow researcher may pipeline the cluster.json ﬁle to the baseline model and try for a different set of pruning strategies.
This will save them the work of re-clustering the ﬁlters and re-calculating/re-evaluating the TMI scores for every layer.
A.3. O MITTING S PEED U P A NALYSIS
We purposefully omitted the speedup analysis between the original and the pruned network mainly due to lack of optimization of grouped convolution on current ML platforms.
As an example, on
PyTorch , a grouped convolution with groups = 8 is much slower than the standard convolution despite the former one has much less parameters and FLOPs.
Please direct to our GitHub repository where we discuss in length of why PyTorch is slowing us done, what’s the reason behind it, and empirically show that it is indeed PyTorch lacks of optimization that causing this problem.
We also discuss why this will not be an issue for long (by presenting ML platforms’ commitments of optimizing the speed of grouped convolution), why this is an achievable goal (as scholars have already accelerated grouped convolution on said platforms (Gibson et al., 2020; Qin et al., 2018)), and how this is not likely to affect the serious implementation of our method for its intended purposes — as a group convolution network can be deployed as standard convolution on multiple edge devices in a parallel fashion (Su et al., 2020).
A.3. S PECIFIC EXPERIMENTS REQUESTED BY REVIEWERS
Upon the requests of two reviewers, we have additionally conducted experiments with our proposed method on VGG-16 (Simonyan & Zisserman, 2014) and on ResNet-56 on CIFAR-10 against pruning method GAL by Lin et al.
(but with a more aggressive pruning rate as ↓ Params and ↓ FLOPs are ≈ 60% instead of ≈ 43% ).
Please refer to our OpenReview entry (former experiment, latter experiment, context of the latter experiment) for details — as these requests should only make sense under their Q&A context, and we have not (or for some literature, we are not able to) conduct the pruning procedure investigations like we did in Table 2 as deﬁned in Section 4.1.
A. A BLATION S TUDIES
In this section we will anatomize our method in the following aspects.
The general idea is we ﬁrst show the basic format(s) of our approach works — as if the results achieved by following our approach would be better than the results achieved by going against our approach — then, we show that our approach works better than some common alternative approaches with similar functionalities.
First, we conﬁrm the effectiveness of our TMI scoring system by conducting the following ablation studies:
1. TMI/MI scores and ﬁlter pruning : where we investigate if ﬁlters with higher TMI score or larger magnitude increase will lead to better accuracy retention.
This shows our TMI scoring system works at a ﬁlter level.
TMI-driven clustering and accuracy retention : where we compare the TMI-driven clustering results with shufﬂed clustering results in terms of accuracy retention.
This shows our TMI scoring system works in terms of ﬁlter clustering.
TMI-driven clustering v. Other ﬁlter clustering schemes : where we investigate if the TMI-driven clustering schemes are better than some common alternative clustering schemes.
TMI-GKP v. Grouped convolution-only : where we investigate if the performance of our method is from the procedure we proposed, or it is simply due to the adaptation of grouped convolution.
Then, we investigate our greedy grouped kernel pruning algorithm with the following ablation studies:
Greedy-induced pruning and accuracy retention : where we compare the grouped kernel pruning results induced by our greedy approach to two approaches that take the “inverse” and “complement” of our greedy approach.
This shows the basic principle of our greedy approach works.
Greedy-induced pruning v. Other grouped kernel pruning policies : where we compare the greedy approach against some alternative kernel pruning policies applied on grouped kernel pruning.
This shows our greedy approach works better than some common alternative kernel pruning policies.
Last, we provide some experiments to evaluate the real-world runtime of the pruning procedure of
TMI-GKP and ablation studies on hyperparameters.
A.4. TMI/MI SCORES AND FILTER PRUNING
We ﬁrst address the question of whether the TMI score (or the original “MI” magnitude increase score from Zhou et al.
(2019)) has a relationship with accuracy retention when relaxed to a ﬁlter level.
Although the question is straightforward, it is up to different interpretations as the TMI score was proposed as a tool to determine which clustering result is the optimal one — which requires multiple tickets evaluation.
Thus, it is hard to determine the TMI scores of a ﬁlter, as it is sensitive to the choice of k in Equation 1.
To address such problem, we run Equation 1 on the same set of k s as our TMI-GKP algorithm, then we rank (sort) all ﬁlters according to their TMI scores under each k .
Namely, when k = 35 , we may have ﬁlter A to be rank 1 , ﬁlter B to be rank 2 ... and for k = 36 we may have B to be rank 1 and ﬁlter C to be rank 2 .
We then sum all ranks of each ﬁlter across different k s, where the ﬁlter with smallest sum (highest sum of ranks) is considered the one that is most preferred by the TMI system.
We denote this sum the TMI Filter Ranking Score .
We opt to use rank-per-each- k , but not simply adding TMI scores of the same ﬁlter on different k s together and rank all ﬁlters, because the former approach is a) less sensitive to potential extreme value introduced by a certain k and b) similar to the proposed scoring mechanism deﬁned in Equation 4, which is used in TMI-GKP.
We zero-mask different ﬁlters according to the following ﬁve criteria.
All experiments are conducted on ResNet-32 (baseline accuracy: 92.82%) with pruning rate set to 50% (half of the ﬁlters per each layer are zero masked).
TMI preferred : We kept ﬁlters that have the lowest TMI Filter Ranking Scores , and zeromasked the rest.
TMI complement : We kept ﬁlters that have the highest TMI Filter Ranking Scores , and zero-masked the rest.
Since the pruning rate is set to 50%, this is essentially taking the complement of the above scheme.
MI preferred : We kept ﬁlters that have the highest magnitude increase since initialization, and zero-masked the rest.
MI complement : The complement of above scheme.
Random :
Half of the ﬁlters per each layer were randomly zero-masked.
[Table]
The above experiments clearly demonstrate that both better TMI ﬁlter ranking scores and larger MI scores are correlated with better accuracy retention.
In addition, the 1 . 23% lead of TMI preferred to MI preferred scheme also implies that our TMI system — with multiple tickets evaluation — is superior than the original magnitude increase system introduced in Zhou et al.
at the ﬁlter level.
A.4. TMI- DRIVEN CLUSTERING AND ACCURACY RETENTION
Knowing that TMI preferred ﬁlters might lead to better accuracy retention.
We are interested in learning if the clustering results produced by the TMI system may also lead to better accuracy retention.
Thus, we implement the following criterion:
• TMI shufﬂed : Take a clustering result determined by the TMI scoring system ( argmin of Equation 5) and shufﬂe its ﬁlters across different groups.
By “shufﬂe,” we mean that if a set of ﬁlters were originally in the same group, after the shufﬂe, no two of the above-mentioned ﬁlters will be in the same group anymore.
[Table]
The experiment results conﬁrmed that our TMI system may provide positive contribution even under the ﬁlter clustering context.
Although the improvement is not as signiﬁcant as the one provide by the greedy approach (see A.4.6), the improvement is consistent.
Yet without it, our method may not exceed SOTA performance at all.
We further include a discussion in A.4.6 on why empirical evidence suggests the TMI-driven clustering may not guarantee on ﬁnding the best clustering in terms of accuracy retention, but rather a robust policy with more comprehensive considerations done to deliver a “better” and very usable solution.
A.4. TMI- DRIVEN CLUSTERING V . O THER FILTER CLUSTERING SCHEMES
We set the greedy grouped kernel pruning procedure to ﬁxed and feed our method with different schemes on ﬁlter clustering: three clustering schemes individually and a uniform random assignment of the three clustering schemes.
The results with ResNet-32 on CIFAR-10 suggest the TMI-driven clustering may deliver the best accuracy retention. Method
BA ( % ) ∆ Acc ( % )
[Table]
( % ) ↓ FLOPs( % )
ResNet-32 on CIFAR-10: FLOPs: 6.95E7 Params: 4.29E
TMI Clustering w/ Greedy 92.
↑ 0.
K-PCA + k -Means w/
Greedy 92.
↑ 0.
Spectral + k -Means w/ Greedy 92.
↓ 0.
t -SNE + DBSCAN w/
Greedy 92.
↓ 0. Random Clustering Schemes w/ Greedy 92. ↑ 0. However, by conducting experiments for the A.4.6, we notice that for the combination of ResNet110 on CIFAR-10, forcing the clustering scheme as Spectral Clustering may achieve slightly superior accuracy retention to TMI-GKP ( +0 . 06% ).
But by running the same clustering scheme on other experiment combinations like ResNet-20 and ResNet-32 on CIFAR-10, spectral clustering induces much lower results ( − 0 .
33% and − 0 .
16% in comparison to TMI-GKP).
This implies the TMIdriven clustering may not guarantee on ﬁnding the best clustering in terms of accuracy retention, but rather a robust policy with more comprehensive considerations done to deliver a “better” solution than many other structured pruning methods when combined with our greedy-induced pruning procedure.
We observe similar phenomena on the K-PCA + k -Means clustering scheme as it only shows slightly lower performance ( − 0 .
21% in comparison to TMI-GKP) on ResNet-32 on CIFAR-10.
But the same clustering scheme performs consistently worse than TMI-GKP on ResNet-20/56/100 with
CIFAR-10: coming as respectively − 0 .
38% , − 0 .
29% , and − 0 .
24% to our TMI-GKP method; which implies TMI-GKP is likely more robust on providing a “better” solution across different networks.
A.4. TMI-GKP V . G ROUPED CONVOLUTION - ONLY
Since the architecture of grouped convolution itself may induce a reduction of parameters, we are also interested in learning how much it contributes to accuracy retention.
We modiﬁed the original network as a vanilla grouped convolution, trained it with the same training settings as in A.3.1, and compared it against TMI-GKP. With a comparable amount of parameters and FLOPs reduction, the results achieved with grouped convolution-only are well below our TMI-GKP. [Table] [Table]
[Table]
[Table]
64E [Table] Conv ( n
[Table]
[Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] 27E
[Table]
[Table] 53E [Table] ( n
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
A.4. G REEDY - INDUCED PRUNING AND ACCURACY RETENTION
Our greedy grouped kernel pruning approach is developed on the assumption of (grouped) kernels that are most distinctive from each other are better .
We now put this assumption to test with the following two schemes:
Greedy complement : where we keep the group kernels that was originally pruned in TMIGKP.
Greedy reverse : where we ﬂip the arg max on line 11 of Algorithm A.2.1 to arg min with β = 0 in Equation 5.
This means the algorithm is now searching for the next grouped kernel that is “most similar” to the selected grouped kernels.
Here are the experiment results with pruning rate set to 43 .
[Table]
The experiment results once again demonstrate the signiﬁcance of our greedy approach.
In addition, we observe Greedy reverse to have better accuracy retention to Greedy complement .
We believe this is because the pruning strategy produced by Greedy reverse may have overlaps with the one produced by TMI-GKP; yet the pruning strategy produced Greedy complement is mutually exclusive with the one produced by TMI-GKP.
This indirectly suggests the grouped kernels selected by our greedy approach are certainly “the better” ones, as only a partial overlap with the TMI-GKP’s pruning strategy may lead to noticeably better accuracy retention.
Upon request, we further investigated Greedy reverse with β = 1 on ResNet-20/32/56/110 on
CIFAR-10, where such scheme is identical to Greedy reverse with the exception of setting β to 1 .
This means the algorithm is still searching for the next grouped kernel that is “most similar” to the selected grouped kernels.
But among the C (cid:96) in candidate pruning strategies, it will pick the one with best outer homogeneity (pruned and kept ﬁlters are most similar).
The results are ↓ 1.52%, ↓ 0.42%, ↓ 0.45%, and ↓ 0.61% respectively to the baseline of four aforementioned networks.
This is in line with our anticipation.
As per the design of Algorithm 2 and Equation 5, the set of grouped kernels preserved by greedy reverse with β = 1 should be very similar to the set group kernels preserved by greedy reverse with β = 0 .
This is because both of them will preserve a set of grouped kernels that are very similar to each other; and as 56.25% of grouped kernels per layer are preserved, the two policies might very likely end up on the same (or similar) set of grouped kernels.
Empirical evidence supports this hypothesis, as the performance differences between the two policies are marginal.
A.4. G REEDY - INDUCED PRUNING V . O THER GROUPED KERNEL PRUNING POLICIES
We set the ﬁlter clustering scheme to ﬁx as Spectral Clustering and feed our method with different commonly applied grouped kernel pruning policies ( L 2 and center pruning).
Experiments of
ResNet-32/110 on CIFAR-10 have conﬁrmed that our greedy-induced pruning scheme may yield better accuracy retention in a controlled setting.
Method
BA ( % ) ∆ Acc ( % ) ↓ Params ( % ) ↓ FLOPs( % )
ResNet-32 on CIFAR-10: FLOPs: 6.95E7 Params: 4.29E
Greedy-induced Pruning 92.
↑ 0.
L 2 Pruning 92.
↓ 0.
Center Pruning 92.
↓ 0.
ResNet-110 on CIFAR-10: FLOPs: 2.55E8 Params: 1.73E
Greedy-induced Pruning 94.
↑ 0.
L 2 Pruning 93.
↓ 0.
Center Pruning 93.
↓ 0.
A
R UNTIME ANALYSIS OF -GKP
The following experiments are conducted on a 2.00GHz 4 core Intel Xeon CPU and Tesla V100.
Evaluated on 35 potential ticket epochs.
In our code implementation, we ﬁrst cluster a layer, then prune it, then move on to the next layer.
So the runtime is a mixed product of both the TMI ﬁlter clustering procedure and the greedy grouped kernel pruning procedure.
Please refer to the table below for the runtime of the pruning procedure of our TMI-GKP method.
All experiments are against the CIFAR-10 dataset.
[Table]
Clustering and Pruning Runtime
[Table]
[Table]
[Table]
We also separately analyze the runtime of our greedy grouped kernel pruning procedure.
This is done by assigning a pre-determined permutation matrix to the network (same effect as clustering, as it permutes a convolutional layer from Figure 1(a) to Figure 1(b)), so the actual greedy-only approach will be even slightly faster).
Also note the runtime of this greedy procedure is theoretically related to the choice of γ in Equation 5, but the greedy procedure itself is so fast to the point the value of γ does not matter anymore. [Table]
[Table]
We consider this sort of runtime is totally tolerable as an overhead.
When compared to methods involve iterative prune-train cycles (Wang et al., 2021), custom loss function (Wu et al., 2018), or feature maps analysis (Yu et al., 2017), our method is signiﬁcantly more efﬁcient and applicable to a broader set of pre-trained networks.
A.4. E FFECTS OF DIFFERENT HYPERPARAMETERS CHOICES
As mentioned in A.3.2, the two hyperparameters we tuned are β and γ (derived from γ ratio ) in
Equation 5.
We hereby provide a set of experiments to show how different choices of such two parameters will affect the accuracy retention of the pruned network.
All experiments were done on
ResNet-32 on CIFAR-10 with pruning rate set to 43 .
75% (baseline: 92 .
Note the experiments below were done with setting one parameter ﬁxed and adjusting the other, so the two tables should be inspect in a collective manner.
We ﬁrst ﬁxed γ ratio to 0 .
2 — namely, for every pruned grouped kernel, 20% of all kept grouped kernels which are most similar to such pruned grouped kernel were evaluated — and try with different β settings.
The term “auto” implies β = (cid:0) V ∗ 2 (cid:1) /
( | g (cid:96) \ V ∗ | · γ ) with respect to Equation 5, please refer to A.3.2 on how we inferred this value.
Fixed / Tuned param [Equation]
[Equation]
[Equation] [Equation]
auto γ ratio =
The experiment results suggest a relatively more “balanced” relationship between the two terms in
Equation 5 may lead to better accuracy retention.
This implies both the inner heterogeneity of kept grouped kernels and outer homogeneity between kept and pruned grouped kernels should be taken into consideration — which is exactly how Equation 5 was designed.
We then ﬁxed β to “auto” and try with different γ ratio settings:
Fixed / Tuned param γ ratio = 0 .γ
ratio = 0 .γ ratio =
.γ
ratio = 0 .γ ratio =
.γ
ratio = 1 β = auto 92.
The experiment results suggest γ ratio should be a relatively small value as increasing its value may lead to worse accuracy retention of the pruned network.
This is a rather intuitive result, as when γ ratio = 1 every pruned grouped kernel is evaluated against all kept grouped kernels.
In such case, the second term of Equation 5 can no longer reveal if there are some kept grouped kernels that are similar to a pruned one, because all kept kernels are evaluated, yet all kept kernels are already distinctive from each others due to the selection procedure introduced in A.2.1.
Additionally, we have conducted/disclosed more experiments with speciﬁc settings required by one reviewer (e.g., with β = and γ ratio = 0 .
1 on ResNets; choice of β being 2 verses auto).
Please refer to our OpenReview entry for more details — as these speciﬁcally requested experiments should only make sense under the Q&A context, and they do not ﬁt well to the structure of our ablation studies.
A. D IMENSIONALITY R EDUCTION AND C LUSTERING S CHEMES A.5. K ERNEL P RINCIPAL C OMPONENT A NALYSIS (K-PCA)
Kernel PCA is considered an improved version of PCA.
The choice of kernel functions will significantly affect the experimental results.
Global kernels (e.g. polynomial, Sigmoid) and local kernels (e.g. RBF, Laplacian) are the most commonly used kernel functions for K-PCA.
They capture different characteristics of the data: the former one demonstrates better extrapolation abilities, and the latter one has better interpolation abilities (Jordaan, 2004).
Since the ﬁlters are usually not linearly separable and often there are fewer ﬁlters than the number of features, we will mix the global and local kernel functions to beneﬁt both extrapolation and interpolation properties.
Speciﬁcally, we mixed a polynomial and an RBF kernel.
We use a parameter λ to control the balance of these two functions.
[Equation]
where λ ∈ [0 , 1) , parameters a
and c in polynomial kernel have been ﬁxed to 1 and 0.
d = 2 is the degree of polynomial kernel and σ in RBF kernel is the reciprocal of the width of the radial basis function.
In practice, such λ is set ﬁxed to 0 .
A.5. S PECTRAL C LUSTERING
Spectral Clustering is yet another maturely studied and widely adopted clustering method.
Unlike in
K-PCA, for this method we simply utilize the implementation of Von Luxburg (2007).
Speciﬁcally in TMI-GKP, we utilize the variation of mutual-KNN as the similarity graph and cosine distance as the adjacency matrix based on empirical observations.
We rely on Yikun Zhang’s version of implementation.
A.5. E QUAL -S IZED k -M EANS
To ensure we may have n equal-sized groups out of each ﬁlter group, we utilize the default Samesize k -Means Variation offered by Schubert & Zimek (2019).
Speciﬁcally, we rely on Nathan
Danielsen’s version of implementation.
A.5. t -SNE AND DBSCAN
We explore the option of t -SNE and DBSCAN because we want to add a density-based clustering scheme from the two distance-based offerings.
For implementation details: we set the perplexity of t -SNE to be the size of the ﬁlter group (namely, C (cid:96) in /n in layer W (cid:96) ) and iterate through different combinations of n components , (cid:15) , and min samples until the DBSCAN algorithm may yield a clustering result of ≥ n + 1 groups (for n components ∈ [2 , n ) ).
After that, we execute the same data-point reassignment procedure as listed in Schubert & Zimek (2019) until n equal-sized groups are achieved.
A. A CKNOWLEDGEMENT
We would like to acknowledge and give our warmest thanks to John Mays for his timely help on analytical writing.
[Table]