Original: L OTTERY T ICKETS CAN HAVE S TRUCTURAL S PARSITY
Revised: L OTTERY T ICKETS CAN HAVE S TRUCTURAL S PARSITY

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The lottery ticket hypothesis (LTH) has shown that dense models contain highly sparse subnetworks (i.e., winning tickets ) that can be trained in isolation to match full accuracy.
Revised: The lottery ticket hypothesis (LTH) has shown that dense models contain highly sparse subnetworks (i.e., winning tickets ) that can be trained in isolation to match full accuracy.

Original: Despite many exciting efforts being made, there is one “commonsense” seldomly challenged: a winning ticket is found by iterative magnitude pruning (IMP) and hence the resultant pruned subnetworks have only unstructured sparsity.
Revised: Despite many exciting efforts being made, there is one “commonsense” seldomly challenged: a winning ticket is found by iterative magnitude pruning (IMP) and hence the resultant pruned subnetworks have only unstructured sparsity.

Original: That gap limits the appeal of winning tickets in practice, since the highly irregular sparse patterns are challenging to accelerate on hardware.
Revised: That gap limits the appeal of winning tickets in practice, since the highly irregular sparse patterns are challenging to accelerate on hardware.

Original: Meanwhile, directly substituting structured pruning for unstructured pruning in IMP damages performance more severely and is usually unable to locate winning tickets.
Revised: Meanwhile, directly substituting structured pruning for unstructured pruning in IMP damages performance more severely and is usually unable to locate winning tickets.

Original: In this paper, we demonstrate the ﬁrst positive result that a structurally sparse winning ticket can be effectively found in general.
Revised: In this paper, we demonstrate the ﬁrst positive result that a structurally sparse winning ticket can be effectively found in general.

Original: The core idea is to append “post-processing techniques” after each round of (unstructured) IMP, to enforce the formation of structural sparsity.
Revised: The core idea is to append “post-processing techniques” after each round of (unstructured) IMP, to enforce the formation of structural sparsity.

Original: Speciﬁcally, we ﬁrst “re-ﬁll” pruned elements back in some channels deemed to be important, and then “re-group” non-zero elements to create ﬂexible group-wise structural patterns.
Revised: Speciﬁcally, we ﬁrst “re-ﬁll” pruned elements back in some channels deemed to be important, and then “re-group” non-zero elements to create ﬂexible group-wise structural patterns.

Original: Both our identiﬁed channel- and group-wise structural subnetworks win the lottery, with substantial inference speedups readily supported by practical hardware.
Revised: Both our identiﬁed channel- and group-wise structural subnetworks win the lottery, with substantial inference speedups readily supported by practical hardware.

Original: Extensive experiments, conducted on diverse datasets across multiple network backbones, consistently validate our proposal, showing that the hardware acceleration roadblock of LTH is now removed .
Revised: Extensive experiments, conducted on diverse datasets across multiple network backbones, consistently validate our proposal, showing that the hardware acceleration roadblock of LTH is now removed .

Original: Speciﬁcally, the structural winning tickets obtain up to { 64 .
Revised: Speciﬁcally, the structural winning tickets obtain up to { 64 .

Original: 84% } running time savings at { 36% ∼ 80% , 74% , 58% } sparsity on { CIFAR, Tiny-ImageNet, ImageNet } , while maintaining comparable accuracy.
Revised: 84% } running time savings at { 36% ∼ 80% , 74% , 58% } sparsity on { CIFAR, Tiny-ImageNet, ImageNet } , while maintaining comparable accuracy.

Original: All the codes and pre-trained models will be publicly released.
Revised: All the codes and pre-trained models will be publicly released.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Recently, the machine learning research community has devoted considerable efforts and ﬁnancial outlay to scaling deep neural networks (DNNs) to enormous sizes ( 175 billion parameter-counts in
Revised: Recently, the machine learning research community has devoted considerable efforts and ﬁnancial outlay to scaling deep neural networks (DNNs) to enormous sizes ( 175 billion parameter-counts in

Original: GPT-3 (Brown et al., 2020)).
Revised: GPT-3 (Brown et al., 2020)).

Original: Although such overparameterization simpliﬁes the training of DNNs and dramatically improves their generalization (Bartlett et al., 2021; Du et al., 2018; Kaplan et al., 2020), it may severely obstruct the practical usage on resource-limited platforms like mobile devices, due to its large memory footprint and inference time (Hoeﬂer et al., 2021).
Revised: Although such overparameterization simpliﬁes the training of DNNs and dramatically improves their generalization (Bartlett et al., 2021; Du et al., 2018; Kaplan et al., 2020), it may severely obstruct the practical usage on resource-limited platforms like mobile devices, due to its large memory footprint and inference time (Hoeﬂer et al., 2021).

Original: Pruning as one of the effective remedies can be dated back to LeCun et al.
Revised: Pruning as one of the effective remedies can be dated back to LeCun et al.

Original: (1990): it can eliminate substantial redundant model parameters and boost the computational and storage efﬁciency of DNNs.
Revised: (1990): it can eliminate substantial redundant model parameters and boost the computational and storage efﬁciency of DNNs.

Original: Such beneﬁts drive numerous interests in designing model pruning algorithms (Han et al., 2015a;b;
Revised: Such beneﬁts drive numerous interests in designing model pruning algorithms (Han et al., 2015a;b;

Original: Ren et al., 2018;
Revised: Ren et al., 2018;

Original: He et al., 2017; Liu et al., 2017).
Revised: He et al., 2017; Liu et al., 2017).

Original: Among this huge family, an emerging representative studies the prospect of training sparse subnetworks in lieu of the full dense models without impacting performance (Frankle & Carbin, 2019; Chen et al., 2020b).
Revised: Among this huge family, an emerging representative studies the prospect of training sparse subnetworks in lieu of the full dense models without impacting performance (Frankle & Carbin, 2019; Chen et al., 2020b).

Original: For instance, Frankle & Carbin (2019) demonstrates that dense models contain sparse, matching subnetworks (Frankle et al., 2019) (a.k.a.
Revised: For instance, Frankle & Carbin (2019) demonstrates that dense models contain sparse, matching subnetworks (Frankle et al., 2020a) (a.k.a.

Original: winning tickets ) capable of training in isolation from the original initialization to match or even surpass the full accuracy.
Revised: winning tickets ) capable of training in isolation from the original initialization to match or even surpass the full accuracy.

Original: This phenomenon is referred to as the lottery tickets hypothesis (LTH), which indicates several impressive observations: ( i ) usually extreme sparsity levels (e.g., 90% , 95% ) can be achieved without sacriﬁcing the test accuracy; ( ii ) the located winning ticket maintains undamaged expressive power as its dense counterpart, and can be easily trained from scratch or early-epoch weights (Renda et al., 2020; Frankle et al., 2020a) to recover the full performance.
Revised: This phenomenon is referred to as the lottery tickets hypothesis (LTH), which indicates several impressive observations: ( i ) usually extreme sparsity levels (e.g., 90% , 95% ) can be achieved without sacriﬁcing the test accuracy; ( ii ) the located winning ticket maintains undamaged expressive power as its dense counterpart, and can be easily trained from scratch or early-epoch weights (Renda et al., 2020; Frankle et al., 2020a) to recover the full performance.

Original: These advances are positive signs about the substantial potential of sparse DNNs.
Revised: These advances are positive signs about the substantial potential of sparse DNNs.

Original: However, almost all LTH literature investigates unstructured sparsity only.
Revised: However, almost all LTH literature investigates unstructured sparsity only.

Original: In practical scenarios, it brings little hardware efﬁciency beneﬁts due to the poor data locality and low parallelism (He et al., 2017; Mao et al., 2017; Wen et al., 2016) caused by highly irregular sparse patterns.
Revised: In practical scenarios, it brings little hardware efﬁciency beneﬁts due to the poor data locality and low parallelism (He et al., 2017; Mao et al., 2017; Wen et al., 2016) caused by highly irregular sparse patterns.

Original: Meanwhile, most of the accelerators are optimized for dense matrix operations (Han et al., 2016), which means there is limited speedup for unstructured pruned subnetworks even the sparsity level exceeds 95% (Wen et al., 2016).
Revised: Meanwhile, most of the accelerators are optimized for dense matrix operations (Han et al., 2016), which means there is limited speedup for unstructured pruned subnetworks even the sparsity level exceeds 95% (Wen et al., 2016).

Original: Structural pruning (He et al., 2017; Liu et al., 2017) as an alternative to exploring sparse subnetworks, removes the entire ﬁlter or channel in DNNs to gain more computational efﬁciency at the cost of (more) accuracy degradation.
Revised: Structural pruning (He et al., 2017; Liu et al., 2017) as an alternative to exploring sparse subnetworks, removes the entire ﬁlter or channel in DNNs to gain more computational efﬁciency at the cost of (more) accuracy degradation.

Original: As shown in Figure 1, traditional channel-wise structural pruning approaches (He et al., 2017; Liu et al., 2017; Bartoldson et al., 2019;
Revised: As shown in Figure 1, traditional channel-wise structural pruning approaches (He et al., 2017; Liu et al., 2017; Bartoldson et al., 2019;

Original: Molchanov et al., 2019) quickly degrade performance and cannot lead to winning tickets, which was also echoed in You et al.(2020b). [Figure] ResNet- [Figure]
Revised: Molchanov et al., 2019) quickly degrade performance and cannot lead to winning tickets, which was also echoed in You et al.

Original: , CIFAR-

Original: [Figure]
Revised: [Figure]

Original: In our paper, we present the ﬁrst study into the structural lottery tickets , which explores hardwarefriendly structural sparsity (including channel-wise and group-wise patterns) in order to ﬁnd lottery tickets.
Revised: In our paper, we present the ﬁrst study into the structural lottery tickets , which explores hardwarefriendly structural sparsity (including channel-wise and group-wise patterns) in order to ﬁnd lottery tickets.

Original: Speciﬁcally, we start from unstructured sparse subnetworks, and then adopt proposed reﬁlling techniques to create channel-wise structural sparsity by growing back the pruned elements within the most important channels and abandoning the rest.
Revised: Speciﬁcally, we start from unstructured sparse subnetworks, and then adopt proposed reﬁlling techniques to create channel-wise structural sparsity by growing back the pruned elements within the most important channels and abandoning the rest.

Original: Our results (Section 4) show such reﬁned channel-wise structural subnetworks win the lottery at a moderate sparsity level with ∼ 50% running time savings on an
Revised: Our results (Section 4) show such reﬁned channel-wise structural subnetworks win the lottery at a moderate sparsity level with ∼ 50% running time savings on an

Original: Nvidia 2080 TI.
Revised: Nvidia 2080 TI.

Original: In order to push the compression ratio higher, we introduce a regrouping algorithm based on hypergraph partitioning (Rumi et al., 2020) to establish group-wise structural patterns which are more amenable to pruning due to the shape ﬂexibility of grouped dense blocks.
Revised: In order to push the compression ratio higher, we introduce a regrouping algorithm based on hypergraph partitioning (Rumi et al., 2020) to establish group-wise structural patterns which are more amenable to pruning due to the shape ﬂexibility of grouped dense blocks.

Original: These group-wise structural winning tickets achieve ∼ 60% running time savings at 50% ∼ 80% sparsity without any performance degradation compared to the dense models.
Revised: These group-wise structural winning tickets achieve ∼ 60% running time savings at 50% ∼ 80% sparsity without any performance degradation compared to the dense models.

Original: Our main contributions lie in the following aspects:
Revised: Our main contributions lie in the following aspects:

Original: • To our best knowledge, we are the ﬁrst to demonstrate the existence of structurally sparse winning tickets at non-trivial sparsity levels (i
Revised: • To our best knowledge, we are the ﬁrst to demonstrate the existence of structurally sparse winning tickets at non-trivial sparsity levels (i

Original: e., > 30% ), and with both channel-wise and group-wise sparse patterns.
Revised: e., > 30% ), and with both channel-wise and group-wise sparse patterns.

Original: • We propose the reﬁlling technique and introduce the regrouping algorithm to form channelwise and group-wise structural sparsity, respectively.
Revised: • We propose the reﬁlling technique and introduce the regrouping algorithm to form channelwise and group-wise structural sparsity, respectively.

Original: Such reﬁned structural subnetworks match the trainability and expressiveness of dense networks, while enabling the inference speedup on practical hardware platforms like GPU machines.
Revised: Such reﬁned structural subnetworks match the trainability and expressiveness of dense networks, while enabling the inference speedup on practical hardware platforms like GPU machines.

Original: • Extensive experiments validate our proposal on diverse datasets (i
Revised: • Extensive experiments validate our proposal on diverse datasets (i

Original: e., CIFAR-10/100, TinyImageNet, and ImageNet) across multiple network architectures, including ResNets, VGG, and MobileNet.
Revised: e., CIFAR-10/100, TinyImageNet, and ImageNet) across multiple network architectures, including ResNets, VGG, and MobileNet.

Original: Speciﬁcally, our structural winning tickets achieve 53 .
Revised: Speciﬁcally, our structural winning tickets achieve 53 .

Original: 75% ∼ 64 .
Revised: 75% ∼ 64 .

Original: 93% GPU running time savings at 45% ∼ 80% channel- and group-wise sparsity.
Revised: 93% GPU running time savings at 45% ∼ 80% channel- and group-wise sparsity.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Pruning.
Revised: Pruning.

Original: Network pruning is a technique that aims at eliminating the unnecessary model parameters (Blalock et al., 2020), which can effectively shrink models for the deployment on resourceconstrained devices (LeCun et al., 1990; Hanson & Pratt, 1988).
Revised: Network pruning is a technique that aims at eliminating the unnecessary model parameters (Blalock et al., 2020), which can effectively shrink models for the deployment on resourceconstrained devices (LeCun et al., 1990; Hanson & Pratt, 1988).

Original: Pruning algorithms are roughly categorized into two groups: (1) unstructured pruning (LeCun et al., 1990; Han et al.
Revised: Pruning algorithms are roughly categorized into two groups: (1) unstructured pruning (LeCun et al., 1990; Han et al.

Original: , 2015a;b; Ren et al., 2018; Zhang et al., 2018a) with irregular sparse patterns; (2) structural pruning (He et al., 2017;
Revised: , 2015a;b; Ren et al., 2018; Zhang et al., 2018) with irregular sparse patterns; (2) structural pruning (He et al., 2017;

Original: Liu et al., 2017; Li et al., 2016;
Revised: Liu et al., 2017; Li et al., 2016;

Original: Hu et al., 2016; Wen et al., 2016; Hong et al., 2018) with structural sparse patterns such as channel-wise, block-wise, column-wise, etc..
Revised: Hu et al., 2016; Wen et al., 2016; Hong et al., 2018) with structural sparse patterns such as channel-wise, block-wise, column-wise, etc..

Original: Within the group of unstructured pruning, Han et al.
Revised: Within the group of unstructured pruning, Han et al.

Original: (2015a;b) remove insigniﬁcant connections of models in the post-training stage, with respect to certain heuristics like weight/gradient magnitudes; during training sparsiﬁcation is also another popular trend for pruning by leveraging (cid:96) 0 regularization (Louizos et al., 2017) or alternating direction method of multipliers (ADMM)
Revised: (2015a;b) remove insigniﬁcant connections of models in the post-training stage, with respect to certain heuristics like weight/gradient magnitudes; during training sparsiﬁcation is also another popular trend for pruning by leveraging (cid:96) 0 regularization (Louizos et al., 2017) or alternating direction method of multipliers (ADMM)

Original: (Ren et al.,
Revised: (Ren et al.,

Original: ; Zhang et al., 2018a).
Revised: ; Zhang et al., 2018).

Original: Recently, several pruning-at-initialization methods (Wang et al., 2020;
Revised: Recently, several pruning-at-initialization methods (Wang et al., 2020;

Original: Lee et al., 2019b; Tanaka et al., 2020) are proposed to identify critical unstructured connections for gradient-ﬂow preserving, without any training.
Revised: Lee et al., 2019b; Tanaka et al., 2020) are proposed to identify critical unstructured connections for gradient-ﬂow preserving, without any training.

Original: Although the unstructured sparse model has superior performance, it usually suffers from poor data locality and low parallelism (He et al., 2017; Mao et al., 2017; Wen et al., 2016), which make it hard to speed up in real-world applications.
Revised: Although the unstructured sparse model has superior performance, it usually suffers from poor data locality and low parallelism (He et al., 2017; Mao et al., 2017; Wen et al., 2016), which make it hard to speed up in real-world applications.

Original: On the contrary, structural pruning is more hardware-friendly at the cost of notable accuracy loss when the compression ratio increases.
Revised: On the contrary, structural pruning is more hardware-friendly at the cost of notable accuracy loss when the compression ratio increases.

Original: He et al.
Revised: He et al.

Original: ; Liu et al.
Revised: ; Liu et al.

Original: (2017) slim the network channels via (cid:96) 1 regularization, and Bartoldson et al.
Revised: (2017) slim the network channels via (cid:96) 1 regularization, and Bartoldson et al.

Original: (2019) selects important channels according to heuristics of feature maps.
Revised: (2019) selects important channels according to heuristics of feature maps.

Original: To combine the beneﬁts of structural and unstructured pruning, hybrid pruning strategies have been introduced to pursue more general structural spares patterns which are also capable of acceleration.
Revised: To combine the beneﬁts of structural and unstructured pruning, hybrid pruning strategies have been introduced to pursue more general structural spares patterns which are also capable of acceleration.

Original: For example, convolution kernels with half regular sparsity (Chen et al., 2018) or pattern-based structural sparsity (Ma et al., 2020a) or vector-wise (Zhu et al., 2019) and group-wise (Rumi et al., 2020) regular sparsity.
Revised: For example, convolution kernels with half regular sparsity (Chen et al., 2018) or pattern-based structural sparsity (Ma et al., 2020) or vector-wise (Zhu et al., 2019) and group-wise (Rumi et al., 2020) regular sparsity.

Original: The lottery tickets hypothesis (LTH).
Revised: The lottery tickets hypothesis (LTH).

Original: The lottery ticket hypothesis (LTH) (Frankle & Carbin, 2019) conjectures that there exists a sparse subnetwork called winning ticket within a dense network, whose performance can match with the dense network when training from the same initialization.
Revised: The lottery ticket hypothesis (LTH) (Frankle & Carbin, 2019) conjectures that there exists a sparse subnetwork called winning ticket within a dense network, whose performance can match with the dense network when training from the same initialization.

Original: With the assistance of weight rewinding techniques (Renda et al., 2020; Frankle et al., 2020a), the original LTH can be scaled up to larger networks and datasets.
Revised: With the assistance of weight rewinding techniques (Renda et al., 2020; Frankle et al., 2020a), the original LTH can be scaled up to larger networks and datasets.

Original: The existence of winning tickets are broadly veriﬁed under diverse contexts, such as image classiﬁcation (Frankle & Carbin, 2019;
Revised: The existence of winning tickets are broadly veriﬁed under diverse contexts, such as image classiﬁcation (Frankle & Carbin, 2019;

Original: Liu et al., 2019; Wang et al., 2020; Evci et al., 2019; Frankle et al., 2020b; Savarese et al., 2020;
Revised: Liu et al., 2019; Wang et al., 2020; Evci et al., 2019; Frankle et al., 2020b; Savarese et al., 2020;

Original: You et al., 2020a; Ma et al., 2021a; Chen et al., 2020a), object detection Girish et al.
Revised: You et al., 2020; Ma et al., 2021a; Chen et al., 2020a), object detection Girish et al.

Original: natural language processing Gale et al.
Revised: natural language processing Gale et al.

Original: ; Yu et al.
Revised: ; Yu et al.

Original: (2020); Prasanna et al.
Revised: (2020); Prasanna et al.

Original: ; Chen et al.
Revised: ; Chen et al.

Original: (2020b;c), generative adversarial networks Chen et al.
Revised: (2020b;c), generative adversarial networks Chen et al.

Original: (2021d)
Revised: (2021d)

Original: ; Kalibhat et al.
Revised: ; Kalibhat et al.

Original: ; Chen et al.
Revised: ; Chen et al.

Original: (2021a)
Revised: (2021a)

Original: , graph neural networks Chen et al.
Revised: , graph neural networks Chen et al.

Original: (2021b), reinforcement learning Yu et al.
Revised: (2021b), reinforcement learning Yu et al.

Original: (2020), and
Revised: (2020), and

Original: life-long learning Chen et al.
Revised: life-long learning Chen et al.

Original: (2021c).
Revised: (2021c).

Original: However, all of the above LTH literature only locate unstructured sparse winning tickets, which can hardly bring hardware efﬁciency boost on real-world applications.
Revised: However, all of the above LTH literature only locate unstructured sparse winning tickets, which can hardly bring hardware efﬁciency boost on real-world applications.

Original: As the most related work, You et al.
Revised: As the most related work, You et al.

Original: (2019) ﬁnds structural winning tickets at only low sparsity levels around 30% in few cases.
Revised: (2020) ﬁnds structural winning tickets at only low sparsity levels around 30% in few cases.

Original: It again reveals the complication and difﬁculty of identifying computation-friendly sparse patterns.
Revised: It again reveals the complication and difﬁculty of identifying computation-friendly sparse patterns.

Original: Another concurrent work (Alabdulmohsin et al., 2021) investigates a generalized LTH with weight space factorization, which is orthogonal to our work.
Revised: Another concurrent work (Alabdulmohsin et al., 2021) investigates a generalized LTH with weight space factorization, which is orthogonal to our work.

Original: Sparse convolutional neural network (CNN) acceleration on GPU.
Revised: Sparse convolutional neural network (CNN) acceleration on GPU.

Original: Previous works have explored the acceleration of sparse convolution operations in two different directions.
Revised: Previous works have explored the acceleration of sparse convolution operations in two different directions.

Original: One direction is to design efﬁcient implementation of unstructured pruned networks for improved data locality and utilization of hardware (Chen, 2018; Park et al., 2016).
Revised: One direction is to design efﬁcient implementation of unstructured pruned networks for improved data locality and utilization of hardware (Chen, 2018; Park et al., 2016).

Original: For example, Dong et al.
Revised: For example, Dong et al.

Original: (2019) proposes “Acorns” to accelerate the sparse computations of convolution kernels with an input sparsity.
Revised: (2019) proposes “Acorns” to accelerate the sparse computations of convolution kernels with an input sparsity.

Original: Peng et al.
Revised: Peng et al.

Original: (2017) has proposed a matrix splitting algorithm for efﬁcient CNN inference.
Revised: (2017) has proposed a matrix splitting algorithm for efﬁcient CNN inference.

Original: Nvidia’s cuSPARSE 1 library contains various efﬁcient sparse matrix computation algorithms like SpMM on
Revised: Nvidia’s cuSPARSE 1 library contains various efﬁcient sparse matrix computation algorithms like SpMM on

Original: GPUs, drawing great attention in efﬁcient scientiﬁc computing.
Revised: GPUs, drawing great attention in efﬁcient scientiﬁc computing.

Original: Furthermore, advanced approaches are developed based on SpMM, such as Adaptive Sparse Tiling (ASpT) (Hong et al., 2019).
Revised: Furthermore, advanced approaches are developed based on SpMM, such as Adaptive Sparse Tiling (ASpT) (Hong et al., 2019).

Original: ASpT signiﬁcantly improves the data usage of SpMM and achieves the current state-of-the-art performance among SpMM implementation variants.
Revised: ASpT signiﬁcantly improves the data usage of SpMM and achieves the current state-of-the-art performance among SpMM implementation variants.

Original: Another direction focuses on more hardware-friendly pruning methods (Chen et al., 2018; Ma et al., 2020b; Niu et al., 2020).
Revised: Another direction focuses on more hardware-friendly pruning methods (Chen et al., 2018; Ma et al., 2020; Niu et al., 2020).

Original: During the model pruning, these works aim to maintain certain regular sparse patterns, which beneﬁt the hardware processing/computing of corresponding sparse matrices.
Revised: During the model pruning, these works aim to maintain certain regular sparse patterns, which beneﬁt the hardware processing/computing of corresponding sparse matrices.

Original: However, Chen et al.
Revised: However, Chen et al.

Original: (2018) achieves unsatisfactory compression ratio, while the pruning methods used in Ma et al.
Revised: (2018) achieves unsatisfactory compression ratio, while the pruning methods used in Ma et al.

Original: (2020b) and Niu et al.
Revised: (2020) and Niu et al.

Original: (2020) require dedicated compiler optimization to accelerate network execution.
Revised: (2020) require dedicated compiler optimization to accelerate network execution.

Original:  M ETHODOLOGY 3. N OTATIONS AND P RELIMINARIES
Revised:  M ETHODOLOGY 3. N OTATIONS AND P RELIMINARIES

Original: Sparse subnetworks and pruning methods.
Revised: Sparse subnetworks and pruning methods.

Original: In this paper, we mainly follow the routine notations in Frankle & Carbin (2019); Renda et al.
Revised: In this paper, we mainly follow the routine notations in Frankle & Carbin (2019); Renda et al.

Original: For a network f ( x ; θ ) with input samples x and model parameters θ , a sparse subnetwork is a network f
Revised: For a network f ( x ; θ ) with input samples x and model parameters θ , a sparse subnetwork is a network f

Original: ( x ; m (cid:12) θ ) with a binary pruning mask m ∈ { 0 , 1 } | θ | , where (cid:12) is the element-wise product.
Revised: ( x ; m (cid:12) θ ) with a binary pruning mask m ∈ { 0 , 1 } | θ | , where (cid:12) is the element-wise product.

Original: In other words, it is a copy of dense network f
Revised: In other words, it is a copy of dense network f

Original: ( x ; θ ) with some weights ﬁxed to 0 .
Revised: ( x ; θ ) with some weights ﬁxed to 0 .

Original: If the non-ﬁxed remaining weights are distributed irregularly, we call it unstructured sparse patterns (e.g., Figure 2, left ); if they are clustered into channels or groups, we name it structural sparse patterns (e.g., Figure 2, right ).
Revised: If the non-ﬁxed remaining weights are distributed irregularly, we call it unstructured sparse patterns (e.g., Figure 2, left ); if they are clustered into channels or groups, we name it structural sparse patterns (e.g., Figure 2, right ).

Original: To obtain the desired sparse subnetworks, we consider and benchmark multiple classical pruning algorithms: (1) random pruning ( RP ) which usually works as a necessary baseline for the sanctity check (Frankle & Carbin, 2019); (2) one-shot magnitude pruning ( OMP ) by eliminating a part of model parameters with the globally smallest magnitudes (Han et al., 2015a); (3) the lottery ticket hypothesis (Frankle & Carbin, 2019) with iterative weight magnitude pruning ( LTH-IMP or IMP for simplicity) (Han et al., 2015a).
Revised: To obtain the desired sparse subnetworks, we consider and benchmark multiple classical pruning algorithms: (1) random pruning ( RP ) which usually works as a necessary baseline for the sanctity check (Frankle & Carbin, 2019); (2) one-shot magnitude pruning ( OMP ) by eliminating a part of model parameters with the globally smallest magnitudes (Han et al., 2015a); (3) the lottery ticket hypothesis (Frankle & Carbin, 2019) with iterative weight magnitude pruning ( LTH-IMP or IMP for simplicity) (Han et al., 2015a).

Original: As adopted in LTH literature (Frankle & Carbin, 2019), we identify the sparse lottery tickets by iteratively removing the 20% of remaining weight with the globally smallest magnitudes, and rewinding model weights to the original random initialization (Frankle &
Revised: As adopted in LTH literature (Frankle & Carbin, 2019), we identify the sparse lottery tickets by iteratively removing the 20% of remaining weight with the globally smallest magnitudes, and rewinding model weights to the original random initialization (Frankle &

Original: Carbin, 2019) or early training epochs (Frankle et al., 2020b; Chen et al., 2020a).
Revised: Carbin, 2019) or early training epochs (Frankle et al., 2020b; Chen et al., 2020a).

Original: In this paper, the model weights are rewound to the eighth epoch (i.e., the 5% of the entire training process) for all
Revised: In this paper, the model weights are rewound to the eighth epoch (i.e., the 5% of the entire training process) for all

Original: CIFAR, Tiny-ImageNet, and ImageNet experiments.
Revised: CIFAR, Tiny-ImageNet, and ImageNet experiments.

Original: (4) pruning at initialization mechanisms.
Revised: (4) pruning at initialization mechanisms.

Original: We choose several representative approaches such as SNIP (Lee et al., 2019a), GraSP (Wang et al., 2020), and SynFlow (Tanaka et al., 2020), which explore sparse patterns at random initialization with some gradient ﬂow based criterion.
Revised: We choose several representative approaches such as SNIP (Lee et al., 2019a), GraSP (Wang et al., 2020), and SynFlow (Tanaka et al., 2020), which explore sparse patterns at random initialization with some gradient ﬂow based criterion.

Original: (5) Alternating Direction Method of Multipliers ( ADMM ) for punning.
Revised: (5) Alternating Direction Method of Multipliers ( ADMM ) for punning.

Original: It is a well-known optimization-based pruning method (Niu et al., 2020; Zhang et al., 2018b), which can obtain superior compression ratios with little performance degradation for deep neural networks.
Revised: It is a well-known optimization-based pruning method (Niu et al., 2020; Zhang et al., 2018), which can obtain superior compression ratios with little performance degradation for deep neural networks.

Original: Note that all pruning approaches are mainly conducted over networks without counting their classiﬁcation heads (Frankle & Carbin, 2019; Ma et al., 2021b).
Revised: Note that all pruning approaches are mainly conducted over networks without counting their classiﬁcation heads (Frankle & Carbin, 2019; Ma et al., 2021b).

Original: Structural winning tickets.
Revised: Structural winning tickets.

Original: We begin by extending the original lottery tickets hypothesis to the context of structural sparse patterns.
Revised: We begin by extending the original lottery tickets hypothesis to the context of structural sparse patterns.

Original: A subnetwork f ( x ; m (cid:12) θ ) is a structural winning ticket for an algorithm A T t if it satisﬁes: (cid:172) training subnetworks f ( x ; m (cid:12) θ ) with algorithm A T t results in performance measurement on task T no lower than training dense networks f ( x ; θ ) with algorithm
Revised: A subnetwork f ( x ; m (cid:12) θ ) is a structural winning ticket for an algorithm A T t if it satisﬁes: (cid:172) training subnetworks f ( x ; m (cid:12) θ ) with algorithm A T t results in performance measurement on task T no lower than training dense networks f ( x ; θ ) with algorithm

Original: A T t , where θ is the original random initialization θ 0 or early rewound weights like θ 5% , and t is the training iterations; (cid:173) the non-zero elements in pruning mask m are clustered as channels, groups or other hardware-friendly structural patterns.
Revised: A T t , where θ is the original random initialization θ 0 or early rewound weights like θ 5% , and t is the training iterations; (cid:173) the non-zero elements in pruning mask m are clustered as channels, groups or other hardware-friendly structural patterns.

Original: Implementation details.
Revised: Implementation details.

Original: We conduct experiments on diverse combinations of network architectures and datasets.
Revised: We conduct experiments on diverse combinations of network architectures and datasets.

Original: Speciﬁcally, we adopt Wide-ResNet-32-2 (Zagoruyko & Komodakis, 2016) (or
Revised: Speciﬁcally, we adopt Wide-ResNet-32-2 (Zagoruyko & Komodakis, 2016) (or

Original: WRN-32-2)
Revised: WRN-32-2)

Original: , ResNet-18 (He et al., 2016) (or RN-18), MobileNet-v1 (or MBNet-v1) (Howard et al., 2017), and VGG-16 (Simonyan & Zisserman, 2014) on both CIFAR-10 (Krizhevsky et al., 2009) and CIFAR-100 datasets.
Revised: , ResNet-18 (He et al., 2016) (or RN-18), MobileNet-v1 (or MBNet-v1) (Howard et al., 2017), and VGG-16 (Simonyan & Zisserman, 2014) on both CIFAR-10 (Krizhevsky et al., 2009) and CIFAR-100 datasets.

Original: ResNet-50 (or RN-50) is evaluated on both Tiny-ImageNet (Le & Yang, 2015) and ImageNet (Deng et al., 2009) datasets.
Revised: ResNet-50 (or RN-50) is evaluated on both Tiny-ImageNet (Le & Yang, 2015) and ImageNet (Deng et al., 2009) datasets.

Original: Table 1 includes more training and evaluation details of our experiments.
Revised: Table 1 includes more training and evaluation details of our experiments.

Original: [Table]
Revised: [Table]

Original: 3. R EFILLING FOR S TRUCTURAL P ATTERNS
Revised: 3. R EFILLING FOR S TRUCTURAL P ATTERNS

Original: It is well-known that the irregular sparsity patterns from unstructured magnitude pruning block the acceleration on practical hardware devices.
Revised: It is well-known that the irregular sparsity patterns from unstructured magnitude pruning block the acceleration on practical hardware devices.

Original: To overcome the limitation, we propose a simple reﬁlling strategy to reorganize the unstructured sparse patterns and to make them more hardware friendly.
Revised: To overcome the limitation, we propose a simple reﬁlling strategy to reorganize the unstructured sparse patterns and to make them more hardware friendly.

Original: Speciﬁcally, we ﬁrst select important channels from the unstructured subnetwork according to certain criteria.
Revised: Speciﬁcally, we ﬁrst select important channels from the unstructured subnetwork according to certain criteria.

Original: The number of picked channels are depended on the desired sparsity level.
Revised: The number of picked channels are depended on the desired sparsity level.

Original: Then, the pruned elements are grown back to be trainable (i.e., unpruned) and are reset to the same random initialization or early rewound weights.
Revised: Then, the pruned elements are grown back to be trainable (i.e., unpruned) and are reset to the same random initialization or early rewound weights.

Original: Lastly , the rest parameters in the remaining insigniﬁcant channels will be removed.
Revised: Lastly , the rest parameters in the remaining insigniﬁcant channels will be removed.

Original: In this way, we reﬁll important channels and empty the rest to create a channel-wise structural sparse pattern that essentially brings computational reductions.
Revised: In this way, we reﬁll important channels and empty the rest to create a channel-wise structural sparse pattern that essentially brings computational reductions.

Original: Note that the picking criterion can be the number of remaining weights in the channel, or the channel’s weight statistics or feature statistics or salience scores, which are comprehensively investigated in the ablation (Section A2).
Revised: Note that the picking criterion can be the number of remaining weights in the channel, or the channel’s weight statistics or feature statistics or salience scores, which are comprehensively investigated in the ablation (Section A2).

Original: The complete pipeline and illustration are summarized in Algorithm 2 and
Revised: The complete pipeline and illustration are summarized in Algorithm 2 and

Original: Figure 2, respectively.
Revised: Figure 2, respectively.

Original: [Figure]
Revised: [Figure]

Original: Initial Unstructured Sparse Mask
Revised: Initial Unstructured Sparse Mask

Original: Channel-wise Structural Sparse Mask
Revised: Channel-wise Structural Sparse Mask

Original: Refilling or Refilling+
Revised: Refilling or Refilling+

Original: Regrouping
Revised: Regrouping

Original: Group-wise Structural Sparse Mask
Revised: Group-wise Structural Sparse Mask

Original: Remaining Weights
Revised: Remaining Weights

Original: Extra Refilled Channel for Refilling+
Revised: Extra Refilled Channel for Refilling+

Original: Refilled Weights
Revised: Refilled Weights

Original: Pruned Weights
Revised: Pruned Weights

Original: Input: f ( x ; θ 0 ) , unstructured sparsity s
Revised: Input: f ( x ; θ 0 ) , unstructured sparsity s

Original: Output: f ( x ; m (cid:12) θ i ) 1 Set the pruning mask m = 1 ∈ R | θ | 2 Train f ( x ; θ 0 ) to rewinding step i : f ( x ; θ i ) =
Revised: Output: f ( x ; m (cid:12) θ i ) 1 Set the pruning mask m = 1 ∈ R | θ | 2 Train f ( x ; θ 0 ) to rewinding step i : f ( x ; θ i ) =

Original: A T i ( f ( x ; θ 0 )) 3 while not reach sparsity s do
Revised: A T i ( f ( x ; θ 0 )) 3 while not reach sparsity s do

Original: Train f
Revised: Train f

Original: ( x ; m (cid:12) θ i ) to step t : f ( x ; m (cid:12) θ t ) =
Revised: ( x ; m (cid:12) θ i ) to step t : f ( x ; m (cid:12) θ t ) =

Original: A T t − i ( f ( x ; m (cid:12) θ i ))
Revised: A T t − i ( f ( x ; m (cid:12) θ i ))

Original: Pruning 20% of remaining weight of m (cid:12) θ t , and update m 6 end
Revised: Pruning 20% of remaining weight of m (cid:12) θ t , and update m 6 end

Original: Input: f ( x ; m (cid:12) θ i ) with unstructured sparsity s from Algorithm
Revised: Input: f ( x ; m (cid:12) θ i ) with unstructured sparsity s from Algorithm

Original: Output: f ( x ; m (cid:12) θ i ) with channel-wise structural mask m at sparsity ˜ s 1 Calculate importance scores of each channel according to certain criterion 2 Pick top- k channels in m , reﬁll back their 0 (pruned) elements with 1 (trainable) and update m , maintaining ˜ s ∼ s 3 Pick and reﬁll extra channels in m with ˜ s + < s # Optional for Refill+
Revised: Output: f ( x ; m (cid:12) θ i ) with channel-wise structural mask m at sparsity ˜ s 1 Calculate importance scores of each channel according to certain criterion 2 Pick top- k channels in m , reﬁll back their 0 (pruned) elements with 1 (trainable) and update m , maintaining ˜ s ∼ s 3 Pick and reﬁll extra channels in m with ˜ s + < s # Optional for Refill+

Original: Input: f ( x ; m (cid:12) θ i ) with unstructured sparsity s from Algorithm 1, hyperparameters t 1 , t 2 , b 1 , and b
Revised: Input: f ( x ; m (cid:12) θ i ) with unstructured sparsity s from Algorithm 1, hyperparameters t 1 , t 2 , b 1 , and b

Original: Output: f ( x ; m ) (cid:12) θ i with group-wise structural mask m at sparsity s ∗ 1 while dense block can be found do
Revised: Output: f ( x ; m ) (cid:12) θ i with group-wise structural mask m at sparsity s ∗ 1 while dense block can be found do

Original: Divide the rows of the sparse pruning mask m into t 1 groups using hypergraph partitioning (hMETIS) a for group c i ∈ { c 1 , c 2 , . .
Revised: Divide the rows of the sparse pruning mask m into t 1 groups using hypergraph partitioning (hMETIS) a for group c i ∈ { c 1 , c 2 , . .

Original: c t 1 } do if c i has ≥ b 1 rows then
Revised: c t 1 } do if c i has ≥ b 1 rows then

Original: Select columns in c
Revised: Select columns in c

Original: i
Revised: i

Original: that has no less than t 2 non-zero items if ≥ b 2 columns are selected then
Revised: that has no less than t 2 non-zero items if ≥ b 2 columns are selected then

Original: Group and Reﬁll the selected columns as well as rows to a dense block, and update m end end end 11 end 12 Set other elements out of dense blocks toa http://glaros.dtc.umn.edu/gkhome/ metis/
Revised: Group and Reﬁll the selected columns as well as rows to a dense block, and update m end end end 11 end 12 Set other elements out of dense blocks toa http://glaros.dtc.umn.edu/gkhome/ metis/

Original: hmetis
Revised: hmetis

Original: overview
Revised: overview

Original: Here we provide a detailed description of how many and which channels we choose to reﬁll.
Revised: Here we provide a detailed description of how many and which channels we choose to reﬁll.

Original: Our main experiments adopt the (cid:96) 1 norm of channel weights as the picking criterion to score the channel importance due to its superior performance.
Revised: Our main experiments adopt the (cid:96) 1 norm of channel weights as the picking criterion to score the channel importance due to its superior performance.

Original: Let θ l ∈ R c out × n denotes the parameters of the convolutional layer l , where c out is the number of output channel and n is the continued product of the number of input channel, channel height and weight, as shown in Figure 2.
Revised: Let θ l ∈ R c out × n denotes the parameters of the convolutional layer l , where c out is the number of output channel and n is the continued product of the number of input channel, channel height and weight, as shown in Figure 2.

Original: θ li ∈ R
Revised: θ li ∈ R

Original: n represents the weights in the i th kernel and m li ∈ { 0 , 1 } | θ li | is the corresponding mask.
Revised: n represents the weights in the i th kernel and m li ∈ { 0 , 1 } | θ li | is the corresponding mask.

Original: We ﬁrst calculate the (cid:96)norm of m li (cid:12) θ li , which is a summation of the absolute value of remaining weights in the kernel i .
Revised: We ﬁrst calculate the (cid:96)norm of m li (cid:12) θ li , which is a summation of the absolute value of remaining weights in the kernel i .

Original: Then we use it to pick the top- k scored kernels, which will be fully reﬁlled.
Revised: Then we use it to pick the top- k scored kernels, which will be fully reﬁlled.

Original: k = (cid:100)
Revised: k = (cid:100)

Original: s l × c out × n (cid:101) , where s l is the original layerwise sparsity and c out × n is the total number of weights in kernel i .
Revised: s l × c out × n (cid:101) , where s l is the original layerwise sparsity and c out × n is the total number of weights in kernel i .

Original: Meanwhile, the rest c out − k kernels are dropped for efﬁciency gains.
Revised: Meanwhile, the rest c out − k kernels are dropped for efﬁciency gains.

Original: Furthermore, we propose a soft version, reﬁlling+ , to make a redemption for the aggressive nature of wiping out all remaining channels.
Revised: Furthermore, we propose a soft version, reﬁlling+ , to make a redemption for the aggressive nature of wiping out all remaining channels.

Original: It picks and re-actives an extra proportion of channels to slow down the network capacity reduction, as indicated by shallow blue blocks in Figure 2.
Revised: It picks and re-actives an extra proportion of channels to slow down the network capacity reduction, as indicated by shallow blue blocks in Figure 2.

Original: 3. R EGROUPING FOR S TRUCTURAL P ATTERNS
Revised: 3. R EGROUPING FOR S TRUCTURAL P ATTERNS

Original: Although proposed reﬁlling(+) reorganizes the unstructured mask and produces useful channelwise structural subnetworks, it is rigid and inelastic since the smallest manageable unit is a kernel.
Revised: Although proposed reﬁlling(+) reorganizes the unstructured mask and produces useful channelwise structural subnetworks, it is rigid and inelastic since the smallest manageable unit is a kernel.

Original: In our words, the dense matrices in identiﬁed structural patterns have a restricted shape where one dimension must align with the kernel size n , i.e., the continued product of the number of input channels, channel height and weight.
Revised: In our words, the dense matrices in identiﬁed structural patterns have a restricted shape where one dimension must align with the kernel size n , i.e., the continued product of the number of input channels, channel height and weight.

Original: Motivated by Rumi et al.
Revised: Motivated by Rumi et al.

Original: (2020), we introduce a regrouping strategy (Figure 2) to create more ﬁne-grained group-wise structural patterns with ﬂexible shapes for remaining dense matrices.
Revised: (2020), we introduce a regrouping strategy (Figure 2) to create more ﬁne-grained group-wise structural patterns with ﬂexible shapes for remaining dense matrices.

Original: (cid:66) How to perform regrouping?
Revised: (cid:66) How to perform regrouping?

Original: Regrouping aims to ﬁnd and extract dense blocks of non-pruned elements in the sparse weight matrix.
Revised: Regrouping aims to ﬁnd and extract dense blocks of non-pruned elements in the sparse weight matrix.

Original: These blocks have diverse shapes, as demonstrated in Figure 2, which are usually smaller in size compared to the original sparse matrix.
Revised: These blocks have diverse shapes, as demonstrated in Figure 2, which are usually smaller in size compared to the original sparse matrix.

Original: Note that a channel/kernel can be regarded as a special case of the dense block.
Revised: Note that a channel/kernel can be regarded as a special case of the dense block.

Original: As described in Algorithm 3, to achieve the goal, we ﬁrst need to ﬁnd similar rows and columns, and then bring them together.
Revised: As described in Algorithm 3, to achieve the goal, we ﬁrst need to ﬁnd similar rows and columns, and then bring them together.

Original: Speciﬁcally, We adopt the Jaccard similarity (Rumi et al., 2020; Jiang et al., 2020) among non-zero columns as the similarity between two rows in the sparse matrix, which is calculated as a cardinality ratio of the intersections to the union of non-zero columns.
Revised: Speciﬁcally, We adopt the Jaccard similarity (Rumi et al., 2020; Jiang et al., 2020) among non-zero columns as the similarity between two rows in the sparse matrix, which is calculated as a cardinality ratio of the intersections to the union of non-zero columns.

Original: For instance, kernel 1 and kernel 2 in Figure 2 (upper left) share three columns in eight non-zero distinct columns, and their similarity is 38 .
Revised: For instance, kernel 1 and kernel 2 in Figure 2 (upper left) share three columns in eight non-zero distinct columns, and their similarity is 38 .

Original: Then, if two rows have a larger similarity, it can form a denser block when we group them together.
Revised: Then, if two rows have a larger similarity, it can form a denser block when we group them together.

Original: Take Figure 2 as an example.
Revised: Take Figure 2 as an example.

Original: We can group kernel 1 , 2 , 3 ’s non-zero columns 1 , 3 , 6 , 11 with at least two elements together, which leads to the ﬁrst orange dense block.
Revised: We can group kernel 1 , 2 , 3 ’s non-zero columns 1 , 3 , 6 , 11 with at least two elements together, which leads to the ﬁrst orange dense block.

Original: More precisely, we take the hypergraph partitioning in the regrouping algorithm to generate dense blocks.
Revised: More precisely, we take the hypergraph partitioning in the regrouping algorithm to generate dense blocks.

Original: It treats each row and column from the sparse matrix as a node and hyperedge in the hypergraph, where
Revised: It treats each row and column from the sparse matrix as a node and hyperedge in the hypergraph, where

Original: hyperedge (i.e., column) connects the corresponding nodes (i.e., row).
Revised: hyperedge (i e., column) connects the corresponding nodes (i.e., row).

Original: Then, the pairwise similarity is leveraged to locate an optimal partitioning, which can be achieved with hMETIS2 .
Revised: Then, the pairwise similarity is leveraged to locate an optimal partitioning, which can be achieved with hMETIS

Original: More details are referred to Rumi et al.
Revised: More details are referred to Rumi et al.

Original: After obtaining the desired dense blocks, we enable all their parameters to be trainable by reﬁlling the corresponding pruned elements.
Revised: After obtaining the desired dense blocks, we enable all their parameters to be trainable by reﬁlling the corresponding pruned elements.

Original: Note that reﬁlling these pruned weights does not cause any efﬁciency loss since the size of the blocks is ﬁxed, while it potentially maximizes the usage of these blocks and brings accuracy gains.
Revised: Note that reﬁlling these pruned weights does not cause any efﬁciency loss since the size of the blocks is ﬁxed, while it potentially maximizes the usage of these blocks and brings accuracy gains.

Original: Meanwhile, the rest parameters not included in the dense blocks will be discarded, i.e., setting the corresponding position in binary mask m to zero, for reducing the computational overhead as illustrated in Figure2.
Revised: Meanwhile, the rest parameters not included in the dense blocks will be discarded, i e., setting the corresponding position in binary mask m to zero, for reducing the computational overhead as illustrated in Figure

Original: It is because any parameters outside the dense blocks require extra weights loading and have little data reuse (Rumi et al., 2020), which harms the trade-off of accuracy and efﬁciency.
Revised: It is because any parameters outside the dense blocks require extra weights loading and have little data reuse (Rumi et al., 2020), which harms the trade-off of accuracy and efﬁciency.

Original: (cid:66) How reﬁlled / regrouped dense blocks be beneﬁcial?
Revised: (cid:66) How reﬁlled / regrouped dense blocks be beneﬁcial

Original: We notice that the common tools like cuDNN  have a signiﬁcant drawback that the inference time does not linearly change with the number of kernels, since they are only optimized for kernel matrices with a multiple of 32 rows (Radu et al.,2019).
Revised: We notice that the common tools like cuDNN (Chetlur et al., 2014) have a signiﬁcant drawback that the inference time does not linearly change with the number of kernels, since they are only optimized for kernel matrices with a multiple of 32 rows (Radu et al.,

Original: For example, as stated in Rumi et al.
Revised: For example, as stated in Rumi et al.

Original: (2020), a convolutional layer with 10 kernels might have a similar inference time with a convolutional layer with 32 kernels.
Revised: , a convolutional layer with 10 kernels might have a similar inference time with a convolutional layer with 32 kernels.

Original: However, the number of kernels in these dense blocks is almost arbitrary, so a more sophisticated GEMM-based efﬁcient implementation (Rumi et al., 2020) is needed to accelerate better our reﬁlled / regrouped structural patterns.
Revised: However, the number of kernels in these dense blocks is almost arbitrary, so a more sophisticated GEMM-based efﬁcient implementation (Rumi et al., 2020) is needed to accelerate better our reﬁlled / regrouped structural patterns.

Original: Following Rumi et al.
Revised: Following Rumi et al.

Original: (2020), we split a kernel with r rows into two parts: one has [ r/ 32] × 32 rows and the other one has r mod 32 rows.
Revised: (2020), we split a kernel with r rows into two parts: one has [ r/ 32] × 32 rows and the other one has r mod 32 rows.

Original: First, we directly apply the standard GEMM-based convolution algorithm with shared memory to cache the input and output matrix.
Revised: First, we directly apply the standard GEMM-based convolution algorithm with shared memory to cache the input and output matrix.

Original: For the second part, due to the poor data reuse of input matrices, we choose caching the kernel and output matrices for an improved cache hit rate and overall performance.
Revised: For the second part, due to the poor data reuse of input matrices, we choose caching the kernel and output matrices for an improved cache hit rate and overall performance.

Original: More details are referred to Rumi et al.
Revised: More details are referred to Rumi et al.

Original:  T HE E XISTENCE OF S TRUCTURAL W INNING T ICKETS
Revised:  T HE E XISTENCE OF S TRUCTURAL W INNING T ICKETS

Original: Tiny-ImageNet and ImageNet.
Revised: Tiny-ImageNet and ImageNet.

Original: In this section, we reveal the existence of our proposed structural winning tickets on ImageNet and Tiny-ImageNet with ResNet-50 backbone.
Revised: In this section, we reveal the existence of our proposed structural winning tickets on ImageNet and Tiny-ImageNet with ResNet-50 backbone.

Original: Results of unstructured
Revised: Results of unstructured

Original: IMP , channel-wise structural IMP-Refill(+) , and group-wise structural IMP-Regroup are collected in the Figure 3.
Revised: IMP , channel-wise structural IMP-Refill(+) , and group-wise structural IMP-Regroup are collected in the Figure 3.

Original: The end-to-end inference time 3 of obtained structural winning tickets with extreme sparsity levels are presented, which is calculated on a single 2080 TI GPU with a batch size of 64 .
Revised: The end-to-end inference time 3 of obtained structural winning tickets with extreme sparsity levels are presented, which is calculated on a single 2080 TI GPU with a batch size of 64 .

Original: Extreme sparsity is deﬁned as maximum sparsity when the subnetwork has superior accuracy than its dense counterpart.
Revised: Extreme sparsity is deﬁned as maximum sparsity when the subnetwork has superior accuracy than its dense counterpart.

Original: [Figure]
Revised: [Figure]

Original: From Tiny-ImageNet results in Figure 3 ( left ), several positive observations can be drawn: (cid:182)
Revised: From Tiny-ImageNet results in Figure 3 ( left ), several positive observations can be drawn: (cid:182)

Original: Structural winning tickets with 60% channel-wise structural sparsity and 74% group-wise structural sparsity are located by IMP-Refill and IMP-Regroup respectively, which validate the effectiveness of our proposals.
Revised: Structural winning tickets with 60% channel-wise structural sparsity and 74% group-wise structural sparsity are located by IMP-Refill and IMP-Regroup respectively, which validate the effectiveness of our proposals.

Original: (cid:183)
Revised: (cid:183)

Original: Although at the high sparsity levels (i.e., > 50% ), IMP-Refill+ outperforms IMP-Refill if they are from the same unstructured IMP subnetworks.
Revised: Although at the high sparsity levels (i.e., > 50% ), IMP-Refill+ outperforms IMP-Refill if they are from the same unstructured IMP subnetworks.

Original: Considering the overall trade-off between channel-wise strcutural sparsity and accuracy, IMP-Refill appears a clear advantage.
Revised: Considering the overall trade-off between channel-wise strcutural sparsity and accuracy, IMP-Refill appears a clear advantage.

Original: A possible explanation is that reﬁlling+ seems to bring undesired channels which potentially result in a degraded performance trade-off.
Revised: A possible explanation is that reﬁlling+ seems to bring undesired channels which potentially result in a degraded performance trade-off.

Original: (cid:184)
Revised: (cid:184)

Original: IMP-Regroup performs better at high sparsities.
Revised: IMP-Regroup performs better at high sparsities.

Original: It is within expectation since ﬁne-grained group-wise structural patterns tend to make the networks be more amenable to pruning.
Revised: It is within expectation since ﬁne-grained group-wise structural patterns tend to make the networks be more amenable to pruning.

Original: (cid:185) Extreme channel- / group-wise structural winning tickets with 45% ∼ 50% / 74% sparsity from IMP-Refill(+) /
Revised: (cid:185) Extreme channel- / group-wise structural winning tickets with 45% ∼ 50% / 74% sparsity from IMP-Refill(+) /

Original: IMP-Regroup achieve 57 .
Revised: IMP-Regroup achieve 57 .

Original: 53% ∼ 61 .
Revised: 53% ∼ 61 .

Original: 84% GPU running time savings, without sacriﬁcing accuracies.
Revised: 84% GPU running time savings, without sacriﬁcing accuracies.

Original: As for large-scale ImageNet experiments, the conclusion are slightly different: (cid:182)
Revised: As for large-scale ImageNet experiments, the conclusion are slightly different: (cid:182)

Original: There is almost no difference between the performance of IMP-Refill and IMP-Refill+ , and both can not ﬁnd channel-wise structural winning tickets.
Revised: There is almost no difference between the performance of IMP-Refill and IMP-Refill+ , and both can not ﬁnd channel-wise structural winning tickets.

Original: But it seems to suggest our picking rule (i.e., channel weights’ (cid:96) 1 norm) provides a great estimation for channel importance, although it is too aggressive for ImageNet experiments.
Revised: But it seems to suggest our picking rule (i.e., channel weights’ (cid:96) 1 norm) provides a great estimation for channel importance, although it is too aggressive for ImageNet experiments.

Original: (cid:183)
Revised: (cid:183)

Original: The group-wise structural winning ticket at 31% sparsity is still exist in (RN-50, ImageNet), while the low sparsity brings limited 1% time savings.
Revised: The group-wise structural winning ticket at 31% sparsity is still exist in (RN-50, ImageNet), while the low sparsity brings limited 1% time savings.

Original: For a better efﬁciency and performance trade-off, IMP-Regroup is capable of locating structural subnetworks at 51% / 58% sparsity with 53 .
Revised: For a better efﬁciency and performance trade-off, IMP-Regroup is capable of locating structural subnetworks at 51% / 58% sparsity with 53 .

Original: 84% time savings and 0 .
Revised: 84% time savings and 0 .

Original: 95% accuracy drop.
Revised: 95% accuracy drop.

Original: CIFAR with diverse network architectures.
Revised: CIFAR with diverse network architectures.

Original: We then validate our approaches on CIFAR-10/ (C10/100) with diverse network backbones including Wide-ResNet-32-2, MobileNet-v1, VGG-16, and ResNet-18.
Revised: We then validate our approaches on CIFAR-10/ (C10/100) with diverse network backbones including Wide-ResNet-32-2, MobileNet-v1, VGG-16, and ResNet-18.

Original: Based on the extensive results in Figure 4 and 5, we ﬁnd: (cid:182)
Revised: Based on the extensive results in Figure 4 and 5, we ﬁnd: (cid:182)

Original: On { (WRN-32-2,C10), (WRN-32-2,C100), (MBNet-v1,C10), (MBNet-v1,C100), (VGG-16,C10), (VGG-16,C100), (RN18,C10), (RN-18,C100) } schemes, we consistently disclose the existence of structural winning tickets with { 53% , 28% , 67% , 0% , 60% , 40% , 50% , 0% } channel-wise sparsity and { 66% , 36% , 72% , 56% , 80% , 80% , 78% , 78% } group-wise sparsity from IMP-Refill(+) and IMP-Regroup , respectively.
Revised: On { (WRN-32-2,C10), (WRN-32-2,C100), (MBNet-v1,C10), (MBNet-v1,C100), (VGG-16,C10), (VGG-16,C100), (RN18,C10), (RN-18,C100) } schemes, we consistently disclose the existence of structural winning tickets with { 53% , 28% , 67% , 0% , 60% , 40% , 50% , 0% } channel-wise sparsity and { 66% , 36% , 72% , 56% , 80% , 80% , 78% , 78% } group-wise sparsity from IMP-Refill(+) and IMP-Regroup , respectively.

Original: (cid:183)
Revised: (cid:183)

Original: With the same network, pursuing channel-wise sparse patterns on CIFAR-100 is more challenging than it on CIFAR-10, possibly due to the larger dataset complexity.
Revised: With the same network, pursuing channel-wise sparse patterns on CIFAR-100 is more challenging than it on CIFAR-10, possibly due to the larger dataset complexity.

Original: On the same dataset, larger networks tend to have larger extreme sparsities for both channel- and group-wise structural winning tickets, with the exception of IMP-Refill(+) on (RN-18, C100).
Revised: On the same dataset, larger networks tend to have larger extreme sparsities for both channel- and group-wise structural winning tickets, with the exception of IMP-Refill(+) on (RN-18, C100).

Original: (cid:184)
Revised: (cid:184)

Original: At the middle sparsity levels (i.e., < 50% ), IMP-Regroup behaves closely to IMP-Refill(+) , while
Revised: At the middle sparsity levels (i.e., < 50% ), IMP-Regroup behaves closely to IMP-Refill(+) , while

Original: IMP-Regroup has a superior performance at high sparsity levels.
Revised: IMP-Regroup has a superior performance at high sparsity levels.

Original: (cid:185) Up to { 57 .
Revised: (cid:185) Up to { 57 .

Original: 93% } GPU running time savings are obtained by group-wise structural winning tickets with undamaged performance on { (VGG-16,C10), (VGG-16,C100), (RN-18,C10), (RN-18,C100) } , which surpass IMP , IMP-Refill(+) , and dense models by a signiﬁcant efﬁciency margin.
Revised: 93% } GPU running time savings are obtained by group-wise structural winning tickets with undamaged performance on { (VGG-16,C10), (VGG-16,C100), (RN-18,C10), (RN-18,C100) } , which surpass IMP , IMP-Refill(+) , and dense models by a signiﬁcant efﬁciency margin.

Original: A exception is that IMP-Refill on (VGG-16,C10) achieves the best time savings, i.e., 63 .
Revised: A exception is that IMP-Refill on (VGG-16,C10) achieves the best time savings, i.e., 63 .

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: Layer-wise speedups.
Revised: Layer-wise speedups.

Original: Figure 6 shows the layer-wise speedup performance of convolution operations in VGG-16’s extreme structured winning tickets from different algorithms.
Revised: Figure 6 shows the layer-wise speedup performance of convolution operations in VGG-16’s extreme structured winning tickets from different algorithms.

Original: IMP-Regroup presents impressive layer-wise speedups up to 6 .
Revised: IMP-Regroup presents impressive layer-wise speedups up to 6 .

Original: 67 x compared to others, especially on the last a few layers (e
Revised: 67 x compared to others, especially on the last a few layers (e

Original: [Figure]
Revised: [Figure]

Original: g., conv. 12 ).
Revised: g., conv. 12 ).

Original: The possible reasons lie in two aspects: ( i ) the latter layers reach a larger compression ratio and have greater potentials for acceleration; ( ii ) the regrouping algorithm prefers convolutional layers (i
Revised: The possible reasons lie in two aspects: ( i ) the latter layers reach a larger compression ratio and have greater potentials for acceleration; ( ii ) the regrouping algorithm prefers convolutional layers (i

Original: [Figure]
Revised: [Figure]

Original: e., latter layers in VGG-16) with a larger number of kernels which beneﬁts to group appropriate dense blocks, as also suggested by Rumi et al. 
Revised: e., latter layers in VGG-16) with a larger number of kernels which beneﬁts to group appropriate dense blocks, as also suggested by Rumi et al. [Figure] ee dup

Original: [Figure]
Revised: [Figure]

Original:  A BLATION S TUDY AND V ISUALIZATION
Revised:  A BLATION S TUDY AND V ISUALIZATION

Original: Different sources of unstructured masks.
Revised: Different sources of unstructured masks.

Original: Intuitively, the initial unstructured sparse mask should plays an essential role in the achievable performance of our proposed “post-processing techniques”.
Revised: Intuitively, the initial unstructured sparse mask should plays an essential role in the achievable performance of our proposed “post-processing techniques”.

Original: We therefore conduct a comprehensive ablation study about the various sources of the initial sparse masks in Figure 7, including IMP , OMP , RP , SNIP , GraSP , SynFlow , and ADMM .
Revised: We therefore conduct a comprehensive ablation study about the various sources of the initial sparse masks in Figure 7, including IMP , OMP , RP , SNIP , GraSP , SynFlow , and ADMM .

Original: The details of comparison methods are included in Section 3.1.
Revised: The details of comparison methods are included in Section 3.1.

Original: We observe that IMP and OMP provide initial unstructured masks with the top- 2 highest quality for our regrouping algorithm, in terms of the train-from-scratch accuracy of grouped structural subnetworks.
Revised: We observe that IMP and OMP provide initial unstructured masks with the top- 2 highest quality for our regrouping algorithm, in terms of the train-from-scratch accuracy of grouped structural subnetworks.

Original: [Figure]
Revised: [Figure]

Original: Different initialization for the re-training.
Revised: Different initialization for the re-training.

Original: Initialization (Frankle & Carbin, 2019; Renda et al., 2020) as another key factor in LTH, also contributes signiﬁcantly to the existence of winning tickets.
Revised: Initialization (Frankle & Carbin, 2019; Renda et al., 2020) as another key factor in LTH, also contributes signiﬁcantly to the existence of winning tickets.

Original: To exhaustively investigate the effect from different initialization (e.g., rewound weights), we launch experiments started from diverse rewound weights ( { 5% , 10% , 20% , 50% , 100% } of total training epochs) as well as a random re-initialization.
Revised: To exhaustively investigate the effect from different initialization (e.g., rewound weights), we launch experiments started from diverse rewound weights ( { 5% , 10% , 20% , 50% , 100% } of total training epochs) as well as a random re-initialization.

Original: As shown in Figure 7, using 50% rewound weight reach the overall best performance; other weight rewinding setups perform similarly and clearly surpass random reinitializing at sparsity levels > 30% .
Revised: As shown in Figure 7, using 50% rewound weight reach the overall best performance; other weight rewinding setups perform similarly and clearly surpass random reinitializing at sparsity levels > 30% .

Original: Group-aware IMP .
Revised: Group-aware IMP .

Original: This work mainly focuses on the post-processing of unstructured sparse masks.
Revised: This work mainly focuses on the post-processing of unstructured sparse masks.

Original: Another possibility is integrating regrouping into IMP by alternatively performing unstructured magnitude pruning and regrouping, which we term as group-aware IMP .
Revised: Another possibility is integrating regrouping into IMP by alternatively performing unstructured magnitude pruning and regrouping, which we term as group-aware IMP .

Original: From
Revised: From

Original: Fig.
Revised: Fig.

Original: 7, it has a worse performance due to the stricter constraint on sparse patterns, compared to IMP-Regroup .
Revised: 7, it has a worse performance due to the stricter constraint on sparse patterns, compared to IMP-Regroup .

Original: Visualization of sparse masks.
Revised: Visualization of sparse masks.

Original: Figure 8 visualizes different types of obtained sparse masks from (VGG-16,C10).
Revised: Figure 8 visualizes different types of obtained sparse masks from (VGG-16,C10).

Original: Sub-ﬁgures (a,b,c) plot the mask matrices of size c out × n for certain layers.
Revised: Sub-ﬁgures (a,b,c) plot the mask matrices of size c out × n for certain layers.

Original: Similar to the illustration in Figure 2, IMP-Refill masks show clear kernel-wise sparse patterns across the rows, and IMP-Regroup masks present ﬁne-grained structural sparse patterns capable of forming neat dense blocks after regrouping.
Revised: Similar to the illustration in Figure 2, IMP-Refill masks show clear kernel-wise sparse patterns across the rows, and IMP-Regroup masks present ﬁne-grained structural sparse patterns capable of forming neat dense blocks after regrouping.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we challenge the “common sense” that an identiﬁed IMP winning ticket can only have unstructured sparsity, which severely limits its practical usage due to the irregular patterns.
Revised: In this paper, we challenge the “common sense” that an identiﬁed IMP winning ticket can only have unstructured sparsity, which severely limits its practical usage due to the irregular patterns.

Original: We for the ﬁrst time demonstrate the existence of structural winning tickets by leveraging post-processing techniques, i.e., reﬁlling(+) and regrouping .
Revised: We for the ﬁrst time demonstrate the existence of structural winning tickets by leveraging post-processing techniques, i.e., reﬁlling(+) and regrouping .

Original: The located channel- and group-wise structural subnetworks achieve signiﬁcant inference speedups up to 6 .
Revised: The located channel- and group-wise structural subnetworks achieve signiﬁcant inference speedups up to 6 .

Original: 67 x on hardware platforms.
Revised: 67 x on hardware platforms.

Original: In this sense, our positive results bridge the gap between the lottery ticket hypothesis and practical accelerations in real-world scenarios.
Revised: In this sense, our positive results bridge the gap between the lottery ticket hypothesis and practical accelerations in real-world scenarios.

Original: We would be interested in examining LTH with more effective structural sparsity for real-time mobile computing in future work. 
Revised: We would be interested in examining LTH with more effective structural sparsity for real-time mobile computing in future work. A M ORE I MPLEMENTATION D ETAILS

Revised: Group-aware IMP .

Revised: Here we provides the detailed procedures of group-aware IMP in Algorithm 4.

Revised: Intuitively, it embeds regrouping (Algorithm 3) into IMP (Algorithm 1) by performing regrouping on the unstructured mask m from each IMP round.

Revised: Input: f ( x ; θ 0 ) , group-wise structural sparsity s

Revised: Output: f ( x ; m (cid:12) θ i ) with group-wise structural sparse mask s 1 Set the pruning mask m = 1 ∈ R | θ | 2 Train f ( x ; θ 0 ) to rewinding step i : f ( x ; θ i ) =

Revised: A T i ( f ( x ; θ 0 )) 3 while not reach sparsity s do

Revised: Train f

Revised: ( x ; m (cid:12) θ i ) to step t : f ( x ; m (cid:12) θ t ) =

Revised: A T t − i ( f ( x ; m (cid:12) θ i ))

Revised: Pruning 20% of remaining weight of m (cid:12) θ t , and update m

Revised: Reﬁning the unstructured mask m by performing regrouping , as shown in Algorithm7 end

Revised: Proﬁling.

Revised: To compute the GPU running time of regrouped convolution layers, we adopt their

Revised: CUDA C/C++ implementation.

Revised: Our results do not include the running time of normalization and activation layers, following the standard in Rumi et al.

Revised: For a fair calculation, we feed the same input features to convolution layers that belong to the same model.

Revised: For ResNet-18 and VGG16, the size of the input features is (64 , 64 , 127 , 127) .

Revised: For ResNet-50, the size of input features is (64 , 64 , 64 , 64) .

Revised: The GPU we use for proﬁling is NVIDIA RTX 2080 TI, with a CUDA version of 10 . 2 and a cuDNN (Chetlur et al., 2014) version of 7 . 6 . 5 .

Revised: A M ORE E XPERIMENT R ESULTS

Revised: Different channel picking criterion for reﬁlling.

Revised: We ablation the channel picking criterion for

Revised: IMP-Refill(+) , including (cid:182) the (cid:96) 1 norm of channel’s remaining weight, (cid:183) the (cid:96) 1 or (cid:96) 2 norms of channel’s feature map, (cid:184) the number of remaining weights in the channel, (cid:185) the channel’s saliency score (Molchanov et al., 2019).

Revised: Experiment results are collected in Figure A9, which demonstrate the superior performance of IMP-Refill w. (cid:96) 1 of channel weights (yellow curve).

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: A