Original: T RAINABILITY P RESERVING N EURAL P RUNING
Revised: T RAINABILITY P RESERVING N EURAL P RUNING

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Many recentpruning works show trainability plays a critical role in  networkstructured pruning – unattended broken trainability can lead to severe underperformance and unintentionally amplify the effect of finetuning learning rate, resulting in biased (or even misinterpreted) benchmark results.
Revised: Many recent works have shown trainability plays a central role in neural network pruning – unattended broken trainability can lead to severe under-performance and unintentionally amplify the effect of retraining learning rate, resulting in biased (or even misinterpreted) benchmark results.

Original: In this paper, we present trainability preserving pruning (TPP), a scalable method to preserve network trainability against pruning, aiming for improved pruning performance.
Revised: This paper introduces trainability preserving pruning (TPP), a scalable method to preserve network trainability against pruning, aiming for improved pruning performance and being more robust to retraining hyper-parameters ( e.g. , learning rate).

Original: TPP regularizes the gram matrix of convolutional filters to decorrelate the pruned filters from the retained filters.
Revised: Specifically, we propose to penalize the gram matrix of convolutional filters to decorrelate the pruned filters from the retained filters in TPP.

Original: In addition to the convolutional layers, per the spirit of preserving the trainability of the whole network, we also propose to regularize the batch normalization parameters.
Revised: In addition to the convolutional layers, per the spirit of preserving the trainability of the whole network, we also propose to regularize the batch normalization parameters.

Original: Empirically, TPP performs on par with the ground-truth trainability recovery method on linear MLP networks.
Revised: Empirical studies on linear MLP networks show that TPP can perform on par with the oracle trainability recovery scheme.

Original: On non-linear networks (ResNet56/VGG19 on CIFAR10/100), our TPP outperforms the other counterpart schemes by an obvious margin.
Revised: On nonlinear ConvNets (ResNet56/VGG19) on CIFAR10/100, TPP outperforms the other counterpart approaches by an obvious margin.

Original: Moreover,extensive results on ImageNet with ResNets show  TPP consistently performs more favorably against other top-performing structured pruning approaches. 
Revised: Moreover, results on ImageNet-1K with ResNets show that TPP consistently performs more favorably against other top-performing structured pruning approaches. Code: https://github.com/MingSun-Tse/TPP.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Neuralnetwork pruning aims to remove the parameters without seriously compromising the performance.
Revised: Neural pruning aims to remove the parameters without seriously compromising the performance.

Original: It normally consists of three steps (Reed, 1993; Han et al., 2015; 2016b): pretrain a dense model; prune the unnecessary connections or neurons; finetune to regain performance.
Revised: It normally consists of three steps (Reed, 1993; Han et al., 2015; 2016b; Li et al., 2017; Liu et al., 2019b; Wang et al., 2021b; Gale et al., 2019; Hoefler et al., 2021; Wang et al., 2023): pretrain a dense model; prune the unnecessary connections or neurons; retrain to regain performance.

Original: Pruning is usually categorized into two groups, unstructured pruning ( a.k.a.
Revised: Pruning is usually categorized into two classes, unstructured pruning ( a.k.a.

Original: element-wise  pruning) and structured pruning ( a.k.a.
Revised: element-wise pruning or fine-grained pruning) and structured pruning ( a.k.a.

Original: filter pruning or channel  pruning).
Revised: filter pruning or channel pruning or coarsegrained pruning).

Original: The former chooses a scalar weight as the basic pruning element; the latter chooses a  3d filter  as the basic pruning element.
Revised: Unstructured pruning chooses a single weight as the basic pruning element; while structured pruning chooses a group of weights ( e.g. , 3d filter or a 2d channel) as the basic pruning element.

Original: In general, structured pruning is morefavored for acceleration because of the regular sparsity; unstructured pruning results in irregular sparsity,which can be considerable without performance degradation but hard to exploit for acceleration without customized hardware  (Han et al., 2016a; 2017).
Revised: In general, structured pruning fits more for acceleration because of the regular sparsity. Unstructured pruning, in contrast, results in irregular sparsity, hard to exploit for acceleration unless customized hardware and libraries are available (Han et al., 2016a; 2017; Wen et al., 2016).

Original: Recent structured pruning works (Renda et al., 2020; Le & Hua, 2021) showed an interesting phenomenon: During finetuning, a larger learning rate (LR) helps achieve a significantly better final performance ( e.g. , ResNet34 pruned at speedup 1.32 × can be improved by over 1% top-1 accuracy on ImageNet (Deng et al., 2009) using finetuning LR 1e-2 s 1e-3 ).
Revised: Recent papers (Renda et al., 2020; Le & Hua, 2021) report an interesting phenomenon: During retraining, a larger learning rate (LR) helps achieve a significantly better final performance ( e.g. , ResNet34 pruned at speedup 1.32 × can be improved by over 1% top-1 accuracy on ImageNet (Deng et al., 2009) using finetuning LR 1e-2 s 1e-3 ).

Original: The reason behind is argued to be relevant to the trainability of neural networks (Lee et al., 2020; Lubana & Dick, 2021; Wang et al., 2021a).
Revised: The reason behind is argued to be relevant to the trainability of neural networks (Wang et al., 2023; Lee et al., 2020; Lubana & Dick, 2021).

Original: They mainly make two observations to explain the LR effect mystery (Wang et al., 2021a).
Revised: They mainly make two observations to explain the LR effect mystery (Wang et al., 2023).

Original: (1) The weights removing operation immediately breaks the  trainability or dynamical isometry (Saxe et al., 2014)
Revised: The weight removal operation immediately breaks the network trainability or dynamical isometry (Saxe et al., 2014)

Original: (the ideal case of trainability) of the trained network.
Revised: (the ideal case of trainability) of the trained network.

Original: The SGD optimization in finetuning can inherently rectify it without extra explicit guidance; a larger LR help recover it faster (and possibly better), thus making the final performance much stronger, especially when the finetuning epochs are insufficient.
Revised: The broken trainability slows down the optimization in retraining, where a larger LR helps the network converge faster, so a better performance is observed earlier – using a smaller LR can do as well, but needs more epochs.

Original: Although these works (Lee et al., 2020; Lubana & Dick, 2021; Wang et al., 2021a) provide a plausibly sound explanation, a more practical issue is how to recover the broken trainability(or dynamical isometry) or maintain it during pruning.
Revised: Although these works (Lee et al., 2020; Lubana & Dick, 2021; Wang et al., 2021a; 2023) provide a plausibly sound explanation, a more practical issue is how to recover the broken trainability or maintain it during pruning .

Original: In this regard, (Wang et al., 2021a) proposes to apply weight orthogonalization based on QR decomposition (Trefethen & Bau III, 1997; Mezzadri, 2006) to the prunedmodel.
Revised: In this regard, (Wang et al., 2021a) proposes to apply weight orthogonalization based on QR decomposition (Trefethen & Bau III, 1997; Mezzadri, 2006) to the pruned

Original: However, their method is shown to only work for linear MLP networks.

Original: On modern deep convolutional neural networks (CNNs), how to maintain trainability after pruning is still elusive.

Original: In this paper, we present trainability preserving pruning (TPP), a new filter pruning method based on regularization (see Fig.

Original: 1 for an overview) that maintains trainability well during pruning.

Original: By our observation, the primary cause that pruning breaks dynamical isometry lies in the dependency

Original: Residual Block
Revised: Residual Block

Original: Skip Connection
Revised: Skip Connection

Original: F (i-1)
Revised: F (i-1)

Original: F (i)
Revised: F (i)

Original: F (i+1)
Revised: F (i+1)

Original: F (i+2)
Revised: F (i+2)

Original: W (i)
Revised: W (i)

Original: W (i+1)  4   4    7 BN 

Revised: W (i+1)  4   4    7 BN

Original: BN
Revised: BN

Original: Large-magnitude (important) filters and their BN params: Not regularized
Revised: Large-magnitude (important) filters and their BN params: Not regularized

Original: Small-magnitude (unimportant) filters and their BN params:Apply TPPregularization Eq.
Revised: Small-magnitude (unimportant) filters and their BN params:Apply TPPregularization Eq.

Original: (3), Eq.
Revised: (3), Eq.

Original: (5) filter 1,4, filter 2,3,5,7, among parameters.
Revised: (5) filter 1,4, filter 2,3,5,7, model.

Revised: However, their method is shown to only work for linear MLP networks.

Revised: On modern deep convolutional neural networks (CNNs), how to maintain trainability after pruning is still elusive.

Revised: This work presents trainability preserving pruning (TPP), a new and novel filter pruning algorithm (see Fig.

Revised: 1 for an overview) that maintains trainability via a regularized training process.

Revised: By our observation, the primary cause that pruning breaks dynamical isometry lies in the dependency among parameters.

Original: The main idea of our method is thus to decorrelate the pruned weights from the kept weights so as to cut off the dependency, so that the following weights removing operation barely hurts the trainability of the network.
Revised: The primary idea of our approach is thus to decorrelate the pruned weights from the kept weights so as to “cut off” the dependency, so that the subsequent sparsifying operation barely hurts the network trainability.

Original: Specifically, we propose to regularize the gram matrix of weights: All the entries representing the correlation between the pruned filters ( i.e. , unimportant filters) and the kept filters ( i.e. , important filters) are encouraged to diminish to zero.
Revised: Specifically, we propose to regularize the gram matrix of weights: All the entries representing the correlation between the pruned filters ( i.e. , unimportant filters) and the kept filters ( i.e. , important filters) are encouraged to diminish to zero.

Original: This is the first technical contribution of our method.
Revised: This is the first technical contribution of our method.

Original: The second one lies in how to treat the other entries.
Revised: The second one lies in how to treat the other entries.

Original: Conventional dynamical isometry wisdom suggests orthogonality (namely, 1 self-correlation and 0 cross-correlation) even among the kept filters, while we find directly translating the idea here is unnecessary or even harmful because the too strong penalty will constrain the optimization, leading to deteriorated local minimum.
Revised: Conventional dynamical isometry wisdom suggests orthogonality (namely, 1 self-correlation and 0 cross-correlation) even among the kept filters, while we find directly translating the idea here is unnecessary or even harmful because the too strong penalty will constrain the optimization, leading to deteriorated local minimum.

Original: Rather, we propose to not impose any regularization on the correlation entries of kept filters.
Revised: Rather, we propose to not impose any regularization on the correlation entries of kept filters.

Original: Finally, modern deep models are typically equipped with batch normalization (BN) (Ioffe &
Revised: Finally, modern deep models are typically equipped with batch normalization (BN) (Ioffe &

Original: Szegedy, 2015).
Revised: Szegedy, 2015).

Original: However, previous filter pruning papers rarely explicitly take BN into account (except two (Liu et al., 2017; Ye et al., 2018); the differences of our work from theirs will be discussed in Sec. 3.2) to mitigate the side effect when it is removed because its associated filter is removed.
Revised: However, previous filter pruning papers rarely explicitly take BN into account (except two (Liu et al., 2017; Ye et al., 2018); the differences of our work from theirs will be discussed in Sec. 3.2) to mitigate the side effect when it is removed because its associated filter is removed.

Original: Since they are also a part of the whole trainable parameters in the network, unattended removal of them will also lead to severely crippled trainability (especially at large sparsity).
Revised: Since they are also a part of the whole trainable parameters in the network, unattended removal of them will also lead to severely crippled trainability (especially at large sparsity).

Original: Therefore, BN parameters (including both the scale and bias parameters) should also be explicitly taken into account when designing a pruning algorithm.
Revised: Therefore, BN parameters (both the scale and bias included) should also be explicitly taken into account when designing a pruning algorithm.

Original: As such, we propose to regularize the learnable parameters of BN to minimize the influence of its absence.
Revised: As such, we propose to regularize the learnable parameters of BN to minimize the influence of its absence.

Original: Empirically, the proposed pruning algorithm is rather easy to implement on popular deep learning platforms and robust to hyper-parameter variations.
Revised: Empirically, the proposed pruning algorithm is rather easy to implement on popular deep learning platforms and robust to hyper-parameter variations.

Original: On the ResNet50 ImageNet benchmark, it delivers encouraging results compared to many recent SOTA filter pruning algorithms.
Revised: On the ResNet50 ImageNet benchmark, TPP delivers encouraging results compared to many recent SOTA filter pruning algorithms.

Original: Contributions.
Revised: Contributions.

Original: (1) We present the first filter pruning method (trainability preserving pruning) that can effectively maintain trainability during pruning for modern deep networks, through a customized weight gram matrix as regularization target.
Revised: (1) We present the first filter pruning method (trainability preserving pruning) that can effectively maintain trainability during pruning for modern deep networks, through a customized weight gram matrix as regularization target.

Original: (2) Apart from weight regularization, we also propose to regularize the BN parameters to offset the side-effect of their eventual absence. This has been overlooked by most previous pruning papers, while we show it is pretty important to preserve trainability, especially when the pruning ratio is large.
Revised: (2) Apart from weight regularization, a BN regularizer is introduced to allow for their subsequent absence in pruning – this issue has been overlooked by most previous pruning papers, although it will be shown to be pretty important to preserve train- ability, especially in the large sparsity regime.

Original: (3) Practically, the proposed method is scalable to modern large-scale deepneural networks ( e.g. , ResNets) and datasets ( e.g. , ImageNet).
Revised: (3) Practically, the proposed method can easily scale to modern large-scale deep networks ( e.g. , ResNets) and datasets ( e.g. , ImageNet-1K).

Original: It achieves promising pruning performance in the comparison to many state-of-the-art filter pruning methods.
Revised: It achieves promising pruning performance in the comparison to many state-of-the-art filter pruning methods.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Neural network pruning .
Revised: Network pruning .

Original: In terms of the sparsity structure induced by pruning, pruning methods mainly fall into structured pruning(filter pruning or channel pruning) (Li et al., 2017; Wen et al., 2016;
Revised: Pruning mainly falls into structured pruning (Li et al., 2017; Wen et al., 2016;

Original: He et al., 2017; 2018; Wang et al., 2021b) and unstructured pruning (Han et al., 2015; 2016b; LeCun et al., 1990; Hassibi & Stork, 1993; Singh & Alistarh, 2020).
Revised: He et al., 2017; 2018; Wang et al., 2021b) and unstructured pruning (Han et al., 2015; 2016b; LeCun et al., 1990; Hassibi & Stork, 1993; Singh & Alistarh, 2020), according to the sparsity structure.

Original: Structured pruning results in regular sparsity after pruning, easy to be translated to acceleration on commodity hardware.

Original: In contrast, unstructured pruning produces irregular sparsity, beneficial to compression while hard to leverage for practical acceleration (Wen et al., 2016;

Original: Wang et al., 2019) unless with special hardware support (Han et al., 2016a; 2017). For more comprehensive coverage, we recommend surveys (Sze et al., 2017; Cheng et al., 2018a;b; Deng et al., 2020;  Wang et al., 2022).
Revised: For more comprehensive coverage, we recommend surveys (Sze et al., 2017; Cheng et al., 2018; Deng et al., 2020; Hoefler et al., 2021; Wang et al., 2022).

Original: This paper targets structured sparsity (filter pruning) because it is more imperative to make modern networks ( e.g. , ResNets (He et al., 2016)) faster rather than smaller compared to the early single-branch convolutional networks.
Revised: This paper targets filter pruning because it is more imperative to make modern networks ( e.g. , ResNets (He et al., 2016)) faster rather than smaller compared to the early single-branch convolutional networks.

Original: It has been broadly observed (LeCun et al., 1990; Han et al., 2015) and also intuitively straightforward that a naive way of pruning( e.g. , random pruning) of a normally-sized ( i.e. , notintentionally severely over-parameterized) network will lead to significant performance drop.
Revised: Random pruning of a normally-sized ( i.e. , not severely over-parameterized) network will lead to significant performance drop.

Original: Namely, we need to cleverly choose some “unimportant” parameters to remove to avoid severe performance degradation.
Revised: We need to cleverly choose some unimportant parameters to remove.

Original: Such a criterion for choosing is  pruning criterion .
Revised: Such a criterion for choosing is called pruning criterion .

Original: In the area, there have been two major paradigms to address the pruning criterion problem dating back to the 1990s: regularization-based methods and importance-based  methods (Reed, 1993).
Revised: In the area, there have been two major paradigms to address the pruning criterion problem dating back to the 1990s: regularization-based methods and importance-based ( a.k.a. saliency-based) methods (Reed, 1993).

Original: Specifically, the regularization-based methods select unimportant parameters by adding a sparsity-inducing penalty termjointly optimized with the original objective function ( e.g. , (Wen et al., 2016; Lebedev & Lempitsky, 2016;
Revised: Specifically, the regularization-based approaches choose unimportant parameters via a sparsityinducing penalty term ( e.g. , Wen et al. ; Lebedev & Lempitsky (2016); Louizos et al.

Original: Louizos et al., 2018; Liu et al., 2017; Ye et al., 2018; Zhang* et al., 2021; Zhang et al., 2021)).
Revised: Liu et al. ; Ye et al. ; Zhang et al. (2021a; 2022; 2021b)).

Original: This paradigm can be applied to a random or pretrained network.
Revised: This paradigm can be applied to a random or pretrained network.

Original: Importance-based methods choose unimportant parameters by certain established mathematical formula (typically based on the Taylor expansion of the loss objective function)
Revised: Importance-based methods choose unimportant parameters via some proposed importance formula, derived from the Taylor expansion of the loss objective function ( e.g. , LeCun et al.

Original: ( e.g. , (LeCun et al., 1990; Hassibi & Stork, 1993; Han et al., 2015; 2016b; Li et al., 2017; Molchanov et al., 2017; 2019)).
Revised: (1990); Hassibi & Stork (1993) ; Han et al. (2015; 2016b) ; Li et al.

Revised: Molchanov et al.

Original: This paradigm is majorly applied to a pretrained network.
Revised: This paradigm is majorly applied to a pretrained network.

Original: Despite the differences, it is worth noting that these two paradigms are not firmly unbridgeable, i.e. , we can develop approaches that take advantage of both ideas, such as (Ding et al., 2018; Wang et al., 2019; 2021b) – these methods identify unimportant weights per a certain importance criterion; then, they utilize a penalty term to produce sparsity.
Revised: Despite the differences, it is worth noting that these two paradigms are not firmly unbridgeable, i.e. , we can develop approaches that take advantage of both ideas, such as (Ding et al., 2018; Wang et al., 2019; 2021b) – these methods identify unimportant weights per a certain importance criterion; then, they utilize a penalty term to produce sparsity.

Original: Our TPP method in this paper is also in this line.
Revised: Our TPP method in this paper is also in this line.

Original: Trainability, dynamical isometry, and orthogonality .
Revised: Trainability, dynamical isometry, and orthogonality .

Original: Trainability describes the easiness of optimization of a neural network.
Revised: Trainability describes the easiness of optimization of a neural network.

Original: Dynamical isometry, the perfect case of trainability, is first introduced by (Saxe et al., 2014), stating that singular values of the Jacobian matrix are close to 1.
Revised: Dynamical isometry, the perfect case of trainability, is first introduced by (Saxe et al., 2014), stating that singular values of the Jacobian matrix are close to 1.

Original: It can be achieved (for linear MLP models) by the orthogonality of weight matrix at initialization.
Revised: It can be achieved (for linear MLP models) by the orthogonality of weight matrix at initialization.

Original: Recent works on this topic mainly focus on how to maintain dynamical isometry during training instead of only for initialization (Xie et al., 2017; Huang et al., 2018; Bansal et al., 2018;
Revised: Recent works on this topic mainly focus on how to maintain dynamical isometry during training instead of only for initialization (Xie et al., 2017; Huang et al., 2018; Bansal et al., 2018;

Original: Huang et al., 2020; Wang et al., 2020).
Revised: Huang et al., 2020; Wang et al., 2020).

Original: These methods are developed independent of pruning, thus not directly relevant to the proposed method in this work.
Revised: These methods are developed independent of pruning, thus not directly relevant to the proposed method in this work.

Original: However, the insights from these works inspire us to our proposed approach (see Sec. 3.2) and possibly more in the future.
Revised: However, the insights from these works inspire us to our proposed approach (see Sec. 3.2) and possibly more in the future.

Original: Several pruning papers study the network trainability issue in the context of network pruning, such as (Lee et al., 2020; Lubana & Dick, 2021; Vysogorets & Kempe, 2021).
Revised: Several pruning papers study the network trainability issue in the context of network pruning, such as (Lee et al., 2020; Lubana & Dick, 2021; Vysogorets & Kempe, 2021).

Original: Our work is different from theirs primarily in that we focus on  pruning  a pretrained model while they tackle the pruning at initialization problem.
Revised: These works mainly discuss the trainability issue of a randomly initialization network. In contrast, we focus on the pruning case of a pretrained network.

Original:  M ETHODOLOGY 3. P RELIMINARIES : D YNAMICAL I SOMETRY AND O RTHOGONALITY
Revised:  M ETHODOLOGY 3. P REREQUISITES : D YNAMICAL I SOMETRY AND O RTHOGONALITY

Original: The definition of dynamical isometry is that theinput-output Jacobian of a network has as many singular values (JSVs) as possible close to 1 (Saxe et al., 2014).
Revised: The definition of dynamical isometry is that the Jacobian of a network has as many singular values (JSVs) as possible close to 1 (Saxe et al., 2014).

Original: With it, the error signal can preserve its norm under propagation without serious amplification or attenuation, which in turn helps the convergence of (very deep) networks.
Revised: With it, the error signal can preserve its norm under propagation without serious amplification or attenuation, which in turn helps the convergence of (very deep) networks.

Original: For a single fully-connected layer W , a sufficient and necessary condition to realize dynamical isometry is orthogonality, i.e. , W T W =
Revised: For a single fully-connected layer W , a sufficient and necessary condition to realize dynamical isometry is orthogonality, i.e. , W T W [Equation] I ,

Original: [Equation]
Revised: [Equation]

Original: where I stands for identity matrix.
Revised: where I represents the identity matrix.

Original: Orthogonality of a weight matrix can be easily realized by matrix orthogonalization techniques such as QR decomposition (Trefethen & Bau III, 1997; Mez- zadri, 2006).
Revised: Orthogonality of a weight matrix can be easily realized by matrix orthogonalization techniques such as QR decomposition (Trefethen & Bau III, 1997; Mezzadri, 2006).

Original: Exact (namely all the Jacobian singular values are exactly 1) dynamical isometry can be achieved for linear networks since multiple linear layers essentially reduce to a single 2d weight matrix.{+[Figure] Target gram matrix WW T 𝑎 !! 𝑎 "" … 
+}
Revised: Exact (namely all the Jacobian singular values are exactly 1) dynamical isometry {+[Figure] Target gram matrix WW T 𝑎 !! 𝑎 "" … 
+}

Original: In contrast, the convolutional and non-linear cases are much complicated.
Revised: N … 𝑎 !" 𝑎 "! … … [Figure] orthogonality

Original: {+Target gram matrix WW T 1  … 
+}
Revised: {+Target gram matrix WW T 1  … 
+}

Revised: N …

Revised: [Figure]

Original: {+orthogonality for pruning Target gram matrix WW T 1  … 
 N … can be achieved for linear networks since multiple linear layers essentially reduce to a single 2d weight contrast the convolutional and non-linear are much complicated.+} Previous work (Wang et al., 2021a) has shown that merely considering convolution or ReLU (Nair & Hinton, 2010) renders the weight orthogonalization method much less effective in terms of recovering dynamical isometry after pruning, let alone considering modern deep networks with BN (Ioffe &
Revised: {+orthogonality for pruning Target gram matrix WW T 1  … 
 N … can be achieved for linear networks since multiple linear layers essentially reduce to a single 2d weight contrast the convolutional and non-linear are much complicated.+} Previous work (Wang et al., 2023) has shown that merely considering convolution or ReLU (Nair & Hinton, 2010) renders the weight orthogonalization method much less effective in terms of recovering dynamical isometry after pruning, let alone considering modern deep networks with BN (Ioffe &

Original: Szegedy, 2015) and residuals (He et al., 2016).
Revised: Szegedy, 2015) and residuals (He et al., 2016).

Original: The primary goal of our paper is to bridge this gap.
Revised: The primary goal of our paper is to bridge this gap.

Original: Following the seminal work of (Saxe et al., 2014), several papers propose to maintain orthogonality during training instead of sorely for the initialization.
Revised: Following the seminal work of (Saxe et al., 2014), several papers propose to maintain orthogonality during training instead of sorely for the initialization.

Original: There are primarily two groups of orthogonalization methods for convolutional neural networks: kernel orthogonality (Xie et al., 2017; Huang et al., 2018; 2020) and orthogonal convolution (Wang et al., 2020):
Revised: There are primarily two groups of orthogonalization methods for CNNs: kernel orthogonality (Xie et al., 2017; Huang et al., 2018; 2020) and orthogonal convolution (Wang et al., 2020),

Original: [Equation]
Revised: [Equation]

Original: where clearly the difference lies in the weight matrix K vs. K . (1) K denotes the original weight matrix in a convolutional layer.
Revised: Clearly the difference lies in the weight matrix K vs. K : (1) K denotes the original weight matrix in a convolutional layer.

Original: Weights of a conv layer make up a 4d tensor R N × C × H × W , where N stands for thenumber of output channels, C for thenumber of input channels, H and W for the height and width of conv kernel.
Revised: Weights of a conv layer make up a 4d tensor R N × C × H × W ( N stands for the output channel number, C for the input channel number, H and W for the height and width of CONV kernel).

Original: Then, K is a reshaped version of the 4d tensor: K ∈ R N × CHW (if N < CHW ; otherwise, K ∈ R CHW × N ).
Revised: Then, K is a reshaped version of the 4d tensor: K ∈ R N × CHW (if N < CHW ; otherwise, K ∈ R CHW × N ).

Original: In contrast, K ∈ R NH fo W fo × CH fi W fi stands for the doubly block-Toeplitz (DBT) matrix representation of K ( H fo stands for the output feature map height, H fi for the input feature map height.
Revised: In contrast, K ∈ R NH fo W fo × CH fi W fi stands for the doubly blockToeplitz representation of K ( H fo stands for the output feature map height, H fi for the input feature map height.

Original: W fo and W fi can be inferred the same way for width).
Revised: W fo and W fi can be inferred the same way for width).

Original: Wang et al.
Revised: Wang et al.

Original: (2020) have shown that orthogonal convolution is more effective than kernel orthogonality (Xie et al., 2017) in that the latter is only a necessary but insufficient condition of the former.
Revised: (2020) have shown that orthogonal convolution is more effective than kernel orthogonality (Xie et al., 2017) in that the latter is only a necessary but insufficient condition of the former.

Original: In this work, we will evaluate both methods to see how effective they are in recovering trainability.
Revised: In this work, we will evaluate both methods to see how effective they are in recovering trainability.

Original: 3. T RAINABILITY P RESERVING P RUNING (TPP)
Revised: 3. T RAINABILITY P RESERVING P RUNING (TPP)

Original: Our TPP method has two parts.
Revised: Our TPP method has two parts.

Original: First, we explain how we come up with the proposed scheme and how it intuitively is better than the straight idea of directly applying orthogonality regularization methods (Xie et al., 2017; Wang et al., 2020) here.
Revised: First, we explain how we come up with the proposed scheme and how it intuitively is better than the straight idea of directly applying orthogonality regularization methods (Xie et al., 2017; Wang et al., 2020) here.

Original: Second, we propose to regularize batch normalization layers given their prevailing use  as a standard component in deep neural networks nowadays.
Revised: Second, a batch normalization regularizer is introduced given the prevailing use of BN as a standard component in deep CNNs nowadays.

Original: (1) Trainability vs. orthogonality .
Revised: (1) Trainability vs. orthogonality .

Original: From previous works (Lee et al., 2020;
Revised: From previous works (Lee et al., 2020;

Original: Lubana & Dick, 2021;
Revised: Lubana & Dick, 2021;

Original: Wang et al., 2021a), we know recovering the broken trainability (or dynamical isometry) incurred by pruning is very important.
Revised: Wang et al., 2023), we know recovering the broken trainability (or dynamical isometry) incurred by pruning is very important.

Original: Considering orthogonality regularization can encouragedynamical isometry, a pretty straightforward solution is to build upon the existing kernel orthogonality regularization schemes.
Revised: Considering orthogonality regularization can encourage isometry, a pretty straightforward solution is to build upon the existing kernel orthogonality regularization schemes.

Original: Specifically, kernel orthogonality regularizes the weight gram matrix to be close to identity matrix (see Fig. 2(a)).
Revised: Specifically, kernel orthogonality regularizes the weight gram matrix towards an identity matrix (see Fig. 2(a)).

Original: In our case, we aim to remove some filters, so naturally we can regularize the weight gram matrix to be close to a partial identity matrix , with the diagonal entries at the pruned filters zeroed (see Fig. 2(b)).
Revised: In our case, we aim to remove some filters, so naturally we can regularize the weight gram matrix to be close to a partial identity matrix , with the diagonal entries at the pruned filters zeroed (see Fig. 2(b)).

Original: The above scheme is simple and straightforward.
Revised: The above scheme is simple and straightforward.

Original: However, it is not in its best shape by our empirical observation.
Revised: However, it is not in its best shape by our empirical observation.

Original: It imposes too strong unnecessary constraint on the remaining weights, which will in turn hurt the optimization.
Revised: It imposes too strong unnecessary constraint on the remaining weights, which will in turn hurt the optimization.

Original: Therefore, we propose to seek a weaker constraint, not demanding the perfect trainability ( i.e. , dynamical isometry realized by orthogonality), but only a benign status, which describes a state of the neural network where gradients can flow effectively through the model without being interrupted.
Revised: Therefore, we propose to seek a weaker constraint, not demanding the perfect trainability ( i.e. , exact isometry realized by orthogonality), but only a benign status, which describes a state of the neural network where gradients can flow effectively through the model without being interrupted.

Original: Orthogonality requires the Jacobian singular values to be exactly 1; in contrast, a benign trainability only requires them not to be extremely large or small so that the network can be trained normally.
Revised: Orthogonality requires the Jacobian singular values to be exactly 1; in contrast, a benign trainability only requires them not to be extremely large or small so that the network can be trained normally.

Original: To this end, we propose to decorrelate the kept filters from the pruned ones: in the target gram matrix, all the entries associated with the pruned filters are zero; all the other entries stay as they are (see Fig. 2(c)).
Revised: To this end, we propose to decorrelate the kept filters from the pruned ones: in the target gram matrix, all the entries associated with the pruned filters are zero; all the other entries stay as they are (see Fig. 2(c)).

Original: This scheme will be empirically justified (Tab. 3).
Revised: This scheme will be empirically justified (Tab. 3).

Original: Specifically, for the l -th layer, we sort the filters by their L 1 norms and select those with the smallest L 1 norms as unimportant filters (denoted as set S l ).
Revised: Specifically, all the filters in a layer are sorted based on their L 1 -norms. Then, we consider those with the smallest L 1 norms as unimportant filters , denoted as set S l (so the proposed method can also be regarded as a magnitude-based pruning method).

Original: Then, the proposed regularization term is,
Revised: Then, the proposed regularization term is,

Original: [Equation]
Revised: [Equation]

Original: where W denotes the weight matrix; 1 represents the matrix full of 1; m is a 0/1-valued column mask vector; ⊙ is the Hadamard (element-wise) product; and || · || F denotes the Frobenius norm.[-(c) De-correlate pruned from kept Target gram matrix WW T 𝑎 !! 𝑎 "" … 
-]
Revised: where W denotes the weight matrix; 1 represents the matrix full of 1; m is a 0/1-valued column mask vector; ⊙ is the Hadamard (element-wise) product; and || · || F denotes the Frobenius norm.[-(c) De-correlate pruned from kept Target gram matrix WW T 𝑎 !! 𝑎 "" … 
-]

Original: N … 𝑎 !"

Original: 𝑎 "! … … (a) Kernel orthogonality

Original: [-Target gram matrix WW T 1  … 
-]
Revised: [-Target gram matrix WW T 1  … 
-]

Original: N … (b) Kernel orthogonality for pruning

Original: [-Target gram matrix WW T 1  … 
-]
Revised: [-Target gram matrix WW T 1  … 
-]

Original: N …

Original: (2) BN regularization .
Revised: (2) BN regularization .

Original: Per the idea of preserving trainability, BN is not ignorable since BN layers are also trainable.
Revised: Per the idea of preserving trainability, BN is not ignorable since BN layers are also trainable.

Original: Removing filters will change the internal feature distributions.
Revised: Removing filters will change the internal feature distributions.

Original: If the learned BN statistics do not change accordingly, the error will accumulate and result in deteriorated performance (especially for deep networks).
Revised: If the learned BN statistics do not change accordingly, the error will accumulate and result in deteriorated performance (especially for deep networks).

Original: Consider the following BN formulation (Ioffe & Szegedy, 2015),
Revised: Consider the following BN formulation (Ioffe & Szegedy, 2015),

Original: [Equation]
Revised: [Equation]

Original: where ∗ stands for convolution; µ stands for the running mean, σ 2 for the running variance; ϵ is a small amount for numerical stability; the two learnable parameters are γ and β .
Revised: where ∗ stands for convolution; µ / σ 2 refers to the running mean/variance; ϵ , a small number, is used for numerical stability. Note, the two learnable parameters are γ and β .

Original: Although unimportant weights are enforced with regularization for sparsity, their magnitude can barely be exact zero, making the subsequent removal of filters biased .
Revised: Although unimportant weights are enforced with regularization for sparsity, their magnitude can barely be exact zero, making the subsequent removal of filters biased .

Original: This will modify the feature distribution and render the BN statistics incorrect.
Revised: This will modify the feature distribution and render the BN statistics incorrect.

Original: Using these biased BN statistics would be improper and damages trainability.
Revised: Using these biased BN statistics would be improper and damages trainability.

Original: To mitigate this influence of BN, we propose to regularize both the γ and β of pruned feature map channels to zero. The penalty term can be formulated as
Revised: To mitigate this influence of BN, we propose to regularize both the γ and β of pruned feature map channels to zero, which gives us the following BN penalty term,

Original: [Equation]
Revised: [Equation]

Original: The merits of BN regularization will be justified in our experiments (Tab. 4).
Revised: The merits of BN regularization will be justified in our experiments (Tab. 4).

Original: To sum, with the proposed regularization terms, the total error function is
Revised: To sum, with the proposed regularization terms, the total error function is

Original: [Equation]
Revised: [Equation]

Original: where L cls means the original classification loss.
Revised: where L cls means the original classification loss.

Original: The coefficient λ grows by a predefined constant ∆ per K u iterations (up to a ceiling τ ) during training to ensure the pruned parameters are rather close to zero (inspired by Wang et al.
Revised: The coefficient λ grows by a predefined constant ∆ per K u iterations (up to a ceiling τ ) during training to ensure the pruned parameters are rather close to zero (inspired by Wang et al.

Original: (2019; 2021b)).
Revised: (2019; 2021b)).

Original: See Algorithm 1 in Appendix for more details.
Revised: See Algorithm 1 in Appendix for more details.

Original: Discussion .
Revised: Discussion .

Original: Prior works (Liu et al., 2017; Ye et al., 2018) also propose to regularize BN for pruning.
Revised: Prior works (Liu et al., 2017; Ye et al., 2018) also study regularizing BN for pruning.

Original: Our BN regularization method is starkly different from theirs.
Revised: Our BN regularization method is starkly different from theirs.

Original: In terms of the motivation or goal,
Revised: In terms of the motivation or goal,

Original: Liu et al.
Revised: Liu et al.

Original: ; Ye et al.
Revised: ; Ye et al.

Original: (2018) regularize γ to learn unimportant filters, namely, regularizing BN is to indirectly decide which filters are unimportant.
Revised: (2018) regularize γ to learn unimportant filters, namely, regularizing BN is to indirectly decide which filters are unimportant.

Original: In contrast, in our method, unimportant filters are decided by their L 1 -norms.
Revised: In contrast, in our method, unimportant filters are decided by their L 1 -norms.

Original: We adopt BN regularization for a totally different consideration – to mitigate the side effect of breaking dynamical isometry, which is not mentioned at all in their works.
Revised: We adopt BN regularization for a totally different consideration – to mitigate the side effect of breaking trainability, which is not mentioned at all in their works.

Original: In terms of specific technique, Liu et al.
Revised: In terms of specific technique, Liu et al.

Original: ; Ye et al.
Revised: ; Ye et al.

Original: (2018) only regularize the scale factor γ (because it is enough to decide which filters are unimportant) while we regularize both the learnable parameters because only regularizing one still impairs dynamical isometry of the network.
Revised: (2018) only regularize the multiplier factor γ (because it is enough to decide which filters are unimportant) while we regularize both the learnable parameters because only regularizing one still impairs trainability of the network.

Original: Besides, we employ different regularization strength for different parameters (by the group of important filters vs. unimportant filters), while Liu et al.
Revised: Besides, we employ different regularization strength for different parameters (by the group of important filters vs. unimportant filters), while Liu et al.

Original: ; Ye et al.
Revised: ; Ye et al.

Original: (2018) simply use the same penaltystrength for all parameters – this is another key difference because regularizing all parameters (including those that are meant to be kept) will damage dynamical isometry, which is exactly what we want to avoid. In short, in either general motivation or specific technical details, our proposed BN regularization is distinct from previous works (Liu et al., 2017; Ye et al., 2018).
Revised: (2018) simply adopt a uniform penalty

Original: [Figure]
Revised: [Figure]

Original: runs. For reference, the unpruned model has mean JSV 2.4987, test accuracy 92.77.
Revised: For reference, the unpruned model has mean JSV 2.4987, test accuracy 92.77.

Revised: strength for all parameters – this is another key difference because regularizing all parameters (including those that are meant to be kept) will damage trainability, which is exactly what we want to avoid.

Revised: In short, in either general motivation or specific technical details, our proposed BN regularization is distinct from previous works (Liu et al., 2017; Ye et al., 2018).

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTS

Original: Datasets and networks .
Revised: Networks and datasets .

Original: We first conduct analyses with MLP-7-Linear network on MNIST (LeCun et al., 1998).
Revised: We first present some analyses with MLP-7-Linear network on MNIST (LeCun et al., 1998).

Original: Then compare our method to other plausible solutions with ResNet56 (He et al.
Revised: Then compare our method to other plausible solutions with the

Original: , 2016)/VGG19 (Simonyan & Zisserman, 2015)  on CIFAR10/100 (Krizhevsky, 2009).
Revised: ResNet56 (He et al., 2016) and VGG19 (Simonyan & Zisserman, 2015) networks, on the CIFAR and 100 datasets (Krizhevsky, 2009), respectively.

Original: Next we evaluate our algorithm on the large-scale ImageNet dataset (Deng et al., 2009) with ResNet34 and 50 (He et al., 2016).
Revised: Next we evaluate our algorithm on ImageNet1K (Deng et al., 2009) with ResNet34 and 50 (He et al., 2016).

Original: Finally, we conduct an ablation study to show the efficacy of two main technical novelties in our approach.
Revised: Finally, we present ablation studies to show the efficacy of two main technical novelties in our approach.

Original: On ImageNet, we take the official PyTorch (Paszke et al., 2019)pre-trained models as base models to maintain comparability with other methods.
Revised: On ImageNet, we use public torchvision models (Paszke et al., 2019) as the unpruned models for fair comparison with other papers.

Original: On other datasets, we train our own base models with comparable accuracies to those reported in their original papers.
Revised: On other datasets, we train our own base models with comparable accuracies to those reported in their original papers.

Original: See the Appendix (Tab. 5) for the specific training settings due to limited space here.
Revised: See the Appendix (Tab. 5) for the specific training settings.

Original: Comparison methods .
Revised: Comparison methods .

Original: We compare with Wang et al.
Revised: We compare with Wang et al.

Original: (2021a), which proposes a method OrthP  to recover broken dynamical isometry for pruning pretrained models.
Revised: (2021a), which proposes a method, OrthP , to recover broken trainability after pruning pretrained models.

Original: Besides, since maintaining orthogonality is the key and there are many existing orthogonality regularization papers (Xie et al., 2017; Wang et al., 2020; Huang et al., 2018; 2020; Wang et al., 2020), a straightforward solution is to combine them with L 1-norm pruning (Li et al., 2017) to see if they can help maintain or recover the broken dynamical isometry.
Revised: Furthermore, since weight orthogonality is closely related to network trainability and there have been plenty of orthogonality regularization approaches (Xie et al., 2017; Wang et al., 2020; Huang et al., 2018; 2020; Wang et al., 2020), a straightforward solution is to combine them with L 1 pruning (Li et al., 2017) to see whether they can help maintain or recover the broken trainability.

Original: There are two possible combination schemes: 1) apply orthogonal regularization methods beforeapplying L 1-norm pruning, 2) apply orthogonal regularization methods after L 1-norm pruning, namely, in finetuning.
Revised: Two plausible combination schemes are easy to see: 1) apply orthogonality regularization methods before L 1 pruning, 2) apply orthogonality regularization methods after L 1 pruning, i.e. , in retraining.

Original: Two representative orthogonality regularization methods are selected because of their proved effectiveness: kernel orthogonality (KernOrth) (Xie et al., 2017) and convolutional orthogonality (OrthConv) (Wang et al., 2020), so in total there arecombinations: L 1 + KernOrth, L 1 + OrthConv, KernOrth + L 1 , OrthConv + L 1 .
Revised: Two representative orthogonality regularization methods are selected because of their proved effectiveness: kernel orthogonality (KernOrth) (Xie et al., 2017) and convolutional orthogonality (OrthConv) (Wang et al., 2020), so in total there are four combinations: L 1 + KernOrth, L 1 + OrthConv, KernOrth + L 1 , OrthConv + L 1 .

Original: Comparison metrics .
Revised: Comparison metrics .

Original: (1) We compare the final test accuracy after finetuning with the similar FLOPs budget – this is currently the most prevailing metric to compare different filter pruning methods in classification.
Revised: (1) We examine the final test accuracy after retraining with the similar FLOPs budget – this is currently the most prevailing metric to compare different filter pruning methods in classification.

Original: Concretely, we compare two settings: a relatively large finetuning LR ( 1e-2 ) and a small one ( 1e-3 ).
Revised: Concretely, we compare two settings: a relatively large finetuning LR ( 1e-2 ) and a small one ( 1e-3 ).

Original: We introduce these settings because previous works (Renda et al., 2020; 
Revised: We introduce these settings because previous works (Renda et al., 2020; Le &

Original: Le & Hua, 2021; Wang et al., 2021a) showed that finetuning LR has a great impact on the final performance.
Revised: Hua, 2021; Wang et al., 2021a; 2023) have showed that finetuning LR has a great impact on the final performance.

Original: From this metric, we can see how sensitive different methods are to the finetuning LR.
Revised: From this metric, we can see how sensitive different methods are to the finetuning LR.

Original: (2) We also compare the test accuracy before finetuning – from this metric, we will see how robust different methods are in the face of weight removal. 
Revised: (2) We also compare the test accuracy before finetuning – from this metric, we will see how robust different methods are in the face of weight removal. [Table]

Original: 4. A NALYSIS : MLP-7-L INEAR +MNIST AND R ES N ET 56+CIFAR 
Revised: 4. A NALYSIS : MLP-7-L INEAR ON MNIST AND R ES N ET 56 ON CIFAR

Original: MLP-7-Linear is a seven-layer linear MLP.
Revised: MLP-7-Linear is a seven-layer linear MLP.

Original: It is adopted in Wang et al.
Revised: It is adopted in Wang et al.

Original: (2021a)
Revised: (2021a)

Original: for analysis because linear MLP is the only network that can achieve exact dynamical isometry (all JSVs are exactly 1) so far.
Revised: for analysis because linear MLP is the only network that can achieve exact dynamical isometry (all JSVs are exactly 1) so far.

Original: Their proposed dynamical isometry recovery method OrthP (Wang et al., 2021a) is shown[Table] to achieve exact isometry on linear MLP networks.
Revised: Their proposed dynamical isometry recovery method, OrthP (Wang et al., 2021a), is shown to achieve exact isometry on linear MLP networks.

Original: Since we claim our method TPP can maintain dynamical isometry too, conceivably [Table] our method should play a similar role to OrthP in pruning.
Revised: Since we claim our method TPP can maintain dynamical isometry too, conceivably, our method should play a similar role to OrthP in pruning.

Original: To confirm this, we prune the MLP-7-Linear network with our method.
Revised: To confirm this, we prune the MLP-7-Linear network with our method.

Original: TPP can perform as well as OrthP on linear MLP .
Revised: TPP can perform as well as OrthP on linear MLP .

Original: Results are presented in Fig.
Revised: In Fig.

Original: Fig. 3(b) is the one equipped with OrthP, which can exactly recover dynamical isometry (note its mean JSV right after pruning is 1.0000), so it works as the oracle here.
Revised: 3, (b) is the one equipped with OrthP, which can exactly recover dynamical isometry (note its mean JSV right after pruning is 1.0000), so it works as the oracle here.

Original: (1) OrthP improves the best accuracy from 91.36/90.54 to 92.79/92.77; using TPP, we obtain 92.81/92.77.
Revised: (1) OrthP improves the best accuracy from 91.36/90.54 to 92.79/92.77; using TPP, we obtain 92.81/92.77.

Original: Namely, in terms of accuracy, our method is as good as the oracle scheme.
Revised: Namely, in terms of accuracy, our method is as good as the oracle scheme.

Original: (2) Note the mean JSV right after pruning – the L 1norm pruning destroys the mean JSV from 2.4987 to 0.0040, and OrthP brings it back to 1.0000.
Revised: (2) Note the mean JSV right after pruning – the L 1 pruning destroys the mean JSV from 2.4987 to 0.0040, and OrthP brings it back to 1.0000.

Original: In comparison, TPP achieves 3.4875, at the same order of magnitude of 1.0000, also as good as OrthP. These demonstrate, in terms of either the final evaluation metric (test accuracy) or the trainability measure (mean JSV), our TPP performs as well as the oracle method OrthP on the linear MLP.
Revised: In comparison, TPP achieves 3.4875, at the same order of magnitude of 1.0000, also as good as OrthP. These demonstrate, in terms of either the final evaluation metric (test accuracy) or the trainability measure (mean JSV), our TPP performs as well as the oracle method OrthP on the linear MLP.

Original: Loss surface analysis with ResNet56+CIFAR10 .
Revised: Loss surface analysis with ResNet56 on CIFAR10 .

Original: We further analyze the loss surfaces (Li et al., 2018) of pruned networks (before finetuning) by different methods.
Revised: We further analyze the loss surfaces (Li et al., 2018) of pruned networks (before finetuning) by different methods.

Original: Our result (due to limited space, we defer this result to Appendix; see Fig. 4) suggests that the loss surface of our method is flatter than other methods, implying the loss landscape is easier for optimization.
Revised: Our result (due to limited space, we defer this result to Appendix; see Fig. 4) suggests that the loss surface of our method is flatter than other methods, implying the loss landscape is easier for optimization.

Original: 4. R ES N ET 56+CIFAR10 / VGG19+CIFAR 
Revised: 4. R ES N ET 56 ON CIFAR10 / VGG19 ON CIFAR

Original: Here we compare our method to other plausible solutions on the CIFAR10/100 datasets (Krizhevsky, 2009) with non-linear convolutional architectures.
Revised: Here we compare our method to other plausible solutions on the CIFAR datasets (Krizhevsky, 2009) with non-linear convolutional architectures.

Original: The results in Tab. 1 (for CIFAR10) and Tab.(for CIFAR100, deferred to Appendix due to limited space here) show that, (1) OrthP does not work well – L 1 + OrthP underperforms the original L 1 under all the five pruning ratios for both ResNet56 and VGG19.
Revised: The results in Tab. 1 (for CIFAR10) and Tab. 10 (for CIFAR100, deferred to Appendix due to limited space here) show that, (1) OrthP does not work well – L 1 + OrthP underperforms the original L 1 under all the five pruning ratios for both ResNet56 and VGG19.

Original: This further confirms the weight orthogonalization method proposed for linear networks indeed does not generalize to non-linear CNNs.
Revised: This further confirms the weight orthogonalization method proposed for linear networks indeed does not generalize to non-linear CNNs.

Original:  For KernOrth vs. OrthConv, the results look mixed – OrthConv is better when applied before the L 1-norm pruning.
Revised: (2) For KernOrth vs. OrthConv, the results look mixed – OrthConv is better when applied before the L 1 pruning.

Original: This is reasonable since OrthConv is shown more effective than KernOrth in enforcing more isometry (Wang et al., 2020), which in turn can stand more damage of pruning.
Revised: This is reasonable since OrthConv is shown more effective than KernOrth in enforcing more isometry (Wang et al., 2020), which in turn can stand more damage of pruning.

Original: (3) Of special note is that, none of the above five methods actually outperform the L 1-norm pruning or the simple scratch training.
Revised: (3) Of special note is that, none of the above five methods actually outperform the L 1 pruning or the simple scratch training.

Original: It means that neither enforcing more isometry before pruning nor  compensating isometry after pruning can help dynamical isometry recovery.
Revised: It means that neither enforcing more isometry before pruning nor [Table] compensating isometry after pruning can help dynamical isometry recover.

Original: In stark contrast, ourproposed TPP method outperforms L 1-norm pruning and scratch consistently against different pruning ratio (only one exception is pruning ratio 0.7 on ResNet56, but our method is still the second best and the gap to the best is only marginal: 93.46 vs. 93.51).Besides, note that the accuracy trend: [Table]
Revised: In stark contrast, our TPP method outperforms L 1 pruning and scratch consistently against different pruning ratios (only one exception is pruning ratio 0.7 on ResNet56, but our method is still the second best and the gap to the best is only marginal: 93.46 vs. 93.51).

Original:  in general, with a larger pruning ratio [Table] the advantage of TPP over L 1 or Scratch is more pronounced .
Revised: Besides, note that the accuracy trend: in general, with a larger pruning ratio, the advantage of TPP over L 1 or Scratch is more pronounced .

Original: This is because a larger pruning ratio means the dynamical isometry is damaged more, where our method can help more, thus harvest more performance gains.
Revised: This is because a larger pruning ratio means the trainability is hurt more, where our method can help more, thus harvest more performance gains.

Original: We will see similar trends many times.
Revised: We will see similar trends many times.

Original: (4) In Tabs.
Revised: (4) In Tabs.

Original: 1, we also present the results when the initial finetuning LR is 1e-3 .
Revised: 1, we also present the results when the initial finetuning LR is 1e-3 .

Original: In (Wang et al., 2021a), the authors argue that if the broken dynamical isometry can be well maintained/recovered, the final performance gap between LR 1e-2 and 1e-3 will be diminished.
Revised: Wang et al. (2021a) argue that if the broken dynamical isometry can be well maintained/recovered, the final performance gap between LR 1e-2 and 1e-3 will be diminished.

Original: Since we claim our method is able to maintain dynamical isometry, the performance gap should become smaller.
Revised: Now that we claim TPP is able to maintain dynamical isometry, the performance gap should become smaller.

Original: This is empirically verified in the table (note the Acc. diff.
Revised: This is empirically verified in the table.

Original: of TPP vs. that of L 1 ). In general, the accuracy gap between LR 1e-2 and LR 1e-3 of TPP is smaller than that of L 1-norm pruning.
Revised: In general, the accuracy gap between LR 1e-2 and LR 1e-3 of TPP is smaller than that of L 1 pruning.

Original: Two exceptions are PR 0.9/0.95 on ResNet56 – LR 1e-3 is unusually better than LR 1e-2 for L 1-norm pruning.
Revised: Two exceptions are PR 0.9/0.95 on ResNet56: LR 1e-3 is unusually better than LR 1e-2 for L 1 pruning.

Original: For now, we do not have a very concrete explanation for this but notably, it does not appear on VGG19.

Original: Hence, it probably is related to the residual architecture.

Original: We will keep exploring this in the future.

Original: Despite that, the general picturefrom the table is that the accuracy gap between LR 1eand 1e-2 turns smaller with TPP.
Revised: Despite them, the general picture is that the accuracy gap between LR 1eand 1e-2 turns smaller with TPP.

Original: This is a sign that dynamical isometry is effectively maintained.
Revised: This is a sign that trainability is effectively maintained.

Original: 4. I MAGE N ET B ENCHMARK
Revised: 4. I MAGE N ET B ENCHMARK

Original: We further evaluate TPP on ImageNet (Deng et al., 2009) in comparison to many existing filter pruning algorithms.
Revised: We further evaluate TPP on ImageNet (Deng et al., 2009) in comparison to many existing filter pruning algorithms.

Original: Results are shown in Tab. 2.
Revised: Results are shown in Tab. 2.

Original: Our method is consistently better than the others across different speedup ratios.
Revised: Our method is consistently better than the others across different speedup ratios.

Original: Moreover, with a larger speedup ratio, the advantage of our method is more pronounced.
Revised: Moreover, with a larger speedup ratio, the advantage of our method is more pronounced.

Original: For example, TPP outperforms Taylor-FO (Molchanov et al., 2019) by 1.15% in terms of the top-1 accuracy drop at the 2.31 × speedup track; at 3.06 × speedup, TPP leads
Revised: For example, TPP outperforms Taylor-FO (Molchanov et al., 2019) by 1.15% in terms of the top-1 accuracy drop at the 2.31 × speedup track; at 3.06 × speedup, TPP leads

Original: Taylor-FO (Molchanov et al., 2019) by 2.87%.
Revised: Taylor-FO (Molchanov et al., 2019) by 2.87%.

Original: This shows TPP is more robust to more aggressive pruning .
Revised: This shows TPP is more robust to more aggressive pruning .

Original: The reason is easy to see – more aggressive pruning hurts trainability (or dynamical isometry) more
Revised: The reason is easy to see – more aggressive pruning hurts trainability (or dynamical isometry) more

Original: (Lee et al., 2019; Wang et al., 2021a), where our method can find more use.
Revised: (Lee et al., 2019; Wang et al., 2023), where our method can find more use.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: We further compare to more strong pruning methods.
Revised: We further compare to more strong pruning methods.

Original: Notably, DMCP (Guo et al., 2020),
Revised: Notably, DMCP (Guo et al., 2020),

Original: LeGR (Chin et al., 2020), EagleEye (Li et al., 2020)
Revised: LeGR (Chin et al., 2020), EagleEye (Li et al., 2020)

Original: [Table]
Revised: [Table]

Original: and CafeNet (Su et al., 2021) have been shown outperformed by CHEX (Hou et al., 2022) (see their Tab. 1) with ResNet50 on ImageNet.
Revised: and CafeNet (Su et al., 2021) have been shown outperformed by CHEX (Hou et al., 2022) (see their Tab. 1) with ResNet50 on ImageNet.

Original: Therefore, here we only compare to CHEX.
Revised: Therefore [Table] here we only compare to CHEX.

Original: Following CHEX
Revised: Following CHEX

Original: [Table]
Revised: [Table]

Original: we employ more advanced training recipe ( e.g. , cosine LR schedule) referring to TIMM (Wightman et al., 2021).
Revised: we employ more advanced training recipe ( e.g. , cosine LR schedule) referring to TIMM (Wightman et al., 2021).

Original: Results in Tab.
Revised: Results in Tab.

Original: 2 show that our method surpasses CHEX at different FLOPs.
Revised: 2 show that our method surpasses CHEX at different FLOPs.

Original: 4. A BLATION S TUDY
Revised: 4. A BLATION S TUDY

Original: Finally [Table] we conduct ablation studies to demonstrate the merits of  two key contributions of our method: 1) proposing not to over-penalize the kept weights in orthogonalization ( i.e. , (c) vs. (b) in Fig. 2), 2) proposing to regularize the two learnable parameters in BN.
Revised: This section presents ablation studies to demonstrate the merits of TPP’s two major innovations: 1) proposing not to over-penalize the kept weights in orthogonalization ( i.e. , (c) vs. (b) in Fig. 2), 2) proposing to regularize the two learnable parameters in BN.

Original: The results are presented in Tabs.
Revised: The results are presented in Tabs.

Original: 3 and 4, where we compare the accuracy right after pruning ( i.e. , without finetuning).
Revised: 3 and 4, where we compare the accuracy right after pruning ( i.e. , without finetuning).

Original: We have the following major observations: (1) Tab.
Revised: We have the following major observations: (1) Tab.

Original: 3 shows using “decorrelate” (Fig. 2(c)) is better than “diagonal” (Fig. 2(b)) on the whole.
Revised: 3 shows using “decorrelate” (Fig. 2(c)) is better than “diagonal” (Fig. 2(b)) on the whole.

Original: Akin to Tabs.
Revised: Akin to Tabs.

Original: 1 and 10, under a larger pruning ratio, the advantage of “decorrelate” is more evident, except for too large sparsity (0.95 for
Revised: 1 and 10, at a greater sparsity ratio, the advantage of “decorrelate” is more evident, except for too large sparsity (0.95 for

Original: ResNet56, 0.9 for VGG19) because too large sparsity will break the isometry beyond repair.
Revised: ResNet56, 0.9 for VGG19) because too large sparsity will break the trainability beyond repair.

Original: (2) For

Original:  BN regularization, in Tab. 4, when it is switched off, the performance degrades.
Revised: For BN regularization, in Tab. 4, when it is switched off, the performance degrades.

Original: It also poses the similar trend: BN regularization is more helpful under the larger sparsity .
Revised: It also poses the similar trend: BN regularization is more helpful under the larger sparsity .

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Trainability maintenance is shown critical in neural network structured pruning, while few works have realized it on the modern large-scale non-linear deep networks.
Revised: Trainability maintenance is shown critical in neural network structured pruning, while few works have realized it on the modern large-scale non-linear deep networks.

Original: To achieve this goal, we present a new filter pruning method named trainability preserving pruning (TPP) based on regularization.
Revised: Towards this end, we present a new filter pruning method named trainability preserving pruning (TPP) based on regularization.

Original: Specifically, we propose a modified weight gram matrix as regularization target which does not unnecessarily over-penalize the weights.
Revised: Specifically, we propose a modified weight gram matrix as regularization target which does not unnecessarily over-penalize the weights.

Original: Besides, we propose to regularize the BN parameters to mitigate its damage to trainability.
Revised: Besides, we propose to regularize the BN parameters to mitigate its damage to trainability.

Original: TPP performs as effectively as the ground-truth trainability recovery method and more effective than other counterpart approaches based on weight orthogonality.
Revised: TPP performs as effectively as the ground-truth trainability recovery method and more effective than other counterpart approaches based on weight orthogonality.

Original: Furthermore, TPP also delivers promising performance when compared to many recent SOTA filter pruning approaches.
Revised: Furthermore, TPP also matches or beats many recent SOTA filter pruning approaches.

Original: To our best knowledge, this is the first work that explicitly tackles the trainability preserving problem in structured pruning that scales to the large-scale ImageNet dataset.
Revised: As far as we are concerned, TPP is the first approach that explicitly tackles the trainability preserving problem in structured pruning that scales to the large-scale ImageNet dataset.

Original: A A PPENDIX A. I MPLEMENTATION D ETAILS
Revised: MPLEMENTATION D ETAILS

Original: Code reference .
Revised: Code reference .

Original: We mainly refer to the following code implementations in this work.
Revised: We mainly refer to the following code implementations in this work.

Original: They are all open-licensed.
Revised: They are all open-licensed.

Original: • Official PyTorch ImageNet example 1 ; • GReg-1/GReg-2 (Wang et al., 2021b) 2 ; • OrthConv (Wang et al., 2020) 3 ; •
Revised: • Official PyTorch ImageNet example 1 ; • GReg-1/GReg-2 (Wang et al., 2021b) 2 ; • OrthConv (Wang et al., 2020) 3 ; •

Original: Rethinking the value of network pruning (Liu et al., 2019b) 4 .
Revised: Rethinking the value of network pruning (Liu et al., 2019b) 4 .

Original: Data split .
Revised: Data split .

Original: All the datasets in this paper are public datasets with standard APIs in PyTorch (Paszke et al., 2019).
Revised: All the datasets in this paper are public datasets with standard APIs in PyTorch (Paszke et al., 2019).

Original: We employs these standard APIs for the train/test data split to keep fair comparison with other methods.
Revised: We employs these standard APIs for the train/test data split to keep fair comparison with other methods.

Original: Training settings .
Revised: Training setups and hyper-parameters .

Original: The specific training settings are summarized in Tab. 5.
Revised: Tab. 5 summarizes the detailed training setups.

Original: For the hyper-parameters that are introduced in our TPP method: regularization granularity ∆ , regularization ceiling τ and regularization update interval K u , we summarize them in Tab. 6.
Revised: For the hyper-parameters that are introduced in our TPP method: regularization granularity ∆ , regularization ceiling τ and regularization update interval K u , we summarize them in Tab. 6.

Original: These hyper-parameters mainly refer to the official implementation of GReg-1 (Wang et al., 2021b)  since we tap into a similar growing regularization scheme as it does.
Revised: We mainly refer to the official code of GReg-1 (Wang et al., 2021b) when setting up these hyper-parameters, since we tap into a similar growing regularization scheme as GReg-1 does.

Revised: For small datasets (CIFAR and MNIST), each reported result is averaged by at least 3 random runs , mean and std reported.

Revised: For ImageNet-1K, we cannot run multiple times due to our limited resource budget.

Revised: This said, in general the results on ImageNet have been shown pretty stable.

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Hardware and running time .
Revised: Hardware and running time .

Original: We conduct all our experiments using 4 NVIDIA V100 GPUs (16GB memory per GPU).
Revised: We conduct all our experiments using 4 NVIDIA V100 GPUs (16GB memory per GPU).

Original: It takes roughly 41 hrs to prune ResNet50 on ImageNet using our TPP method (pruning and 90-epoch finetuning both included).
Revised: It takes roughly 41 hrs to prune ResNet50 on ImageNet using our TPP method (pruning and 90-epoch finetuning both included).

Original: Among them, 12 hrs (namely, close to 30%) are spent on pruning and 29 hrs are spent on finetuning (about 20 mins per epoch) [Table] [Table]
Revised: Among them, 12 hrs (namely, close to 30%) are spent on pruning and 29 hrs are spent on finetuning (about 20 mins per epoch).

Original: Layer-wise pruning ratios .
Revised: Layerwise pruning ratios .

Original: The layer-wise pruning ratios are pre-specified in this paper.
Revised: The layerwise pruning ratios are pre-specified in this paper.

Original: For the
Revised: For the

Original: ImageNet benchmark, we exactly follow GReg-1/GReg-

Revised: ImageNet benchmark, we exactly follow GReg-1/GReg-

Original: (Wang et al., 2021b) for the layer-wise pruning ratios to keep fair comparison to it.
Revised: (Wang et al., 2021b) for the layerwise pruning ratios to keep fair comparison to it.

Original: The specific numbers are summarized in Tab. 7.
Revised: The specific numbers are summarized in Tab. 7.

Original: Each number is the pruning ratio shared by all the layers of a stage in ResNet34/50.
Revised: Each number is the pruning ratio shared by all the layers of the same stage in ResNet34/50.

Revised: On top of these ratios, some layers are skipped, such as the last CONV layer in a residual block.

Revised: The best way to examine the detailed layerwise pruning ratio is to check the code at: https://github.com/MingSunTse/TPP.

Original: [Table]
Revised: [Table]

Original: A. S ENSITIVITY A NALYSIS OF H YPER - PARAMETERS
Revised: B S ENSITIVITY A NALYSIS OF H YPER - PARAMETERS

Original: Among the three hyper-parameters in Tab. 6, regularization ceiling τ works as a termination condition.
Revised: Among the three hyper-parameters in Tab. 6, regularization ceiling τ works as a termination condition.

Original: We only requires it to be large enough to ensure the weights are compressed to a very small amount.
Revised: We only requires it to be large enough to ensure the weights are compressed to a very small amount.

Original: It does not have to be 1.
Revised: It does not have to be 1.

Original: The final performance is also less sensitive to it.
Revised: The final performance is also less sensitive to it.

Original: The pruned performance seems to be more sensitive to the other two hyper-parameters, so here we conduct hyper-parameter sensitivity analysis to check their robustness.
Revised: The pruned performance seems to be more sensitive to the other two hyper-parameters, so here we conduct hyper-parameter sensitivity analysis to check their robustness.

Original: Results are presented in Tab. 8 and Tab. 9.
Revised: Results are presented in Tab. 8 and Tab. 9.

Original: Pruning ratio 0.7 (for ResNet56) and 0.5 (for VGG19) are chosen here because the resulted sparsity is the most representative ( i.e. , not too large or small).
Revised: Pruning ratio 0.7 (for ResNet56) and 0.5 (for VGG19) are chosen here because the resulted sparsity is the most representative ( i.e. , not too large or small).

Original: (1) For K u , in general, a larger K u tends to deliver a better result.
Revised: (1) For K u , in general, a larger K u tends to deliver a better result.

Original: This is no surprise since a larger
Revised: This is no surprise since a larger

Original: K u implies more time for the network to adapt and recover when undergoing the penalty.
Revised: K u allows more iterations for the network to adapt and recover when undergoing the penalty.

Original: (2) For ∆ , we do not see a clear pattern here: either a small or large ∆ can achieve the best result (for different networks).
Revised: For ∆ , we do not see a clear pattern here: either a small or large ∆ can achieve the best result (for different networks).

Original: On the whole, when varying the hyper-parameters within a reasonable range, the performance is pretty robust, no catastrophic failures.
Revised: On the whole, when varying the hyper-parameters within a reasonable range, the performance is pretty robust, no catastrophic failures.

Original: Moreover, note that the default setting is actually not the best for both K u and ∆ .
Revised: Moreover, note that the default setting is actually not the best for both K u and ∆ .

Original: This is because we did not hard-tune our hyper-parameters, however, they still achieve encouraging performance compared to the counterpart methods, as we have shown in the main text. 
Revised: This is because we did not heavily search the best hyper-parameters; however, they still achieve encouraging performance compared to the counterpart methods, as we have shown in the main text. [Table] [Table]

Original: Each is randomly run for 3 times
Revised: The details of our TPP method is summarized in Algorithm 1. 1: Input : Pretrained model Θ , layerwise pruning ratio r l of l -th layer, for l ∈ { 1 , 2 , · · · , L } . 2: Input : Regularization ceiling τ , penalty coefficient update interval K u , penalty granularity

Original: [Table]
Revised: ∆ .

Original: A. M ORE R ESULTS
Revised: 3: Init : Iteration i = 0 . λ

Original: Loss surface visualization .  
Revised: j = 0 for all filter j . Set pruned filter indices S l by L 1 -norm sorting. 4: while λ j ≤ τ , for j ∈ S l do 5: if i % K u = 0 then 6: λ j = λ j + ∆ for j ∈ S l . ▷ Update regularization co-efficient in Eq. (

Revised: ) 7: end if 8:

Original: The loss surface visualization of ResNet56 on CIFAR10 is presented in Fig.
Revised: Network forward, loss (Eq. (6)) backward, parameter update by stochastic gradient descent. Update iteration: i = i + 1 .

Original: VGG19 + CIFAR100 . The results of VGG19 on CIFAR100 is shown in Tab. 10.
Revised: 10: end while 11: Remove filters in S l to obtain a smaller model Θ ′ . 12: Finetune Θ ′ to regain accuracy.

Original: [Table] [Figure] [Table] [Figure]
Revised: 13: Output : Finetuned model Θ ′ .

Original: [Table] [Figure] [Table] [Figure]
Revised: D R ESULTS O MITTED FROM THE M AIN T EXT

Original: [Table] [Figure] [Table] [Figure]
Revised: Loss surface visualization .

Original: [Table] [Figure] [Table] [Figure]
Revised: The loss surface visualization of ResNet56 on CIFAR10 is presented in Fig.

Original: [Table] [Figure] [Table] [Figure]
Revised: VGG19 on CIFAR100 .

Original: [Table] [Figure] [Table] [Figure]
Revised: The results of VGG19 on CIFAR100 is shown in Tab. 10.

Original: Examination of the early finetuning phase .
Revised: Examination of the early finetuning phase .

Original: To further understand how pruning hurts trainability and how our TPP method maintains it, in Tab. 11, we list the mean JSVs of the first 10-epoch finetuning (at pruning ratio 0.9).
Revised: To further understand how pruning hurts trainability and how our TPP method maintains it, in Tab. 11, we list the mean JSVs of the first 10-epoch finetuning (at pruning ratio 0.9).

Original: Note the obvious mean JSV gap between LR 0.01 and LR 0.
Revised: Note the obvious mean JSV gap between LR 0.01 and LR 0.

Original: without OrthP : LR 0.01 can reach mean JSV 0.65 just after 1 epoch of finetuning while LR 0.
Revised: without OrthP : LR 0.01 can reach mean JSV 0.65 just after 1 epoch of finetuning while LR 0.

Original: takes over 8 epochs.
Revised: takes over 8 epochs.

Original: When OrthP is used, this gap greatly shrinks.
Revised: When OrthP is used, this gap greatly shrinks.

Original: We also list the test accuracy of the first 10-epoch finetuning in Tab. 12.
Revised: We also list the test accuracy of the first 10-epoch finetuning in Tab. 12.

Original: Particularly note that the test accuracy correlates well with the mean JSV trend under each setting, implying that the damaged trainability primarily answers for the under-performance of LR 0.001 after pruning.
Revised: Particularly note that the test accuracy correlates well with the mean JSV trend under each setting, implying that the damaged trainability primarily answers for the under-performance of LR 0.001 after pruning.

Original: Then, when TPP is used in place of OrthP, we can see after 1-epoch finetuning, the model can achieve mean JSV above 1 and test accuracy over 90%, which are particularly similar to the effect of using OrthP. These re-iterate that the proposed TPP method can work just as effectively as the ground-truth trainability recovery method OrthP on this toy setup.
Revised: Then, when TPP is used in place of OrthP, we can see after 1-epoch finetuning, the model can achieve mean JSV above 1 and test accuracy over 90%, which are particularly similar to the effect [Figure] [Figure]

Original: The details of our TPP method is summarized in Algorithm 1.
Revised: [Table]

Original: A. N EWLY ADDED RESULTS DURING REBUTTAL
Revised: Initial finetuning LR 1e-

Original: [Table]
Revised: [Table]

Original: LR= 10 − 2 , w/o OrthP 0.00 0.65 0.89 1.01 0.97 1.10 1.14 1.27 1.33 1.29 1.
Revised: of using OrthP. These re-iterate that the proposed TPP method can work just as effectively as the ground-truth trainability recovery method OrthP on this toy setup. E M ORE A NALYTICAL R ESULTS In this section, we add more analytical results to help readers better understand how TPP works. E. TPP + Random B ASE M ODELS

Original: LR= 10 − 3 , w/o OrthP
Revised: Pruning is typically conducted on a pretrained model. TPP is also brought up in this context. This said, the fundamental idea of TPP, i.e. , the proposed regularized training which can preserve trainability from the sparsifying action, is actually independent of the base model. Therefore, we may expect TPP can also surpass the L 1 pruning method (Li et al., 2017) in the case of pruning a random model – this is confirmed in Tab. 13.

Original: [Table] [Table]
Revised: As expected, the damaged trainability problem does not only exit for pruning pretrained models, but also exits for pruning random models; so TPP performs better than L 1 pruning, especially in the large sparsity regime. Moreover, in Fig. 5, we show the mean JSV and test accuracy when pruning a random model with different schemes ( L 1 , L 1 +OrthP, and our TPP).

Original: [Table] [Table] [Table]
Revised: We observe that, in general, the JSV and test accuracy of pruning a random model pose similar patterns to pruning a pretrained model (Fig. 3):

Original: [Table]
Revised: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: LR= 10 − 2 , w/ OrthP
Revised: • Using L 1 , the LR=

Original: [Table]
Revised: 0.01 achieves “better” (not really better, but due to damaged trainability, the performance of LR=0.001 has been underestimated, see Wang et al.

Original: [Table]
Revised: (2023) for more detailed discussions) JSV and test accuracy; note the test accuracy solid line is above the dashed line by an obvious margin.

Original: LR= 10 − 3 , w/ OrthP [Table] LR= [Table]
Revised: • While using L 1 +OrthP or our TPP, the LR=0.001 can actually match LR=0.01.

Original: , w/
Revised: Same as the case of pruning a pretrained model, here, TPP behaves similarly to the oracle trainability recovery method OrthP. • To summarize, the trainability-preserving effect of TPP also generalizes to the case of pruning random networks.

Original: [Table]
Revised: In Fig.

Original: [Table]
Revised: 6, we show the mean JSV and test accuracy over the overall process (including the regularized training and retraining) of TPP applied to a random model.

Original: LR= 10 − 3 , w/ TPP 
Revised: E. TPP + M ORE A DVANCED P RUNING C RITERIA

Original: [Table]
Revised: In the main text, we demonstrate the effectiveness of TPP, which employs the magnitude ( L 1 -norm) of filters to decide masks. It is of interest whether the effectiveness of TPP can carry over to other more advanced pruning criteria.

Original: [Table]
Revised: Here we combine TPP with other 3 more advanced criteria:

Original: LR
Revised: SSL (Wen et al., 2016), DeepHoyer (Yang et al., 2020), and Taylor-FO (Molchanov et al., 2019). SSL and DeepHoyer are regularization-based pruning methods like ours; differently, their pruned indices (as well as the pruning ratio) for each layer is not pre-defined, but “learned” by the regularized training. As such, the layerwise pruning ratios are not uniform (see Fig.

Original: [Table]
Revised: 7 for an example).

Original: LR
Revised: Taylor-FO is a more complex pruning criterion than magnitude by taking into account the first-order gradient information. Specifically, we first use these methods to decide the layerwise pruned indices, given a total pruning ratio.

Original: [Table]
Revised: Then, we inherit these layerwise pruned indices when using our TPP method.

Original: M ea n J S V lr=0.001 lr=0.
Revised: M ea n J S V lr=0.001 lr=0.

Original: 80 Epoch
Revised: 80 Epoch

Original: T e s t acc
Revised: T e s t acc

Original: u r ac
Revised: u r ac

Original: y
Revised: y

Original: ( % ) lr=0.001 lr=0.
Revised: ( % ) lr=0.001 lr=0.

Original: M ea n J S V lr=0.001 lr=0.
Revised: M ea n J S V lr=0.001 lr=0.

Original: 80 Epoch
Revised: 80 Epoch

Original: T e s t acc
Revised: T e s t acc

Original: u r ac
Revised: u r ac

Original: y
Revised: y

Original: ( % ) lr=0.001 lr=0.
Revised: ( % ) lr=0.001 lr=0.

Original: M ea n J S V lr=0.001 lr=0.
Revised: M ea n J S V lr=0.001 lr=0.

Original: 80 Epoch
Revised: 80 Epoch

Original: T e s t acc
Revised: T e s t acc

Original: u r ac
Revised: u r ac

Original: y
Revised: y

Original: ( % )lr=0.001 lr=0.
Revised: ( % ) lr=0.

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original: [Figure]

Original: lr=0.
Revised: lr=0.

Original: lr=0.

Original: [Figure]
Revised: [Figure]

Original: Epoch

Original: [Figure]
Revised: [Figure]

Original: lr=

Original: [Figure]
Revised: [Figure]

Original: training and finetuning with TPP (network:
Revised: and retraining with TPP (network:

Original: MLP-7-Linear, dataset: MNIST) when pruning a random model .
Revised: MLP-7-Linear, dataset: MNIST) when pruning a random model .

Original: The model spends 100 epochs on regularized training (LR schedule at this stage is the same, 0.001, fixed); then pruned by L 1norm pruning (Li et al., 2017); then finetuned with different LR schedules (0.01 vs. 0.001).
Revised: The model spends 100 epochs on regularized training (LR schedule at this stage is the same, 0.001, fixed); then pruned by Lpruning (Li et al., 2017); then retrained with different LR schedules (0.01 vs. 0.001).

Original: As seen, pruning the TPP regularized model does not make the mean JSV drop significantly at the pruning point (epoch 100), namely, trainability preserved .
Revised: As seen, pruning the TPP regularized model does not make the mean JSV drop significantly at the pruning point (epoch 100), namely, trainability preserved .

Original: In stark contrast, when applying L 1-norm pruning to a normally trained ( i.e. , not TPP regularized) model, mean JSV drops from 2.4987 to 0.0040 (see
Revised: In stark contrast, when applying L 1 pruning to a normally trained ( i.e. , not TPP regularized) model, mean JSV drops from 2.4987 to 0.0040 (see

Original: Fig. 3), namely, trainability not preserved.
Revised: Fig. 3), namely, trainability not preserved.

Original: Note, when trainability is preserved, finetuning LR 0.
Revised: Note, when trainability is preserved, finetuning LR 0.

Original: and 0.001 do not pose obvious test accuracy gap; while trainability is not preserved, the gap would be obvious – see Fig. 5(a).
Revised: and 0.001 do not pose obvious test accuracy gap; while trainability is not preserved, the gap would be obvious – see Fig. 5(a).

Original: [Figure]
Revised: Results are shown in Tab.

Original: [Figure]
Revised: We observe, at small PRs (0.3-0.7), TPP performs similarly to L 1 . This agrees with Tab. 1, where TPP is comparable to L 1 . At large PRs (0.9, 0.95), the advantage of TPP starts to expose more – at PR 0.9/0.95, TPP beats L 1 by 0.9/1.44 with SSL learned pruned indices, which is a statistically significant advantage as indicated by the std (and again, when PR is larger, the advantage of TPP is generally more pronounced). This table shows the advantage of

Original: We can observe that at large PRs ( significantly the the
Revised: TPP indeed can carry over to other layerwise pruning ratios derived from more advanced pruning criteria. [Table]

Original: L
Revised: [Figure] [Table] [Figure] [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: Sparsity

Revised: [Figure]

Revised: [Table]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Original: Li et al because of better
Revised: [Table] L 1 (w/ Taylor [Table] FO pruned indices

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: Taylor

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Original: [Figure]
Revised: [Figure]

Revised: E. T RAINING C URVE P LOTS : TPP vs. L 1 PRUNING

Revised: In Fig.

Revised: [Figure]

Revised: we show the

Revised: [Figure]

Revised: [Figure]

Revised: E. P RUNING R ES N ET 50 ON I MAGE N ET WITH L ARGER S PARSITY R ATIOS

Revised: It is noted that our method only beats some of the compared methods marginally ( < 0.5% top-1 accuracy) in low sparsity regime (around 2 ×∼ 3 × speedup, see Tab. 2).

Revised: This is mainly because when the sparsity is low, network trainability is not seriously damaged, thus our trainability -preserving method cannot fully expose its advantage.

Revised: Here we showcase a scenario that trainability is intentionally damaged more dramatically.

Revised: In Tab. 2, when pruning ResNet50, researchers typically do not prune all the layers – the last CONV layer in a residual block is usually spared (Li et al., 2017; Wang et al., 2021b), for the sake of performance.

Revised: Here we intentionally prune all the layers (only excluding the first CONV and last classifier FC) in ResNet50.

Revised: For the L 1 pruning method (Li et al., 2017)

Revised: (we report a stronger version re-implemented by Wang et al.

Revised: , different layers are pruned independently since the layerwise pruning ratio has been specified.

Revised: All the hyper-parameters of the retraining process are maintained the same for fair comparison, per the spirit brought up in Wang et al.

Revised: The results are shown in Tab. 15.

Revised: As seen, in the case that trainability is damaged more, our TPP beats L 1 by 0.77 to 2.17 top-1 accuracy on ImageNet, much more significant than Tab. 2.

Revised: F M ORE D ISCUSSIONS

Revised: Can TPP be useful for finding lottery ticket subnetwork in a filter level?

Revised: [Table]

Revised: To our best knowledge, filter-level winning tickets (WT) are still hard to find even using the original

Revised: LTH pipeline.

Revised: Few attempts in this direction have succeeded – E.g. , Chen et al.

Revised: (2022) tried, but they can only achieve a bit marginal sparsity ( 30%) with filter-level WT (see their Fig.

Revised: 3, ResNet on ImageNet) while weight-level WT typically can be found at over 90% sparsity.

Revised: This said, we do think this paper can contribute in that direction since preserving trainability is also a central issue in

Revised: LTH, too.