Original: SOSP: E FFICIENTLY C APTURING G LOBAL C ORRELATIONS BY S ECOND -O RDER S TRUCTURED P RUNING
Revised: SOSP: E FFICIENTLY C APTURING G LOBAL C ORRELATIONS BY S ECOND -O RDER S TRUCTURED P RUNING

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Pruning neural networks reduces inference time and memory costs.
Revised: Pruning neural networks reduces inference time and memory costs.

Original: On standard hardware, these beneﬁts will be especially prominent if coarse-grained structures, like feature maps, are pruned.
Revised: On standard hardware, these beneﬁts will be especially prominent if coarse-grained structures, like feature maps, are pruned.

Original: We devise two novel saliency-based methods for second-order structured pruning (SOSP) which include correlations among all structures and layers.
Revised: We devise two novel saliency-based methods for second-order structured pruning (SOSP) which include correlations among all structures and layers.

Original: Our main method SOSP-H employs an innovative second-order approximation, which enables saliency evaluations by fast Hessian-vector products.
Revised: Our main method SOSP-H employs an innovative second-order approximation, which enables saliency evaluations by fast Hessian-vector products.

Original: SOSP-H thereby scales like a ﬁrst-order method despite taking into account the full
Revised: SOSP-H thereby scales like a ﬁrst-order method despite taking into account the full

Original: Hessian.
Revised: Hessian.

Original: We validate SOSP-H by comparing it to our second method SOSP-I that uses a well-established Hessian approximation, and to numerous state-of-the-art methods.
Revised: We validate SOSP-H by comparing it to our second method SOSP-I that uses a well-established Hessian approximation, and to numerous state-of-the-art methods.

Original: While SOSP-H performs on par or better in terms of accuracy, it has clear advantages in terms of scalability and efﬁciency.
Revised: While SOSP-H performs on par or better in terms of accuracy, it has clear advantages in terms of scalability and efﬁciency.

Original: This allowed us to scale SOSP-H to large-scale vision tasks, even though it captures correlations across all layers of the network.
Revised: This allowed us to scale SOSP-H to large-scale vision tasks, even though it captures correlations across all layers of the network.

Original: To underscore the global nature of our pruning methods, we evaluate their performance not only by removing structures from a pretrained network, but also by detecting architectural bottlenecks.
Revised: To underscore the global nature of our pruning methods, we evaluate their performance not only by removing structures from a pretrained network, but also by detecting architectural bottlenecks.

Original: We show that our algorithms allow to systematically reveal architectural bottlenecks, which we then widen to further increase the accuracy of the networks.
Revised: We show that our algorithms allow to systematically reveal architectural bottlenecks, which we then widen to further increase the accuracy of the networks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep neural networks have consistently grown in size over the last years with increasing performance.
Revised: Deep neural networks have consistently grown in size over the last years with increasing performance.

Original: However, this increase in size leads to slower inference, higher computational requirements and higher cost.
Revised: However, this increase in size leads to slower inference, higher computational requirements and higher cost.

Original: To reduce the size of the networks without affecting their performance, a large number of pruning algorithms have been proposed (e.g., LeCun et al., 1990; Hassibi et al., 1993; Reed, 1993;
Revised: To reduce the size of the networks without affecting their performance, a large number of pruning algorithms have been proposed (e.g., LeCun et al., 1990; Hassibi et al., 1993; Reed, 1993;

Original: Han et al., 2015; Blalock et al., 2020).
Revised: Han et al., 2015; Blalock et al., 2020).

Original: Pruning can either be unstructured , i.e. removing individual weights, or structured , i.e. removing entire substructures like nodes or channels.
Revised: Pruning can either be unstructured , i.e. removing individual weights, or structured , i.e. removing entire substructures like nodes or channels.

Original: Single-shot pruning methods, as investigated in this work, usually consist of three steps: 1) training, 2) pruning, 3) another training step often referred to as ﬁne-tuning .
Revised: Single-shot pruning methods, as investigated in this work, usually consist of three steps: 1) training, 2) pruning, 3) another training step often referred to as ﬁne-tuning .

Original: Unstructured pruning can signiﬁcantly reduce the number of parameters of a neural network with only little loss in the accuracy, but the resulting networks often show only a marginal improvement in training and inference time, unless specialized hardware is used (He et al., 2017).
Revised: Unstructured pruning can signiﬁcantly reduce the number of parameters of a neural network with only little loss in the accuracy, but the resulting networks often show only a marginal improvement in training and inference time, unless specialized hardware is used (He et al., 2017).

Original: In contrast, structured pruning can directly reduce inference time and even training time when applied at initialization (Lee et al., 2018).
Revised: In contrast, structured pruning can directly reduce inference time and even training time when applied at initialization (Lee et al., 2018).

Original: To exploit these advantages, in this work, we focus on structured pruning.
Revised: To exploit these advantages, in this work, we focus on structured pruning.

Original: Global pruning removes structure by structure from all available structures of a network until a predeﬁned percentage of pruned structures is reached.
Revised: Global pruning removes structure by structure from all available structures of a network until a predeﬁned percentage of pruned structures is reached.

Original: Recent examples for global structured pruning methods are NN Slimming (Liu et al., 2017), C-OBD and EigenDamage (Wang et al., 2019a).
Revised: Recent examples for global structured pruning methods are NN Slimming (Liu et al., 2017), C-OBD and EigenDamage (Wang et al., 2019a).

Original: Local pruning, on the other hand, ﬁrst subdivides all global structures into subsets (e.g. layers) and removes a percentage of structures of each subset.
Revised: Local pruning, on the other hand, ﬁrst subdivides all global structures into subsets (e.g. layers) and removes a percentage of structures of each subset.

Original: Recent examples for local pruning methods are HRank (Lin et al., 2019), CCP (Peng et al., 2019), FPGM (He et al., 2019) and Variational Pruning (Zhao et al., 2019).
Revised: Recent examples for local pruning methods are HRank (Lin et al., 2019), CCP (Peng et al., 2019), FPGM (He et al., 2019) and Variational Pruning (Zhao et al., 2019).

Original: Most local pruning schemes use a predeﬁned layer-wise pruning ratio, which ﬁxes the percentage of structures removed per layer.
Revised: Most local pruning schemes use a predeﬁned layer-wise pruning ratio, which ﬁxes the percentage of structures removed per layer.

Original: While this prevents the layers from collapsing, it also reduces some of the degrees of freedom, since some layers may be less important than others.
Revised: While this prevents the layers from collapsing, it also reduces some of the degrees of freedom, since some layers may be less important than others.

Original: Other local pruning methods like AMC (He et al., 2018) learn the layer-wise pruning ratios in a ﬁrst step.
Revised: Other local pruning methods like AMC (He et al., 2018) learn the layer-wise pruning ratios in a ﬁrst step.

Original: A key challenge for global saliency-based pruning is to ﬁnd an objective which can be efﬁciently calculated to make the approach scalable to large-scale, modern neural networks.
Revised: A key challenge for global saliency-based pruning is to ﬁnd an objective which can be efﬁciently calculated to make the approach scalable to large-scale, modern neural networks.

Original: While second-order pruning methods are usually more accurate than ﬁrst-order methods (Molchanov et al., 2019), calculating the full second-order saliency objective is intractable for modern neural networks.
Revised: While second-order pruning methods are usually more accurate than ﬁrst-order methods (Molchanov et al., 2019), calculating the full second-order saliency objective is intractable for modern neural networks.

Original: Therefore, most saliency-based pruning methods such as OBD (e.g., LeCun et al., 1990) or C-OBD (Wang et al., 2019a) evaluate the effect of removing a single weight or structure on the loss of the neural network in isolation.
Revised: Therefore, most saliency-based pruning methods such as OBD (e.g., LeCun et al., 1990) or C-OBD (Wang et al., 2019a) evaluate the effect of removing a single weight or structure on the loss of the neural network in isolation.

Original: However, this neglects possible correlations between different structures, which are captured by the off-diagonal second-order terms, and hence, can signiﬁcantly harm the estimation of the sensitivities.
Revised: However, this neglects possible correlations between different structures, which are captured by the off-diagonal second-order terms, and hence, can signiﬁcantly harm the estimation of the sensitivities.

Original: Finding a global second-order method that both considers off-diagonal terms and scales to modern neural networks is still an unsolved research question.
Revised: Finding a global second-order method that both considers off-diagonal terms and scales to modern neural networks is still an unsolved research question.

Original: Our main goal in this work is to devise a simple and efﬁcient second-order pruning method that considers all global correlations for structured sensitivity pruning.
Revised: Our main goal in this work is to devise a simple and efﬁcient second-order pruning method that considers all global correlations for structured sensitivity pruning.

Original: In addition, we want to highlight the beneﬁts that such methods may have over other structured global and local pruning schemes.
Revised: In addition, we want to highlight the beneﬁts that such methods may have over other structured global and local pruning schemes.

Original: Our contributions are as follows:
Revised: Our contributions are as follows:

Original: • We develop two novel saliency-based pruning methods for second-order structured pruning (SOSP) and analyze them theoretically.
Revised: • We develop two novel saliency-based pruning methods for second-order structured pruning (SOSP) and analyze them theoretically.

Original: We show that both of our methods drastically improve on the complexity of a naive second-order approach, which is usually intractable for modern neural networks.
Revised: We show that both of our methods drastically improve on the complexity of a naive second-order approach, which is usually intractable for modern neural networks.

Original: Further, we show that our SOSP-H method, which is based on fast Hessian-vector products, has the same low complexity as ﬁrst-order methods, while taking the full Hessian into account.
Revised: Further, we show that our SOSP-H method, which is based on fast Hessian-vector products, has the same low complexity as ﬁrst-order methods, while taking the full Hessian into account.

Original: • We compare the performance and the scaling of SOSP-H to that of SOSP-I, which is based on the well-known Gauss-Newton approximation.
Revised: • We compare the performance and the scaling of SOSP-H to that of SOSP-I, which is based on the well-known Gauss-Newton approximation.

Original: While both methods perform on par, SOSP-H shows better scaling.
Revised: While both methods perform on par, SOSP-H shows better scaling.

Original: We then benchmark our SOSP methods against a variety of state-of-the-art pruning methods and show that they achieve comparable or better results at lower computational costs for pruning.
Revised: We then benchmark our SOSP methods against a variety of state-of-the-art pruning methods and show that they achieve comparable or better results at lower computational costs for pruning.

Original: • We exploit the structure of the pruning masks found by our SOSP methods to widen architectural bottlenecks , which further improves the performance of the pruned networks.
Revised: • We exploit the structure of the pruning masks found by our SOSP methods to widen architectural bottlenecks, which further improves the performance of the pruned networks.

Original: We diagnose layers with disproportionally low pruning ratios as architectural bottlenecks.
Revised: We diagnose layers with disproportionally low pruning ratios as architectural bottlenecks.

Original: PyTorch code implementing our method will be published upon acceptance of this manuscript.

Original: Related work is discussed in the light of our results in the Discussion section (Sec. 4).
Revised: Related work is discussed in the light of our results in the Discussion section (Sec. 4).

Original:  SOSP: S ECOND - ORDER STRUCTURED PRUNING
Revised:  SOSP: S ECOND - ORDER STRUCTURED PRUNING

Original: A neural network (NN) maps an input x ∈ R d to an output f θ ( x ) ∈ R D , where θ ∈ R P are its
Revised: A neural network (NN) maps an input x ∈ R d to an output f θ ( x ) ∈ R D , where θ ∈ R P are its

Original: P parameters.
Revised: P parameters.

Original: NN training proceeds, after random initialization θ = θ 0 of the weights, by minibatch stochastic gradient descent on the empirical loss L ( θ ) :
Revised: NN training proceeds, after random initialization θ = θ 0 of the weights, by minibatch stochastic gradient descent on the empirical loss L ( θ ) :

Original: = 1 N (cid:80) Nn =1 (cid:96) ( f θ ( x n ) , y n ) , given the training dataset { ( x 1 , y 1 ) , . . .
Revised: = 1 N (cid:80) Nn =1 (cid:96) ( f θ ( x n ) , y n ) , given the training dataset { ( x 1 , y 1 ) , . . .

Original: , ( x N , y N ) } .
Revised: , ( x N , y N ) } .

Original: In the classiﬁcation case, y ∈ { 1 , . . .
Revised: In the classiﬁcation case, y ∈ { 1 , . . .

Original: , D } is a discrete ground-truth label and (cid:96) ( f θ ( x ) , y ) :
Revised: , D } is a discrete ground-truth label and (cid:96) ( f θ ( x ) , y ) :

Original: = − log σ ( f θ ( x ))
Revised: = − log σ ( f θ ( x ))

Original: y the cross-entropy loss, with σ : R D → R D the softmax-function.
Revised: y the cross-entropy loss, with σ : R D → R D the softmax-function.

Original: For regression, y ∈ R D and (cid:96) ( f θ ( x ) , y ) = 12 (cid:107) f
Revised: For regression, y ∈ R D and (cid:96) ( f θ ( x ) , y ) = 12 (cid:107) f

Original: θ ( x ) − y (cid:107) 2 is the squared loss.
Revised: θ ( x ) − y (cid:107) 2 is the squared loss.

Original: Structured pruning aims to remove weights or rather entire structures from a NN f θ with parameters θ .
Revised: Structured pruning aims to remove weights or rather entire structures from a NN f θ with parameters θ .

Original: A structure can be a ﬁlter (channel) in a convolutional layer, a neuron in a fully-connected layer, or an entire layer in a parallel architecture.
Revised: A structure can be a ﬁlter (channel) in a convolutional layer, a neuron in a fully-connected layer, or an entire layer in a parallel architecture.

Original: We assume the NN in question has been segmented into
Revised: We assume the NN in question has been segmented into

Original: S structures s = 1 , . .
Revised: S structures s = 1 , . .

Original: , S , which can potentially be pruned.
Revised: , S , which can potentially be pruned.

Original: We deﬁne the notation θ s ∈ R P as the vector whose only nonzero components are those weights from θ that belong to structure s .
Revised: We deﬁne the notation θ s ∈ R P as the vector whose only nonzero components are those weights from θ that belong to structure s .

Original: Then, a pruning mask is a set M = { s 1 , . . .
Revised: Then, a pruning mask is a set M = { s 1 , . . .

Original: , s m } of structures, and applying a mask M to a NN f θ means to consider the NN with parameter vector θ \ M :
Revised: , s m } of structures, and applying a mask M to a NN f θ means to consider the NN with parameter vector θ \ M :

Original: = θ − (cid:80) s ∈ M θ s .
Revised: = θ − (cid:80) s ∈ M θ s .

Original: We now develop our pruning methods that incorporate global correlations into their saliency assessment by efﬁciently including the second-order loss terms.
Revised: We now develop our pruning methods that incorporate global correlations into their saliency assessment by efﬁciently including the second-order loss terms.

Original: Our method SOSP-I allows a direct interpretation in terms of individual loss sensitivities, while our main method SOSP-H remains very efﬁcient for large-scale networks due to its Hessian-vector product approximation.
Revised: Our method SOSP-I allows a direct interpretation in terms of individual loss sensitivities, while our main method SOSP-H remains very efﬁcient for large-scale networks due to its Hessian-vector product approximation.

Original: The basic idea behind both our pruning methods is to select the pruning mask M so as to (approximately) minimize the joint effect on the network loss
Revised: The basic idea behind both our pruning methods is to select the pruning mask M so as to (approximately) minimize the joint effect on the network loss

Original: [Equation]
Revised: [Equation]

Original: of removing all structures in M , subject to a constraint on the overall pruning ratio.
Revised: of removing all structures in M , subject to a constraint on the overall pruning ratio.

Original: To circumvent this exponentially large search space, we approximate the loss up to second order, so that
Revised: To circumvent this exponentially large search space, we approximate the loss up to second order, so that

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: collapses to single-structure contributions plus pairwise correlations.
Revised: collapses to single-structure contributions plus pairwise correlations.

Original: The ﬁrst-order terms λ 1 ( s ) :
Revised: The ﬁrst-order terms λ 1 ( s ) :

Original: = θ s · d L ( θ ) /dθ
Revised: = θ s · d L ( θ ) /dθ

Original: in (1) are efﬁcient to evaluate by computing the gradient d L ( θ ) /dθ
Revised: in (1) are efﬁcient to evaluate by computing the gradient d L ( θ ) /dθ

Original: ∈ R P once and then a (sparse) dot product for every s .
Revised: ∈ R P once and then a (sparse) dot product for every s .

Original: In contrast, the network Hessian
Revised: In contrast, the network Hessian

Original: H ( θ ) :
Revised: H ( θ ) :

Original: = d 2 L ( θ ) /dθ
Revised: = d 2 L ( θ ) /dθ

Original: 2 ∈ R P × P in (1) is prohibitively expensive to compute or store in full.
Revised: 2 ∈ R P × P in (1) is prohibitively expensive to compute or store in full.

Original: We therefore propose two different schemes to efﬁciently overcome this obstacle.
Revised: We therefore propose two different schemes to efﬁciently overcome this obstacle.

Original: We name the full methods SOSP-I (individual sensitivities) and SOSP-H (Hessian-vector product).
Revised: We name the full methods SOSP-I (individual sensitivities) and SOSP-H (Hessian-vector product).

Original: 2. SOSP-I: S ALIENCY FROM INDIVIDUAL SENSITIVITIES
Revised: 2. SOSP-I: S ALIENCY FROM INDIVIDUAL SENSITIVITIES

Original: SOSP-I approximates each individual term θ
Revised: SOSP-I approximates each individual term θ

Original: Ts H ( θ ) θ s (cid:48) in (1) efﬁciently, as we will show in Eq.
Revised: Ts H ( θ ) θ s (cid:48) in (1) efﬁciently, as we will show in Eq.

Original: We will therefore consider an upper bound to Eq.
Revised: We will therefore consider an upper bound to Eq.

Original: (1) which measures all sensitivities individually:
Revised: (1) which measures all sensitivities individually:

Original: [Equation]
Revised: [Equation]

Original: The absolute values are to prevent cancellations among the individual sensitivities of the network loss to the removal of structures, i.e. the derivatives λ 1 ( s ) , and individual correlations θ Ts H ( θ ) θ s (cid:48) .
Revised: The absolute values are to prevent cancellations among the individual sensitivities of the network loss to the removal of structures, i.e. the derivatives λ 1 ( s ) , and individual correlations θ Ts H ( θ ) θ s (cid:48) .

Original: While objectives other than λ
Revised: While objectives other than λ

Original: I 2 are equally possible in the method, including λ 2 and variants with the absolute values not pulled in all the way, we found λ
Revised: I 2 are equally possible in the method, including λ 2 and variants with the absolute values not pulled in all the way, we found λ

Original: I 2 to empirically perform best overall.
Revised: I 2 to empirically perform best overall.

Original: Then, SOSP-I iteratively selects the structures to prune, based on the objective (2): Starting from an empty pruning mask M = {} , we iteratively add to M the structure s / ∈ M that minimizes the overall sensitivity
Revised: Then, SOSP-I iteratively selects the structures to prune, based on the objective (2): Starting from an empty pruning mask M = {} , we iteratively add to M the structure s / ∈ M that minimizes the overall sensitivity

Original: λ
Revised: λ

Original: I 2 ( M ∪ { s } ) .
Revised: I 2 ( M ∪ { s } ) .

Original: In practice, the algorithm pre-computes the matrix Q ∈ R S × S ,
Revised: In practice, the algorithm pre-computes the matrix Q ∈ R S × S ,

Original: [Equation]
Revised: [Equation]

Original: and selects at each iteration a structure s / ∈ M to prune by
Revised: and selects at each iteration a structure s / ∈ M to prune by

Original: [Equation]
Revised: [Equation]

Original: terminating at the desired pruning ratio.
Revised: terminating at the desired pruning ratio.

Original: To compute Q efﬁciently, we show in App.
Revised: To compute Q efﬁciently, we show in App.

Original: B that the Hessian terms can be approximated as
Revised: B that the Hessian terms can be approximated as

Original: [Equation]
Revised: [Equation]

Original: Herein, the NN gradient φ ( x ) := ∇ θ f θ ( x ) ∈ R D
Revised: Herein, the NN gradient φ ( x ) := ∇ θ f θ ( x ) ∈ R D

Original: × P forms the basis of the well-established GaussNewton approximation (see App.
Revised: × P forms the basis of the well-established GaussNewton approximation (see App.

Original: B for more details) used for the Hessian, and the matrices R n ∈ R D
Revised: B for more details) used for the Hessian, and the matrices R n ∈ R D

Original: × D are diagonal plus a rank-1 contribution (App. B).
Revised: × D are diagonal plus a rank-1 contribution (App. B).

Original: For further efﬁciency gains, the sum runs over a random subsample of size N (cid:48) <
Revised: For further efﬁciency gains, the sum runs over a random subsample of size N (cid:48) <

Original: N .
Revised: N .

Original: In practice, one pre-computes all (sparse) products φ ( x n ) θ s ∈ R D starting from the efﬁciently computable gradient φ ( x n ) , before aggregating a batch onto the terms θT s H ( θ )
Revised: In practice, one pre-computes all (sparse) products φ ( x n ) θ s ∈ R D starting from the efﬁciently computable gradient φ ( x n ) , before aggregating a batch onto the terms θ

Original:  θ s (cid:48) .
Revised: Ts H ( θ ) θ s (cid:48) .

Original: Eq.
Revised: Eq.

Original: (5) also has an interpretation as output correlations between certain network modiﬁcations, without derivatives (App. C).
Revised: (5) also has an interpretation as output correlations between certain network modiﬁcations, without derivatives (App. C).

Original: 2. SOSP-H: S ALIENCY FROM H ESSIAN - VECTOR PRODUCT
Revised: 2. SOSP-H: S ALIENCY FROM H ESSIAN - VECTOR PRODUCT

Original: SOSP-H treats the second-order terms in (1) in a different way, motivated by the limit of large pruning ratios: At high pruning ratios, the sum (cid:80) s (cid:48) ∈
Revised: SOSP-H treats the second-order terms in (1) in a different way, motivated by the limit of large pruning ratios: At high pruning ratios, the sum (cid:80) s (cid:48) ∈

Original: M θ s (cid:48) in (1) can be approximated by (cid:80) Ss (cid:48)
Revised: M θ s (cid:48) in (1) can be approximated by (cid:80) Ss (cid:48)

Original: =1 θ s (cid:48)
Revised: =1 θ s (cid:48)

Original: =:
Revised: =:

Original: θ struc (this equals θ if every NN weight belongs to some structure s ).
Revised: θ struc (this equals θ if every NN weight belongs to some structure s ).

Original: The second-order term (cid:80) 
Revised: The second-order term (cid:80) s,s (cid:48)

Original: [Equation]
Revised: ∈ M θ

Original:  H [Equation] θ ) θ struc
Revised: Ts H ( θ ) θ s (cid:48)

Original: [Equation]
Revised: ≈ (cid:0) (cid:80) s ∈ M θ Ts (cid:1)(cid:0)

Original:  thus becomes tractable since the Hessian-vector product H ( θ ) θ struc is efﬁciently computable by a variant of the backpropagation algorithm.
Revised: H ( θ ) θ struc (cid:1) thus becomes tractable since the Hessian-vector product H ( θ ) θ struc is efﬁciently computable by a variant of the backpropagation algorithm.

Original: To account for each structure s and for the ﬁrst- and second-order contributions separately, as above, we place absolute value signs in (1) so as to arrive at the ﬁnal objective λ
Revised: To account for each structure s and for the ﬁrst- and second-order contributions separately, as above, we place absolute value signs in (1) so as to arrive at the ﬁnal objective λ

Original: H 2 ( M ) :
Revised: H 2 ( M ) :

Original: = (cid:80) s ∈ M λ H 2 ( s ) with
Revised: = (cid:80) s ∈ M λ H 2 ( s ) with

Original: [Equation]
Revised: [Equation]

Original: To minimize λ H 2 ( M ) , SOSP-H starts from an empty pruning mask M = {} and successively adds to
Revised: To minimize λ H 2 ( M ) , SOSP-H starts from an empty pruning mask M = {} and successively adds to

Original: M a structure s / ∈ M with smallest λ H 2 ( s ) , until the desired pruning ratio is reached.
Revised: M a structure s / ∈ M with smallest λ H 2 ( s ) , until the desired pruning ratio is reached.

Original: Unlike the Gauss-Newton approximation in SOSP-I, SOSP-H uses the exact Hessian H ( θ ) in the Hessian-vector product.
Revised: Unlike the Gauss-Newton approximation in SOSP-I, SOSP-H uses the exact Hessian H ( θ ) in the Hessian-vector product.

Original: But SOSP-H can therefore not account for individual absolute s - s (cid:48) - correlations, as some of those may cancel in the last term in Eq.
Revised: But SOSP-H can therefore not account for individual absolute s - s (cid:48) - correlations, as some of those may cancel in the last term in Eq.

Original: (6), contrary to (2).
Revised: (6), contrary to (2).

Original: Both SOSP-H and SOSP-I reduce to the same ﬁrst-order pruning method when neglecting the second order, i.e. setting H ( θ ) :
Revised: Both SOSP-H and SOSP-I reduce to the same ﬁrst-order pruning method when neglecting the second order, i.e. setting H ( θ ) :

Original: = 0 ; we compare to the resulting ﬁrst-order method in Sec.
Revised: = 0 ; we compare to the resulting ﬁrst-order method in Sec.

Original: 3.1 (see also App. A.1).
Revised: 3.1 (see also App. A.1).

Original: 2. C OMPUTATIONAL COMPLEXITY
Revised: 2. C OMPUTATIONAL COMPLEXITY

Original: We detail here the computational complexities of our methods (for the experimental evaluation see
Revised: We detail here the computational complexities of our methods (for the experimental evaluation see

Original: Sec. 3.2).
Revised: Sec. 3.2).

Original: The approximation of Q in (3) requires complexity O ( N (cid:48)
Revised: The approximation of Q in (3) requires complexity O ( N (cid:48)

Original: D ( F + P )) =
Revised: D ( F + P )) =

Original: O ( N (cid:48) DF ) for computing all φ ( x n ) θ s , where F ≥ P denotes the cost of one forward pass through the network ( F ≈ P for fully-connected NNs), plus O ( N (cid:48) DS 2 ) for the sum in (5).
Revised: O ( N (cid:48) DF ) for computing all φ ( x n ) θ s , where F ≥ P denotes the cost of one forward pass through the network ( F ≈ P for fully-connected NNs), plus O ( N (cid:48) DS 2 ) for the sum in (5).

Original: This is tractable for modern
Revised: This is tractable for modern

Original: NNs, while including the exact H ( θ ) would have complexity at least O ( N (cid:48) DSF ) .
Revised: NNs, while including the exact H ( θ ) would have complexity at least O ( N (cid:48) DSF ) .

Original: Once Q has been computed, the selection procedure based on (4) has overall complexity
Revised: Once Q has been computed, the selection procedure based on (4) has overall complexity

Original: O ( S 3 ) , which is feasible for most modern convolutional NNs (Sec. 3.2).
Revised: O ( S 3 ) , which is feasible for most modern convolutional NNs (Sec. 3.2).

Original: The total complexity of the SOSP-I method is thus
Revised: The total complexity of the SOSP-I method is thus

Original: [Equation]
Revised: [Equation]

Original: SOSP-H requires computational complexity O ( N (cid:48) DF ) both to compute all the ﬁrst-order as well as all the second-order terms in Eq.
Revised: SOSP-H requires computational complexity O ( N (cid:48) DF ) both to compute all the ﬁrst-order as well as all the second-order terms in Eq.

Original: Together with the sorting of the saliency values λ H 2 ( s ) , SOSP-H has thus the same low overall complexity as a ﬁrst-order method, namely
Revised: Together with the sorting of the saliency values λ H 2 ( s ) , SOSP-H has thus the same low overall complexity as a ﬁrst-order method, namely

Original: [Equation]
Revised: [Equation]

Original: Due to its weak dependency on S , in practice, SOSP-H efﬁciently scales to large modern networks (see Sec. 3.2), and may even be used for unstructured second-order pruning, where S = P .
Revised: Due to its weak dependency on S , in practice, SOSP-H efﬁciently scales to large modern networks (see Sec. 3.2), and may even be used for unstructured second-order pruning, where S = P .

Original: Both of our methods scale much better than naively including all off-diagonal Hessian terms, which is intractable for modern NNs due to its O ( N (cid:48) DSF ) scaling.
Revised: Both of our methods scale much better than naively including all off-diagonal Hessian terms, which is intractable for modern NNs due to its O ( N (cid:48) DSF ) scaling.

Original: Since SOSP-I builds on individual absolute sensitivities and on the established Gauss-Newton approximation, we use SOSP-I in the following in particular to validate our more efﬁcient main method SOSP-H.
Revised: Since SOSP-I builds on individual absolute sensitivities and on the established Gauss-Newton approximation, we use SOSP-I in the following in particular to validate our more efﬁcient main method SOSP-H.

Original:  R ESULTS
Revised:  R ESULTS

Original: To evaluate our methods, we train and prune VGGs (Simonyan & Zisserman, 2014),
Revised: To evaluate our methods, we train and prune VGGs (Simonyan & Zisserman, 2014),

Original: ResNets (He et al., 2016), PlainNet (He et al., 2016), and DenseNets (Huang et al., 2017) on the Cifar10/ (Krizhevsky et al., 2009) and ImageNet (Deng et al., 2009) datasets.
Revised: ResNets (He et al., 2016), PlainNet (He et al., 2016), and DenseNets (Huang et al., 2017) on the Cifar10/ (Krizhevsky et al., 2009) and ImageNet (Deng et al., 2009) datasets.

Original: Stochastic gradient descent with an initial learning rate of 0.1, a momentum of 0.9 and weight decay of 10 − 4 is used to train these networks.
Revised: Stochastic gradient descent with an initial learning rate of 0.1, a momentum of 0.9 and weight decay of 10 − 4 is used to train these networks.

Original: For ResNet-32/56 and VGG-Net on Cifar10/100, we use a batch size of 128, train forepochs and reduce the learning rate by a factor of 10 after 120 and 160 epochs.
Revised: For ResNet-32/56 and VGG-Net on Cifar10/100, we use a batch size of 128, train forepochs and reduce the learning rate by a factor of 10 after 120 and 160 epochs.

Original: For DenseNeton Cifar10/100, we train for 300 epochs and reduce the learning rate after 150 and 225 epochs.
Revised: For DenseNeton Cifar10/100, we train for 300 epochs and reduce the learning rate after 150 and 225 epochs.

Original: To ﬁne-tune the network after pruning, we exactly repeat this learning rate schedule.
Revised: To ﬁne-tune the network after pruning, we exactly repeat this learning rate schedule.

Original: For ResNets on
Revised: For ResNets on

Original: ImageNet, we use a batch size of 256, train for 128 epochs and use a cosine learning rate decay.
Revised: ImageNet, we use a batch size of 256, train for 128 epochs and use a cosine learning rate decay.

Original: For all networks, we prune feature maps (i.e. channels) from all layers except the last fully-connected layer; for ResNets, we also exclude the downsampling-path from pruning.
Revised: For all networks, we prune feature maps (i.e. channels) from all layers except the last fully-connected layer; for ResNets, we also exclude the downsampling-path from pruning.

Original: We approximate the
Revised: We approximate the

Original: Hessians by a subsample of size N (cid:48)
Revised: Hessians by a subsample of size N (cid:48)

Original: = 1000 (see Sec. 2.1).
Revised: = 1000 (see Sec. 2.1).

Original: We report the best or average ﬁnal test accuracy over 3 trials if not noted otherwise.
Revised: We report the best or average ﬁnal test accuracy over 3 trials if not noted otherwise.

Original: The experiments were run on an internal cluster with

Original: Nvidia Tesla V100 GPUs.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: 3. C OMPARISON ON MEDIUM - SIZED DATASETS
Revised: 3. C OMPARISON ON MEDIUM - SIZED DATASETS

Original: In this section, we compare the performance of our SOSP methods and benchmark them against existing pruning algorithms on various medium-sized datasets and networks.
Revised: In this section, we compare the performance of our SOSP methods and benchmark them against existing pruning algorithms on various medium-sized datasets and networks.

Original: First, we compare against other recent global pruning methods, then against pruning methods that learn layer-wise pruning ratios, and lastly against local structured pruning methods, i.e. with pre-speciﬁed layer-wise pruning ratios.
Revised: First, we compare against other recent global pruning methods, then against pruning methods that learn layer-wise pruning ratios, and lastly against local structured pruning methods, i.e. with pre-speciﬁed layer-wise pruning ratios.

Original: For all comparisons we report the achieved test accuracy, the number of parameters of the pruned network, and the MACs (often referred to as FLOPs).
Revised: For all comparisons we report the achieved test accuracy, the number of parameters of the pruned network, and the MACs (often referred to as FLOPs).

Original: To facilitate direct comparisons, we report test accuracies in the same way as the competing methods (e.g. best trial or average over trials), but additionally report mean and standard deviation of the test error for our models in App. A.
Revised: To facilitate direct comparisons, we report test accuracies in the same way as the competing methods (e.g. best trial or average over trials), but additionally report mean and standard deviation of the test error for our models in App. A.

Original: Our count of the network parameters and MACs is based on the actual pruned network architecture (cf.
Revised: Our count of the network parameters and MACs is based on the actual pruned network architecture (cf.

Original: App.
Revised: App.

Original: D), even though our saliency measure associates with each structure only the weights into this structure (see Sec. 2).
Revised: D), even though our saliency measure associates with each structure only the weights into this structure (see Sec. 2).

Original: Before comparing our SOSP methods to other recently published pruning algorithms, we perform ablation studies to motivate the use of our second-order objective which takes correlations into account.
Revised: Before comparing our SOSP methods to other recently published pruning algorithms, we perform ablation studies to motivate the use of our second-order objective which takes correlations into account.

Original: First, we compare the pruning accuracy of a ﬁrst-order method (dropping all second-order terms in either Eq.
Revised: First, we compare the pruning accuracy of a ﬁrst-order method (dropping all second-order terms in either Eq.

Original: (3) or (6)) to our second-order SOSP objectives.
Revised: (3) or (6)) to our second-order SOSP objectives.

Original: The results (see App. A.1) clearly indicate, that second-order information improves the pruning performance.
Revised: The results (see App. A.1) clearly indicate, that second-order information improves the pruning performance.

Original: Second, we investigate whether the off-diagonal elements of the second-order terms improve the ﬁnal accuracies.
Revised: Second, we investigate whether the off-diagonal elements of the second-order terms improve the ﬁnal accuracies.

Original: We perform this comparison using a variant of SOSP-I, where the off-diagonal terms (cid:80) s (cid:48)
Revised: We perform this comparison using a variant of SOSP-I, where the off-diagonal terms (cid:80) s (cid:48)

Original: Q s,s (cid:48) in Eq.
Revised: Q s,s (cid:48) in Eq.

Original: (4) are dropped.
Revised: (4) are dropped.

Original: The results (see App. A.2) again suggest that adding these off-diagonal terms leads to a consistent improvement in ﬁnal accuracy.
Revised: The results (see App. A.2) again suggest that adding these off-diagonal terms leads to a consistent improvement in ﬁnal accuracy.

Original: In summary, we conclude that second-order pruning objectives that consider global correlations perform best.
Revised: In summary, we conclude that second-order pruning objectives that consider global correlations perform best.

Original: We now ﬁrst compare our SOSP methods to global pruning methods on VGG-Net, ResNet-32 and
Revised: We now ﬁrst compare our SOSP methods to global pruning methods on VGG-Net, ResNet-32 and

Original: DenseNet-40.
Revised: DenseNet-40.

Original: We use the same variants and implementations of these networks as used by Neural
Revised: We use the same variants and implementations of these networks as used by Neural

Original: Network Slimming (NN Slimming; Liu et al., 2017) as well as EigenDamage and C-OBD (Wang et al., 2019a), e.g. capping the layer-wise ratio of removed structures at 95% for VGGs to prevent layer collapse and increasing the width of ResNet-32 by a factor 4.
Revised: Network Slimming (NN Slimming; Liu et al., 2017) as well as EigenDamage and C-OBD (Wang et al., 2019a), e.g. capping the layer-wise ratio of removed structures at 95% for VGGs to prevent layer collapse and increasing the width of ResNet-32 by a factor 4.

Original: C-OBD is a structured variant of the original OBD algorithm (Hassibi et al., 1993), which neglects all cross-structure correlations that, in contrast, SOSP takes into account.
Revised: C-OBD is a structured variant of the original OBD algorithm (Hassibi et al., 1993), which neglects all cross-structure correlations that, in contrast, SOSP takes into account.

Original: The results over three trials for high pruning ratios are shown in
Revised: The results over three trials for high pruning ratios are shown in

Original: Tab. 1 and for moderate pruning ratios in App. A.3.
Revised: Tab. 1 and for moderate pruning ratios in App. A.3.

Original: To enable the comparison to NN
Revised: To enable the comparison to NN

Original: Slimming on an already pretrained VGG, we included the results of NN Slimming applied to a baseline network obtained without modiﬁcations to its initial training, i.e. without L 1 -regularization on the batchnormalization parameters.
Revised: Slimming on an already pretrained VGG, we included the results of NN Slimming applied to a baseline network obtained without modiﬁcations to its initial training, i.e. without L 1 -regularization on the batchnormalization parameters.

Original: For moderate pruning ratios, all pruning schemes approximately retain the baseline performance for VGG-Net and ResNet-32 on Cifar10 and VGG-Net on Cifar100 (see Tab.
Revised: For moderate pruning ratios, all pruning schemes approximately retain the baseline performance for VGG-Net and ResNet-32 on Cifar10 and VGG-Net on Cifar100 (see Tab.

Original: The only exception is the accuracy for C-OBD applied to VGG-Net on Cifar100, which drops by approximately 1% .
Revised: The only exception is the accuracy for C-OBD applied to VGG-Net on Cifar100, which drops by approximately 1% .

Original: For ResNet-32 on Cifar100 the accuracy after pruning is approximately 1% lower than the baseline, for all pruning schemes.
Revised: For ResNet-32 on Cifar100 the accuracy after pruning is approximately 1% lower than the baseline, for all pruning schemes.

Original: In the regime of larger pruning ratios of approximately 97% ,
Revised: In the regime of larger pruning ratios of approximately 97% ,

Original: SOSP and EigenDamage signiﬁcantly outperform NN Slimming and C-OBD.
Revised: SOSP and EigenDamage signiﬁcantly outperform NN Slimming and C-OBD.

Original: SOSP performs on par with EigenDamage, except for ResNet-32 on Cifar100, where SOSP outperforms EigenDamage by almost 3% .
Revised: SOSP performs on par with EigenDamage, except for ResNet-32 on Cifar100, where SOSP outperforms EigenDamage by almost 3% .

Original: For DenseNet-40, we achieve similar results compared to NN Slimming.
Revised: For DenseNet-40, we achieve similar results compared to NN Slimming.

Original: However, note that NN Slimming depends on the above L 1 -modiﬁcation of network pretraining.
Revised: However, note that NN Slimming depends on the above L 1 -modiﬁcation of network pretraining.

Original: Next, we compare SOSP-H to AMC (He et al., 2018) which, instead of ranking the ﬁlters globally, learns layer-wise pruning ratios and then ranks ﬁlters within each layer.
Revised: Next, we compare SOSP-H to AMC (He et al., 2018) which, instead of ranking the ﬁlters globally, learns layer-wise pruning ratios and then ranks ﬁlters within each layer.

Original: For a fair comparison, we extract their learned layer-wise pruning ratios for PlainNet-20 (He et al., 2016) and rank the ﬁlters in each layer according to their 2-norm (see also Han et al.
Revised: For a fair comparison, we extract their learned layer-wise pruning ratios for PlainNet-20 (He et al., 2016) and rank the ﬁlters in each layer according to their 2-norm (see also Han et al.

Original: The results in App.
Revised: The results in App.

Original: A.4 show that
Revised: A.5 show that

Original: SOSP performs better than AMC when compared over parameters, the main objective of SOSP, and on par with AMC when compared over MACs, the main objective of AMC.
Revised: SOSP performs better than AMC when compared over parameters, the main objective of SOSP, and on par with AMC when compared over MACs, the main objective of AMC.

Original: Lastly, we compare our SOSP methods against four recently published local, i.e. layer-wise, pruning algorithms: FPGM He et al.
Revised: Lastly, we compare our SOSP methods against four recently published local, i.e. layer-wise, pruning algorithms: FPGM He et al.

Original: , GAL (Lin et al., 2019), CCP (Peng et al., 2019) (see Fig. 7),
Revised: , GAL (Lin et al., 2019), CCP (Peng et al., 2019) (see Fig. 7),

Original: Variational Pruning (VP; Zhao et al., 2019) and HRank (Lin et al., 2020).
Revised: Variational Pruning (VP; Zhao et al., 2019) and HRank (Lin et al., 2020).

Original: For ResNet-56, our SOSP methods outperform all other methods across all pruning ratios (see Fig.
Revised: For ResNet-56, our SOSP methods outperform all other methods across all pruning ratios (see Fig.

Original: 1a and b).
Revised: 1a and b).

Original: For DenseNet-40,
Revised: For DenseNet-40,

Original: SOSP achieves better accuracies when compared over parameters (Fig. 1c) and is on par with the best other methods over MACs (Fig. 1d).
Revised: SOSP achieves better accuracies when compared over parameters (Fig. 1c) and is on par with the best other methods over MACs (Fig. 1d).

Original: The reason for this discrepancy is probably that the SOSP objective is agnostic to the number of MACs (image size) in each individual layer.
Revised: The reason for this discrepancy is probably that the SOSP objective is agnostic to the number of MACs (image size) in each individual layer.

Original: 3. S CALABILITY AND APPLICATION TO LARGE - SCALE DATASETS
Revised: 3. S CALABILITY AND APPLICATION TO LARGE - SCALE DATASETS

Original: Before going to large datasets, we compare the scalability of our methods SOSP-I and SOSP-H.
Revised: Before going to large datasets, we compare the scalability of our methods SOSP-I and SOSP-H.

Original: As the preceding section shows, both methods perform basically on par with each other in terms of accuracy.
Revised: As the preceding section shows, both methods perform basically on par with each other in terms of accuracy.

Original: This conﬁrms that SOSP-H is not degraded by the approximations leading to the efﬁcient Hessian-vector product, or is helped by use of the exact Hessian.
Revised: This conﬁrms that SOSP-H is not degraded by the approximations leading to the efﬁcient Hessian-vector product, or is helped by use of the exact Hessian.

Original: In terms of efﬁciency, however, SOSP-H shows clear advantages compared to SOSP-I, for which the algorithm to select the structures to be pruned scales with O ( S 3 ) (see Sec. 2.3), potentially dominating the overall runtime for large-scale networks.
Revised: In terms of efﬁciency, however, SOSP-H shows clear advantages compared to SOSP-I, for which the algorithm to select the structures to be pruned scales with O ( S 3 ) (see Sec. 2.3), potentially dominating the overall runtime for large-scale networks.

Original: Measurements of the actual runtimes show that already for medium-sized networks SOSP-H is more efﬁcient than SOSP-I, see Fig.
Revised: Measurements of the actual runtimes show that already for medium-sized networks SOSP-H is more efﬁcient than SOSP-I, see Fig.

Original: 2 and App.
Revised: 2 and App.

Original: A.5.
Revised: A.6.

Original: Since SOSP-I becomes impractical for large-scale networks like other second-order methods (Molchanov et al., 2019; Wang et al., 2019a), for the ImageNet dataset we will only evaluate SOSP-H and refer to it as SOSP.
Revised: Since SOSP-I becomes impractical for large-scale networks like other second-order methods (Molchanov et al., 2019; Wang et al., 2019a), for the ImageNet dataset we will only evaluate SOSP-H and refer to it as SOSP.

Original: On ImageNet, we compare our results to the literature for ResNet-18 and ResNet-50, see Tab.or Suppl.
Revised: On ImageNet, we compare our results to the literature for ResNet-18 and ResNet-50, see Tab.or Suppl.

Original: Fig.
Revised: Fig.

Original: Because SOSP assesses the sensitivity of each structure independently of the contributed MACs, it has a bias towards pruning small-scale structures.
Revised: Because SOSP assesses the sensitivity of each structure independently of the contributed MACs, it has a bias towards pruning small-scale structures.

Original: This tendency is strongest for ResNet-50, due to its 1 × 1 -convolutions.
Revised: This tendency is strongest for ResNet-50, due to its 1 × 1 -convolutions.

Original: Since these 1 × 1 -convolutions tend to contribute disproportionately little to the overallnumber of MACs, we devised a scaled variant of SOSP, which 
Revised: Since these 1 × 1 -convolutions tend to contribute disproportionately little to the overall of MACs, we devised a scaled variant of SOSP, which divides

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: divides the saliency of every structure by its kernel size (e
Revised: the saliency of every structure by its kernel size (e.g. 1, 3, or 7).

Revised: Compared to the vanilla SOSP

Original: [Table]
Revised: [Table]

Original: g. 1, 3, or 7). Compared to the vanilla SOSP, the scaled variant of SOSP is able to remove larger percentages of MACs with similar drops in accuracy (see Tab. 2).
Revised: the scaled variant of SOSP is able to remove larger percentages of MACs with similar drops in accuracy (see Tab. 2).We further show results on MobieleNetV2 for Imagenet in the appendix (see App. A.5).

Original: For both networks SOSP outperforms FPGM and HRank, especially when considering the main objective of SOSP, which is to reduce the number of parameters or structures.
Revised: For both networks SOSP outperforms FPGM and HRank, especially when considering the main objective of SOSP, which is to reduce the number of parameters or structures.

Original: Since CCP uses a different way of calculating the MACs, which leads to consistently higher pruning ratios, we added an alternative MAC count to enable a fair comparison (for details, see App. D).
Revised: Since CCP uses a different way of calculating the MACs, which leads to consistently higher pruning ratios, we added an alternative MAC count to enable a fair comparison (for details, see App. D).

Original: Since HRank and
Revised: Since HRank and

Original: FPGM do not mention their MAC counting convention, we assume they use the same convention as we do.
Revised: FPGM do not mention their MAC counting convention, we assume they use the same convention as we do.

Original: Taking this into account, our scaled SOSP variant is able to prune more MACs than
Revised: Taking this into account, our scaled SOSP variant is able to prune more MACs than

Original: CCP, while having the same ﬁnal accuracy.
Revised: CCP, while having the same ﬁnal accuracy.

Original: Further, we compare SOSP to Taylor (Molchanov et al., 2019), a global pruning method that uses only the ﬁrst-order terms for its saliency ranking.
Revised: Further, we compare SOSP to Taylor (Molchanov et al., 2019), a global pruning method that uses only the ﬁrst-order terms for its saliency ranking.

Original: SOSP clearly outperforms Taylor, even though both methods have the same scaling.
Revised: SOSP clearly outperforms Taylor, even though both methods have the same scaling.

Original: This is also in line with the results of our ablation study (see Sec.
Revised: This is also in line with the results of our ablation study (see Sec.

Original: 3.1 and esp.
Revised: 3.1 and esp.

Original: App. A.1), where we show that including second-order information leads to improved accuracies.
Revised: App. A.1), where we show that including second-order information leads to improved accuracies.

Original: 3. I DENTIFYING & W IDENING A RCHITECTURAL B OTTLENECKS
Revised: 3. I DENTIFYING & W IDENING A RCHITECTURAL B OTTLENECKS

Original: In the previous sections we evaluated the SOSP algorithm by removing those structures considered as least signiﬁcant and then assessed the algorithm based on the performance of the pruned model.
Revised: In the previous sections we evaluated the SOSP algorithm by removing those structures considered as least signiﬁcant and then assessed the algorithm based on the performance of the pruned model.

Original: Here we instead follow an alternative path to validate the importance ranking introduced by SOSP, namely, we increase the size of these NN layers ranked as most signiﬁcant.
Revised: Here we instead follow an alternative path to validate the importance ranking introduced by SOSP, namely, we increase the size of these NN layers ranked as most signiﬁcant.

Original: The idea for this comes from a common feature of the pruning masks found by SOSP for VGG and ResNet-56 (see Suppl.
Revised: The idea for this comes from a common feature of the pruning masks found by SOSP for VGG and ResNet-56 (see Suppl.

Original: Fig.
Revised: Fig.

Original: 8(d,g,e,h) and Fig.
Revised: 8(d,g,e,h) and Fig.

Original: 3 (c,d,f,g)), namely that some layers are barely pruned while others are pruned by more than 80%.
Revised: 3 (c,d,f,g)), namely that some layers are barely pruned while others are pruned by more than 80%.

Original: This could indicate towards architectural bottlenecks; widening these could further improve the performance of the model.
Revised: This could indicate towards architectural bottlenecks; widening these could further improve the performance of the model.

Original: We consider a layer an architectural bottleneck if it has a considerably lower pruning ratio compared to the other layers.
Revised: We consider a layer an architectural bottleneck if it has a considerably lower pruning ratio compared to the other layers.

Original: Thus, widening these layers may improve the overall performance, and subsequent pruning could allow for even smaller models with higher accuracies compared to pruning the original model.
Revised: Thus, widening these layers may improve the overall performance, and subsequent pruning could allow for even smaller models with higher accuracies compared to pruning the original model.

Original: To utilize this insight we devise a procedure that we call expand-pruning .
Revised: To utilize this insight we devise a procedure that we call expand-pruning .

Original: The idea is to ﬁrst calculate the pruning ratios of the trained network and then to identify architectural bottlenecks, i.e. the layers with the lowest pruning ratios.
Revised: The idea is to ﬁrst calculate the pruning ratios of the trained network and then to identify architectural bottlenecks, i.e. the layers with the lowest pruning ratios.

Original: Next, we widen these speciﬁc layers by a factor of two, which has been proven to work well empirically.
Revised: Next, we widen these speciﬁc layers by a factor of two, which has been proven to work well empirically.

Original: Finally, we randomly initialize, train, prune, and ﬁne-tune the expanded network (for a schematic, see Fig. 3a).
Revised: Finally, we randomly initialize, train, prune, and ﬁne-tune the expanded network (for a schematic, see Fig. 3a).

Original: As a naive baseline that we call the widen-pruning procedure, we widen every layer in the network with a constant factor, instead of widening speciﬁc layers; this constant factor is chosen such that the overall number of parameters matches that of the expand-prune procedure (e.g., leading to the width multiplier 1 . 1 in Fig. 3).
Revised: As a naive baseline that we call the widen-pruning procedure, we widen every layer in the network with a constant factor, instead of widening speciﬁc layers; this constant factor is chosen such that the overall number of parameters matches that of the expand-prune procedure (e.g., leading to the width multiplier 1 . 1 in Fig. 3).

Original: [Figure]
Revised: [Figure]

Original: expand
Revised: expand

Original: [Figure]
Revised: [Figure]

Revised: pruning

Original: [Figure]
Revised: [Figure]

Original: t
Revised: pruning [Figure]

Original: [Figure]
Revised: [Figure]

Original: e
Revised: e

Original: [Figure]
Revised: [Figure]

Original: w i d t h m u l t i p

Original: [Figure]
Revised: [Figure]

Original: e
Revised: i d t

Original: [Figure]
Revised: [Figure]

Revised: m u l t i p

Original: [Figure]
Revised: [Figure]

Original: We evaluate the expand-pruning procedure for ResNet-56 and VGG on Cifar10, for whichwe expand the least pruned of the three main building blocks and the ﬁve least pruned layers, respectively (cf. red width multipliers in Fig.
Revised: We evaluate the expand-pruning procedure for -56 and VGG on Cifar10, for which expand the least pruned of the three main building blocks and the ﬁve least pruned layers, respectively ( red width multipliers in Fig.

Original: 3(c,f) selected on the basis of the pruning masks shown in Suppl.
Revised: 3(c,f) selected on the basis of the pruning masks shown in Suppl.

Original: Fig.
Revised: Fig.

Original: 8(d,g)).
Revised: 8(d,g)).

Original: Note that a more ﬁne-grained widening of bottlenecks in ResNet-56, e.g. on layer level, is not possible without changing the overall ResNet architecture.
Revised: Note that a more ﬁne-grained widening of bottlenecks in ResNet-56, e.g. on layer level, is not possible without changing the overall ResNet architecture.

Original: In summary, a selective widening of bottlenecks results in smaller network models overall with higher accuracy than pruning the vanilla network or unselectively increasing the network size (compare expand-pruning and widen-pruning in Fig.
Revised: In summary, a selective widening of bottlenecks results in smaller network models overall with higher accuracy than pruning the vanilla network or unselectively increasing the network size (compare expand-pruning and widen-pruning in Fig.

Original: 3(b,e)).
Revised: 3(b,e)).

Original: While in principle any global pruning method could be used for the expand-pruning procedure, SOSP is especially suited since it does not require to modify the network architecture like
Revised: While in principle any global pruning method could be used for the expand-pruning procedure, SOSP is especially suited since it does not require to modify the network architecture like

Original: EigenDamage.
Revised: EigenDamage.

Original: Further, SOSP can be applied directly at initialization (see App. A.9), unlike e.g.NN
Revised: Further, SOSP can be applied directly at initialization (see App. A.10), unlike e.g.

Original:  Slimming, allowing for a similar expand scheme directly at initialization (see App. A.10).
Revised: NN Slimming, allowing for a similar expand scheme directly at initialization (see App. A.11).

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: In this work we have demonstrated the effectiveness and scalability of our second-order structured pruning algorithms (SOSP).
Revised: In this work we have demonstrated the effectiveness and scalability of our second-order structured pruning algorithms (SOSP).

Original: While both algorithms perform similarly well, SOSP-H is more easily scalable to large-scale networks and datasets.
Revised: While both algorithms perform similarly well, SOSP-H is more easily scalable to large-scale networks and datasets.

Original: Therefore, we generally recommend to use SOSP-H over SOSP-I, especially for large-scale datasets and networks.
Revised: Therefore, we generally recommend to use SOSP-H over SOSP-I, especially for large-scale datasets and networks.

Original: The only regime where SOSP-I becomes a valuable alternative is for small pruning ratios and small- to medium-sized networks.
Revised: The only regime where SOSP-I becomes a valuable alternative is for small pruning ratios and small- to medium-sized networks.

Original: Further, we showed that the pruning masks found by SOSP can be used to systematically detect and widen architectural bottlenecks, further improving the performance of pruned networks.
Revised: Further, we showed that the pruning masks found by SOSP can be used to systematically detect and widen architectural bottlenecks, further improving the performance of pruned networks.

Original: Compared to other global pruning methods, SOSP captures correlations between structures by a simple, effective and scalable algorithm that requires to modify neither the training nor the architecture of the network model to be pruned.
Revised: Compared to other global pruning methods, SOSP captures correlations between structures by a simple, effective and scalable algorithm that requires to modify neither the training nor the architecture of the network model to be pruned.

Original: SOSP achieves comparable or better accuracies on benchmark datasets, while having the same low complexity as ﬁrst-order methods.
Revised: SOSP achieves comparable or better accuracies on benchmark datasets, while having the same low complexity as ﬁrst-order methods.

Original: The C-OBD algorithm (Wang et al., 2019a) is a structured generalization of the original unstructured OBD algorithm (LeCun et al., 1990).
Revised: The C-OBD algorithm (Wang et al., 2019a) is a structured generalization of the original unstructured OBD algorithm (LeCun et al., 1990).

Original: In contrast to OBD, C-OBD accounts for correlations within each structure, but does not capture correlations between different structures within and across layers.
Revised: In contrast to OBD, C-OBD accounts for correlations within each structure, but does not capture correlations between different structures within and across layers.

Original: We show that considering these global correlations consistently improves the performance, especially for large pruning ratios (Tab. 1).
Revised: We show that considering these global correlations consistently improves the performance, especially for large pruning ratios (Tab. 1).

Original: This is in line with the results of our ablation study in App.
Revised: This is in line with the results of our ablation study in App.

Original: A.2 (see also Sec. 3.1).
Revised: A.2 (see also Sec. 3.1).

Original: The objective of EigenDamage (Wang et al., 2019a) to include second-order correlations is similar to ours, but the approaches are signiﬁcantly different: EigenDamage uses the Fisher-approximation, which is similar to SOSP-I’s Gauss-Newton approximation and exhibits a similar scaling behaviour, and then, in addition to further approximations, applies low-rank approximations that require the substitution of each layer by a bottleneck-block structure.
Revised: The objective of EigenDamage (Wang et al., 2019a) to include second-order correlations is similar to ours, but the approaches are signiﬁcantly different: EigenDamage uses the Fisher-approximation, which is similar to SOSP-I’s Gauss-Newton approximation and exhibits a similar scaling behaviour, and then, in addition to further approximations, applies low-rank approximations that require the substitution of each layer by a bottleneck-block structure.

Original: Our SOSP method is simpler, easier to implement and does not require to modify the network architecture, but nevertheless performs on par with EigenDamage.
Revised: Our SOSP method is simpler, easier to implement and does not require to modify the network architecture, but nevertheless performs on par with EigenDamage.

Original: The approach of NN Slimming (Liu et al., 2017) is more heuristic than
Revised: The approach of NN Slimming (Liu et al., 2017) is more heuristic than

Original: SOSP and is easy to implement.
Revised: SOSP and is easy to implement.

Original: However, networks need to be pretrained with L 1 -regularization on the batch-normalization parameters, otherwise the performance is severely harmed (Tab. 1 and
Revised: However, networks need to be pretrained with L 1 -regularization on the batch-normalization parameters, otherwise the performance is severely harmed (Tab. 1 and

Original: Supp.
Revised: Supp.

Original: Tab. 4).
Revised: Tab. 4).

Original: SOSP on the other hand does not require any modiﬁcations to the network training and thus can be applied to any network.
Revised: SOSP on the other hand does not require any modiﬁcations to the network training and thus can be applied to any network.

Original: A recent variant of NN Slimming was developed by Zhuang et al.
Revised: A recent variant of NN Slimming was developed by Zhuang et al.

Original: (2020) who optimize their hyperparameters to reduce the number of MACs.
Revised: (2020) who optimize their hyperparameters to reduce the number of MACs.

Original: Using the number
Revised: Using the number

Original: MACs as an objective for SOSP is left for future studies.
Revised: MACs as an objective for SOSP is left for future studies.

Original: Most other existing global second-order structured pruning methods either approximate the Hessian as a diagonal matrix (Theis et al., 2018;
Revised: Most other existing global second-order structured pruning methods either approximate the Hessian as a diagonal matrix (Theis et al., 2018;

Original: Liu et al., 2021) or use an approximation similar to Gauss-Newton, which we used for SOSP-I (e.g., the Fisher information matrix in Molchanov et al., 2019).
Revised: Liu et al., 2021) or use an approximation similar to Gauss-Newton, which we used for SOSP-I (e.g., the Fisher information matrix in Molchanov et al., 2019).

Original: These approaches based on the
Revised: These approaches based on the

Original: Gauss-Newton approximation or its variants usually share the same disadvantageous scaling as that encountered by SOSP-I, and thus cannot scale to ImageNet as SOSP-H can.
Revised: Gauss-Newton approximation or its variants usually share the same disadvantageous scaling as that encountered by SOSP-I, and thus cannot scale to ImageNet as SOSP-H can.

Original: In addition to the above comparison to other global pruning methods, we also compared our methods to simpler local pruning methods that keep the pruning ratios constant for each layer and, consequently, scale well to large-scale datasets.
Revised: In addition to the above comparison to other global pruning methods, we also compared our methods to simpler local pruning methods that keep the pruning ratios constant for each layer and, consequently, scale well to large-scale datasets.

Original: The pruning method closest to our SOSP-I method is the one by Peng et al.
Revised: The pruning method closest to our SOSP-I method is the one by Peng et al.

Original: While both works consider second-order correlations between structures, theirs is based on a ranking scheme different from our absolute sensitivities in λ I 2 and considers only intra-layer correlations.
Revised: While both works consider second-order correlations between structures, theirs is based on a ranking scheme different from our absolute sensitivities in λ I 2 and considers only intra-layer correlations.

Original: Furthermore, they employ an auxiliary classiﬁer with a hyperparameter, yielding accuracy improvements that are difﬁcult to disentangle from the effect of second-order pruning.
Revised: Furthermore, they employ an auxiliary classiﬁer with a hyperparameter, yielding accuracy improvements that are difﬁcult to disentangle from the effect of second-order pruning.

Original: Going beyond a constant pruning ratio for each layer, Su et al.
Revised: Going beyond a constant pruning ratio for each layer, Su et al.

Original: (2020) discovered, for
Revised: (2020) discovered, for

Original: ResNets, that pruning at initialization seems to preferentially prune initial layers and thus proposed a pruning scheme based on a “keep-ratio” per layer which increases with the depth of the network.
Revised: ResNets, that pruning at initialization seems to preferentially prune initial layers and thus proposed a pruning scheme based on a “keep-ratio” per layer which increases with the depth of the network.

Original: Our experiments conﬁrm some of the ﬁndings of Su et al.
Revised: Our experiments conﬁrm some of the ﬁndings of Su et al.

Original: but we also show that the speciﬁc network architectures found by pruning can drastically vary between different networks and especially between initialization and after training (histograms in Suppl.
Revised: but we also show that the speciﬁc network architectures found by pruning can drastically vary between different networks and especially between initialization and after training (histograms in Suppl.

Original: Fig.
Revised: Fig.

Original: While all local pruning methods specify pruning ratios for each layer, our method performs automatic selection across layers.
Revised: While all local pruning methods specify pruning ratios for each layer, our method performs automatic selection across layers.

Original: Further,
Revised: Further,

Original: SOSP performs on par or better than AMC (He et al., 2018) which learns these layer-wise pruning ratios as a ﬁrst step, instead of specifying them in advance like the above local methods.
Revised: SOSP performs on par or better than AMC (He et al., 2018) which learns these layer-wise pruning ratios as a ﬁrst step, instead of specifying them in advance like the above local methods.

Original: This is surprising since AMC uses many iterations to learn the layer-wise pruning ratios, while SOSP uses only a single step.
Revised: This is surprising since AMC uses many iterations to learn the layer-wise pruning ratios, while SOSP uses only a single step.

Original: Further, there exist even more elaborate schemes to ﬁnd these subnetworks, in the spirit of AMC, such as Autocompress (Liu et al., 2020), EagleEye (Li et al., 2020), DMCP (Guo et al., 2020) and CafeNet (Su et al., 2021).
Revised: Further, there exist even more elaborate schemes to ﬁnd these subnetworks, in the spirit of AMC, such as Autocompress (Liu et al., 2020), EagleEye (Li et al., 2020), DMCP (Guo et al., 2020) and CafeNet (Su et al., 2021).

Original: It is difﬁcult to enable a fair quantitative comparison between these iterative methods and our single-shot method SOSP.
Revised: It is difﬁcult to enable a fair quantitative comparison between these iterative methods and our single-shot method SOSP.

Original: SOSP’s automatic selection of important structures allows us to identify and widen architectural bottlenecks.
Revised: SOSP’s automatic selection of important structures allows us to identify and widen architectural bottlenecks.

Original: However, as the size of structures is usually identical within layers, our global pruning method has a bias towards pruning small structures, absent from local pruning methods.
Revised: However, as the size of structures is usually identical within layers, our global pruning method has a bias towards pruning small structures, absent from local pruning methods.

Original: We proposed a simple solution to remove some of this bias by scaling each structure by the inverse of the kernel size.
Revised: We proposed a simple solution to remove some of this bias by scaling each structure by the inverse of the kernel size.

Original: Alternatively, to better reﬂect the computational costs in real-world applications, each structure could also be normalized by the number of its required MACs or memory (van Amersfoort et al., 2020; Liu et al., 2021).
Revised: Alternatively, to better reﬂect the computational costs in real-world applications, each structure could also be normalized by the number of its required MACs or memory (van Amersfoort et al., 2020; Liu et al., 2021).

Original: Further, introducing additional hyperparameters for each layer optimized to reduce the number of MACs as done by Chin et al.
Revised: Further, introducing additional hyperparameters for each layer optimized to reduce the number of MACs as done by Chin et al.

Original: (2020) could potentially lead to further improvements, but would also signiﬁcantly increase the complexity due to the need of hyperparameter optimization.
Revised: (2020) could potentially lead to further improvements, but would also signiﬁcantly increase the complexity due to the need of hyperparameter optimization.

Original: Liu et al.
Revised: Liu et al.

Original: (2021) form groups of network structures that depend on each other, and compute the saliency for each group collectively, to better estimate the importance.
Revised: (2021) form groups of network structures that depend on each other, and compute the saliency for each group collectively, to better estimate the importance.

Original: While this proved to be beneﬁcial, it is orthogonal to the investigation of ﬁnding good second-order approximations.
Revised: While this proved to be beneﬁcial, it is orthogonal to the investigation of ﬁnding good second-order approximations.

Original: Recently, unstructured (Lee et al., 2018; Wang et al., 2019b; Tanaka et al., 2020) and structured (van Amersfoort et al., 2020;
Revised: Recently, unstructured (Lee et al., 2018; Wang et al., 2019b; Tanaka et al., 2020) and structured (van Amersfoort et al., 2020;

Original: Hayou et al., 2021) pruning schemes were proposed that are applicable to networks at initialization.
Revised: Hayou et al., 2021) pruning schemes were proposed that are applicable to networks at initialization.

Original: While these methods fail to achieve similar accuracies compared to pruning after training, we show in a small ablation study (see App. A.8) that our SOSP method at initialization can, after comparable training time, achieve accuracies similar to pruning after training.
Revised: While these methods fail to achieve similar accuracies compared to pruning after training, we show in a small ablation study (see App. A.9) that our SOSP method at initialization can, after comparable training time, achieve accuracies similar to pruning after training.

Original: In accordance with Elsken et al.
Revised: In accordance with Elsken et al.

Original: , our results suggest that pruning can be used to optimize the architectural hyperparameters of established networks (Liu et al., 2018) or super-graphs (Noy et al., 2020); see also App. A.11.
Revised: , our results suggest that pruning can be used to optimize the architectural hyperparameters of established networks (Liu et al., 2018) or super-graphs (Noy et al., 2020); see also App. A.12.

Original: We envision our second-order sensitivity analysis to be a valuable tool to identify and widen bottlenecks.
Revised: We envision our second-order sensitivity analysis to be a valuable tool to identify and widen bottlenecks.

Original: For example, whenever a building block of the neural network cannot be compressed, this building block may be considered a bottleneck of the architecture and could be inﬂated to improve the overall trade-off between accuracy and computational cost.
Revised: For example, whenever a building block of the neural network cannot be compressed, this building block may be considered a bottleneck of the architecture and could be inﬂated to improve the overall trade-off between accuracy and computational cost.

Original: Supplementary Material for
Revised: Supplementary Material for

Original: SOSP: Efﬁciently Capturing Global Correlations by Second-Order Structured Pruning
Revised: SOSP: Efﬁciently Capturing Global Correlations by Second-Order Structured Pruning

Original: A A DDITIONAL D ATA AND E XPERIMENTS A. C OMPARISON OF F IRST -O RDER S TRUCTURED P RUNING WITH SOSP
Revised: A A DDITIONAL D ATA AND E XPERIMENTS A. C OMPARISON OF F IRST -O RDER S TRUCTURED P RUNING WITH SOSP

Original: See Figure 4 and Table 3 and their captions (cf.
Revised: See Figure 4 and Table 3 and their captions (cf.

Original: also Sec.
Revised: also Sec.

Original: 3.1 for further descriptions).
Revised: 3.1 for further descriptions).

Original: The experiments are in line with our comparison to Taylor (Molchanov et al., 2019) on ImageNet, which is basically a ﬁrst-order method (see Sec. 3.2).
Revised: The experiments are in line with our comparison to Taylor (Molchanov et al., 2019) on ImageNet, which is basically a ﬁrst-order method (see Sec. 3.2).

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: A. C OMPARISON OF A CCURACIES A CHIEVED BY SOSP-I WITH AND WITHOUT C ROSS -S TRUCTURE C ORRELATIONS
Revised: A. C OMPARISON OF A CCURACIES A CHIEVED BY SOSP-I WITH AND WITHOUT C ROSS -S TRUCTURE C ORRELATIONS

Original: See Figure 5 and its caption (cf.
Revised: See Figure 5 and its caption (cf.

Original: also Sec.
Revised: also Sec.

Original: 3.1 for further descriptions).
Revised: 3.1 for further descriptions).

Original: A. R ESULTS FOR M EDIUM P RUNING R ATES FOR C OMPARING G LOBAL P RUNING M ETHODS
Revised: A. R ESULTS FOR M EDIUM P RUNING R ATES FOR C OMPARING G LOBAL P RUNING M ETHODS

Original: See Table 4 and its caption.
Revised: See Table 4 and its caption.

Original: (For high pruning rates, see Table 1 in Sec. 3.1.)
Revised: (For high pruning rates, see Table 1 in Sec. 3.1.)

Original: A. C OMPARISON TO P RUNING -M ETHODS THAT FIND L AYER -W ISE P RUNING R ATES
Revised: A. C OMPARISON TO P RUNING -M ETHODS THAT FIND L AYER -W ISE P RUNING R ATES

Original: See Table 5 and its captions (cf.
Revised: See Table 5 and its captions (cf.

Original: also Sec.
Revised: also Sec.

Original: 3.1 for further descriptions).
Revised: 3.1 for further descriptions).

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original:  a c y
Revised: %)     a cc u r a c y

Original: ResNet [Figure] [Figure] ResNet
Revised: ResNet3 # parameters (10 6 )

Original: [Figure] SOSP [Figure] correlations SOSP 
Revised: ResNetSOSP without correlations SOSP with correlations 0.

Original: [Figure] correlations [Figure]
Revised: 0.8 # Pruned Structures (in %)

Original: VGG [Figure] Net [Figure]
Revised: VGG-Net  4 # parameters (10 6 )

Original: [Figure]

Original: [Figure] [Figure] [Figure] ) d)
Revised: VGG-Net a) b) c) d)

Original: [Table]
Revised: [Table]

Revised: A. C OMPARISON OF SOSP ON M OBILE N ET V2 FOR I MAGE N ET

Revised: See Table 6 for a comparison of SOSP for MobileNetV2 for Imagenet

Original:  A. R UNTIME C OMPARISONS OF SOSP-H AND SOSP-I
Revised: . A. R UNTIME C OMPARISONS OF SOSP-H AND SOSP-I

Original: The results of Table 6 complement the results of Fig.
Revised: The results of Table 7 complement the results of Fig.

Original: 2 and show that while SOSP-I is still practical for small-scale networks and datasets for large scale networks and datasets it becomes impractical.
Revised: 2 and show that while SOSP-I is still practical for small-scale networks and datasets for large scale networks and datasets it becomes impractical.

Original: SOSP-H on the other side says practical across all network dataset sizes.
Revised: SOSP-H on the other side says practical across all network dataset sizes.

Original: A. P LOTS FOR C OMPARISON OF SOSP ON R ES N ET -18/50 FOR L ARGE -S CALE D ATASETS
Revised: A. P LOTS FOR C OMPARISON OF SOSP ON R ES N ET -18/50 FOR L ARGE -S CALE D ATASETS

Original: See Figure 6 and its caption (cf.
Revised: See Figure 6 and its caption (cf.

Original: also Sec. 3.2 for further descriptions) 
Revised: also Sec. 3.2 for further descriptions) [Table] [Table]

Original: . A. C OMPARISON OF SOSP ON R ES N ET -56 AGAINST CCP
Revised: A. C OMPARISON OF SOSP ON R ES N ET -56 AGAINST CCP

Original: See Figure 7 and its caption (cf.
Revised: See Figure 7 and its caption (cf.

Original: also Sec. 3 for further comparisons on ResNet-56)
Revised: also Sec. 3 for further comparisons on ResNet-56)

Original: [Table]

Original: [Table]
Revised: [Table]

Original: A. P RUNING AT I NITIALIZATION
Revised: A. P RUNING AT I NITIALIZATION

Original: Traditionally, pruning methods are applied to pretrained networks, as also done in the main part of our work, but recently there has been growing attention on pruning at initialization following the works of Lee et al.
Revised: Traditionally, pruning methods are applied to pretrained networks, as also done in the main part of our work, but recently there has been growing attention on pruning at initialization following the works of Lee et al.

Original: and Frankle & Carbin (2018).
Revised: and Frankle & Carbin (2018).

Original: Since SOSP employs the absolute value of the sensitivities, it can also be applied to a randomly initialized network without any modiﬁcations. Thus
Revised: Since SOSP employs the absolute value of the sensitivities

Original: [Table]
Revised: [Table]

Revised: it can also be applied to a randomly initialized network without any modiﬁcations.

Original:  SOSP can also be seen as an efﬁcient second-order generalization of SNIP (Lee et al., 2018; van Amersfoort et al., 2020).
Revised: Thus, SOSP can also be seen as an efﬁcient second-order generalization of SNIP (Lee et al., 2018; van Amersfoort et al., 2020).

Original: While EigenDamage can in principle be modiﬁed and applied to a randomly initialized network as well, NN Slimming can not be applied at initialization.
Revised: While EigenDamage can in principle be modiﬁed and applied to a randomly initialized network as well, NN Slimming can not be applied at initialization.

Original: Usually pruning at initialization leads to worse accuracies than pruning after training (Liu et al., 2018).
Revised: Usually pruning at initialization leads to worse accuracies than pruning after training (Liu et al., 2018).

Original: However, pruning an already trained network is often followed by ﬁne-tuning, effectively training the network twice (Fig. 8a).
Revised: However, pruning an already trained network is often followed by ﬁne-tuning, effectively training the network twice (Fig. 8a).

Original: For comparability with pruning at initialization, we unify the overall training schedule between these two settings and consequently apply two training cycles after pruning the randomly initialized network (Fig. 8b; for further discussion, see App. A.9).
Revised: For comparability with pruning at initialization, we unify the overall training schedule between these two settings and consequently apply two training cycles after pruning the randomly initialized network (Fig. 8b; for further discussion, see App. A.10).

Original: We observe that applying SOSP at initialization performs almost equally well than applying SOSP after training (see ResNet-56 and VGG in Fig.
Revised: We observe that applying SOSP at initialization performs almost equally well than applying SOSP after training (see ResNet-56 and VGG in Fig.

Original: 8c and f, respectively).
Revised: 8c and f, respectively).

Original: In conclusion, applying SOSP at initialization can signiﬁcantly reduce the time and resources required for network training with no or only minor degradation in accuracy.
Revised: In conclusion, applying SOSP at initialization can signiﬁcantly reduce the time and resources required for network training with no or only minor degradation in accuracy.

Original: A. P RUNING AT I NITIALIZATION VS P RUNING A P RETRAINED N EURAL N ETWORK
Revised: A. P RUNING AT I NITIALIZATION VS P RUNING A P RETRAINED N EURAL N ETWORK

Original: In this section we investigate further why pruning a randomly initialized network tends to achieve lower accuracies compared to pruning a pretrained network (Lee et al., 2018; van Amersfoort et al., 2020).
Revised: In this section we investigate further why pruning a randomly initialized network tends to achieve lower accuracies compared to pruning a pretrained network (Lee et al., 2018; van Amersfoort et al., 2020).

Original: We compare the ﬁnal accuracies of pruning and ﬁne-tuning a randomly initialized and pretrained network (see Fig. 9).
Revised: We compare the ﬁnal accuracies of pruning and ﬁne-tuning a randomly initialized and pretrained network (see Fig. 9).

Original: While pruning a pretrained network leads to considerably higher accuracies compared to pruning a randomly initialized network, the random baseline curves show the same difference.
Revised: While pruning a pretrained network leads to considerably higher accuracies compared to pruning a randomly initialized network, the random baseline curves show the same difference.

Original: Thus, to be able to compare pruning before and after training one needs to device settings that enable a fair comparison, ensuring similar accuracies for random pruning accross both settings.
Revised: Thus, to be able to compare pruning before and after training one needs to device settings that enable a fair comparison, ensuring similar accuracies for random pruning accross both settings.

Original: [Figure]
Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Original: [Figure] 
Revised: [Figure] [Table] [Figure] [Table]

Original: [Figure] 
Revised: [Figure] [Table] [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Revised: A. E XPAND -I NIT D ATA

Original: This section provides the experimental results of the corresponding expand-procedure at initialization (see Sec. 3.3).
Revised: This section provides the experimental results of the corresponding expand-procedure at initialization (see Sec. 3.3).

Original: The results of Fig.
Revised: The results of Fig.

Original: 8 indicate that architectural bottlenecks exist not only for pretrained networks but also for randomly initialized networks.
Revised: 8 indicate that architectural bottlenecks exist not only for pretrained networks but also for randomly initialized networks.

Original: Therefore, we device also an expand scheme before training.
Revised: Therefore, we device also an expand scheme before training.

Original: In this scheme the mask for the expand-procedure is not calculated for a pretrained[Figure] network but for a randomly initialized network.
Revised: In this scheme the mask for the expand-procedure is not calculated for a pretrained network but for a randomly initialized network.

Original: Following the expand scheme the network is pruned and then only ﬁne-tuned once.
Revised: Following the expand scheme the network is pruned and then only ﬁne-tuned once.

Original: The results are displayed in Fig.
Revised: The results are displayed in Fig.

Original: [Figure]

Original: i o

Original: [Figure]

Original: V GG a cc u r a c y

Original: [Figure]
Revised: [Figure]

Original: p r un i n g r a t

Original: i o

Original: [Figure]

Original: w i d t h m u l t i p li e r

Original: [Figure]
Revised: [Figure]

Original: w i d t h m u l t i p

Original: [Figure]

Original: e

Original: [Figure]
Revised: [Figure]

Original: A. S HUFFLE E XPERIMENTS
Revised: A. S HUFFLE E XPERIMENTS

Original: See Table 7 and its caption. 
Revised: See Table 8 and its caption. SOSP

Revised: SOSP prunes structures

Original: [Table]
Revised: [Table]

Original: A. M EAN AND S TANDARD D EVIATION OF F INAL A CCURACIES FOR THE G LOBAL P RUNING C OMPARISON
Revised: A. M EAN AND S TANDARD D EVIATION OF F INAL A CCURACIES FOR THE G LOBAL P RUNING C OMPARISON

Original: See Table 8 and its caption.
Revised: See Table 9 and its caption.

Original: [Table]
Revised: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]
Revised: [Figure]

Original: A. T ABULAR D ATA AND M EAN AND S TANDARD D EVIATION FOR L AYER -W ISE P RUNING C OMPARISON
Revised: A. T ABULAR D ATA AND M EAN AND S TANDARD D EVIATION FOR L AYER -W ISE P RUNING C OMPARISON

Original: This section provides additional data complementing the results of Fig.
Revised: This section provides additional data complementing the results of Fig.

Original: The numerical data of
Revised: The numerical data of

Original: Fig. 1 is shown in Tab. 9 and the mean and standard deviation of the ﬁnal accuracies for both SOSP algorithms is shown in Tab. 10 and Fig.
Revised: Fig. 1 is shown in Tab. 10 and the mean and standard deviation of the ﬁnal accuracies for both SOSP algorithms is shown in Tab. 11 and Fig.

Original: [Figure]
Revised: [Figure]

Original: [Table] 
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: B A PPROXIMATIVE S ECOND -O RDER D ERIVATIVES
Revised: B A PPROXIMATIVE S ECOND -O RDER D ERIVATIVES

Original: In order to approximate the Hessian terms θ Ts H ( θ ) θ s (cid:48) in the matrix Q efﬁciently (see Eq.
Revised: In order to approximate the Hessian terms θ Ts H ( θ ) θ s (cid:48) in the matrix Q efﬁciently (see Eq.

Original: (2) and (3)), we omit from H ( θ )
Revised: (2) and (3)), we omit from H ( θ )

Original: =
Revised: =

Original: 1 N (cid:80) n ∇ 2 θ (cid:96) ( f θ ( x n ) , y n ) those terms that involve the expensive second-order derivatives ∇ 2 θ f θ ( x n ) of the NN outputs, while including second-order couplings due to (cid:96) .
Revised: 1 N (cid:80) n ∇ 2 θ (cid:96) ( f θ ( x n ) , y n ) those terms that involve the expensive second-order derivatives ∇ 2 θ f θ ( x n ) of the NN outputs, while including second-order couplings due to (cid:96) .

Original: This is equivalent to approximating H ( θ )
Revised: This is equivalent to approximating H ( θ )

Original: ≈ H ( f lin θ ) :
Revised: ≈ H ( f lin θ ) :

Original: = 1 N (cid:80) n ∇ 2 θ (cid:96) ( f lin θ ( x n ) , y n ) for the linearized f 
Revised: = 1 N (cid:80) n ∇ 2 θ (cid:96) ( f lin θ ( x n ) , y n ) for the linearized f θ

Original: [Equation]
Revised: [Equation]

Original: ( x )
Revised: x )

Original: [Equation]
Revised: [Equation]

Original: f linθ
Revised: f linθ

Original: [Equation]
Revised: [Equation]

Original: ( x ) :
Revised: ( x ) :

Original: = f θ ( x ) + φ ( x ) · ( θ (cid:48) − θ ) with φ ( x ) :
Revised: = f θ ( x ) + φ ( x ) · ( θ (cid:48) − θ ) with φ ( x ) :

Original: = ∇ θ f θ ( x ) ∈ R D × P , which is well motivated by the NTK limit (Jacot et al., 2018) for large NNs at both initialization and after training.
Revised: = ∇ θ f θ ( x ) ∈ R D × P , which is well motivated by the NTK limit (Jacot et al., 2018) for large NNs at both initialization and after training.

Original: The terms in the sum become then
Revised: The terms in the sum become then

Original: [Equation]
Revised: [Equation]

Original: where R n ∈ R D
Revised: where R n ∈ R D

Original: × D is diagonal for squared loss, and has an additional rank-1 contribution for cross-entropy (see App.
Revised: × D is diagonal for squared loss, and has an additional rank-1 contribution for cross-entropy (see App.

Original: B.1 and App.
Revised: B.1 and App.

Original: B.2, respectively).
Revised: B.2, respectively).

Original: Similar Hessian approximations were employed before in NNs (Hassibi et al., 1993;
Revised: Similar Hessian approximations were employed before in NNs (Hassibi et al., 1993;

Original: Wang et al., 2019a; Peng et al., 2019;
Revised: Wang et al., 2019a; Peng et al., 2019;

Original: Liu et al., 2021;
Revised: Liu et al., 2021;

Original: Theis et al., 2018) and also in the Gauss-Newton optimization method (Fletcher, 2013).
Revised: Theis et al., 2018) and also in the Gauss-Newton optimization method (Fletcher, 2013).

Original: Below we derive our approximation of the second-order loss derivatives Eq.
Revised: Below we derive our approximation of the second-order loss derivatives Eq.

Original: (9) that lead to the desired Eq.
Revised: (9) that lead to the desired Eq.

Original: (5), and especially the particular matrix structures of R n in these expressions (diagonal plus rank-1), which allow for a more efﬁcient matrix multiplication (App. B.4).
Revised: (5), and especially the particular matrix structures of R n in these expressions (diagonal plus rank-1), which allow for a more efﬁcient matrix multiplication (App. B.4).

Original: Our approximation is based on omitting from the exact loss derivative those terms that involve the (expensive) second-order derivatives ∇ 2 θ f θ ( x n ) of the NN outputs f θ ( x n ) ∈ R D .
Revised: Our approximation is based on omitting from the exact loss derivative those terms that involve the (expensive) second-order derivatives ∇ 2 θ f θ ( x n ) of the NN outputs f θ ( x n ) ∈ R D .

Original: We however still include second-order couplings due to the loss function (cid:96) .
Revised: We however still include second-order couplings due to the loss function (cid:96) .

Original: Our approximation is thus to approximate the NN output
Revised: Our approximation is thus to approximate the NN output

Original: [Equation]
Revised: [Equation]

Original: to ﬁrst order, where φ ( x ) := ∇ θ f θ ( x ) ∈ R D
Revised: to ﬁrst order, where φ ( x ) := ∇ θ f θ ( x ) ∈ R D

Original: × P is the ﬁrst-order derivative of the NN output, and then to approximate the second-order derivatives of the NN loss as follows:
Revised: × P is the ﬁrst-order derivative of the NN output, and then to approximate the second-order derivatives of the NN loss as follows:

Original: [Equation]
Revised: [Equation]

Original: We now compute this approximation ∇ 2 θ (cid:96) (cid:0) f linθ ( x n ) , y n (cid:1) for both the squared loss (cid:96) ( f, y ) :
Revised: We now compute this approximation ∇ 2 θ (cid:96) (cid:0) f linθ ( x n ) , y n (cid:1) for both the squared loss (cid:96) ( f, y ) :

Original: = 12 (cid:107) f − y (cid:107) 2 as well as for the cross-entropy loss (cid:96) ( f, y ) :
Revised: = 12 (cid:107) f − y (cid:107) 2 as well as for the cross-entropy loss (cid:96) ( f, y ) :

Original: = − log σ ( f ) y , where σ : R D → R D denotes the softmax function.
Revised: = − log σ ( f ) y , where σ : R D → R D denotes the softmax function.

Original: In this computation we use the following facts:
Revised: In this computation we use the following facts:

Original: [Equation]
Revised: [Equation]

Original: which follow directly from (10)
Revised: which follow directly from (10)

Original: [Equation]
Revised: [Equation]

Original: B. S ECOND -O RDER A PPROXIMATION FOR S QUARED L OSS
Revised: B. S ECOND -O RDER A PPROXIMATION FOR S QUARED L OSS

Original: For the squared loss
Revised: For the squared loss

Original: [Equation]
Revised: [Equation]

Original: we obtain:
Revised: we obtain:

Original: [Equation]
Revised: [Equation]

Original: where R n := 1 D × D is here the D × D -identity matrix.
Revised: where R n := 1 D × D is here the D × D -identity matrix.

Original: This is Eq.
Revised: This is Eq.

Original: (9) for the squared loss.
Revised: (9) for the squared loss.

Original: Due to this diagonal form of R n , the matrix multiplication ( φ ( x n ) θ s ) T R n ( φ ( x n ) θ s (cid:48) ) in Eq.
Revised: Due to this diagonal form of R n , the matrix multiplication ( φ ( x n ) θ s ) T R n ( φ ( x n ) θ s (cid:48) ) in Eq.

Original: (5) has, for each pair ( s, s (cid:48) ) , a computational complexity linear in the dimension D (which is the number of
Revised: (5) has, for each pair ( s, s (cid:48) ) , a computational complexity linear in the dimension D (which is the number of

Original: NN outputs), rather than quadratic:
Revised: NN outputs), rather than quadratic:

Original: [Equation]
Revised: [Equation]

Original: where ( φ ( x n ) θ s )
Revised: where ( φ ( x n ) θ s )

Original: j ∈ R denotes the j -th component of the vector φ ( x n ) θ s ∈ R D .
Revised: j ∈ R denotes the j -th component of the vector φ ( x n ) θ s ∈ R D .

Original: It is for this reason that the overall computational complexity of SOSP-I is linear in D (see Sect. 2.1), rather than of order O
Revised: It is for this reason that the overall computational complexity of SOSP-I is linear in D (see Sect. 2.1), rather than of order O

Original: ( D 2 ) .
Revised: ( D 2 ) .

Original: B. S ECOND -O RDER A PPROXIMATION FOR C ROSS -E NTROPY L OSS
Revised: B. S ECOND -O RDER A PPROXIMATION FOR C ROSS -E NTROPY L OSS

Original: Note that the ﬁrst derivatives of the softmax-function σ : R D → R D , deﬁned by σ ( f ) i :=
Revised: Note that the ﬁrst derivatives of the softmax-function σ : R D → R D , deﬁned by σ ( f ) i :=

Original: e f i / (cid:80) Dk =

Revised: e f i / (cid:80) Dk =

Original: e f k are:
Revised: e f k are:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: We can thus compute for the cross-entropy loss, where δ y · ∈ R D denotes the vector with entry 1 in component y and entries 0 everywhere else:
Revised: We can thus compute for the cross-entropy loss, where δ y · ∈ R D denotes the vector with entry 1 in component y and entries 0 everywhere else:

Original: [Equation]
Revised: [Equation]

Original: To compute ∇ θ σ ( f linθ ( x n ) in (25)
Revised: To compute ∇ θ σ ( f linθ ( x n ) in (25)

Original: [Equation]
Revised: [Equation]

Original: we consider its i -th component:
Revised: we consider its i -th component:

Original: [Equation]
Revised: [Equation]

Original: where R n ∈ R D × D is the matrix with entries
Revised: where R n ∈ R D × D is the matrix with entries

Original: [Equation]
Revised: [Equation]

Original: Thus [Equation] plugging back into (25),∇
Revised: Thus, plugging back into (25),

Original: [Equation]
Revised: [Equation]

Original: which is Eq.
Revised: which is Eq

Revised: [Equation]

Original: (9) for the cross-entropy loss
Revised: (9) for the cross-entropy loss

Original: [Equation]
Revised: [Equation]

Original: B. S ECOND -O RDER A PPROXIMATION FOR OTHER L OSS F UNCTIONS
Revised: B. S ECOND -O RDER A PPROXIMATION FOR OTHER L OSS F UNCTIONS

Original: The same developments as above apply to any twice differentiable loss function (cid:96) :
Revised: The same developments as above apply to any twice differentiable loss function (cid:96) :

Original: The Hessian of the loss with the linearized model f linθ instead of the full network f θ can be written as
Revised: The Hessian of the loss with the linearized model f linθ instead of the full network f θ can be written as

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where R n =∇
Revised: where R n =

Original: [Equation]

Original: [Equation]

Original: f

Original: [Equation]
Revised: [Equation]

Original: ) ∈ R D
Revised: ∈ R D

Original: × D has a different form depending on the speciﬁc loss function.
Revised: × D has a different form depending on the speciﬁc loss function.

Original: Thus, a general loss function (cid:96) still allows the use of the approximation used by SOSPI.
Revised: Thus, a general loss function (cid:96) still allows the use of the approximation used by SOSPI.

Original: The only caveat is that R n in general can be any D × D -matrix (without sparsity or low-rank properties), so that matrix multiplication may in general not be as efﬁcient as for the above loss functions (described in detail in App.
Revised: The only caveat is that R n in general can be any D × D -matrix (without sparsity or low-rank properties), so that matrix multiplication may in general not be as efﬁcient as for the above loss functions (described in detail in App.

Original: B.4); in this case the second complexity term for SOSP-I in Eq.
Revised: B.4); in this case the second complexity term for SOSP-I in Eq.

Original: (7) can worsen from O ( N (cid:48) DS 2 ) to O ( N (cid:48)
Revised: (7) can worsen from O ( N (cid:48) DS 2 ) to O ( N (cid:48)

Original: D 2 S 2 ) .
Revised: D 2 S 2 ) .

Original: B. E FFICIENT MATRIX MULTIPLICATION FOR SQUARED LOSS AND CROSS - ENTROPY LOSS
Revised: B. E FFICIENT MATRIX MULTIPLICATION FOR SQUARED LOSS AND CROSS - ENTROPY LOSS

Original: Note that R n is a sum of a diagonal matrix R diag n (ﬁrst part in Eq.
Revised: Note that R n is a sum of a diagonal matrix R diag n (ﬁrst part in Eq.

Original: (31)) and a rank-1 matrix
Revised: (31)) and a rank-1 matrix

Original: R rank-1 n (second part in Eq.
Revised: R rank-1 n (second part in Eq.

Original: Due to this special matrix form, the matrix multiplication ( φ ( x n ) θ s ) T R n ( φ ( x n ) θ s (cid:48) ) in Eq.
Revised: Due to this special matrix form, the matrix multiplication ( φ ( x n ) θ s ) T R n ( φ ( x n ) θ s (cid:48) ) in Eq.

Original: (5) has, for each pair ( s, s (cid:48) ) , a computational complexity linear in the dimension D (the number of NN outputs), rather than quadratic.
Revised: (5) has, for each pair ( s, s (cid:48) ) , a computational complexity linear in the dimension D (the number of NN outputs), rather than quadratic.

Original: For the diagonal part (cid:0)
Revised: For the diagonal part (cid:0)

Original: R diag n (cid:1) ij = σ ( f θ ( x n )) i δ ij , the reason for this is similar to the one for the squared error:
Revised: R diag n (cid:1) ij = σ ( f θ ( x n )) i δ ij , the reason for this is similar to the one for the squared error:

Original: [Equation]
Revised: [Equation]

Original: For the rank-1 part (cid:0) R rank-1 n (cid:1) ij = − σ ( f θ ( x n ))
Revised: For the rank-1 part (cid:0) R rank-1 n [Equation] ij = − σ [Equation]

Original:  i σ
Revised: f [Equation] x n )) i σ

Original: [Equation]
Revised: [Equation]

Original: f θ ( x n ))
Revised: f [Equation] ( x n ))

Original: j , the O ( D ) -efﬁcient computation is
Revised: j , the O ( D ) -efﬁcient computation is

Original: [Equation]
Revised: [Equation]

Original: Again, for these reasons, the overall computational complexity of SOSP-I is linear in D (see Sect. 2.1), instead of quadratic in D .
Revised: Again, for these reasons, the overall computational complexity of SOSP-I is linear in D (see Sect. 2.1), instead of quadratic in D .

Original: This can make a signiﬁcant difference for some datasets (e.g. D =classes on ImageNet).
Revised: This can make a signiﬁcant difference for some datasets (e.g. D =classes on ImageNet).

Original: An alternative O ( D ) -efﬁcient way of computing our second-order approximation follows by continuing from Eq.
Revised: An alternative O ( D ) -efﬁcient way of computing our second-order approximation follows by continuing from Eq.

Original: (25), noting that ∇ θ f linθ ( x n ) =
Revised: (25), noting that ∇ θ f linθ ( x n ) =

Original: ∇ θ f θ ( x n ) by (13):
Revised: ∇ θ f θ ( x n ) by (13):

Original: [Equation]
Revised: [Equation]

Original: where we deﬁned φ σ ( x n ) :
Revised: where we deﬁned φ σ ( x n ) :

Original: = ∇ θ ( σ ( f θ ( x n )))
Revised: = ∇ θ ( σ ( f θ ( x n )))

Original: ∈ R D × P .
Revised: ∈ R D × P .

Original: Thus, each term in the sum (5) can be written as
Revised: Thus, each term in the sum (5) can be written as

Original: [Equation]
Revised: [Equation]

Original: which again has complexity O ( D ) .
Revised: which again has complexity O ( D ) .

Original: For this it is necessary to pre-compute each φ σ ( x n ) in addition to φ ( x n ) , but both have the same complexity.
Revised: For this it is necessary to pre-compute each φ σ ( x n ) in addition to φ ( x n ) , but both have the same complexity.

Original: We ﬁnally note that our approximation of the second-order derivatives (i.e., of the Hessian) is somewhat different from the approximation made in (Peng et al., 2019) for the cross-entropy case:
Revised: We ﬁnally note that our approximation of the second-order derivatives (i.e., of the Hessian) is somewhat different from the approximation made in (Peng et al., 2019) for the cross-entropy case:

Original: While we dropped all second-order derivatives ∇ 2 θ f θ ( x n ) of the pre -softmax activations, Peng et al.
Revised: While we dropped all second-order derivatives ∇ 2 θ f θ ( x n ) of the pre -softmax activations, Peng et al.

Original: (2019) dropped all second-order derivatives of the post -softmax activiations, i.e. all terms ∇ 2 θ ( σ ( f θ ( x n )) .
Revised: (2019) dropped all second-order derivatives of the post -softmax activiations, i.e. all terms ∇ 2 θ ( σ ( f θ ( x n )) .

Original: A consequence of this difference is that our approximation (33) (or (42)) does not depend on the labels y n (this is already apparent from our intermediate step Eq.
Revised: A consequence of this difference is that our approximation (33) (or (42)) does not depend on the labels y n (this is already apparent from our intermediate step Eq.

Original: (25)), whereas the approximation made in (Peng et al., 2019) does depend on the labels y n .
Revised: (25)), whereas the approximation made in (Peng et al., 2019) does depend on the labels y n .

Original: The fact that our second-order approximation is independent of the y n is similar to the second-order approximation in the squared-loss case (see App.
Revised: The fact that our second-order approximation is independent of the y n is similar to the second-order approximation in the squared-loss case (see App.

Original: B.1 above, and also (Peng et al., 2019)).
Revised: B.1 above, and also (Peng et al., 2019)).

Original: Note furthermore that the ﬁrst-order terms in the loss approximation (see e.g. in Eq.
Revised: Note furthermore that the ﬁrst-order terms in the loss approximation (see e.g. in Eq.

Original: (2) or (3)) are the same in our method as in (Peng et al., 2019)), and these do depend on the labels y n (cf.
Revised: (2) or (3)) are the same in our method as in (Peng et al., 2019)), and these do depend on the labels y n (cf.

Original: the expression in square bracket in Eq.
Revised: the expression in square bracket in Eq.

Original: C S ECOND -O RDER A PPROXIMATION C ORRESPONDS TO O UTPUT -C ORRELATION
Revised: C S ECOND -O RDER A PPROXIMATION C ORRESPONDS TO O UTPUT -C ORRELATION

Original: The purpose of this section is to provide a better intuition of the second-order components of our loss approximation.
Revised: The purpose of this section is to provide a better intuition of the second-order components of our loss approximation.

Original: First, we reformulate the expression for the second-order terms in Eq.
Revised: First, we reformulate the expression for the second-order terms in Eq.

Original: We use the fact that for any ReLU-NN without batch normalization layers it holds almost everywhere that
Revised: We use the fact that for any ReLU-NN without batch normalization layers it holds almost everywhere that

Original: [Equation]
Revised: [Equation]

Original: 43) where θ s is the weight vector, where all structures of the layer that contains structure s are set to zero expect for the weights of structure s itself.
Revised: where θ s is the weight vector, where all structures of the layer that contains structure s are set to zero expect for the weights of structure s itself.

Original: The identity is straightforward to derive, therefore we show the relation for a feed-forward neural network with zero biases, but the proof for a convolutional neural network is almost identical .
Revised: The identity is straightforward to derive, therefore we show the relation for a feed-forward neural network with zero biases, but the proof for a convolutional neural network is almost identical .

Original: Let f θ be a fully-connected neural network with
Revised: Let f θ be a fully-connected neural network with

Original: L layers and
Revised: L layers and

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 1 ( x ) [Equation] W 1 x , where W l are the weight-matrices and h l ( x ) the output functions of the l -th layer and l h l ( x ) ≥ 

Revised: 1 ( x ) ≥ 0 W 1 x , where W l are the weight-matrices and h l ( x ) the output functions of the l -th layer and l h l ( x ) ≥

Original: the diagonal matrix with the step function on the diagonal elements corresponding to the components of h l ( x ) ≥ 0 .
Revised: the diagonal matrix with the step function on the diagonal elements corresponding to the components of h l ( x ) ≥ 0 .

Original: Now assuming that structure s is contained in the i -th layer, the only non-vanishing components of the vector θ s are the once associated with structure s .
Revised: Now assuming that structure s is contained in the i -th layer, the only non-vanishing components of the vector θ s are the once associated with structure s .

Original: Thus, one can evaluate the gradient to get
Revised: Thus, one can evaluate the gradient to get

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where W si is the weight matrix of layer i where all components are set to zero except for those belonging to structure s .
Revised: where W si is the weight matrix of layer i where all components are set to zero except for those belonging to structure s .

Original: Using this, one directly receives ∇ θ f θ( x
Revised: Using this, one directly receives ∇ θ f θ

Original: [Equation]
Revised: [Equation]

Original: θ s = f

Original: [Equation]

Original: s ( x )

Original: Next, applying this identity to Eq.
Revised: Next, applying this identity to Eq.

Original: 5 gives
Revised: 5 gives

Original: [Equation]
Revised: [Equation]

Original: N (cid:48) (cid:88) n = ( φ ( x n ) [Equation] s ) T R n ( φ ( x n ) [Equation]
Revised: N (cid:48) (cid:88) n = ( φ ( x n ) θ s ) T R n ( φ ( x n ) θ s (cid:48) )

Original: s (cid:48) )

Original: =
Revised: =

Original: N (cid:48)
Revised: N (cid:48)

Original: N (cid:48) (cid:88) n = ( ∇
Revised: N (cid:48) (cid:88) n = ( ∇

Original: [Equation]
Revised: [Equation]

Original: f
Revised: f

Original: [Equation]
Revised: [Equation]

Original: ( x n )
Revised: ( x n )

Original: [Equation]
Revised: [Equation]

Original: s )
Revised: s )

Original: [Equation]
Revised: [Equation]

Original: n 
Revised: n (

Original: [Equation]
Revised: [Equation]

Original: f [Equation] ( x n )
Revised: f θ ( x n )

Original: [Equation]
Revised: [Equation]

Original:  (cid:48) )
Revised: s (cid:48) )

Original: =
Revised: =

Original: N (cid:48)
Revised: N (cid:48)

Original: N (cid:48) (cid:88) n 
Revised: N (cid:48) (cid:88) n =

Original: [Equation]
Revised: [Equation]

Original: f
Revised: f

Revised: [Equation]

Revised: [Equation]

Revised: n

Original: [Equation]
Revised: [Equation]

Original:  ) , where the explicit form of the R n matrix is provided in the previous section.
Revised: n f [Equation] (cid:48) ( x n ) , where the explicit form of the R n matrix is provided in the previous section.

Original: From this relation one can now see that the second-order components of our pruning objective correspond to outputcorrelations.
Revised: From this relation one can now see that the second-order components of our pruning objective correspond to outputcorrelations.

Original: This connection could also explain, why our second-order pruning methods do not improve the pruning performance at initialization, but do improve performance after training, since at initialization different structures may not have as strong output-correlations as the learned features after training.
Revised: This connection could also explain, why our second-order pruning methods do not improve the pruning performance at initialization, but do improve performance after training, since at initialization different structures may not have as strong output-correlations as the learned features after training.

Original: D C OUNTING OF P ARAMETERS AND MAC S
Revised: D C OUNTING OF P ARAMETERS AND MAC S

Original: Here we provide details on how we compute the number of parameters of our pruned NNs, and the number of MACs required for the evaluation of a pruned NN on one input point.
Revised: Here we provide details on how we compute the number of parameters of our pruned NNs, and the number of MACs required for the evaluation of a pruned NN on one input point.

Original: The description is speciﬁc to the ResNets and DenseNets used in our work; the main complication arises for the ResNet architecture (in particular for the residual connections), as we describe below.
Revised: The description is speciﬁc to the ResNets and DenseNets used in our work; the main complication arises for the ResNet architecture (in particular for the residual connections), as we describe below.

Original: While our parameter and MAC counting is exact, it is somewhat intricate.
Revised: While our parameter and MAC counting is exact, it is somewhat intricate.

Original: We do not know whether this exact counting has been implemented in other papers on pruning as well (see the comparisons in Tables 1 and 2), since these other works did not elaborate on their counting.
Revised: We do not know whether this exact counting has been implemented in other papers on pruning as well (see the comparisons in Tables 1 and 2), since these other works did not elaborate on their counting.

Original: Therefore, the comparability with the counts from other papers is not necessarily given.
Revised: Therefore, the comparability with the counts from other papers is not necessarily given.

Original: When we compare our exact counting to a more straightforward but approximative counting, we ﬁnd that our exact counts for
Revised: When we compare our exact counting to a more straightforward but approximative counting, we ﬁnd that our exact counts for

Original: ResNet50 (Table 2) yield substantially higher parameter and MAC numbers than the straightforward approximate counting.
Revised: ResNet50 (Table 2) yield substantially higher parameter and MAC numbers than the straightforward approximate counting.

Original: The straightforward approximate counting would yield MAC-pruning-ratios that are 12-18 percentage points higher (better) than our exact numbers.
Revised: The straightforward approximate counting would yield MAC-pruning-ratios that are 12-18 percentage points higher (better) than our exact numbers.

Original: We now explain our exact counting ﬁrst.
Revised: We now explain our exact counting ﬁrst.

Original: The number of parameters of a convolutional layer l equals the number F inl of input ﬁlters of the layer times the number F outl of output ﬁlters times the kernel size K whl (which equals the kernel width times the kernel height): C l = F inl · F outl · K whl .
Revised: The number of parameters of a convolutional layer l equals the number F inl of input ﬁlters of the layer times the number F outl of output ﬁlters times the kernel size K whl (which equals the kernel width times the kernel height): C l = F inl · F outl · K whl .

Original: In addition to this count, there are 2 F outl parameters for the batchnorm layer following each convolutional layer (our networks do not have bias terms in the convolutional layers, which would add another F outl ).
Revised: In addition to this count, there are 2 F outl parameters for the batchnorm layer following each convolutional layer (our networks do not have bias terms in the convolutional layers, which would add another F outl ).

Original: The subtlety is now that, within the chain of convolutional layers in the ResNet architecture, the number F inl +1 of input ﬁlters into the following convolutional layer l + 1 does not necessarily equal the number of output ﬁlters F outl of the present convolutional layer l .
Revised: The subtlety is now that, within the chain of convolutional layers in the ResNet architecture, the number F inl +1 of input ﬁlters into the following convolutional layer l + 1 does not necessarily equal the number of output ﬁlters F outl of the present convolutional layer l .

Original: Namely, this can happen if the output of layer l is added to the output of a residual connection to compute the input into l + 1 .
Revised: Namely, this can happen if the output of layer l is added to the output of a residual connection to compute the input into l + 1 .

Original: At such a point in the chain of convolutional layers, the number of input ﬁlters F inl +1 into layer l +depends on both the output ﬁlters of layer l as well as on the output ﬁlters of the residual connection.
Revised: At such a point in the chain of convolutional layers, the number of input ﬁlters F inl +1 into layer l +depends on both the output ﬁlters of layer l as well as on the output ﬁlters of the residual connection.

Original: More precisely, only those ﬁlters can be removed from the input into layer l + 1 which are absent from both the output of layer l as well as from the output of the residual connection.
Revised: More precisely, only those ﬁlters can be removed from the input into layer l + 1 which are absent from both the output of layer l as well as from the output of the residual connection.

Original: (Note, thus, that the number F inl +1 of input ﬁlters into layer l + 1 cannot be determined by only knowing the number of output ﬁlters F outl and the number of output ﬁlters of the residual connection.
Revised: (Note, thus, that the number F inl +1 of input ﬁlters into layer l + 1 cannot be determined by only knowing the number of output ﬁlters F outl and the number of output ﬁlters of the residual connection.

Original: Rather, the answer depends on which output ﬁlters have been pruned.)
Revised: Rather, the answer depends on which output ﬁlters have been pruned.)

Original: To determine F inl +1 , two kinds of residual connections have to be distinguished:
Revised: To determine F inl +1 , two kinds of residual connections have to be distinguished:

Original: (a) Residual connection is an identity skip connection.
Revised: (a) Residual connection is an identity skip connection.

Original: In this case, the output ﬁlters of the residual connection are exactly the output ﬁlters of a previous layer: either the output ﬁlters of layer l (cid:48) :
Revised: In this case, the output ﬁlters of the residual connection are exactly the output ﬁlters of a previous layer: either the output ﬁlters of layer l (cid:48) :

Original: = l − 2 (when the skip connection is in a “BasicBlock”) or of layer l (cid:48) :
Revised: = l − 2 (when the skip connection is in a “BasicBlock”) or of layer l (cid:48) :

Original: = l −(when the skip connection is in a “BottleneckBlock”).
Revised: = l −(when the skip connection is in a “BottleneckBlock”).

Original: F inl +1 thus equals the number of ﬁlters that are un-pruned in the output of both layer l (cid:48) and un-pruned in the output of layer l .
Revised: F inl +1 thus equals the number of ﬁlters that are un-pruned in the output of both layer l (cid:48) and un-pruned in the output of layer l .

Original: (b) Residual connection is a downsampling layer.
Revised: (b) Residual connection is a downsampling layer.

Original: As we exclude downsampling layers from pruning (see Sec. 3), the number of output ﬁlters of a downsampling layer in the pruned network equals the number of outputs of the downsampling layer in the original network.
Revised: As we exclude downsampling layers from pruning (see Sec. 3), the number of output ﬁlters of a downsampling layer in the pruned network equals the number of outputs of the downsampling layer in the original network.

Original: Therefore, if the output of layer l is added to the output of a downsampling layer, then F inl + in the pruned network takes the same value as in the original (un-pruned) network,
Revised: Therefore, if the output of layer l is added to the output of a downsampling layer, then F inl + in the pruned network takes the same value as in the original (un-pruned) network,

Original: i
Revised: i

Original: e. F inl
Revised: e. F inl

Original: + is the original number of input ﬁlters into layer l + 1 .
Revised: + is the original number of input ﬁlters into layer l + 1 .

Original: The same reasoning and procedure applies to determining the number of input ﬁlters into any downsampling layer (i.e., this number is the same as the number of inputs into the convolutional layer that has the same input as the downsampling layer; note, a downsampling layer is a convolutional layer as well, but not part of the “chain of convolutional layers”), and also to determine the number of inputs (“input neurons”) into the ﬁnal fully-connected layer (i.e., the number of “input neurons” into the last fully-connected layer is the same as would be the number of input ﬁlters into a convolutional layer at this stage).
Revised: The same reasoning and procedure applies to determining the number of input ﬁlters into any downsampling layer (i.e., this number is the same as the number of inputs into the convolutional layer that has the same input as the downsampling layer; note, a downsampling layer is a convolutional layer as well, but not part of the “chain of convolutional layers”), and also to determine the number of inputs (“input neurons”) into the ﬁnal fully-connected layer (i.e., the number of “input neurons” into the last fully-connected layer is the same as would be the number of input ﬁlters into a convolutional layer at this stage).

Original: Having determined the number of input and output ﬁlters (neurons) into all channels in this way, our parameter count sums up the parameter numbers for all convolutional layers (incl.
Revised: Having determined the number of input and output ﬁlters (neurons) into all channels in this way, our parameter count sums up the parameter numbers for all convolutional layers (incl.

Original: downsampling layers), batchnorm layers, and fully-connected layers (incl.
Revised: downsampling layers), batchnorm layers, and fully-connected layers (incl.

Original: bias terms).
Revised: bias terms).

Original: This is the exact number of parameters needed to specify the pruned NN and also to build the pruned NN, since these parameters specify all surviving ﬁlters (and the fully-connected layers).
Revised: This is the exact number of parameters needed to specify the pruned NN and also to build the pruned NN, since these parameters specify all surviving ﬁlters (and the fully-connected layers).

Original: Our exact MAC count is similar, also based on the “true” F inl and F outl as just determined.
Revised: Our exact MAC count is similar, also based on the “true” F inl and F outl as just determined.

Original: The
Revised: The

Original: MAC count of a convolutional layer is M l = C l · S wl · S hl , where S wl and S hl are the numbers of width-wise and height-wise applications of each ﬁlter; for our networks, S wl equals the spatial picture width at layer l divided by the width-stride, and similarly for S hl .
Revised: MAC count of a convolutional layer is M l = C l · S wl · S hl , where S wl and S hl are the numbers of width-wise and height-wise applications of each ﬁlter; for our networks, S wl equals the spatial picture width at layer l divided by the width-stride, and similarly for S hl .

Original: Finally – and as mentioned above – we brieﬂy describe the more straightforward but approximate counting method, that would yield pruning ratios that can appear quite a bit better.
Revised: Finally – and as mentioned above – we brieﬂy describe the more straightforward but approximate counting method, that would yield pruning ratios that can appear quite a bit better.

Original: In this approximate way of counting, we take, within the chain of convolutional layers in the ResNet architecture, as input ﬁlters into layer l + 1 exactly the output ﬁlters from layer l , i.e. F inl +1 :
Revised: In this approximate way of counting, we take, within the chain of convolutional layers in the ResNet architecture, as input ﬁlters into layer l + 1 exactly the output ﬁlters from layer l , i.e. F inl +1 :

Original: = F outl .
Revised: = F outl .

Original: Consequently for any downsampling layer, we take as its number of input ﬁlters the number of output ﬁlters of its preceding convolutional layer, and as its number of output ﬁlters we take the number of input ﬁlters into the following convolutional layer (as determined by the previous sentence).
Revised: Consequently for any downsampling layer, we take as its number of input ﬁlters the number of output ﬁlters of its preceding convolutional layer, and as its number of output ﬁlters we take the number of input ﬁlters into the following convolutional layer (as determined by the previous sentence).

Original: We compute the number of parameters and MACs then according to the same formulas as in the exact method, but with potentially other values for F in l and F out l for all convolutional layers (incl.
Revised: We compute the number of parameters and MACs then according to the same formulas as in the exact method, but with potentially other values for F in l and F out l for all convolutional layers (incl.

Original: the downsampling layers, and the number of input nodes into the last fully-connected layer).
Revised: the downsampling layers, and the number of input nodes into the last fully-connected layer).

Original: Note, this approximate count is actually the exact count for a version of the pruned network where the size of the residual connections (identity skip connections and downsampling layers) has been adapted in a “natural” way to the sizes of the pruned convolutional layers in the chain of convolutional layers.
Revised: Note, this approximate count is actually the exact count for a version of the pruned network where the size of the residual connections (identity skip connections and downsampling layers) has been adapted in a “natural” way to the sizes of the pruned convolutional layers in the chain of convolutional layers.

Original: In particular, this count can therefore be computed by just knowing the numbers of pruned ﬁlters in each convolutional layer, instead of knowing exactly which of the ﬁlters in each convolutional layer have been pruned.
Revised: In particular, this count can therefore be computed by just knowing the numbers of pruned ﬁlters in each convolutional layer, instead of knowing exactly which of the ﬁlters in each convolutional layer have been pruned.

Original: It is apparent from the way of approximate counting just described that its parameter and MAC count will be smaller or equal to the exact count (described futher above); both counts conincide for the original (un-pruned) network.
Revised: It is apparent from the way of approximate counting just described that its parameter and MAC count will be smaller or equal to the exact count (described futher above); both counts conincide for the original (un-pruned) network.

Original: For pruned networks, however, the difference in both counts can be substantial, esp.
Revised: For pruned networks, however, the difference in both counts can be substantial, esp.

Original: for the ResNet50 network (Table 2).
Revised: for the ResNet50 network (Table 2).

Original: For example, whereas the MAC-pruning-ratio of our SOSP-I(0.3) method is 15% (see Table 2), its MAC-pruning-ratio according to the approximate counting would equal 27%.
Revised: For example, whereas the MAC-pruning-ratio of our SOSP-I(0.3) method is 15% (see Table 2), its MAC-pruning-ratio according to the approximate counting would equal 27%.

Original: In case that the competitor papers used this simpler approximate counting (which we do not know), we should also use this approximate counting to evaluate our method, which would thus appear more favorable, especially on the ImageNet experiments and especially on
Revised: In case that the competitor papers used this simpler approximate counting (which we do not know), we should also use this approximate counting to evaluate our method, which would thus appear more favorable, especially on the ImageNet experiments and especially on

Original: ResNet-50 (Table 2).
Revised: ResNet-50 (Table 2).

Original: On a ﬁnal note, we remark that the paper Tang et al.
Revised: On a ﬁnal note, we remark that the paper Tang et al.

Original: (2020) introducing the SCOP method, mentioned a discrepancy between the theoretically computed number of MACs and the experimentally measured value for this quantity, hinting at least at some inconsistency in the counting.
Revised: (2020) introducing the SCOP method, mentioned a discrepancy between the theoretically computed number of MACs and the experimentally measured value for this quantity, hinting at least at some inconsistency in the counting.