Original: L OOKAHEAD : A F AR - SIGHTED A LTERNATIVE OF M AGNITUDE - BASED P RUNING
Revised: L OOKAHEAD : A F AR - SIGHTED A LTERNATIVE OF M AGNITUDE - BASED P RUNING

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Magnitude-based pruning is one of the simplest methods for pruning neural networks.
Revised: Magnitude-based pruning is one of the simplest methods for pruning neural networks.

Original: Despite its simplicity, magnitude-based pruning and its variants demonstrated remarkable performances for pruning modern architectures.
Revised: Despite its simplicity, magnitude-based pruning and its variants demonstrated remarkable performances for pruning modern architectures.

Original: Based on the observation thatthe magnitude-based pruning indeed minimizes the Frobenius distortion of a linear operator corresponding to a single layer, we develop a simple pruning method, coined lookahead pruning, by extending the single layer optimization to a multi-layer optimization.
Revised: Based on the observation that magnitude-based pruning indeed minimizes the Frobenius distortion of a linear operator corresponding to a single layer, we develop a simple pruning method, coined lookahead pruning, by extending the single layer optimization to a multi-layer optimization.

Original: Our experimental results demonstrate that the proposed method consistently outperforms the magnitude pruning on various networks including VGG and ResNet, particularly in the high-sparsity regime.
Revised: Our experimental results demonstrate that the proposed method consistently outperforms magnitude-based pruning on various networks, including VGG and ResNet, particularly in the high-sparsity regime.

Revised: See https://github.com/alinlab/lookahead_pruning for codes.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The “magnitude-equals-saliency” approach has been long underlooked as an overly simplistic baseline among all imaginable techniques to eliminate unnecessary weights from over-parametrized neural networks.
Revised: The “magnitude-equals-saliency” approach has been long underlooked as an overly simplistic baseline among all imaginable techniques to eliminate unnecessary weights from over-parametrized neural networks.

Original: Since the early works of LeCun et al.
Revised: Since the early works of LeCun et al.

Original: ; Hassibi & Stork (1993) which provided more theoretically grounded alternative of magnitude-based pruning (MP) based on second derivatives of  loss function, a wide range of methods including Bayesian / information-theoretic approaches (Neal, 1996; Louizos et al., 2017; Molchanov et al., 2017; Dai et al., 2018), (cid:96) pregularization (Wen et al., 2016; Liu et al., 2017; Louizos et al., 2018), sharing redundant channels (Zhang et al., 2018; Ding et al., 2019), and reinforcement learning approaches (Lin et al., 2017;
Revised: ; Hassibi & Stork (1993) which provided more theoretically grounded alternatives of magnitude-based pruning (MP) based on second derivatives of the loss function, a wide range of methods including Bayesian / informationtheoretic approaches (Neal, 1996; Louizos et al., 2017; Molchanov et al., 2017; Dai et al., 2018), (cid:96) p -regularization (Wen et al., 2016; Liu et al., 2017; Louizos et al., 2018), sharing redundant channels (Zhang et al., 2018; Ding et al., 2019), and reinforcement learning approaches (Lin et al., 2017;

Original: Bellec et al., 2018; He et al., 2018) has been proposed as more sophisticated alternatives.
Revised: Bellec et al., 2018; He et al., 2018) have been proposed as more sophisticated alternatives.

Original: On the other hand, the capabilities of MP heuristics are gaining attention once more.
Revised: On the other hand, the capabilities of MP heuristics are gaining attention once more.

Original: Combined with minimalistic techniques including iterative pruning (Han et al., 2015) and dynamic reestablishment of connections (Zhu & Gupta, 2017), a recent large-scale study by Gale et al.
Revised: Combined with minimalistic techniques including iterative pruning (Han et al., 2015) and dynamic reestablishment of connections (Zhu & Gupta, 2017), a recent large-scale study by Gale et al.

Original: (2019) claims that MP can achieve a state-of-the-art trade-off of sparsity and accuracy on ResNet-50.
Revised: (2019) claims that MP can achieve a state-of-the-art trade-off between sparsity and accuracy on ResNet-50.

Original: The unreasonable effectiveness of magnitude scores often extends beyond the strict domain of network pruning; a recent experiment by Frankle & Carbin (2019) suggests an existence of an automatic subnetwork discovery mechanism underlying the standard gradient-based optimization procedures of deep, overparametrized neural networks by showing that the MP algorithm ﬁnds an efﬁcient trainable subnetwork.
Revised: The unreasonable effectiveness of magnitude scores often extends beyond the strict domain of network pruning; a recent experiment by Frankle & Carbin (2019) suggests the existence of an automatic subnetwork discovery mechanism underlying the standard gradient-based optimization procedures of deep, overparametrized neural networks by showing that the MP algorithm ﬁnds an efﬁcient trainable subnetwork.

Original: These observations constitute a call to revisit the “magnitude-equals-saliency” approach for a better understanding of  deep neural network itself.
Revised: These observations constitute a call to revisit the “magnitude-equals-saliency” approach for a better understanding of the deep neural network itself.

Original: As an attempt to better understand the nature of MP methods, we study a generalization of magnitude scores under a functional approximation framework; by viewing MP as a relaxed minimization of distortion in layerwise operators introduced by zeroing out parameters, we consider a multi-layer extension of the distortion minimization problem.
Revised: As an attempt to better understand the nature of MP methods, we study a generalization of magnitude scores under a functional approximation framework; by viewing MP as a relaxed minimization of distortion in layerwise operators introduced by zeroing out parameters, we consider a multi-layer extension of the distortion minimization problem.

Original: Minimization of the newly suggested distortion measure which ‘looks ahead’ the impact of pruning on neighboring layers gives birth to a novel pruning strategy, coined lookahead pruning (LAP).
Revised: Minimization of the newly suggested distortion measure, which ‘looks ahead’ the impact of pruning on neighboring layers, gives birth to a novel pruning strategy, coined lookahead pruning (LAP).

Original: In this paper, we focus on  comparison of the proposed LAP scheme to its MP counterpart.
Revised: In this paper, we focus on the comparison of the proposed LAP scheme to its MP counterpart.

Original: We empirically demonstrate that LAP consistently outperformsthe MP under various setups including linear networks, fully-connected networks, and deep convolutional and residual networks.
Revised: We empirically demonstrate that LAP consistently outperforms MP under various setups, including linear networks, fully-connected networks, and deep convolutional and residual networks.

Original: In particular,the LAP consistently enables more than × 2 gain in the compression rate of the considered models, with  increasing beneﬁts under the high-sparsity regime.
Revised: In particular, LAP consistently enables more than × 2 gain in the compression rate of the considered models, with [Figure] increasing beneﬁts under the high-sparsity regime.

Original: Apart from its performance,the lookahead pruningmethod enjoys additional attractive properties:
Revised: Apart from its performance, lookahead pruning enjoys additional attractive properties:

Original: [Figure] • Easy-to-use: Like magnitude-based pruning, the proposed LAP is a simple score-based approach agnostic to model and data, which can be implemented by computationally light elementary tensor operations.
Revised: • Easy-to-use: Like magnitude-based pruning, the proposed LAP is a simple score-based approach agnostic to model and data, which can be implemented by computationally light elementary tensor operations.

Original: Unlike most Hessian-based methods, LAP does not rely on an availability of training data except for the retraining phase.
Revised: Unlike most Hessian-based methods, LAP does not rely on the availability of training data except for the retraining phase.

Original: It also has no hyper-parameter to tune, in contrast to other sophisticated training-based and optimization-based schemes.
Revised: It also has no hyper-parameter to tune, in contrast to other sophisticated training-based and optimization-based schemes.

Original: • Versatility: As our method simply replaces the “magnitude-as-saliency” criterion with a lookahead alternative, it can be deployed jointly with algorithmic tweaks developed for magnitudebased pruning, such as iterative pruning and retraining (Han et al., 2015) or joint pruning and training with dynamic reconnections (Zhu & Gupta, 2017; Gale et al., 2019).
Revised: • Versatility: As our method simply replaces the “magnitude-as-saliency” criterion with a lookahead alternative, it can be deployed jointly with algorithmic tweaks developed for magnitudebased pruning, such as iterative pruning and retraining (Han et al., 2015) or joint pruning and training with dynamic reconnections (Zhu & Gupta, 2017; Gale et al., 2019).

Original: The remainder of this manuscript is structured as follows:
Revised: The remainder of this manuscript is structured as follows:

Original: In Section 2, we introduce a functional approximation perspective toward MP and motivate LAP and its variants as a generalization of MP for multiple layer setups; in Section 3 we explore the capabilities of LAP and its variants with simple models, then move on to apply LAP to larger-scale models.
Revised: In Section 2, we introduce a functional approximation perspective toward MP and motivate LAP and its variants as a generalization of MP for multiple layer setups; in Section 3 we explore the capabilities of LAP and its variants with simple models, then move on to apply LAP to larger-scale models.

Original:  L OOKAHEAD : A FAR - SIGHTED LAYER APPROXIMATION
Revised:  L OOKAHEAD : A FAR - SIGHTED LAYER APPROXIMATION

Original: We begin by a more formal description of the magnitude-based pruning (MP)
Revised: We begin by a more formal description of the magnitude-based pruning (MP)

Original: algorithm (Han et al., 2015).
Revised: algorithm (Han et al., 2015).

Original: Given an L -layer neural network associated with weight tensors W 1 , . . .
Revised: Given an L -layer neural network associated with weight tensors W 1 , . . .

Original: , W L , the MP algorithm removes connections with  smallest absolute weights from each weight tensor, until the desired level of sparsity has been achieved.
Revised: , W L , the MP algorithm removes connections with the smallest absolute weights from each weight tensor until the desired level of sparsity has been achieved.

Original: This layerwise procedure is equivalent to ﬁnding a mask
Revised: This layerwise procedure is equivalent to ﬁnding a mask

Original: M whose entries are either 0 or 1, incurring a smallest Frobenius distortion , measured by[Figure] [Equation] [Figure]
Revised: M whose entries are either 0 or 1, incurring a smallest Frobenius distortion , measured by

Original: [Equation]
Revised: [Equation]

Original: where (cid:12) denotes the Hadamard product, (cid:107) · (cid:107) 0 denotes the entry-wise (cid:96) 0 -norm, and s is a sparsity constraint imposed by some operational criteria.
Revised: where (cid:12) denotes the Hadamard product, (cid:107) · (cid:107) 0 denotes the entrywise (cid:96) 0 -norm, and s is a sparsity constraint imposed by some operational criteria.

Original: Aiming to minimize the Frobenius distortion (Eq.
Revised: Aiming to minimize the Frobenius distortion (Eq.

Original: , the MP algorithm naturally admits a functional approximation interpretation.
Revised: , the MP algorithm naturally admits a functional approximation interpretation.

Original: For the case of a fully-connected layer, the maximal difference between the output from a pruned and an unpruned layer can be bounded as
Revised: For the case of a fully-connected layer, the maximal difference between the output from a pruned and an unpruned layer can be bounded as

Original: [Equation]
Revised: [Equation]

Original: Namely, the product of the layerwise Frobenius distortion upper bounds the output distortion of the network incurred by pruning weights.
Revised: Namely, the product of the layerwise Frobenius distortion upper bounds the output distortion of the network incurred by pruning weights.

Original: Note that this perspective on MP as a worst-case distortion minimization was already made in Dong et al.
Revised: Note that this perspective on MP as a worst-case distortion minimization was already made in Dong et al.

Original: (2017), which inspired an advent of the layerwise optimal brain surgery (L-OBS) procedure.
Revised: (2017), which inspired an advent of the layerwise optimal brain surgery (L-OBS) procedure.

Original: A similar idea holds for convolutional layers.
Revised: A similar idea holds for convolutional layers.

Original: For the case of a two-dimensional convolution with a single input and a single output channel, the corresponding linear operator takes a form of a doubly block circulant matrix constructed from the associated kernel tensor (see, e.g., Goodfellow et al.
Revised: For the case of a two-dimensional convolution with a single input and a single output channel, the corresponding linear operator takes a form of a doubly block circulant matrix constructed from the associated kernel tensor (see, e.g., Goodfellow et al.

Original: Here, the Frobenius distortion of doubly block circulant matrices can be controlled by the
Revised: Here, the Frobenius distortion of doubly block circulant matrices can be controlled by the

Original: Frobenius distortion of the weight tensor of the convolutional layer.
Revised: Frobenius distortion of the weight tensor of the convolutional layer.

Revised: do 4:

Revised: Eq all entry w

Revised: W i 5:

Revised: Set w s i a s

Original: 2. L OOKAHEAD DISTORTION AS A BLOCK APPROXIMATION ERROR
Revised: i th smallest element {L i w : w an entry of W i } 6: Set M

Revised: ←

Revised: W i

Revised: 8: end

Revised: 2. L OOKAHEAD DISTORTION AS A BLOCK APPROXIMATION ERROR

Original: The myopic optimization (Eq.
Revised: The myopic optimization (Eq.

Original: based on the per-layer
Revised: based on the per-layer

Original: Frobenius distortion falls short even in the simplest case of the two-layer linear neural network with one-dimensional output, where we consider predictors of a form (cid:98) Y = u (cid:62)
Revised: Frobenius distortion falls short even in the simplest case of the two-layer linear neural network with one-dimensional output, where we consider predictors taking form (cid:98) Y = u (cid:62)

Original: Wx and try to minimize the Frobenius distortion of u (cid:62) W (equivalent to (cid:96) 2 distortion in this case).
Revised: Wx and try to minimize the Frobenius distortion of u (cid:62) W (equivalent to (cid:96) 2 distortion in this case).

Original: Here, if u i is extremely large, pruning any nonzero element in the i -th row of W may incur a signiﬁcant Frobenius distortion.
Revised: Here, if u i is extremely large, pruning any nonzero element in the i -th row of W may incur a signiﬁcant Frobenius distortion.

Original: Motivated by this observation, we consider a block approximation analogue of the magnitude-based pruning objective Eq.
Revised: Motivated by this observation, we consider a block approximation analogue of the magnitude-based pruning objective Eq.

Original: Consider an L -layer neural networkwith associated  weight tensors
Revised: Consider an L -layer neural network associated with weight tensors

Original: W 1 , . .
Revised: W 1 , . .

Original: , W L , and assume linear activation for simplicity (will be extended to nonlinear cases later in this section).
Revised: , W L , and assume linear activation for simplicity (will be extended to nonlinear cases later in this section).

Original: Let J ( W i ) denote the Jacobian matrix corresponding to the linear operator characterized by W i .
Revised: Let J ( W i ) denote the Jacobian matrix corresponding to the linear operator characterized by W i .

Original: For pruning the i -th layer, we take into account the weight tensors of neighboring layers
Revised: For pruning the i -th layer, we take into account the weight tensors of adjacent layers

Original: W i − 1 , W i +1 in addition to the original weight tensor W i .
Revised: W i − 1 , W i +1 in addition to the original weight tensor W i .

Original: In particular, we propose to minimize the
Revised: In particular, we propose to minimize the

Original: Frobenius distortion of the operator block J ( W i
Revised: Frobenius distortion of the operator block J ( W i

Original: [Equation]
Revised: [Equation]

Original: 1 ) J ( W
Revised: 1 ) J [Equation] W

Original: [Equation]
Revised: [Equation]

Original: J ( Wi
Revised: J [Equation] W

Original: [Equation]
Revised: [Equation]

Original: ) , i.e.,
Revised: , i.e.,

Original: [Equation]
Revised: [Equation]

Original: An explicit minimization of the block distortion (Eq.
Revised: An explicit minimization of the block distortion (Eq.

Original: (3)), however, is computationally intractable in general (see Appendix D for a more detailed discussion).
Revised: (3)), however, is computationally intractable in general (see Appendix D for a more detailed discussion).

Original: To avoid an excessive computational overhead, we propose to use the following score-based pruning algorithm, coined lookahead pruning (LAP), for approximating Eq.
Revised: To avoid an excessive computational overhead, we propose to use the following score-based pruning algorithm, coined lookahead pruning (LAP), for approximating Eq.

Original: For each entry w of W i , we pruneaway the weights  with the smallest value of lookahead distortion (in a single step), deﬁned as
Revised: For each tensor W i , we prune the weights w with the smallest value of lookahead distortion (in a single step), deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where W i | w =0 denotes the tensor whose entries are equal to the entries of W i except for having zeroed out w .
Revised: where W i | w =0 denotes the tensor whose entries are equal to the entries of W i except for having zeroed out w .

Original: We let both W 0 and W L +1 to be tensors consisting of ones.
Revised: We let both W 0 and W L +1 to be tensors consisting of ones.

Original: In other words, lookahead distortion (Eq.
Revised: In other words, lookahead distortion (Eq.

Original: (4)) measures the distortion (in Frobenius norm) induced by pruning w while all other weights remain intact.
Revised: (4)) measures the distortion (in Frobenius norm) induced by pruning w while all other weights remain intact.

Original: For three-layer blocks consisting only of fully-connected layers and convolutional layers, Eq.
Revised: For three-layer blocks consisting only of fully-connected layers and convolutional layers, Eq.

Original: (4) reduces to the following compact formula: for an edge w connected to the j -th input neuron/channel and the k -th output neuron/channel of the i -th layer, where its formal derivation is presented in Appendix E.
Revised: (4) reduces to the following compact formula: for an edge w connected to the j -th input neuron/channel and the k -th output neuron/channel of the i -th layer, where its formal derivation is presented in Appendix E.

Original: [Equation]
Revised: [Equation]

Original: where | w | denotes the weight of w , W [ j, :] denotes the slice of W composed of weights connected to the j -th output neuron/channel, and W [: , k ] denotes the same for the k -th input neuron/channel.
Revised: where | w | denotes the weight of w , W [ j, :] denotes the slice of W composed of weights connected to the j -th output neuron/channel, and W [: , k ] denotes the same for the k -th input neuron/channel.

Original: In LAP, we compute the lookahead distortion for all weights, and then remove weights with  smallest distortions in a single step (as done in MP).
Revised: In LAP, we compute the lookahead distortion for all weights, and then remove weights with the smallest distortions in a single step (as done in MP).

Original: A formal description of LAP is presented in Algorithm 1.
Revised: A formal description of LAP is presented in Algorithm 1.

Original: We also note the running time of LAP is comparable with that of MP (see Appendix G).
Revised: We also note the running time of LAP is comparable with that of MP (see Appendix G).

Original: LAP on linear networks.
Revised: LAP on linear networks.

Original: To illustrate the beneﬁt of lookahead, we evaluate the performance of
Revised: To illustrate the beneﬁt of lookahead, we evaluate the performance of

Original: MP and LAP on a linear fully-connected network with a single hidden layer of 1,000 nodes, trained with  MNIST image classiﬁcation dataset.
Revised: MP and LAP on a linear fully-connected network with a single hidden layer of 1,000 nodes, trained with the MNIST image classiﬁcation dataset.

Original: Fig.
Revised: Fig.

Original: 2a and Fig.
Revised: 2a and Fig.

Original: 2b depict the test accuracy of models pruned with each methods, before and after retraining steps.
Revised: 2b depict the test accuracy of models pruned with each method, before and after retraining steps.

Original: As can be expected from the discrepancy between the minimization objectives (Eqs.
Revised: As can be expected from the discrepancy between the minimization objectives (Eqs.

Original: (1) and (3)), networks pruned with LAP outperform networks pruned with MP at every sparsity level, in terms
Revised: (1) and (3)), networks pruned with LAP outperform networks pruned with MP at every sparsity level, in terms

Original: [Figure]
Revised: [Figure]

Original: of its performance before a retraining phase.
Revised: of its performance before a retraining phase.

Original: Remarkably, we observe that test accuracy of models pruned with LAP monotonically increases from 91.2% to 92.3% as the sparsity level increases, until the fraction of surviving weights reaches 1 .
Revised: Remarkably, we observe that test accuracy of models pruned with LAP monotonically increases from 91.2% to 92.3% as the sparsity level increases, until the fraction of surviving weights reaches 1.28%.

Original: At the same sparsity level, models pruned with MP achieves only 71.9% test accuracy.
Revised: At the same sparsity level, models pruned with MP achieves only 71.9% test accuracy.

Original: We also observe that LAP leads MP at every sparsity level even after a retraining phase, with an increasing margin as we consider a higher level of sparsity.
Revised: We also observe that LAP leads MP at every sparsity level even after a retraining phase, with an increasing margin as we consider a higher level of sparsity.

Original: Understanding LAP with nonlinear activations.
Revised: Understanding LAP with nonlinear activations.

Original: Most neural network models in practice deploy nonlinear activation functions, e.g., rectiﬁed linear units (ReLU).
Revised: Most neural network models in practice deploy nonlinear activation functions, e.g., rectiﬁed linear units (ReLU).

Original: Although the lookahead distortion was originally derived using linear activation functions, LAP can also be used for nonlinear networks, as the quantity L i ( w ) remains relevant to the original block approximation point of view.
Revised: Although the lookahead distortion has been initially derived using linear activation functions, LAP can also be used for nonlinear networks, as the quantity L i ( w ) remains relevant to the original block approximation point of view.

Original: This is especially true when the network is severely over-parametrized.
Revised: This is especially true when the network is severely over-parametrized.

Original: To see this, consider a case where one aims to prune a connection in the ﬁrst layer of a two-layer fully-connected network with
Revised: To see this, consider a case where one aims to prune a connection in the ﬁrst layer of a two-layer fully-connected network with

Original: ReLU, i.e.,
Revised: ReLU, i.e.,

Original: [Equation]
Revised: [Equation]

Original: where σ ( x ) = max { 0 , x } is applied entrywise.
Revised: where σ ( x ) = max { 0 , x } is applied entrywise.

Original: Under the over-parametrized scenario, zeroing out a single weight may alter the activation pattern of connected neurons with only negligible probability, which allows one to decouple the probability of activation of each neuron from the act of pruning each connection.
Revised: Under the over-parametrized scenario, zeroing out a single weight may alter the activation pattern of connected neurons with only negligible probability, which allows one to decouple the probability of activation of each neuron from the act of pruning each connection.

Original: This enables us to approximate the root mean square distortion of the network output introduced by pruning w of W 1 by √ p k L 1 ( w ) , where k is the index of the output neuron that w is connected to, and p k denotes the probability of activation for the k -th neuron.
Revised: This enables us to approximate the root mean square distortion of the network output introduced by pruning w of W 1 by √ p k L 1 ( w ) , where k is the index of the output neuron that w is connected to, and p k denotes the probability of activation for the k -th neuron.

Original: In this sense,
Revised: In this sense,

Original: LAP (Algorithm 1) can be understood as assuming i.i.d.
Revised: LAP (Algorithm 1) can be understood as assuming i.i.d.

Original: activations of neurons, due to a lack ofan additional access to training data.
Revised: activations of neurons, due to a lack of additional access to training data.

Original: In other word, LAP admits a natural extension to the regime where we assumean additional access to training data during the pruning phase.
Revised: In other words, LAP admits a natural extension to the regime where we assume additional access to training data during the pruning phase.

Original: This variant, coined
Revised: This variant, coined

Original: LAP-act, will be formally described in Appendix F, with experimental comparisons to another datadependent baseline of optimal brain damage (OBD) (LeCun et al., 1989).
Revised: LAP-act, will be formally described in Appendix F, with experimental comparisons to another datadependent baseline of optimal brain damage (OBD) (LeCun et al., 1989).

Original: Another theoretical justiﬁcation of using the lookahead distortion (Eq.
Revised: Another theoretical justiﬁcation of using the lookahead distortion (Eq.

Original: (5)) for neural networks with nonlinear activation functions comes from recent discoveries regarding the implicit bias imposed by training via stochastic gradient descent (Du et al., 2018).
Revised: (5)) for neural networks with nonlinear activation functions comes from recent discoveries regarding the implicit bias imposed by training via stochastic gradient descent (Du et al., 2018).

Original: See Appendix M for a detailed discussion.
Revised: See Appendix M for a detailed discussion.

Original: As will be empiricically shown in Section 3.1, LAP is an effective pruning strategy for sigmoids and tanh activations, that are not piece-wise linear as ReLU.
Revised: As will be empirically shown in Section 3.1, LAP is an effective pruning strategy for sigmoids and tanh activations, that are not piece-wise linear as ReLU.

Original: 2. L OOKAHEAD PRUNING WITH BATCH NORMALIZATION
Revised: 2. L OOKAHEAD PRUNING WITH BATCH NORMALIZATION

Original: Batch normalization (BN), introduced by Ioffe & Szegedy (2015), aims to normalize the output of a layer per batch by scaling and shifting the outputs with trainable parameters.
Revised: Batch normalization (BN), introduced by Ioffe & Szegedy (2015), aims to normalize the output of a layer per batch by scaling and shifting the outputs with trainable parameters.

Original: Based on our functional approximation perspective, having batch normalization layers in a neural network is not an issue for
Revised: Based on our functional approximation perspective, having batch normalization layers in a neural network is not an issue for

Original: MP, which relies on the magnitudes of weights; batch normalization only affects the distribution of the input for each layer, not the layer itself.
Revised: MP, which relies on the magnitudes of weights; batch normalization only affects the distribution of the input for each layer, not the layer itself.

Original: On the other hand, as the lookahead distortion (Eq.
Revised: On the other hand, as the lookahead distortion (Eq.

Original: (3)) characterizes the distortion of the multi-layer block, one must take into account batch normalization when assessing the abstract importance of each connection.
Revised: (3)) characterizes the distortion of the multi-layer block, one must take into account batch normalization when assessing the abstract importance of each connection.

Original: The revision of lookahead pruning under the presence of batch normalization can be done fairly simply.
Revised: The revision of lookahead pruning under the presence of batch normalization can be done fairly simply.

Original: Note that such a normalization process can be expressed as
Revised: Note that such a normalization process can be expressed as

Original: [Equation]
Revised: [Equation]

Original: for some a, b ∈ R dim ( x ) .
Revised: for some a, b ∈ R dim ( x ) .

Original: Hence, we revisethe lookahead pruning to prune the connections with a minimum value of
Revised: Hence, we revise lookahead pruning to prune the connections with a minimum value of

Original: [Equation]
Revised: [Equation]

Original: where a i [ k ] denotes the k -th index scaling factor for the BN layer placed at the output of the i -th fully-connected or convolutional layer (if BN layer does not exist, let a i [ k ] = 1 ).
Revised: where a i [ k ] denotes the k -th index scaling factor for the BN layer placed at the output of the i -th fully-connected or convolutional layer (if BN layer does not exist, let a i [ k ] = 1 ).

Original: This modiﬁcation of LAP makes it an efﬁcient pruning strategy, as will be empirically veriﬁed in Section 3.3.
Revised: This modiﬁcation of LAP makes it an efﬁcient pruning strategy, as will be empirically veriﬁed in Section 3.3.

Original: 2. V ARIANTS OF LOOKAHEAD PRUNING
Revised: 2. V ARIANTS OF LOOKAHEAD PRUNING

Original: As the LAP algorithm (Algorithm 1) takes into account current states of the neighboring layers,
Revised: As the LAP algorithm (Algorithm 1) takes into account current states of the neighboring layers,

Original: LAP admits several variants in terms of lookahead direction,  order of pruning, and sequential pruning methods; these methods are extensively studied in Section 3.2 Along with “vanilla” LAP, we consider in total, six variants, which we now describe below:
Revised: LAP admits several variants in terms of lookahead direction, the order of pruning, and sequential pruning methods; these methods are extensively studied in Section 3.2. Along with “vanilla” LAP, we consider in total, six variants, which we now describe below:

Original: Mono-directional LAPs.
Revised: Mono-directional LAPs.

Original: To prune a layer, LAP considers both preceding and succeeding layers.
Revised: To prune a layer, LAP considers both preceding and succeeding layers.

Original: Looking forward, i.e., only considering the succeeding layer, can be viewed as an educated modiﬁcation of the internal representation the present layer produces.
Revised: Looking forward, i.e., only considering the succeeding layer, can be viewed as an educated modiﬁcation of the internal representation the present layer produces.

Original: Looking backward, on the other hand, can be interpreted as only taking into account the expected structure of input coming into the present layer.
Revised: Looking backward, on the other hand, can be interpreted as only taking into account the expected structure of input coming into the current layer.

Original: The corresponding variants, coined LFP and LBP, are tested.
Revised: The corresponding variants, coined LFP and LBP, are tested.

Original: Order of pruning.
Revised: Order of pruning.

Original: Instead of using the unpruned tensors of preceding/succeeding layers, we also consider performing LAP  onthe basis of already-pruned layers.
Revised: Instead of using the unpruned tensors of preceding/succeeding layers, we also consider performing LAP based on already-pruned layers.

Original: This observation brings up a question of the order of pruning; an option is to prune in a forward direction, i.e., prune the preceding layer ﬁrst and use the pruned weight to prune the succeeding, and the other is to prune backward.
Revised: This observation brings up a question of the order of pruning; an option is to prune in a forward direction, i.e., prune the preceding layer ﬁrst and use the pruned weight to prune the succeeding, and the other is to prune backward.

Original: Both methods are tested, which are referred to as LAP-forward and LAP-backward, respectively.
Revised: Both methods are tested, which are referred to as LAP-forward and LAP-backward, respectively.

Original: Sequential pruning.
Revised: Sequential pruning.

Original: We also consider a sequential version of LAP-forward/backward methods.
Revised: We also consider a sequential version of LAP-forward/backward methods.

Original: More speciﬁcally, if we aim to prune total p % of weights from each layer, we divide the pruning budget into ﬁve pruning steps and gradually prune ( p/ 5) % of the weights per step in forward/backward direction.
Revised: More speciﬁcally, if we aim to prune total p % of weights from each layer, we divide the pruning budget into ﬁve pruning steps and gradually prune ( p/ 5) % of the weights per step in forward/backward direction.

Original: Sequential variants will be marked with a sufﬁx “-seq”.
Revised: Sequential variants will be marked with a sufﬁx “-seq”.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we compare the empirical performance of LAP with that of MP.
Revised: In this section, we compare the empirical performance of LAP with that of MP.

Original: More speciﬁcally, we validate the applicability of LAP to nonlinear activation functions in Section 3.1.
Revised: More speciﬁcally, we validate the applicability of LAP to nonlinear activation functions in Section 3.1.

Original: In Section 3.2, we test LAP variants from Section 2.3.
Revised: In Section 3.2, we test LAP variants from Section 2.3.

Original: In Section 3.3, we test LAP on VGG (Simonyan & Zisserman, 2015), ResNet (He et al., 2016), and Wide ResNet (WRN, Zagoruyko & Komodakis (2016)).
Revised: In Section 3.3, we test LAP on VGG (Simonyan & Zisserman, 2015), ResNet (He et al., 2016), and Wide ResNet (WRN, Zagoruyko & Komodakis (2016)).

Original: Experiment setup.
Revised: Experiment setup.

Original: We consider ﬁve neural network architectures: (1) The fully-connected network (FCN) under consideration is consist of four hidden layers, each with 500 neurons.
Revised: We consider ﬁve neural network architectures: (1) The fully-connected network (FCN) under consideration is consist of four hidden layers, each with 500 neurons.

Original: The convolutional network (Conv-6) consists of six convolutional layers, followed by a fully-connected classiﬁer with two hidden layers with 256 neurons each; this model is identical to that appearing in the work of Frankle & Carbin (2019) suggested as a scaled-down variant of VGG.
Revised: The convolutional network (Conv-6) consists of six convolutional layers, followed by a fully-connected classiﬁer with two hidden layers with 256 neurons each; this model is identical to that appearing in the work of Frankle & Carbin (2019) suggested as a scaled-down variant of VGG.

Original: 2 (3) VGG-19 is used, with an addition of batch normalization layers after each convolutional layers, and a reduced number of fully-connected layers from three to one.
Revised: 2 (3) VGG-19 is used, with an addition of batch normalization layers after each convolutional layers, and a reduced number of fully-connected layers from three to one.

Original: 3 (4) ResNets of depths { 18 , 50 } are used.
Revised: 3 (4) ResNets of depths { 18 , 50 } are used.

Original: (5) WRN of 16 convolutional layers and widening factor 8 (WRN-16-8) is used.
Revised: (5) WRN of 16 convolutional layers and widening factor 8 (WRN-16-8) is used.

Original: All networks used
Revised: All networks used

Original: ReLU activation function, except for the experiments in Section 3.1.
Revised: ReLU activation function, except for the experiments in Section 3.1.

Original: We mainly consider image classiﬁcation tasks.
Revised: We mainly consider image classiﬁcation tasks.

Original: In particular, FCN is trained on MNIST dataset (Lecun et al., 1998), Conv-6, VGG, and ResNet are trained on CIFAR-10 dataset (Krizhevsky & Hinton, 2009), and VGG, ResNet, and
Revised: In particular, FCN is trained on MNIST dataset (Lecun et al., 1998), Conv-6, VGG, and ResNet are trained on CIFAR-10 dataset (Krizhevsky & Hinton, 2009), and VGG, ResNet, and

Original: WRN are trained on Tiny-ImageNet.
Revised: WRN are trained on Tiny-ImageNet.

Original: 4 We focus on the one-shot pruning of MP and LAP, i.e., models are trained with a single training-pruning-retraining cycle.
Revised: 4 We focus on the one-shot pruning of MP and LAP, i.e., models are trained with a single training-pruning-retraining cycle.

Original: All results in this section are averaged over ﬁve independent trials.
Revised: All results in this section are averaged over ﬁve independent trials.

Original: We provide more details on setups in Appendix A.
Revised: We provide more details on setups in Appendix A.

Original: [Figure]
Revised: [Figure]

Original: 3. N ETWORKS WITH NONLINEAR ACTIVATION FUNCTIONS
Revised: 3. N ETWORKS WITH NONLINEAR ACTIVATION FUNCTIONS

Original: We ﬁrst compare the performance of LAP with that of MP on FCN using three different types of activation functions: sigmoid, and tanh, and ReLU.
Revised: We ﬁrst compare the performance of LAP with that of MP on FCN using three different types of activation functions: sigmoid, and tanh, and ReLU.

Original: Figs.
Revised: Figs.

Original: 3a to 3c depict the performance of models pruned with LAP (Green) and MP (Red) under various levels of sparsity.
Revised: 3a to 3c depict the performance of models pruned with LAP (Green) and MP (Red) under various levels of sparsity.

Original: Although LAP was motivated primarily from linear networks and partially justiﬁed for positivehomogenous activation functions such as ReLU, the experimental results show that LAP consistently outperforms MP even on networks using sigmoidal activation functions.
Revised: Although LAP was motivated primarily from linear networks and partially justiﬁed for positivehomogenous activation functions such as ReLU, the experimental results show that LAP consistently outperforms MP even on networks using sigmoidal activation functions.

Original: We remark that LAP outperforms MP by a larger margin as fewer weights survive (less than 1 %).
Revised: We remark that LAP outperforms MP by a larger margin as fewer weights survive (less than 1 %).

Original: Such a pattern will be observed repeatedly in the remaining experiments of this paper.
Revised: Such a pattern will be observed repeatedly in the remaining experiments of this paper.

Original: In addition, we also check whether LAP still exhibits better test accuracy before retraining under the usage of nonlinear activation functions, as in the linear network case (Fig. 2b).
Revised: In addition, we also check whether LAP still exhibits better test accuracy before retraining under the usage of nonlinear activation functions, as in the linear network case (Fig. 2b).

Original: Fig.
Revised: Fig.

Original: 3d illustrates the test accuracy of pruned FCN using ReLU on  MNIST dataset before retraining.
Revised: 3d illustrates the test accuracy of pruned FCN using ReLU on the MNIST dataset before retraining.

Original: We observe that the network pruned by LAP continues to perform better than MP in this case; the network pruned by LAP retains the original test accuracy until only 38 % of the weights survive, and shows less than 1 % performance drop with only 20 % of the weights remaining.
Revised: We observe that the network pruned by LAP continues to perform better than MP in this case; the network pruned by LAP retains the original test accuracy until only 38% of the weights survive, and shows less than 1% performance drop with only 20% of the weights remaining.

Original: On the other hand, MP requires 54 % and 30 % to achieve the same level of performance, respectively.
Revised: On the other hand, MP requires 54% and 30% to achieve the same level of performance, respectively.

Original: In other words, the models pruned with MP requires about 50 % more survived parameters than the models pruned with LAP to achieve a similar level of performance before being retrained using additional training batches.
Revised: In other words, the models pruned with MP requires about 50% more survived parameters than the models pruned with LAP to achieve a similar level of performance before being retrained using additional training batches.

Original: 3. E VALUATING LAP VARIANTS
Revised: 3. E VALUATING LAP VARIANTS

Original: Now we evaluate LAP and its variants introduced in Section 2.3 on FCN and Conv-6, each trained on MNIST and CIFAR-10, respectively.
Revised: Now we evaluate LAP and its variants introduced in Section 2.3 on FCN and Conv-6, each trained on MNIST and CIFAR-10, respectively.

Original: Table 1 summarizes the experimental results on FCN and
Revised: Table 1 summarizes the experimental results on FCN and

Original: Table 2 summarizes the results on Conv-6.
Revised: Table 2 summarizes the results on Conv-6.

Original: In addition to the baseline comparison with MP, we also compare with random pruning (RP), where the connection to be pruned was decided completely independently.
Revised: In addition to the baseline comparison with MP, we also compare with random pruning (RP), where the connection to be pruned was decided completely independently.

Original: We observe that LAP performs consistently better than MP and RP with similar or smaller variance in any case.
Revised: We observe that LAP performs consistently better than MP and RP with similar or smaller variance in any case.

Original: In the case of an extreme sparsity, LAP enjoys a signiﬁcant performance gain; over 75 % gain on FCN and 14 % on Conv-6.
Revised: In the case of an extreme sparsity, LAP enjoys a signiﬁcant performance gain; over 75% gain on FCN and 14% on Conv-6.

Original: This performance gain comes from a better training accuracy, instead of a better generalization; see Appendix L for more information.
Revised: This performance gain comes from a better training accuracy, instead of a better generalization; see Appendix L for more information.

Original: Comparing mono-directional lookahead variants, we observe that LFP performs better than LBP in the low-sparsity regime, while LBP performs better in the high-sparsity regime; in any case,
Revised: Comparing mono-directional lookahead variants, we observe that LFP performs better than LBP in the low-sparsity regime, while LBP performs better in the high-sparsity regime; in any case,

Original: LAP performed better than both methods.
Revised: LAP performed better than both methods.

Original: Intriguingly, the same pattern appeared in the case of the ordered pruning.
Revised: Intriguingly, the same pattern appeared in the case of the ordered pruning.

Original: Here, LAP-forward can be considered an analogue of LBP in the sense that they both consider layers closer to the input to be more important.
Revised: Here, LAP-forward can be considered an analogue of LBP in the sense that they both consider layers closer to the input to be more critical.

Original: Likewise, LAP-backward can be considered an analogue of LFP.
Revised: Likewise, LAP-backward can be considered an analogue of LFP.

Original: We observe that LAP-forward performs better than LAP-backward in the high-sparsity regime, and vice versa in the low-sparsity regime.
Revised: We observe that LAP-forward performs better than LAP-backward in the high-sparsity regime, and vice versa in the low-sparsity regime.

Original: Our interpretation is as follows:
Revised: Our interpretation is as follows:

Original: Whenever the sparsity level is low,the importance of a carefully curating the input signal is not signiﬁcant due to high redundancies in  natural image signal.
Revised: Whenever the sparsity level is low, carefully curating the input signal is not important due to high redundancies in the natural image signal.

Original: This causes a relatively low margin of increment by looking backward in comparison to looking forward.
Revised: This causes a relatively low margin of increment by looking backward in comparison to looking forward.

Original: When the sparsity level is high, the input signal is scarce, and the relative importance of preserving the input signal is higher.
Revised: When the sparsity level is high, the input signal is scarce, and the relative importance of preserving the input signal is higher.

Original: Finally, we observe that employing forward/backward ordering and sequential methods leads toa better performance, especially in the high-sparsity regime.
Revised: Finally, we observe that employing forward/backward ordering and sequential methods leads to better performance, especially in the high-sparsity regime.

Original: There is no clear beneﬁt of adopting directional methods in the low-sparsity regime.
Revised: There is no clear beneﬁt of adopting directional methods in the low-sparsity regime.

Original: The relative gain in performance with respect to
Revised: The relative gain in performance with respect to

Original: LAP is either marginal, or unreliable.
Revised: LAP is either marginal or unreliable.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 3. D EEPER NETWORKS : VGG, R ES N ET , AND WRN
Revised: 3. D EEPER NETWORKS : VGG, R ES N ET , AND WRN

Original: We also compare empirical performances of MP with LAP on deeper networks.
Revised: We also compare empirical performances of MP with LAP on deeper networks.

Original: We trained VGGand ResNet-18 on CIFAR-10 (Tables 3 and 4), and VGG-19, ResNet-50, and WRN-16-8 on TinyImageNet (Tables 5 to 7).
Revised: We trained VGGand ResNet-18 on CIFAR-10 (Tables 3 and 4), and VGG-19, ResNet-50, and WRN-16-8 on TinyImageNet (Tables 5 to 7).

Original: For models trained on CIFAR-10, we also test LAP-forward to verify the observation that it outperforms LAP in the high-sparsity regime on such deeper models.
Revised: For models trained on CIFAR-10, we also test LAP-forward to verify the observation that it outperforms LAP in the high-sparsity regime on such deeper models.

Original: We also report additional experimental results on VGG- { 11 , 16 } trained on CIFAR-10 in Appendix B. For models trained on Tiny-ImageNet, top-1 error rates are reported in Appendix C.
Revised: We also report additional experimental results on VGG- { 11 , 16 } trained on CIFAR-10 in Appendix B. For models trained on Tiny-ImageNet, top-1 error rates are reported in Appendix C.

Original: From Tables 3 to 7, we make the following two observations: First, as in Section 3.2, the models pruned with LAP consistently achieve a higher or similar level of accuracy compared to models pruned with MP, at all sparsity levels.
Revised: From Tables 3 to 7, we make the following two observations: First, as in Section 3.2, the models pruned with LAP consistently achieve a higher or similar level of accuracy compared to models pruned with MP, at all sparsity levels.

Original: In particular, test accuracies tend to decay at a much slower rate with LAP.
Revised: In particular, test accuracies tend to decay at a much slower rate with LAP.

Original: In Table 3, for instance, we observe that the models pruned by LAP retain test accuracies of 70 ∼ 80% even with less than 2% of weights remaining.
Revised: In Table 3, for instance, we observe that the models pruned by LAP retain test accuracies of 70 ∼ 80% even with less than 2% of weights remaining.

Original: In contrast, the performance of models pruned with MP falls drastically, to below 30% accuracy.
Revised: In contrast, the performance of models pruned with MP falls drastically, to below 30% accuracy.

Original: This observation is consistent on both CIFAR-10 and Tiny-ImageNet datasets.
Revised: This observation is consistent on both CIFAR-10 and Tiny-ImageNet datasets.

Original: Second, the advantages of considering an ordered pruning method (LAP-forward) over LAP is limited.
Revised: Second, the advantages of considering an ordered pruning method (LAP-forward) over LAP is limited.

Original: While we observe from Table 3 that LAP-forward outperforms both MP and LAP in the highsparsity regime, the gain is marginal considering standard deviations.
Revised: While we observe from Table 3 that LAP-forward outperforms both MP and LAP in the highsparsity regime, the gain is marginal considering standard deviations.

Original: LAP-forward is consistently worse than LAP (by at most 1% in absolute scale) in the low-sparsity regime.
Revised: LAP-forward is consistently worse than LAP (by at most 1% in absolute scale) in the low-sparsity regime.

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we interpret magnitude-based pruning as a solution to the minimization of the Frobenius distortion of a single layer operation incurred by pruning.
Revised: In this work, we interpret magnitude-based pruning as a solution to the minimization of the Frobenius distortion of a single layer operation incurred by pruning.

Original: Based on this framework, we consider the minimization of the Frobenius distortion of multi-layer operation, and propose a novel lookahead pruning (LAP) scheme as a computationally efﬁcient algorithm to solve the optimization.
Revised: Based on this framework, we consider the minimization of the Frobenius distortion of multi-layer operation, and propose a novel lookahead pruning (LAP) scheme as a computationally efﬁcient algorithm to solve the optimization.

Original: Although
Revised: Although

Original: LAP was motivated from linear networks, it extends to nonlinear networks which indeed minimizes the root mean square lookahead distortion assuming i.i.d.
Revised: LAP was motivated from linear networks, it extends to nonlinear networks which indeed minimizes the root mean square lookahead distortion assuming i.i.d.

Original: activations.
Revised: activations.

Original: We empirically show its effectiveness on networks with nonlinear activation functions, and test the algorithm on various network architectures including VGG, ResNet and WRN, where LAP consistently performs better than MP. 
Revised: We empirically show its effectiveness on networks with nonlinear activation functions, and test the algorithm on various network architectures including VGG, ResNet and WRN, where LAP consistently performs better than MP. Acknowledgments.

Revised: We thank Seunghyun Lee for providing helpful feedbacks and suggestions in preparing the early version of the manuscript.

Revised: JL also gratefully acknowledges Jungseul Ok and Phillip M. Long for enlightening discussions about theoretical natures of neural network pruning.

Revised: This research was supported by the Engineering Research Center Program through the National Research Foundation of Korea (NRF), funded by the Korean Government MSIT (NRF2018R1A5A1059921).

Original: A E XPERIMENTAL SETUP
Revised: A E XPERIMENTAL SETUP

Original: Models and datasets.
Revised: Models and datasets.

Original: We consider four neural network architectures: (1) The fully-connected network (FCN) under consideration is composed of four hidden layers, each with 500 hidden neurons.
Revised: We consider four neural network architectures: (1) The fully-connected network (FCN) under consideration is composed of four hidden layers, each with 500 hidden neurons.

Original: The convolutional network (Conv-6) consists of six convolutional layers, followed by a fully-connected classiﬁer with two hidden layers with 256 hidden neurons each; this model is identical to that appearing in the work of Frankle & Carbin (2019) suggested as a scaled-down variant of
Revised: The convolutional network (Conv-6) consists of six convolutional layers, followed by a fully-connected classiﬁer with two hidden layers with 256 hidden neurons each; this model is identical to that appearing in the work of Frankle & Carbin (2019) suggested as a scaled-down variant of

Original: VGG.
Revised: VGG.

Original: 5 (3) VGGs of depths { 11 , 16 , 19 } were used, with an addition of batch normalization layers after each convolutional layers, and a reduced number of fully-connected layers from three to one.(4) ResNets with depth { 18 , 50 } are used.
Revised: 5 (3) VGGs of depths { 11 , 16 , 19 } were used, with an addition of batch normalization layers after each convolutional layers, and a reduced number of fully-connected layers from three to one.(4) ResNets with depth { 18 , 50 } are used.

Original: (5) Wide ResNets with depth 16 and widening factoris used.
Revised: (5) Wide ResNets with depth 16 and widening factoris used.

Original: All networks are initialized via the method of Glorot & Bengio (2010), except for ResNets and WRN.
Revised: All networks are initialized via the method of Glorot & Bengio (2010), except for ResNets and WRN.

Original: We use the ReLU activation function except for the experiments in Section 3.1.
Revised: We use the ReLU activation function except for the experiments in Section 3.1.

Original: We focus on image classiﬁcation tasks.
Revised: We focus on image classiﬁcation tasks.

Original: FCN is trained with MNIST dataset
Revised: FCN is trained with MNIST dataset

Original: (Lecun et al., 1998), Conv-6,
Revised: (Lecun et al., 1998), Conv-6,

Original: VGG- { 11 , 16 , 19 } and ResNet-18 are trained with CIFAR-10 dataset (Krizhevsky & Hinton, 2009), and VGG-19, ResNet-50, WRN-16-8 ware trained with Tiny-ImageNet dataset.
Revised: VGG- { 11 , 16 , 19 } and ResNet-18 are trained with CIFAR-10 dataset (Krizhevsky & Hinton, 2009), and VGG-19, ResNet-50, WRN-16-8 ware trained with Tiny-ImageNet dataset.

Original: Optimizers and hyperparameters.
Revised: Optimizers and hyperparameters.

Original: We use Adam optimizer (Kingma & Ba, 2015) with batch size 60 .
Revised: We use Adam optimizer (Kingma & Ba, 2015) with batch size 60 .

Original: We use a learning rate of 1 . 2 · 10 − 3 for FCN and 3 · 10 − 4 for all other models.
Revised: We use a learning rate of 1 . 2 · 10 − 3 for FCN and 3 · 10 − 4 for all other models.

Original: For FCN, we use [50k, 50k] for the initial training phase and retraining phase.
Revised: For FCN, we use [50k, 50k] for the initial training phase and retraining phase.

Original: For Conv-6, we use [30k, 20k] steps.
Revised: For Conv-6, we use [30k, 20k] steps.

Original: For VGG-11 and ResNet-18, we use [35k, 25k] steps.
Revised: For VGG-11 and ResNet-18, we use [35k, 25k] steps.

Original: For VGG-16, we use [50k, 35k].
Revised: For VGG-16, we use [50k, 35k].

Original: For
Revised: For

Original: VGG-19, ResNet-50, and WRN-16-8 we use [60k, 40k].
Revised: VGG-19, ResNet-50, and WRN-16-8 we use [60k, 40k].

Original: We do not use any weight decay, learning rate scheduling, or regularization.
Revised: We do not use any weight decay, learning rate scheduling, or regularization.

Original: Sparsity levels.
Revised: Sparsity levels.

Original: To determine the layerwise pruning ratio, we largely follow the the guidelines of
Revised: To determine the layerwise pruning ratio, we largely follow the the guidelines of

Original: Han et al. (2015); Frankle & Carbin (2019):
Revised: Han et al. (2015); Frankle & Carbin (2019):

Original: For integer values of τ , we keep p τ fraction of weights in all convolutional layers and q τ fraction in all fully-connected layers, except for the last layer where we use (1 + q ) / 2 instead.
Revised: For integer values of τ , we keep p τ fraction of weights in all convolutional layers and q τ fraction in all fully-connected layers, except for the last layer where we use (1 + q ) / 2 instead.

Original: For FCN, we use ( p, q ) =
Revised: For FCN, we use ( p, q ) =

Original: For Conv-6, VGGs ResNets, and
Revised: For Conv-6, VGGs ResNets, and

Original: WRN, we use (0 . 85 , 0 . 8) .
Revised: WRN, we use (0 . 85 , 0 . 8) .

Original: For ResNet- { 18 , 50 } , we do not prune the ﬁrst convolutional layer.
Revised: For ResNet- { 18 , 50 } , we do not prune the ﬁrst convolutional layer.

Original: The range of sparsity for reported ﬁgures in all tables is decided as follows: we start from τ where test error rate starts falling below that of an unpruned model and report the results at τ, τ + 1 , τ + 2 , . . .
Revised: The range of sparsity for reported ﬁgures in all tables is decided as follows: we start from τ where test error rate starts falling below that of an unpruned model and report the results at τ, τ + 1 , τ + 2 , . . .

Original: for FCN and Conv-6, τ, τ + 2 , τ + 4 , . . .
Revised: for FCN and Conv-6, τ, τ + 2 , τ + 4 , . . .

Original: for VGGs, ResNet-50, and WRN, and τ, τ + 3 , τ + 6 , . . .
Revised: for VGGs, ResNet-50, and WRN, and τ, τ + 3 , τ + 6 , . . .

Original: for ResNet-18.
Revised: for ResNet-18.

Original: B A DDITIONAL VGG EXPERIMENTS
Revised: B A DDITIONAL VGG EXPERIMENTS

Original: [Table] [Table]
Revised: [Table] [Table]

Original: C T OP -1 ERROR RATES FOR T INY -I MAGE N ET EXPERIMENTS
Revised: C T OP -1 ERROR RATES FOR T INY -I MAGE N ET EXPERIMENTS

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]

Original: [Table]
Revised: [Table]

Original: D NP-H ARDNESS OF E Q . (3)
Revised: D NP-H ARDNESS OF E Q . (3)

Original: In this section, we show that the optimization in Eq.
Revised: In this section, we show that the optimization in Eq.

Original: (3) is NP-hard by showing the reduction from the following binary quadratic programming which is NP-hard (Murty & Kabadi, 1987):
Revised: (3) is NP-hard by showing the reduction from the following binary quadratic programming which is NP-hard (Murty & Kabadi, 1987):

Original: [Equation]
Revised: [Equation]

Original: for some symmetric matrix A ∈ R n × n .
Revised: for some symmetric matrix A ∈ R n × n .

Original: Without loss of generality, we assume that the minimum eigenvalue of A (denoted with λ ) is negative; if not, Eq.
Revised: Without loss of generality, we assume that the minimum eigenvalue of A (denoted with λ ) is negative; if not, Eq.

Original: (9) admits a trivial solution x = (0 , . .
Revised: (9) admits a trivial solution x = (0 , . .

Original: Assuming λ < 0 , Eq.
Revised: Assuming λ < 0 , Eq.

Original: (9) can be reformulated as:
Revised: (9) can be reformulated as:

Original: [Equation]
Revised: [Equation]

Original: where H =
Revised: where H =

Original: A − λI .
Revised: A − λI .

Original: Here, one can easily observe that the above optimization can be solved by solving the below optimization for s = 1 , . . . , n
Revised: Here, one can easily observe that the above optimization can be solved by solving the below optimization for s = 1 , . . . , n

Original: [Equation]
Revised: [Equation]

Original: Finally, we introduce the below equality
Revised: Finally, we introduce the below equality

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation] 
Revised: [Equation] [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: ) where 1 denotes a vector of ones, U is a matrix consisting of the eigenvectors of H as its column vectors, and Λ is a diagonal matrix with corresponding (positive) eigenvalues of H as its diagonal elements.
Revised: ) where 1 denotes a vector of ones, U is a matrix consisting of the eigenvectors of H as its column vectors, and Λ is a diagonal matrix with corresponding (positive) eigenvalues of H as its diagonal elements.

Original: The above equality shows that Eq.
Revised: The above equality shows that Eq.

Original: (11) is a special case of Eq.
Revised: (11) is a special case of Eq.

Original: (3) by choosing W 1 = √ Λ U (cid:62) , W 2 = 1 , W 3 = 1 and M = 1 − x .
Revised: (3) by choosing W 1 = √ Λ U (cid:62) , W 2 = 1 , W 3 = 1 and M = 1 − x .

Original: This completes the reduction from Eq.
Revised: This completes the reduction from Eq.

Original: (9) to Eq.
Revised: (9) to Eq.

Original: E D ERIVATION OF E Q . (5)
Revised: E D ERIVATION OF E Q . (5)

Original: In this section, we provide a derivation of Eq.
Revised: In this section, we provide a derivation of Eq.

Original: (5) for the fully-connected layers.
Revised: (5) for the fully-connected layers.

Original: The convolutional layers can be handled similarly by substituting the multiplications in Eqs.
Revised: The convolutional layers can be handled similarly by substituting the multiplications in Eqs.

Original: (16) and (17) by the convolutions.
Revised: (16) and (17) by the convolutions.

Original: The Jacobian matrix of the linear operator correponding to a fully-connected layer is the weight matrix itself, i.e. J ( W i )
Revised: The Jacobian matrix of the linear operator correponding to a fully-connected layer is the weight matrix itself, i.e. J ( W i )

Original: =
Revised: =

Original: W i .
Revised: W i .

Original: From this, lookahead distortion can be reformulated as
Revised: From this, lookahead distortion can be reformulated as

Original: [Equation]
Revised: [Equation]

Original: Now, we decompose the matrix product
Revised: Now, we decompose the matrix product

Original: W i +

Revised: W i +

Original: W i W i − 1 in terms of entries of W i as below:
Revised: W i W i − 1 in terms of entries of W i as below:

Original: [Equation]
Revised: [Equation]

Original: where W i [ k, j ] , W i +

Revised: where W i [ k, j ] , W i +

Original: [: , k ] , and W i − 1 [ j, :] denote ( j, k ) -th element of W i , k -th column of W i +1 , and j -th row of W i − 1 , respectively.
Revised: [: , k ] , and W i − 1 [ j, :] denote ( j, k ) -th element of W i , k -th column of W i +1 , and j -th row of W i − 1 , respectively.

Original: The contribution of a single entry w :=
Revised: The contribution of a single entry w :=

Original: W i [ k, j ] to the product
Revised: W i [ k, j ] to the product

Original: W i +

Revised: W i +

Original: W i W i − 1 is equivalent to w · W i +

Revised: W i W i − 1 is equivalent to w · W i +

Original: [: , k ] W i − 1 [ j, :] .
Revised: [: , k ] W i − 1 [ j, :] .

Original: Therefore, in terms of the Frobenius distortion, we conclude that
Revised: Therefore, in terms of the Frobenius distortion, we conclude that

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Revised: j

Revised: [Equation]

Revised: [Equation]

Original: which completes the derivation of Eq.
Revised: which completes the derivation of Eq.

Original: (5) for fully-connected layers.
Revised: (5) for fully-connected layers.

Original: F LAP- ACT : IMPROVING LAP USING TRAINING DATA
Revised: F LAP- ACT : IMPROVING LAP USING TRAINING DATA

Original: Recall two observations made from the example of two-layer fully connected network with ReLU activation appearing in Section 2.1: LAP is designed to reﬂect the lack of knowledge about the training data at the pruning phase; once the activation probability of each neuron can be estimated, it is possible to reﬁne LAP to account for this information.
Revised: Recall two observations made from the example of two-layer fully connected network with ReLU activation appearing in Section 2.1: LAP is designed to reﬂect the lack of knowledge about the training data at the pruning phase; once the activation probability of each neuron can be estimated, it is possible to reﬁne LAP to account for this information.

Original: In this section, we continue our discussion on the second observation.
Revised: In this section, we continue our discussion on the second observation.

Original: In particular, we study an extension of LAP called lookahead pruning with activation (LAP-act) which prunes the weight with smallest value of
Revised: In particular, we study an extension of LAP called lookahead pruning with activation (LAP-act) which prunes the weight with smallest value of

Original: [Equation]
Revised: [Equation]

Original: Here, (cid:99) W i is a scaled version of W i and (cid:98) w is the corresponding scaled value of w , deﬁned by
Revised: Here, (cid:99) W i is a scaled version of W i and (cid:98) w is the corresponding scaled value of w , deﬁned by

Original: [Equation]
Revised: [Equation]

Original: where I ij denotes the set of output indices in the j -th output neuron/channel of i -th layer (for fully connected layers, this is a singleton).
Revised: where I i,j denotes the set of ReLU indices in the j -th output neuron/channel of i -th layer. For example, I i,j = { j } for fully connected layers and I i,j is a set of ReLU indices in the j -th channel for convolutional layers.

Original: Also, p k denotes the neuron’s probability of activation, which can be estimated by passing the training data.
Revised: Also, p k denotes the k -th ReLU’s probability of activation, which can be estimated by passing the training data.

Original: We derive LAP-act (Eq.
Revised: We derive LAP-act (Eq.

Original: (18)) in Appendix F.1 and perform preliminary empirical validations in
Revised: (18)) in Appendix F.1 and perform preliminary empirical validations in

Original: Appendix F.2 with using optimal brain damage (OBD) as a baseline.
Revised: Appendix F.2 with using optimal brain damage (OBD) as a baseline.

Original: We also evaluate a variant of LAP using Hessian scores of OBD instead of magnitude scores.
Revised: We also evaluate a variant of LAP using Hessian scores of OBD instead of magnitude scores.

Original: It turns out that in the small networks (FCN, Conv-6), LAP-act outperforms OBD.
Revised: It turns out that in the small networks (FCN, Conv-6), LAP-act outperforms OBD.

Original: F. D ERIVATION OF LAP- ACT
Revised: F. D ERIVATION OF LAP- ACT

Original: Consider a case where one aims to prune a connection of a network with ReLU, i.e.,
Revised: Consider a case where one aims to prune a connection of a network with ReLU, i.e.,

Original: [Equation]
Revised: [Equation]

Original: where σ ( x ) = max { 0 , x } is applied entrywise.
Revised: where σ ( x ) = max { 0 , x } is applied entrywise.

Original: Under the over-parametrized scenario, zeroing out a single weight may alter the activation pattern of connected neurons with only negligible probability, which allows one to decouple the probability of activation of each neuron from the act of pruning each connection.
Revised: Under the over-parametrized scenario, zeroing out a single weight may alter the activation pattern of connected neurons with only negligible probability, which allows one to decouple the probability of activation of each neuron from the act of pruning each connection.

Original: From this observation, we ﬁrst construct the below random distortion, following the philosophy of the linear lookahead distortion Eq.
Revised: From this observation, we ﬁrst construct the below random distortion, following the philosophy of the linear lookahead distortion Eq.

Original: [Equation]
Revised: [Equation]

Original:  where (cid:101) J ( W i ) denotes a random matrix where (cid:101)
Revised: 21) where (cid:101) J ( W i ) denotes a random matrix where (cid:101)

Original: J ( W i )
Revised: J ( W i )

Original: [ k, :]
Revised: [ k, :]

Original: =
Revised: =

Original: g i [ k ] · J ( W i )
Revised: g i [ k ] · J ( W i )

Original: [ k, :]
Revised: [ k, :]

Original: and g i [ k ] is a 0-1 random variable corresponding to the activation, i.e., g i [ k ] = 

Revised: and g i [ k ] is a 0random variable corresponding to the activation, i.e., g i [ k ] =

Original: if and only if the k -th output of the i -th layer is activated.
Revised: if and only if the k -th output, i.e., ReLU, of the i -th layer is activated.

Original: However, directly computing the expected distortion with respect to the real activation distribution might be computationally expensive.
Revised: However, directly computing the expected distortion with respect to the real activation distribution might be computationally expensive.

Original: To resolve this issue, we approximate the root mean-squared lookahead distortion by applying the mean-ﬁeld approximation to the activation probability of neurons, i.e., all activations are assumed to be independent, as
Revised: To resolve this issue, we approximate the root mean square lookahead distortion by applying the mean-ﬁeld approximation to the activation probability of neurons, i.e., all activations are assumed to be independent, as

Original: [Equation]
Revised: [Equation]

Original: (22) where g =
Revised: where g =

Original: [ g i ] i , p ( g ) denotes the empirical activation distribution of all neurons and (cid:81)
Revised: [ g i ] i , p ( g ) denotes the empirical activation distribution of all neurons and (cid:81)

Original: i,k p ( g i [ k ]) denotes the mean-ﬁeld approximation of p ( g ) .
Revised: i,k p ( g i [ k ]) denotes the mean-ﬁeld approximation of p ( g ) .

Original: Indeed, the lookahead distortion with ReLU nonlinearity (Eq.
Revised: Indeed, the lookahead distortion with ReLU nonlinearity (Eq.

Original: (22)) or three-layer blocks consisting only of the fully-connected layers and the convolutional layers can be easily computed by using the rescaled weight matrix (cid:99) W i :
Revised: (22)) or three-layer blocks consisting only of the fully-connected layers and the convolutional layers can be easily computed by using the rescaled weight matrix (cid:99) W i :

Original: [Equation]
Revised: [Equation]

Original: where I i,j denotes a set of output indices in the j -th output neuron/channel ofthe i -th layer.
Revised: where I i,j denotes the set of ReLU indices in the j -th output neuron/channel of i -th layer.

Revised: For example, I i,j = { j } for fully connected layers and I i,j is a set of ReLU indices in the j -th channel for convolutional layers.

Original: Finally, for an edge w connected to the j -th input neuron/channel and the k -th output neuron/channel of the i -th layer, Eq.
Revised: Finally, for an edge w connected to the j -th input neuron/channel and the k -th output neuron/channel of the i -th layer, Eq.

Original: (22) reduces to
Revised: (22) reduces to

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  where (cid:98) w denotes the rescaled value of w .
Revised: 24) where (cid:98) w denotes the rescaled value of w .

Original: This completes the derivation of Eq.
Revised: This completes the derivation of Eq.

Original: F. E XPERIMENTS WITH LAP- ACT
Revised: F. E XPERIMENTS WITH LAP- ACT

Original: We compare the performance of three algorithms utilizing training data at the pruning phase: optimal brain damage (OBD) which approximates the loss via second order Taylor seris approximation with the Hessian diagonal (LeCun et al., 1989), LAP using OBD instead of weight magnitudes (OBD+LAP), and LAP-act as described in this section.
Revised: We compare the performance of three algorithms utilizing training data at the pruning phase: optimal brain damage (OBD) which approximates the loss via second order Taylor seris approximation with the Hessian diagonal (LeCun et al., 1989), LAP using OBD instead of weight magnitudes (OBD+LAP), and LAP-act as described in this section.

Original: We compare the performances of three algorithms under the same experimental setup as in Section 3.2.
Revised: We compare the performances of three algorithms under the same experimental setup as in Section 3.2.

Original: To compute the Hessian diagonal for
Revised: To compute the Hessian diagonal for

Original: OBD and OBD+LAP, we use a recently introduced software package called “BackPACK,” (Anonymous, 2020), which is the only open-source package supporting an efﬁcient of Hessians, up to our knowledge.
Revised: OBD and OBD+LAP, we use a recently introduced software package called “BackPACK,” (Dangel et al., 2020), which is the only open-source package supporting an efﬁcient of Hessians, up to our knowledge.

Original: Note that the algorithms evaluated in this section are also evaluated for global pruning experiments in Appendix I.
Revised: Note that the algorithms evaluated in this section are also evaluated for global pruning experiments in Appendix I.

Original: The experimental results for FCN and Conv-6 are presented in Tables 13 and 14.
Revised: The experimental results for FCN and Conv-6 are presented in Tables 13 and 14.

Original: Comparing to algorithms relying solely on the model parameters for pruning (MP/LAP in Tables 1 and 2), we observe that OBD performs better in general, especially in the high sparsity regime.
Revised: Comparing to algorithms relying solely on the model parameters for pruning (MP/LAP in Tables 1 and 2), we observe that OBD performs better in general, especially in the high sparsity regime.

Original: This observation is coherent to the ﬁndings of LeCun et al.
Revised: This observation is coherent to the ﬁndings of LeCun et al.

Original: Intriguingly, however, we observe that applying lookahead critertion to OBD (OBD+LAP) signiﬁcantly enhances to OBD signiﬁcantly enhances the performance in the high sparsity regime.
Revised: Intriguingly, however, we observe that applying lookahead critertion to OBD (OBD+LAP) signiﬁcantly enhances to OBD signiﬁcantly enhances the performance in the high sparsity regime.

Original: We hypothesize that LAP helps capturing a correlation among scores (magnitude or Hessian-based) of adjacent layers.
Revised: We hypothesize that LAP helps capturing a correlation among scores (magnitude or Hessian-based) of adjacent layers.

Original: Also, we observe that LAP-act consistently exhibits a better performance compared to OBD.
Revised: Also, we observe that LAP-act consistently exhibits a better performance compared to OBD.

Original: This result is somewhat surprising, in the sense that LAP-act only utilizes (easier-to-estimate) information about activation probabilities of each neuron to correct lookahead distortion.
Revised: This result is somewhat surprising, in the sense that LAP-act only utilizes (easier-to-estimate) information about activation probabilities of each neuron to correct lookahead distortion.

Original: The average running time of OBD, OBD+LAP, and LAP-act is summarized in Table 15.
Revised: The average running time of OBD, OBD+LAP, and LAP-act is summarized in Table 15.

Original: We use
Revised: We use

Original: Xeon E5-2630v

Revised: Xeon E5-2630v

Original: 2.20GHz for pruning edges, and additionally used a single NVidia GeForce GTX1080 for the computation of Hessian diagonals (used for OBD, OBD+LAP) and activation probabiility (for LAP-act).
Revised: 2.20GHz for pruning edges, and additionally used a single NVidia GeForce GTX1080 for the computation of Hessian diagonals (used for OBD, OBD+LAP) and activation probabiility (for LAP-act).

Original: We observe that LAP-act runs in a signiﬁcantly less running time than
Revised: We observe that LAP-act runs in a signiﬁcantly less running time than

Original: OBD/OBD+LAP, and the gap widens as the number of parameters and the dimensionality of the dataset increases (from MNIST to CIFAR-10).
Revised: OBD/OBD+LAP, and the gap widens as the number of parameters and the dimensionality of the dataset increases (from MNIST to CIFAR-10).

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: G C OMPUTATIONAL COST OF LOOKING AHEAD
Revised: G C OMPUTATIONAL COST OF LOOKING AHEAD

Original: In this section, we brieﬂy describe how a computation of lookahead distortion Eq.
Revised: In this section, we brieﬂy describe how a computation of lookahead distortion Eq.

Original: (5) can be done efﬁciently, and provide experimental comparisons of average computation times for MP and LAP.
Revised: (5) can be done efﬁciently, and provide experimental comparisons of average computation times for MP and LAP.

Original: It turns out that most of the computational load for LAP comes from the sorting procedure, and tensor operations introduce only a minimal overhead.
Revised: It turns out that most of the computational load for LAP comes from the sorting procedure, and tensor operations introduce only a minimal overhead.

Original: MP comprises of three steps: (1) computing the absolute value of the tensor, (2) sorting the absolute values, and (3) selecting the cut-off threshold and zero-ing out the weights under the threshold.
Revised: MP comprises of three steps: (1) computing the absolute value of the tensor, (2) sorting the absolute values, and (3) selecting the cut-off threshold and zero-ing out the weights under the threshold.

Original: Steps (2) and (3) remain the same in LAP, and typically takes O ( n log n ) steps ( n denotes the number of parameters in a layer).
Revised: Steps (2) and (3) remain the same in LAP, and typically takes O ( n log n ) steps ( n denotes the number of parameters in a layer).

Original: On the other hand, Step (1) is replaced by computing the lookahead distortion
Revised: On the other hand, Step (1) is replaced by computing the lookahead distortion

Original: [Equation]
Revised: [Equation]

Original: for each parameter w .
Revised: for each parameter w .

Original: Fortunately, this need not be computed separately for each parameter.
Revised: Fortunately, this need not be computed separately for each parameter.

Original: Indeed, one can perform tensor operations to compute the squared lookahead distortion, which has the same ordering with lookahead distortion.
Revised: Indeed, one can perform tensor operations to compute the squared lookahead distortion, which has the same ordering with lookahead distortion.

Original: For fully-connected layers with 2-dimensional Jacobians, the squared lookahead distortion for W i +1 ∈
Revised: For fully-connected layers with 2-dimensional Jacobians, the squared lookahead distortion for W i +1 ∈

Original: R
Revised: R

Original: d i +1 ×
Revised: d i +1 ×

Original: d i , W i ∈ R
Revised: d i , W i ∈ R

Original: d i × d i [Equation] 1 , W i 
Revised: d i × d i − 1 , W i −

Original: [Equation]

Original: 1 ∈ R d i − 1 ×
Revised: 1 ∈ R d i − 1 ×

Original: di
Revised: d

Original: [Equation] 2 is
Revised: i − 2 is

Original: [Equation]
Revised: [Equation]

Original: (25) where 1 i denotes all-one matrix of size d i − 2 ×
Revised: where 1 i denotes all-one matrix of size d i − 2 ×

Original: d i ; multiplying 1 i denotes summing operation along an axis and duplicating summed results into the axis, and (cid:12) 2 denotes the element-wise square operation.
Revised: d i ; multiplying 1 i denotes summing operation along an axis and duplicating summed results into the axis, and (cid:12) 2 denotes the element-wise square operation.

Original: The case of convolutional layers can be handled similarly.
Revised: The case of convolutional layers can be handled similarly.

Original: We note that an implementation of Eq.
Revised: We note that an implementation of Eq.

Original: (25) is very simple.
Revised: (25) is very simple.

Original: Indeed, the following PyTorch code segment calculates a lookahead score matrix: def lookahead_score(W,W_prev,W_next):
Revised: Indeed, the following PyTorch code segment calculates a lookahead score matrix: def lookahead_score(W,W_prev,W_next):

Original: W_prev_sq=(W_prev**2).sum(dim=1)
Revised: W_prev_sq=(W_prev**2).sum(dim=1)

Original: W_prev_mat=W_prev_sq.view(1,-1).repeat(W.size(0),1)
Revised: W_prev_mat=W_prev_sq.view(1,-1).repeat W.size(0)

Original: W_next_sq =
Revised: W_next_sq =

Original: (W_next ** 2).sum(dim=0)
Revised: (W_next ** 2).sum dim=0)

Original: W_next_mat = W_next_sq.view(-1,1).repeat(1,W.size(1)) return (W**2)*W_prev_mat*W_next_mat
Revised: W_next_ = W_next_sq.view repeat 1)) return W next mat

Original: Combined with modern tensor computation frameworks, computing Eq.
Revised: Combined with modern tensor computation frameworks, computing Eq.

Original: (25) does not introduce heavy overhead.
Revised: (25) does not introduce heavy overhead.

Original: To show this, we compare the computation time of MP and LAP for six neural networks in Table 16, where we ﬁxed the layerwise pruning rate to be uniformly 90%.
Revised: To show this, we compare the computation time of MP and LAP for six neural networks in Table 16, where we ﬁxed the layerwise pruning rate to be uniformly 90%.

Original: The codes are implemented with PyTorch, and the computations have taken place on 40 CPUs of Intel Xeon
Revised: The codes are implemented with PyTorch, and the computations have taken place on 40 CPUs of Intel Xeon

Original: E5-2630v4 @
Revised: E5-2630v4 @

Original: 2.20GHz.
Revised: 2.20GHz.

Original: All ﬁgures are averaged over 100 trials.
Revised: All ﬁgures are averaged over 100 trials.

Original: We make two observations from Table 16.
Revised: We make two observations from Table 16.

Original: First, the time required for LAP did not exceed 150% of the time required for MP, conﬁrming our claim on the computational beneﬁts of LAP.
Revised: First, the time required for LAP did not exceed 150% of the time required for MP, conﬁrming our claim on the computational beneﬁts of LAP.

Original: Second, most of the added computation comes from considering the factors from batch normalization, without which the added computation load is ≈ 5%.
Revised: Second, most of the added computation comes from considering the factors from batch normalization, without which the added computation load is ≈ 5%.

Original: [Table]
Revised: [Table]

Original: H L OOKAHEAD FOR CHANNEL PRUNING
Revised: H L OOKAHEAD FOR CHANNEL PRUNING

Original: In the main text, LAP is compared tothe MP in the context of unstructured pruning, where we do not impose any structural constraints on the set of connections to be pruned together.
Revised: In the main text, LAP is compared to MP in the context of unstructured pruning, where we do not impose any structural constraints on the set of connections to be pruned together.

Original: On the other hand, the magnitude-based pruning methods are also being used popularly as a baseline for channel pruning (Ye et al., 2018), which falls under the category of structured pruning.
Revised: On the other hand, the magnitude-based pruning methods are also being used popularly as a baseline for channel pruning (Ye et al., 2018), which falls under the category of structured pruning.

Original: MP in channel pruning is typically done by removing channels with smallest aggregated weight magnitudes; this aggregation can be done by either taking (cid:96) 1 -norm or (cid:96) 2 -norm of magnitudes.
Revised: MP in channel pruning is typically done by removing channels with smallest aggregated weight magnitudes; this aggregation can be done by either taking (cid:96) 1 -norm or (cid:96) 2 -norm of magnitudes.

Original: Similarly, we can consider channel pruning scheme based on an (cid:96) 1 or (cid:96) 2 aggregation of LAP distortions, which we will call LAP- (cid:96) 1 and LAP- (cid:96) 2 (as opposed to MP- (cid:96) 1 and MP- (cid:96) 2 ).
Revised: Similarly, we can consider channel pruning scheme based on an (cid:96) 1 or (cid:96) 2 aggregation of LAP distortions, which we will call LAP- (cid:96) 1 and LAP- (cid:96) 2 (as opposed to MP- (cid:96) 1 and MP- (cid:96) 2 ).

Original: We compare the performances of LAP-based channel pruning methods to MP-based channel pruning methods, along with another baseline of random channel pruning (denoted with RP).
Revised: We compare the performances of LAP-based channel pruning methods to MP-based channel pruning methods, along with another baseline of random channel pruning (denoted with RP).

Original: We test with
Revised: We test with

Original: Conv-6 (Table 17) and VGG-19 (Table 18) networks on CIFAR-10 dataset.
Revised: Conv-6 (Table 17) and VGG-19 (Table 18) networks on CIFAR-10 dataset.

Original: All reported ﬁgures are averaged over ﬁve trials, experimental settings are identical to the unstructure pruning experiments unless noted otherwise.
Revised: All reported ﬁgures are averaged over ﬁve trials, experimental settings are identical to the unstructure pruning experiments unless noted otherwise.

Original: Similar to the case of unstructured pruning, we observe that LAP-based methods consistently outperform MP-based methods.
Revised: Similar to the case of unstructured pruning [Table] we observe that LAP-based methods consistently outperform MP-based methods.

Original: Comparing (cid:96) 1 with (cid:96) 2 aggregation, we note that LAP- (cid:96) 2 performs better than LAP- (cid:96) 1 in both experiments, by a small margin.
Revised: Comparing (cid:96) 1 with (cid:96) 2 aggregation, we note that LAP- (cid:96) 2 performs better than LAP- (cid:96) 1 in both experiments, by a small margin.

Original: Among MP-based methods, we do not observe any similar dominance.
Revised: Among MP-based methods, we do not observe any similar dominance.

Original: models achieve 11.97% error rate

Original: [Table] [Table]
Revised: [Table] [Table]

Original: I L OOKAHEAD FOR G LOBAL P RUNING
Revised: I L OOKAHEAD FOR G LOBAL P RUNING

Original: In this section, we present global pruning results for MP, LAP, OBD, OBD+LAP and LAP-act in
Revised: In this section, we present global pruning results for MP, LAP, OBD, OBD+LAP and LAP-act in

Original: Table 19 and Table 20.
Revised: Table 19 and Table 20.

Original: In this methods, we prune a fraction of weights with smallest scores (e.g. weight magnitude, lookahead distortion, Hessian-based scores) among all weights in the whole network.
Revised: In this methods, we prune a fraction of weights with smallest scores (e.g. weight magnitude, lookahead distortion, Hessian-based scores) among all weights in the whole network.

Original: The sufﬁx “-normalize” in the tables denotes that the score is normalized by the Frobenius norm of the corresponding layer’s score.
Revised: The sufﬁx “-normalize” in the tables denotes that the score is normalized by the Frobenius norm of the corresponding layer’s score.

Original: For MP, LAP, OBD+LAP and LAP-act, we only report the results for global pruning with normalization, as the normalized versions outperform the unnormalized ones.
Revised: For MP, LAP, OBD+LAP and LAP-act, we only report the results for global pruning with normalization, as the normalized versions outperform the unnormalized ones.

Original: In the case of OBD, whose score is already globally designed, we report the results for both unnormalized and normalized versions.
Revised: In the case of OBD, whose score is already globally designed, we report the results for both unnormalized and normalized versions.

Original: As demonstrated in Section 3.2 for ﬁxed layerwise pruning rates, we observe that LAP and its variants perform better than their global pruning baselines,
Revised: As demonstrated in Section 3.2 for ﬁxed layerwise pruning rates, we observe that LAP and its variants perform better than their global pruning baselines,

Original: i
Revised: i

Original: [Table]
Revised: [Table]

Original: e. MP-normalize and OBD.
Revised: e. MP-normalize and OBD.

Original: We also note that LAP-normalize performs better than MP with pre-speciﬁed layerwise pruning rates (appeared in Section 3.2), with a larger gap for higher levels of sparsity.
Revised: We also note that LAP-normalize performs better than MP with pre-speciﬁed layerwise pruning rates (appeared in Section 3.2), with a larger gap for higher levels of sparsity.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: J LAP- ALL : L OOKING AHEAD THE WHOLE NETWORK
Revised: J LAP- ALL : L OOKING AHEAD THE WHOLE NETWORK

Original: We also report some experimental results on a variant of lookahead pruning, coined LAP-all, which treats (a linearized version of) the whole network as an operator block.
Revised: We also report some experimental results on a variant of lookahead pruning, coined LAP-all, which treats (a linearized version of) the whole network as an operator block.

Original: More speciﬁcally, one attempts to minimize the Frobenius distortion of the operator block
Revised: More speciﬁcally, one attempts to minimize the Frobenius distortion of the operator block

Original: [Equation]
Revised: [Equation]

Original: where J i + j : i :=
Revised: where J i + j : i :=

Original: J ( W i + j ) J ( W i + j − 1 ) · · · J ( W i ) .
Revised: J ( W i + j ) J ( W i + j − 1 ) · · · J ( W i ) .

Original: We test LAP-all on FCN under the same setup as in Section 3.
Revised: We test LAP-all on FCN under the same setup as in Section 3.

Original: [Table]
Revised: [Table]

Original: and report the results in Table 21.
Revised: and report the results in Table 21.

Original: All ﬁgures are averaged over ﬁve trials.
Revised: All ﬁgures are averaged over ﬁve trials.

Original: We observe that LAP-all achieves a similar level of performance to LAP, while LAP-all underperforms under a high-sparsity regime.
Revised: We observe that LAP-all achieves a similar level of performance to LAP, while LAP-all underperforms under a high-sparsity regime.

Original: We suspect that such shortfall originates from the accumulation of error terms incurred by ignoring the effect of activation functions, by which the beneﬁts of looking further fades.
Revised: We suspect that such shortfall originates from the accumulation of error terms incurred by ignoring the effect of activation functions, by which the beneﬁts of looking further fades.

Original: An in-depth theoretical analysis for the determination of an optimal “sight range” of LAP would be an interesting future direction.
Revised: An in-depth theoretical analysis for the determination of an optimal “sight range” of LAP would be an interesting future direction.

Original: [Table]
Revised: [Table]

Original: K C OMPARISON WITH S MALLER N ETWORKS
Revised: K C OMPARISON WITH S MALLER N ETWORKS

Original: As a sanity check
Revised: As a sanity check

Original: [Table]
Revised: [Table]

Original: we compare the performance of large neural networks pruned via MP and LAP to the performance of a small network.
Revised: we compare the performance of large neural networks pruned via MP and LAP to the performance of a small network.

Original: In particular, we prune VGG-16, VGG-19, and ResNet18 trained on CIFAR-10 dataset
Revised: In particular, we prune VGG-16, VGG-19, and ResNet18 trained on CIFAR-10 dataset

Original: [Table]
Revised: [Table]

Original: to have a similar number of parameters to MobileNetV2 (Sandler et al., 2018).
Revised: to have a similar number of parameters to MobileNetV2 (Sandler et al., 2018).

Original: For training and pruning VGGs and ResNet
Revised: For training and pruning VGGs and ResNet

Original: [Table]
Revised: [Table]

Original: we follows the prior setup in Appendix
Revised: we follows the prior setup in Appendix

Original: A while we use the same setup for training MobileNetV2 (Adam optimizer with learning rate of 3 · 10 − 4 with batch size 60 , and trained 60k steps).
Revised: A while we use the same setup for training MobileNetV2 (Adam optimizer with learning rate of 3 · 10 − 4 with batch size 60 , and trained 60k steps).

Original: We observe that models pruned via LAP (and
Revised: We observe that models pruned via LAP (and

Original: MP) exhibit better performance compared to MobileNetV2, even when pruned to have a smaller number of parameters.
Revised: MP) exhibit better performance compared to MobileNetV2, even when pruned to have a smaller number of parameters.

Original: [Table]
Revised: [Table]

Original: L W HERE IS THE PERFORMANCE GAIN OF LAP COMING FROM ?
Revised: L W HERE IS THE PERFORMANCE GAIN OF LAP COMING FROM ?

Original: In this section, we brieﬂy discuss where the beneﬁts of the sub-network discovered by LAP comes from; does LAP subnetwork have a better generalizability or expressibility?
Revised: In this section, we brieﬂy discuss where the beneﬁts of the sub-network discovered by LAP comes from; does LAP subnetwork have a better generalizability or expressibility?

Original: For this purpose, we look into the generalization gap, i.e., the gap between the training and test accuracies, of the hypothesis learned via LAP procedure.
Revised: For this purpose, we look into the generalization gap, i.e., the gap between the training and test accuracies, of the hypothesis learned via LAP procedure.

Original: Below we present a plot of test accuracies (Fig. 4a) and a plot of generalization gap (Fig. 4b) for FCN trained with MNIST dataset.
Revised: Below we present a plot of test accuracies (Fig. 4a) and a plot of generalization gap (Fig. 4b) for FCN trained with MNIST dataset.

Original: The plot hints us that the network structure learned by LAP may not necessarily have a smaller generalizability.
Revised: The plot hints us that the network structure learned by LAP may not necessarily have a smaller generalizability.

Original: Remarkably, the generalization gap of the MP-pruned models and the LAP-pruned models are very similar to each other; the beneﬁts of LAP subnetwork compared to MP would be that it can express a better-performing architecture with a network of similar sparsity and generalizability.
Revised: Remarkably, the generalization gap of the MP-pruned models and the LAP-pruned models are very similar to each other; the beneﬁts of LAP subnetwork compared to MP would be that it can express a better-performing architecture with a network of similar sparsity and generalizability.

Original: [Figure]
Revised: [Figure]

Original: M C ONNECTIONS TO IMPLICIT BIAS OF SGD
Revised: M C ONNECTIONS TO IMPLICIT BIAS OF SGD

Original: Another theoretical justiﬁcation of using the lookahead distortion (Eq.
Revised: Another theoretical justiﬁcation of using the lookahead distortion (Eq.

Original: (5)) for neural networks with nonlinear activation functions comes from recent discoveries regarding the implicit bias imposed by training procedures using stochastic gradient descent.
Revised: (5)) for neural networks with nonlinear activation functions comes from recent discoveries regarding the implicit bias imposed by training procedures using stochastic gradient descent.

Original: More speciﬁcally, Du et al.
Revised: More speciﬁcally, Du et al.

Original: (2018) proves the following result, generalizing the ﬁndings of Arora et al.
Revised: (2018) proves the following result, generalizing the ﬁndings of Arora et al.

Original: For any two neighboring layers of fully-connected neural network using positive homogeneous activation functions, the quantity(cid:107)
Revised: For any two neighboring layers of fully-connected neural network using positive homogeneous activation functions, the quantity

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (26) remains constant for any hidden neuron j over training via gradient ﬂow.
Revised: remains constant for any hidden neuron j over training via gradient ﬂow.

Original: In other words, the total outward ﬂow of weights is tied to the inward ﬂow of weights for each neuron.
Revised: In other words, the total outward ﬂow of weights is tied to the inward ﬂow of weights for each neuron.

Original: This observation hints at the possibility of a relative undergrowth of weight magnitude of an ‘important’ connection, in the case where the connection shares the same input/output neuron with other ‘important’ connections.
Revised: This observation hints at the possibility of a relative undergrowth of weight magnitude of an ‘important’ connection, in the case where the connection shares the same input/output neuron with other ‘important’ connections.

Original: From this viewpoint, the multiplicative factors in Eq.
Revised: From this viewpoint, the multiplicative factors in Eq.

Original: (5) take into account the abstract notion of neuronal importance score, assigning signiﬁcance to connections to the neuron through which more gradient signals have ﬂowed through.
Revised: (5) take into account the abstract notion of neuronal importance score, assigning signiﬁcance to connections to the neuron through which more gradient signals have ﬂowed through.

Original: Without considering such factors, LAP reduces to the ordinary magnitude-based pruning.
Revised: Without considering such factors, LAP reduces to the ordinary magnitude-based pruning.