Original: D YNAMIC S PARSE T RAINING : F IND E FFICIENT S PARSE N ETWORK FROM SCRATCH WITH T RAINABLE M ASKED L AYERS
Revised: D YNAMIC S PARSE T RAINING : F IND E FFICIENT S PARSE N ETWORK FROM SCRATCH WITH T RAINABLE M ASKED L AYERS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We present a novel network pruning algorithm called Dynamic Sparse Training that can jointly ﬁnd the optimal network parameters and sparse network structure in a uniﬁed optimization process with trainable pruning thresholds.
Revised: We present a novel network pruning algorithm called Dynamic Sparse Training that can jointly ﬁnd the optimal network parameters and sparse network structure in a uniﬁed optimization process with trainable pruning thresholds.

Original: These thresholds can have ﬁne-grained layer-wise adjustments dynamically via backpropagation.
Revised: These thresholds can have ﬁne-grained layer-wise adjustments dynamically via backpropagation.

Original: We demonstrate that our dynamic sparse training algorithm can easily train very sparse neural network models with little performance loss using the same training epochs as dense models.
Revised: We demonstrate that our dynamic sparse training algorithm can easily train very sparse neural network models with little performance loss using the same training epochs as dense models.

Original: Dynamic Sparse Training achieves prior art performance compared with other sparse training algorithms on various network architectures.
Revised: Dynamic Sparse Training achieves prior art performance compared with other sparse training algorithms on various network architectures.

Original: Additionally, we have several surprising observations that provide strong evidence to the effectiveness and efﬁciency of our algorithm.
Revised: Additionally, we have several surprising observations that provide strong evidence to the effectiveness and efﬁciency of our algorithm.

Original: These observations reveal the underlying problems of traditional three-stage pruning algorithms and present the potential guidance provided by our algorithm to the design of more compact network architectures
Revised: These observations reveal the underlying problems of traditional three-stage pruning algorithms and present the potential guidance provided by our algorithm to the design of more compact network architectures

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Despite the impressive success that deep neural networks have achieved in a wide range of challenging tasks, the inference in deep neural network is highly memory-intensive and computationintensive due to the over-parameterization of deep neural networks.
Revised: Despite the impressive success that deep neural networks have achieved in a wide range of challenging tasks, the inference in deep neural network is highly memory-intensive and computationintensive due to the over-parameterization of deep neural networks.

Original: Network pruning (LeCun et al.
Revised: Network pruning (LeCun et al.

Original: ; Han et al.
Revised: ; Han et al.

Original: ; Molchanov et al.
Revised: ; Molchanov et al.

Original: (2017)) has been recognized as an effective approach to improving the inference efﬁciency in resource-limited scenarios.
Revised: (2017)) has been recognized as an effective approach to improving the inference efﬁciency in resource-limited scenarios.

Original: Traditional pruning methods consist of dense network training followed with pruning and ﬁne-tuning iterations.
Revised: Traditional pruning methods consist of dense network training followed with pruning and ﬁne-tuning iterations.

Original: To avoid the expensive pruning and ﬁne-tuning iterations, many sparse training methods (Mocanu et al., 2018; Bellec et al., 2017; Mostafa & Wang, 2019; Dettmers & Zettlemoyer, 2019) have been proposed, where the network pruning is conducted during the training process.
Revised: To avoid the expensive pruning and ﬁne-tuning iterations, many sparse training methods (Mocanu et al., 2018; Bellec et al., 2017; Mostafa & Wang, 2019; Dettmers & Zettlemoyer, 2019) have been proposed, where the network pruning is conducted during the training process.

Original: However, all these methods suffers from following three problems:
Revised: However, all these methods suffers from following three problems:

Original: Coarse-grained predeﬁned pruning
Revised: Coarse-grained predeﬁned pruning

Original: schedule .
Revised: schedule .

Original: Most of the existing pruning methods use predeﬁned pruning schedule with many additional hyperparameters like pruning a % parameter each time and then ﬁne-tuning for b epochs with totally c pruning steps.
Revised: Most of the existing pruning methods use predeﬁned pruning schedule with many additional hyperparameters like pruning a % parameter each time and then ﬁne-tuning for b epochs with totally c pruning steps.

Original: It is non-trivial to determine these hyperparameters for network architectures with various degree of complexity.
Revised: It is non-trivial to determine these hyperparameters for network architectures with various degree of complexity.

Original: Therefore, usually a ﬁxed pruning schedule is adopted for all the network architectures, which means that a very simple network architecture like LeNet-300-100 will have the same pruning schedule as a far more complex network like ResNet-152.
Revised: Therefore, usually a ﬁxed pruning schedule is adopted for all the network architectures, which means that a very simple network architecture like LeNet-300-100 will have the same pruning schedule as a far more complex network like ResNet-152.

Original: Besides, almost all the existing pruning methods conduct epoch-wise pruning, which means that the pruning is conducted between two epochs and no pruning operation happens inside each epoch.
Revised: Besides, almost all the existing pruning methods conduct epoch-wise pruning, which means that the pruning is conducted between two epochs and no pruning operation happens inside each epoch.

Original: Failure to properly recover the pruned weights .
Revised: Failure to properly recover the pruned weights .

Original: Almost all the existing pruning methods conduct ”hard” pruning that prunes weights by directly setting their values to 0 .
Revised: Almost all the existing pruning methods conduct ”hard” pruning that prunes weights by directly setting their values to 0 .

Original: Many works (Guo et al., 2016; Mocanu et al., 2018;
Revised: Many works (Guo et al., 2016; Mocanu et al., 2018;

Original: He et al., 2018) have argued that the importance of network weights are not ﬁxed and will change dynamically during the pruning and training process.
Revised: He et al., 2018) have argued that the importance of network weights are not ﬁxed and will change dynamically during the pruning and training process.

Original: Previously unimportant weights may tend to be important.
Revised: Previously unimportant weights may tend to be important.

Original: So the ability to recover the pruned weights is of high signiﬁcance.
Revised: So the ability to recover the pruned weights is of high signiﬁcance.

Original: However, directly setting the pruned weights to 0 results in the loss of historical parameter importance, which makes it difﬁcult to determine: 1) whether and when each pruned weight should be recovered, 2) what values should be assigned to the recovered weights.
Revised: However, directly setting the pruned weights to 0 results in the loss of historical parameter importance, which makes it difﬁcult to determine: 1) whether and when each pruned weight should be recovered, 2) what values should be assigned to the recovered weights.

Original: Therefore, existing methods that claim to be able to recover the pruned weights simply choose a predeﬁned portion of pruned weights to recover and these recover weights are randomly initialized or initialized to the same value.
Revised: Therefore, existing methods that claim to be able to recover the pruned weights simply choose a predeﬁned portion of pruned weights to recover and these recover weights are randomly initialized or initialized to the same value.

Original: Failure to properly determine layer-wise pruning rates .
Revised: Failure to properly determine layer-wise pruning rates .

Original: Modern neural network architectures usually contain dozens of layers with various number of parameters.
Revised: Modern neural network architectures usually contain dozens of layers with various number of parameters.

Original: Therefore, the degrees of parameter redundancy are very different among the layers.
Revised: Therefore, the degrees of parameter redundancy are very different among the layers.

Original: For simplicity, some methods prune the same percentage of parameter at each layer, which is not optimal.
Revised: For simplicity, some methods prune the same percentage of parameter at each layer, which is not optimal.

Original: To obtain dynamic layer-wise pruning rates, a single global pruning threshold or layer-wise greedy algorithms are applied.
Revised: To obtain dynamic layer-wise pruning rates, a single global pruning threshold or layer-wise greedy algorithms are applied.

Original: Using a single global pruning threshold is exceedingly difﬁcult to assess the local parameter importance of the individual layer, since each layer has a signiﬁcantly different amount of parameter and contribution to the model performance.
Revised: Using a single global pruning threshold is exceedingly difﬁcult to assess the local parameter importance of the individual layer, since each layer has a signiﬁcantly different amount of parameter and contribution to the model performance.

Original: This makes pruning algorithms based on a single global threshold inconsistent and non-robust.
Revised: This makes pruning algorithms based on a single global threshold inconsistent and non-robust.

Original: The problem of layer-by-layer greedy pruning methods is that the unimportant neurons in an early layer may have a signiﬁcant inﬂuence on the responses in later layers, which may result in propagation and ampliﬁcation of the reconstruction error (Yu et al., 2018).
Revised: The problem of layer-by-layer greedy pruning methods is that the unimportant neurons in an early layer may have a signiﬁcant inﬂuence on the responses in later layers, which may result in propagation and ampliﬁcation of the reconstruction error (Yu et al., 2018).

Original: We propose a novel end-to-end sparse training algorithm that properly solves the above problems.
Revised: We propose a novel end-to-end sparse training algorithm that properly solves the above problems.

Original: With only one additional hyperparameter used to set the ﬁnal model sparsity, our method can achieve dynamic ﬁne-grained pruning and recovery during the whole training process.
Revised: With only one additional hyperparameter used to set the ﬁnal model sparsity, our method can achieve dynamic ﬁne-grained pruning and recovery during the whole training process.

Original: Meanwhile, the layerwise pruning rates will be adjusted automatically with respect to the change of parameter importance during the training and pruning process.
Revised: Meanwhile, the layerwise pruning rates will be adjusted automatically with respect to the change of parameter importance during the training and pruning process.

Original: Our method achieves state-of-the-art performance compared with other sparse training algorithms.
Revised: Our method achieves state-of-the-art performance compared with other sparse training algorithms.

Original: The proposed algorithm has following promising properties:
Revised: The proposed algorithm has following promising properties:

Original: • Step-wise pruning and recovery.
Revised: • Step-wise pruning and recovery.

Original: A training epoch usually will have tens of thousands of training steps, which is the feed-forward and back-propagation pass for a single mini-batch.
Revised: A training epoch usually will have tens of thousands of training steps, which is the feed-forward and back-propagation pass for a single mini-batch.

Original: Instead of pruning between two training epochs with predeﬁned pruning schedule, our method prunes and recovers the network parameter at each training step, which is far more ﬁne-grained than existing methods.
Revised: Instead of pruning between two training epochs with predeﬁned pruning schedule, our method prunes and recovers the network parameter at each training step, which is far more ﬁne-grained than existing methods.

Original: • Neuron-wise or ﬁlter-wise trainable thresholds.
Revised: • Neuron-wise or ﬁlter-wise trainable thresholds.

Original: All the existing methods adopt a single pruning threshold for each layer or the whole architecture.
Revised: All the existing methods adopt a single pruning threshold for each layer or the whole architecture.

Original: Our method deﬁnes a threshold vector for each layers.
Revised: Our method deﬁnes a threshold vector for each layers.

Original: Therefore, our method adopts neuron-wise pruning thresholds for fully connected and recurrent layer and ﬁlter-wise pruning thresholds for convolutional layer.
Revised: Therefore, our method adopts neuron-wise pruning thresholds for fully connected and recurrent layer and ﬁlter-wise pruning thresholds for convolutional layer.

Original: Additionally, all these pruning thresholds are trainable and will be updated automatically via back-propagation.
Revised: Additionally, all these pruning thresholds are trainable and will be updated automatically via back-propagation.

Original: • Dynamic pruning schedule.
Revised: • Dynamic pruning schedule.

Original: The training process of deep neural network consists of many hyperparameters.
Revised: The training process of deep neural network consists of many hyperparameters.

Original: The learning rate is perhaps the most important hyperparameter.
Revised: The learning rate is perhaps the most important hyperparameter.

Original: Usually, the learning rate will decay during the training process.
Revised: Usually, the learning rate will decay during the training process.

Original: Our method can automatic adjust the layer-wise pruning rates under different learning rate to get the optimal sparse network structure.
Revised: Our method can automatic adjust the layer-wise pruning rates under different learning rate to get the optimal sparse network structure.

Original: • Consistent sparse pattern.
Revised: • Consistent sparse pattern.

Original: Our algorithm can get consistent layer-wise sparse pattern under different model sparsities, which indicates that our method can automatic determine the optimal layer-wise pruning rates given the target model sparsity.
Revised: Our algorithm can get consistent layer-wise sparse pattern under different model sparsities, which indicates that our method can automatic determine the optimal layer-wise pruning rates given the target model sparsity.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Traditional Pruning Methods : LeCun et al.
Revised: Traditional Pruning Methods : LeCun et al.

Original: (1990) presented the early work about network pruning using second-order derivatives as the pruning criterion.
Revised: (1990) presented the early work about network pruning using second-order derivatives as the pruning criterion.

Original: The effective and popular training, pruning and ﬁne-tuning pipeline was proposed by Han et al.
Revised: The effective and popular training, pruning and ﬁne-tuning pipeline was proposed by Han et al.

Original: (2015), which used the parameter magnitude as the pruning criterion.
Revised: (2015), which used the parameter magnitude as the pruning criterion.

Original: Narang et al.
Revised: Narang et al.

Original: (2017) extended this pipeline to prune the recurrent neural networks with a complicated pruning strategy.
Revised: (2017) extended this pipeline to prune the recurrent neural networks with a complicated pruning strategy.

Original: Molchanov et al.
Revised: Molchanov et al.

Original: (2016) introduced ﬁrst-order Taylor term as the pruning criterion and conduct global pruning.
Revised: (2016) introduced ﬁrst-order Taylor term as the pruning criterion and conduct global pruning.

Original: Li et al.
Revised: Li et al.

Original: (2016) used (cid:96) 1 regularization to force the unimportant parameters to zero.
Revised: (2016) used (cid:96) 1 regularization to force the unimportant parameters to zero.

Original: Sparse Neural Network Training : Recently, some works attempt to ﬁnd the sparse network directly during the training process without the pruning and ﬁne-tuning stage.
Revised: Sparse Neural Network Training : Recently, some works attempt to ﬁnd the sparse network directly during the training process without the pruning and ﬁne-tuning stage.

Original: Inspired by the growth and extinction of neural cell in biological neural networks, Mocanu et al.
Revised: Inspired by the growth and extinction of neural cell in biological neural networks, Mocanu et al.

Original: (2018) proposed a pruneregrowth procedure called Sparse Evolutionary Training (SET) that allows the pruned neurons and connections to revive randomly.
Revised: (2018) proposed a pruneregrowth procedure called Sparse Evolutionary Training (SET) that allows the pruned neurons and connections to revive randomly.

Original: However, the sparsity level need to be set manually and the random recover of network connections may provoke unexpected effects to the network.
Revised: However, the sparsity level need to be set manually and the random recover of network connections may provoke unexpected effects to the network.

Original: DEEP-R proposed by Bellec et al.
Revised: DEEP-R proposed by Bellec et al.

Original: (2017) used Bayesian sampling to decide the pruning and regrowth conﬁguration, which is computationally expensive.
Revised: (2017) used Bayesian sampling to decide the pruning and regrowth conﬁguration, which is computationally expensive.

Original: Dynamic Sparse Reparameterization (Mostafa & Wang, 2019) used dynamic parameter reallocation to ﬁnd the sparse structure.
Revised: Dynamic Sparse Reparameterization (Mostafa & Wang, 2019) used dynamic parameter reallocation to ﬁnd the sparse structure.

Original: However, the pruning threshold can only get halved if the percentage of parameter pruned is too high or get doubled if that percentage is too low for a certain layer.
Revised: However, the pruning threshold can only get halved if the percentage of parameter pruned is too high or get doubled if that percentage is too low for a certain layer.

Original: This coarse-grained adjustment of pruning threshold signiﬁcantly limits the ability of Dynamic Sparse Reparameterization.
Revised: This coarse-grained adjustment of pruning threshold signiﬁcantly limits the ability of Dynamic Sparse Reparameterization.

Original: Additionally, a predeﬁned pruning ratio and fractional tolerance are required.
Revised: Additionally, a predeﬁned pruning ratio and fractional tolerance are required.

Original: Dynamic Network Surgery (Guo et al., 2016) proposed pruning and splicing procedure that can prune or recover network connections according to the parameter magnitude but it requires manually determined thresholds that are ﬁxed during the sparse learning process.
Revised: Dynamic Network Surgery (Guo et al., 2016) proposed pruning and splicing procedure that can prune or recover network connections according to the parameter magnitude but it requires manually determined thresholds that are ﬁxed during the sparse learning process.

Original: This layer-wise thresholds are extremely hard to manually set.
Revised: This layer-wise thresholds are extremely hard to manually set.

Original: Meanwhile, Fixing the thresholds makes it hard to adapt to the rapid change of parameter importance.
Revised: Meanwhile, Fixing the thresholds makes it hard to adapt to the rapid change of parameter importance.

Original: Dettmers &
Revised: Dettmers &

Original: Zettlemoyer (2019) proposed sparse momentum that used the exponentially smoothed gradients as the criterion for pruning and regrowth.
Revised: Zettlemoyer (2019) proposed sparse momentum that used the exponentially smoothed gradients as the criterion for pruning and regrowth.

Original: A ﬁxed percentage of parameters are pruned at each pruning step.
Revised: A ﬁxed percentage of parameters are pruned at each pruning step.

Original: The pruning ratio and momentum scaling rate need to be searched from a relatively high parameter space.
Revised: The pruning ratio and momentum scaling rate need to be searched from a relatively high parameter space.

Original:  D YNAMIC S PARSE T RAINING 3. N OTATION
Revised:  D YNAMIC S PARSE T RAINING 3. N OTATION

Original: Deep neural network consists of a set of parameters { W i : 1 ≤ i ≤ C } , where W i denotes the parameter matrix at layer i and C denotes the number of layers in this network.
Revised: Deep neural network consists of a set of parameters { W i : 1 ≤ i ≤ C } , where W i denotes the parameter matrix at layer i and C denotes the number of layers in this network.

Original: For each fully connected layer and recurrent layer, the corresponding parameter is W i ∈ R c
Revised: For each fully connected layer and recurrent layer, the corresponding parameter is W i ∈ R c

Original: o × c i , where c o is the output dimension and c
Revised: o × c i , where c o is the output dimension and c

Original: i is the input dimension.
Revised: i is the input dimension.

Original: For each convolutional layer, there exists a convolution kernel K
Revised: For each convolutional layer, there exists a convolution kernel K

Original: i ∈ R c
Revised: i ∈ R c

Original: o × c i × w × h , where c o is the number of output channels, c
Revised: o × c i × w × h , where c o is the number of output channels, c

Original: i is the number of input channels, w and h are the kernel sizes.
Revised: i is the number of input channels, w and h are the kernel sizes.

Original: Each ﬁlter in a convolution kernel K i can be ﬂattened to a vector.
Revised: Each ﬁlter in a convolution kernel K i can be ﬂattened to a vector.

Original: Therefore, a corresponding parameter matrix W i ∈ R c o × z can be derived from each convolution kernel K
Revised: Therefore, a corresponding parameter matrix W i ∈ R c o × z can be derived from each convolution kernel K

Original: i ∈ R c
Revised: i ∈ R c

Original: o × c i × w × h , where
Revised: o × c i × w × h , where

Original: z =
Revised: z =

Original: c i × w × h .
Revised: c i × w × h .

Original: Actually, the pruning process is equivalent to ﬁnding a binary parameter mask M i for each parameter matrix W i .
Revised: Actually, the pruning process is equivalent to ﬁnding a binary parameter mask M i for each parameter matrix W i .

Original: Thus, a set of binary parameter masks { M i : 1 ≤ i ≤ C } will be found by network pruning.
Revised: Thus, a set of binary parameter masks { M i : 1 ≤ i ≤ C } will be found by network pruning.

Original: Each element for these parameter masks
Revised: Each element for these parameter masks

Original: M i is either 1 or 0 .
Revised: M i is either 1 or 0 .

Original: 3. T HRESHOLD VECTOR AND D YNAMIC PARAMETER MASK
Revised: 3. T HRESHOLD VECTOR AND D YNAMIC PARAMETER MASK

Original: Pruning can be regarded as applying a binary mask M to each parameter W .
Revised: Pruning can be regarded as applying a binary mask M to each parameter W .

Original: This binary parameter mask M preserves the information about the sparse structure.
Revised: This binary parameter mask M preserves the information about the sparse structure.

Original: Given the parameter set { W 1 , W 2 , · · · , W C } , our method will dynamically ﬁnd the corresponding parameter masks { M 1 , M 2 , · · · , M C } .
Revised: Given the parameter set { W 1 , W 2 , · · · , W C } , our method will dynamically ﬁnd the corresponding parameter masks { M 1 , M 2 , · · · , M C } .

Original: To achieve this, for each parameter matrix W ∈ R c
Revised: To achieve this, for each parameter matrix W ∈ R c

Original: o × c i , a trainable pruning threshold vector t ∈ R c o is deﬁned.
Revised: o × c i , a trainable pruning threshold vector t ∈ R c o is deﬁned.

Original: Then we utilize a unit step function S ( x ) as shown in Figure 2(a) to get the masks according to the magnitude of parameters and corresponding thresholds as present below.
Revised: Then we utilize a unit step function S ( x ) as shown in Figure 2(a) to get the masks according to the magnitude of parameters and corresponding thresholds as present below.

Original: [Equation]
Revised: [Equation]

Original: With the dynamic parameter mask M , the corresponding element in mask M ij will be set to 0 if
Revised: With the dynamic parameter mask M , the corresponding element in mask M ij will be set to 0 if

Original: W ij needs to be pruned.
Revised: W ij needs to be pruned.

Original: This means that the weight W ij is masked out by the 0 at M ij to get a sparse parameter W (cid:12) M .
Revised: This means that the weight W ij is masked out by the 0 at M ij to get a sparse parameter W (cid:12) M .

Original: The value of underlying weight W ij will not change, which preserves the historical information about the parameter importance.
Revised: The value of underlying weight W ij will not change, which preserves the historical information about the parameter importance.

Original: For a fully connected layer or recurrent layer with parameter W ∈ R c o × c i and threshold vector t ∈ R
Revised: For a fully connected layer or recurrent layer with parameter W ∈ R c o × c i and threshold vector t ∈ R

Original: c o , each weight W ij will have a neuron-wise threshold t i , where W ij is the j th weight associated with the i th output neuron.
Revised: c o , each weight W ij will have a neuron-wise threshold t i , where W ij is the j th weight associated with the i th output neuron.

Original: Similarly, the thresholds are ﬁlter-wise for convolutional layer.
Revised: Similarly, the thresholds are ﬁlter-wise for convolutional layer.

Original: Besides, a threshold matrix or a single scalar threshold can also be chosen.
Revised: Besides, a threshold matrix or a single scalar threshold can also be chosen.

Original: More details are present in Appendix A.2.
Revised: More details are present in Appendix A.2.

Original: 3. T RAINABLE MASKED LAYERS
Revised: 3. T RAINABLE MASKED LAYERS

Original: With the threshold vector and dynamic parameter mask, the trainable masked fully connected, convolutional and recurrent layer are introduced as shown in Figure 1, where X is the input of current layer and Y is the output.
Revised: With the threshold vector and dynamic parameter mask, the trainable masked fully connected, convolutional and recurrent layer are introduced as shown in Figure 1, where X is the input of current layer and Y is the output.

Original: For fully connected and recurrent layers, instead of the dense parameter
Revised: For fully connected and recurrent layers, instead of the dense parameter

Original: W , the sparse product W (cid:12) M will be used in the batched matrix multiplication, where (cid:12) denote
Revised: W , the sparse product W (cid:12) M will be used in the batched matrix multiplication, where (cid:12) denote

Original: Hadamard product operator.
Revised: Hadamard product operator.

Original: For convolutional layers,
Revised: For convolutional layers,

Original: each convolution kernel K ∈ R c o × c
Revised: each convolution kernel K ∈ R c o × c

Original: i × w × h can be ﬂatten to get W ∈ R c
Revised: i × w × h can be ﬂatten to get W ∈ R c

Original: o × z .
Revised: o × z .

Original: Therefore, the sparse kernel can be obtained by similar process as fully connected layers.
Revised: Therefore, the sparse kernel can be obtained by similar process as fully connected layers.

Original: This sparse kernel will be used for subsequent convolution operation.
Revised: This sparse kernel will be used for subsequent convolution operation.

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: In order to make the elements in threshold vector t trainable via back-propagation, the derivative of the binary step function S ( x ) is required.
Revised: In order to make the elements in threshold vector t trainable via back-propagation, the derivative of the binary step function S ( x ) is required.

Original: However, its original derivative is an impulse function whose value is zero almost everywhere and inﬁnite at zero as shown in Figure 2(b).
Revised: However, its original derivative is an impulse function whose value is zero almost everywhere and inﬁnite at zero as shown in Figure 2(b).

Original: Thus the original derivative of binary step function S ( x ) cannot be applied in back-propagation and parameter updating directly.
Revised: Thus the original derivative of binary step function S ( x ) cannot be applied in back-propagation and parameter updating directly.

Original: Some previous works (Hubara et al.
Revised: Some previous works (Hubara et al.

Original: ; Rastegari et al.
Revised: ; Rastegari et al.

Original: ; Zhou et al.
Revised: ; Zhou et al.

Original: (2016)) demonstrated that by providing a derivative estimation, it is possible to train networks containing such binarization function.
Revised: (2016)) demonstrated that by providing a derivative estimation, it is possible to train networks containing such binarization function.

Original: A clip function called straight through estimator (STE) (Bengio et al., 2013) was employed in these works and is illustrated in Figure 2(c).
Revised: A clip function called straight through estimator (STE) (Bengio et al., 2013) was employed in these works and is illustrated in Figure 2(c).

Original: Furthermore, Xu & Cheung (2019) discussed the derivative estimation in balance of tight approximation and smooth back-propagation.
Revised: Furthermore, Xu & Cheung (2019) discussed the derivative estimation in balance of tight approximation and smooth back-propagation.

Original: We adopt this long-tailed higher-order estimator H ( x ) in our method.
Revised: We adopt this long-tailed higher-order estimator H ( x ) in our method.

Original: As shown in Figure 2(d), it has a wide active range between [ − 1 , 1] with non-zero gradient to avoid gradient vanishing during training.
Revised: As shown in Figure 2(d), it has a wide active range between [ − 1 , 1] with non-zero gradient to avoid gradient vanishing during training.

Original: On the other hand, the gradient value near zero is a piecewise polynomial function giving tighter approximation than STE.
Revised: On the other hand, the gradient value near zero is a piecewise polynomial function giving tighter approximation than STE.

Original: The estimator is represented as
Revised: The estimator is represented as

Original: [Equation]
Revised: [Equation]

Original: With this derivative estimator, the elements in the vector threshold t can be trained via backpropagation.
Revised: With this derivative estimator, the elements in the vector threshold t can be trained via backpropagation.

Original: Meanwhile, in trainable masked layers, the network parameter W can receive two branches of gradient, namely the performance gradient for better model performance and the structure gradient for better sparse structure, which helps to properly update the network parameter under sparse network connectivity.
Revised: Meanwhile, in trainable masked layers, the network parameter W can receive two branches of gradient, namely the performance gradient for better model performance and the structure gradient for better sparse structure, which helps to properly update the network parameter under sparse network connectivity.

Original: The structure gradient enables the pruned (masked) weights to be updated via back-propagation.
Revised: The structure gradient enables the pruned (masked) weights to be updated via back-propagation.

Original: The details about the feed-forward and back-propagation of the trainable masked layer is present in Appendix A.3.
Revised: The details about the feed-forward and back-propagation of the trainable masked layer is present in Appendix A.3.

Original: Therefore, the pruned (masked) weights, the unpruned (unmasked) weights and the elements in the threshold vector can all be updated via back-propagation at each training step.
Revised: Therefore, the pruned (masked) weights, the unpruned (unmasked) weights and the elements in the threshold vector can all be updated via back-propagation at each training step.

Original: The proposed method will conduct ﬁne-grained step-wise pruning and recovery automatically.
Revised: The proposed method will conduct ﬁne-grained step-wise pruning and recovery automatically.

Original: 3. S PARSE REGULARIZATION TERM
Revised: 3. S PARSE REGULARIZATION TERM

Original: Now that the thresholds of each layer is trainable, higher percentage of pruned parameter is desired.
Revised: Now that the thresholds of each layer is trainable, higher percentage of pruned parameter is desired.

Original: To get the parameter masks M with high sparsity, higher pruning thresholds are needed.
Revised: To get the parameter masks M with high sparsity, higher pruning thresholds are needed.

Original: To achieve this, we add a sparse regularization term L s to the training loss that penalizes the low threshold value.
Revised: To achieve this, we add a sparse regularization term L s to the training loss that penalizes the low threshold value.

Original: For a trainable masked layer with threshold t ∈ R c o , the corresponding regularization term is R = (cid:80)
Revised: For a trainable masked layer with threshold t ∈ R c o , the corresponding regularization term is R = (cid:80)

Original: c
Revised: c

Original: o i =1 exp( − t i ) .
Revised: o i =1 exp( − t i ) .

Original: Thus, the sparse regularization term L s for the a deep neural network with
Revised: Thus, the sparse regularization term L s for the a deep neural network with

Original: C trainable masked layers is:
Revised: C trainable masked layers is:

Original: [Equation]
Revised: [Equation]

Original: We use exp( − x ) as the regularization function since it is asymptotical to zero as x increases.
Revised: We use exp( − x ) as the regularization function since it is asymptotical to zero as x increases.

Original: Consequently, it penalizes low thresholds without encouraging them to become extremely large.
Revised: Consequently, it penalizes low thresholds without encouraging them to become extremely large.

Original: 3. D YNAMIC SPARSE TRAINING
Revised: 3. D YNAMIC SPARSE TRAINING

Original: The traditional fully connected, convolutional and recurrent layers can be replaced with the corresponding trainable masked layers in deep neural networks.
Revised: The traditional fully connected, convolutional and recurrent layers can be replaced with the corresponding trainable masked layers in deep neural networks.

Original: Then we can train a sparse neural network directly with back-propagation algorithm given the training dataset D
Revised: Then we can train a sparse neural network directly with back-propagation algorithm given the training dataset D

Original: = { ( x 1 , y 1 ) , ( x 2 , y 2 ) , · · · , ( x N , y N ) } , the network parameter W and the layer-wise threshold t as follows:
Revised: = { ( x 1 , y 1 ) , ( x 2 , y 2 ) , · · · , ( x N , y N ) } , the network parameter W and the layer-wise threshold t as follows:

Original: [Equation]
Revised: [Equation]

Original: where L ( · ) is the loss function, e.g. cross-entropy loss for classiﬁcation and α is the scaling coefﬁcient for the sparse regularization term, which is able to control the percentage of parameter remaining.
Revised: where L ( · ) is the loss function, e.g. cross-entropy loss for classiﬁcation and α is the scaling coefﬁcient for the sparse regularization term, which is able to control the percentage of parameter remaining.

Original: The sparse regularization term L s tends to increase the threshold t for each layer thus getting higher model sparsity.
Revised: The sparse regularization term L s tends to increase the threshold t for each layer thus getting higher model sparsity.

Original: However, higher sparsity tends to increase the loss function, which reversely tends to reduce the threshold and level of sparsity.
Revised: However, higher sparsity tends to increase the loss function, which reversely tends to reduce the threshold and level of sparsity.

Original: Consequently, the training process of the thresholds can be regarded as a contest between the sparse regularization term and the loss function in the sense of game theory.
Revised: Consequently, the training process of the thresholds can be regarded as a contest between the sparse regularization term and the loss function in the sense of game theory.

Original: Therefore, our method is able to dynamically ﬁnd the sparse structure that properly balances the model sparsity and performance.
Revised: Therefore, our method is able to dynamically ﬁnd the sparse structure that properly balances the model sparsity and performance.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: The proposed method is evaluated on MNIST, CIFAR-10 and ImageNet with various modern network architectures including fully connected, convolutional and recurrent neural networks.
Revised: The proposed method is evaluated on MNIST, CIFAR-10 and ImageNet with various modern network architectures including fully connected, convolutional and recurrent neural networks.

Original: To quantify the pruning performance, the layer remaining ratio is deﬁned to be k l = n/m , where n is the number of element equal to 1 in the mask M and m is the total number of element in M .
Revised: To quantify the pruning performance, the layer remaining ratio is deﬁned to be k l = n/m , where n is the number of element equal to 1 in the mask M and m is the total number of element in M .

Original: The model remaining ratio k m is the overall ratio of non-zero element in the parameter masks for all trainable masked layers.
Revised: The model remaining ratio k m is the overall ratio of non-zero element in the parameter masks for all trainable masked layers.

Original: The model remaining percentage is deﬁned as k m × 100% .
Revised: The model remaining percentage is deﬁned as k m × 100% .

Original: For all trainable masked layers, the trainable thresholds are initialized to zero since it is assumed that originally the network is dense.
Revised: For all trainable masked layers, the trainable thresholds are initialized to zero since it is assumed that originally the network is dense.

Original: The detailed experimental setup is present in Appendix A.1.
Revised: The detailed experimental setup is present in Appendix A.1.

Original: 4. P RUNING PERFORMANCE ON VARIOUS DATASETS
Revised: 4. P RUNING PERFORMANCE ON VARIOUS DATASETS

Original: MNIST.
Revised: MNIST.

Original: Table 1 presents the pruning results of proposed method for Lenet-300-100 (LeCun et al., 1998), Lenet-5-Caffe and LSTM (Hochreiter & Schmidhuber, 1997).
Revised: Table 1 presents the pruning results of proposed method for Lenet-300-100 (LeCun et al., 1998), Lenet-5-Caffe and LSTM (Hochreiter & Schmidhuber, 1997).

Original: Both LSTM models have two
Revised: Both LSTM models have two

Original: LSTM layers with hidden size 128 for LSTM-a and 256 for
Revised: LSTM layers with hidden size 128 for LSTM-a and 256 for

Original: LSTM-b.
Revised: LSTM-b.

Original: Our method can prune almost 98% parameter with little loss of performance on Lenet-300-100 and Lenet-5-Caffe.
Revised: Our method can prune almost 98% parameter with little loss of performance on Lenet-300-100 and Lenet-5-Caffe.

Original: For the LSTM models adapted for the sequential MNIST classiﬁcation, our method can ﬁnd sparse models with better performance than dense baseline with over 99% parameter pruned.
Revised: For the LSTM models adapted for the sequential MNIST classiﬁcation, our method can ﬁnd sparse models with better performance than dense baseline with over 99% parameter pruned.

Original: CIFAR-10.
Revised: CIFAR-10.

Original: The pruning performance of our method on CIFAR-10 is tested on VGG (Simonyan & Zisserman, 2014) and WideResNet (Zagoruyko & Komodakis, 2016).
Revised: The pruning performance of our method on CIFAR-10 is tested on VGG (Simonyan & Zisserman, 2014) and WideResNet (Zagoruyko & Komodakis, 2016).

Original: We compare our method
Revised: We compare our method

Original: [Table]
Revised: [Table]

Original: with other sparse learning algorithms on CIFAR-10 as presented in Table 2.
Revised: with other sparse learning algorithms on CIFAR-10 as presented in Table 2.

Original: The state-of-the-art algorithms, DSR (Mostafa & Wang, 2019) and Sparse momentum (Dettmers & Zettlemoyer, 2019), are selected for comparison.
Revised: The state-of-the-art algorithms, DSR (Mostafa & Wang, 2019) and Sparse momentum (Dettmers & Zettlemoyer, 2019), are selected for comparison.

Original: Dynamic Sparse Training (DST) outperforms them in almost all the settings as present in Table
Revised: Dynamic Sparse Training (DST) outperforms them in almost all the settings as present in Table

Original: [Table]
Revised: [Table]

Original: ImageNet.
Revised: ImageNet.

Original: For ResNet-50 on ImageNet (ILSVRC2012)
Revised: For ResNet-50 on ImageNet (ILSVRC2012)

Original: [Table]
Revised: [Table]

Original: we present the performance and comparison with other methods in Table 3.
Revised: we present the performance and comparison with other methods in Table 3.

Original: Dynamic Sparse Training (DST) achieves better top-1 and topaccuracy with slightly higher model sparsity.
Revised: Dynamic Sparse Training (DST) achieves better top-1 and topaccuracy with slightly higher model sparsity.

Original: The number of parameters for the dense ResNetmodel is 25.56 M.
Revised: The number of parameters for the dense ResNetmodel is 25.56 M.

Original: [Table]
Revised: [Table]

Original: 4. P RUNING PERFORMANCE ON VARIOUS REMAINING RATIO
Revised: 4. P RUNING PERFORMANCE ON VARIOUS REMAINING RATIO

Original: By varying the scaling coefﬁcient α for sparse regularization term, we can control the model remaining ratios of sparse models generated by dynamic sparse training.
Revised: By varying the scaling coefﬁcient α for sparse regularization term, we can control the model remaining ratios of sparse models generated by dynamic sparse training.

Original: The relationships between α , model remaining ratio and sparse model accuracy of VGG, WideResNet-16-8 and WideResNet-28on CIFAR-10 are presented in Figure 3.
Revised: The relationships between α , model remaining ratio and sparse model accuracy of VGG, WideResNet-16-8 and WideResNet-28on CIFAR-10 are presented in Figure 3.

Original: As demonstrated, the model remaining ratio keeps decreasing with increasing α .
Revised: As demonstrated, the model remaining ratio keeps decreasing with increasing α .

Original: With a moderate α value, it is easy to obtain a sparse model with comparable or even higher accuracy than the dense counterpart.
Revised: With a moderate α value, it is easy to obtain a sparse model with comparable or even higher accuracy than the dense counterpart.

Original: On the other side, if the α value is too large that makes the model remaining percentage less than 5%, there will be a noticeable accuracy drop.
Revised: On the other side, if the α value is too large that makes the model remaining percentage less than 5%, there will be a noticeable accuracy drop.

Original: As demonstrated in Figure 3, the choice of α ranges from 10 − 9 to 10 − 4 .
Revised: As demonstrated in Figure 3, the choice of α ranges from 10 − 9 to 10 − 4 .

Original: Depending on the application scenarios, we can either get models with similar or better performance as dense counterparts by a relatively small α or get a highly sparse model with little performance loss by a larger α .
Revised: Depending on the application scenarios, we can either get models with similar or better performance as dense counterparts by a relatively small α or get a highly sparse model with little performance loss by a larger α .

Original: [Figure]
Revised: [Figure]

Original:  D ISCUSSION 5. F INE - GRAINED STEP - WISE PRUNING
Revised:  D ISCUSSION 5. F INE - GRAINED STEP - WISE PRUNING

Original: Figure 4(a) demonstrates the change of layer remaining ratios for Lenet-300-100 trained with DST at each training step during the ﬁrst training epoch.
Revised: Figure 4(a) demonstrates the change of layer remaining ratios for Lenet-300-100 trained with DST at each training step during the ﬁrst training epoch.

Original: And Figure 4(b) presents the change of layer remaining ratios during the whole training process (20 epochs).
Revised: And Figure 4(b) presents the change of layer remaining ratios during the whole training process (20 epochs).

Original: As present in these two ﬁgures, instead of decreasing by manually set ﬁxed step size as in other pruning methods, our method makes the layer remaining ratios change smoothly and continuously at each training step.
Revised: As present in these two ﬁgures, instead of decreasing by manually set ﬁxed step size as in other pruning methods, our method makes the layer remaining ratios change smoothly and continuously at each training step.

Original: Meanwhile, as shown in Figure 4(a), the layer remaining ratios ﬂuctuate dynamically within the ﬁrst 100 training steps, which indicates that DST can achieve step-wise ﬁne-grained pruning and recovery.
Revised: Meanwhile, as shown in Figure 4(a), the layer remaining ratios ﬂuctuate dynamically within the ﬁrst 100 training steps, which indicates that DST can achieve step-wise ﬁne-grained pruning and recovery.

Original: Meanwhile, for multilayer neural networks, the parameters in different layers will have different relative importance.
Revised: Meanwhile, for multilayer neural networks, the parameters in different layers will have different relative importance.

Original: For example, Lenet-300-100 has three fully connected layers.
Revised: For example, Lenet-300-100 has three fully connected layers.

Original: Changing the parameter of the last layer (layer 3) can directly affect the model output.
Revised: Changing the parameter of the last layer (layer 3) can directly affect the model output.

Original: Thus, the parameter of layer 3 should be pruned more carefully.
Revised: Thus, the parameter of layer 3 should be pruned more carefully.

Original: The input layer (layer 1) has the largest amount of parameters and takes the raw image as input.
Revised: The input layer (layer 1) has the largest amount of parameters and takes the raw image as input.

Original: Since the images in MNIST dataset consist of many unchanged pixels (the background) that have no contribution to the classiﬁcation process, it is expected that the parameters that take these invariant background as input can be pruned safely.
Revised: Since the images in MNIST dataset consist of many unchanged pixels (the background) that have no contribution to the classiﬁcation process, it is expected that the parameters that take these invariant background as input can be pruned safely.

Original: Therefore, the remaining ratio should be the highest for layer 3 and the lowest for layer 1 if a Lenet-300-100 model is pruned.
Revised: Therefore, the remaining ratio should be the highest for layer 3 and the lowest for layer 1 if a Lenet-300-100 model is pruned.

Original: To check the pruning effect of our method on these three layers, Lenet-300-100 model is sparsely trained by the default hyperparameters setting present in Appendix A.1.
Revised: To check the pruning effect of our method on these three layers, Lenet-300-100 model is sparsely trained by the default hyperparameters setting present in Appendix A.1.

Original: The pruning trends of these three layers during dynamic sparse training is present in Figure 4(b).
Revised: The pruning trends of these three layers during dynamic sparse training is present in Figure 4(b).

Original: During the whole sparse training process, the remaining ratios of layer 1 and layer 2 keep decreasing and the remaining ratio of layer 3 maintains to be 1 after the ﬂuctuation during the ﬁrst epoch.
Revised: During the whole sparse training process, the remaining ratios of layer 1 and layer 2 keep decreasing and the remaining ratio of layer 3 maintains to be 1 after the ﬂuctuation during the ﬁrst epoch.

Original: The remaining ratio of layer 1 is the lowest and decrease to less that 10% quickly, which is consistent with the expectation.
Revised: The remaining ratio of layer 1 is the lowest and decrease to less that 10% quickly, which is consistent with the expectation.

Original: In the meantime, the test accuracy of the sparse model is almost the same as the test accuracy on dense model in the whole training process.
Revised: In the meantime, the test accuracy of the sparse model is almost the same as the test accuracy on dense model in the whole training process.

Original: This indicates that our method can properly balance the model remaining ratio and the model performance by continuous ﬁne-grained pruning throughout the entire sparse training procedure.
Revised: This indicates that our method can properly balance the model remaining ratio and the model performance by continuous ﬁne-grained pruning throughout the entire sparse training procedure.

Original: Similar training tendency can be observed in other network architectures.
Revised: Similar training tendency can be observed in other network architectures.

Original: The detailed results for other architectures are present in Appendix A.4.
Revised: The detailed results for other architectures are present in Appendix A.4.

Original: [Figure]
Revised: [Figure]

Original: 5. D YNAMIC PRUNING SCHEDULE
Revised: 5. D YNAMIC PRUNING SCHEDULE

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Dynamic adjustment regarding learning rate decay.
Revised: Dynamic adjustment regarding learning rate decay.

Original: To achieve better performance, it is a common practice to decay the learning rate for several times during the training process of deep neural network.
Revised: To achieve better performance, it is a common practice to decay the learning rate for several times during the training process of deep neural network.

Original: Usually, the test accuracy will decrease immediately just after the learning rate decay and then tend toward ﬂatness.
Revised: Usually, the test accuracy will decrease immediately just after the learning rate decay and then tend toward ﬂatness.

Original: Similar phenomenon is observed for VGG-16 on CIFAR-10 trained by
Revised: Similar phenomenon is observed for VGG-16 on CIFAR-10 trained by

Original: DST as present in Figure 5(b), where the learning rate decay from 0.1 to 0.01 at 80 epoch.
Revised: DST as present in Figure 5(b), where the learning rate decay from 0.1 to 0.01 at 80 epoch.

Original: Like the layer 3 of Lenet-300-100, the second fully connected layer (FC 2) of VGG-16 is the output layer, hence its remaining ratio is expected to be relatively high.
Revised: Like the layer 3 of Lenet-300-100, the second fully connected layer (FC 2) of VGG-16 is the output layer, hence its remaining ratio is expected to be relatively high.

Original: But a surprising observation is that the remaining ratio of FC 2 is quite low at around 0.1 for the ﬁrst 80 epochs and increases almost immediately just after the 80 epoch as present in Figure 5(a).
Revised: But a surprising observation is that the remaining ratio of FC 2 is quite low at around 0.1 for the ﬁrst 80 epochs and increases almost immediately just after the 80 epoch as present in Figure 5(a).

Original: We suppose that this is caused by the decay of learning rate from 0.1 to 0.01 at 80 epoch.
Revised: We suppose that this is caused by the decay of learning rate from 0.1 to 0.01 at 80 epoch.

Original: Before the learning rate decay, the layers preceding to the output layer fail to extract enough useful features for classiﬁcation due to the relatively coarse-grained parameter adjustment incurred by high learning rate.
Revised: Before the learning rate decay, the layers preceding to the output layer fail to extract enough useful features for classiﬁcation due to the relatively coarse-grained parameter adjustment incurred by high learning rate.

Original: This means that the corresponding parameters that take those useless features as input can be pruned, which leads to the low remaining ratio of the output layer.
Revised: This means that the corresponding parameters that take those useless features as input can be pruned, which leads to the low remaining ratio of the output layer.

Original: The decayed learning rate enables ﬁne-grained parameter update that makes the neural network model converge to the good local minima quickly (Kawaguchi, 2016), where most of the features extracted by the preceding layers turn to be helpful for the classiﬁcation.
Revised: The decayed learning rate enables ﬁne-grained parameter update that makes the neural network model converge to the good local minima quickly (Kawaguchi, 2016), where most of the features extracted by the preceding layers turn to be helpful for the classiﬁcation.

Original: This makes previously unimportant network connections in the output layer become important thus the remaining ratio of this layer get abrupt increase.
Revised: This makes previously unimportant network connections in the output layer become important thus the remaining ratio of this layer get abrupt increase.

Original: There are two facts that support our assumptions for this phenomenon.
Revised: There are two facts that support our assumptions for this phenomenon.

Original: The ﬁrst fact is the sudden increase of the test accuracy from around 85% to over 92% just after the learning rate decay.
Revised: The ﬁrst fact is the sudden increase of the test accuracy from around 85% to over 92% just after the learning rate decay.

Original: Secondly, the remaining ratios of the preceding convolutional layers are almost unchanged after the remaining ratio of the output layer increases up to 1, which means that the remaining parameters in these layers are necessary to ﬁnd the critical features.
Revised: Secondly, the remaining ratios of the preceding convolutional layers are almost unchanged after the remaining ratio of the output layer increases up to 1, which means that the remaining parameters in these layers are necessary to ﬁnd the critical features.

Original: The abrupt change of the remaining ratio of the output layer indicates that our method can dynamically adjust the pruning schedule regarding the change of hyperparameters during the training process.
Revised: The abrupt change of the remaining ratio of the output layer indicates that our method can dynamically adjust the pruning schedule regarding the change of hyperparameters during the training process.

Original: Dynamic adjustment regarding different initial learning rate.
Revised: Dynamic adjustment regarding different initial learning rate.

Original: To further investigate the effect of learning rate, VGG-16 model is trained by different initial learning rate (0.01) with other hyperparameters unchanged.
Revised: To further investigate the effect of learning rate, VGG-16 model is trained by different initial learning rate (0.01) with other hyperparameters unchanged.

Original: The corresponding training curves are presented in Figure 6.
Revised: The corresponding training curves are presented in Figure 6.

Original: As shown in
Revised: As shown in

Original: Figure 6(a), the remaining ratio of FC 2 increases up to 1 after around 40 epochs when the test accuracy reaches over 90%.
Revised: Figure 6(a), the remaining ratio of FC 2 increases up to 1 after around 40 epochs when the test accuracy reaches over 90%.

Original: This means that with proper update of network parameters due to a smaller initial learning rate, the layers preceding to the output layer (FC 2) tend to extract useful features before the learning rate decay.
Revised: This means that with proper update of network parameters due to a smaller initial learning rate, the layers preceding to the output layer (FC 2) tend to extract useful features before the learning rate decay.

Original: It can be observed in Figure 6(b) that the test accuracy only increase about 2% from 90% to 92% roughly after the learning rate decay at the 80 epoch.
Revised: It can be observed in Figure 6(b) that the test accuracy only increase about 2% from 90% to 92% roughly after the learning rate decay at the 80 epoch.

Original: Meanwhile, one can see that our method adopts different pruning schedule regarding different initial choices of training hyperparameters.
Revised: Meanwhile, one can see that our method adopts different pruning schedule regarding different initial choices of training hyperparameters.

Original: Model performance under dynamic schedule.
Revised: Model performance under dynamic schedule.

Original: Figure 5(b) and Figure 6(b) show the model remaining ratio, test accuracy of sparse model (sparse accuracy) and test accuracy of dense counterparts (dense accuracy) after each epoch during the whole training process.
Revised: Figure 5(b) and Figure 6(b) show the model remaining ratio, test accuracy of sparse model (sparse accuracy) and test accuracy of dense counterparts (dense accuracy) after each epoch during the whole training process.

Original: For sparse model trained with initial learning rate 0.1, the ﬁnal model remaining percentage is 8.82%.
Revised: For sparse model trained with initial learning rate 0.1, the ﬁnal model remaining percentage is 8.82%.

Original: The test accuracy of this sparse model is 93.93%, which is better than 93.75% of the dense counterparts.
Revised: The test accuracy of this sparse model is 93.93%, which is better than 93.75% of the dense counterparts.

Original: Similarly, for sparse model trained with initial learning rate 0.01, the ﬁnal model remaining percentage is 12.45%.
Revised: Similarly, for sparse model trained with initial learning rate 0.01, the ﬁnal model remaining percentage is 12.45%.

Original: The test accuracy of this sparse model is 92.74%, which is also better than 92.54% of the dense model.
Revised: The test accuracy of this sparse model is 92.74%, which is also better than 92.54% of the dense model.

Original: Considering the model performance during the whole training process, when trained with initial learning rate 0.01, the sparse accuracy is consistent higher than dense accuracy during the whole training process as present in Figure 6(b).
Revised: Considering the model performance during the whole training process, when trained with initial learning rate 0.01, the sparse accuracy is consistent higher than dense accuracy during the whole training process as present in Figure 6(b).

Original: Meanwhile, one can see from Figure 5(b) that the running average of sparse accuracy is also higher than that of dense accuracy during the whole training process when the initial learning rate is 0.1.
Revised: Meanwhile, one can see from Figure 5(b) that the running average of sparse accuracy is also higher than that of dense accuracy during the whole training process when the initial learning rate is 0.1.

Original: 5. I NFORMATION REVEALED FROM CONSISTENT SPARSE PATTERN
Revised: 5. I NFORMATION REVEALED FROM CONSISTENT SPARSE PATTERN

Original: Many works have tried to design more compact model with mimic performance to overparameterized models (He et al., 2016; Howard et al., 2017; Zhang et al., 2018).
Revised: Many works have tried to design more compact model with mimic performance to overparameterized models (He et al., 2016; Howard et al., 2017; Zhang et al., 2018).

Original: Network architecture search has been viewed as the future paradigm of deep neural network design.
Revised: Network architecture search has been viewed as the future paradigm of deep neural network design.

Original: However, it is extremely difﬁcult to determine whether the designed layer consists of redundancy.
Revised: However, it is extremely difﬁcult to determine whether the designed layer consists of redundancy.

Original: Therefore, typical network architecture search methods rely on evolutionary algorithms (Liu et al., 2017) or reinforcement learning (Baker et al., 2016), which is extremely time-consuming and computationally expensive.
Revised: Therefore, typical network architecture search methods rely on evolutionary algorithms (Liu et al., 2017) or reinforcement learning (Baker et al., 2016), which is extremely time-consuming and computationally expensive.

Original: Network pruning can actually be reviewed as a kind of architecture search process (Liu et al., 2018; Frankle & Carbin, 2018) thus the sparse structure revealed from network pruning may provide some guidance to the network architecture design.
Revised: Network pruning can actually be reviewed as a kind of architecture search process (Liu et al., 2018; Frankle & Carbin, 2018) thus the sparse structure revealed from network pruning may provide some guidance to the network architecture design.

Original: However, The layer-wise equal remaining ratio generated by the uniﬁed pruning strategy fails to indicate the different degree of redundancy for each layer.
Revised: However, The layer-wise equal remaining ratio generated by the uniﬁed pruning strategy fails to indicate the different degree of redundancy for each layer.

Original: And the global pruning algorithm is non-robust, which fails to offer consistent guidance.
Revised: And the global pruning algorithm is non-robust, which fails to offer consistent guidance.

Original: Here we demonstrate another interesting observation called consistent sparse pattern during dynamic sparse training that provides useful information about the redundancy of individual layers as the guidance for compact network architecture design.
Revised: Here we demonstrate another interesting observation called consistent sparse pattern during dynamic sparse training that provides useful information about the redundancy of individual layers as the guidance for compact network architecture design.

Original: For the same architecture trained by our method with various α values, the relative relationship of sparsity among different layers keeps consistent.
Revised: For the same architecture trained by our method with various α values, the relative relationship of sparsity among different layers keeps consistent.

Original: The sparse patterns of VGG-16 on CIFAR-10 are present in Figure 7 with three different α .
Revised: The sparse patterns of VGG-16 on CIFAR-10 are present in Figure 7 with three different α .

Original: In all conﬁgurations, the last four convolutional layers (Conv 10-13) and the ﬁrst fully connected layers (FC 1) are highly sparse.
Revised: In all conﬁgurations, the last four convolutional layers (Conv 10-13) and the ﬁrst fully connected layers (FC 1) are highly sparse.

Original: Meanwhile, some layers (Conv 3-7, FC 2) keep a high amount of parameters after pruning.
Revised: Meanwhile, some layers (Conv 3-7, FC 2) keep a high amount of parameters after pruning.

Original: This consistent sparse pattern indicates that these heavily pruned layers consist of high redundancy and a large number of parameters can be reduced in these layers to get more compact models.
Revised: This consistent sparse pattern indicates that these heavily pruned layers consist of high redundancy and a large number of parameters can be reduced in these layers to get more compact models.

Original: This phenomenon also exists in other network architectures, which is present in details in the appendix A.5.
Revised: This phenomenon also exists in other network architectures, which is present in details in the appendix A.5.

Original: Therefore, with this consistent sparse pattern, after designing a new network architecture, our method can be applied to get useful information about the layer-wise redundancy in this new architecture.
Revised: Therefore, with this consistent sparse pattern, after designing a new network architecture, our method can be applied to get useful information about the layer-wise redundancy in this new architecture.

Original: [Figure]
Revised: [Figure]

Original: R
Revised: R

Original: A A PPENDIX A. E XPERIMENTAL SETUP
Revised: A A PPENDIX A. E XPERIMENTAL SETUP

Original: MNIST : LeNet-300-100 and LeNet-5-Caffe are trained using SGD with momentum of 0.9 and batch size of 64 with 20 training epochs.
Revised: MNIST : LeNet-300-100 and LeNet-5-Caffe are trained using SGD with momentum of 0.9 and batch size of 64 with 20 training epochs.

Original: The learning rate is 0.01 without learning rate decay.
Revised: The learning rate is 0.01 without learning rate decay.

Original: Meanwhile, the default scaling factor α for sparse regularization term is 0.0005 for both network architectures.
Revised: Meanwhile, the default scaling factor α for sparse regularization term is 0.0005 for both network architectures.

Original: LSTM models are trained using Adam optimization scheme (Kingma & Ba, 2014) with default
Revised: LSTM models are trained using Adam optimization scheme (Kingma & Ba, 2014) with default

Original: Adam hyperparameter setting for 20 epochs.
Revised: Adam hyperparameter setting for 20 epochs.

Original: The batch size is 100 and the default learning rate is 0.001.
Revised: The batch size is 100 and the default learning rate is 0.001.

Original: Meanwhile, the default scaling factor α for sparse regularization term is 0.001.
Revised: Meanwhile, the default scaling factor α for sparse regularization term is 0.001.

Original: CIFAR-10 Models on CIFAR-10 are trained using SGD with momentum 0.9 and batch size ofwith 160 training epochs.
Revised: CIFAR-10 Models on CIFAR-10 are trained using SGD with momentum 0.9 and batch size ofwith 160 training epochs.

Original: The initial learning rate is 0.1 and decayed by 0.1 at 80 and 120 epoch.
Revised: The initial learning rate is 0.1 and decayed by 0.1 at 80 and 120 epoch.

Original: The default scaling factor α for sparse regularization term is 5 × 10 − 6 for all tested architectures.
Revised: The default scaling factor α for sparse regularization term is 5 × 10 − 6 for all tested architectures.

Revised: ImageNet.

Revised: ResNet-50 models are trained using SGD with momentum 0.9 and batch size ofwith 90 training epochs.

Revised: The default initial learning rate is 0.1 and decayed by 10 at 30, 60,epochs.

Original: For all trainable masked layers, the trainable thresholds are initialized to zero.
Revised: For all trainable masked layers, the trainable thresholds are initialized to zero.

Original: Additionally, we ﬁnd that extremely high scaling coefﬁcient α will make the sparse regularization term dominates the training loss thus making the mask M to be all zero in certain layer, which may impede the training process.
Revised: Additionally, we ﬁnd that extremely high scaling coefﬁcient α will make the sparse regularization term dominates the training loss thus making the mask M to be all zero in certain layer, which may impede the training process.

Original: To prevent this, the pruning threshold t will be reset to zero if more than 99% elements in the mask are zero in a certain layer despite the LSTM models.
Revised: To prevent this, the pruning threshold t will be reset to zero if more than 99% elements in the mask are zero in a certain layer despite the LSTM models.

Original: This mechanism makes the training process smoother and enables a wider range choice of α .
Revised: This mechanism makes the training process smoother and enables a wider range choice of α .

Original: A. A NALYSIS OF DIFFERENT THRESHOLD CHOICES
Revised: A. A NALYSIS OF DIFFERENT THRESHOLD CHOICES

Original: Beside a threshold vector, a threshold scalar t or threshold matrix T ∈
Revised: Beside a threshold vector, a threshold scalar t or threshold matrix T ∈

Original: R
Revised: R

Original: c
Revised: c

Original: o × c i can also be adopted for each trainable masked layer with parameter W ∈ R c
Revised: o × c i can also be adopted for each trainable masked layer with parameter W ∈ R c

Original: o × c i .
Revised: o × c i .

Original: [Equation]
Revised: [Equation]

Original: All the three choices of trainable thresholds are tested on various architectures.
Revised: All the three choices of trainable thresholds are tested on various architectures.

Original: Considering the effects on network pruning, the matrix threshold has almost the same model remaining ratio compared with the vector threshold.
Revised: Considering the effects on network pruning, the matrix threshold has almost the same model remaining ratio compared with the vector threshold.

Original: The scalar threshold is less robust than the vector and matrix threshold.
Revised: The scalar threshold is less robust than the vector and matrix threshold.

Original: In terms of the storage overhead, the matrix threshold almost doubles the amount of parameter during the training process in each architectures.
Revised: In terms of the storage overhead, the matrix threshold almost doubles the amount of parameter during the training process in each architectures.

Original: Meanwhile, the vector threshold only adds less than 1% additional network parameter.
Revised: Meanwhile, the vector threshold only adds less than 1% additional network parameter.

Original: The extra trainable threshold will also bring additional computation.
Revised: The extra trainable threshold will also bring additional computation.

Original: In both feed forward and backpropagation phase, the additional computations are matrix-element-wise operations ( O ( n 2 ) ), that is apparently light-weighted compared to the original batched matrix multiplication ( O ( n 3 ) ).
Revised: In both feed forward and backpropagation phase, the additional computations are matrix-element-wise operations ( O ( n 2 ) ), that is apparently light-weighted compared to the original batched matrix multiplication ( O ( n 3 ) ).

Original: For practical deployment, only the masked parameter W (cid:12) M need to be stored, thus no overhead will be introduced.
Revised: For practical deployment, only the masked parameter W (cid:12) M need to be stored, thus no overhead will be introduced.

Original: Therefore, considering the balance between the overhead and the beneﬁt incurred by these three choices, the vector threshold is chosen for trainable masked layers.
Revised: Therefore, considering the balance between the overhead and the beneﬁt incurred by these three choices, the vector threshold is chosen for trainable masked layers.

Original: A. M ORE DETAILS ABOUT THE TRAINABLE MASKED LAYER
Revised: A. M ORE DETAILS ABOUT THE TRAINABLE MASKED LAYER

Original: Feed forward process.
Revised: Feed forward process.

Original: Considering a single layer in deep neural network with input x and dense parameter W .
Revised: Considering a single layer in deep neural network with input x and dense parameter W .

Original: The normal layer will conduct matrix-vector multiplication as W x or convolution as Conv ( W , x ) .
Revised: The normal layer will conduct matrix-vector multiplication as W x or convolution as Conv ( W , x ) .

Original: In trainable masked layers, since a sparse mask M is obtained for each layer.
Revised: In trainable masked layers, since a sparse mask M is obtained for each layer.

Original: The sparse parameter W (cid:12) M will be adopted in the corresponding matrix-vector multiplication or convolution as ( W (cid:12) M ) x and Conv ( W (cid:12) M , x ) .
Revised: The sparse parameter W (cid:12) M will be adopted in the corresponding matrix-vector multiplication or convolution as ( W (cid:12) M ) x and Conv ( W (cid:12) M , x ) .

Original: Back-propagation process.
Revised: Back-propagation process.

Original: Referring to Figure 1, we denotes P = W (cid:12) M and the gradient received by P in back-propagation as d P .
Revised: Referring to Figure 1, we denotes P = W (cid:12) M and the gradient received by P in back-propagation as d P .

Original: Considering the gradients from right to left:
Revised: Considering the gradients from right to left:

Original: • The performance gradient is d P (cid:12) M • The gradient received by M is d P (cid:12) W • The gradient received by Q is d P (cid:12) W (cid:12) H ( Q ) , where H ( Q ) is the result of H ( x ) applied to Q elementwisely.
Revised: • The performance gradient is d P (cid:12) M • The gradient received by M is d P (cid:12) W • The gradient received by Q is d P (cid:12) W (cid:12) H ( Q ) , where H ( Q ) is the result of H ( x ) applied to Q elementwisely.

Original: • The structure gradient is d P (cid:12) W (cid:12) H ( Q ) (cid:12) sgn ( W ) , where sgn ( W ) is the result of sign function applied to W elementwisely.
Revised: • The structure gradient is d P (cid:12) W (cid:12) H ( Q ) (cid:12) sgn ( W ) , where sgn ( W ) is the result of sign function applied to W elementwisely.

Original: • The gradient received by the vector threshold t is d t ∈ R
Revised: • The gradient received by the vector threshold t is d t ∈ R

Original: c
Revised: c

Original: o .
Revised: o .

Original: We denote d T = − d P (cid:12) W (cid:12) H ( Q ) , then d T ∈ R c o × c
Revised: We denote d T = − d P (cid:12) W (cid:12) H ( Q ) , then d T ∈ R c o × c

Original: i .
Revised: i .

Original: And we will have d t i
Revised: And we will have d t i

Original: =
Revised: =

Original: (cid:80) c
Revised: (cid:80) c

Original: i j =1 T ij for 1 ≤ i ≤ c o .
Revised: i j =1 T ij for 1 ≤ i ≤ c o .

Original: The gradient received by the parameter W is dW = d P (cid:12)
Revised: The gradient received by the parameter W is d

Original: [Equation]
Revised: W = d P (cid:12) M + d P (cid:12) W (cid:12) H ( Q ) (cid:12) sgn ( W )

Original: Since we add (cid:96) 2 regularization in the training process, all the elements in W are distributed within [ − 1 , 1] .
Revised: Since we add (cid:96) 2 regularization in the training process, all the elements in W are distributed within [ − 1 , 1] .

Original: Meanwhile, almost all the elements in the vector threshold are distributed within [0 , 1] .
Revised: Meanwhile, almost all the elements in the vector threshold are distributed within [0 , 1] .

Original: The exceptions are the situation as shown in Figure 3(a) and Figure 4(a) where the last layer get no weight pruned (masked).
Revised: The exceptions are the situation as shown in Figure 3(a) and Figure 4(a) where the last layer get no weight pruned (masked).

Original: Regarding the process of getting Q , all the elements in Q are within [ − 1 , 1] .
Revised: Regarding the process of getting Q , all the elements in Q are within [ − 1 , 1] .

Original: Therefore H ( Q ) is a dense matrix.
Revised: Therefore H ( Q ) is a dense matrix.

Original: Then W , H ( Q ) and sgn ( W ) are all dense matrices and the pruned (masked) weights can receive the structure gradient d P (cid:12) W (cid:12) H ( Q ) (cid:12) sgn ( W )
Revised: Then W , H ( Q ) and sgn ( W ) are all dense matrices and the pruned (masked) weights can receive the structure gradient d P (cid:12) W (cid:12) H ( Q ) (cid:12) sgn ( W )

Original: A. C HANGE OF REMAINING RATIO IN OTHER NETWORK ARCHITECTURES
Revised: A. C HANGE OF REMAINING RATIO IN OTHER NETWORK ARCHITECTURES

Original: Here we present the change of remaining ratios during dynamic sparse training for the other tested architectures.
Revised: Here we present the change of remaining ratios during dynamic sparse training for the other tested architectures.

Original: Since WideResNet models only have one fully connected layer at the end as the output layer, the ﬁrst ﬁve convolutional layers and the last fully connected layer are present.
Revised: Since WideResNet models only have one fully connected layer at the end as the output layer, the ﬁrst ﬁve convolutional layers and the last fully connected layer are present.

Original: Figure 8 demonstrates the corresponding result for WideResNet-16-8.
Revised: Figure 8 demonstrates the corresponding result for WideResNet-16-8.

Original: Figure 9 demonstrates the corresponding result for WideResNet-16-10.
Revised: Figure 9 demonstrates the corresponding result for WideResNet-16-10.

Original: And Figure 10 demonstrates the corresponding result for
Revised: And Figure 10 demonstrates the corresponding result for

Original: WideResNet-28-8.
Revised: WideResNet-28-8.

Original: The similar phenomenon can be observed in all these three network architectures for various α .
Revised: The similar phenomenon can be observed in all these three network architectures for various α .

Original: A. C ONSISTENT SPARSE PATTERN IN OTHER NETWORK ARCHITECTURES
Revised: A. C ONSISTENT SPARSE PATTERN IN OTHER NETWORK ARCHITECTURES

Original: Here we present the consistent sparse pattern for the other tested architectures.
Revised: Here we present the consistent sparse pattern for the other tested architectures.

Original: Figure 11 demonstrates the corresponding result for WideResNet-16-8.
Revised: Figure 11 demonstrates the corresponding result for WideResNet-16-8.

Original: And Figure 12 demonstrates the corresponding result for WideResNet-16-10.
Revised: And Figure 12 demonstrates the corresponding result for WideResNet-16-10.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure] 
Revised: [Figure] [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]