Original: P ARAMETER EFFICIENT TRAINING OF DEEP CONVOLUTIONAL NEURAL NETWORKS BY DYNAMIC SPARSE REPARAMETERIZATION
Revised: P ARAMETER EFFICIENT TRAINING OF DEEP CONVOLUTIONAL NEURAL NETWORKS BY DYNAMIC SPARSE REPARAMETERIZATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Modern deep neural networks are highly overparameterized, and often of huge sizes.
Revised: Modern deep neural networks are highly overparameterized, and often of huge sizes.

Original: A number of post-training model compression techniques, such as distillation, pruning and quantization, can reduce the size of network parameters by a substantial fraction with little loss in performance.
Revised: A number of post-training model compression techniques, such as distillation, pruning and quantization, can reduce the size of network parameters by a substantial fraction with little loss in performance.

Original: However, training a small network of the post-compression size de novo typically fails to reach the same level of accuracy achieved by compression of a large network, leading to a widely-held belief that gross overparameterization is essential to effective learning.
Revised: However, training a small network of the post-compression size de novo typically fails to reach the same level of accuracy achieved by compression of a large network, leading to a widely-held belief that gross overparameterization is essential to effective learning.

Original: In this work, we argue that this is not necessarily true.
Revised: In this work, we argue that this is not necessarily true.

Original: We describe a dynamic sparse reparameterization technique that closed the performance gap between a model compressed by pruning and a model of the post-compression size trained de novo .
Revised: We describe a dynamic sparse reparameterization technique that closed the performance gap between a model compressed through iterative pruning and a model of the post-compression size trained de novo .

Original: We applied our method to training deep residual networks and showed that it outperformed existingstatic reparameterization techniques, yielding the best accuracy for a given parameter budget for training.
Revised: We applied our method to training deep residual networks and showed that it outperformed existing reparameterization techniques, yielding the best accuracy for a given parameter budget for training.

Original: Compared to other dynamic reparameterization methods that reallocate non-zero parameters during training, our approachbroke free from a few key limitations and achievedmuch better performance at lower computational cost.
Revised: Compared to existing dynamic reparameterization methods that reallocate non-zero parameters during training, our approach achieved better performance at lower computational cost.

Original: Our method is not only of practical value for training under stringent memory constraints, but also potentially informative to theoretical understanding of generalization properties of overparameterized deep neural networks.
Revised: Our method is not only of practical value for training under stringent memory constraints, but also potentially informative to theoretical understanding of generalization properties of overparameterized deep neural networks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep neural network’s success is due in no small part to two of its surprising properties that starkly defy conventional wisdom.
Revised: Deep neural network’s success is due in no small part to two of its surprising properties that starkly defy conventional wisdom.

Original: First, training of modern deep neural networks is a high-dimensional non-convex optimization problem, a kind prohibitively difﬁcult in general from an optimization theoretic viewpoint.
Revised: First, training of modern deep neural networks is a high-dimensional non-convex optimization problem, a kind prohibitively difﬁcult in general from an optimization theoretic viewpoint.

Original: Yet ﬁrst-order gradient-based methods, stochastic gradient descent (SGD) and its variants, prove to be very effective; training seldom gets trapped in bad local minima.
Revised: Yet ﬁrst-order gradient-based methods, stochastic gradient descent (SGD) and its variants, prove to be very effective; training seldom gets trapped in bad local minima.

Original: Second, overparameterization (i.e. having more model parameters than training data examples) does not undermine generalization; highly overparameterized models in practice seldom overﬁt, a phenomenon at odds with traditional principles of statistical learning theory.
Revised: Second, overparameterization (i.e. having more model parameters than training data examples) does not undermine generalization; highly overparameterized models in practice seldom overﬁt, a phenomenon at odds with traditional principles of statistical learning theory.

Original: Simply put, huge models are not hard to train, and trained huge models are not bad ones.
Revised: Simply put, huge models are not hard to train, and trained huge models are not bad ones.

Original: Emerging evidence has attributed these effects to the geometry of high-dimensional loss landscapes of overparameterized deep neural networks (Dauphin et al., 2014; Choromanska et al., 2014; Goodfellow et al., 2014; Im et al., 2016;
Revised: Emerging evidence has attributed these effects to the geometry of high-dimensional loss landscapes of overparameterized deep neural networks (Dauphin et al., 2014; Choromanska et al., 2014; Goodfellow et al., 2014; Im et al., 2016;

Original: Wu et al., 2017; Liao & Poggio, 2017; Cooper, 2018; Novak et al., 2018), and to the implicit regularization properties of SGD (Brutzkus et al., 2017; Zhang et al., 2018a;
Revised: Wu et al., 2017; Liao & Poggio, 2017; Cooper, 2018; Novak et al., 2018), and to the implicit regularization properties of SGD (Brutzkus et al., 2017; Zhang et al., 2018a;

Original: Poggio et al., 2017), though a thorough theoretical understanding is not yet complete.
Revised: Poggio et al., 2017), though a thorough theoretical understanding is not yet complete.

Original: On the other hand, practitioners, blessed in this reality, do not seem to shy away from building gigantic models.
Revised: On the other hand, practitioners, blessed in this reality, do not seem to shy away from building gigantic models.

Original: Models with parameters in the tens and hundreds of millions are common in practice, with the current record holder having a parameter count in the range of a hundred billion (Shazeer et al., 2017).
Revised: Models with parameters in the tens and hundreds of millions are common in practice, with the current record holder having a parameter count in the range of a hundred billion (Shazeer et al., 2017).

Original: However, huge models are computationally expensive, both spatially (in terms of memory requirements) and temporally (in terms of number of operations), at both training time and inference time.
Revised: However, huge models are computationally expensive, both spatially (in terms of memory requirements) and temporally (in terms of number of operations), at both training time and inference time.

Original: Thus, scalable methods for controlling the computational complexity of deep neural networks are key to making them more useful in practice.
Revised: Thus, scalable methods for controlling the computational complexity of deep neural networks are key to making them more useful in practice.

Original: To achieve efﬁcient inference, a number of well-established model compression techniques are highly effective in reducing the post-training model size with little degradation in performance.
Revised: To achieve efﬁcient inference, a number of well-established model compression techniques are highly effective in reducing the post-training model size with little degradation in performance.

Original: These include distillation (e.g. Bucilua et al.
Revised: These include distillation (e.g. Bucilua et al.

Original: ; Hinton et al.
Revised: ; Hinton et al.

Original: , quantization (e.g. Hubara et al.
Revised: , quantization (e.g. Hubara et al.

Original: ; McDonnell (2018)), low-rank decomposition (e.g. Denil et al.
Revised: ; McDonnell (2018)), low-rank decomposition (e.g. Denil et al.

Original: ; Jaderberg et al.
Revised: ; Jaderberg et al.

Original: (2014)), and pruning (e.g. Han et al.
Revised: (2014)), and pruning (e.g. Han et al.

Original: ; Zhang et al.
Revised: ; Zhang et al.

Original: (2018b))
Revised: (2018b))

Original: , to name a few.
Revised: , to name a few.

Original: Despite their success, post-training compression methods require the full overparameterized model to be trained in the ﬁrst place.
Revised: Despite their success, post-training compression methods require the full overparameterized model to be trained in the ﬁrst place.

Original: Training remains expensive.
Revised: Training remains expensive.

Original: In order to make training more efﬁcient, signiﬁcant efforts have been invested into numerical innovations for training at limited precision (e.g.
Revised: In order to make training more efﬁcient, signiﬁcant efforts have been invested into numerical innovations for training at limited precision (e.g.

Original: Courbariaux et al.
Revised: Courbariaux et al.

Original: (2016); K¨oster et al.
Revised: (2016); K¨oster et al.

Original: In contrast, however, little progress has been made in effective training at signiﬁcantly reduced number of parameters.
Revised: In contrast, however, little progress has been made in effective training at signiﬁcantly reduced number of parameters.

Original: One obvious alternative toward this goal is to seek novel network architectures, ones that are more parameter efﬁcient 1 than the original.
Revised: One obvious alternative toward this goal is to seek novel network architectures, ones that are more parameter efﬁcient 1 .

Original: In fact, recent architectural innovations in deep convolutional neural networks (CNNs) not only achieved better generalization performance, but parameter efﬁciency as well (see Section 5).
Revised: In fact, recent architectural innovations in deep convolutional neural networks (CNNs) not only achieved better generalization performance, but parameter efﬁciency as well (see Section 5).

Original: Instead of inventing new networks, a more desirable approach is to achieve higher parameter efﬁciency directly by reparameterizing an existing model architecture.
Revised: Instead of inventing new networks, an alternative approach is to achieve higher parameter efﬁciency directly by reparameterizing an existing model architecture.

Original: In general, any differentiable reparameterization can be used to augment training of a given model.
Revised: In general, any differentiable reparameterization can be used to augment training of a given model.

Original: Let an original network (or a layer therein) be denoted by y = f
Revised: Let an original network (or a layer therein) be denoted by y = f

Original: ( x ; θ ) , parameterized by θ ∈ Θ .
Revised: ( x ; θ ) , parameterized by θ ∈ Θ .

Original: Reparameterize it by φ ∈ Φ through θ =
Revised: Reparameterize it by φ ∈ Φ through θ =

Original: g ( φ ; ψ ) , where g is differentiable w.r.t.
Revised: g ( φ ; ψ ) , where g is differentiable w.r.t.

Original: φ but not necessarily w.r.t.
Revised: φ but not necessarily w.r.t.

Original: ψ .
Revised: ψ .

Original: Denote the reparameterized network by f ψ , considering ψ as metaparameters 2 :
Revised: Denote the reparameterized network by f ψ , considering ψ as metaparameters 2 :

Original: [Equation]
Revised: [Equation]

Original: Training of f ψ is done by backpropagation further through g , as ∂∂ φ = ∂g∂ φ ∂∂g .
Revised: Training of f ψ is done by backpropagation further through g , as ∂∂ φ = ∂g∂ φ ∂∂g .

Original: If it is so chosen that dim( Φ ) < dim( Θ ) and f ψ ≈ f
Revised: If it is so chosen that dim( Φ ) < dim( Θ ) and f ψ ≈ f

Original: in terms of its generalization performance, then f ψ is a more parameter efﬁcient approximation of f .
Revised: in terms of its generalization performance, then f ψ is a more parameter efﬁcient approximation of f .

Original: Sparse reparameterization is a special case where g is a linear projection; φ is the non-zero entries (i.e. “weights”) and ψ their indices (i.e. “connectivity”) in the original parameter θ .
Revised: Sparse reparameterization is a special case where g is a linear projection; φ is the non-zero entries (i.e. “weights”) and ψ their indices (i.e. “connectivity”) in the original parameter θ .

Original: Likewise, parameter sharing (or more generally linear constraints of parameters ) is a similar special case of linear reparameterization where φ is the tied parameters and ψ their indices.
Revised: Likewise, parameter sharing (or more generally linear constraints of parameters ) is a similar special case of linear reparameterization where φ is the tied parameters and ψ their indices.

Original: In this sense a convolution layer can be considered a reparameterization of a fully connected layer.
Revised: In this sense a convolution layer can be considered a reparameterization of a fully connected layer.

Original: Further, if metaparameters ψ are ﬁxed during the course of training, the reparameterization is static , whereas if ψ is adjusted adaptively during training, we call it dynamic reparameterization.
Revised: Further, if metaparameters ψ are ﬁxed during the course of training, the reparameterization is static , whereas if ψ is adjusted adaptively during training, we call it dynamic reparameterization.

Original: Now, the question is: given a full model f , is it possible to ﬁnd a more efﬁcient reparameterization f ψ that, trained de novo , can generalize comparably well?
Revised: Now, the question is: given a full model f , is it possible to ﬁnd a more efﬁcient reparameterization f ψ that, trained de novo , can generalize comparably well?

Original: The success of various model compression techniques suggests that such reparameterizations might exist for most well-known models, and a recent study (Frankle & Carbin, 2018) made successful post hoc identiﬁcations of sparse reparameterized small networks with precisely such properties.
Revised: The success of various model compression techniques suggests that such reparameterizations might exist for most well-known models, and a recent study (Frankle & Carbin, 2018) made successful post hoc identiﬁcations of sparse reparameterized small networks with precisely such properties.

Original: Nevertheless,all attempts of training small networks de novo have yielded results signiﬁcantly underperforming those compressed from the original models (Zhu & Gupta, 2017).
Revised: Nevertheless, attempts at training small networks de novo typically yield results signiﬁcantly underperforming networks obtained by compressing larger models (Zhu & Gupta, 2017).

Original: This has led to a commonly held belief that gross overparameterization is necessary for effective training, and it is further hypothesized that this necessity is due to the haphazard occurrences of good reparameterizations and their sensitivity to initialization (Frankle & Carbin, 2018).
Revised: This has led to a commonly held belief that gross overparameterization is necessary for effective training.

Original: To date, a method capable of training a  sparse network de novo to match the performance of a dense model compressed to the same size remains out of reach. In this study we present a dynamic sparse reparameterization technique that successfully overcame this challenge.
Revised: Here we argue that this is not true by presenting a dynamic sparse reparameterization technique able to train sparse models de novo without the need to compress a large model, a desirable feature for training on memory- and power-constrained devices.

Original: Ourmajor discoveries and contributions are listed as follows.
Revised: Our contributions are listed as follows.

Original: Ours is the ﬁrst systematic method able to train  sparse models directly without an increased parameter footprint during the entire course of training, and still achieve performance  on par with post-training compression of  dense models,  the best result at a given sparsity.We described the ﬁrst dynamic reparameterization method for training convolutional networks.
Revised: We showed that it is possible to train a small sparse network directly without a larger than inference-time parameter footprint at any stages of training, yet still achieving generalization performance at least on par with post-training iterative pruning of large dense models, yielding the most parameter efﬁcient model at a given sparsity.

Original: We showed that ourdynamic sparse reparameterization signiﬁcantly outperformed static ones. Our method not only outperformed existing dynamic sparse reparameterization techniques, but also incurred much lower computational costs.
Revised: We showed that our method is more scalable and efﬁcient, and leads to signiﬁcantly better accuracy than existing dynamic sparse reparameterization training techniques.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Training of differentiably reparameterized networks has been proposed in numerous studies before.
Revised: Training of differentiably reparameterized networks has been proposed in numerous studies before.

Original: Dense reparameterization Several dense reparameterization techniques sought to reduce the size of fully connected layers.
Revised: Dense reparameterization Several dense reparameterization techniques sought to reduce the size of fully connected layers.

Original: These include low-rank decomposition (Denil et al., 2013), fastfood transform (Yang et al., 2014), ACDC transform (Moczulski et al., 2015),
Revised: These include low-rank decomposition (Denil et al., 2013), fastfood transform (Yang et al., 2014), ACDC transform (Moczulski et al., 2015),

Original: HashedNet (Chen et al., 2015), low displacement rank (Sindhwani et al., 2015) and block-circulant matrix parameterization (Treister et al., 2018).
Revised: HashedNet (Chen et al., 2015), low displacement rank (Sindhwani et al., 2015) and block-circulant matrix parameterization (Treister et al., 2018).

Original: Note that similar reparameterizations were also used to introduce certain algebraic properties to the parameters for purposes other than reducing model sizes, e.g. to make training more stable as in unitary evolution RNNs (Arjovsky et al., 2015) and in weight normalization (Salimans &
Revised: Note that similar reparameterizations were also used to introduce certain algebraic properties to the parameters for purposes other than reducing model sizes, e.g. to make training more stable as in unitary evolution RNNs (Arjovsky et al., 2015) and in weight normalization (Salimans &

Original: Kingma, 2016), to inject inductive biases (Thomas et al., 2018), and to alter (Dinh et al., 2017) or to measure (Li et al., 2018) properties of the loss landscape.
Revised: Kingma, 2016), to inject inductive biases (Thomas et al., 2018), and to alter (Dinh et al., 2017) or to measure (Li et al., 2018) properties of the loss landscape.

Original: All dense reparameterization methods to date are static.
Revised: All dense reparameterization methods to date are static.

Original: Sparse reparameterization Successful training of sparse reparameterized networks invariably employed iterative pruning and retraining (e.g. Han et al.
Revised: Sparse reparameterization Successful training of sparse reparameterized networks usually employs iterative pruning and retraining, e.g. Han et al.

Original: ; Narang et al.
Revised: ; Narang et al.

Original: (2017); Zhu & Gupta (2017)).
Revised: (2017); Zhu & Gupta (2017) 3 .

Original: Training typically started with a large  model and sparsity was scheduled to increase during the course of learning.
Revised: Training typically starts with a large pre-trained model and sparsity is gradually increased during the course of ﬁne-tuning.

Original: Training a small,  sparse model de novo always fared much worse than training a large one to begin with (Zhu & Gupta, 2017).
Revised: Training a small, static, and sparse model de novo always fared much worse than training a large one to begin with (Zhu & Gupta, 2017).

Original: Frankle & Carbin (2018) successfully identiﬁed small and sparse subnetworks post-training which, when trained in isolation, reached a similar accuracy as the enclosing big network.
Revised: Frankle & Carbin (2018) successfully identiﬁed small and sparse subnetworks post-training which, when trained in isolation, reached a similar accuracy as the enclosing big network.

Original: They further showed that these subnetworks were sensitive to initialization, and hypothesized that the role of overparameterization is to provide a large number of candidate subnetworks, thereby increasing the likelihood that one of these subnetworks will have the necessary structure and initialization needed for effective learning.
Revised: They further showed that these subnetworks were sensitive to initialization, and hypothesized that the role of overparameterization is to provide a large number of candidate subnetworks, thereby increasing the likelihood that one of these subnetworks will have the necessary structure and initialization needed for effective learning.

Original: Most closely related to our work are dynamic sparse reparameterization techniques that emerged only recently.
Revised: Most closely related to our work are dynamic sparse reparameterization techniques that emerged only recently.

Original: Like ours, these methods adaptively alter, by certain heuristic rules, reparameterization during training.
Revised: Like ours, these methods adaptively alter, by certain heuristic rules, reparameterization during training.

Original: Sparse evolutionary training (Mocanu et al., 2018) used magnitude-based pruning and random growth at the end of each training epoch.
Revised: Sparse evolutionary training (Mocanu et al., 2018) used magnitude-based pruning and random growth at the end of each training epoch.

Original: NeST (Dai et al., 2017; 2018) iteratively grew and pruned parameters and neurons during training; parameter growth was guided by gradient and pruning by magnitude.
Revised: NeST (Dai et al., 2017; 2018) iteratively grew and pruned parameters and neurons during training; parameter growth was guided by gradient and pruning by magnitude.

Original: Deep rewiring (Bellec et al., 2017) combined sparse reparameterization with stochastic parameter updates for training.
Revised: Deep rewiring (Bellec et al., 2017) combined sparse reparameterization with stochastic parameter updates for training.

Original: These methods were mostly concerned with sparsifying fully connected layers and applied to relatively small and shallow networks.
Revised: These methods were mostly concerned with sparsifying fully connected layers and applied to relatively small and shallow networks.

Original: As will be discussed in Section 5, our method, more scalable and computationally efﬁcient than these previous approaches, fully closed the generalization gap for the ﬁrst time between training a compact sparse network and compression of a large deep CNN.
Revised: We show that the method we propose in this paper is more scalable and computationally efﬁcient than these previous approaches, while achieving better performance on deep convolutional networks.

Original:  M ETHODS
Revised:  M ETHODS

Original: In this section, we describe our dynamic sparse reparameterization method.
Revised: In this section, we describe our dynamic sparse reparameterization method.

Original: Some notations ﬁrst.
Revised: Some notations ﬁrst.

Original: Let all reparameterized weight tensors in the original network be denoted by { W l } , where l = 1 , · · · , L indexes layers.
Revised: Let all reparameterized weight tensors in the original network be denoted by { W l } , where l = 1 , · · · , L indexes layers.

Original: Let N l be the number of parameters in W l , and N = (cid:80) l N l the total parameter count.
Revised: Let N l be the number of parameters in W l , and N = (cid:80) l N l the total parameter count.

Original: Sparse reparameterize W l = g ( φ l ; ψ l ) , where function g places components of parameter φ l into positions in W l indexed by ψ l ∈ Ψ M l ( { 1 , · · · , N l } ) 3 , s.t. W l,ψ l,i =
Revised: Sparse reparameterize W l = g ( φ l ; ψ l ) , where function g places components of parameter φ l into positions in W l indexed by ψ l ∈ Ψ M l ( { 1 , · · · , N l } ) 4 , s.t. W l,ψ l,i =

Original: φ l,i , ∀ i indexing components.
Revised: φ l,i , ∀ i indexing components.

Original: Let M l < N l be the dimensionality of φ l and ψ l , i.e. the number of non-zero weights in W l .
Revised: Let M l < N l be the dimensionality of φ l and ψ l , i.e. the number of non-zero weights in W l .

Original: Deﬁne s l = 1 − M l N l as the sparsity of W l .
Revised: Deﬁne s l = 1 − M l N l as the sparsity of W l .

Original: Global sparsity is then deﬁned as s = 1 − MN where M = (cid:80) l M l .
Revised: Global sparsity is then deﬁned as s = 1 − MN where M = (cid:80) l M l .

Revised: During the whole course of training, we kept global sparsity constant, speciﬁed by hyperparameter s ∈ (0 , 1) .

Revised: Reparameterization was initialized by uniformly sampling positions in each weight tensor

Original: Input: (cid:110)(cid:16) φ ( t ) l , ψ ( t ) l (cid:17)(cid:111) , M ( t ) , H ( t ) (cid:46) From step t
Revised: Input: (cid:110)(cid:16) φ ( t ) l , ψ ( t ) l (cid:17)(cid:111) , M ( t ) , H ( t ) (cid:46) From step t

Original: Output: (cid:110)(cid:16) φ ( t +1) l , ψ ( t +1) l (cid:17)(cid:111) , M ( t +1) , H ( t +1) (cid:46)
Revised: Output: (cid:110)(cid:16) φ ( t +1) l , ψ ( t +1) l (cid:17)(cid:111) , M ( t +1) , H ( t +1) (cid:46)

Original: To step t +
Revised: To step t +

Original: Need: K, δ (cid:46)
Revised: Need: K, δ (cid:46)

Original: Target number of parameters to be pruned and its fractional tolerance 1 for l ∈ { 1 , · · · , L } do (cid:46)
Revised: Target number of parameters to be pruned and its fractional tolerance 1 for l ∈ { 1 , · · · , L } do (cid:46)

Original: For each reparameterized weight tensor Π ( t ) l ← (cid:110) i : (cid:12)(cid:12) φ ( t ) l,i (cid:12)(cid:12) <
Revised: For each reparameterized weight tensor Π ( t ) l ← (cid:110) i : (cid:12)(cid:12) φ ( t ) l,i (cid:12)(cid:12) <

Original: H ( t ) (cid:111)
Revised: H ( t ) (cid:111)

Original: (cid:46) Indices of components of φ ( t ) l to be
Revised: (cid:46) Indices of components of φ ( t ) l to be

Original: (cid:16) K ( t ) l , R ( t ) l (cid:17) ← (cid:16)(cid:12)(cid:12) Π ( t
Revised: (cid:16) K ( t ) l , R ( t ) l (cid:17) ← (cid:16)(cid:12)(cid:12) Π ( t

Original: [Table]
Revised: [Table]

Original: l (cid:12)(cid:12) , M ( t ) l − (cid:12)(cid:12) Π ( t ) l (cid:12)(cid:12)(cid:17) (cid:46) Numbers of pruned and surviving weights 4 if (cid:80) l K ( t ) l < (1 − δ )
Revised: l (cid:12)(cid:12) , M ( t ) l − (cid:12)(cid:12) Π ( t ) l (cid:12)(cid:12)(cid:17) (cid:46) Numbers of pruned and surviving weights 4 if (cid:80) l K ( t ) l < (1 − δ )

Original: K then (cid:46)
Revised: K then (cid:46)

Original: Too few parameters pruned
Revised: Too few parameters pruned

Original: H ( t +1) ← 2 H ( t ) (cid:46)
Revised: H ( t +1) ← 2 H ( t ) (cid:46)

Original: Increase pruning threshold 6 else if (cid:80) l K ( t ) l > (1 + δ ) K then (cid:46)
Revised: Increase pruning threshold 6 else if (cid:80) l K ( t ) l > (1 + δ ) K then (cid:46)

Original: Too many parameters pruned
Revised: Too many parameters pruned

Original: H ( t +1) ← 12 H ( t ) (cid:46)
Revised: H ( t +1) ← 12 H ( t ) (cid:46)

Original: Decrease pruning threshold 8 else (cid:46)
Revised: Decrease pruning threshold 8 else (cid:46)

Original: A proper number of parameters pruned
Revised: A proper number of parameters pruned

Original: H ( t +1) ← H ( t ) (cid:46) Maintain pruning threshold 10 for l ∈ { 1 , · · · , L } do (cid:46)
Revised: H ( t +1) ← H ( t ) (cid:46) Maintain pruning threshold 10 for l ∈ { 1 , · · · , L } do (cid:46)

Original: For each reparameterized weight tensor
Revised: For each reparameterized weight tensor

Original: G ( t ) l ← (cid:106) R
Revised: G [Equation] t ) l ← [Equation]

Original: [Equation]
Revised: [Equation]

Original: t ) l 
Revised: l R

Original: [Equation]
Revised: [Equation]

Original: l R ( t ) l (cid:80) l K
Revised: K

Original: [Equation]
Revised: [Equation]

Original: ) l (cid:109)

Original: (cid:46) Redistribute parameters for growth ˜ ψ ( t ) l ∼ U (cid:104)
Revised: (cid:46) Redistribute parameters for growth ˜ ψ ( t ) l ∼ U [Equation]

Original: Ψ G ( t ) l (cid:16) { 1 , · · · , N l
Revised: [Equation] (cid:16) [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: \ (cid:110) ψ ( t ) l,
Revised: (cid:110) [Equation] ( t ) [Equation]

Original: [Equation]
Revised: [Equation]

Original: (cid:111)(cid:17)(cid:105) (cid:46)
Revised: (cid:111)(cid:17)(cid:105) (cid:46)

Original: Sample zero positions to grow new weights
Revised: Sample zero positions to grow new weights

Original: M ( t +1) l ← M ( t ) l− K
Revised: M ( t +1) l ← M ( t ) l

Original: [Equation]
Revised: [Equation]

Original: (cid:46)
Revised: (cid:46)

Original: New parameter count (cid:16) φ
Revised: New parameter count (cid:16) φ

Original: [Equation]
Revised: [Equation]

Original: t+1) l , ψ ( t +1) l (cid:17) ← (cid:18)(cid:20) φ ( t ) l,i/
Revised: t

Original: [Equation]
Revised: [Equation]

Original: Π ( t ) l ,(cid:21) , (cid:20) ψ (

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation] [Equation] [Equation]
Revised: , ψ ( t +

Original: [Equation]
Revised: [Equation]

Original: , ˜ ψ ( [Equation]
Revised: ) l (cid:17) ← (cid:18)(cid:20) φ ( t ) l,i/

Original: [Equation]
Revised: [Equation]

Original: (cid:21)(cid:19) (cid:46)
Revised: ,(cid:21) , (cid:20) ψ (

Original: New reparameterization

Original: During the whole course of training, we kept global sparsity constant, speciﬁed by hyperparameter s ∈ (0 , 1) .

Original: Reparameterization was initialized by uniformly sampling positions in each weight tensor at the global sparsity s , i.e. ψ (0) l ∼

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]

Original: N

Original: [Equation]
Revised: [Equation]

Original:  } ) (cid:105) , ∀ l , where M (0) l = (cid:4) ( [Equation] N l (cid:7) .
Revised: (cid:21)(cid:19) (cid:46) New reparameterization at the global sparsity s , i.e. ψ (0) l ∼ U (cid:104) Ψ M (0) l ( { 1 , · · · , N l } ) (cid:105) , ∀ l , where M (0) l = (cid:4) (1 − s ) N l (cid:7) .

Original: Associated parameters φ (0) l were randomly initialized.
Revised: Associated parameters φ (0) l were randomly initialized.

Original: Dynamic reparameterization was done periodically by repeating the following steps during training:
Revised: Dynamic reparameterization was done periodically by repeating the following steps during training:

Original: Train the model (currently reparameterized by (cid:110)(cid:16) φ ( t ) l , ψ ( t ) l (cid:17)(cid:111) ) for P batch iterations; 2.
Revised: Train the model (currently reparameterized by (cid:110)(cid:16) φ ( t ) l , ψ ( t ) l (cid:17)(cid:111) ) for P batch iterations; 2.

Original: Reallocate free parameters within and across weight tensors following Algorithm 1 to arrive at new reparameterization (cid:110)(cid:16) φ ( t +1) l , ψ ( t +1) l (cid:17)(cid:111) .
Revised: Reallocate free parameters within and across weight tensors following Algorithm 1 to arrive at new reparameterization (cid:110)(cid:16) φ ( t +1) l [Equation] t +

Original:  The adaptive reallocation is in essence a two-step procedure: a global pruning followed by a tensorwise growth.
Revised: [Equation] (cid:17)(cid:111) . The adaptive reallocation is in essence a two-step procedure: a global pruning followed by a tensorwise growth.

Original: Speciﬁcally our algorithm has the following key features:
Revised: Speciﬁcally our algorithm has the following key features:

Original: • Pruning was based on magnitude of weights, by comparing all parameters to a global threshold H , making the algorithm much more scalable than methods relying on layer-speciﬁc pruning.
Revised: • Pruning was based on magnitude of weights, by comparing all parameters to a global threshold H , making the algorithm much more scalable than methods relying on layer-speciﬁc pruning.

Original: • We made H adaptive, subject to a simple setpoint control dynamics that ensured roughly K weights to be pruned globally per iteration.
Revised: • We made H adaptive, subject to a simple setpoint control dynamics that ensured roughly K weights to be pruned globally per iteration.

Original: This proved to be much cheaper computationally than pruning exactly K smallest weights, which requires sorting all weights in the network.
Revised: This proved to be much cheaper computationally than pruning exactly K smallest weights, which requires sorting all weights in the network.

Original: • Growth was by uniformly sampling zero weights and tensor-speciﬁc, thereby achieving a reallocation of parameters across layers.
Revised: • Growth was by uniformly sampling zero weights and tensor-speciﬁc, thereby achieving a reallocation of parameters across layers.

Original: The heuristic guiding growth is
Revised: The heuristic guiding growth is

Original: [Equation]
Revised: [Equation]

Revised: where K ( t ) l and R ( t ) l = M ( t ) l − K ( t ) l are the pruned and surviving parameter counts, respectively.

Original: This rule allocated more free parameters to weight tensors with more surviving entries, while keeping the global sparsity the same by balancing numbers of parameters pruned and grown 4 .
Revised: This rule allocated more free parameters to weight tensors with more surviving entries, while keeping the global sparsity the same by balancing numbers of parameters pruned and grown 5 .

Original: [Table]
Revised: [Table]

Original: The entire procedure can be fully speciﬁed by hyperparameters (cid:0) s, P, K, δ, H (0) (cid:1) .
Revised: The entire procedure can be fully speciﬁed by hyperparameters (cid:0) s, P, K, δ, H (0) (cid:1) .

Original: For speciﬁc values used in experiments see implementational details in Appendix A
Revised: For speciﬁc values used in experiments see implementational details in Appendix A

Original: [Table]
Revised: [Table]

Original:  E XPERIMENTAL RESULTS
Revised:  E XPERIMENTAL RESULTS

Original: We evaluated our method in three sets of experiments (Table 1).
Revised: We evaluated our method in three sets of experiments (Table 1).

Original: We chose more modern convolutional networks, Resnet (He et al., 2015) and Wide Resnet (Zagoruyko & Komodakis, 2016), with parameter efﬁciency superior to earlier models, e.g. AlexNet (Krizhevsky et al., 2012) and VGG (Simonyan &
Revised: We chose more modern convolutional networks, Resnet (He et al., 2015) and Wide Resnet (Zagoruyko & Komodakis, 2016), with parameter efﬁciency superior to earlier models, e.g. AlexNet (Krizhevsky et al., 2012) and VGG (Simonyan &

Original: Zisserman, 2014), thanks to the adoption of skip connections and use of global average pooling over fully connected layers.
Revised: Zisserman, 2014), thanks to the adoption of skip connections and use of global average pooling over fully connected layers.

Original: Such a setup makes a much stronger case for our method because compression is much harder for these recent models.
Revised: Such a setup makes a much stronger case for our method because compression is much harder for these recent models.

Original: Pre-activation batch normalization were used in all cases.
Revised: Pre-activation batch normalization was used in all cases.

Original: Dynamic sparse reparameterization was applied to all weight tensors of fully connected and convolutional layers (with the exception of downsampling convolutions and the ﬁrst convolutional layer taking the input image), while all biases and parameters of normalization layers were kept dense 5 .
Revised: Dynamic sparse reparameterization was applied to all weight tensors of fully connected and convolutional layers (with the exception of downsampling convolutions and the ﬁrst convolutional layer taking the input image), while all biases and parameters of normalization layers were kept dense 6 .

Original: At a speciﬁc sparsity s , we compared our method ( dynamic sparse ) against four baselines:
Revised: At a speciﬁc sparsity s , we compared our method ( dynamic sparse ) against six baselines:

Original: • Full dense : original large and dense model, with N parameters; • Thin dense : original model with less wide layers, such that it had (1 − s )
Revised: • Full dense : original large and dense model, with N parameters; • Thin dense : original model with less wide layers, such that it had (1 − s )

Original: N parameters; • Static sparse : original model reparameterized to sparsity s , then trained with connectivity ﬁxed; • Compressed sparse : state-of-the-art compression of the original model by iterative pruning and retraining the original model to target sparsity s (see Appendix A for details of implementation).
Revised: N parameters; • Static sparse : original model reparameterized to sparsity s , then trained with connectivity ﬁxed; • Compressed sparse : state-of-the-art compression of the original model by iterative pruning and retraining the original model to target sparsity s (Zhu & Gupta, 2017); • DeepR : sparse model trained by using Deep Rewiring (Bellec et al., 2017); • SET : sparse model trained by using Sparse Evolutionary Training (Mocanu et al., 2018). Note that compressed sparse is a compression method that starts training with a dense model, whereas DeepR and SET , like ours, are dynamic reparameterization techniques that maintain sparsity throughout training. See Appendix A for hyperparameters used in the experiments.

Original: LeNet-300-100 on MNIST Like previous work on post-training compression, e.g. Han et al.
Revised: LeNet-300-100 on MNIST Like previous work on post-training compression, e.g. Han et al.

Original: and dynamic reparameterization, e.g. Bellec et al.
Revised: and dynamic reparameterization, e.g. Bellec et al.

Original: (2017), we ﬁrst experimented with a simple
Revised: (2017), we ﬁrst experimented with a simple

Original: LeNet-300-100 trained on MNIST.
Revised: LeNet-300-100 trained on MNIST.

Original: We found that static sparse grossly underperformed compressed sparse , a gap effectively closed by our dynamic sparse method (Figure 1a)6 , with the following nuances.
Revised: We found that static sparse grossly underperformed compressed sparse , a gap effectively closed by our dynamic sparse method (Figure 1a) with the following nuances.

Original: Dynamic sparse slightly outperformed compressed sparse at very high sparsity (i.e. low parameter count), a trend reversed, albeit weakly, at lower sparsities.
Revised: Dynamic sparse slightly outperformed compressed sparse at very high sparsity (i.e. low parameter count), a trend reversed, albeit weakly, at lower sparsities.

Original: A breakdown of post-training sparsities across layers of the network (Figure 1b) reveals reliable patterns emerged from the dynamics of our adaptive reparameterization algorithm. 
Revised: A breakdown of post-training sparsities across layers of the network (Figure 1b) reveals reliable patterns emerged from the dynamics of our adaptive reparameterization algorithm. This is consistent [Figure] [Figure]

Original: This is consistent with previous observations (Bellec et al., 2017) that, given a ﬁxed global sparsity, it was wiser to allocate more free parameters to the last layer, than to impose constant sparsity on all layers.
Revised: y

Original: [-WRN-28-2 on CIFAR
-]
Revised: [-WRN-28-2 on CIFAR
-][Figure] r [Figure]

Original: Next, we experimented with aWide Resnet model WRN-28-2 (Zagoruyko & Komodakis, 2016) trained to classify CIFAR10 images (see Appendix A for details of imple-
Revised: a

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original: [Figure]

Original: L a y
Revised: s i t y

Original: [Figure]
Revised: [Figure]

Original: s p a r{+ﬁxed global sparsity, it was wiser to allocate more free parameters to the last layer, than to impose constant sparsity on all layers. WRN-28-2 on CIFAR
 Next, we experimented with a Wide Resnet model WRN-28-2 (Zagoruyko & Komodakis, 2016) trained to classify CIFAR10 images (see Appendix A for details of implementation).+}
Revised: with previous observations (Bellec et al., 2017) that, given a {+ﬁxed global sparsity, it was wiser to allocate more free parameters to the last layer, than to impose constant sparsity on all layers. WRN-28-2 on CIFAR
 Next, we experimented with a Wide Resnet model WRN-28-2 (Zagoruyko & Komodakis, 2016) trained to classify CIFAR10 images (see Appendix A for details of implementation).+}

Original: s i t y
Revised: As shown in Figure 2a, static sparse and thin dense signiﬁcantly underperformed the state-of-the-art compressed sparse model, whereas our method dynamic sparse signiﬁcantly outperformed it. Deep rewiring signiﬁcantly lagged all other method. While the performance of SET [Table]

Original: [Figure]
Revised: [Figure]

Original: mentation).
Revised: s i

Original: As shown in Figure 2a, static sparse and thin dense signiﬁcantly underperformed the state-of-the-art compressed sparse model, whereas our method dynamic sparse signiﬁcantly outperformed it. Again, consistent sparsity patterns emerged from the adaptive reparameterization dynamics of Algorithm 1 (Figure 2b, c & d).
Revised: [Figure] was on par with compressed sparse , it lagged behind dynamic sparse at high sparsity levels. At low sparsity levels SET largely closed the gap to compressed sparse .

Revised: Again, consistent sparsity patterns emerged from the adaptive reparameterization dynamics of

Original:  We observed two rough trends: (a) larger parameter tensors tended to be sparser than smaller ones, and (b) deeper layers tended to be sparser than shallower ones.
Revised: Algorithm 1 (Figure 2b, c & d). We observed two rough trends: (a) larger parameter tensors tended to be sparser than smaller ones, and (b) deeper layers tended to be sparser than shallower ones.

Original: Resnet-50 on Imagenet Finally, we experimented with the Resnet-50 bottleneck architecture (He et al., 2015) trained on Imagenet (see Appendix A for details of implementation).
Revised: Resnet-50 on Imagenet Finally, we experimented with the Resnet-50 bottleneck architecture (He et al., 2015) trained on Imagenet (see Appendix A for details of implementation).

Original: We tested two global sparsity levels, 0 . 8 and 0 .
Revised: We tested two global sparsity levels, 0 . 8 and 0 .

Original: 9 (Table 2).
Revised: 9 (Table 2).

Original: Again, our method ( dynamic sparse ) outperformed stateof-the-art sparse compression ( compressed sparse ), which in turn outperformed static sparse and thin 
Revised: Again, our method ( dynamic sparse ) outperformed state-of-the-art sparse compression ( compressed sparse ), which in turn outperformed static sparse and thin dense baselines.

Original: [Table]
Revised: We also list in Table 2 two representative methods of structured pruning (see Appendix C),

Original: r [Table]
Revised: ThiNet (Luo et al., 2017) and Sparse Structure Selection (Huang & Wang, 2017), which, consistent with recent criticisms (Liu et al., 2018), underperformed static dense baselines.

Original: i
Revised: Consistent with the aforementioned experiments with LeNet-100-10 and WRN-28-2, reliable sparsity patterns across layers emerged from dynamic parameter reallocation during training, displaying the same empirical trends described above (Figure 3). Computational overhead of dynamic reparameterization We assessed the additional computational cost incurred by reparameterization steps (Algorithm 1) during training, and compared ours

Original: [Table]
Revised: [Table]

Original: dense baselines. Furthermore, consistent with the aforementioned experiments with LeNet-100and WRN-28-2, reliable sparsity patterns across layers emerged from dynamic parameter reallocation during training, displaying the same empirical trends described above (Figure 3). 
Revised: with existing dynamic sparse reparameterization techniques, DeepR and SET (Table 3). Because both SET and ours reallocate parameters only intermittently (every few hundred training iterations), the computational overhead was negligible for the experiments presented here 7 . DeepR, however, requires adding noise to gradient updates as well as reallocating parameters every training iteration, and therefore led to a signiﬁcantly larger overhead.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: In this work, we described and validated the best known solution so far to the following problem: given a small, ﬁxed budget of parameters for a deep CNN throughout training time, how to train it to yield the best generalization performance.
Revised: In this work, we described and validated the best known solution so far to the following problem: given a small, ﬁxed budget of parameters for a deep CNN throughout training time, how to train it to yield the best generalization performance.

Original: Our method used a dynamic reparameterization technique that adaptively reallocated free parameters across the network based on a simple heuristic.
Revised: Our method used a dynamic reparameterization technique that adaptively reallocated free parameters across the network based on a simple heuristic.

Original: We demonstrated that this method not only fared much better than static reparameterization techniques, but also signiﬁcantly outperformed even the state-of-the-art sparse compression methods.
Revised: We demonstrated that this method not only fared much better than static reparameterization techniques, but also signiﬁcantly outperformed even the state-of-the-art sparse compression methods.

Original: Note that compression is a much more forgiving situation where the parameter budget is not imposed during the entire course, but only at the end, of training.
Revised: Note that compression is a much more forgiving situation where the parameter budget is not imposed during the entire course, but only at the end, of training.

Original: Thus, our method yielded the most parameter efﬁcient sparse reparameterization of a given CNN.
Revised: Thus, our method yielded the most parameter efﬁcient sparse reparameterization of a given CNN.

Original: The past few years have seen a steady improvement in parameter efﬁciency of state-of-the-art CNN models thanks to innovations in network architectures.
Revised: The past few years have seen a steady improvement in parameter efﬁciency of state-of-the-art CNN models thanks to innovations in network architectures.

Original: Speciﬁcally, the introduction of residual modules and the use of global pooling in place of fully connected layers are largely responsible, as convolutional layers are more parameter efﬁcient than fully connected layers in general.
Revised: Speciﬁcally, the introduction of residual modules and the use of global pooling in place of fully connected layers are largely responsible, as convolutional layers are more parameter efﬁcient than fully connected layers in general.

Original: For example,
Revised: For example,

Original: AlexNet (Krizhevsky et al., 2012) had a majority of its parameters in fully connected layers, and this fraction has been decreasing in VGG (Simonyan & Zisserman, 2014),
Revised: AlexNet (Krizhevsky et al., 2012) had a majority of its parameters in fully connected layers, and this fraction has been decreasing in VGG (Simonyan & Zisserman, 2014),

Original: Inception (Szegedy et al., 2014), and more recently in Resnets (He et al., 2015) it was reduced to close to none.
Revised: Inception (Szegedy et al., 2014), and more recently in Resnets (He et al., 2015) it was reduced to close to none.

Original: Meanwhile, compact architectures with far fewer parameters also emerged at reasonable costs of performance degradation, e.g. SqueezeNet (Iandola et al., 2016) and MobileNet (Howard et al., 2017), for niche use cases where computing resources are limited.
Revised: Meanwhile, compact architectures with far fewer parameters also emerged at reasonable costs of performance degradation, e.g. SqueezeNet (Iandola et al., 2016) and MobileNet (Howard et al., 2017), for niche use cases where computing resources are limited.

Original: Here we took a different, but complementary, approach.
Revised: Here we took a different, but complementary, approach.

Original: Instead of searching the space of network architectures for new models with superior parameter efﬁciency, we sought to reparameterize a given model to make it more parameter efﬁcient.
Revised: Instead of searching the space of network architectures for new models with superior parameter efﬁciency, we sought to reparameterize a given model to make it more parameter efﬁcient.

Original: Such techniques have a history as long as modern convolutional networks.
Revised: Such techniques have a history as long as modern convolutional networks.

Original: Most early methods used dense reparameterization (Denil et al., 2013;
Revised: Most early methods used dense reparameterization (Denil et al., 2013;

Original: Yang et al., 2014; Moczulski et al., 2015; Sindhwani et al., 2015), effective in reducing redundancy in fully connected layers of big sizes, such as in AlexNet.
Revised: Yang et al., 2014; Moczulski et al., 2015; Sindhwani et al., 2015), effective in reducing redundancy in fully connected layers of big sizes, such as in AlexNet.

Original: These became less relevant as models became increasingly free of large linear layers but heavy in convolutional ones 7 .
Revised: These became less relevant as models became increasingly free of large linear layers but heavy in convolutional ones 8 .

Original: Sparse reparameterization, on the other hand, has been shown useful in compression of state-of-the-art deep CNNs (Han et al., 2015; Narang et al., 2017).
Revised: Sparse reparameterization, on the other hand, has been shown useful in compression of state-of-the-art deep CNNs (Han et al., 2015; Narang et al., 2017).

Original: However, a prominent empirical observation from these compression studies is that training a small sparse model de novo could never reach the same accuracy achieved by one of the same size compressed down from a large model (Zhu & Gupta, 2017).
Revised: However, a prominent empirical observation from these compression studies is that training a small sparse model de novo could never reach the same accuracy achieved by one of the same size compressed down from a large model (Zhu & Gupta, 2017).

Original: Why does it seem impossible to train a compact network but easy to compress a large one?
Revised: Why does it seem impossible to train a compact network but easy to compress a large one?

Original: The “lottery ticket hypothesis” (Frankle & Carbin, 2018) offered a plausible explanation.
Revised: The “lottery ticket hypothesis” (Frankle & Carbin, 2018) offered a plausible explanation.

Original: The authors identiﬁed, by post-training pruning, trainable sparse models that, albeit sensitive to initialization, generalized comparably well (called “winning tickets”).
Revised: The authors identiﬁed, by post-training pruning, trainable sparse models that, albeit sensitive to initialization, generalized comparably well (called “winning tickets”).

Original: They further posited that, the necessity of starting training with a large model is because only a big network with combinatorially a huge number of small subnetworks has a high probability of “winning the initialization lottery”.
Revised: They further posited that, the necessity of starting training with a large model is because only a big network with combinatorially a huge number of small subnetworks has a high probability of “winning the initialization lottery”.

Original: Similarly, here we argue that post hoc identiﬁcation of trainable compact (not necessarily sparse) reparameterization is actually rather trivial also because the optimization trajectory is typically a very low-dimensional object .
Revised: Similarly, here we argue that post hoc identiﬁcation of trainable compact (not necessarily sparse) reparameterization is actually rather trivial also because the optimization trajectory is typically a very low-dimensional object .

Original: Take for example the training of Resnet-152 (parameter count 60.2M) as described by He et al.
Revised: Take for example the training of Resnet-152 (parameter count 60.2M) as described by He et al.

Original: (2015): parameter updates for 600K iterations reliably send a random initialization to a state-of-the-art solution.
Revised: (2015): parameter updates for 600K iterations reliably send a random initialization to a state-of-the-art solution.

Original: One could simply take a trivial post hoc reparameterization as a linear projection onto the span of the 600K parameter updates which, even if they are all linearly independent from each other, amount to less than 1% of the total parameter count.
Revised: One could simply take a trivial post hoc reparameterization as a linear projection onto the span of the 600K parameter updates which, even if they are all linearly independent from each other, amount to less than 1% of the total parameter count.

Original: The low dimensionality of optimization trajectory is consistent with emerging evidence suggesting that optima of overparameterized deep neural networks are usually high-dimensional manifolds with low co-dimensions (Cooper, 2018), and that such optima have superior generalization properties (Wu et al., 2017) and favored by SGD (Zhang et al., 2018a; Poggio et al., 2017).
Revised: The low dimensionality of optimization trajectory is consistent with emerging evidence suggesting that optima of overparameterized deep neural networks are usually high-dimensional manifolds with low co-dimensions (Cooper, 2018), and that such optima have superior generalization properties (Wu et al., 2017) and favored by SGD (Zhang et al., 2018a; Poggio et al., 2017).

Original: What prohibits any post hoc reparameterized compact model from being effectively trained de novo is however its sensitivity to initialization–though optimization trajectories are each low-dimensional, their linear spans do not necessarily overlap when starting from different initial points.
Revised: What prohibits any post hoc reparameterized compact model from being effectively trained de novo is however its sensitivity to initialization–though optimization trajectories are each low-dimensional, their linear spans do not necessarily overlap when starting from different initial points.

Original: An obvious way “to win the lottery” is to purchase all available tickets (i.e. having a huge model to cover all possible linear spans of optimization trajectories), but it is also conceivably feasible “to cheat” by adaptively changing the number on a single ticket as the lottery result is being announced (i.e. dynamic reparameterization that continually re-orients a low-dimensional parameter manifold tangentially along an optimization trajectory).
Revised: An obvious way “to win the lottery” is to purchase all available tickets (i.e. having a huge model to cover all possible linear spans of optimization trajectories), but it is also conceivably feasible “to cheat” by adaptively changing the number on a single ticket as the lottery result is being announced (i.e. dynamic reparameterization that continually re-orients a low-dimensional parameter manifold tangentially along an optimization trajectory).

Original: We believe our method did exactly this.
Revised: We believe our method did exactly this.

Original: Though we are the ﬁrst to apply dynamic reparameterization to deep CNNs and our method the ﬁrst to outperform state-of-the-art sparse compression techniques, it should be noted that ours is not the ﬁrst dynamic sparse reparameterization approach.

Original: We were inspired by, and successfully exceeded the limitations of, three previous methods, DeepR (Bellec et al., 2017), SET (Mocanu et al., 2018) and NeST (Dai et al., 2017; 2018).
Revised: We were inspired by previous methods of dynamic sparse reparameterization, e.g. DeepR (Bellec et al., 2017) and SET (Mocanu et al., 2018) (also see Appendix C). Compared to these techniques, our method has a number of advantages.

Original: Speciﬁcally, in addition to producing better  sparse models, our approach was more scalable and much cheaper computationally.
Revised: First, ours produced better performing sparse CNNs, fully closed the gap to compression of large dense models.

Original: First, compared to DeepR, our method reallocated free parameters much less frequently (once per hundreds of training iterations), and did not require random walk in the parameter space, only random sampling of dimensions. Second, by using an adaptive threshold for pruning, we avoided expensive sorting operations as used by SET.
Revised: Second, our reparameterization incurred the least computational overhead during training by infrequent and cheap parameter reallocation.

Original: Third, our  reallocation of parameters across layers was automatic, without the need of manually conﬁguring sparsity for each layer; the heuristic we used discovered similar optimal allocation patterns as manual choices 8 . 
Revised: Finally, our method is highly scalable thanks to its automatic reallocation of parameters across layers, without the need of manually conﬁguring sparsity for each layer 9 . A noteworthy observation from our experiments is that the highest parameter efﬁciency was achieved by non-structured sparsiﬁcation (see Appendix C for details).

Original: Finally, our parameter reallocation, like DeepR, was  by random sampling, which is much cheaper in memory consumption than gradient-based growth heuristic such as NeST, which requires gradients of zero entries of a sparse tensor to be tracked.
Revised: Accuracy suffered signiﬁcantly when structure was imposed on the sparsity patterns (see Appendix D). Because computations involving ﬁne-grained sparsity cannot be readily accelerated on GPUs, a number of compression methods by structured pruning (see Appendix C) remove entire channels from deep CNNs to yield smaller networks with dense parameter tensors. However, many of these structured pruning methods produced compressed models no better than direct training of a thin dense model (Liu et al., 2018) (also see

Original: We believe this study might have profound implications. If training compact sparse models directlyproved to be competitive as our work suggested, it would warrant a renewed look on practical choices of best hardware architectures for training, between computing devices optimized for dense linear algebra versus those suited for sparse 
Revised: Table 2 for examples compared with our method). Our ﬁndings suggest that it is possible to train sparse models directly to reach generalization performances equal to or better than those of dense or sparse networks of comparable size produced by compression. This warrants a renewed look at the practical choices of best hardware architectures for training, between computing devices optimized for dense linear algebra versus those suited for sparse operations.

Original: A PPENDIX A D ETAILS OF IMPLEMENTATION
Revised: A PPENDIX A D ETAILS OF IMPLEMENTATION

Original: We implemented all models and reparameterization mechanisms using pytorch .
Revised: We implemented all models and reparameterization mechanisms using pytorch .

Original: Experiments were run on GPUs, and all sparse tensors were represented as dense tensors ﬁltered by a binary mask 9 .
Revised: Experiments were run on GPUs, and all sparse tensors were represented as dense tensors ﬁltered by a binary mask 10 .

Original: Source code to reproduce all experiments is available on GitHub:
Revised: Source code to reproduce all experiments is available on GitHub:

Original: Link suppressed for the sake of anonymity during review process .
Revised: Link suppressed for the sake of anonymity during review process .

Original: Training Hyperparameter settings for training are listed in the ﬁrst block of Table 3.
Revised: Training Hyperparameter settings for training are listed in the ﬁrst block of Table 4.

Original: Standard mild data augmentation were used in all experiments for CIFAR10 (random translation, cropping and horizontal ﬂipping) and for Imagenet (random cropping and horizontal ﬂipping).
Revised: Standard mild data augmentation was used in all experiments for CIFAR10 (random translation, cropping and horizontal ﬂipping) and for Imagenet (random cropping and horizontal ﬂipping).

Original: Dynamic reparameterization Hyperparameter settings for dynamic sparse reparameterization (Algorithm 1) are listed in the second block of Table 3.

Original: [Table]

Original: Hyperparameters for dynamic sparse reparameterization

Original: Number of parameters to prune ( K ) 20, 200,

Original: Fractional tolerance of K ( δ ) 0.

Original: Initial pruning threshold ( H (0) ) 0. Reparameterization period ( P ) schedule (epoch range: P ) 1 - 25: 26 - 50: 51 - 75: 76 - 100: 1 - 25: 26 - 80: 81 - 140: 141 - 200: 1 - 25: 26 - 50: 51 - 75: 76 - 100: Hyperparameters for sparse compression (iterative pruning and re-training) Number of pruning iterations ( T )
Revised: Sparse compression baseline We compared our method against iterative pruning methods (Han et al., 2015; Zhu & Gupta, 2017).

Original: Number of training epochs between pruning iterations

Original: Number of training epochs post-pruning Number of epochs during pruning Learning rate schedule during pruning (epoch range: learning rate) 1 - 20: 21 - 30: 31 - 40: 0. [Figure]
Revised: We start from a full dense model trained with hyperparameters provided in the ﬁrst block of Table 4 and then gradually prune the network to a target sparsity in T steps.

Original: [Table] [Table] Sparse compression baseline We compared our method against state-of-the-art model compression methods using iterative pruning during training (Han et al., 2015; Zhu & Gupta, 2017), starting from a full dense model trained with hyperparameters provided in the ﬁrst block of Table 3 and then gradually prune thenetwork to a target sparsity in T steps. The pruning schedule we used was
Revised: As in Zhu & Gupta (2017), the pruning schedule we used was

Original: [Equation]
Revised: [Equation]

Original: where t = 0 , 1 , · · · , T indexes pruning steps, and s the target sparsity reached at the end of training.
Revised: where t = 0 , 1 , · · · , T indexes pruning steps, and s the target sparsity reached at the end of training.

Original: Thus, this baseline (labeled as compressed sparse in the paper) was effectively trained for more iterations (original training phase plus compression phase) than our dynamic sparse method, making it a so far strongestpossible baseline to benchmark our method. 
Revised: Thus, this baseline (labeled as compressed sparse in the paper) was effectively trained for more iterations (original training phase plus compression phase) than our dynamic sparse method, to the best of our knowledge a so far strongest baseline to benchmark our method. Hyperparameter settings for sparse compression are listed in the second block of Table 4. Dynamic reparameterization (ours)

Original: Hyperparameter settings for  sparse compression are listed in the third block of Table 3.
Revised: Hyperparameter settings for dynamic sparse reparameterization (Algorithm 1) are listed in the third block of Table 4.

Revised: Sparse Evolutionary Training (SET)

Original: Exceptions For reasons described below, we made a few minor exceptions in sparsiﬁcation for certain model layers in our experiments.
Revised: Because the larger-scale experiments here (WRN-28-2 on CIFAR10 and Resnet-50 on Imagenet) were not attempted by Mocanu et al. (2018), no speciﬁc settings for reparameterization in these cases were available in the original paper. In order to make a fair comparison, we used the same hyperparameters as those used in our dynamic reparameterization scheme (third block in Table 4).

Revised: At each reparameterization step, the weights in each layer were sorted by magnitude and the smallest fraction was pruned.

Revised: An equal number of parameters were then randomly allocated in the same layer and initialized to zero.

Revised: For control, the total number of reallocated weights at each step was chosen to be the same as our dynamic reparameterization method, as was the schedule for reparameterization.

Revised: Deep Rewiring (DeepR)

Revised: The fourth block in Table 4 contain hyperparameters for the DeepR experiments.

Revised: We refer the reader to Bellec et al.

Revised: (2017) for details of the deep rewiring algorithm and for explanation of the hyperparameters.

Revised: Exceptions For reasons described below, we made a few minor exceptions in sparsiﬁcation for certain model layers in our experiments.

Original: • The sparsity of the last linear layer in LeNet-300-100 was allowed to be sparsiﬁed to at most 90% sparsity.
Revised: • The sparsity of the last linear layer in LeNet-300-100 was allowed to be sparsiﬁed to at most 90% sparsity.

Original: In cases where global sparsity was over 90% , we redistribute free parameters from earlier layers to maintain the total parameter count.
Revised: In cases where global sparsity was over 90% , we redistribute free parameters from earlier layers to maintain the total parameter count.

Original: This is because, if the last layer was allowed to be overly sparse, performance sustained huge degradation unsuited for meaningful comparison.
Revised: This is because, if the last layer was allowed to be overly sparse, performance sustained huge degradation unsuited for meaningful comparison.

Original: • The last linear layer of WRN-28-2 was always kept dense.
Revised: • The last linear layer of WRN-28-2 was always kept dense.

Original: It has a negligible fraction of parameter count. 
Revised: It has a negligible fraction of parameter count. [Table]

Original: A PPENDIX B C OMPARISON TO H ASHED N ET
Revised: Fractional tolerance of K ( δ ) 0. Initial pruning threshold ( H (0) ) 0. Reparameterization period ( P ) schedule (epoch range: P ) 1 - 25: 26 - 50: 51 - 75: 76 - 100:    1 - 25: 26 - 80: 81 - 140: 141 - 200:    1 - 25: 26 - 50: 51 - 75: 76 - 100: Hyperparameters for Sparse Evolutionary Training ( SET ) (Mocanu et al., 2018)

Revised: Number of parameters to prune at each re-parameterization step20, 200,

Revised: Reparameterization period ( P ) schedule (epoch range: P ) - 1 - 25: 26 - 80: 81 - 140: 141 - 200: 1 - 25: 26 - 50: 51 - 75: 76 - 100:

Revised: Hyperparameters for Deep Rewiring ( DeepR ) (Bellec et al., 2017)

Revised: L 1 regularization multiplier ( α ) - 10 −10 −

Revised: Temperature ( T ) schedule (epoch range: T ) - 1 - 25: 26 - 80: 81 - 140: 141 - 200:

Revised: 10 −10 −10 −10 −1 - 25: 26 - 50: 51 - 75: 76 - 100: 10 −10 −10 −10 −

Original: We also compared our dynamic sparse reparameterization method to a number of static dense reparameterization techniques, e.g. Denil et al.
Revised: We also compared our dynamic sparse reparameterization method to a number of static dense reparameterization techniques, e.g. Denil et al.

Original: ; Yang et al.
Revised: ; Yang et al.

Original: Moczulski et al.
Revised: Moczulski et al.

Original: Sindhwani et al.
Revised: Sindhwani et al.

Original: ; Chen et al.
Revised: ; Chen et al.

Original: ; Treister et al.
Revised: ; Treister et al.

Original: Instead of sparsiﬁcation, these methods impose structure on large parameter tensors by parameter sharing.
Revised: Instead of sparsiﬁcation, these methods impose structure on large parameter tensors by parameter sharing.

Original: Most of these methods have not been used for convolutional layers except for recent ones (Chen et al., 2015; Treister et al., 2018).
Revised: Most of these methods have not been used for convolutional layers except for recent ones (Chen et al., 2015; Treister et al., 2018).

Original: We found that HashedNet (Chen et al., 2015) had the best performance over other static dense reparameterization methods, and also benchmarked our method against it.
Revised: We found that HashedNet (Chen et al., 2015) had the best performance over other static dense reparameterization methods, and also benchmarked our method against it.

Original: Instead of reparameterizing a parameter tensor with N entries to a sparse one with M < N non-zero components, HashedNet ’s reparameterization is to put M free parameters into N positions in the parameter through a random mapping from { 1 , · · · , N } to { 1 , · · · , M } computed by cheap hashing, resulting in a dense parameter tensor with shared components.
Revised: Instead of reparameterizing a parameter tensor with N entries to a sparse one with M < N non-zero components, HashedNet ’s reparameterization is to put M free parameters into N positions in the parameter through a random mapping from { 1 , · · · , N } to { 1 , · · · , M } computed by cheap hashing, resulting in a dense parameter tensor with shared components.

Original: Results of LeNet-300-100-10 on MNIST are presented in Figure 4a, those of WRN-28-2 on CIFAR in Figure 4b, and those of Resnet-50 on Imagenet in Table 4.
Revised: Results of LeNet-300-100-10 on MNIST are presented in Figure 4a, those of WRN-28-2 on CIFAR in Figure 4b, and those of Resnet-50 on Imagenet in Table 5.

Original: For a certain global sparsity s of our 
Revised: For a certain global sparsity s of our method, we compare it against a HashedNet with all reparameterized tensor hashed such that each had a fraction 1 − s of effective parameter count. We found that our method dynamic sparse signiﬁcantly outperformed HashedNet .

Original: [Figure] 
Revised: [Figure] [Table]

Original: method, we  compareit against a HashedNet with all reparameterized tensor hashed such that each had a fraction 1 − s of effective parameter count. We found that our method dynamic sparse signiﬁcantly outperformed HashedNet .
Revised: As an extension to Section 2 of the main text, here we elaborate on existing methods related to ours, how they compare with and contrast to each other, and what features, apart from effectiveness, distinguished our approach from all previous ones. We conﬁne the scope of comparison to training methods that produce smaller versions (i.e. ones with fewer parameters) of a given modern (i.e. post-AlexNet) deep convolutional neural network model.

Original: A PPENDIX C B ENEFITS OF PARAMETER REALLOCATION ACROSS LAYERS
Revised: We list representative methods in Table 6. We classify these methods by three key features. Strict parameter budget throughout training and inference This feature was discussed in depth in the main text. Most of the methods to date are compression techniques, i.e. they start training with a fully parameterized, dense model, and then reduce parameter counts.

Revised: To the best of our knowledge, only three methods, namely DeepR (Bellec et al., 2017), SET (Mocanu et al., 2018) and ours, strictly impose, throughout the entire course of training, a ﬁxed small parameter budget, one that is equal to

Revised: [Table]

Revised: Bellec et al.,

Revised: [Table]

Revised: NN Synthesis Tool (NeST)

Revised: (Dai et al., 2017; 2018)

Revised: [Table]

Revised: tf.contrib.model pruning (Zhu & Gupta, 2017)

Revised: [Table]

Revised: RNN Pruning (Narang et al., 2017)

Revised: [Table]

Revised: Deep Compression (Han et al., 2015)

Revised: [Table]

Revised: Group-wise Brain Damage (Lebedev & Lempitsky, 2015)

Revised: [Table]

Revised: Pruning (Li et al., 2016)

Revised: no channel no

Revised: Structured Sparsity Learning (SSL) (Wen et al., 2016)

Revised: no channel/kernel/layer yes

Revised: ThiNet (Luo et al., 2017)

Revised: no channel no

Revised: LASSO-regression Channel Pruning (He et al., 2017)

Revised: no channel no

Revised: Network Slimming (Liu et al., 2017)

Revised: no channel yes

Revised: Sparse Structure Selection (SSS) (Huang & Wang, 2017)

Revised: no layer yes

Revised: Principal Filter Analysis (PFA) (Suau et al., 2018)

Revised: no channel yes/no

Revised: We the size of the ﬁnal sparse model for inference.

Revised: We make a distinction between these direct training methods (ﬁrst block) and compression methods (second and third blocks of Table 6) 11 .

Revised: This distinction is meaningful in two ways: (a) practically, direct training methods are more memoryefﬁcient on appropriate computing substrate by requiring parameter storage of no more than the ﬁnal compressed model size; (b) theoretically, these methods, if performing on par with or better than compression methods (as this work suggests), shed light on an important question: whether gross overparameterization is necessary for good generalization performance?

Revised: Granularity of sparsity The granularity of sparsity refers to the additional structure imposed on the placement of the non-zero entries of a sparsiﬁed parameter tensor.

Revised: The ﬁnest-grained case, namely non-structured , allows each individual weight in a parameter tensor to be zero or non-zero independently.

Revised: Early compression techniques, e.g. Han et al.

Revised: (2015), and more recent pruning-based compression methods based thereon, e.g. Zhu & Gupta (2017), are non-structured (second block of

Revised: Table 6).

Revised: So are all direct training methods like ours (ﬁrst block of Table 6).

Revised: Non-structured sparsity makes its computations difﬁcult to accelerate on GPUs.

Revised: To tackle this problem, a class of compression methods, called structured pruning (third block in Table 6), constrain “sparsity” to a much coarser granularity.

Revised: Typically, pruning is performed to an entire convolution channel, e.g. ThiNet (Luo et al., 2017), even whole layers or residual blocks (Huang & Wang, 2017).

Revised: This way, the compressed “sparse” model has essentially smaller and/or fewer dense parameter tensors, and computation can thus be accelerated on GPUs the same way as dense neural networks.

Revised: These structured compression methods, however, did not make a useful baseline in this work, for the following reasons.

Revised: First, because they produce dense models, their relevance to our method (non-structured, non-compression) is far more remote than non-structured compression techniques yielding sparse models, for a meaningful comparison.

Revised: Second, typical structured pruning methods substantially underperformed non-structured ones (see Table 2 for two examples, ThiNet and SSS), and emerging evidence has called into question the fundamental value of structured pruning: Mittal et al.

Revised: (2018) found that the channel pruning criteria used in a number of state-of-the-art structured pruning methods performed no better than random channel elimination, and Liu et al.

Revised: (2018) found that ﬁne-tuning in a number of state-of-the-art pruning methods fared no better than direct training of a randomly initialized pruned model which, in the case of channel/layer pruning, is simply a less wide and/or less deep dense model (see Table 2 for comparison of ThiNet and SSS against thin dense ).

Revised: In addition, we also performed extra experiments in which we constrained our method to perform “structured sparsiﬁcation” and obtained signiﬁcantly worse results, see Appendix D.

Revised: Predeﬁned versus automatically discovered sparsity levels across layers The last key feature (rightmost column of Table 6) for our classiﬁcation of methods is whether the sparsity levels of different layers of the network is automatically discovered during training or predeﬁned by manual conﬁguration.

Revised: The value of automatic sparsiﬁcation, e.g. ours, is twofold.

Revised: First, it is conceptually more general because parameter reallocation heuristics can be applied to diverse model architectures, whereas layer-speciﬁc conﬁguration has to be cognizant of network architecture, and at times also of the task to learn.

Revised: Second, it is practically more scalable because it obviates manual conﬁgurations of layer-wise sparsity, agnostic of the depth and size of the network, keeping the overhead of hyperparameter tuning constant rather than scaling with model depth/size.

Revised: In addition to efﬁciency, we also show in Appendix E extra experiments on how automatic parameter reallocation across layers contributed to its effectiveness.

Revised: In conclusion, to the best of our knowledge, our method is unique from all existing ones because it

Revised: • strictly maintained a ﬁxed parameter footprint throughout the entire course of training, and • automatically discovered layer-wise sparsity levels

Revised: during training

Revised: A PPENDIX D S TRUCTURED VERSUS NON - STRUCTURED SPARSITY

Revised: As discussed in Appendix C, we benchmarked our direct sparse training methods against the best performing non-structured pruning method (Zhu & Gupta, 2017) known to us.

Revised: Compression by structured channel-wise pruning, on the other hand, is not better than even the thin dense baseline (Table 2), consistent with recent criticisms (Mittal et al., 2018;

Revised: Liu et al., 2018).

Revised: We asked how our method would perform if it were constrained to training sparse models at a coarser granularity.

Revised: To answer this question, we performed additional experiments for WRN-28-2 trained on

Revised: CIFAR10 and Resnet-50 trained on Imagenet, as described in the following.

Revised: Consider a weight tensor of a convolution layer, of size C out × C in × 3 × 3 , where C out and C in are the number of output and input channels, respectively.

Revised: Our method performed dynamic sparse reparameterization by pruning and reallocating individual weights of the 4-dimensional parameter tensor–the ﬁnest granularity.

Revised: To adapt our procedure to coarse-grain sparsity on groups of parameters, we modiﬁed our Algorithm 1 in the following ways:

Revised: the pruning step now removed entire groups of weights by comparing their L 1 -norms with the adaptive threshold; 2.

Revised: the adaptive threshold was updated based on the difference between the target number and the actual number of groups to prune/grow at each step; 3.

Revised: the growth step reallocated groups of weights within and across parameter tensors using the same heuristic (Equation 2).

Revised: We show results at kernel-level granularity (i.e. groups are 3 × 3 kernels) in Fig.

Revised: 5 and Table 7, for WRN-28-2 on CIFAR10 and Resnet-50 on Imagenet, respectively.

Revised: We made two observations.

Revised: First, enforcing kernel-level sparsity leads to signiﬁcantly worse accuracy compared to unstructured sparsity.

Revised: Second, at this coarser granularity, our method still outperformed the thin dense baseline model with the same number of parameters.

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: When we further coarsened the granularity of sparsity to channel level (i

Revised: [Table]

Revised: e. groups are C in × 3 ×slices that generate output feature maps), our method utterly failed to produce performant models

Revised: [Table]

Original: In order to assess whether and how much our parameter reallocation heuristic (Equation 2) contributed to the effectiveness of our method, we did a set of control experiments in which all free parameter reallocation were constrained to within parameter tensors,i.e. parameter reallocation across layers were disabled, and sparsity stayed constant (uniformly initialized) for each layer.
Revised: In order to assess whether and how much our parameter reallocation heuristic (Equation 2) contributed to the effectiveness of our method [Figure] we did a set of control experiments in which all free parameter reallocation were constrained to within parameter tensors,

Revised: i

Revised: [Figure]

Revised: e. parameter reallocation across layers were disabled, and sparsity stayed constant (uniformly initialized) for each layer.

Original: The results are presented in Figure 5 for LeNet-300-100-10 on MNIST.
Revised: The results are presented in Figure 6 for LeNet-300-100-10 on MNIST.

Original: We found that removing inter-layer parameter allocation yielded worse performance, particularly at  high sparsity levels.
Revised: We found that removing inter-layer parameter allocation yielded worse performance, particularly at extremely high sparsity levels.

Original: This suggested that parameter reallocation across the entire model was beneﬁcial to our dynamic reparameterization.
Revised: This suggested that parameter reallocation across the entire model was beneﬁcial to our dynamic reparameterization [Figure]