Original: L EARNING TO S HARE : S IMULTANEOUS P ARAMETER T YING AND S PARSIFICATION IN D EEP L EARNING
Revised: L EARNING TO S HARE : S IMULTANEOUS P ARAMETER T YING AND S PARSIFICATION IN D EEP L EARNING

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Deep neural networks (DNNs) usually contain millions, maybe billions, of parameters/weights, makingboth storage and computation very expensive. This has motivated a large body of work to reduce the complexityof the neural network by using sparsity-inducing regularizers.
Revised: Deep neural networks (DNNs) may contain millions, even billions, of parameters/weights, making storage and computation very expensive and motivating a large body of work aimed at reducing their complexity by using, e.g., sparsityinducing regularization.

Original: Another well-known approach for controlling the complexity of DNNs is parameter sharing/tying, where certain sets of weightsare forced to share a common value.
Revised: Parameter sharing/tying is another well-known approach for controlling the complexity of DNNs by forcing certain sets of weights to share a common value.

Original: Some forms of weight sharing are hard-wired to express certain invariances, with a notable example being the shift-invariance of convolutional layers.
Revised: Some forms of weight sharing are hard-wired to express certain invariances; a notable example is the shift-invariance of convolutional layers.

Original: However,there may be other groups of weightsthat may be tied together during the learning process, thus further reducing the complexity of the network.
Revised: However, other groups of weights may be tied together during the learning process to further reduce the network complexity.

Original: In this paper, we adopt a recently proposedsparsity-inducing regularizer,named GrOWL ( group ordered weighted (cid:96) 1 ), which encourages sparsity and, simultaneously, learns which groups of parameters should share a common value.
Revised: In this paper, we adopt a recently proposed regularizer, GrOWL ( group ordered weighted (cid:96) 1 ), which encourages sparsity and, simultaneously, learns which groups of parameters should share a common value.

Original: GrOWL has been proven effective in linear regression, being able to identify and cope with strongly correlated covariates.
Revised: GrOWL has been proven effective in linear regression, being able to identify and cope with strongly correlated covariates.

Original: Unlike standard sparsity-inducing regularizers ( e.g. , (cid:96) 1 a.k.a.
Revised: Unlike standard sparsity-inducing regularizers ( e.g. , (cid:96) 1 a.k.a.

Original: Lasso), GrOWL not only eliminates unimportant neurons by setting all the corresponding weights to zero, but also explicitly identiﬁes strongly correlated neurons by tying the corresponding weights to a common value.
Revised: Lasso), GrOWL not only eliminates unimportant neurons by setting all their weights to zero, but also explicitly identiﬁes strongly correlated neurons by tying the corresponding weights to a common value.

Original: This ability of GrOWL motivates the following two-stage procedure: (i) use GrOWL regularization in the trainingprocess to simultaneously identify signiﬁcant neurons and groups of parameter that should be tied together; (ii) retrain the network, enforcing the structure that was unveiled in the previous phase, i.e. , keeping only the signiﬁcant neurons and enforcing the learned tying structure.
Revised: This ability of GrOWL motivates the following two-stage procedure: (i) use GrOWL regularization during training to simultaneously identify signiﬁcant neurons and groups of parameters that should be tied together; (ii) retrain the net- work , enforcing the structure that was unveiled in the previous phase

Revised: i

Revised: e. , keeping only the signiﬁcant neurons and enforcing the learned tying structure.

Original: We evaluate the proposed approach on several benchmark datasets, showing that it can dramatically compress the network with slight or even no loss on generalization performance.
Revised: We evaluate this approach on several benchmark datasets, showing that it can dramatically compress the network with slight or even no loss on generalization accuracy

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep neural networks (DNNs) have recently revolutionized machine learning by dramatically advancing the state-of-the-artperformance in many applications, ranging from speech and image recognition to playing video games (Goodfellow et al., 2016).
Revised: Deep neural networks (DNNs) have recently revolutionized machine learning by dramatically advancing the state-of-the-art in several applications, ranging from speech and image recognition to playing video games (Goodfellow et al., 2016).

Original: A typical DNN consists of a sequence of concatenated layers of functions, potentially involving millions or billions of parameters; by using extremely large training datasets, these architectures are able to learn extremely complex non-linear mappings, features, and dependencies.
Revised: A typical DNN consists of a sequence of concatenated layers, potentially involving millions or billions of parameters; by using very large training sets, DNNs are able to learn extremely complex non-linear mappings, features, and dependencies.

Original: A large amount of research has been devoted to the use of regularization in DNN learning (Goodfellow et al., 2016), aiming at reducing the generalization error.
Revised: A large amount of research has focused on the use of regularization in DNN learning (Goodfellow et al., 2016), as a means of reducing the generalization error.

Original: It has been shown that the parametrization of many DNNs is very redundant, in the sense that a large fraction of the parameters can be predicted from the remaining ones, with no accuracy loss (Denil et al., 2013).
Revised: It has been shown that the parametrization of many DNNs is very redundant, with a large fraction of the parameters being predictable from the remaining ones, with no accuracy loss (Denil et al., 2013).

Original: Several regularization methods have been proposed to tackle the potential over-ﬁtting caused by this redundancy.
Revised: Several regularization methods have been proposed to tackle the potential over-ﬁtting due to this redundancy.

Original: Arguably, the earliest and simplest choice is the classical (cid:96) 2 norm, known as weight decay in the early neural networks literature (Rumelhart et al., 1988), and as ridge regression in statistics.
Revised: Arguably, the earliest and simplest choice is the classical (cid:96) 2 norm, known as weight decay in the early neural networks literature (Rumelhart et al., 1986), and as ridge regression in statistics.

Original: In the past two decades, sparsity-inducing regularization based on the (cid:96) 1 norm, also known as Lasso (Tibshirani, 1996), and variants thereof, became standard tools in statistics and machine learning, including in deep learning (Goodfellow et al., 2016).
Revised: In the past two decades, sparsity-inducing regularization based on the (cid:96) 1 norm (often known as Lasso) (Tibshirani, 1996), and variants thereof, became standard tools in statistics and machine learning, including in deep learning (Goodfellow et al., 2016).

Original: In recent work, Scardapane et al.
Revised: Recently, Scardapane et al.

Original: (2017) use group-Lasso (a variant of Lasso that assumes thatthe parameters are organized in groups and encourages sparsity at the group level (Yuan & Lin, 2006)) in deep learning.
Revised: (2017) used group-Lasso (a variant of Lasso that assumes that parameters are organized in groups and encourages sparsity at the group level (Yuan & Lin, 2006)) in deep learning.

Original: One of the effects of Lasso or group-Lasso regularization in learning a DNN is that many of the parameters may become exactly zero, thus reducing the amount of memory needed to store the model, and lowering the computational cost of applying it.
Revised: One of the effects of Lasso or group-Lasso regularization in learning a DNN is that many of the parameters may become exactly zero, thus reducing the amount of memory needed to store the model, and lowering the computational cost of applying it.

Original: It has been pointed out by several authors that a major drawback of Lasso (or group-Lasso) regularization is that in the presence of groups of highly correlated covariates/features, it tends to select only one or an arbitrary convex combination of features from each group (Bondell & Reich, 2008;
Revised: It has been pointed out by several authors that a major drawback of Lasso (or group-Lasso) regularization is that in the presence of groups of highly correlated covariates/features, it tends to select only one or an arbitrary convex combination of features from each group (Bondell & Reich, 2008;

Original: B ¨ uhlmann et al., 2013; Figueiredo & Nowak, 2016; Oswal et al., 2016; Zou & Hastie, 2005).
Revised: B ¨ uhlmann et al., 2013; Figueiredo & Nowak, 2016; Oswal et al., 2016; Zou & Hastie, 2005).

Original: Moreover, the learning process tends to be unstable, in the sense that subsets of parameters that end up being selected may change dramatically with minor changes in the data or algorithmic procedure.
Revised: Moreover, the learning process tends to be unstable, in the sense that subsets of parameters that end up being selected may change dramatically with minor changes in the data or algorithmic procedure.

Original: In DNNs, it is almost unavoidable to encounter correlated features, not only due to the high dimensionality of the input to each layer, but also because neurons tend to co-adapt, yielding strongly correlated features that are passed as input to the subsequent layer (Srivastava et al., 2014).
Revised: In DNNs, it is almost unavoidable to encounter correlated features, not only due to the high dimensionality of the input to each layer, but also because neurons tend to co-adapt, yielding strongly correlated features that are passed as input to the subsequent layer (Srivastava et al., 2014).

Original: In this work, we propose using, as a regularizer for learning DNNs, the group version of the ordered weighted (cid:96) 1 (OWL) norm Figueiredo & Nowak (2016), termed group-OWL (GrOWL), which was recently proposed by Oswal et al.
Revised: In this work, we propose using, as a regularizer for learning DNNs, the group version of the ordered weighted (cid:96) 1 (OWL) norm (Figueiredo & Nowak, 2016), termed group-OWL (GrOWL), which was recently proposed by Oswal et al.

Original: In a linear regression context, GrOWL regularization has been shown to avoid the above mentioned deﬁciency of group-Lasso regularization.
Revised: In a linear regression context, GrOWL regularization has been shown to avoid the above mentioned deﬁciency of group-Lasso regularization.

Original: In addition to being a sparsity-inducing regularizer, GrOWL is able to explicitly identify groups of correlated features and set the corresponding parameters/weights to be very close or exactly equal to each other, thus taking advantage of correlated features, rather than being negatively affected by them.
Revised: In addition to being a sparsity-inducing regularizer, GrOWL is able to explicitly identify groups of correlated features and set the corresponding parameters/weights to be very close or exactly equal to each other, thus taking advantage of correlated features, rather than being negatively affected by them.

Original: In deep learning parlance, this corresponds to adaptive parameter sharing/tying , where instead of having to deﬁne a priori which sets of parameters are forced to share a common value, these sets are learned during the training process.
Revised: In deep learning parlance, this corresponds to adaptive parameter sharing/tying , where instead of having to deﬁne a priori which sets of parameters are forced to share a common value, these sets are learned during the training process.

Original: We exploit this ability of GrOWL regularization to encourage parameter sparsity and group-clustering in a two-stage procedure depicted in Fig.
Revised: We exploit this ability of GrOWL regularization to encourage parameter sparsity and group-clustering in a two-stage procedure depicted in Fig.

Original: 1: we ﬁrst use GrOWL to identify the signiﬁcant parameters/weights of the network and, simultaneously, the correlated cluster information of the selected features; then, we retrain the network only in terms of the selected features, while enforcing the weights within the same cluster to share a common value.
Revised: 1: we ﬁrst use GrOWL to identify the signiﬁcant parameters/weights of the network and, simultaneously, the correlated cluster information of the selected features; then, we retrain the network only in terms of the selected features, while enforcing the weights within the same cluster to share a common value.

Original: The experiments reported in this paper conﬁrm that using GrOWL regularization in learning DNNs does encourage sparsity, by eliminating small weights, and  yields parameter sharing, by tying groups of weights to have the same absolute values.
Revised: The experiments reported below conﬁrm that using GrOWL regularization in learning DNNs encourages sparsity and also yields parameter sharing, by forcing groups of weights to share a common absolute value.

Original: We test the proposed approach on two benchmark datasets, MNIST and CIFAR-10, comparing its performance with weight decay and group-Lasso regularization, and exploring the accuracy-memory trade-off.
Revised: We test the proposed approach on two benchmark datasets, MNIST and CIFAR-10, comparing it with weight decay and group-Lasso regularization, and exploring the accuracy-memory trade-off.

Original: Our results indicate that GrOWL is able to reduce the number of free parameters in the network without degrading the accuracy, as compared to otherregularization approaches.
Revised: Our results indicate that GrOWL is able to reduce the number of free parameters in the network without degrading the accuracy, as compared to other approaches.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: In order to relieve the burden on both required memory and data for training and storing DNNs, a substantial amount of work has focused on reducing the number of free parametersthat need to be estimated, namely by enforcing weight sharing.
Revised: In order to relieve the burden on both required memory and data for training and storing DNNs, a substantial amount of work has focused on reducing the number of free parameters to be estimated, namely by enforcing weight sharing.

Original: The classical instance ofparameter sharing is found in the convolutional layers of DNNs (Goodfellow et al., 2016).
Revised: The classical instance of sharing is found in the convolutional layers of DNNs (Goodfellow et al., 2016).

Original: In fact,the use of weight-sharing as a simplifying technique for neural networks can be traced back to more than three decades ago (LeCun, 1987; Rumelhart & McClelland, 1986).
Revised: In fact, weight-sharing as a simplifying technique for NNs can be traced back to more than 30 years ago (LeCun, 1987; Rumelhart & McClelland, 1986).

Original: Recently,with the goal of reducing the storage and communication costs of DNNs, there has been a surge of interest in compressing the description of these models.
Revised: Recently, there has been a surge of interest in compressing the description of DNNs, with the aim of reducing their storage and communication costs.

Original: Various methods have been proposed that approximate or quantize the learned weights after the training process.
Revised: Various methods have been proposed to approximate or quantize the learned weights after the training process.

Original: Denton et al.
Revised: Denton et al.

Original: (2014) have shown that, in some cases, it is possible to replace the original weight matrix with a low-rank approximation.
Revised: (2014) have shown that, in some cases, it is possible to replace the original weight matrix with a low-rank approximation.

Original: Alternatively, Aghasi et al.
Revised: Alternatively, Aghasi et al.

Original: (2016) propose retraining the network layer by layer, keeping the layer inputs and outputs close to the originally trained model, while seeking a sparse transform matrix, whereas Gong et al.
Revised: (2016) propose retraining the network layer by layer, keeping the layer inputs and outputs close to the originally trained model, while seeking a sparse transform matrix, whereas Gong et al.

Original: (2014) propose the use of vector quantization to compress the parameters of DNNs.
Revised: (2014) propose using vector quantization to compress the parameters of DNNs.

Original: Network pruning is another relevant line of work.
Revised: Network pruning is another relevant line of work.

Original: Early work by LeCun et al.
Revised: In early work, LeCun et al.

Original: and Hassibi et al.
Revised: and Hassibi &

Original:  (1993) uses the information provided by  Hessian of the loss function to remove less important weights of the network, which requires expensive computation of second order derivatives.
Revised: Stork (1993) use the information provided by the Hessian of the loss function to remove less important weights; however, this requires expensive computation of second order derivatives.

Original: Recently,
Revised: Recently,

Original: Han et al.
Revised: Han et al.

Original: (2016) reduce the number of parameters by  an order of magnitude using an iterative algorithm that alternates between learning the parameters and removing those below a certain threshold.
Revised: (2016) reduce the number of parameters by up to an order of magnitude by alternating between learning the parameters and removing those below a certain threshold.

Original: Li et al.
Revised: Li et al.

Original: (2016) propose to prune ﬁlters, which seeks sparsity with respect to neurons, rather than connectivities; that approach relieves the burden on requiring sparse libraries or special hardware to deploy the network.
Revised: (2016) propose to prune ﬁlters, which seeks sparsity with respect to neurons, rather than connections; that approach relieves the burden on requiring sparse libraries or special hardware to deploy the network.

Original: All these methods either require multiple training/retraining iterations or a careful choice ofpruning thresholds.
Revised: All those methods either require multiple training/retraining iterations or a careful choice of thresholds.

Original: There is a large body of work onusing sparsity-inducing regularization for training DNNs.
Revised: There is a large body of work on sparsity-inducing regularization in deep learning.

Original: For example, Collins & Kohli (2014) exploit (cid:96) 1 and (cid:96) 0 regularization to encourage weight sparsity; however, the sparsity level achieved is typically modest, making this approach not competitive in terms of DNN compression.
Revised: For example, Collins & Kohli (2014) exploit (cid:96) 1 and (cid:96) 0 regularization to encourage weight sparsity; however, the sparsity level achieved is typically modest, making that approach not competitive for DNN compression.

Original: Group-Lasso has also been used in training DNNs, which allows seeking sparsity in terms of neurons (Scardapane et al., 2017; Alvarez & Salzmann, 2016;
Revised: Group-Lasso has also been used in training DNNs; it allows seeking sparsity in terms of neurons (Scardapane et al., 2017; Alvarez & Salzmann, 2016;

Original: Zhou et al., 2016; Murray & Chiang, 2015) or other structures, e.g. , ﬁlters, channels, ﬁlter shapes, and layer depth (Wen et al., 2016).
Revised: Zhou et al., 2016; Murray & Chiang, 2015) or other structures, e.g. , ﬁlters, channels, ﬁlter shapes, and layer depth (Wen et al., 2016).

Original: However, as mentioned above, both Lasso and group-Lasso can fail to select important features if they are strongly correlated with each other. In Section 4, we will further illustrate this fact on both synthetic  and real data.
Revised: However, as mentioned above, both Lasso and group-Lasso can fail in the presence of strongly correlated features (as illustrated in Section 4, with both synthetic data and real data.

Original: A recent stream of work has focused on using further parameter sharing in convolutional DNNs.
Revised: A recent stream of work has focused on using further parameter sharing in convolutional DNNs.

Original: By tying weights in an appropriate way, Dieleman et al.
Revised: By tying weights in an appropriate way, Dieleman et al.

Original: (2016) obtain a convolutional DNN with rotation invariance.
Revised: (2016) obtain a convolutional DNN with rotation invariance.

Original: On the task of analyzing positions in the game Go , Clark & Storkey (2015) showed improved performance by constraining features to be invariant to reﬂections along the x-axis, y-axis, and diagonal-axis.
Revised: On the task of analyzing positions in the game Go , Clark & Storkey (2015) showed improved performance by constraining features to be invariant to reﬂections along the x-axis, y-axis, and diagonal-axis.

Original: Finally, Chen et al.
Revised: Finally, Chen et al.

Original: (2015) used a hash function to randomly group the weights such that those in a hash bucket share the same value.
Revised: (2015) used a hash function to randomly group the weights such that those in a hash bucket share the same value.

Original: In contrast, with GrOWL regularization, we aim to learn weight sharing from the data itself, rather than specifying it a priori .
Revised: In contrast, with GrOWL regularization, we aim to learn weight sharing from the data itself, rather than specifying it a priori .

Original: Dropout -type methods have been proposed to ﬁght over-ﬁtting and are very popular, arguably due to their simplicity of implementation.
Revised: Dropout -type methods have been proposed to ﬁght over-ﬁtting and are very popular, arguably due to their simplicity of implementation (Srivastava et al., 2014).

Original: It has been shown that dropout effectively reduces over-ﬁtting and prevents different neurons from co-adapting.
Revised: Dropout has been shown to effectively reduce over-ﬁtting and prevent different neurons from co-adapting.

Original: Decorrelation is another popular technique in deep learning pipelines (Bengio & Bergstra, 2009; Cogswell et al., 2015; Rodr ´ ıguez et al., 2016). Unlike sparsity-inducing regularizers, decorrelation methods try to make full use of the model’s capacity by decorrelating the neurons.
Revised: Decorrelation is another popular technique in deep learning pipelines (Bengio & Bergstra, 2009; Cogswell et al., 2015; Rodr ´ ıguez et al., 2016); unlike sparsity-inducing regularizers, these methods try to make full use of the model’s capacity by decorrelating the neurons.

Original: Although dropout and decorrelation can reduce over-ﬁtting, they do not compress the network, hence do not address the issue of high memory cost.
Revised: Although dropout and decorrelation can reduce over-ﬁtting, they do not compress the network, hence do not address the issue of high memory cost.

Original: It should also be mentioned that our proposal can be seen as complementary to dropout and decorrelation, in the following sense: whereas dropout and decorrelation can reduce co-adaption of nodes during training, GrOWL regularization copes with co-adaptation by tying together the weights associated to co-adapted nodes.
Revised: It should also be mentioned that our proposal can be seen as complementary to dropout and decorrelation: whereas dropout and decorrelation can reduce co-adaption of nodes during training, GrOWL regularization copes with co-adaptation by tying together the weights associated to co-adapted nodes.

Original:  G ROUP -OWL R EGULARIZATION FOR D EEP L EARNING 3. T HE G ROUP -OWL N ORM
Revised:  G ROUP -OWL R EGULARIZATION FOR D EEP L EARNING 3. T HE G ROUP -OWL N ORM

Original: We start by recalling the deﬁnition of the group-OWL (GrOWL) regularizer and very brieﬂy reviewing some of its relevant properties (Oswal et al., 2016).
Revised: We start by recalling the deﬁnition of the group-OWL (GrOWL) regularizer and very brieﬂy reviewing some of its relevant properties (Oswal et al., 2016).

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: Given a matrix W ∈ R n × m , let w [ i ] · denote the row of W with the i -th largest (cid:96)norm.
Revised: Given a matrix W ∈ R n × m , let w [ i ] · denote the row of W with the i -th largest (cid:96)norm.

Original: Let λ ∈ R n + , with 0 < λ 1 ≥ λ 2 ≥ · · · ≥ λ n ≥ 0 .
Revised: Let λ ∈ R n + , with 0 < λ 1 ≥ λ 2 ≥ · · · ≥ λ n ≥ 0 .

Original: The GrOWL regularizer (which is a norm) Ω λ : R n × m → R is deﬁned as
Revised: The GrOWL regularizer (which is a norm) Ω λ : R n × m → R is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: This is a group version of the OWL regularizer (Figueiredo & Nowak, 2016), also known as WSL ( weighted sorted (cid:96) 

Revised: This is a group version of the OWL regularizer (Figueiredo & Nowak, 2016), also known as WSL ( weighted sorted (cid:96)

Original: (Zeng & Figueiredo, 2014)) and SLOPE (Bogdan et al., 2015), where the groups are the rows of its matrix argument.
Revised: (Zeng & Figueiredo, 2014)) and SLOPE (Bogdan et al., 2015), where the groups are the rows of its matrix argument.

Original: It is clear that GrOWL includes group-Lasso as a special case when λ 1 = λ n .
Revised: It is clear that GrOWL includes group-Lasso as a special case when λ 1 = λ n .

Original: As a regularizer for multiple/multi-task linear regression, each row of W contains the regression coefﬁcients of a given feature, for the m tasks.
Revised: As a regularizer for multiple/multi-task linear regression, each row of W contains the regression coefﬁcients of a given feature, for the m tasks.

Original: It has been shown that by adding the
Revised: It has been shown that by adding the

Original: GrOWL regularizer to a standard squared error loss function, the resulting estimate of W has the following property: rows associated with highly correlated covariates are very close or even exactly equal to each other (Oswal et al., 2016).
Revised: GrOWL regularizer to a standard squared-error loss function, the resulting estimate of W has the following property: rows associated with highly correlated covariates are very close or even exactly equal to each other (Oswal et al., 2016).

Original: In the linear case, GrOWL encourages correlated features to form predictive clusters corresponding to the groups of rows that are nearly or exactly equal.
Revised: In the linear case, GrOWL encourages correlated features to form predictive clusters corresponding to the groups of rows that are nearly or exactly equal.

Original: The rationale underlying this paper is that when used as a regularizer for DNN learning, GrOWL will induce both sparsity and parameters tying, as illustrated in Fig.
Revised: The rationale underlying this paper is that when used as a regularizer for DNN learning, GrOWL will induce both sparsity and parameters tying, as illustrated in Fig.

Original: 2 and explained below ingreater detail.
Revised: 2 and explained below in detail.

Original: 3. L AYER -W ISE G R OWL R EGULARIZATION F OR F EEDFORWARD N EURAL N ETWORKS
Revised: 3. L AYER -W ISE G R OWL R EGULARIZATION F OR F EEDFORWARD N EURAL N ETWORKS

Original: A typical feed-forward DNN with L layers can be treated as a function f of the following form: 
Revised: A typical feed-forward DNN with L layers can be treated as a function f of the following form: f ( x, θ ) ≡ h L = f L ( h L − 1 W L + b L ) , h L − 1 =

Original: [Equation]
Revised: f L − 1 ( h L − 2 W L − 1 + b L − 1 ) , . . . , h 1 = f 1 ( xW 1 + b 1 ) θ = (cid:0) W 1 , b 1 , . .

Original:  denotes the set of parameters of the network, and each f i is a componentwise nonlinear activation function, with the rectiﬁed linear unit (ReLU), the sigmoid , and the hyperbolic tangent being common choices for this function (Goodfellow et al., 2016).
Revised: , W L , b L (cid:1) denotes the set of parameters of the network, and each f i is a componentwise nonlinear activation function, with the rectiﬁed linear unit (ReLU), the sigmoid , and the hyperbolic tangent being common choices for this function (Goodfellow et al., 2016).

Original: Given labelled data D = (cid:0) ( x (1) , y (1) ) , ..., ( x ( m ) , y ( m ) ) (cid:1) , DNN learning may be formalized as an optimization problem,
Revised: Given labelled data D = (cid:0) ( x (1) , y (1) ) , ..., ( x ( m ) , y ( m ) ) (cid:1) , DNN learning may be formalized as an optimization problem,

Original: [Equation]
Revised: [Equation]

Original: where[Equation] [Equation] and L (cid:0) y, ˆ y (cid:1) is the loss incurred when the DNN predicts ˆ y for y , and R is a regularizer.
Revised: where L (cid:0) y, ˆ y (cid:1) is the loss incurred when the DNN predicts ˆ y for y , and R is a regularizer.

Original: The regularizerherein adopted is a sum of GrOWL penalties, each applied to each layer of the neural network, i.e.,
Revised: Here, we adopt as regularizer a sum of GrOWL penalties, each for each layer of the neural network, i.e.,

Original: [Equation]
Revised: [Equation]

Original: where N l denotes the number of neurons in the l -th layer and 0 < λ ( l ) 1 ≥ λ ( l ) 2 ≥ · · · ≥ λ ( l ) N l − 1 ≥ 0 .
Revised: where N l denotes the number of neurons in the l -th layer and 0 < λ ( l ) 1 ≥ λ ( l ) 2 ≥ · · · ≥ λ ( l ) N l − 1 ≥ 0 .

Original: Notice that R ( θ ) does not depend on b 1 , ..., b L , which means these parameters are not regularized, as is common practice.
Revised: Since R ( θ ) does not depend on b 1 , ..., b L , the biases are not regularized, as is common practice.

Original: As indicated in Eq.
Revised: As indicated in Eq.

Original: (3), the number of groups in each GrOWL regularizer equals the number of neurons in the previous layer, i.e., λ ( l ) ∈ R N l − 1 .
Revised: (3), the number of groups in each GrOWL regularizer is the number of neurons in the previous layer, i.e., λ ( l ) ∈ R N l − 1 .

Original: In other words, we treat the weights associated with each input feature as a group .
Revised: In other words, we treat the weights associated with each input feature as a group .

Original: For fully connected layers, where W l ∈ R N l − 1 × N l , each group is a row of the weight matrix.
Revised: For fully connected layers, where W l ∈ R N l − 1 × N l , each group is a row of the weight matrix.

Original: When considering convolutional layers, where W l ∈ R F w × F h × N l − 1 × N l , with F w and F h denoting the width and height, respectively, of each ﬁlter, we ﬁrst reshape W l to a 2-dimensional array, i.e., W l → W 2 D l , where W 2 D l ∈ R N l − 1 × ( F w F h N l ) , and then apply GrOWL on the reshaped matrix.
Revised: In convolutional layers, where W l ∈ R F w × F h × N l − 1 × N l , with F w and F h denoting the width and height, respectively, of each ﬁlter, we ﬁrst reshape W l to a 2-dimensional array, i.e., W l → W 2 D l , where W 2 D l ∈ R N [Equation] × ( F w F h N l ) , and then apply GrOWL on the reshaped matrix.

Original: That is, if the l -th layer is convolutional, then
Revised: That is, if the l -th layer is convolutional, then

Original: [Equation]
Revised: [Equation]

Original: Each row of W 2 D l represents the operation on an input channel.
Revised: Each row of W 2 D l represents the operation on an input channel.

Original: The rationale to apply the GrOWL regularizer to each row of the reshaped weight matrix is that GrOWL can select the relevant features of the network, while encouraging the coefﬁcient rows of each layer associated with strongly correlated features from the previous layer to be nearly or exactly equal, as depicted in Fig.
Revised: The rationale to apply the GrOWL regularizer to each row of the reshaped weight matrix is that GrOWL can select the relevant features of the network, while encouraging the coefﬁcient rows of each layer associated with strongly correlated features from the previous layer to be nearly or exactly equal, as depicted in Fig.

Original: The goal is to signiﬁcantly reduce the complexity by: (i) pruning unimportant neurons of the previous layer that correspond to zero rows of the (reshaped) weight matrix of the current layer; (ii) grouping the rows associated with highly correlated features of the previous layer, thus encouraging the coefﬁcient rows in each of these groups to be very close to each other.
Revised: The goal is to signiﬁcantly reduce the complexity by: (i) pruning unimportant neurons of the previous layer that correspond to zero rows of the (reshaped) weight matrix of the current layer; (ii) grouping the rows associated with highly correlated features of the previous layer, thus encouraging the coefﬁcient rows in each of these groups to be very close to each other.

Original: As a consequence, in the retraining process, we could further compress the neural network by enforcing the parameters within each neuron that belong to the same cluster to share same values.
Revised: As a consequence, in the retraining process, we can further compress the neural network by enforcing the parameters within each neuron that belong to the same cluster to share same values.

Original: In the work of Alvarez & Salzmann (2016), each group is predeﬁned as the set of parameters associated to a neuron, and group-Lasso regularization is applied to seek group sparsity, which corresponds to zeroing out redundant neurons of each layer.
Revised: In the work of Alvarez & Salzmann (2016), each group is predeﬁned as the set of parameters associated to a neuron, and group-Lasso regularization is applied to seek group sparsity, which corresponds to zeroing out redundant neurons of each layer.

Original: In contrast, we treat the ﬁlters corresponding with the same input channel as a group, and GrOWL is applied to prune the redundant groups and thus remove the associated unimportant neurons of  previous layer , while grouping associated parameters of the current layer that correspond with highly correlated input features to different clusters.
Revised: In contrast, we treat the ﬁlters corresponding to the same input channel as a group, and GrOWL is applied to prune the redundant groups and thus remove the associated unimportant neurons of the previous layer , while grouping associated parameters of the current layer that correspond with highly correlated input features to different clusters.

Original: Moreover,  shown in Section 4, group-Lasso can fail at selecting all relevant features of previous layers, and for the selected ones the corresponding coefﬁcient groups are quite dissimilar from each other, making it impossible to further compress the DNN by enforcing parameter tying.
Revised: Moreover, as shown in Section 4, group-Lasso can fail at selecting all relevant features of previous layers, and for the selected ones the corresponding coefﬁcient groups are quite dissimilar from each other, making it impossible to further compress the DNN by enforcing parameter tying.

Original: 3. P ROXIMAL G RADIENT A LGORITHM
Revised: 3. P ROXIMAL G RADIENT A LGORITHM

Original: To solve (2), we use a proximal gradient algorithm (Bauschke & Combettes, 2011), which has the following general form: at the t -th iteration, the parameter estimates are updated according to
Revised: To solve (2), we use a proximal gradient algorithm (Bauschke & Combettes, 2011), which has the following general form: at the t -th iteration, the parameter estimates are updated according to

Original: [Equation]
Revised: [Equation]

Original:  where, for some convex function Q , prox Q denotes its proximity operator  (Bauschke & Combettes, 2011),which is deﬁned as prox Q ( ξ ) =
Revised: (5) where, for some convex function Q , prox Q denotes its proximity operator (or simply “prox”) (Bauschke & Combettes, 2011), deﬁned as prox Q ( ξ ) =

Original: arg min ν Q ( ν ) + 1 2 (cid:107) ν − ξ (cid:107) 22 .
Revised: arg min ν Q ( ν ) + 12 (cid:107) ν − ξ (cid:107) 22 .

Original: In the previous expression, (cid:107) ν − ξ (cid:107) 22 denotes the sum of the squares of the differences between the corresponding components of ν and ξ , regardless of their organization (in our case, a collection of matrices and vectors).
Revised: In Eq. (5), (cid:107) ν − ξ (cid:107) 22 denotes the sum of the squares of the differences between the corresponding components of ν and ξ , regardless of their organization (here, a collection of matrices and vectors).

Original: Since R ( θ ) , as deﬁned in (3), is separable across the weight matrices of different layers and zero for b 1 , ..., b L , the corresponding proximity operator is also separable, thus
Revised: Since R ( θ ) , as deﬁned in (3), is separable across the weight matrices of different layers and zero for b 1 , ..., b L , the corresponding prox is also separable, thus

Original: [Equation]
Revised: [Equation]

Original: It was proved by Oswal et al.
Revised: It was shown by Oswal et al.

Original: (2016) that the proximity operator ofthe GrOWLnorm can be computed as follows.
Revised: (2016) that the prox of GrOWL can be computed as follows.

Original: For some matrix V ∈ R N × M , let U =
Revised: For some matrix V ∈ R N × M , let U =

Original: prox Ω λ ( V ) , and v i and u
Revised: prox Ω λ ( V ) , and v i and u

Original: i denote the corresponding i -th rows.
Revised: i denote the corresponding i -th rows.

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where ˜ v =
Revised: where ˜ v =

Original: [ (cid:107) v 1 (cid:107) , (cid:107) v 2 (cid:107) , · · · , (cid:107) v N (cid:107) ] .
Revised: [ (cid:107) v 1 (cid:107) , (cid:107) v 2 (cid:107) , · · · , (cid:107) v N (cid:107) ] .

Original: The proximal operator prox Ω λ ( l ) for a vector in R N (in which case GrOWL coincides with OWL) can be computed with O ( n log n ) complexity, where the core computation is the so-called pool adjacent violators algorithm (PAVA (de Leeuw et al., 2009)) for isotonic regression.
Revised: For vectors in R N (in which case GrOWL coincides with OWL), prox Ω λ ( l ) can be computed with O ( n log n ) cost, where the core computation is the socalled pool adjacent violators algorithm (PAVA (de Leeuw et al., 2009)) for isotonic regression.

Original: We provide one of the existing algorithms in Appendix A; for details of the algorithm, the reader is referred to the work of Bogdan et al.
Revised: We provide one of the existing algorithms in Appendix A; for details, the reader is referred to the work of Bogdan et al.

Original: and Zeng & Figueiredo (2015).
Revised: and Zeng & Figueiredo (2015).

Original: In this paper, we apply the proximal gradient descent per epoch, which generally yields better performance.
Revised: In this paper, we apply the proximal gradient algorithm per epoch, which generally performs better.

Original: The training method is summarized in Algorithm 1.Algorithm
Revised: The training method is summarized in Algorithm 1.

Original: Input : parameters of the OWL regularizers λ ( l ) , ..., λ ( L ) , learning rate η for each epoch T do for each iteration t in epoch T do

Original: Update the parameter via backpropagation (BP) end for

Original: Apply proximity operator via (6) end for

Original: 3. I MPLEMENTATION D ETAILS 3.4. S ETTING THE G R OWL W EIGHTS
Revised: I MPLEMENTATION D ETAILS 3.4. S ETTING THE G R OWL W EIGHTS

Original: GrOWL is a family of regularizers, with different variants obtained by choosing different weight sequences λ 1 , . . .
Revised: GrOWL is a family of regularizers, with different variants obtained by choosing different weight sequences λ 1 , . . .

Original: , λ n .
Revised: , λ n .

Original: In this paper, we propose the following choice:
Revised: In this paper, we propose the following choice:

Original: [Equation]
Revised: [Equation]

Original: where p ∈ { 1 , ...n } is a parameter.
Revised: where p ∈ { 1 , ...n } is a parameter.

Original: The ﬁrst p weights follow a linear decay, while the remaining ones are all equal to Λ 1 .
Revised: The ﬁrst p weights follow a linear decay, while the remaining ones are all equal to Λ 1 .

Original: Notice that, if p = n , the above setting is equivalent to OSCAR (Bondell & Reich, 2008).
Revised: Notice that, if p = n , the above setting is equivalent to OSCAR (Bondell & Reich, 2008).

Original: Roughly speaking, Λ 1 controls the sparsifying strength of the regularizer, while Λcontrols the clustering property (correlation identiﬁcation ability) of GrOWL (Oswal et al., 2016).
Revised: Roughly speaking, Λ 1 controls the sparsifying strength of the regularizer, while Λcontrols the clustering property (correlation identiﬁcation ability) of GrOWL (Oswal et al., 2016).

Original: Moreover, by setting the weights to a common constant beyond index p means that clustering is only encouraged among the p largest coefﬁcients, i.e., only among relevant coefﬁcient groups.
Revised: Moreover, by setting the weights to a common constant beyond index p means that clustering is only encouraged among the p largest coefﬁcients, i.e., only among relevant coefﬁcient groups.

Original: Finding adequate choices for p , Λ 1 , and Λ 2 is crucial for jointly selecting the relevant featuresof the network and identifying the underlying correlations among them.
Revised: Finding adequate choices for p , Λ 1 , and Λ 2 is crucial for jointly selecting the relevant features and identifying the underlying correlations.

Original: In practice, we ﬁnd that with properly chosen p , GrOWL is able to ﬁnd more correlationsof the underlying data than OSCAR.
Revised: In practice, we ﬁnd that with properly chosen p , GrOWL is able to ﬁnd more correlations than OSCAR.

Original: We explore different choices of p in Section 4.1.
Revised: We explore different choices of p in Section 4.1.

Original: 3.4. P ARAMETER T YING
Revised: 3.4. P ARAMETER T YING

Original: After the initial training phase, at each layer l , rows of W l that corresponds to highly correlated outputs of layer l − 1 have been made similar or even exactly equal.
Revised: After the initial training phase, at each layer l , rows of W l that corresponds to highly correlated outputs of layer l − 1 have been made similar or even exactly equal.

Original: To further compress the DNN, we force rows that are close to each other to be identical.
Revised: To further compress the DNN, we force rows that are close to each other to be identical.

Original: We ﬁrst group the rows into different clusters 1 according to the pairwise similarity metric
Revised: We ﬁrst group the rows into different clusters 1 according to the pairwise similarity metric

Original: [Equation]
Revised: [Equation]

Original: where W l,i and W l,j denote the i -th and j -th rows of W l , respectively.
Revised: where W l,i and W l,j denote the i -th and j -th rows of W l , respectively.

Original: With the cluster information obtained by using GrOWL, we enforce parameter sharing for the rows that belong to a same cluster by replacing their values with the averages (centroid) of the rows in that cluster.
Revised: With the cluster information obtained by using GrOWL, we enforce parameter sharing for the rows that belong to a same cluster by replacing their values with the averages (centroid) of the rows in that cluster.

Original: In the following retraining process , let G ( l ) k denote the k -th cluster of the l -th layer, then centroid g ( l ) k of this cluster is updated via
Revised: In the subsequent retraining process , let G ( l ) k denote the k -th cluster of the l -th layer, then centroid g ( l ) k of this cluster is updated via

Original: [Equation]
Revised: [Equation]

Original:  N UMERICAL R ESULTS
Revised:  N UMERICAL R ESULTS

Original: We assess the performance of the proposed method on two benchmark datasets: MNIST and CIFAR10.
Revised: We assess the performance of the proposed method on two benchmark datasets: MNIST and CIFAR-10.

Original: We consider two different networks and compare GrOWL with group-Lasso and weight decay, in terms of the compression vs accuracy trade-off.
Revised: We consider two different networks and compare GrOWL with group-Lasso and weight decay, in terms of the compression vs accuracy trade-off.

Original: For fair comparison, the training-retraining pipeline is used with the different regularizers.
Revised: For fair comparison, the training-retraining pipeline is used with the different regularizers.

Original: After the initial training phase, the rows that are close to each other are clustered together and forced to share common values in the retraining phase.
Revised: After the initial training phase, the rows that are close to each other are clustered together and forced to share common values in the retraining phase.

Original: We implement all models using Tensorﬂow Abadi et al.
Revised: We implement all models using Tensorﬂow Abadi et al.

Original: To evaluate the effect of the different regularizers, we use the following quantities: sparsity = # zero params # total params ; compression rate = # total params # unique params ; parameter sharing =
Revised: We evaluate the effect of the different regularizers using the following quantities: sparsity = (# zero params) / (# total params), compression rate = (# total params) / (# unique params), and parameter sharing =

Original: # nonzero params # unique params .
Revised: (# nonzero params) / (# unique params).

Original: 4. D IFFERENT C HOICES OF G R OWL P ARAMETERS
Revised: 4. D IFFERENT C HOICES OF G R OWL P ARAMETERS

Original: We ﬁrst evaluate GrOWL norm regularization on syntheticdata. The covariance matrix Σ of the data  X is set as a block diagonal matrix such that each block represents the covariances of a cluster of correlated features, and there is a gap g between two blocks.
Revised: First, we consider a synthetic data matrix X with block-diagonal covariance matrix Σ , where each block corresponds to a cluster of correlated features, and there is a gap g between two blocks.

Original: Within each cluster, the covariance between two features X i and X j is cov ( X i , X j ) = 0 .
Revised: Within each cluster, the covariance between two features X i and X j is cov ( X i , X j ) = 0 .

Original: 96 | i − j | . we assume that the features from different clusters are independent of each other, while In this section, we set n = 784 , K = 10 and block sizeequals 50 ,  g = 28 .
Revised: 96 | i − j | , while features from different clusters are generated independently of each other. We set n = 784 , K = 10 , block size 50 , and gap g = 28 .

Original: We generate 10000 trainingexamples and 1000 testing examples.
Revised: We generate 10000 training and 1000 testing examples.

Original: We train a neural network with a single fully connected layer that consists of 300 hidden units.
Revised: We train a NN with a single fully-connected layer ofhidden units.

Original: In Fig 3, we plot the the ﬁrst 25000 entries of the sorted  similarity matrices  obtained by applying GrOWL with different p (Eq 9) values.
Revised: Fig 3 shows the ﬁrst 25000 entries of the sorted pairwise similarity matrices (Eq 10) obtained by applying GrOWL with different p (Eq 9) values.

Original: Although, by setting the weights  to a common constant beyond index p means that clustering is only encouraged among the p largest coefﬁcients, i.e., only among relevant coefﬁcient groups, Figure 3 shows that, with probably chosen p , GrOWL is able to encourage more parameter tying than OSCAR ( p = n ).
Revised: By setting the weights beyond index p to a common constant implies that clustering is only encouraged among the p largest coefﬁcients, i.e. , relevant coefﬁcient groups; however, Fig. 3 shows that, with properly chosen p , GrOWL yields more parameter tying than OSCAR ( p = n ).

Original: On the other hand,choosing smaller p values allows us to use large Λ 2 , which encourages parameter tying among comparatively loose correlations.
Revised: On the other hand, smaller p values allow using large Λ 2 , encouraging parameter tying among relatively loose correlations.

Original: In practice, we found that choosing p around the target portion of nonzero parameters over total parameters yields better performance in general.
Revised: In practice, we ﬁnd that for p around the target fraction of nonzero parameters leads to good performance in general.

Original: The intuition is that we only want to identify correlations among the remaining important parameters/connections.
Revised: The intuition is that we only need to identify correlations among the selected important features.

Original: Fig 3 shows that weight decay (denoted as (cid:96) 2 )can also push parameters together, though the parameter tying effect is not as obvious as that obtained by applying GrOWL.
Revised: Fig. 3 shows that weight decay (denoted as (cid:96) 2 ) also pushes parameters together, though the parameter-tying effect is not as clear as that of GrOWL.

Original: As has been observed in the literature (Bondell & Reich, 2008), weight decay often achieves better generalization as compared with sparsity-inducing regularizers.
Revised: As has been observed in the literature (Bondell & Reich, 2008), weight decay often achieves better generalization than sparsity-inducing regularizers.

Original: It achieves this via parameter shrinkage, especially in the highly correlated region, but it cannot yielda sparse model.
Revised: It achieves this via parameter shrinkage, especially in the highly correlated region, but it does not yield sparse models.

Original: In the following section, we explore the compression performance of GrOWL by comparing it with both group-Lasso and weight decay.
Revised: In the following section, we explore the compression performance of GrOWL by comparing it with both group-Lasso and weight decay.

Revised: We also explore how to further improve the accuracy vs compression trade-off by using sparsity-inducing regularization together with weight decay ( (cid:96) 2 ).

Original: For each case, the baseline performance is provided as the best performance obtained by running the original neural network (without compression) after sweeping the hyper-parameter on the weight decay regularizer over a range of values.
Revised: For each case, the baseline performance is provided as the best performance obtained by running the original neural network (without compression) after sweeping the hyper-parameter on the weight decay regularizer over a range of values.

Original: 4. F ULLY C ONNECTED N EURAL N ETWORK ON MNIST
Revised: 4. F ULLY C ONNECTED N EURAL N ETWORK ON MNIST

Original: The MNIST dataset contains 28 × 28 (784) pixels images of handwritten digits (0, 1, ..., 9), each located at the center of the image.
Revised: The MNIST dataset contains centered images of handwritten digits (0–9), of size 28 × 28 (784) pixels.

Original: Fig 4 (a) shows the ( 784 × 784 ) correlation matrix of the dataset (the margins are zero due to the redundant background of the images).
Revised: Fig 4 (a) shows the ( 784 × 784 ) correlation matrix of the dataset (the margins are zero due to the redundant background of the images).

Original: We use a network with a single fully connected layer of 300 hidden units.
Revised: We use a network with a single fully connected layer of 300 hidden units.

Original: The network is trained for 300 epochs and then retrained for an additional 100 epochs, both with momentum.
Revised: The network is trained for 300 epochs and then retrained for an additional 100 epochs, both with momentum.

Original: The initial learning rate is set to 0. for boththe training and retraining phases, andit is reduced by a factor of 0.96 every 10 epochs.
Revised: The initial learning rate is set to 0.001, for both training and retraining, and is reduced by a factor of 0.96 every 10 epochs.

Original: We set p = 0 .
Revised: We set p = 0 .

Original: 5 , and Λ 1 , Λ 2 are selected through grid search.
Revised: 5 , and Λ 1 , Λ 2 are selected by grid search.

Original: Pairwise similarities (see Eq.
Revised: Pairwise similarities (see Eq.

Original: (10)) between the rows of the weight matrices learned with different regularizers are shown in Fig.
Revised: (10)) between the rows of the weight matrices learned with different regularizers are shown in Fig.

Original: 4 (b–f).
Revised: 4 (b–f).

Original: As we can see, GrOWL (+ (cid:96) 2 ) identiﬁes more correlations than group-Lasso (+ (cid:96) 2 ), and the similarity patterns in Fig.
Revised: As we can see, GrOWL (+ (cid:96) 2 ) identiﬁes more correlations than group-Lasso (+ (cid:96) 2 ), and the similarity patterns in Fig.

Original: 4 (b, c) are very close to that of the data (Fig. 4(a)).
Revised: 4 (b, c) are very close to that of the data (Fig. 4(a)).

Original: On the other hand, weight decay also identiﬁes correlations between parameter rows, but it does not induce a sparse model.
Revised: On the other hand, weight decay also identiﬁes correlations between parameter rows, but it does not induce sparsity.

Original: Moreover, as shown in Table 1, GrOWL obtains a higher level of parameter sharing thandoes weight decay.
Revised: Moreover, as shown in Table 1, GrOWL yields a higher level of parameter sharing than weight decay, matching what we observed on synthetic data in Section 4.1. [Table]

Original: The compression vs accuracy trade-off of the different regularizers is summarized in Table1.
Revised: The compression vs accuracy trade-off of the different regularizers is summarized in Table

Original: As we can see, by applyingthe (cid:96) 2 norm regularizer together with group-Lasso or GrOWL, we achieve a higher compression rate while yielding even better accuracy.
Revised: [Table] where we see that applying (cid:96) 2 regularization together with group-Lasso or GrOWL leads to a higher compression ratio, with negligible effect on the accuracy.

Original: Table 1 also shows that,with even smaller sparsities after the initial training phase, GrOWL ( + (cid:96) 2 ) compresses the network more than group-Lasso ( + (cid:96) 2 ) does, due to the signiﬁcant amount of correlation it identiﬁes. This also implies that group-Lassocan only select a subset of the correlated features while GrOWL selects all of them.
Revised: Table 1 also shows that, even with lower sparsity after the initial training phase, GrOWL ( + (cid:96) 2 ) compresses the network more than group-Lasso ( + (cid:96) 2 ), due to the signiﬁcant amount of correlation it identiﬁes; this also implies that group-Lasso only selects a subset of the correlated features, while GrOWL selects all of them.

Original: On the other hand, group-Lasso suffers from randomly selecting a subset of correlated features.
Revised: On the other hand, group-Lasso suffers from randomly selecting a subset of correlated features; this effect is illustrated in Fig.

Original: We illustrate this effect in Fig. 5, which plots the indices of nonzero rows, showing that GrOWL  stably selects relevant features while group-Lasso  does not.
Revised: 5, which plots the indices of nonzero rows, showing that GrOWL ( + (cid:96) 2 ) stably selects relevant features while group-Lasso ( + (cid:96) 2 ) does not.

Original: The mean ratios of changed indices 2 are 11.09% and 0.58% for group-Lasso and GrOWL, respectively. parameter sharing for baseline model. We train each model for 5 times and report the average values together with their standard deviations [Table]
Revised: The mean ratios of changed indices 2 are 11.09%, 0.59%, 32.07%, and 0.62% for group-Lasso, GrOWL, group-Lasso+ (cid:96) 2 , and GrOWL+ (cid:96) 2 , respectively.

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: 4. VGG-16 ON CIFAR- 
Revised: 4. VGG-16 ON CIFAR-

Original: To evaluate the proposed method on large DNNs, we consider a VGG-like (Simonyan & Zisserman, 2014) architecture proposed by Zagoruyko (2015) on the CIFAR-10 dataset.
Revised: To evaluate the proposed method on large DNNs, we consider a VGG-like (Simonyan & Zisserman, 2014) architecture proposed by Zagoruyko (2015) on the CIFAR-10 dataset.

Original: The network  is summarized in Table 3; comparing with the original VGG of Simonyan & Zisserman (2014), their fully connected layers are replaced with two small ones with 14.3% of the total parameters.
Revised: The network architecture is summarized in Appendix C; comparing with the original VGG of Simonyan & Zisserman (2014), their fully connected layers are replaced with two much smaller ones.

Original: A batch normalization layer is added after each convolutional layer and the ﬁrst fully connected layer.
Revised: A batch normalization layer is added after each convolutional layer and the ﬁrst fully connected layer.

Original: Unlike Zagoruyko (2015), we don’t use dropout.
Revised: Unlike Zagoruyko (2015), we don’t use dropout.

Original: We ﬁrst train  under different regularizers overepochs, then retrain  for another 50 epochs, using the learning rate decay scheme described by He et al. 
Revised: We ﬁrst train the network under different regularizers for 150 epochs, then retrain it for another 50 epochs, using the learning rate decay scheme described by He et al. Layers [Table]

Original: (2016): the initial rates for the training and retraining phases are set to 0 .
Revised: (2016): the initial rates for the training and retraining phases are set to 0 .

Original: 01 and 0 .
Revised: 01 and 0 .

Original: 001 , respectively; the learning rate is multiplied by 0 .
Revised: 001 , respectively [Table] the learning rate is multiplied by 0 .

Original: 1 every 60 epochs of the training phase, and every 20 epochs of the retraining phase.
Revised: 1 every 60 epochs of the training phase [Table] and every 20 epochs of the retraining phase.

Original: For GrOWL ( + (cid:96) 2 ), we set p = 0 .
Revised: For GrOWL ( + (cid:96) 2 ), we set p = 0 .

Original: 1 n (see Eq.
Revised: 1 n (see Eq.

Original: (9)) for all layers, where n denotes the number of rows of  each layer.Layers Weight Decay [Table]
Revised: (9)) for all layers, where n denotes the number of rows of the (reshaped) weight matrices of each layer.

Original: The results are summarized in Table 2.
Revised: The results are summarized in Table 2.

Revised: For all of the regularizers, we use the afﬁnity propagation algorithm (with preference value 3 set to 0 . 8 ) to cluster the rows at the end of initial training process.

Original: In the experiments, we found that it isquite hard to encourage parameter tying in the ﬁrst 7 convolutional layers. The reason may be that the ﬁlters of these ﬁrst 7 convolutional layers have comparatively large feature maps (from 32 × 32 to 8 × 8 ), which are only loosely correlated.
Revised: Our experiments showed that it is hard to encourage parameter tying in the ﬁrst 7 convolutional layers; this may be because the ﬁlters of these ﬁrst 7 convolutional layers have comparatively large feature maps (from 32 × 32 to 8 × 8 ), which are only loosely correlated.

Original: We illustrate this reasoning in Fig. 6, showing the cosine similarity between the vectorized output channels of layers 1, 6, 10, and 11, at the end of the training phase; it can be seen that the outputs of layers 10 and 11 have many more signiﬁcant similarities than that of layer 6.
Revised: We illustrate this reasoning in Fig. 6, showing the cosine similarity between the vectorized output channels of layers 1, 6, 10, and 11, at the end of the training phase; it can be seen that the outputs of layers 10 and 11 have many more signiﬁcant similarities than that of layer 6.

Original: Although the output channels of layer 1 also have certain similarities, as seen in Table 2, neither
Revised: Although the output channels of layer 1 also 3 In Table 4 (Appendix C), we explore the effect of different choices of this value.

Original:  GrOWL ( + (cid:96) 2 ) nor weight decay tends to tie the associated weights.
Revised: have certain similarities, as seen in Table 2, neither GrOWL ( + (cid:96) 2 ) nor weight decay tends to tie the associated weights.

Original: This may mean that the network is maintaining the diversity of the inputs in the ﬁrst few convolutional layers.
Revised: This may mean that the network is maintaining the diversity of the inputs in the ﬁrst few convolutional layers.

Original: Although GrOWL and weight decay both encourage parameter tying in layers 9-13, weight decay does it with less intensity and does not yield a sparse model, thus it cannot signiﬁcantly compress the network.
Revised: Although GrOWL and weight decay both encourage parameter tying in layers 9-13, weight decay does it with less intensity and does not yield a sparse model, thus it cannot signiﬁcantly compress the network.

Original: Li et al.
Revised: Li et al.

Original: (2016) propose to prune small weights after the initial training phase with weight decay, then retrain the reduced network; however, this type of method only achieves roughly 3 X compression 4 .
Revised: (2016) propose to prune small weights after the initial training phase with weight decay, then retrain the reduced network; however, this type of method only achieves compressionratios around 3.

Original: As mentioned by Li et al.
Revised: As mentioned by Li et al.

Original: (2016), layers 3-7 can be very sensitive to pruning; however, both GrOWL ( + (cid:96) 2 ) and group-Lasso ( + (cid:96) 2 ) effectively compress them witha minor accuracy loss.
Revised: (2016), layers 3-7 can be very sensitive to pruning; however, both GrOWL ( + (cid:96) 2 ) and group-Lasso ( + (cid:96) 2 ) effectively compress them, with minor accuracy loss.

Original: On the other hand, similar to what we observed by running the simple fully connected neural network on MNIST, the accuracy and memory trade-off imrpoves signiﬁcantly by applying GrOWL or group-Lasso together with (cid:96) 2 .
Revised: On the other hand, similar to what we observed by running the simple fully-connected network on MNIST, the accuracy-memory trade-off improves signiﬁcantly by applying GrOWL or group-Lasso together with (cid:96) 2 .

Original: However, Table 2 also shows that theaccuracy vs. memory trade-off achieved by GrOWL ( + (cid:96) 2 ) and group-Lasso ( + (cid:96) 2 ) are almost the same.
Revised: However, Table 2 also shows that the trade-off achieved by GrOWL ( + (cid:96) 2 ) and group-Lasso ( + (cid:96) 2 ) are almost the same.

Original: We suspect that this is caused by the fact thatthe CIFAR-10dataset is simple enough that one could still expect a good performance after strong network compression.
Revised: We suspect that this is caused by the fact that CIFAR-10 is simple enough that one could still expect a good performance after strong network compression.

Original: We believe this gap in the compression vs accuracy trade-off can be further increased in larger networks on more complex datasets.
Revised: We believe this gap in the compression vs accuracy trade-off can be further increased in larger networks on more complex datasets.

Original: We leave this question for future work.
Revised: We leave this question for future research.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have proposed using the recent GrOWL regularizer for simultaneous parameter sparsity and tying in DNN learning.
Revised: We have proposed using the recent GrOWL regularizer for simultaneous parameter sparsity and tying in DNN learning.

Original: By leveraging on GrOWL’s capability of simultaneously pruning redundant parameters and tying parameters associated with highly correlated features, we can signiﬁcantly reduce the model complexity, with a slight or even no loss in generalization accuracy.
Revised: By leveraging on GrOWL’s capability of simultaneously pruning redundant parameters and tying parameters associated with highly correlated features, we achieve signiﬁcant reduction of model complexity, with a slight or even no loss in generalization accuracy.

Original: We evaluate the proposed method on both a fully connected neural network and a deep convolutional neural network.
Revised: We evaluate the proposed method on both a fully connected neural network and a deep convolutional neural network.

Original: The results show that GrOWL can compress large DNNs by a factor of 11 .
Revised: The results show that GrOWL can compress large DNNs by factors ranging from 11 .

Original: 4 to 14 . 5 .
Revised: 4 to 14 . 5 , with negligible loss on accuracy.

Original: The correlation patterns identiﬁed by GrOWL are close to those of the input features to each layer.
Revised: The correlation patterns identiﬁed by GrOWL are close to those of the input features to each layer.

Original: This may be important to reveal the structure of the features, contributing to the interpretability of deep learning models.
Revised: This may be important to reveal the structure of the features, contributing to the interpretability of deep learning models.

Original: On the other hand, by automatically tying together the parameters corresponding to highly correlated features, GrOWL alleviates the negative effect of strong correlations that might be induced by the noisy input or the co-adaption tendency of deep neural networks.
Revised: On the other hand, by automatically tying together the parameters corresponding to highly correlated features, GrOWL alleviates the negative effect of strong correlations that might be induced by the noisy input or the co-adaption tendency of DNNs.

Original: However, the gap in the accuracy and memory trade-off obtained by applying GrOWL and groupLasso to compress neural networks gets decreased as we move to large DNNs.
Revised: The gap in the accuracy vs memory trade-off obtained by applying GrOWL and group-Lasso decreases as we move to large DNNs.

Original: Although we suspect this can be caused by running a much larger network on a simple dataset, this motivates us to explore different ways to apply GrOWL to compress neural networks.
Revised: Although we suspect this can be caused by running a much larger network on a simple dataset, it motivates us to explore different ways to apply GrOWL to compress neural networks.

Original: One possible method is applying GrOWL within each neuron by predeﬁning each 2D convolutional ﬁlter as a group (instead all 2D convolutional ﬁlters corresponding to the same input features).
Revised: One possible approach is to apply GrOWL within each neuron by predeﬁning each 2D convolutional ﬁlter as a group (instead all 2D convolutional ﬁlters corresponding to the same input features).

Original: By doing so, we encourage parameter sharing among much smaller units, which in turn would further improve the diversity vs parameter sharing trade-off.
Revised: By doing so, we encourage parameter sharing among much smaller units, which in turn would further improve the diversity vs parameter sharing trade-off.

Original: We leave this as future work.
Revised: We leave this for future work.

Original: A PPENDIX B A FFINITY P ROPAGATION
Revised: A PPENDIX B A FFINITY P ROPAGATION

Original: Afﬁnity Propagation is a clustering method based on sending messages between pairs of data samples.
Revised: Afﬁnity Propagation is a clustering method based on sending messages between pairs of data samples.

Original: The idea is to use these messages to determine the most representative data samples, which are called exemplars, then create clusters using these exemplars.
Revised: The idea is to use these messages to determine the most representative data samples, which are called exemplars, then create clusters using these exemplars.

Original: Provided with the precomputed data similarity s ( i, j ) , i (cid:54) =
Revised: Provided with the precomputed data similarity s ( i, j ) , i (cid:54) =

Original: j and preference s ( i, i ) , there are two types information being sent between samples iteratively: 1) responsibility r ( i, k ) , which measures how likely that sample k should be the exemplar of sample i ; 2) availability a ( k, i ) , which is the evidence that sample i should choose sample k as its exemplar.
Revised: j and preference s ( i, i ) , there are two types information being sent between samples iteratively: 1) responsibility r ( i, k ) , which measures how likely that sample k should be the exemplar of sample i ; 2) availability a ( k, i ) , which is the evidence that sample i should choose sample k as its exemplar.

Original: The algorithm is described in 3.
Revised: The algorithm is described in 3.

Original: Initialization : r ( i, k ) = 0 , a ( k, i ) = 0 for all i, k while not converge do
Revised: Initialization : r ( i, k ) = 0 , a ( k, i ) = 0 for all i, k while not converge do

Original: Responsibility updates:
Revised: Responsibility updates:

Original: [Equation]
Revised: [Equation]

Original: Availability updates: 
Revised: Availability updates: a (

Original: [Equation]
Revised: [Equation]

Revised: ←

Revised: [Equation]

Revised: 

Original:  end while
Revised:  end while

Original: Making assignments:
Revised: Making assignments:

Original: [Equation]
Revised: [Equation]

Original: Unlike k-means or agglomerative algorithm, Afﬁnity Propagation does not require the number of clusters as an input.
Revised: Unlike k-means or agglomerative algorithm [Table] Afﬁnity Propagation does not require the number of clusters as an input.

Original: We deem this as a desired property for enforcing parameter sharing in neural network compression because it’s impossible to have the exact number of clusters as a prior information.
Revised: We deem this as a desired property for enforcing parameter sharing in neural network compression because it’s impossible to have the exact number of clusters as a prior information.

Original: In practice, the input preference of Afﬁnity Propagation determines how likely each sample will be chosen as an exemplar and its value will inﬂuence the number of clusters created.[Figure]
Revised: In practice, the input preference of Afﬁnity Propagation determines how likely each sample will be chosen as an exemplar and its value will inﬂuence the number of clusters created.

Original: A PPENDIX D VGG-16 ON CIFAR- 
Revised: A PPENDIX C VGG-16 ON CIFAR-

Original: [Table]
Revised: [Table]

Revised: For each regularizer, we use different preference values to run Algorithm 3 to cluster the rows at the end of initial training process.

Revised: Then we retrain the neural network correspondingly.

Revised: The results are reported as the averages over 5 training and retraining runs.

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]