Original: L EARNING TO S HARE : S IMULTANEOUS P ARAMETER T YING AND S PARSIFICATION IN D EEP L EARNING
Revised: L EARNING TO S HARE : S IMULTANEOUS P ARAMETER T YING AND S PARSIFICATION IN D EEP L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Deep neural networks (DNNs) usually contain millions, maybe billions, of parameters/weights, making both storage and computation very expensive.
Revised: Deep neural networks (DNNs) usually contain millions, maybe billions, of parameters/weights, making both storage and computation very expensive.

Original: Moreover, although this complexity allows DNNs to learn complex mappings, it may also allow them to over-ﬁt, which has motivated  the use of regularization methods, e.g. , by encouraging weight sparsity.
Revised: This has motivated a large body of work to reduce the complexity of the neural network by using sparsity-inducing regularizers.

Original: Another well-known approach for controlling the complexity of DNNs is parameter sharing/tying, where certain sets of weights are forced to share a common value.
Revised: Another well-known approach for controlling the complexity of DNNs is parameter sharing/tying, where certain sets of weights are forced to share a common value.

Original: Some forms of weight sharing are hard-wired to express certain invariances, with a notable example being the shift-invariance of convolutional layers.
Revised: Some forms of weight sharing are hard-wired to express certain invariances, with a notable example being the shift-invariance of convolutional layers.

Original: However, there may be other groups of weights that may be tied together during the learning process, thus further reducing the complexity of the network.
Revised: However, there may be other groups of weights that may be tied together during the learning process, thus further reducing the complexity of the network.

Original: In this paper, we adopt a recently proposed sparsity-inducing regularizer, named GrOWL ( group ordered weighted (cid:96) 1 ), which encourages sparsity and, simultaneously, learns which groups of parameters should share a common value.
Revised: In this paper, we adopt a recently proposed sparsity-inducing regularizer, named GrOWL ( group ordered weighted (cid:96) 1 ), which encourages sparsity and, simultaneously, learns which groups of parameters should share a common value.

Original: GrOWL has been proven effective in linear regression, being able to identify and cope with strongly correlated covariates.
Revised: GrOWL has been proven effective in linear regression, being able to identify and cope with strongly correlated covariates.

Original: Unlike standard sparsity-inducing regularizers ( e.g. , (cid:96) 1 a.k.a.
Revised: Unlike standard sparsity-inducing regularizers ( e.g. , (cid:96) 1 a.k.a.

Original: Lasso), GrOWL not only eliminates unimportant neurons by setting all the corresponding weights to zero, but also explicitly identiﬁes strongly correlated neurons by tying the corresponding weights to a common value.
Revised: Lasso), GrOWL not only eliminates unimportant neurons by setting all the corresponding weights to zero, but also explicitly identiﬁes strongly correlated neurons by tying the corresponding weights to a common value.

Original: This ability of GrOWL motivates the following two-stage procedure: (i) use GrOWL regularization in the training process to simultaneously identify signiﬁcant neurons and groups of parameter that should be tied together; (ii) retrain the network, enforcing the structure that was unveiled in the previous phase, i.e. , keeping only the signiﬁcant neurons and enforcing the learned tying structure.
Revised: This ability of GrOWL motivates the following two-stage procedure: (i) use GrOWL regularization in the training process to simultaneously identify signiﬁcant neurons and groups of parameter that should be tied together; (ii) retrain the network, enforcing the structure that was unveiled in the previous phase, i.e. , keeping only the signiﬁcant neurons and enforcing the learned tying structure.

Original: We evaluate the proposed approach on several benchmark datasets, showing that it can dramatically compress the network with slight or even no loss on generalization performance.
Revised: We evaluate the proposed approach on several benchmark datasets, showing that it can dramatically compress the network with slight or even no loss on generalization performance.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep neural networks (DNNs) have recently revolutionized machine learning by dramatically advancing the state-of-the-art performance in many applications, ranging from speech and image recognition to playing video games (Goodfellow et al., 2016).
Revised: Deep neural networks (DNNs) have recently revolutionized machine learning by dramatically advancing the state-of-the-art performance in many applications, ranging from speech and image recognition to playing video games (Goodfellow et al., 2016).

Original: A typical DNN consists of a sequence of concatenated layers of functions, potentially involving millions or billions of parameters; by using extremely large training datasets, these architectures are able to learn extremely complex non-linear mappings, features, and dependencies.
Revised: A typical DNN consists of a sequence of concatenated layers of functions, potentially involving millions or billions of parameters; by using extremely large training datasets, these architectures are able to learn extremely complex non-linear mappings, features, and dependencies.

Original: A large amount of research has been devoted to the use of regularization in DNN learning (Goodfellow et al., 2016), aiming at reducing the generalization error.
Revised: A large amount of research has been devoted to the use of regularization in DNN learning (Goodfellow et al., 2016), aiming at reducing the generalization error.

Original: It has been shown that the parametrization of many DNNs is very redundant, in the sense that a large fraction of the parameters can be predicted from the remaining ones, with no accuracy loss (Denil et al., 2013).
Revised: It has been shown that the parametrization of many DNNs is very redundant, in the sense that a large fraction of the parameters can be predicted from the remaining ones, with no accuracy loss (Denil et al., 2013).

Original: Several regularization methods have been proposed to tackle the potential over-ﬁtting caused by this redundancy.
Revised: Several regularization methods have been proposed to tackle the potential over-ﬁtting caused by this redundancy.

Original: Arguably, the earliest and simplest choice is the classical (cid:96) 2 norm, known as weight decay in the early neural networks literature (Rumelhart et al., 1988), and as ridge regression in statistics.
Revised: Arguably, the earliest and simplest choice is the classical (cid:96) 2 norm, known as weight decay in the early neural networks literature (Rumelhart et al., 1988), and as ridge regression in statistics.

Original: In the past two decades, sparsity-inducing regularization based on the (cid:96) 1 norm, also known as Lasso (Tibshirani, 1996), and variants thereof, became standard tools in statistics and machine learning, including in deep learning (Goodfellow et al., 2016).
Revised: In the past two decades, sparsity-inducing regularization based on the (cid:96) 1 norm, also known as Lasso (Tibshirani, 1996), and variants thereof, became standard tools in statistics and machine learning, including in deep learning (Goodfellow et al., 2016).

Original: In recent work, Scardapane et al.
Revised: In recent work, Scardapane et al.

Original: (2017) use group-Lasso (a variant of Lasso that assumes that the parameters are organized in groups and encourages sparsity at the group level (Yuan & Lin, 2006)) in deep learning.
Revised: (2017) use group-Lasso (a variant of Lasso that assumes that the parameters are organized in groups and encourages sparsity at the group level (Yuan & Lin, 2006)) in deep learning.

Original: One of the effects of Lasso or group-Lasso regularization in learning a DNN is that many of the parameters may become exactly zero, thus reducing the amount of memory needed to store the model, and lowering the computational cost of applying it.
Revised: One of the effects of Lasso or group-Lasso regularization in learning a DNN is that many of the parameters may become exactly zero, thus reducing the amount of memory needed to store the model, and lowering the computational cost of applying it.

Original: It has been pointed out by several authors that a major drawback of Lasso (or group-Lasso) regularization is that in the presence of groups of highly correlated covariates/features, it tends to select only one or an arbitrary convex combination of features from each group (Bondell & Reich, 2008;
Revised: It has been pointed out by several authors that a major drawback of Lasso (or group-Lasso) regularization is that in the presence of groups of highly correlated covariates/features, it tends to select only one or an arbitrary convex combination of features from each group (Bondell & Reich, 2008;

Original: B ¨ uhlmann et al., 2013; Figueiredo & Nowak, 2016; Oswal et al., 2016; Zou & Hastie, 2005).
Revised: B ¨ uhlmann et al., 2013; Figueiredo & Nowak, 2016; Oswal et al., 2016; Zou & Hastie, 2005).

Original: Moreover, the learning process tends to be unstable, in the sense that subsets of parameters that end up being selected may change dramatically with minor changes in the data or algorithmic procedure.
Revised: Moreover, the learning process tends to be unstable, in the sense that subsets of parameters that end up being selected may change dramatically with minor changes in the data or algorithmic procedure.

Original: In DNNs, it is almost unavoidable to encounter correlated features, not only due to the high dimensionality of the input to each layer, but also because neurons tend to co-adapt, yielding strongly correlated features that are passed as input to the subsequent layer (Srivastava et al., 2014).
Revised: In DNNs, it is almost unavoidable to encounter correlated features, not only due to the high dimensionality of the input to each layer, but also because neurons tend to co-adapt, yielding strongly correlated features that are passed as input to the subsequent layer (Srivastava et al., 2014).

Original: In this work, we propose using, as a regularizer for learning DNNs, the group version of the ordered weighted (cid:96) 1 (OWL) norm Figueiredo & Nowak (2016), termed group-OWL (GrOWL), which was recently proposed by Oswal et al.
Revised: In this work, we propose using, as a regularizer for learning DNNs, the group version of the ordered weighted (cid:96) 1 (OWL) norm Figueiredo & Nowak (2016), termed group-OWL (GrOWL), which was recently proposed by Oswal et al.

Original: In a linear regression context, GrOWL regularization has been shown to avoid the above mentioned deﬁciency of group-Lasso regularization.
Revised: In a linear regression context, GrOWL regularization has been shown to avoid the above mentioned deﬁciency of group-Lasso regularization.

Original: In addition to being a sparsity-inducing regularizer, GrOWL is able to explicitly identify groups of correlated features and set the corresponding parameters/weights to be very close or exactly equal to each other, thus taking advantage of correlated features, rather than being negatively affected by them.
Revised: In addition to being a sparsity-inducing regularizer, GrOWL is able to explicitly identify groups of correlated features and set the corresponding parameters/weights to be very close or exactly equal to each other, thus taking advantage of correlated features, rather than being negatively affected by them.

Original: In deep learning parlance, this corresponds to adaptive parameter sharing/tying , where instead of having to deﬁne a priori which sets of parameters are forced to share a common value, these sets are learned during the training process.
Revised: In deep learning parlance, this corresponds to adaptive parameter sharing/tying , where instead of having to deﬁne a priori which sets of parameters are forced to share a common value, these sets are learned during the training process.

Original: We exploit this ability of GrOWL regularization to encourage parameter sparsity and group-clustering in a two-stage procedure depicted in Fig.
Revised: We exploit this ability of GrOWL regularization to encourage parameter sparsity and group-clustering in a two-stage procedure depicted in Fig.

Original: 1: we ﬁrst use GrOWL to identify the signiﬁcant parameters/weights of the network and, simultaneously, the correlated cluster information of the selected features; then, we retrain the network only in terms of the selected features, while enforcing the weights within the same cluster to share a common value.
Revised: 1: we ﬁrst use GrOWL to identify the signiﬁcant parameters/weights of the network and, simultaneously, the correlated cluster information of the selected features; then, we retrain the network only in terms of the selected features, while enforcing the weights within the same cluster to share a common value.

Original: The experiments reported in this paper conﬁrm that using GrOWL regularization in learning DNNs does encourage sparsity, by eliminating small weights, and yields parameter sharing, by tying groups of weights to have the same absolute values.
Revised: The experiments reported in this paper conﬁrm that using GrOWL regularization in learning DNNs does encourage sparsity, by eliminating small weights, and yields parameter sharing, by tying groups of weights to have the same absolute values.

Original: We test the proposed approach on two benchmark datasets, MNIST and CIFAR-10, comparing its performance with weight decay and group-Lasso regularization, and exploring the accuracy-memory trade-off.
Revised: We test the proposed approach on two benchmark datasets, MNIST and CIFAR-10, comparing its performance with weight decay and group-Lasso regularization, and exploring the accuracy-memory trade-off.

Original: Our results indicate that GrOWL is able to reduce the number of free parameters in the network without degrading the accuracy, as compared to other regularization approaches.
Revised: Our results indicate that GrOWL is able to reduce the number of free parameters in the network without degrading the accuracy, as compared to other regularization approaches.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: In order to relieve the burden on both required memory and data for training and storing DNNs, a substantial amount of work has focused on reducing the number of free parameters that need to be estimated, namely by enforcing weight sharing.
Revised: In order to relieve the burden on both required memory and data for training and storing DNNs, a substantial amount of work has focused on reducing the number of free parameters that need to be estimated, namely by enforcing weight sharing.

Original: The classical instance of parameter sharing is found in the convolutional layers of DNNs (Goodfellow et al., 2016).
Revised: The classical instance of parameter sharing is found in the convolutional layers of DNNs (Goodfellow et al., 2016).

Original: In fact, the use of weight-sharing as a simplifying technique for neural networks can be traced back to more than three decades ago (LeCun, 1987; Rumelhart & McClelland, 1986).
Revised: In fact, the use of weight-sharing as a simplifying technique for neural networks can be traced back to more than three decades ago (LeCun, 1987; Rumelhart & McClelland, 1986).

Original: Recently, with the goal of reducing the storage and communication costs of DNNs, there has been a surge of interest in compressing the description of these models.
Revised: Recently, with the goal of reducing the storage and communication costs of DNNs, there has been a surge of interest in compressing the description of these models.

Original: Various methods have been proposed that approximate or quantize the learned weights after the training process.
Revised: Various methods have been proposed that approximate or quantize the learned weights after the training process.

Original: Denton et al.
Revised: Denton et al.

Original: (2014) have shown that, in some cases, it is possible to replace the original weight matrix with a low-rank approximation.
Revised: (2014) have shown that, in some cases, it is possible to replace the original weight matrix with a low-rank approximation.

Original: Alternatively, Aghasi et al.
Revised: Alternatively, Aghasi et al.

Original: (2016) propose retraining the network layer by layer, keeping the layer inputs and outputs close to the originally trained model, while seeking a sparse transform matrix, whereas Gong et al.
Revised: (2016) propose retraining the network layer by layer, keeping the layer inputs and outputs close to the originally trained model, while seeking a sparse transform matrix, whereas Gong et al.

Original: (2014) propose the use of vector quantization to compress the parameters of DNNs.
Revised: (2014) propose the use of vector quantization to compress the parameters of DNNs.

Original: Network pruning is another relevant line of work.
Revised: Network pruning is another relevant line of work.

Original: Early work by LeCun et al.
Revised: Early work by LeCun et al.

Original: and
Revised: and

Original: Hassibi et al.
Revised: Hassibi et al.

Original: (1993) uses the information provided by Hessian of the loss function to remove less important weights of the network, which requires expensive computation of second order derivatives.
Revised: (1993) uses the information provided by Hessian of the loss function to remove less important weights of the network, which requires expensive computation of second order derivatives.

Original: Recently,
Revised: Recently,

Original: Han et al.
Revised: Han et al.

Original: (2016) reduce the number of parameters by an order of magnitude using an iterative algorithm that alternates between learning the parameters and removing those below a certain threshold.
Revised: (2016) reduce the number of parameters by an order of magnitude using an iterative algorithm that alternates between learning the parameters and removing those below a certain threshold.

Original: Li et al.
Revised: Li et al.

Original: (2016) propose to prune ﬁlters, which seeks sparsity with respect to neurons, rather than connectivities; that approach relieves the burden on requiring sparse libraries or special hardware to deploy the network.
Revised: (2016) propose to prune ﬁlters, which seeks sparsity with respect to neurons, rather than connectivities; that approach relieves the burden on requiring sparse libraries or special hardware to deploy the network.

Original: All these methods either require multiple training/retraining iterations or a careful choice of pruning thresholds.
Revised: All these methods either require multiple training/retraining iterations or a careful choice of pruning thresholds.

Original: There is a large body of work on using sparsity-inducing regularization for training DNNs.
Revised: There is a large body of work on using sparsity-inducing regularization for training DNNs.

Original: For example, Collins & Kohli (2014) exploit (cid:96) 1 and (cid:96) 0 regularization to encourage weight sparsity; however, the sparsity level achieved is typically modest, making this approach not competitive in terms of DNN compression.
Revised: For example, Collins & Kohli (2014) exploit (cid:96) 1 and (cid:96) 0 regularization to encourage weight sparsity; however, the sparsity level achieved is typically modest, making this approach not competitive in terms of DNN compression.

Original: Group-Lasso has also been used in training DNNs, which allows seeking sparsity in terms of neurons (Scardapane et al., 2017; Alvarez & Salzmann, 2016;
Revised: Group-Lasso has also been used in training DNNs, which allows seeking sparsity in terms of neurons (Scardapane et al., 2017; Alvarez & Salzmann, 2016;

Original: Zhou et al., 2016; Murray & Chiang, 2015) or other structures, e.g. , ﬁlters, channels, ﬁlter shapes, and layer depth (Wen et al., 2016).
Revised: Zhou et al., 2016; Murray & Chiang, 2015) or other structures, e.g. , ﬁlters, channels, ﬁlter shapes, and layer depth (Wen et al., 2016).

Original: However, as mentioned above, both Lasso and group-Lasso can fail at selecting important features if they are strongly correlated with each other.
Revised: However, as mentioned above, both Lasso and group-Lasso can fail to select important features if they are strongly correlated with each other.

Original: In Section 4 and Appendix B, we will further illustrate this fact on both synthetic and real data.
Revised: In Section 4, we will further illustrate this fact on both synthetic and real data.

Original: A recent stream of work has focused on using further parameter sharing in convolutional DNNs.
Revised: A recent stream of work has focused on using further parameter sharing in convolutional DNNs.

Original: By tying weights in an appropriate way, Dieleman et al.
Revised: By tying weights in an appropriate way, Dieleman et al.

Original: (2016) obtain a convolutional DNN with rotation invariance.
Revised: (2016) obtain a convolutional DNN with rotation invariance.

Original: On the task of analyzing positions in the game Go , Clark & Storkey (2015) showed improved performance by constraining features to be invariant to reﬂections along the x-axis, y-axis, and diagonal-axis.
Revised: On the task of analyzing positions in the game Go , Clark & Storkey (2015) showed improved performance by constraining features to be invariant to reﬂections along the x-axis, y-axis, and diagonal-axis.

Original: Finally, Chen et al.
Revised: Finally, Chen et al.

Original: (2015) used a hash function to randomly group the weights such that those in a hash bucket share the same value.
Revised: (2015) used a hash function to randomly group the weights such that those in a hash bucket share the same value.

Original: In contrast, with GrOWL regularization, we aim to learn weight sharing from the data itself, rather than specifying it a priori .
Revised: In contrast, with GrOWL regularization, we aim to learn weight sharing from the data itself, rather than specifying it a priori .

Original: Dropout -type methods have been proposed to ﬁght over-ﬁtting and are very popular, arguably due to their simplicity of implementation.
Revised: Dropout -type methods have been proposed to ﬁght over-ﬁtting and are very popular, arguably due to their simplicity of implementation.

Original: For each training example, either activations (Srivastava et al., 2014) or connections (Wan et al., 2013) are randomly dropped and the error is then back-propagated only through the remaining ones.

Original: It has been shown that dropout effectively reduces over-ﬁtting and prevents different neurons from co-adapting.
Revised: It has been shown that dropout effectively reduces over-ﬁtting and prevents different neurons from co-adapting.

Original: Decorrelation is another popular technique in deep learning pipelines (Bengio & Bergstra, 2009; Cogswell et al., 2015; Rodr´ıguez et al., 2016).
Revised: Decorrelation is another popular technique in deep learning pipelines (Bengio & Bergstra, 2009; Cogswell et al., 2015; Rodr ´ ıguez et al., 2016).

Original: Unlike sparsity-inducing regularizers, decorrelation methods try to make full use of the model’s capacity by decorrelating the neurons.
Revised: Unlike sparsity-inducing regularizers, decorrelation methods try to make full use of the model’s capacity by decorrelating the neurons.

Original: Although dropout and decorrelation can reduce over-ﬁtting, they do not compress the network, hence do not address the issue of high memory cost.
Revised: Although dropout and decorrelation can reduce over-ﬁtting, they do not compress the network, hence do not address the issue of high memory cost.

Original: It should also be mentioned that our proposal can be seen as complementary to dropout and decorrelation, in the following sense: whereas dropout and decorrelation can reduce co-adaption of nodes during training, GrOWL regularization copes with co-adaptation by tying together the weights associated to co-adapted nodes.
Revised: It should also be mentioned that our proposal can be seen as complementary to dropout and decorrelation, in the following sense: whereas dropout and decorrelation can reduce co-adaption of nodes during training, GrOWL regularization copes with co-adaptation by tying together the weights associated to co-adapted nodes.

Original:  G ROUP -OWL R EGULARIZATION FOR D EEP L EARNING 3. T HE G ROUP -OWL N ORM
Revised:  G ROUP -OWL R EGULARIZATION FOR D EEP L EARNING 3. T HE G ROUP -OWL N ORM

Original: We start by recalling the deﬁnition of the group-OWL (GrOWL) regularizer and very brieﬂy reviewing some of its relevant properties (Oswal et al., 2016).
Revised: We start by recalling the deﬁnition of the group-OWL (GrOWL) regularizer and very brieﬂy reviewing some of its relevant properties (Oswal et al., 2016).

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: Given a matrix W ∈ R n × m , let(cid:107) w(cid:107) [ i ] · denote the row of W with the i -th largest (cid:96)norm.
Revised: Given a matrix W ∈ R n × m , let w [ i ] · denote the row of W with the i -th largest (cid:96)norm.

Original: Let λ ∈ R n + , with 0 < λ 1 ≥ λ 2 ≥ · · · ≥ λ n ≥ 0 .
Revised: Let λ ∈ R n + , with 0 < λ 1 ≥ λ 2 ≥ · · · ≥ λ n ≥ 0 .

Original: The GrOWL regularizer (which is a norm) Ω λ : R n × m → R is deﬁned as
Revised: The GrOWL regularizer (which is a norm) Ω λ : R n × m → R is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: This is a group version of the OWL regularizer (Figueiredo & Nowak, 2016), also known as WSL ( weighted sorted (cid:96) 

Revised: This is a group version of the OWL regularizer (Figueiredo & Nowak, 2016), also known as WSL ( weighted sorted (cid:96)

Original: (Zeng & Figueiredo, 2014)) and SLOPE (Bogdan et al., 2015), where the groups are the rows of its matrix argument.
Revised: (Zeng & Figueiredo, 2014)) and SLOPE (Bogdan et al., 2015), where the groups are the rows of its matrix argument.

Original: It is clear that GrOWL includes group-Lasso as a special case when λ 1 = λ n .
Revised: It is clear that GrOWL includes group-Lasso as a special case when λ 1 = λ n .

Original: As a regularizer for multiple/multi-task linear regression, each row of W contains the regression coefﬁcients of a given feature, for the m tasks.
Revised: As a regularizer for multiple/multi-task linear regression, each row of W contains the regression coefﬁcients of a given feature, for the m tasks.

Original: It has been shown that by adding the
Revised: It has been shown that by adding the

Original: GrOWL regularizer to a standard squared error loss function, the resulting estimate of W has the following property: rows associated with highly correlated covariates are very close or even exactly equal to each other (Oswal et al., 2016).
Revised: GrOWL regularizer to a standard squared error loss function, the resulting estimate of W has the following property: rows associated with highly correlated covariates are very close or even exactly equal to each other (Oswal et al., 2016).

Original: In the linear case, GrOWL encourages correlated features to form predictive clusters corresponding to the groups of rows that are nearly or exactly equal.
Revised: In the linear case, GrOWL encourages correlated features to form predictive clusters corresponding to the groups of rows that are nearly or exactly equal.

Original: The rationale underlying this paper is that when used as a regularizer for DNN learning, GrOWL will induce both sparsity and parameters tying, as illustrated in Fig.
Revised: The rationale underlying this paper is that when used as a regularizer for DNN learning, GrOWL will induce both sparsity and parameters tying, as illustrated in Fig.

Original: 2 and explained below in greater detail.
Revised: 2 and explained below in greater detail.

Original: 3. L AYER -W ISE G R OWL R EGULARIZATION F OR F EEDFORWARD N EURAL N ETWORKS
Revised: 3. L AYER -W ISE G R OWL R EGULARIZATION F OR F EEDFORWARD N EURAL N ETWORKS

Original: A typical feed-forward DNN with L layers can be treated as a function f of the following form:f ( x, θ ) ≡ h L = f L ( h L [Equation] W L + b L ) ,
Revised: A typical feed-forward DNN with L layers can be treated as a function f of the following form:

Original: [Equation]
Revised: [Equation]

Original: − 1 ( h L − 2 W L − 1 + b L − 1 ) , ..., h 1 = f 1 ( xW 1 + b 1 ) θ = (cid:0) W 1 , b 1 , . .

Original: , W L , b L (cid:1) denotes the set of parameters of the network, and each f i is a componentwise nonlinear activation function, with the rectiﬁed linear unit (ReLU), the sigmoid , and the hyperbolic tangent being common choices for this function (Goodfellow et al., 2016).
Revised: denotes the set of parameters of the network, and each f i is a componentwise nonlinear activation function, with the rectiﬁed linear unit (ReLU), the sigmoid , and the hyperbolic tangent being common choices for this function (Goodfellow et al., 2016).

Original: Given labelled data D = (cid:0) ( x (1) , y (1) ) , ..., ( x ( m ) , y ( m ) ) (cid:1) , DNN learning may be formalized as an optimization problem,
Revised: Given labelled data D = (cid:0) ( x (1) , y (1) ) , ..., ( x ( m ) , y ( m ) ) (cid:1) , DNN learning may be formalized as an optimization problem,

Original: [Equation]
Revised: [Equation]

Original: , (2) and L (cid:0) y, ˆ y (cid:1) is the loss incurred when the DNN predicts ˆ y for y , and R is a regularizer.
Revised: where [Equation] [Equation] and L (cid:0) y, ˆ y (cid:1) is the loss incurred when the DNN predicts ˆ y for y , and R is a regularizer.

Original: The regularizer herein adopted is a sum of GrOWL penalties, each applied to each layer of the neural network, i.e.,
Revised: The regularizer herein adopted is a sum of GrOWL penalties, each applied to each layer of the neural network, i.e.,

Original: [Equation]
Revised: [Equation]

Original: where N l denotes the number of neurons in the l -th layer and 0 < λ ( l ) 1 ≥ λ ( l ) 2 ≥ · · · ≥ λ ( l ) N l − 1 ≥ 0 .
Revised: where N l denotes the number of neurons in the l -th layer and 0 < λ ( l ) 1 ≥ λ ( l ) 2 ≥ · · · ≥ λ ( l ) N l − 1 ≥ 0 .

Original: Notice that R ( θ ) does not depend on b 1 , ..., b L , which means these parameters are not regularized, as is common practice.
Revised: Notice that R ( θ ) does not depend on b 1 , ..., b L , which means these parameters are not regularized, as is common practice.

Original: As indicated in Eq.
Revised: As indicated in Eq.

Original: (3), the number of groups in each GrOWL regularizer equals the number of neurons in the previous layer, i.e., λ ( l ) ∈ R N l − 1 .
Revised: (3), the number of groups in each GrOWL regularizer equals the number of neurons in the previous layer, i.e., λ ( l ) ∈ R N l − 1 .

Original: In other words, we treat the weights associated with each input feature as a group .
Revised: In other words, we treat the weights associated with each input feature as a group .

Original: For fully connected layers, where W l ∈ R N l − 1 × N l , each group is a row of the weight matrix.
Revised: For fully connected layers, where W l ∈ R N l − 1 × N l , each group is a row of the weight matrix.

Original: When considering convolutional layers, where W l ∈ R F w × F h × N l − 1 × N l , with F w and F h denoting the width and height, respectively, of each ﬁlter, we ﬁrst reshape W l to a 2-dimensional array, i.e., W l → W 2 D l , where W 2 D l ∈ R N l [Equation] 1 × ( F w F h N l ) , and then apply GrOWL on the reshaped matrix.
Revised: When considering convolutional layers, where W l ∈ R F w × F h × N l − 1 × N l , with F w and F h denoting the width and height, respectively, of each ﬁlter, we ﬁrst reshape W l to a 2-dimensional array, i.e., W l → W 2 D l , where W 2 D l ∈ R N l − 1 × ( F w F h N l ) , and then apply GrOWL on the reshaped matrix.

Original: That is, if the l -th layer is convolutional, then
Revised: That is, if the l -th layer is convolutional, then

Original: [Equation]
Revised: [Equation]

Original: Each row of W 2 D l represents the operation on an input channel.
Revised: Each row of W 2 D l represents the operation on an input channel.

Original: The rationale to apply the GrOWL regularizer to each row of the reshaped weight matrix is that GrOWL can select the relevant features of the network, while encouraging the coefﬁcient rows of each layer associated with strongly correlated features from the previous layer to be nearly or exactly equal, as depicted in Fig.
Revised: The rationale to apply the GrOWL regularizer to each row of the reshaped weight matrix is that GrOWL can select the relevant features of the network, while encouraging the coefﬁcient rows of each layer associated with strongly correlated features from the previous layer to be nearly or exactly equal, as depicted in Fig.

Original: The goal is to signiﬁcantly reduce the complexity by: (i) pruning unimportant neurons of the previous layer that correspond to zero rows of the (reshaped) weight matrix of the current layer; (ii) grouping the rows associated with highly correlated features of the previous layer, thus encouraging the coefﬁcient rows in each of these groups to be very close to each other.
Revised: The goal is to signiﬁcantly reduce the complexity by: (i) pruning unimportant neurons of the previous layer that correspond to zero rows of the (reshaped) weight matrix of the current layer; (ii) grouping the rows associated with highly correlated features of the previous layer, thus encouraging the coefﬁcient rows in each of these groups to be very close to each other.

Original: As a consequence, in the retraining process, we could further compress the neural network by enforcing the parameters within each neuron that belong to the same cluster to share same values.
Revised: As a consequence, in the retraining process, we could further compress the neural network by enforcing the parameters within each neuron that belong to the same cluster to share same values.

Original: In the work of Alvarez & Salzmann (2016), each group is predeﬁned as the set of parameters associated to a neuron, and group-Lasso regularization is applied to seek group sparsity, which corresponds to zeroing out redundant neurons of each layer.
Revised: In the work of Alvarez & Salzmann (2016), each group is predeﬁned as the set of parameters associated to a neuron, and group-Lasso regularization is applied to seek group sparsity, which corresponds to zeroing out redundant neurons of each layer.

Original: In contrast, we treat the ﬁlters corresponding with the same input channel as a group, and GrOWL is applied to prune the redundant groups and thus remove the associated unimportant neurons of previous layer , while grouping associated parameters of the current layer that correspond with highly correlated input features to different clusters.
Revised: In contrast, we treat the ﬁlters corresponding with the same input channel as a group, and GrOWL is applied to prune the redundant groups and thus remove the associated unimportant neurons of previous layer , while grouping associated parameters of the current layer that correspond with highly correlated input features to different clusters.

Original: Moreover, shown in Section 4, group-Lasso can fail at selecting all relevant features of previous layers, and for the selected ones the corresponding coefﬁcient groups are quite dissimilar from each other, making it impossible to further compress the DNN by enforcing parameter tying.
Revised: Moreover, shown in Section 4, group-Lasso can fail at selecting all relevant features of previous layers, and for the selected ones the corresponding coefﬁcient groups are quite dissimilar from each other, making it impossible to further compress the DNN by enforcing parameter tying.

Original: 3. P ROXIMAL G RADIENT A LGORITHM
Revised: 3. P ROXIMAL G RADIENT A LGORITHM

Original: To solve (2), we use a proximal gradient algorithm (Bauschke & Combettes, 2011), which has the following general form: at the t -th iteration, the parameter estimates are updated according to
Revised: To solve (2), we use a proximal gradient algorithm (Bauschke & Combettes, 2011), which has the following general form: at the t -th iteration, the parameter estimates are updated according to

Original: [Equation]
Revised: [Equation]

Original: where, for some convex function Q , prox Q denotes its proximity operator (Bauschke & Combettes, 2011), which is deﬁned as prox Q ( ξ ) =
Revised: where, for some convex function Q , prox Q denotes its proximity operator (Bauschke & Combettes, 2011), which is deﬁned as prox Q ( ξ ) =

Original: arg min θ Q ( ν ) + 12 (cid:107) ν − ξ (cid:107) 22 .
Revised: arg min ν Q ( ν ) + 1 2 (cid:107) ν − ξ (cid:107) 22 .

Original: In the previous expression, (cid:107) ν − ξ (cid:107) 22 denotes the sum of the squares of the differences between the corresponding components of ν and ξ , regardless of their organization (in our case, a collection of matrices and vectors).
Revised: In the previous expression, (cid:107) ν − ξ (cid:107) 22 denotes the sum of the squares of the differences between the corresponding components of ν and ξ , regardless of their organization (in our case, a collection of matrices and vectors).

Original: Since R ( θ ) , as deﬁned in (3), is separable across the weight matrices of different layers and zero for b 1 , ..., b L , the corresponding proximity operator is also separable, thus
Revised: Since R ( θ ) , as deﬁned in (3), is separable across the weight matrices of different layers and zero for b 1 , ..., b L , the corresponding proximity operator is also separable, thus

Original: [Equation]
Revised: [Equation]

Original: It was proved by Oswal et al.
Revised: It was proved by Oswal et al.

Original: (2016) that the proximity operator of the GrOWL norm can be computed as follows.
Revised: (2016) that the proximity operator of the GrOWL norm can be computed as follows.

Original: For some matrix V ∈ R N × M , let U =
Revised: For some matrix V ∈ R N × M , let U =

Original: prox Ω λ ( V ) , and v i and u
Revised: prox Ω λ ( V ) , and v i and u

Original: i denote the corresponding i -th rows.
Revised: i denote the corresponding i -th rows.

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: where ˜ v =
Revised: where ˜ v =

Original: [ (cid:107) v 1 (cid:107) , (cid:107) v 2 (cid:107) , · · · , (cid:107) v N (cid:107) ] .
Revised: [ (cid:107) v 1 (cid:107) , (cid:107) v 2 (cid:107) , · · · , (cid:107) v N (cid:107) ] .

Original: The proximal operator prox Ω λ ( l ) for a vector in R N (in which case GrOWL coincides with OWL) can be computed with O ( n log n ) complexity, where the core computation is the so-called pool adjacent violators algorithm (PAVA (de Leeuw et al., 2009)) for isotonic regression.
Revised: The proximal operator prox Ω λ ( l ) for a vector in R N (in which case GrOWL coincides with OWL) can be computed with O ( n log n ) complexity, where the core computation is the so-called pool adjacent violators algorithm (PAVA (de Leeuw et al., 2009)) for isotonic regression.

Original: We provide one of the existing algorithms in Appendix A; for details of the algorithm, the reader is referred to the work of Bogdan et al.
Revised: We provide one of the existing algorithms in Appendix A; for details of the algorithm, the reader is referred to the work of Bogdan et al.

Original: and Zeng & Figueiredo (2015).
Revised: and Zeng & Figueiredo (2015).

Original: In this paper, we apply the proximal gradient descent per epoch, which generally yields better performance.
Revised: In this paper, we apply the proximal gradient descent per epoch, which generally yields better performance.

Original: The training method is summarized in Algorithm 1.
Revised: The training method is summarized in Algorithm 1.

Original: Algorithm
Revised: Algorithm

Original: Input : parameters of the OWL regularizers λ ( l ) , ..., λ ( L ) , learning rate η for each epoch T do for each iteration t in epoch T do
Revised: Input : parameters of the OWL regularizers λ ( l ) , ..., λ ( L ) , learning rate η for each epoch T do for each iteration t in epoch T do

Original: Update the parameter via backpropagation (BP) end for
Revised: Update the parameter via backpropagation (BP) end for

Original: Apply proximity operator via (6) end for
Revised: Apply proximity operator via (6) end for

Original: 3. I MPLEMENTATION D ETAILS 3.4. S ETTING THE G R OWL W EIGHTS
Revised: 3. I MPLEMENTATION D ETAILS 3.4. S ETTING THE G R OWL W EIGHTS

Original: GrOWL is a family of regularizers, with different variants obtained by choosing different weight sequences λ 1 , . . .
Revised: GrOWL is a family of regularizers, with different variants obtained by choosing different weight sequences λ 1 , . . .

Original: , λ n .
Revised: , λ n .

Original: In this paper, we propose the following choice:
Revised: In this paper, we propose the following choice:

Original: [Equation]
Revised: [Equation]

Original: where p ∈ { 1 , ...n } is a parameter.
Revised: where p ∈ { 1 , ...n } is a parameter.

Original: The ﬁrst p weights follow a linear decay, while the remaining ones are all equal to Λ 1 .
Revised: The ﬁrst p weights follow a linear decay, while the remaining ones are all equal to Λ 1 .

Original: Notice that, if p = n , the above setting is equivalent to OSCAR (Bondell & Reich, 2008).
Revised: Notice that, if p = n , the above setting is equivalent to OSCAR (Bondell & Reich, 2008).

Original: Roughly speaking, Λ 1 controls the sparsifying strength of the regularizer, while Λcontrols the clustering property (correlation identiﬁcation ability) of GrOWL (Oswal et al., 2016).
Revised: Roughly speaking, Λ 1 controls the sparsifying strength of the regularizer, while Λcontrols the clustering property (correlation identiﬁcation ability) of GrOWL (Oswal et al., 2016).

Original: Moreover, by setting the weights to a common constant beyond index p means that clustering is only encouraged among the p largest coefﬁcients, i.e., only among relevant coefﬁcient groups.
Revised: Moreover, by setting the weights to a common constant beyond index p means that clustering is only encouraged among the p largest coefﬁcients, i.e., only among relevant coefﬁcient groups.

Original: Finding adequate choices for p , Λ 1 , and Λ 2 is crucial for jointly selecting the relevant features of the network and identifying the underlying correlations among them.
Revised: Finding adequate choices for p , Λ 1 , and Λ 2 is crucial for jointly selecting the relevant features of the network and identifying the underlying correlations among them.

Original: In practice, we ﬁnd that with properly chosen p , GrOWL is able to ﬁnd more correlations of the underlying data than OSCAR.
Revised: In practice, we ﬁnd that with properly chosen p , GrOWL is able to ﬁnd more correlations of the underlying data than OSCAR.

Original: We explore different choices of p in Appendix B.
Revised: We explore different choices of p in Section 4.1.

Original: 3.4. P ARAMETER T YING
Revised: 3.4. P ARAMETER T YING

Original: After the initial training phase, at each layer l , rows of W l that corresponds to highly correlated outputs of layer l − 1 have been made similar or even exactly equal.
Revised: After the initial training phase, at each layer l , rows of W l that corresponds to highly correlated outputs of layer l − 1 have been made similar or even exactly equal.

Original: To further compress the DNN, we force rows that are close to each other to be identical.
Revised: To further compress the DNN, we force rows that are close to each other to be identical.

Original: We ﬁrst group the rows into different clusters 1 according to the pairwise similarity metric
Revised: We ﬁrst group the rows into different clusters 1 according to the pairwise similarity metric

Original: [Equation]
Revised: [Equation]

Original: where W l,i and W l,j denote the i -th and j -th rows of W l , respectively.
Revised: where W l,i and W l,j denote the i -th and j -th rows of W l , respectively.

Original: With the cluster information obtained by using GrOWL, we enforce parameter sharing for the rows that belong to a same cluster by replacing their values with the averages (centroid) of the rows in that cluster.
Revised: With the cluster information obtained by using GrOWL, we enforce parameter sharing for the rows that belong to a same cluster by replacing their values with the averages (centroid) of the rows in that cluster.

Original: In the following retraining process , let G ( l ) k denote the k -th cluster of the l -th layer, then centroid g ( l ) k of this cluster is updated via
Revised: In the following retraining process , let G ( l ) k denote the k -th cluster of the l -th layer, then centroid g ( l ) k of this cluster is updated via

Original: [Equation]
Revised: [Equation]

Original:  N UMERICAL R ESULTS
Revised:  N UMERICAL R ESULTS

Original: We assess the performance of the proposed method on two benchmark datasets: MNIST and CIFAR10.
Revised: We assess the performance of the proposed method on two benchmark datasets: MNIST and CIFAR10.

Original: We consider two different networks and compare GrOWL with group-Lasso and weight decay, in terms of the compression vs accuracy trade-off.
Revised: We consider two different networks and compare GrOWL with group-Lasso and weight decay, in terms of the compression vs accuracy trade-off.

Original: For fair comparison, the training-retraining pipeline is used with the different regularizers.
Revised: For fair comparison, the training-retraining pipeline is used with the different regularizers.

Original: After the initial training phase, the rows that are close to each other are clustered together and forced to share common values in the retraining phase.
Revised: After the initial training phase, the rows that are close to each other are clustered together and forced to share common values in the retraining phase.

Original:  To evaluate the effect of the different regularizers, we use the following quantities: sparsity = # zero 
Revised: We implement all models using Tensorﬂow Abadi et al. To evaluate the effect of the different regularizers, we use the following quantities: sparsity = # zero params # total params ; compression rate = # total params # unique params ; parameter sharing = # nonzero params # unique params . 4. D IFFERENT C HOICES OF G R OWL P ARAMETERS

Original: [Equation]
Revised: We ﬁrst evaluate GrOWL norm regularization on synthetic data. The covariance matrix Σ of the data X is set as a block diagonal matrix such that each block represents the covariances of a cluster of correlated features, and there is a gap g between two blocks. Within each cluster, the covariance between two features X i and X j is cov ( X i , X j ) = 0 . 96 | i − j | .

Revised: we assume that the features from different clusters are independent of each other, while In this section, we set n = 784 , K = 10 and block size equals 50 , g = 28 .

Revised: We generate 10000 training examples and 1000 testing examples.

Revised: We train a neural network with a single fully connected layer that consists of 300 hidden units.

Revised: In Fig 3, we plot the the ﬁrst 25000 entries of the sorted similarity matrices obtained by applying GrOWL with different p (Eq 9) values.

Revised: Although, by setting the weights to a common constant beyond index p means that clustering is only encouraged among the p largest coefﬁcients, i.e., only among relevant coefﬁcient groups, Figure 3 shows that, with probably chosen p , GrOWL is able to encourage more parameter tying than OSCAR ( p = n ).

Revised: On the other hand, choosing smaller p values allows us to use large Λ 2 , which encourages parameter tying among comparatively loose correlations.

Revised: In practice, we found that choosing p around the target portion of nonzero parameters over total parameters yields better performance in general.

Revised: The intuition is that we only want to identify correlations among the remaining important parameters/connections.

Revised: Fig 3 shows that weight decay (denoted as (cid:96) 2 ) can also push parameters together, though the parameter tying effect is not as obvious as that obtained by applying GrOWL.

Revised: As has been observed in the literature (Bondell & Reich, 2008), weight decay often achieves better generalization as compared with sparsity-inducing regularizers.

Revised: It achieves this via parameter shrinkage, especially in the highly correlated region, but it cannot yield a sparse model.

Revised: In the following section, we explore the compression performance of GrOWL by comparing it with both group-Lasso and weight decay.

Revised: For each case, the baseline performance is provided as the best performance obtained by running the original neural network (without compression) after sweeping the hyper-parameter on the weight decay regularizer over a range of values.

Original: 4. F ULLY C ONNECTED N EURAL N ETWORK ON MNIST
Revised: 4. F ULLY C ONNECTED N EURAL N ETWORK ON MNIST

Original: The MNIST dataset contains 28 × 28 (784) pixels images of handwritten digits (0, 1, ..., 9), each located at the center of the image.
Revised: The MNIST dataset contains 28 × 28 (784) pixels images of handwritten digits (0, 1, ..., 9), each located at the center of the image.

Original: Fig 3 (a) shows the ( 784 × 784 ) correlation matrix of the dataset (the margins are zero due to the redundant background of the images).
Revised: Fig 4 (a) shows the ( 784 × 784 ) correlation matrix of the dataset (the margins are zero due to the redundant background of the images).

Original: We use a network with a single fully connected layer of 300 hidden units.
Revised: We use a network with a single fully connected layer of 300 hidden units.

Original: The network is trained for 200 epochs and then retrained for an additional 100 epochs, both with momentum.
Revised: The network is trained for 300 epochs and then retrained for an additional 100 epochs, both with momentum.

Original: The initial learning rate is set to 10 −for both the training and retraining phases, and it is reduced by a factor of 0.96 every 10 epochs.
Revised: The initial learning rate is set to 0. for both the training and retraining phases, and it is reduced by a factor of 0.96 every 10 epochs.

Original: The hyper-parameters are selected through grid search
Revised: The hyper-parameters are selected through grid search.

Original: The pairwise similarities (see Eq.
Revised: Pairwise similarities (see Eq.

Original: (10)) between the rows of the weight matrices learned withthe different regularizerstested are shown in Fig.
Revised: (10)) between the rows of the weight matrices learned with different regularizers are shown in Fig.

Original: 3 (b,c,d).
Revised: 4 (b–f).

Original: For the same level of sparsity ( ∼ 47%), GrOWL  identiﬁes more correlations than group-Lasso, and the similarity pattern in Fig.
Revised: As we can see, GrOWL (+ (cid:96) 2 ) identiﬁes more correlations than group-Lasso (+ (cid:96) 2 ), and the similarity patterns in Fig.

Original: 3 (b) is very close to that of the data (Fig. 3(a)).
Revised: 4 (b, c) are very close to that of the data (Fig. 4(a)).

Original: On the other hand, weight decay also identiﬁes correlations between parameter rows, but it does not induce a sparse model.
Revised: On the other hand, weight decay also identiﬁes correlations between parameter rows, but it does not induce a sparse model.

Original: Moreover, as shown in Table 1, GrOWL obtains a higher level of parameter sharing than  weight decay.
Revised: Moreover, as shown in Table 1, GrOWL obtains a higher level of parameter sharing than does weight decay.

Original: The compression vs accuracy trade-off of the different regularizers is summarized in Table 1, which shows that GrOWL achieves the highest compression and accuracy. More speciﬁcally, when
Revised: The compression vs accuracy trade-off of the different regularizers is summarized in Table 1.

Original: GrOWL and group-Lasso (1) achieve the same sparsity level after the initial training phase, GrOWL  compresses the network∼ 3 times more thangroup-Lasso, due to the signiﬁcant amount of correlation it identiﬁes. On the other hand, when group-Lasso (2) reaches the same level of compression as
Revised: As we can see, by applying the (cid:96) 2 norm regularizer together with group-Lasso or GrOWL, we achieve a higher compression rate while yielding even better accuracy. Table 1 also shows that, with even smaller sparsities after the initial training phase, GrOWL ( + (cid:96) 2 ) compresses the network more than group-Lasso ( + (cid:96) 2 )

Original: GrOWL, i.e., the same amount of free parameters, its accuracy is about 1.6% lower than GrOWL.
Revised: does, due to the signiﬁcant amount of correlation it identiﬁes. This also implies that group-Lasso can only select a subset of the correlated features while GrOWL selects all of them.

Original: A possible explanation is that group-Lasso suffers from randomly selecting a subset of correlated features.
Revised: On the other hand, group-Lasso suffers from randomly selecting a subset of correlated features.

Original: We illustrate this effect in Fig. 4, which plots the indices of nonzero rows, showing that GrOWL stably selects relevant features while group-Lasso does not.
Revised: We illustrate this effect in Fig. 5, which plots the indices of nonzero rows, showing that GrOWL stably selects relevant features while group-Lasso does not.

Original: The mean ratios of changed in- dices 2 are 39.75% and 1.56% for group-Lasso and GrOWL, respectively.
Revised: The mean ratios of changed indices 2 are 11.09% and 0.58% for group-Lasso and GrOWL, respectively.

Original: This suggests, combined with Table 1, that stably selecting all of the relevant features may be important for achieving a high generalization performance.
Revised: parameter sharing for baseline model. We train each model for 5 times and report the average values together with their standard deviations [Table] [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: 4. VGG-16 ON CIFAR- 
Revised: 4. VGG-16 ON CIFAR-

Original: To evaluate the proposed method on large DNNs [Table] we consider a VGG-like (Simonyan & Zisserman, 2014) architecture proposed by Sergey (2015) on the CIFAR-10 dataset.
Revised: To evaluate the proposed method on large DNNs, we consider a VGG-like (Simonyan & Zisserman, 2014) architecture proposed by Zagoruyko (2015) on the CIFAR-10 dataset.

Original: The network is summarized in Table 2; comparing with the original VGG of Simonyan & Zisserman (2014), their fully connected layers are replaced with two small ones with 14.3% of the total parameters.
Revised: The network is summarized in Table 3; comparing with the original VGG of Simonyan & Zisserman (2014), their fully connected layers are replaced with two small ones with 14.3% of the total parameters.

Original: A batch normalization layer is added after each convolutional layer and the ﬁrst fully connected layer.
Revised: A batch normalization layer is added after each convolutional layer and the ﬁrst fully connected layer.

Original: Unlike Sergey (2015) [Table] we don’t use dropout.
Revised: Unlike Zagoruyko (2015), we don’t use dropout.

Original: We ﬁrst train under different regularizers over 160 epochs, then retrain for another 40 epochs, using the learning rate decay scheme described by He et al.
Revised: We ﬁrst train under different regularizers overepochs, then retrain for another 50 epochs, using the learning rate decay scheme described by He et al.

Original: (2016): the initial rates for the training and retraining phases are set to 0 .
Revised: (2016): the initial rates for the training and retraining phases are set to 0 .

Original: 01 and 0 .
Revised: 01 and 0 .

Original: 001 , respectively; the learning rate is multiplied by 0 .
Revised: 001 , respectively; the learning rate is multiplied by 0 .

Original: 1 every 60 epochs of the training phase, and every 20 epochs of the retraining phase.
Revised: 1 every 60 epochs of the training phase, and every 20 epochs of the retraining phase.

Original: For GrOWL, we set p = 0 .
Revised: For GrOWL ( + (cid:96) 2 ), we set p = 0 .

Original: 1 n (see Eq.
Revised: 1 n (see Eq.

Original: (9)) for all layers, where n denotes the number of rows of each layer.
Revised: (9)) for all layers, where n denotes the number of rows of each layer.

Revised: Layers Weight Decay

Revised: [Table]

Original: The results are summarized in Table 2.
Revised: The results are summarized in Table 2.

Original: In the experiments, we found that it is quite hard to encourage parameter tying in the ﬁrst 7 convolutional layers.
Revised: In the experiments, we found that it is quite hard to encourage parameter tying in the ﬁrst 7 convolutional layers.

Original: The reason may be that the ﬁlters of these ﬁrst 7 convolutional layers have comparatively large feature maps (from 32 × 32 to 8 × 8 ), which are only loosely correlated.
Revised: The reason may be that the ﬁlters of these ﬁrst 7 convolutional layers have comparatively large feature maps (from 32 × 32 to 8 × 8 ), which are only loosely correlated.

Original: We illustrate this reasoning in Fig. 5, showing the cosine similarity between the vectorized output channels of layers 1, 6, 10, and 11, at the end of the training phase; it can be seen that the outputs of layers 10 and 11 have many more signiﬁcant similarities than that of layer 6.
Revised: We illustrate this reasoning in Fig. 6, showing the cosine similarity between the vectorized output channels of layers 1, 6, 10, and 11, at the end of the training phase; it can be seen that the outputs of layers 10 and 11 have many more signiﬁcant similarities than that of layer 6.

Original: Although the output channels of layer 1 also have certain similarities, as seen in Table 2, neither
Revised: Although the output channels of layer 1 also have certain similarities, as seen in Table 2, neither

Original: GrOWL  nor weight decay tends to tie the associated weights.
Revised: GrOWL ( + (cid:96) 2 ) nor weight decay tends to tie the associated weights.

Original: This may mean that the network is maintaining the diversity of the inputs in the ﬁrst few convolutional layers.
Revised: This may mean that the network is maintaining the diversity of the inputs in the ﬁrst few convolutional layers.

Original: [Table] Although GrOWL and weight decay both encourage parameter tying in layers 8- [Table] weight decay does it with less intensity and does not yield a sparse model
Revised: Although GrOWL and weight decay both encourage parameter tying in layers 9-13, weight decay does it with less intensity and does not yield a sparse model, thus it cannot signiﬁcantly compress the network.

Original: [Table]

Original: thus it cannot signiﬁcantly compress the network.

Original: Li et al.
Revised: Li et al.

Original: (2016) propose to prune small weights after the initial training phase with weight decay, then retrain the reduced network; however, this type of method only achieves roughly 3 X compression 3 .
Revised: (2016) propose to prune small weights after the initial training phase with weight decay, then retrain the reduced network; however, this type of method only achieves roughly 3 X compression 4 .

Original: As mentioned by Li et al.
Revised: As mentioned by Li et al.

Original: (2016), layers 3-7 can be very sensitive to pruning; however, both GrOWL  and group-Lasso  effectively compress them with a minor accuracy loss.
Revised: (2016), layers 3-7 can be very sensitive to pruning; however, both GrOWL ( + (cid:96) 2 ) and group-Lasso ( + (cid:96) 2 ) effectively compress them with a minor accuracy loss.

Original: Compared with group-Lasso, GrOWL can further compress the  network  by  2 . 5 X , due to its ability to identify the correlations and tie the corresponding parameters.
Revised: On the other hand, similar to what we observed by running the simple fully connected neural network on MNIST, the accuracy and memory trade-off imrpoves signiﬁcantly by applying GrOWL or group-Lasso together with (cid:96) 2 . However, Table 2 also shows that the accuracy vs. memory trade-off achieved by GrOWL ( + (cid:96) 2 ) and group-Lasso ( + (cid:96) 2 ) are almost the same.

Original: The CIFAR-10 dataset is simple enough, so that one could still expect a good performance after strong network compression.
Revised: We suspect that this is caused by the fact that the CIFAR-10 dataset is simple enough that one could still expect a good performance after strong network compression.

Original: We believe this gap in the compression vs accuracy trade-off can be further increased in larger networks on more complex datasets.
Revised: We believe this gap in the compression vs accuracy trade-off can be further increased in larger networks on more complex datasets.

Original: We leave this question for future work.
Revised: We leave this question for future work.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have proposed using the recent GrOWL regularizer for simultaneous parameter sparsity and tying in DNN learning.
Revised: We have proposed using the recent GrOWL regularizer for simultaneous parameter sparsity and tying in DNN learning.

Original: By leveraging on GrOWL’s capability of simultaneously pruning redundant parameters and tying parameters associated with highly correlated features, we can signiﬁcantly reduce the model complexity, with a slight or even no loss in generalization accuracy.
Revised: By leveraging on GrOWL’s capability of simultaneously pruning redundant parameters and tying parameters associated with highly correlated features, we can signiﬁcantly reduce the model complexity, with a slight or even no loss in generalization accuracy.

Original: We evaluate the proposed method on both a fully connected neural network and a deep convolutional neural network.
Revised: We evaluate the proposed method on both a fully connected neural network and a deep convolutional neural network.

Original: The results show that GrOWL can compress large DNNs by a factor of more than 13 .
Revised: The results show that GrOWL can compress large DNNs by a factor of 11 .

Original:  5 , outperforming group-Lasso in terms of the accuracy vs memory trade-off.
Revised: 4 to 14 . 5 .

Original: The correlation patterns identiﬁed by GrOWL are close to those of the input features to each layer.
Revised: The correlation patterns identiﬁed by GrOWL are close to those of the input features to each layer.

Original: This may be important to reveal the structure of the features, contributing to the interpretability of deep learning models.
Revised: This may be important to reveal the structure of the features, contributing to the interpretability of deep learning models.

Original: On the other hand, by automatically tying together the parameters corresponding to highly correlated features, GrOWL alleviates the negative effect of strong correlations that might be induced by the noisy input or the co-adaption tendency of deep neural networks.
Revised: On the other hand, by automatically tying together the parameters corresponding to highly correlated features, GrOWL alleviates the negative effect of strong correlations that might be induced by the noisy input or the co-adaption tendency of deep neural networks.

Original: Future work will explore the use of GrOWL in the layers corresponding to large output features, either within each neuron, by predeﬁning each 2D convolutional ﬁlter as a group, or to obtain new sharing structures.
Revised: However, the gap in the accuracy and memory trade-off obtained by applying GrOWL and groupLasso to compress neural networks gets decreased as we move to large DNNs. Although we suspect this can be caused by running a much larger network on a simple dataset, this motivates us to explore different ways to apply GrOWL to compress neural networks.

Original: A PPENDIX A P ROX G R OWL
Revised: One possible method is applying

Original: Various methods have been proposed tocompute the proximal mapping of OWL (ProxOWL) . It has been proven that the computation complexity of these methods is O ( n log n ) which is just slightly worse than the soft thresholding method for solving (cid:96) 1 norm regularization.
Revised: GrOWL within each neuron by predeﬁning each 2D convolutional ﬁlter as a group (instead all 2D convolutional ﬁlters corresponding to the same input features). By doing so, we encourage parameter sharing among much smaller units, which in turn would further improve the diversity vs parameter sharing trade-off.

Original: In this paper, we use
Revised: We leave this as future work.

Original: Algorithm 2 that was originally proposed in Bogdan et al. Algorithm Input : z and λ Let (cid:101) λ
Revised: A PPENDIX B A FFINITY P ROPAGATION

Original: = ηλ and (cid:101) z = | Pz | be a nonincreasing vector, where P is a permutation matrix.
Revised: Afﬁnity Propagation is a clustering method based on sending messages between pairs of data samples.

Original: while (cid:101) z − (cid:101) λ is not nonincreasing: do Identify strictly increasing subsequences, i.e., segments i : j such that [Equation]
Revised: The idea is to use these messages to determine the most representative data samples, which are called exemplars, then create clusters using these exemplars. Provided with the precomputed data similarity s ( i, j ) , i (cid:54) =

Original: [Equation] Replace the values of (cid:101) z and (cid:101) λ over such segments by their average value: for k ∈ { i, i + 1 , · · · , j } (cid:101)
Revised: j and preference s ( i, i ) , there are two types information being sent between samples iteratively: 1) responsibility r ( i, k ) , which measures how likely that sample k should be the exemplar of sample i ; 2) availability a ( k, i ) , which is the evidence that sample i should choose sample k as its exemplar.

Original: z
Revised: The algorithm is described in 3.

Original:  k ← [Equation] i + [Equation]
Revised: Initialization : r ( i, k ) = 0 , a ( k, i ) = 0 for all i, k while not converge do

Original: ←
Revised: Responsibility updates:

Original: [Equation]
Revised: [Equation]

Original: (cid:88)
Revised: Availability updates:

Original: [Equation]
Revised: [Equation]

Original: (13) end while
Revised: end while

Original: Output: (cid:98) z = sign ( z ) P T ( (cid:101) z [Equation] (cid:101)
Revised: Making assignments:

Original: [Equation]
Revised: [Equation]

Original: + .

Original: We ﬁrst evaluate GrOWL norm regularization on synthetic data. The covariance matrix Σ of the data X is set as a block diagonal matrix such that each block represents the covariances of a cluster of correlated features, and there is a gap g between two blocks. Within each cluster, the covariance between two features X i and X j is cov ( X i , X j ) = 0 .
Revised: Unlike k-means or agglomerative algorithm, Afﬁnity Propagation does not require the number of clusters as an input.

Original: 96 | i − j | .

Original: we assume that the features from different clusters are independent of each other, while In this section, we set n = 784 , K = 10 and block size equals 50 , g = 28 .

Original: We generate 10000 training examples and 1000 testing examples.

Original: We train a  neural network with a single fully connected layer that consists of 300 hidden units. In Fig 6 (a) -(c), we visualize the similarities (Eq 10) between rows of the weight matrix learned by different regularizers. As we can see, both Weight Decay and GrOWL can capture the correlations of the data by tying the corresponding rows together.
Revised: We deem this as a desired property for enforcing parameter sharing in neural network compression because it’s impossible to have the exact number of clusters as a prior information.

Original: However, as it’s implied by Fig 6 (a), (c) and (d), in the highly correlated regime of the data, GrOWL ties the associated rows even tighter than weight decay does.

Original: Moreover, GrOWL can capture more correlation information of the data while encouraging sparsity simultaneously.

Original: On the other hand, although Group Lasso can prune away the redundant rows, it fails at selecting all relevant features and for the selected features, the associated rows are quite dissimilar with each other. In contrast, GrOWL can select all of the important relevant features and tying them together. This capability of GrOWL can help interpret the model better while allowing more space for parameter sharing in the retraining phase. In Figure 6, we plot the the ﬁrst 25000 entries of the sorted similarity matrices obtained by applying
Revised: In practice, the input preference of Afﬁnity Propagation determines how likely each sample will be chosen as an exemplar and its value will inﬂuence the number of clusters created.

Original: GrOWL with different p (Eq 9) values. Although, by setting the weights to a common constant beyond index p means that clustering is only encouraged among the p largest coefﬁcients, i.e., only among relevant coefﬁcient groups, Figure 6 shows that, with probably chosen p , GrOWL is able to encourage more parameter tying than OSCAR ( p = n ). This implies that with probably chosen p , GrOWL is able to identify more correlations of the input features than OSCAR.
Revised: [Figure]

Original: A PPENDIX C VGG-16 ON CIFAR- 
Revised: A PPENDIX D VGG-16 ON CIFAR-

Original: In this section, we visualize the similarity pattern between the output channels of each layer. For each layer, we also plot the similarity between the rows of the (reshaped) weight matrices of the next layer. As we can see in Figures 7, 8, and 9 GrOWL and weight decay, the similarity pattern of the output channels of each layer and that of the corresponding weight matrix of the next layer match well with each other; while for Group Lasso, they do not match well with each other.
Revised: [Table] [Table]