Original: A Better Way to Decay: Proximal Gradient Training Algorithms for Neural Nets
Revised: A Better Way to Decay: Proximal Gradient Training Algorithms for Neural Nets

Original: author names withheld

Original: Under Review for OPT

Original: Abstract
Revised: Abstract

Original: Weight decay is one of the most widely used forms of regularization in deep learning, and has been shown to improve generalization and robustness.
Revised: Weight decay is one of the most widely used forms of regularization in deep learning, and has been shown to improve generalization and robustness.

Original: The optimization objective driving weight decay is a sum of losses plus a term proportional to the sum of squared weights.
Revised: The optimization objective driving weight decay is a sum of losses plus a term proportional to the sum of squared weights.

Original: This paper argues that stochastic gradient descent (SGD) may be an inefficient algorithm for this objective.
Revised: This paper argues that stochastic gradient descent (SGD) may be an inefficient algorithm for this objective.

Original: For neural networks with ReLU activations, solutions to the weight decay objective are equivalent to those of a different objective in which the regularization term is instead a sum of products of ℓ 2 (not squared) norms of the input and output weights associated each ReLU.
Revised: For neural networks with ReLU activations, solutions to the weight decay objective are equivalent to those of a different objective in which the regularization term is instead a sum of products of ℓ 2 (not squared) norms of the input and output weights associated each ReLU.

Original: This alternative (and effectively equivalent) regularization suggests a novel proximal gradient algorithm for network training.
Revised: This alternative (and effectively equivalent) regularization suggests a novel proximal gradient algorithm for network training.

Original: Theory and experiments support the new training approach, showing that it can converge much faster to the sparse solutions it shares with standard weight decay training
Revised: Theory and experiments support the new training approach, showing that it can converge much faster to the sparse solutions it shares with standard weight decay training.

Original: 1. Introduction
Revised: 1. Introduction

Original: Weight decay is the most prevalent form of explicit regularization in deep learning, which corresponds to regularizing the sum of squared weights in the model.
Revised: Weight decay is the most prevalent form of explicit regularization in deep learning, which corresponds to regularizing the sum of squared weights in the model.

Original: It has been shown to improve the generalization performance of deep neural networks [1, 16, 35] and even plays a role in making models more robust [7, 25].
Revised: It has been shown to improve the generalization performance of deep neural networks [1, 17, 41] and even plays a role in making models more robust [8, 28].

Original: This paper argues that weight decay regularization can be equivalently and more effectively incorporated into training via shrinkage and thresholding.
Revised: This paper argues that weight decay regularization can be equivalently and more effectively incorporated into training via shrinkage and thresholding.

Original: Moreover, the new training algorithms produce better solutions with more desirable properties compared to standard weight decay (see initial result in Fig.
Revised: Moreover, the new training algorithms produce better solutions with more desirable properties compared to standard weight decay (see initial result in Fig.

Original: 1 and 2).
Revised: 1 and 2).

Original: Weight decay can produce solutions with similar qualities, but only after an inordinate number of training epochs.
Revised: Weight decay can produce solutions with similar qualities, but only after an inordinate number of training epochs.

Original: This may explain why characteristics like sparsity are not associated with weight decay training.
Revised: This may explain why characteristics like sparsity are not associated with weight decay training.

Original: Shrinkage and thresholding converges to such solutions much faster.
Revised: Shrinkage and thresholding converges to such solutions much faster.

Original: To gain some intuition into the connection between weight decay and thresholding let us consider a key aspect of most neural networks.
Revised: To gain some intuition into the connection between weight decay and thresholding let us consider a key aspect of most neural networks.

Original: Deep architectures include many types of processing steps, but the basic neuron or unit is common in almost all.
Revised: Deep architectures include many types of processing steps, but the basic neuron or unit is common in almost all.

Original: Consider a single unit of the form v σ ( w T x + b ) , where σ is a fixed activation function and v , w , b denote its trainable output, input, and bias weights.
Revised: Consider a single unit of the form v σ ( w T x + b ) , where σ is a fixed activation function and v , w , b denote its trainable output, input, and bias weights.

Original: This single unit is homogeneous if v σ ( w T x + b ) =
Revised: This single unit is homogeneous if v σ ( w T x + b ) =

Original: α v σ ( α − 1 ( w T x + b )) for all α > 0 .
Revised: α v σ ( α − 1 ( w T x + b )) for all α > 0 .

Original: Weight decay regularization of this unit corresponds to adding a term proportional to 12 (cid:0) ∥ w ∥ 22 + ∥ v ∥ 22 ) to the optimization objective.
Revised: Weight decay regularization of this unit corresponds to adding a term proportional to 12 (cid:0) ∥ w ∥ 22 + ∥ v ∥ 22 ) to the optimization objective.

Original: Among all the equivalent representations of the unit, it is easy to verify that
Revised: Among all the equivalent representations of the unit, it is easy to verify that

Original: α 2 = ∥
Revised: α 2 = ∥

Original: w ∥ 2 / ∥ v ∥ 2 produces the smallest regularization term by the AM-GM inequality.
Revised: w ∥ 2 / ∥ v ∥ 2 produces the smallest regularization term by the AM-GM inequality.

Original: Thus, at a minimum of the objective we have 12 (cid:0) ∥ w ∥ 22 + ∥ v ∥ 22 ) =
Revised: Thus, at a minimum of the objective we have 12 (cid:0) ∥ w ∥ 2 2 + ∥ v ∥ 2 2 ) =

Original: ∥ w ∥ 2 ∥ v ∥ 2 .
Revised: ∥ w ∥ 2 ∥ v ∥ 2 .

Original: This simple fact is understood [8, 24], albeit perhaps not widely.
Revised: This simple fact is understood [9, 26], albeit perhaps not widely.

Original: The form ∥ w ∥ 2 ∥ v ∥ 2 (coined as ℓ 2 -P ATH -N ORM ) is reminiscent of ℓ 1 -type regularization functions, such as the lasso and group lasso regularizers.
Revised: The form ∥ w ∥ 2 ∥ v ∥ 2 (coined as ℓ 2 -P ATH -N ORM ) is reminiscent of ℓ 1 -type regularization functions, such as the lasso and group lasso regularizers.

Original: In this paper, we propose to replace the weight decay term with the effectively equivalent proximal operation that corresponds to a shrinkage and thresholding step for each homogeneous unit.
Revised: In this paper [Figure] we propose to replace the weight decay term with the effectively equivalent proximal operation that corresponds to a shrinkage and thresholding step for each homogeneous unit.

Original: Theory (Sec. 3) and experiments (Sec. 5) have shown this leads to solutions that are sparser, have smaller Lipschitz constants, and are more robust than solutions based on weight decay training. © .
Revised: Theory (Sec. 3) and experiments (Sec. 5) have shown this leads to solutions that are sparser, have smaller Lipschitz constants, and are more robust than solutions based on weight decay training

Original: [Figure]
Revised: [Figure]

Original: 2. Related Works
Revised: 2. Related Works

Original: It has been well-understood that weight decay helps improve the robustness and generalization of the trained model [16, 19].
Revised: It has been well-understood that weight decay helps improve the robustness and generalization of the trained model [17, 20].

Original: Lines of work have focused on theoretically analyzing the equivalent form of weight decay: [8] was the first work to highlight an equivalence between weight decay and lasso on the output weights of a neural network.
Revised: Lines of work have focused on theoretically analyzing the equivalent form of weight decay: [9] was the first work to highlight an equivalence between weight decay and lasso on the output weights of a neural network.

Original: Later [23, 24] proved that for a shallow, single output network, training the network with squared ℓ 2 regularization on all the weights (i e. weight decay) is equivalent to regularizing with the ∥ w ∥ 2 ∥ v ∥ 1 norm in the ( w , v ) homogeneous units. 
Revised: Later [25, 26] proved that for a shallow, single output network, training the network with squared ℓ 2 regularization on all the weights (i.e. weight decay) is equivalent to regularizing with the ∥ w ∥ 2 ∥ v ∥ 1 norm in the ( w , v ) homogeneous units. [29, 30] demonstrated regularizing the shallow neural network with ∥ w ∥ 2 ∥ v ∥ 1 will lead to finite width neural network which are the optimal solution to learning problems in second-order Radon-domain Bounded Variation R BV 2 function space. Furthermore they demonstrate how the ∥ w ∥ 2 ∥ v ∥ 1 “path regularizer" is indeed the natural norm for the functions in R BV 2 .

Revised: This result was later generalized for the deep neural network case in [31].

Original: In [27] they also used the equivalent form of weight decay to understand the minimum norm required to fit any function to an infinitely wide univariate shallow network.
Revised: In [32] they also used the equivalent form of weight decay to understand the minimum norm required to fit any function to an infinitely wide univariate shallow network.

Original: More recently, [5] used this equivalence to utilize convex geometry in understanding the optimal solution for neural networks trained with weight decay.
Revised: More recently, [6] used this equivalence to utilize convex geometry in understanding the optimal solution for neural networks trained with weight decay.

Original: Based on the rescaling equivalence on homogeneuous units, [30] proposed an algorithm to iteratively minimize the sum of squared weights for the weight decay objective.
Revised: Based on the rescaling equivalence on homogeneous units, [35] proposed an algorithm to iteratively minimize the sum of squared weights for the weight decay objective.

Original: Another property of weight decay is that it encourages sparse structure, as confirmed in [13].
Revised: Another property of weight decay is that it encourages sparse structure, as confirmed in [14].

Original: With weight decay, several work [26, 36] have also identified the neural collapse phenomenon, indicating that only K number of neurons are needed in the penultimate layer, where K is the number of class.
Revised: [5, 38] also indicate weight decay explicitly promotes low-rank solution for parallel network.

Original: However, the sparse structure is not fully revealed when training with weight decay for finite iterations.
Revised: However, the sparse structure is not fully revealed when training with weight decay for finite iterations.

Original: Our proposed algorithm, on the other hand, directly exploits the sparse structure with a weight decay equivalent form of regularization.
Revised: Our proposed algorithm, on the other hand, directly exploits the sparse structure with a weight decay equivalent form of regularization.

Original: Other recent works propose related forms of regularization, and argue that these are sometimes better than weight decay:
Revised: Other recent works propose related forms of regularization, and argue that these are sometimes better than weight decay:

Original: In [22] introduced the “path regularizer", a generalization of the regularizer in [24] for deep neural networks and showed how it can lead to solutions that generalize better and are more robust [4, 14].
Revised: In [24] introduced the “path regularizer", a generalization of the regularizer in [26] for deep neural networks and showed how it can lead to solutions that generalize better and are more robust [4, 15].

Original: Similarly, [20] utilize the homogeneity of ReLU neural network and proposed “scale shift invariant” algorithm.
Revised: Similarly, [21] utilize the homogeneity of ReLU neural network and proposed “scale shift invariant” algorithm.

Original: Proximal gradient type of algorithm has been proposed for 1-path-norm in [17], where they focus on the ∥ w ∥ 1 ∥ v ∥ 1 norm of a homogeneous unit ( w , v ) in shallow networks.
Revised: Proximal gradient type of algorithm has been proposed for 1-path-norm in [18], where they focus on the ∥ w ∥ 1 ∥ v ∥ 1 norm of a homogeneous unit ( w , v ) in shallow networks.

Original: 3. Training Neural Networks with Weight Decay
Revised: 3. Training Neural Networks with Weight Decay

Original: Let W denote the weights of a multilayer neural network.
Revised: Let W denote the weights of a multilayer neural network.

Original: Most standard “training" algorithms fit neural networks to data by minimizing an objective of the form
Revised: Most standard “training" algorithms fit neural networks to data by minimizing an objective of the form

Original: [Equation]
Revised: [Equation]

Original: where L ( W ) is a loss function on the training data, R ( W ) is the sum of squared weights, and λ ≥ 0 .
Revised: where L ( W ) is a loss function on the training data, R ( W ) is the sum of squared weights, and λ ≥ 0 .

Original: When minimized by gradient descent methods, R ( W ) leads to the common practice known as “weight decay.
Revised: When minimized by gradient descent methods, R ( W ) leads to the common practice known as “weight decay.

Original: " We will call F λ ( W ) the weight decay objective and λ the weight decay parameter .
Revised: " We will call F λ ( W ) the weight decay objective and λ the weight decay parameter .

Revised: The neural network may have a general architecture (fully connected, convolutional, etc) and may involve many types of units and operations (e.g., nonlinear activation functions, pooling/subsampling, etc).

Original: Assume that the activation functions of the neurons in the network are homogeneous : A function σ is homogeneous if it satisfies σ ( αx ) =
Revised: This paper focuses on those units in the architecture that are homogeneous . A function σ is homogeneous if it satisfies σ ( αx ) =

Original: ασ ( x ) for any α > 0 .
Revised: ασ ( x ) for any α > 0 .

Original: For example, the popular Rectified
Revised: For example, the popular Rectified

Original: Linear Unit (ReLU) is homogeneous.
Revised: Linear Unit (ReLU), Leaky ReLU and PReLU are homogeneous.

Original: Consider a neuron with input weights w ∈ R p and output weights v ∈ R q .
Revised: Consider a neuron with input weights w ∈ R p and output weights v ∈ R q .

Original: The neuron produces the mapping x
Revised: The neuron produces the mapping x

Original: (cid:55)→ v σ ( w T x ) .
Revised: (cid:55)→ v σ ( w T x ) .

Original: Because σ is homogeneous, v σ ( w T x )
Revised: Because σ is homogeneous, v σ ( w T x )

Original: =
Revised: =

Original: α v σ ( α − 1 w T x ) , for every α > 0 .
Revised: α v σ ( α − 1 w T x ) , for every α > 0 .

Original: The following NeuralNetwork Balance Theorem provides an important characterization of representations with the minimum sum of squared weights.
Revised: The following Neural Balance Theorem provides an important characterization of representations with the minimum sum of squared weights.

Original: Theorem 1 (NeuralNetwork Balance Theorem) Let f be a function represented by a neural network and consider a representation of f with the minimum sum of squared weights.
Revised: Theorem 1 (Neural Balance Theorem) Let f be a function represented by a neural network and consider a representation of f with the minimum sum of squared weights.

Original: Then the weights satisfy the following balancing constraints .
Revised: Then the weights satisfy the following balancing constraints .

Original: Let w and v denote the input and output weights of any homogeneous unit in this representation.
Revised: Let w and v denote the input and output weights of any homogeneous unit in this representation.

Original: Then ∥ w ∥ 2 = ∥ v ∥ 2 .
Revised: Then ∥ w ∥ 2 = ∥ v ∥ 2 .

Original: Proof Assume there exists a representation f with minimum sum of squared weights, but does not satisfy the constraint for a certain unit.
Revised: Proof Assume there exists a representation f with minimum sum of squared weights, but does not satisfy the constraint for a certain unit.

Original: Because the  unit is homogeneous, its input and output weights, w and v , can be scaled by α > 0 and 1 /α , respectively, without changing the function.
Revised: Because the homogeneous unit is homogeneous, its input and output weights, w and v , can be scaled by α > 0 and 1 /α , respectively, without changing the function.

Original: The solution to the optimization min α> 0 ∥
Revised: The solution to the optimization min α> 0 ∥

Original: α w ∥ 22 + ∥ α − 1 v ∥ 22 is α = (cid:112) ∥ v ∥ 2 / ∥ w ∥ 2 .
Revised: α w ∥ 22 + ∥ α − 1 v ∥ 22 is α = (cid:112) ∥ v ∥ 2 / ∥ w ∥ 2 .

Original: Thus, we can rescale the input and output weights to meet the constraint while preserving f yet reducing the sum of squared weights, contradicting the  assumption of the theorem.
Revised: Thus, we can rescale the input and output weights to meet the constraint while preserving f yet reducing the sum of squared weights, contradicting the beginning assumption in the proof.

Revised: Remark 2 Versions of Neural Balance Theorem (NBT) and its consequences have been discussed in the literature [9, 24, 25, 27, 30–32], but usually in the setting of fully connected ReLU architectures.

Revised: We note here that NBT holds for any architecture (fully connected, convolutional, pooling layers, etc.) and every homogeneous unit in the architecture.

Original: Theorem 1 tells us that the norms of the input and output weights of each homogeneous unit must be equal to each other at a minimum of F λ (see Appendix A.1 for further discussion).
Revised: Theorem 1 tells us that the norms of the input and output weights of each homogeneous unit must be equal to each other at a minimum of F λ (see Appendix A.1 for further discussion).

Original: To illustrate a key implication of this, consider a neural network with L layers of homogeneous neurons.
Revised: To illustrate a key implication of this, consider a neural network with L layers of homogeneous neurons.

Original: Let w i,k and v i,k denote the input and output weights of the i th neuron in the k th layer.
Revised: Let w i,k and v i,k denote the input and output weights of the i th neuron in the k th layer.

Original: Theorem 1 shows that a global minimizer of the weight decay objective F λ with any λ > 0 must satisfy the balancing constraints of the theorem.
Revised: Theorem 1 shows that a global minimizer of the weight decay objective F λ with any λ > 0 must satisfy the balancing constraints of the theorem.

Original: At a global minimum the i th neuron in the k th layer contributes the term 12 (cid:0) [Equation] ,k [Equation]
Revised: At a global minimum the i th neuron in the k th layer contributes the term 12 (cid:0) ∥ w i,k ∥ 22 + ∥ v i,k ∥ 22 (cid:1) =

Original: i,k

Original: [Equation]

Original: = ∥ [Equation] i,k ∥ 2 ∥ v i,k ∥ 2 to the overall sum of squares R ( W ) .
Revised: ∥ w i,k ∥ 2 ∥ v i,k ∥ 2 to the overall sum of squares R ( W ) .

Original: Therefore, at a global minimum of F λ
Revised: Therefore, at a global minimum of F λ

Original: [Equation]
Revised: [Equation]

Original: n k is the number of neurons in the k th layer.
Revised: n k is the number of neurons in the k th layer.

Original: The expression accounts for the fact that certain weights may appear twice (associated with the input/output of two different neurons).
Revised: The expression accounts for the fact that certain weights may appear twice (associated with the input/output of two different neurons).

Original: Though we mainly discuss in the multi-layer perceptron regime, Theorem 1 also applied to convolutional neural network, as presented in Appendix A.2.
Revised: Though we mainly discuss in the multi-layer perceptron regime, Theorem 1 also applied to convolutional neural network, as presented in Appendix A.2.

Original: Note that ∥ w ∥ 2 ∥ v ∥ 2 is the Lipschitz constant of the function v σ ( w T x ) , which shows that weight decay encourages solutions that are Lipschitz smooth (detailed proof in Appendix A.3).
Revised: Note that ∥ w ∥ 2 ∥ v ∥ 2 is the Lipschitz constant of the function v σ ( w T x ) , which shows that weight decay encourages solutions that are Lipschitz smooth (detailed proof in Appendix A.3).

Original: 4. A New Algorithm for Minimizing the Weight Decay Objective
Revised: 4. A New Algorithm for Minimizing the Weight Decay Objective

Original: In this section we propose a new neural network training algorithm.
Revised: In this section we propose a new neural network training algorithm.

Original: It exploits the fact that a global minimizer to the weight decay objective is related to a sum of norm-products, as shown in Eqn.
Revised: It exploits the fact that a global minimizer to the weight decay objective is related to a sum of norm-products, as shown in Eqn.

Original: The following expression for the sum of squared weights will be helpful in deriving the new algorithm.
Revised: The following expression for the sum of squared weights will be helpful in deriving the new algorithm.

Original: Again consider a neural network with homogeneous neurons and L layers.
Revised: Again consider a neural network with homogeneous neurons and L layers.

Original: Since the inputs weights of a neuron in layer k involve the output weights of neurons in the preceding layer k − 1 , we will associate the weights with odd number layers: R ( W ) =
Revised: Since the inputs weights of a neuron in layer k involve the output weights of neurons in the preceding layer k − 1 , we will associate the weights with odd number layers: R ( W ) =

Original: (cid:80) [Equation]
Revised: (cid:80) ⌊ L/ 2 ⌋ j =1 (cid:80) i (cid:0) ∥

Original:  c (cid:80) i ∥ v i,L ∥ 22 , where c = 

Revised: w i, 2 j − 1 ∥ 22 + ∥ v i, 2 j − 1 ∥ 22 (cid:1) + c (cid:80) i ∥ v i,L ∥ 22 , where c =

Original: if L is even and 1 if L is odd.
Revised: if L is even and 1 if L is odd.

Original: Each weight appears only once in the expression above, which is convenient for optimization.
Revised: Each weight appears only once in the expression above, which is convenient for optimization.

Original: Theorem 1 implies for any weights that minimize F λ we have
Revised: Theorem 1 implies for any weights that minimize F λ we have

Original: [Equation]
Revised: [Equation]

Original: Theorem 2 For any weights W let (cid:101) R 
Revised: Theorem 3 For any weights W let (cid:101) R ( W ) : = (cid:80)

Original: [Equation]
Revised: [Equation]

Original: Then the solutions to min W L (
Revised: (cid:80)

Original: [Equation]
Revised: [Equation]

Original: +

Original: [Equation]
Revised: [Equation]

Original:  W ) and min W L ( W ) + λ (cid:101) R ( W ) are equivalent.
Revised: Then the solutions to min W L ( W ) + λ 2 R ( W ) and min W L ( W ) + λ (cid:101) R ( W ) are equivalent.

Original: Specifically, a minimizer of the first optimization is a minimizer of the second, and a minimizer of the second minimizes the first (after possibly rescaling the weights so ∥ w i, 2 j − 1 ∥ 2 = ∥ v i, 2 j − 1 ∥ 2 for each term).
Revised: Specifically, a minimizer of the first optimization is a minimizer of the second, and a minimizer of the second minimizes the first (after possibly rescaling the weights so ∥ w i, 2 j − 1 ∥ 2 = ∥ v i, 2 j − 1 ∥ 2 for each term).

Original: The proof is presented in Appendix A.4.
Revised: The proof is presented in Appendix A.4.

Original: The product ∥ w i,j ∥ 2 ∥ v i,j ∥ 2 is sometimes called the path-norm of the neuron.
Revised: The product ∥ w i,j ∥ 2 ∥ v i,j ∥ 2 is sometimes called the path-norm of the neuron.

Original: So we will call the objective
Revised: So we will call the objective

Original: [Equation]
Revised: [Equation]

Original: the path-norm objective , where again 
Revised: the path-norm objective , where again (cid:101) R ( W ) = (cid:80) ⌊ L/ 2 ⌋ j =1 (cid:80) i ∥ w i [Equation]

Revised: 2 j − 1 ∥

Revised: [Equation]

Revised: v

Revised: [Equation]

Revised: i

Original: [Equation]
Revised: [Equation]

Original:  Theorem 2 shows that minimizing G λ is equivalent to minimizing the weight decay objective F λ .
Revised: v [Equation] L ∥ 22 . Theorem 3 shows that minimizing G λ is equivalent to minimizing the weight decay objective F λ .

Original: Our new neural network training algorithm is designed to minimize the G λ .
Revised: Our new neural network training algorithm is designed to minimize the G λ .

Original: The key observation is that the product terms ∥ w ∥ 2 ∥ v ∥ 2 are non-smooth, which means that minimizers may be sparse.
Revised: The key observation is that the product terms ∥ w ∥ 2 ∥ v ∥ 2 are non-smooth, which means that minimizers may be sparse.

Original: In fact, as λ increases fewer and fewer terms (neurons) will be nonzero.
Revised: In fact, as λ increases fewer and fewer terms (neurons) will be nonzero.

Original: This is remarkable, since the sparsity of solutions is not apparent from simple inspection of the original weight decay objective, as indicate in Fig.
Revised: This is remarkable, since the sparsity of solutions is not apparent from simple inspection of the original weight decay objective, as indicate in Fig.

Original: 1 ( right ).
Revised: 1 ( right ).

Original: The product terms are reminiscent of group lasso regularization terms.
Revised: The product terms are reminiscent of group lasso regularization terms.

Original: Proximal gradient descent methods have been widely applied to this (group) lasso type of regularization schemes in the linear/convex cases [6, 9, 11, 34].
Revised: Proximal gradient descent methods have been widely applied to this (group) lasso type of regularization schemes in the linear/convex cases [7, 10, 12, 40].

Original: This motivates us to the following Algorithm 1.
Revised: This motivates us to the following Algorithm 1.

Original: In the algorithnm, we project each w to the unit sphere, which is also equivalent with the proximal operator of the indicator function ∞· 1 {∥ w ∥ [Equation] ∞
Revised: In the algorithnm, we project each w to the unit sphere, which is also equivalent with the proximal operator of the indicator function ∞

Original: [Equation]
Revised: [Equation]

Original: Shown in [3], with mild assumptions on gradient noise and proper learning rate scheduler beyond T , when running nonstop, the objective value evaluated at every step of our Algorithm 1 converges.
Revised: Shown in [3], with mild assumptions on gradient noise and proper learning rate scheduler beyond T , when running nonstop, the objective value evaluated at every step of our Algorithm 1 converges.

Original: Moreover, every limit point of the iterates { W k } ∞ k =1 is composite critical almost surely.
Revised: Moreover, every limit point of the iterates { W k } ∞ k =1 is composite critical almost surely.

Original: As indicated in Eqn. 2, we may have weights associated to non-homogeneous units: { v i,L } n L i =1 .

Original: Standard weight decay will be applied to these additional weights. Algorithm 1 can also be applied Update input : update based batch gradient 
Revised: The algorithm is presented as a full batch gradient step, but can be easily modified to SGD/mini-batch style algorithms.

Original: ← w t − , 2 j − 1 − γ ∂L ( W ) ∂ w t − 1 i, 2 j − 1 | W t − 1 . project to have unit norm: w ti, 2 j − 1 ← arg min ∥ w ∥ 2 =1 ∥ w − y ∥ 2 = y ∥
Revised: Algorithm Optimization of λ as discussed in Thm. unit norm: w ti, 2 j − 1 ← arg min ∥ w ∥ 2 =1 ∥ w − y ∥ 2 = y ∥

Original: y
Revised: y

Original: ∥ 2 .
Revised: ∥ 2 .

Original: Update homogeneous output weights: update based  gradient:  ← v t − 1 i, 2 j − 1 − γ ∂L  W ) ∂ v t − 1 i, 2 j − 1 | W t − 1 .
Revised: Update homogeneous output weights: update based on batch gradient: z ← v t − 1 i, 2 j − 1 − γ ∂L ( W ) ∂ v t − 1 i, 2 j − 1 | W t − 1 .

Original:  ||
Revised: apply proximal operator: v ti, 2 j − 1 ← Prox 2 ( z ) with Prox 2 ( z ) i = (cid:40)|| z ||

Original: z || 2 ≤ λ  γ z i − λ · γ z i || 
Revised: ≤ λ · γ z i − λ · γ z i || z ||o.w.

Original: z
Revised: end for end for As indicated in Eqn. 2, we may have weights associated to non-homogeneous units: { v i,L } n L i =1 . Standard weight decay will be applied to these additional weights.

Original: || for to convolutional neural networks, where we treat each channel of a given convolutional layer as an 1-homogeneous unit (see details in Appendix B.1).
Revised: Algorithm 1 can also be applied to convolutional neural networks, where we treat each channel of a given convolutional layer as an 1-homogeneous unit (see details in Appendix B.1).

Original: Motivated by the homogeneity of layers and
Revised: Motivated by the homogeneity of layers and

Original: Theorem 1, we also apply a layer-wise balancing procedure (described in Appendix B.2) to ensure the total ℓ 2 -P ATH -N ORM among each coupling of layers to be equal at every iteration.
Revised: Theorem 1, we also apply a layer-wise balancing procedure (described in Appendix B.2) to ensure the total ℓ 2 -P ATH -N ORM among each coupling of layers to be equal at every iteration.

Original: 5. Experimental Results
Revised: 5. Experimental Results

Original: In this section, we will present experimental results to support the following claims: Proximal algorithm for ℓ 2 -P ATH -N ORM provides a) faster convergence b) better generalization c) sparser solutions than weight decay.
Revised: In this section, we will present experimental results to support the following claims: Proximal algorithm for ℓ 2 -P ATH -N ORM provides a) faster convergence b) better generalization c) sparser solutions than weight decay.

Original: For simplicity, we use MLP- d - n for d fully-connected feedforward layers, each with n neurons.
Revised: For simplicity, we use MLP- d - n for d fully-connected feedforward layers, each with n neurons.

Original: MLP- d - n factorized indicates each layer of MLP- d - n is factorized into 2 linear layers, with hidden neurons to be n as well.
Revised: MLP- d - n factorized indicates each layer of MLP- d - n is factorized into 2 linear layers, with hidden neurons to be n as well.

Original: Our proximal algorithm is evaluated on the following task: (Task 1) MNIST [18] subset with a factorized MLP-3-400,
Revised: Our proximal algorithm is evaluated on the following task: (Task 1) MNIST [19] subset on factorized MLP-3-400,

Original: (Task 2) MNIST with an MLP-6-400, (Task 3) CIFAR

Revised: (Task 2) MNIST on MLP-6-400, (Task 3) CIFAR

Original: [15] with VGG19 [29], (Task 4)  MNIST with an MLP-3-800.
Revised: [16] on VGG19 [34], (Task 4) SVHN on VGG19, (Task 5) MNIST on MLP-3-800.

Original: For MNIST subset, we randomly subsample 100 images per class.
Revised: For MNIST subset, we randomly subsample 100 images per class.

Original: Please refer to details of the experiment settings in Appendix C.1.
Revised: Please refer to details of the experiment settings in Appendix C.1.

Original: The detail of applying Algorithm 1 to convolutional neural networks such as VGG is discussed in Appendix B.1.
Revised: The detail of applying Algorithm 1 to convolutional neural networks such as VGG19 is discussed in Appendix B.1.

Original: 5.1. Generalization on Test Data and Corrupted Data
Revised: 5.1. Generalization on Test Data and Corrupted Data

Original: Performance on the unseen dataset, or dataset sampled from other distribution, measures the generalization ability of the model.
Revised: Performance on the unseen dataset, or dataset sampled from other distribution, measures the generalization ability of the model.

Original: Following the experimental design in [10, 28, 35], the following modification of the data and label are investigated: i)
Revised: Following the experimental design in [11, 33, 41], the following modification of the data and label are investigated: i)

Original: True labels : the original dataset without modification.
Revised: True labels : the original dataset without modification.

Original: ii) Corrupted dataset : for MNIST, we train on an unmodified dataset, then evaluate the average accuracy across different types of corruption on the MNIST-C dataset [21].
Revised: ii) Corrupted dataset : for MNIST, we train on an unmodified dataset, then evaluate the average accuracy across different types of corruption on the MNIST-C dataset [22].

Revised: For

Revised: CIFAR10, we train on unmodified dataset, then evaluate the average accuracy across different types of corruption on the CIFAR10-C dataset [13].

Original: iii)
Revised: iii)

Original: Partially corrupted labels : the label of training image is corrupted to be uniform random class with probability p .
Revised: Partially corrupted labels : the label of training image is corrupted to be uniform random class with probability p .

Original: We train on this modified dataset, and report the result on unmodified dataset.
Revised: We train on this modified dataset, and report the result on unmodified dataset.

Original: In this section, we present the result of (Task 1) on modification i, ii, and iii; result of (Task 2) on modification i and ii;and result of (Task 3) on modification i and ii in Table 1.
Revised: In this section, we present the result of (Task 1) on modification i, ii, and iii; result of (Task 2) on modification i and ii; result of (Task 3) on modification i and ii; and result of (Task 4) on modification i in Table 1.

Original: For each experiment, we did a grid search on the hyper-parameter choice of λ and learning rate , and pick the best set of parameter based on the validation accuracy.
Revised: For each experiment, we did a grid search on the hyper-parameter choice of λ and learning rate , and pick the best set of parameter based on the validation accuracy.

Revised: The details of hyper-parameter search is in Appendix C.1.

Original: Notice that although we didn’t explicitly prune the model, our proposed ∥ w ∥ 2 ∥ v ∥ 2 proximal algorithm naturally enforce some level of sparsity.
Revised: Notice that although we did not explicitly prune the model, our proposed ∥ w ∥ 2 ∥ v ∥ 2 proximal algorithm inherently enforce the sparse structure: in the

Revised: Table 1, we calculate the structural sparsity of the model, namely the percentage of the active units in the grouped layers, and the result confirm our proximal method can naturally prune the model to have some level of sparsity.

Original: We also investigate the efficiency of finding the structural sparse solution when explicitly enforce the model to prune, and present the result in Appendix C.3.
Revised: We also investigate the efficiency of finding the structural sparse solution when explicitly enforce the model to prune, and present the result in Appendix C. [Table]

Original: [Table]
Revised: [Table]

Original: 6. Conclusion and Future Work
Revised: 6. Conclusion and Future Work

Original: In this work [Table] we primarily identify the one-homogeneous units through grouping of consecutive layers.
Revised: This work shows that proximal gradient algorithms may offer advantages in neural network training compared to standard weight decay.

Original: Alternative one-homogeneous grouping schemes can be explored for future work. Moreover, our algorithm currently assumes that one-homogeneous units to be non-overlapping. For future work, we would like to answer the following question: Do we have equivalent weight decay regularizers if these units overlap?
Revised: There are several directions for possible future work, including investigating alternative formulations of the proximal gradient method that treat all homogeneous units in the same manner (rather than grouping weights into disjoint sets) and adaptive learning rate procedures like those used in other proximal gradient methods [Table]

Original: Appendix Content A Supplementary of Theory
Revised: Appendix Content A Supplementary of Theory

Original: A.1 Discussion on the Theorem 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A.1 Discussion on the Theorem 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A.2 Extension of Theorem 1 to Convolutional Layers . . . . . . . . . . . . . . . . . .
Revised: A.2 Extension of Theorem 1 to Convolutional Layers . . . . . . . . . . . . . . . . . .

Original: A.3 Proof for Lipschitz constant bound . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A.3 Proof for Lipschitz constant bound . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A.4 Proof for Theorem. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A.4 Proof for Theorem. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: B Supplementary of Algorithm
Revised: B Supplementary of Algorithm

Original: B.1 Homogeneous Units are Everywhere . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: B.1 Homogeneous Units are Everywhere . . . . . . . . . . . . . . . . . . . . . . . . .

Original: B.2 Layer-wise Balancing Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: B.2 Layer-wise Balancing Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: B.2.
Revised: B.2.

Original: Corollary of Theorem 1: Layer-wise Balancing . . . . . . . . . . . . . . .
Revised: Corollary of Theorem 1: Layer-wise Balancing . . . . . . . . . . . . . . .

Original: B.2.
Revised: B.2.

Original: Layer-wise Balance Algorithm .
Revised: Layer-wise Balance Algorithm .

Original: C Supplementary of Experiments
Revised: C Supplementary of Experiments

Original: C.1 Experiment Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: C.1 Experiment Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: C.1.
Revised: C.1.

Original: Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: C.1.
Revised: C.1.

Original: Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: C.1.
Revised: C.1.

Original: Hyper Parameter Choice . . .
Revised: Hyper Parameter Choice . . .

Original: C.1.
Revised: C.1.

Original: Standard Error Calculation . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Standard Error Calculation . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: C.2 Experiments Details for Figure 1 and 2 . . . . . . . . . . . . . . . . . . . . . . . .
Revised: C.2 Experiments Details for Figure 1 and 2 . . . . . . . . . . . . . . . . . . . . . . . .

Original: C.3 Proximal Algorithm Finds Structural Sparse Solution Faster . . . . . . . . . . . .
Revised: C.3 Proximal Algorithm Finds Structural Sparse Solution Faster . . . . . . . . . . . .

Original: Appendix A. Supplementary of Theory A.1. Discussion on the Theorem
Revised: Appendix A. Supplementary of Theory A.1. Discussion on the Theorem

Original: At minimum sum of squared weights, Theorem 1 state that each homogeneous unit in the function f have equivalent ℓ 2 norm for the input and output weights of the unit:
Revised: At minimum sum of squared weights, Theorem 1 state that each homogeneous unit in the function f have equivalent ℓ 2 norm for the input and output weights of the unit:

Original: [Equation]
Revised: [Equation]

Original: Note that the balancing constraint is a necessary, but not a sufficient condition for the minimum sum of squared weights representation.
Revised: Note that the balancing constraint is a necessary, but not a sufficient condition for the minimum sum of squared weights representation.

Original: In details, there are different equivalence of function:
Revised: In details, there are different equivalence of function:

Original: Definition 3 (Rescaling Equivalence) Function f with weights W f
Revised: Definition 4 (Rescaling Equivalence) Function f with weights W f

Original: and g with weights W g are rescaling equivalent if W f = s ( W g ) where s ( · ) is the scaling operator on the homogeuous units in weights W .
Revised: and g with weights W g are rescaling equivalent if W f = s ( W g ) where s ( · ) is the scaling operator on the homogeuous units in weights W .

Original: This rescaling equivalence is the main force we leverage for finding the effectively equivalent norm.
Revised: This rescaling equivalence is the main force we leverage for finding the effectively equivalent norm.

Original: Definition 4 (Functional Equivalence) Function f with weights W f and g with weights W g are functional equivalent if ∀ x , f ( x ) =
Revised: Definition 5 (Functional Equivalence) Function f with weights W f and g with weights W g are functional equivalent if ∀ x , f ( x ) =

Original: g ( x ) .
Revised: g ( x ) .

Original: It is clear from the definition that rescaling equivalence implies functional equivalence, but the reverse is not true.
Revised: It is clear from the definition that rescaling equivalence implies functional equivalence, but the reverse is not true.

Original: With this definition, we will properly state the relation of balancing constraint and minimum representation as follows (also discussed in [30]):
Revised: With this definition, we will properly state the relation of balancing constraint and minimum representation as follows (also discussed in [35]):

Original: Remark 5 For function f with weights W , the minimum norm up to rescaling equivalence may not be the minimum representation up to functional equivalence.
Revised: Remark 6 For function f with weights W , the minimum norm up to rescaling equivalence may not be the minimum representation up to functional equivalence.

Original: So the minimum sum of squared weights representation implies balancing constraint, however balancing constraint doesn’t necessarily imply minimum norm representation.
Revised: So the minimum sum of squared weights representation implies balancing constraint, however balancing constraint doesn’t necessarily imply minimum norm representation.

Original: We are not aware of the algorithm to find the minimum sum of squared weights representation.
Revised: We are not aware of the algorithm to find the minimum sum of squared weights representation.

Original: However up to rescaling equivalence, [30] proposed an iterative algorithm to minimize the sum of squared weights, which requires C cycles to converge.
Revised: However up to rescaling equivalence, [35] proposed an iterative algorithm to minimize the sum of squared weights, which requires C cycles to converge.

Original: The number of cycles after each SGD step is an hyperparameter, by default they set to 1.
Revised: The number of cycles after each SGD step is an hyperparameter, by default they set to 1.

Original: If set to C , which means constraint the norm to be balanced for every homogeneous units after each SGD step,  the algorithm converge to worse solution compared to without this constraint.
Revised: If set to C , which means constraint the norm to be balanced for every homogeneous units after each SGD step, from our experiment findings, the algorithm converge to worse solution compared to without this constraint.

Original: A.2. Extension of Theorem 1 to Convolutional Layers
Revised: A.2. Extension of Theorem 1 to Convolutional Layers

Original: Let ⃝ ∗ denote the sliding-window convolutional operator, where for any two-dimensional tensors
Revised: Let ⃝ ∗ denote the sliding-window convolutional operator, where for any two-dimensional tensors

Original: X, Y and Z , we have Z i,j =
Revised: X, Y and Z , we have Z i,j =

Original: ( X ⃝ ∗ Y )
Revised: ( X ⃝ ∗ Y )

Original: i,j = (cid:80) a ∈ [ L 1 ] ,b ∈
Revised: i,j = (cid:80) a ∈ [ L 1 ] ,b ∈

Original: [ L 2] X a,b Y i + a,j + b with L 1 and L 2 denoting the width and height of X .
Revised: [ L 2 ] X a,b Y i + a,j + b with L 1 and L 2 denoting the width and height of X .

Original: A convolutional homogeneous unit then takes the following form with one hidden channel 
Revised: A convolutional homogeneous unit then takes the following form with one hidden channel [Equation] [Equation] =

Original: [Equation]
Revised: [Equation]

Original: where x , w and v are three-dimensional tensors and with slight abuse of notation σ ( · ) is applied element-wise.
Revised: where x , w and v are three-dimensional tensors and with slight abuse of notation σ ( · ) is applied element-wise.

Original: x takes the channel first notation, where the first dimension indexes number of channels while the last two dimensions indexes width and height.
Revised: x takes the channel first notation, where the first dimension indexes number of channels while the last two dimensions indexes width and height.

Original: Both x and w have channel size
Revised: Both x and w have channel size

Original: C 1 while v has channel size C 2 .
Revised: C 1 while v has channel size C 2 .

Original: Below, we restate the equivalence from Theorem 1 for convolutional units:
Revised: Below, we restate the equivalence from Theorem 1 for convolutional units:

Original: Theorem 6 (Convolutional Neural Network Balance Theorem) Let f be a function represented by a neural network and consider a representation of f with the minimum sum of squared weights.
Revised: Theorem 7 (Convolutional Neural Network Balance Theorem) Let f be a function represented by a neural network and consider a representation of f with the minimum sum of squared weights.

Original: Furthermore, let vec ( · ) denote the vectorize operator of a tensor.
Revised: Furthermore, let vec ( · ) denote the vectorize operator of a tensor.

Original: Then the weights satisfy the following balancing constraints .
Revised: Then the weights satisfy the following balancing constraints .

Original: Let w and v denote the input and output weights of any convolutional homogeneous unit µ ( x ) =
Revised: Let w and v denote the input and output weights of any convolutional homogeneous unit µ ( x ) =

Original: [ v j ⃝ ∗ σ ( (cid:80) i ∈ [ C 1 ] w i ⃝ ∗ x i )]
Revised: [ v j ⃝ ∗ σ ( (cid:80) i ∈ [ C 1 ] w i ⃝ ∗ x i )]

Original: j ∈ [ C 2 ] in this representation.
Revised: j ∈ [ C 2 ] in this representation.

Original: Then ∥ vec ( w ) ∥ 2 = ∥ vec ( v ) ∥ 2 .
Revised: Then ∥ vec ( w ) ∥ 2 = ∥ vec ( v ) ∥ 2 .

Original: We also note that one can take a channel-wise homogeneous pooling operation on the hidden channel, i.e., a homogeneous unit of the form µ ( x ) =
Revised: We also note that one can take a channel-wise homogeneous pooling operation on the hidden channel, i.e., a homogeneous unit of the form µ ( x ) =

Original: [ v j ⃝ ∗ P ( σ ( (cid:80) i ∈ [ C 1 ] w i ⃝ ∗ x i ))]
Revised: [ v j ⃝ ∗ P ( σ ( (cid:80) i ∈ [ C 1 ] w i ⃝ ∗ x i ))]

Original: j ∈ [ C 2 ] , where
Revised: j ∈ [ C 2 ] , where

Original: P is some homogeneous pooling function such as max pooling or average pooling.
Revised: P is some homogeneous pooling function such as max pooling or average pooling.

Original: Due to the homogeneity of the pooling layer, the above results still hold for these homogeneous units with pooling layers.
Revised: Due to the homogeneity of the pooling layer, the above results still hold for these homogeneous units with pooling layers.

Original: This suggests we can apply the argument across blocks in a convolutional network.
Revised: This suggests we can apply the argument across blocks in a convolutional network.

Original: A.3. Proof for Lipschitz constant bound
Revised: A.3. Proof for Lipschitz constant bound

Original: As indicated in Sec. 3, at the global minimum of F λ , we can write the weight decay objective as Eqn.
Revised: As indicated in Sec. 3, at the global minimum of F λ , we can write the weight decay objective as Eqn.

Original: This shows that increasing the weight decay parameter λ penalizes the average of the norm-products ∥ w ∥ 2 ∥ v ∥ 2 .
Revised: This shows that increasing the weight decay parameter λ penalizes the average of the norm-products ∥ w ∥ 2 ∥ v ∥ 2 .

Original: This observation provides insight into the effects of weight decay regularization: Let η ( x ) :
Revised: This observation provides insight into the effects of weight decay regularization: Let η ( x ) :

Original: = v σ ( w T x ) be a homogeneous neuron with input weights w ∈ R
Revised: = v σ ( w T x ) be a homogeneous neuron with input weights w ∈ R

Original: p
Revised: p

Original: and
Revised: and

Original: output weights v ∈ R q , and assume that σ is 1 -Lipschitz.
Revised: output weights v ∈ R q , and assume that σ is 1 -Lipschitz.

Original: Hölder’s inequality implies that for all x , x ′ ∈ R m ,
Revised: Hölder’s inequality implies that for all x , x ′ ∈ R m ,

Original: [Equation]
Revised: [Equation]

Original: which shows ∥ w ∥ 2 ∥ v ∥ 2 is a bound on the Lipschitz constant of η .
Revised: which shows ∥ w ∥ 2 ∥ v ∥ 2 is a bound on the Lipschitz constant of η .

Original: Thus, weight decay regularization encourages solutions in which the individual unit functions have small Lipschitz constants on average, a property known to be related to generalization and robustness.
Revised: Thus, weight decay regularization encourages solutions in which the individual unit functions have small Lipschitz constants on average, a property known to be related to generalization and robustness.

Original: A.4. Proof for Theorem
Revised: A.4. Proof for Theorem

Original: Restate the Theorem 2 Here.
Revised: Restate the Theorem 3 Here.

Original: Theorem For any weights W let (cid:101)
Revised: Theorem For any weights W let (cid:101)

Original: [Equation]
Revised: [Equation]

Original: Then the solutions to min W L ( W ) + λ 2 R ( W ) and min W L ( W ) + λ (cid:101)
Revised: Then the solutions to min W L ( W ) + λ 2 R ( W ) and min W L ( W ) + λ (cid:101)

Original: R ( W ) are equivalent.
Revised: R ( W ) are equivalent.

Original: Specifically, a minimizer of the first optimization is a minimizer of the second, and a minimizer of the second minimizes the first (after possibly rescaling the weights so ∥ w i, 2 j − 1 ∥ 2 = ∥ v i, 2 j − 1 ∥ 2 for each term).
Revised: Specifically, a minimizer of the first optimization is a minimizer of the second, and a minimizer of the second minimizes the first (after possibly rescaling the weights so ∥ w i, 2 j − 1 ∥ 2 = ∥ v i, 2 j − 1 ∥ 2 for each term).

Original: Proof Suppose that (cid:99) W is a solution to min W L ( W ) + λ (cid:101) R ( W ) , but there exists a W such that
Revised: Proof Suppose that (cid:99) W is a solution to min W L ( W ) + λ (cid:101) R ( W ) , but there exists a W such that

Original: [Equation]
Revised: [Equation]

Original: Theorem 1 shows that R ( W ) = 2 (cid:101) R ( W ) , which contradicts the claim that (cid:99) W is a solution to min W L ( W ) + λ (cid:101) R ( W ) .
Revised: Theorem 1 shows that R ( W ) = 2 (cid:101) R ( W ) , which contradicts the claim that (cid:99) W is a solution to min W L ( W ) + λ (cid:101) R ( W ) .

Original: Next let W be a solution to min W L ( W ) + λ 2 R ( W ) , but suppose it does not minimize L ( W ) + λ (cid:101) R ( W ) .
Revised: Next let W be a solution to min W L ( W ) + λ 2 R ( W ) , but suppose it does not minimize L ( W ) + λ (cid:101) R ( W ) .

Original: Then there exists a (cid:99) W such that
Revised: Then there exists a (cid:99) W such that

Original: [Equation]
Revised: [Equation]

Original: If necessary, rescale the weights (cid:99) W so that ∥ (cid:98) w i, 2 j − 1 ∥ 2 = ∥ (cid:98) v i, 2 j − 1 ∥ 2 for each term in (cid:101) R ( W ) ; this does not affect the value of L ( (cid:99) W ) .
Revised: If necessary, rescale the weights (cid:99) W so that ∥ (cid:98) w i, 2 j − 1 ∥ 2 = ∥ (cid:98) v i, 2 j − 1 ∥ 2 for each term in (cid:101) R ( W ) ; this does not affect the value of L ( (cid:99) W ) .

Original: Then R ( (cid:99) W ) / 2 = (cid:101) R ( (cid:99) W ) , which shows that L ( W )+ λ 2 R ( W ) >
Revised: Then R ( (cid:99) W ) / 2 = (cid:101) R ( (cid:99) W ) , which shows that L ( W )+ λ 2 R ( W ) >

Original: L ( (cid:99) W ) + λ 2 R ( (cid:99) W ) , contradicting the assumption that W is a solution to the first optimization.
Revised: L ( (cid:99) W ) + λ 2 R ( (cid:99) W ) , contradicting the assumption that W is a solution to the first optimization.

Original: Appendix B. Supplementary of Algorithm B.1. Homogeneous Units are Everywhere
Revised: Appendix B. Supplementary of Algorithm B.1. Homogeneous Units are Everywhere

Original: In the paper we mainly discuss the theorem and algorithm related to the multi-layer perceptron, but the identification of homogeneous units is not limited to multi-layer perceptron.
Revised: In the paper we mainly discuss the theorem and algorithm related to the multi-layer perceptron, but the identification of homogeneous units is not limited to multi-layer perceptron.

Original: In this section, we formally identify the homogeneous units in common neural network architectures: multi-layer perceptron (MLP) and convolutional neural network (CNN).
Revised: In this section, we formally identify the homogeneous units in common neural network architectures: multi-layer perceptron (MLP) and convolutional neural network (CNN).

Original: Multi-layer Perceptron (MLP).
Revised: Multi-layer Perceptron (MLP).

Original: A multi-layer perceptron f MLP ( x ; W ) :
Revised: A multi-layer perceptron f MLP ( x ; W ) :

Original: X → Y with M linear layers takes the following recursive parameterization
Revised: X → Y with M linear layers takes the following recursive parameterization

Original: [Equation]
Revised: [Equation]

Original: and h
Revised: and h

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: Here, the linear layers are parameterized by weights W = { W k ∈ R n k +1 × n k } Lk
Revised: Here, the linear layers are parameterized by weights W = { W k ∈ R n k +1 × n k } Lk

Original: =1 , where n k is the dimension of the ( k − 1) -th hidden layer, n 1 is the input dimension, and n L +1 is the output dimension.
Revised: =1 , where n k is the dimension of the ( k − 1) -th hidden layer, n 1 is the input dimension, and n L +1 is the output dimension.

Original: The activation function σ ( x ) = max { 0 , x } is applied element-wise.
Revised: The activation function σ ( x ) = max { 0 , x } is applied element-wise.

Original: Let (cid:102) W denote the matrix of W with its first column removed, W i and W : ,i denote the i -th row and column of W , respectively.
Revised: Let (cid:102) W denote the matrix of W with its first column removed, W i and W : ,i denote the i -th row and column of W , respectively.

Original: Consider for every two layers, we have
Revised: Consider for every two layers, we have

Original: [Equation]
Revised: [Equation]

Original: as part of the computation for h k +2 .
Revised: as part of the computation for h k +2 .

Original: Therefore, for every two consecutive layers with weights W 2 j −and (cid:102) W 2 j where j ∈ (cid:2) ⌊ L 2 ⌋ (cid:3) , we identify n 2 j − 1 homogeneous units — one for each hidden neuron.
Revised: Therefore, for every two consecutive layers with weights W 2 j −and (cid:102) W 2 j where j ∈ (cid:2) ⌊ L 2 ⌋ (cid:3) , we identify n 2 j − 1 homogeneous units — one for each hidden neuron.

Original: In our experiments, for regular multi-layer perceptron, we use exactly this coupling scheme, where every two layers are combined and viewed as n 2 j − 1 homogeneous units.
Revised: In our experiments, for regular multi-layer perceptron, we use exactly this coupling scheme, where every two layers are combined and viewed as n 2 j − 1 homogeneous units.

Original: Factorized MLP.
Revised: Factorized MLP.

Original: Multi-layer perceptron can be equivalent factorized as shown in [32].
Revised: Multi-layer perceptron can be equivalent factorized as shown in [37].

Original: The linear layers are parameterized by weights W = { W k ∈ R n k +1 × n k } Lk
Revised: The linear layers are parameterized by weights W = { W k ∈ R n k +1 × n k } Lk

Original: =1 , where n k is the dimension of the ( k − 1) -th hidden layer.
Revised: =1 , where n k is the dimension of the ( k − 1) -th hidden layer.

Original: For each W k , we can further factorize it into W k = Q k P k for k ∈ { 2 , 3 , · · · , L − 1 } , where P k ∈ R ( n k +1 − 1) × n k , Q k ∈ R n k +1 × ( n k +1 − 1) , and
Revised: For each W k , we can further factorize it into W k = Q k P k for k ∈ { 2 , 3 , · · · , L − 1 } , where P k ∈ R ( n k +1 − 1) × n k , Q k ∈ R n k +1 × ( n k +1 − 1) , and

Original: [Equation]
Revised: [Equation]

Original: Here the bias term is not required for the factorization.
Revised: Here the bias term is not required for the factorization.

Original: Now let g k = P k
Revised: Now let g k = P k

Original: (cid:20)h k (cid:21) ∈ R n k +1 , and let (cid:101)
Revised: (cid:20)h k (cid:21) ∈ R n k +1 , and let (cid:101)

Original: P denote the matrix of P with its first column removed, then (cid:102) W = Q (cid:101)
Revised: P denote the matrix of P with its first column removed, then (cid:102) W = Q (cid:101)

Original: P has the first column removed.
Revised: P has the first column removed.

Original: We have part of the computation for h k +2 to be:
Revised: We have part of the computation for h k +2 to be:

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: where Q i and P : ,i denote the i -th row and column of Q and P , respectively.
Revised: where Q i and P : ,i denote the i -th row and column of Q and P , respectively.

Original: Therefore, for every consecutive layers with factorized weights Q k and P k +1 , we identify n k +1 − 1 homogeneous units — one for each hidden neuron.
Revised: Therefore, for every consecutive layers with factorized weights Q k and P k +1 , we identify n k +1 − 1 homogeneous units — one for each hidden neuron.

Original: Note that for fixed dimensions and number of layers, the class of factorized MLP is equivalent with the class of the original MLPs with only activated layers.
Revised: Note that for fixed dimensions and number of layers, the class of factorized MLP is equivalent with the class of the original MLPs with only activated layers.

Original: Furthermore, with a factorized MLP, we bypass the issue of having to group even number of layers.
Revised: Furthermore, with a factorized MLP, we bypass the issue of having to group even number of layers.

Original: Instead, we are able to optimize an equivalent class of function and identify a homogeneous unit for each hidden neuron.
Revised: Instead, we are able to optimize an equivalent class of function and identify a homogeneous unit for each hidden neuron.

Original: Convolutional neural networks.
Revised: Convolutional neural networks.

Original: A 2D convolutional backbone network is a sequence of function composition of convolutional and pooling layers that maps 3D tensors to 3D tensors.
Revised: A 2D convolutional backbone network is a sequence of function composition of convolutional and pooling layers that maps 3D tensors to 3D tensors.

Original: The output of a backbone network is usually then flattened and passed through an MLP.
Revised: The output of a backbone network is usually then flattened and passed through an MLP.

Original: We focus on an L layers convolutional backbone network here, which takes the following recursive parameterization f CNN
Revised: We focus on an L layers convolutional backbone network here, which takes the following recursive parameterization f CNN

Original: [Equation]
Revised: [Equation]

Original: (cid:88)
Revised: (cid:88)

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (cid:88)
Revised: (cid:88)

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original:   
Revised:   

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ] , ∀ k ∈
Revised: ] , ∀ k ∈

Original: [ L − 1] and h 1 = x.
Revised: [ L − 1] and h 1 = x.

Original: Here n k denotes the number of hidden/output channels and W = { ( W k , B k ) }
Revised: Here n k denotes the number of hidden/output channels and W = { ( W k , B k ) }

Original: Lk =1 are the weights parameterizing the neural network.
Revised: Lk =1 are the weights parameterizing the neural network.

Original: Each layer weight W k is a four-dimensional tensor of dimensions n k +1 × n k × l × l ′ and b k ∈ R n L +1 .
Revised: Each layer weight W k is a four-dimensional tensor of dimensions n k +1 × n k × l × l ′ and b k ∈ R n L +1 .

Original: ⊕ is an element-wise addition operator which adds the later scalar argument to the former tensor.
Revised: ⊕ is an element-wise addition operator which adds the later scalar argument to the former tensor.

Original: P k is a channel-wise pooling layer such as average pooling and max pooling, or the identity function.
Revised: P k is a channel-wise pooling layer such as average pooling and max pooling, or the identity function.

Original: Each hidden layer h k is then a three-dimensional tensor of dimensions n k × l × l ′ .
Revised: Each hidden layer h k is then a three-dimensional tensor of dimensions n k × l × l ′ .

Original: By substituting in one more recursive step and for any homogeneous activation function σ , we get h
Revised: By substituting in one more recursive step and for any homogeneous activation function σ , we get h

Original: k
Revised: k

Original: +

Revised: +

Original: =
Revised: =

Original: 
Revised: 

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: [Equation]
Revised: [Equation]

Original:   
Revised:   

Original:  i ∈
Revised:  i ∈

Original: [ n
Revised: [ n

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original:  σ    (cid:88)
Revised:  σ    (cid:88)

Original: j ′
Revised: j ′

Original: W k − 1 j,j ′ ⃝ ∗ h j − 1 j ′ 
Revised: W k − 1 j,j ′ ⃝ ∗ h j − 1 j ′ 

Original: 
Revised: 

Original: ⊕ b kj   
Revised: ⊕ b kj   

Original:  
Revised:  

Original:  ⊕ b ki   i ∈ [ n k +1 ]     .
Revised:  ⊕ b ki   i ∈ [ n k +1 ]     .

Original: Therefore, for each j ∈ [ n k ] , we have (cid:104) W ki,j ⃝ ∗ P k − 1 (cid:16) σ (cid:16)(cid:16)(cid:80) j ′ W k − 1 j,j ′ ⃝ ∗ h k − 1 j ′ (cid:17) ⊕ b k − 1 j (cid:17)(cid:17)(cid:105) i ∈ [ n k +1 ] as part of the computation.
Revised: Therefore, for each j ∈ [ n k ] , we have (cid:104) W ki,j ⃝ ∗ P k − 1 (cid:16) σ (cid:16)(cid:16)(cid:80) j ′ W k − 1 j,j ′ ⃝ ∗ h k − 1 j ′ (cid:17) ⊕ b k − 1 j (cid:17)(cid:17)(cid:105) i ∈ [ n k +1 ] as part of the computation.

Original: With the one-homogeneity of the channel-wise pooling layer P k − 1 , we can get an equivalent function by scaling (cid:16) αW k − 1 j,j ′ , αb k − 1 j (cid:17) and 1 α W ki,j , we can therefore obtain a homogeneous unit for every hidden channel j of the k -th layer h k .
Revised: With the one-homogeneity of the channel-wise pooling layer P k − 1 , we can get an equivalent function by scaling (cid:16) αW k − 1 j,j ′ , αb k − 1 j (cid:17) and 1 α W ki,j , we can therefore obtain a homogeneous unit for every hidden channel j of the k -th layer h k .

Original: Since the channel-wise pooling layer can be either identity, or max pooling, or average pooling layer, we can group the convolutional layers across max/average pooling layers.
Revised: Since the channel-wise pooling layer can be either identity, or max pooling, or average pooling layer, we can group the convolutional layers across max/average pooling layers.

Original: Therefore, for every two consecutive convolutional layer (possibly across the pooling layer), with weights (cid:16)
Revised: Therefore, for every two consecutive convolutional layer (possibly across the pooling layer), with weights (cid:16)

Original: W 2 k − 1 j,j ′ , b 2 k − 1 j (cid:17) and W 2 k
Revised: W 2 k − 1 j,j ′ , b 2 k − 1 j (cid:17) and W 2 k

Original: i,j , where k ∈ { 1 , 2 , · · · , ⌊ L 2 ⌋} , we identify n 2 k − 1 homogeneous units, one for each channel.
Revised: i,j , where k ∈ { 1 , 2 , · · · , ⌊ L 2 ⌋} , we identify n 2 k − 1 homogeneous units, one for each channel.

Original: B.2. Layer-wise Balancing Procedure
Revised: B.2. Layer-wise Balancing Procedure

Original: In this section, we present a layer-wise balancing procedure that enforces balancing constraints across layers.
Revised: In this section, we present a layer-wise balancing procedure that enforces balancing constraints across layers.

Original: This is motivated by the fact that the sums of ℓ 2 -P ATH -N ORM of every two consecutive layers should be equal at the minimum norm solution.
Revised: This is motivated by the fact that the sums of ℓ 2 -P ATH -N ORM of every two consecutive layers should be equal at the minimum norm solution.

Original: Below, we will first formally show this observation as a corollary of Theorem 1 and then present the specific layer-wise balancing algorithm.
Revised: Below, we will first formally show this observation as a corollary of Theorem 1 and then present the specific layer-wise balancing algorithm.

Original: B.2.1. C OROLLARY OF T HEOREM 1: L AYER - WISE B ALANCING
Revised: B.2.1. C OROLLARY OF T HEOREM 1: L AYER - WISE B ALANCING

Original: As indicated in Theorem 1, for the minimum sum of squared weight representation, the ℓ 2 norm of the input vector and output vector of the unit are the same.
Revised: As indicated in Theorem 1, for the minimum sum of squared weight representation, the ℓ 2 norm of the input vector and output vector of the unit are the same.

Original: Thus for the coupling of ( j − 1 , j ) -th layer, we have
Revised: Thus for the coupling of ( j − 1 , j ) -th layer, we have

Original: [Equation]
Revised: [Equation]

Original: and for the coupling of ( j, j + 1) -th layer, we have
Revised: and for the coupling of ( j, j + 1) -th layer, we have

Original: [Equation]
Revised: [Equation]

Original: which indicate the j − 1 , j and j + 1 -th layer have the same amount of sum of squared weights.
Revised: which indicate the j − 1 , j and j + 1 -th layer have the same amount of sum of squared weights.

Original: Since j is arbitrary, it is easily verified that at the minimum norm representation, each layer share the same amount of sum of squared weights.
Revised: Since j is arbitrary, it is easily verified that at the minimum norm representation, each layer share the same amount of sum of squared weights.

Original: Now consider the ( j, j + 1) -th and ( k, k + 1) -th coupling layer, we have
Revised: Now consider the ( j, j + 1) -th and ( k, k + 1) -th coupling layer, we have

Original: [Equation]
Revised: [Equation]

Original: Again, as in Theorem 1 indicate, at minimum norm representation, we have ∥ w ∥ 2 = ∥ v ∥ 2 , therefore n j (cid:88)
Revised: Again, as in Theorem 1 indicate, at minimum norm representation, we have ∥ w ∥ 2 = ∥ v ∥ 2 , therefore n j (cid:88)

Original: [Equation]
Revised: [Equation]

Original: So the sum of the ℓ 2 -P ATH -N ORM per coupling of layers are the same for the minimum norm solution.
Revised: So the sum of the ℓ 2 -P ATH -N ORM per coupling of layers are the same for the minimum norm solution.

Original: Our proposed proximal algorithm doesn’t naturally enforce this, so we will apply the layer-wise balance procedure along the proximal algorithm, as indicated in the next section.
Revised: Our proposed proximal algorithm doesn’t naturally enforce this, so we will apply the layer-wise balance procedure along the proximal algorithm, as indicated in the next section.

Original: B.2.2. L AYER - WISE B ALANCE A LGORITHM
Revised: B.2.2. L AYER - WISE B ALANCE A LGORITHM

Original: In Algorithm 1, we consider bias term as part of the weight parameter, and thus calculate ℓ 2 -P ATHN ORM with the bias term added.
Revised: In Algorithm 1, we consider bias term as part of the weight parameter, and thus calculate ℓ 2 -P ATHN ORM with the bias term added.

Original: However to apply layer-wise balance, we will consider only regularizing the weight parameter, and leave the bias term for standard weight decaying.
Revised: However to apply layer-wise balance, we will consider only regularizing the weight parameter, and leave the bias term for standard weight decaying.

Original: Let (cid:102) W denote the matrix of W with its first column removed, W i and W : ,i denote the i -th row and column of W , respectively.
Revised: Let (cid:102) W denote the matrix of W with its first column removed, W i and W : ,i denote the i -th row and column of W , respectively.

Original: For every group of layers j ∈ { 1 , 2 , · · · , ⌊ L 2 ⌋} , we will obtain equality in the following term
Revised: For every group of layers j ∈ { 1 , 2 , · · · , ⌊ L 2 ⌋} , we will obtain equality in the following term

Original: [Equation]
Revised: [Equation]

Original: through the layer-wise balance procedure as indicated in Algorithm 2, and make sure the following function is equivalent after adjustment:
Revised: through the layer-wise balance procedure as indicated in Algorithm 2, and make sure the following function is equivalent after adjustment:

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: where h 2 j − 1 ∈ R n 2 j − 1 is the input to the (2 j − 1)
Revised: where h 2 j − 1 ∈ R n 2 j − 1 is the input to the (2 j − 1)

Original: -th layer, b 2 j − 1 =
Revised: -th layer, b 2 j − 1 =

Original: W 2 j − 1 : ,, b 2 j = W 2 j : , 1 and b 

Revised: W 2 j − 1 : ,, b 2 j = W 2 j : , 1 and b

Original: j +1 =
Revised: j +1 =

Original: W 2 j +1 : ,are the bias terms for (2 j − 1) -th, 2 j -th and (2 j + 1) -th layer respectively.
Revised: W 2 j +1 : ,are the bias terms for (2 j − 1) -th, 2 j -th and (2 j + 1) -th layer respectively.

Original: In practice, this layer-wise balance is processed after doing batch gradient update on the homogeneous output weights, and before applying proximal operator.
Revised: In practice, this layer-wise balance is processed after doing batch gradient update on the homogeneous output weights, and before applying proximal operator.

Original: For CNN backbone network, similar layer-wise balance procedure is applied on every group of the layers with proper defined ℓ 2 -P ATH -N ORM norm as indicated in Sec.
Revised: For CNN backbone network, similar layer-wise balance procedure is applied on every group of the layers with proper defined ℓ 2 -P ATH -N ORM norm as indicated in Sec.

Original: A.2.
Revised: A.2.

Original: Since the convolution operator is piece-wise linear in weights, the layer-wise balancing procedure still holds for adjusting the 4D weight parameter in CNN.
Revised: Since the convolution operator is piece-wise linear in weights, the layer-wise balancing procedure still holds for adjusting the 4D weight parameter in CNN.

Original: Note: After the proximal step for homogeneous input weights, we always have (cid:13)(cid:13)(cid:13)(cid:102) W 2 j − 1 i (cid:13)(cid:13)(cid:13) 2 = 1 , thus this term is omitted in the calculation of ℓ 2 -P ATH -N ORM norm below.
Revised: Note: After the proximal step for homogeneous input weights, we always have (cid:13)(cid:13)(cid:13)(cid:102) W 2 j − 1 i (cid:13)(cid:13)(cid:13) 2 = 1 , thus this term is omitted in the calculation of ℓ 2 -P ATH -N ORM norm below.

Original: Calculated the geometric mean of the ℓ 2 -P ATH -N ORM norm of each layer:
Revised: Calculated the geometric mean of the ℓ 2 -P ATH -N ORM norm of each layer:

Original: [Equation]
Revised: [Equation]

Original: (cid:13)(cid:13)(cid:13)(cid:102) W
Revised: (cid:13)(cid:13)(cid:13)(cid:102) W

Original: [Equation]
Revised: [Equation]

Original: (cid:13)(cid:13)(cid:13)(cid:33) for j = 1 , 2 , · · · , ⌊ L 2 ⌋ do
Revised: (cid:13)(cid:13)(cid:13)(cid:33) for j = 1 , 2 , · · · , ⌊ L 2 ⌋ do

Original: Calculate the scale α applied to layer 2 j : α j = X/ (cid:13)(cid:13)(cid:13)(cid:102) W 2 j (cid:13)(cid:13)(cid:13)
Revised: Calculate the scale α applied to layer 2 j : α j = X/ (cid:13)(cid:13)(cid:13)(cid:102) W 2 j (cid:13)(cid:13)(cid:13)

Original: Update the output weight: (cid:102) W 2 j ← α j · (cid:102) W 2 j , where · is applied in an element-wise manner.
Revised: Update the output weight: (cid:102) W 2 j ← α j · (cid:102) W 2 j , where · is applied in an element-wise manner.

Original: For the following block of function:
Revised: For the following block of function:

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: Adjust other parameter to maintain the equivalence of the neural network function:
Revised: Adjust other parameter to maintain the equivalence of the neural network function:

Original: [Equation]
Revised: [Equation]

Original: end for
Revised: end for

Original: Appendix C. Supplementary of Experiments C.1. Experiment Setup
Revised: Appendix C. Supplementary of Experiments C.1. Experiment Setup

Original: In this section we will describe the dataset and network we use in our experiments.
Revised: In this section we will describe the dataset and network we use in our experiments.

Original: All our experiments are conducted on Nvidia 3090 GPUs.
Revised: All our experiments are conducted on Nvidia 3090 GPUs.

Original: C.1.1. D ATASETS
Revised: C.1.1. D ATASETS

Original: In this work, we demonstrate the performance of the proposed proximal algorithm for ℓ 2 -P ATHN ORM N ORM on the following dataset:
Revised: In this work, we demonstrate the performance of the proposed proximal algorithm for ℓ 2 -P ATHN ORM N ORM on the following dataset:

Original: a) MNIST [18] consist of 10 classes of hand-written digits, each class has 6000 training images and 1000 test images.
Revised: a) MNIST [19] consist of 10 classes of hand-written digits, each class has 6000 training images and 1000 test images.

Original: When training, we randomly split the dataset into 55000 training data, and 5000 validation data.
Revised: When training, we randomly split the dataset into 55000 training data, and 5000 validation data.

Original: We use the validation data to decide the hyper-parameters.
Revised: We use the validation data to decide the hyper-parameters.

Original: Each image has shape 28 × 28 .
Revised: Each image has shape 28 × 28 .

Original: When training, we normalize the input data.
Revised: When training, we normalize the input data.

Original: b) MNIST-C [21] is a robustness benchmark on MNIST dataset, which applies 15 standard corruption to the MNIST dataset, namely 1) shot noise, 2) impluse noise, 3) glass blur, 4) motion blur, 5) shear, 6) scale, 7) rotate, 8) brightness, 9) translate, 10) stripe, 11) fog, 12) spatter, 13) dotted line, 14) zigzag, and 15) canny edge.
Revised: b) MNIST-C [22] is a robustness benchmark on MNIST dataset, which applies 15 standard corruption to the MNIST dataset, namely 1) shot noise, 2) impluse noise, 3) glass blur, 4) motion blur, 5) shear, 6) scale, 7) rotate, 8) brightness, 9) translate, 10) stripe, 11) fog, 12) spatter, 13) dotted line, 14) zigzag, and 15) canny edge.

Original: When validate our result on MNIST-C dataset, we train on clean MNIST dataset, and pick the best model based on clean validation set, then test its performance on the corrupted MNIST-C dataset to measure the generalization.
Revised: When validate our result on MNIST-C dataset, we train on clean MNIST dataset, and pick the best model based on clean validation set, then test its performance on the corrupted MNIST-C dataset to measure the generalization.

Original: c) CIFAR

Revised: c) CIFAR

Original: [15] has 10 classes of real images.
Revised: [16] has 10 classes of real images.

Original: Each class has 5000 training images andtest images.
Revised: Each class has 5000 training images andtest images.

Original: When training, we randomly split 45000 images for training, and 5000 images for validation.
Revised: When training, we randomly split 45000 images for training, and 5000 images for validation.

Original: Each image has shape 32 × 32 .
Revised: Each image has shape 32 × 32 .

Original: When training, we random crop, random horizontal flip, and normalize the input data.
Revised: When training, we random crop, random horizontal flip, and normalize the input data.

Original: d) CIFAR10-C [12] is a robustness benchmark on CIFAR10 dataset, which applies 18 standard corruption to the CIFAR10 dataset, namely 1) Gaussian noise, 2) shot noise, 3) impluse noise, 4) defocus blur,
Revised: d) CIFAR10-C [13] is a robustness benchmark on CIFAR10 dataset, which applies 18 standard corruption to the CIFAR10 dataset, namely 1) Gaussian noise, 2) shot noise, 3) impluse noise, 4) defocus blur,

Original: 5) frosted glass blur, 6) motion blur, 7) zoom blur, 8) snow, 9) frost, 10) fog, 11) brightness, 12) contrast, 13) elastic, 14) pixelate, and 15) JPEG.
Revised: 5) frosted glass blur, 6) motion blur, 7) zoom blur, 8) snow, 9) frost, 10) fog, 11) brightness, 12) contrast, 13) elastic, 14) pixelate, and 15) JPEG.

Original: When validate our result on
Revised: When validate our result on

Original: CIFAR10-C dataset, we train on clean CIFAR10 dataset, and pick the best model based on clean validation set, then test its performnce on the corrupted CIFAR10-C dataset.
Revised: CIFAR10-C dataset, we train on clean CIFAR10 dataset, and pick the best model based on clean validation set, then test its performnce on the corrupted CIFAR10-C dataset.

Revised: e) SVHN [23] is the Street View House Numbers dataset, with 73257 digits for training,digits for testing.

Revised: We randomly split 67257 images for training, and 6000 images for validation.

Revised: Each image has shape 32 × 32 .

Revised: When training, we normalize the input data.

Revised: We didn’t use the additional dataset to boost the performance.

Original: C.1.2. M ODELS
Revised: C.1.2. M ODELS

Original: In this work, we demonstrate the performance of the proposed proximal algorithm for ℓ 2 -P ATHN ORM N ORM on the following models:
Revised: In this work, we demonstrate the performance of the proposed proximal algorithm for ℓ 2 -P ATHN ORM N ORM on the following models:

Original: a) MLP- d - n model consist of d fully connected layers, each with n neurons.
Revised: a) MLP- d - n model consist of d fully connected layers, each with n neurons.

Original: Details of the MLP- d - n architecture is shown in Table 2.
Revised: Details of the MLP- d - n architecture is shown in Table 2.

Original: b) MLP- d - n factorized model consist of d fully connected layers, each with n neurons.
Revised: b) MLP- d - n factorized model consist of d fully connected layers, each with n neurons.

Original: For each layer, we factorize it into two linear layers, with hidden neurons to be n as well.
Revised: For each layer, we factorize it into two linear layers, with hidden neurons to be n as well.

Original: Details of the MLP- d - n factorized architecture is shown in Table 2.
Revised: Details of the MLP- d - n factorized architecture is shown in Table 2.

Original: c) VGG19 is introduced in [29], which is widely used for computer vision task.
Revised: c) VGG19 is introduced in [34], which is widely used for computer vision task.

Original: Instead offully-connected layer as the classifier, to apply VGG19 on CIFAR10, we use 1 fully-connected layer instead 1 .
Revised: Instead offully-connected layer as the classifier, to apply VGG19 on CIFAR10, we use 1 fully-connected layer instead 1 .

Original: There are 16 convolutional layers.
Revised: There are 16 convolutional layers.

Original: After every two or four convolutional layers, there follows a max-pooling layer to reduce the feature map size by half.
Revised: After every two or four convolutional layers, there follows a max-pooling layer to reduce the feature map size by half.

Original: In Sec. 5.1, we demonstrate the generalization result of weight decay and proximal algorithm for ∥ w ∥ 2 ∥ v ∥ 2 on:
Revised: In Sec. 5.1, we demonstrate the generalization result of weight decay and proximal algorithm for ∥ w ∥ 2 ∥ v ∥ 2 on:

Original: (Task 1) MNIST subset on MLP-3-400 factorized (Task 2) MNIST on MLP-6- (Task 3) CIFAR10 on VGG 
Revised: (Task 1) MNIST subset on MLP-3-400 factorized (Task 2) MNIST on MLP-6- (Task 3) CIFAR10 on VGG (Task 4) SVHN on VGG

Original: In Sec.
Revised: In Sec.

Original: C.3, we evaluate the ability to obtain sparse solution of weight decay, lasso and group lasso on
Revised: C.3, we evaluate the ability to obtain sparse solution of weight decay, lasso and group lasso on

Original: (Task 4) MNIST on MLP-3-
Revised: (Task 5) MNIST on MLP-3-

Original: When evaluating the proximal algorithm for ∥ w ∥ 2 ∥ v ∥ 2 , we need to couple the layers into groups.
Revised: When evaluating the proximal algorithm for ∥ w ∥ 2 ∥ v ∥ 2 , we need to couple the layers into groups.

Original: We choose to factorize it, and thus evaluate the algorithm on:
Revised: We choose to factorize it, and thus evaluate the algorithm on:

Original: (Task 4) MNIST on MLP-3-800 factorized
Revised: (Task 5) MNIST on MLP-3-800 factorized

Original: [Table]
Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Original: C.
Revised: [Figure] [Table] [Figure] [Table]

Original: H
Revised: [Figure] [Table] [Figure] [Table]

Original: YPER
Revised: [Figure] [Table] [Figure] [Table]

Original: P
Revised: [Figure] [Table] [Figure] [Table]

Original: ARAMETER
Revised: [Figure] [Table] [Figure] [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Original:  C HOICE
Revised: [Table] [Figure] C.1.3. H YPER P ARAMETER C HOICE

Original: For generalization experiments in Sec. 5.1, we did grid search for the learning rate as well as the λ as follows:
Revised: For generalization experiments in Sec. 5.1, we did grid search for the learning rate as well as the λ as follows:

Original: [Equation] [Equation]
Revised: (Task 1) & (Task 2)

Original: • learning rate : 0.003, 0.01, 0.03, 0.1, 0.3, 0.
Revised: • learning rate : 0.003, 0.01, 0.03, 0.1, 0.3, 0.

Original: • λ : 0.00001, 0.00003, 0.0001, 0.0003, 0.001, 0.003, 0.
Revised: • λ : 0.00001, 0.00003, 0.0001, 0.0003, 0.001, 0.003, 0.

Original: (Task 3) : • learning rate : 0.01, 0.03, 0.1, 0.
Revised: (Task 3) : • learning rate : 0.01, 0.03, 0.1, 0.

Original: • λ : 0.00001, 0.00003, 0.0001, 0.0003, 0.001, 0.003, 0.
Revised: • λ : 0.00001, 0.00003, 0.0001, 0.0003, 0.001, 0.003, 0.

Revised: (Task 4) : • learning rate : 0.03, 0.1, 0.3, 0.

Revised: • λ : 0.0001, 0.0003, 0.001, 0.

Original: (Task 4) : • learning rate : 0.01, 0.
Revised: (Task 5) : • learning rate : 0.01, 0.

Original: • λ : 0.0001, 0.001, 0.003, 0.
Revised: • λ : 0.0001, 0.001, 0.003, 0.

Original: C.1.4. S TANDARD E RROR C ALCULATION
Revised: C.1.4. S TANDARD E RROR C ALCULATION

Original: For each task, we run the grid search experiments on random seed 42 , and pick the set of hyper parameter ( learning rate ∗ , λ ∗ ) based on the best validation accuracy.
Revised: For each task, we run the grid search experiments on random seed 42 , and pick the set of hyper parameter ( learning rate ∗ , λ ∗ ) based on the best validation accuracy.

Original: Then for each task, we run with ( learning rate ∗ , λ ∗ ) for another three times, with random seed 43 , 44 , and 45 .
Revised: Then for each task, we run with ( learning rate ∗ , λ ∗ ) for another three times, with random seed 43 , 44 , and 45 .

Original: Again in each run, the test accuracy is picked based on the best validation accuracy.
Revised: Again in each run, the test accuracy is picked based on the best validation accuracy.

Original: With four runs, we evaluate the mean and standard error of the experiments, and present the result in the table.
Revised: With four runs, we evaluate the mean and standard error of the experiments, and present the result in the table.

Original: C.2. Experiments Details for Figure 1 and
Revised: C.2. Experiments Details for Figure 1 and

Original: Left figure in Fig.
Revised: Left figure in Fig.

Original: For the convergence figure on the left, we evaluate on the (Task 1) with learning rate = 0 . 01 , and λ = 0 . 0001 .
Revised: For the convergence figure on the left, we evaluate on the (Task 1) with learning rate = 0 . 01 , and λ = 0 . 0001 .

Original: With same λ and learning rate , the proximal algorithm for ∥ w ∥ 2 ∥ v ∥ 2 finds better solution faster, and minimize the weight decay objective faster.
Revised: With same λ and learning rate , the proximal algorithm for ∥ w ∥ 2 ∥ v ∥ 2 finds better solution faster, and minimize the weight decay objective faster.

Original: Middle figure in Fig.
Revised: Middle figure in Fig.

Original: For the histogram in the middle, we evaluate on the (Task 2) .
Revised: For the histogram in the middle, we evaluate on the (Task 2) .

Original: The models we investigate have the best validation accuracy with learning rate = 0 .
Revised: The models we investigate have the best validation accuracy with learning rate = 0 .

Original: 1 and λ = 0 .
Revised: 1 and λ = 0 .

Original: To measure the local Lipschitz constant for the unseen data, we compare the spectral norm of the
Revised: To measure the local Lipschitz constant for the unseen data, we compare the spectral norm of the

Original: Jacobian (of the model with respect to the input) on all 10,000 MNIST test samples, and plot their values in the histogram shown.
Revised: Jacobian (of the model with respect to the input) on all 10,000 MNIST test samples, and plot their values in the histogram shown.

Original: We see that for the model trained with weight decay the spectral norm of the Jacobian is generally larger than the model trained with our approach, indicating that the proximal ∥ w ∥ 2 ∥ v ∥ 2 regularization leads to models with a lower local Lipschitz constant and therefore more robust.
Revised: We see that for the model trained with weight decay the spectral norm of the Jacobian is generally larger than the model trained with our approach, indicating that the proximal ∥ w ∥ 2 ∥ v ∥ 2 regularization leads to models with a lower local Lipschitz constant and therefore more robust.

Original: Right figure in Fig.
Revised: Right figure in Fig.

Original: For the sparsity figure on the right, we do experiment on a synthetic dataset with spatial variants.
Revised: For the sparsity figure on the right, we do experiment on a synthetic dataset with spatial variants.

Original: To generate the dataset, we sample in uniform randomness 64 data point from the following function:
Revised: To generate the dataset, we sample in uniform randomness 64 data point from the following function:

Original: [Equation]
Revised: [Equation]

Original: which has variant spatial frequency.
Revised: which has variant spatial frequency.

Original: The model used to fit the data is an one-hidden layer fullyfeedforward network, with 640 neurons, input dimension 1, and output dimension 1.
Revised: The model used to fit the data is an one-hidden layer fullyfeedforward network, with 640 neurons, input dimension 1, and output dimension 1.

Original: We train with weight decay, as well as ∥ w ∥ 2 ∥ v ∥ 2 proximal algorithm.
Revised: We train with weight decay, as well as ∥ w ∥ 2 ∥ v ∥ 2 proximal algorithm.

Original: Consider i -th neuron with w i ∈ R input and v i ∈ R output, we count an neuron to be active if | w i
Revised: Consider i -th neuron with w i ∈ R input and v i ∈ R output, we count an neuron to be active if | w i

Original: ||
Revised: ||

Original: v i | > 0 .
Revised: v i | > 0 .

Original: With this, we obtain the number of active neurons during the training time, and plot the figure as shown.
Revised: With this, we obtain the number of active neurons during the training time, and plot the figure as shown.

Original: From the plot, we can see weight decay can also sparsify the model, though with much slower speed compared to the ∥ w ∥ 2 ∥ v ∥ 2 proximal algorithm.
Revised: From the plot, we can see weight decay can also sparsify the model, though with much slower speed compared to the ∥ w ∥ 2 ∥ v ∥ 2 proximal algorithm.

Original: The decision boundary figure is generated on the training task of fitting a 1-hidden-layer shallow network to the data points presented in the figure.
Revised: The decision boundary figure is generated on the training task of fitting a 1-hidden-layer shallow network to the data points presented in the figure.

Original: Fix the weight decay parameter λ to be 0.0001 and learning rate to be 0.1, we run both algorithm for same amount of iterations (till converge).
Revised: Fix the weight decay parameter λ to be 0.0001 and learning rate to be 0.1, we run both algorithm for same amount of iterations (till converge).

Original: The choice of learning rate and weight decay parameter is arbitrary.
Revised: The choice of learning rate and weight decay parameter is arbitrary.

Original: In general, we found that with same weight decay parameter (the same objective), larger learning rate will see the model getting over-thresholding (for both algorithm), and thus turning sparse.
Revised: In general, we found that with same weight decay parameter (the same objective), larger learning rate will see the model getting over-thresholding (for both algorithm), and thus turning sparse.

Original: The sparse structure may degrade the performance.
Revised: The sparse structure may degrade the performance.

Original: Many approaches [2, 33] adapt learning rate in proximal gradient algorithm, which is a good direction for the future work.
Revised: Many approaches [2, 39] adapt learning rate in proximal gradient algorithm, which is a good direction for the future work.

Original: C.3. Proximal Algorithm Finds Structural Sparse Solution Faster
Revised: C.3. Proximal Algorithm Finds Structural Sparse Solution Faster

Original: Weight decay will eventually find sparse solution, however with an inordinate number of training steps as indicate in Fig.
Revised: Weight decay will eventually find sparse solution, however with an inordinate number of training steps as indicate in Fig.

Original: 1 ( right ).
Revised: 1 ( right ).

Original: Proximal algorithm helps accelerating the training process.
Revised: Proximal algorithm helps accelerating the training process.

Original: To demonstrate this, we perform experiments on (Task 4) , and compare the sparse solution found by 1) weight decay, 2) proximal algorithm for ℓ 2 -P ATH -N ORM ; as well as well-known regularizations that enforce sparsity in the solution: 3) lasso [31], and 4) group lasso [34].
Revised: To demonstrate this, we perform experiments on (Task 5) , and compare the sparse solution found by 1) weight decay, 2) proximal algorithm for ℓ 2 -P ATH -N ORM ; as well as well-known regularizations that enforce sparsity in the solution: 3) lasso [36], and 4) group lasso [40].

Original: As the proximal algorithm operates on each one homogeneous units (instead of weight parameter), in this section we mainly focus on structural sparsity of the model.
Revised: As the proximal algorithm operates on each one homogeneous units (instead of weight parameter), in this section we mainly focus on structural sparsity of the model.

Original: [Figure]

Original: For each experiments, we first prune the model in each training step (prune one unit if either its input vector w or output vector v is zeroed out).
Revised: For each experiments, we first prune the model in each training step (prune one unit if either its input vector w or output vector v is zeroed out).

Original: After training and pruning for 30000 iterations, we take the model checkpoint from iterations { 5000 , 10000 , · · · , 30000 } , set the unit to be inactive if ∥ w ∥ 2 ∥ v ∥ 2 < 10 − 5 , and then train this sparse model for another 10000 iterations.
Revised: After training and pruning for 30000 iterations, we take the model checkpoint from iterations { 5000 , 10000 , · · · , 30000 } , set the unit to be inactive if ∥ w ∥ 2 ∥ v ∥ 2 < 10 − 5 , and then train this sparse model for another 10000 iterations.

Original: Since different λ and learning rate may lead to different level of sparsity, we try with learning rate in { 0 . 01 , 0 . 03 } , and λ ∈ { 0 . 0001 , 0 . 001 , 0 . 003 , 0 . 01 } for all method.
Revised: Since different λ and learning rate may lead to different level of sparsity, we try with learning rate in { 0 . 01 , 0 . 03 } , and λ ∈ { 0 . 0001 , 0 . 001 , 0 . 003 , 0 . 01 } for all method.

Original: Performance of the sparse solution is presented in Fig.
Revised: Performance of the sparse solution is presented in Fig.

Original: For each sparsity level s , we present the best solution with sparsity < s .
Revised: For each sparsity level s , we present the best solution with sparsity < s .

Original: Compared with proximal algorithm for ℓ 2 -P ATH -N ORM , weight  decay only finds solution of sparsity > 64% .
Revised: Compared with proximal algorithm for ℓ 2 -P ATH -N ORM , weight [Figure] decay only finds solution of sparsity > 64% .

Original: Though the proximal algorithm doesn’t enforce sparsity, it finds competitive solution as lasso, and outperform group lasso in the sparse regime.
Revised: Though the proximal algorithm doesn’t enforce sparsity, it finds competitive solution as lasso, and outperform group lasso in the sparse regime.