Original: A PAC-B AYESIAN A PPROACH TO S PECTRALLY -N ORMALIZED M ARGIN B OUNDS FOR N EURAL N ETWORKS
Revised: A PAC-B AYESIAN A PPROACH TO S PECTRALLY -N ORMALIZED M ARGIN B OUNDS FOR N EURAL N ETWORKS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We present a generalization bound for feedforward neural networks with ReLU activations in terms of the product of the spectral norm of the layers and the
Revised: We present a generalization bound for feedforward neural networks with ReLU activations in terms of the product of the spectral norm of the layers and the

Original: Frobenius norm of the weights.
Revised: Frobenius norm of the weights.

Original: The key ingredient is a bound on the changes in the output of a network with respect to perturbation of its weights, thereby bounding the sharpness of the network.
Revised: The key ingredient is a bound on the changes in the output of a network with respect to perturbation of its weights, thereby bounding the sharpness of the network.

Original: We combine this perturbation bound with the PAC-Bayes analysis to derive the generalization bound.
Revised: We combine this perturbation bound with the PAC-Bayes analysis to derive the generalization bound

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Learning with deep neural networks has enjoyed great success across a wide variety of tasks.
Revised: Learning with deep neural networks has enjoyed great success across a wide variety of tasks.

Original: Even though learning neural networks is a hard problem, even for one hidden layer (Blum & Rivest, 1993), simple optimization methods such as stochastic gradient descent (SGD) have been able to minimize the training error.
Revised: Even though learning neural networks is a hard problem, even for one hidden layer (Blum & Rivest, 1993), simple optimization methods such as stochastic gradient descent (SGD) have been able to minimize the training error.

Original: More surprisingly, solutions found this way also have small test error, even though the networks used have more parameters than the number of training samples, and have the capacity to easily ﬁt random labels (Zhang et al., 2017).
Revised: More surprisingly, solutions found this way also have small test error, even though the networks used have more parameters than the number of training samples, and have the capacity to easily ﬁt random labels (Zhang et al., 2017).

Original: Harvey et al.
Revised: Harvey et al.

Original: (2017) provided a generalization error bound by showing that the VC dimension of d-layer networks is depth times the number parameters improving over earlier bounds by Bartlett et al.
Revised: (2017) provided a generalization error bound by showing that the VC dimension of d-layer networks is depth times the number parameters improving over earlier bounds by Bartlett et al.

Original: Such VC bounds which depend on the number of parameters of the networks, cannot explain the generalization behavior in the over parametrized settings, where the number of samples is much smaller than the number of parameters.
Revised: Such VC bounds which depend on the number of parameters of the networks, cannot explain the generalization behavior in the over parametrized settings, where the number of samples is much smaller than the number of parameters.

Original: For linear classiﬁers we know that the generalization behavior depends on the norm and margin of the classiﬁer and not on the actual number of parameters.
Revised: For linear classiﬁers we know that the generalization behavior depends on the norm and margin of the classiﬁer and not on the actual number of parameters.

Original: Hence, a generalization bound for neural networks that only depends on the norms of its layers, and not the actual number of parameters, can explain the good generalization behavior of the over parametrized networks.
Revised: Hence, a generalization bound for neural networks that only depends on the norms of its layers, and not the actual number of parameters, can explain the good generalization behavior of the over parametrized networks.

Original: Bartlett & Mendelson (2002) showed generalization bounds for feedforward networks in terms of unit-wise (cid:96) 1 norm with exponential dependence on depth.
Revised: Bartlett & Mendelson (2002) showed generalization bounds for feedforward networks in terms of unit-wise (cid:96) 1 norm with exponential dependence on depth.

Original: In a more recent work, Neyshabur et al.
Revised: In a more recent work, Neyshabur et al.

Original: (2015a) provided generalization bounds for general class of group norms including Frobenius norm with same exponential dependence on the depth and showed that the exponential dependence is unavoidable in the worst case.
Revised: (2015a) provided generalization bounds for general class of group norms including Frobenius norm with same exponential dependence on the depth and showed that the exponential dependence is unavoidable in the worst case.

Original: Bartlett et al.
Revised: Bartlett et al.

Original: (2017) showed a margin based generalization bound that depends on spectral norm and (cid:96) 1 norm of the layers of the networks.
Revised: (2017a) showed a margin based generalization bound that depends on spectral norm and (cid:96) 1 norm of the layers of the networks.

Original: They show this bound using a complex covering number argument.
Revised: They show this bound using a complex covering number argument.

Original: This bound does not depend directly on the number of parameters of the network but depends on the norms of its layers.
Revised: This bound does not depend directly on the number of parameters of the network but depends on the norms of its layers.

Original: However the (cid:96) 1 term has high dependence on the number of hidden units if the weights are dense, which is typically the case for modern deep learning architectures.
Revised: However the (cid:96) 1 term has high dependence on the number of hidden units if the weights are dense, which is typically the case for modern deep learning architectures.

Original: Keskar et al.
Revised: Keskar et al.

Original: (2016) suggested a sharpness based measure to predict the difference in generalization behavior of networks trained with different batch size SGD.
Revised: (2016) suggested a sharpness based measure to predict the difference in generalization behavior of networks trained with different batch size SGD.

Original: However sharpness is not a scale invariant measure and cannot predict the generalization behavior (Neyshabur et al., 2017).
Revised: However sharpness is not a scale invariant measure and cannot predict the generalization behavior (Neyshabur et al., 2017).

Original: Instead sharpness when combined with the norms of the network can predict the generalization behavior according to the PAC-Bayes framework (McAllester, 1999).
Revised: Instead sharpness when combined with the norms of the network can predict the generalization behavior according to the PAC-Bayes framework (McAllester, 1999).

Original: Dziugaite & Roy (2017) numerically evaluated a generalization error bound from the PAC-Bayes framework showing that it can predict the difference in generalization behavior of networks trained on true vs random labels.
Revised: Dziugaite & Roy (2017) numerically evaluated a generalization error bound from the PAC-Bayes framework showing that it can predict the difference in generalization behavior of networks trained on true vs random labels.

Revised: This generalization result is more of computational in nature and gives much tighter (non-vacuous) bounds compared with the ones in either (Bartlett et al., 2017a) or the ones in this paper.

Original: In this paper we present and prove a margin based generalization bound for feedforward neural networks with ReLU activations, that depends on the product of the spectral norm of the weights in each layer, as well as the Frobenius norm of the weights.
Revised: In this paper we present and prove a margin based generalization bound for feedforward neural networks with ReLU activations, that depends on the product of the spectral norm of the weights in each layer, as well as the Frobenius norm of the weights.

Original: Our generalization bound shares much similarity with a margin based generalization bound recently presented by Bartlett et al. 
Revised: Our generalization bound shares much similarity with a margin based generalization bound recently presented by Bartlett et al. (2017a).

Original: Both bounds depend similarly on the product of the spectral norms of each layer, multiplied by a factor that is additive across layers.
Revised: Both bounds depend similarly on the product of the spectral norms of each layer, multiplied by a factor that is additive across layers.

Original: In addition, Bartlett et al.
Revised: In addition, Bartlett et al.

Original: (2017) bound depends on the elementwise (cid:96) 1 -norm of the weights in each layer, while our bound depends on the Frobenius (elementwise (cid:96) 2 ) norm of the weights in each layer, with an additional multiplicative dependence on the “width”.
Revised: (2017a) bound depends on the elementwise (cid:96) 1 -norm of the weights in each layer, while our bound depends on the Frobenius (elementwise (cid:96) 2 ) norm of the weights in each layer, with an additional multiplicative dependence on the “width”.

Original: The two bounds are thus not directly comparable, and as we discuss in Section 3, each one dominates in a different regime, roughly depending on the sparsity of the weights.
Revised: The two bounds are thus not directly comparable, and as we discuss in Section 3, each one dominates in a different regime, roughly depending on the sparsity of the weights.

Original: We also discuss in what regimes each of these bounds could dominate a VC-bound based on the overall number of weights.
Revised: We also discuss in what regimes each of these bounds could dominate a VC-bound based on the overall number of weights.

Original: More importantly, our proof technique is entirely different, and arguably simpler, than that of Bartlett et al.
Revised: More importantly, our proof technique is entirely different, and arguably simpler, than that of Bartlett et al.

Original:  We derive our bound using PAC-Bayes analysis, and more speciﬁcally a generic PACBayes margin bound (Lemma 1).
Revised: (2017a). We derive our bound using PAC-Bayes analysis, and more speciﬁcally a generic PACBayes margin bound (Lemma 1).

Original: The main ingredient is a perturbation bound (Lemma 2), bounding the changes in the output of a network when the weights are perturbed, thereby its sharpness, in terms of the product of the spectral norm of the layers.
Revised: The main ingredient is a perturbation bound (Lemma 2), bounding the changes in the output of a network when the weights are perturbed, thereby its sharpness, in terms of the product of the spectral norm of the layers.

Original: This is an entirely different analysis approach from the covering number analysis of Bartlett et al.
Revised: This is an entirely different analysis approach from the covering number analysis of Bartlett et al.

Original:  We hope our analysis can give more direct intuition into the different ingredients in the bound and will allow modifying the analysis, e.g. by using different prior and perturbation distributions in the PAC-Bayes bound, to obtain tighter bounds, perhaps with dependence on different layer-wise norms.
Revised: (2017a). We hope our analysis can give more direct intuition into the different ingredients in the bound and will allow modifying the analysis, e.g. by using different prior and perturbation distributions in the PAC-Bayes bound, to obtain tighter bounds, perhaps with dependence on different layer-wise norms.

Original: We note that  prior bounds in terms of elementwise or unit-wise norms (such as the Frobenius norm and elementwise (cid:96) 1 norms of layers), without a spectral norm dependence, all have a multiplicative dependence across layers or exponential dependence on depth (Bartlett & Mendelson, 2002;
Revised: We note that other prior bounds in terms of elementwise or unit-wise norms (such as the Frobenius norm and elementwise (cid:96) 1 norms of layers), without a spectral norm dependence, all have a multiplicative dependence across layers or exponential dependence on depth (Bartlett & Mendelson, 2002;

Original: Neyshabur et al., 2015b), or are for constant depth networks (Bartlett, 1998).
Revised: Neyshabur et al., 2015b), or are for constant depth networks (Bartlett, 1998).

Original: Here only the spectral norm is multiplied across layers, and thus if the spectral norms are close to one, the exponential dependence on depth can be avoided.
Revised: Here only the spectral norm is multiplied across layers, and thus if the spectral norms are close to one, the exponential dependence on depth can be avoided.

Revised: After the initial preprint of this paper, Bartlett et al.

Revised: (2017b) presented an improved bound that replaces the dependency on (cid:96) 1 norm of the layers (Bartlett et al., 2017a) with (cid:96) 2 , 1 norm.

Revised: This new generalization bound is strictly better than our Frobenius norm bound, and improves over existing results.

Original: 1. P RELIMINARIES
Revised: 1. P RELIMINARIES

Original: Consider the classiﬁcation task with input domain X B,n = (cid:8) x ∈ R n | | (cid:80) ni
Revised: Consider the classiﬁcation task with input domain X B,n = (cid:8) x ∈ R n | | (cid:80) ni

Original: =1 x 2 i ≤ B 2 (cid:9) and output domain R k where the output of the model is a score for each class and the class with the maximum score will be selected as the predicted label.
Revised: =1 x 2 i ≤ B 2 (cid:9) and output domain R k where the output of the model is a score for each class and the class with the maximum score will be selected as the predicted label.

Original: Let f w ( x ) : X B,n → R k be the function computed by a d layer feed-forward network for the classiﬁcation task with parameters w = vec (cid:0) { W i } di =1 (cid:1) , f w ( x ) =
Revised: Let f w ( x ) : X B,n → R k be the function computed by a d layer feed-forward network for the classiﬁcation task with parameters w = vec (cid:0) { W i } di =1 (cid:1) , f w ( x ) =

Original: W d φ ( W d − 1 φ ( ....φ ( W 1 x ))) , here φ is the ReLU activation function.
Revised: W d φ ( W d − 1 φ ( ....φ ( W 1 x ))) , here φ is the ReLU activation function.

Original: Let f i w ( x ) denote the output of layer i before activation and h be an upper bound on the number of output units in each layer.
Revised: Let f i w ( x ) denote the output of layer i before activation and h be an upper bound on the number of output units in each layer.

Original: We can then deﬁne fully connected feedforward networks recursively: f 1 w ( x ) =
Revised: We can then deﬁne fully connected feedforward networks recursively: f 1 w ( x ) =

Original: W 1 x and f i w ( x ) =
Revised: W 1 x and f i w ( x ) =

Original: W i φ ( f i − 1 w ( x )) .
Revised: W i φ ( f i − 1 w ( x )) .

Original: Let (cid:107) .
Revised: Let (cid:107) .

Original: (cid:107) F , (cid:107) .
Revised: (cid:107) F , (cid:107) .

Original: (cid:107) 1 and (cid:107) .
Revised: (cid:107) 1 and (cid:107) .

Original: (cid:107) 2 denote the Frobenius norm, the element-wise (cid:96)norm and the spectral norm respectively.
Revised: (cid:107) 2 denote the Frobenius norm, the element-wise (cid:96)norm and the spectral norm respectively.

Original: We further denote the (cid:96) p norm of a vector by | .
Revised: We further denote the (cid:96) p norm of a vector by | .

Original: | p .
Revised: | p .

Original: Margin Loss.
Revised: Margin Loss.

Original: For any distribution D and margin γ > 0 , we deﬁne the expected margin loss as follows:
Revised: For any distribution D and margin γ > 0 , we deﬁne the expected margin loss as follows:

Original: [Equation]
Revised: [Equation]

Original: Let (cid:98) L γ ( f w ) be the empirical estimate of the above expected margin loss.
Revised: Let (cid:98) L γ ( f w ) be the empirical estimate of the above expected margin loss.

Original: Since setting γ =corresponds to the classiﬁcation loss, we will use L 0 ( f w ) and (cid:98) L 0 ( f w ) to refer to the expected risk and the training error.
Revised: Since setting γ =corresponds to the classiﬁcation loss, we will use L 0 ( f w ) and (cid:98) L 0 ( f w ) to refer to the expected risk and the training error.

Original: The loss L γ deﬁned this way is bounded between 0 and 1.
Revised: The loss L γ deﬁned this way is bounded between 0 and 1.

Original: 1. PAC-B AYESIAN FRAMEWORK
Revised: 1. PAC-B AYESIAN FRAMEWORK

Original: The PAC-Bayesian framework (McAllester, 1998; 1999) provides generalization guarantees for randomized predictors, drawn form a learned distribution Q (as opposed to a learned single predictor) that depends on the training data.
Revised: The PAC-Bayesian framework (McAllester, 1998; 1999) provides generalization guarantees for randomized predictors, drawn form a learned distribution Q (as opposed to a learned single predictor) that depends on the training data.

Original: In particular, let f w be any predictor (not necessarily a neural network) learned from the training data and parametrized by w .
Revised: In particular, let f w be any predictor (not necessarily a neural network) learned from the training data and parametrized by w .

Original: We consider the distribution Q over predictors of the form f w + u , where u is a random variable whose distribution may also depend on the training data.
Revised: We consider the distribution Q over predictors of the form f w + u , where u is a random variable whose distribution may also depend on the training data.

Original: Given a “prior” distribution P over the set of predictors that is independent of the training data, the PAC-Bayes theorem states that with probability at least 1 − δ over the draw of the training data, the expected error of f w + u can be bounded as follows (McAllester, 2003):
Revised: Given a “prior” distribution P over the set of predictors that is independent of the training data, the PAC-Bayes theorem states that with probability at least 1 − δ over the draw of the training data, the expected error of f w + u can be bounded as follows (McAllester, 2003):

Original: [Equation]
Revised: [Equation]

Original: To get a bound on the expected risk L 0 ( f w ) for a single predictor f w , we need to relate the expected perturbed loss, E u
Revised: To get a bound on the expected risk L 0 ( f w ) for a single predictor f w , we need to relate the expected perturbed loss, E u

Original: [ L 0 ( f w + u )] in the above equation with L 0 ( f w ) .
Revised: [ L 0 ( f w + u )] in the above equation with L 0 ( f w ) .

Original: Toward this we use the following lemma that gives a margin-based generalization bound derived from the PAC-Bayesian bound (2):
Revised: Toward this we use the following lemma that gives a margin-based generalization bound derived from the PAC-Bayesian bound (2):

Original: Lemma 1.
Revised: Lemma 1.

Original: Let f w ( x ) :
Revised: Let f w ( x ) :

Original: X → R k be any predictor (not necessarily a neural network) with parameters w , and P be any distribution on the parameters that is independent of the training data.
Revised: X → R k be any predictor (not necessarily a neural network) with parameters w , and P be any distribution on the parameters that is independent of the training data.

Original: Then, for any γ, δ > 0 , with probability ≥ 1 − δ over the training set of size m , for any w , and any random perturbation u s.t.
Revised: Then, for any γ, δ > 0 , with probability ≥ 1 − δ over the training set of size m , for any w , and any random perturbation u s.t.

Original: P u (cid:2) max x [Equation] | f [Equation]
Revised: P u (cid:2) max x ∈X | f w + u ( x ) − f w ( x ) | ∞ <

Original: +

Original: [Equation]

Original: ( x ) − f

Original: [Equation]

Original: x

Original: [Equation] | [Equation] < γ 4 (cid:3) ≥ 12 , we have:
Revised: γ 4 (cid:3) ≥ 12 , we have:

Original: [Equation]
Revised: [Equation]

Original: In the above expression the KL is evaluated for a ﬁxed w and only u is random, i.e. the distribution of w + u is the distribution of u shifted by w .
Revised: In the above expression the KL is evaluated for a ﬁxed w and only u is random, i.e. the distribution of w + u is the distribution of u shifted by w .

Original: The lemma is analogous to similar analysis of
Revised: The lemma is analogous to similar analysis of

Original: Langford & Shawe-Taylor (2003) and McAllester (2003) obtaining PAC-Bayes margin bounds for linear predictors, and the proof, presented in Section 4, is essentially the same.
Revised: Langford & Shawe-Taylor (2003) and McAllester (2003) obtaining PAC-Bayes margin bounds for linear predictors, and the proof, presented in Section 4, is essentially the same.

Original: As we state the lemma, it is not speciﬁc to linear separators, nor neural networks, and holds generally for any real-valued predictor.
Revised: As we state the lemma, it is not speciﬁc to linear separators, nor neural networks, and holds generally for any real-valued predictor.

Original: We next show how to utilize the above general PAC-Bayes bound to prove generalization guarantees for feedforward networks based on the spectral norm of its layers.
Revised: We next show how to utilize the above general PAC-Bayes bound to prove generalization guarantees for feedforward networks based on the spectral norm of its layers.

Original:  G ENERALIZATION B OUND
Revised:  G ENERALIZATION B OUND

Original: In this section we present our generalization bound for feedfoward networks with ReLU activations, derived using the PAC-Bayesian framework.
Revised: In this section we present our generalization bound for feedfoward networks with ReLU activations, derived using the PAC-Bayesian framework.

Original: Langford & Caruana (2001), and more recently Dziugaite & Roy (2017) and Neyshabur et al.
Revised: Langford & Caruana (2001), and more recently Dziugaite & Roy (2017) and Neyshabur et al.

Original: (2017), used PAC-Bayes bounds to analyze generalization behavior in neural networks, evaluating the KL-divergence, “perturbation error” L [ f w + u ] − L [ f w ] , or the entire bound numerically.
Revised: (2017), used PAC-Bayes bounds to analyze generalization behavior in neural networks, evaluating the KL-divergence, “perturbation error” L [ f w + u ] − L [ f w ] , or the entire bound numerically.

Original: Here, we use the PAC-Bayes framework as a tool to analytically derive a margin-based bound in terms of norms of the weights.
Revised: Here, we use the PAC-Bayes framework as a tool to analytically derive a margin-based bound in terms of norms of the weights.

Original: As we saw in Lemma 1, the key to doing so is bounding the change in the output of the network when the weights are perturbed.
Revised: As we saw in Lemma 1, the key to doing so is bounding the change in the output of the network when the weights are perturbed.

Original: In the following lemma, we bound this change in terms of the spectral norm of the layers:
Revised: In the following lemma, we bound this change in terms of the spectral norm of the layers:

Original: Lemma 2 (Perturbation Bound) .
Revised: Lemma 2 (Perturbation Bound) .

Original: For any B, d > 0 , let f w : X B,n → R k be a d -layer neural network with ReLU activations.
Revised: For any B, d > 0 , let f w : X B,n → R k be a d -layer neural network with ReLU activations.

Original: Then for any w , and x ∈ X B,n , and any perturbation u = vec (cid:0) { U i } d i =1 (cid:1) such that (cid:107) U i 
Revised: Then for any w , and x ∈ X B,n , and any perturbation u = vec (cid:0) { U i } di =1 (cid:1) such that (cid:107) U i (cid:107)

Original: [Equation]
Revised: [Equation]

Original: 2 ≤1 d
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: W i (cid:107) 2 , the change in the output of the network can be bounded as follows:
Revised: W i (cid:107) 2 , the change in the output of the network can be bounded as follows:

Original: [Equation]
Revised: [Equation]

Original: This lemma characterizes the change in the output of a network with respect to perturbation of its weights, thereby bounding the sharpness of the network as deﬁned in Keskar et al.
Revised: This lemma characterizes the change in the output of a network with respect to perturbation of its weights, thereby bounding the sharpness of the network as deﬁned in Keskar et al.

Original: The proof of this lemma is presented in Section 4.
Revised: The proof of this lemma is presented in Section 4.

Original: Next we use the above perturbation bound and the
Revised: Next we use the above perturbation bound and the

Original: PAC-Bayes result (Lemma 1) to derive the following generalization guarantee.
Revised: PAC-Bayes result (Lemma 1) to derive the following generalization guarantee.

Original: Theorem 1 (Generalization Bound) .
Revised: Theorem 1 (Generalization Bound) .

Original: For any B, d, h > 0 , let f w : X B,n → R k be a d -layer feedforward network with ReLU activations.
Revised: For any B, d, h > 0 , let f w : X B,n → R k be a d -layer feedforward network with ReLU activations.

Original: Then, for any δ, γ > 0 , with probability ≥ 1 − δ over a training set of size m , for any w , we have:
Revised: Then, for any δ, γ > 0 , with probability ≥ 1 − δ over a training set of size m , for any w , we have:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The proof involves mainly two steps.
Revised: The proof involves mainly two steps.

Original: In the ﬁrst step we calculate what is the maximum allowed perturbation of parameters to satisfy a given margin condition γ , using Lemma 2.
Revised: In the ﬁrst step we calculate what is the maximum allowed perturbation of parameters to satisfy a given margin condition γ , using Lemma 2.

Original: In the second step we calculate the KL term in the PAC-Bayes bound in Lemma 1, for this value of the perturbation.
Revised: In the second step we calculate the KL term in the PAC-Bayes bound in Lemma 1, for this value of the perturbation.

Original: Let β = (cid:16)(cid:81) di
Revised: Let β = (cid:16)(cid:81) di

Original: =1 (cid:107) W
Revised: =1 (cid:107) W

Original: i (cid:107) 2 (cid:17) 1 /d and consider a network with the normalized weights (cid:102) W i =
Revised: i (cid:107) 2 (cid:17) 1 /d and consider a network with the normalized weights (cid:102) W i =

Original: β (cid:107) W
Revised: β (cid:107) W

Original: i (cid:107) 2 W i .
Revised: i (cid:107) 2 W i .

Original: Due to the homogeneity of the ReLU, we have that for feedforward networks with ReLU activations f (cid:101) w = f w , and so the (empirical and expected) loss (including margin loss) is the same for w and (cid:101) w .
Revised: Due to the homogeneity of the ReLU, we have that for feedforward networks with ReLU activations f (cid:101) w = f w , and so the (empirical and expected) loss (including margin loss) is the same for w and (cid:101) w .

Original: We can also verify that (cid:16)(cid:81) di
Revised: We can also verify that (cid:16) (cid:81) di

Original: =1 (cid:107) W
Revised: =1 (cid:107) W

Original: i (cid:107) 2 (cid:17) = (cid:16)(cid:81) di =1 (cid:13)(cid:13)(cid:13)(cid:102) W i (cid:13)(cid:13)(cid:13) 2 (cid:17) and (cid:107) W
Revised: i (cid:107) 2 (cid:17) = (cid:16) (cid:81) di =1 (cid:13) (cid:13)(cid:13)(cid:102) W i (cid:13) (cid:13)(cid:13) 2 (cid:17) and (cid:107) W

Original: i (cid:107) F (cid:107) W i (cid:107) 

Revised: i (cid:107) F (cid:107) W i (cid:107)

Original: = (cid:107) ˜ W
Revised: = (cid:107) ˜ W

Original: i (cid:107) F (cid:107) ˜
Revised: i (cid:107) F (cid:107) ˜

Original: W
Revised: W

Original: i (cid:107) 2 , and so the excess error in the Theorem statement is also invariant to this transformation.
Revised: i (cid:107) 2 , and so the excess error in the Theorem statement is also invariant to this transformation.

Original: It is therefore sufﬁcient to prove the Theorem only for the normalized weights ˜ w , and hence we assume w.l.o.g. that the spectral norm is equal across layers, i.e. for any layer
Revised: It is therefore sufﬁcient to prove the Theorem only for the normalized weights ˜ w , and hence we assume w.l.o.g. that the spectral norm is equal across layers, i.e. for any layer

Original: i , (cid:107) W
Revised: i , (cid:107) W

Original: i (cid:107) 2 = β .
Revised: i (cid:107) 2 = β .

Original: Choose the distribution of the prior P to be N (0 , σ 2 I ) , and consider the random perturbation u ∼ N (0 , σ 2 I ) , with the same σ , which we will set later according to β .
Revised: Choose the distribution of the prior P to be N (0 , σ 2 I ) , and consider the random perturbation u ∼ N (0 , σ 2 I ) , with the same σ , which we will set later according to β .

Original: More precisely, since the prior cannot depend on the learned predictor w or its norm, we will set σ based on an approximation ˜ β .
Revised: More precisely, since the prior cannot depend on the learned predictor w or its norm, we will set σ based on an approximation ˜ β .

Original: For each value of ˜ β on a pre-determined grid, we will compute the PAC-Bayes bound, establishing the generalization guarantee for all w for which
Revised: For each value of ˜ β on a pre-determined grid, we will compute the PAC-Bayes bound, establishing the generalization guarantee for all w for which

Original: | β − ˜ β | ≤ 1 d β , and ensuring that each relevant value of β is covered by some ˜ β on the grid.
Revised: | β − ˜ β | ≤ 1 d β , and ensuring that each relevant value of β is covered by some ˜ β on the grid.

Original: We will then take a union bound over all ˜ β on the grid.
Revised: We will then take a union bound over all ˜ β on the grid.

Original: For now, we will consider a ﬁxed ˜ β and the w for which |
Revised: For now, we will consider a ﬁxed ˜ β and the w for which |

Original: β − ˜ β | ≤ 1 d β , and hence 1 1 e β d − 1 ≤ ˜ β
Revised: β − ˜ β | ≤ 1 d β , and hence 1 1 e β d − 1 ≤ ˜ β

Original: d − 1 ≤ eβ d − 1 .
Revised: d − 1 ≤ eβ d − 1 .

Original: Since u ∼ N (0 , σ 2 I ) , we get the following bound for the spectral norm of U
Revised: Since u ∼ N (0 , σ 2 I ) , we get the following bound for the spectral norm of U

Original: i (Tropp, 2012):
Revised: i (Tropp, 2012):

Original: [Equation]
Revised: [Equation]

Original: Taking a union bond over the layers, we get that, with probability ≥ 12 , the spectral norm of the perturbation U i in each layer is bounded by σ (cid:112) 2 h ln(4 dh ) .
Revised: Taking a union bond over the layers, we get that, with probability ≥ 12 , the spectral norm of the perturbation U i in each layer is bounded by σ (cid:112) 2 h ln(4 dh ) .

Original: Plugging this spectral norm bound into
Revised: Plugging this spectral norm bound into

Original: Lemma 2 we have that with probability at least 12 ,
Revised: Lemma 2 we have that with probability at least 12 ,

Original: [Equation]
Revised: [Equation]

Original: where we choose σ = [Equation]
Revised: where we choose σ = γ 42 dB ˜ β

Original: [Equation] ) to get the last inequality.
Revised: d − 1 √ h ln(4 hd ) to get the last inequality.

Original: Hence, the perturbation u with the above value of σ satisﬁes the assumptions of the Lemma 1.
Revised: Hence, the perturbation u with the above value of σ satisﬁes the assumptions of the Lemma 1.

Original: We now calculate the KL-term in Lemma 1 with the chosen distributions for P and u , for the above value of σ .
Revised: We now calculate the KL-term in Lemma 1 with the chosen distributions for P and u , for the above value of σ .

Original: [Equation]
Revised: [Equation]

Original: [-|| P ) ≤ | w | 2 
-]
Revised: [-|| P ) ≤ | w | 2 
-]

Original: σ 2 = 42 2 d 2 B 2 ˜ β 2 d − 2 h ln(4 hd ) 2 γd (cid:88) i

Original: = (cid:107) W i (cid:107) 2 F ≤ O (cid:32) B 2 d 2 h ln( dh ) β 2 d γd (cid:88) i =
Revised: 1 (cid:0) 1 + 1 d (cid:1)

Original: (cid:107) W

Original: i (cid:107) 2 F β(cid:33) ≤ O (cid:32) B 2 d 2 h ln( dh )Π di =1 (cid:107) W i (cid:107) 22 γd (cid:88) i = (cid:107) W
Revised: d − 1 ≤ (cid:0) 1 + 1 d (cid:1) d ≤ e , as 1 + x ≤ e x , for all x .

Original: i (cid:107) 2 F (cid:107) W i (cid:107)(cid:33) . Hence, for any ˜ β , with probability ≥ 1  −δ and for all w such that, | β − ˜ β | ≤ 1  d β , we have:
Revised: Similarly (cid:16) 1 + 1 d − 1 (cid:17) d − 1 ≤ e , gives 1 e ≤

Original: [Equation]
Revised: [Equation]

Original: (cid:0) (cid:1)
Revised: 1 − 1 d (cid:1)

Original:  − 
Revised: d − 1 .

Original: (cid:1) , as ,  for all Similarly (cid:16)
Revised: Hence, for any ˜ β , with probability ≥ 1 − δ and for all w such that, | β − ˜ β | ≤ 1 d β , we have:

Original: − (cid:17)
Revised: [Equation]

Original: ≤ e , gives 1 e ≤

Original: Finally we need to take a union bound over different choices of ˜ β .
Revised: Finally we need to take a union bound over different choices of ˜ β .

Original: Let us see how many choices of ˜ β
Revised: Let us see how many choices of ˜ β

Original: we need to ensure we always have ˜ β in the grid s.t.
Revised: we need to ensure we always have ˜ β in the grid s.t.

Original: | ˜
Revised: | ˜

Original: β − β | ≤ 1 d β .
Revised: β − β | ≤ 1 d β .

Original: We only need to consider values of β in the range (cid:0) γ 2 B (cid:1) 1 /d ≤ β ≤ (cid:16) γ √ m 2 B (cid:17) 1 /d .
Revised: We only need to consider values of β in the range (cid:0) γ 2 B (cid:1) 1 /d ≤ β ≤ (cid:16) γ √ m 2 B (cid:17) 1 /d .

Original: For β outside this range the theorem statement holds trivially: Recall that the LHS of the theorem statement, L 0 ( f w ) is always bounded by 1 .
Revised: For β outside this range the theorem statement holds trivially: Recall that the LHS of the theorem statement, L 0 ( f w ) is always bounded by 1 .

Original: If β d < γ 2 B , then for any x , | f w ( x ) | ≤ β d B ≤ γ/ 2 and therefore (cid:98) L γ = 1 .
Revised: If β d < γ 2 B , then for any x , | f w ( x ) | ≤ β d B ≤ γ/ 2 and therefore (cid:98) L γ = 1 .

Original: Alternately, if β d > γ √ m 2 B , then the second term in equation 2 is greater than one.
Revised: Alternately, if β d > γ √ m 2 B , then the second term in equation 2 is greater than one.

Original: Hence, we only need to consider values of β in the range discussed above.
Revised: Hence, we only need to consider values of β in the range discussed above.

Original: | ˜
Revised: | ˜

Original: β − β | ≤ 1 d (cid:0) γ 2 B (cid:1) 1 /d is a sufﬁcient condition to satisfy the required condition that | ˜ β − β | ≤ 1 d β in the above range, thus we can use a cover of size dm 12 d .
Revised: β − β | ≤ 1 d (cid:0) γ 2 B (cid:1) 1 /d is a sufﬁcient condition to satisfy the required condition that | ˜ β − β | ≤ 1 d β in the above range, thus we can use a cover of size dm 1 2 d .

Original: Taking a union bound over the choices of ˜ β in this cover and using the bound in equation (4) gives us the theorem statement.
Revised: Taking a union bound over the choices of ˜ β in this cover and using the bound in equation (4) gives us the theorem statement.

Original:  C OMPARISON TO E XISTING G ENERALIZATION B OUNDS
Revised:  C OMPARISON TO E XISTING G ENERALIZATION B OUNDS

Original: In this section we will compare the bound of Theorem 1 with a similar spectral norm based margin bound recently obtained by Bartlett et al.
Revised: In this section we will compare the bound of Theorem 1 with a similar spectral norm based margin bound recently obtained by Bartlett et al.

Original: (2017), as well as examine whether and when these bounds can improve over VC-based generalization guarantees.
Revised: (2017a;b), as well as examine whether and when these bounds can improve over VC-based generalization guarantees.

Original: The VC-dimension of fully connected feedforward neural networks with ReLU activation with d layers and h units per layer is ˜Θ( d 2 h 2 ) (Harvey et al.
Revised: The VC-dimension of fully connected feedforward neural networks with ReLU activation with d layers and h units per layer is ˜Θ( d 2 h 2 ) (Harvey et al.

Original: , yielding a generalization guarantee of the form:
Revised: , yielding a generalization guarantee of the form:

Original: [Equation]
Revised: [Equation]

Original: where here and throughout this section we ignore logarithmic factors that depend on - the failure probability δ , the sample size m , the depth d and the number of units h .
Revised: where here and throughout this section we ignore logarithmic factors that depend on - the failure probability δ , the sample size m , the depth d and the number of units h .

Original: Bartlett et al.
Revised: Bartlett et al.

Original: (2017) showed a generalization bound for neural networks based on the spectral norm of its layers, using a different proof approach based on covering number arguments.
Revised: (2017a) showed a generalization bound for neural networks based on the spectral norm of its layers, using a different proof approach based on covering number arguments.

Original: For feedforward depth- d networks with ReLU activations, and when inputs are in X B,n , i.e. are of norm bounded by | x | 2 ≤ B , their generalization guarantee, ignoring logarithmic factors, ensures that, with high probability, for any w ,
Revised: For feedforward depth- d networks with ReLU activations, and when inputs are in X B,n , i.e. are of norm bounded by | x | 2 ≤ B , their generalization guarantee, ignoring logarithmic factors, ensures that, with high probability, for any w ,

Original: [Equation]
Revised: [Equation]

Original: Comparing our Theorem 1 and the Bartlett et al.
Revised: Comparing our Theorem 1 and the Bartlett et al.

Original: (2017) bound (6), the factor O (cid:16)
Revised: (2017a) bound (6), the factor O (cid:16)

Original: [Equation]
Revised: [Equation]

Original:  B 
Revised: 2 B 2 Π di

Original: [Equation]
Revised: [Equation]

Revised: 1 (cid:107) W

Original: [Equation]
Revised: [Equation]

Original:  appears in both bounds.
Revised: (cid:107) 22 (cid:17) appears in both bounds.

Original: The main difference is in the multiplicative factors, d 2 h (cid:80) di
Revised: The main difference is in the multiplicative factors, d 2 h (cid:80) di

Original: =1 (cid:107) W
Revised: =1 (cid:107) W

Original: i (cid:107) [Equation] F / (cid:107) W 
Revised: i (cid:107) 2 F / (cid:107) W i

Original: i (cid:107) 2in Theorem 1 compared to (cid:16)(cid:80) di =1 (cid:0) (cid:107) W
Revised: (cid:107)in Theorem 1 compared to (cid:16)(cid:80) d i =1 (cid:0) (cid:107) W

Original: i (cid:107) 1 / (cid:107) W i (cid:107) 2 (cid:1) 2 / 3 (cid:17) 3 in (6).
Revised: i (cid:107) 1 / (cid:107) W i (cid:107) 2 (cid:1) 2 / 3 (cid:17) 3 in (6).

Original: To get a sense of how these two bounds compare, we will consider the case where the norms of the weight matrices are uniform across layers—this is a reasonable situation as we already saw that the bounds are invariant to re-balancing the norm between the layers.
Revised: To get a sense of how these two bounds compare, we will consider the case where the norms of the weight matrices are uniform across layers—this is a reasonable situation as we already saw that the bounds are invariant to re-balancing the norm between the layers.

Original: But for the sake of comparison, we further assume that not only is the spectral norm equal across layers (this we can assume w.l.o.g.)
Revised: But for the sake of comparison, we further assume that not only is the spectral norm equal across layers (this we can assume w.l.o.g.)

Original: but also the Frobenius (cid:107) W i (cid:107) F and element-wise (cid:96) 1 norm (cid:107) W i (cid:107) 1 are uniform across layers (we acknowledge that this setting is somewhat favorable to our bound compared to (6)).
Revised: but also the Frobenius (cid:107) W i (cid:107) F and element-wise (cid:96) 1 norm (cid:107) W i (cid:107) 1 are uniform across layers (we acknowledge that this setting is somewhat favorable to our bound compared to (6)).

Original: In this case the numerator in the generalization bound of Theorem 1 scales as:
Revised: In this case the numerator in the generalization bound of Theorem 1 scales as:

Original: [Equation]
Revised: [Equation]

Original: while numerator in (6) scales as:
Revised: while numerator in (6) scales as:

Original: [Equation]
Revised: [Equation]

Original: Comparing between the bounds thus boils down to comparing √ h (cid:107) W i (cid:107) F with (cid:107) W i (cid:107) 1 .
Revised: Comparing between the bounds thus boils down to comparing √ h (cid:107) W i (cid:107) F with (cid:107) W i (cid:107) 1 .

Original: Recalling that W i is at most a h × h matrix, we have that (cid:107) W
Revised: Recalling that W i is at most a h × h matrix, we have that (cid:107) W

Original: i
Revised: i

Original: (cid:107) F ≤ (cid:107) W
Revised: (cid:107) F ≤ (cid:107) W

Original: i (cid:107) 1 ≤ h (cid:107) W i (cid:107) F .
Revised: i (cid:107) 1 ≤ h (cid:107) W i (cid:107) F .

Original: When the weights are fairly dense and are of uniform magnitude, the second inequality will be tight, and we will have √ h (cid:107) W i (cid:107) F (cid:28) (cid:107) W
Revised: When the weights are fairly dense and are of uniform magnitude, the second inequality will be tight, and we will have √ h (cid:107) W i (cid:107) F (cid:28) (cid:107) W

Original: i
Revised: i

Original: (cid:107) 1 , and Theorem 1 will dominate.
Revised: (cid:107) 1 , and Theorem 1 will dominate.

Original: When the weights are sparse with roughly a constant number of signiﬁcant weights per unit (i.e. weight matrix with sparsity Θ( h ) ), the bounds will be similar.
Revised: When the weights are sparse with roughly a constant number of signiﬁcant weights per unit (i.e. weight matrix with sparsity Θ( h ) ), the bounds will be similar.

Original: Bartlett et al.
Revised: Bartlett et al.

Original: (2017) bound will dominate when the weights are extremely sparse, with much fewer signiﬁcant weights than units, i.e. when most units do not have any incoming or outgoing weights of signiﬁcant magnitude.
Revised: (2017a) bound will dominate when the weights are extremely sparse, with much fewer signiﬁcant weights than units, i.e. when most units do not have any incoming or outgoing weights of signiﬁcant magnitude.

Original: It is also insightful to ask in what regime each bound could potentially improve over the VC-bound (5) and thus provide a non-trivial guarantee.
Revised: It is also insightful to ask in what regime each bound could potentially improve over the VC-bound (5) and thus provide a non-trivial guarantee.

Original: To this end, we consider the most “optimistic” scenario where B [-
-]
Revised: To this end, we consider the most “optimistic” scenario where B [-
-]2 γ 2 Π

Original: γ 2 Π di =1 (cid:107) W
Revised: d i =1 (cid:107) W

Original: i (cid:107) 22 = Θ(1) (it certainly cannot be lower than one if we have a non-trivial margin loss).
Revised: i (cid:107) 2 2 = Θ(1) (it certainly cannot be lower than one if we have a non-trivial margin loss).

Original: As before, we also take the norms of the weight matrices to be uniform across layers, yielding the multiplicative factors in (7) and (8), which we must compare to the VC-dimension ˜Θ( d 2 h 2 ) .
Revised: As before, we also take the norms of the weight matrices to be uniform across layers, yielding the multiplicative factors in (7) and (8), which we must compare to the VC-dimension ˜Θ( d 2 h 2 ) .

Original: We get that the bound of Theorem 1 is smaller than the VC bound if
Revised: We get that the bound of Theorem 1 is smaller than the VC bound if

Original: [Equation]
Revised: [Equation]

Original: We always have (cid:107) W
Revised: We always have (cid:107) W

Original: i (cid:107) F ≤ √ h (cid:107) W i (cid:107) 2 , and this is tight only for orthogonal matrices, where all eigenvalues are equal.
Revised: i (cid:107) F ≤ √ h (cid:107) W i (cid:107) 2 , and this is tight only for orthogonal matrices, where all eigenvalues are equal.

Original: Satisfying (9), and thus having Theorem 1 potentially improving over the
Revised: Satisfying (9), and thus having Theorem 1 potentially improving over the

Original: VC-bound, thus only requires fairly mild eigenvalue concentration (i.e. having multiple units be similar to each other), reduced rank or row-level sparsity in the weight matrices.
Revised: VC-bound, thus only requires fairly mild eigenvalue concentration (i.e. having multiple units be similar to each other), reduced rank or row-level sparsity in the weight matrices.

Original: Note that we cannot expect to improve over the VC-bound for unstructured “random” weight matrices—we can only expect norm-based guarantees to improve over the VC bound if there is some speciﬁc degenerate structure in the weights, and as we indeed see is the case here.
Revised: Note that we cannot expect to improve over the VC-bound for unstructured “random” weight matrices—we can only expect norm-based guarantees to improve over the VC bound if there is some speciﬁc degenerate structure in the weights, and as we indeed see is the case here.

Original: A similar comparison with Bartlett et al.
Revised: A similar comparison with Bartlett et al.

Original: (2017) bound (6) and its multiplicative factor (8), yields the following condition for improving over the VC bound:
Revised: (2017a) bound (6) and its multiplicative factor (8), yields the following condition for improving over the VC bound:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Since (cid:107) W i (cid:107) 1 can be as large as h 1 .
Revised: Since (cid:107) W i (cid:107) 1 can be as large as h 1 .

Original: 5 (cid:107) W i (cid:107) 2 , in some sense more structure is required here in order to satisfy (10), such as elementwise sparsity combined with low-rank row structure.
Revised: 5 (cid:107) W i (cid:107) 2 , in some sense more structure is required here in order to satisfy (10), such as elementwise sparsity combined with low-rank row structure.

Original: As discussed above, Theorem 1 and Bartlett et al.
Revised: As discussed above, Theorem 1 and Bartlett et al.

Original: (2017) bound can each be better in different regimes.
Revised: (2017a) bound can each be better in different regimes.

Original: Also in terms of comparison to the VC-bound, it is possible for either one to improve over the VC bound while the other doesn’t (i.e. for either (9) or (10) to be satisﬁed without the other one being satisﬁed), depending on the sparsity structure in the weights.
Revised: Also in terms of comparison to the VC-bound, it is possible for either one to improve over the VC bound while the other doesn’t (i.e. for either (9) or (10) to be satisﬁed without the other one being satisﬁed), depending on the sparsity structure in the weights.

Revised: After the initial preprint of this paper, Bartlett et al.

Revised: (2017b) presented an improved bound replacing the (cid:96) 1 norm term in the bound Bartlett et al.

Revised: (2017a) with (cid:96) 2 , 1 norm (sum of (cid:96) 2 norms of each unit).

Revised: This new bound depending on (cid:107) W i (cid:107) 2 , 1 is always better than the bound based on (cid:107) W i (cid:107) 1 and our bound based on √ h (cid:107) W i (cid:107) F .

Revised: They match when each hidden unit has the same norm, making (cid:107) W i (cid:107) 2 , 1 ≈ √ h (cid:107) W i (cid:107) F .

Original:  P ROOFS OF L EMMAS
Revised:  P ROOFS OF L EMMAS

Original: In this section we present the proofs of Lemmas 1 and 2.
Revised: In this section we present the proofs of Lemmas 1 and 2.

Original: Proof of Lemma 1 .
Revised: Proof of Lemma 1 .

Original: Let w (cid:48)
Revised: Let w (cid:48)

Original: =
Revised: =

Original: w + u .
Revised: w + u .

Original: Let S w be the set of perturbations with the following property:
Revised: Let S w be the set of perturbations with the following property:

Original: [Equation]
Revised: [Equation]

Original: Let q be the probability density function over the parameters w (cid:48) .
Revised: Let q be the probability density function over the parameters w (cid:48) .

Original: We construct a new distribution ˜ Q over predictors f ˜ w where ˜ w is restricted to S w with the probability density function:
Revised: We construct a new distribution ˜ Q over predictors f ˜ w where ˜ w is restricted to S w with the probability density function:

Original: [Equation]
Revised: [Equation]

Original: ∈ S w otherwise .

Original: Here Z is a normalizing constant and by the lemma assumption
Revised: Here Z is a normalizing constant and by the lemma assumption

Original: Z = P [ w (cid:48)
Revised: Z = P [ w (cid:48)

Original: ∈ S w ] ≥ 12 .
Revised: ∈ S w ] ≥ 12 .

Original: By the deﬁnition of ˜ Q , we have:
Revised: By the deﬁnition of ˜ Q , we have:

Original: [Equation]
Revised: [Equation]

Original: Therefore, the perturbation can change the margin between two output units of f w by at most γ 2 ; i.e. for any perturbed parameters ˜ w drawn from ˜ Q :
Revised: Therefore, the perturbation can change the margin between two output units of f w by at most γ 2 ; i.e. for any perturbed parameters ˜ w drawn from ˜ Q :

Original: [Equation]
Revised: [Equation]

Original: Since the above bound holds for any x in the domain X B,n , we can get the following a.s.:
Revised: Since the above bound holds for any x in the domain X B,n , we can get the following a.s.:

Original: [Equation]
Revised: [Equation]

Original: Now using the above inequalities together with the equation (2), with probability 1 − δ over the training set we have:
Revised: Now using the above inequalities together with the equation (2), with probability 1 − δ over the training set we have:

Original: [Equation]
Revised: [Equation]

Original: The last inequality follows from the following calculation.
Revised: The last inequality follows from the following calculation.

Original: Let S c w denote the complement set of S w and ˜
Revised: Let S c w denote the complement set of S w and ˜

Original: q c denote the density function q restricted to S c w and normalized.
Revised: q c denote the density function q restricted to S c w and normalized.

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: where H ( Z ) = − Z ln Z − (1 − Z ) ln(1 − Z ) ≤ 1 is the binary entropy function.
Revised: where H ( Z ) = − Z ln Z − (1 − Z ) ln(1 − Z ) ≤ 1 is the binary entropy function.

Original: Since KL is always positive, we get,
Revised: Since KL is always positive, we get,

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma 2 .
Revised: Proof of Lemma 2 .

Original: Let ∆ i = (cid:12)(cid:12)
Revised: Let ∆ i = (cid:12)(cid:12)

Original: f i w + u ( x ) − f i w [Equation] x ) [Equation]
Revised: f i w + u ( x ) − f i w ( x ) (cid:12)(cid:12) 2 .

Original: We will prove using induction that for any i ≥ 0 :
Revised: We will prove using induction that for any i ≥ 0 :

Original: [Equation]
Revised: [Equation]

Original: The above inequality together with (cid:0) 1 + 1 d (cid:1) d ≤ e proves the lemma statement.
Revised: The above inequality together with (cid:0) 1 + 1 d (cid:1) d ≤ e proves the lemma statement.

Original: The induction base clearly holds since ∆ 

Revised: The induction base clearly holds since ∆

Original: = | x − x | 2 = 0 .
Revised: = | x − x | 2 = 0 .

Original: For any i ≥ 1 , we have the following:
Revised: For any i ≥ 1 , we have the following:

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Revised: W i

Revised: [Equation]

Revised: [Equation]

Revised: i

Revised: [Equation]

Revised: [Equation]

Revised: i

Revised: [Equation]

Revised: [Equation]

Revised: i

Revised: [Equation]

Revised: [Equation]

Revised: i

Revised: [Equation]

Revised: i

Revised: [Equation]

Revised: [Equation]

Revised: i

Revised: [Equation]

Revised: i

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: (cid:12)(cid:12)

Revised: [Equation]

Revised: [Equation]

Revised: i

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: i

Revised: [Equation]

Revised: i

Revised: [Equation]

Revised: (cid:107)

Revised: [Equation]

Revised: (cid:12)(cid:12)

Revised: [Equation]

Revised: i

Revised: [Equation]

Revised: [Equation]

Revised: i

Revised: [Equation]

Revised: i

Revised: [Equation]

Revised: [Equation]

Revised: i

Original:  where the last inequality is by the Lipschitz property of the activation function and using φ (0)
Revised: [Equation] i [Equation] where the last inequality is by the Lipschitz property of the activation function and using φ (0)

Original: = 0 .
Revised: = 0 .

Original: The (cid:96) 2 norm of outputs of layer i is bounded by | x | 2 Π ij =1 (cid:107) W j (cid:107) 2 and by the lemma assumption we have (cid:107) U
Revised: The (cid:96) 2 norm of outputs of layer i is bounded by | x | 2 Π ij =1 (cid:107) W j (cid:107) 2 and by the lemma assumption we have (cid:107) U

Original: i +1 (cid:107) 2 ≤ 1 d (cid:107) W
Revised: i +1 (cid:107) 2 ≤ 1 d (cid:107) W

Original: i +1 (cid:107) 2 .
Revised: i +1 (cid:107) 2 .

Original: Therefore, using the induction step, we get the following bound:
Revised: Therefore, using the induction step, we get the following bound:

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper we presented new perturbation bounds for neural networks thereby giving a bound on its sharpness.
Revised: In this paper, we presented new perturbation bounds for neural networks thereby giving a bound on its sharpness.

Original: We also discussed how PAC-Bayes framework can be used to derive generalization bounds based on the sharpness of a model class.
Revised: We also discussed how PAC-Bayes framework can be used to derive generalization bounds based on the sharpness of a model class.

Original: Applying this to the feedforward networks, we showed that a tighter generalization bound can be achieved based on the spectral norm and Frobenius norm of the layers.
Revised: Applying this to the feedforward networks, we showed that a tighter generalization bound can be achieved based on the spectral norm and Frobenius norm of the layers.

Original: The simplicity of the proof compared to that of covering number arguments in
Revised: The simplicity of the proof compared to that of covering number arguments in

Original: Bartlett et al.
Revised: Bartlett et al.

Original: (2017) suggest that the PAC-Bayes framework might be an important tool in analyzing the generalization behavior of neural networks.
Revised: (2017a) suggest that the PAC-Bayes framework might be an important tool in analyzing the generalization behavior of neural networks.