D ETERMINISTIC PAC-B AYESIAN GENERALIZATION BOUNDS FOR DEEP NETWORKS VIA GENERALIZING NOISE - RESILIENCE
Anonymous authors
Paper under double-blind review
A BSTRACT
The ability of overparameterized deep networks to generalize well has been linked to the fact that stochastic gradient descent (SGD) ﬁnds solutions that lie in ﬂat, wide minima in the training loss – minima where the output of the network is resilient to small random noise added to its parameters.
So far this observation has been used to provide generalization guarantees only for neural networks whose parameters are either stochastic or compressed .
In this work, we present a general
PAC-Bayesian framework that leverages this observation to provide a bound on the original network learned – a network that is deterministic and uncompressed.
What enables us to do this is a key novelty in our approach: our framework allows us to show that if on training data, the interactions between the weight matrices satisfy certain conditions that imply a wide training loss minimum, these conditions themselves generalize to the interactions between the matrices on test data, thereby implying a wide test loss minimum.
We then apply our general framework in a setup where we assume that the pre-activation values of the network are not too small (although we assume this only on the training data).
In this setup, we provide a generalization guarantee for the original (deterministic, uncompressed) network, that does not scale with product of the spectral norms of the weight matrices – a guarantee that would not have been possible with prior approaches.
I NTRODUCTION
Modern deep neural networks contain millions of parameters and are trained on relatively few samples.
Conventional wisdom in machine learning suggests that such models should massively overﬁt on the training data, as these models have the capacity to memorize even a randomly labeled dataset of similar size (Zhang et al., 2017).
Yet these models have achieved state-of-the-art generalization error on many real-world tasks.
This observation has spurred an active line of research (Neyshabur et al., 2015; Soudry et al., 2018; Brutzkus et al., 2018) that has tried to understand what properties are possessed by stochastic gradient descent (SGD) training of deep networks that allows these networks to generalize well.
One particularly promising line of work (Neyshabur et al., 2017; Arora et al., 2018) has been bounds that utilize the noise-resilience of deep networks on training data (i.e., how much the training loss of the network changes with noise injected into the parameters, or roughly, how wide is the training loss minimum).
While these have yielded generalization bounds that do not have a severe exponential dependence on depth (unlike other bounds that grow with the product of spectral norms of the weight matrices), these bounds are quite limited: they either apply to a stochastic version of the classiﬁer (where the parameters are drawn from a distribution) or a compressed version of the classiﬁer (where the parameters are modiﬁed and represented using fewer bits).
In this paper, we revisit the PAC-Bayesian analysis of deep networks in Neyshabur et al.
We provide a general framework that allows one to use noise-resilience of the deep network on training data to provide a bound on the original deterministic and uncompressed network.
We achieve this by arguing that if on the training data, the interaction between the ‘activated weight matrices’ (weight matrices where the weights incoming from/outgoing to inactive units are zeroed out) satisfy certain conditions which results in a wide training loss minimum, these conditions themselves generalize to the weight matrix interactions on the test data.
After presenting this general PAC-Bayesian framework, we specialize it to the case of deep ReLU networks, showing that we can provide a generalization bound that accomplishes two goals simultaneously: i) it applies to the original network and ii) it does not scale exponentially with depth in terms of the products of the spectral norms of the weight matrices; instead our bound scales with more meaningful terms that capture the interactions between the weight matrices and do not have such a severe dependence on depth in practice.
We note that all but one of these terms are indeed quite small on networks in practice.
However, one particularly (empirically) large term that we use is the reciprocal of the magnitude of the network pre-activations on the training data (and so our bound would be small only in the scenario where the pre-activations are not too small).
We emphasize that this drawback is more of a limitation in how we characterize noise-resilience through the speciﬁc conditions we chose for the ReLU network, rather than a drawback in our PAC-Bayesian framework itself.
Our hope is that, since our technique is quite general and ﬂexible, by carefully identifying the right set of conditions, in the future, one might be able to derive a similar generalization guarantee that is smaller in practice.
To the best of our knowledge, our approach of generalizing noise-resilience of deep networks from training data to test data in order to derive a bound on the original network that does not scale with products of spectral norms, has neither been considered nor accomplished so far, even in limited situations.
B ACKGROUND AND RELATED WORK
One of the most important aspects of the generalization puzzle that has been studied is that of the ﬂatness/width of the training loss at the minimum found by SGD.
The general understanding is that ﬂatter minima are correlated with better generalization behavior, and this should somehow help explain the generalization behavior (Hochreiter & Schmidhuber, 1997; Hinton & van Camp, 1993;
Keskar et al., 2017).
Flatness of the training loss minimum is also correlated with the observation that on training data, adding noise to the parameters of the network results only in little change in the output of the network – or in other words, the network is noise-resilient.
Most generalization bounds that do not make use of noise-resilience of the network have generalization guarantees that in practice have strong exponential dependence on the depth of the network.
More precisely, these bounds scale either with the product of the spectral norms of the weight matrices (Neyshabur et al., 2018; Bartlett et al., 2017) or their Frobenius norms (Golowich et al., 2018).
In practice, the weight matrices have a spectral norm that is as large as 2 or 3 , and an even larger
Frobenius norm that scales with √ H where H is the width of the network i.e., maximum number of hidden units per layer.
Thus, the generalization bound scales as say, 2 D or H D / 2 , where D is the depth of the network.
At a high level, the reason these bounds suffer from such an exponential dependence on depth is that they effectively perform a worst case approximation of how the weight matrices interact with each other.
For example, the product of the spectral norms arises from a naive approximation of the
Lipschitz constant of the neural network, which would hold only when the singular values of the weight matrices all align with each other.
However, in practice, for most inputs to the network, the interactions between the activated weight matrices are not as adverse.
By using noise-resilience of the networks, prior approaches (Arora et al., 2018; Neyshabur et al., 2017) have been able to derive bounds that replace the above worst-case approximation with smaller terms that realistically capture these interactions.
However, these works are limited in critical ways.
Arora et al.
(2018) use noise-resilience of the network to modify and “compress” the parameter representation of the network, and derive a generalization bound on the compressed network.
While this bound enjoys a better dependence on depth because its applies to a compressed network, the high probability, lies near its expected value, near 2 and the Frobenius norm near its expected value which is √ H .
Since SGD is observed not to move too far away from the initialization regardless of H (Nagarajan &
Kolter, 2017), these values are more or less preserved for the ﬁnal weight matrices.
main drawback of this bound is that it does not apply on the original network.
On the other hand,
Neyshabur et al.
(2017) take advantage of noise-resilience on training data by incorporating it within a PAC-Bayesian generalization bound (McAllester, 1999a).
However, their ﬁnal guarantee is only a bound on the expected test loss of a stochastic network.
In this work, we revisit the idea in Neyshabur et al.
(2017), by pursuing the PAC-Bayesian framework (McAllester, 1999a) to answer this question.
The standard PAC-Bayesian framework provides generalization bounds for the expected loss of a stochastic classiﬁer, where the stochasticity typically corresponds to Gaussian noise injected into the parameters output by the learning algorithm.
However, if the classiﬁer is noise-resilient on both training and test data, one could extend the PAC-Bayesian bound to a standard generalization guarantee on the deterministic classiﬁer.
Other works have used PAC-Bayesian bounds in different ways in the context of neural networks.
Langford & Caruana (2001); Dziugaite & Roy (2017) optimize the stochasticity and the weights of the network in order to numerically compute good (i.e., non-vacuous) generalization bounds on the stochastic network.
Neyshabur et al.
(2018) derive generalization bounds on the original, deterministic network by working from the PAC-Bayesian bound on the stochastic network.
However, as stated earlier, their work does not make use of noise resilience in the networks learned by SGD.
O UR C ONTRIBUTIONS
The key contribution in our work is a general PAC-Bayesian framework for deriving generalization bounds while leveraging the noise resilience of a deep network.
While our approach is applied to deep networks, we note that it is quite general (as will be apparent from
Section 3) and can be applied to other classiﬁers.
In our framework, we consider a set of conditions that when satisﬁed by the network, makes the output of the network noise-resilient at a particular input datapoint.
For example, these conditions could characterize the interactions between the activated weight matrices at a particular input.
To provide a generalization guarantee, we assume that the learning algorithm has found weights such that these conditions hold for the weight interactions in the network on training data (which effectively implies a wide training loss minimum).
Then, as a key step, we generalize these conditions over to the weight interactions on test data (which effectively implies a wide test loss minimum) 2 .
Thus, with the guarantee that the classiﬁer is noise-resilient both on training and test data, we derive a generalization bound on the test loss of the original network.
Our framework relies on a novel technique to convert the PAC-Bayesian bound on a stochastic classiﬁer to the generalization bound on the deterministic classiﬁer.
This technique is arguably more powerful than known PAC-Bayesian approaches, in leveraging noise-resilience, and is thus of independent theoretical interest.
Finally, we apply our framework to a speciﬁc set up of ReLU based feedforward networks.
In particular, we ﬁrst instantiate the above abstract framework with a set of speciﬁc conditions, and then use the above framework to derive a bound on the original network.
While very similar conditions have already been identiﬁed in prior work (Arora et al., 2018; Neyshabur et al., 2017) (see Appendix G for an extensive discussion of this), our contribution here is in showing how these conditions generalize from training to test data.
Crucially, like these works, our bound does not have severe exponential dependence on depth in terms of products of spectral norms.
We note that in reality, all but one of our conditions on the network do hold on training data as necessitated by the framework.
The strong, non-realistic condition we make is that the pre-activation values of the network are sufﬁciently large, although only on training data; however, in practice a small proportion of the pre-activation values can be arbitrarily small.
Our generalization bound scales inversely with the smallest absolute value of the pre-activations on the training data, and hence in practice, our bound would be large.
Intuitively, we make this assumption to ensure that under sufﬁciently small parameter perturbations, the activation states of the units are guaranteed not to ﬂip.
It is worth noting that Arora et al.
Neyshabur et al.
(2017) too require similar, but more realistic assumptions about pre-activation values that effectively assume only a small proportion of units ﬂip under noise.
However, even under our stronger condition that no such units exist, it is not apparent how these approaches would yield a similar bound on the deterministic, uncompressed network without generalizing their conditions to test data.
We hope that in the future our work could be developed further to accommodate the more realistic conditions from Arora et al.
; Neyshabur et al.
A GENERAL PAC-B AYESIAN FRAMEWORK
In this section, we present our general PAC-Bayesian framework that uses noise-resilience of the network to convert a PAC-Bayesian generalization bound on the stochastic classiﬁer to a generalization bound on the deterministic classiﬁer.
N OTATION .
Let KL (⋅∥⋅) denote the KL-divergence, ∥⋅∥ denote the (cid:96) 2 norm of a vector, ∥⋅∥ ∞ the (cid:96) ∞ norm of a vector, and ∥⋅∥ 2 , ∥⋅∥ F denote the spectral and Frobenius norm of a matrix, respectively.
Consider a K -class learning task where the labeled datapoints ( x ,y ) are drawn from an underlying distribution D over X × { 1 , 2 , ⋯ ,K } where X ∈ R N and ∀ x ∈ X ,
∥ x ∥ ≤ B .
We consider a classiﬁer parametrized by weights W .
For a given input x and class k , we denote the output of the classiﬁer by f ( x ; W)
[ k ] .
In our PAC-Bayesian analysis, we will use U ∼ N( 0 ,σ 2 ) to denote parameters whose entries are sampled independently from a Gaussian, and W + U to denote the entrywise addition of the two sets of parameters.
We use ∥W∥ 2 F to denote ∑ Dd = 1 ∥ W d ∥ 2 F .
Finally, given a training set S of m samples, we let ( x ,y ) ∼ S to denote uniform sampling from the set.
See Appendix A for more notations.
T RADITIONAL PAC-B AYESIAN BOUNDS .
The PAC-Bayesian framework (McAllester, 1999a;b) allows us to derive generalization bounds for a stochastic classiﬁer.
Speciﬁcally, let ˜ W be a random variable in the parameter space whose distribution is learned based on training data S .
Let P be a prior distribution in the parameter space chosen independent of the training data.
Let L( ˜ W , x ,y ) be the loss of the classiﬁer ˜ W on a datapoint ( x ,y ) , where the loss between 0 and 1; in this paper, we take the loss to be the 0-1 classiﬁcation error.
The PAC-Bayesian framework yields the following generalization bound on the stochastic classiﬁer that holds with probability 1 − δ over the draw of the training set S of m samples:
[Equation]
Typically, and in the rest of this discussion, ˜ W is a Gaussian with covariance σ 2 I for some σ >centered at the weights W learned based on the training data.
Furthermore, we will set P to be a
Gaussian with covariance σ 2 I centered at the random initialization of the network like in Dziugaite & Roy (2017), instead of at the origin, like in Neyshabur et al.
This is because the resulting
KL-divergence – which depends on the distance between the means of the prior and the posterior – is known to be smaller, and to save a √ H factor in the bound (Nagarajan & Kolter, 2017).
3. O UR FRAMEWORK
To extend the above PAC-Bayesian bound to a standard generalization bound on a deterministic classiﬁer W , we need to replace the training and the test loss of the stochastic classiﬁer with that of the original, deterministic classiﬁer.
However, in doing so, we will have to introduce extra terms in the upper bound to account for the perturbation suffered by the train and test loss under the
Gaussian perturbation of the parameters.
To tightly bound these two terms, we need that the network is noise-resilient on training and test data respectively.
Our hope is that if the learning algorithm has found weights such that the network is noise-resilient on the training data, we can then generalize this noise-resilience over to test data as well, allowing us to better bound the excess terms.
Below, we lay out the key components of our framework, and constraints on how these should be designed; following this we present our main result and some intuition about our proof technique.
I NPUT - DEPENDENT PROPERTIES OF WEIGHTS
In our framework, we characterize noise-resilience of the model at a given input x , through conditions on how the weights of the model interact with each other for that input.
The idea is that when these conditions involving the weights and the input are satisﬁed, if we add noise to the weights, the output of the classiﬁer for that input will provably suffer only little perturbation.
For example, in the next section, we will consider conditions of the form “the preactivation values of the hidden units in layer d , have magnitude larger than some small positive constant”.
We will more generally refer to quantities involved in these conditions, like the pre-activation values, as “input-dependent properties of the weights”.
In this section, we will formulate these input-dependent properties and the conditions on them, for an abstract classiﬁer.
Consider a classiﬁer for which noise-resilience can be guaranteed at a particular input if R different conditions are known to be satisﬁed (in the case of deep networks, we will have a condition for each layer, and so R will scale with D ). Let the r th condition be a bound involving a particular set { ρ ( 1 ) r (W , x ,y ) ,ρ ( 2 ) r (W , x ,y ) , ⋯ , } of input-dependent properties of the weights – each element in this set is a scalar value that depends on the weights and the input (such as the pre-activation value of a particular hidden unit in layer d ).
Given this r th set of properties, we assume that the r th condition can be expressed as T r (W , x ,y ) > 0 , where T r is a function that aggregates these properties, e.g., T r (W , x ,y ) =
min r ρ ( l ) r
(W , x ,y ) .
For convenience, we instantiate the R th set to be the K classiﬁer outputs { f D ( x ; W)[ 1 ] , ⋯ ,f D ( x ; W)
[ K ]} and deﬁne T R (W , x ,y ) ∶= ( f ( x ; W)
[ y ] − max j ≠
y f ( x ; W)[ j ])/ 2 .
Then T R (W , x ,y ) > [-
 corre- sponds-]0 corresponds to correct classiﬁcation (zero error).
C ONSTRAINT 1: A GGREGATION OF THE PROPERTIES
In this framework, we restrict the aggregator functions T r to be linear in terms of the corresponding properties, in the sense that, ∀ r,
[Equation]
C ONSTRAINT 2: O RDERING OF THE SETS OF PROPERTIES
Next, we impose a crucial constraint on how the properties are grouped and indexed, that will help us show that these properties themselves are noise-resilient.
Roughly speaking, we want that for a given input, the properties in the r -th set are noise-resilient, whenever the ﬁrst r − 1 sets of properties satisfy the bounds imposed on them through the conditions T q (W , x ,y ) > 0 for q = 1 , 2 , ⋯ r − 1 .
For example, in the case of ReLU networks, we will show that the perturbation in the pre-activation values of the d th layer is small as long as the absolute pre-activation values in the layers below d − 1 are large, and a few other norm-bounds on the lower layer weights are satisﬁed.
We formalize the above requirement by deﬁning a set of expressions ∆ 1 ( σ ) , ⋯ , ∆ R ( σ ) that bound the perturbation in the properties as follows.
For any r ≤ R and for any x ∈ X : if ∀ q < r,T q (W , x ,y ) > 0 then [Equation]
and
[Equation]
In words, the ﬁrst event within the probability term above is the event that the properties involved in the r th condition suffer a large perturbation. The second is the event that the properties involved in the ﬁrst r − 1 conditions do not suffer much perturbation. Given that the ﬁrst r − 1 conditions hold for x , the second event implies that the ﬁrst r − 1 conditions still hold approximately under perturbation. Thus, the above constraint requires that there should be very few perturbations that perturb the r th set of properties signiﬁcantly while preserving the ﬁrst r − 1 conditions.
(When we instantiate the framework, we have to derive closed form expressions for ∆ r ( σ ) satisfying the above inequality, like we will do in Lemma E.1 for ReLU networks.)
T HEOREM S TATEMENT
In this setup, we have the following ‘margin-based’ generalization guarantee on the original network.
That is, we bound the 0-1 test error of the network by a margin-based error on the training data.
Our generalization guarantee – which scales linearly with the number of conditions R – holds under the setting that the training algorithm always ﬁnds weights such that on the training data, the condition that T r (W , x ,y ) is sufﬁciently large is satisﬁed for all r = 1 , ⋯ ,R − 1 .
Theorem 3.1.
Let γ class > 0 .
Let ∆ ⋆ 1 , ∆ ⋆ 2 , ⋯ , ∆ ⋆ R be a set of positive constants that will act as bounds on the input-dependent properties. Let ∆ ⋆ R = γ class / 2 .
Let T 1 ,T 2 , ⋯ ,T R be a set of aggregator functions that satisfy Constraint 1.
Let σ ∗ be the maximum value of σ such that for all r = 1 , ⋯ ,R ,
Constraint 2 holds with the perturbation bounds ∆ r ( σ ) satisfying ∆ r ( σ ) ≤ ∆ ⋆ r .
Then, for any δ > 0 , with probability 1 − δ over the draw of samples S from D m , for any W we have that: if W satisﬁes T r (W , x ,y ) > ∆ ⋆ r for all r < R and for all ( x ,y ) ∈ S , then
[Equation]
The crux of our proof (in Appendix D) lies in generalizing the conditions T r (W , x ,y ) >
∆ ⋆ r satisﬁed on the training data to test data one after the other .
Crucially, after we generalize the ﬁrst r −conditions from training data to test data, we will have from Equation 2 that the r th set of properties are noise-resilient on both training and test data.
Given the noise-resilience on the r th set of properties on test data, we can generalize the r th condition to test data too.
We emphasize a key, fundamental tool that we develop for use in this theorem.
Speciﬁcally, in
Theorem C.1 we present a new technique to convert a generic PAC-Bayesian bound on a stochastic classiﬁer, to a generalization bound on the deterministic classiﬁer.
We argue in Section C.1 this technique is more powerful than prior approaches (Neyshabur et al., 2018; Langford & ShaweTaylor, 2002; McAllester, 2003) in leveraging the noise-resilience of a generic classiﬁer.
The high level argument is that, to convert the PAC-Bayesian bound, previous works relied on a looser output perturbation bound, one that holds on all possible inputs, with high probability over all perturbations i.e., a bound on max x ∥ f ( x ; W) − f ( x ; W + U)∥ ∞ w.h.p over draws of U .
In contrast, our technique relies on a subtly different but signiﬁcantly tighter bound; speciﬁcally, we rely on a bound on the output perturbation that holds with high probability given an input i.e., a bound on ∥ f
( x ; W) − f ( x ; W + U)∥ ∞ w.h.p over draws of U for each x .
When we do instantiate our framework as in the next section, this subtle difference is critical in being able to bound the output perturbation without suffering from a factor proportional to the product of the spectral norms of the weight matrices (which is the case in Neyshabur et al.
A PPLICATION OF OUR FRAMEWORK TO R E LU N ETWORKS
N OTATION In this section, we apply our framework to feedforward fully connected ReLU networks of depth D (we care about D > 2 ) and width H and derive a generalization bound on the original network that does not scale with the product of spectral norms of the weight matrices. Let φ (⋅) denote the ReLU activation. We consider a network parameterized by W =
( W 1 ,W 2 , ⋯ ,W D ) such that the output of the network is computed as f ( x ; W) = W D φ ( W D − 1 ⋯ φ ( W 1 x )) . We denote the value of the h th hidden unit on the d th layer before and after the activation by g d ( x ; W) [ h ] and f d ( x ; W)
[ h ] respectively.
We deﬁne J d / d ′ ( x ; W) ∶= ∂g d ( x ; W)/ ∂g d ′ ( x ; W) to be the Jacobian of the pre-activations of layer d with respect to the pre-activations of layer d ′ for d ′ ≤ d (each row in this Jacobian corresponds to a unit in layer d ).
In short, we will call this, Jacobian d / d ′ .
Informally, we consider a setting where the learning algorithm satisﬁes the following conditions on the training data that make it noise-resilient on training data: a) the Jacobian of any layer with respect to a lower layer, has rows with a small (cid:96) 2 norm, b) the (cid:96) 2 norm of the hidden layers are all small and c) the pre-activation values are all sufﬁciently large in magnitude.
We cast these conditions in terms of the functions T r in the general framework.
These properties are quite similar to those already explored in Arora et al.
; Neyshabur et al.
(2017); we provide more intuition about these properties, and how we cast them in our framework in Appendix E.1.
We prove in Lemma E.1 in Appendix E a guarantee equivalent to the abstract inequality in Equation 2.
Essentially, we show that under random perturbations of the parameters, the perturbation in the output of the network and the perturbation in the input-dependent properties involved in (a), (b), (c) themselves can all be bounded in terms of each other.
The highlight of these perturbation bounds is that none of them grow with the spectral norms of the network.
Having set up the conditions of the framework as required, we then instantiate the bound provided by the framework as described below.
Our generalization bound scales with the bounds on the properties in (a) and (b) above as satisﬁed on the training data, and with the reciprocal of the property in (c) i.e., the smallest absolute value of the pre-activations on the training data.
Additionally, our bound has an explicit quadratic dependence on the depth of the network, which arises from the fact that we generalize O( D 2 ) conditions.
Most importantly, our bound does not have a dependence on the product of the spectral norms of the weight matrices.
Theorem 4.1.
( shorter version ; see Appendix F for the complete statement) Let ζ ⋆ d / d ′ ,α ⋆ d ,γ ⋆ d be a set of positive constants.
For any margin γ class > 0 , and any δ > 0 , with probability 1 − δ over the draw of samples from D m , for any W , if W satisﬁes the following for every training data point ( x ,y ) ∈ S :
(a) Sufﬁciently small maximum row (cid:96) 2 norm of Jacobian d / d ′ , max h ∥
J d / d ′ ( x ; W)
[ h ]∥ ≤ ζ ⋆ d / d ′ for all d ′ ,d such that 1 ≤ d ′ ≤ d ≤ D .
(b) Sufﬁciently small (cid:96) 2 norm of layer d ’s output: ∥ f d ( x ; W)∥ ≤ α ⋆ d for all d such that 0 ≤ d < D .
(c) Sufﬁciently large pre-activations on layer d : min h ∣ g d ( x ; W)[ h ]∣ ≥ γ ⋆ d for d such that 1 ≤ d < D .
then, for such W , we have that:
[Equation]
Here 1 / σ ⋆ equals ˜ O(√
H max {B jac , B out , B hidden-preact , B output-preact }) , where 4 :
[Equation]
In Figure 1, we demonstrate that in practice the row (cid:96) 2 norms of the Jacobian and the (cid:96) 2 norms of the hidden layer outputs are orders of magnitude smaller than products of the spectral norms.
In Figure 2, we train networks of varying depth and demonstrate that the terms B jac , B out , B output-preact also do not exhibit such severe exponential dependence on depth.
(In Appendix F, we demonstrate that these terms decay with width, although our emphasis is not on the width-dependence.)
However, the bottleneck in our bound is B hidden-preact , which scales inversely with the magnitude of the smallest absolute pre-activation value of the network.
In practice, this term can be arbitrarily large, even though it does not depend on the product of spectral norms/depth (Figure 2).
This is because some hidden units can have arbitrarily small absolute pre-activation values – although this is true only for a small proportion of these units.
To give an idea of the typical magnitude of the pre-activation values, in Figure 2, we plot a modiﬁed term called ‘median-based’- B hidden-preact , where we replace γ ⋆ d in the denominator with a different constant that bounds the median absolute pre-activation value for each training datapoint; ‘median-based’- B hidden-preact turns out to be a signiﬁcantly smaller quantity.
As noted before, we emphasize that the drawback highlighted above is a limitation in how we characterize noise-resilience through our conditions rather than a drawback in our general PACBayesian framework itself.
Speciﬁcally, using the assumed lower bound on the pre-activation magnitudes we can ensure that, under noise, the activation states of the units do not ﬂip; then the noise propagates through the network in a tractable, “linear” manner.
Improving this analysis is an important direction for future work.
For example, one could modify our analysis to allow perturbations large enough to ﬂip a small proportion of the activation states; one could potentially formulate such realistic conditions by drawing inspiration from the conditions in Neyshabur et al.
(2017); Arora et al.
However, we note that even though these prior approaches made more realistic assumptions about the magnitudes of the pre-activation values [Figure] the key limitation in these approaches is that even under our non-realistic assumption [Figure]
their approaches would yield bounds only on stochastic/compressed networks.
Generalizing noise-resilience from training data to test data is crucial to extending these bounds to the original network, which we accomplish.
[Figure]
[Figure]
While it is hard to quantify how B hidden-preact varies, all other values do not increase with depth as rapidly as the product of the spectral norms (i.e., for a additive increase of 1 in the depth, the bound does not increase by a multiplicative factor of 2 .).
S UMMARY AND FUTURE WORK
In this work, we introduced a novel PAC-Bayesian framework for leveraging the noise-resilience of deep neural networks on training data, to derive a generalization bound on the original uncompressed, deterministic network.
The main philosophy of our approach is to ﬁrst generalize the noise-resilience from training data to test data; this allows us to convert a PAC-Bayesian bound on a stochastic network to a standard margin-based generalization bound.
We also build a fundamental technique for converting PAC-Bayesian bounds, that is more powerful than prior approaches.
We apply our approach to ReLU based networks and derive a bound that scales with terms that capture the interactions between the weight matrices better than the product of spectral norms.
Our work brings up two concrete directions for future work.
The most important direction is that of removing the dependence on our strong assumption that the magnitude of the pre-activation values of the network are not too small on training data.
Another interesting direction would be to tighten the dependence on D 2 in our bound in Theorem 4.1.
More generally, our framework for generalizing noise-resilience is a promising tool for tapping the noise-resilience of the network to provide generalization guarantees on the original network; we hope that by further studying and applying this tool, we will be better equipped to answer the generalization puzzle in the future
A PPENDIX A N OTATIONS ( CONTINUED )
We will use upper-case symbols to denote matrices, and lower-case bold-face symbols to denote vectors.
In order to make the mathematical statements/derivations easier to read, if we want to emphasize a term, say x , we write, x .
Recall that we consider a neural netork of depth D (i.e., D − 1 hidden layers and one output layer) mapping from R N → R K , where K is the number of class labels in the learning task.
The layers are fully connected with H units in each hidden layer, and with ReLU activations φ (⋅) on all the hidden units and linear activations on the output units.
We denote the parameters of the network using the symbol W , which in turn denotes a set of weight matrices W 1 ,W 2 , ⋯ ,W D .
Here, W 1 ∈ R H × N , and
W D ∈ R K × H
and for all other layers d ≠ 1 ,D , W d ∈ R H
× H .
We will use the notation W d to denote the ﬁrst d weight matrices.
We denote the vector of weights input to the h th unit on the d th layer (which corresponds to the h th row in W d ) as w dh .
For any input x ∈ R N , we denote the function computed by the network on that input as f ( x ; W) =
W D φ ( W D − 1 ⋯ φ ( W 1 x )) .
For any d = 1 , ⋯ ,D − 1 , we denote the output of the d th hidden layer after the activation by f d ( x ; W) .
We denote the corresponding pre-activation values for that layer by g d ( x ; W) .
We denote the value of the h th hidden unit on the d th layer after and before the activation by f d ( x ; W)
[ h ] and g d ( x ; W)
[ h ] respectively.
Note that for the output layer d = D , these two values are equal as we assume only a linear activation.
For d = 0 , we deﬁne f 0 ( x ; W) = x .
As a result, we have the following recursions:
[Equation]
For layers d ′ ,d such that d ′ ≤ d , let us deﬁne J d / d ′ ( x ; W) to be the Jacobian corresponding to the pre-activation values of layer d with respect to the pre-activation values of layer d ′ on an input x .
That is,
[Equation]
In other words, this corresponds to the product of the ‘activated’ portion of the matrices
W d ′ + 1 ,W d ′ + 2 , ⋯ ,W d , where the weights corresponding to inactive inputs are zeroed out.
In short, we will call this ‘Jacobian d / d ′ ’.
Note that each row in this Jacobian corresponds to a unit on the d th layer, and each column corresponds to a unit on the d ′ th layer.
We will denote the parameters of a random initialization of the network by Z =
( Z 1 ,Z 2 , ⋯ ,Z d ) .
Let
D be an underlying distribution over R N × { 1 , 2 , ⋯ ,K } from which the data is drawn.
In our PAC-Bayesian analysis, we will use U to denote a set of D weight matrices U 1 ,U 2 , ⋯ ,U D whose entries are sampled independently from a Gaussian.
Furthermore, we will use U d to denote only the ﬁrst d of the randomly sampled weight matrices, and W + U d to denote a network where the d random matrices are added to the ﬁrst d weight matrices in W .
Note that W + U 0 = W .
Thus, f ( x ; W + U d ) is the output of a network where the ﬁrst d weight matrices have been perturbed.
In our analysis, we will also need to study a perturbed network where the hidden units are frozen to be at the activation state they were at before the perturbation; we will use the notation W[+U
d
] to denote the weights of such a network.
For our statements regarding probability of events, we will use ∧ , ∨ , and ¬ to denote the intersection, union and complement of events (to disambiguate from the set operators).
B U SEFUL L EMMAS
In this section, we present some results.
The ﬁrst two results below will be useful for our noise resilience analysis.
H OEFFDING B OUND
Lemma B.1.
For i = 1 , 2 , ⋯ ,n , let X i be independent random variables sampled from a Gaussian with mean µ i and variance σ 2 i .
Then for all t ≥ 0 , we have:
[Equation]
Or alternatively, for δ ∈ ( 0 , 1 ]
[Equation]
Note that an identical inequality holds good symmetrically for the event ∑ ni = 1 X i − µ i ≤ − t , and so the probability that the event ∣∑ ni = 1 X
i − µ i ∣ > t holds, is at most twice the failure probability in the above inequalities.
P RODUCT OF AN ENTRYWISE G AUSSIAN MATRIX AND A VECTOR
Lemma B.2.
Let U be a H 1 × H 2 matrix where each entry is sampled from N( 0 ,σ 2 ) .
Let x be an arbitrary vector in R H 2 .
Then, U x ∼ N( 0 , ∥ x ∥ 22 σ 2 I ) .
Proof.
U x is a random vector sampled from a multivariate Gaussian with mean E [ U x ] =and co-variance E
[ U xx T U T ] .
The ( i,j ) th entry in this covariance matrix is E
[( u Ti x )( u Tj x )] where u i and u j are the i th and j th row in U .
When i =
j , E [( u Ti x )( u Tj x )]
=
E [∥ u Ti x ∥ 2 ] =
∑ H 2 h = 1 E [ u 2 ih ] x 2 h
= σ 2 ∥ x ∥ 22 .
When i ≠ j , since u i and u j are independent random variables, we will have E [( u Ti x )( u Tj x )]
=
∑ H 2 h = 1 E [ u ih x h ]
∑ H 2 h = 1 E [ u jh x h ] = 0 .
KL DIVERGENCE OF G AUSSIANS .
We will use the following KL divergence equality to bound the generalization error in our PAC-Bayesian analyses.
Lemma B.3.
Let P be the spherical Gaussian N( µ 1 ,σ 2 I ) and Q be the spherical Gaussian
N( µ 2 ,σ 2 I ) .
Then, the KL-divergence between Q and P is:
[Equation]
C PAC-B AYESIAN T HEOREM
In this section, we will present our main PAC-Bayesian theorem that will guide our result about generalization.
Concretely, our result extends the generalization bound provided by conventional
PAC-Bayesian analysis McAllester (2003) – which is a generalization bound on the expected loss of a distribution of classiﬁers i.e., a stochastic classiﬁer – to a generalization bound on a deterministic classiﬁer.
The way we reduce the PAC-Bayesian bound to a standard generalization bound, is different from the one pursued in previous works (Neyshabur et al., 2018; Langford & Shawe-Taylor, 2002;
McAllester, 2003).
The generalization bound that we state below is a bit more general than standard generalization bounds.
Typically, generalization bounds are on the classiﬁcation error; however, as discussed in the main paper we will be dealing with generalizing multiple different conditions on the interactions between the weights of the network from the training data to test data.
So to state a bound that is general enough, we consider a set of generic functions T r (W , x , y ) for r = 1 , 2 , ⋯ R (as initially considered in the main paper in Section 3).
Recall that each of these functions compute a scalar value that corresponds to some input-dependent property of the network with parameters W for the datapoint ( x ,y ) .
As an example, this property could simply be the margin of the function on the y th class i.e., f ( x ; W)
[ y ] − max j ≠
y f ( x ; W)[ j ] , though we consider many other input-dependent properties of the network when we apply this theorem.
Theorem C.1.
Let P be a prior distribution over the parameter space that is chosen independent of the training dataset.
Let U be a random variable sampled entrywise from N( 0 ,σ 2 ) .
Let T r (⋅ , ⋅ , ⋅) and ∆ r > 0 for r = 1 , 2 , ⋯ R , be a set of functions and their corresponding margins.
We deﬁne the network W to be noise-resilient with respect to all these functions, at a given data point ( x ,y ) if:
[Equation]
Let µ D ({( T r , ∆ r )}
Rr = 1 , W) denote the probability over the random draw of a point ( x ,y )
drawn from D , that the network with weights W is not noise-resilient at ( x ,y ) according to Equation 3.
That is, let µ
[Equation]
Similarly, let ˆ µ S ({( T r , ∆ r )}
Rr = 1 , W) denote the fraction of data points ( x ,y ) in a dataset S for which the network is not noise-resilient according to Equation 3.
Then for any δ , with probability 1 − δ over the draws of a sample set S
= {( x i ,y i )
∼ D ∣ i = 1 , 2 , ⋯ ,m } , for any W we have:
[Equation]
The reader maybe curious about how one would bound the term µ D in the above bound, as this term corresponds to noise-resilience with respect to test data.
This is precisely what we bound later when we generalize the noise-resilience-related conditions satisﬁed on train data over to test data.
C. K EY ADVANTAGE OF OUR BOUND .
The above approach differs from previous approaches used by Neyshabur et al.
; Langford &
Shawe-Taylor (2002); McAllester (2003) in how strong a noise-resilience we require of the classiﬁer to provide the generalization guarantee.
The stronger the noise-resilience requirement, the more price we have to pay when we jump from the PAC-Bayesian guarantee on the stochastic classiﬁer to a guarantee on the deterministic classiﬁer.
We argue that our noise-resilience requirement is a much milder condition and therefore promises tighter guarantees .
More concretely, to arrive at a reasonable generalization guarantee in our setup, we would need that µ D and ˆ µ S are both only as large as O( 1 /√ m ) .
In other words, we would want the following for ( x ,y ) ∼ D and for ( x ,y ) ∼ S :
[Equation]
Previous works require a noise resilience condition of the form that with high probability a particular perturbation does not perturb the classiﬁer output on any input .
For example, the noise-resilience condition used in Neyshabur et al.
(2018) written in terms of our notations, would be:
[Equation]
The main difference between the above two formulations is in what makes a particular perturbation (un)favorable for the classiﬁer.
In our case, we deem a perturbation unfavorable only after ﬁxing the datapoint.
However, in the earlier works, a perturbation is deemed unfavorable if it perturbs the classiﬁer output sufﬁciently on some datapoint from the domain of the distribution.
While this difference is subtle, the earlier approach would lead to a much more pessimistic analysis of these perturbations.
In our analysis, this weakened noise resilience condition will be critical in analyzing the Gaussian perturbations more carefully than in Neyshabur et al.
(2018) i.e., we can bound the perturbation in the classiﬁer output more tightly by analyzing the Gaussian perturbation for a ﬁxed input point.
Note that one way our noise resilience condition would seem stronger in that on a given datapoint we want less than 1 /√ m mass of the perturbations to be unfavorable for us, while in previous bounds, there can be as much as 1 / 2 probability mass of perturbations that are unfavorable.
In our analysis, this will only weaken our generalization bound by a ln √ m factor in comparison to previous bounds (while we save other signiﬁcant factors).
C. P ROOF FOR T HEOREM C.
Proof.
The starting point of our proof is a standard PAC-Bayesian theorem McAllester (2003) which bounds the generalization error of a stochastic classiﬁer.
Let P be a data-independent prior over the parameter space.
Let L(W , x ,y ) be any loss function that takes as input the network parameter, and a datapoint x and its true label y and outputs a value in [ 0 , 1 ] .
Then, we have that, with probability 1 − δ over the draw of S ∼ D m , for every distribution Q over the parameter space, the following holds:
[Equation]
In other words, the statement tells us that except for a δ proportion of bad draws of m samples, the test loss of the stochastic classiﬁer ˜ W ∼ Q would be close to its train loss.
This holds for every possible distribution Q , which allows us to cleverly choose Q based on S .
As is the convention, we choose Q to be the distribution of the stochastic classiﬁer picked from N(W ,σ 2 I ) i.e., a Gaussian perturbation of the deterministic classiﬁer W .
R ELATING TEST LOSS OF STOCHASTIC CLASSIFIER TO DETERMINISTIC CLASSIFIER .
Now our task is to bound the loss for the deterministic classiﬁer W , Pr ( x ,y )
∼D [∃ r ∣ T r (W , x ,y ) < 0 ] .
To this end, let us deﬁne the following margin-based variation of this loss for some c ≥ 0 :
[Equation]
and so we have Pr ( x ,y )
∼D [∃ r ∣ T r (W , x ,y ) < 0 ]
=
E ( x ,y )∼D
[L 0 (W , x ,y )] .
First, we will bound the expected L 0 of a deterministic classiﬁer by the expected L 1 / 2 of the stochastic classiﬁer; then we will bound the test L 1 / 2 of the stochastic classiﬁer using the PAC-Bayesian bound.
We will split the expected loss of the deterministic classiﬁer into an expectation over datapoints for which it is noise-resilient with respect to Gaussian noise and an expectation over the rest.
To write this out, we deﬁne, for a datapoint ( x ,y ) , N (W , x ,y ) to be the event that W is noise-resilient at ( x ,y ) as deﬁned in Equation 3 in the theorem statement:
E ( x ,y )
∼D [L 0 (W , x ,y )]
=
E ( x ,y )∼D
[L 0 (W , x ,y )∣ N
(W , x ,y )]
Pr ( x ,y )∼D
[ N (W , x ,y )] + E ( x ,y )
∼D [L 0 (W , x ,y )∣ ¬ N (W , x ,y )] [Table]
≤ Pr ( x ,y )∼D [¬ N (W , x ,y )] [Table] [Equation]
[Equation]
To further continue the upper bound on the left hand side, we turn our attention to the stochastic classiﬁer’s loss on the noise-resilient part of the distribution D (we will lower bound this term in terms of the ﬁrst term on the right hand side above).
For simplicity of notations, we will write D ′ to denote the distribution D conditioned on N (W , x ,y ) .
Also, let U ( ˜ W , x ,y ) be the favorable event that for a given data point ( x ,y ) and a draw of the stochastic classiﬁer, ˜ W , it is the case that for every r , ∣ T r (W , x ,y ) − T r ( ˜ W , x ,y )∣ ≤ ∆ r / 2 .
Then, the stochastic classiﬁer’s loss L 1 / 2 on D ′ is:
[Equation]
splitting the inner expectation over the favorable and unfavorable perturbations, and using linearity of expectations,
[Equation]
to lower bound this, we simply ignore the second term (which is positive)
[Equation]
Next, we use the following fact: if L 1 / 2 ( ˜ W , x ,y ) = 0 , then for all r , T r ( ˜ W ,x,y ) ≥ ∆ r / 2 and if ˜ W is a favorable perturbation of W , then for all r , T r (W ,x,y ) ≥ T r ( ˜ W , x ,y )
− ∆ r / 2 >i.e., L 1 / 2 ( ˜ W , x ,y ) = 0 implies L 0 (W , x ,y )
= 0 .
Hence if ˜ W is a favorable perturbation then,
L 1 / 2 ( ˜ W , x ,y ) ≥ L 0 (W , x ,y ) .
Therefore, we can lower bound the above expression by replacing the stochastic classiﬁer with the deterministic classiﬁer (and thus ridding ourselves of the expectation over Q ):
[Equation]
Since the favorable perturbations for a ﬁxed datapoint drawn from D ′ have sufﬁciently high probability (that is, Pr ˜ W∼ Q [ U ( ˜ W , x ,y )] ≥ 1 − 1 /√ m ), we have:
[Equation]
Thus, we have a lower bound on the stochastic classiﬁer’s loss that is in terms of the deterministic classiﬁer’s loss on the noise-resilient datapoints.
Rearranging it, we get an upper bound on the latter:
[Equation]
Thus, we have an upper bound on the expected loss of the deterministic classiﬁer W on the noiseresilient part of the distribution.
Plugging this back in the ﬁrst term of the upper bound on the deterministic classiﬁer’s loss on the whole distribution D in Equation 5 we get :
E
[Equation]
x ,y )
∼D
[L
[Equation]
W , x ,y )]
[Equation]
˜
[Equation]
x
[Equation]
y
[Equation]
[Equation]
∆
[Equation]
rearranging, we get:
[Equation]
rewriting the expectation over D ′ explicitly as an expectation over D conditioned on N (W , x ,y ) , we get:
[Equation]
the ﬁrst term above is essentially an expectation of a loss over the distribution D with the loss set to be zero over the non-noise-resilient datapoints and set to be L 1 / 2 over the noise-resilient datapoints; thus we can upper bound it with the expectation of the L 1 / 2 loss over the whole distribution D :
[Equation]
Now observe that we can upper bound the ﬁrst term here using the PAC-Bayesian bound by plugging in L 1 / 2 for the generic L in Equation 4; however, the bound would still be in terms of the stochastic classiﬁer’s train error.
To get the generalization bound we seek, which involves the deterministic classiﬁer’s train error, we need to take one ﬁnal step mirroring these tricks on the train loss.
R ELATING THE STOCHASTIC CLASSIFIER ’ S TRAIN LOSS TO DETERMINISTIC CLASSIFIER ’ S TRAIN LOSS .
Our analysis here is almost identical to the previous analysis.
Instead of working with the distribution D and D ′ we will work with the training data set S and a subset of it S ′ for which noise resilience property is satisﬁed by W .
Below, to make the presentation neater, we use ( x ,y ) ∼ S to denote uniform sampling from S .
First, we upper bound the stochastic classiﬁer’s train loss ( L 1 / 2 ) as follows:
[Equation]
splitting over the noise-resilient points S ′ ( ( x ,y ) ∈ S for which N (W , x ,y ) holds) like in Equation 5, we can upper bound as:
[Equation]
We can upper bound the ﬁrst term by ﬁrst splitting it over the favorable and unfavorable perturbations like we did before:
[Equation]
To upper bound this, we apply a similar argument.
First, if L 1 / 2 ( ˜ W , x ,y ) = 1 , then ∃ r such that
T r ( ˜ W ,x,y ) <
∆ r / 2 and if ˜ W is a favorable perturbation then for that value of r , T r (W ,x,y ) <
T r ( ˜ W , x ,y )
+ ∆ r / 2 < ∆ r .
Thus if ˜ W is a favorable perturbation then, L 1 (W , x ,y ) = 1 whenever L 1 / 2 ( ˜ W , x ,y )
=
i.e., L 1 / 2 ( ˜ W , x ,y ) ≤ L 1 (W , x ,y ) .
Next, we use the fact that the unfavorable perturbations for a ﬁxed datapoint drawn from S ′ have sufﬁciently low probability i.e.,
Pr ˜ W∼ Q [¬ U ( ˜ W , x ,y )] ≤ 1 /√ m .
Then, we get the following upper bound on the above equations, by replacing the stochastic classiﬁer with the deterministic classiﬁer (and thus ignoring the expectation over Q ):
[Equation]
⎤⎥⎥⎥⎥⎥⎥⎥⎦
[Equation]
⎤⎥⎥⎥⎥⎥⎥⎥⎦
[Equation]
Plugging this back in the ﬁrst term of Equation [Equation] we get:
[Equation]
since the ﬁrst term is effectively the expectation of a loss over the whole distribution with the loss set to be zero on the non-noise-resilient points and set to L 1 over the rest, we can upper bound it by setting the loss to be L 1 over the whole distribution:
[Equation]
Applying the above upper bound and the bound in Equation 6 into the PAC-Bayesian result of
Equation 4 yields our result (Note that combining these equations would produce the term 1 √ m + 1 √ m −which is at most 2 √ m − 1 , which we reﬂect in the ﬁnal bound. ).
D P ROOF FOR T HEOREM 3.
In this section, we present the proof for the (abstract) generalization guarantee presented in Section 3.
Our proof is based on the following recursive inequality that holds for all r <
R (we will prove a similar, but slightly different inequality for r
= R ):
[Equation]
⎠ ·„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„‚„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„¶ generalization error for condition r
That is, we bound the probability mass of test points such that any one of the ﬁrst r condition fails, in terms of the probability mass of points where one of the ﬁrst r − 1 conditions fail and an error that corresponds to how much error there can be in generalizing the r th condition from the training data.
First, we consider the base case when r = 1 , and apply the PAC-Bayes-based guarantee from Theorem C.1 on the singleton set of functions consisting only of T 1 .
First we have from our assumption that on all the training data, the condition T 1 (W , x ,y ) >
∆ ⋆ 1 is satisﬁed.
Thus, the ﬁrst term in the upper bound in Theorem C.1 is zero.
Next, we can show that the terms ˆ µ S and ˆ µ D would be zero too.
This follows from the fact that Equation 2 holds in this framework.
Applying this equation for r = 1 , for σ = σ ⋆ , we get that for all ∀ x ∈ X the following inequality holds:
[Equation]
Since, σ ⋆ was chosen such that ∆ 1 ( σ ⋆ ) ≤ ∆ ⋆ 1 , we effectively have:
[Equation]
Then, using Equation 1, we have that
[Equation]
Effectively this establishes that the noise-resilience requirement of Equation 3 holds on all possible inputs, for T 1 , thus proving our claim that the terms ˆ µ S and ˆ µ D would be zero.
Thus, we will get that
[Equation]
which proves the recursion statement for the base case.
To prove the recursion for some arbitrary r < R , we again apply the PAC-Bayes-based guarantee from Theorem C.1, but on the set of ﬁrst r functions.
Again, we will have that the ﬁrst term in the guarantee would be zero, since the corresponding conditions are satisﬁed on the training data.
Now, to bound the proportion of bad points ˆ µ S and ˆ µ D , we claim that the network is noise-resilient according to Equation 3 for any input that satisﬁes the r − 1 conditions, T q (W , x ,y ) > 0 for q = 1 , 2 , ⋯ ,r − 1 .
Since these conditions are assumed to be satisﬁed by a margin on the training data, it immediately implies that ˆ µ S ({( T q , ∆ q )} rq = 1 , W) = 0 .
On the test data, we can bound µ D ({( T q , ∆ q )} rq = 1 , W) in terms of Pr ( x ,y )
∼D [∃ q < r T q (W , x ,y ) < 0 ] , thus giving rise to the recursion in Equation 8.
Now, to prove our claim, consider an input x such that T q (W , x ,y ) > 0 for q = 1 , 2 , ⋯ ,r − 1 .
We then have the following:
[Equation]
x ,y )
[Equation]
∆
[Equation]
+
[Equation]
] because ∆ q ( σ ⋆ ) ≤ ∆ ⋆ q [Equation] l
[Equation]
] the above inequality follows from the constraint we imposed on T r in Equation
[Equation]
Thus, we have that x satisﬁes the noise-resilience condition from Equation 3 if it also satisﬁes
T q (W , x ,y ) > 0 for q = 1 , 2 , ⋯ ,r − 1 .
This proves the recursion in Equation 8.
Finally, we can apply a similar argument for T R with a small change since the ﬁrst term in the guarantee from Theorem C.1 is not explicitly assumed to be zero; we will get an inequality in terms of the number of training points that are not classiﬁed correctly by a margin, giving rise to the margin-based bound: [Equation] +
[Equation]
√ 2 KL (N(W [Equation] σ 2 I )∥ P ) m −⎞ ⎠
By using the fact that the test error is upper bounded by the left hand side in the above equation, applying the recursion on the right hand side R times, we get our ﬁnal result.
E P ERTURBATION B OUNDS E. O VERVIEW OF THE BOUNDS .
In this section, we will quantify the noise resilience of a network in different aspects.
Each of our bounds has the following structure: we ﬁx an input point ( x ,y ) , and then say that with high probability over a Gaussian perturbation of the network’s parameters, a particular input-dependent property of the network (say the output of the network, or the pre-activation value of a particular unit h at a particular layer d , or say the Frobenius norm sof its active weight matrices), changes only by a small magnitude proportional to the variance σ 2 of the Gaussian perturbation of the parameters.
A key feature of our bounds is that they do not involve the product of the spectral norm of the weight matrices and hence save us an exponential factor in the ﬁnal generalization bound.
Instead, the bound in the perturbation of a particular property will be in terms of i) the magnitude of the some ‘preceding’ properties (typically, these are properties of the lower layers) of the network, and ii) how those preceding properties themselves respond to perturbations.
For example, an upper bound in the perturbation of the d th layer’s output would involve the (cid:96) 2 norm of the lower layers d ′ <
d , and how much they would blow up under these perturbations.
More concretely, the bound on the perturbation in the norm of rows of the Jacobian d / d ′ depends on i) the magnitude of the norms related to all the lower layer Jacobians, and also the ‘higher’ Jacobians of layer d i.e., d / d ′′ for d ′′ > d ′ and ii) the perturbation suffered by the higher Jacobians of layer d .
Second, the bound on the perturbation of both the (cid:96) 2 norm and the pre-activations of layer d depend on i) the maximum row (cid:96) 2 norm of the
Jacobians d / d ′ for each d ′ <
d and the (cid:96) 2 norms of all lower layers d ′ < d ii) the perturbation suffered by the (cid:96) 2 norms of all lower layers.
Intuitively, when the Jacobian of layer d with respect to some d ′ has small (cid:96) 2 row norms, it diminishes the effect of noise injected into the matrix immediately after layer d
/immediately before layer d ′ .
Similarly, when the (cid:96) 2 norm of the output of the d th layer is small, it diminishes the effect of noise injected into the subsequent matrix.
Finally, assuming sufﬁciently large pre-activation values prevents hidden units from crossing non-linearities with high probability; this then allows us to propagate the
Gaussian noise ‘linearly’ through the network.
E. S OME NOTATIONS .
To formulate our lemma statement succinctly, we design a notation wherein we deﬁne a set of ‘tolerance parameters’ which we will use to denote the extent of perturbation suffered by a particular property of the network.
Let ˆ C denote a ‘set’ (more on what we mean by a set below) of positive tolerance values, consisting of the following elements:
ˆ ζ d / d ′ for each layer d = 1 , 2 , ⋯ ,D , and d ′ = 1 , ⋯ ,d (a tolerance value for the (cid:96) 2 norm of each row of the Jacobians at layer d ) 2.
ˆ α d , for each layer d = 1 , ⋯ ,D − 1 (a tolerance value for the (cid:96) 2 norm of the output of layer d ) 3.
ˆ γ d for each layer d = 1 , ⋯ ,D .
(a
tolerance value for
the
magnitude
of the
pre activations
of
layer d )
Notes about (abuse of) notation:
• We call ˆ C a ‘set’ to denote a group of related constants into a single symbol.
Each element in this set has a particular semantic associated with it, unlike the standard notation of a set, and so when we refer to, say ˆ ζ d / d ′ ∈
ˆ C , we are indexing into the set to pick a particular element.
• We will use the subscript ˆ C d to index into a subset of only those tolerance values corresponding to layers from 1 until d .
Next we deﬁne two events.
The ﬁrst event formulates the scenario that for a given input, a particular perturbation of the weights until layer d brought about very little change in the properties of these layers (within some tolerance levels).
The second event formulates the scenario that the perturbation did not ﬂip the activation states of the network.
Deﬁnition E.1.
Given an input x , and an arbitrary set of constants ˆ C ′ , for any perturbation U of W , we denote by P ERT -B OUND (W +U , ˆ C ′ , x ) the event that:
• for each ˆ ζ ′ d / d ′ ∈
ˆ C ′ , the maximum perturbation in the (cid:96) 2 norm of a row of the Jacobian d / d ′ is bounded as max h ∣∥ J d / d ′ ( x ; W)[ h ]∥ − ∥ J d / d ′ ( x ; W +U )
[ h ]∥∣ ≤ ˆ ζ ′ d / d ′ .
• for each ˆ α ′ d ∈ ˆ C ′ , the perturbation in the (cid:96) 2 norm of layer d activations is bounded as ∣∥ f d ( x ; W)∥ − ∥ f d ( x ; W +U )∥∣ ≤ ˆ α ′ d . • for each ˆ γ ′ d ∈ ˆ C ′ , the maximum perturbation in the preactivation of hidden units on layer d is bounded as max
h ∣ f d ( x ; W)
[ h ] − f d ( x ; W +U )
[ h ]∣ ≤ ˆ γ ′ d .
N OTE :
A subtle point in the above deﬁnition (which we take advantage of, when we state our results) is that if we supply only a subset of ˆ C (say ˆ C d instead of the whole of ˆ C ) to the above event,
P ERT -B OUND (W + U , ⋅ , x ) , then it would denote the event that the perturbations suffered by only that subset of properties is within the respective tolerance values.
Next, we deﬁne the event that the perturbations do not affect the activation states of the network.
Deﬁnition E.2.
For any perturbation U of the matrices W , let U NCHANGED -A CTS d (W + U , x ) denote the event that none of the activation states of the ﬁrst d layers change on perturbation.
E. M AIN LEMMA .
Our results here are styled similar to the equations required by Equation 2 presented in the main paper.
For a given input point and for a particular property of the network, roughly, we bound the the probability that a perturbation affects the value of the property while none of the preceding preceding properties themselves are perturbed beyond a certain tolerance level.
Lemma E.1.
Let ˆ C be a set of constants.
For any ˆ δ > 0 , deﬁne ˆ C ′ in terms of ˆ C , for all d = 1 , 2 , ⋯ ,D as follows:
[Equation]
√ 2ln
[Table]
[Equation]
Let U d be sampled entrywise from N( 0 ,σ 2 ) for any d .
Then, the following statements hold good:
Bound on perturbation of (cid:96) 2 norm on the rows of the Jacobians d / d ′ .
For all d ′ =
d,d − 1 , ⋯ , 1 :
[Equation]
ERT
[Equation]
OUND
[Equation]
Bound on perturbation of of (cid:96) 2 norm of the output of layer d .
For all d = 1 , 2 , ⋯ ,D ,
[Equation]
CTS
[Equation]
W
[Equation]
Bound on perturbation of pre-activations at layer d .
For all d = 1 , 2 , ⋯ ,D , Pr
[Equation]
W [Equation] U , { ˆ γ
[Equation]
} , x
[Equation]
Proof.
For the most part of this discussion
[Equation]
we will consider a perturbed network where all the hidden units are frozen to be at the same activation state as they were at, before the perturbation (we denote the weights of such a network by W[+U] and its output at the d th layer by f d ( x ; W[+U]) ).
By having the activations states frozen, the Gaussian perturbations propagate as Gaussian perturbations and we can enjoy the well-established properties of the Gaussian even after they propagate.
N OTE .
We will prove the parts of the above lemma in an order different from how it is stated; speciﬁcally, we will study the perturbation of the Jacobian’s row-wise (cid:96) 2 norm the last, as it is slightly more involved.
However, the reason for stating the lemma in that order is to keep it consistent with the order in which our generalization analysis applies parts of it later.
P ERTURBATION BOUND ON THE (cid:96) 2 NORM OF LAYER d .
We bound the change in the (cid:96) 2 norm of the d th layer’s output by applying a triangle inequality 5 after splitting it into a sum of vectors.
Each summand here (which we deﬁne as v d ′ for each d ′ ≤ d ) is the difference in the d th layer output on introducing noise in weight matrix d ′ after having introduced noise into all the ﬁrst d ′ − 1 weight matrices.
[Equation]
because the activations are ReLU, we can replace this with the perturbation of the pre-activation
[Equation]
[Equation]
⎞⎟⎟⎟⎠XXXXXXXXXXXXXXXXXXX
[Equation]
[Equation]
Here, v d ′ ,h is the perturbation in the preactivation of hidden unit h on layer d ′ , brought about by perturbation of the d ′ th weight matrix in a network where the ﬁrst d ′ − 1 weight matrices have already been perturbed. proof.
Now, for each h
[Table]
we bound v d ′ ,h in Equation 9.
Since the activations have been frozen we can rewrite each v d ′ ,h as the product of the h th row of the unperturbed network’s Jacobian d / d ′ , followed by only the perturbation matrix U d ′ , and then the output of the layer d ′ − 1 .
Concretely, we have 6 7 : v d ′ ,h = 1 × H d ′
[Table]
spherical Gaussian
What do these random variables v d ′ ,h look like?
Conditioned on U d ′ − 1 , the second part of our expansion of v d ′ ,h , namely, U d ′ f d ′ − 1 ( x ; W[+U d ′ − 1 ]) is a multivariate spherical Gaussian (see Lemma B.2) of the form N( 0 ,σ
∥ f d ′ − 1 ( x ; W[+U d
′ − 1 ])∥ 2 I ) .
As a result, conditioned on U d ′ − 1 , v d ′ ,h is a univariate Gaussian N( 0 ,σ 2 ∥ J d / d [Equation] ( x ; W)
[ h ]∥
∥ f d ′ [Equation] 1 ( x ; W[+U d
[Equation] 1 ])∥ 2 ) .
Then, we can apply a standard Gaussian tail bound (see Lemma B.1) to conclude that with probability 1 − ˆ δ / DH over the draws of U d ′ (conditioned on any U d ′ − 1 ), v d ′ ,h is bounded as:
[Equation]
Then, by a union bound over all the hidden units on layer d , and for each d ′ , we have that with probability 1 − ˆ δ , Equation 9 is upper bounded as:
[Equation]
Finally, we prove the probability bound in the lemma statement.
To simplify notations, let us denote ˆ
C d −
⋃{ ˆ ζ d / d ′ } dd ′ = 1 by ˆ C prev .
Furthermore, we will drop redundant symbols in the arguments of the events we have deﬁned.
Then, recall that we want to we upper bound the following probability (we ignore the arguments W + U and x for brevity):
[Equation]
Recall that Equation 11 is a bound on the perturbation of the (cid:96) 2 norm of the d th layer’s output when the activation states are explicitly frozen.
If the perturbation we randomly draw happens to satisfy U NCHANGED -A CTS d − 1 then this bound holds good even in the case where the activation states are not explicitly frozen.
Furthermore, when P ERT -B OUND ( ˆ C prev ) holds, the bound in Equation 11 can be upper-bounded by ˆ α ′ d as deﬁned in the lemma statement, because under
P ERT -B OUND ( ˆ C prev ) , the middle term in Equation 11 can be upper bounded using triangle inequality as ∥ f d ′ − 1 ( x ; W[+U d
′ − 1 ])
∥ ≤ ∥
f d ′ − 1 ( x ; W)∥ + ˆ α d ′ − 1 .
Hence, the event above happens only for the perturbations for which Equation 11 fails and hence we have that the above is upper bounded by ˆ δ .
P ERTURBATION BOUND ON THE PREACTIVATION VALUES OF LAYER d .
Following the same analysis as above, the bound we are seeking here is essentially max h ∑ dd ′ =
∣ v d ′ ,h ∣ .
The bound follows similarly from Equation 10.
P
ON
(cid:96) d / d
We split this term like we did in the previous section, and apply triangle equality as follows:
[Equation]
⎞⎟⎟⎟⎟⎟⎠XXXXXXXXXXXXXXXXXXXXX
[Equation]
[Equation]
Here, we have deﬁned y d ′′ h to be the vector that corresponds to the difference in the h th row of the
Jacobian d / d ′ brought about by perturbing the d ′′ th weight matrix, given that the ﬁrst d ′′ − 1 matrices have already been perturbed.
We use h to iterate over the units in the d th layer and h ′ to iterate over the units in the d ′ th layer.
Now, under the frozen activation states, when we perturb the weight matrices from 1 uptil d ′ , since these matrices are not involved in the Jacobian d / d ′ , fortunately, the Jacobian d / d ′ is not perturbed (as the set of active weights in d / d ′ are the same when we perturb W as W[+U d ′ ] ).
So, we will only need to bound y d ′′ ,h,h ′ for d ′′ > d ′ .
What does the distribution of y d ′′ ,h,h ′ look like for d ′′ > d ′ ?
We can expand 8 y d ′′ ,h,h ′ as the product of i) the h th row of the Jacobian d / d ′′ ii) the perturbation matrix U d ′′ and iii)
the h ′
th column of the
Jacobian d ′ / d ′′ − 1 for the perturbed network:
[Equation]
[Table] spherical
[Table]
Conditioned on U d ′′ − 1 , the second part of this expansion, namely, U d ′′ J d ′′ − 1 / d ′ ( x ; W[+U d ′′ − 1 ])
[∶ ,h ′ ] is a multivariate spherical
Gaussian (see
Lemma
B.2) of the form
N( 0 ,σ 2 ∥ J d ′′ − 1 / d ′ ( x ; W[+U d ′′ − 1 ])
[∶ ,h ′ ]∥ 2 I ) .
As a result, conditioned on U d ′′ − 1 , y d ′′ ,h,h ′ is a univariate Gaussian N( 0 ,σ 2 ∥ J d / d ′′ ( x ; W)
[ h ]∥ 2 ∥ J d ′′ − 1 / d ′ ( x ; W[+U d ′′ − 1 ])
[∶ ,h ′ ]∥ 2 ) .
Then, by applying a standard Gaussian tail bound we have that with probability 1 − ˆ δDH 2 over the draws of U d ′′ conditioned on U d ′′ − 1 , each of these quantities is bounded as:
[Equation]
By a union bound, we then get that with probability 1 − ˆ δ over the draws of U d , we can upper bound
Equation 12 as:
[Equation]
Finally we get the result of the lemma by a similar argument as in the case of the perturbation bound on the output of each layer.
F P ROOF FOR T HEOREM 4.
In this section, we will prove our the generalization bound for the speciﬁc case of ReLU based deep networks, that we proved in our main paper.
F. N OTATIONS .
To make the presentation of our theorem cleaner, we will set up some notations.
First, we deﬁne the following positive constants which will correspond to norm bounds on different properties of the network on the training data :
ζ ⋆ d / d ′ for each layer d = 1 , 2 , ⋯ ,D , and d ′ = 1 , ⋯ ,d
(we will use this to bound (cid:96) 2 norms of rows in the Jacobians of the network for a training input) 2.
α ⋆ d , for each hidden layer d = 1 , ⋯ ,D − 1 , (we will use this to bound (cid:96) 2 norms of the outputs of the layers of the network on a training input) 3.
γ ⋆ d for each layer d = 1 , ⋯ ,D , (we will use this to bound magnitudes of the preactivations values of the network on a training input).
We will also deﬁne α ⋆ 0 to be an (cid:96) 2 -norm bound on the input x .
As we did in the previous section, to simplify our discussion, we will group these constants into a ‘set’ C ⋆ .
Given these training set related constants, we also deﬁne C † to be the following constants corresponding to weaker norm-bounds related to the test data :
ζ † d / d ′ ∶= 2 ζ ⋆ d / d ′ for each layer d = 1 , 2 , ⋯ ,D , and d ′ = 1 , ⋯ ,d
(we will use this to bound (cid:96)norms of rows in the Jacobians of the network for a test input) 2.
α † d ∶= 2 α ⋆ d , for each hidden layer d
= 1 , ⋯ ,D − 1 , (we will use this to bound (cid:96) 2 norms of the outputs of the layers of the network on a test input) 3.
γ † d ∶= γ ⋆ d / 2 for each layer d = 1 , ⋯ ,D , (we will use this to bound magnitudes of the preactivations values of the network on a test input).
Note about (abuse of) notation: We reiterate a point about our notation which we also made in
Appendix E. We call C ⋆ and C † a ‘set’ to denote a group of related constants by a single symbol.
Each element in this set has a particular semantic associated with it, unlike the standard notation of a set.
Now, for any given set of constants C , for a particular weight conﬁguration W , and for a given input x , we deﬁne the following event which holds when the network satisﬁes certain norm-bounds deﬁned by the constants C (that are favorable for noise-resilience).
Deﬁnition F.1.
For a set of constants C , for network parameters W and for any input x , we deﬁne
N ORM -B OUND (W , C , x ) to be the event that all the following hold good:
for all ζ d / d ′ ∈ C , max h ∥ J d / d ′ ( x ; W)[ h ]∥ < ζ d / d ′ (Rows of Jacobian do not have too large an (cid:96) 2 norm).
for all α d ∈ C , ∥ f d ( x ; W)∥ < α d (Output of the layer does not have too large an (cid:96) 2 norm).
for all γ d ∈ C , min h ∣ f d ( x ; W)
[ h ]∣ > γ d .
(Pre-activation values are not too small).
N OTE :
(Similar to a note under Deﬁnition E.1)
A subtle point in the above deﬁnition (which we will make use of, to state our theorems) is that if we supply only a subset of C to the above event,
N ORM -B OUND (W , ⋅ , x ) then it would denote the event that only those subset of properties satsify the respective norm bounds.
F. M AIN R ESULT
Below, we present our main result for this section, a generalization bound on a class of networks that satisfy certain norm bounds on the training data .
We provide a more intuitive presentation of these bounds after the proof in Appendix F.4.
Theorem.
4.1 Consider a set of positive training-set related constants C ⋆ .
Then, for any δ > 0 , with probability 1 − δ over the draw of samples from D m , for any W such that, for every ( x ,y ) ∈ S ,
N ORM -B OUND (W , C ⋆ , x ) holds, we have that:
[Equation]
where 1 σ ⋆ is 40 √ H
[Equation]
-preact } ; here we deﬁne:
[Equation]
R EMARK
Observe that the above generalization bound applies to a speciﬁc class of W that satisﬁes certain norm bounds on the training data.
One could extend the bound to all values of W for theoretical purposes, by ‘covering’ all possible values of the constants deﬁned in the norm bounds (see Neyshabur et al.
(2018) for example) at the cost of extra factors.
However, from a practical point of view, the statement in this form is sufﬁcient because we are interested in a generalization bound for a small class of networks that SGD restricts itself to, and not all possible networks.
In practice, to apply this bound, one could run SGD on many random draws of the training dataset and identify training-set related constants C ⋆ such that the network learned from all these random draws satisfy the norm bounds deﬁned by C ⋆ .
F. P ROOF FOR T HEOREM 4.
Proof.
To apply the framework
[Equation]
we will have to ﬁrst deﬁne the abstract functions T r and quantities ∆ ⋆ r used in Theorem 3.1 .
These functions will be composed of the following properties corresponding to different values of d,d ′ and h (the ρ functions described in the main framework): ∥ J d / d ′ ( x ; W)[ h ]∥ , ∥ f d ( x ; W)∥ and f d ( x ; W)
[ h ] .
Deﬁnition F.2.
We deﬁne the following set of functions T ⋆ , and the corresponding set of margin values C ▵ :
[Equation]
for d
[Equation]
D
[Equation]
for d = 1 , 2 , ⋯ ,D
[Equation]
for d = 1 , 2 , ⋯ ,D
[Equation]
and for the output layer D :
[Equation]
We will use the notation C ▵ / 2 to denote the values in the margin set divided by 2 .
Note that the above functions satisfy the constraint Equation 1 as required by our abstract framework.
O N THE CHOICE OF THE ABOVE FUNCTIONS AND MARGIN VALUES .
Below, we justify our choice of these functions and margins, by describing how inequalities of the form T r (W , x ,y ) <
∆ + r ⋆ or
T r (W , x ,y ) < 0 convert to norm-bounds on the input-dependent properties of the network.
When we apply the above quantities of T and ∆ ⋆ in Theorem C.1, the guarantees will be in terms of the test loss Pr ( x,y )
∼D [ T (W , x ) < 0 ] and train loss Pr ( x,y )∼ S
[ T (W , x ) < ∆ ] which would be equivalent to the norm-bounds shown in the table below.
Observe that the second column (the 0-margin error column) corresponds to the event that a datapoint does not satisfy the weaker normbound deﬁned by test-set constants C † (which was deﬁned in the beginning of this section).
On the other hand, the last column (the ∆ -margin error column) corresponds to the event that a datapoint does not satisfy the norm-bounds deﬁned by the train-set constants C ⋆ .
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Now to apply the abstract generalization bound in Theorem 3.1, recall that we need to come up with an ordering of the functions above such that we can realize an inequality of the form given in Equation 2.
To this end, we induce an ordering based on the perturbation bounds presented in
Lemma E.1.
The order in which we traverse the properties is as follows.
We will go from layer 1 uptil D ; within each layer we will ﬁrst deal with the Jacobians d / d , d / d − 1 , so on until d / 1 in that order 9 .
Then we will deal with the (cid:96) 2 norm of the output of the layer and the pre-activation values (the order between these two will not actually matter).
Note that ∣ T ⋆ ∣ ≤ 3 D 2 since there are at most D + 2 properties in each layer.
Deﬁnition F.3.
Sequence 1 :
T 1 / 1 , jac ,
T 1 , output ,
T 1 , preact , followed by
T 2 / 1 , jac ,
T 2 , output ,
T 2 , preact , ⋮ ⋮ ⋮ followed by
T D / D, jac , ⋯ ,T D / 1 , jac ,
T D, output
T D, preact .
The next step in our proof is to show that inequalities of the form presented in Equation 2 hold good.
To do this, we instantiate Lemma E.1 with σ = σ ⋆ as in Theorem 4.1, ˆ δ = 1 3 D 2 √ m and ˆ C = C ▵ / 2 .
Then, it can be veriﬁed that the values of the tolerance parameters ˆ C ′ in Lemma E.1 can be upper bounded by the corresponding value in C ▵ / 2 .
Succinctly, we say:
[Equation]
Given this, we will focus on showing that an inequality of the form Equation 2 holds for
T d / d ′ , jac .
A similar approach would apply for the other properties.
For brevity, we will deﬁne
C
[Equation]
⋃{
[Equation]
and ˆ
[Equation]
ˆ [Equation] First, we note that the precondition for Equation 2 translates to N ORM -B OUND (W + U , C †prev , x ) (as shown in Table 1).
Given this condition, we ﬁrst argue that if P ERT -B OUND (W + U , ˆ C prev , x ) holds, then so does U NCHANGED -A CTS d − 1 (W + U , x ) .
This is because, the event P ERT -B OUND (W +
U , ˆ C prev , x ) implies that the pre-activation values of layer d − 1 suffer a perturbation of at most ˆ γ d − 1 , which from Equation 13 is at most γ ▵ d − 1 / 2 = γ ⋆ d − 1 / 4 i.e., max h ∣ f d − 1 ( x ; W + U)
[ h ] − f d − 1 ( x ; W)[ h ]∣ ≤ γ ⋆ d − 1 / 4 .
However, since N ORM -B OUND (W , C †prev , x ) holds, we have that the preactivation values of this layer have a magnitude of at least γ † d − 1 = γ ⋆ d − 1 / 2 before perturbation i.e., min h ∣ f d − 1 ( x ; W)
[ h ]∣ ≥ γ ⋆ d − 1 / 2 .
From these two equations, we have that the hidden units even at layer d − 1 of the network do not change their activation state (i.e., the sign of the pre-activation does not change) under this perturbation.
We can similarly argue for the layers below d − 1 , thus proving that U NCHANGED -A CTS d − 1 (W + U , x ) holds under P ERT -B OUND (W + U , ˆ C prev , x ) .
Then, observe that the guarantee from Lemma E.1 boils down to:
[Equation]
Now, from Equation 13 we have that, if P ERT -B OUND (W + U , { ˆ ζ ′ d / d ′ } , x ) holds, then
P ERT -B OUND (W + U , { ζ ▵
[Equation]
/ d ′ / 2 } , x ) holds.
This implies that we can replace the ﬁrst term as below:
[Equation]
This inequality is precisely of the same form as Lemma E.1.
Through identical arguments for the other functions, our generalization bound holds.
Note that in the ﬁnal bound, we use Lemma B.3 to simplify the KL-divergence term between the posterior centered at W and the prior at the random initialization Z
. F. I NTERPRETING THE BOUNDS .
Below we show empirically how the terms B jac , B out , B hidden-preact , B output-preact vary with the width of a network after ﬁxing the depth D of the network to be 6 .
For all examples, we use SGD with learning rate 0 .
01 to train the network on the MNIST dataset by minimizing cross entropy loss until all datapoints are classiﬁed by a margin of at least 10 .
We see in Figures 3 that the ﬁrst two terms B jac , B out are width-independent; on the other hand,
B output-preact decreases with width as 1 /√ H and this is because the numerator of this term is widthindependent, while the denominator of this term, which is
√
Hγ class , increases with width (we have chosen γ class to be a constant of 10 in all our experiments).
Furthermore, in Figure 2 we see that all these three terms show an increase with depth that is only nearly linear or mildly exponential – these terms certainly do not grow as rapidly as the product of the spectral norms.
This is because, in practice the terms ζ ⋆ d / d ′ and α ⋆ d are width-independent, and do not grow as rapidly as the product of the spectral norms.
As far as the term B hidden-preact is concerned, recall from our discussion in the main paper that the minimum pre-activation value γ ⋆ d of the network tends to be quite small in practice, and therefore the term B hidden-preact can be quite large; furthermore, this term exhibits considerable variance across different widths/depths and different training runs.
As we noted, for future work, it would be important to understand how the dependence of the bound on this term can be better improved.
Theoretically speaking, the best scenario for B hidden-preact can be realized when the preactivation values of each layer (which has a total (cid:96) 2 norm that is width-independent in practice) are equally spread out across the hidden units.
Then we will have that the smallest pre-activation value is guaranteed to be as large as Ω ( 1 /√ H ) .
Then, B hidden-preact would behave like the other terms, in terms of its width and depth dependence.
In fact, we simulate this in practice, by plotting a redeﬁned version of
B hidden-preact where instead of plugging in the minimum pre-activation value, we plug in the median pre-activation value (that is, for each datapoint we compute the median pre-activation value for that layer, and then ﬁnd the lowest value of this median across all datapoints).
We observe that since the median pre-activation value is only as small as Ω ( 1 /√ H ) in practice, the ‘median-based- B hidden-preact ’ has similar variation with respect to the width and the depth as the other terms B jac , B out , B output-preact and is also quite small.
See Figure 4 and Figure 2 for how the original B hidden-preact and the ‘medianbased- B hidden-preact ’ vary with width and depth.
[Figure]
ppe r bound
[Figure] [Figure]
G C OMPARISON OF OUR NOISE - RESILIENCE CONDITIONS WITH RELATED WORK
Recall from the discussion in the introduction in the main paper that, prior works (Neyshabur et al., 2017; Arora et al., 2018) have also characterized noise resilience in terms of conditions on the interactions between the activated weight matrices.
Below, we discuss the conditions assumed by these works, which parallel the conditions we have studied in our paper (such as the bounded (cid:96) 2 norm in each layer).
There are two main high level similarities between the conditions studied across these works.
First, these conditions – all of which characterize the interactions between the activated weights matrices in the network – are assumed only for the training inputs; such an assumption implies noise-resilience of the network on training inputs.
Second, there are two kinds of conditions assumed.
The ﬁrst kind allows one to bound the propagation of noise through the network under the assumption that the activation states do not ﬂip; the second kind allows one to bound the extent to which the activation states do ﬂip.
C ONDITIONS IN N EYSHABUR ET AL . (2017)
Using noise-resilience conditions assumed about the network on the training data, Neyshabur et al.
(2017) derive a PAC-Bayes based generalization bound on a stochastic network.
The ﬁrst condition in Neyshabur et al.
(2017) characterizes how the
Jacobians of different parts of the network interact with each other.
Speciﬁcally, consider layers d,d ′ and d ′′ such that d ′′ ≤ d ′ ≤ d .
Then, consider the Jacobian of layer d ′ with respect to layer d ′′ and the
Jacobian of layer d with respect to d ′ .
Then, they require that ∥ J d / d ′ ( x ; W)∥ F ∥ J d ′ − 1 / d ′′ ( x ; W)∥ F =
O(∥ J d / d ′′ ( x ; W)∥ F ) .
This speciﬁc condition allows one to bound how the noise injected into the parameters propagate through the network under the assumption that the activation states do not ﬂip.
In our paper, we pick an orthogonal approach by assuming an upper bound on the Jacobian (cid:96) 2 norms and the layer output norms, which allows us to bound the propagation of noise under unchanged activation states.
The second condition in Neyshabur et al.
(2017) is that under a noise of variance σ 2 , the number of units that ﬂip their activation state in a particular layer must be bounded as O( Hσ ) i.e., smaller the noise, the smaller the proportion of units that ﬂip their activation state.
This condition is similar to (although milder than) our lower bounds on the magnitudes of the pre-activation values (which allow us to pick a sufﬁciently large noise that does not ﬂip the activation states).
Note that a bound on the Jacobian norms corresponds to a bound on the weights input to the active units in the network.
However, since Neyshabur et al.
(2017) allow a few units to ﬂip activation states, they additionally require a bound on the weights input to the inactive units too.
Speciﬁcally, for every layer, the maximum row (cid:96) 2 norm of the weight matrix W i is upper bounded in terms of the Frobenius norm of the Jacobian J d / d − 1 ( x ; W) .
C ONDITIONS IN A RORA ET AL . (2018)
In contrast to our work and Neyshabur et al. (2017), Arora et al.
(2018) use their assumed noise-resilience conditions to derive a bound on a compressed network.
Another small technical difference here is that, the kind of noise analysed here is Gaussian noise injected into the activations of each layer of the network (and not exactly the weights).
The ﬁrst condition here characterizes the interaction between the Jacobian of layer d with respect to d ′ and the output of layer d ′ .
Speciﬁcally, this is a lower bound on the so-called ‘interlayer cushion’, which is evaluated as
[Equation]
Essentially when the interlayer cushion is sufﬁciently large, it means that the output of layer d ′ is well-aligned with the larger singular directions of the Jacobian matrix above it; as a result it can be shown that noise injected at/below layer d ′ diminishes as it propagates through the weights above layer d ′ , assuming the activation states do not ﬂip.
Again, our analysis is technically orthogonal to this style of analysis as we bound the propogation of the noise under unchanged activation states assuming that the norms of the Jacobians and the layer outputs are bounded.
Another important condition in Arora et al.
(2018) is that of “interlayer smoothness” which effectively captures how far the set of activation states between two layers, say d ′ and d , ﬂip under noise.
Roughly speaking, the assumption made here is that when noise is injected into layer d ′ , there is not much difference between a) the output of the d th layer with the activation states of the units in layers d ′ until d frozen at their original state and b) the output of the d th layer with the activation states of the units in layers d ′ to d allowed to ﬂip under the noise.
As stated before, this condition is a relaxed version of our condition that essentially implies that none of the activation states ﬂip.