Beneﬁts of Additive Noise in Composing Classes with Bounded Capacity
Abstract
We observe that given two (compatible) classes of functions F and H with small capacity as measured by their uniform covering numbers, the capacity of the composition class H  F can become prohibitively large or even unbounded.
We then show that adding a small amount of Gaussian noise to the output of F before composing it with H can effectively control the capacity of H  F , offering a general recipe for modular design.
To prove our results, we deﬁne new notions of uniform covering number of random functions with respect to the total variation and Wasserstein distances.
We instantiate our results for the case of multi-layer sigmoid neural networks.
Preliminary empirical results on MNIST dataset indicate that the amount of noise required to improve over existing uniform bounds can be numerically negligible (i.e., element-wise i.i.d. Gaussian noise with standard deviation 10  240 ).
Introduction
Let F be a class of functions from X to Y , and H a class of functions from Y to Z .
Assuming that
F and H have bounded “capacity”, can we bound the capacity of their composition, i e.,
H  F = { h  f | f 2 F , h 2 H} ?
Here, by capacity we mean learning-theoretic quantities such as VC dimension, fat-shattering dimension, and (uniform) covering numbers associated with these classes (see Vapnik (1999); Anthony et al.
Shalev-Shwartz and Ben-David (2014); Mohri et al. (2018) for an introduction).
Being able to control the capacity of composition of function classes is useful, as it offers a modular approach to design sophisticated classes (and therefore learning algorithms) out of simpler ones.
To be concrete, we want to know if the uniform covering number (as deﬁned in the next section) of H  F can be “effectively” bounded as a function of the uniform covering numbers of
F and H .
The answer to the above questions is true when F is a set of binary valued functions (i.e., Y = { 0 , 1 } in the above).
More generally, the capacity of the composition class (as measured by the uniform covering number) can be bounded as long as |Y| is relatively small (see Proposition 7).
But what if Y is an inﬁnite set, such as the natural case of
Y =
Unfortunately, in this case the capacity of
H  F (as measured by the covering number) can become unbounded (or excessively large) even[-
-] when both
F and H have bounded (or small) capacities; see Propositions 8 and 9.
Given the above observation, we ask whether there is a general and systematic way to control the capacity of the composition of bounded-capacity classes.
More speciﬁcally, we are interested in the case where the domain sets are multi-dimensional real-valued vectors (e.g., X ⇢ R d , Y ⇢ R p ,
Z ⇢ R q ).
The canonical examples of such classes are those associated with neural networks.
A common approach to control the capacity of H  F is assuming that H and F have bounded capacity and
H consists of Lipschitz functions (with respect to appropriate metrics).
Then the capacity of H  F can be bounded as long as H has a small “global cover” (see Remark 13).
This observation has been used to bound the capacity of neural networks in terms of the magnitude of their weights (Bartlett, 1996).
More generally, the capacity of neural networks that admit Lipschitz continuity can be bounded based on their group norms and spectral norms (Neyshabur et al., 2015;
Bartlett et al., 2017; Golowich et al., 2018).
One beneﬁt of this approach is that the composition of
Lipschitz classes is still Lipschitz (although with a larger Lipschitz constant).
While building classes of functions from composition of Lipschitz classes is useful, it does not necessarily work as a general recipe.
In fact, some commonly used classes of functions do not admit a small Lipschitz constant.
Consider the class of single-layer neural networks deﬁned over bounded input domain [  B, B ] d and with the sigmoid activation function.
While the sigmoid activation function itself is Lipschitz, the Lipschitz constant of the network depends on the magnitude of the weights.
Indeed, we empirically observe that this can turn Lipschitzness-based bounds on the covering number of neural networks worse than classic VC-based bounds.
Another limitation of using Lipschitz classes is that they cannot be easily “mixed and matched” with other (bounded-capacity) classes.
For example, suppose F is a class of L -Lipschitz functions (e.g., multi-layer sigmoid neural networks with many weights but small magnitudes).
Also, assume H is a non-Lipschitz class with bounded uniform covering number (e.g., one layer sigmoid neural network with unbounded weights).
Then although both F and H have bounded capacity, H  F is not Lipschitz and its capacity cannot be generally controlled.
We take a different approach for composing classes of functions.
A key observation that we make and utilize is that adding a little bit of noise while “gluing” two classes can help in controlling the capacity of their composition.
In order to prove such results, we deﬁne and study uniform covering numbers of random functions with respect to total variation and Wasserstein metrics.
The bounds for composition then come naturally through the use of data processing inequality for the total variation distance metric.
Contributions and Organization.
• Section 3 provides the necessary notations and includes the observations that composing real-valued functions can be more challenging than binary valued functions (Propositions 7,8, and 9).
• In Section 4, we deﬁne a new notion of covering number for random functions (Deﬁnition 10) with respect to total variation (TV) and Wasserstein distances.
• The bulk of our technical results appear in Section 5.
These include a composition result for random classes with respect to the TV distance (Lemma 18) that is based on the data processing inequality.
We also show how one can translate TV covering numbers to conventional k .
k 2 counterparts (Theorem 17) and vice versa (Corollary 21).
A useful tool is Theorem 20 which exploits kernel density estimation techniques to translate Wasserstein covers to TV covers when we add Gaussian noise to the output of functions.
• Section 6 provides a stronger type of covering number for classes of single-layer noisy neural networks with the sigmoid activation function (Theorem 25).
• In Section 7, we use the tools developed in the previous sections and prove a novel bound on the k .
k 2 covering number of noisy deep neural networks (Theorem 26).
• In Section 8 we deﬁne NVAC, a metric for comparing generalization bounds (Deﬁnition 28) based on the number of samples required to make the bound non-vacuous.
• We offer some preliminary experiments, comparing various generalization bounds in Section 9.
We observe that even a negligible amount of Gaussian noise can improve NVAC over other approaches without affecting the accuracy of the model on train or test data.
Related work
Adding various types of noise have been empirically shown to be beneﬁcial in training neural networks.
In dropout noise (Srivastava et al., 2014) (and its variants such as DropConnect (Wan et al.,
the output of some of the activation functions (or weights) are randomly set to zero.
These approaches are thought to act as a regularizer.
Another example is Denoising AutoEncoders (Vincent et al., 2008) which adds noise to the input of the network while training stacked autoencoders. There has been efforts on studying the theory behind the effects of noise in neural networks. Jim et al. (1996) study the effects of different types of additive and multiplicative noise on convergence speed and generalization of recurrent neural networks (RNN) and suggest that noise can help to speed up the convergence on local minima surfaces.
Lim et al.
(2021) formalize the regularization effects of noise in RNNs and show that noisy RNNs are more stable and robust to input perturbations.
Wang et al.
and Gao and Zhou (2016) analyze the networks with dropout noise and ﬁnd bounds on
Rademacher complexities that are dependent on the product of norms and dropout probability.
It is noteworthy that our techniques and results are quite different, and require a negligible amount of additive noise to work, while existing bounds for dropout improve over conventional bounds only if the amount of noise is substantial.
Studying dropout noise with the tools developed in this paper is a direction for future research.
Studying PAC learning and its sample complexity is by now a mature ﬁeld; see Vapnik (1999);
Shalev-Shwartz and Ben-David (2014); Mohri et al.
In the case of neural networks, standard
Vapnik-Chervonenkis-based complexity bounds have been established (Baum and Haussler, 1988;
Maass, 1994; Goldberg and Jerrum, 1995; Vidyasagar, 1997; Sontag et al., 1998; Koiran and Sontag, 1998; Bartlett et al., 1998; Bartlett and Maass, 2003; Bartlett et al., 2019).
These offer generalization bounds that depend on the number of parameters of the neural network.
There is also another line of work that aims to prove a generalization bound that mainly depends on the norms of the weights and Lipschitz continuity properties of the network rather than the number of parameters (Bartlett, 1996; Anthony et al., 1999; Zhang, 2002; Bartlett, 1996; Neyshabur et al., 2015; Bartlett et al., 2017;
Neyshabur et al., 2018; Golowich et al., 2018; Arora et al., 2018; Nagarajan and Kolter, 2018; Long and Sedghi, 2020).
We provide a more detailed discussion of some of these results in Appendix H.
Finally, we refer the reader to Anthony et al.
for an introductory discussion on this subject.
The above-mentioned bounds are usually vacuous for commonly used data sets and architectures.
Dziugaite and Roy (2017) (and later Zhou et al.
(2019)) show how to achieve a non-vacuous bound using the PAC Bayesian framework.
These approaches as well as compression-based methods (Arora et al., 2018) are, however, examples of “two-step” methods; see Appendix H for more details.
It has been argued that uniform convergence theory may not fully explain the performance of neural networks (Zhang et al., 2021; Nagarajan and Kolter, 2019).
One conjecture is that implicit bias of gradient descent (Gunasekar et al., 2017; Arora et al., 2019; Ji et al., 2020; Chizat and Bach, 2020; Ji and Telgarsky, 2021) can lead to benign overﬁtting (Belkin et al., 2018, 2019; Bartlett et al., 2020); see Bartlett et al.
(2021) for a recent overview.
In a recent line of work, generalization has been studied from the perspective of information theory (Russo and Zou, 2016; Xu and Raginsky, 2017; Russo and Zou, 2019; Steinke and Zakynthinou, 2020), showing that a learning algorithm will generalize if the (conditional) mutual information between the training sample and the learned model is small. Utilizing these results, a number of generic generalization bounds have been proved for Stochastic Gradient Langevin Descent (SGLD) (Raginsky et al., 2017; Haghifam et al., 2020) as well as Stochastic Gradient Descent (SGD) Neu et al.
Somewhat related to our “noise analysis”, these approaches (virtually) add noise to the parameters to control the mutual information.
In contrast, we add noise between modules for composition (e.g., in between layers of a neural network).
Furthermore, we prove uniform (covering number) bounds while these approaches are for generic SGD/SGLD and are mostly agnostic to the structure of the hypothesis class.
Investigating the connections between our analysis and information-theoretic techniques is a direction for future research.
Notations and background
Notation.
X ✓ R d and Y ✓ R p denote two (domain) sets.
For x 2 X , let k x k 1 , k x k 2 , and k x k 1 denote the ` 1 , ` 2 , and ` 1 norm of the vector x , respectively.
We denote the cardinality of a set S by | S | .
The set of natural numbers smaller or equal to m are denoted by [ m ] .
A hypothesis is a Borel function f : R d !
R p , and a hypothesis class F is a set of hypotheses.
We also deﬁne the random counterparts of the above deﬁnitions and use an overline to distinguish them from the non-random versions.
X denotes the set of all random variables deﬁned over X that admit a generalized density function.
3 We sometimes abuse the notation and write x 2 X rather than x 2 X (e.g., x 2 R d is a random variable taking values in R d ).
By y = f ( x ) we denote a random variable that is the result of mapping x using a Borel function f : R d !
R p .
We use f : R d !
R p to indicate that the mapping itself can be random.
We use F to signal that the class can include random hypotheses.
We conﬂate the notation for random hypotheses so that they can be applied to both random and non-random inputs (e.g., f ( x ) and f ( x ) ).
Deﬁnition 1 (Composition of two hypothesis classes) .
We denote by h  f the function h
( f ( x )) (assuming the range of f and the domain of h are compatible).
The composition of two hypothesis classes F and H is deﬁned by H  F = { h  f | h 2 H , f 2 F} .
Composition of classes of random hypotheses is deﬁned similarly by
[Equation]
The following singleton class G  will be used to create noisy functions (e.g., using G   F ).
Deﬁnition 2 (The Gaussian Noise Class) .
The d -dimensional noise class with scale  is denoted by G  ,d = { g  ,d } .
Here, g  ,d : R d !
R d is a random function deﬁned by g  ,d ( x ) =
x + z , where z ⇠ N ( 0 ,  2 I d ) .
When it is clear from the context we drop d and write G  = { g  } .
In the rest of this section, we deﬁne the standard notion of uniform covering numbers for hypothesis classes.
Intuitively, classes with larger uniform covering numbers have more capacity/ﬂexibility, and therefore require more samples to be learned.
Deﬁnition 3 (Covering number) .
Let ( X , ⇢ ) be a metric space.
We say that a set A ⇢ X is ✏ -covered by a set C ✓ A with respect to ⇢ , if for all a 2 A there exists c 2 C such that ⇢ ( a, c )  ✏ .
The cardinality of the smallest set C that ✏ -covers
A is denoted by N ( ✏ , A, ⇢ ) and it is referred to as the ✏ -covering number of A with respect to metric ⇢ .
Deﬁnition 4 (Extended metrics) .
Let ( X , ⇢ ) be a metric space.
Let u
= ( a 1 , . .
. , a m ) , v = ( b 1 , . . . , b m ) 2 X m for m 2 N .
The 1 -extended and ` 2 -extended metrics over X m are deﬁned by ⇢ 1 ,m ( u, v ) =
sup 1  i  m ⇢ ( a i , b i ) and ⇢ ` 2 ,m ( u, v ) =
q 1 m
P mi =1 ( ⇢ ( a i , b i )) 2 , respectively.
We drop m and use ⇢ 1 or ⇢ ` 2 if it is clear from the context.
Remark 5.
The extended metrics are used in Deﬁnition 6 and capture the distance of two hypotheses on an input sample of size m .
A typical example of ⇢ is the Euclidean distance over R p , for which the extended metrics are denoted by k .
k 1 ,mand k .
k ` 2 ,m. Unlike 1 -extended metric, the ` 2 -extended metric is normalized by 1 / p m , and therefore we have ⇢ ` 2 ,m ( u, v )  ⇢ 1 ,m ( u, v ) for all u, v 2 X m .
Deﬁnition 6 (Uniform covering number) .
Let ( Y , ⇢ ) be a metric space and F a hypothesis class of functions from X to Y .
For a set of inputs S = { x 1 , x 2 , . . .
, x m } ✓ X , we deﬁne the restriction of
F to S as F | S
= { ( f ( x 1 ) , f ( x 2 ) , . . .
, f ( x m )) :
f 2 F} ✓ Y m .
The uniform ✏ -covering numbers of hypothesis class F with respect to metrics ⇢ 1 , ⇢ ` 2 are denoted by N U ( ✏ , F , m, ⇢ 1 ) and
N U ( ✏ , F , m, ⇢ ` 2 ) and are the maximum values of N ( ✏ , F | S , ⇢ 1 ,m ) and N ( ✏ , F | S , ⇢ ` 2 ,m ) over all
S ✓ X with | S | = m , respectively.
It is well-known that the Rademacher complexity and therefore the generalization gap of a class can be bounded based on logarithm of the uniform covering number.
For sake of brevity, we defer those results to Appendix F. Therefore, our main object of interest is bounding (logarithm of) the uniform covering number.
The following propositions show that there is a stark difference between classes of functions with ﬁnite range versus continuous valued functions when it comes to bounding the uniform covering number of composite classes; the proofs can be found in Appendix B.
Proposition 7.
Let Y be a ﬁnite domain ( |Y| = k ) and ⇢ ( y, ˆ y )
=
11 { y 6 =
ˆ y } be a metric over Y .
For any class F of functions from X to Y and any class H of functions from Y to R d we have N
U ( ✏ , H  F , m, k . k 1 2 )  N 1 .N U ( ✏ , H , mN 1 , k .
k 1 2 )
where N 1 = N U (0 . 5 , F , m, ⇢ 1 ) .
Proposition 8.
Let F = { f w ( x ) =
wx | w 2 (0 , 1) , x 2 (0 , 1) } be a class of functions and
H =
{ h ( y ) =
1 /y | y 2 (0 , 1) } be a singleton class.
Then, N U ( ✏ , F , m, k .
k ` 2 2 )  d 2 / ✏ 2 e and
N U ( ✏ , H , m, k .
k ` 2 2 )
= 1 , but N U ( ✏ , H  F , m, k .
k ` 2 2 ) is unbounded.
Proposition 9.
For every ✏ > ✏ 0 > 0 , there exist hypothesis classes F and H such that for every m we have N U ( ✏ 0 , H , m, k .
k 1 2 )  m +
and N U ( ✏ 0 , F , m, k .
k 1 2 )
= 1 , yet N U ( [Equation]
Covering random hypotheses
We want to establish the beneﬁts of adding (a little bit of) noise when composing hypothesis classes.
Therefore, we need to analyze classes of random hypotheses.
One way to do this is to replace each hypothesis with its expectation, creating a deterministic version of the hypothesis class.
Unfortunately, this approach misses the whole point of having noisy hypotheses (and their beneﬁts in composition).
Instead, we extend the deﬁnition of uniform covering numbers to classes of random hypotheses F .
The following is basically the random counterpart of Deﬁnition 6.
Deﬁnition 10 (Uniform covering number for classes of random hypotheses) .
Let ( Y , ⇢ ) be a metric space and F a class of random hypotheses from X to Y .
For a set of random variables S = { x 1 , x 2 , . . .
, x m } ✓ X , we deﬁne the restriction of F to S as F | S
= { ( f ( x 1 ) , f ( x 2 ) , . . .
, f ( x m )) :
f 2 F} ✓ Y m .
Let  ✓ X .
The uniform ✏ -covering numbers of F with respect to  and metrics ⇢and ⇢ ` 2 are deﬁned by
[Equation]
Remark 11.
Unlike in Deﬁnition 6 where ⇢ is usually the k .
k 2 metric in the Euclidean space, here in
Deﬁnition 10 ⇢ is deﬁned over random variables.
More speciﬁcally, we will use the Total Variation and Wasserstein metrics as concrete choices for ⇢ .
Remark 12.
The speciﬁc choices that we use for  are
•  =
X
d : the set of all random variables deﬁned over R d that admit a generalized density function. •  = X B,d : the set of all random variables deﬁned over [  B, B ] d that admit a generalized density function.
=  d = {  x | x 2 R d } and  =  B,d = {  x | x 2 [  B, B ] d } , where  x is the random variable associated with Dirac delta measure on x . •  = G  ,d  X B,d = { g  ,d ( x ) | x 2 X B,d } : all members of X B,d after being “smoothed” by adding (convolving with) Gaussian noise.
Remark 13.
Some hypothesis classes that we work with have “global” covers, in the sense that the uniform covering number does not depend on m .
We therefore use the following notation
[Equation]
We now deﬁne Total Variation (TV) and Wasserstein metrics over probability measures rather than random variables, but with a slight abuse of notation we will use them for random variables too.
Deﬁnition 14 (Total Variation Distance) .
Let µ and ⌫ denote two probability measures over X and let ⌦ be the Borel sigma-algebra over X .
The TV distance between µ and ⌫ is deﬁned by
[Equation]
Furthermore, if µ and ⌫ have densities f
and g then
[Equation]
Deﬁnition 15 (Wasserstein Distance) .
Let µ and ⌫ denote two probability measures over X , and ⇧ ( µ, ⌫ ) be the set of all their couplings.
The Wasserstein distance between µ and ⌫ is deﬁned by
[Equation]
The following proposition makes it explicit that the conventional uniform covering number with respect to k .
k 2 (Deﬁnition 6) can be regarded as a special case of Deﬁnition 10.
Proposition 16.
Let F be a class of (deterministic) hypotheses from R d to R p .
Then
N U ( ✏ , F , m, k . k 1 2 )
=
N U ( ✏ , F , d 1 W , m,  d ) and N U ( ✏ , F , m, k .
k ` 2 2 )
=
N U ( ✏ , F , d ` 2 W , m,  d ) .
The proposition is the direct consequence of the Deﬁnitions 6 and 10 once we note that the Wasserstein distance between Dirac random variables is just their ` 2 distance, i.e., d W (  x ,  y )
=
k x  y k 2 .
Bounding the uniform covering number
This section provides tools that can be used in a general recipe for bounding the uniform covering number.
The ultimate goal is to bound the (conventional) k .
k 1 2 and k .
k ` 2 2 uniform covering numbers for (noisy) compositions of hypothesis classes.
In order to achieve this, we will show how one can turn TV covers into k .
k 2 covers (Theorem 17) and vice versa (Corollary 21).
But what is the point of going back and forth between k .
k 2 and TV covers?
Basically, the data processing inequality ensures an effective composition (Lemma 18) for TV covers.
Our analysis goes through a number of steps, connecting covering numbers with respect to k .
k 2 , Wasserstein, and TV distances.
The missing proofs of this section can be found in Appendix C.
The following theorem considers the deterministic class H associated with expectations of random hypotheses from F , and shows that bounding the uniform covering number of F with respect to TV distance is enough for bounding the uniform covering number of H with respect to k .
k 2 distance.
Theorem 17 (From a TV cover to a k .
k 2 cover) .
Consider any class F of random hypotheses f : R d !
[  B, B ] p with bounded output.
Deﬁne the (nonrandom) hypothesis class H
=
{ h : R d !
[  B, B ] p | h ( x ) =
E f ⇥ f ( x ) ⇤ , f 2 F} .
Then for every ✏ > 0 , m 2 N these two inequalities hold:
[Equation]
But what is the point of working with the TV distance?
An important ingredient of our analysis is the use of data processing inequality which holds for the TV distance (see Lemma 31).
The following lemma uses this fact, and shows how one can compose classes with bounded TV covers.
Lemma 18 (Composing classes with bounded TV covers) .
Let F be a class of random hypotheses from R d to R p , and H be a class of random hypotheses from R p to R q .
For every ✏ , ✏ 0 > 0 , and every m 2 N these three inequalities hold:
[Equation]
where N=
[Equation]  and N=
N U ⇣
[Equation]
⌘ .
Remark 19.
In Lemma 18, for H , we required the stronger notion of cover with respect to X d (i.e., the input to the hypotheses can be any random variable with a density function), whereas for F a cover with respect to  d sufﬁced in some cases.
As we will see below, ﬁnding a cover with respect to  d is easier since one can reuse conventional k .
k 2 covers.
However, ﬁnding covers with respect to
X d is more challenging.
In the next section we show how to do this for a class of neural networks.
The next step is bounding the uniform covering number with respect to the TV distance (TV covering number for short).
It will be useful to be able to bound TV covering number with Wasserstein covering number.
However, this is generally impossible since closeness in Wasserstein distance does not imply closeness in TV distance.
Yet, the following theorem establishes that one can bound
the
TV covering number as long as some Gaussian noise is added to the output of the hypotheses.
Theorem 20 (From a Wasserstein cover to a TV cover) .
Let F be a class of random hypotheses from
R d to R p , and G  ,p be a Gaussian noise class.
Then for every ✏ > 0 and m 2 N we have
[Equation]
Intuitively, the Gaussian noise smooths out densities of random variables that are associated with applying transformation in F to random variables in X d or  d .
As a result, the proof of Theoremhas a step on relating the Wasserstein distance between two smoothed (by adding random Gaussian noise) densities to their total variation distance (see Lemma 35 in Appendix C).
Finally, we can use Proposition 16 to relate the Wasserstein covering number with the k .
k 2 covering number.
The following corollary is the result of Proposition 16 and Theorem 20 that is stated for both d ` 2 TV and d 1 TV extended metrics.
Corollary 21 (From a k .
k 2 cover to a TV cover) .
Let F be a class of hypotheses f : R d !
R p and
G  ,p be a Gaussian noise class.
Then for every ✏ > 0 and m 2 N we have
[Equation]
The following theorem shows that we can get a stronger notion of TV cover with respect to X B,d from a k .
k 2 global cover, given that some Gaussian noise is added to the output of hypotheses.
Theorem 22 (From a global k .
k 2 cover to a global TV cover) .
Let F be a class of hypotheses f : R d !
R p and G  ,p be a Gaussian noise class.
Then for every ✏ > 0 and m 2 N we have
[Equation]
The proof involves ﬁnding a Wasserstein covering number and using Theorem 20 to obtain TV covering number.
Uniform TV covers for single-layer neural networks
In this section, we study the uniform covering number of single-layer neural networks with respect to the total variation distance.
This will set the stage for the next section, where we want to use the tools from Section 5 to bound covering numbers of deeper networks.
We start with the following deﬁnition for the class of single-layer neural networks.
Deﬁnition
(Single-Layer Sigmoid Neural Networks) .
Let  : R p !
[0 , 1] p be the elementwise sigmoid activation function deﬁned by  (( x (1) , . . .
, x ( p ) ))
=
(  ( x (1) ) , . . .
,  ( x ( p ) )) , where  ( x ) = 1 {+
 ++} e  x is the sigmoid function.
The class of single-layer neural networks with d inputs and p outputs is deﬁned by NET[ d, p ] =
{ f W : [Equation]
[Equation] [Equation] p
[Equation] [Equation]
Remark 24.
We choose sigmoid function for simplicity, but our analysis for ﬁnding uniform covering numbers of neural networks (Theorem 25) is not speciﬁc to the sigmoid activation function.
We present a stronger version of Theorem 25 in Appendix D which works for any activation function that is Lipschitz, monotone, and bounded.
As mentioned in Remark 19, Lemma 18 requires stronger notion of covering numbers with respect to X d and TV distance.
In fact, the size of this kind of cover is inﬁnite for deterministic neural networks deﬁned above.
In contrast, Theorem 25 shows that one can bound this covering number as long as some Gaussian noise is added to the input and output of the network.
The proof is quite technical, starting with estimating the smoothed input distribution ( g  ( x ) ) with mixtures of Gaussians using kernel density estimation (see Lemma 58 in Appendix J).
Then a cover for mixtures of Gaussians with respect to Wasserstein distance is found.
Finally, Theorem 20 helps to ﬁnd the cover with respect to total variation distance.
For a complete proof of theorem see Appendix D.
Theorem 25 (A global total variation cover for noisy neural networks with unbounded weights) .
For every p, d 2 N , ✏ > 0 ,  < 5 d/ ✏ we have
[Equation]
Note that the dependence of the bound on 1 /  is polynomial.
he assumption  ⌧ 5 d/ ✏ holds for any reasonable application (we will use  ⌧ 1 in the experiments).
In contrast to the analyses that exploit Lipschitz continuity, the above theorem does not require any assumptions on the norms of weights.
Theorem 25 is a key tool in analyzing the uniform covering number of deeper networks.
In the following, we discuss how one can use Theorem 25 and techniques provided in Section 5 to obtain bounds on covering number for deeper networks.
For a T -layer neural network, it is useful to separate the ﬁrst layer from the rest of the network.
The following theorem offers a bound on the uniform covering number of (the expectation of) a noisy network based on the usual k .
k ` 2 2 covering number of the ﬁrst layer and the TV covering number of the subsequent layers.
Theorem 26.
Let NET[ d, p 1 ] , NET[ p 1 , p 2 ] , . .
, NET[ p T  1 , p T ] be T classes of neural networks.
Denote the T -layer noisy network by
[Equation]
and let H = { h : R d ! [0 , 1] p T | h ( x [Equation]
= E f [Equation] f 2 F} .
Denote the uniform covering numbers of compositions of neural network classes with the Gaussian noise class (with respect to d 1 TV ) as
[Equation]
and the uniform covering number of G   NET[ d, p 1 ] with respect to k .
k ` 2 2 as
[Equation]
Then we have
[Equation]
The k .
k ` 2 2 covering number of the ﬁrst layer (i.e., N 1 in above) can be bounded using standard approaches in the literature.
For instance, in Appendix H we will use the bound of Lemma 14.7 in
Anthony et al.
Other N i ’s can be bounded using Theorem 25 (See Corollary 28 in Pour and
Ashtiani (2022) and Theorem 50 in Appendix H).
The above bound does not depend on the norm of weights and therefore we can use it for networks with large weights.
The proof of Theorem 26 involves applying Corollary 21 to turn the k .
k 2 cover of ﬁrst layer into a TV cover.
We then ﬁnd a TV cover for rest of the network by applying Lemma 18 recursively to compose all the other layers.
We will compose the ﬁrst layer with the rest of the network and bound the covering number by another application of Lemma 18.
Finally, we turn the TV covering number (of the entire network) back into k .
k ` 2 2 covering number using Theorem 17.
The complete proof can be found in Appendix E.
One can generalize the above analysis in the following way: instead of separating the ﬁrst layer, one can basically “break” the network from any layer, use existing k .
k 2 covering number bounds for the ﬁrst few layers, and Theorem 25 for the rest.
See Lemma 37 in Appendix E for details.
NVAC: a metric for comparing generalization bounds
We want to provide tools to compare different approaches in ﬁnding covering numbers and their suggested generalization bounds.
First, we deﬁne the notion of a generalization bound for classiﬁcation.
Let Y =
[ k ] and F be a class of functions from X to R k .
Let A be an algorithm that receives a labeled sample S =
(( x 1 , y 1 ) , . . .
, ( x m , y m )) 2 ( X ⇥ Y ) m
and outputs a function ˆ h 2 F .
Note that the output of this function is a real vector so it can capture margin-based classiﬁers too.
Let l 0  1 : R k ⇥ [ k ] !
{ 0 , 1 } be the “thresholded” 0-1 loss function deﬁned by l 0  1 ( u, y )
= 11 { argmax i u ( i ) 6 = y } where u ( i ) is the i -th dimension of u .
Deﬁnition 27 (Generalization Bound for Classiﬁcation) .
A (valid) generalization bound for A with respect to l 0  1 and another (surrogate) loss function l is a function GB : F ⇥ ( X ⇥ Y ) m !
R such that for every distribution
D over X ⇥ Y , if S ⇠ D m , then with probability at least 0.99 (over the randomness of S ) we have [Equation] i
[Equation]
For example, GB (ˆ h, S )
= 2 is a useless but valid generalization bound.
Various generalization bounds that have been proposed in the literature are examples of a GB .
Note that GB can depend both on S (for instance on | S | ) and on ˆ h (for example, on the norm of the weights of network).
It is not straightforward to empirically compare generalization bounds since they are often vacuous for commonly used applications.
Jiang et al.
(2019) address this by looking at other metrics, such as the correlation of each bound with the actual generalization gap.
While these metrics are informative, it is also useful to know how far off each bound is from producing a “non-vacuous” bound (Dziugaite and Roy, 2017).
Therefore, we will take a more direct approach and propose the following metric.
Deﬁnition 28 (NVAC) .
Let ˆ h be a hypothesis, S 2 ( X ⇥ Y ) m a sample, and GB a generalization bound for algorithm A .
Let S n denote a sample of size mn which includes n copies of S .
Let n ⇤ be the smallest integer such that the following holds:
[Equation]
We deﬁne NVAC to be | S n ⇤ | =
mn ⇤ .
Informally speaking, NVAC is an upper bound on the minimum number of samples required to obtain a non-vacuous generalization bound.
Approaches that get tighter upper bounds on covering number will generally result in smaller NVACs.
In Appendix G, we will show how one can calculate NVAC using the uniform covering number bounds.
Experiments
In this section, we empirically compare different approaches in bounding the covering number using the NVAC metric.
We compare the following approaches in bounding covering number:
Theorem 26,
Norm-based (Theorem 14.17 in Anthony et al.
, Lipschitzness-based (Theorem 14.5 in
Anthony et al.
, Pseudo-dim-based (Theorem 14.2 in Anthony et al. (1999)), and Spectral (Bartlett et al.
More details about these bounds can be found in Appendix H.
We train fully connected neural networks on MNIST dataset.
We use a network with an input layer, an output layer, and three hidden layers each containing 250 hidden neurons as the baseline architecture.
See Appendix I for the details of the learning settings.
The left two graphs in Figure 1 depict NVACs as functions of the depth and width of the network.
It can be observed that our approach achieves the smallest NVAC.
The Norm-based bound is the worst and is removed from the graph (see Appendix I).
Overall, bounds that are based on the norm of the weights (even the spectral norm) perform poorly compared to those that are based on the parameter count.
This is an interesting observation since we have millions of parameters ( ⇡ 3 ⇥ 10 9 ) in some of the wide networks and one would assume approaches based on norm of weights should be able to explain generalization behaviour better.
Aside from the fact that our bound does not have any dependence on the norms of weights, there are several reasons why it performs better.
First, the NVAC in Spectral and Norm-based approaches have an extra polynomial dependence on 1 / ✏ , compared to all other approaches.
Moreover, these bounds depend on product of norms and group norms which can get quite large.
Finally, our method works naturally for multi-output layers, while the Pseudo-dim-based, Lipschitzness-based, and Norm-based approaches work for real-valued output (and therefore one needs to bound the cover for each output separately).
The covering number bound of Theorem 26 has a polynomial dependence on 1 /  .
Therefore, NVAC has a mild logarithmic dependence on 1 /  (see Appendix G for details).
The third graph in Figurecorroborates that even a negligible amount of noise (  ⇡ 10  240 ) is sufﬁcient to get tighter bounds on NVAC compared to other approaches.
Finally, the right graph in Figure 1 shows that even with a considerable amount of noise (e.g,  = 0 . 2 ), the train and test accuracy of the model remain almost unchanged.
This is perhaps expected, as the dynamics of training neural networks with gradient descent is already noisy even without adding Gaussian noise.
Overall, our preliminary experiment shows that small amount of noise does not affect the performance, yet it enables us to prove tighter generalization bounds.
Limitations and Future Work.
Our analysis is based on the assumption that the activation function is bounded.
Therefore, extending the results to ReLU neural networks is not immediate, and is left for [Figure] future work.
Also, our empirical analysis is preliminary and is mostly used as a sanity check.
Further empirical evaluations can help to better understand the role of noise in training neural networks.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] The main claims of the paper in abstract and introduction are stated as theorems, lemmas and propositions, which are proved in appendices due to space constraints.
(b) Did you describe the limitations of your work?
[Yes] We discuss the limitations of our experiments and possible future work in “Limitations and Future Work”.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
This is mainly a theoretical paper and is unlikely to have any negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] The theorems and lemmas are proved in appendices due to space constraints.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] The codes will be provided as a supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Appendix I. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix I 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] We experiment on the publicly available MNIST dataset, which is mentioned in the main paper.
(b) Did you mention the license of the assets?
[Yes] We provide our code as a supplemental material which does not require a license.
We use MNIST dataset and discuss it in Appendix I. (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] We provide our code in the supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] We use MNIST dataset which is a publicly available.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] We use MNIST dataset which is a collection of handwritten digits and does not contain personal information or offensive content.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]