O N T IGHTER G ENERALIZATION B OUNDS FOR D EEP N EURAL N ETWORKS : CNN S , R ES N ETS , AND B EYOND
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose a generalization error bound for a general family of deep neural networks based on the depth and width of the networks, as well as the spectral norm of weight matrices.
Through introducing a novel characterization of the Lipschitz properties of neural network family, we achieve a tighter generalization error bound.
We further obtain a result that is free of linear dependence on norms for bounded losses.
Besides the general deep neural networks, our results can be applied to derive new bounds for several popular architectures, including convolutional neural networks (CNNs), residual networks (ResNets), and hyperspherical networks (SphereNets).
When achieving same generalization errors with previous arts, our bounds allow for the choice of much larger parameter spaces of weight matrices, inducing potentially stronger expressive ability for neural networks.
I NTRODUCTION
We aim to provide a theoretical justiﬁcation for the enormous success of deep neural networks (DNNs) in real world applications (He et al., 2016; Collobert et al., 2011; Goodfellow et al., 2016).
In particular, our paper focuses on the generalization performance of a general class of DNNs.
The generalization bound is a powerful tool to characterize the predictive performance of a class of learning models for unseen data.
Early studies investigate the generalization ability of shallow neural networks with no more than one hidden layer (Bartlett, 1998; Anthony & Bartlett, 2009).
More recently, studies on the generalization bounds of deep neural networks have received increasing attention (Dinh et al., 2017; Bartlett et al., 2017; Golowich et al., 2017; Neyshabur et al., 2015; 2017).
There are two major questions of our interest in these analysis of the generalization bounds:
(Q1)
Can we establish tighter generalization error bounds for deep neural networks in terms of the network dimensions and structure of the weight matrices?
(Q2)
Can we develop generalization bounds for neural networks
with special architectures?
For (Q1), (Neyshabur et al., 2015; Bartlett et al., 2017; Neyshabur et al., 2017; Golowich et al., 2017) have established results that characterize the generalization bounds in terms of the depth
D and width p of networks and norms of rank- r weight matrices.
For example, Neyshabur et al.
(2015) provide an exponential bound on D based on the Frobenius norm (cid:107) W d (cid:107) F , where W d is the weight matrix of d -th layer; Bartlett et al.
; Neyshabur et al.
(2017) provide a polynomial bound on p and D based on (cid:107) W d (cid:107) 2 (spectral norm) and (cid:107) W d (cid:107) 2 , 1 (sum of the Euclidean norms for all rows of W d ).
Golowich et al.
(2017) provide a nearly size independent bound based on (cid:107) W d (cid:107) F .
Nevertheless, the generalization bound depends on other than the spectral norms of the weight matrices may be too loose.
In speciﬁc, (cid:107) W d (cid:107) F ( (cid:107) W d (cid:107) 2 , 1 ) is in general √ r ( r ) times larger than (cid:107) W d (cid:107) 2 .
Given m training data points and suppose (cid:107) W d (cid:107) 2 = 1 for ease of discussion, Bartlett et al.
(2017) and Neyshabur et al.
(2017) demonstrate generalization error bounds as (cid:101)
O ( (cid:112) D 3 pr/m ) , and Golowich et al.
(2017) achieve a bound (cid:101) O
( r D/ 2 min( m − 1 / 4 , (cid:112) D/m )) , where (cid:101) O ( · ) represents the rate by ignoring logarithmic factors.
In comparison, we show a tighter generalization error bound as (cid:101) O
( (cid:112) Dpr/m ) , which is signiﬁcantly smaller than existing results and achieved based on a new
Lipschitz analysis for DNNs in terms of both the input and weight matrices.
We notice that some recent results characterize the generalization bound in more structured ways, e.g., by considering speciﬁc error-resilience parameters (Arora et al., 2018), which can achieve empirically improved
[Table] [Table]
Our results
Theorem
[Equation] [Equation]
generalization bounds than existing ones based on the norms of weight matrices.
However, it is not clear how the weight matrices explicitly control these parameters, which makes the results less interpretable.
Thus, we do not compare with these types of results.
We summarize the comparison between existing norm based generalization bounds with our results in Table 1, as well as the results when (cid:107) W d (cid:107)
= 1 for more explicit comparison in terms of the network sizes (i.e, depth and width).
For (Q2), we consider several widely used architectures to demonstrate, including convolutional neural networks (CNNs) (Krizhevsky et al., 2012), residual networks (ResNets)
(He et al., 2016), and hyperspherical networks (SphereNets) (Liu et al., 2017b).
By taking their structures of weight matrices into consideration, we provide tight characterization of their resulting capacities.
In particular, we consider orthogonal ﬁlters and normalized weight matrices, which show good performance in both optimization and generalization (Mishkin & Matas, 2015; Xie et al., 2017).
This is closely related with normalization frameworks, e.g., batch normalization (Ioffe & Szegedy, 2015) and layer normalization (Ba et al., 2016), which have achieved great empirical performance (Liu et al., 2017a;
He et al., 2016).
Take CNNs as an example.
By incorporating the orthogonal structure of convolutional ﬁlters, we achieve (cid:101) O (cid:0)(cid:0) ks (cid:1) D 2 √ Dk 2 / √ m (cid:1) , while Bartlett et al.
; Neyshabur et al.
(2017) achieve (cid:101) O (cid:0)(cid:0) ks (cid:1)
D − 1 2 (cid:112) D 3 p 2 / √ m (cid:1) and Golowich et al.
(2017) achieve (cid:101) O (cid:16) p D 2 min (cid:110) 1 4 √ m , (cid:113)
Dm (cid:111)(cid:17) (rank ( W d )
=
p in CNNs), where k is the ﬁlter size that satisﬁes k (cid:28) p and s is stride size that is usually of the same order with k ; see Section 4.1 for details.
Here we achieve stronger results in terms of both depth D and width p for CNNs, where our bound only depend on k rather than p .
Some recent result achieved results that is free of the linear dependence on the weight matrix norms by considering networks with bounded outputs (Zhou & Feng, 2018).
We can achieve similar results using bounded loss functions as discussed in Section 3.2, but do not restrict ourselves to this scenario in general.
Analogous improvement is also attained for ResNets and SphereNets.
In addition, we consider some widely used operations for width expansion and reduction, e.g., padding and pooling, and show that they do not increase the generalization bound.
Further numerical evaluation is provided for quantitative comparison in Section 4.5.
Our tighter bounds result in potentially stronger expressive power, hence higher training/testing accuracy for the DNNs.
In particular, when achieving the same order of generalization errors, we allow the choice of a larger parameter space with deeper/wider networks and larger matrix spectral norms.
We further show numerically that a larger parameter space can lead to better empirical performance.
Quantitative analysis for the expressive power of DNNs is of great interest on its own, which includes (but not limited to) studying how well DNNs can approximate general class of functions and distributions (Cybenko, 1989;
Hornik et al., 1989; Funahashi, 1989; Barron, 1993; 1994; Lee et al., 2017; Petersen & Voigtlaender, 2017; Hanin & Sellke, 2017), and quantifying the computation hardness of learning neural networks; see e.g., Shamir (2016); Eldan & Shamir (2016);
Song et al.
We defer our investigation toward this to future efforts.
Notation.
Given an integer n > 0 , we deﬁne [ n ] = { 1 , . . . , n } .
Given a vector x ∈ R p , we denote x i as the i -th entry, and x i : j as a sub-vector indexed from i -th to j -th entries of x .
Given a matrix
A ∈ R n × m , we denote A ij as the entry corresponding to i -th row and j -th column, A i ∗ ( A ∗ i ) as the i -th row (column),
A
I 1 I 2 as a submatrix of A indexed by the set of rows I 1 ⊆
[ n ] and columns
I 2 ⊆
[ m ] , (cid:107) A (cid:107) as a generic norm, (cid:107) A (cid:107) 2 as the spectral norm, (cid:107) A (cid:107) F as the Frobenius norm, and (cid:107) A (cid:107) 2 , 1 = (cid:80) ni
=1 (cid:107) A i ∗ (cid:107) 2 .
Given two real values a, b ∈ R + , we write a (cid:46) ( (cid:38) )
b if a ≤ ( ≥ ) cb for some generic constant c > 0 .
We use the standard notations O ( · ) , Θ ( · ) , and Ω ( · ) to denote limiting behaviors ignoring constants, and (cid:101) O ( · ) , (cid:101) Θ ( · ) and (cid:101) Ω ( · ) to further ignore logarithmic factors.
P RELIMINARIES
We provide a brief description of the DNNs.
Given an input x ∈ R p 0 , the output of a D -layer network is deﬁned as f D f ( W D , x ) =
f W D ( · · · f W 1 ( x ))
∈
R p D , where f W d ( y ) =
σ d ( W d · y ) : R p d − 1 → R p d with an entry-wise activation function σ d ( · ) .
We specify σ d as the rectiﬁed linear unit (ReLU) activation (Nair & Hinton, 2010) for ease of discussion.
The extension to more general activations, e.g., Lipschitz continuous functions, is straightforward.
Given any two layers i, j ∈ [ D ] with i < j and input x , we denote J i : j as the Jacobian from layer i to layer j , i.e., f W j ( · · · f W i ( x )) =
J xi : j · x .
Note that J xi : j depends on x .
For convenience, we denote f W i ( x ) =
J xi,i · x when i =
j and denote J xi :
j = I when i > j .
Then we denote DNNs with bounded weight W D
= { W d ∈ R p d × p d − 1 } Dd =1 , Jacobian, and rank as
[Equation] [Equation]
, rank
[Equation]
d ) [Equation] r d (cid:9) , (1) where x ∈ R p 0 is an input, and { B d } , (cid:8) B Jac i : j (cid:9) are real positive constants.
For convenience, we also denote B Jac \ d, 2 = max d ∈ [ D ] B Jac 1:( d − 1) , 2 B Jac ( d +1): D, 2 .
We will specify the norm (cid:107)·(cid:107) and the corresponding upper bounds B d , e.g., (cid:107)·(cid:107) 2 and B d, 2 , or (cid:107)·(cid:107) F and B d, F , when necessary.
Given a loss function g ( · , · ) , we denote a class of loss functions measuring the discrepancy between a DNN’s output f ( W D , x ) and the corresponding observation y ∈ Y m for a given input x ∈ X m as
[Equation]
where the sets of bounded inputs X m and the corresponding observations Y m are
X m = { x i ∈
R p 0 | (cid:107) x i (cid:107) 2 ≤ R for all i ∈ [ m ] } ⊂ X and Y m = { y i ∈
[ p D ] for all i ∈ [ m ] } ⊂ Y .
Then the empirical Rademacher complexity (ERC) of G (cid:0) F D, (cid:107)·(cid:107) (cid:1) given X m and Y m is
[Equation]
where {± 1 } m ∈ R m is the set of vectors only containing entries +1 and − 1 , and (cid:15) ∈
R m is a vector with Rademacher entries, i.e., (cid:15) i
=
+1 or − 1 with equal probabilities.
Take the classiﬁcation as an example.
For multi-class classiﬁcation, suppose p D = N class is the number of classes.
Consider g with bounded outputs, namely the ramp risk .
Speciﬁcally, for an input x belonging to class y ∈ [ N class ] , we denote ν =
( f ( W D , x ))
y − max i (cid:54) =
y ( f ( W D , x )) i .
For a given real value γ > 0 , the class of ramp risk functions with parameter γ is G γ (cid:0) F D, (cid:107)·(cid:107) (cid:1) =
(cid:8) g γ ( f ( W D , x ) , y )
| f D ∈ F D, (cid:107)·(cid:107) (cid:9) , where g γ isγ -Lipschitz continuous, deﬁned in (3).
For conve-
[Equation]
nience, we denote g γ ( f ( W D , x ) , y ) as g γ ( f ( W D , x )) (or g γ ) in the rest of the paper.
Then the generalization error bound for PAC learning (Bartlett et al., 2017) (Lemma 3.1) states the following.
Given any real δ ∈ (0 , 1) and g γ , with probability at least 1 − δ , we have that for any f ( · , · ) ∈ F D, (cid:107)·(cid:107) , the generalization error is upper bounded with respect to (w.r.t.)
the ERC satisﬁes E
[Equation]
The right hand side of (4) is viewed as a guaranteed error bound for the gap between the testing and the empirical training performance.
Since the ERC is generally the dominating term in (4), a small
R m is desired for DNNs given the loss function g γ .
Analogous results hold for regression tasks; see e.g., Kearns & Vazirani (1994); Mohri et al. (2012) for details.
G ENERALIZATION E RROR B OUND FOR DNN S
Our major focus here is to provide tighter bounds w.r.t.
the network sizes based on the spectral norm of weight matrices.
A reﬁned result, which is free of the linear dependence on the spectral norm, is provided when the loss function is additionally bounded.
3. A T IGHTER ERC B OUND FOR DNN S
We ﬁrst provide the ERC bound for the class of DNNs deﬁned in (1) and Lipschitz loss functions in the following theorem.
The proof is provided in Appendix A.
Theorem 1.
Let g γ be a 1 γ -Lipschitz loss function and F D, (cid:107)·(cid:107) 2 be the class of DNNs deﬁned in (1), p d = p , r d = r for all d ∈ [ D ] , and C 1 =
[Equation]
Then the ERC satisﬁes
[Equation]
For convenience, we treat R/γ as a constant here.
Note that C 1 depends on the norm of Jacobian, which is signiﬁcantly smaller than the product of matrix norms that is exponential on D .
For example, when we constrain the network to be obtained from stochastic gradient descent using random initialized weights, then B Jac \ d, 2 (cid:28) (cid:81) d B d, 2 .
Empirical distributions of B Jac \ d,and (cid:81) d B d, 2 are provided in Appendix F.2, where B Jac \ d, 2 ’s are constants that are orders of magnitude smaller than (cid:81) d B d, 2 .
Thus, logC 1 can be considered as a constant that is (almost) independent of D .
In other words, we achieve (cid:101) O
(Π Dd =1 B d, 2 · (cid:112) Dpr/m ) in Theorem 1, which is tighter than existing results based on the network sizes and norms of weight matrices, as shown in Table 1.
In particular, Neyshabur et al.
(2015) show an exponential dependence on D , i.e.,
O (2 D Π Dd =1 B d, F / √ m ) , which can be signiﬁcantly larger than ours.
Bartlett et al.
; Neyshabur et al.
(2017) demonstrate polynomial dependence on sizes and the spectral norm of weights, i.e., (cid:101)
O
(Π Dd =1 B d, 2 · (cid:112) D 3 pr/m ) .
Our result in (5) is tighter by an order of D , which is signiﬁcant in practice.
More recently, Golowich et al.
(2017) demonstrate a bound w.r.t the Frobenius norm as (cid:101) O (cid:0) Π Dd =1 B d, F · min (cid:8)(cid:113)
Dm , m − 14 · log 34 ( m ) (cid:112) log ( C 2 ) (cid:9)(cid:1) , where C 2 =
R · Π Dd =1 B d, F sup x ∈X (cid:107) f
( W D ,x ) (cid:107) 2 .
This has a tighter dependence on network sizes.
Nevertheless, (cid:107) W d (cid:107) F is generally √ r times larger than (cid:107) W d (cid:107) 2 , which results in an exponential dependence p D/ 2 compared with the bound based on the spectral norm.
Moreover, log( C 2 ) is linear on D except that the stable ranks (cid:107) W d (cid:107) F / (cid:107) W d (cid:107) 2 across all layers are close to 1 (rather than almost independent on D as in (5) without low-rank constraints).
In addition, it has m − 14 dependence rather than m − 12 except when D = O ( √ m ) .
Note that our bound is based on a novel characterization of Lipschitz properties of DNNs, which may be of independent interest from the learning theory point of view.
We refer to Appendix A for details.
We also remark that when achieving the same order of generalization errors, our result allows the choices of larger dimensions ( D, p ) and spectral norms of weight matrices, which lead to stronger expressive power for DNNs.
For example, when achieving the same bound with (cid:107) W d (cid:107) 2 = 1 in spectral norm based results (e.g. in ours) and (cid:107) W d (cid:107) F = 1 in Frobenius norm based results (e.g., in
Golowich et al.
, they only have (cid:107) W d (cid:107)
= O (1 / √ r ) in Frobenius norm based results.
The later results in a much smaller space for eligible weight matrices as r is of order p in general (i.e., r = δp for some constant δ ∈ (0 , 1) ), which may lead to weaker expressive ability of DNNs.
We also demonstrate numerically that when norms of weight matrices are constrained to be very small, both training and testing performance degrade signiﬁcantly; see further discussion in Section 4.5.
A quantitative analysis for the tradeoff between the expressive ability and the generalization for DNNs is deferred to a future effort.
3. A S PECTRAL N ORM F REE ERC B OUND
When, in addition, the loss function is bounded, we have the ERC bound free of the linear dependence on the spectral norm, as in the following corollary.
The proof is provided in Appendix B.
Corollary 1.
In addition to the conditions in Theorem 1, suppose we further let g γ be bounded, i.e., | g γ | ≤ b .
Then the ERC satisﬁes
[Equation]
The boundedness of G γ holds for certain loss functions, e.g., the ramp risk deﬁned in (3).
When b is constant (e.g., b = 1 for the ramp risk) and R (cid:81)
Dd =1 B d, 2 > γ , we have that the ERC reduces to (cid:101) O
( (cid:112) Dpr/m ) .
This is close to the VC dimension of DNNs, which can be signiﬁcantly tighter than existing norm based bounds in general.
Similar norm free results hold for the architectures discussed in Section 4 using argument for Corollary 1, which we skip due to space limit.
Moreover, our bound (6) is also tighter than recent results that are free of linear dependence on (cid:81)
Dd =1 B d, 2 (Zhou & Feng, 2018; Arora et al., 2018).
Speciﬁcally, Zhou & Feng (2018) show that the generalization bound for CNNs is (cid:101) O (cid:0) D (cid:112) pr 2 /m (cid:1) , which results in a bound larger than (6) by O ( √ Dr ) .
Arora et al.
(2018) derive a bound for a compressed network in terms of some error-resilience parameters, which is (cid:101)
O ( (cid:112) D 3 p 2 /m )
since the cushion parameter therein is of the order µ = O (1 / √ p ) .
Further discussion is provided in Appendix F.1.
E XPLORING N ETWORK S TRUCTURES
The generic result in Section 3 does not highlight explicitly the potential impacts for speciﬁc structures of the networks.
In this section, we consider several popular architectures of DNNs, including convolutional neural networks (CNNs) (Krizhevsky et al., 2012), residual networks (ResNets)
(He et al., 2016), and hyperspherical networks (SphereNets) (Liu et al., 2017b), and provide sharp characterization of the corresponding generalization bounds.
In particular, we consider orthogonal ﬁlters and normalized weight matrices, which have shown good performance in both optimization and generalization (Mishkin & Matas, 2015; Huang et al., 2017).
Such constraints can be enforced using regularizations on ﬁlters and weight matrices, which is very efﬁcient to implement in practice.
This is also closely related with normalization approaches, e.g., batch normalization (Ioffe & Szegedy, 2015) and layer normalization (Ba et al., 2016), which have achieved tremendous empirical success.
4. CNN S WITH O RTHOGONAL F ILTERS
CNNs are one of the most powerful architectures in deep learning, especially in tasks related to images and videos (Goodfellow et al., 2016).
We consider a tight characterization of the generalization bound for CNNs by generating the weight matrices using unit norm orthogonal ﬁlters, which has shown great empirical performance (Huang et al., 2017; Xie et al., 2017).
Speciﬁcally, we generate the weight matrices using a circulant approach, as follows.
For the convolutional operation at the dth layer, we have n d channels of convolution ﬁlters, each of which is generated from a k ddimensional feature using a stride side s d .
Suppose that s d divides both k d and p d − 1 , i.e., k d − 1 s d and p d − 1 s d are integers, then we have p d = n d · p d − 1 s d .
This is equivalent to ﬁxing the weight matrix at the d -th layer to be generated as in (7), where for all j ∈ [ n d ] , each W ( j ) d
[Equation]
is formed in a circulant-like way using a vector w ( d,j ) ∈ R k d with unit norms for all j as in (8).
[Equation]

[Equation]
When the stride size s d = 1 , W ( j ) d corresponds to a standard circulant matrix (Davis, 2012).
The following lemma establishes that when (cid:8) w ( d,j ) (cid:9) n d j =1 are orthogonal vectors with unit Euclidean norms, the generalization bound only depend on s d and k d that are independent of the width p d .
The proof is provided in Appendix C.
Corollary 2.
Let g γ be a 1 γ -Lipschitz and bounded loss function, i.e., | g γ | ≤ b , and F D, (cid:107)·(cid:107) 2 be the class of CNNs deﬁned in (1).
Suppose the weight matrices in CNNs are formed as in (7) and (8) with s d = s , k d = k , and s divides both k and p d for all d ∈ [ D ] , where (cid:8) w ( d,j ) (cid:9) n d j =1 satisﬁes w ( j ) (cid:62) w ( i ) = 0 for all i, j ∈ [ n d ] and i (cid:54)
=
j with (cid:13)(cid:13) w [Equation] d,j ) (cid:13)(cid:13) 2 = 1 for all j ≤ n d .
Denote C 1 =
[Equation]
f
[Equation]
[Equation]
x )) .
Then the ERC for CNNs satisﬁes
[Equation]
Since n d ≤ k in our setting, the ERC for CNNs is proportional to √ Dk 2 instead of √ Dpr .
For the orthogonal ﬁltered considered in Corollary 2, we have (cid:107) W d (cid:107) F = √ p d and (cid:107) W d (cid:107) 2 , 1 = p d , which lead to the bounds of CNNs in existing results in Table 2.
In practice, one usually has k d (cid:28) p d , which exhibit a signiﬁcant improvement over existing results, i.e., √ Dk 2 (cid:28) (cid:112) D 3 p 2 .
Even without the orthogonal constraint on ﬁlters, the rank r in CNNs is usually of the same order with width p , which also makes the existing bound undesirable.
On the other hand, it is widely used in practice that k d = µs d for some small constant µ ≥ 1 in CNNs, then we have ( k d /s d ) D/ 2 (cid:28) p D/ 2 resulted from
R m (cid:0) G γ (cid:0) F D, (cid:107)·(cid:107) F (cid:1)(cid:1) .
Remark 1.
Corollary 2 demonstrates the generalization bound of CNNs when the input at each layer is a vector.
For matrix inputs, e.g., images, similar results hold by considering vectorized input and permuting columns of W d .
Speciﬁcally, suppose √ k d and √ p d −are integers for ease of discussion.
Consider the input as a p d − 1 dimensional vector obtained by vectorizing a √ p d − 1 × √ p d − 1 input matrix.
When the 2-dimensional (matrix) convolutional ﬁlters are of size √ k d × √ k d , we form the rows of each W ( j ) d by concatenating √ k d vectors { w ( j,i ) } √ k d i =1 padded with 0’s, each of which is a concatenation of one row of the ﬁlter of size √ k d with some zeros as follows: bounds of CNNs.
We suppose R and γ are generic constants for ease of illustration.
The results of
CNNs in existing works are obtained by substituting the corresponding norms of the weight matrices generated by orthogonal ﬁlters, i.e.,
[Table]
Correspondingly, the stride size is s 2 d k d on average and we have (cid:107) W d (cid:107) 2 ≤ k d s d if (cid:13)(cid:13) w ( j,i ) (cid:13)(cid:13) 2 = 1 for all i, j ; see Appendix E for details.
This is equivalent to permuting the columns of W d generated as in (8) by vectorizing the matrix ﬁlters in order to validate the convolution of the ﬁlters with all patches of the matrix input.
Remark 2.
A more practical scenario for CNNs is when a network has a few fully connected layers after the convolutional layers.
Suppose we have D C convolutional layers and D F fully connected layers.
From the analysis in Corollary 2, when s d = k d for convolutional layers and (cid:107) W d (cid:107) 2 = 1 for fully connected layers, we have that the overall ERC satisﬁes (cid:101)
[Equation]
(cid:16)
[Equation]
4. R ES N ETS WITH S TRUCTURED W EIGHT M ATRICES
Residual networks (ResNets) (He et al., 2016) is one of the most powerful architectures that allows training of tremendously deep networks.
Given an input x ∈ R p 0 , the output of a Dlayer ResNet is deﬁned as f ( V D , U D , x ) =
f V D ,U D ( · · · f V 1 ,U 1 ( x ))
∈
R p D , where f V d ,U d ( x ) =
σ ( V d · σ ( U d x ) + x ) .
For any two layers i, j ∈ [ D ] with i < j , we denote J i : j as the Jacobian from layer i to layer j , i.e., f V i ,U j ( · · · f V i ,U i ( x ))
=
J i : j · x .
Then we denote the class of ResNets with bounded weight V D
= { V d ∈ R p d × q d } Dd =1 , U D
= { U d ∈ R q d × p d − 1 } Dd =1 and Jacobian as
[Equation]
We then provide an upper bound of the ERC for ResNets in the following corollary.
The proof is provided in Appendix D.
Corollary 3.
Let g γ be a 1 γ -Lipschitz and bounded loss function, i.e., | g γ | ≤ b , and F RN D, (cid:107)·(cid:107) 2 be the class of ResNets deﬁned in (9) with p d = p and q d = q for all d ∈ [ D ] .
Denote C 1 =
[Equation]
Then the ERC satisﬁes
[Table]
Compared with the D -layer networks without shortcuts (1), ResNets have a stronger dependence on the input due to the shortcuts structure, which leads to ( B V d , 2 B U d , 2 + 1) dependence for each layer.
When B V d , 2 and B U d , 2 are of order 1 / √ D , we still have (cid:81)
Dd =
( B V d , 2 B U d , 2 + 1) =
O (1) .
This partially explains the observation in practice that ResNets have good performance when the weight matrices have relatively small scales.
Also note that to achieve the same bound for R m
[Table]
we require B V d , F , B U d , F ≤ c , which leads to a much smaller parameter space than the space corresponding to B V d , 2 , B U d , 2 ≤ c for the same c .
4. H YPERSPHERICAL N ETWORKS
We also consider the hyperspherical networks (SphereNets)
(Liu et al., 2017b), which demonstrate improved performance than the vanilla DNNs.
In speciﬁc, the SphereNets has the same architecture with DNNs deﬁned in (1), except that the weight matrix can be viewed as (cid:102) W d = S W d W d , where S W d is a diagonal matrix with the i -th diagonal entries being the Euclidean norm of the i -th row of W d .
Note that we do not normalize the input x as in (Liu et al., 2017b) for ease of the discussion.
A direct result of applying Theorem 1 implies that R m (cid:16) G γ (cid:16) F SN D, (cid:107)·(cid:107) 2 (cid:17)(cid:17) =
(cid:101)
O
( R · (cid:81)
Dd =1 B (cid:102) W d , 2 · √ Dpr/ ( γ √ m )) .
Such a self-normalization architecture has a beneﬁt that B (cid:102) W d ,is small (close to 1) in general when the weights are spread out.
In addition, it has lower computational costs than the weight normalization based on the spectral norm directly, and improved empirical results over batch normalization have been observed (Liu et al., 2017b;a).
4. E XTENSION TO W IDTH -C HANGE O PERATIONS
Change the width for certain layers is a widely used operation, e.g., for CNNs and ResNets, which can be viewed as a linear transformation in many cases.
In speciﬁc, denote x { d } ∈ R p d as the output of the d -th layer.
Then we can use a transformation matrix T d ∈ R p
d +1 × p d to denote the operation to change the dimension between the output of the d -th layer and the input of the ( d + 1) -th layer as f W d +1 (cid:0) x { d } (cid:1)
=
σ (cid:0) W d +1 T d x { d } (cid:1) .
Denote the set of layers with width changes by I T ⊆
[ D ] .
Combining with Theorem 1, we have that the ERC satisﬁes
R
[Equation]
(cid:17) .
Next, we illustrate several popular examples to show that Π t ∈I T (cid:107) T t (cid:107) 2 is a size independent constant.
We refer to Goodfellow et al.
(2016) for more operations of changing the width.
Width Expansion.
Two popular types of width expansion are padding and 1 × 1 convolution.
For ease of discussion, suppose p d +
=
s · p d for some positive integer s ≥ 1 .
Taking padding with 0 as an example, where we plug in ( s − 1) zeros before each entry of x { d } , which is equivalent to setting
T d ∈ R sp d × p d with ( T d ) ij = 1 if i = js , and ( T d ) ij = 0 otherwise.
This implies that (cid:107) T d (cid:107) 2 = 1 .
For 1 × 1 convolution, suppose that the convolution features are { c 1 , . . .
, c s } .
Then we expand width by performing convolution (essentially entry-wise product) using s features respectively.
This is equivalent to setting T d ∈ R sp d × p d with ( T d ) ij =
c k if i =
j +( k − 1) s for k ∈ [ s ] , and ( T d ) ij
=otherwise.
It implies that (cid:107) T d (cid:107) 2 = (cid:112)(cid:80) si =1 c 2 i .
When (cid:80) si =1 c 2 i ≤ 1 , we have (cid:107) T d (cid:107) 2 ≤ 1 .
Width Reduction.
Two popular types of width reduction are average pooling and max pooling.
Suppose p d +
= p
d s is an integer for some positive integer s .
For average pooling, we pool each nonoverlapping s features into one feature.
This is equivalent to setting T d ∈ R pds × p d with ( T d ) ij = 1 /s if j = ( i − 1) s
+ k for k ∈ [ s ] , and ( T d ) ij = 0 otherwise.
This implies that (cid:107) T d (cid:107)
=
(cid:112) 1 /s .
For max pooling, we choose the largest entry in each nonoverlapping feature segment of length s .
Denote the set
I s =
{ ( i − 1) × s + 1 , . .
, i · s } .
This is equivalent to setting T d ∈ R pds × p d with ( T d ) ij = 1 if | ( x { d } )
j | ≥ | ( x { d } ) k
| ∀ k ∈
I s , k (cid:54) =
j , and ( T d ) ij = 0 otherwise.
This implies that (cid:107) T d (cid:107) 2 = 1 .
For pooling with overlapping features, similar results hold.
4. N UMERICAL E VALUATION
To better illustrate the difference between our result and existing ones, we demonstrate some comparison results in Figure 1 using real data.
In speciﬁc, we train a simpliﬁed VGG19-net (Simonyan & Zisserman, 2014) using 3 × 3 convolution ﬁlters (with unit norm constraints) on the
CIFAR-10 dataset (Krizhevsky & Hinton, 2009).
We ﬁrst compare with the capacity terms in Bartlett et al.
(2017) (Bound1),
Neyshabur et al.
(2017) (Bound2), and
Golowich et al.
(2017) (Bound3) by ignoring the common factor R γ √ m as follows:
Ours: Π Dd =1 B d, 2 (cid:113) k
(cid:80) Dd =1 n d •
Bound1:
Π Dd =1 B d, 2 (cid:18)(cid:80)
Dd =1 B 2 / 3 d, 2 → 1 B 2 / 3 d, 2 (cid:19) 3 /•
Bound2:
Π Dd =1 B d,(cid:114) D 2 p (cid:80)
Dd =1 p d B 2 d, F B 2 d,• Bound3:
Π Dd =1 B d, F √ D
[Figure]
Note that since we may have more ﬁlters n d than their dimension k , we do not assume orthogonality here.
Thus we simply use the upper bounds of norms B d rather than the form as in Table 2.
Following the analysis of Theorem 1, we have (cid:113) k
(cid:80) Dd =1 n d dependence rather than √ Dpr as k (cid:80) Dd
=1 n d is the total number free parameter for CNNs, where n d is the number of ﬁlters at d -th layer.
For the same network and corresponding weight matrices, we see from Figure 1 (a) that our result ( 10 4 ∼ 10 5 ) is signiﬁcantly smaller than Bartlett et al.
; Neyshabur et al.
(2017) ( 10 8 ∼ 10 9 ) and Golowich et al.
(2017) ( 10 14 ∼ 10 15 ).
As we have discussed, our bound beneﬁts from tighter dependence on the dimensions.
Note that k (cid:80) Dd =1 n d is approximately of order Dk 2 , which is signiﬁcantly smaller than (cid:0) (cid:80) Dd =1 B 2 / 3 d, 2 → 1 /B 2 / 3 d, 2 (cid:1) 3 in Bartlett et al.
and D 2 p (cid:80) Dd =1 p d B 2 d, F /B 2 d, 2 in
Neyshabur et al.
(2017) (both are of order D 3 pr ).
In addition, this veriﬁes that spectral dependence is signiﬁcantly tighter than Frobenius norm dependence in Golowich et al.
Further, we show the training accuracy, testing accuracy, and the empirical generalization error using different scales on the norm of the ﬁlters in Figure 1 (b).
We see that the generalization errors decrease when the norm of ﬁlters decreases.
However, note that when the norms are too small, the accuracies drop signiﬁcantly due to a potentially much smaller parameter space.
Thus, the scales (norms) of the weight matrices should be nether too large (induce large generalization error) nor too small (induce low accuracy) and choosing proper scales is important in practice as existing works have shown.
On the other hand, this also support our claim that when R m (cid:0) G γ (cid:0) F D, (cid:107)·(cid:107) F (cid:1)(cid:1) (or other existing bound) attains the same order with our R m (cid:0) G γ (cid:0) F D, (cid:107)·(cid:107) 2 (cid:1)(cid:1) , we have better training/testing performance.
Further experimental result that compare our bound (Corollary 1) with (Zhou & Feng, 2018; Arora et al., 2018) in the bounded output case is provided in Appendix F.1.
D ISCUSSION
Our investigation on the generalization bound establishes that the spectral norm is a tighter characterization for the norm based analysis of the ERC on DNNs, compared with other norms (e.g., Frobenius norm).
This is also closely related the efﬁcient optimization of DNNs.
For example, effective initializations generally require the spectral norms (rather than the Frobenius norm) of weight matrices to be approximately constant (Glorot & Bengio, 2010).
Therefore, properly choosing the structure of weight matrices can signiﬁcantly affect both generalization and optimization performance of DNNs, as is observed in various applications (Goodfellow et al., 2016).
On the other hand, the lower bound in Golowich et al.
(2017) states that R m (cid:0) G γ (cid:0) F D, (cid:107)·(cid:107) 2 (cid:1)(cid:1)
=
Ω (cid:16) R · (cid:81) Dd =1 B d, 2 γ (cid:112) pm (cid:17) , which implies that further improvement maybe achievable.
A P ROOF OF T HEOREM
Our analysis is based on the characterization of the Lipschitz property of a given function on both input and parameters.
Such an idea can potentially provide tighter bound on the model capacity in terms of these Lipschitz constants and the number of free parameters, including other architectures of DNNs.
We ﬁrst provide an upper bound for the Lipschitz constant of f ( W D , x ) in terms of the input x .
Lemma 1.
Given W D , for any f ( W D , · ) ∈ F D, (cid:107)·(cid:107) 2 and x 1 , x 2 ∈ R p 0 , we have
[Equation]
Proof.
We prove by induction.
Speciﬁcally, we have
[Equation]
where ( i ) comes from the entry-wise 1 –Lipschitz continuity of σ ( · ) .
For the ﬁrst layer, we have
[Equation]
By repeating the argument above, we complete the proof.
Next, we provide an upper bound for the Lipschitz constant of f ( W D , x ) in terms of the parameters
W D .
Lemma 2.
Given any x ∈ R p 0 satisfying (cid:107) x (cid:107) 2 ≤ R , for any f ( W D , x ) , f (cid:16)(cid:102) W D , x (cid:17) ∈ F D, (cid:107)·(cid:107)with W D
= { W d } Dd =1 and (cid:102) W D
=
(cid:110)(cid:102) W d (cid:111)
D d =1 , and denote B Jac \ d, 2 = max d ∈ [ D ] B Jac 1:( d − 1) , 2 B Jac ( d +1): D, 2 , then we have
[Equation]
Proof.
Given x and two sets of weight matrices { W d } Dd =1 , (cid:110)(cid:102) W d (cid:111)
D d =1 , we have
[Equation]
f W D
[Equation]
f W D
[Equation]
[Equation] [Equation]
f W
[Equation] [Equation]
− f
[Equation]
W
[Equation]
W
[Equation]
[Table]
where ( i ) is derived from adding and subtracting intermediate neural network functions recurrently, where f W D (cid:16) · · · f W d +1 (cid:16) f (cid:102) W d (cid:16) · · · f (cid:102) W 1 ( x ) (cid:17)(cid:17)(cid:17) share the same output of activation functions from d + 1 -th layer to D -the layer with f W D (cid:0)
f W D − 1 ( · · · f W 1 ( x )) (cid:1) , ( ii ) is from ﬁxing the activation function output, and ( iii ) is from the entry-wise 1 –Lipschitz continuity of σ ( · ) .
On the other hand, for any d ∈ [ D ] , we further have
[Equation]
where ( i ) is from the entry-wise 1 –Lipschitz continuity of σ ( · ) and ( ii ) is from recursively applying the same argument.
In addition, we denote W d = U d V (cid:62) d and (cid:102) W d
= (cid:101) U
d (cid:101) V (cid:62) d , where U d , V d , (cid:101)
U d , (cid:101) V d ∈ R p × r and (cid:107) U (cid:107)
= (cid:107) V (cid:107)
= (cid:13)(cid:13)(cid:13)(cid:101) U (cid:13)(cid:13)(cid:13) 2 = (cid:13)(cid:13)(cid:13)(cid:101) V (cid:13)(cid:13)(cid:13)
= (cid:107) W d (cid:107) 1 / 2 2 .
Then we have (cid:13)(cid:13)(cid:13)
[Equation]
Applying (10) recursively and combining (11) and (12)
[Equation]
we obtain the desired result as
[Equation]
[Table]
U
[Table]
Lemma 3.
Suppose g ( w, x ) is L w -Lipschitz over w ∈ R h with (cid:107) w (cid:107) 2 ≤ K and α = sup g ∈G ,x ∈X m | g ( w, x ) | .
Then the ERC of G = { g ( w, x ) } satisﬁes
[Equation]
Proof.
For any w, (cid:101) w ∈ R h and X m = { x i } mi =1 , we consider the matric ∆ ( g 1 , g 2 ) =
max x i ∈X m | g 1 ( x i ) − g 2 ( x i ) | , which satisﬁes
[Equation]
Since g is a parametric function with h parameters, then we have the covering number of G under the metric ∆ in (13) satisﬁes
[Equation]
Then using the standard Dudley’s entropy integral bound on the ERC (Mohri et al., 2012), we have the ERC satisﬁes
[Equation]
Since we have
[Equation]
Then we have
[Equation]
where ( i ) is obtained by taking β
= α (cid:112) h/m .
By deﬁnition, we have
[Equation]
From Lemma 1 and 1 γ -Lipschitz continuity of g , we also have
[Equation]
From Lemma 2, we have
[Equation]
Moreover, when p d = p for all d ∈ [ D ] , we have
[Equation]
Combining the results above with Lemma 3 and h = 2 Dpr , we have
[Equation]
B P ROOF OF C OROLLARY
The analysis follows Theorem 1, except that the bound for α in (15) satisﬁes
[Equation]
since g satisﬁes | g | ≤ b and 1 γ -Lipschitz continuous.
Then we have the desired result.
C P ROOF OF C OROLLARY
We ﬁrst show that using unit norm ﬁlters for all d ∈ [ D ] and n d ≤ k d , we have
[Equation]
First note that when n d = k d , due to the orthogonality of (cid:8) w ( d,j ) (cid:9) k d j =1 , for all i, q ∈ [ k d ] , i (cid:54) =
q , we have
[Equation]
When n d = k d , we have for all i ∈
[ p d − 1 ] , the diagonal entries of W (cid:62) d W d satisfy
[Equation]
18) where ( i ) is from (17).
For the off-diagonal entries of W (cid:62) d W d , i.e., for i (cid:54) =
q , i, q ∈ [ p d ] , we have
[Equation]
) where ( i ) is from (17).
Combining (18) and (19), we have that W (cid:62) d W d is a diagonal matrix with
[Equation]
d .
For n d < n k , we have that W d is a row-wise submatrix of that when n d = k d , denoted as (cid:102) W d .
Let
S ∈ R ndkdsd × p d be a row-wise submatrix of an identity matrix corresponding to sampling the row of
W d to form (cid:102) W d .
Then we have that (16) holds, and since
[Equation]
Suppose k 1 = · · · = k D =
k for ease of discussion.
Then following the same argument as in the proof of Theorem 1 and Lemma 3, we have
[Equation]
n
[Equation]
Using the fact that the number of parameters in each layer is no more than kn d rather than 2 pr , we have
[Equation]
D P ROOF OF C OROLLARY
The analysis is analogous to the proof for Theorem 1, but with different construction of the intermediate results.
We ﬁrst provide an upper bound for the Lipschitz constant of f ( V D , U D , x ) in terms of x .
Lemma 4.
Given V D and U D , for any f ( V D , U D , · ) ∈ F D, (cid:107)·(cid:107) 2 and x 1 , x 2 ∈ R p 0 , we have
[Equation]
Proof.
Consider the ResNet layer, for any x 1 , x 2 ∈ R k , we have
[Equation]
where ( i ) is the fact that σ is 1 –Lipschitz, and ( ii ) is from repeating the arguments of ( i ) and ( ii ) .
By recursively applying the argument above, we have the desired result.
Next, we provide an upper bound for the Lipschitz constant of f ( V D , U D , x ) in terms of V D and
U D .
Lemma 5.
Given any x ∈ R p 0 satisfying (cid:107) x (cid:107) 2 ≤ R , for any f ( V D , U D , x ) , f (cid:16)(cid:101) V D , (cid:101) U D , x (cid:17) ∈
F D, (cid:107)·(cid:107) 2 with V D
=
{ V d } Dd =1 , U D
= { U d } Dd =1 , (cid:101) V D
= (cid:110)(cid:101) V d (cid:111)
D d =1 , and (cid:101) U D
= (cid:110)(cid:101) U d (cid:111)
D d =1 , and denote
B
[Equation] [Equation]
, 2 = max d
[Equation]
D ] B Jac
[Equation]
[Equation] [Equation]
B
[Equation] [Equation]
[Table]
Proof.
Given x and two sets of weight matrices { W d } Dd =1 , (cid:110)(cid:102) W d
[Table]
D d =1 , we have
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
(21) where ( i ) and ( ii ) from the entry-wise 1 –Lipschitz continuity of σ ( · ) .
In addition
[Equation]
for any d ∈ [ D ]
[Equation]
we further have
[Equation]
[Equation]
Combining (21) and (22)
[Equation]
we obtain
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
On the other hand, for any d ∈ [ D ] , we have
[Equation]
where ( i ) is from the entry-wise 1 –Lipschitz continuity of σ ( · ) and ( ii ) is from recursively applying the same argument.
Let p 1 = · · · = p D = p and q 1 = · · · = q D = q .
Then following the same argument as in the proof of Theorem 1 and (23)
[Equation]
we have
[Equation]
2 + B U d , 2 ) , and h = 2 Dpq,
Combining Lemma
[Equation]
and Lemma 4, Lemma 5, we have
[Equation]
E S PECTRAL B OUND FOR W d IN CNN S WITH M ATRIX F ILTERS
We provide further discussion on the upper bound of the spectral norm for the weight matrix W d in
CNNs with matrix ﬁlters.
In particular, by denoting W d using submatrices as in (7), i
[Equation]
e.,
[Equation]
we have that each block matrix W ( j ) d is of the form
[Equation]

[Equation]
where W ( j ) d ( i, l ) ∈ R
[Table]
for all i ∈ (cid:20)
[Table]
(cid:21) and l ∈ (cid:2)
√ p d − 1 (cid:3) .
Particularly, offdiagonal blocks are zero matrices, i.e., W ( j ) d ( i, l ) =
for i (cid:54)
=
l .
For diagonal blocks, we have
W ( j ) d ( i, i ) =
 
[Table]
  .
(25) where w ( j, 1) { sd 1 } = w ( j, 1) 1: sd √ kd ∈ R sd √ kd and w ( j, 1) (cid:26) sd √
[Table]
= w
[Table]
Combining (24) and (25), we have that the stride for W ( j ) d is s 2 d k d .
Using the same analysis for Corollary 2.
We have (cid:107) W d (cid:107) 2 = 1 if (cid:113)(cid:80) i (cid:13)(cid:13) w ( j,i ) (cid:13)(cid:13) 2 2 = k d s d .
For image inputs, we need an even smaller matrix W ( j ) d ( i, i ) with fewer rows than (25), denoted as
W ( j ) d ( i, i )
[Table]
 
[Table]
[Equation]
 .
Then (cid:107) W d (cid:107) 2 ≤ 1 still holds if (cid:113)(cid:80)
i (cid:13)(cid:13) w ( j,i ) (cid:13)(cid:13) 2
= k d s d since W d generated using (26) is a submatrix of W d generated using (25).
F E XTENDED N UMERICAL R ESULTS F. C OMPARISON WITH E XISTING R ESULTS WITH B OUNDED O UTPUT
We compared our norms based bound with several other norm based results in Section 4.5.
In this section, we further compare our result with bounded output in Corollary 1 with Zhou & Feng (2018); Arora et al.
Analogous to Section 4.5, we train a simpliﬁed VGG19-net (Simonyan & Zisserman, 2014) using 3 × 3 convolution ﬁlters (with unit norm constraints) on the CIFARdataset (Krizhevsky & Hinton, 2009).
We compare with the capacity terms in Zhou & Feng (2018)
(Bound 1) and Arora et al.
(Bound 2) by ignoring the factor R γ √ m as follows:
• Ours: b (cid:113) k (cid:80)
Dd =1 n d .
Note that b = 1 in this case.
• Bound1:
(cid:113) Dk (cid:80) Dd =1 rank ( W d ) · n d .
Note that their activation functions for the intermediate layers are sigmoid and the last layer is softmax, with squared error, which essentially has a bounded output.
We also take the last layer as convolution layer for ease of discussion.
[Figure]
, where C = Ω(1) is the activation contraction, β = Ω(1) is the well-distributed Jacobian, µ d = O (1 / √ p ) is the layer cushion, and µ d → = O (1 / √ p ) is the interlayer cushion.
See more details in Arora et al.
[Figure]
The resulting bounds on the trained networks are provided in Figure 2.
We observe that our bound is smaller than Zhou & Feng (2018); Arora et al.
by at least an order of magnitude.
Speciﬁcally, our bound is of order ≈ 10 2 , while Zhou & Feng (2018) result in a bound of order > 10 3 and Arora et al.
(2018) result in a bound of order > 10 4 .
This coincide with our discussion in Section 3.2 and allows us to obtain non-vacuous bound (generalization bound < 1 ) with moderate training sample sizes (e.g., m = Ω(10 4 ) ).
Also note that compared with the norm based based results in Figure 1, the results here based on the bounded output are signiﬁcantly tighter in general.
We regard this as a trade-off between the looser norm based bounds that allow more explicit interpretability in terms of the weight matrix structures and the tighter bounds that are more obscure in terms of dependence on the network structures.
A result that have both merits are desired as a future direction.
F
C OMPARISON BETWEEN B J AC \ d, 2 AND (cid:81) d B d,
We demonstrate the empirical difference between B Jac \ d, 2 and (cid:81)
d B d, 2 .
Using the same setting of the network and dataset as above, we provide the empirical distribution of B Jac \ d, 2 and (cid:81)
d B d, 2 over the training set, which is provided in Figure 3.
We can observe that the values of B Jac \ d, 2 are approximately 2 orders smaller than the values of (cid:81)
d B d, 2 , which support our claim that B Jac \ d, 2 is a tighter quantiﬁcation then (cid:81)
d B d, 2 .
[Figure]