A S PECTRAL A PPROACH TO G ENERALIZATION AND O PTIMIZATION IN N EURAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
The recent success of deep neural networks stems from their ability to generalize well on real data; however, Zhang et al.
(Zhang et al., 2016) have observed that neural networks can easily overﬁt random labels.
This observation demonstrates that with the existing theory, we cannot adequately explain why gradient methods can ﬁnd generalizable solutions for neural networks.
In this work, we use a Fourierbased approach to study the generalization properties of gradient-based methods over 2-layer neural networks with sinusoidal activation functions.
We prove that if the underlying distribution of data has nice spectral properties such as bandlimitedness, then the gradient descent method will converge to generalizable local minima.
We also establish a Fourier-based generalization bound for bandlimited spaces, which generalizes to other activation functions.
Our generalization bound motivates a grouped version of path norms for measuring the complexity of 2-layer neural networks with ReLU activation functions.
We demonstrate numerically that regularization of this group path norm results in neural network solutions that can ﬁt true labels without losing test accuracy while not overﬁtting random labels.
I NTRODUCTION
Deep neural networks (DNNs) have achieved state-of-the-art performance on a wide array of diverse tasks (LeCun et al., 2015).
A given DNN architecture represents a highly rich space of hypotheses.
However, numerous empirical results have demonstrated that a simple stochastic gradient descent (SGD) learner can efﬁciently search over this space to ﬁnd a solution that achieves high performance on both training and test data.
Despite many successful applications of DNNs to practical tasks such as computer vision (Krizhevsky et al., 2012), natural language processing (Collobert & Weston, 2008) and speech recognition (Hinton et al., 2012), our basic understanding of the factors that drive DNN generalization is still lacking.
Addressing generalization for DNNs is hard for two fundamental reasons: 1) Empirical risk minimization for neural networks is a non-convex optimization problem with possibly many local minima, and 2) Two different local minima with the same training performance can achieve signiﬁcantly different performance on test data.
For these reasons, the neural network optimization method plays an important role in the generalizability of the local minima found.
For example, SGD has been empirically shown to outperform large-batch gradient descent (Keskar et al., 2016).
Also, the performance of gradient methods can be improved upon by incorporating the geometry of observed data (Duchi et al., 2011; Neyshabur et al., 2015a).
For DNNs, however, a good optimization method is not sufﬁcient for guaranteeing good generalization.
Zhang et al.
(Zhang et al., 2016) empirically demonstrate that a neural network trained by SGD can easily overﬁt random labels on the CIFAR-10 (Krizhevsky & Hinton, 2009) data.
Yet, the same neural network ﬁtted by the same SGD algorithm achieves good generalization performance for the original
CIFAR-10 labels.
This observation challenges the ability of traditional learning theory to explain why
SGD learns generalizable hypotheses over neural networks.
To shed light on this phenomenon, two recent works have developed generalization bounds and complexity measures for neural networks which can distinguish the local minima found for true and random labels.
(Bartlett et al., 2017) proves a margin-based generalization bound and shows how it correlates with the generalization risk of DNNs when ﬁtting true and random labels.
(Neyshabur et al., 2017) explores different complexity scores for DNNs and how they behave differently for true and random labels.
The complexity measures investigated in these works can effectively distinguish generalizable from poorly-generalizable local b1.
[Figure]
minima.
They do not explain, however, why SGD converges to generalizable local minima when there exist poorly-generalizable local minima which can also perfectly ﬁt the training set.
To approach this question, one needs to understand the key characteristic of CIFAR-10’s original labeling which differentiates it from random labels and how it is exploited by SGD to achieve good generalization performance.
In this work, we approach this problem in the Fourier domain where non-random labeling schemes behave completely differently from random labeling schemes.
While signals recoverable from few measurements possess nice spectral properties such as bandlimitedness, fully random stochastic processes are not bandlimited and not recoverable from any ﬁnite number of measurements.
Using spectral analysis, we focus on characterizing spectral properties of an underlying distribution which can be exploited by gradient-based methods to converge to generalizable local minima.
We address this problem for 2-layer neural networks (see Figure 1a) with sinusoidal activation functions, where we show that if the underlying labeling scheme has limited bandwidth and Fourier (cid:96) 1 -norm (i.e. "nice" Fourier properties), we expect a gradient-based method to achieve good generalization performance.
To arrive at this result, we ﬁrst develop a Fourier-based generalization bound for 2-layer neural networks in terms of bandwidth and Fourier (cid:96) 1 -norm.
Next, we prove that the local minima found by the gradient descent method over a 2-layer neural network with sine activation have bandwidth and Fourier (cid:96) 1 -norm bounded in terms of the spectral properties of the underlying labeling scheme.
As a byproduct of our Fourier analysis, we derive generalization bounds for 2-layer neural networks with general activation functions.
For bandlimited activation functions with ﬁnite Fourier (cid:96) 1 -norm, such as sinusoidal or Gaussian activation 1 , our bound is tighter than the generalization bound obtained using only the Lipschitz constant of the activation function.
For ReLU-type activation functions, our generalization bound is comparable to Lipschitz-based bounds; however, it leads to a grouped version of the path norms developed in (Neyshabur et al., 2015a).
We therefore call this capacity norm group path norm which can be used as an additive penalty to regularize 2-layer neural networks with ReLU activation.
Our numerical experiments suggest that the generalization gap can be effectively tightened by regularizing the group path norm.
Figure 1b demonstrates how group path norm regularization can help close the generalization gap for both true and random labels.
R ELATED WORK
Generalization has been a topic of central interest in statistical learning theory (Vapnik, 1999; ShalevShwartz & Ben-David, 2014).
Generalization bounds have been derived using the stability of a learning algorithm
(Bousquet & Elisseeff, 2002) and various complexity measures of a function space such as VC-dimension (Vapnik, 2013) and Rademacher complexity (Bartlett & Mendelson, 2002).
(Hardt et al., 2015) develops a stability-based generalization result for SGD as the learning algorithm, which holds for both convex and non-convex loss functions.
We note that spectral analysis has provided a powerful framework for studying neural networks.
(Barron, 1993) uses a Fourier-based approach to prove the universal approximation theorem for 2-layer neural networks.
Similarly, (Lee et al., 2017) applies Fourier analysis to extend Barron’s result to a general feedforward neural network.
(Rippel et al., 2015) uses a spectral approach to model and analyze convolutional neural networks (CNNs) and introduce the spectral pooling scheme for
CNNs.
Also, our Fourier-based approach to analyze SGD’s performance for 2-layer neural networks follows the same prinicples as the analysis performed in (Shamir, 2016) to prove the hardness of ﬁtting periodic labeling schemes via gradient-based methods.
We should note that in this work we use only periodic activation functions and not periodic labeling schemes.
Therefore, the hardness result shown in (Shamir, 2016) does not affect our numerical experiments.
In general, theoretical studies of neural networks can be categorized into three main categories: 1)
Approximation: Neural networks have been proven to be powerful in expressing a very rich class of functions (Cybenko, 1989) and in general deeper networks need fewer neurons to express the same class of functions (Eldan & Shamir, 2016; Liang & Srikant, 2016).
2) Generalization:
Tight bounds have been shown on the VC dimesnion of feedforward neural networks (Anthony & Bartlett, 2009;
Harvey et al., 2017).
Also, norm-based Rademacher complexity bounds have been developed at (Bartlett & Mendelson, 2002; Neyshabur et al., 2015b).
Sharpness of local minima and its connection to their generalizibility have been the focus of several recent works (Keskar et al., 2016;
Dinh et al., 2017; Neyshabur et al., 2017) 3) Optimization: theoretical studies have shown both positive (Andoni et al., 2014; Daniely, 2017) and negative (Shalev-Shwartz et al., 2017) results about the performance of gradient-based methods in training neural networks.
P RELIMINARIES 3. S UPERVISED LEARNING AND GENERALIZATION
Suppose that we are given n samples ( x i , y i ) ni
=1 drawn i.i.d.
from a population distribution P X ,Y .
Here X denotes the random vector of features and Y denotes the target variable.
Using these n samples, the goal of a supervised learner is to ﬁnd a prediction rule f from a function space F which can predict Y for an unseen test sample X .
Therefore, given loss function (cid:96) the supervised learner wants to ﬁnd f ∗ ∈ F minimizing the population risk, deﬁned as E (cid:2) (cid:96) (cid:0) f
( X ) , Y (cid:1)(cid:3) averaged under the population distribution.
However, the supervised learner does not know the population distribution P X ,Y and has only access to the n training samples.
The supervised learner can minimize the empirical risk, deﬁned as 1 /n (cid:80) ni =1 (cid:96) (cid:0) f
( x i ) , y i (cid:1) and ﬁnd f emp n .
Since we only observe a limited number of samples, the empirical risk would be different from the population risk.
The generalization risk, deﬁned for f ∈ F as E [ (cid:96) ( f ( X ) , Y )] − 1 n (cid:80) ni =1 (cid:96) ( f ( x i ) , y i ) , is the difference among the population risk and empirical risk for f .
Studying the behavior of f emp n ’s generalization risk for different function spaces and learning algorithms is a topic of central interest in statistical learning theory.
3. F OURIER T RANSFORM AND B ANDLIMITED F UNCTIONS
Consider a real-valued function f : R k
→ R .
The Fourier transform of this function, which we denote by (cid:98) f , is deﬁned as
[Equation]
Some important examples of Fourier transform are:
• Sinusoidal function: f ( x ) = exp(
πi ω T x ) , then (cid:98) f ( ξ ) =
δ ( ξ − ω ) where δ denotes the Dirac delta function, which also implies – f ( x ) = cos(2 π ω T x ) , then (cid:98) f ( ξ )
= 1 / 2 (cid:2) δ ( ξ + ω ) + δ ( ξ − ω ) (cid:3) .
– f ( x ) = sin(2 π ω T x ) , then (cid:98) f ( ξ ) =
i/ 2 (cid:2) δ ( ξ + ω ) − δ ( ξ − ω ) (cid:3) .
• Gaussian function: f ( x ) =
( √ 2 πσ ) k exp (cid:0) −(cid:107) x (cid:107) 22 / 2 σ 2 (cid:1) , then (cid:98) f ( ξ ) =
exp (cid:0) − σ 2 (cid:107) ξ (cid:107) 22 / 2 (cid:1) .
Thus, the Fourier transform of a Gaussian function preserves the Gaussian shape.
A function f is called B -bandlimited if (cid:98) f ( ξ ) = 0 for every ξ where (cid:107) ξ (cid:107) 2 > B .
The smallest B for which this property holds is called the bandwidth of f .
We use B ( f ) to denote the bandwidth of function f .
We also use (cid:107) (cid:98) f (cid:107) 1 to denote the (cid:96) 1 -norm of f ’s Fourier transform,
[Equation]
which we call the Fourier (cid:96) 1 -norm of f .
Fourier (cid:96) 1 -norm can be interpreted as the absolute volume under f ’s Fourier transform, and is an approximate measure of (cid:98) f ’s sparsity.
Fourier (cid:96) 1 -norm is both scale and shift invariant, i.e. if we deﬁne g ( x ) =
f ( Wx + b ) for a real-valued f and W ∈ R r × k and b ∈ R r for some r ≤ k , then (cid:107) (cid:98) g
(cid:107)
= (cid:107) (cid:98) f
(cid:107) 1 .
Some other useful properties of Fourier transform are:
• Synthesis: f ( x ) =
(cid:82) (cid:98) f
( ξ ) exp (cid:0)
πi ξ T x (cid:1) d ξ , which also implies (cid:107) (cid:98) f (cid:107) 1 = f (0) if (cid:98) f is real and non-negative.
• Shift: (cid:98) f b ( ξ )
= exp(2 πi b T ξ ) (cid:98) f ( ξ ) where f b ( x ) :=
f ( x − b ) , which implies (cid:107) (cid:99) f b (cid:107)
=
(cid:107) (cid:98) f
(cid:107)and B ( f b ) =
B ( f ) .
• Derivative: (cid:99) ∇ f ( ξ )
= 2 πi (cid:98) f ( ξ ) ξ , where ∇ f denotes the gradient of f . • Isometry: (cid:82) f
( x ) g ( x ) d x = (cid:82) (cid:98) f ( ξ ) (cid:98) g ( ξ ) d ξ where z denotes the complex conjugate of z . • Convolution: (cid:99) fg = (cid:98)
f
(cid:63) (cid:98) g where (cid:63) denotes the convolution operator
i
e. (cid:98) f
(cid:63) (cid:98) g ( ξ ) :
= (cid:82) (cid:98) f
( η ) (cid:98) g ( ξ − η )
d η .
Therefore, B ( fg ) ≤ B ( f ) + B ( g ) and (cid:107) (cid:99) fg (cid:107) 1 ≤ (cid:107)
[Table]
A F OURIER - BASED G ENERALIZATION B OUND
Consider a supervised learning task with n training samples (cid:0) x i , y i (cid:1) n i =1 and function space F .
We are interested in uniform convergence bounds on the generalization risk.
A standard approach to bound the generalization risk is based on the notion of Rademacher complexity.
Given samples (cid:0) x
i , y i (cid:1) n i =1 , the empirical Rademacher complexity of F is deﬁned as
[Equation]
where σ i ’s are i.i.d.
random variables uniformly distributed over {− 1 , +1 } .
In fact, the Rademacher complexity of F measures how well F can ﬁt some random labels over input x i ’s.
The following result shows how to bound the generalization risk over F through its Rademacher complexity.
Theorem 1 (Bartlett & Mendelson (2002)) .
Consider a ρ -Lipschitz loss function (cid:96) ( f ( x ) , y ) bounded as | (cid:96) ( z, y )
| ≤ c .
Then, for any δ > 0 , with probability at least 1 − δ
[Equation]
Since the Rademacher complexity of norm-bounded linear functions can be appropriately bounded (Kakade et al., 2009), one can effectively apply Theorem 1 to bound generalization risk over normbounded linear functions.
To use Theorem 1 in the Fourier domain, here we provide a Rademacher complexity bound for bandlimited functions with bounded Fourier (cid:96) 1 -norm.
We apply the following
Rademacher complexity bound to bound generalization risk for 2-layer neural networks in Section 5, and also to analyze the performance of gradient-based methods with sinusoidal activation functions in Section 6.
Theorem 2.
Consider function space F = (cid:8) f : R k → R s.t.
B ( f ) ≤ B, (cid:107) (cid:98) f (cid:107) 1 ≤ V (cid:9) of Bbandlimited functions with V -bounded Fourier (cid:96) 1 -norm.
Then, the empirical Rademacher complexity for samples ( x i , y i ) ni =1 is bounded as
[Equation]
Proof.
We defer the proof to the Appendix.
Corollary 1.
Assume that (cid:107) X (cid:107) 2 ≤ C holds almost surely and the loss function (cid:96) is ρ -Lipschitz.
Then, for any δ > 0 with probability at least 1 − δ the following generalization bound holds for any
B -bandlimited function f with V
-bounded Fourier (cid:96) 1 -norm:
[Equation]
Proof.
The corrollary is a direct result of applying the bound in Theorem 2 to Theorem 1.
The above corollary bounds the generalization risk uniformly over all bandlimited f ’s such that
B ( f ) ≤ B and (cid:107) (cid:98) f
(cid:107) 1 ≤ V .
Next, we apply the above results to 2-layer neural networks.
A PPLICATION OF T HEOREM 2 TO 2- LAYER N EURAL N ETWORKS
Consider a 2-layer neural network including d neurons with activation function φ in the hidden layer (See Figure 1a).
The output of this neural network is
[Equation]
If φ has bounded bandwidth and Fourier (cid:96) 1 -norm, we can apply Theorem 2 to bound the Rademacher complexity and hence generalization risk over the 2-layer neural network.
Here, we use (cid:107) W (cid:107) 2 , ∞ to denote the maximum (cid:96) 2 -norm (cid:107) w i (cid:107) 2 among all rows of W .
Corollary 2.
Let F φ = (cid:8) f
( x ) = a T φ ( Wx + b ) : (cid:107) W (cid:107) 2 , ∞ ≤ W, (cid:107) a (cid:107) 1 ≤ A (cid:9) be the class of 2-layer neural networks where B ( φ )
=
B and (cid:107) (cid:98) φ (cid:107) 1 = V .
Then, the empirical Rademacher complexity of F φ for samples ( x i , y i ) ni =1 is bounded as follows
[Equation]
Proof.
We defer the proof to the Appendix.
Notice that for bandlimited activation functions with bounded Fourier (cid:96) 1 -norm, the above generalization bound is increasing logarithmically with (cid:107) W (cid:107) 2 , ∞ .
For example, this result holds for sinusoidal activation φ ( x ) = sin(2 πx ) where (cid:107) (cid:98) φ (cid:107) 1 = 1 , B ( φ )
= 1 .
On the other hand, the existing Rademacher complexity bounds which use only the Lipschitz constant of the activation function are linear in W ’s norm (Bartlett & Mendelson, 2002).
Therefore, by exploiting the spectral properties of φ , Corollary 2 results in a tighter generalization bound than the bounds using only the Lipschitz constant of φ .
However, an unbounded function such as ReLU φ ( x ) = max( x, 0) has an inﬁnite Fourier (cid:96) 1 -norm.
Therefore, Corollary 2 does not directly apply to these functions.
The following theorem uses a boundedness assumption on input X to apply Theorem 2 to ReLU-type activation functions.
Although the following bound is growing faster than logarithmically with W ’s norm, it introduces new capacity norms for 2-layer ReLU-based networks.
Theorem 3.
Suppose that φ α ( x ) = max { x, αx } where α ∈ [0 , 1] is an arbitrary constant.
Consider the pair of dual norms ( (cid:107) · (cid:107) p , (cid:107) · (cid:107) q ) where 1 ≤ p, q ≤ ∞ and 1 /p
+ 1 /q
= 1 .
Assume that (cid:107) x i (cid:107) p ≤ C holds for all x i ’s.
Then, for F φ α = (cid:8) f a , W ( x ) =
a T φ α
( Wx
[Equation]
: (cid:80) di =1 | a i |(cid:107) w i (cid:107) q ≤ V (cid:9)
[Equation]
Proof.
We relegate the proof to the Appendix.
The above bound uses the complexity score (cid:80) di =1 | a i |(cid:107) w i (cid:107) q for each f a , W ( x )
= a T φ α
( Wx ) .
We can rewrite this complexity score in the following way, which is an (cid:96) 1 ,q -group norm on the product of weights for each path from the input nodes to the output node of the 2-layer neural network,
[Equation]
q (cid:0)
[Equation]
a , W (cid:1) =
[Equation]
Here w i,j denotes the weight on the link from the j th node of the input layer to the i th node of the hidden layer.
Based on the path-norm function deﬁned at (Neyshabur et al., 2015a), we call χ q (cid:0) f a , W (cid:1) the group path norm.
For q = 1 , χ 1 -group path norm leads to the (cid:96) 1 -path norm for 2-layer neural networks.
We can use group path norms as an additive regularization penalty to learn over 2-layer neural networks.
In our numerical experiments, we test the performance of χ 2 -group path norm and (cid:96) 1 -path norm regularization to control the generalization risk over 2-layer neural networks.
F OURIER A NALYSIS OF G RADIENT - BASED METHODS FOR 2- LAYER N EURAL N ETWORKS WITH S INE A CTIVATION
In this section, we apply Fourier analysis for a 2-layer neural network with sinusoidal activation.
We aim to understand the connection between generalizibility of local minima found by gradient-based methods and spectral properties of the population distribution P X ,Y .
As a simplifying assumption, let’s assume that target variable Y is a deterministic function Y ( x ) of input X , which we call the labeling scheme.
In our analysis, we consider the squared-error loss (cid:96) ( y, y (cid:48) )
= ( y − y (cid:48) ) 2 .
We speciﬁcally ask this question: how can spectral properties of labeling scheme Y ( x ) and population density function P X ( x ) affect the generalization performance of a gradient-based method?
To address this question, we use a similar strategy to the analysis performed in (Mei et al., 2016) by establishing generalization results for both the empirical risk and the gradient of empirical risk.
First, we show that the bandwidth and Fourier (cid:96) 1 -norm for the local minima of the population risk can be bounded in terms of the bandwidth and Fourier (cid:96) 1 -norm of Y ( x ) and
P X ( x ) .
Next, we establish a generalization result for the gradient of the empirical risk, proving that the gradient of empirical risk would stay close to the gradient of population risk given that Y ( x ) has limited bandwidth and Fourier (cid:96) 1 -norm.
These two results show that by assuming a labeling scheme with limited bandwidth and Fourier (cid:96) 1 -norm, the local minima found by the gradient descent (in general large-batch gradient descent) method will generalize well.
6. P OPULATION R ISK WITH S INUSOIDAL A CTIVATION
Consider f a , W , b ( x ) = (cid:80) dj
=1 a j sin(2 π w Tj x + b j ) coming from a 2-layer neural network with d sinusoidal hidden units.
Given the labeling scheme Y ( x ) the population risk will be
[Equation] [Equation]
where the expectation is according to the population density function P X ( x ) .
Lemma 1.
Consider the population risk in (11) .
Assume w j satisﬁes
∀
i (cid:54) =
j : min (cid:8) (cid:107) w i − w j (cid:107) 2 , (cid:107) w i + w j (cid:107) 2 (cid:9) >
B ( P X ) .
Then, if ( a , W , b ) is assumed to be a local minimum of the population risk,
[Equation]
Proof.
We defer the proof to the Appendix.
Lemma 1 says that if the component a j sin(2 π w Tj x ) becomes isolated for a local minimum, by which we mean there are no other component a i sin(2 π w Ti x ) with min {(cid:107) w i − w j (cid:107) 2 , (cid:107) w i + w j (cid:107) 2 } less than P X ’s bandwidth, then the value of a j for that local minimum is nicely bounded in terms of the population distribution.
This result leads to the following Theorem which describes the Fourier properties of the local minima of the population risk.
Theorem 4.
Consider the minimization problem of the population risk (11) .
If a local minimum ( a ∗ , W ∗ , b ∗ ) satisﬁes the isolated components condition, i.e. for any two different i, j we have min (cid:8) (cid:107) w ∗ i − w ∗ j (cid:107) 2 , (cid:107) w ∗ i + w ∗ j (cid:107) 2 (cid:9) >
2 B ( P X ) , then for the local minimum function f a ∗ , W ∗ , b ∗
• B ( f a ∗ , W ∗ , b ∗ ) ≤ B ( Y ) + B ( P X ) , • (cid:107) (cid:98) f
a ∗ , W ∗ , b ∗ (cid:107) 1 ≤ 2 (cid:107) (cid:98) Y (cid:107) 1 .
Proof.
We defer the proof to the Appendix.
Theorem 4 implies that the bandwidth of the local minima of the population risk is less than the sum of bandwidths for Y and P X .
Also, the Fourier (cid:96) 1 -norm for the local minima of the population distribution is bounded by twice the Fourier (cid:96) 1 -norm of Y .
Remark 1.
To apply Theorem 4, the bandwidth of P X needs to be smaller than half the distance among w ∗ i ’s.
For example, suppose that X ∼ N ( µ , σ 2 I k × k ) has a multivariate Gaussian distribution with mean µ and diagonal covariance matrix with standard deviation σ .
Then, the above theorem shows that if for any i, j we have min (cid:8) (cid:107) w ∗ i − w ∗ j (cid:107) 2 , (cid:107)
w
∗
i
+
w ∗ j (cid:107) 2 (cid:9) > 2 C/σ for some constant C , then
• B ( f a ∗ , W ∗ , b ∗ ) ≤ B ( Y ) + O (cid:0) √ k/σ (cid:1) , • (cid:107) (cid:98) f
a ∗ , W ∗ , b ∗ (cid:107) 1 ≤ 2(1 + d exp
[Equation]
Proof.
See the proof of Theorem 4 in the Appendix.
6. G ENERALIZATION TO THE E MPIRICAL R ISK
Theorem 4 characterizes the Fourier properties of the local minima for the population risk.
However, we want to investigate the generalization performance of the local minima of the empirical risk deﬁned for training samples ( x i , Y ( x i ))
ni =1 as
[Equation]
To address this question
[Equation]
note that the bandwidth and Fourier (cid:96) 1 -norm of the loss’s gradient with respect to each a j are bounded in terms of the bandwidth and Fourier (cid:96) 1 -norm of Y ( x ) as
[Equation]
We can apply Corollary 1 to show that not only the empirical risk uniformly converges to the population risk but also the gradient of the empirical risk will stay close to the gradient of the population risk.
Corollary 3.
Consider f a , W , b ( x
[Equation]
= (cid:80) dj
=1 a j sin( w Tj x + b j ) and squared error loss (cid:96) .
Then, given that (cid:107) X (cid:107) 2 ≤ C , for any δ > 0 with probability at least 1 − δ we have ∀ j, a , W , b s.t.
[Equation]
Proof.
The corollary is a direct result of Corollary (1) given (14) and (15).
Note that the generalization bound holds with probability 1 − δ for the derivative with respect to all a j ’s, since the bounds in (14) and (15) hold for all j ’s.
We emphasize that to prove Theorem 4 we need to analyze the risk function’s derivative only with respect to a j ’s. Hence, generalization of the empirical risk’s gradient with respect to a j ’s, which
[Figure]
[Figure]
is shown in the above corollary under certain assumptions, is sufﬁcient to apply an approximate version of Theorem 4 in section 8.6 to a local minimum ( a ∗ , W ∗ , b ∗ ) found by the gradient descent approach initialized at a low (cid:107) a (cid:107) 1 and (cid:107) W (cid:107) 2 , which satisﬁes the isolated components assumption.
We can conclude that with probability at least 1 − δ the (cid:96) 1 -norm of f a ∗ , W ∗ , b ∗ ’s Fourier transform outside the bandwidth B ( Y ) + B ( P X ) is bounded by O (cid:0) dV
[Equation]
(cid:1) , and also
[Equation]
Based on the above discussion, if a large-batch gradient descent method starts learning from f a , W , b with low (cid:107) a (cid:107) 1 and (cid:107) W (cid:107) 2 , ∞ and also we assume that the bandwidth and the Fourier (cid:96) 1 -norm for Y ( x ) are properly bounded, Theorem 4 combined with Corollary 1 will guarantee good generalization performance for the local minima found by the gradient descent method.
N UMERICAL E XPERIMENTS
For all experiments described in this section, we implemented and trained the two-layer neural network described in Figure 1a using TensorFlow 1.3.0.
We used SGD to train the model forepochs with an initial learning rate of 0.01.
The learning rate decayed slightly each epoch at a rate of 0.95 every 390 epochs.
We used h = 512 hidden units and a batch size of 128.
When working with CIFAR10 data, we preprocessed the data as described in (Zhang et al., 2016), resulting in each training sample having dimension d = 2352 .
Initial weights from the ﬁrst layer were sampled from
N (0 , 0 .
01 /d )
and initial weights from the second layer were sampled from N (0 , 0 .
01 /h ) .
7. SGD GRADUALLY LEARNS HIGHER F OURIER (cid:96) 1 - NORM , B ANDWIDTH HYPOTHESES
We ﬁrst numerically demonstrate that how Fourier (cid:96) 1 -norm and bandwidth both increases during training via SGD.
Motivated by the analysis from Section 6, we use the squared-error as our loss function and sine as our activation function.
Our samples consist of cats and airplanes from the
CIFAR10 dataset with the labels mapped to − 1 and 1 .
We use 5000 and 2000 samples from each category for training and test, respectively.
We arbitrarily chose two of the ten classes to accommodate our choice of loss function.
We evaluate the network’s performance for both random and true labels.
Figure 2a shows that without regularization, SGD learns to perfectly ﬁt both the true and random labels, which is consistent with the results from Zhang et al.
Additionally, the random labels are harder to learn, requiring more epochs before achieving a perfect ﬁt.
Figures 2b and 2c conﬁrm that both Fourier (cid:96) 1 -norm and bandwidth consistently increase with training, highlighting how SGD gradually ﬁnds more complex hypotheses in order to ﬁt the data.
Finally, we see in ﬁgures 2d and 2e how both Fourier (cid:96) 1 -norm and bandwidth increase with generalization risk (the difference between test mean squared-error (MSE) and training MSE) with almost perfect correlation.
This suggests that, as implied by the theory above, regularizing Fourier (cid:96) 1 -norm and bandwidth could improve generalizability of the ﬁnal learned model.
7. G ROUP PATH NORM REGULARIZATION FOR R E LU ACTIVATION
We regularize group path norm for ReLU activation as motivated by Theorem 3.
Although χ 2 -group path norm is not convex, it is differentiable and we can use it as an additive penalty and ﬁnd a local minimum via SGD.
Using the same experimental setup as from section 7.1, we swap sine for ReLU and test the network’s performance for both random and true labels.
Figure 3a conﬁrms that, like before, the network can ﬁt both true and random labels.
The generalization gap, however, remains large for random labels.
By regularizing the (cid:96) 2 -norm of all the weights, we see that the generalization gap closes for both the true labels and the random labels without compromising test accuracy signiﬁcantly (Figure 3b).
This result is further improved when we use the χ 2 -group path norm and (cid:96) 1 -path norm (Figure 3c and 3d), demonstrating that direct regularization of Fourier (cid:96) 1 -norm leads to better generalization.
We cross-validated the value of λ for each regularization technique, and we chose the λ that resulted in the smallest generalization gap with comparable validation performance.
To fairly compare different regularization strategies, we tested ﬁve lambda values for each strategy and then reported the performance on the test set for the lambda value that resulted in the best performance on the validation set.
We repeated the experiment using all 50000 CIFAR10 training samples (and 10000 test samples).
We included all 10 classes and switched to cross-entropy loss.
The results are shown in Figure 1b.
Again, we see that while all regularization techniques give similar test performance, the generalization gap is closed signiﬁcantly for the χ 2 -group path norm and (cid:96) 1 -path norm.
A PPENDIX 8. P ROOF OF T HEOREM
We use a high-dimensional grid in the Fourier domain to approximate the Fourier transform of a B -bandlimited function.
Consider the ball (cid:8) ξ : (cid:107) ξ (cid:107) 2 ≤ B (cid:9) .
Using the bounds on the covering number for (cid:96) 2 -norm, for any 0 < (cid:15) <
B we can ﬁnd a set of points { ξ j : 1 ≤ j ≤ (3 B/(cid:15) ) k } such that for any ξ with (cid:107) ξ (cid:107) 2 ≤ B , there exists some ξ j with (cid:107) ξ − ξ j (cid:107) 2 ≤ (cid:15) .
Let S j
=
ξ : (cid:107) ξ − ξ j (cid:107) 2 ≤ (cid:15) } for each 1 ≤ j ≤ (3 B/(cid:15) )
k .
Note that { ξ : (cid:107) ξ (cid:107) 2 ≤ B } ⊂ ∪ j S j .
We then deﬁne S (cid:48)
j = S j \ ∪ j − 1 t =1 S t to have a group of disjoint sets S (cid:48)
j covering { ξ : (cid:107) ξ (cid:107) 2 ≤ B } .
Since any f ∈ F is assumed to be B -bandlimited, for f ∈ F f (
[Equation]
(cid:90) (cid:98)
[Equation]
Then, for any f ∈ F = { f : B ( f ) ≤ B, (cid:107) (cid:98) f (cid:107) 1 ≤ V } we have
[Equation]
[Table]
[Equation]
[Table]
Here, (a) is a direct application of (17).
(b) holds as exp( ibz ) =
cos( bz ) + i sin( bz ) is b -Lipschitz as a function of z ∈ R for any real number b > 0 .
(c) holds because according to our deﬁnitions S (cid:48) j ⊆ S j and S j = { ξ : (cid:107)
ξ − ξ j (cid:107) 2 ≤ (cid:15) } .
Therefore, the following function space F (cid:15) can approximate any f ∈ F = (cid:8) f : B ( f ) ≤ B, (cid:107) (cid:98) f
(cid:107)
[Table] [Table]
[Table] [Table]
where | z | denotes the absolute value of complex number z ,
[Equation]
Then, F (cid:15) is the space of (cid:96) 1 -norm bounded linear functions in terms of the input vector (cid:2) exp(2 πi ξ Tj x ) (cid:3) j .
Now, we can apply a well-known bound (Shalev-Shwartz & Ben-David, 2014) on the Rademacher complexity of (cid:96) 1 -norm bounded linear space F lin , 1 = (cid:8) f
[Equation]
R k
→ R s.t
[Equation]
a
[Equation]
1 ≤ A (cid:9) as
[Equation]
Applying the above bound, we can bound the Rademacher complexity of F (cid:15) as
[Equation]
emp n
[Equation]
n .
Since for each f ∈ F there exists ˜ f ∈ F (cid:15) such that ∀ (cid:107) x (cid:107) 2 ≤ C : | f
[Equation]
x
[Equation]
− ˜ f
( x ) | ≤ 2 π(cid:15)CV ,
R
[Equation]
1 n n (cid:88) i =
σ i
[Equation]
( x i ) (cid:21) ≤ E σ (cid:20) sup ˜ f ∈F (cid:15) 1 n n (cid:88) i =
σ i ˜ f
( x i ) (cid:21)
[Equation]
Finally, combining (20) and (21) we obtain:
[Equation]
If we choose the value (cid:15) = 1 2 πn max i
[Equation]
x i (cid:107) 2 , then we get
[Equation]
where the last inequality follows from the fact that 1 ≤ k, n .
Therefore, the proof is complete.
8. P ROOF OF C OROLLARY
First, we prove the following lemma.
Lemma 2.
Given function f : R k
→ R and matrix W ∈ R k × k , we deﬁne g ( x ) =
f ( Wx ) .
Then,
• B ( g ) ≤ (cid:107) W (cid:107) 2 B ( f ) with (cid:107) W (cid:107) 2 denoting the spectral norm of W , • (cid:107) (cid:98) g
(cid:107)
= (cid:107) (cid:98) f
(cid:107)
Proof.
From the properties of the Fourier transform we know
[Equation]
Therefore, (cid:98) g ( W T ξ (cid:48) )
=
[Equation]
|
[Equation]
f (cid:0) ξ
[Equation]
(cid:1) and if (cid:107) ξ
[Equation]
(cid:107) 2 ≤ B ( f ) , then (cid:107) W T ξ
[Equation]
(cid:107) 2 ≤ (cid:107) W (cid:107) 2 B ( f ) gives an upperbound on B ( g ) .
Also, (cid:107)
[Equation]
g
[Equation]
It can be seen that this result remains valid even if W is not an invertible matrix, which will complete the proof for Corollary 2.
However, we continue proving Corollary 2 without using this fact.
As shown in the above lemma, Fourier (cid:96) 1 -norm and bandwidth are invariant to an orthonormal transformation W .
Given f i ( x ) = a i φ ( w Ti x ) , we deﬁne g
i ( x ) =
f i ( A i x ) where A i is an orthonormal matrix with w i as an eigenvector.
Note that (cid:107) (cid:98)
f i (cid:107)
= (cid:107) (cid:98) g
i (cid:107) 1 and B ( f i ) =
B ( g i ) .
However, g i ( x ) is a function of only one of the coordinates, which we can assume, without loss of generality, to be the ﬁrst coordinate.
Hence, g i ( x ) =
a i φ ( (cid:107) w i (cid:107) 2 x 1 ) for the ﬁrst coordinate x 1 , implying (cid:98) g
i ( ξ ) = a i (cid:107) w i (cid:107) 2 (cid:98) φ ( ξ 1 (cid:107) w i (cid:107) 2 ) .
δ 2 ( ξ 2 ) . . .
δ k ( ξ k ) where δ j is the Dirac delta function across the j th dimension.
Hence, we can use the above lemma in the 1-dimensional case to show (cid:107) (cid:98)
g i (cid:107) 1 = | a i |(cid:107) (cid:98) φ (cid:107) 1 and B ( g i ) = (cid:107) w i (cid:107) 2 B ( φ ) .
As a result, (cid:107) (cid:98) f
i (cid:107) 1 = | a i |(cid:107) (cid:98) φ (cid:107) 1 ,
B ( f i ) ≤ (cid:107) w i (cid:107) 2 B ( φ ) .
Hence, for f ( x ) = a T φ ( Wx + b ) =
(cid:80) di
=1 a i φ ( w Ti x + b i ) we have (cid:107) (cid:98) f (cid:107) 1 ≤ (cid:107)
[Equation]
The corollary is then a direct application of Theorem 2.
8. P ROOF OF T HEOREM
Given a ReLU-type activation function φ α ( z ) =
max { z, αz } ,
[Equation]
Since (cid:13)(cid:13) w (cid:107) w (cid:107) q
(cid:13)(cid:13)
q = 1 , if (cid:107) x (cid:107) p ≤ C , then (cid:12)(cid:12) ( w (cid:107) w (cid:107) q ) T x (cid:12)(cid:12) ≤ C and hence the input to φ α in the R.H.S. of (28) is always between − C and C .
Suppose that function ψ α satisﬁes ψ α ( z )
=
φ α ( z ) for z ∈ [ − C, C ] .
Then, based on the above discussion, we can bound the Rademacher complexity of F φ α by ﬁnding a bound on the Rademacher complexity of
F ψ α = (cid:8) f v , U ( x ) = v
T ψ α ( Ux ) : (cid:107) v (cid:107) 1 ≤ V, ∀ i : (cid:107) u
i (cid:107) q = 1 (cid:9) .
To ﬁnd a good candidate for ψ α , we use a symmetrization trick to deﬁne ψ α ( z ) =
  − αC if
[Equation]
C
[Equation]
if
[Equation]
Note that ψ α ( z ) =
(1 − α )
C h ( z − CC ) + 2 αC h ( z − C 2 C ) − αC
where h ( z ) = max { 0 , 1 − | z |} .
It can be seen that (cid:98) h ( ξ )
= (cid:0) sin( πξ ) πξ (cid:1) 2 which is real and positive everywhere.
Therefore, (cid:107) (cid:98) h (cid:107) 1 = h
= 1 which means that (cid:107) (cid:99) ψ α (cid:107) 1 ≤ C (1 + 2 α ) ≤ 3 C .
Since | (cid:98) h ( ξ ) | ≤ 1 ξ 2 , we have (cid:12)(cid:12)(cid:99) ψ α ( ξ ) (cid:12)(cid:12) ≤ 1 ξ 2 .
For B > 0 , we let the B -ﬁltered ψ α,B be a function with the following Fourier transform:
[Equation]
otherwise.
[Equation]
Then, since (cid:12)(cid:12)(cid:99) ψ α ( ξ ) (cid:12)(cid:12) ≤ 1 ξ 2 we have
[Equation]
Thus, for any B > 0 the deﬁned ψ α,B approximates φ α with a maximum error of 2 B uniformly over [ − C, C ] .
ψ α,B also satisﬁes (cid:107) (cid:91) ψ α,B (cid:107) 1 ≤ 3 C and B ( ψ α,B ) =
B .
Applying Corollary 2, we get ∀ B > 0 :
[Equation]
n
[Equation]
B .
[Equation] [Equation]
Here we can bound max i (cid:107) x
i (cid:107) 2 ≤ √ k max i (cid:107) x i (cid:107) ∞ ≤ √ kC , and choose B = n to get
[Equation]
which completes the proof.
8. P ROOF OF L EMMA
Note that
[Equation]
To show the last equality, we use the isolatedness assumption for w j , i.e. ∀ t (cid:54) =
j : min (cid:8) (cid:107) w t − w j (cid:107) 2 , (cid:107) w t + w j (cid:107) 2 (cid:9) >
B ( P X ) , and also (cid:107) w j (cid:107) 2 ≥ B
P X ) / 2 .
Then, for each t
[Equation]
Also
[Equation]
by applying the convolution property of Fourier transform we can show
[Equation]
Finally if ( a , W , b ) is a local minimum for the population risk, for all t ’s we have ∇ a t E P X (cid:2) (cid:96) (cid:0) f a , W , b ( X ) , Y ( X ) (cid:1)(cid:3) = 0 .
Therefore, due to the isolatedness assumption of w j we have
[Equation] [Equation]
8. P ROOF OF T HEOREM
Since the isolatedness assumption holds for all j ’s, by Lemma 1,
[Equation]
If (cid:107) w ∗ t (cid:107) 2 > B ( Y ) + B ( P X ) holds for some t , (37) implies
[Equation]
Hence, a ∗ t will be 0 , implying there will be no component in f a ∗ , W ∗ , b ∗ with (cid:107) w ∗ t (cid:107) 2 > B ( Y ) + B ( P X ) .
This discussion proves the ﬁrst part of Theorem, i.e. B ( f a ∗ , W ∗ , b ∗ ) ≤ B ( Y ) + B ( P X ) .
To show the second part, note that
[Equation]
Y ( ξ ) (cid:99) P
[Equation]
( w
[Equation]
ξ ) (cid:12)(cid:12)(cid:12)(cid:12) d ξ =d (cid:88) t = (cid:90) (cid:12)(cid:12)(cid:98) Y ( ξ ) (cid:12)(cid:12) (cid:12)(cid:12) (cid:99) P X (
[Equation]
ξ ) (cid:12)(cid:12)
[Equation]
ξ = 2 (cid:90) (cid:12)(cid:12)(cid:98) Y ( ξ ) (cid:12)(cid:12)(cid:20)
[Equation]
(cid:88) t = (cid:12)(cid:12) (cid:99) P X (
[Equation]
ξ ) (cid:12)(cid:12)(cid:21) d ξ ( b ) ≤ 2 d (cid:88) t = (cid:90) (cid:12)(cid:12)(cid:98) Y ( ξ ) (cid:12)(cid:12)
d ξ = 2 (cid:107) (cid:98) Y (cid:107) 1 .
Here, (a) comes from Lemma 1.
Also, since min (cid:8) (cid:107) w ∗ t − w ∗ r (cid:107) 2 , (cid:107) w ∗ t + w ∗ r (cid:107) 2 (cid:9) > 2 B ( P X ) is assumed for any t (cid:54) =
r , for any ξ at most one element in (cid:2) (cid:99) P X ( w ∗ t − ξ ) (cid:3) d t =1 can be nonzero.
Because if both (cid:99) P X ( w ∗ t − ξ ) and (cid:99) P X ( w ∗ r − ξ ) are nonzero for r (cid:54)
=
t , then (cid:107) w ∗ t − ξ (cid:107) ≤ B ( P X ) and also (cid:107) w ∗ r − ξ (cid:107) ≤ B ( P X ) which results in (cid:107) w ∗ t − w ∗ r (cid:107) ≤ 2 B ( P X ) which is a contradiction.
Hence,
[Equation]
which proves (b) and completes the proof.
8.5. A PPLYING T HEOREM 4 TO MULTIVARIATE G AUSSIAN X
Assume X ∼ N ( µ , σ
I k × k ) has a multivariate Gaussian distribution with mean µ and standard deviation σ .
Then, the Fourier transform ˆ P X has a Gaussian shape with mean 0 and standard deviation 1 /σ .
Hence, if for any i, j we have min (cid:8) (cid:107) w ∗ i − w ∗ j (cid:107) 2 , (cid:107) w ∗ i + w ∗ j (cid:107) 2 (cid:9) > 2 C/σ for some constant C , the approximation error term which should be added to the upperbound in Equation (39) is 2 (cid:107) (cid:98) Y (cid:107)
d exp( − C 2 / 2) .
Also, given any (cid:15) > 0 the
Fourier (cid:96) 1 norm outside the bandwidth O ( √ k log(1 /(cid:15) ) /σ ) is at most (cid:15) .
Therefore, Theorem 4 implies
• B ( f a ∗ , W ∗ , b ∗ ) ≤ B ( Y ) + O (cid:0) √ k/σ (cid:1) , • (cid:107) (cid:98) f
a ∗ , W ∗ , b ∗ (cid:107)
1 ≤ 2(1 + d exp( − C 2 / 2))
(cid:107) (cid:98) Y (cid:107) 1 .
8. A PPROXIMATE VERSION OF T HEOREM
Here we show an approximate version of Theorem 4 which applies to approximate population local minima.
Theorem 5.
Consider minimizing the population risk (11) .
Consider an approximate local minimum ( a ∗ , W ∗ , b ∗ )
where |∇ a j E (cid:2) (cid:96) (cid:0) f a ∗ , W ∗ , b ∗ ( X ) , Y ( X ) (cid:1)(cid:1)(cid:3) | ≤ (cid:15) for all j ’s. If for any two different i, j we have min (cid:8) (cid:107) w ∗ i − w ∗ j (cid:107) 2 , (cid:107) w ∗ i + w ∗ j (cid:107) 2 (cid:9) > 2 B ( P X ) , then the Fourier (cid:96) 1 -norm of f a ∗ , W ∗ , b ∗ outside the bandwidth B ( Y ) +
B ( P
X ) is bounded by d (cid:15) and (cid:107) (cid:98) f a ∗ , W ∗ , b ∗ (cid:107) 1 ≤ 2 (cid:107) (cid:98) Y (cid:107)
+ d (cid:15).
Proof.
Since the isolated components condition holds, we can apply Lemma 1’s proof to show under the above assumptions
[Equation]
Then, a simple modiﬁcation of Theorem 4’s proof according to the above inequality proves the above theorem.
8. P ROOF OF T HEOREM 4 WITHOUT THE ISOLATED COMPONENTS ASSUMPTION
What happens if a component w i is not isolated from the other components which has been assumed in Theorem 4?
As a simplifying assumption, we assume that b = 0 and (cid:99) P X is real.
We can write ∀
[Equation]
Here S w j ’s, which are centered around w j ’s, are disjoint sets covering the bandwidth region for (cid:98) Y , i.e. { ξ : (cid:107) ξ (cid:107) 2 ≤ B ( Y ) } ⊆
(cid:83) j S w j .
Note that in (41) we have approximated the convolution integral as
[Equation]
Letting the gradient element in (41) be zero for all a i ’s at a local minimum ( a ∗ , W ∗ ) of the population risk, the following approximation holds in general case:
[Equation]
[Equation]
Here, the matrix (cid:2) (cid:99) P X ( w i − w j ) (cid:3) 1 ≤ i,j ≤ d is positive-deﬁnite and hence invertible, because (cid:99) P X is the Fourier transform of P X and due to Bochner’s theorem a positive-deﬁnite kernel function.
Therefore, the system of linear equations (cid:2)
[Equation]