O VER - PARAMETERIZATION I MPROVES G ENERALIZATION IN THE XOR D ETECTION P ROBLEM
Anonymous authors
Paper under double-blind review
A BSTRACT
Empirical evidence suggests that neural networks with ReLU activations generalize better with over-parameterization.
However, there is currently no theoretical analysis that explains this observation.
In this work, we study a simpliﬁed learning task with over-parameterized convolutional networks that empirically exhibits the same qualitative phenomenon.
For this setting, we provide a theoretical analysis of the optimization and generalization performance of gradient descent.
Speciﬁcally, we prove data-dependent sample complexity bounds which show that overparameterization improves the generalization performance of gradient descent.
I NTRODUCTION
Most successful deep learning models use a number of parameters that is larger than the number of parameters that are needed to get zero-training error.
This is typically referred to as overparameterization .
Indeed, it can be argued that over-parameterization is one of the key techniques that has led to the remarkable success of neural networks.
However, there is still no theoretical account for its effectiveness.
One very intriguing observation in this context is that over-parameterized networks with ReLU activations, which are trained with gradient based methods, often exhibit better generalization error than smaller networks (Neyshabur et al., 2014; 2018; Novak et al., 2018).
This somewhat counterintuitive observation suggests that ﬁrst-order methods which are trained on over-parameterized networks have an inductive bias towards solutions with better generalization performance.
Understanding this inductive bias is a necessary step towards a full understanding of neural networks in practice. Providing theoretical guarantees for this phenomenon is extremely challenging due to two main reasons.
First, to show a generalization gap, one needs to prove that large networks have better sample complexity than smaller ones. However, current generalization bounds that are based on complexity measures do not offer such guarantees. Second, analyzing the dynamics of ﬁrst-order methods on networks with ReLU activations is a major challenge.
Indeed, there do not exist optimization guarantees even for simple learning tasks such as the classic XOR problem in two dimensions.
To advance this issue, we focus on a particular learning setting that captures key properties of the over-parameterization phenomenon.
We consider a high-dimensional extension of the XOR problem, which we refer to as the “XOR Detection problem (XORD)”.
The XORD is a pattern recognition task where the goal is to learn a function which classiﬁes binary vectors according to whether they contain a two-dimensional binary XOR pattern (i.e., (1 , 1) or ( − 1 , − 1) ).
This problem contains the classic XOR problem as a special case when the vectors are two dimensional.
We consider learning this function with gradient descent trained on an over-parameterized convolutional neural network (i.e., with multiple channels) with ReLU activations and three layers: convolutional, max pooling and fully connected.
As can be seen in Fig.
1, over-parameterization improves generalization in this problem as well.
Therefore it serves as a good test-bed for understanding the role of over-parameterization.
In this work we provide an analysis of optimization and generalization of gradient descent for
XORD.
We show that for various input distributions, ranges of accuracy and conﬁdence parameters, sufﬁciently over-parameterized networks have better sample complexity than a small network which can realize the ground truth classiﬁer.
To the best of our knowledge, this is the ﬁrst example which shows that over-paramaterization can provably improve generalization for a neural network with ReLU activations.
Our analysis provides a clear distinction between the inductive bias of gradient descent for overparameterized and small networks.
It reveals that over-parameterized networks are biased towards global minima that detect more patterns in the data than global minima found by small networks.
Thus, even though both networks succeed in optimization, the larger one has better generalization performance. We provide experiments which show that the same phenomenon occurs in a more general setting with more patterns in the data and non-binary input.
We further show that our analysis can predict the behavior of over-parameterized networks trained on MNIST and guide a compression scheme for over-parameterized networks with a mild loss in accuracy (Sec. 6).
R ELATED W ORK
In recent years there have been many works on theoretical aspects of deep learning.
We will refer to those that are most relevant to this work.
First, we note that we are not aware of any work that shows that generalization performance provably improves with over-parameterization.
This distinguishes our work from all previous works.
Several works study convolutional networks with ReLU activations and their properties (Du et al., 2017a;b; Brutzkus & Globerson, 2017).
All of these works consider convolutional networks with a single channel.
Daniely (2017) and Li & Liang (2018) provide guarantees for SGD in general settings.
However, their analysis holds for over-parameterized networks with an extremely large number of neurons that are not used in practice (e.g., the number of neurons is a very large polynomial of certain problem parameters).
Furthermore, we consider a 3-layer convolutional network with max-pooling which is not studied in these works.
Soltanolkotabi et al.
(2018), Du & Lee (2018) and Li et al.
(2017) study the role of overparameterization in the case of quadratic activation functions.
Brutzkus et al.
(2018) provide generalization guarantees for over-parameterized networks with Leaky ReLU activations on linearly separable data.
Neyshabur et al.
(2018) prove generalization bounds for neural networks.
However, these bounds are empirically vacuous for over-parameterized networks and they do not prove that networks found by optimization algorithms give low generalization bounds.
P ROBLEM F ORMULATION
We begin with some notations and deﬁnitions.
Let d ≥ 4 be an integer.
We consider a classiﬁcation problem in the space {± 1 } 2 d .
Namely, the space of vectors of 2 d coordinates where each coordinate can be +1 or − 1 .
Given a vector x ∈ {± 1 } 2 d , we consider its partition into d sets of two coordinates as follows x
= ( x 1 , ..., x d ) where x i ∈ {± 1 } 2 .
We refer to
each such x
i as a pattern in x .
Neural Architecture: We consider learning with the following three-layer neural net model.
The ﬁrst layer is a convolutional layer with non-overlapping ﬁlters and multiple channels, the second layer is max pooling and the third layer is a fully connected layer with 2 k hidden neurons and weights ﬁxed to values ± 1 .
Formally, for an input x = ( x 1 , ..., x d ) ∈ R 2 d where x i ∈ R 2 , the output of the network is given by:
[Equation]
where W ∈ R 2 k × 2 is the weight matrix whose rows are the w ( i ) vectors followed by the u ( i ) vectors, and σ ( x ) = max { 0 , x } is the ReLU activation applied element-wise.
See Figure 4 for an illustration of this architecture.
Remark 3.1.
Because there are only 4 different patterns, the network is limited in terms of the number of different rules it can implement.
Speciﬁcally, it is easy to show that its VC dimension is at most 15 (see Sec. 10).
Despite this limited expressive power, there is a generalization gap between small and large networks in this setting, as can be seen in Figure 1, and in our analysis below.
Data Generating Distribution: Next we deﬁne the classiﬁcation rule we will focus on.
Let
P XOR correspond to the following two patterns: P XOR = { (1 , 1) , ( − 1 , − 1) } .
Deﬁne the classiﬁcation rule:
[Equation]
Namely, f ∗ detects whether a pattern in P XOR appears in the input.
In what follows, we refer to
P XOR as the set of positive patterns and {± 1 } 2 \ P XOR as the set of negative patterns.
Let D be a distribution over X × {± 1 } such that for all ( x , y ) ∼ D we have y = f ∗ ( x ) .
We say that a point ( x , y ) is positive if y = 1 and negative otherwise.
Let D + be the marginal distribution over {± 1 } 2 d of positive points and D − be the marginal distribution of negative points.
In the following deﬁnition we introduce the notion of diverse points, which will play a key role in our analysis.
Deﬁnition 3.2 ( Diverse Points).
We say that a positive point ( x , 1) is diverse if for all z ∈ {± 1 }there exists 1 ≤ i ≤ d
such that x i =
z .
We say that a negative point ( x , − 1) is diverse if for all z ∈ {± 1 } 2 \ P XOR there exists 1 ≤ i ≤ d
such that x i =
z .
For φ ∈ {− , + } deﬁne p φ to be the probability that x is diverse with respect to D φ .
For example, if both D + and D − are uniform, then by the inclusion-exclusion principle it follows that p + = 1 − 4 · 3 d − 6 · 2 d +4 4 d and p − = 1 − 1 2 d − 1 . For each set of binary patterns A ⊆ {± 1 } 2 deﬁne p
A to be the probability to sample a point which contains all patterns in A and no patterns in A c (the complement of A ).
Let A 1 = { 2 } , A 2 = { 4 } ,
A 3 = { 2 , 4 , 1 } and A 4 = { 2 , 4 , 3 } .The
following quantity will be useful in our analysis:
[Equation] Learning Setup:
Our analysis will focus on the problem of learning f ∗ from training data with a three layer neural net model.
The learning algorithm will be gradient descent, randomly initialized.
As in any learning task in practice, f ∗ is unknown to the training algorithm.
Our goal is to analyze the performance of gradient descent when given data that is labeled with f ∗ .
We assume that we are given a training set S = S + ∪ S − ⊆ {± 1 } 2 d × {± 1 } 2 where S + consists of m IID points drawn from D + and S − consists of m IID points drawn from D − .
Importantly, we note that the function f ∗ can be realized by the above network with k = 2 .
Indeed, the network N deﬁned by the ﬁlters w (1) =
(3 , 3) , w (2) =
( − 3 , − 3) , u (1) =
( − 1 , 1) , u (2) =
(1 , − 1) satisﬁes sign ( N ( x ))
=
f ∗ ( x ) for all x ∈ {± 1 } 2 d .
It can be seen that for k = 1 , f ∗ cannot be realized.
Therefore, any k > 2 is an over-parameterized setting.
Training Algorithm: We will use gradient descent to optimize the following hinge-loss function.
[Equation]
for γ ≥ 1 .
We assume that gradient descent runs with a constant learning rate η
and the weights are randomly initiliazed with IID Gaussian weights with mean 0 and standard deviation σ g .
Furthermore, only the weights of the ﬁrst layer, the convolutional ﬁlters, are trained.
We will need the following notation.
Let W t be the weight matrix in iteration t of gradient descent.
For 1 ≤ i ≤ k , denote by w ( i ) t ∈ R 2 the i th convolutional ﬁlter at iteration t .
Similarly, for 1 ≤ i ≤ k we deﬁne u ( i ) t ∈ R 2 to be the k
+ i convolutional ﬁlter at iteration t .
We assume that each w ( i ) 0 and u ( i ) 0 is initialized as a Gaussian random variable where the entries are IID and distributed as N (0 , σ 2 g ) .
In each iteration, gradient descent performs the update W t +1 =
W t − η ∂(cid:96)∂W ( W t ) .
M AIN R ESULT
In this section we state our main result that demonstrates the generalization gap between overparameterized networks and networks with k = 2 .
Deﬁne the generalization error to be the difference between the 0-1 test error and the 0-1 training error.
For any (cid:15) , δ and training algorithm let m ( (cid:15), δ ) be the sample complexity of a training algorithm, namely, the number of minimal samples the algorithm needs to get at most (cid:15) generalization error with probability at least 1 − δ .
We consider running gradient descent in two cases, when k ≥ 120 and k = 2 .
In the next section we exactly deﬁne under which set of parameters gradient descent runs, e.g., which constant learning rates.
Fix parameters p + and p − of a distribution D and denote by c <
10 − 10 a negligible constant.
Assume that gradient descent is given a sample of points drawn from D + and D − .
We denote the sample complexity of gradient descent in the cases k ≥ 120 and k = 2 , by m 1 and m 2 , respectively.
The following result shows a data dependent generalization gap (recall the deﬁnition of p ∗ in Eq. 3).
Theorem 4.1.
Let D be a distribution with paramaters p + , p − and p ∗ .
Let δ ≥ 1 − p + p − (1 − c − 16 e − 8 ) and 0 ≤ (cid:15) <
p ∗ .
Then m 1 ( (cid:15), δ ) ≤ 2 whereas m 2 ( (cid:15), δ ) ≥ 2log ( 48 δ 33(1 − c ) ) log( p + p − ) .
The proof follows from Theorem 5.2 and Theorem 5.3 which we state in the next section.
The proof is given in Sec. 8.8.
One surprising fact of this theorem is that m 1 (0 , δ ) ≤ 2 .
Indeed, our analysis shows that for an over-parameterized network and for sufﬁciently large p + and p − , one diverse positive point and one diverse negative sufﬁce for gradient descent to learn f ∗ with high probability.
We note that even in this case, the dynamics of gradient descent is highly complex.
This is due to the randomness of the initialization and to the fact that there are multiple weight ﬁlters in the network, each with different dynamics.
See Sec.
5 for further details.
We will illustrate the guarantee of Theorem 4.1 with several numerical examples.
In all of the examples we assume that for the distribution D , the probability to sample a positive point is 12 and p ∗ =
min (cid:110) 1 − p + 4 , 1 − p − 4 (cid:111)
(it is easy to constuct such distributions).
In the ﬁrst example, we assume that p + = p − = 0 .
98 and δ = 1 − 0 .
98 2 (1 − c − 16 e − 8 ) ≤ 0 . 05 .
In this case we get that for any 0 ≤ (cid:15) < 0 .
005 , m 1 ( (cid:15), δ ) ≤ 2 whereas m 2 ( (cid:15), δ ) ≥ 129 .
For the second example consider the case where p + = p − = 0 .
It follows that for δ = 0 .
16 and any 0 ≤ (cid:15) < 0 .
02 it holds that m 1 ( (cid:15), δ ) ≤ 2 and m 2 ( (cid:15), δ ) ≥ 17 .
For (cid:15) = 0 and any δ > 0 , by setting p + and p − to be sufﬁciently close to 1, we can get an arbitrarily large gap between m 1 ( (cid:15), δ ) and m 2 ( (cid:15), δ ) .
In contrast, for sufﬁciently small p + and p − , e.g., in which p + , p − ≤ 0 . 7 , our bound does not guarantee a generalization gap.
P ROOF S KETCH AND I NSIGHTS
In this section we sketch the proof of Theorem 4.1.
The theorem follows from two theorems: Theorem 5.2 for over-parameterized networks and Theorem 5.3 for networks with k = 2 .
We formally show this in Sec. 8.8.
In Sec.
5.1 we state Theorem 5.2 and outline its proof.
In Sec.
5.2 we state
Theorem 5.3 and shortly outline its proof.
Finally, for completeness, in Sec.
9 we also provide a convergence guarantee for the XOR problem with inputs in {± 1 } , which in our setting is the case of d = 1 .
In what follows, we will need the following formal deﬁnition for a detection of a pattern by a network.
Deﬁnition 5.1.
Let c d ≥ 0 be a constant.
For each positive pattern v p deﬁne D v p = (cid:80) ki =1 σ (cid:0) w ( i ) · v p (cid:1) and for each negative pattern v n deﬁne D v n =
(cid:80) ki
=1 σ (cid:0) u
( i ) · v n (cid:1) .
We say that a pattern v (positive or negative) is detected by the network N W with conﬁdence c d if D v > c d .
The above deﬁnition captures a desired property of a network, namely, that its ﬁlters which are connected with a positive coefﬁcient in the last layer, have high correlation with the positive patterns and analogously for the remaining ﬁlters and negative patterns.
We note however, that the condition in which a network detects all patterns is not equivalent to realizing the ground truth f ∗ .
The former can hold without the latter and vice versa.
Theorem 5.2 and Theorem 5.3 together imply a clear characterization of the different inductive biases of gradient descent in the case of small ( k = 2 ) and over-parameterized networks.
The characterization is that over-parameterized networks are biased towards global minima that detect all patterns in the data, whereas small networks with k = 2 are biased towards global minima that do not detect all patterns (see Deﬁnition 5.1).
In Sec.
8.5 we show this empirically in the XORD problem and in a generalization of the XORD problem.
In the following sections we will need several notations.
Deﬁne x 1 = (1 , 1) , x 2 = (1 , − 1) , x 3 = ( − 1 , − 1) , x 4 = ( − 1 , 1) to be the four possible patterns in the data and the following sets:
[Equation]
We denote by x + a positive diverse point and x − a negative diverse point.
Deﬁne the following sum:
[Equation]
Finally, in all of the results in this section we will denote by c < 10 − 10 a negligible constant.
5. S AMPLE C OMPLEXITY U PPER B OUND FOR O VER - PARAMETERIZED N ETWORKS
The main result in this section is given by the following theorem.
Theorem 5.2.
Let S = S
+ ∪ S − be a training set as in Sec. 3.
Assume that gradient descent runs with parameters η = c η k where c η ≤ 1 410 , σ g ≤ c η 16 k 3 2 , k ≥ 120 and γ ≥ 8 .
Then, with probability at least ( p + p − ) m (cid:0) 1 − c − 16 e − 8 (cid:1) after running gradient descent for T ≥ 28( γ +1+8 c η ) c
η iterations, it converges to a global minimum which satisﬁes:
1. sign ( N W T ( x ))
=
f ∗ ( x ) for all x ∈ {± 1 } 2 d . 2.
Let α ( k ) =
k 4 +2 √ k k 4 − 2 √ k .
Then for c d ≤ 1 − 5 cη 4 α ( k )+1 , all patterns are detected with conﬁdence c d .
This result shows that given a small training set size, and sufﬁciently large p + and p − , overparameterized networks converge to a global minimum which realizes the classiﬁer f ∗ with high probability and in a constant number of iterations.
Furthermore, this global minimum detects all patterns in the data with conﬁdence that increases with over-parameterization.
The full proof of Theorem 5.2 is given in Sec. 8.6.
We will now sketch its proof.
With probability at least ( p + p − ) m all training points are diverse and we will condition on this event.
From Sec.
10 we can assume WLOG that the training set consists of one positive diverse point x
+ and one negative diverse point x − (since the network will have the same output on all same-label diverse points).
We note that empirically over-parameterization improves generalization even when the training set contains non-diverse points (see Fig.
1 and Sec. 8.2).
Now, to understand the dynamics of gradient descent it is crucial to understand the dynamics of the sets in Eq.
This follows since the gradient updates are expressed via these sets.
Concretely, let j ∈ W + t
( i 1 ) ∩ W − t ( i 2 )
then the gradient update is given as follows:
[Equation]
Similarly, for j ∈ U + t ( i 1 ) ∩ U − t
( i 2 ) the gradient update is given by:
[Equation]
Furthermore, the values of N W ( x + ) and N W ( x − ) depend on these sets and their corresponding weight vectors, via sums of the form S + t , deﬁned above.
The proof consists of a careful analysis of the dynamics of the sets in Eq.
5 and their corresponding weight vectors.
For example, one result of this analysis is that for all t ≥ 1 and i ∈ { 1 , 3 } we have
W + t ( i ) =
W +0 ( i ) and the size of W + t
( i ) is at least k 2 − 2 √ k with high probability.
There are two key technical observations that we apply in this analysis.
First, with a small initialization and with high probability, for all 1 ≤ j ≤ k and 1 ≤ i ≤ 4 it holds that (cid:12)(cid:12)(cid:12) w ( j ) 0 · x i (cid:12)(cid:12)(cid:12) ≤ η 4 and (cid:12)(cid:12)(cid:12) u ( j ) 0 · x i (cid:12)(cid:12)(cid:12) ≤ η 4 .
This allows us to keep track of the dynamics of the sets in Eq.
5 more easily.
For example, by this observation it follows that if for some j ∗ ∈ W + t (2) it holds that j ∗ ∈ W + t +1 (4) , then for all j such that j ∈ W + t (2) it holds that j ∈ W + t +1 (4) .
Hence, we can reason about the dynamics of several ﬁlters all at once, instead of each one separately.
Second, by concentration of measure we can estimate the sizes of the sets in Eq.
5 at iteration t = 0 .
Combining this with results of the kind W + t
( i ) =
W +0 ( i ) for all t , we can understand the dynamics of these sets throughout the optimization process.
The theorem consists of optimization and generalization guarantees.
For the optimization guarantee we show that gradient descent converges to a global minimum.
To show this, the idea is to characterize the dynamics of S + t using the characterization of the sets in Eq.
5 and their corresponding weight vectors.
We show that as long as gradient descent did not converge to a global minimum,
S + t cannot decrease in any iteration and it is upper bounded by a constant.
Furthermore, we show that there cannot be too many consecutive iterations in which S + t does not increase.
Therefore, after sufﬁciently many iterations gradient descent will converge to a global minimum.
We will now outline the proof of the generalization guarantee.
Denote the network learned by gradient descent by N W T .
First, we show that the network classiﬁes all positive points correctly.
Deﬁne the following sums for all 1 ≤ i ≤ 4 :
[Equation]
First we notice that for all positive z we have N W T ( z ) (cid:39) min { X + T (1) , X + T (3) } − Y + T (2) − Y + T (4) .
Then by the fact that N W T ( x + ) ≥ γ at the global minimum, we can show that X + T (1) + X + T (3) is sufﬁciently large.
As mentioned previously, by concentration of measure, (cid:12)(cid:12) W +0 ( i ) (cid:12)(cid:12) is sufﬁciently large.
Then by a symmetry argument we show that this implies that both X + T (1) and X + T (3) are sufﬁciently large. This shows that patterns x 1 and x 3 are detected.
Finally, we show that Y + T (2) + Y + T (4) is not too large due to an upper bound on − N W T ( x − ) .
Hence, we can show that each positive point is classiﬁed correctly.
The proof that all negative points are classiﬁed correctly and patterns x 2 and x 4 are detected is similar but slightly more technical.
We refer the reader to Sec. 8. for further details.
5. S AMPLE C OMPLEXITY L OWER B OUND FOR S MALL N ETWORKS ( k = 2 )
The following theorem provides generalization lower bounds of global minima in the case that
k
=and in a slightly more general setting than the one given in Theorem 5.2.
Theorem 5.3.
Let S = S
+ ∪ S − be a training set as in Sec. 3.
Assume that gradient descent runs with parameters
η = c η k where c η ≤ 141 , σ g ≤ c η 16 k 32 , k = 2 and γ ≥ 1 .
Then the following holds
With probability at least ( p + p − ) m (1 − c ) 3348 , gradient descent converges to a global minimum that has non-zero test error.
Furthermore, for c d ≥ 2 c η , there exists at least one pattern which is not detected by the global minimum with conﬁdence c d . 2.
The non-zero test error above is at least p ∗ .
The theorem shows that for a training set that is not too large and given sufﬁciently large p + and p − , with constant probability, gradient descent will converge to a global minimum that is not the classiﬁer f ∗ .
Furthermore, this global minimum does not detect at least one pattern.
The proof of the theorem is given in Sec.
We will now provide a short outline of the proof.
Let w (1) T , w (2) T , u (1) T and u (2) T be the ﬁlters of the network at the iteration T in which gradient descent converges to a global minimum.
The proof shows that gradient descent will not learn f ∗ if one of the following conditions is met: a)
[Equation]
Then by using a symmetry argument which is based on the symmetry of the initialization and the training data it can be shown that one of the above conditions is met with high constant probability.
Finally, it can be shown that if one of these conditions hold, then at least one pattern is not detected.
E XPERIMENTS
We perform several experiments that corroborate our theoretical ﬁndings.
In Sec.
8.5 we empirically demonstrate our insights on the inductive bias of gradient descent.
In Sec.
6.2 we evaluate a model compression scheme implied by our results, and demonstrate its success on the MNIST dataset.
6. P ATTERN D ETECTION
In this section we perform experiments to examine the insights from our analysis on the inductive bias of gradient descent.
Namely, that over-parameterized networks are biased towards global minima that detect more patterns in the data than global minima found by smaller networks.
We check this both on the XORD problem which contains 4 possible patterns in the data and on an instance of an extension of the XORD problem, that we refer to as the Orthonormal Basis Detection (OBD) problem, which contains 60 patterns in the data.
In Sec.
8.5 we provide details on the experimental setups.
[Figure] Due to space considerations, we will not formally deﬁne the OBD problem in this section.
We refer the reader to Sec.
8.5 for a formal deﬁnition.
Informally, The OBD problem is a natural extension of the XORD problem that contains more possible patterns in the data and allows the dimension of the ﬁlters of the convolutional network to be larger.
The patterns correspond to a set of orthonormal vectors and their negations.
The ground truth classiﬁer in this problem can be realized by a convolutional network with 4 channels.
In Fig.
2 we show experiments which conﬁrm that in the OBD problem as well, overparameterization improves generalization.
We further show the number of patterns detected in %0 training error solutions for different number of channels, in both the XORD and OBD problems.
It can be clearly seen that for both problems, over-parameterized networks are biased towards %0 training error solutions that detect more patterns, as predicted by the theoretical results.
6. N ETWORK C OMPRESSION
By inspecting the proof of Theorem 5.2, one can see that the dynamics of the ﬁlters of an overparameterized network are such that they either have low norm, or they have large norm and they point to the direction of one of the patterns (see, e.g., Lemma 8.4 and Lemma 8.6).
This suggests that by clustering the ﬁlters of a trained over-parameterized network to a small number of clusters, one can create a signiﬁcantly smaller network which contains all of the detectors that are needed for good generalization performance.
Then, by training the last layer of the network, it can converge to a good solution.
Following this insight, we tested this procedure on the MNIST data set and a 3 layer convolutional network with convolutional layer with multiple channels and 3 × 3 kernels, max pooling layer and fully connected layer.
We trained an over-parameterized network withchannels, clustered its ﬁlters with k-means into 4 clusters and used the cluster centers as initialization for a small network with 4 channels.
Then we trained only the fully connected layer of the small network.
In Fig.
3 we show that for various training set sizes, the performance of the small network improves signiﬁcantly with the new initialization and nearly matches the performance of the overparameterized network.
C ONCLUSIONS
In this paper we consider a simpliﬁed learning task on binary vectors and show that overparameterization can provably improve generalization performance of a 3-layer convolutional network trained with gradient descent.
Our analysis reveals that in the XORD problem overparameterized networks are biased towards global minima which detect more relevant patterns in the data. While we prove this only for the XORD problem and under the assumption that the training set contains diverse points, our experiments clearly show that a similar phenomenon occurs in other settings as well. We show that this is the case for XORD with non-diverse points (Figure 1) and in the more general OBD problem which contains 60 patterns in the data and is not restricted to binary inputs (Figure 2).
Furthermore, our experiments on MNIST hint that this is the case in MNIST as well (Figure 3).By clustering the detected patterns of the large network we could achieve better accuracy with a small network. This suggests that the larger network detects more patterns with gradient descent even though its effective size is close to that of a small network.
We believe that these insights and our detailed analysis can guide future work for showing similar results in more complex tasks and provide better understanding of this phenomenon.
It would also be interesting to further study the implications of such results on model compression and on improving training algorithms.
A PPENDIX 8. N ETWORK A RCHITECTURE IN THE XORD P ROBLEM 8. E XPERIMENTAL S ETUPS 8.2. E XPERIMENT IN F IGURE
We tested the generalization performance in the setup of Section3.
We considered networks with number of channels 4,6,8,20,50,100 and 200.
The distribution in this setting has p + = 0 .
5 and p − = [-
 .and-]0 .
9 and the training sets are of size 12 (6 positive, 6 negative).
Note that in this case the training set contains non-diverse points with high probability.
The ground truth network can be realized by a network with 4 channels.
For each number of channels we trained a convolutional network 100 times and averaged the results.
In each run we sampled a new training set and new initialization of the weights according to a gaussian distribution with mean 0 and standard deviation 0.00001.
For each number of channels c , we ran gradient descent with learning rate 0 .
04 c and stopped it if it did not improve the cost for 20 consecutive iterations or if it reached 30000 iterations.
The last iteration was taken for the calculations.
We plot both average test error over all 100 runs and [Figure] average test error only over the runs that ended at 0% train error.
In this case, for each number of channels 4,6,8,20,50,100,200 the number of runs in which gradient descent converged to a 0% train error solution is 62, 79, 94, 100, 100, 100, 100, respectively.
8. H INGE L OSS C ONFIDENCE
Figure 5 shows that setting γ = 5 gives better performance than setting γ = 1 in the XORD problem.
The setting is similar to the setting of Section 8.2.1.
Each point is an average test error of 100 runs.
8. E XPERIMENTS FOR S ECTION 5.
Theorem 5.3 holds for any η ≤ 182 and σ ≤ 1 1900 .
Because the result is a lower bound, it is desirable to understand the behaviour of gradient descent for values outside these ranges.
In Figure 6 we empirically show that for values outside these ranges, there is a generalization gap between gradient descent for k = 2 and gradient descent for larger k .
8. E XPERIMENTAL S ETUP IN S ECTION
We will ﬁrst formally deﬁne the OBD problem.
Fix an even dimension parameter d 1 ≥ 2 .
In this problem, we assume there is an orthonormal basis B = { v 1 , ..., v d 1 } of R d 1 .
Divide B into two equally sized sets B 1 and B 2 , each of size d 1 2 .
Now deﬁne the set of positive patterns to be P
= { v | v ∈ B 1 } ∪ {− v | v ∈ B 1 } and negative patterns to be N = { v | v ∈ B 2 } ∪ {− v | v ∈ B 2 } .
Let P OBD
=
P ∪ N .
For d 2 > 0 , we assume the input domain is X ⊆
R
d 1 d 2 and each x ∈ X is a vector such that x = ( x 1 , ..., x d 2 ) where each x i ∈ P OBD .
We deﬁne the ground truth classiﬁer f OBD : X → {± 1 } such that f OBD ( x ) = 1 if and only there exists at least one x i such that x i ∈ P .
Notice that for d 1 = 2 and by normalizing the four vectors in {± 1 } 2 to have unit norm, we get the XORD problem.
We note that the positive patterns in the XORD problem are deﬁned to be
P XOR and the negative patterns are {± 1 } 2 \ P XOR .
Let D be a distribution over X 2 d × {± 1 } such that for all ( x , y ) ∼ D , y = f OBD ( x ) .
As in the
XORD problem we deﬁne the distributions D + and D − .
We consider the following learning task which is the same as the task for the XORD problem.
We assume that we are given a training set S =
S + ∪ S − ⊆ {± 1 } d 1 d 2 × {± 1 } where S + consists of m IID points drawn from D + and S − consists of m IID points drawn from D − .
The goal is to train a neural network with randomly initialized gradient descent on S and obtain a network N : R d 1 d 2 → R such that sign ( N ( x ))
=
f OBD ( x ) for all x ∈ {± 1 } d 1 d 2 .
We consider the same network as in the XORD problem (Eq. 1), but now the ﬁlters of the convolution layer are d 1 -dimensional.
Formally, for an input x = ( x 1 , ..., x d ) ∈ X the output of the network is given by
[Equation]
where W ∈ R 2 k × d 1 is the weight matrix which contains in the ﬁrst k rows the vectors w ( i
) ∈
R d 1 , in the next k rows the vectors u ( i )
∈ R
d 1 and σ ( x ) = max { 0 , x } is the ReLU activation applied element-wise.
We performed experiments in the case that d 1 = 30 , i.e., in which there arepossible patterns.
In Figure 2a, for each number of channels we trained a convolutional network given in Eq.
9 with gradient descent for 100 runs and averaged the results.
The we sampled 25 positive points andnegative points in the following manner.
For each positive point we sampled with probability 0 .one of the numbers [4,6,8,10] twice with replacement.
Denote these numbers by m 1 and m 2 .
Then we sampled m 1 different positive patterns and m 2 different negative patterns.
Then we ﬁlled a 60 d 1dimensional vectors with all of these patterns.
A similar procedure was used to sample a negative point.
We considered networks with number of channels 4,6,8,20,100 and 200 and 500.
Note that the ground truth network can be realized by a network with 4 channels.
For each number of channels we trained a convolutional network 100 times and averaged the results.
For each number of channels c , we ran gradient descent with learning rate 0 .
2 c and stopped it if it did not improve the cost forconsecutive iterations or if it had 0% training error for 200 consecutive iterations or if it reached 30000 iterations.
The last iteration was taken for the calculations. We plot both average test error over all 100 runs and average test error only over the runs that ended at 0% train error.
For each number of channels 4,6,8,20,100,200,500 the number of runs in which gradient descent converged to a 0% train error solution is 96, 99, 100, 100, 100, 100, 100, respectively.
For each 0% train error solution we recorded the number of patterns detected {+with c d = 0 . 
+} according to the Deﬁnition 5.1 (generalized to the OBD problem).
In the XORD problem we recorded similarly the number of patterns detected in experiments which are identical to the experiments in Section 8.2.1, except that in this case p + = p − = 0 . 98 .
8. P ROOF OF T HEOREM 5.
We will ﬁrst need a few notations.
Deﬁne x 1 = (1 , 1) , x 2 = (1 , − 1) , x 3 = ( − 1 , − 1) , x 4 = ( − 1 , 1) and the following sets:
[Equation]
We can use these deﬁnitions to express more easily the gradient updates.
Concretely, let j ∈
W + t ( i 1 ) ∩ W − t
( i 2 ) then the gradient update is given as follows:
[Equation]
Similarly, for j ∈ U + t ( i 1 ) ∩ U − t
( i 2 ) the gradient update is given by:
[Equation]
We denote by x + a positive diverse point and x − a negative diverse point.
Deﬁne the following sums for φ ∈ { + , −} :
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
Note that R + t
=
R − t since for z ∈ { x + , x − } there exists i 1 , i
such that z
i 1 = x 2 , z i 2 = x 4 .
By the conditions of the theorem, with probability at least ( p + p − ) m all the points in the training set are diverse.
From now on we will condition on this event.
Furthermore, without loss of generality, we can assume that the training set consists of one diverse point x + and one negative points x − .
This follows since the network and its gradient have the same value for two different positive diverse points and two different negative points.
Therefore, this holds for the loss function deﬁned in Eq.as well.
We will now proceed to prove the theorem.
In Section 8.6.1 we prove results on the ﬁlters at initialization.
In Section 8.6.2 we prove several auxiliary lemmas.
In Section 8.6.3 we prove upper bounds on S − t , P + t and P − t for all iterations t .
In Section 8.6.4 we characterize the dynamics of S + t and in Section [-8.5.
-]{+8.6.
+}
we prove an upper bound on it together with upper bounds on N W t ( x + ) and − N W t ( x − ) for all iterations t .
We provide an optimization guarantee for gradient descent in Section 8.6.6.
We prove generalization guarantees for the points in the positive class and negative class in Section 8.6.7 and Section 8.6.8, respectively.
We complete the proof of the theorem in Section 8.6.9.
8.6. I NITIALIZATION G UARANTEES
Lemma 8.1.
With probability at least 1 − 4 e − 8 , it holds that
[Equation]
and
[Equation]
Proof.
Without loss of generality consider (cid:12)(cid:12) W +0 (1) ∪ W +0 (3) (cid:12)(cid:12) .
Since P (cid:2) j ∈ W +0 (1) ∪ W +0 (3) (cid:3) = 12 , we get by Hoeffding’s inequality
[Equation]
The result now follows by the union bound.
Lemma 8.2.
With probability ≥ 1 − √ 2 k √ πe 8 k , for all 1 ≤ j ≤ k and 1 ≤ i ≤ 4 it holds that (cid:12)(cid:12)(cid:12) w ( j ) 0 · x i (cid:12)(cid:12)(cid:12) ≤ η 4 and (cid:12)(cid:12)(cid:12) u ( j [Equation]
Proof.
Let Z be a random variable distributed as N (0 , σ 2 ) .
Then by Proposition 2.1.2 in Vershynin (2017), we have
[Equation]
Therefore, for all 1 ≤ j ≤ k and 1 ≤ i ≤ 4 ,
[Equation]
and
[Equation]
The result follows by applying a union bound over all 2 k weight vectors and the four points x i , 1 ≤ i ≤ 4 .
From now on we assume that the highly probable event in Lemma 8.2 holds.
Lemma 8.3.
N W t ( x + ) < 1 and − N W t ( x − ) < 1 for 0 ≤ t ≤ 2 .
Proof.
By Lemma 8.2 we have
N W 0 ( x + ) =
k (cid:88) i =
[Equation]
≤ ηk 4 < γ and similarly − N W 0 ( x − ) < 1 .
Therefore, by Eq. 10 and Eq.
11 we get:
For i ∈ { 1 , 3 } , l ∈ { 2 , 4 } , j ∈ W + 0 ( i ) ∩ W − 0 ( l ) , it holds that w ( j ) 1 = w ( j ) 0 − η x l + η x i .
For i ∈ { 2 , 4 } and j ∈ W +0 ( i ) , it holds that w ( j ) 1 = w ( j ) 0 .
For i ∈ { 1 , 3 } , l ∈ { 2 , 4 } , j ∈ U
+0 ( i ) ∩ U − 0 ( l ) , it holds that u ( j ) 1 = u ( j ) 0 − η
x
i + η x l . 4.
For i ∈ { 2 , 4 } and j ∈ U +0 ( i ) , it holds that u ( j ) 2 = u ( j ) 0 .
Applying Lemma 8.2 again and using the fact that η ≤ 18 k
we have N W 1 ( x + ) < γ and − N W 1 ( x − ) < 1 .
Therefore we get,
For i ∈ { 1 , 3 } , l ∈ { 2 , 4 } , j ∈ W +0 ( i ) ∩ W − 0 ( l ) , it holds that w ( j ) 2 = w ( j )
+ 2 η x i .
For i ∈ { 2 , 4 } and j ∈ W +0 ( i ) , it holds that w ( j ) 2 = w ( j ) 0 .
For i ∈ { 1 , 3 } , l ∈ { 2 , 4 } , j ∈ U
+0 ( i ) ∩ U − 0 ( l ) , it holds that u ( j ) 2 = u ( j ) 0 − η
x
i + η x l . 4.
For i ∈ { 2 , 4 } and j ∈ U +0 ( i ) , it holds that u ( j ) 2 = u ( j ) 0 .
As before, by Lemma 8.2 we have N W 2 ( x + ) < γ and − N W 2 ( x − ) < 1 .
A UXILIARY L EMMAS
Lemma 8.4.
For all t ≥ 1 we have W + t
( i ) =
W +0 ( i ) for i ∈ { 1 , 3 } .
Proof.
We will ﬁrst prove that W +0 ( i ) ⊆
W + t
( i ) for all t ≥ 1 .
To prove this, we will show by induction on t ≥ 1 , that for all j ∈ W +0 ( i ) ∩ W +0 ( l ) , where l ∈ { 2 , 4 } the following holds: j ∈ W + t (
i w ( j ) t · x l = w ( j ) 0 · x l − η or w ( j ) t · x l = w (0) t · x l . 3. w ( j ) t · x i > η .
The claim holds for t = 1 by the proof of Lemma 8.3.
Assume it holds for t = T .
By the induction hypothesis there exists an l (cid:48) ∈ { 2 , 4 } such that j ∈ W + T ( i ) ∩ W − T ( l (cid:48) ) .
By Eq.
10 we have,
[Equation]
[Equation]
where a ∈ { 0 , 1 } and b ∈ {− 1 , 0 } .
If w ( j ) T · x l = w ( j ) 0 · x l then l (cid:48)
=
l and either w ( j ) T +1 · x l = w ( j ) 0 · x l if b = 0 or w ( j ) T +1 · x l = w ( j ) 0 · x l − η if b = − 1 .
Otherwise, assume that w ( j ) T · x l = w ( j ) 0 · x l − η .
By Lemma 8.2 we have 0 < w ( j ) 0 · x l < η 4 .
Therefore − η < w ( j ) T · x l < 0 and l (cid:48) (cid:54) =
l .
It follows that either w ( j ) T +1 · x l = w ( j ) 0 · x
l − η if b = 0 or w ( j ) T +1 · x l = w ( j ) 0 · x l if b = − 1 .
In both cases, we have (cid:12)(cid:12)(cid:12) w ( j ) T +1 · x l (cid:12)(cid:12)(cid:12) <
η .
Furthermore, by
Eq.
12, w ( j ) T +1 · x i ≥ w ( j ) T · x i > η .
Hence, arg max 1 ≤ l ≤ 4 w ( j ) T +1 · x l = i which by deﬁnition implies that j ∈ W + T +1 ( i ) .
This concludes the proof by induction which shows that W +0 ( i ) ⊆
W + t
( i ) for all t ≥ 1 .
In order to prove the lemma, it sufﬁces to show that W +0 (2) ∪ W +0 (4) ⊆
W + t (2) ∪ W + t (4) .
This follows since (cid:83) 4 i
=1 W + t
( i ) =
{ 1 , 2 , ..., k } .
We will show by induction on t ≥ 1 , that for all j ∈
W +0 (2) ∪ W +0 (4) , the following holds:
j ∈ W + t (2)
∩ W + t (4) .
w ( j ) t = w ( j )
+ m x 2 for m ∈ Z .
The claim holds for t = 1 by the proof of Lemma 8.3.
Assume it holds for t = T .
By the induction hypothesis j ∈ W + T (2) ∩ W + T (4) .
Assume without loss of generality that j ∈ W + T (2) .
This implies that j ∈ W − T (2) as well.
Therefore, by Eq.
10 we have
[Equation]
where a ∈ { 0 , 1 } and b ∈ { 0 , − 1 } .
By the induction hypothesis, w ( j ) T +1 =
w ( j )
+ m x 2 for m ∈ Z .
If a = 1 or b = 0 we have for i ∈ { 1 , 3 } ,
[Equation]
where the ﬁrst inequality follows since j ∈ W + T (2) and the second by Eq. 13.
This implies that j ∈ W + T +1 (2) ∩ W + T +1 (4) .
Otherwise, assume that a = 0 and b = − 1 .
By Lemma 8.2 we have w ( j ) 0 · x 2 < η 4 .
Since j ∈ W + T (2) , it follows by the induction hypothesis that w ( j ) T = w ( j )
+ m x 2 , where m ∈ Z and m ≥ 0 .
To see this, note that if m < 0 , then w ( j ) T · x 2 < 0 and j / ∈ W + T (2) , which is a contradiction.
Let i ∈ { 1 , 3 } .
If m = 0 , then w ( j ) T +1 = w ( j ) 0 − x 2 , w ( j ) T +1 · x 4 > η 2 and w ( j ) T +1 · x i = w ( j ) 0 · x i < ηby Lemma 8.2.
Therefore, j ∈ W + T +1 (4) .
Otherwise, if m > 0 , then w ( j ) T +1 · x 2 ≥ w ( j ) 0 · x 2 > w ( j ) 0 · x i = w ( j ) T +1 · x i .
Hence, j ∈ W + T +1 (2) , which concludes the proof.
Lemma 8.5.
For all t ≥
we have U +
(2) ∪ U +0 (4) ⊆
U + t (2) ∪ U + t (4) .
Proof.
Let j ∈ U +0 (2) ∪ U +0 (4) .
It sufﬁces to prove that u ( j ) t = u ( j )
+ α t η x 2 for α t ∈ Z .
This follows since the inequalities (cid:12)(cid:12)(cid:12) u ( j ) 0 · x 1 (cid:12)(cid:12)(cid:12) <
(cid:12)(cid:12)(cid:12) u ( j ) 0 · x 2 (cid:12)(cid:12)(cid:12) ≤ η 4 imply that in this case j ∈ U + t (2) ∪
U + t (4) .
Assume by contradiction that there exist an iteration t for which u ( j ) t = u ( j )
+ α t η x 2 + β t η x i where β t ∈ {− 1 , 1 } , α t ∈ Z , i ∈ { 1 , 3 } and u ( j ) t − 1 = u
( j )
+ α t − 1 η x 2 where α t − 1 ∈ Z .
9 Since the coefﬁcient of x i changed in iteration t , we have j ∈ U + t − 1 (1) ∪ U + t − 1 (3) .
However, this contradicts the claim above which shows that if u ( j ) t − 1 = u ( j )
+ α t − 1 η x 2 , then j ∈ U + t − 1 (2) ∪
U + t − 1 (4) .
Lemma 8.6.
Let i ∈ { 1 , 3 } and l ∈ { 2 , 4 } .
For all t ≥ 0 , if j ∈ U +0 ( i ) ∩ U − 0 ( l ) , then there exists a t ∈ { 0 , − 1 } , b t ∈ N such that u ( j ) t
[Equation]
Proof.
First note that by Eq.
11 we generally have u ( j ) t = u ( j )
+ αη x i + βη x l where α, β ∈ Z .
Since (cid:12)(cid:12)(cid:12) u ( j ) 0 · x 1 (cid:12)(cid:12)(cid:12) ≤ η 4 , by the gradient update in Eq.
11 it holds that a t ∈ { 0 , − 1 } .
Indeed, a 0 =and by the gradient update
if a t − 1 = 0 or a t − 1 = − 1 then a t ∈ {− 1 , 0 } .
Assume by contradiction that there exists an iteration t > 0 such that b t = − 1 and b t − 1 = 0 .
Note that by Eq.
11 this can only occur if j ∈ U + t − 1 ( l ) .
We have u ( j ) t − 1 = u ( j )
+ a t − 1 η x i where a t − 1 ∈ { 0 , − 1 } .
Observe that (cid:12)(cid:12)(cid:12) u ( j ) t − 1 · x i (cid:12)(cid:12)(cid:12) ≥ (cid:12)(cid:12)(cid:12) u
( j ) 0 ·
x i (cid:12)(cid:12)(cid:12) by the fact that (cid:12)(cid:12)(cid:12) u ( j ) 0 · x i (cid:12)(cid:12)(cid:12) ≤ η 4 .
Since u ( j ) 0 · x i > u ( j ) 0 · x l = u ( j ) t − 1 · x l we have j ∈ U + t − 1 (1) ∪ U + t − 1 (3) , a contradiction.
Lemma 8.7.
Let
X + t = (cid:88)
j
[Equation]
[Equation]
(cid:104)
[Equation]
w [Equation] [Equation] (cid:16) w ( i
[Equation]
· x [Equation] (cid:17)(cid:111)(cid:105) and Y + t = (cid:88)
j
[Equation]
(3) (cid:104)
[Equation]
(cid:17) [Equation] (cid:16)
[Equation]
( i [Equation] · x [Equation]
(cid:17)(cid:111)(cid:105)
Then for all t , X + t − X [Equation]
Proof.
We will prove the claim by induction on t .
For t = 0 this clearly holds.
Assume it holds for t = T .
Let j 1 ∈ W + T (1) and j 2 ∈ W + T (3) .
By Eq.
10, the gradient updates of the corresponding weight vector are given as follows:
[Equation]
and
[Equation]
where a ∈ { 0 , 1 } and b 1 , b 2 ∈ {− 1 , 0 , 1 } .
By Lemma 8.4, j 1 ∈ W + T +1 (1) and j 2 ∈ W + T +1 (3) .
Therefore,
[Equation]
and
[Equation]
By Lemma 8 .
4 we have (cid:12)(cid:12) W + t [Equation]
[Equation] and (cid:12)(cid:12) W + t (3) (cid:12)(cid:12) =
(cid:12)(cid:12) W +0 (3) (cid:12)(cid:12) for all t .
It follows that
X + T +
[Equation]
− X +
[Equation]
where the second equality follows by the induction hypothesis.
This proves the claim.
8.6. B OUNDING P + t , P − t AND S − t
Lemma 8.8.
The following holds
S − t ≤ (cid:12)(cid:12) W + t (1) ∪ W + t (3) (cid:12)(cid:12) η for all t ≥ 1 . 2.
P + t ≤ (cid:12)(cid:12) U + t (1) ∪ U + t (3) (cid:12)(cid:12) η for all t ≥ 1 . 3.
P − t ≤ (cid:12)(cid:12) U + t (1) ∪ U + t (3) (cid:12)(cid:12) η for all t ≥ 1 .
Proof.
In Lemma 8.4 we showed that for all t ≥ 0 and j ∈ W + t (1) ∪ W + t (3) it holds that (cid:12)(cid:12)(cid:12) w ( j ) t · x 2 (cid:12)(cid:12)(cid:12) ≤ η .
This proves the ﬁrst claim.
The second claim follows similarly.
Without loss of generality, let j ∈ U + t (1) .
By Lemma 8.5 it holds that U + t (cid:48) (1) ⊆
U +0 (1) ∪ U +0 (3) for all t (cid:48) ≤ t .
Therefore, by Lemma 8.6 we have (cid:12)(cid:12)(cid:12) u ( j ) t x 1 (cid:12)(cid:12)(cid:12) <
η , from which the claim follows.
For the third claim, without loss of generality, assume by contradiction that for j ∈ U + t (1) it holds that (cid:12)(cid:12)(cid:12) u ( j ) t · x 2 (cid:12)(cid:12)(cid:12) >
η .
Since (cid:12)(cid:12)(cid:12) u ( j ) t · x 1 (cid:12)(cid:12)(cid:12) <
η by Lemma 8.6, it follows that j ∈ U + t (2) ∪ U + t (4) , a contradiction.
Therefore, (cid:12)(cid:12)(cid:12) u ( j ) t · x 2 (cid:12)(cid:12)(cid:12) ≤ η for all j ∈ U + t (1) ∪ U + t (3) , from which the claim follows.
8.6. D YNAMICS OF S + t
Lemma 8.9.
The following holds:
If N W t ( x + ) < γ and − N W t ( x − ) < 1 , then S + t +1 = S + t + η (cid:12)(cid:12) W + t (1) ∪ W + t (3) (cid:12)(cid:12) .
If N W t ( x + ) ≥ γ and − N W t ( x − ) < 1 , then S + t +1 = S + t . 3.
If N W t ( x + ) < γ and − N W t ( x − ) ≥ 1 , then S + t +1 = S + t + η (cid:12)(cid:12) W + t (1) ∪ W + t (3) (cid:12)(cid:12) .
Proof.
The equality follows since for each i ∈ { 1 , 3 } , l ∈ { 2 , 4 } and j ∈ W + t ( i ) ∩ W − t ( l ) we have w ( j ) t +1 = w ( j ) t + η x i
− η x l and W + t +1 (1) ∪ W + t + 3) = W
+ t
∪ W + t 3) by Lemma 8.4.
In this case for each i ∈ { 1 , 3 } , l ∈ { 2 , 4 } and j ∈ W + t ( i )
∩ W − t ( l ) we have w ( j ) t
+1 = w ( j ) t − η x l and W + t +1 (1) ∪ W + t +1 (3)
=
W + t (1) ∪ W + t (3) by Lemma 8.4.
This equality follows since for each i ∈ { 1 , 3 } , l ∈ { 2 , 4 } and j ∈ W + t ( i ) ∩ W − t ( l ) we have w ( j ) t +1 = w ( j ) t + η x i and W + t +1 (1) ∪ W + t +1 (3)
=
W + t (1) ∪ W + t (3) by Lemma 8.4.
U PPER B OUNDS ON N W t ( x + ) , − N W t ( x − ) AND S + t
Lemma 8.10.
Assume that N W t ( x + ) ≥ γ and − N W t ( x − ) <
1 for T ≤ t < T + b where b ≥ 2 .
Then N W T + b ( x + ) ≤
[Equation]
Proof.
Deﬁne R + t
= Y + t − Z
+ t where
[Equation]
and
[Equation]
Let l ∈ { 2 , 4 } , t = T and j ∈ U + t
+1 ( l ) .
Then, either j ∈ U + t (2) ∪ U + t (4) or j ∈ U + t (1) ∪ U + t (3) .
In the ﬁrst case, u ( j ) t +1 = u ( j ) t + η x l .
Note that this implies that U + t (2) ∪ U + t (4) ⊆
U + t +1 (2) ∪ U + t +1 (4) (since x l will remain the maximal direction).
Therefore, (cid:88)
[Equation]
In the second case, where we have j ∈ U + t (1) ∪ U + t (3) , it holds that u ( j ) t +1 =
u ( j ) t + η x l , j ∈ U − t ( l ) and u ( j ) t +1 · x l > η .
Furthermore, by Lemma 8.6, u ( j ) t · x i < η for i ∈ { 1 , 3 } .
Note that by Lemma 8.6, any j 1 ∈ U + t (1) ∪ U + t (3) satisﬁes j 1 ∈ U
+ t +1 (2) ∪ U
+ t +1 (4) .
By all these observations, we have
[Equation]
−
[Equation]
[Table] [Equation] [Table] ≥(15)
By Eq.
14 and Eq.
15, it follows that, Z + t +1 + P + t +1 ≥ Z + t +1 ≥
Z + t + P + t + η (cid:12)(cid:12) U
+ t (2) ∪ U + t (4) (cid:12)(cid:12) .
By induction we have Z + t + b + P + t + b ≥ Z + t + P + t
+ (cid:80) b − 1 i =
η (cid:12)(cid:12) U + t
+ i (2) ∪ U + t
+ i (4) (cid:12)(cid:12) .
By Lemma 8.
for any 1 ≤ i ≤ b − 1 we have (cid:12)(cid:12) U
+ t
+ i (2) ∪ U + t
+ i (4) (cid:12)(cid:12) = { 1 , ..., k } .
Therefore, Z + t + b + P + t + b ≥
Z + t + P + t + ( b − 1)
c η .
Now, assume that j ∈ W + T ( l ) for l ∈ { 2 , 4 } .
Then w ( j ) T +1 = w ( j ) T − η x l .
Thus either max
[Equation]
w
[Equation]
[Equation] [Equation] =
[Equation]
in the case that j ∈ W + T +1 ( l ) , or max
[Equation]
if j / ∈ W + T +1 ( l ) .
Applying these observations b times, we see that Y + T + b − Y + T is at most η (cid:12)(cid:12) W + T + b (2) ∪ W + T + b (4) (cid:12)(cid:12) =
η (cid:12)(cid:12) W +0 (2) ∪ W +0 (4) (cid:12)(cid:12) where the equality follows by Lemma 8.4.
By Lemma 8.9, we have S + T + b =
S + T .
Hence, we can conclude that
[Equation]
Lemma 8.11.
Assume that N W t ( x + ) < γ and − N W t ( x − ) ≥ 1 for T ≤ t < T + b where b ≥ 1 .
Then − N W T + b ( x − ) ≤ − N W T ( x − )
− bη (cid:12)(cid:12) W +0 (2) ∪ W +0 (4) (cid:12)(cid:12) + c η .
Proof.
Deﬁne
Y − t = (cid:88)
j ∈ W + t (2) ∪ W
[Equation]
t (4) (cid:104) max (cid:110) σ (cid:16) w ( i ) · x +1 (cid:17) , ..., σ (cid:16) w ( i ) · x + d (cid:17)(cid:111)(cid:105) and
Z − t = k (cid:88)
j = (cid:104) max (cid:110) σ (cid:16) u
( j ) ·
x +1 (cid:17) , ..., σ (cid:16) u
[Equation]
j
[Equation]
(cid:17)(cid:111)(cid:105)
First note that by Lemma 8.4 we have W + t +1 (2) ∪ W + t +1 (4)
=
W + t (2) ∪ W + t (4) .
Next, for any l ∈ { 2 , 4 } and j ∈ W + t ( l ) we have w ( j ) t +1 = w ( j ) t + η x l .
Therefore,
[Equation]
where the second equality follows by Lemma 8.4.
Assume that j ∈ U + T ( l ) for l ∈ { 1 , 3 } .
Then u ( j ) T
+1 =
u ( j ) T − η x l and max
[Equation]
[Equation]
To see this, note that by Lemma 8.6 and Lemma 8.5 it holds that u ( j ) T
= u ( j )
+ a T η x l where a T ∈ {− 1 , 0 } .
Hence, u ( j ) T +1 =
u ( j )
+ a T +1 η x l where a T +1 ∈ {− 1 , 0 } .
Since (cid:12)(cid:12)(cid:12) u ( j ) 0 · x 2 (cid:12)(cid:12)(cid:12) <
η 4 it follows that u ( j ) T +1 · x 2 = u ( j ) T · x 2 = u ( j ) 0 · x 2 and thus Eq.
16 holds.
Now assume that j ∈ U + T ( l ) for l ∈ { 2 , 4 } .
Then
[Equation]
if l ∈ { 2 , 4 } and j ∈ U + T +1 ( l ) , or [Equation] ≤
[Equation]
if l ∈ { 2 , 4 } and j / ∈ U + T +1 ( l ) .
Applying these observations b times, we see that Z − T + b − Z − T is at most η (cid:12)(cid:12)
U + T + b (2) ∪ U + T + b (4) (cid:12)(cid:12) .
Furthermore, for j ∈ W + T ( l ) , l ∈ { 1 , 3 } , it holds that w ( j ) T +1 = w ( j ) T + η x l .
Therefore
[Equation]
and since W + T +1 (1) ∪ W + T +1 (3)
= W + T (1) ∪ W + T (3) by Lemma 8.4, we get S − T + b = S − T .
Hence, we can conclude that
[Equation]
Lemma 8.12.
For all t , N W t ( x + ) ≤ γ + 3 c η , − N W t ( x − ) ≤ 1 + 3 c η and S + t ≤ γ + 1 + 8 c η .
Proof.
The claim holds for t = 0 .
Consider an iteration T .
If N W T ( x + ) <
γ then N W T +1 ( x + ) ≤
N W T ( x + )+2 ηk ≤
γ +2 c η .
Now assume that N W t ( x + ) ≥ γ for T ≤ t ≤ T + b and N W T − 1 ( x + ) < γ .
By Lemma 8.10, it holds that N W T + b ( x + ) ≤ N W T ( x + ) + ηk ≤ N W T ( x + ) +
c η ≤ γ + 3 c η , where the last inequality follows from the previous observation.
Hence, N W t ( x + ) ≤ γ + 3 c η for all t .
The proof of the second claim follows similarly.
It holds that − N W T +1 ( x − ) <
1 + 2 c η if − N W T ( x − ) < 1 .
Otherwise if − N W t ( x − ) ≥ 1 for T ≤ t ≤ T + b and − N W T − 1 ( x − ) <then − N W T + b ( x − ) ≤ 1 + 3 c η by Lemma 8.11.
The third claim holds by the following identities and bounds N W T ( x + ) − N W T ( x − ) =
S + T − P + T +
P − T − S − T , P − T ≥ 0 , (cid:12)(cid:12) P + T (cid:12)(cid:12) ≤ c η , (cid:12)(cid:12) S − T (cid:12)(cid:12) ≤ c η and N W T ( x + ) − N W T ( x − ) ≤ γ + 1 + 6 c η by the previous claims.
8.6. O PTIMIZATION
We are now ready to prove a global optimality guarantee for gradient descent.
Proposition 8.13.
Let k > 16 and γ ≥ 1 .
With probabaility at least 1 − √ 2 k √ πe 8 k − 4 e − 8 , after
T = 7( γ +1+8 c
η ) ( k 2 − 2 √ k ) η iterations, gradient descent converges to a global minimum.
Proof.
First note that with probability at least 1 − √ 2 k √ πe 8 k − 4 e − 8 the claims of Lemma 8.1 and
Lemma 8.2 hold.
Now, if gradient descent has not reached a global minimum at iteration t then either N W t ( x + ) < γ or − N W t ( x − ) < 1 .
If − N W t ( x + ) < γ then by Lemma 8.9 it holds that
[Equation]
where the last inequality follows by Lemma 8.1.
If N W t ( x + ) ≥ γ and − N W t ( x − ) < 1 we have S
+ t +1 =
S + t by Lemma 8.9.
However, by Lemma 8.10, it follows that after 5 consecutive iterations t < t (cid:48) <
t + 6 in which N W t (cid:48) ( x + ) ≥ γ and − N W t (cid:48) ( x − ) < 1 , we have N W t +6 ( x + ) < γ .
To see this, ﬁrst note that for all t , N W t ( x + ) ≤ γ +3 c η by Lemma 8.12.
Then, by Lemma 8.10 we have
[Equation]
≤ γ + 3 c η − 5 c η + c η < γ where the second inequality follows by Lemma 8.1 and the last inequality by the assumption on k .
Assume by contradiction that GD has not converged to a global minimum after T = 7( γ +1+8 c
η
( k 2 − 2 √ k ) η iterations.
Then, by the above observations, and the fact that S +0 > 0 with probability 1 , we have
[Equation]
However, this contradicts Lemma 8.12.
8.6. G ENERALIZATION ON P OSITIVE C LASS
We will ﬁrst need the following three lemmas.
Lemma 8.14.
With probability at least 1 − 4 e − 8 , it holds that
[Equation]
and
[Equation]
Proof.
The proof is similar to the proof of Lemma 8.1.
Lemma 8.15.
Assume that gradient descent converged to a global minimum at iteration T .
Then there exists an iteration T 2 < T for which S + t ≥ γ + 1 − 3 c η for all t ≥ T 2 and for all t < T 2 , − N W t ( x − ) < 1 .
Proof.
Assume that for all 0 ≤ t ≤ T 1 it holds that N W t ( x + ) < γ and − N W t ( x − ) < 1 .
By continuing the calculation of Lemma 8.3 we have the following:
For i ∈ { 1 , 3 } , l ∈ { 2 , 4 } , j ∈ W +0 ( i ) ∩ W − 0 ( l ) , it holds that w ( j ) T 1 = w ( j )
+ T 1 η x i − 12 (1 − ( − 1) T 1 )
η x l . 2.
For i ∈ { 2 , 4 } and j ∈ W +0 ( i ) , it holds that w ( j ) T 1 = w ( j ) 0 .
For i ∈ { 1 , 3 } , l ∈ { 2 , 4 } , j ∈ U
+0 ( i ) ∩ U − 0 ( l ) , it holds that u ( j ) T 1 = u ( j ) 0 − η x
i + η x l . 4.
For i ∈ { 2 , 4 } and j ∈ U +0 ( i ) , it holds that u ( j ) T 1 = u ( j ) 0 .
Therefore, there exists an iteration T 1 such that N W T 1 ( x + ) ≥ γ and − N W T 1 ( x − ) < 1 and for all t < T 1 , N W t ( x + ) < γ and − N W t ( x − ) < 1 .
Let T 2 ≤ T be the ﬁrst iteration such that − N W T 2 ( x − ) ≥ 1 .
We claim that for all T 1 ≤ t ≤ T 2 we have N W T 1 ( x + ) ≥ γ − 2 c η .
It sufﬁces to show that for all T 1 ≤ t < T 2 the following holds:
If N W t ( x + ) ≥ γ then N W t +1 ( x + ) ≥ γ − 2 c η . 2.
If N W t ( x + ) < γ then N W t +1 ( x + ) ≥ N W t ( x + ) .
The ﬁrst claim follows since at any iteration N W t ( x + ) can decrease by at most 2 ηk = 2 c η .
For the second claim, let t (cid:48) <
t be the latest iteration such that N W t (cid:48)
( x + ) ≥ γ .
Then at iteration t (cid:48) it holds that − N W t (cid:48) ( x − ) <
1 and N W t (cid:48) ( x + ) ≥ γ .
Therefore, for all i ∈ { 1 , 3 } , l ∈ { 2 , 4 } and j ∈ U
+0 ( i ) ∩ U +0 ( l ) it holds that u ( j ) t (cid:48)
+1 =
u ( j ) t (cid:48)
+ η x l .
Hence, by Lemma 8.5 and Lemma 8.6 it holds that U + t (cid:48) +1 (1) ∪ U
+ t (cid:48) +1 (3) = ∅ .
Therefore, by the gradient update in Eq. 11, for all 1 ≤ j ≤ k , and all t (cid:48) < t (cid:48)(cid:48) ≤
t we have u ( j ) t (cid:48)(cid:48)
+
= u ( j ) t (cid:48)(cid:48) , which implies that N W t (cid:48)(cid:48) +
( x + ) ≥ N W t (cid:48)(cid:48) ( x + ) .
For t (cid:48)(cid:48)
=
t we get N W t +1 ( x + ) ≥ N W t ( x + ) .
The above argument shows that N W T 2 ( x + ) ≥ γ − 2 c η and − N W T 2 ( x − ) ≥ 1 .
Since N W T 2 ( x + ) −
N W T 2 ( x − )
=
S + T 2 − P + T 2 + P − T 2 − S − T 2 , P − T 2 , S − T 2 ≥ 0 and (cid:12)(cid:12)
P − T 2 (cid:12)(cid:12) ≤ c η it follows that S + T 2 ≥ γ +1 − 3 c η .
Finally, by Lemma 8.9 we have S + t ≥ γ + 1 − 3 c η for all t ≥ T 2 .
Lemma 8.16.
Let
[Equation]
and
[Equation]
Assume that k ≥ 64 and gradient descent converged to a global minimum at iteration T .
Then,
X + T ≤ 34 c η and Y + T ≤ 1 + 38 c η .
Proof.
Notice that by the gradient update in Eq. 10 and Lemma 8.2, X + t can be strictly larger than max (cid:8) X + t − 1 , η (cid:12) (cid:12) W + t (2) ∪ W + t (4) (cid:12) (cid:12) (cid:9) only if N W t − 1 ( x + ) < γ and − N W t − 1 ( x − ) ≥ 1 .
Furthermore, in this case X + t − X + t − 1 = η (cid:12) (cid:12) W + t (2) ∪ W + t (4) (cid:12)(cid:12) .
By Lemma 8.9, S + t increases in this case by η (cid:12)(cid:12) W + t (1) ∪ W + t (3) (cid:12)(cid:12) .
We know by Lemma 8.15 that there exists T 2 < T such that S + T 2 ≥ γ +1 − 3 c
η
and that N W t ( x + ) < γ and − N W t ( x − ) ≥ 1 only for t > T 2 .
Since S + t ≤ γ + 1 + 8 c η for all t by Lemma 8.12, there can only be at most
[Equation]
η | W + T (1) ∪ W + T (3) | iterations in which N W t ( x + ) < γ and − N W t ( x − ) ≥ 1 .
It follows that
[Equation]
where the second inequality follows by Lemma 8.1 and the third inequality by the assumption on k .
At convergence we have N W T ( x − ) =
S − T + X + T − Y + T − P − T ≥ − 1 − 3 c η by Lemma 8.12 (recall that R − t =
R + t
= X
+ t − Y + t ).
Furthermore, P − T ≥ 0 and by Lemma 8.8 we have S − T ≤ c η .
Therefore, we get Y + T ≤ 1 + 38 c η .
We are now ready to prove the main result of this section.
Proposition 8.17.
Deﬁne β ( γ ) =
γ − 40 14 c η 39 c η +1 .
Assume that γ ≥ 2 and k ≥ 64 (cid:16) β ( γ )
+1 β
( γ ) − 1 (cid:17) 2 .
Then with probability at least 1 − √ 2 k √ πe 8 k − 8 e − 8 , gradient descent converges to a global minimum which classiﬁes all positive points correctly.
Proof.
With probability at least 1 − √ 128 k √ πe k 2 − 8 e − 8 Proposition 8.13, and Lemma 8.14 hold.
It sufﬁces to show generalization on positive points.
Assume that gradient descent converged to a global minimum at iteration T .
Let ( z , 1) be a positive point.
Then there exists z i ∈ { (1 , 1) , ( − 1 , − 1) } .
Assume without loss of generality that z
i
=
( − 1 , − 1)
= x 3 .
Deﬁne
[Equation]
for i ∈ [4] .
Notice that
N
[Equation]
Since N W T ( x + ) ≥ γ , − N W T ( x − ) ≥ 1 , (cid:12)(cid:12) P − T (cid:12)(cid:12) ≤ c η by Lemma 8.8 and P + T , S − T ≥ 0 , we obtain X + T (
[Equation]
+ X
+
[Equation]
3) ≥
[Equation]
1 − c η (18)
Furthermore, by Lemma 8.7 we have
[Equation]
(19) and by Lemma 8.14,
[Equation]
Let α ( k ) =
k 4 +2 √ k k 4 − 2 √ k .
By Lemma 8.2 we have (cid:12) (cid:12) X +0 (1) (cid:12) (cid:12) ≤ ηk 4 ≤ c η 4 .
Combining this fact with Eq.and Eq.
20 we get
X
[Equation]
which implies together with Eq. 18 that X + T (3) ≥ γ [-+1 − 5 cη 4 
-]+ [Equation] [Equation]
[Equation]
α ( k ) .
Therefore,
[Equation]
where the ﬁrst inequality is true because k (cid:88)
[Equation]
The second inequality in Eq.
21 follows since P + T ≤ c η and by appyling Lemma 8.16.
Finally, the last inequality in Eq.
21 follows by the assumption on k .
Hence, z is classiﬁed correctly.
8.6. G ENERALIZATION ON N EGATIVE C LASS
We will need the following lemmas.
Lemma 8.18.
With probability at least 1 − 8 e − 8 , it holds that
[Equation]
[Table]
Proof.
The proof is similar to the proof of Lemma 8.1 and follows from the fact that
[Equation]
Lemma 8.19.
Let
X − t = (cid:88) j ∈ U +0 (2) (cid:104) max (cid:110) σ (cid:16)
[Equation]
j )
[Equation]
[Equation]
(cid:17) , ..., σ (cid:16)
[Equation]
( j ) t · x − d (cid:17)(cid:111)(cid:105) and Y − t = (cid:88)
j ∈ U +0 (4) (cid:104) max (cid:110) σ (cid:16) u
( j ) t · x − 1 (cid:17) , ..., σ (cid:16) u ( j ) t · x − d (cid:17)(cid:111)(cid:105) Then for all t , there exists X, Y ≥ 0 such that | X | ≤ η (cid:12)(cid:12)
U +0 (2) (cid:12)(cid:12) , | Y | ≤ η (cid:12)(cid:12)
U +0 (4) (cid:12)(cid:12) and X − t − X | U {++
+}
(2) | =
Y − t − Y | U +0 (4)
| .
Proof.
First, we will prove that for all t there exists a t ∈ Z such that for j 1 ∈ U − 0 (2) and j 2 ∈ U − 0 (4) it holds that u ( j 1 ) t = u ( j 1 )+ a t η x 2 and u ( j 2 ) t = u ( j 2 )− a t η x 2 .
11 We will prove this by induction on t .
For t = 0 this clearly holds.
Assume it holds for an iteration t .
Let j 1 ∈ U − 0 (2) and j 2 ∈ U − 0 (4) .
By the induction hypothesis, there exists a T ∈
Z such that u ( j 1 ) t = u ( j 1 )+ a t η x 2 and u ( j 2 ) t = u ( j 2 )− a t η x 2 .
Since for all 1 ≤ j ≤ k it holds that (cid:12)(cid:12)(cid:12) u ( j ) 0 · x 2 (cid:12)(cid:12)(cid:12) < η 4 , it follows that either
U − 0 (2) ⊆
U − t (2) and U − 0 (4) ⊆
U − t (4) or U − 0 (2) ⊆
U − t (4) and U − 0 (4) ⊆
U − t (2) .
In either case, by Eq. 11, we have the following update at iteration t + 1 :
[Equation]
and
[Equation]
where a ∈ {− 1 , 0 , 1 } .
Hence, u ( j 1 ) t +1 = u ( j 1 )
[Equation] a t + a ) η x 2 and u ( j 2 ) t = u ( j 2 )− ( a t + a ) η x 2 .
This concludes the proof by induction.
Now, consider an iteration t , j 1 ∈ U +0 (2) , j 2 ∈ U +0 (4) and the integer a t deﬁned above.
If a t ≥then
[Equation]
(cid:110) σ (cid:16) u
( j 1 )· x − 1 (cid:17) , ..., σ (cid:16) u ( j 1 )· x − d (cid:17)(cid:111) =
ηa t and max (cid:110) σ (cid:16) u ( j 2 ) t · x − 1 (cid:17) , ..., σ (cid:16) u ( j 2 ) t · x − d (cid:17)(cid:111) − max (cid:110) σ (cid:16) u ( j 2 )· x − 1 (cid:17) , ..., σ (cid:16) u ( j 2 )· x − d (cid:17)(cid:111) =
ηa
t
Deﬁne X
=
X − 0 and Y
=
Y − 0 then | X | ≤ η (cid:12)(cid:12) U − 0 (2) (cid:12)(cid:12) , | Y | ≤ η (cid:12)(cid:12) U − 0 (4) (cid:12)(cid:12) and
X − t − X (cid:12)(cid:12)
U − 0 (2) (cid:12)(cid:12) =
(cid:12) (cid:12) U − 0 (2) (cid:12) (cid:12) ηa t (cid:12)(cid:12) U − 0 (2) (cid:12)(cid:12) = ηa t = (cid:12) (cid:12) U − 0 (4) (cid:12) (cid:12) ηa t (cid:12)(cid:12) U − 0 (4) (cid:12)(cid:12) =
Y − t − Y (cid:12)(cid:12) U − 0 (4) (cid:12)(cid:12) which proves the claim in the case that a t ≥ 0 . If a t < 0 it holds that
[Table] [Table] [Table] [Table]
[Equation]
[Table]
and
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
Deﬁne
X = (cid:88)
[Table]
[Equation]
[Table]
[Equation]
[Table]
and
Y =
[Table]
{+, σ (cid:16)(cid:16) u ( j ) 
+}
+ x 2 (cid:17) · x − d (cid:17)(cid:111)(cid:105)
Since for all 1 ≤ j ≤ k it holds that (cid:12) (cid:12)(cid:12) u
( j ) 0 · x 2 (cid:12)(cid:12)(cid:12) <
η 4 , we have | X | ≤ η (cid:12)(cid:12)
U − 0 (2) (cid:12) (cid:12) , | Y | ≤ η (cid:12)(cid:12) U − 0 (4) (cid:12) (cid:12) .
Furthermore,
[Equation]
which concludes the proof.
Lemma 8.20.
Let
X
[Equation]
and
[Equation]
[Table]
Then for all t ,
[Equation]
Proof.
We will ﬁrst prove that for all t there exists an integer a t ≥ 0 such that for j 1 ∈ (cid:0)
U +0 (1) ∪ U +0 (3) (cid:1)
∩
U − 0 (2) and j 2 ∈ (cid:0) U +0 (1) ∪ U +0 (3) (cid:1) ∩ U − 0 (4) it holds that u ( j 1 ) t · x 2 = u ( j 1 )· x 2 + ηa t and u ( j 2 ) t · x 4 = u ( j 2 )· x 4 + ηa t .
We will prove this by induction on t .
For t = 0 this clearly holds.
Assume it holds for an iteration t .
Let j 1 ∈ (cid:0) U +0 (1) ∪ U +0 (3) (cid:1)
∩ U − 0 (2) and j 2 ∈ (cid:0) U +0 (1) ∪ U +0 (3) (cid:1)
∩ U − 0 (4) .
By the induction hypothesis, there exists an integer a t ≥such that u ( j 1 ) t · x 2 = u ( j 1 )· x 2 + ηa t and u ( j 2 ) t · x 4 = u ( j 2 )· x 4 + ηa t .
Since for all 1 ≤ j ≤ k it holds that (cid:12)(cid:12)(cid:12) u ( j ) 0 · x 1 (cid:12)(cid:12)(cid:12) <
η 4 , it follows that if a t ≥ 1 we have the following update at iteration T + 1 :
[Equation]
and u
[Equation]
where a ∈ {− 1 , 0 , 1 } .
Hence, u ( j 1 ) t +1 · x 2 = u ( j 1 )· x 2 + η ( a t + a ) and u ( j 2 ) t +1 · x 4 = u ( j 2 )· x 4 + η ( a t + a ) .
Otherwise, if a t = 0 then [Equation]
and
[Equation]
such that a ∈ { 0 , 1 } and b 1 , b 2 ∈ {− 1 , 0 , 1 } .
Hence, u ( j 1 ) t +1 · x 2 = u ( j 1 )· x 2 + η ( a t + a ) and u ( j 2 ) t +1 · x 4 = u ( j 2 )· x 4 + η ( a t + a ) .
This concludes the proof by induction.
Now, consider an iteration t , j 1 ∈ (cid:0) U +0 (1) ∪ U +0 (3) (cid:1)
∩ U − 0 (2) and j 2 ∈ (cid:0) U +0 (1) ∪ U +0 (3) (cid:1)
∩ U − 0 (4) and the integer a t deﬁned above.
We have, max (cid:110) [Equation] u ( [Equation]
1 ) t · x
[Equation]
(cid:17) , ..., σ (cid:16) u (
[Equation]
[Equation]
t · x
[Equation]
d (cid:17)(cid:111) − max (cid:110) σ (cid:16) u ( j [Equation] · x − 1 (cid:17) , ..., σ (cid:16) u ( j 1 )· x − d (cid:17)(cid:111) = ηa t and max (cid:110) σ (cid:16) u ( j 2 ) t · x
[Equation]
1 (cid:17) , ..., σ (cid:16) u ( j
[Equation]
t · x − d (cid:17)(cid:111) − max (cid:110) σ (cid:16) u ( j 2 )· x − 1 (cid:17) , ..., σ (cid:16) u ( j 2 )· x − d (cid:17)(cid:111) =
ηa
t
It follows that
[Equation]
which concludes the proof.
We are now ready to prove the main result of this section.
Proposition 8.21.
Deﬁne β
= 1 − 36 14 c η 35 c η .
Assume that k > 64 (cid:16) β
+1 β − 1 (cid:17) 2 .
Then with probability at least 1 − √ 2 k √ πe 8 k − 8 e − 8 , gradient descent converges to a global minimum which classiﬁes all negative points correctly.
Proof.
With probability at least 1 − √ 2 k √ πe 8 k − 16 e − 8 Proposition 8.13 and Lemma 8.18 hold.
It sufﬁces to show generalization on negative points.
Assume that gradient descent converged to a global minimum at iteration T .
Let ( z , − 1) be a negative point.
Assume without loss of generality that z
i
=
x
2 for all 1 ≤ i ≤ d .
Deﬁne the following sums for l ∈ { 2 , 4 } ,
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table] [Equation] [Table] First, we notice that
[Equation]
and
[Equation]
imply that
[Equation]
We note that by the analysis in Lemma 8.18, it holds that for any t , j 1 ∈ U +0 (2) and j 2 ∈ U +0 (4) , either j 1 ∈ U + t (2) and j 2 ∈ U + t (4) , or j 1 ∈ U + t (4) and j 2 ∈ U + t (2) .
We assume without loss of generality that j 1 ∈ U + T (2) and j 2 ∈ U + T (4) .
It follows that in this case N W T ( z ) ≤ S − T + X − T −
Z − T (2) − Y − T (2) .
12 Otherwise we would replace Y − T (2) with Y − T (4) and vice versa and continue with the same proof.
Let α ( k ) =
k 4 +2 √ k k 4 − 2 √ k .
By Lemma 8.20 and Lemma 8.
[Equation]
≤
[Equation]
and by Lemma 8.19 and Lemma 8.18 there exists Y ≤ c η such that:
[Equation]
Plugging these inequalities in Eq.
24 we get:
[Equation]
which implies that
[Equation]
By Lemma 8.16 we have X − T ≤ 34 c η .
Hence, by using the inequality S − T ≤ c
η we conclude that
[Equation]
where the last inequality holds for k > 64 (cid:16) β +1 β − 1 (cid:17) 2 .
Therefore, z is classiﬁed correctly.
8.6. F INISHING THE P ROOF
First, for k ≥ 120 , with probability at least 1 − √ 2 k √ πe 8 k − 16 e − 8 , Proposition 8.13, Lemma 8.
and Lemma 8.18 hold.
Also, for the bound on T , note that in this case 28( γ +1+8 c η ) c η ≥ 7( γ +1+8 c η ) ( k 2 − 2 √ k ) η . Deﬁne β [Equation]
and
[Equation]
[-Deﬁne β 1 = γ − 40 14 c η 39 c η +
 and β 2 = 1 − 36 14 c η 35 c η-] and let β = max { β 1 , β 2 } .
For γ ≥ 8 and c η ≤ 1410 it holds that 64 (cid:16) β +1 β − 1 (cid:17) 2 < 120 .
By Proposition 8.17 and Proposition 8.21, it follows that for k ≥gradient descent converges to a global minimum which classiﬁes all points correctly.
We will now prove pattern detection results.
In the case of over-paramterized networks, in Proposition 8.17 we proved that X
+ (1) , X + (3) ≥ γ +1 − 5 cη
+ α ( k ) .
Since for i ∈ { 1 , 3 } it holds that
D x i ≥ X
+ ( i ) , it follows that patterns x 1 and x 3 are detected.
Similarly, in Proposition 8.
our analysis implies that, without loss of generality, Y − T (2) + Z − T (2) , Y − T (4) + Z − T (4) ≥ 1 − 5 cη 4 α ( k )+1 .
Since, for l ∈ { 2 , 4 } , D x l ≥ Y − T ( l ) + Z − T ( l ) (under the assumption that we assumed without loss of generality), it follows that patterns x 2 and x 4 are detected.
The conﬁdence of the detection is at least 1 − 5 cη 4 α ( k )+1 .
8. P ROOF OF T HEOREM 5.
We refer to Eq.
17 in the proof of Proposition 8.13.
To show convergence and provide convergence rates of gradient descent, the proof uses Lemma 8.1.
However, to only show convergence, it sufﬁces to bound the probability that W +0 (1) ∪ W +0 (3) (cid:54) = ∅ and that the initialization satisﬁes Lemma 8.2.
Given that Lemma 8.2 holds (with probability at least 1 − (cid:113) 8 π e − 32 ), then W +0 (1) ∪ W +0 (3) (cid:54) = ∅ holds with probability 34 .
By the argument above, with probability at least ( p + p − {+) m (cid:16) 1 − (cid:113) 
+}
π e − 32 (cid:17) 34 all training points are diverse, Lemma 8.2 holds with k = 2 and W +0 (1) ∪ W +0 (3) (cid:54) = ∅ which implies that gradient descent converges to a global minimum.
For the rest of the proof we will condition on the corresponding event.
Let T be the iteration in which gradient descent converges to a global minimum.
Note that T is a random variable.
Denote the network at iteration T by N .
For all z ∈ R 2 d denote
N ( z ) =
[Equation]
Let E denote the event for which at least one of the following holds:
(a) W + T (1) = ∅ .
(b) W + T (3)
= ∅ .
(c) u (1) · x 2 > 0 and u (2) · x 2 > 0 .
(d) u (1) · x 4 > 0 and u (2) · x 4 > 0 .
Our proof will proceed as follows.
We will ﬁrst show that if E occurs then gradient descent does not learn f ∗ , i.e., the network N does not satisfy sign ( N ( x )) =
f ∗ ( x ) for all x ∈ {± 1 } 2 d .
Then, we will show that P [ E ] ≥ 1112 .
This will conclude the proof.
Assume that one of the ﬁrst two items in the deﬁnition of the event E occurs.
Without loss of generality assume that W + T (1) = ∅ and recall that x − denotes a negative vector which only contains the patterns x 2 , x 4 and let z + ∈ R 2 d be a positive vector which only contains the patterns x 1 , x 2 , x 4 .
By the assumption W + T (1) = ∅ and the fact that x 1 = − x 3 it follows that for all j = 1 , 2 ,
[Equation]
Furthermore, since z + contains more distinct patterns than x − , it follows that for all j = 1 , 2 ,
[Equation]
Hence, N ( z + ) ≤ N ( x − ) .
Since at a global minimum N ( x − ) ≤ − 1 , we have N ( z + ) ≤ − 1 and z 2 is not classiﬁed correctly.
Now assume without loss of generality that the third item in the deﬁnition of E occurs.
Let z − be the negative vector with all of its patterns equal to x 4 .
It is clear that N ( z − )
≥and therefore z − is not classiﬁed correctly.
This concludes the ﬁrst part of the proof.
We will now proceed to show that P [ E ] ≥ 1112 .
Denote by A i the event that item i in the deﬁnition of E occurs and for an event A denote by
A c its complement.
Thus E c = ∩ 4 i =
A ci and P [ E c ] =
P [ A c 3 ∩
A c 4 | A c 1 ∩ A c 2 ] P [ A c 1 ∩ A c 2 ] .
We will ﬁrst calculate P [ A c 1 ∩ A c 2 ] .
By Lemma 8.4, we know that for i ∈ { 1 , 3 } , W +0 ( i ) =
W + T ( i ) .
Therefore, it sufﬁces to calculate the probabilty that W +0 (1) (cid:54) = ∅ and W +0 (3) (cid:54) = ∅ , provided that W +0 (1) ∪ W +0 (3) (cid:54) = ∅ .
Without conditioning on W +0 (1) ∪ W +0 (3) (cid:54) = ∅ , for each 1 ≤ i ≤ 4 and 1 ≤ j ≤ 2 the event that j ∈ W +0 ( i ) holds with probability 14 .
Since the initializations of the ﬁlters are independent, we have P [ A c 1 ∩ A c 2 ] = 16 .
We will show that P [ A c 3 ∩ A c 4 | A c 1 ∩ A c 2 ] = 12 by a symmetry argument.
This will ﬁnish the proof of the theorem.
For the proof, it will be more convenient to denote the matrix of weights at iteration t as a tuple of 4 vectors, i.e., W t = (cid:16) w (1)0 , w (2)0 , u (1)0 , u (2)0 (cid:17) .
Consider two initializations W (1) = (cid:16) w (1)0 , w (2)0 , u (1)0 , u (2)0 (cid:17) and W (2) = (cid:16) w (1)0 , w (2)0 , − u (1)0 , u (2)0 (cid:17) and let W (1) t and W (2) t be the corresponding weight values at iteration t .
We will prove the following lemma:
Lemma 8.22.
For all t ≥ 0 , if W (
[Equation]
t
[Equation]
then
[Equation]
Proof.
We will show this by induction on t .
This holds by deﬁnition for t = 0 .
Assume it holds for an iteration t .
Denote W (2) t +1 =
( z 1 , z 2 , v 1 , v 2 ) .
We need to show that z 1 = w (1) t +1 , z 2 = w (2) t +1 , v 1 = − u (1) t +1 and v 2 = u (2) t +1 .
By the induction hypothesis it holds that
N W (1) t ( x + ) =
N W (2) t ( x + ) and N W (1) t ( x − ) =
N W (2) t ( x − ) .
This follows since for diverse points (either positive or negative), negating a neuron does not change the function value.
Thus, according to Eq. 10 and Eq. 11 we have z 1 = w (1) t +1 , z 2 = w (2) t +1 and v [-
-]2 = u (2) t +1 .
We are left to show that v 1 = − u (1) t +1 .
This follows from Eq.
11 and the following facts:
(a) x 3 = − x 1 .
(b) x 2 = − x 4 .
(c) arg max 1 ≤ l ≤ 4 u · x l =
if and only if arg max 1 ≤ l ≤ 4 − u · x l = 3 .
(d) arg max 1 ≤ l ≤ 4 u · x l = 2 if and only if arg max 1 ≤ l ≤ 4 − u · x l = 4 .
(e) arg max l ∈{ 2 , 4 } u · x l = 2 if and only if arg max l ∈{ 2 , 4 } − u · x
l = 4 .
To see this, we will illustrate this through one case, the other cases are similar.
Assume, for example, that arg max 1 ≤ l ≤ 4 u (1) t · x l = 3 and arg max l ∈{ 2 , 4 } u (1) t · x l = 2 and assume without loss of generality that N W (1) t ( x + ) =
N W (2) t ( x + ) < γ and N W (1) t ( x − ) =
N W (2) t ( x − ) > − 1 .
Then, by Eq. 11, u (1) t +1 =
u (1) t − x 3 + x 2 .
By the induction hypothesis and the above facts it follows that v 1 = − u (1) t − x 1 + x 4 = − u
(1) t + x 3 − x 2 = − u (1) t +1 .
This concludes the proof.
Consider an initialization of gradient descent where w (1)0 and w (2)0 are ﬁxed and the event that we conditioned on in the beginning of the proof and A c 1 ∩ A c 2 hold.
Deﬁne the set Bto be the set of all pair of vectors ( v 1 , v 2 ) such that if u (1) = v 1 and u (1) = v 2 then at iteration T , u (1) · x 2 > 0 and u (2) · x 2 > 0 .
Note that this deﬁnition implicitly implies that this initialization satisﬁes the condition in Lemma 8.2 and leads to a global minimum.
Similarly, let B 2 be the set of all pair of vectors ( v 1 , v 2 ) such that if u (1) = v 1 and u (1)0 = v 2 then at iteration T , u (1) · x 4 > 0 and u (2) · x 2 > 0 .
First, if ( v 1 , v 2 ) ∈ B 1 then ( − v 1 , v 2 ) satisﬁes the conditions of Lemma 8.2.
Second, by Lemma 8.22, it follows that if ( v 1 , v 2 ) ∈ B 1 then initializating with ( − v 1 , v 2 ) , leads to the same values of N W t ( x + ) and N W t ( x − ) in all iterations 0 ≤ t ≤ T .
Therefore, initializing with ( − v 1 , v 2 ) leads to a convergence to a global minimum with the same value of T as the initialization with ( v 1 , v 2 ) .
Furthermore, if ( v 1 , v 2 ) ∈ B 1 , then by Lemma 8.22, initializing with u (1) = − v 1 and u (1)
= v 2 results in u (1) · x 2 < 0 and u (2) · x 2 > 0 .
It follows that ( v 1 , v 2 ) ∈
Bif and only if ( − v 1 , v 2 ) ∈ B 2 .
For l 1 , l
[Equation]
∈
[Equation]
deﬁne
[Equation]
Then, by symmetry of the initialization and the latter arguments it follows that P 2 , 2 = P 4 , 2 .
By similar arguments we can obtain the equalities P 2 , 2 = P 4 , 2 = P 4 , 4 =
P 2 , 4 .
Since all of these four probabilities sum to 1 , each is equal to 14 .
Taking expectations of these probabilities with respect to the values of w (1)0 and w (2)0 (given that Lemma 8.2 and
A c 1 ∩ A c 2 hold) and using the law of total expectation, we conclude that
[Equation]
Finally, we show results for detection of a pattern.
To see this, we will show that if one of the four conditions of the event E deﬁned above is met, then for c d > 2 c η , the network does not detect all patterns.
If one of the last two conditions hold, then this is true even for c d ≥ 0 .
Now, assume without loss of generality that W + T (1) = ∅ .
In this case by Lemma 8.4 and Lemma 8.2, it follows that
[Equation]
and therefore, x 1 cannot be detected with conﬁdence greater than 2 c η .
Let Z 1 be the set of positive points which contain only the patterns x 1 , x 2 , x 4 , Z 2 be the set of positive points which contain only the patterns x 3 , x 2 , x 4 .
Let Z 3 be the set which contains the negative point with all patterns equal to x 2 and Z 4 be the set which contains the negative point with all patterns equal to x 4 .
By the proof of the previous section, if the event E holds, then there exists 1 ≤ i ≤ 4 , such that gradient descent converges to a solution at iteration T which errs on all of the points in Z i .
Therefore, its test error will be at least p ∗ (recall Eq. 3).
8. P ROOF OF T HEOREM 4.
Let δ ≥ 1 − p + p − (1 − c − 16 e − 8 ) .
By Theorem 5.2, given 2 samples, one positive and one negative, with probability at least 1 − δ ≤ p + p − (1 − c − 16 e − 8 ) , gradient descent will converge to a global minimum that has 0 test error.
Therefore, for all (cid:15) ≥ 0 , m ( (cid:15), δ ) ≤ 2 .
On the other hand, by Theorem 5.3, if m <
[Equation]
p − ) then with probability greater than
[Equation]
gradient descent converges to a global minimum with test error at least p ∗ .
It follows that for 0 ≤ (cid:15) <
p ∗ , m ( (cid:15), δ ) ≥ 2log ( [Equation] [Equation] [Equation]
XOR
In this section we assume that we are given a training set S ⊆ {± 1 } 2 × {± 1 } 2 consisting of points ( x 1 , 1) , ( x 2 , − 1) , ( x 3 , 1) , ( x 4 , − 1) , where x 1 = (1 , 1) , x 2 = ( − 1 , 1) , x 3 = ( − 1 , − 1) and x 4 = (1 , − 1) .
Our goal is to learn the XOR function with gradient descent.
Note that in the case of two dimensions, the convolutional network introduced in Section 3 reduces to the following two-layer fully connected network.
[Equation]
We consider running gradient descent with a constant learning rate η ≤ c η k , c η ≤ 1 and IID gaussian initialization with mean 0 and standard deviation σ g = c
η 16 k 3 / 2 .
We assume that gradient descent minimizes the hinge loss
[Equation]
where optimization is only over the ﬁrst layer.
We will show that gradient descent converges to the global minimum in a constant number of iterations.
For each point x i ∈ S deﬁne the following sets of neurons:
[Equation] [Equation]
Lemma 9.1.
For i ∈ { 1 , 3 } , if j ∈ W +0 ( i ) then j ∈ W + t ( i ) for all t > 0 and if j ∈ W − 0 ( i ) then j ∈ W − t ( i ) for all t > 0 .
Similarly, for i ∈ { 2 , 4 } , if j ∈ U + t
( i ) then j ∈ U + t
( i ) for all t > 0 and if j ∈ W − t ( i ) then j ∈ U − t
( i ) for all t > 0 .
Proof.
Without loss of generality it sufﬁces to prove the claim for W + t (1) .
This follows by symmetry and the fact that j ∈ W + t (1) if and only if j ∈ W − t (3) .
The proof is by induction.
Assume that j ∈ W it, + .
The gradient of w ( i ) with respect to a point ( x , y ) is given by ∂(cid:96) ( x ,y ) ∂
w ( i ) ( W ) = − yσ (cid:48)
( w ( i ) · x ) x 1 yN W ( x ) < 1 .
Therefore, by the facts w ( j ) t · x 3 < 0 and x 1 · x 2 , x 1 · x 4 = 0 it follows that w ( j ) t +1 · x 1 > 0 , which concludes the proof.
Lemma 9.2.
With probability at least 1 − 8 e − 8 , for all 1 ≤ j ≤
[Equation]
Proof.
Without loss of generality consider (cid:12)(cid:12) W +0 (1) (cid:12)(cid:12) .
Since the sign of a one dimensional Gaussian random variable is a Bernoulli random variable, we get by Hoeffding’s inequality
[Equation]
Since (cid:12)(cid:12) W +0 (1) (cid:12)(cid:12) + (cid:12)(cid:12) W +0 (3) (cid:12)(cid:12) =
k with probability 1, we get that if (cid:12)(cid:12)(cid:12)(cid:12) W +0 (1) (cid:12)(cid:12) − k 2 (cid:12)(cid:12) <
2 √ k then (cid:12)(cid:12)(cid:12)(cid:12) W +0 (3) (cid:12)(cid:12) − k 2 (cid:12)(cid:12) < 2 √ k .
The result now follows by symmetry and the union bound.
For each point x i , deﬁne the following sums:
[Equation]
We will prove the following lemma regarding S + t (1) , S − t (1) , R + t (1) , R − t (1) for i = 1 .
By symmetry, analogous lemmas follow for i (cid:54) = 1 .
Lemma 9.3.
The following holds with probability ≥ 1 − √ 2 k √ πe 8 k :
For all t ≥ 0 , R + t (1) + R − t (1) ≤ kη .
For all t ≥ 0 , S − t (1) = 0 .
Let t ≥ 0 .
If − yN W t ( x 1 ) < 1 , then S + t +1 (1) ≥ S + t (1) + (cid:12)(cid:12) W + t (1) (cid:12)(cid:12)
η 4 .
Otherwise, if − yN W t ( x 1 ) ≥ 1 then S + t +1 (1) =
S + t (1) .
Proof.
For t =
the claim holds by Lemma 8.2.
Assume by contradiction that there exists t > 0 , such that R + t (1) + R − t (1) > kη .
It follows that, without loss of generality, there exists j ∈ U + t (1) such that σ (cid:16) u ( j ) t · x 1 (cid:17) >
η .
In each iteration u ( j ) t · x 1 can increase by at most η , therefore by Lemma 8.2 there exists 0 ≤ t (cid:48) <
t such that 0 < σ (cid:16) u ( j ) t (cid:48) ·
x 1 (cid:17) ≤ η and σ (cid:16) u ( j ) t (cid:48) +1 · x 1 (cid:17) > η .
However, in this case u ( j ) t (cid:48) +1 =
u ( j ) t (cid:48)
+ α x 1 + β x 2 + γ x 4 where α ≤ 0 .
Therefore, u ( j ) t (cid:48) +1 · x 1 < u ( j ) t (cid:48) · x 1 ≤ η , a contradiction.
This is a direct consequence of Lemma 9.1.
By Lemma 9.1, if j ∈ W + t (1) and − yN W t ( x 1 ) < 1 then w ( j ) t +1 = w ( j ) t + η 4 x 1 + β x 2 + γ x 4 , from which the ﬁrst part of the claim follows.
The second claim follows similarly.
Proposition 9.4.
Assume that k ≥ 25 .
With probability ≥ 1 − √ 2 k √ πe 8 k − 8 e − 8 , for all i , if until iteration T there were at least l ≥ k k − 2 √ k + 10 iterations, in which − yN W t ( x i ) < 1 , then it holds that − yN W t ( x i ) ≥ 1 for all t ≥ T .
Proof.
Without loss of generality assume that i
= 1 .
By Lemma 9.3 and Lemma 8.2, with probability ≥ 1 − √ 2 k √ πe 8 k − 8 e − 8 , if − yN W t ( x 1 ) < 1 then S + t +1 (1) ≥ S + t (1) + (cid:16) k 2 − 2 √ k (cid:17) η .
Therefore, by Lemma 9.3, for all t ≥ T
[Equation]
where the last ineqaulity follows by the assumption on l .
Theorem 9.5.
Assume that k ≥ 25 .
With probability ≥ 1 − √ 2 k √ πe 8 k − 8 e − 8 , after at most 4 k k − 2 √ k + iterations, gradient descent converges to a global minimum.
Proof.
Proposition 9.4 implies that there are at most
[Equation]
√
[Equation]
40 iterations in which there exists ( x i , y i ) such that − y i N W t ( x i ) < 1 .
After at most that many iterations, gradient descent converges to a global minimum.
VC D IMENSION
As noted in Remark 3.1, the VC dimension of the model we consider is at most 15 .
To see this, we ﬁrst deﬁne for any z ∈ {± 1 } 2 d the set P z ⊆ {± 1 } 2 which contains all the distinct two dimensional binary patterns that z has.
For example, for a positive diverse point z it holds that P z
= {± 1 } 2 .
Now, for any points z (1) , z (2) ∈ {± 1 } 2 d such that P z (1) =
P z (2) and for any ﬁlter w ∈ R 2 it holds that max j σ (cid:16) w · z (1) j (cid:17) =
max j σ (cid:16) w · z (2) j (cid:17) .
Therefore, for any W , N W ( z (1) ) =
N W ( z (2) ) .
Speciﬁcally, this implies that if both z (1) and z (2) are diverse then N W ( z (1) ) =
N W ( z (2) ) .
Since there are 15 non-empty subsets of {± 1 } 2 , it follows that for any k the network can shatter a set of at most 15 points, or equivalently, its VC dimension is at most 15.
Despite these expressive power limitations, there is a generalization gap between small and large networks in this setting, as can be seen in Figure 1.