B OUNDS ON O VER -P ARAMETERIZATION FOR G UARANTEED E XISTENCE OF D ESCENT P ATHS IN S HALLOW R E LU N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
We study the landscape of squared loss in neural networks with one-hidden layer and ReLU activation functions.
Let m and d be the widths of hidden and input layers, respectively.
We show that there exist poor local minima with positive curvature for some training sets of size n ≥ m + 2 d − 2 .
By positive curvature of a local minimum, we mean that within a small neighborhood the loss function is strictly increasing in all directions.
Consequently, for such training sets, there are initialization of weights from which there is no descent path to global optima.
It is known that for n ≤ m , there always exist descent paths to global optima from all initial weights.
In this perspective, our results provide a somewhat sharp characterization of the over-parameterization required for “existence of descent paths” in the loss landscape.
I NTRODUCTION
We consider shallow neural networks of the form shown in Fig.
The network comprises a hidden layer and an input layer of widths m and d , respectively; and is to be trained over a training set of size n .
Our results concern the slightly over-parameterized regime where n ≈ m .
We study the existence of poor local minima that have positive curvature in the empirical squared loss landscape.
It is well-known that poor local minima exist in the loss landscape of shallow networks of arbitrary width.
In fact, in a shallow network with ReLU activation functions, it is easy to construct training sets whose empirical loss landscape has high plateaus.
1 It is however not fully understood that under what conditions poor local minima may have positive curvature.
This paper presents results that improve this understanding.
Non-existence of spurious local minima is closely connected to the so called descent path property : a loss landscape is said to have the descent path property if starting from any initial point there is a path of descent loss to a global minimum.
From optimization perspective, the descent path property favors descent optimization algorithms like the pure gradient descent (GD) method.
For SGD as well, non-existence of poor local minima is known to be a favorable property for guaranteed convergence (Ge et al., 2015; Jin et al., 2017; Lee et al., 2016).
The descent path property is shown to be satisﬁed in over-parameterized shallow networks with sufﬁciently large widths (Venturi et al., 2018).
The results we present in this work, tighten the existing bounds on the over-parameterization required to guarantee this property.
1. B ACKGROUND
Over the past few years, deep neural networks have achieved tremendous performance in various artiﬁcial intelligence applications such as computer vision, reinforcement learning, and natural language processing, etc.
Despite their remarkable success in practice, theoretical aspects of this success remain a mystery.
It has long been an open problem why simple local search algorithms for training deep neural networks, like stochastic gradient descent (SGD), typically converge to local minima with low training error despite the highly non-convex behavior of empirical loss.
It has been observed, e.g., in (Choromanska et al., 2015), that these methods may get stuck in poor local minima
(i.e., local minima with empirical loss much larger than the global optimum) for small networks, while the problem fades away as the number of parameters grows larger.
Such observations are often explained by studying the loss landscape in over-parameterized regime where the number of parameters in the network exceeds the training sample size.
Recently, several attempts have been made to characterize properties of squared loss landscape by conditioning on the layers’ dimensions and sample size.
Soudry and Hoffer (2017) showed that weights of a neural network can be adjusted such that the empirical loss is zero almost surely if m > 4 (cid:100) n/ (2 d − 2) (cid:101)
≈ (2 n ) /d .
This result is consistent with experimental observations that neural networks can ﬁt training data if the number of parameters (here approximately 2 n ) is greater than the sample size.
They also proved for normally distributed input that as n goes to inﬁnity, the ratio between the volume of poor ﬂat local minima regions to the volume of ﬂat global minima fades exponentially if d = ˜Ω( √ n ) and m = ˜Ω( n/d ) .
Safran and Shamir (2016) showed that if the number of neurons in the hidden layer is Ω( n rank( X ) ) (where X is the matrix containing all input), then with high probability, random initialization of weights will put them in a region of parameter space at which the loss surface has a basin-like structure, i.e., every local minimum in that region is global.
In another work (Safran and Shamir, 2017), the same authors provide a computer-assisted proof to show that spurious local minima are common in the expected loss landscape of shallow under-parameterized (small-width) networks.
Xie et al.
(2016) showed that if the input data is drawn uniformly at random from the unit sphere, and if m = ˜Ω( n β ) and d = ˜Ω( n β ) with β ∈ (0 , 1) being the decay exponent of the smallest eigenvalue of a kernel matrix, then every critical point is a global minimum.
Li et al.
(2018) proved that for any continuous activation function and under the assumption that data samples are distinct, there exist no poor local minima with positive curvature if m ≥ n .
In the same spirit,
Venturi et al.
(2018) showed that for any continuous activation function, there is always a descent path to an optimal solution with zero loss in the empirical loss landscape if m ≥ n .
Several works have proposed similar results in other settings and under different assumptions.
Soudry and Carmon (2016) showed that in a network of leaky ReLU activation functions with randomized perturbation of slopes, all differentiable local minima are global minima if m ≥ n/d . Kawaguchi (2016) proved that in shallow networks with linear activation functions, every local minimum is a global minimum and all the saddle points are strict in the sense that they have a direction of strictly negative curvature.
Soltanolkotabi et al.
(2019) showed that the same result carries over to quadratic activation functions under the assumption that the last layer comprises at east d positive and d negative weights.
Du and Lee (2018) established similar results for quadratic activation functions, assuming m ≥ √ 2 n .
For deep neural networks with linear activation functions, Freeman and Bruna (2016) showed that all local minima are global minima if there is a hidden layer whose number of neurons exceeds the minimum of the widths of input and output layers.
For deep neural networks with analytical activation functions, Nguyen and Hein (2017) proved a similar property under the assumptions that the number of neurons in some hidden layer is greater than sample size and the network has a pyramidal structure.
Such studies on the properties of loss landscape do not only provide insights into the complication of training, but are also beneﬁcial for proving performance guarantees for some local search algorithms.
For the class of loss functions whose landscape satisfy the properties of: a) all local minima are global, b) all saddle points are strict, it has been shown in several works (Ge et al., 2015; Jin et al., 2017; Lee et al., 2016) that perturbed gradient descent converges to global optima in polynomial time.
Another direction of research concerns the convergence of speciﬁc optimization algorithm such as pure gradient descent and SGD without assuming such properties for the loss landscape (Du et al., 2018; Allen-Zhu et al., 2018; Du et al., 2018).
1. O UR CONTRIBUTIONS
We study the amount of over-parameterization required for guaranteed existence of descent paths to zero loss in the empirical loss landscape.
Previous works suggest that zero loss is always possible for m ≥ 2 n/d (Soudry and Hoffer, 2017).
On the other hand, the best existing bound for guaranteed existence of descent paths to this zero loss requires m ≥ n neurons in the hidden layer (Venturi et al., 2018).
Prior to the present work, it was not known whether the “descent path property” holds for m < n .
Even for m ∈ (2 n/d, n ) , where zero empirical risk is known to be achievable (Soudry and Hoffer, 2017), the existence of descent paths was in question.
In this work, we tighten this gap and prove that there are training sets, under which in any network of width m ≤ n − 2 d + 2 , there
(cid:1850) (cid:3031)(cid:3400)(cid:2869) (cid:1875) (cid:1877)(cid:3548) (cid:1874) (cid:3040) (cid:1874) (cid:2869) (cid:1874) (cid:2870) exist initial weights that have no descent path to global minima.
We do this by showing that the loss landscape, in this regime, admits poor local minima with positive curvature.
We also provide evidences and make conjectures that these results carry over to networks of width m = n − 4 , which, if true, provides a sharp characterization of the over-parameterization required for guaranteed existence of descent paths.
We also wish to point that unlike most previous works, we do not restrict to differentiable local minima; for a simple argument shows that local minima with positive curvature cannot be differentiable if m > n/d (cf.
Appendix A).
1. O UTLINE
We continue by discussing details of the system model and introducing our key deﬁnitions in Section 2.
We then present, in Section 3, the main results of the paper.
Proof of the main results are then given in Section 4.
We ﬁnally discuss implications and possible extensions of our results in Section 5 along with a number of open problems and directions for future research.
P RELIMINARIES 2. M ODEL
We consider shallow networks of the form shown in Fig.
The network takes d -dimensional inputs denoted by X .
There is a single hidden layer comprising m neurons with ReLU activation function.
For simplicity of our proofs, we only consider even values of m .
We denote the input weights of r -th neuron by a d -dimensional vector w r , for r = 1 , . . .
, m .
We then let w ∈ R
md be the vector representation of all weights in the ﬁrst layer.
The output layer has a single neuron, whose activation function is linear with an m -dimensional weight vector denoted by v .
The network outputs a scalar ˆ y ( w, v ) =
(cid:80) m r =1 v r w Tr X 1 ( w Tr X ≥ 0) .
We ﬁx a training set ( X 1 , y 1 ) , . . .
, ( X n , y n ) of size n , and consider the landscape of empirical squared loss function:
[Equation]
2. P ROPERTIES OF THE LANDSCAPE
We ﬁrst provide a formal deﬁnition for the descent path property, which is a necessary condition for guaranteed performance of descent optimization algorithms.
Deﬁnition 1 (Descent path property) .
Consider a continuous function f : R d → R and let f ∗ =
inf x ∈ R d f ( x ) be its inﬁmum.
We say that f has the descent path property if for any x ∈ R d , there exists a continuous curve with γ : [0 , 1] → R d such
that γ (0) = x , f (cid:0) γ (1) (cid:1) =
f ∗ , and f (cid:0) γ ( t ) (cid:1) is a non-increasing function of t .
The descent path property is a necessary condition for any descent optimization algorithm to provably ﬁnd a global minimum from all initial conditions.
It was shown in Venturi et al.
(2018) that the empirical loss landscape of a shallow neural network with ReLU activation and squared loss has the descent path property if the size of training data is no larger than the width of the hidden layer, i.e., n ≤ m .
We now characterize a class of local minima of speciﬁc form in the following deﬁnition.
Deﬁnition 2 (Cupped minima) .
Given a function f : R d → R , we call x ∈ R d a cupped minimum of f if there are (cid:15), δ > 0 such that for any y in the δ -neighborhood of x , we have f ( y ) ≥ f ( x )+ (cid:15) (cid:107) y − x (cid:107) 2 .
By a sub-optimal cupped minimum we mean a cupped minimum that is not a global minimum.
Note that every cupped minimum is a local minimum, but not every local minimum is cupped (e.g., ﬂat local minima are not curved downwards, and hence are not cupped).
Also note that a function is not necessarily differentiable at its cupped minima.
We study cupped minima of the loss function in equation 1.
Note however that for any α > 0 , F ( αw, v/α )
=
F ( w, v ) .
Therefore, F ( · , · ) has no cupped minima if both arguments are taken as variables.
For that matter, when talking about cupped minima of F , we ﬁx a v and consider F ( · , v ) as a function of its ﬁrst argument.
Interestingly, existence of cupped minima for F ( · , v ) leads to violation of descent path property for F ( · , · ) over both arguments, as shown in the following lemma.
The proof is given in Appendix B.
Lemma 1. Consider a shallow network with loss function F in equation 1, and a pair of weights ( w, v ) .
Suppose that w is a sub-optimal cupped minimum of F ( · , v ) , and that w r (cid:54) = 0 , for r = 1 , . . .
, m .
Then, F ( · , · ) has no descent path (cid:0) w ( t ) , v ( t ) (cid:1) , initiated at ( w, v ) , to its global minima.
M AIN RESULTS
The following theorem and corollary state the main results of the paper.
Theorem 1.
For any d ≥ 4 , m ≥ 8 + 4 (cid:4) 3 / ( d − 3) (cid:5) , and n ≥ m + 2 d − 2 , there exists a training set of size n such that the empirical loss function F of a shallow neural network of width m has the following property.
For any m -dimensional vector v , with m/ 2 number of positive and m/ 2 number of negative entries, F ( · , v ) has exponentially many sub-optimal cupped minima.
The proof is constructive and is given in Section 4.
In particular, we devise a training sequence ( X 1 , y 1 ) , . . .
, ( X n , y n ) such that for weights ( w, v ) at the cupped minima, we have (cid:107) v r w r (cid:107) = 1 / √ m , for r = 1 , . . .
, m .
Moreover, (cid:107) X
i (cid:107) ≤ 1 , | y i | ≤ 2 , and | e i | = 1 for i = 1 , . .
. , n (cf.
Remark 2).
According to Theorem 1, there are training sequences tailored to give rise to sub-optimal cupped minima.
However, we wish to point that the existence of such cupped minima does not stem from measure-zero incidents like placement of several data points on a low dimensional plane.
In fact, in view of Lemma 1, any path that starts from a cupped minimum and end up in a global minimum would have an uphill climb of at least (cid:15) , for some (cid:15) > 0 .
Since the loss surface is a continuous function of ( X i , y i ) , a small perturbation of ( X i , y i ) ’s leads continuously to a small change in F .
Therefore, for small enough perturbations of ( X i , y i ) , any path to the set of global minima would still witness a positive uphill-climb.
Hence, the descent path property remains out of order, even when the training data is slightly perturbed.
Based on the above intuition, we can establish the following corollary 2 ,
Corollary 1.
For any d ≥ 4 , m ≥ 8 + 4 (cid:4) 3 / ( d − 3) (cid:5) , and n ≥ m + 2 d − 2 ; and when the inputs
X and labels y are randomly drawn from independent normal distributions, there is a non-zero probability that F ( · , · ) does not have the descent path property.
Note that Corollary 1 does not imply Theorem 1, because a landscape could be cupped-min-free even if it the descent path property is not satisﬁed.
It was shown in Venturi et al.
(2018) that n ≤ m is sufﬁcient for the descent path property to hold.
In contrast, Corollary 1 show that if n ≥ m + 2 d − 2 , then the descent path property is not necessarily in effect.
This leave a gap of size 2 d − 2 for the edge of over-parameterization required to guarantee the descent path property.
We believe that this edge lies sharp at n = m .
We conjecture a stronger version of Theorem 1, that cupped minima can emerge for training data sizes as small as m = n − 4 .
Conjecture 1.
Statement of Theorem 1 holds for all d ≥ 4 , m ≥ 2 d + 4 , and n ≥ m + 4 .
See Remark 1 for insights into the possibility of this conjecture.
P ROOF OF THE MAIN RESULT
In this section, we present the proof of Theorem 1 organized in a sequence of four subsections.
We ﬁrst present some preliminaries in Subsection 4.1.
In Subsection 4.2, we introduce a geometric structure called “ ( d, t, k ) -conﬁguration”, based on which we construct, in Subsection 4.3, the training set that gives rise to cupped minima in the loss landscape.
Finally, in Subsection 4.4, we prove the existence of cupped mimima in the devised setting.
In order to provide intuitions on the loss landscape at cupped minima and motivate our construction of the training set in Subsection 4.3, we make a short note on different types of cupped minima in
Appendix A.
We also defer the proofs of some lemmas from this section to appendices for improved readability.
4. P RELIMINARIES
Consider weights ( w, v ) and let ( w (cid:48) , v (cid:48) ) be another set of weights such that for r = 1 , . . .
, m , v r and v (cid:48)
r have the same sign and w (cid:48) r
= ( v r /v (cid:48) r ) w r .
Then, F ( w, v ) = F ( w (cid:48) , v (cid:48) ) .
Moreover, it is no difﬁcult to see that w is a cupped minimum of F ( · , v ) if and only if w (cid:48) is a cupped minimum of F ( · , v (cid:48) ) .
For this reason, it sufﬁces to prove existence of cupped minima for a ﬁxed vector v .
Note also that where w r ’s are distinct, any permutation of the order of neurons would give rise to a new cupped minimum.
Hence, existence of a cupped minimum for F ( · , v ) implies existence of exponentially many cupped minima for F ( · , v ) .
We denote by e i
= ˆ y i ( w, v ) − y i the estimation error for input X i .
We let u d = [0 , . . .
, 0 , 1] T be a d -dimensional vector with the last entry equal to one and all other entries equal to zero.
For a region P ⊆
R d , we denote its interior and and its convex-hull by int( P ) and Conv( P ) , respectively.
Assuming differentiability of F at w , the partial derivatives of the loss function with respect to w r , r = 1 , . .
, m , is as follows
[Equation]
4. A GEOMETRIC CONFIGURATION
We introduce a geometric structure for sets of points in R d .
This conﬁguration will be used in
Subsection 4.3 to construct a landscape with cupped minima.
Deﬁnition 3 ( ( d, t, k ) -Conﬁguration) .
Given integers d, t, k ≥ 0 and disjoint sets A , ˜ A , B , and ˜ B of points in R d , we say that (cid:0) A, ˜ A, B, ˜ B (cid:1) forms a ( d, t, k ) -conﬁguration if the following properties are satisﬁed:
(p1) Each of A and B consists of t points, and each of ˜ A and ˜ B consists of k points.
(p2) (A)
The convex hull of A (cid:83) ˜
A forms a polytope P A that has exactly t + k vertices.
Equivalently, no point in A (cid:83) ˜
A is a convex combination of other points in A (cid:83)
˜
A .
(B) Similarly, the convex hull of B (cid:83) ˜
B forms a polytope P B that has exactly t + k vertices.
(p3)
We have 0 ∈ int( P A ) and 0 ∈ int( P B ) .
(p4)
There exists a constant β ∈ (0 , 1) such that (A) For any a ∈ A , βa lies on a facet of P B .
We denote this facet by S B ( a ) .
(B) For any b ∈ B , βb lies on a facet of P A .
We denote this facet by S A ( b ) .
(p5) (A)
For any a ∈ A , S B ( a ) is a ( d − 1) -dimensional simplex.
(B) For any b ∈ B , S A ( b ) is a ( d − 1) -dimensional simplex.
(p6) (A)
For any a ∈ A , there exist scalars α 1 , . .
. , α d ∈ (0 , 1) such that a = (cid:80) di
=
α i s i ( a ) , where s 1 ( a ) , . . .
, s d ( a ) are the vertices of simplex S B ( a ) .
(B) For any b ∈ B , there exist scalars α 1 , . .
. , α d ∈ (0 , 1) such that b = (cid:80) di
=
α i s i ( b ) , where s 1 ( b ) , . . .
, s d ( b ) are the vertices of simplex S A ( b ) .
(p7) (A)
For any pair a and a (cid:48) of distinct points in A , we have S B ( a ) (cid:54) =
S B ( a (cid:48) ) .
Moreover, letting H a be the hyperplane that contains S B ( a ) , a and a (cid:48)(cid:48) lie on opposite sides of H a , for all
a (cid:48)(cid:48) ∈ A (cid:83) ˜
A with a (cid:48)(cid:48) (cid:54) = a . [Figure]
[Figure] [Figure] (B) For any pair b and b (cid:48) of distinct points in B , we have S A ( b ) (cid:54) =
S A ( b (cid:48) ) .
Moreover, letting H b be the hyperplane that contains S A ( b ) , b and b (cid:48)(cid:48) lie on opposite sides of H b , for all b (cid:48)(cid:48) ∈ B (cid:83) ˜ B with b (cid:48)(cid:48) (cid:54) =
b .
(p8) Consider a 2 t × 2 t matrix M whose rows and columns are associated to points
p ∈ A (cid:83) B and q ∈ A (cid:83) B , and whose entries are as follows [Equation] p
[Equation]
p
[Equation]
q
[Equation]
p
[Equation]
A
[Equation]
q
[Equation]
B
[Equation]
p
[Equation]
B
[Equation]
q
[Equation] A [Equation] where H q is the hyperplane deﬁne in Property (p7) , and d ( · , · ) is the euclidean distance.
The property requires M to be full-rank.
Among the above properties, the most difﬁcult of all is Property (p4), and the requirement that it involves the same β for all points in A (cid:83) B .
In fact, elimination of Property (p4) gives rise to trivial conﬁgurations.
In the two dimensional space, for any t ≥ 4 there exists a (2 , t, 0) -conﬁguration of the form illustrated in Fig.
In the following proposition, we generalize this observation to higher dimensions.
Proposition 1.
For any d ≥ 2 and t ≥ 4 , there exists a ( d, t, d − 2) -conﬁguration.
The proof is constructive, and is given in Appendix C.
We conjecture that there also exist ( d, t, 0) - conﬁgurations.
Conjecture 2.
For any d ≥ 2 and t ≥ 2 d , there exists a ( d, t, 0) -conﬁguration.
Remark 1.
Using a conﬁguration given by Conjecture 2 instead of the conﬁguration from Proposition 1 in the construction and the proof that follow, we obtain a proof for Conjecture 1.
In this view, establishing Conjecture 2 would also resolve Conjecture 1.
4. C ONSTRUCTING A LANDSCAPE WITH CUPPED MINIMA
Here we present a set of training data ( X 1 , y 1 ) , . . .
, ( X n , y n ) and a set of wights ( w 1 , v 1 ) , . .
, ( w m , v m ) such that the empirical loss surface corresponding to ( X 1 , y 1 ) , . . .
, ( X n , y n ) has a sub-optimal cupped minimum at ( w 1 , v 1 ) , . .
, ( w m , v m ) .
Without loss of generality, we assume n = m + 2 d − 2 .
Extension to larger values of n is straightforward via replication.
Let (cid:0) A, ˜ A, B, ˜ B (cid:1) be a ( d − 1 , m/ 2 , d − 3)
-conﬁguration in the ( d − 1) -dimensional space, as in Proposition 1.
In view of Property (p3), let (cid:15) 0 > 0 be such that P A and P B contain the (cid:15) 03 e.g., for A and ˜ A chosen uniformly at random over the unit sphere, and letting ( B, ˜ B ) be a small rotation of ( A, ˜ A ) around an arbitrary axis, it is neighborhood of 0 .
Let
[Equation]
We proceed by introducing the data points X 1 , . . .
, X n .
An illustration of these data points in the three dimensional space is shown in Fig.
Data points X : We consider a total number of n = m + 2 d − 2 data points as follows.
• For each a ∈ A (cid:83)
˜
A , we consider a new data point X a as follows.
Let [ z 1 , . . .
, z d − 1 ] ∈ R d −be the representation of a in the Cartesian coordinates.
We let X a = (cid:2) z 1 , . . .
z d − 1 , 1 (cid:3) .
• For each b ∈ B (cid:83) ˜ B , we consider a new data point X b as follows.
Let [ z 1 , . . .
, z d − 1 ] ∈ R d −be the representation of b in the Cartesian coordinates.
We let X b = − (cid:2) z 1 , . . .
, z d − 1 , 1 (cid:3) .
• We consider two extra points
X A
+ , X A − , X B + , and X B − as follows.
We let X A − (cid:44) ξu d , X B − (cid:44) − ξu d , and
[Equation]
where u d = [0 , . . . , 0 , 1] T , and ξ and β are deﬁned in equation 4 and Property (p4), respectively.
Weights at cupped minimum: We associate each of m neurons to a point p in A (cid:83) B , in a one-one manner; and denote the vector of input weights and the output weight of that neuron by w p and v p , respectively.
These weights are chosen as follows:
• For each a ∈ A , we let v a = − 1 / √ m .
• For each b ∈ B , we let v b = 1 / √ m .
• For each a ∈ A , consider the facet S B ( a ) deﬁned in Property (p4), and let s 1 ( a ) , . . .
, s d − 1 ( a ) be the vertices of S B ( a ) (as in Property (p6)).
We let w a be the unique vector such that (cid:107) w a (cid:107) = 1 and
[Equation]
• For each b ∈ B , consider the facet S A ( b ) and let s 1 ( b ) , . . .
, s d − 1 ( b ) be the vertices of S A ( b ) .
We let w b be the unique vector such that (cid:107) w b (cid:107) = {+
+} and
[Equation]
Labels y :
Having determined the data points X and the weights ( w, v ) , the output ˆ y ( w, v ) of the network is determined for all input X .
In the following, we choose the true labels y to obtain a desired error e
= ˆ y − y for each input data.
In particular:
• For each a ∈ A (cid:83)
˜
A , we associate to X a a label y a
so that e a (cid:44) ˆ y a ( w, v ) − y a = 1 .
• For each b ∈ B (cid:83) ˜ B , we associate to X b a label
y b so that e b (cid:44) ˆ y b ( w, v )
− y b
= − 1 .
• We choose the labels associated to X A
+ , X A − , X B + , and X B − such that e A + = e B + =and e A − =
e B − = − 1 .
This completes the description of the training set.
As shown in in Soudry and Carmon (2016), there exist weights that achieve zero loss if m > 4 (cid:100) n/ (2 d − 2) (cid:101) .
In our case, n = m + 2 d − 2 , and its easy to check that m > 4 (cid:6) ( m + 2 d − 2) / (2 d − 2) (cid:7) for all d ≥ 4 and m ≥ 8 + 4 (cid:4) 3 / ( d − 3) (cid:5) .
It follows that in our setting the global optimum has zero loss, showing that the above ( w, v ) is sub-optimal.
Remark 2.
In the above construction, the norms of inputs vectors may be very large.
If we scale down the inputs such that (cid:107) X i (cid:107) ≤ 1 / √ m for i = 1 , . . . , n , and modify the corresponding labels
y i such that e i remains unchanged, then a same set of weights will still be a cupped minimum for the landscape deﬁned in terms of new ( X i , y i ) ’
s. Moreover, for this setting, it is easy to check that (cid:107) w r (cid:107) =
1 , (cid:107) e i (cid:107) = 1 , and | y i | ≤ 2 .
4. P ROVING THE CUPPED MINIMA PROPERTY
Let
[Equation]
Then, δ > 0 .
For any θ ∈ R md with (cid:107) θ (cid:107) =
and for any t ∈ [0 , δ ] , let
[Equation]
We show that there is an (cid:15) > 0 such that for any unit-norm θ and any t ∈ [0 , δ ] ,
[Equation]
Lemma 2.
For any θ ∈ R md with (cid:107) θ (cid:107) = 1 , F θ ( · ) is a quadratic and convex function over [0 , δ ] .
The proof is give in Appendix D and relies on the fact that neuron activations do not alter at w + θt for t ∈ [0 , δ ] .
Consider now the following m -dimensional subspace of R md
[Equation]
For any θ ∈ R md , let θ (cid:107) and θ ⊥ be the orthogonal projections of θ on H w and H ⊥ w , respectively.
Then, θ = θ (cid:107) + θ ⊥ .
In order to establish equation 13, we need lower bounds on F (cid:48) θ (0) and F (cid:48)(cid:48) θ , which we derive in the next two lemmas.
Lemma 3.
There exists µ > 0 such that for any θ ∈ R md with (cid:107) θ (cid:107) = 1 , we have
[Equation]
The proof is given in Appendix E, and relies in a subtle way on the choice of data points in subsection 4.3.
We now bound the curvature of F θ .
Lemma 4.
There exist constants η 1 , η 2 > 0 such that for any θ ∈ R md with (cid:107) θ (cid:107) = 1 , and for any t ∈ (0 , δ ) ,
[Equation]
The proof is given in Appendix F, and relies on Property (p8) of the underlying conﬁguration.
Consider now the second order polynomial p ( x ) =
η 1 δ (cid:0) 1 − x 2 (cid:1) − η 2 δx − µx , where µ , η 1 , and ηare the constants in Lemmas 3 and 4.
Since p (0) > 0 and p (1) < 0 , the polynomial p has exactly one root in the interval (0 , 1) , which we denote by x 0 .
Let [Equation]
Lemma 5.
For any θ ∈ R md with (cid:107) θ (cid:107) = 1 , and any t ∈ [0 , δ ] , we have F θ ( t ) ≥
F θ (0) + (cid:15)t 2 .
This lemma is a simple consequence of Lemmas 3 and 4, and its proof is given in Appendix G. It follows from Lemma 5 that for any w (cid:48) in the δ -neighborhood of w , we have F ( w (cid:48) , v ) ≥ F ( w, v ) + (cid:15) (cid:107) w (cid:48) − w (cid:107) 2 .
This shows that w is a cupped minimum for F ( · , v ) , and completes the proof of
Theorem 1.
D ISCUSSION
The guaranteed existence of descent paths in shallow networks of ReLU neurons was previously established (Venturi et al., 2018), given an over-parameterization of m ≥ n (where m and n are the number of neurons and the size of training data, respectively).
This left an uncertainty gap of 2 n/d < m < n , where zero empirical risk is known to be achievable (for m ≥ 2 n/d ) (Soudry and
Hoffer, 2017), but the existence of descent paths was in question.
In this work, we have tightened this uncertainty gap to n − 2 d
+2 < m < n , by proving that for any m ≤ n − 2 d
+2 , there are input data and initial weights for which a descent path does not exist. This conclusion we reach by establishing the existence of cupped minima for m ≤ n − 2 d + 2 , and for the right choice of input data. Compared to similar existing results for other activation functions, our results suggest that the edge m ≈ n of over-parameterization required for elimination of sub-optimal cupped minima in ReLU networks is much higher than that of networks with quadratic activation functions, m ≈ √ 2 n (Du and Lee, 2018), and linear activation functions, m ≈ n/d (Kawaguchi, 2016), and is almost as high as general continuous activation functions of any form, m ≤ n (Venturi et al., 2018). Non-existence of spurious local minima and the decent path property favor the convergence of decent optimization methods like GD.
However, for different variants of noisy GD, like SGD and Langevin dynamics, it is quite common for the empirical loss to ﬂuctuate during the training. Nevertheless, for theoretical analysis purposes it usually helps to take the noise away, for example by tending the step size to zero. The resulting GD, which always follows a descent path, is usually easier to analyze and can also help to study the SGD dynamics. On the other hand, from a practical view, convergence of
SGD in local-min-free landscapes is well-studied.
Aside from addressing Conjecture 2, there remain several open problems, which we review next.
As an important direction for future research, it would be interesting if one could obtain bounds on the probability of existence of cupped minima over random data sets, underneath the edge of over-parameterization.
In particular, we showed in Corollary 1 that this probability is non-zero, however we gave no clue on either the size or scaling of this probability.
Another class of problems concerns basins of local minima, and how they affect dynamics of ﬁrst order optimization algorithms.
As a step toward this goal, one might characterize the true over-parameterization regime in which the basins of poor local minima have considerable volume.
Among other directions are extensions of our results to deep ReLU networks, shallow non-ReLU networks and shallow ReLU networks under loss functions more general than the squared loss.
A D IFFERENT TYPES OF CUPPED MINIMA IN TERMS OF DIFFERENTIABILITY
Here we discuss different types of cupped minima and provide elementary intuitions on the loss landscape at cupped minima.
We ﬁrst characterize curvature of the loss function at differentiable points.
For r = 1 , . . .
, m , let J r be an n × n diagonal matrix whose ( i, i ) entry equals 1 (cid:0) w Tr X i ≥ 0 (cid:1) .
Let G be an n × md matrix of the form:
[Equation]
where X is a d × n matrix that has X i in its i -th column.
If F is differentiable at ( w, v ) , its gradient is ∇ w F ( w, v ) =
G T [ e 1 , . . .
, e m ] T , where e i =
ˆ y
i − y
i is the output error for input X i .
Moreover, if F is differentiable at ( w, v ) , its Hessian with respect to w equals
[Equation]
We classify cupped minima into three categories in terms of differentiability.
Speciﬁcally, for a cupped minimum w of F ( · , v ) , we consider three cases:
Type 1) F is differentiable at w .
Type 2) F ( w + θt, v ) as a function of t is non-differentiable at t = 0 , for all θ ∈ R md .
Type 3)
There are θ 1 , θ 2 ∈ R md such that F ( w + θ 1 t, v ) is differentiable at t = 0 , while F ( w + θ 2 t, v ) is non-differentiable at t = 0 .
Fig.
4 illustrates examples of loss surface at the above three types of cupped minima.
We now argue that the ﬁrst two types are not possible in the loss landscape of shallow networks.
If F ( · , v ) is differentiable at w , its Hessian given in equation 19 equals G T G .
Since G is an n × md matrix, assuming md > n , G T G would have zero eigenvalues.
Therefore, w cannot be a cupped minimum of F ( · , v ) .
It follows that there exists no differentiable cupped minimum (nor saddle point) if md > n .
For non-differentiable points, note that F ( w + wt, v ) as a function of t is a differentiable quadratic function.
This is because the output scales proportionally with w .
Therefore, cupped minima of the second type are not possible, as well.
In the same spirit, it can be shown that F ( w + θt, v ) is differentiable as a function of t , if θ ∈ R md belongs to the m dimensional subspace H w deﬁned in equation 14.
For the above reasons, all cupped minima, if any, are of the third type.
Therefore in the construction of Subsection 4.3, we introduce a training set and a pair of weights ( w, v ) such that w is a cupped minimum of F ( · , v ) , and F ( w + θt, v ) is differentiable in t only for θ ∈ H w .
B P ROOF OF L EMMA
If v has a zero entry, v r = 0 for some r , then F ( · , v ) is a constant function with respect to w r , and thereby has no cupped minima.
Therefore, we assume that v has no zero entries.
Let
[Equation]
For t > 0 , we deﬁne
[Equation]
Let
[Equation]
[Figure]
We show that t 1 < t 0 .
If t 0 < ∞ , then continuity of v ( t ) implies that v r ( t 0 )
= 0 , for some r ≤ m .
Therefore, ˜ w r ( t 0 ) =
(cid:54) =
w r .
It then follows from the continuity of ˜ w r ( · ) that there is an (cid:15) >
such that ˜ w r ( t 0 − (cid:15) )
(cid:54) =
w r .
Consequently, t 1 < t 0 .
The inequality t 1 < t 0 implies that there is an (cid:15) > 0 such that ˜ w r ( t 1 + (cid:15) )
(cid:54) = w , and for any t ∈ [0 , t 1 + (cid:15) ] , we have sgn (cid:0) v r ( t ) (cid:1) =
sgn ( v r ) , r = 1 , . .
, m .
Therefore, for any t ∈ [0 , t 1 + (cid:15) ] , we have
F (cid:0) ˜ w ( t ) , v (cid:1) =
F (cid:0) w ( t ) , v ( t ) (cid:1) .
Since w is a cupped minimum of F ( · , v ) , there is an s ∈ [ t 1 , t 1 + (cid:15) ] such that F (cid:0) ˜ w ( t ) , v (cid:1) >
F ( w, v ) .
This shows that (cid:0) w ( t ) , v ( t ) (cid:1) cannot be a descent path for F ( · , · ) , and completes the proof of Lemma 1.
C P ROOF OF P ROPOSITION
For i = 3 , . . .
, d , let
[Equation]
Fix a constant
[Equation]
Let γ be a uniform random variable
[Equation]
We take a sample from the above distribution and ﬁx a γ for the rest of the proof.
We now introduce the points in the conﬁguration.
For i = 0 , . . .
, t − 1 , we consider points a i ∈ A and b i ∈ B as follows
[Equation]
For i =
[Equation] [Equation]
d
[Equation]
we consider points ˜ a
i ∈ ˜ A and ˜ b i ∈ ˜ B as follows
[Equation]
Fig.
5 shows an illustration of these points for d = 3 .
We proceed by verifying Properties (p1)–(p8).
Property (p1) : Property (p1) is straightforward from the above construction.
Property (p2) :
Considering only the ﬁrst two coordinates, it is easy to see that no point of A is a convex combination of other points in A (cid:83)
˜
A .
Also note that for i = 3 , . . .
, d , ˜ a
i is the only point
[Figure]
[Equation]
in A (cid:83) ˜
A whose i -th entry is negative.
Therefore, no point in ˜ A can be represented as a convex combination of other points in A (cid:83) ˜
A .
A similar argument works for Part (B) of Property (p2).
Property (p3) : Let
[Equation] [Equation]
A simple induction on j shows that
[Equation]
for j = 4 , . . .
, d .
We show that
[Equation]
For the ﬁrst two coordinates, equation 31 is easy.
Let a ij and ˜ a ij be the j -th entries of a i and ˜ a i , respectively.
For the j -th coordinate, j = 3 , . . .
, d , we then have
[Equation]
where the second equality is due to equation 30.
This establishes equation 31.
It then follows from equation 31 that 0 is a convex combination of points in A (cid:83)
˜
A .
Consequently, 0 ∈ int( P A ) .
A similar argument shows that 0 ∈ int( P B ) .
Properties (p4) – (p7) :
We only prove Part (B) for each of these properties; as similar proofs work also for Part (A)’s.
Moreover, because of the rotational symmetry of A and B in the ﬁrst two coordinates, it sufﬁces to prove of the Properties (p4)–(p7) only for b 0 .
Properties (p4) and (p5) :
For the j -th coordinate, j = 3 , . . .
, d , we have
[Equation]
where b 0 j is the j -th entry of b 0 deﬁned in equation 25.
Similarly, for the ﬁrst two coordinates, we have:
[Equation]
It then follows from equation 32 and equation 33 that
[Equation] [Equation]
Let
[Equation]
Then, from equation
[Equation]
[Equation]
Therefore, βb 0 is a convex combination of a 0 , a 1 , ˜ a 3 , . . .
, ˜ a d , and therefore lies on the simplex S that has a 0 , a 1 , ˜ a 3 , . .
˜ a d as its vertices.
Next, we show that S is a facet of P A .
Consider a vector z ∈ R d with entries
[Equation]
Then, a simple backward induction, with base case i =
d , shows that for i = 3 , . . .
, d , we have
[Equation]
= z i + 1 .
In the same vein,
[Equation]
It follows that for i = 3 , . . .
, d ,
[Equation] [Equation]
Moreover, for i = 0 , 1 ,
[Equation]
where the second equality is due to equation 38.
Let H be the hyperplane that passes through a 0 , a 1 , ˜ a 3 , . . .
, ˜ a d .
It follows from equation 39 and equation 40 that for any p ∈ (cid:8) a 0 , a 1 , ˜ a 3 , . . .
, ˜ a d (cid:9) , we have z T p = 1 .
Therefore, z is orthogonal to H .
Consequently,
[Equation]
For i = 2 , . . .
, t − 1 , we have
[Equation]
where the second equality is due to equation 40, and the inequality is because z 1 > 0 and cos(2 π ( i − 1 / 2) /t )
− cos( π/t ) < 0 .
It follows from equation 41 and equation 42 that all points of A \{ a 0 , a 1 } lie on one side of H , while all points of ˜ A (cid:83) { a 0 , a 1 } lie on H .
Then, H is a tangent hyperplane to
P A .
Thus, the simplex S is a facet of P A .
This completes the proofs of Properties (p4) and (p5).
Moreover, from the deﬁnition H b in Property (p7), we have
[Equation] [Equation]
Property (p6) :
Since t ≥ 4 , we have 2 cos( π/ 2) > 1 .
Moreover, recall from equation 23 that γ < 1 .
Property (p6) then follows from equation 34 and the fact that S is a facet of P A .
Property (p7) :
For i = 0 , . . .
, t − 1 ,
[Equation]
where the second equality is due to equation 40 and deﬁnitions of a 0 and b i , and the third equality is from equation 38.
It follows that
[Equation]
where the inequality is because the ﬁrst two terms on the left hand side of the inequality are positive.
In the same vein, for i = 1 , . . .
, t −
[Equation]
where the second inequality follows from the deﬁnition of z 1 in equation 37 and the fact that z 1 > 1 , and the third inequality is from the deﬁnition of γ in equation 23 and the fact that 2 γ ˜ c ≤ cos( π/t ) − cos(2 π/t ) .
It then follows from equation 41, equation 45, and equation 46 that b
and b i lie on opposite sides of H , for i = 1 , . . .
, t − 1 .
On the other hand, since ˜ b i = − ˜ a i , for i = 3 , . . .
, d , it follows from equation 39 that z T ˜ b
i = − z T ˜ a
i = − 1 < 1 .
Therefore, for i = 1 , . . . , 3 , ˜ b
i and b 0 lie on opposite sides of H .
This completes the proof of Property (p7).
Property (p8) :
We have
[Equation]
where the ﬁrst equality is due to equation 43 and equation 41, and the second equality follows from the equality in equation 45.
Similarly, from equation 42, we have for i = 0 , . . .
, t −
[Equation]
For i, j ∈ { 0 , . . .
, t − 1 } , let ˆ m a
i ,b j (cid:44) cos (cid:0) 2 π ( i − j − 1 / 2) /t (cid:1) − cos( π/t ) and ˆ m b i ,a j (cid:44) cos (cid:0) 2 π ( i − j + 1 / 2) /t (cid:1) − cos( π/t ) .
Then, it follows from equation 48 and rotational symmetry of A and B in the ﬁrst two coordinates that for i, j ∈ { 0 , . . .
, t − 1 } ,
[Equation]
Note that for any p, q ∈ A (cid:83) B , ˆ m p,q is a constant independent of the value of γ .
Let ˆ M be a 2 t × 2 t matrix, with entries
[Equation]
for p, q ∈ A (cid:83) B .
Then, all entries of ˆ M are constants independent of γ .
Let ˆ λ 1 , . . .
, ˆ λ 2 t be the eigenvalues of ˆ m .
It follows that ˆ λ 1 , . . .
, ˆ λ 2 t are also constants independent of γ .
Consider the matrix M deﬁned in equation 3.
It follows from equation 47 and equation 49 that for any p, q ∈ A (cid:83) B ,
[Equation]
if p ∈ A and q ∈ B, OR p ∈ B and q ∈ A,
[Equation]
otherwise .
[Equation]
Consider the order (cid:0) a 0 , . .
. , a t − 1 , b 0 , . . .
, b t − 1 (cid:1) on the elements of A (cid:83) B .
Then,
[Equation]
where I is the 2 t × 2 t identity matrix and the second equality is from the deﬁnition of z 1 in equation 37.
Denote the eigenvalues of M by λ 1 , . . .
, λ 2 t .
Then, from elementary linear algebra,
[Equation]
for i = 1 , . . .
, 2 t .
Therefore, there is at most one value of γ for which λ i = 0 .
Then, in view of equation 23, we have Pr( λ i = 0)
= 0 , over the random choice of γ .
Thus, with probability one, M has no zero eigenvalues and is thereby full-rank.
As an immediate consequence, M is full-rank for suitable choice of γ .
This establishes Property (p8) and completes the proof of Proposition 1.
D P ROOF OF L EMMA
Consider a block representation of θ as follows
[Equation]
where each θ r is a d -dimensional vector.
It follows from the deﬁnition of δ that for any t ∈ (0 , δ ) , if w Tr X i > 0 , then ( w r + θ r t ) T X i > 0 ; and if w Tr X i < 0 , then ( w r + θ r t ) T X i < 0 .
Therefore, for r = 1 , . . .
, m and i = 1 , . .
. , n , and for any t ∈ [0 , δ ) ,
[Equation]
Consequently, for any t ∈ [0 , δ ] and i = 1 , . . .
, n , we have ˆ
[Equation]
It follows that ˆ y i (cid:0) w + θt, v (cid:1) is a linear function of t over the interval t ∈ [0 , δ ] .
Therefore, F θ ( t ) = (cid:80) ni
[Equation]
w
[Equation]
, v ) − y i (cid:1) 2 is a quadratic and convex function of t , for t ∈ [0 , δ ] .
E P ROOF OF L EMMA
We ﬁrst characterize active neurons for different inputs.
For two subsets S 1 , S 2 ⊂ R d we let
S 1 \ S 2 = S 1 (cid:84)
S c 2 .
Recall the deﬁnition of s i ( a ) and s i ( b ) from Property (p6).
Claim 1.
For any a ∈ A , we have
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
a
[Equation]
Similarly, for any b ∈ B , we have
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
˜
[Equation]
Proof of Claim 1.
Fix a b ∈ B .
We begin by introducing some notations.
Let H be the ( d − 2) - dimensional hyperplane in the ( d − 1) -dimensional space that passes through s 1 ( b ) , . . .
, s d − 1 ( b ) .
In the same spirit, let ˜ H be the ( d − 1)
-dimensional
subspace in the d -dimensional space that passes through X s 1 ( b ) , . .
X s d − 1 ( b ) , equivalently ˜ H is the subspace orthogonal to w b .
We denote the convex hull of X s 1 ( b ) , . . .
X s d − 1 ( b ) by C B (cid:44) Conv (cid:0)(cid:8) X s 1 ( b ) , . . .
X s d − 1 ( b ) (cid:9)(cid:1) .
Similarly, we let
Q A (cid:44) conv (cid:0)(cid:8) X a | a ∈ A (cid:83) ˜
A (cid:9)(cid:1) and Q B (cid:44) Conv (cid:0)(cid:8)
X b | b ∈ B (cid:83) ˜ B
(cid:9)(cid:1) .
Before presenting the proofs of properties equation 57–equation 72, we review make some easy observations.
Recall the deﬁnition of (cid:15) 0 from the paragraph proceeding equation 4.
Let B d − 1 (cid:15)be the intersection of the (cid:15)
-ball centered at 0 with the orthogonal space of u d .
Then, from the deﬁnition of (cid:15) , we have
[Equation]
For x ∈ R d , let π ( x ) be the projection of x on the span of ﬁrst d − 1 coordinates, i.e., the orthogonal space of u d .
Then,
[Equation]
where the ﬁrst equality is from the deﬁnition of X A
+ and the second inequality follows from the deﬁnition of ξ in equation 4.
Therefore, π ( X A + ) / (cid:0) ξ + 1 /β − n − 1 (cid:1) ∈ B
d − 1 (cid:15) 0 .
Consequently,
[Equation]
where the ﬁrst equality is orthogonal decomposition of X A + , and last inclusion follows from equation 73.
In the same vein, we can show that
[Equation]
We proceed to verify equation 65–equation 72.
Eq.
equation 65 follows from equation 9.
Recall the deﬁnitions X
A − = ξu d and X B − = − ξu d .
Then, equation 10 implies equation 69 and equation 71.
Since X s 1 ( b ) , . . .
X s d − 1 ( b ) deﬁne a boundary of the ( d − 1)
-dimensional
convex set Q A , and ˜ H passes through X s 1 ( b ) , . . .
X s d − 1 ( b ) , then all points in Q A lie on a same side of ˜ H .
In other words, either we have w Tb x ≥ 0 , for all x ∈ Q A ; or we have w Tb x ≤ 0 , for all x ∈ Q A .
In view of
Property (p3), u d ∈ int( Q A ) .
It then follows from equation 10 that for any x ∈ Q A , we have w Tb x ≥ 0 .
Consequently, for any x ∈ Q A \ ˜ H , we have w Tb x > 0 .
In particular,
[Equation]
where the ﬁrst inequality is because Property (p5) implies that X a (cid:54)∈ ˜ H for a ∈ (cid:0) A (cid:83)
˜
A (cid:1) \ (cid:8) s 1 ( b ) , . . .
, s d − 1 ( b ) (cid:9) , and the last two inequalities are due to equation 75 and equation 76, respectively.
This establishes equation 67, equation 68, and equation 70.
For equation 66, it follows from Property (p4)
that b and the origin, 0 , lie on opposite sides of hyperplane H .
Consequently, − X b and u d also lie on opposite sides of hyperplane ˜ H .
Therefore, w Tb X b and w Tb u d have a same sing.
It then follows from equation 10 that w Tb X b > 0 , and equationfollows.
For equation 72, it follows from Property (p7) that for any b (cid:48) ∈ B (cid:83)
˜ B with b (cid:48) (cid:54) =
b , X b and X b (cid:48) lie on opposite sides of ˜ H .
Eq.
equation 66 then implies that w T b X b (cid:48) < 0 .
This establishes equation 72, and completes the proof of Claim 1.
In light of Claim 1, it is easy to see for r = 1 , . . .
, m and i = 1 , . .
n that if w Tr X i = 0 , then
[Equation]
In our next claim, we examine a linear combination of data points for which a particular neuron is active.
Claim 2.
For r = 1 , . . .
, m , there exist constants γ r 1 , . .
. , γ rm such that
[Equation]
Proof of Claim 2.
Fix a b ∈ B .
We prove the claim for the neuron associated to b .
It follows from
Claim 1 that
[Equation]
where the second equality is due to the deﬁnitions of e a , e b , e A + , and e A − , and the third equality is from the deﬁnitions of X A + in equation 5.
On the other hand, it follows from Property (p6) that there exist scalars α 1 , . . .
, α d ∈ (0 , 1) such that b = (cid:80) d − 1 i =
α i s i ( b ) .
Therefore, from the deﬁnition of X b ,
[Equation]
Moreover, Property (p4) implies that (cid:80) d − 1 i =
α i = 1 /β .
Then, from equation 80,
[Equation]
For i = 1 , . . .
, d − 1 , let γ i
=
− α i .
Then, Claim 1 implies that
[Equation]
where last equality in due to equation 81.
Combing equation 79 and equation 82, we obtain equationfor w r = w b .
A similar argument implies equation 78 for w r = w a , a ∈ A .
This completes the proof of Claim 2.
Back to the proof of Lemma 3, for r = 1 , . . .
, m , let
[Equation]
for the constant γ ri deﬁned in Claim 2.
It follows that (cid:15) r > 0 , for r = 1 , . . .
, m .
For any r ≤ m , X s 1 ( r ) , . . .
X s d − 1 ( r ) are linearly independent and, by deﬁnition, are all orthogonal to w r .
Therefore, there exists a constant (cid:15) (cid:48) r > 0 such that for any ζ r ∈ R d with (cid:107) ζ r (cid:107)
= 1 , we have max i =1 ,...,d − 1 | ζ Tr X s i ( r ) | ≥ (cid:15) (cid:48) r (cid:107) ζ ⊥ r (cid:107) , where ζ ⊥ r is the projection of ζ r on the null-space of w r .
Consequently, for any ζ r ∈ R d with (cid:107) ζ r (cid:107) = 1 ,
[Equation]
In particular, considering the block-vector representation of θ in equation 54, we obtain for r = 1 , . . .
, m ,
[Equation]
Let µ (cid:44) min r (cid:15) r (cid:15) (cid:48) r / √ m .
Then, µ > 0 .
It then follows from Claim 2 that, for r = 1 , . . .
, m , v r n (cid:88) i =
[Equation]
(cid:88)
[Equation]
where the third equality is due to equation
[Equation]
the fourth equality is because θ Tr X i and 1 ( θ Tr X i > 0) − γ ri have always the same sign, the ﬁrst inequality is by deﬁnition of (cid:15) r in equation 83, the third inequality follows from equation
[Equation]
and the last inequality is from the deﬁnition of µ
[Equation]
On the other hand
[Equation]
equation 2 implies that
[Equation]
where the second equality is due to equation 55 and the ﬁrst inequality follows from equation 85.
This completes the proof of Lemma 3.
F P ROOF OF L EMMA
We begin by a claim.
Given a q ∈ A (cid:83) B , recall the deﬁnition of hyperplane H q from Property (p7).
Claim 3.
For any pair of points p, q ∈ A (cid:83) B , we have
[Equation]
Proof of Claim 3.
In the ( d − 1) -dimensional space, let ω be the unit normal vector of H q .
Recall from Property (p6) that s 1 ( q ) , . . .
, s d − 1 ( q ) are located on H q .
Let,
[Equation]
Then,
[Equation]
Without loss of generality suppose that q ∈ A .
Let ˜ ω be the lifting of ω from the ( d − 1) -dimensional space to the d -dimensional space by appending ω by a new coordinate with zero coefﬁcient, i.e., ˜ ω is a d -dimensional vector with ˜ ω
i = ω
i , for i = 1 , . . .
d − 1 , and ˜ ω
d = 0 .
For γ deﬁned in equation 88, let
[Equation]
Then, we have (cid:107) z (cid:107) =
1 and z T u d < 0 .
Moreover, for i = 1 , . . .
, d − 1 ,
[Equation]
where the second equality is because s i ( q ) ∈ B (cid:83) ˜ B for q ∈ A .
It follows from equation 91 and the deﬁnition of w q in equation 7 and equation 8 that w q = z .
Therefore,
[Equation]
where the second equality is from the deﬁnition of z in equation
[Equation]
and the last equality is due to equation 88 and equation 89.
This completes the proof of Claim
[Equation]
We now proceed to the proof of Lemma 4.
Fix an arbitrary θ ∈ R d with (cid:107) θ (cid:107) = 1 .
Without loss of generality 4 assume that F ( · , · ) is differentiable at (cid:0) w + δθ/ 2 , v (cid:1) .
For r = 1 , . . .
, m , let J r be a diagonal matrix whose ( i, i ) entry, for i = 1 , . . .
, n , equals
[Equation]
where the equality is due to equation 55.
Recall the deﬁnition of matrix G in equation 18:
[Equation] [Equation]
Then, for any t ∈ (0 , δ ) ,
[Equation]
[Equation]
where the third equality is due to equation 19.
On the other hand, since θ = θ (cid:107) + θ ⊥ , we have
[Equation]
In the following claim, we elaborate on (cid:107) Gθ (cid:107) (cid:107) .
Claim 4.
There exists a constant η 1 > 0 such that (cid:107) Gθ (cid:107) (cid:107) 2 ≥ 2 η 1 (cid:107) θ (cid:107) (cid:107) 2 , for all θ ∈ R d .
Proof of Claim 4.
Recall that θ (cid:107) is the projection of θ on subspace H w deﬁned in equation 14.
Then, there exist constants α 1 . . .
, α m such that
[Equation]
Let α be the vector representation of α 1 , . . .
, α m .
Then,
[Equation]
Consider the n × m matrix
[Equation]
Then, from the deﬁnition of matrix G in equation 18,
[Equation]
Each column of ˜ G corresponds to a neuron, and thereby is associated to a point in A (cid:83) B .
In the same vein, every row of ˜ G is associated to an input vector.
By removing some rows of ˜ G , we devise a matrix ˜ M so that each row of ˜ M is associated to an input X p for p
∈ A (cid:83) B .
Therefore, ˜ M is an m × m matrix, whose rows and columns are associated to the points in A (cid:83) B .
It follows that ˜
Mα is a vector obtained by removing some entries from vector ˜ Gα .
As a result,
[Equation]
In the following, we capitalize on Property (p8) to show that ˜ M is full-rank.
For q ∈ A (cid:83) B , let
[Equation]
For p, q ∈ A (cid:83) B , the entry in row p and column q of ˜ M equals
[Equation]
where the ﬁrst equality is from the deﬁnition of ˜ M , the second equality follows from the deﬁnitions of ˜ G and J q in equation 99 and equation 93, the third equality is because X Tp w
[Equation]
= 0 , and the fourth equality is due to Claim 3.
Then, Claim 1 implies that for any p, q ∈ A (cid:83) B , ˜ M pq = (cid:26) γ q d (cid:0) p, H q (cid:1) , if p = q, OR
p ∈ A and q ∈ B, OR p ∈ B and q ∈ A, 0 , otherwise .
Compared to matrix M deﬁned in equation 3, each column q of ˜ M equals the column q of M multiplied by a non-zero constant γ q .
In view of Property (p8), M is full-rank.
It follows that ˜ M is full-rank, as well.
Let σ be the smallest singular value of ˜ M .
Since ˜ M is full-rank, we have σ > 0 .
Moreover,
[Equation]
Then,
[Equation]
where the equations are due to equation 100, equation 101, equation 104, and equation 98, respectively.
Claim 4 then follows for η 1 = σ 2 / 2 .
Back to the proof of Lemma 4, we denote by σ max the largest singular value of G .
Let η 2 (cid:44) σ 2max .
Then,
[Equation]
where the second inequality is from the deﬁnition of σ max , the third inequality is because (cid:107) θ (cid:107) (cid:107) ≤ (cid:107) θ (cid:107)
= 1 , the last equality is by the deﬁnition of η 2 , and the last inequality follows from Claim 4.
This completes the proof of Lemma 4.
G P ROOF OF L EMMA
Fix a θ ∈ R md with (cid:107) θ (cid:107) = 1 .
Recall the deﬁnition of x 0 from the paragraph proceeding equation 17.
If (cid:107) θ ⊥ (cid:107) ≥ x 0 , then for any t ∈ [0 , δ ] ,
[Equation]
where the ﬁrst inequality is from convexity of F θ in Lemma 2, the second inequality is due to
Lemma 3, the equality is by the deﬁnition of (cid:15) in equation 17, and the last inequality is because t ≤ δ .
On the other hand, if (cid:107) θ ⊥ (cid:107) < x 0 , then for any t ∈ [0 , δ ] ,
[Equation]
where the ﬁrst equality is because F θ is quadratic (c.f. Lemma 2), the ﬁrst inequality follows from
Lemma 3, the second inequality is due to Lemma 4, the third inequality is because (cid:107) θ ⊥ (cid:107) < x 0 , and the last two equalities are due to p ( x 0 )
=
and the deﬁnition of (cid:15) in equation 17, respectively.
Combining the above two cases, we obtain Lemma 5.