T HE C ONVEX G EOMETRY OF B ACKPROPAGATION : N EURAL N ETWORK G RADIENT F LOWS C ONVERGE TO E XTREME P OINTS OF THE D UAL C ONVEX P ROGRAM
A BSTRACT
We study non-convex subgradient ﬂows for training two-layer ReLU neural networks from a convex geometry and duality perspective.
We characterize the implicit bias of unregularized non-convex gradient ﬂow as convex regularization of an equivalent convex model.
We then show that the limit points of non-convex subgradient ﬂows can be identiﬁed via primal-dual correspondence in this convex optimization problem.
Moreover, we derive a sufﬁcient condition on the dual variables which ensures that the stationary points of the non-convex objective are the KKT points of the convex objective, thus proving convergence of non-convex gradient ﬂows to the global optimum.
For a class of regular training data distributions such as orthogonal separable data, we show that this sufﬁcient condition holds.
Therefore, non-convex gradient ﬂows in fact converge to optimal solutions of a convex optimization problem.
We present numerical results verifying the predictions of our theory for non-convex subgradient descent.
I NTRODUCTION
Neural networks (NNs) exhibit remarkable empirical performance in various machine learning tasks.
However, a full characterization of the optimization and generalization properties of NNs is far from complete.
Non-linear operations inherent to the structure of NNs, over-parameterization and the associated highly nonconvex training problem makes their theoretical analysis quite challenging.
In over-parameterized models such as NNs, one natural question arises: Which particular solution does gradient descent/gradient ﬂow ﬁnd in unregularized NN training problems?
Suppose that X ∈ R N ×
d
is the training data matrix and y ∈ { 1 , − 1 } N is the label vector.
For linear classiﬁcation problems such as logistic regression, it is known that gradient descent (GD) exhibits implicit regularization properties, see, e.g., (Soudry et al., 2018; Gunasekar et al., 2018).
To be precise, under certain assumptions, GD converges to the following solution which maximizes the margin:
[Equation]
Here we denote [ N ] = { 1 , . . .
, N } .
Recently, there are several results on the implicit regularization of the (stochastic) gradient descent method for NNs.
In (Lyu & Li, 2019), for the multi-layer homogeneous network with exponential or cross-entropy loss, with separable training data, it is shown that the gradient ﬂow (GF) and GD ﬁnds a stationary point of the following non-convex max-margin problem:
[Equation]
where f ( θ ; x ) represents the output of the neural network with parameter θ given input x .
In (Phuong & Lampert, 2021), by further assuming the orthogonal separability of the training data, it is shown that all neurons converge to one of the two max-margin classiﬁers.
One corresponds to the data with positive labels, while the other corresponds to the data with negative labels.
However, as the max-margin problem of the neural network (2) is a non-convex optimization problem, the existing results only guarantee that it is a stationary point of (2), which can be a local minimizer or even a saddle point.
In other words, the global optimality is not guaranteed.
In a different line of work (Pilanci & Ergen, 2020; Ergen & Pilanci, 2020; 2021b), exact convex optimization formulations of two and three-layer ReLU NNs are developed, which have global optimality guarantees in polynomial-time when the data has a polynomial number of hyperplane arrangements, e.g., in any ﬁxed dimension or with convolutional networks of ﬁxed ﬁlter size.
The convex optimization framework was extended to vector output networks (Sahiner et al., 2021b), quantized networks (Bartan & Pilanci, 2021b), autoencoders (Sahiner et al., 2021c; Gupta et al., 2021), networks with polynomial activation functions (Bartan & Pilanci, 2021a), networks with batch normalization (Ergen et al., 2021), univariate deep ReLU networks, deep linear networks (Ergen &
Pilanci, 2021c) and Generative Adversarial Networks (Sahiner et al., 2021a).
In this work, we ﬁrst derive an equivalent convex program corresponding to the maximal margin problem (2).
We then consider non-convex subgradient ﬂow for unregularized logistic loss.
We show that the limit points of non-convex subgradient ﬂow can be identiﬁed via primal-dual correspondence in the convex optimization problem.
We then present a sufﬁcient condition on the dual variable to ensure that all stationary points of the non-convex max-margin problem are KKT points of the convex max-margin problem.
For certain regular datasets including orthogonal separable data, we show that this sufﬁcient condition on the dual variable holds, thus implies the convergence of gradient ﬂow on the unregularized problem to the global optimum of the non-convex maximalo margin problem (2).
Consequently, this enables us to fully characterize the implicit regularization of unregularized gradient ﬂow or gradient descent as convex regularization applied to a convex model.
1. R ELATED W ORK
There are several works studying the property of two-layer ReLU networks trained by gradient descent/gradient ﬂow dynamics.
The following papers study the gradient descent like dynamics in training two-layer ReLU networks for regression problems.
Ma et al.
(2020) show that for two-layer
ReLU networks, only a group of a few activated neurons dominate the dynamics of gradient descent.
In (Mei et al., 2018), the limiting dynamics of stochastic gradient descent (SGD) is captured by the distributional dynamics from a mean-ﬁeld perspective and they utlize this to prove a general convergence result for noisy SGD.
Li et al.
(2020) focus on the case where the weights of the second layer are non-negative and they show that the over-parameterized neural network can learn the ground-truth network in polynomial time with polynomial samples.
In (Zhou et al., 2021), it is shown that mildly over-parameterized student network can learn the teacher network and all student neurons converge to one of the teacher neurons.
Beyond (Lyu & Li, 2019) and (Phuong & Lampert, 2021), the following papers study the classiﬁcation problems.
In (Chizat & Bach, 2018), under certain assumptions on the training problem, with overparameterized model, the gradient ﬂow can converge to the global optimum of the training problem.
For linear separable data, utilizing the hinge loss for classiﬁcation, Wang et al.
(2019) introduce a perturbed stochastic gradient method and show that it can attain the global optimum of the training problem.
Similarly, for linear separable data, Yang et al.
(2021) introduce a modiﬁed loss based on the hinge loss to enable (stochastic) gradient descent ﬁnd the global minimum of the training problem, which is also globally optimal for the training problem with the hinge loss.
1. P ROBLEM SETTING
We focus on two-layer neural networks with ReLU activation, i.e., f ( θ , X ) =
( XW 1 ) + w 2 , where
[Equation]
× m , w 2 ∈ R m and θ = ( W 1 , w 2 ) represents the parameter.
Due to the ReLU activation, this neural network is homogeneous, i.e., for any scalar c > 0 , we have f ( c θ ; X ) =
c 2 f ( θ ; X ) .
The training problem is given by
[Equation]
where (cid:96) ( q ) :
R → R + is the loss function.
We focus on the logistic, i.e, cross-entropy loss, i.e., (cid:96) ( q ) = log(
+ exp( − q )) .
Then, we brieﬂy review gradient descent and gradient ﬂow.
The gradient descent takes the update rule
[Equation]
where g ( t ) ∈ ∂ ◦ L ( θ ( t )) and ∂ ◦ represents the Clarke’s subdifferential.
The gradient ﬂow can be viewed as the gradient descent with inﬁnitesimal step size.
The trajectory of the parameter θ during training is an arc θ : [0 , + ∞ )
→ Θ , where Θ = { θ = ( W 1 , w 2 ) | W 1 ∈ R
d × m , W 2 ∈ R m } .
More precisely, the gradient ﬂow is given by the differential inclusion
[Equation]
for t ≥ 0 , almost everywhere.
M AIN R ESULTS
In this section, we present our main results and defer the detailed analysis to the following sections.
Consider the more general multi-class version of the problem with K classes.
Suppose that ¯ y ∈
[ K ] N is the label vector.
Let Y =
( y n,k ) n ∈ [ N ] ,k ∈ [ K ] ∈ R N × K be the encoded label matrix such that y n
[Equation]
Similarly, we consider the following two-layer vector-output neural networks with ReLU activation:
[Equation]
where we write Θ = ( θ 1 , .
, θ K ) .
For k = 1 , . . .
, K , we have θ k =
( W ( k ) 1 , w ( k ) 2 )
where
W ( k )∈ R N × m and w ( k )∈
R m .
One can view each of the K outputs of F ( Θ , X ) as the output of a two-layer scalar-output neural network.
Consider the following training problem:
[Equation]
According to (Lyu & Li, 2019), the gradient ﬂow and the gradient descent ﬁnds a stationary point of the following non-convex max-margin problem:
[Equation]
Denote the set of all possible hyperplane arrangement as
[Equation]
and let p = |P| .
We can also write P =
{ D 1 , . . .
, D p } .
From (Cover, 1965), we have an upper bound p ≤ 2 r (cid:16) e ( N − 1) r (cid:17) r where r = rank ( X ) .
We ﬁrst reformulate (5) as convex optimization.
Proposition 1 The non-convex problem (5) is equivalent to the following convex program
[Equation]
where y k is the k -th column of Y .
The dual problem of (7) is given by
[Equation]
[Equation]
where λ k is the k -th column of Λ .
We present the detailed derivation of the convex formulation (7) and its dual problem (8) in the appendix.
Given u ∈ R d , we deﬁne D ( u ) =
diag ( I ( Xu > 0)) .
For two vectors u , v ∈ R d , we deﬁne the cosine angle between u and v by cos ∠ ( u , v ) =
u T v (cid:107) u (cid:107) 2 (cid:107) v (cid:107) 2 .
2. O UR CONTRIBUTIONS
The following theorem illustrate that for neurons satisfying sign ( y Tk ( Xw ( k ) 1 ,i ) + ) = sign ( w ( k ) 2 ,i ) at initialization, w ( k ) 1 ,i align to the direction of ± X T D ( w ( k ) 1 ,i )
y k at a certain time T , depending on sign ( w ( k ) 2 ,i k, + ) at initialization.
In Section 2.3, we show that these are dual extreme points of (7).
Theorem 1 Consider the K -class classiﬁcation training problem (4) for any dataset.
Suppose that the neural network is scaled at initialization such that (cid:107) w ( k ) 1 ,i (cid:107) 2 = | w ( k ) 2 ,i | for i ∈ [ m ] and k ∈ [ K ] .
Assume that at initialization, for k ∈ [ K ] , there exists neurons ( w ( k ) 1 ,i k , w ( k ) 2 ,i k ) such that
[Equation]
where s ∈ { 1 , − 1 } .
Consider the subgradient ﬂow applied to the non-convex problem (4) .
Let δ ∈ (0 , 1) .
Suppose that the initialization is sufﬁciently close to the origin.
Then, for k ∈ [ K ] , there exist T = T ( δ, k ) such that
[Equation]
Next, we impose conditions on the dataset to prove a stronger global convergence results on the ﬂow.
We say that the dataset ( X , ¯ y ) is orthogonal separable among multiple classes if for all n, n
[Equation]
∈ [ N ] ,
[Equation]
For orthogonal separable dataset among multiple classes, the subgradient ﬂow for the non-convex problem (4) can ﬁnd the global optimum of (5) up to a scaling constant.
Theorem 2 Suppose that ( X , ¯ y ) ∈
R N × d ×
[ K ] N is orthogonal separable among multiple classes.
Consider the non-convex subgradient ﬂow applied to the non-convex problem (4) .
Suppose that the initialization is sufﬁciently close to the origin and scaled as in Theorem 1.
Then, the non-convex subgradient ﬂow converges to the global optimum of the convex program (7) and hence the non-convex objective (5) up to scaling.
Therefore, the above result characterizes the implicit regularization of unregularized gradient ﬂow as convex regularization , i.e., group (cid:96) 1 norm, in the convex formulation (7).
It is remarkable that group sparsity is enforced by small initialization magnitude with no explicit form of regularization.
2. C ONVEX G EOMETRY OF N EURAL G RADIENT F LOW
Suppose that λ ∈ R N .
Here we provide an interesting geometric interpretation behind the formula
[Equation]
which describes a dual extreme point to which hidden neurons approach to as predicted by
Theorem 1.
We now explain the geometric intuition behind this result.
Consider an ellipsoid { Xu : (cid:107) u (cid:107) 2 ≤ 1 } .
A positive extreme point of this ellipsoid along the direction λ is deﬁned by arg max u : (cid:107) u (cid:107) 2 ≤ 1 λ T Xu , which is given by the formula X T λ (cid:107)
X T λ (cid:107) 2 .
Next, we consider the rectiﬁed ellipsoid set Q := { ( Xu )
+ : (cid:107) u (cid:107) 2 ≤ 1 } introduced in (Ergen & Pilanci, 2021a) and shown in Figure 1.
The constraint max u : (cid:107) u (cid:107) 2 ≤ 1 | λ T ( Xu ) + | ≤ 1 on λ is equivalent to λ ∈ Q ∗ .
Here Q ∗ is the absolute polar set of Q , which appears as a constraint in the convex program (8) and is deﬁned as the following convex set
[Equation]
An extreme point of this non-convex body along the direction λ is given by the solution of the problem
[Equation]
Here, ( λ , u ) are primal-dual pairs as they appear in the convex dual program (8).
First, note that a stationary point of gradient ﬂow on the objective in (11) is given by the identity c u ∈ ∂ ◦ u λ T ( Xu ) + where c is a constant.
In particular, by picking the zero as the subgradient of ( x Tn u ) + when x Tn u = 0 ,
[Equation]
Note that the formula cos ∠ ( u , X T D ( u ) λ ) > 1 − δ appearing in Theorem 1 shows that gradient ﬂow reaches the extreme points of projected ellipsoids { D j [Equation]
2 ≤ 1 } in the direction of λ = y k , where D j ∈ P corresponds to a valid hyperplane arrangement.
This interesting phenomenon is depicted in Figures 3 and 4.
The one-dimensional spikes in Figures 1 and 3 are projected ellipsoids.
Detailed setup for Figure 1 to 4 and additional experiments can be found in Appendix F.
[Figure]
Q := { ( Xu ) + : (cid:107) u (cid:107) 2 ≤ 1 } and its extreme points (spikes).
[Figure] [Figure] [Figure]
[Figure]
ˆ w 1 ,i = w 1 ,i (cid:107) w 1 ,i (cid:107) 2 is the normalized vector of the i -th hidden neuron in the ﬁrst layer.
C ONVEX MAX - MARGIN PROBLEM
In this section, we consider the equivalent convex model of the max-margin problem and its optimality conditions.
We primarily focus on the binary classiﬁcation problem for simplicity, which are later extended to the multi-class case.
We can reformulate the nonconvex max-margin problem (2) as
[Equation]
where Y = diag ( y ) .
This is a nonconvex optimization problem due to the ReLU activation and the two-layer structure of neural network.
Analogous to the convex formulation introduced in (Pilanci & Ergen, 2020) for regularized training problem of neural network, we can provide a convex optimization formulation of (13) and derive the dual problem.
Proposition 2 The problem (13) is equivalent to
[Equation]
The dual problem of (14) is given by
[Equation]
The following proposition gives a characterization of the KKT point of the non-convex max-margin problem (2).
The deﬁnition of B -subdifferential can be found in Appendix A.
Proposition 3 Let ( W 1 , w 2 , λ ) be a KKT point of the non-convex max-margin problem (2) (in terms of B-subdifferential).
Suppose that w 2 ,i (cid:54) = 0 for certain i ∈ [ m ] .
Then, there exists a diagonal matrix ˆ D i ∈ R N × N satisfying
[Equation]
such that
[Equation]
Based on the characterization of the KKT point of the non-convex max-margin problem (2), we provide an equivalent condition to ensure that it is also the KKT point of the convex max-margin problem (14).
Theorem 3 The KKT point of the non-convex max-margin problem (13) (in terms of B-subdifferential) corresponds to a KKT point of the convex max-margin problem (14) if λ is dual feasible, i.e.,
[Equation]
This condition is equivalent to for all D j ∈ P , the dual variable λ satisﬁes that
[Equation]
3. D UAL FEASIBILITY OF THE DUAL VARIABLE
A natural question arises: is it possible to examine whether λ is feasible in the dual problem?
We say the dataset ( X , y ) is orthogonal separable if for all n, n (cid:48) ∈
[ N ] ,
[Equation]
For orthogonal separable data, as long as the induced diagonal matrices in Proposition 3 cover the positive part and the negative part of the labels, the KKT point of the non-convex max-margin problem (2) is the KKT point of the convex max-margin problem (14).
Proposition 4 Suppose that ( X , y ) is orthogonal separable.
Suppose that the KKT point of the nonconvex problem include two neurons ( w 1 ,i + , w 2 ,i + ) and ( w 1 ,i − , w 2 ,i − ) such that the corresponding diagonal matrices ˆ D i +
and ˆ D i − deﬁned in Proposition 3 satisfy that
[Equation]
Then, the dual variable λ is dual feasible, i.e., satisfying (16) .
The spike-free matrices discussed in (Ergen & Pilanci, 2021a) also makes examining the dual feasibility of λ easier.
The deﬁnition of spike-free matrices can be found in Appendix A
Proposition 5 Suppose that X is spike-free.
Suppose that the KKT point of the non-convex problem include two neurons ( w 1 ,i + , w 2 ,i + ) and ( w 1 ,i − , w 2 ,i − ) such that the corresponding diagonal matrices ˆ
D i +
and ˆ D i − deﬁned in Proposition 3 satisfy that
[Equation]
Then, the dual variable λ is dual feasible, i.e., satisfying (16) .
Remark 1 For the spike-free data, the constraint on the dual problem is equivalent to
[Equation]
or equivalently
[Equation]
S UB - GRADIENT FLOW DYNAMICS OF LOGISTIC LOSS
In this section, we consider the following sub-gradient ﬂow of the logistic loss (3)
[Equation]
18) where the n -th entry of (cid:101) λ ( t )
∈
R N is deﬁned
[Equation]
For simplicity, we omit the term ( t ) .
For instance, we write w 1 ,i =
w 1 ,i ( t ) .
To be speciﬁc, when w T 1 ,i x n = 0 , we select 0 as the subgradient of w 2 ,i ( w T 1 ,i x n ) + with respect to w 1 ,i .
Denote σ i = sign ( Xu i ) .
For σ ∈ { 1 , − 1 , 0 } N , we deﬁne
[Equation]
For simplicity, we also write
[Equation] [Equation]
Then, we can rewrite sub-gradient ﬂow of the logistic loss (3) as follows:
[Equation] [Equation]
Assume that the neural network is scaled at initialization, i.e., (cid:107) w 1 ,
i
[Equation]
[Equation] 22 = w 22 ,i (0) for i ∈ [ m ] .
Then, the neural network is scaled for t ≥ 0 .
Lemma 1 Suppose that (cid:107) w 1 ,i (0) (cid:107) 2 = | w 2 ,i (0) | > 0 for i ∈ [ m ] .
Then, for any t > 0 , we have (cid:107) w 1 ,i ( t ) (cid:107) 2 = | w 2 ,i ( t ) | > 0 .
According to Lemma 1, for all t ≥ 0 , sign ( w 2 ,i ( t ))
=
sign ( w 2 ,i (0)) .
Therefore, we can simply write s i
=
s i ( t )
= sign ( w 2 ,i ( t )) .
As the neural network is scaled for t ≥ 0 , it is interesting to study the dynamics of w 1 ,i in the polar coordinate.
We write w 1 ,i ( t ) =
e r i ( t ) u i ( t ) , where (cid:107) u i ( t ) (cid:107) 2 = 1 .
The gradient ﬂow in terms of polar coordinate writes
[Equation]
Let x max = max i ∈ [ n ] (cid:107) x
i (cid:107) 2 .
Deﬁne g min to be
[Equation]
0 , where we denote (24)
[Equation]
As the set Q ⊆ { 1 , − 1 , 0 } N is ﬁnite, we note that g min > 0 .
We note that when max n ∈ [ N ] | q n | ≈ 0 , we have (cid:101) λ
≈ y 4 .
The following lemma shows that with initializations sufﬁciently close to 0 , (cid:107) g ( u ( t ) , (cid:101) λ ( t )) − g ( u ( t ) , y / 4) (cid:107) 2 and (cid:13)(cid:13)(cid:13) ddt g ( u ( t ) , (cid:101) λ ( t ))
(cid:13)(cid:13)(cid:13) 2 can be very small.
Lemma 2 Suppose that T > 0 and δ > 0 .
Suppose that ( u ( t ) , r ( t )) follows the gradient ﬂow (23) with s = 1 and the initialization u (0) =
u 0 and r (0)
=
r 0 .
Suppose that r 0 is sufﬁciently small.
Then, the following two statements hold.
• For all t ≤ T , we have (cid:107) g
( u ( t ) , (cid:101) λ ( t ))
− g ( u ( t ) , y / 4) (cid:107) 2 ≤ g min δ 8 .
• For t ≤ T such that sign ( Xu ( s )) is constant in a small neighbor of t , we have (cid:13)(cid:13)(cid:13) ddt g ( u ( t ) , (cid:101) λ ( t ))
(cid:13)(cid:13)(cid:13) 2 ≤ g 2min δ 16 .
Based on the above lemma on the property of g ( u ( t ) , (cid:101) λ ( t )) , we introduce the following lemma to upper-bound the time such that cos ∠ ( u ( t ) , g ( u ( t ) , (cid:101) λ ( t ))) approaches 1 − δ or sign ( Xu ( t )) changes.
Lemma 3 Let δ ∈ (0 , 1) .Suppose
that u 0 satisﬁes that (cid:107) u 0 (cid:107) 2 = 1 and (cid:101) λ (0) T ( Xu 0 ) + > 0 .
Suppose that ( u ( t ) , r ( t )) follows the gradient ﬂow (23) with s = 1 and the initialization u (0) =
u 0 and r (0)
=
r 0 .
Let v ( t ) =
g ( u ( t ) , (cid:101) λ ( t ))
(cid:107) g
( u ( t ) , (cid:101) λ ( t ))
(cid:107) 2 .
We write v 0 = v (0) , σ 0 = σ (0) and g
= (cid:107) g ( σ 0 , y / 4) (cid:107) 2 .
Denote
[Equation]
For c ∈ (0 , 1 − δ ] , deﬁne
[Equation]
Suppose that r 0 is sufﬁciently small such that the statements in Lemma 2 holds for T
= T ∗ .
Then, at least one of the following event happens
• There exists a time T such that we have sign ( Xu ( t ))
=
sign ( Xu 0 ) for t ∈ [0 , T ) and sign ( Xu ( t )) (cid:54) =
sign ( Xu 0 ) .
Let u 1 = u
( T ) and v 1 =
lim t → T − 0 v ( t ) .
If u T 1 v 1 ≤ 1 − δ , then the time T satisﬁes that T ≤ T shift ( v T 1 u 1 ) .
Otherwise, there exists a time T (cid:48) satisfying T (cid:48) ≤ T ∗ , such that we have sign ( Xu ( t ))
=
sign ( Xu 0 ) for t ∈ [0 , T (cid:48) ] and u ( T (cid:48) )
T v ( T (cid:48) ) ≥ 1 − δ .
• There exists a time T ≤ T ∗ , such that we have sign ( Xu ( t ))
=
sign ( Xu 0 ) for t ∈ [0 , T ] and u ( T ) T v ( T ) ≥ 1 − δ .
Corollary 1 Suppose that there exists a time T such that we have sign ( Xu ( t ))
=
sign ( Xu 0 ) for t ∈ [0 , T ) and sign ( Xu ( t )) (cid:54) =
sign ( Xu 0 ) .
If T > T shift ( v T 1 u 1 ) =
[Equation]
(cid:19) , then, we have u T 1 v 1 > 1 − δ .
Proposition 6 Consider the sub-gradient ﬂow (23) with s = 1 and the initialization u (0) =
u 0 and r (0)
=
r 0 .
Here at initilization the neuron u 0 satisﬁes that (cid:107) u 0 (cid:107) 2 = 1 and y T ( Xu 0 ) + > 0 .
Let v ( t ) =
g ( u ( t ) , (cid:101) λ ( t ))
(cid:107) g
( u ( t ) , (cid:101) λ ( t ))
(cid:107) 2 .
For any δ > 0 , for sufﬁciently small r 0 , there exists a time T
=
O (log( δ − 1 ))
such that u ( T ) T v ( T ) ≥ 1 − δ and cos ∠ ( u ( T ) , g ( u ( T ) , y )) ≥
1 − δ .
Remark
The statement of proposition is similar to Lemma 4 in (Maennel et al., 2018).
However, their proof contains a problem because they did not consider the change of sign ( Xw ) along the gradient ﬂow.
Our proof in Appendix D.4 corrects this error.
We next study the properties of orthogonal separable datasets.
Denote B = { w ∈ R d : (cid:107) w (cid:107) 2 ≤ 1 } .
The following lemma give a sufﬁcient condition on w to satisfy the condition in Proposition 4.
Lemma 4 Assume that ( X , y ) is orthogonal separable.
Suppose that w ∈ B is a local maximizer of y T ( Xw ) + in B and ( Xw ) + (cid:54) = 0 .
Then, (cid:104) w , x n (cid:105) > 0 for n ∈ [ N ] such that y n = 1 .
Suppose that w ∈ B is a local minimizer of y T ( Xw ) + in B and ( Xw ) + (cid:54) = 0 .
Then, (cid:104) w , x n (cid:105) > 0 for n ∈ [ N ] such that y n = − 1 .
We show an equivalent condition of u ∈ B being the local maximizer/minimizer of y T ( Xu ) + in B .
Proposition 7 Assume that ( X , y ) is orthogonal separable.
Then, u ∈ B is a local maximizer of y T ( Xu ) + in B is equivalent to cos ∠ ( u , g ( u , y ))
= 1 .
Similarly, u ∈ B is a local minimizer of y T ( Xu ) + in B is equivalent to cos ∠ ( u , g ( u , y ))
= − 1 .
Based on Proposition 4 and 7, we present the main theorem.
Theorem 4 Suppose that the dataset is orthogonal separable and θ ( t ) follows the gradient ﬂow.
Suppose that the neural network is scaled at initialization, i.e., (cid:107) w 1 ,i (0) (cid:107) 2 = | w 2 ,i (0) | for all i ∈ [ m ] .
For almost all initializations which are sufﬁciently close to zero, the limiting point of θ ( t ) (cid:107) θ ( t ) (cid:107) 2 is θ ∗ (cid:107) θ ∗ (cid:107) 2 , where θ ∗ is a global minimizer of the max-margin problem (2) .
We present a sketch of the proof.
According to Proposition 6, for initialization sufﬁciently close to zero, there exist two neurons and time T + , T − > 0 such that cos ∠ ( w 1 ,i + ( T + ) , g ( w 1 ,i + ( T + ) , y )) ≥ 1 − δ and cos ∠ ( w 1 ,i − ( T − ) , g ( w 1 ,i − ( T − ) , y ))
≤ − (1 − δ ) .
This implies that w 1 ,i + ( T + ) and w 1 ,i − ( T + ) are sufﬁciently close to certain stationary points of gradient ﬂow maximizing/minimizing y T ( Xu + ) over B , i.e., { u ∈ B| cos( u , g ( u , y ))
= ± 1 } .
As the dataset is orthogonal separable, from Proposition 7 and Lemma 4, the induced masking matrices ˆ D
i + ( T + ) and ˆ D i − ( T − )
by w 1 ,i + ( T + ) / w 1 ,i − ( T − ) in Proposition 3 satisfy that ˆ D i + ( T + ) ≥ diag ( I ( y = 1)) and ˆ D i − ( T − ) ≥ diag ( I ( y = − 1)) .
According to Lemma 3 in (Phuong & Lampert, 2021), for t ≥ max { T + , T − } , we also have ˆ D i + ( t ) ≥ diag ( I ( y = 1)) and ˆ D i − ( t ) ≥ diag ( I ( y = − 1)) .
According to Theorem 3 and Proposition 4, the KKT point of the non-convex problem (2) that gradient ﬂow converges to corresponds to the KKT point of the convex problem (14).
C ONCLUSION
We provide a convex formulation of the non-convex max-margin problem for two-layer ReLU neural networks and uncover a primal-dual extreme point relation between non-convex subgradient ﬂow.
Under the assumptions on the training data, we show that ﬂows converge to KKT points of the convex max-margin problem, hence a global optimum of the non-convex objective.
A CKNOWLEDGEMENTS
This work was partially supported by the National Science Foundation under grants ECCS-2037304,
DMS-2134248, and US Army Research Ofﬁce.
A D EFINITIONS AND NOTIONS
We introduce several useful deﬁnitions and notions which will be utilized in the proof.
A. D EFINITIONS
Deﬁnition 1 Let O ⊂ R n be an open set and let F : O → R be locally Lipschitz continuous at x ∈ O .
Let D F be the differentiable points of F in O .
The B -subdifferential of F at x is deﬁned by
[Equation]
The set ∂ ◦ F ( x ) = co( ∂ B F ( x )) is called Clarke’s subdifferential, where co denotes the convex hull.
Deﬁnition
A matrix A is spike-free if and only if the following conditions hold: for all (cid:107) u (cid:107) 2 ≤ 1 , there exists (cid:107) z (cid:107) 2 ≤ 1 such that
[Equation]
This is equivalent to say that
[Equation]
A. N OTIONS
We use the following letters for indexing.
• The index n is for the n -th data sample x n .
• We use the index i to represent the i -th neuron-pair ( w 1 ,i , w 2 ,i ) .
• The index j is for the j -th masking matrix D i ∈ P .
B P ROOFS IN S ECTIONB. P ROOF FOR P ROPOSITION
Consider the following loss function ˜
l : R N × R N → R ∪ { + ∞}
[Equation]
y [Equation] n ∈ [ N ] , +
[Equation]
otherwise .
For a given y ∈ { 1 , − 1 } N , ˜ (cid:96) ( z , y ) is a convex loss function of z .
The non-convex max-margin is equivalent to
[Equation] [Equation]
According to Appendix A.13 in (Pilanci & Ergen, 2020), the problem (32) is equivalent to
[Equation] [Equation]
This is equivalent to (14).
For ﬁxed y ∈ { 1 , − 1 } N , the Fenchel conjugate function of ˜ (cid:96) ( z , y ) with respect to z can be computed by
[Equation] otherwise .
According to Theorem 6 in (Pilanci & Ergen, 2020), the dual problem of (14) writes
[Equation]
which is equivalent to
[Equation]
By taking λ = − ˆ λ , we derive (15).
This completes the proof.
B. P ROOF FOR P ROPOSITION
For the non-convex max-margin problem (13), consider the Lagrange function
[Equation]
where Y λ (cid:23) 0 .
The KKT point of the non-convex max-margin problem (13) (in terms of Bsubdifferential) satisﬁes
[Equation]
The KKT condition on the i -th column of W 1 is equivalent to
[Equation]
(38) where g n,i ∈
∂ B ( z )
+ | z = x
Tn w 1 ,i .
In other words, we have
[Equation]
Let ˆ D i =
diag ([ g 1 ,i , . .
. , g N,i ]) .
Then, we can write that
[Equation]
[Equation]
From the deﬁnition of g n,i , we have g n,
i
[Equation]
Therefore, we can compute that
[Equation]
In summary, we have
[Equation]
Suppose that w 2 ,i (cid:54) = 0 .
This implies that
[Equation] [Equation]
This completes the proof.
B. P ROOF FOR T HEOREM
P ROOF We can write the Lagrange function for the convex max-margin problem (14) as
[Equation]
) where z j , z (cid:48) j ∈ R N satisﬁes that z j ≥ 0 , z (cid:48)
j ≥ 0 for j ∈ [ p ] and λ ∈ R N satisﬁes that diag ( y ) λ ≥ 0 .
The KKT point shall satisfy the following KKT conditions:
[Equation]
Let ( W 1 , w 2 , λ ) be the KKT point of the non-convex problem (2) and λ satisﬁes (17).
Let ˆ D i be the diagonal matrix deﬁned in Proposition 3 with respect to w 1 ,i and denote ¯ P
= {
ˆ D i | i ∈ [ m ] } .
Without the loss of generality, we may assume that { ¯ D i } mi =1 are different.
(Otherwise, we can merge two neurons w 1 ,i 1 and w 1 ,i 2 with ¯ D
i 1 =
¯ D
i 2 together.)
Suppose that D j ∈ ˆ P , i.e., D j = ˆ D i for certain i ∈ [ m ] .
By letting u (cid:48)
j = w 1 ,i w 2 ,i , z (cid:48)
j = 0 , u j = − w 1 ,i w 2 ,i and z j = 0 , the following identities hold.
[Equation]
[Equation]
Therefore, for index
j satisfying D i ∈
ˆ P , the ﬁrst two KKT conditions in (46) hold.
For D j / ∈ ˆ P , we can let u j = u (cid:48)
j = 0 .
As λ satisﬁes (17), we have
[Equation]
According to Lemma 4 in (Pilanci & Ergen, 2020), this implies that there exist z j , z (cid:48)
j ≥ 0 such that
[Equation]
[Equation]
Therefore, the ﬁrst two KKT conditions in (46) hold.
From our choice of u j , z j , u (cid:48)
j , z j , the last two KKT conditions in (46) hold.
We also note that
[Equation]
As ( W 1 , w 2 , λ ) is the KKT point of the non-convex problem, the third KKT condition in (46) holds.
This completes the proof.
C P ROOFS IN S ECTION 3.
In this section, we present several proofs for propositions in Section 3.1.
C. P ROOF FOR P ROPOSITION
We start with two lemmas.
Lemma 5 Suppose that u 0 =
X T ˆ D 0 λ
and (cid:107) u 0 (cid:107) 2 ≤ 1 .
For any masking matrix D j ∈ P such that ( D j − ˆ D 0 ) I ( λ > 0) = 0 , we have
[Equation]
P ROOF
According to Lemma 4 in (Pilanci & Ergen, 2020), the constraint (52) is equivalent to that there exist z j ∈ R N such that z j ≥ 0 and
[Equation]
Consider the index n ∈ [ N ] such that ( D j − ˆ D 0 ) nn (cid:54) = 0 .
As ( D j − ˆ D 0 ) I ( λ > 0)
= 0 , we have λ n ≤ 0 .
We let ( z j ) n = − λ n .
If ( ˆ D 0 ) nn = 0 , then we have ( D j ) nn = 1 and
[Equation]
If ( ˆ D 0 ) nn = 1 , then we have ( D j ) nn = 0 and
[Equation]
D
[Equation]
ˆ
[Equation]
nn λ n x
[Equation]
For other index n ∈ [ N ] , we simply let ( z j ) n = 0 .
Then, we have
[Equation]
n .
[Equation]
Based on our choice of z j , we have z j ≥ 0 and for n ∈ [ N ]
[Equation]
[Equation]
This implies that
[Equation]
[Equation]
Hence, we have
[Equation]
[Equation]
Therefore, (cid:107) X T D j λ + X T (2 D j − I )
z j (cid:107) 2 ≤ 1 .
Lemma 6 Suppose that the data is orthogonal separable and Y λ ≥ 0 .
Suppose that u 0 =
X T ˆ D 0 λ
and (cid:107) u 0 (cid:107) 2 ≤ 1 .
For any masking matrix D j such that ˆ D 0 − D j ≥ 0 , we have (cid:107) X T D j λ (cid:107) 2 ≤ (cid:107) u 0 (cid:107) 2 ≤ 1 .
Therefore, (52) holds.
P ROOF
We note that u 0 = X T ( ˆ D 0 − D j ) λ + XD j λ .
Denote a = X T ( ˆ D 0 − D j ) λ and b =
X T D j λ .
We note that
[Equation]
As diag ( y ) λ ≥ 0 , λ n has the same signature with y n .
Therefore, from the orthogonal separability of the data, we have
[Equation] [Equation]
This immediately implies that a T b ≥ 0 .
Therefore,
[Equation]
[Equation]
This completes the proof.
Based on Lemma 5 and Lemma 6, we present the proof for Proposition 3.
Let u + = w 1 ,i + w 2 ,i + .
From the proof of Proposition 3, we note that (cid:107) u
+ (cid:107) 2 = 1 .
For any masking matrix D j ∈ P , let ˜ D
= ˆ D i + D j .
As ˆ D i + ≥ ˜ D , according to Lemma 6, we have
[Equation]
As Y λ ≥ 0 and ˆ D i + ≥ diag ( I ( y = 1)) , we have ( D j − ˜ D ) I ( λ > 0)
=
D j
( I − ˆ D
i + ) I ( λ > 0)
= 0 .
From Lemma 5, we note that λ satisﬁes (52).
Similarly, we can show that − λ also satisﬁes (52).
This completes the proof.
C. P ROOF FOR P ROPOSITION
P ROOF Note that Y λ ≥ 0 .
Let Y + = diag
( I ( y = 1)) and Y − =
diag ( I ( y = − 1)) .
We claim that
[Equation]
Firstly, we note that
[Equation]
This implies that max (cid:107) u (cid:107)≤ 1 λ T ( Xu ) + ≤ max (cid:107) u (cid:107)≤ 1 λ T Y + ( Xu ) + .
On the other hand, suppose that u ∈ arg max (cid:107) u (cid:107)≤ 1 λ T Y + ( Xu ) + .
As X is spike-free, there exists z such that (cid:107) z (cid:107) 2 ≤ 1 and Xz =
( Xu )
+ .
Therefore, we have
[Equation]
[Equation]
This implies that max (cid:107) u (cid:107)≤ 1 λ T ( Xu )
+ ≥ max (cid:107) u (cid:107)≤ 1 λ T Y + ( Xu )
+ .
For any D j ∈ P with D j ≥ Y + .
We note that
[Equation]
Combining with (65), this implies that max (cid:107) u (cid:107)≤ 1 λ T (
[Equation]
+ [Equation] max (cid:107) u (cid:107)≤ 1 λ T D
( Xu )
+ .
Let us go back to the original problem.
Let u + = w 1 ,i + w 2 ,i + .
We note that ( Xw + ) =
ˆ D i + Xw +
=
ˆ D i + XX T
ˆ D
i + λ .
Therefore, we have
[Equation]
Thus, for any (cid:107) u (cid:107) 2 ≤ 1 , suppose that ( Xu ) + =
Xz , where (cid:107) z (cid:107) 2 ≤ 1 .
Then, we have
[Equation]
Therefore, max (cid:107) u (cid:107)≤ [Equation] ( Xu
[Equation]
1 λ T D + ( Xu ) + ≤ 1 .
Similarly, we have
[Equation]
This completes the proof.
D P ROOFS IN S ECTIOND. P ROOF FOR L EMMAP ROOF
According to the sub-gradient ﬂow (22), we can compute that
[Equation]
Let T
=
sup
{ T |(cid:107)
w 1 ,i ( t ) (cid:107) 2 = | w 2 ,i ( t ) | > 0 , ∀ i ∈ [ n ] , t ∈ [0 , T ) } .
For t ∈ [0 , T 0 ) , as the neural network is scaled, it is sufﬁcient study the dynamics of w 1 ,i in the polar coordinate.
Let us write w 1 ,i ( t ) =
e r i ( t ) u i ( t ) , where (cid:107) u i ( t ) (cid:107) 2 = 1 .
Then, in terms of polar coordinate, the projected gradient ﬂow follows
[Equation]
Without the loss of generality, we may assume that w 2 ,i (0) (cid:54) = 0 for i ∈ [ m ] .
Denote
[Equation]
From the deﬁnition of (cid:101) λ , we have (cid:107) (cid:101) λ (cid:107) ∞ ≤ 1 / 4 .
Therefore, we have
[Equation]
Therefore, for ﬁnite t > 0 , we have
[Equation]
74) which implies that | w 2 ,i ( t ) | > 0 .
This implies that T 0 = ∞ .
D. P ROOF OF L EMMA
P ROOF
As we have (cid:107) w 1 ,i (cid:107) 2 = | w 2 ,i | , for n ∈ [ N ] , we can compute that [Equation]
[Equation]
Note that ˜ λ n = − y n (cid:96) (cid:48) ( q n ) and y n 4 = − y n (cid:96) (cid:48) (0) .
As (cid:96) (cid:48) is 14 -Lipschitz continuous, we have
[Equation]
For any ˆ σ ∈ Q , as ˜ λ n ∈ [0 , 1 / 4] for n ∈ [ N ] , we have
[Equation]
(77) where c 1 = 14 (cid:107) X (cid:107) 2 F > 0 is a constant.
Therefore, we can bound (cid:107) g
(ˆ σ , (cid:101) λ ( t ))
(cid:107) by (cid:107) g (ˆ σ, (cid:101)
[Equation]
where we let
[Equation]
Let r ( t ) =
max i ∈ [ m ] r i ( t ) .
We note that
[Equation]
where c 2 = max { nc 1 , d max } > 0 is a constant.
If we start with r (0) (cid:28) 0 , then, r ( t ) cannot grow much faster than c 2 t .
Let ˜ r ( t ) satisfy the following ODE:
[Equation]
The solution is given by
[Equation]
where a > 0 is a parameter depending on the initialization.
For any initial r (0) [Equation] we have a unique a satisfying ˜ r a (0) =
r (0) .
Therefore, we have r ( t ) ≤ ˜ r a ( t ) and
[Equation]
According to the bound (83)
[Equation]
by choosing a sufﬁciently small r 0 , (which leads to a sufﬁciently small a )
[Equation]
such that
[Equation]
Therefore, for t ≤ T [Equation] we have
[Equation] [Equation]
Hence, we have
[Equation]
[Equation]
We can compute that
[Equation]
As l (2) ( q ) ∈ (0 , 1 / 4]
[Equation]
we can compute that
[Equation] [Equation]
Therefore
[Equation]
we have
[Equation]
Suppose that sign ( Xu ( s ))
= σ ( t ) holds for s in a small neighbor of t .
Then, we have
[Equation]
This completes the proof.
D. P ROOF OF L EMMA
P ROOF Let T
=
sup
{ T | sign ( Xu ( t ))
=
sign ( Xu 0 ) , ∀ t ∈ [0 , T ) } .
We analyze the dynamics of u ( t ) in the interval [0 , min { T 0 , T ∗ } ] .
For t ≤ min { T 0 , T ∗ } , as the statements in Lemma 2 hold, we can compute that
[Equation]
Here we utilize that g 0 ≥ g min , where g min is deﬁned in (24).
Let z ( t ) satisﬁes the ODE
[Equation]
[Equation]
with initialization z (0) = v T 0 u 0 .
Then, we note that
[Equation] [Equation]
where c 3 = (cid:18) √ 1 − δ/v T 0 u 0 −
[Equation]
(cid:19) −.
We can compute that z ( T
[Equation]
= [Equation] According to the comparison theorem, for t ≤ min { T 0 , T 3 } , we have
[Equation]
We ﬁrst consider the case where T 0 = ∞ .
As T 0 = ∞ , we have
[Equation]
Therefore, the second event holds for T ≤ T ∗ .
Otherwise, we have T 0 < ∞ .
Recall that u 1 = u
( T 0 ) and v 1 = lim t
↑ T 0 v ( t ) .
Let T 1 = sup
{ T | v ( t ) T u ( t ) <
v T 1 u 1 , ∀ t ∈ [0 , T ) } and T 2 = sup { T | v ( t ) T u ( t ) < 1 − δ, ∀ t ∈ [0 , T ) } .
If T 2 ≤ T 0 , for t ∈ [0 , T 2 ] , we have
[Equation]
Therefore, v ( t ) T u ( t ) monotonically increases in [0 , T 2 ] .
As v ( t ) T u ( t ) ≥ z ( t ) for t ∈ [0 , T 0 ] , we have that z ( T 2 ) ≤ v ( T 2 ) T u ( T 2 )
=
1 − δ
=
z ( T 3 ) .
Hence, we have T 2 ≤ T ∗ .
Therefore, the second condition of the ﬁrst event holds at T
=
T 2 .
Then, we consider the case where T 2 ≥ T 0 .
For t ≤ T 0 , we have v ( t ) T u ( t ) ≤ 1 − δ .
This implies that v T 1 u 1 ≤ 1 − δ .
Apparently, we have T 1 ≤ T 0 .
If T 1 < T 0 , as T 0 ≤ T 2 , for t ∈ [0 , T 0 ] , the inequality
(97) holds.
This implies that lim t → T 0 − 0 v ( t ) T u ( T 0 ) > v ( T 1 ) T u ( T 1 ) =
lim t → T 0 − 0 v ( t ) T u ( T 0 ) , which leads to a contradiction.
Therefore, we have T
=
T 1 .
We note that
[Equation]
As u ( t ) T g ( u ( t ) , (cid:101) λ ( t )) ≥ z ( t ) for t ∈ [0 , T 0 ] , we have that z ( T 1 ) ≤ u T 1 v 1 ≤ z ( T 4 ) .
Hence, we have
T
= T 1 ≤ T shift ( u T 1 v 1 ) .
This completes the proof.
D. P ROOF OF P ROPOSITION
We ﬁrst introduce a lemma.
Lemma 7 Let a , b ∈ R d and 0 < δ < c .
Suppose that (cid:107) a − b (cid:107) 2 ≤ δ and (cid:107) a (cid:107) 2 ≥ c .
Then, we have (cid:13)(cid:13)(cid:13)(cid:13) a (cid:107) a (cid:107) 2 − b (cid:107) b (cid:107) 2 (cid:13)(cid:13)(cid:13)(cid:13) 2 ≤ 2 δ c .
P ROOF
As δ < c , we have (cid:107) b (cid:107) 2 > (cid:107) a (cid:107) 2 − (cid:107) a − b (cid:107) 2 ≥ c − δ > 0 .
We ﬁrst note that (cid:12) (cid:12) (cid:107) a (cid:107) − 1 2 − (cid:107) b (cid:107) − 1 2 (cid:12) (cid:12) =
|(cid:107) a (cid:107) 2 − (cid:107) b (cid:107)
| (cid:107) a (cid:107)
[Equation]
[Equation]
Therefore, we can compute that
[Equation]
This completes the proof.
Then we present the proof of Proposition 6.
P ROOF
As y T ( Xu 0 ) + > 0 , with sufﬁciently small initialization and sufﬁciently small δ > 0 , we also have (cid:101) λ (0) T ( Xu 0 )
+ ≥ y T
( Xu 0 ) + / 4 − (cid:107) X
(cid:107) 2 (cid:107) (cid:101) λ
(0) − y / 4 (cid:107) 2 > 0 .
We prove that there exists a time T such that u ( T ) T v ( T ) ≥ 1 − 34 δ by contradiction.
Denote v 0 = v (0) .
For all possible values of (cid:107) g
( u , y / 4) (cid:107) 2 , we can arrange them from the smallest to the largest by g (1) < g (2)
< · · · < g ( p ) .
Let T i = 1 2 √ 1 − δ/ 8 g ( i ) (cid:18) log √
[Equation]
√
[Equation]
√
[Equation]
[Equation]
1 ( i )
[Equation]
√ 1 − δ/ 8 − g − 1 ( i ) v T 0 u(cid:19) and T = (cid:80) pi
=1 T i .
Suppose that r 0 is sufﬁciently small such that statements in Lemma 2 holds for T .
According to Lemma 3, we can ﬁnd 0 =
t 0 < t 1 < . . .
such that for i = 1 , . .
, sign ( Xu ( t )) is constant on [ t i − 1 , t i ) and sign ( Xu ( t i − 1 ))
(cid:54)
=
sign ( Xu ( t i )) .
We write u i =
u
( t i ) , g i =
(cid:107) g
( u ( t i ) , y / 4) (cid:107) 2 , g − i =
lim t ↑ t i g ( u ( t ) , (cid:101) λ ( t )) , g i =
g ( u ( t i ) , (cid:101) λ ( t )) , (102) v − i =
g − i (cid:107) g − i (cid:107)
and v i
=
g i (cid:107) g
i (cid:107) 2 .
We note that g − i =
g
+ i − 1 .
According to Lemma 3, we have t
[Equation]
Here we utilize that g i − 1 ≥ g min
[Equation]
where g min is deﬁned in (24).
This implies that
[Equation]
We can show that for t satisfying t ≥
[Equation]
and t ≤ T , we have (cid:107) g
( u ( t ) , λ ) (cid:107) 2 > g min .
According to Lemma ??
, as g i ≥ g min , we have
[Equation]
This implies that
[Equation]
or equivalently, for any t >
[Equation]
we have
[Equation]
Here we utilize that g ( u ( t ) , λ ( t ))
T u ( t ) is continuous w.r.t.
t .
Therefore, for t ≥
[Equation]
, we have
[Equation]
This implies that
[Equation]
If (cid:107) g
( u ( t ) [Equation] λ ) (cid:107) 2 = g min
[Equation]
as the statements in Lemma 2 hold
[Equation]
we can compute that
[Equation]
which implies that
[Equation]
Therefore, we have
[Equation]
g ( u ( t ) , (cid:101) λ ( t )))
T u
[Equation]
This leads to a contradiction.
Analogously, we can show that for t ≥ (cid:80) ij =1 T i , we have (cid:107) g
u ( t ) , y / 4) (cid:107) 2 > g ( i ) .
Thus, by taking t ≥ (cid:80) pi =1 T i , we have (cid:107) g
( u ( t ) , y / 4) (cid:107) 2 > g ( p ) =
g max .
However, from the deﬁnition of g max , we have (cid:107) g
( u ( t ) , y / 4) (cid:107) 2 ≤ g max .
This leads to a contradiction.
Therefore, there exists a time
T = (cid:80) pi
=1 T i =
O (log δ − 1 ) such that v ( T ) T u ( T ) ≥ 1 − 34 δ .
We note that (cid:107) g ( u ( T ) , y / 4) (cid:107) 2 ≥ g min .
As the statements in Lemma (2) hold, we have
[Equation]
According to Lemma 7, we have
[Equation]
[Equation]
This implies that
[Equation]
Hence, we have
[Equation]
This completes the proof.
D. P ROOF OF L EMMA
P ROOF This is proved in Lemma 2 in (Phuong & Lampert, 2021).
Here we provide an alternative proof.
It is sufﬁcient to prove for the case of local maximizer.
Suppose that w is a local maximizer of y T ( Xw ) + in B .
We ﬁrst consider the case where y T ( Xw ) + > 0 .
If there exists n ∈ [ N ] such that (cid:104) w , x n (cid:105) ≤ 0 and y n = 1 .
Consider v = x n / (cid:107) x n (cid:107) 2 and let w (cid:15) = w +
[Equation]
v (cid:107) w + (cid:15) v (cid:107) 2 , where (cid:15) > 0 .
For index n (cid:48) ∈
[ N ] such that y n (cid:48)
= 1 , as the dataset is orthogonal separable, we have x Tn (cid:48) x n > 0 and
[Equation]
This implies that ( x Tn (cid:48) w (cid:15) )
+ ≥ ( x Tn (cid:48) w )
+ .
For y n (cid:48)
=
− 1 , as the data is orthogonal separable, we note that x Tn (cid:48) x n ≤ 0 and
[Equation]
[Equation]
This implies that ( x Tj w (cid:15) ) + ≤ ( x Tj w ) + .
In summary, we have y T ( X ( w + (cid:15) v ))
+ =
N (cid:88) n = y [Equation] [Equation] (cid:88)
[Equation]
If (cid:104) w , x n (cid:105) < 0 , then w T v < 0 .
This implies that with sufﬁciently small (cid:15) , we have (cid:107) w + (cid:15) v (cid:107) 2 < (cid:107) w (cid:107) 2 = 1 .
Therefore,
[Equation]
119) which leads to a contradiction.
If (cid:104) w , x n (cid:105) = 0 , we note that
[Equation] [Equation]
This implies that
[Equation] [Equation]
We also note that (cid:107) w + (cid:15) v (cid:107) 2 = √ 1 + (cid:15) 2 = 1 + O ( (cid:15) 2 ) .
Therefore, with sufﬁciently small (cid:15) , we have
[Equation]
We then consider the case where y T ( Xw ) + < 0 .
Apparently, we can make y T ( Xw )
+ larger by replacing w by (1 − (cid:15) ) w , where (cid:15) ∈ (0 , 1) , which leads to a contradiction.
Finally, we consider the case where y T ( Xw ) + = 0 .
This implies that (cid:88)
[Equation]
[Equation] As ( Xw ) + (cid:54) = 0 , this implies that there exists at least for one index n ∈ [ N ] such that y n = 1 and x Tn w > 0 .
Let v = x n / (cid:107) x n (cid:107) 2 .
We note that 1 (cid:107) w + (cid:15) v (cid:107) 2 y T
( X ( w + (cid:15) v )) + > 0 for (cid:15) > 0 .
This leads to a contradiction.
D. P ROOF OF P ROPOSITION
It is sufﬁcient to consider the case of the local maximizer.
Denote Q = { σ ∈ {− 1 , 0 , 1 } N | diag ( σ ) ∈
P} .
For σ , σ (cid:48) ∈ Q , we say σ ⊆ σ (cid:48) if for all index n ∈ [ N ] with σ n (cid:54) = 0 , σ (cid:48) n = σ n .
We say σ ∈ Q is open if σ n (cid:54) = 0 for n ∈ [ N ] .
Deﬁne
S σ = { u | sign ( Xu ) = σ } .
We start with the two lemmas.
Lemma 8 Let λ ∈ R N .
Suppose that u 0 satisﬁes that u 0 =
g ( u 0 , λ ) (cid:107) g
( u 0 , λ ) (cid:107) 2 .
Let σ =
sign
u 0 ) .
Then, v ∈ B 2 is a local maximizer of λ T ( Xu ) + in B 2 if for any open σ (cid:48) satisfying σ ⊆ σ (cid:48) , we have (cid:107) g ( σ , y ) (cid:107) 2 = (cid:107) g
( σ (cid:48) , y ) (cid:107) 2 .
P ROOF Suppose that σ is open.
Then, S σ is an open set.
In a small neighbor around u 0 = g ( u 0 , λ ) (cid:107) g
( u 0 , λ ) (cid:107) 2 = g ( σ , λ ) (cid:107) g
( σ , λ ) (cid:107) 2 , λ T ( Xu ) + = u T g ( σ , λ ) is a linear function of u .
The Riemannian gradient of u T g ( σ , λ ) at v is zero.
This implies that v locally maximizes λ T ( Xu ) + .
Suppose that there exists at least one zero in σ .
Consider any v ∈ B satisfying u T 0 v = 0 .
Let (cid:15) >be a small constant such that for any s ∈ (0 , (cid:15) ] , u
+ s v ∈ S σ (cid:48) where σ ⊆ σ (cid:48) .
Let u s = u
+ s v √ 1+ s 2 .
Suppose that (cid:107) g ( σ (cid:48)(cid:48) , λ ) (cid:107) 2 ≤ (cid:107) g ( σ , λ ) (cid:107) 2 for all open σ (cid:48)(cid:48) satisfying σ ⊆ σ (cid:48)(cid:48) .
For any σ (cid:48) with σ ⊆ σ (cid:48) , we construct σ (cid:48)(cid:48) by σ (cid:48)(cid:48) i = − 1 for n ∈ [ N ] such that σ (cid:48) n = 0 and σ (cid:48)(cid:48) n = σ (cid:48) n for n ∈ [ N ] such that σ (cid:48) n = 0 .
We note that (cid:107) g ( σ (cid:48)(cid:48) , λ ) (cid:107) 2 ≥ (cid:107) g ( σ (cid:48) , λ ) (cid:107) 2 .
Thus, (cid:107) g
( σ (cid:48) , λ ) (cid:107) 2 ≤ (cid:107) g ( σ (cid:48)(cid:48) , λ ) (cid:107) 2 ≤ (cid:107) g ( σ , λ ) (cid:107) 2 .
As | λ T ( Xu s ) + | = | g ( σ (cid:48) , λ ) T u s | ≤ (cid:107) g ( σ (cid:48) , λ ) (cid:107) 2 , we have | λ T ( Xu s ) + | ≤ (cid:107) g
( σ (cid:48) , λ ) (cid:107) 2 ≤ (cid:107) g ( σ , λ ) (cid:107) 2 .
Therefore, u is a local maximizer of λ T ( Xu ) + .
Lemma 9 Suppose that the dataset is orthogonal separable.
Let λ ∈ R N satisfy that diag ( y )
λ ≥ 0 .
Suppose that u 0 satisﬁes that u 0 =
g ( u 0 , λ ) (cid:107) g
( u 0 , λ ) (cid:107) 2 .
Then, for any σ (cid:48) satisfying σ ⊆ σ (cid:48) , we have (cid:107) g ( σ (cid:48) , λ ) (cid:107)
= (cid:107) g
( σ , λ ) (cid:107) 2 .
P ROOF
If there exists n ∈ [ N ] such that σ n = 1 and y n = − 1 , as the data is orthogonal separable, we note that
[Equation]
which contradicts with sign ( x Tn g ( σ , λ ))
=
sign ( x Tn u 0 )
= σ n = 1 .
Suppose that there exists n ∈ [ N ] such that σ n and y n = 1 .
Then, as the dataset is orthogonal separable, then, for index n 1 ∈ [ N ] such that σ n 1 = 0 , we note that y n 1 (cid:54) = 1 .
Otherwise,
[Equation]
126) which contradicts with sign ( x Tn 1 g ( σ , λ ))
=
sign ( x Tn 1 u 0 ) =
σ n 1 = 0 .
This also implies that the index set { n ∈ [ N ] | σ n > 0 } include all data with y n = 1 .
If there exists σ (cid:48) such that σ ⊆ σ (cid:48) and (cid:107) g
( σ (cid:48) , λ ) (cid:107) 2 > (cid:107) g
( σ , λ ) (cid:107) 2 .
Then, there exists at least one index n ∈ [ N ] such that σ n ≤ 0 and σ (cid:48) n = 1 .
However, from the previous derivation, we note that y n = − 1 and
[Equation]
which contradicts with σ (cid:48) n = 1 .
By combining Lemma 8 and 9, we complete the proof.
D. P ROOF OF T HEOREM
P ROOF For almost all initialization, we can ﬁnd two neurons such that sign ( w 2 ,i + )
= sign ( y T ( Xw 1 ,
i + ) + )
= 1 and sign ( w 2 ,i − )
= sign ( y T ( Xw 1 ,i − )
+ )
= − 1 at initialization.
By choosing a sufﬁciently small δ > 0 in Proposition 6, there exist two neurons w 1 ,i + , w 1 ,i − and times T + , T − > 0 such that cos ∠ ( w 1 ,i + ( T + ) , g ( w 1 ,i + ( T + ) , y )) > 1 − δ and cos ∠ ( w 1 ,i + ( T + ) , g ( w 1 ,i + ( T + ) , y ))
< − (1 − δ ) .
This implies that w 1 ,i + ( T + ) and w 1 ,i − ( T + ) are sufﬁciently close to certain stationary points of gradient ﬂow maximizing/minimizing y T ( Xu + ) over B , i.e., { u ∈ B| cos( u , g ( u , y ))
= ± 1 } .
As the dataset is orthogonal separable, according to
Lemma 4 and Proposition 7, the corresponding diagonal matrices ˆ D i + ( T + ) and ˆ D i − ( T − ) satisfy that ˆ D i + ( T + ) ≥ diag ( I ( y = 1)) and ˆ D
i − ( T − ) ≥ diag ( I ( y = − 1)) .
According to Lemma 3 in (Phuong & Lampert, 2021), we have ˆ D i + ( t ) ≥ diag ( I ( y = 1)) and D i − ( t ) ≥ diag ( I ( y = − 1)) hold for t ≥ max { T + , T − } .
With t → ∞ , according to Proposition 4, the dual variable λ in the KKT point of the non-convex max-margin problem (13) is dual feasible, i.e., λ satisﬁes (16).
Suppose that θ ∗ is a limiting point of (cid:110) θ ( t ) (cid:107) θ ( t ) (cid:107) 2 (cid:111) t ≥ 0 and λ ∗ is the corresponding dual variable.
From Theorem 1, we note that the pair ( θ ∗ , λ ∗ ) corresponds to the KKT point of the convex max-margin problem (14).
E P ROOFS OF MAIN RESULTS ON MULTI - CLASS CLASSIFICATION E. P ROOF OF P ROPOSITION
The neural network training problem (4) can be separated into K subproblems.
Each of these subproblems corresponds to the neural network training problem (3) for binary classiﬁcation.
For each subproblem, by applying Proposition 2, we complete the proof.
E. P ROOF OF T HEOREM
We note that the neural network training problem (4) can be separated into K subproblems.
Each of these subproblems corresponds to the neural network training problem (3) for binary classiﬁcation.
By applying Proposition 6 with to each subproblem with y =
y k , we complete the proof.
E. P ROOF OF T HEOREM
Similarly, the corresponding non-convex max-margin problem (5) and the convex max-margin problem (7) can be separated into K subproblems.
Each of these subproblems corresponds to the nonconvex max-margin problem (2) and the convex max-margin problem (14) for binary classiﬁcation.
By applying Theorem 4 to each subproblem with y =
y k , we complete the proof.
F N UMERICAL EXPERIMENT F. D ETAILS ON F IGURE
We provide the experiment setting in Figure 1 and 5 as follows.
The dataset is given by X = (cid:20) 1 .− 0 .− 0 .1 .(cid:21)
∈ R 2 × 2 and y =
(cid:20) 1 − 1 (cid:21) ∈
R 2 .
Here we have N = 2 and d = 2 .
We note that this dataset is orthogonal separable but not spike-free.
We plot the ellipsoid set and the rectiﬁed ellipsoid set in Figure 6.
We enumerate all possible hyperplane arrangements in the set P and solve the convex max-margin problem (14) via CVXPY to obtain the following non-zero neurons
[Equation]
We note that the dual problem (15) is equivalent to
[Equation]
The above problem is a second-order cone program (SOCP) and can be solved via standard convex optimization frameworks such as CVX and CVXPY.
We solve (129) to obtain the optimal dual variable λ .
For the geometry of the dual problem, as the dataset is orthogonal separable, the set { λ : max (cid:107) u (cid:107) 2 ≤ 1 | λ T ( Xu ) + | ≤ 1 } reduces to { λ : max (cid:107) u (cid:107) 2 ≤ 1 | λ T ( Xu ∗ 1 ) + | ≤ 1 , λ T ( Xu ∗ 2 ) + | ≤ 1 } , where u ∗ 1 , u ∗ 2 correspond to two vectors at the spikes of the rectiﬁed ellipsoid set.
We draw the sets { λ : max (cid:107) u (cid:107) 2 ≤ 1 | λ T ( Xu ) + | ≤ 1 } , { λ : , the optimal dual variable λ and the direction of y in
Figure 2.
For each D j ∈ P , we solve for the vector u j which maximize/minimize λ T D
j Xu j with the constraints (cid:107) u j (cid:107) 2 ≤ 1 and (2 D j − I )
Xu j ≥ 0 .
We plot the rectiﬁed ellipsoid set { ( Xu ) + |(cid:107) u (cid:107) 2 ≤ 1 } , vectors u j , neurons in the optimal solution to (14) scaled to unit (cid:96) 2 -norm
and the direction of λ in Figure 1.
We note that each neuron u ∗ j in the optimal solution from (14) (scaled to unit (cid:96) 2 -norm) maximize/minimize the corresponding λ T D
j Xu j given (2 D j − I )
Xu ∗ j ≥ 0 .
Then, we consider a two-layer ReLU network with m = 10 neurons and apply the gradient descent method to train on the logistic loss (3).
Let ˆ w 1 ,i =
w 1 ,i (cid:107) w 1 ,i (cid:107) 2 for i ∈ [ m ] .
We plot ˆ w 1 ,i and ( X ˆ w 1 ,i )
+ at iteration { 10 l | l = 0 , . . .
, 4 } along with neurons in the optimal solution to (14) scaled to unit (cid:96) 2 -norm in Figure 5.
Certain neurons do not move, while the activated neurons trained by gradient descent tend to converge to the direction of the neurons in the optimal solution to (14).
We repeat the training on the logistic loss (3) with the gradient descent method several times and we plot the trajectories in Figure 7.
[Figure]
F. E XPERIMENT ON SPIKE - FREE DATASET
We repeat the previous numerical experiment on a non-spike-free dataset: X = (cid:20) 1 .
47 1 .(cid:21) ∈
R 2 × 2 and y =
(cid:20) 11 (cid:21) ∈ R 2 .
Similarly, we plot the ellipsoid set and the rectiﬁed set in Figure 8.
We enumerate all possible hyperplane arrangements in the set P and solve the convex max-margin problem (14) via CVXPY to obtain the following non-zero neuron
[Equation]
We plot the rectiﬁed ellipsoid set { ( Xu ) + |(cid:107) u (cid:107) 2 ≤ 1 } , vectors u j , neurons in the optimal solution to (14) scaled to unit (cid:96) 2 -norm
and the direction of λ in Figure 9.
We also plot ˆ w 1 ,i and ( X ˆ w 1 ,i ) + at iteration { 10 l | l = 0 , . . .
, 4 } along with neurons in the optimal solution to (14) scaled to unit (cid:96) 2 -norm in Figure 10.
[Figure] [Figure]