G RADIENT D ESCENT M AXIMIZES THE M ARGIN OF H OMOGENEOUS N EURAL N ETWORKS
A BSTRACT
In this paper, we study the implicit regularization of the gradient descent algorithm in homogeneous neural networks, including fully-connected and convolutional neural networks with ReLU or LeakyReLU activations.
In particular, we study the gradient descent or gradient ﬂow (i.e., gradient descent with inﬁnitesimal step size) optimizing the logistic loss or cross-entropy loss of any homogeneous model (possibly non-smooth), and show that if the training loss decreases below a certain threshold, then we can deﬁne a smoothed version of the normalized margin which increases over time.
We also formulate a natural constrained optimization problem related to margin maximization, and prove that both the normalized margin and its smoothed version converge to the objective value at a KKT point of the optimization problem.
Our results generalize the previous results for logistic regression with one-layer or multi-layer linear networks, and provide more quantitative convergence results with weaker assumptions than previous results for homogeneous smooth neural networks.
We conduct several experiments to justify our theoretical ﬁnding on MNIST and CIFAR-10 datasets.
Finally, as margin is closely related to robustness, we discuss potential beneﬁts of training longer for improving the robustness of the model.
I NTRODUCTION
A major open question in deep learning is why gradient descent or its variants, are biased towards solutions with good generalization performance on the test set.
To achieve a better understanding, previous works have studied the implicit bias of gradient descent in different settings.
One simple but insightful setting is linear logistic regression on linearly separable data.
In this setting, the model is parameterized by a weight vector w and the class prediction for any data point x is determined by the sign of w (cid:62) x .
Therefore, only the direction w / (cid:107) w (cid:107) 2 is important for making prediction.
Soudry et al.
(2018a;b); Ji & Telgarsky (2018; 2019c); Nacson et al.
(2019c) investigated this problem and proved that the direction of w converges to the direction that maximizes the L 2 -margin while the norm of w diverges to + ∞ , if we train w with (stochastic) gradient descent on logistic loss.
Interestingly, this convergent direction is the same as that of any regularization path : any sequence of weight vectors { w t } such that every w t is a global minimum of the L 2 -regularized loss L ( w )+ λ t 2 (cid:107) w (cid:107) 22 with λ t → 0 (Rosset et al., 2004).
Indeed, the trajectory of gradient descent is also pointwise close to a regularization path (Suggala et al., 2018).
The aforementioned linear logistic regression can be viewed as a single-layer neural network.
A natural and important question is to what extent gradient descent has similiar implicit bias for modern deep neural networks.
For theoretical analysis, a natural candidate is to consider homogeneous neural networks .
Here a neural network Φ is said to be (positively) homogeneous if there is a number
L > 0 (called the order ) such that the network output Φ( θ ; x ) , where θ stands for the parameter and x stands for the input, satisﬁes the following:
[Equation]
It is important to note that many neural networks are homogeneous (Neyshabur et al., 2015a; Du et al., 2018).
For example, deep fully-connected neural networks or deep CNNs with ReLU or
LeakyReLU activations can be made homogeneous if we remove all the bias terms, and the order L is exactly equal to the number of layers.
In (Wei et al., 2019), it is shown that the regularization path does converge to the max-margin direction for homogeneous neural networks with cross-entropy or logistic loss.
This result suggests that gradient descent or gradient ﬂow may also converges to the max-margin direction by assuming homogeneity, and this is indeed true for some sub-classes of homogeneous neural networks.
For gradient ﬂow, this convergent direction is proven for linear fully-connected networks (Ji & Telgarsky, 2019a).
For gradient descent on linear fully-connected and convolutional networks, (Gunasekar et al., 2018b) formulate a constrained optimization problem related to margin maximization and prove that gradient descent converges to the direction of a KKT point or even the max-margin direction, under various assumptions including the convergence of loss and gradient directions.
In an independent work, (Nacson et al., 2019a) generalize the result in (Gunasekar et al., 2018b) to smooth homogeneous models (we will discuss this work in more details in Section 2).
1. M AIN R ESULTS
In this paper, we identify a minimal set of assumptions for proving our theoretical results for homogeneous neural networks on classiﬁcation tasks.
Besides homogeneity, we make two additional assumptions:
Exponential-type Loss Function.
We require the loss function to have certain exponential tail (see Appendix A for the details).
This assumption is not restrictive as it includes the most popular classﬁcation losses: exponential loss, logistic loss and cross-entropy loss.
Separability.
The neural network can separate the training data during training (i
e., the neural network can achieve 100% training accuracy) 1 .
While the ﬁrst assumption is natural, the second requires some explanation.
In fact, we assume that at some time t 0 , the training loss is smaller than a threshold, and the threshold here is chosen to be so small that the training accuracy is guaranteed to be 100% (e.g., for the logistic loss and cross-entropy loss, the threshold can be set to ln 2 ).
Empirically, state-of-the-art CNNs for image classiﬁcation can even ﬁt randomly labeled data easily (Zhang et al., 2017).
Recent theoretical work on over-parameterized neural networks (Allen-Zhu et al., 2019; Zou et al., 2018) show that gradient descent can ﬁt the training data if the width is large enough.
Furthermore, in order to study the margin, ensuring the training data can be separated is inevitable; otherwise, there is no positive margin between the data and decision boundary.
Our Contribution.
Similar to linear models, for homogeneous models, only the direction of parameter θ is important for making predictions, and one can see that the margin γ ( θ ) scales linearly with (cid:107) θ (cid:107) L 2 , when ﬁxing the direction of θ .
To compare margins among θ in different directions, it makes sense to study the normalized margin , ¯ γ ( θ ) :
= γ ( θ ) / (cid:107) θ (cid:107) L 2 .
In this paper, we focus on the training dynamics of the network after t 0 (recall that t 0 is a time that the training loss is less than the threshold).
Our theoretical results can answer the following questions regarding the normalized margin.
First, how does the normalized margin change during training?
The answer may seem complicated since one can easily come up with examples in which ¯ γ increases or decreases in a short time interval.
However, we can show that the overall trend of the normalized margin is to increase in the following sense: there exists a smoothed version of the normalized margin, denoted as ˜ γ , such that (1) | ˜ γ − ¯ γ | → 0 as t → ∞ ; and (2) ˜ γ is non-decreasing for t > t 0 .
Second, how large is the normalized margin at convergence?
To answer this question, we formulate a natural constrained optimization problem which aims to directly maximize the margin.
We show that every limit point of { θ ( t ) / (cid:107) θ ( t ) (cid:107) 2 : t > 0 } is along the direction of a KKT point of the maxmargin problem.
This indicates that gradient descent/gradient ﬂow performs margin maximization implicitly in deep homogeneous networks.
This result can be seen as a signiﬁcant generalization of previous works (Soudry et al., 2018a;b; Ji & Telgarsky, 2019a; Gunasekar et al., 2018b) from linear classiﬁers to homogeneous classiﬁers. [Figure]
As by-products of the above results, we derive tight asymptotic convergence/growth rates of the loss and weights.
It is shown in (Soudry et al., 2018a;b; Ji & Telgarsky, 2018; 2019c) that the loss decreases at the rate of O (1 /t ) , the weight norm grows as O (log t ) for linear logistic regression.
In this work, we generalize the result by showing that the loss decreases at the rate of O (1 / ( t (log t ) 2 − 2 /L )) and the weight norm grows as O ((log t ) 1 /L ) for homogeneous neural networks with exponential loss, logistic loss, or cross-entropy loss.
Experiments.
The main practical implication of our theoretical result is that training longer can enlarge the normalized margin.
To justify this claim empiricaly, we train CNNs on MNIST and
CIFAR-10 with SGD (see Section K.1).
Results on MNIST are presented in Figure 1.
For constant step size, we can see that the normalized margin keeps increasing, but the growth rate is rather slow (because the gradient gets smaller and smaller).
Inspired by our convergence results for gradient descent, we use a learning rate scheduling method which enlarges the learning rate according to the current training loss, then the training loss decreases exponentially faster and the normalized margin increases signiﬁcantly faster as well.
For feedforward neural networks with ReLU activation, the normalized margin on a training sample is closely related to the L 2 -robustness (the L 2 -distance from the training sample to the decision boundary).
Indeed, the former divided by a Lipschitz constant is a lower bound for the latter.
For example, the normalized margin is a lower bound for the L 2 -robustness on fully-connected networks with ReLU activation (see, e.g., Theorem 4 in (Sokolic et al., 2017)).
This fact suggests that training longer may have potential beneﬁts on improving the robustness of the model.
In our experiments, we observe noticeable improvements of L 2 -robustness on both training and test sets (see Section K.2).
R ELATED W ORK
Implicit Bias in Training Linear Classiﬁers.
For linear logistic regression on linearly separable data, Soudry et al.
(2018a;b) showed that full-batch gradient descent converges in the direction of the max
L 2 -margin solution of the corresponding hard-margin Support Vector Machine (SVM).
Subsequent works extended this result in several ways: Nacson et al.
(2019c) extended the results to the case of stochastic gradient descent; Gunasekar et al.
(2018a) considered other optimization methods; Nacson et al.
(2019b) considered other loss functions including those with poly-exponential tails; Ji & Telgarsky (2018; 2019c) characterized the convergence of weight direction without assuming separability; Ji & Telgarsky (2019b) proved a tighter convergence rate for the weight direction.
Those results on linear logistic regression have been generalized to deep linear networks.
Ji & Telgarsky (2019a) showed that the product of weights in a deep linear network with strictly decreasing loss converges in the direction of the max L 2 -margin solution.
Gunasekar et al.
(2018b) showed more general results for gradient descent on linear fully-connected and convolutional networks with exponential loss, under various assumptions on the convergence of the loss and gradient direction
Margin maximization phenomenon is also studied for boosting methods (Schapire et al., 1998;
Schapire & Freund, 2012; Shalev-Shwartz & Singer, 2010; Telgarsky, 2013) and Normalized Perceptron (Ramdas & Pena, 2016).
Implicit Bias in Training Nonlinear Classiﬁers.
Soudry et al.
(2018a) analyzed the case where there is only one trainable layer of a ReLU network.
Xu et al.
characterized the implicit bias for the model consisting of one single ReLU unit.
Our work is closely related to a recent independent work by (Nacson et al., 2019a) which we discuss in details below.
Comparison with (Nacson et al., 2019a).
Very recently, (Nacson et al., 2019a) analyzed gradient descent for smooth homogeneous models and proved the convergence of parameter direction to a
KKT point of the aforementioned max-margin problem.
Compared with their work, our work adopt much weaker assumptions: (1) They assume the training loss converges to 0 , but in our work we only require that the training loss is lower than a small threshold value at some time t 0 (and we prove the exact convergence rate of the loss after t 0 ); (2) They assume the convergence of parameter direction 3 , while we prove that KKT conditions hold for all limit points of { θ ( t ) / (cid:107) θ ( t ) (cid:107) 2 : t > 0 } , without requiring any convergence assumption; (3) They assume the convergence of the direction of losses (the direction of the vector whose entries are loss values on every data point) and Linear
Independence Constraint Qualiﬁcation (LICQ) for the max-margin problem, while we do not need such assumptions.
Besides the above differences in assumptions, we also prove the monotonicity of the normalized margin and provide tight convergence rate for training loss.
We believe both results are interesting in their own right.
Another technical difference is that their work analyzes discrete gradient descent on smooth homogeneous models (which fails to capture ReLU networks).
In our work, we analyze both gradient descent on smooth homogeneous models and also gradient ﬂow on homogeneous models which could be non-smooth.
Other Works on Implicit Bias.
Banburski et al.
(2019) also studied the dynamics of gradient ﬂow and among other things, provided mathematical insights to the implicit bias towards max margin solution for homogeneous networks.
We note that their analysis of gradient ﬂow decomposes the dynamics to the tangent component and radial component, which is similar to our proof of Theorem 4.1 in spirit.
Wilson et al.
; Ali et al.
; Gunasekar et al.
(2018a) showed that for the linear least-square problem gradient-based methods converge to the unique global minimum that is closest to the initialization in L 2 distance.
Du et al.
; Jacot et al.
; Lee et al.
Arora et al.
(2019b) showed that over-parameterized neural networks of sufﬁcient width (or inﬁnite width) behave as linear models with Neural Tangent Kernel (NTK) with proper initialization and gradient descent converges linearly to a global minimum near the initial point.
Other related works include (Ma et al., 2019; Gidel et al., 2019; Arora et al., 2019a; Suggala et al., 2018; Blanc et al., 2019; Neyshabur et al., 2015b;a).
P RELIMINARIES
Basic Notations.
For any N ∈ N , let [ N ] = { 1 , . .
. , N } .
(cid:107) v (cid:107) 2 denotes the L 2 -norm of a vector v .
The default base of log is e .
For a function f : R d → R , ∇ f ( x ) stands for the gradient at x if it exists.
A function f : X → R d is
C k -smooth if f is k times continuously differentiable.
A function f : X → R is locally Lipschitz if for every x ∈ X there exists a neighborhood U of x such that the restriction of f on U is Lipschitz continuous.
Non-smooth Analysis.
For a locally Lipschitz function f : X → R , the Clarke’s subdifferential (Clarke, 1975; Clarke et al., 2008; Davis et al., 2020) at x ∈ X is the convex set
[Equation]
For brevity, we say that a function z : I → R d on the interval I is an arc if z is absolutely continuous for any compact sub-interval of I .
For an arc z , z (cid:48) ( t ) (or d z dt ( t ) ) stands for the derivative at t if it exists.
Following the terminology in (Davis et al., 2020), we say that a locally Lipschitz function f : R d → R admits a chain rule if for any arc z : [0 , + ∞ ) → R d , ∀ h ∈ ∂ ◦ f
( z ( t )) : ( f ◦ z ) (cid:48) ( t ) = (cid:104) h , z (cid:48) ( t ) (cid:105) holds for a.e. t > 0 (see also Appendix I).
Binary Classiﬁcation.
Let Φ be a neural network, assumed to be parameterized by θ .
The output of Φ on an input x ∈ R d x is a real number Φ( θ ; x ) , and the sign of Φ( θ ; x ) stands for the classiﬁcation result.
A dataset is denoted by D
= { ( x n , y n ) : n ∈ [ N ] } , where x n ∈ R d x stands for a data input and y n ∈ {± 1 } stands for the corresponding label.
For a loss function (cid:96) : R → R , we deﬁne the training loss of Φ on the dataset D to be L ( θ ) :
= (cid:80) Nn =1 (cid:96) ( y n Φ( θ ; x n )) .
Gradient Descent.
We consider the process of training this neural network Φ with either gradient descent or gradient ﬂow.
For gradient descent, we assume the training loss L ( θ ) is C 2 -smooth and describe the gradient descnet process as θ ( t +1)
= θ ( t ) − η ( t )
∇L ( θ ( t )) , where η ( t ) is the learning rate at time t and ∇L ( θ ( t )) is the gradient of L at θ ( t ) .
Gradient Flow.
For gradient ﬂow, we do not assume the differentibility but only some regularity assumptions including locally Lipschitz.
Gradient ﬂow can be seen as gradient descent with inﬁnitesimal step size.
In this model, θ changes continuously with time, and the trajectory of parameter θ during training is an arc θ : [0 , + ∞ ) → R d , t
(cid:55)→ θ ( t ) that satisﬁes the differential inclusion d θ ( t ) dt ∈ − ∂ ◦ L ( θ ( t )) for a.e. t ≥ 0 .
The Clarke’s subdifferential ∂ ◦ L is a natural generalization of the usual differential to non-differentiable functions.
If L ( θ ) is actually a C 1 -smooth function, the above differential inclusion reduces to d θ ( t ) dt
= −∇L
( θ ( t )) for all t ≥ 0 , which corresponds to the gradient ﬂow with differential in the usual sense.
G RADIENT D ESCENT / G RADIENT F LOW ON H OMOGENEOUS M ODEL
In this section, we ﬁrst state our results for gradient ﬂow and gradient descent on homogeneous models with exponential loss (cid:96) ( q ) :=
e − q for simplicity of presentation.
Due to space limit, we defer the more general results which hold for a large family of loss functions (including logistic loss and cross-entropy loss) to Appendix A, F and G.
4. A SSUMPTIONS
Gradient Flow.
For gradient ﬂow, we assume the following:
(A1).
(Regularity).
For any ﬁxed x , Φ( · ; x ) is locally Lipschitz and admits a chain rule; (A2).
(Homogeneity).
There exists L > 0 such that ∀ α > 0 : Φ( α θ ; x ) =
α L Φ( θ ; x ) ; (A3).
(Exponential Loss).
(cid:96) ( q ) =
e − q ; (A4).
(Separability).
There exists a time t 0 such that L ( θ ( t 0 )) < 1 .
(A1) is a technical assumption about the regularity of the network output.
As shown in (Davis et al., 2020), the output of almost every neural network admits a chain rule (as long as the neural network is composed by deﬁnable pieces in an o-minimal structure, e.g., ReLU, sigmoid, LeakyReLU).
(A2) assumes the homogeneity, the main property we assume in this work.
(A3), (A4) correspond to the two conditions introduced in Section 1.
The exponential loss in (A3) is main focus of this section. (A4) is a separability assumption: the condition L ( θ ( t 0 )) <
1 ensures that (cid:96) ( y n Φ( θ ( t 0 ); x n )) < for all n ∈ [ N ] , and thus y n Φ( θ ( t 0 ); x n ) > 0 , meaning that Φ classiﬁes every x n correctly.
Gradient Descent.
For gradient descent, we assume (A2), (A3), (A4) similarly as for gradient ﬂow, and the following two assumptions (S1) and (S5).
(S1).
(Smoothness).
For any ﬁxed x , Φ( · ; x ) is C 2 -smooth on R d \ { 0 } .
(S5).
(Learning rate condition, Informal).
η ( t ) =
η 0 for a sufﬁciently small constant η 0 .
In fact, η ( t ) is even allowed to be as large as O ( L ( t ) − 1 polylog 1 L ( t ) ) .
See Appendix E.1 for the details.
(S5) is natural since deep neural networks are usually trained with constant learning rates.
(S1) ensures the smoothness of Φ , which is often assumed in the optimization literature in order to analyze gradient descent.
While (S1) does not hold for neural networks with ReLU, it does hold for neural networks with smooth homogeneous activation such as the quadratic activation φ ( x ) :
= x 2 (Li et al., 2018b; Du & Lee, 2018) or powers of ReLU φ ( x ) :=
ReLU( x ) α for α > 2 (Zhong et al., 2017;
Klusowski & Barron, 2018; Li et al., 2019).
4. M AIN T HEOREM : M ONOTONICITY OF N ORMALIZED M ARGINS
The margin for a single data point ( x n , y n ) is deﬁned to be q n ( θ ) :
= y n Φ( θ ; x n ) , and the margin for the entire dataset is deﬁned to be q min ( θ ) :
= min n ∈ [ N ] q n ( θ ) .
By homogenity, the margin q min ( θ ) scales linearly with (cid:107) θ (cid:107) L 2 for any ﬁxed direction since q min ( c θ )
=
c L q min ( θ ) .
So we consider the normalized margin deﬁned as below:
[Equation]
We say f is an (cid:15) -additive approximation for the normalized margin if ¯ γ − (cid:15) ≤ f ≤ ¯ γ , and cmultiplicative approximation if c ¯ γ ≤ f ≤ ¯ γ .
Gradient Flow.
Our ﬁrst result is on the overall trend of the normalized margin ¯ γ ( θ ( t )) .
For both gradient ﬂow and gradient descent, we identify a smoothed version of the normalized margin, and show that it is non-decreasing during training.
More speciﬁcally, we have the following theorem for gradient ﬂow.
Theorem 4.1 (Corollary of Theorem A.7) .
Under assumptions (A1) - (A4), there exists an
O ( (cid:107) θ (cid:107) − L 2 ) -additive approximation function ˜
γ ( θ ) for the normalized margin such that the following statements are true for gradient ﬂow:
For a.e. t > t 0 , ddt ˜ γ
( θ ( t )) ≥ 0 ; 2.
For a.e. t > t 0 , either ddt ˜ γ
( θ ( t )) > 0 or ddt θ ( t ) (cid:107) θ ( t ) (cid:107) 2 = 0 ; 3.
L ( θ ( t ))
→ 0 and (cid:107) θ ( t ) (cid:107) 2 → ∞ as t → + ∞ ; therefore, | ¯ γ ( θ ( t ))
− ˜ γ ( θ ( t ))
| → 0 .
More concretely, the function ˜
γ ( θ ) in Theorem 4.1 is deﬁned as
[Equation]
Note that the only difference between ¯ γ ( θ ) and ˜ γ ( θ ) is that the margin q min ( θ ) in ¯ γ ( θ ) is replaced by log 1 L ( θ )
= − LSE( − q 1 ( θ ) , . .
, − q N ( θ )) , where LSE( a 1 , . . . , a N ) = log(exp( a 1 ) + · · · + exp( a N )) is the LogSumExp function.
This is indeed a very natural idea, and previous works on linear models (e.g., (Telgarsky, 2013; Nacson et al., 2019b)) also approximate q min with LogSumExp in the analysis of margin.
It is easy to see why ˜ γ ( θ ) is an O ( (cid:107) θ (cid:107) − L 2 ) -additive approximation for ¯ γ ( θ ) :
e a max ≤ (cid:80) Nn
=1 e a n ≤ Ne a max holds for a max = max { a 1 , . . .
, a N } , so a max ≤ LSE( a 1 , . . . , a N ) ≤ a max + log N ; combining this with the deﬁnition of ˜ γ ( θ ) gives ¯ γ ( θ ) − (cid:107) θ (cid:107) − L 2 log N ≤ ˜ γ ( θ ) ≤ ¯ γ ( θ ) .
Gradient Descent.
For gradient descent, Theorem 4.1 holds similarly with a slightly different function ˆ γ ( θ ) that approximates ¯ γ ( θ ) multiplicatively rather than additively.
Theorem 4.2 (Corollary of Theorem E.2) .
Under assumptions (S1), (A2) - (A4), (S5), there exists an (1 − O
(1 / (log 1 L )))
-multiplicative approximation function ˆ γ ( θ ) for the normalized margin such that the following statements are true for gradient descent:
For all t > t 0 ,
ˆ γ ( θ ( t + 1)) ≥ ˆ γ ( θ ( t )) ; 2.
For all t > t 0 , either ˆ γ ( θ ( t + 1)) >
ˆ γ ( θ ( t )) or θ ( t +1) (cid:107) θ ( t +1)
(cid:107)
=
θ ( t ) (cid:107) θ ( t ) (cid:107) 2 ; 3.
L ( θ ( t ))
→ 0 and (cid:107) θ ( t ) (cid:107) 2 → ∞ as t → + ∞ ; therefore, | ¯ γ ( θ ( t ))
− ˆ γ ( θ ( t ))
| → 0 .
Due to the discreteness of gradient descent, the explicit formula for ˆ γ ( θ ) is somewhat technical, and we refer the readers to Appendix E for full details.
Convergence Rates.
It is shown in Theorem 4.1, 4.2 that L ( θ ( t ))
→ 0 and (cid:107) θ ( t ) (cid:107) 2 → ∞ .
In fact, with a more reﬁned analysis, we can prove tight loss convergence and weight growth rates using the monotonicity of normalized margins.
Theorem 4.3 (Corollary of Theorem A.10 and E.5) .
For gradient ﬂow under assumptions (A1) - (A4) or gradient descent under assumptions (S1), (A2) - (A4), (S5), we have the following tight bounds for training loss and weight norm:
[Equation]
and
[Equation]
where T = t for gradient ﬂow and T = (cid:80) t − 1 τ = t 0 η
( τ ) for gradient descent.
4. M AIN T HEOREM : C ONVERGENCE TO KKT P OINTS
For gradient ﬂow, ˜ γ is upper-bounded by ˜ γ ≤ ¯ γ ≤ sup { q n ( θ ) : (cid:107) θ (cid:107) 2 = 1 } .
Combining this with
Theorem 4.1 and the monotone convergence theorem, it is not hard to see that lim t → + ∞ ¯ γ ( θ ( t )) and lim t →
+ ∞ ˜ γ ( θ ( t )) exist and equal to the same value.
Using a similar argument, we can draw the same conclusion for gradient descent.
To understand the implicit regularization effect, a natural question arises: what optimality property does the limit of normalized margin have?
To this end, we identify a natural constrained optimization problem related to margin maximization, and prove that θ ( t ) directionally converges to its KKT points, as shown below.
We note that we can extend this result to the ﬁnite time case, and show that gradient ﬂow or gradient descent passes through an approximate KKT point after a certain amount of time.
See Theorem A.9 in Appendix A and Theorem E.4 in Appendix E for the details.
We will brieﬂy review the deﬁnition of KKT points and approximate KKT points for a constraint optimization problem in Appendix C.1.
Theorem 4.4 (Corollary of Theorem A.8 and E.3) .
For gradient ﬂow under assumptions (A1) - (A4) or gradient descent under assumptions (S1), (A2) - (A4), (S5), any limit point ¯ θ of (cid:110) θ ( t ) (cid:107) θ ( t ) (cid:107) 2 : t ≥ 0 (cid:111) is along the direction of a KKT point of the following constrained optimization problem (P):
[Equation]
That is, for any limit point ¯ θ , there exists a scaling factor α > 0 such that α ¯ θ satisﬁes Karush-KuhnTucker (KKT) conditions of (P).
Minimizing (P) over its feasible region is equivalent to maximizing the normalized margin over all possible directions.
The proof is as follows.
Note that we only need to consider all feasible points θ with q min ( θ ) > 0 .
For a ﬁxed θ , α θ is a feasible point of (P) iff α ≥ q min ( θ ) − 1 /L .
Thus, the minimum objective value over all feasible points of (P) in the direction of θ is 1 2 (cid:107) θ /q min ( θ ) 1 /L
(cid:107) 2 2 = 12 ¯ γ ( θ ) − 2 /L .
Taking minimum over all possible directions, we can conclude that if the maximum normalized margin is ¯ γ ∗ , then the minimum objective of (P) is 12 ¯ γ − 2 /L ∗ .
It can be proved that (P) satisﬁes the Mangasarian-Fromovitz Constraint Qualiﬁcation (MFCQ)
(See
Lemma C.7).
Thus, KKT conditions are ﬁrst-order necessary conditions for global optimality.
For linear models, KKT conditions are also sufﬁcient for ensuring global optimality; however, for deep homogeneous networks, q n ( θ ) can be highly non-convex.
Indeed, as gradient descent is a ﬁrst-order optimization method, if we do not make further assumptions on q n ( θ ) , then it is easy to construct examples that gradient descent does not lead to a normalized margin that is globally optimal.
Thus, proving the convergence to KKT points is perhaps the best we can hope for in our setting, and it is an interesting future work to prove stronger convergence results with further natural assumptions.
Moreover, we can prove the following corollary, which characterizes the optimality of the normalized margin using SVM with Neural Tangent Kernel (NTK, introduced in (Jacot et al., 2018)) deﬁned at limit points.
The proof is deferred to Appendix C.6.
Corollary 4.5 (Corollary of Theorem 4.4) .
Assume (S1).
Then for gradient ﬂow under assumptions (A2) - (A4) or gradient descent under assumptions (A2) - (A4), (S5), any limit point ¯ θ of { θ ( t ) / (cid:107) θ ( t ) (cid:107) 2 : t ≥ 0 } is along the max-margin direction for the hard-margin SVM with kernel
K ¯ θ ( x , x (cid:48) )
= (cid:10) ∇ Φ x (¯ θ ) , ∇ Φ x (cid:48)
(¯ θ ) (cid:11) , where Φ x ( θ ) :
=
Φ( θ ; x ) .
That is, for some α > 0 , α ¯ θ is the optimal solution for the following constrained optimization problem:
[Equation]
If we assume (A1) instead of (S1) for gradient ﬂow, then there exists a mapping h ( x ) ∈ ∂ ◦
Φ x (¯ θ ) such that the same conclusion holds for K ¯ θ ( x , x (cid:48) )
= (cid:104) h
( x ) , h ( x (cid:48) ) (cid:105) .
4. O THER M AIN R ESULTS
The above results can be extended to other settings as shown below.
Other Binary Classiﬁcation Loss.
The results on exponential loss can be generalized to a much broader class of binary classiﬁcation loss.
The class includes the logistic loss which is one of the most popular loss functions, (cid:96) ( q ) =
log(1+ e − q ) .
The function class also includes other losses with exponential tail, e.g., (cid:96) ( q ) =
e − q 3 , (cid:96) ( q ) =
log(1 + e − q 3 ) .
For all those loss functions, we can use its inverse function (cid:96) − 1 to deﬁne the smoothed normalized margin as follows
[Equation]
Then all our results for gradient ﬂow continue to hold (Appendix A). Using a similar modiﬁcation, we can also extend it to gradient descent (Appendix F).
Cross-entropy Loss.
In multi-class classiﬁcation, we can deﬁne q n to be the difference between the classiﬁcation score for the true label and the maximum score for the other labels, then the margin q min := min n ∈ [ N ] q n and the normalized margin ¯ γ ( θ ) :
= q min ( θ ) (cid:107) θ (cid:107) L 2 can be similarly deﬁned as before.
In Appendix G, we deﬁne the smoothed normalized margin for cross-entropy loss to be the same as that for logistic loss (See Remark A.4). Then we show that Theorem 4.1 and Theorem 4.4 still hold (but with a slightly different deﬁnition of (P)) for gradient ﬂow, and we also extend the results to gradient descent.
Multi-homogeneous Models.
Some neural networks indeed possess a stronger property than homogeneity, which we call multi-homogeneity .
For example, the output of a CNN (without bias terms) is 1 -homogeneous with respect to the weights of each layer.
In general, we say that a neural network Φ( θ ; x ) with θ =
( w 1 , . . .
, w m ) is ( k 1 , . . .
, k m ) -homogeneous
if for any x and any c 1 , . . .
, c m > 0 , we have Φ( c 1 w 1 , . . .
, c m w m ; x )
= (cid:81) mi
=1 c k i i · Φ( w 1 , . . . , w m ; x ) .
In the previous example, an L -layer CNN with layer weights θ =
( w 1 , . . .
, w L ) is (1 , . . .
, 1) -homogeneous.
One can easily see that that ( k 1 , . . .
, k m ) -homogeneity implies L -homogeneity, where L = (cid:80) mi =1 k i , so our previous analysis for homogeneous models still applies to multi-homogeneous models.
But it would be better to deﬁne the normalized margin for multi-homogeneous model as
[Equation]
In this case, the smoothed approximation of ¯ γ for general binary classiﬁcation loss (under some conditions) can be similarly deﬁned for gradient ﬂow:
[Equation]
It can be shown that ˜ γ is also non-decreasing during training when the loss is small enough (Appendix H).
In the case of cross-entropy loss, we can still deﬁne ˜ γ by (5) while (cid:96) ( · ) is set to the logistic loss in the formula.
P ROOF S KETCH : G RADIENT F LOW ON H OMOGENEOUS M ODEL WITH E XPONENTIAL L OSS
In this section, we present a proof sketch in the case of gradient ﬂow on homogeneous model with exponential loss to illustrate our proof ideas.
Due to space limit, the proof for the main theorems on gradient ﬂow and gradient descent in Section 4 are deferred to Appendix A and E respectively.
For convenience, we introduce a few more notations for a L -homogeneous neural network Φ( θ ; x ) .
Let S d − 1 = { θ ∈ R
d : (cid:107) θ (cid:107) 2 = 1 } be the set of L 2 -normalized parameters.
Deﬁne ρ := (cid:107) θ
(cid:107)and ˆ θ := θ (cid:107) θ (cid:107) 2 ∈ S
d − 1 to be the length and direction of θ .
For both gradient descent and gradient ﬂow, θ is a function of time t .
For convenience, we also view the functions of θ , including L ( θ ) , q n ( θ ) , q min ( θ ) , as functions of t .
So we can write L ( t ) :
= L ( θ ( t )) , q n ( t ) :
=
q n ( θ ( t )) , q min ( t ) :
= q min ( θ ( t )) .
Lemma 5.1 below is the key lemma in our proof.
It decomposes the growth of the smoothed normalized margin into the ratio of two quantities related to the radial and tangential velocity components of θ respectively.
We will give a proof sketch for this later in this section.
We believe that this lemma is of independent interest.
Lemma 5.1 (Corollary of Lemma B.1) .
For a.e. t > t 0 , d dt log ρ > 0 and d dt log ˜ γ ≥ L (cid:18) d dt log ρ (cid:19) − 1 (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
d ˆ θ dt (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)  .
Using Lemma 5.1, the ﬁrst two claims in Theorem 4.1 can be directly proved.
For the third claim, we make use of the monotonicity of the margin to lower bound the gradient, and then show L →and ρ → + ∞ .
Recall that ˜ γ is an O ( ρ − L ) -additive approximation for ¯ γ .
So this proves the third claim.
We defer the detailed proof to Appendix B.
To show Theorem 4.4, we ﬁrst change the time measure to log ρ , i.e., now we see t as a function of log ρ .
So the second inequality in Lemma 5.1 can be rewritten as d log ˜ γ d log ρ ≥ L (cid:107) d ˆ θ d log ρ (cid:107) 22 .
Integrating on both sides and noting that ˜ γ is upper-bounded, we know that there must be many instant log ρ such that (cid:107)
d ˆ θ d log ρ (cid:107) 2 is small.
By analyzing the landscape of training loss, we show that these points are “approximate” KKT points.
Then we show that every convergent sub-sequence of { ˆ θ ( t ) : t ≥ 0 } can be modiﬁed to be a sequence of “approximate” KKT points which converges to the same limit.
Then we conclude the proof by applying a theorem from (Dutta et al., 2013) to show that the limit of this convergent sequence of “approximate” KKT points is a KKT point.
We defer the detailed proof to Appendix C.
Now we give a proof sketch for Lemma 5.1, in which we derive the formula of ˜ γ step by step.
In the proof, we obtain several clean close form formulas for several relevant quantities, by using the chain rule and Euler’s theorem for homogeneous functions extensively.
Proof Sketch of Lemma 5.1.
For ease of presentation, we ignore the regularity issues of taking derivatives in this proof sketch.
We start from the equation d L dt = − (cid:10) ∂
◦ L ( θ ( t )) , d θ dt (cid:11) =
− (cid:13)(cid:13)
d θ dt (cid:13)(cid:13) 2which follows from the chain rule (see also Lemma I.3).
Then we note that d θ dt can be decomposed into two parts: the radial component v :
=
ˆ θ ˆ θ (cid:62) d θ dt and the tangent component u :
= ( I − ˆ θ ˆ θ (cid:62) )
d θ dt .
The radial component is easier to analyze.
By the chain rule, (cid:107) v (cid:107) {+
+}
=
ˆ θ (cid:62) d θ dt = 1 ρ (cid:10) θ , d θ dt (cid:11) =
1 ρ · 1 2 dρ 2 dt .
For 12 dρ 2 dt , we have an exact formula:
[Equation]
The last equality is due to (cid:104) ∂ ◦ q n , θ (cid:105) =
Lq n by homogeneity of q n .
This is sometimes called Euler’s theorem for homogeneous functions (see Theorem B.2).
For differentiable q n , it can be easily proved by taking the derivative over c on both sides of q n ( c θ ) =
c
L q n ( θ ) and letting c = 1 .
With (6), we can lower bound 12 dρ 2 dt by
[Equation]
where the last inequality uses the fact that e − q min ≤ L .
(7) also implies that 12 dρ 2 dt > 0 for t > tsince L ( t 0 ) <
1 and L is non-increasing.
As ddt log ρ = 12 ρ 2 dρ 2 dt , this also proves the ﬁrst inequality of Lemma 5.1.
Now, we have (cid:107) v (cid:107) 22 = 1 ρ 2 (cid:16) 12 dρ 2 dt (cid:17) 2 =
dρ 2 dt · ddt log ρ on the one hand; on the other hand, by the chain rule we have d ˆ θ dt = 1 ρ 2 ( ρ d θ dt − dρdt θ )
= 1 ρ 2 ( ρ d θ dt − (ˆ θ (cid:62) d θ dt ) θ )
=
u ρ .
So we have
[Equation]
Dividing 12 dρ 2 dt on the leftmost and rightmost sides, we have
[Equation]
By − d L dt ≥ 0 and (7), the LHS is no greater than − d L dt · (cid:0) L
· L log 1 L (cid:1) − 1 = 1 L log log 1 L .
Thus we have ddt log log 1 L − L [Equation] log [Equation]
ddt log ρ (cid:1) − 1 (cid:13)(cid:13)(cid:13) [Table] 2 , where the LHS is exactly ddt log ˜ γ .
D ISCUSSION AND F UTURE D IRECTIONS
In this paper, we analyze the dynamics of gradient ﬂow/descent of homogeneous neural networks under a minimal set of assumptions.
The main technical contribution of our work is to prove rigorously that for gradient ﬂow/descent, the normalized margin is increasing and converges to a KKT point of a natural max-margin problem.
Our results leads to some natural further questions:
• Can we generalize our results for gradient descent on smooth neural networks to nonsmooth ones?
In the smooth case, we can lower bound the decrement of training loss by the gradient norm squared, multiplied by a factor related to learning rate.
However, in the non-smooth case, no such inequality is known in the optimization literature.
• Can we make more structural assumptions on the neural network to prove stronger results?
In this work, we use a minimal set of assumptions to show that the convergent direction of parameters is a KKT point.
A potential research direction is to identify more key properties of modern neural networks and show that the normalized margin at convergence is locally or globally optimal (in terms of optimizing (P)).
• Can we extend our results to neural networks with bias terms?
In our experiments, the normalized margin of the CNN with bias also increases during training despite that its output is non-homogeneous.
It is very interesting (and technically challenging) to provide a rigorous proof for this fact.
A CKNOWLEDGMENTS
The research is supported in part by the National Natural Science Foundation of China Grant 61822203, 61772297, 61632016, 61761146003and the Zhongguancun Haihua Institute for Frontier Information Technology and Turing AI Institute of Nanjing.
We thank Liwei Wang for helpful suggestions on the connection between margin and robustness.
We thank Sanjeev Arora, Tianle Cai,
Simon Du, Jason D. Lee, Zhiyuan Li, Tengyu Ma, Ruosong Wang for helpful discussions.
A R ESULTS FOR G ENERAL L OSS
In this section, we state our results for a broad class of binary classiﬁcation loss.
A major consequence of this generalization is that the logistic loss, one of the most popular loss functions, (cid:96) ( q ) =
log(1 + e − q ) is included.
The function class also includes other losses with exponential tail, e.g., (cid:96) ( q ) =
e − q 3 , (cid:96) ( q ) =
log(1 + e − q 3 ) .
A. A SSUMPTIONS
We ﬁrst focus on gradient ﬂow.
We assume (A1), (A2) as we do for exponential loss.
For (A3), (A4), we replace them with two weaker assumptions (B3), (B4).
All the assumptions are listed below: (A1).
(Regularity).
For any ﬁxed x , Φ( · ; x ) is locally Lipschitz and admits a chain rule; (A2).
(Homogeneity).
There exists L > 0 such that ∀ α > 0 : Φ( α θ ; x ) =
α L Φ( θ ; x ) ; (B3).
The loss function (cid:96) ( q ) can be expressed as (cid:96) ( q ) =
e − f ( q ) such that (B3.1).
f : R → R is C 1 -smooth.
(B3.2).
f (cid:48) ( q ) > 0 for all q ∈ R .
(B3.3).
There exists b f ≥ 0 such that f (cid:48) ( q ) q is non-decreasing for q ∈ ( b f , + ∞ ) , and f (cid:48) ( q ) q → + ∞ as q → + ∞ .
(B3.4).
Let
g : [ f ( b f )
+ ∞ ) →
[ b f , + ∞ ) be
the
inverse function of
f
on the domain [ b f , + ∞
There exists b g ≥ max { 2 f ( b f ) f b f ) } K ≥ such that g (cid:48) ( x ) ≤ Kg (cid:48) ( θx ) and f (cid:48) ( y ) ≤
Kf (cid:48)
( θy ) for all x ∈ ( b g , + ∞ ) , y ∈ ( g ( b g ) , + ∞ ) and θ ∈ [1 / 2 , 1) .
(B
Separability
There exists a time t 0 such that L ( t 0 ) <
e − f ( b f ) =
(cid:96) ( b f ) .
(A1) and (A2) remain unchanged.
(B3) is satisﬁed by exponential loss (cid:96) ( q ) =
e − q (with f ( q ) = q ) and logistic loss (cid:96) ( q ) =
log(1 + e − q ) (with f ( q ) =
− log log(1 + e − q ) ).
(B4) are essentially the same as (A4) but (B4) uses a threshold value that depends on the loss function.
Assuming (B3), it is easy to see that (B4) ensures the separability of data since (cid:96) ( q n ) <
e − f ( b f ) implies q n > b f ≥ 0 .
For logistic loss, we can set b f =
(see Remark A.2).
So the corresponding threshold value in (B4) is (cid:96) (0) =
log 2 .
Now we discuss each of the assumptions in (B3).
(B3.1) is a natural assumption on smoothness.
(B3.2) requires (cid:96) ( · ) to be monotone decreasing, which is also natural since (cid:96) ( · ) is used for binary classiﬁcation.
The rest of two assumptions in (B3) characterize the properties of (cid:96) (cid:48) ( q ) when q is large enough.
(B3.3) is an assumption that appears naturally from the proof.
For exponential loss, f (cid:48) ( q ) q
= q is always non-decreasing, so we can set b f = 0 .
In (B3.4), the inverse function g is deﬁned.
It is guaranteed by (B3.1) and (B3.2) that g always exists and g is also C 1 -smooth.
Though (B3.4) looks very complicated, it essentially says that f (cid:48)
(Θ( q ))
=
Θ( f (cid:48) ( q )) , g (cid:48)
(Θ( q ))
=
Θ( g (cid:48) ( q )) as q → ∞ .
(B3.4) is indeed a technical assumption that enables us to asymptotically compare the loss or the length of gradient at different data points.
It is possible to base our results on weaker assumptions than (B3.4), but we use (B3.4) for simplicity since it has already been satisﬁed by many loss functions such as the aforementioned examples.
We summarize the corresponding f, g and b f for exponential loss and logistic loss below:
Remark A.1.
Exponential loss (cid:96) ( q ) =
e − q satisﬁes (B3) with
[Equation]
b
[Equation]
Remark A.2.
Logistic loss (cid:96) ( q ) =
log(1 + e − q ) satisﬁes (B3) with f ( q ) =
− log log(1 + e − q )
=
Θ( q ) f (cid:48)
[-e − q (1 + e − q ) log(
 + e − q ) =-][Equation]
) g ( q ) =
− log( e e − q − 1) =
Θ( q ) [Equation]
b f =(cid:96) [Equation] f
[Equation]
The proof for Remark A.1 is trivial.
For Remark A.2, we give a proof below.
Proof for Remark A.2.
By simple calculations, the formulas for f ( q ) , f (cid:48) ( q ) , g ( q ) , g (cid:48) ( q ) are correct.
(B3.1) is trivial.
f (cid:48) ( q ) =
e − q (1+ e − q )log(
+ e − q ) > 0 , so (B3.2) is satisﬁed.
For (B3.3), note that f (cid:48) ( q ) q = q (1+ e q )log(1+ e − q ) .
The denominator is a decreasing function since
[Equation]
Thus, f (cid:48) ( q ) q is a strictly increasing function on R .
As b f is required to be non-negative, we set b f = 0 .
For proving that f (cid:48) ( q ) q → + ∞ and (B4), we only need to notice that f (cid:48) ( q ) ∼ e − q 1 · e − q
=
and g (cid:48)
( x ) = 1 /f (cid:48)
( g ( x )) ∼ 1 .
A. S MOOTHED N ORMALIZED M ARGIN
For a loss function (cid:96) ( · ) satisfying (B3), it is easy to see from (B3.2) that its inverse function (cid:96) − 1 ( · ) must exist.
For this kind of loss functions, we deﬁne the smoothed normalized margin as follows:
Deﬁnition A.3.
For a loss function (cid:96) ( · ) satisfying (B3), the smoothed normalized margin ˜ γ
( θ ) of θ is deﬁned as
[Equation]
where (cid:96) − 1 ( · ) is the inverse function of (cid:96) ( · ) and ρ := (cid:107) θ (cid:107) 2 .
Remark A.4.
For logistic loss (cid:96) ( q ) =
log(1+ e − q ) , ˜ γ ( θ ) =
ρ − L log 1 exp( L ) − 1 ; for exponential loss (cid:96) ( q ) =
e − q , ˜ γ
( θ ) =
ρ − L log 1 L , which is the same as (3) .
Now we give some insights on how well ˜ γ ( θ ) approximates ¯ γ ( θ ) using a similar argument as in
Section 4.2.
Using the LogSumExp function, the smoothed normalized margin ˜ γ ( θ ) can also be written as
[Equation]
LSE is a (log N ) -additive approximation for max .
So we can roughly approximate ˜ γ ( θ ) by
[Equation]
Note that (B3.3) is crucial to make the above approximation reasonable.
Similar to exponential loss, we can show the following lemma asserting that ˜ γ is a good approximation of ¯ γ .
Lemma A.5.
Assuming (B3) 4 , we have the following properties about the margin:
(a) f ( q min ) − log N ≤ log 1 L ≤ f ( q min ) .
(b) If log 1 L > f ( b f ) , then there exists ξ ∈ ( f ( q min ) − log N, f ( q min ))
∩ ( b f , + ∞ ) such that
[Equation]
(c) For a seuqnce of parameters { θ
m ∈ R d : m ∈ N } , if L ( θ m ) → 0 , then | ˜ γ ( θ m ) − ¯ γ ( θ m ) | → 0 .
Proof.
(a) can be easily deduced from e − f ( q min ) ≤ L ≤ Ne − f ( q min ) .
Combining (a) and the monotonicity of g ( · ) , we further have g ( s ) ≤ g (log 1 L ) ≤ q min for s :=
max { f ( b f ) , f ( q min ) − log N } .
By the mean value Theorem, there exists ξ ∈ ( s, f ( q min )) such that g ( s ) =
g ( f ( q min ))
− g (cid:48) ( ξ )( f ( q min ) − s ) ≥ q min − g (cid:48) ( ξ ) log N .
Dividing ρ L on each side of q min − g (cid:48) ( ξ ) log N ≤ g (log 1 L ) ≤ q min proves (b) .
Now we prove (c) .
Without loss of generality, we assume log 1 L ( θ m ) >
f ( b f ) for all θ m .
It follows from (b) that for every θ m there exists ξ m ∈ ( f ( q min ( θ m ))
− log N, f ( q min ( θ m )))
∩ ( b f , + ∞ ) such that
[Equation]
Note that ξ m ≥ f ( q min ( θ m ))
− log N ≥ log
[Equation]
So
[Equation]
f (cid:48) ( g ( ξ m )) g ( ξ m ) →
+ ∞ by (B3.3).
Also note that there exists a constant B 0 such that ¯ γ ( θ m ) ≤ Bfor all m since ¯ γ is continuous on the unit sphere S d − 1 .
So we have
[Equation]
where the ﬁrst inequality follows since ξ m ≤ f ( q min ( θ m )) .
Together with (8), we have | ˜ γ ( θ m ) − ¯ γ ( θ m ) | → 0 .
Remark A.6.
For exponential loss, we have already shown in Section 4.2 that ˜ γ ( θ ) is an O ( ρ − L ) - additive approximation for ¯ γ ( θ ) .
For logistic loss, it follows easily from g (cid:48) ( q ) = Θ(1) and (b) of
Lemma A.5 that ˜ γ ( θ ) is an O ( ρ − L ) -additive approximation for ¯ γ ( θ ) if L is sufﬁciently small.
A. T HEOREMS
Now we state our main theorems.
For the monotonicity of the normalized margin, we have the following theorem.
The proof is provided in Appendix B.
Theorem A.7.
Under assumptions (A1), (A2), (B3) 4 , (B4), the following statements are true for gradient ﬂow:
For a.e. t > t 0 , ddt ˜ γ
( θ ( t )) ≥ 0 ; 2.
For a.e. t > t 0 , either ddt ˜ γ
( θ ( t )) > 0 or ddt θ ( t ) (cid:107) θ ( t ) (cid:107) 2 = 0 ; 3.
L ( θ ( t ))
→ 0 and (cid:107) θ ( t ) (cid:107) 2 → ∞ as t → + ∞ ; therefore, | ¯ γ ( θ ( t ))
− ˜ γ ( θ ( t ))
| → 0 .
For the normalized margin at convergence, we have two theorems, one for inﬁnite-time limiting case, and the other being a ﬁnite-time quantitative result.
Their proofs can be found in Appendix C.
As in the exponential loss case, we deﬁne the constrained optimization problem (P) as follows:
[Equation]
First, we show the directional convergence of θ ( t ) to a KKT point of (P).
Theorem A.8.
Consider gradient ﬂow under assumptions (A1), (A2), (B3), (B4).
For every limit point ¯ θ of (cid:110)
ˆ θ ( t ) : t ≥ 0 (cid:111) , ¯ θ /q min (¯ θ ) 1 /L is a KKT point of (P).
Second, we show that after ﬁnite time, gradient ﬂow can pass through an approximate KKT point.
Theorem A.9.
Consider gradient ﬂow under assumptions (A1), (A2), (B3), (B4).
For any (cid:15), δ > 0 , there exists r := Θ(log δ − 1 ) and ∆ :
= Θ( (cid:15) − 2 ) such that θ /q min ( θ ) 1 /L is an ( (cid:15), δ ) -KKT point at some time t ∗ satisfying log (cid:107)
θ ( t ∗ ) (cid:107)
2 ∈
( r, r + ∆) .
For the deﬁnitions for KKT points and approximate KKT points, we refer the readers to Appendix C.1 for more details.
With a reﬁned analysis, we can also provide tight rates for loss convergence and weight growth.
The proof is given in Appendix D.
Theorem A.10.
Under assumptions (A1), (A2), (B3), (B4), we have the following tight rates for loss convergence and weight growth:
[Equation]
Applying Theorem A.10 to exponential loss and logistic loss, in which g ( x ) =
Θ( x ) , we have the following corollary:
Corollary A.11.
If (cid:96) ( · ) is the exponential or logistic loss, then,
[Equation]
and (cid:107) θ ( t ) (cid:107) 2 = Θ((log t ) 1 /L ) .
B M ARGIN M ONOTONICITY FOR G ENERAL L OSS
In this section, we consider gradient ﬂow and prove Theorem A.7.
We assume (A1), (A2), (B3), (B4) as mentioned in Appendix A.
We follow the notations in Section 5 to deﬁne ρ := (cid:107) θ (cid:107) 2 and ˆ θ := θ (cid:107) θ (cid:107)
2 ∈ S
d − 1 , and sometimes we view the functions of θ as functions of t .
B. P ROOF FOR P ROPOSITION 1 AND
To prove the ﬁrst two propositions, we generalize our key lemma (Lemma 5.1) to general loss.
Lemma B.1.
For ˜ γ deﬁned in Deﬁnition A.3, the following holds for all t > t 0 , d dt log ρ > 0 and d
[Equation]
L (cid:18)
[Equation]
(cid:19)
[Equation]
1 (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
θ dt (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)  .
Before proving Lemma B.1, we review two important properties of homogeneous functions.
Note that these two properties are usually shown for smooth functions.
By considering Clarke’s subdifferential, we can generalize it to locally Lipschitz functions that admit chain rules:
Theorem B.2.
Let F : R d → R be a locally Lipschitz function that admits a chain rule.
If F is k -homogeneous, then
(a) For all x ∈ R d and α > 0 , ∂ ◦ F ( αx )
[Equation]
α k − 1 ∂ ◦ F ( x ) That is, ∂ ◦ F ( αx ) =
{ α k − 1 h : h ∈ ∂ ◦ F ( x ) } .
(b) (Euler’s Theorem for Homogeneous Functions).
For all x ∈ R d , (cid:104) x , ∂ ◦ F ( x ) (cid:105) =
k [Equation] x ) That is, (cid:104) x , h (cid:105) =
k · F ( x ) for all h ∈ ∂ ◦ F ( x ) .
Proof.
Let D be the set of points x such that F is differentiable at x .
According to the deﬁnition of
Clarke’s subdifferential, for proving (a) , it is sufﬁcient to show that
[Equation]
Fix x k ∈ D .
Let U be a neighborhood of x k .
By deﬁnition of homogeneity, for any h ∈ R d and any y ∈
U \ { x k } ,
[Equation]
Taking limits y
→ x k on both sides, we know that the LHS converges to 0 iff the RHS converges to 0 .
Then by deﬁnition of differetiability and gradient, F is differentiable at α x k iff it is differentiable at x k , and ∇ F ( α x k )
=
α k − 1 h iff ∇ F ( x k ) =
h .
This proves (10).
To prove (b) , we ﬁx x ∈ R d .
Let z : [0 , + ∞ ) → R d , α (cid:55)→ α x be an arc.
By deﬁnition of homogeneity, ( F ◦ z )( α ) =
α k F ( x ) for α > 0 .
Taking derivative with respect to α on both sides (for differentiable points), we have
[Equation]
holds for a.e. α > 0 .
Pick an arbitrary α > 0 making (11) hold.
Then by (a) , (11) is equivalent to ∀
h ∈ ∂ ◦ F ( x ) : (cid:10) x , α k − 1 h (cid:11) =
kα k − 1 F ( x ) , which proves (b) .
Applying Theorem B.2 to homogeneous neural networks, we have the following corollary:
Corollary B.3.
Under the assumptions (A1) and (A2), for any θ ∈ R d and x ∈ R d x ,
[Equation]
, where Φ x ( θ ) =
Φ( θ ; x ) is the network output for a ﬁxed input x .
Corollary B.3 can be used to derive an exact formula for the weight growth during training.
Theorem B.4.
For a.e. t ≥ 0 ,
[Equation]
Proof.
The proof idea is to use Corollary B.3 and chain rules (See Appendix I for chain rules in
Clarke’s sense).
Applying the chain rule on t (cid:55)→ ρ 2 = (cid:107) θ (cid:107) 22 yields 12 dρ 2 dt = − (cid:104) θ , h (cid:105) for all h ∈ ∂ ◦ L and a.e. t > 0 .
Then applying the chain rule on θ (cid:55)→ L , we have
[Equation]
By Corollary B.3, (cid:104) θ , h n (cid:105) =
Lq n , and thus 12 dρ 2 dt =
L (cid:80) Nn =1 e − f ( q n ) f [Equation]
( q n ) q n .
For convenience, we deﬁne ν ( t ) :
= (cid:80) Nn =1 e − f ( q n ) f (cid:48) ( q n ) q n for all t ≥ 0 .
Then Theorem B.4 can be rephrased as 12 dρ 2 dt = Lν ( t ) for a.e. t ≥ 0 .
Lemma B.5.
For all t > t 0 ,
[Equation]
Proof.
By Lemma A.5, q n ≥ g (log 1 L ) for all n ∈ [ N ] .
Then by Assumption (B3),
[Equation]
Combining this with the deﬁnitions of ν ( t ) and L gives
[Equation]
Proof for Lemma B.1.
Note that ddt log ρ = 12 ρ 2 dρ 2 dt
=
L ν ( t ) ρ 2 by Theorem B.4.
Then it simply follows from Lemma B.5 that ddt log ρ > 0 for a.e. t > t 0 .
For the second inequality, we ﬁrst prove that log ˜ γ = log (cid:0) (cid:96) − 1 ( L ) /ρ
L (cid:1) = log (cid:0) g (log 1 L ) /ρ
L (cid:1) exists for all t ≥ t 0 .
L ( t ) is non-increasing with t .
So L ( t ) <
e − f ( b f ) for all t ≥ t 0 .
This implies that (1) log 1 L is always in the domain of g ; (2) ρ >
(otherwise L ≥ Ne − f (0) >
e − f ( b f ) , contradicting (B4)).
Therefore, ˜ γ := g (log 1 L ) /ρ
L exists and is always positive for all t ≥ t 0 , which proves the existence of log ˜ γ .
By the chain rule and Lemma B.5, we have
[Equation]
1 L ) · L · (cid:18) − d L dt (cid:19) − L 2 · ν ( t ) ρ≥ν ( t ) · (cid:18) − d L dt (cid:19) − L 2 · ν ( t ) ρ≥ν ( t ) · (cid:18) − d L dt −
L 2 ν ( t )ρ(cid:19) .
On the one hand, − d L dt =
(cid:13)(cid:13)
d θ dt (cid:13)(cid:13) 2 2 for a.e. t > 0 by Lemma I.3; on the other hand, Lν ( t ) =
(cid:10) θ , d θ dt (cid:11) by Theorem B.4.
Combining these together yields
[Equation]
By the chain rule, d ˆ θ dt = 1 ρ ( I − ˆ θ ˆ θ (cid:62) )
d θ dt for a.e. t > 0 .
So we have
[Equation]
B. P ROOF FOR P ROPOSITION
To prove the third proposition, we prove the following lemma to show that L → 0 by giving an upper bound for L .
Since L can never be 0 for bounded ρ , L → 0 directly implies ρ → + ∞ .
For showing | ¯ γ − ˜ γ | → 0 , we only need to apply (c) in Lemma A.5, which shows this when L → 0 .
Lemma B.6.
For all t > t 0 ,
[Equation]
x [Equation] Therefore, L ( t ) → 0 and ρ ( t ) →
+ ∞ as t → ∞ .
Proof for Lemma B.6.
By Lemma I.3 and Theorem B.4,
[Equation]
Using Lemma B.5 to lower bound ν and replacing ρ with (cid:0) g (log 1 L ) /
˜ γ (cid:1) 1 /L by the deﬁnition of ˜ γ , we have
[Equation]
, where the last inequality uses the monotonicity of ˜ γ .
So the following holds for a.e. t ≥ t 0 ,
[Equation]
Integrating on both sides from t 0 to t , we can conclude that
[Equation]
Note that 1 / L is non-decreasing.
If 1 / L does not grow to + ∞ , then neither does G (1 / L ) .
But the
RHS grows to + ∞ , which leads to a contradiction.
So L → 0 .
To make L → 0 , q min must converge to + ∞ .
So ρ → + ∞ .
C C ONVERGENCE TO THE M AX - MARGIN S OLUTION
In this section, we analyze the convergent direction of θ and prove Theorem A.8 and A.9, assuming (A1), (A2), (B3), (B4) as mentioned in Section A.
We follow the notations in Section 5 to deﬁne ρ := (cid:107) θ (cid:107) 2 and ˆ θ := θ (cid:107) θ (cid:107)
2 ∈ S d − 1 , and sometimes we view the functions of θ as functions of t .
C. P KKT C ONDITIONS
We ﬁrst review the deﬁnition of Karush-Kuhn-Tucker (KKT) conditions for non-smooth optimization problems following from (Dutta et al., 2013).
Consider the following optimization problem (P) for x ∈ R d :
[Equation]
∈ [ N ] where f, g 1 , . .
g n : R d → R are locally Lipschitz functions.
We say that x ∈ R d is a feasible point of (P) if x satisﬁes g n ( x ) ≤ 0 for all n ∈ [ N ] .
Deﬁnition C.1 (KKT Point) .
A feasible point x of (P) is a KKT point if x satisﬁes KKT conditions: there exists λ 1 , . . .
, λ N ≥
such that
1. 0 ∈ ∂ [Equation] ( x )
[Equation]
(cid:80) n ∈ [ N ] λ n ∂ ◦ g n ( x ) ; 2. ∀ n ∈ [ N ] : λ n g n ( x ) = 0 . It is important to note that a global minimum of (P) may not be a KKT point, but under some regularity assumptions, the KKT conditions become a necessary condition for global optimality.
The regularity condition we shall use in this paper is the non-smooth version of Mangasarian-Fromovitz
Constraint Qualiﬁcation (MFCQ) (see, e.g., the constraint qualiﬁcation (C.Q.5) in (Giorgi et al., 2004)):
Deﬁnition C.2 (MFCQ) .
For a feasible point x of (P), (P) is said to satisfy MFCQ at x if there exists v ∈ R d such that for all n ∈ [ N ] with g n ( x ) = 0 ,
[Equation]
Following from (Dutta et al., 2013), we deﬁne an approximate version of KKT point, as shown below.
Note that this deﬁnition is essentially the modiﬁed (cid:15) -KKT point deﬁned in their paper, but these two deﬁnitions differ in the following two ways: (1) First, in their paper, the subdifferential is allowed to be evaluated in a neighborhood of x , so our deﬁnition is slightly stronger;
(2) Second, their paper ﬁxes δ = (cid:15) 2 , but in our deﬁnition we make them independent.
Deﬁnition C.3 (Approximate KKT Point) .
For (cid:15), δ > 0 , a feasible point x of (P) is an ( (cid:15), δ ) -KKT point
if there exists λ n ≥ 0 , k ∈ ∂ ◦ f ( x ) , h n ∈ ∂ ◦ g n ( x ) for all n ∈ [ N ] such that
(cid:13)(cid:13)(cid:13) k + (cid:80) n ∈ [ N ] λ n h n ( x ) (cid:13)(cid:13)(cid:13) 2 ≤ (cid:15) ; 2.
∀ n ∈ [ N ] : λ n g n ( x ) ≥ − δ .
As shown in (Dutta et al., 2013), ( (cid:15), δ ) -KKT point is an approximate version of KKT point in the sense that a series of ( (cid:15), δ ) -KKT points can converge to a KKT point.
We restate their theorem in our setting:
Theorem C.4 (Corollary of Theorem 3.6 in (Dutta et al., 2013)) .
Let { x k ∈ R d : k ∈ N } be a sequence of feasible points of (P), { (cid:15) k > 0 : k ∈ N } and { δ k > 0 : k ∈ N } be two sequences.
x k is an ( (cid:15) k , δ k ) -KKT point for every k , and (cid:15) k
→ 0 , δ k → 0 .
If x k → x as k → + ∞ and MFCQ holds at x , then x is a KKT point of (P).
C. KKT C ONDITIONS FOR (P)
Recall that for a homogeneous neural network, the optimization problem (P) is deﬁned as follows:
[Equation]
Using the terminologies and notations in Appendix C.1, the objective and constraints are f ( x ) = 12 (cid:107) x (cid:107) 22 and g n ( x ) = 1 − q n ( x ) .
The KKT points and approximate KKT points for (P) are deﬁned as follows:
Deﬁnition C.5 (KKT Point of (P)) .
A feasible point θ of (P) is a KKT point if there exist λ 1 , . . .
, λ N ≥
such that
θ − (cid:80) Nn =
λ n h n = 0 for some h 1 , . .
, h N satisfying h n ∈ ∂ ◦ q n ( θ ) ; 2.
∀ n ∈ [ N ] : λ n ( q n ( θ ) − 1) = 0 .
Deﬁnition C.6 (Approximate KKT Point of (P)) .
A feasible point θ of (P) is an ( (cid:15), δ ) -KKT point of (P) if there exists λ 1 , . . .
, λ N ≥
such that
(cid:13)(cid:13)(cid:13) θ − (cid:80) Nn =1 λ n h n (cid:13)(cid:13)(cid:13) 2 ≤ (cid:15) for some h 1 , . . .
, h N satisfying h n ∈ ∂ ◦ q n ( θ ) ;
[Equation]
By the homogeneity of q n , it is easy to see that (P) satisﬁes MFCQ, and thus KKT conditions are ﬁrst-order necessary condition for global optimality.
Lemma C.7.
(P) satisﬁes MFCQ at every feasible point θ .
Proof.
Take v := θ .
For all n ∈ [ N ] satisfying q n = 1 , by homogeneity of q n , [Equation] v ,
[Equation]
= [Equation] holds for any h ∈ − ∂ ◦ q n ( θ ) =
∂ ◦ (1 − q n ( θ )) .
[Equation]
Deﬁne β ( t ) :=
[Equation]
[Equation]
ˆ θ , d θ dt (cid:69) to be the cosine of the angle between θ and d θ dt .
Here β ( t ) is only deﬁned for a.e. t > 0 .
Since q n is locally Lipschitz, it can be shown that q n is (globally) Lipschitz on the compact set S d − 1 , which is the unit sphere in R d .
Deﬁne
[Equation]
For showing Theorem A.8 and Theorem A.9, we ﬁrst prove Lemma C.8.
In light of this lemma, if we aim to show that θ is along the direction of an approximate KKT point, we only need to show β → 1 (which makes (cid:15) → 0 ) and L → 0 (which makes δ → 0 ).
Lemma C.8.
Let C 1 , C 2 be two constants deﬁned as
[Equation]
where K, b g are constants speciﬁed in (B3.4).
If log 1 L ≥ b g at time t > t 0 , then ˜ θ :
= θ /q min ( θ ) 1 /L is an ( (cid:15), δ ) -KKT point of (P), where (cid:15) :
= C 1 √ 1 − β and δ :
=
C 2 / (log 1 L ) .
Proof.
Let h ( t ) :
= d θ dt ( t ) for a.e. t > 0 .
By the chain rule, there exist h 1 , . .
, h N such that h n ∈ ∂ ◦ q n and h = (cid:80) n ∈
[ N ] e − f ( q n ) f (cid:48) ( q n ) h n .
Let ˜ h n :=
h n /q 1 − 1 /L
min ∈ ∂ ◦ q n (˜ θ ) (Recall that ˜ θ := θ /q min ( θ ) 1 /L ).
Construct λ n :=
q 1 − 2 /L min ρ · e − f ( q n ) f (cid:48) ( q n ) / (cid:107) h (cid:107) 2 .
Now we only need to show (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
− N (cid:88) n =
[Equation]
n
[Equation]
n (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)  ≤
[Equation]
1 −
[Equation]
Then ˜ θ can be shown to be an ( (cid:15), δ ) -KKT point by the monotonicity ˜ γ ( t ) ≥ ˜ γ ( t 0 ) for t > t 0 .
Proof of (12) .
From our construction, (cid:80) [Equation]
So
[Equation]
where the last equality is by Lemma A.5.
Proof for (13) .
According to our construction,
[Equation]
Note that (cid:107) h (cid:107) 2 ≥ (cid:68) h , ˆ θ (cid:69) =
Lν/ρ .
By Lemma B.5 and Lemma D.1, we have
[Equation]
where the last inequality uses f (˜ γρ L )
= log 1 L and L ≥ e − f ( q min ) .
Combining these gives
N (cid:88)
[Equation]
If q n > q min , then by the mean value theorem there exists ξ n ∈ ( q min , q n ) such that f ( q n ) − f ( q min ) =
f (cid:48) ( ξ n )( q n − q min ) .
By Assumption (B3.4), we know that f (cid:48) ( q n ) ≤ K (cid:100) log 2 ( q n /ξ n ) (cid:101) f (cid:48) ( ξ n ) .
Note that (cid:100) log 2 ( q n /ξ n ) (cid:101) ≤ log 2 (2 B 1 ρ L /q min ) ≤ log 2 (2 B 1 / ˜ γ ) .
Then we have
N (cid:88) n =
[Equation]
where the second inequality uses q − 2 /L
min ρ 2 ≤ ˜ γ − 2 /L by Lemma A.5 and the fact that the function x (cid:55)→ e − x x on (0 , + ∞ ) attains the maximum value e at x = 1 .
By Theorem A.7, we have already known that L → 0 .
So it remains to bound β ( t ) .
For this, we ﬁrst prove the following lemma to bound the integral of β ( t ) .
Lemma C.9.
For all t 2 > t 1 ≥ t 0 ,
[Equation]
t
[Equation]
Proof.
By Lemma B.4, ddt log ρ = 12 ρ 2 dρ 2 dt =
Lνρ 2 for a.e. t > 0 .
By Lemma B.1, for a.e. t ∈ ( t 1 , t 2 ) ,
[Equation]
By the chain rule, d ˆ θ dt = 1 ρ ( I − ˆ θ ˆ θ (cid:62) )
d θ dt .
So we have (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
where the last equality follows from the deﬁnition of β .
Combining 14 and 15, we have
[Equation]
Integrating on both sides from t 1 to t 2 proves the lemma.
A direct corollary of Lemma C.9 is the upper bound for the minimum β 2 − 1 within a time interval:
Corollary C.10.
For all t 2 > t 1 ≥ t 0 , then there exists t ∗ ∈ ( t 1 , t 2 ) such that
[Equation]
Proof.
Denote the RHS as C .
Assume to the contrary that β ( τ ) − 2 − 1 > C for a.e. τ ∈ ( t 1 , t 2 ) .
By
Lemma B.1, log ρ ( τ ) > 0 for a.e. τ ∈ ( t 1 , t 2 ) .
Then by Lemma C.9, we have
[Equation]
which leads to a contradiction.
In the rest of this section, we present both asymptotic and non-asymptotic analyses for the directional convergence by using Corollary C.10 to bound β ( t ) .
C. A SYMPTOTIC A NALYSIS
We ﬁrst prove an auxiliary lemma which gives an upper bound for the change of ˆ θ .
Lemma C.11.
For a.e. t > t 0 , (cid:13)(cid:13)(cid:13)(cid:13) (cid:13)
[Equation]
Proof.
Observe that (cid:13)(cid:13)(cid:13)
d ˆ θ dt (cid:13)(cid:13)(cid:13) 2 = 1 ρ (cid:13)(cid:13)(cid:13) ( I − ˆ θ ˆ θ (cid:62) ) d θ dt (cid:13)(cid:13)(cid:13) 2 ≤ 1 ρ (cid:13)(cid:13)
d θ dt (cid:13)(cid:13) 2 .
It is sufﬁcient to bound (cid:13)(cid:13)
d θ dt (cid:13)(cid:13) 2 .
By the chain rule, there exists h 1 , . .
, h N : [0 , + ∞ ) → R d satisfying that for a.e. t > 0 , h n ( t ) ∈ ∂ ◦ q n and d θ dt = (cid:80) n ∈ [ N ] e − f ( q n ) f (cid:48) ( q n ) h n ( t ) .
By deﬁnition of B 1 , (cid:107) h n (cid:107) 2 ≤ B 1 ρ L − 1 for a.e. t > 0 .
So we have
[Equation]
Note that every summand is positive.
By Lemma A.5, q n is lower-bounded by q n ≥ q min ≥ g (log 1 L ) , so we can replace q n with g (log 1 L ) in the above inequality.
Combining with the fact that (cid:80) n
[Equation]
] e −
[Equation]
n
[Equation]
f [Equation] ( q n ) q n is just ν , we have
[Equation]
So we have (cid:13)(cid:13)(cid:13) d ˆ
[Equation]
dt
[Equation]
≤
[Equation]
dt
[Equation]
[Equation] log ρ . To prove Theorem A.8, we consider each limit point ¯ θ /q min (¯ θ ) 1 /L , and construct a series of approximate KKT points converging to it.
Then ¯ θ /q min (¯ θ ) 1 /L can be shown to be a KKT point by
Theorem C.4.
The following lemma ensures that such construction exists.
Lemma C.12.
For every limit point ¯ θ of (cid:110)
ˆ θ ( t ) : t ≥ 0 (cid:111) , there exists a sequence of { t m : m ∈ N } such that t m ↑ + ∞ , ˆ θ ( t m )
→ ¯ θ , and β ( t m ) → 1 .
Proof.
Let { (cid:15) m > 0 : m ∈ N } be an arbitrary sequence with (cid:15) m → 0 .
Now we construct { t m } by induction.
Suppose t 1 < t 2 < · · · < t m − 1 have already been constructed.
Since ¯ θ is a limit point and ˜ γ ( t ) ↑ ˜ γ ∞ (recall that ˜ γ ∞ :
=
lim t → + ∞ ˜ γ ( t ) ), there exists s m > t m − 1 such that
[Equation]
and
[Equation]
Let s (cid:48) m > s m be a time such that log ρ ( s (cid:48) m )
= log ρ
( s m ) + (cid:15) m .
According to Theorem A.7, log ρ → + ∞ , so s (cid:48) m must exist.
We construct t m ∈ ( s m , s (cid:48) m ) to be a time that β ( t m ) − 2 − 1 ≤ (cid:15) 2 m , where the existence can be shown by Corollary C.10.
Now we show that this construction meets our requirement.
It follows from β ( t m ) − 2 − 1 ≤ (cid:15) 2 m that β ( t m ) ≥ 1 / (cid:112) 1 + (cid:15) 2 m → 1 .
By Lemma C.11, we also know that
[Equation]
This completes the proof.
Proof of Theorem A.8.
Let ˜ θ :=
¯ θ /q min (¯ θ ) 1 /L for short.
Let { t m : m ∈ N } be the sequence constructed as in Lemma C.12.
For each t m , deﬁne (cid:15) ( t m ) and δ ( t m ) as in Lemma C.8.
Then we know that θ ( t m ) /q
min ( t m ) 1 /L is an ( (cid:15) ( t m ) , δ ( t m ))
-KKT point and θ ( t m ) /q
min ( t m ) 1 /L
→ ˜ θ , (cid:15) ( t m ) → 0 , δ ( t m ) → 0 .
By Lemma C.7, (P) satisﬁes MFCQ.
Applying Theorem C.4 proves the theorem.
C. N ON - ASYMPTOTIC A NALYSIS
Proof of Theorem A.9.
Let C 0 :
= 1 L log ˜ γ ∞
˜ γ ( t 0 ) .
Without loss of generality, we assume (cid:15) < √ 62 Cand δ <
C 2 /f ( b f ) .
Let t 1 be the time such that log ρ ( t 1 )
= 1 L log g ( C 2 δ − 1 ) ˜ γ
( t 0 )
= Θ(log δ − 1 ) and t 2 be the time such that log ρ ( t 2 ) − log ρ ( t 1 ) =
12 C 0 C 21 (cid:15) − 2 =
Θ( (cid:15) − 2 ) .
By Corollary C.10, there exists t ∗ ∈ ( t 1 , t 2 ) such that β ( t ∗ ) − 2 − 1 ≤ 2 (cid:15) 2 C − 2 1 .
Now we argue that ˜ θ ( t ∗ ) is an ( (cid:15), δ ) -KKT point.
By Lemma C.8, we only need to show
C 1 (cid:112) 1 − β ( t ∗ ) ≤ (cid:15) and C 2 /f (˜ γ ( t ∗ ) ρ ( t ∗ ) L ) ≤ δ .
For the ﬁrst inequality, by assumption (cid:15) < √ 62 C 1 , we know that β ( t ∗ ) − 2 − 1 < 3 , which implies | β ( t ∗ ) | < 12 .
Then we have 1 − β ( t ∗ ) ≤ 2 (cid:15) 2 C − 2· β ( t ∗ ) 2 1+ β ( t ∗ ) ≤ (cid:15)
2 C − 2 1 .
Therefore, C 1 (cid:112) 1 − β ( t ∗ ) ≤ (cid:15) holds.
For the second inequality, ˜ γ ( t ∗ ) ρ ( t ∗ ) L ≥ ˜ γ ( t ∗ ) · g ( C 2 δ − 1 ) ˜ γ ( t 0 ) ≥ g ( C 2 δ − 1 ) .
Therefore, C 2 /f (˜ γ ( t ∗ ) ρ ( t ∗ ) L ) ≤ δ holds.
C.
P ROOF FOR C OROLLARY 4.
By the homogeneity of q n , we can characterize KKT points using kernel SVM.
Lemma C.13.
If θ ∗ is KKT point of (P), then there exists h n ∈ ∂ ◦ Φ x n ( θ ∗ ) for n ∈ [ N ] such that
L θ ∗ is an optimal solution for the following constrained optimization problem (Q):
[Equation]
Proof.
It is easy to see that (Q) is a convex optimization problem.
For θ = 2 L θ ∗ , from Theorem B.2, we can see that y n (cid:104) θ , h n (cid:105) =
2 q n ( θ ∗ ) ≥ 2 > 1 , which implies Slater’s condition.
Thus, we only need to show that 1 L θ ∗ satisﬁes KKT conditions for (Q).
By the KKT conditions for (P), we can construct h n ∈ ∂ ◦ q n ( θ ∗ ) for n ∈ [ N ] such that θ ∗ − (cid:80)
Nn =1 λ n y n h n = 0 for some λ 1 , . . .
, λ N ≥ 0 and λ n ( q n ( θ ∗ ) − 1) = 0 .
Thus, 1 L θ ∗ satisﬁes
1 L θ ∗ − (cid:80) Nn =1 1 L λ n y n h n = 0 ; 2.
1 L λ n (cid:0) y n (cid:10) 1 L θ ∗ , h n (cid:11) − 1 (cid:1)
=
1 L λ n ( q n ( θ ) − 1) ≥ 0 .
So 1 L θ ∗ satisﬁes KKT conditions for (Q).
Now we prove Corollary 4.5 in Section 4.3.
Proof.
By Theorem A.8, every limit point ¯ θ is along the direction of a KKT point of (P).
Combining this with Lemma C.13, we know that every limit point ¯ θ is also along the max-margin direction of (Q).
For smooth models, h n in (Q) is exactly the gradient ∇ Φ x n (¯ θ ) .
So, (Q) is the optimization problem for SVM with kernel K ¯ θ ( x , x (cid:48) )
= (cid:10) ∇ Φ x (¯ θ ) , ∇ Φ x (cid:48)
(¯ θ ) (cid:11) .
For non-smooth models, we can construct an arbitrary function h ( x ) ∈ ∂ ◦
Φ x (¯ θ ) that ensures h ( x n ) = h n .
Then, (Q) is the optimization problem for SVM with kernel K ¯ θ ( x , x (cid:48) )
= (cid:104) h
( x ) , h ( x (cid:48) ) (cid:105) .
D T IGHT B OUNDS FOR L OSS C ONVERGENCE AND W EIGHT G ROWTH
In this section, we give proof for Theorem A.10, which gives tight bounds for loss convergence and weight growth under Assumption (A1), (A2), (B3), (B4).
D. C ONSEQUENCES OF (B3.4)
Before proving Theorem A.10, we show some consequences of (B3.4).
Lemma D.1.
For f ( · ) and g ( · ) , we have
For all x ∈ [ b g , + ∞ ) , g ( x ) g (cid:48)
( x ) ∈ [ 12 K x, 2 Kx ] ; 2.
For all y ∈ [ g ( b g ) , + ∞ ) , f ( y ) f (cid:48) ( y ) ∈ [ 12 K y, 2 Ky ] .
Thus, g ( x ) =
Θ( xg (cid:48) ( x )) , f ( y )
=
Θ( yf (cid:48) ( y )) .
Proof.
To prove Item 1, it is sufﬁcient to show that
[Equation]
To prove Item 2, we only need to notice that Item 1 implies yf (cid:48)
( y ) =
g ( f ( y )) g (cid:48)
( f ( y )) ∈
[ 12 K f ( y ) , 2 Kf ( y )] for all y ∈
[ g ( b g ) , + ∞ ) .
Recall that (B3.4) directly implies that f (cid:48)
(Θ( x ))
=
Θ( f (cid:48) ( x )) and g (cid:48)
(Θ( x ))
=
Θ( g (cid:48) ( x )) .
Combining this with Lemma D.1, we have the following corollary:
Corollary D.2.
f (Θ( x ))
=
Θ( f ( x )) and g (Θ( x ))
=
Θ( g ( x )) .
Also, note that Lemma D.1 essentially shows that (log f ( x )) (cid:48)
=
Θ(1 /x ) and (log g ( x )) (cid:48) =
Θ(1 /x ) .
So log f ( x ) = Θ(log x ) and log g ( x ) = Θ(log x ) , which means that f and g grow at most polynomially.
Corollary D.3.
f ( x ) = x Θ(1) and g ( x ) = x Θ(1) .
D. P ROOF FOR T HEOREM A.
We follow the notations in Section 5 to deﬁne ρ := (cid:107) θ (cid:107) 2 and ˆ θ := θ (cid:107) θ (cid:107)
2 ∈ S
d − 1 , and sometimes we view the functions of θ as functions of t .
And we use the notations B 0 , B 1 from Appendix C.3.
The key idea to prove Theorem A.10 is to utilize Lemma B.6, in which L ( t ) is bounded from above by 1 G − 1 (Ω( t )) .
So upper bounding L ( t ) reduces to lower bounding G − 1 .
In the following lemma, we obtain tight asymptotic bounds for G ( · ) and G − 1 ( · ) :
Lemma D.4.
For function G ( · ) deﬁned in Lemma B.6 and its inverse function G − 1 ( · ) , we have the following bounds: [Equation] and G
[Equation]
Proof.
We ﬁrst prove the bounds for G ( x ) , and then prove the bounds for G − 1 ( y ) .
Bounding for G ( x ) .
Let C G = (cid:82) exp( b g ) 1 / L ( t 0 ) g (cid:48) (log u ) 2 g (log u ) 2 − 2 /L du . For x ≥ exp( b g ) , [Equation]
[Equation]
/L du
[Equation] (cid:18)
[Equation]
u
[Equation]
(cid:19)
[Equation]
(log
[Equation]
[Equation]
/L (log
[Equation]
du,
By Lemma D.1, G
[Equation]
[Equation] x exp [Equation] b g
[Equation] [Equation] log u )
[Equation]
du =
O (cid:18)
[Equation]
x
[Equation]
/L (log x
[Equation]
x (cid:19) .
On the other hand, for x ≥ exp(2 b g ) , we have
{+G ( x ) ≥ 4 K(cid:90) x √ x g (log u ) 2 /L (log u ) 2 du ≥ g ((log x ) / 2) 2 /L 4 K(cid:90) x √ x (log u ) 
+}
du = Ω (cid:18) g
[Equation]
[Equation]
(cid:19) .
Bounding for G − 1 ( y ) .
Let x = G − 1 ( y ) for y ≥ 0 .
G ( x ) always has a ﬁnite value whenever x is ﬁnite.
So x → + ∞ when y → + ∞ .
According to the ﬁrst part of the proof, we know that y
= Θ (cid:16) g (log x ) 2 /L
(log x ) 2 x (cid:17) .
Taking logarithm on both sides and using Corollary D.3, we have log y = Θ(log x ) .
By Corollary D.2, g (log y )
=
g (Θ(log x ))
=
Θ( g (log x )) .
Therefore,
[Equation]
This implies that x = Θ (cid:16) (log y ) 2 g (log y ) 2 /L y (cid:17) .
For other bounds, we derive them as follows.
We ﬁrst show that g (log 1 L ) =
Θ( ρ L ) .
With this equivalence, we derive an upper bound for the gradient at each time t in terms of L , and take an integration to bound L ( t ) from below.
Now we have both lower and upper bounds for L ( t ) .
Plugging these two bounds to g (log 1 L ) =
Θ( ρ L ) gives the lower and upper bounds for ρ ( t ) .
Proof for Theorem A.10.
We ﬁrst prove the upper bound for L .
Then we derive lower and upper bounds for ρ in terms of L , and use these bounds to give a lower bound for L .
Finally, we plug in the tight bounds for L to obtain the lower and upper bounds for ρ in terms of t .
Upper Bounding L .
By Lemma B.6, we have 1 L ≥ G − 1 (Ω( t )) .
Using Lemma D.4, we have
[Equation]
, which completes the proof.
Bounding ρ in Terms of L .
[Equation]
On the other hand, g (log 1 L ) ≤ q min ≤ B 0 ρ L .
So ρ L = Ω( g (log 1 L )) .
Therefore, we have the following relationship between ρ L and g (log 1 L ) :
[Equation]
Lower Bounding L .
Let h 1 , . .
, h N be a set of vectors such that h n ∈ ∂q n
[Equation]
θ and
[Equation]
By (17) and Corollary D.2, f (cid:48) ( q n ) = f (cid:48) ( O ( ρ L )) =
f (cid:48)
( O ( g (log 1 L )))
=
O ( f (cid:48)
( g (log 1 L )))
=
O (1 /g (cid:48) (log 1 L )) .
Again by (17), we have (cid:107) h n (cid:107) 2 ≤ B 1 ρ L − 1 =
O ( g (log 1 L ) 1 − 1 /L ) .
Combining these two bounds together, it follows from Corollary D.2 that
[Equation]
Thus, −
[Equation]
By deﬁnition of G ( · ) , this implies that there exists a constant c such that ddt G ( 1 L ) ≤ c for any L that is small enough.
We can complete our proof by applying Lemma D.4.
Bounding ρ in Terms of t .
By (17) and the tight bound for L ( t ) , ρ L = Θ( g (log 1 L )) =
Θ( g (Θ(log t ))) .
Using Corollary D.2, we can conclude that ρ L = Θ( g (log t )) .
E G RADIENT D ESCENT : S MOOTH H OMOGENEOUS M ODELS WITH E XPONENTIAL L OSS
In this section, we discretize our proof to prove similar results for gradient descent on smooth homogeneous models.
As usual, the update rule of gradient descent is deﬁned as
[Equation]
Here η ( t ) is the learning rate, and ∇L ( t ) :
= ∇L ( θ ( t )) is the gradient of L at θ ( t ) .
We ﬁrst focus on the exponential loss.
At the end of this section (Appendix F), we discuss how to extend the proof to general loss functions with a similar assumption as (B3).
The main difﬁculty for discretizing our previous analysis comes from the fact that the original version of the smoothed normalized margin ˜ γ ( θ ) :
= ρ − L log 1 L becomes less smooth when ρ → + ∞ .
Thus, if we take a Taylor expansion for ˜ γ ( θ ( t + 1)) from the point θ ( t ) , although one can show that the ﬁrst-order term is positive as in the gradient ﬂow analysis, the second-order term is unlikely to be bounded during gradient descent with a constant step size.
To get a smoothed version of the normalized margin that is monotone increasing, we need to deﬁne another one that is even smoother than ˜ γ .
Technically, recall that d L dt = −(cid:107)∇L(cid:107) 22 does not hold exactly for gradient descent.
However, if the smoothness can be bounded by s ( t ) , then it is well-known that
[Equation]
By analyzing the landscape of L , one can easily ﬁnd that the smoothness is bounded locally by
O ( L · polylog( 1 L )) .
Thus, if we set η ( t ) to a constant or set it appropriately according to the loss, then this discretization error becomes negligible.
Using this insight, we deﬁne the new smoothed normalized margin ˆ γ in a way that it increases slightly slower than ˜ γ during training to cancel the effect of discretization error.
E. A SSUMPTIONS
As stated in Section 4.1, we assume (A2), (A3), (A4) similarly as for gradient ﬂow, and two additional assumptions (S1) and (S5).
(S1).
(Smoothness).
For any ﬁxed x , Φ( · ; x ) is C 2 -smooth on R d \ { 0 } . (A2).
(Homogeneity).
There exists L > 0 such that ∀ α > 0 : Φ( α θ ; x ) =
α L Φ( θ ; x ) ; (A3).
(Exponential Loss).
(cid:96) ( q ) = e − q ; (A4). (Separability).
There exists a time t 0 such that L ( θ ( t 0 )) < 1 .
(S5).
(Learing rate condition).
(cid:80) t ≥ 0 η ( t ) = + ∞ and η ( t ) ≤ H ( L ( θ ( t ))) .
Here H ( L ) is a function of the current training loss.
The explicit formula of H ( L ) is given below:
[Equation]
where C η is a constant, and κ ( x ) , µ ( x ) are two non-decreasing functions.
For constant learning rate η ( t ) =
η 0 , (S5) is satisﬁed when η 0 if sufﬁciently small.
Roughly speaking, C η κ ( x ) is an upper bound for the smoothness of L in a neighborhood of θ when x
= L ( θ ) .
And we set the learning rate η ( t ) to be the inverse of the smoothness multiplied by a factor µ ( x ) =
o (1) .
In our analysis, µ ( x ) can be any non-decreasing function that maps (0 , L ( t 0 )] to (0 , 1 / 2] and makes the integral (cid:82) 1 / 2µ ( x ) dx exist.
But for simplicity, we deﬁne µ ( x ) as
[Equation]
The value of C η will be speciﬁed later.
The deﬁnition of κ ( x ) depends on L .
For 0 < L ≤ 1 , we deﬁne κ ( x ) as κ ( x ) :
= x
[Equation]
x
[Equation]
2 − 2 /L
For L > 1 , we deﬁne κ ( x ) as
[Equation]
where κ max :=
e (2 − 2 /L )
(ln(2 − 2 /L ) − 1) .
The speciﬁc meaning of C η , κ ( x ) and µ ( x ) will become clear in our analysis.
E. S MOOTHED N ORMALIZED M ARGIN
Now we deﬁne the smoothed normalized margins.
As usual, we deﬁne ˜ γ
( θ ) :
= log 1 L ρ L .
At the same time, we also deﬁne
[Equation]
Here φ : (0 , L ( t 0 )]
→ (0 , + ∞ ) is constructed as follows.
Construct the ﬁrst-order derivative of φ ( x ) as
[Equation]
where λ ( x ) :
= (log 1 x ) − 1 .
And then we set φ ( x ) to be
[Equation]
It can be veriﬁed that φ ( x ) is well-deﬁned and φ (cid:48)
( x ) is indeed the ﬁrst-order derivative of φ ( x ) .
Moreover, we have the following relationship among ˆ γ, ˜ γ and ¯ γ .
Lemma E.1.
ˆ γ ( θ ) is well-deﬁned for L ( θ )
≤ L ( t 0 ) and has the following properties:
(a)
If L ( θ ) ≤ L ( t 0 ) , then ˆ γ ( θ ) <
˜ γ ( θ ) ≤ ¯ γ ( θ ) .
(b) Let { θ m ∈ R d : m ∈ N } be a sequence of parameters.
If L ( θ m ) → 0 , then
[Equation]
Proof.
First we verify that ˆ γ is well-deﬁned.
To see this, we only need to verify that
[Equation]
exists for all x ∈ (0 , L ( t 0 )] , then it is trivial to see that φ (cid:48) ( w ) is indeed the derivative of φ ( w ) by
I (cid:48)
( x ) = φ (cid:48)
( x ) + 1 x log 1 x .
Note that I ( x ) exists for all x ∈ (0 , L ( t 0 )] as long as I ( x ) exists for a small enough x > 0 .
By deﬁnition, it is easy to verify that r ( w ) :
= 1+2(1+ λ ( w ) /L )
µ ( w ) w log 1 w is decreasing when w is small enough.
Thus, for a small enough w > 0 , we have
[Equation]
So we have the following for small enough x :
[Equation]
This proves the existence of I ( x ) .
Now we prove (a).
By Lemma A.5, ˜ γ ( θ ) ≤ ¯ γ ( θ ) , so we only need to prove that ˆ γ ( θ ) <
˜ γ ( θ ) .
To see this, note that for all w ≤ L ( t 0 ) , r ( w ) > 1 w log 1 w .
So we have − φ (cid:48) ( w ) >
1 w log 1 w , and this implies I ( x ) < 0 for all x ≤ L ( t 0 ) .
Then it holds for all θ with L ( θ ) ≤ L ( t 0 ) that
[Equation]
To prove (b), we combine (19) and (20), then for small enough L ( θ m ) , we have
[Equation]
· log 1 L ( θ m ) q min ( θ m ) =
exp (cid:32) − O (cid:32) log 1 L ( θ m ) (cid:33)(cid:33) · log 1 L ( θ m ) log 1 L ( θ m )
+ O (1)
= 1 − O (cid:32)(cid:18) log
L ( θ m ) (cid:19) − 1 (cid:33) .
So ˆ γ ( θ ) ¯ γ ( θ ) = 1 − O (cid:16) (log 1 L ( θ m ) ) − 1 (cid:17) → 1 .
Now we specify the value of C η .
By (S1) and (S2), we can deﬁne B 0 , B 1 , B 2 as follows:
[Equation]
Then we set[-C η := 1 2 (cid:0) B 2 
 + ρ ( t 0 ) − L B 2 (cid:1) ˆ γ ( t 0 )-]
[Equation]
E. T HEOREMS
Now we state our main theorems for the monotonicity of the normalized margin and the convergence to KKT points.
We will prove Theorem E.2 in Appendix E.4, and prove Theorem E.3 and E.4 in
Appendix E.5.
Theorem E.2.
Under assumptions (S1), (A2) - (A4), (S5), the following are true for gradient descent:
For all t ≥ t 0 , ˆ γ ( t + 1) ≥ ˆ γ ( t ) ; 2.
For all t ≥ t 0 , either ˆ γ ( t + 1) > ˆ γ ( t ) or ˆ θ ( t + 1) =
ˆ θ ( t ) ; 3.
L ( t ) → 0 and ρ ( t ) → ∞ as t → + ∞ ; therefore, | ¯ γ ( t ) − ˜ γ ( t ) | → 0 .
Theorem E.3.
Consider gradient ﬂow under assumptions (S1), (A2) - (A4), (S5).
For every limit point ¯ θ of (cid:110)
ˆ θ ( t ) : t ≥ 0 (cid:111) , ¯ θ /q min (¯ θ ) 1 /L is a KKT point of (P).
Theorem E.4.
Consider gradient descent under assumptions (S1), (A2) - (A4), (S5).
For any (cid:15), δ > 0 , there exists r := Θ(log δ − 1 ) and ∆ :
= Θ( (cid:15) − 2 ) such that θ /q min ( θ ) 1 /L is an ( (cid:15), δ ) -KKT point at some time t ∗ satisfying log ρ ( t ∗ ) ∈
( r, r + ∆) .
With a reﬁned analysis, we can also derive tight rates for loss convergence and weight growth.
We defer the proof to Appendix E.6.
Theorem E.5.
Under assumptions (S1), (A2) - (A4), (S5), we have the following tight rates for training loss and weight norm:
[Equation]
where T = (cid:80) t − 1 τ = t 0 η ( τ ) .
E. P ROOF FOR T HEOREM E.
We deﬁne ν ( t ) :
= (cid:80) Nn =1 e − q n ( t ) q n ( t ) as we do for gradient ﬂow.
Then we can get a closed form for (cid:104) θ ( t ) , −∇L ( t ) (cid:105) easily from Corollary B.3.
Also, we can get a lower bound for ν ( t ) using
Lemma B.5 for exponential loss directly.
Corollary E.6.
(cid:104)
θ ( t ) ,
−∇L ( t ) (cid:105) =
Lν ( t ) .
If L ( t ) < 1 , then ν ( t ) ≥ L ( t ) λ ( L ( t )) .
As we are analyzing gradient descent, the norm of ∇L and ∇ 2 L appear naturally in discretization error terms.
To bound them, we have the following lemma when ˜ γ ( θ ) and ρ ( θ ) have lower bounds:
Lemma E.7. For any θ , if ˜ γ ( θ ) ≥ ˆ γ ( t 0 ) , ρ ( θ ) ≥ ρ ( t 0 ) , then (cid:107)∇L ( θ ) (cid:107) 2 2 ≤ 2 C η κ ( L ( θ )) L ( θ ) and (cid:13)(cid:13) ∇ 2 L ( θ ) (cid:13)(cid:13) 2 ≤ 2 C η κ ( L ( θ )) .
Proof.
By the chain rule and the deﬁnitions of B 1 , B 2 , we have
[Equation]
Note that ˆ
[Equation]
Combining all these formulas together gives
[Equation]
which completes the proof.
For proving the ﬁrst two propositions in Theorem E.2, we only need to prove Lemma E.8.
(P1) gives a lower bound for ˜ γ .
(P2) gives both lower and upper bounds for the weight growth using ν ( t ) .
(P3) gives a lower bound for the decrement of training loss.
Finally, (P4) shows the monotonicity of ˆ γ , and it is trivial to deduce the ﬁrst two propositions in Theorem E.2 from (P4).
Lemma E.8.
For all t = t 0 , t 0 + 1 , . . .
, we interpolate between θ ( t ) and θ ( t + 1) by deﬁning θ ( t + α )
= θ ( t ) − αη ( t ) ∇L ( t ) for α ∈ (0 , 1) .
Then for all integer t ≥ t 0 , ν ( t ) > 0 , and the following holds for all α ∈ [0 , 1] : (P1). ˜ γ ( t + α ) >
ˆ γ ( t 0 ) .
(P2).
[Equation]
[Equation]
[Equation]
( t )
[Equation] [Equation] [Equation]
(P3).
L ( t + α ) − L ( t ) ≤ − αη ( t )(1 − µ
( L ( t )))
[Equation] ( t ) (cid:107) 22 . (P4).
log ˆ γ ( t + α ) − log ˆ γ ( t ) ≥ ρ ( t ) 2 Lν ( t ) 2 (cid:13)(cid:13)(cid:13)(cid:16) I − ˆ θ ( t )ˆ θ ( t ) (cid:62) (cid:17) ∇L ( t ) (cid:13)(cid:13)(cid:13)
2 2 · log ρ ( t + α ) ρ ( t ) .
To prove Lemma E.8, we only need to prove the following lemma and then use an induction:
Lemma E.9.
Fix an integer T ≥ t 0 .
Suppose that (P1), (P2), (P3), (P4) hold for any t + α ≤ T .
Then if (P1) holds for ( t, α ) ∈ { T } × [0 , A ) for some A ∈ (0 , 1] , then all of (P1), (P2), (P3), (P4) hold for ( t, α ) ∈ { T } × [0 , A ] .
Proof for Lemma E.8.
We prove this lemma by induction.
For t = t 0 , α = 0 , ν ( t ) > 0 by (S4) and
Corollary E.6.
(P2), (P3), (P4) hold trivially since log ˆ γ ( t + α )
=
log ˆ γ ( t ) , L ( t + α )
=
L ( t ) and log ˆ γ ( t + α ) =
log ˆ γ ( t ) .
By Lemma E.1, (P1) also holds trivially.
Now we ﬁx an integer T ≥ t 0 and assume that (P1), (P2), (P3), (P4) hold for any t + α ≤ T (where t ≥ t 0 is an integer and α ∈ [0 , 1] ).
By (P3), L ( t ) ≤ L ( t 0 ) < 1 , so ν ( t ) > 0 .
We only need to show that (P1), (P2), (P3), (P4) hold for t =
T and α ∈ [0 , 1] .
Let A :=
inf { α ∈ [0 , 1] : α = 1 or (P1) does not hold for ( T, α ) } .
If A = 0 , then (P1) holds for ( T, A ) since (P1) holds for ( T − 1 , 1) ; if A > 0 , we can also know that (P1) holds for ( T, A ) by
Lemma E.9.
Suppose that A < 1 .
Then by the continuity of ˜ γ ( T + α ) (with respect to α ), we know that there exists A (cid:48) >
A such that ˜ γ ( T + α ) >
ˆ γ ( t 0 ) for all α ∈ [ A, A (cid:48) ] , which contradicts to the deﬁnition of A .
Therefore, A = 1 .
Using Lemma E.9 again, we can conclude that (P1), (P2), (P3), (P4) hold for t =
T and α ∈ [0 , 1] .
Now we turn to prove Lemma E.9.
Proof for Lemma E.9.
Applying (P3) on ( t, α ) ∈ { t 0 , . .
, T − 1 } × 1 , we have L ( t ) ≤ L ( t 0 ) < 1 .
Then by Corollary E.6, we have ν ( t ) > 0 .
Applying (P2) on ( t, α ) ∈ { t 0 , . .
, T − 1 } × 1 , we can get ρ ( t ) ≥ ρ ( t 0 ) .
Fix t = T .
By (P2) with α ∈ [0 , A ) and the continuity of ˜ γ , we have ˜ γ ( t + A ) ≥ ˆ γ ( t 0 ) .
Thus, ˜ γ ( t + α ) ≥ ˆ γ ( t 0 ) for all α ∈ [0 , A ] .
Proof for (P2).
By deﬁnition, ρ ( t + α ) 2 = [Equation] θ [-( t ) − αη ( t ) ∇L ( t ) (cid:107) 
-][Equation]
t ) [-
-]− [Equation] ∇L [Equation]
[Equation] [Equation] [Equation] (cid:107) 22 − 2 αη ( t ) (cid:104) θ ( t ) , ∇L ( t ) (cid:105) .
By Corollary E.6, we have
[Equation]
So ρ ( t + α ) 2 − ρ ( t ) 2 ≥ 2 Lαη ( t ) ν
( t ) .
For the other direction, we have the following using Corollary E.6 and [-(cid:107)∇L ( t ) (cid:107) 
 = (cid:13)(cid:13)(cid:13)(cid:80) Nn =1 e − q n ( t ) ∇ q n ( t ) (cid:13)(cid:13)(cid:13) 2 ≤ B 1 L ( t ) ρ ( t )-]Lemma E.7,
[Equation]
= 2 Lαη ( t ) ν ( t ) (1 + λ ( L ( t ))
µ ( L ( t )) /L ) .
Proof for (P3).
(P3) holds trivially for α = 0 or ∇L ( t )
= 0 .
So now we assume that α (cid:54) = 0 and ∇L ( t ) (cid:54)
= 0 .
By the update rule (18) and Taylor expansion, there exists ξ ∈ (0 , α ) such that
[Equation]
By Lemma E.7, (cid:13)(cid:13) ∇ 2 L ( t + ξ ) (cid:13)(cid:13) 2 ≤ 2 C η · κ ( L ( t + ξ )) , so we have
[Equation]
Now we only need to show that L ( t + α ) <
L ( t ) for all α ∈ (0 , A ] .
Assuming this, we can have κ ( L ( t + ξ )) ≤ κ ( L ( t )) by the monotonicity of κ , and thus
[Equation]
Now we show that L ( t + α ) <
L ( t ) for all α ∈ (0 , A ] .
Assume to the contrary that α 0 := inf { α (cid:48) ∈ (0 , A ] : L ( t + α (cid:48) )
≥ L ( t ) } exists.
If α 0 = 0 , let p : [0 , 1] → R , α (cid:55)→ L ( t + α ) , then p (cid:48) (0) =
lim
α ↓ 0 L ( t + α ) −L ( t ) α ≥ 0 , but it contradicts to p (cid:48) (0) = − η ( t ) (cid:107)∇L ( t ) (cid:107) 22 < 0 .
If α 0 > 0 , then by the monotonicity of κ we have κ ( L ( t + ξ ))
< κ ( L ( t )) for all ξ ∈ (0 , α 0 ) , and thus
[Equation]
, which leads to a contradiction.
Proof for (P4).
We deﬁne v ( t ) :
= ˆ θ ( t )ˆ θ ( t ) (cid:62) ( −∇L ( t )) and u ( t ) :
= (cid:16) I − ˆ θ [Equation] similarly as in the analysis for gradient ﬂow.
For v ( t ) , we have
[Equation]
Decompose (cid:107)∇L ( t ) {+(cid:107) 
+}
[-1 ρ ( t ) · Lν ( t ) . Decompose (cid:107)∇L ( t ) (cid:107) 
-] = (cid:107) v ( t ) (cid:107)
+ (cid:107) u ( t ) (cid:107) 22 .
Then by (P3), we have
L ( t + α ) − L ( t ) ≤ − αη ( t )(1 − µ ( L ( t )))
(cid:18)ρ ( t ) 2 · L 2 ν ( t )
+ (cid:107) u
(cid:107)(cid:19) .
Multiplying 1+ λ ( L ( t ))
µ ( L ( t )) /L
(1 − µ ( L ( t )))
ν ( t ) on both sides, we have 1 + [Equation] /L (
[Equation]
− µ ( L ( t )))
[Equation]
( t ) [Equation] L (cid:19) [Equation]
ρ ( t [Equation] (cid:107) u (cid:107)ν ( t ) (cid:33)
By Corollary E.6, we can bound ν ( t ) by ν ( t ) ≥ L ( t ) /λ ( L ( t )) . By (P2), we have the inequality αη ( t ) ν
{+( t ) (cid:16) 
+}
+ λ ( L ( t )) µ ( L ( t )) L (cid:17) ≥ ρ ( t + α ) 2 − ρ ( t ) 2 2 L .
So we further have 1 + λ ( L ( t )) µ ( L ( t )) /L
(1 − µ ( L ( t )))
L ( t ) /λ ( L ( t ))
[Equation]
L ( t + α ) −L
[Equation]
t
[Equation]
≤ −
[Equation]
(cid:107)
[Equation]
(cid:107)(cid:19)
From the deﬁnition φ , it is easy to see that − φ (cid:48)
( L ( t )) ≥ 1+ λ ( L ( t ))
µ ( L ( t )) /L (1 − µ ( L ( t ))) L ( t ) /λ ( L ( t )) .
Let ψ ( x ) = − log x , then ψ (cid:48)
( x ) = − 1 x .
Combining these together gives
[Equation]
Then by convexity of φ and ψ , we have
[Equation]
And by deﬁnition of ˆ γ , this can be re-written as
[Equation]
Proof for (P1).
By (P4), log ˆ γ ( t + α ) ≥ log ˆ γ ( t ) ≥ log ˆ γ ( t 0 ) .
Note that φ ( x ) ≥ log log 1 x .
So we have
[Equation]
which completes the proof.
For showing the third proposition in Theorem E.2, we use (P1) to give a lower bound for (cid:107)∇L ( t ) (cid:107) 2 , and use (P3) to show the speed of loss decreasing.
Then it can be seen that L ( t ) → 0 and ρ ( t ) → + ∞ .
By Lemma E.1, we then have | ¯ γ − ˆ γ | → 0 .
Lemma E.10.
Let E 0 :
=
L ( t 0 ) 2 (log 1 L ( t [Equation] ) 2 − 2 /L .
Then for all t > t 0 ,
[Equation]
η
[Equation]
for E ( x ) :
= (cid:90) L ( t 0 ) x min { u 2 (log 1 u ) 2 − 2 /L , E 0 } du.
Therefore, L ( t ) → 0 and ρ ( t ) →
+ ∞ as t → ∞ .
Proof.
For any integer t ≥ t 0 , µ ( L ( t )) ≤ 12 and (cid:107)∇L ( t ) (cid:107) 2 ≥ (cid:107) v ( t ) (cid:107) 2 .
Combining these with (P3), we have
[Equation]
By (P1), ρ ( t ) − 2 ≤ ˆ γ ( t 0 ) 2 /L (cid:16) log 1 L ( t ) (cid:17) − 2 /L .
By Corollary E.6, ν ( t ) 2 ≥ L ( t ) 2 (cid:16) log 1 L ( t ) (cid:17) 2 .
Thus we have
[Equation]
It is easy to see that 1 u 2 (log 1 u ) 2 − 2 /L is unimodal in (0 , 1) , so E (cid:48)
( x ) is non-decreasing and E ( x ) is convex.
So we have
[Equation]
which proves E ( L ( t )) ≥ 12 L 2 ˆ γ ( t 0 ) 2 /L (cid:80) t − 1 τ = t 0 η ( τ ) .
Note that L is non-decreasing.
If L does not decreases to 0 , then neither does E ( L ) .
But the RHS grows to + ∞ , which leads to a contradiction.
So L → 0 .
To make L → 0 , q min must converge to + ∞ .
So ρ → + ∞ .
E. P ROOF FOR T HEOREM E.3 AND E.
The proofs for Theorem E.3 and E.4 are similar as those for Theorem A.8 and A.9 in Appendix C.
Deﬁne β ( t ) :=
[Equation]
( t
[Equation]
(cid:68) ˆ θ , −∇L ( t ) (cid:69) as we do in Appendix C. It is easy to see that Lemma C. still holds if we replace ˜ γ ( t 0 ) with ˆ γ ( t 0 ) .
So we only need to show L → 0 and β → 1 for proving convergence to KKT points.
L → 0 can be followed from Theorem E.2.
Similar as the proof for
Lemma C.9, it follows from Lemma E.8 and (15) that for all t 2 > t 1 ≥ t 0 ,
[Equation]
Now we prove Theorem E.4.
Proof for Theorem E.4.
We make the following changes in the proof for Theorem A.9.
First, we replace ˜ γ ( t 0 ) with ˆ γ ( t 0 ) , since ˜ γ ( t ) ( t ≥ t 0 ) is lower bounded by ˆ γ ( t 0 ) rather than ˜ γ ( t 0 ) .
Second, when choosing t 1 and t 2 , we make log ρ ( t 1 ) and log ρ ( t 2 ) equal to the chosen values approximately with an additive error o (1) , rather than make them equal exactly.
This is possible because it can be shown from (P2) in Lemma E.8 that the following holds:
[Equation]
Dividing ρ ( t ) 2 on the leftmost and rightmost sides, we have
[Equation]
+ o ( ρ ( t ) − L )
=
1 + o (1) , which implies that log ρ ( t + 1) − log ρ ( t ) =
o (1) .
Therefore, for any R , we can always ﬁnd the minimum time t such that log ρ ( t ) ≥ R , and it holds for sure that log ρ ( t ) − R → 0 as R → + ∞ .
For proving Theorem E.3, we also need the following lemma as a variant of Lemma C.11.
Lemma E.11.
For all t ≥ t 0 ,
[Equation]
Proof.
According to the update rule, we have
[Equation]
=
[Equation] [Equation] [Equation]
By [Equation] [Equation] ( t ) (cid:107) 2 ≤ B 1 ν ( t ) ˜
[Equation]
t ) ρ ( t ) . So we can bound the ﬁrst term as
[Equation]
where the last inequality uses the inequality a − ba ≤ log( a/b ) .
Using this inequality again, we can bound the second term by
[Equation]
Combining these together gives
[Equation]
Now we are ready to prove Theorem E.3.
Proof for Theorem E.3.
As discussed above, we only need to show a variant of Lemma C.12 for gradient descent: for every limit point ¯ θ of (cid:110)
ˆ θ ( t ) : t ≥ 0 (cid:111) , there exists a sequence of { t m : m ∈ N } such that t m ↑ + ∞ , ˆ θ ( t m )
→ ¯ θ , and β ( t m ) → 1 .
We only need to change the choices of s m , s (cid:48) m , t m in the proof for Lemma C.12.
We choose s m > t m − 1 to be a time such that
[Equation]
and
[Equation]
Then we let s (cid:48) m > s m be the minimum time such that log ρ ( s (cid:48) m ) ≥ log ρ ( s m ) + (cid:15) m .
According to
Theorem E.2, s m and s (cid:48) m must exist.
Finally, we construct t m ∈ { s m , . . .
, s (cid:48)
m − 1 } to be a time that β ( t m ) − 2 − 1 ≤ (cid:15) 2 m , where the existence can be shown by (22).
To see that this construction meets our requirement, note that β ( t m ) − 2 − 1 ≤ (cid:15) 2 m → 0 and
[Equation]
where the last inequality is by Lemma E.11.
E. P ROOF FOR T HEOREM E.
Proof.
By a similar analysis as Lemma D.4, we have
[Equation]
We can also bound the inverse function E − 1 ( y ) by Θ (cid:16)
[Equation]
With these, we can use a similar analysis as Theorem A.10 to prove Theorem E.5.
First, using a similar proof as for (17), we have ρ L = Θ(log 1 L ) .
So we only need to show L ( t ) =
Θ( 1 T (log T ) 2 − 2 /L ) .
With a similar analysis as for (P3) in Lemma E.9, we have the following bound for L ( τ + 1) − L ( τ ) :
[Equation]
Using the fact that µ ≤ 1 / 2 , we have L ( τ + 1) − L ( τ ) ≥ − 32 η ( τ ) (cid:107)∇L ( τ ) (cid:107) 22 .
By Lemma E.7, (cid:107)∇L ( τ ) (cid:107) 2 ≤ 2 C η κ ( L ( τ )) L ( τ ) .
Using a similar proof as for Lemma E.10, we can show that E ( L ( t )) ≤ O ( T ) .
Combining this with Lemma E.10, we have E ( L ( t )) =
Θ( T ) .
Therefore, L ( t ) = Θ( 1 T [Equation]
T [Equation] 2 − 2 /L ) .
F G RADIENT D ESCENT : G ENERAL L OSS F UNCTIONS
It is worth to note that the above analysis can be extended to other loss functions.
For this, we need to replace (B3) with a strong assumption (S3), which takes into account the second-order derivatives of f .
(S3).
The loss function (cid:96) ( q ) can be expressed as (cid:96) ( q ) =
e − f ( q ) such that
(S3.1).
f : R → R is C 2 -smooth.
(S3.2).
f (cid:48) ( q ) > 0 for all q ∈ R .
(S3.3).
There exists b f ≥ 0 such that f (cid:48) ( q ) q is non-decreasing for q ∈ ( b f , + ∞ ) , and f (cid:48) ( q ) q → + ∞ as q → + ∞ .
(S3.4).
Let g :
[ f ( b f ) , + ∞ ) → [ b f , + ∞ ) be the inverse function of f on the domain [ b f , + ∞ ) .
{+There exists p ≥ 
+}
such that for all x > f ( b f ) , y > b f ,
[Equation]
p x and
[Equation]
It can be veriﬁed that (S3) is satisﬁed by exponential loss and logistic loss.
Now we explain each of the assumptions in (S3).
(S3.2) and (S3.3) are essentially the same as (B3.2) and (B3.3).
(S3.1) and (B3.1) are the same except that (S3.1) assumes f is C 2 -smooth rather than C 1 -smooth.
(S3.4) can also be written in the following form:
(cid:12)(cid:12)(cid:12)(cid:12)
d dx log g (cid:48)
( x ) (cid:12)(cid:12)(cid:12)(cid:12) ≤ p · d dx log x and (cid:12)(cid:12)(cid:12)(cid:12)
d dy log f (cid:48)
[Equation]
That is, log g (cid:48)
( x ) and log f (cid:48) ( y ) grow no faster than O (log x ) and O (log y ) , respectively.
In fact, (B3.4) can be deduced from (S3.4).
Recall that (B3.4) ensures that Θ( g (cid:48) ( x )) =
g (cid:48)
(Θ( x )) and Θ( f (cid:48) ( y ))
=
f (cid:48)
(Θ( y )) .
Thus, (S3.4) also gives us the interchangeability between f (cid:48) , g (cid:48) and Θ .
Lemma F.1.
(S3.4) implies (B3.4) with b g = max { 2 f ( b f ) , f (2 b f ) } and K = 2 p .
Proof.
Fix x ∈ ( b g , + ∞ ) , y ∈ ( g ( b g ) , + ∞ ) and θ ∈ [1 / 2 , 1) .
Integrating (23) on both sides of the inequalities from θx to x and θy to y , we have
[Equation]
and
[Equation]
Therefore, we have g (cid:48)
( x ) ≤ θ − p g (cid:48) ( θx ) ≤ Kg (cid:48) (
[Equation]
and f (cid:48) ( y ) ≤ θ − p f (cid:48) ( θy ) ≤ Kf (cid:48) ( θx ) .
To extend our results to general loss functions, we need to redeﬁne κ ( x ) , λ ( x ) , φ ( x ) , C η , H ( x ) in order.
For κ ( x ) and λ ( x ) , we redeﬁne them as follows:
[Equation]
[Equation]
deﬁned
[Equation]
Using λ ( x ) , we can deﬁne φ ( x ) and ˆ γ ( θ ) as follows.
[Equation]
Using a similar argument as in Lemma E.1, we can show that ˆ γ ( θ ) is well-deﬁned and ˆ γ ( θ )
<
˜ γ ( θ ) :
= g (log 1 L ) /ρ
L ≤ ¯ γ ( θ ) .
When L ( θ ) → 0 , we also have ˆ γ ( θ ) /
¯ γ ( θ ) → 1 .
For C η , we deﬁne it to be the following.
[Equation]
Finally, the deﬁnitions for µ ( x ) :
= (log
[Equation]
(2 log 1 x ) and H ( x ) :
= µ ( x ) / ( C η κ ( x )) in (S5) remain unchanged except that κ ( x ) and C η now use the new deﬁnitions.
Similar as gradient ﬂow, we deﬁne ν ( t ) :
= (cid:80) Nn =1 e − f ( q n ( t ))
f (cid:48) ( q n ( t ))
q n ( t ) .
The key idea behind the above deﬁnitions is that we can prove similar bounds for ν ( t ) , (cid:107)∇L(cid:107) 2 , (cid:107)∇ 2 L(cid:107) 2 as Corollary E. and Lemma E.7.
Lemma F.2.
(cid:104)
θ ( t ) ,
−∇L ( t ) (cid:105) =
Lν ( t ) .
If L ( t ) < e − f ( b f ) , then ν ( t ) ≥ L ( t ) λ
( L ( t )) and λ ( L ( t )) has the lower bound λ ( L ( t ))
≤ 2 p +1 (cid:16) log 1 L ( t ) (cid:17) − 1 .
Proof.
It can be easily proved by combining Theorem B.4, Lemma B.5 and Lemma D.1 together.
Lemma F.3.
For any θ , if L ( θ ) ≤ L ( t 0 ) , ˜ γ ( θ ) ≥ ˆ γ ( t 0 ) , then (cid:107)∇L ( θ ) (cid:107) 22 ≤ 2 C η κ ( L ( θ ))
L ( θ ) and (cid:13)(cid:13) ∇ 2 L ( θ ) (cid:13)(cid:13) 2 ≤ 2 C η κ ( L ( θ )) .
Proof.
Note that a direct corollary of (23) is that f (cid:48) ( x 1 ) ≤ f (cid:48) ( x 2 )( x 1 /x 2 ) p for x 1 ≥ x 2 > b f .
So we have
[Equation]
(cid:19)
[Equation]
log
[Equation]
log
[Equation]
R g (cid:48) (log 1 L ) , where R :=
( B 0 / ˆ γ ( t 0 )) p .
Applying (S3.4), we can also deduce that | f
[Equation]
Now we bound (cid:107)∇L(cid:107) 2 and (cid:13)(cid:13) ∇ 2 L (cid:13)(cid:13) 2 .
By the chain rule, we have (cid:107)∇L(cid:107)
[Equation]
−
N
[Equation]
∇
[Equation]
(cid:1)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ≤
N
[Equation]
For ρ we have ˆ γ ( t 0 )
ρ L ≤ log 1 L ≤ B 0 ρ L , so we can bound ρ 2 L − 2 by ρ 2 L − 2 ≤ M (log 1 L ) 2 L − 2 , where M :=
min (cid:110)
ˆ γ ( t 0 ) − 2+2 /L , B − 2+2 /L(cid:111) .
By Lemma F.2, we have λ ( L ) ≤ 2 p +1 1 log 1 L ≤ 2 p +1 log 1 L ( t 0) , and also g (cid:48) (log 1 L ) ρ L
[Equation]
p
[Equation]
Combining all these formulas together gives
[Equation]
which completes the proof.
With Corollary E.6 and Lemma E.7, we can prove Lemma E.8 with exactly the same argument.
Then Theorem E.2, E.3, E.4 can also be proved similarly.
For Theorem E.5, we can follow the argument for gradient ﬂow to show that it holds with slightly different tight bounds:
Theorem F.4.
Under assumptions (S1), (A2), (S3), (A4), (S5), we have the following tight rates for training loss and weight norm:
[Equation]
T
[Equation]
log T
[Equation]
(cid:19) and ρ ( t ) =
Θ( g (log T ) 1 /L ) , where T = (cid:80) t − 1 τ = t 0 η
( τ ) .
G E XTENSION : M ULTI - CLASS C LASSIFICATION
In this section, we generalize our results to multi-class classiﬁcation with cross-entropy loss.
This part of analysis is inspired by Theorem 1 in (Zhang et al., 2019), which gives a lower bound for the gradient in terms of the loss L .
Since now a neural network has multiple outputs, we need to redeﬁne our notations.
Let C be the number of classes.
The output of a neural network Φ is a vector Φ ( θ ; x ) ∈ R C .
We use Φ j ( θ ; x ) ∈ R to denote the j -th output of Φ on the input x ∈ R d x .
A dataset is denoted by
D = { x n , y n } Nn =1 = { ( x n , y n ) : n ∈ [ N ] } , where x n ∈ R d x is a data input and y n ∈ [ C ] is the corresponding label.
The loss function of Φ on the dataset D is deﬁned as
[Equation]
The margin for a single data point ( x n , y n ) is deﬁned to be q n ( θ ) :
=
Φ y n ( θ ; x n ) − max j (cid:54) =
y n { Φ j ( θ ; x n ) } , and the margin for the entire dataset is deﬁned to be q min ( θ )
=
min n ∈ [ N ] q n ( θ ) .
We deﬁne the normalized margin to be ¯ γ ( θ ) :
= q min (ˆ θ ) =
q min ( θ ) /ρ
L , where ρ := (cid:107) θ (cid:107)
and ˆ θ := θ /ρ
∈
S
d − 1 as usual.
Let (cid:96) ( q ) :
= log(1+ e − q ) be the logistic loss.
Recall that (cid:96) ( q ) satisﬁes (B3).
Let f ( q ) =
− log (cid:96) ( q ) = − log log(1 + e − q ) .
Let g be the inverse function of f .
So g ( q ) =
− log( e e − q − 1) .
The cross-entropy loss can be rewritten in other ways.
Let s nj :=
Φ y n ( θ ; x n ) − Φ j
( θ ; x n ) .
Let ˜ q n := − LSE( {− s nj : j (cid:54) =
y n } )
= − log (cid:16)(cid:80)
j (cid:54) =
y n e − s nj (cid:17) .
Then
L ( θ ) :=
[Equation]
Gradient Flow.
For gradient ﬂow, we assume the following:
(M1).
(Regularity).
For any ﬁxed x and j ∈ [ C ] , Φ j ( · ; x ) is locally Lipschitz and admits a chain rule; (M2).
(Homogeneity).
There exists L > 0 such that ∀ j ∈ [ C ] , ∀ α > 0 : Φ j ( α θ ; x ) = α L Φ j ( θ ; x ) ; (M3).
(Cross-entropy Loss).
L ( θ ) is deﬁned as the cross-entropy loss on the training set; (M4).
(Separability).
There exists a time t 0 such that L ( t 0 ) < log 2 .
If L < log 2 , then (cid:80) j (cid:54) =
y n e − s nj < 1 for all n ∈ [ N ] , and thus s nj > 0 for all n ∈ [ N ] , j ∈ [ C ] .
So (M4) ensures the separability of training data.
Deﬁnition G.1.
For cross-entropy loss, the smoothed normalized margin ˜ γ ( θ ) of θ is deﬁned as
[Equation]
where (cid:96) − 1 ( · ) is the inverse function of the logistic loss (cid:96) ( · ) .
Theorem 4.1 and 4.4 still hold.
Here we redeﬁne the optimization problem (P) to be
[Equation]
Most of our proofs are very similar as before.
Here we only show the proof for the generalized version of Lemma 5.1.
Lemma G.2.
Lemma 5.1 is also true for the smoothed normalized margin ˜ γ deﬁned in Deﬁnition G.1.
Proof.
Deﬁne ν ( t ) by the following formula:
[Equation]
Using a similar argument as in Theorem B.4, it can be proved that 12 dρ 2 dt = Lν ( t ) for a.e. t > 0 .
It can be shown that Lemma B.5, which asserts that ν ( t ) ≥ g (log 1 L ) g (cid:48) (log 1 L ) L , still holds for this new deﬁnition of ν ( t ) .
By deﬁnition, s nj ≥ q n for j (cid:54) = y n .
Also note that e − ˜ q n ≥ e − q n .
So s nj ≥ q n ≥ ˜ q n .
Then we have
[Equation]
Note that L = (cid:80) Nn =1 e − f
(˜ q n ) .
Then using Lemma B.5 for logistic loss can conclude that ν ( t ) ≥ g [Equation] log
[Equation]
L
[Equation]
g
[Equation]
L [Equation]
The rest of the proof for this lemma is exactly the same as that for Lemma 5.1.
Gradient Descent.
For gradient descent, we only need to replace (M1) with (S1) and make the assumption (S5) on the learning rate.
(S1).
(Smoothness).
For any ﬁxed x and j ∈ [ C ] , Φ j ( · ; x ) is C 2 -smooth; (S5).
(Learing rate condition). (cid:80) t ≥ 0 η ( t ) = + ∞ and η ( t ) ≤ H ( L ( θ ( t ))) . Here H ( x ) :
= µ ( x ) / ( C η κ ( x )) is deﬁned to be the same as in Appendix F (when (cid:96) ( · ) is set to logistic loss) except that we use another C η which will be speciﬁed later.
We only need to show that Lemma F.2 and Lemma F.3 continue to hold.
Using the same argument as we do for gradient ﬂow, we can show that γ ( t ) and λ ( x ) do satisfy the propositions in Lemma F.2.
For Lemma F.3, we ﬁrst note the original deﬁnition of C η involves B 0 , B 1 , B 2 , which are undeﬁned in the multi-class setting.
So now we redeﬁne them as
[Equation]
By the property of LSE , we can use B 0 , B 1 , B 2 to bound ˜ q n , ∇ ˜ q n , ∇ 2 ˜ q n .
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13) (cid:13)(cid:13)(cid:13)
[Equation]
Using these bounds, we can prove with the constant C η deﬁned as follows:
[Equation]
M
[Equation]
where M := min (cid:110) ˆ γ ( t 0 ) − 2+2 /L , B − 2+2 /L(cid:111) .
H E XTENSION : M ULTI - HOMOGENEOUS M ODELS
In this section, we extend our results to multi-homogeneous models.
For this, the main difference from the proof for homogeneous models is that now we have to separate the norm of each homogeneous parts of the parameter, rather than consider them as a whole.
So only a small part to proof needs to be changed.
We focus on gradient ﬂow, but it is worth to note that following the same argument, it is not hard to extend the results to gradient descent.
Let Φ( w 1 , . . .
, w m ; x ) be ( k 1 , . . .
, k m ) -homogeneous.
Let ρ i
= (cid:107)
w i (cid:107) 2 and ˆ w i = w
[Equation]
[Equation]
The smoothed normalized margin deﬁned in (5) can be rewritten as follows:
Deﬁnition H.1.
For a multi-homogeneous model with loss function (cid:96) ( · ) satisfying (B3), the smoothed normalized margin ˜ γ ( θ ) of θ is deﬁned as
[Equation]
We only prove the generalized version of Lemma 5.1 here.
The other proofs are almost the same.
Lemma H.2.
For all t > t 0 , d dt log ρ i > 0 for all i ∈ [ m ] and
[Equation]
Proof.
Note that ddt log ρ i = 12 ρ
i
dρ
i dt = k i ν ( t ) ρ 2 i by Theorem B.4.
It simply follows from Lemma B. that d dt log ρ > 0 for a.e. t > t 0 .
And it is easy to see that log ˜ γ = log (cid:0) g (log 1 L ) /ρ
L (cid:1) exists for all t ≥ t 0 .
By the chain rule and Lemma B.5, we have
[Equation]
On the one hand, − d L dt = (cid:80) mi =1 (cid:13)(cid:13)
d w i dt (cid:13)(cid:13) 2 2 for a.e. t > 0 by Lemma I.3; on the other hand, k i ν ( t ) =
(cid:10) w i , d w i dt (cid:11) by Theorem B.4.
Combining these together yields
[Equation]
i [Equation]
dt (cid:13)(cid:13)(cid:13)(cid:13)  .
By the chain rule, d ˆ w i dt =
ρ
i
( I − ˆ w i ˆ w i (cid:62) )
d w i dt for a.e. t > 0 .
So we have
[Equation]
For cross-entropy loss, we can combine the proofs in Appendix G to show that Lemma H.2 holds if we use the following deﬁnition of the smoothed normalized margin:
Deﬁnition H.3.
For a multi-homogeneous model with cross-entropy, the smoothed normalized margin ˜ γ ( θ ) of θ is deﬁned as
[Equation]
where (cid:96) − 1 ( · ) is the inverse function of the logistic loss (cid:96) ( · ) .
The only place we need to change in the proof for Lemma H.2 is that instead of using Lemma B.5, we need to prove ν ( t ) ≥ g (log 1 L ) g (cid:48) (log 1 L ) L in a similar way as in Lemma G.2.
The other parts of the proof are exactly the same as before.
I C HAIN R ULES FOR N ON - DIFFERENTIABLE F UNCTIONS
In this section, we provide some background on the chain rule for non-differentiable functions.
The ordinary chain rule for differentiable functions is a very useful formula for computing derivatives in calculus.
However, for non-differentiable functions, it is difﬁcult to ﬁnd a natural deﬁnition of subdifferential so that the chain rule equation holds exactly.
To solve this issue, Clarke proposed
Clarke’s subdifferential (Clarke, 1975; 1990; Clarke et al., 2008) for locally Lipschitz functions, for which the chain rule holds as an inclusion rather than an equation:
Theorem I.1 (Theorem 2.3.9 and 2.3.10 of (Clarke, 1990)) .
Let z 1 , . . .
, z n : R d → R and f : R n → R be locally Lipschitz functions.
Let ( f ◦ z )( x ) =
f ( z 1 ( x ) , . . .
, z n ( x )) be the composition of f and z .
Then,
[Equation]
For analyzing gradient ﬂow, the chain rule is crucial.
For a differentiable loss function L ( θ ) , we can see from the chain rule that the function value keeps decreasing along the gradient ﬂow d θ ( t ) dt =
[Equation]
But for locally Lipschitz functions which could be non-differentiable, (25) may not hold in general since Theorem I.1 only holds for an inclusion.
Following (Davis et al., 2020; Drusvyatskiy et al., 2015), we consider the functions that admit a chain rule for any arc.
Deﬁnition I.2 (Chain Rule) .
A locally Lipschitz function f : R d → R admits a chain rule if for any arc z : [0 , + ∞ ) → [Equation]
[Equation]
holds for a.e. t > 0 .
It is shown in (Davis et al., 2020; Drusvyatskiy et al., 2015) that a generalized version of (25) holds for such functions:
Lemma I.3 (Lemma 5.2 (Davis et al., 2020)) .
Let L : R d → R be a locally Lipschitz function that admits a chain rule.
Let θ : [0 , + ∞ ) → R d be the gradient ﬂow on L :
[Equation]
for a.e [Equation] t ≥
[Equation]
Then
[Equation]
holds for a.e. t > 0 .
We can see that C 1 -smooth functions admit chain rules.
As shown in (Davis et al., 2020), if a locally Lipschitz function is subdifferentiablly regular or Whitney C 1 -stratiﬁable, then it admits a chain rule.
The latter one includes a large family of functions, e.g., semi-algebraic functions, semianalytic functions, and deﬁnable functions in an o-minimal structure (Coste, 2002; van den Dries & Miller, 1996).
It is worth noting that the class of functions that admits chain rules is closed under composition.
This is indeed a simple corollary of Theorem I.1.
Theorem I.4.
Let z 1 , . . .
, z n : R d → R and f : R n → R be locally Lipschitz functions and assume all of them admit chain rules.
Let ( f ◦ z )( x ) =
f ( z 1 ( x ) , . . .
, z n ( x )) be the composition of f and z .
Then f ◦ z also admits a chain rule.
Proof.
We can see that f ◦ z is locally Lipschitz.
Let x : [0 , + ∞ ) → R d , t (cid:55)→ x ( t ) be an arc.
First, we show that z ◦ x : [0 , + ∞ )
→ R d , t (cid:55)→ z ( x ( t )) is also an arc.
For any closed sub-interval
I , z ( x ( I )) must be contained in a compact set U .
Then it can be shown that the locally Lipschitz continuous function z is (globally) Lipschitz continuous on U .
By the fact that the composition of a Lipschitz continuous and an absolutely continuous function is absolutely continuous, z ◦ x is absolutely continuous on I , and thus it is an arc.
Since f and z admit chain rules on arcs z ◦ x and x respectively, the following holds for a.e. t > 0 ,
[Equation]
Combining these we obtain that for a.e. t > 0 ,
[Equation]
) = n (cid:88) i = α i (cid:104) h i , x (cid:48) ( t ) (cid:105) , for all α ∈ ∂ ◦ f ( z ( x ( t ))) and for all h i ∈
∂ ◦ z i ( x ( t )) .
The RHS can be rewritten as (cid:104) (cid:80) ni
=
α i h
i , x (cid:48) ( t ) (cid:105) .
By Theorem I.1, every k ∈ ∂ ◦ ( f ◦ z )( x ( t )) can be written as a convex combination of a ﬁnite set of points in the form of (cid:80) ni =
α
i h i .
So ( f ◦ z ◦ x ) (cid:48) ( t ) = (cid:104) k , x (cid:48) ( t ) (cid:105) holds for a.e. t > 0 .
J M EXICAN H AT
In this section, we give an example to illustrate that gradient ﬂow does not necessarily converge in direction, even for C ∞ -smooth homogeneous models.
It is known that gradient ﬂow (or gradient descent) may not converge to any point even when optimizing an C ∞ function (Curry, 1944; Zoutendijk, 1976; Palis & De Melo, 2012; Absil et al., 2005).
One famous counterexample is the “Mexican Hat” function described in (Absil et al., 2005):
[Equation]
where C ( r ) =
[Equation]
4 ∈ [0 , 1] .
It can be shown that f is C ∞ -smooth on R 2 but not analytic.
See Figure 2 for a plot for f ( u, v ) .
However, the Maxican Hat function is not homogeneous, and Absil et al.
(2005) did not consider the directional convergence, either.
To make it homogeneous, we introduce an extra variable z , and normalize the parameter before evaluate f .
In particular, we ﬁx L > 0 and deﬁne h ( θ )
= h ( x, y, z ) = ρ L (1 − f ( u, v )) where u = x/ρ, v = y/ρ, ρ = (cid:112) x 2 + y 2 + z 2 .
We can show the following theorem.
Theorem J.1.
Consider gradient ﬂow on L ( θ )
= (cid:80) N n =1 e − q n ( θ ) , where q n ( θ )
=
h ( θ ) for all n ∈ [ N ] .
Suppose the polar representation of ( u, v ) is ( r cos ϕ, r sin ϕ ) .
If 0 < r < 1 and ϕ = 1 1 − r 2 holds at time t = 0 , then θ ( t ) (cid:107) θ ( t ) (cid:107) 2 does not converge to any point, and the limit points of { θ ( t ) (cid:107) θ ( t ) (cid:107) 2 : t > 0 } form a circle { ( x, y, z ) ∈ S 2 : x
2 + y 2 = 1 , z = 0 } .
Proof.
Deﬁne ψ = ϕ − 1 1 − r 2 .
Our proof consists of two parts, following from the idea in (Absil et al., 2005).
First, we show that dψdt = 0 as long as ψ = 0 .
Then we can infer that ψ = 0 for all t ≥ 0 .
Next, we show that r → 1 as t → + ∞ .
Using ψ = 0 , we know that the polar angle ϕ → + ∞ as t → + ∞ .
Therefore, ( u, v ) circles around { ( u, v ) : u 2 + v 2 = 1 } , and thus it does not converge.
Proof for dψ dt = 0 .
For convenience, we use w to denote z/ρ .
By simple calculation, we have the following formulas for partial derivatives:
[Equation]
For gradient ﬂow, we have
[Equation]
By writing down the movement of ( u, v ) in the polar coordinate system, we have
[Equation]
For ψ = 0 , the partial derivatives of f with respect to r and ϕ can be evaluated as follows:
[Equation]
So if ψ = 0 , then dψdt = 0 by the direct calculation below:
[Equation]
Proof for r → 1 .
Let (¯ u, ¯ v ) be a convergent point of { ( u, v ) : t ≥ 0 } .
Deﬁne ¯ r = √ u
+ v 2 .
It is easy to see that r ≤ 1 from the normalization of θ in the deﬁnition.
According to Theorem 4.4, we know that (¯ u, ¯ v ) is a stationary point of ¯ γ ( u ( t ) , v ( t ))
= 1 − f ( u ( t ) , v ( t )) .
If ¯ r = 0 , then f (¯ u, ¯ v ) >
f ( u (0) , v (0)) , which contradicts to the monotonicity of ˜ γ ( t ) =
¯ γ ( t )
= 1 − f ( u ( t ) , v ( t )) .
If ¯ r < 1 , then ¯ ψ = 0 (deﬁned as ψ for (¯ u, ¯ v ) ).
So ∂f∂r = − [Equation]
= 0 , which again leads to a contradiction.
Therefore, ¯ r = 1 , and thus r → 1 .
K E XPERIMENTS
To validate our theoretical results, we conduct several experiments.
We mainly focus on MNIST dataset.
We trained two models with Tensorﬂow.
The ﬁrst one (called the CNN with bias ) is a standard 4-layer CNN with exactly the same architecture as that used in MNIST Adversarial Examples
Challenge 5 .
The layers of this model can be described as conv -32 with ﬁlter size 5 × 5 , max-pool , conv -64 with ﬁlter size 3 × 3 , max-pool , fc -1024, fc -10 in order.
Notice that this model has bias terms in each layer, and thus does not satisfy homogeneity.
To make its outputs homogeneous to its parameters, we also trained this model after removing all the bias terms except those in the ﬁrst layer (the modiﬁed model is called the CNN without bias ).
Note that keeping the bias terms in the ﬁrst layer prevents the model to be homogeneous in the input data while retains the homogeneity in parameters.
We initialize all layer weights by He normal initializer (He et al., 2015) and all bias terms by zero.
In training the models, we use SGD with batch size 100 without momentum.
We normalize all the images to [0 , 1] 32 × 32 by dividing 255 for each pixel.
K. E VALUATION FOR N ORMALIZED M ARGIN
In the ﬁrst part of our experiments, we evaluate the normalized margin every few epochs to see how it changes over time.
From now on, we view the bias term in the ﬁrst layer as a part of the weight in the ﬁrst layer for convenience.
Observe that the CNN without bias is multi-homogeneous in layer weights (see (4) in Section 4.4).
So for the CNN without bias, we deﬁne the normalized margin ¯ γ as the margin divided by the product of the L 2 -norm of all layer weights.
Here we compute the
L 2 -norm of a layer weight parameter after ﬂattening it into a one-dimensional vector.
For the CNN with bias, we still compute the smoothed normalized margin in this way.
When computing the L 2norm of every layer weight, we simply ignore the bias terms if they are not in the ﬁrst layer.
For completeness, we include the plots for the normalized margin using the original deﬁnition (2) in
Figure 3 and 4.
[Figure] [Figure]
[Figure]
://github.com/MadryLab/mnist_challenge
SGD with Constant Learning Rate.
We ﬁrst train the CNNs using SGD with constant learning rate 0 .
After about 100 epochs, both CNNs have ﬁtted the training set.
After that, we can see that the normalized margins of both CNNs increase.
However, the growth rate of the normalized margin is rather slow.
The results are shown in Figure 1 in Section 1.
We also tried other learning rates other than 0 .
01 , and similar phenomena can be observed.
SGD with Loss-based Learning Rate.
Indeed, we can speed up the training by using a proper scheduling of learning rates for SGD.
We propose a heuristic learning rate scheduling method, called the loss-based learning rate scheduling .
The basic idea is to ﬁnd the maximum possible learning rate at each epoch based on the current training loss (in a similar way as the line search method).
See Appendix L.1 for the details.
As shown in Figure 1, SGD with loss-based learning rate scheduling decreases the training loss exponentially faster than SGD with constant learning rate.
Also, a rapid growth of normalized margin is observed for both CNNs.
Note that with this scheduling the training loss can be as small as 10 − 800 , which may lead to numerical issues.
To address such issues, we applied some re-parameterization tricks and numerical tricks in our implementation.
See
Appendix L.2 for the details.
Experiments on CIFAR-10.
To verify whether the normalized margin is increasing in practice, we also conduct experiments on CIFAR-10.
We use a modiﬁed version of VGGNet-16.
The layers of this model can be described as conv -64 × 2 , max-pool , conv -128 × 2 , max-pool , conv - × 3 , max-pool , conv -512 × 3 , max-pool , conv -512 × 3 , max-pool , fc -10 in order, where each conv has ﬁlter size 3 × 3 .
We train two networks: one is exactly the same as the VGGNet we described, and the other one is the VGGNet without any bias terms except those in the ﬁrst layer (similar as in the experiments on MNIST).
The experiment results are shown in Figure 5 and 6.
We can see that the normalize margin is increasing over time
[Figure] [Figure]
Test Accuracy.
Previous works on margin-based generalization bounds (Neyshabur et al., 2018;
Bartlett et al., 2017; Golowich et al., 2018; Li et al., 2018a; Wei et al., 2019; Banburski et al., 2019) usually suggest that a larger margin implies a better generalization bound.
To see whether the generalization error also gets smaller in practice, we plot train and test accuracy for both MNIST and
CIFAR-10.
As shown in Figure 7, the test accuracy changes only slightly after training with lossbased learning rate scheduling for 10000 epochs, although the normalized margin does increase a lot.
We leave it as a future work to study this interesting gap between margin-based generalization bound and generalization error.
Concurrent to this work, Wei & Ma (2020) proposed a generalization bound based on a new notion of margin called all-layer margin, and showed via experiments that enlarging all-layer margin can indeed improve generalization. It would be an interesting research direction to study how different deﬁnitions of margin may lead to different generalization abilities.
[Figure]
K. E VALUATION FOR R OBUSTNESS
Recently, robustness of deep learning has received considerable attention (Szegedy et al., 2013; Biggio et al., 2013; Athalye et al., 2018), since most state-of-the-arts deep neural networks are found to be very vulnerable against small but adversarial perturbations of the input points.
In our experiments, we found that enlarging the normalized margin can improve the robustness.
In particular, by simply training the neural network for a longer time with our loss-based learning rate, we observe noticeable improvements of L 2 -robustness on both the training set and test set.
We ﬁrst elaborate the relationship between the normalized margin and the robustness from a theoretical perspective.
For a data point z =
( x , y ) , we can deﬁne the robustness (with respect to some norm (cid:107) · (cid:107) ) of a neural network Φ for z to be
[Equation]
where X is the data domain (which is [0 , 1] 32 × 32 for MNIST).
It is well-known that the normalized margin is a lower bound of L 2 -robustness for fully-connected networks (See, e.g., Theorem 4 in (Sokolic et al., 2017)).
Indeed, a general relationship between those two quantities can be easily shown.
Note that a data point z is correctly classiﬁed iff the margin for z , denoted as q θ ( z ) , is larger than 0 .
For homogeneous models, the margin q θ ( z ) and the normalized margin q ˆ θ ( z ) for x have the same sign.
If q ˆ θ ( · ) : R d x → R is β -Lipschitz (with respect to some norm (cid:107) · (cid:107) ), then it is easy to see that R θ ( z ) ≥
q ˆ θ ( z ) /β .
This suggests that improving the normalize margin on the training set can improve the robustness on the training set.
Therefore, our theoretical analysis suggests that training longer can improve the robustness of the model on the training set.
This observation does match with our experiment results.
In the experiments, we measure the L 2robustness of the CNN without bias for the ﬁrst time its loss decreases below 10 − 10 , 10 − 15 , 10 − 20 , 10 − 120 (labelled as model-1 to model-4 respectively).
We also measure the L 2 -robustness for the ﬁnal model after training for 10000 epochs (labelled as model-5 ), whose training loss is about 10 − 882 .
The normalized margin of each model is monotone increasing with respect to the number of epochs, as shown in Table 1.
[Figure]
[Table] 10 −100% 99 . We use the standard method for evaluating L 2 -robustness in (Carlini & Wagner, 2017) and the source code from the authors with default hyperparameters 6 .
We plot the robust accuracy (the percentage of data with robustness > (cid:15) ) for the training set in the ﬁgures on the ﬁrst row of Figure 8.
It can be seen from the ﬁgures that for small (cid:15) (e.g., (cid:15) < 0 . 3 ), the relative order of robust accuracy is just the order of model-1 to model-5 .
For relatively large (cid:15) (e.g., (cid:15) > 0 . 3 ), the improvement of model-5 upon model-2 to model-4 becomes marginal or nonexistent in certain intervals of (cid:15) , but model-1 to model-4 still have an increasing order of robust accuracy and the improvement of model-5 upon model-1 is always signiﬁcant.
This shows that training longer can help to improve the L 2 -robust accuracy on the training set.
We also evaluate the robustness on the test set, in which a misclassiﬁed test sample is considered to have robustness 0 , and plot the robust accuracy in the ﬁgures on the second row of Figure 8.
It can be seen from the ﬁgures that for small (cid:15) (e.g., (cid:15) < 0 . 2 ), the curves of the robust accuracy of model-1 to model-5 are almost indistinguishable.
However, for relatively large (cid:15) (e.g., (cid:15) > 0 . 2 ), again, model-1 to model-4 have an increasing order of robust accuracy and the improvement of model-5 upon model-1 is always signiﬁcant.
This shows that training longer can also help to improve the L 2 -robust accuracy on the test set.
We tried various different settings of hyperparameters for the evaluation method (including different learning rates, different binary search steps, etc.) and we observed that the shapes and relative positions of the curves in Figure 8 are stable across different hyperparameter settings.
It is worth to note that the normalized margin and robustness do not grow in the same speed in our experiments, although the theory suggests R θ ( z ) ≥ q ˆ θ ( z ) /β .
This may be because the Lipschitz constant β (if deﬁned locally) is also changing during training.
Combining training longer with existing techniques for constraining Lipschitz number (Anil et al., 2019; Cisse et al., 2017) could potentially alleviate this issue, and we leave it as a future work.
L A DDITIONAL E XPERIMENTAL D ETAILS
In this section, we provide additional details of our experiments.
L. L OSS - BASED L EARNING R ATE S CHEDULING
The intuition of the loss-based learning rate scheduling is as follows.
If the training loss is αsmooth, then optimization theory suggests that we should set the learning rate to roughly 1 /α .
For a homogeneous model with cross-entropy loss, if the training accuracy is 100% at θ , then a simple calculation can show that the smoothness (the L 2 -norm of the Hessian matrix) at θ is O ( ¯ L· poly( ρ )) , where ¯ L is the average training loss and poly( ρ ) is some polynomial.
Motivated by this fact, we parameterize the learning rate η ( t ) at epoch t as
[Equation]
(27) where ¯ L ( t − 1) is the average training loss at epoch t − 1 , and α ( t ) is a relative learning rate to be tuned (Similar parameterization has been considiered in (Nacson et al., 2019b) for linear model).
The loss-based learning rate scheduling is indeed a variant of line search.
In particular, we initialize α (0) by some value, and do the following at each epoch t :
Step 1.
Initially α ( t ) ← α ( t − 1) ; Let ¯ L ( t − 1) be the training loss at the end of the last epoch; Step 2.
Run SGD through the whole training set with learning rate η ( t ) :
= α ( t ) / L ( t − 1) ;
Step 3.
Evaluate the training loss ¯ L ( t ) on the whole training set; Step 4.
If ¯ L ( t ) <
¯ L ( t − 1) , α ( t ) ← α ( t ) · r u and end this epoch; otherwise, α ( t ) ← α ( t ) /r
d and go to Step 2.
In all our experiments, we set α (0) :
= 0 .
1 , r u := 2 1 / 5 ≈ 1 . 149 , r d := 2 1 / 10 ≈ 1 . 072 .
This speciﬁc choice of those hyperparameters is not important; other choices can only affact the computational efﬁciency, but not the overall tendency of normalized margin.
L. A DDRESSING N UMERICAL I SSUES
Since we are dealing with extremely small loss (as small as 10 − 800 ), the current Tensorﬂow implementation would run into numerical issues.
To address the issues, we work as follows.
Let ¯ L B ( θ ) be the (average) training loss within a batch B ⊆
[ N ] .
We use the notations C, s nj , ˜ q n , q n from
Appendix G. We only need to show how to perform forward and backward passes for ¯ L B ( θ ) .
Forward Pass.
Suppose we have a good estimate (cid:101) F for log ¯ L B ( θ ) in the sense that [Equation]
[Equation]
is in the range of float64 .
R B ( θ ) can be thought of a relative training loss with respect to (cid:101) F .
Instead of evaluating the training loss ¯ L B ( θ ) directly, we turn to evaluate this relative training loss in a numerically stable way:
Step 1.
Perform forward pass to compute the values of s nj with float32 , and convert them into float64 ;
Step 2.
Let Q := 30 .
If q n ( θ ) > Q for all n ∈ B , then we compute
[Equation]
where ˜ q n ( θ ) :
= − LSE( {− s nj : j (cid:54) =
y n } )
= − log (cid:16)(cid:80)
j (cid:54) =
y n e − s nj (cid:17) is evaluated in a numerically stable way; otherwise, we compute
[Equation]
where log1p( x ) is a numerical stable implementation of log(1 + x ) .
This algorithm can be explained as follows.
Step 1 is numerically stable because we observe from the experiments that the layer weights and layer outputs grow slowly.
Now we consider Step 2.
If q n ( θ ) ≤ Q for some n ∈ [ B ] , then ¯ L B ( θ ) =
Ω( e − Q ) is in the range of float64 , so we can compute R B ( θ ) by (28) directly except that we need to use a numerical stable implementation of log(1 + x ) .
For q n ( θ ) > Q , arithmetic underﬂow can occur.
By Taylor expansion of log(1 + x ) , we know that when x is small enough log(1 + x )
≈ x in the sense that the relative error | log(1+ x ) − x | log(1+ x ) =
O ( x ) .
Thus, we can do the following approximation
[Equation]
(29) for q n ( θ ) > Q , and only introduce a relative error of O ( Ce − Q ) (recall that C is the number of classes).
Using a numerical stable implementation of LSE , we can compute ˜ q n easily.
Then
the
RHS of (29) can be rewritten as e − (˜ q n ( θ )+ (cid:101) F ) .
Note that computing e − (˜ q n ( θ )+ (cid:101) F ) does not have underﬂow or overﬂow problems if (cid:101) F is a good approximation for log ¯ L B ( θ ) .
Backward Pass.
To perform backward pass, we build a computation graph in Tensorﬂow for the above forward pass for the relative training loss and use the automatic differentiation.
We parameterize the learning rate as η = ˆ η · e (cid:101) F .
Then it is easy to see that taking a step of gradient descent for
L B ( θ ) with learning rate η is equivalent to taking a step for R B ( θ ) with ˆ η .
Thus, as long as ˆ η can ﬁt into float64 , we can perform gradient descent on R B ( θ ) to ensure numerical stability.
The Choice of (cid:101) F .
The only question remains is how to choose (cid:101) F .
In our experiments, we set (cid:101) F ( t ) :
= log ¯ L ( t − 1) to be the training loss at the end of the last epoch, since the training loss cannot change a lot within one single epoch.
For this, we need to maintain log ¯ L ( t ) during training.
This can be done as follows: after evaluating the relative training loss R ( t ) on the whole training set, we can obtain log ¯ L ( t ) by adding (cid:101) F ( t ) and log R ( t ) together.
It is worth noting that with this choice of (cid:101) F , ˆ η ( t ) =
α ( t ) in the loss-based learning rate scheduling.
As shown in the right ﬁgure of Figure 4, α ( t ) is always between 10 − 9 and 10 0 , which ensures the numerical stability of backward pass.