T HE I MPLICIT B IAS OF G RADIENT D ESCENT ON S EPARABLE D ATA
A BSTRACT
We show that gradient descent on an unregularized logistic regression problem, for almost all separable datasets, converges to the same direction as the max-margin solution.
The result generalizes also to other monotone decreasing loss functions with an inﬁmum at inﬁnity, and we also discuss a multi-class generalizations to the cross entropy loss.
Furthermore, we show this convergence is very slow, and only logarithmic in the convergence of the loss itself.
This can help explain the beneﬁt of continuing to optimize the logistic or cross-entropy loss even after the training error is zero and the training loss is extremely small, and, as we show, even if the validation loss increases.
Our methodology can also aid in understanding implicit regularization in more complex models and with other optimization methods.
I NTRODUCTION
It is becoming increasingly clear that implicit biases introduced by the optimization algorithm play a crucial role in deep learning and in the generalization ability of the learned models (Neyshabur et al., 2014; 2015; Zhang et al., 2017;
Keskar et al., 2017; Neyshabur et al., 2017; Wilson et al., 2017).
In particular, minimizing the training error, without any explicit regularization, over models with more parameters and more capacity then the number of training examples, often yields good generalization, despite the empirical optimization problem being highly underdetermined.
That is, there are many global minima of the training objective, most of which will not generalize well, but the optimization algorithm ( e g. gradient descent) biases us toward a particular minimum that does generalize well.
Unfortunately, we still do not have a good understanding of the biases introduced by different optimization algorithms in different situations.
We do have a decent understanding of the implicit regularization introduced by early stopping of stochastic methods or, at an extreme, of one-pass (no repetition) stochastic optimization.
However, as discussed above, in deep learning we often beneﬁt from implicit bias even when optimizing the (unregularized) training error to convergence, using stochastic or batch methods.
For loss functions with attainable, ﬁnite, minimizers, such as the squared loss, we have some understanding of this:
In particular, when minimizing an underdetermined least squares problem using gradient descent starting from the origin, we know we will converge to the minimum Euclidean norm solution.
But the logistic loss, and its generalization the cross-entropy loss which is often used in deep learning, do not admit a ﬁnite minimizer on separable problems.
Instead, to drive the loss toward zero and thus minimize it, the predictor must diverge toward inﬁnity.
Do we still beneﬁt from implicit regularization when minimizing the logistic loss on separable data?
Clearly the norm of the predictor itself is not minimized, since it grows to inﬁnity.
However, for prediction, only the direction of the predictor, i.e. the normalized w ( t ) / (cid:107) w ( t ) (cid:107) , is important.
How does w ( t ) / (cid:107) w ( t ) (cid:107) behave as t → ∞ when we minimize the logistic (or similar) loss using gradient descent on separable data, i.e. , when it is possible to get zero misclassiﬁcation error and thus drive the loss to zero?
In this paper, we show that even without any explicit regularization, for all most all datasets (except a zero measure set), when minimizing linearly separable logistic regression problems using gradient descent, we have that w ( t ) / (cid:107) w ( t ) (cid:107) converges to the L 2 maximum margin separator, i.e. to the solution of the hard margin SVM.
This happens even though the norm (cid:107) w (cid:107) , nor the margin constraint, are in no way part of the objective nor explicitly introduced into optimization.
More generally, we show the same behavior for generalized linear problems with any smooth, monotone strictly decreasing, lower bounded loss with an exponential tail.
Furthermore, we characterize the rate of this convergence, and show that it is rather slow, with the distance to the max-margin predictor decreasing only as O (1 / log( t )) .
This explains why the predictor continues to improve even when the training loss is already extremely small.
We emphasize and demonstrate that this bias is speciﬁc to gradient descent, and changing the optimization algorithm, e.g. using adaptive learning rate methods such as
ADAM Kingma & Ba (2015), changes this implicit bias.
M AIN R ESULTS
Consider a dataset { x n , y n } Nn =1 , with binary labels y n ∈ {− 1 , 1 } .
We analyze learning by minimizing an empirical loss of the form
[Equation]
where w ∈ R d is the weight vector.
A bias term could be added in the usual way, extending x n by an additional ’1’ component.
To simplify notation, we assume that ∀ n : y n = 1 — this is true without loss of generality, since we can always re-deﬁne y n x n as x n .
We are particularly interested in problems that are linearly separable, and the loss is smooth monotone strictly decreasing and non-negative:
Assumption 1.
The dataset is strictly linearly separable: ∃ w ∗ such that ∀ n : w (cid:62)∗ x n > 0 .
Assumption 2.
(cid:96) ( u ) is a positive, differentiable, monotonically decreasing to zero 1 , (so ∀ u : (cid:96) ( u ) > 0 , (cid:96) (cid:48) ( u ) < 0 and lim u →∞ (cid:96) ( u ) =
lim u →∞ (cid:96) (cid:48) ( u ) = 0 ) and a β -smooth function, i.e. its derivative is β -Lipshitz.
Many common loss functions, including the logistic, exp-loss, probit and sigmoidal losses, follow
Assumption 2.
Assumption 2 also straightforwardly implies that L ( w ) is a βσ 2max ( X ) -smooth function, where the columns of X are all samples, and σ max ( X ) is the maximal singular value of X .
Under these conditions, the inﬁmum of the optimization problem is zero, but it is not attained at any ﬁnite w .
Furthermore, no ﬁnite critical point w exist.
We consider minimizing eq.
2.1 using Gradient
Descent (GD) with a ﬁxed learning rate η , i.e., with steps of the form:
[Equation]
We do not require convexity.
Under Assumptions 1 and 2, gradient descent converges to the global minimum ( i.e. to zero loss) even without it:
Lemma 1.
Let w ( t ) be the iterates of gradient descent (eq. 2.2) with η < 2 β − 1 σ − 2max ( X ) and any starting point w (0) .
Under Assumptions 1 and 2, we have: (1) lim
t →∞ L ( w ( t ))
= 0 , (2) lim t →∞ (cid:107) w ( t ) (cid:107) = ∞ , and (3) ∀ n : lim t →∞ w ( t ) (cid:62) x n = ∞ .
Proof.
Since the data is strictly linearly separable, ∃ w ∗ which linearly separates the data, and therefore
[Equation]
For any ﬁnite w , this sum cannot be equal to zero, as a sum of negative terms, since ∀ n : w (cid:62)∗ x n >and ∀ u : (cid:96) (cid:48) ( u ) < 0 .
Therefore, there are no ﬁnite critical points w , for which ∇L ( w ) = 0 .
But gradient descent on a smooth loss with an appropriate stepsize is always guaranteed to converge to a critical point: ∇L ( w ( t ))
→ 0 (see, e.g. Lemma 5 in Appendix A.5, slightly adapted from Ganti (2015), Theorem 2).
This necessarily implies that (cid:107) w ( t ) (cid:107) → ∞ while ∀ n : w ( t ) (cid:62) x n > 0 for large enough t —since only then (cid:96) (cid:48) (cid:16) w ( t ) (cid:62) x n (cid:17) → 0 .
Therefore, L ( w ) → 0 , so GD converges to the global minimum.
The main question we ask is: can we characterize the direction in which w ( t ) diverges?
That is, does the limit lim t →∞ w ( t ) / (cid:107) w ( t ) (cid:107) always exists, and if so, what is it?
In order to analyze this limit, we will need to make a further assumption on the tail of the loss function:
Deﬁnition 2.
A function f ( u ) has a “tight exponential tail”, if there exist positive constants c, a, µ + , µ − , u + and u − such that
[Equation]
Assumption 3.
The negative loss derivative − (cid:96) (cid:48) ( u ) has a tight exponential tail (Deﬁnition 2).
For example, the exponential loss (cid:96) ( u ) =
e − u and the commonly used logistic loss (cid:96) ( u ) =
log (1 + e − u )
both follow this assumption with a = c = 1 .
We will assume a = c = 1 — without loss of generality, since these constants can be always absorbed by re-scaling x n and η .
We are now ready to state our main result:
Theorem 3.
For almost all datasets ( i.e. , except for a measure zero) which are strictly linearly separable (Assumption 1) and given a β -smooth decreasing loss function (Assumption 2) with an exponential tail (Assumption 3), gradient descent (as in eq. 2.2) with stepsize η < 2 β − 1 σ − 2max ( X ) and any starting point w (0) will behave as:
[Equation]
where the residual ρ ( t ) is bounded and so
[Equation]
where ˆ w is the L 2 max margin vector (the solution to the hard margin SVM):
[Equation]
Since the theorem holds for almost all datasets, in particular, it holds with probability 1 if { x n } N n = are sampled from an absolutely continuous distribution.
Proof Sketch We ﬁrst understand intuitively why an exponential tail of the loss entail asymptotic convergence to the max margin vector: Assume for simplicity that (cid:96) ( u ) =
e − u exactly, and examine the asymptotic regime of gradient descent in which ∀ n : w ( t ) (cid:62) x n → ∞ , as is guaranteed by Lemma 1.
If w ( t ) / (cid:107) w ( t ) (cid:107) converges to some limit w ∞ , then we can write w ( t ) =
g ( t ) w ∞ + ρ ( t ) such that g ( t ) → ∞ , ∀ n : x (cid:62) n w ∞ > 0 , and lim t →∞ ρ ( t ) /g
( t ) = 0 .
The gradient can then be written as:
[Equation]
As g ( t ) → ∞ and the exponents become more negative, only those samples with the largest ( i.e. , least negative) exponents will contribute to the gradient.
These are precisely the samples with the smallest margin argmin n w (cid:62)∞ x n , aka the “support vectors”.
The negative gradient (eq. 2.5) would then asymptotically become a non-negative linear combination of support vectors.
The limit w ∞ will then be dominated by these gradients, since any initial conditions become negligible as (cid:107) w ( t ) (cid:107) → ∞
(from Lemma 1).
Therefore, w ∞ will also be non-negative linear combination of support vectors, and so will its scaling ˆ w = w ∞ / (cid:0) min n w (cid:62)∞ x n (cid:1) .
We therefore have: ˆ
[Equation]
=
N
[Equation]
n =
[Equation]
n
[Equation]
n
[Equation]
n
[Equation]
0 and ˆ w (cid:62) x n
[Equation] [Equation]
OR (cid:0)
[Equation] [Equation]
ˆ w (cid:62)
[Equation] [Equation]
These are precisely the KKT condition for the SVM problem (eq. 2.4) and we can conclude that ˆ w is indeed its solution and w ∞ is thus proportional to it.
To prove Theorem 3 rigorously, we need to show that w ( t ) / (cid:107) w ( t ) (cid:107) has a limit, that g ( t ) = log ( t ) and to bound the effect of various residual errors, such as gradients of non-support vectors and the fact that the loss is only approximately exponential.
To do so, we substitute eq.
2.3 into the gradient descent dynamics (eq. 2.2), with w ∞ =
ˆ w being the max margin vector and g ( t ) =
log t .
We then show that the increment in the norm of ρ ( t ) is bounded by C 1 t − ν for some C 1 > 0 and ν > 1 , which is a converging series.
This happens because the increment in the max margin term, ˆ w [log ( t + 1) − log ( t )]
≈ ˆ w t − 1 , cancels out the dominant t − 1 term in the gradient −∇L ( w ( t ))
(eq. 2.5 with g ( t ) =
log ( t ) and w (cid:62)∞ x n = 1 ).
A complete proof can be found in Appendix A.
More reﬁned analysis: characterizing the residual We can furthermore characterize the asymptotic behavior of ρ ( t ) .
To do so, we need to refer to the KKT conditions (eq. 2.6) of the SVM problem (eq. 2.4) and the associated support vectors S = argmin n ˆ w (cid:62) x n .
The following reﬁnement of Theorem 3 is also proved in Appendix A:
Theorem 4.
Under the conditions and notation of Theorem 3, if, in addition the support vectors span the data ( i.e. rank ( X S ) = rank ( X ) where the columns of X are all samples and of X S are the support vectors), then lim t →∞ ρ ( t ) =
˜ w , where ˜ w is unique, given w (0) , and a solution to
[Equation]
Note these equations are well-deﬁned for almost all datasets, since (see Lemma 8 in Appendix F) then there are at most d support vectors, α n are unique and ∀ n ∈ S : α n (cid:54) = 0 .
I MPLICATIONS : R ATES OF CONVERGENCE
The solution in eq.
2.3 implies that w ( t ) / (cid:107) w ( t ) (cid:107) converges to the normalized max margin vector ˆ w / (cid:107) ˆ w (cid:107) .
Moreover, this convergence is very slow— logarithmic in the number of iterations.
Speciﬁcally, in Appendix B we show that Theorem 3 implies the following tight rates of convergence:
The
normalized weight vector
converges
to normalized max margin vector
in L norm
[Equation]
and in angle
[Equation]
and the margin converges as
[Equation]
this slow convergence is in sharp contrast to the convergence of the (training) loss:
[Equation]
A simple construction (also in Appendix B) shows that the rates in the above equations are tight.
Thus, the convergence of w ( t ) to the max-margin ˆ w can be logarithmic in the loss itself, and we might need to wait until the loss is exponentially small in order to be close to the max-margin solution.
This can help explain why continuing to optimize the training loss, even after the training error is zero and the training loss is extremely small, still improves generalization performance—our results suggests that the margin could still be improving signiﬁcantly in this regime.
[Figure]
y 1 = y 2 = 1 , and x 3 = − x 1 , x 4 = − x 2 with y 3 =
y 4 = − 1 (the L 2 normalized max margin vector is then ˆ w =
(1 , 1) / √ 2 with margin equal to √ 2 ), and 12 other random datapoints ( 6 from each class), that are not on the margin.
We used a learning rate η = 1 /σ max ( X ) , where σ max ( X ) is the maximal singular value of X , momentum γ = 0 .
9 for GDMO, and initialized at the origin.
A numerical illustration of the convergence is depicted in Figure 1.
As predicted by the theory, the norm (cid:107) w ( t ) (cid:107) grows logarithmically (note the semi-log scaling), and w ( t ) converges to the max-margin separator, but only logarithmically, while the loss itself decreases very rapidly (note the log-log scaling).
An important practical consequence of our theory, is that although the margin of w ( t ) keeps improving, and so we can expect the population (or test) misclassiﬁcation error of w ( t ) to improve for many datasets, the same cannot be said about the expected population loss (or test loss)!
At the limit, the direction of w ( t ) will converge toward the max margin predictor ˆ w .
Although ˆ w has zero training error, it will not generally have zero misclassiﬁcation error on the population, or on a test or a validation set.
Since the norm of w ( t ) will increase, if we use the logistic loss or any other convex loss, the loss incurred on those misclassiﬁed points will also increase.
More formally, consider the logistic loss (cid:96) ( u )
= log(1+ e − u ) and deﬁne also the hinge-at-zero loss h ( u ) = max(0 , − u ) .
Since ˆ w classiﬁes all training points correctly, we have that on the training set (cid:80) N n =1 h ( ˆ w (cid:62) x n ) = 0 .
However, on the population we would expect some errors and so E
[ h ( ˆ w (cid:62) x )] > 0 .
Since w ( t ) ≈ ˆ w log t and (cid:96) ( αu ) → αh ( u ) as α → ∞ , we have:
[Equation]
That is, the population loss increases logarithmically while the margin and the population misclassiﬁcation error improve.
Roughly speaking, the improvement in misclassiﬁcation does not out-weight the increase in the loss of those points still misclassiﬁed.
The increase in the test loss is practically important because the loss on a validation set is frequently used to monitor progress and decide on stopping.
Similar to the population loss, the validation loss
L val ( w ( t ))
= (cid:80) x ∈V (cid:96) (cid:16) w ( t ) (cid:62) x (cid:17) calculated on an independent validation set V , will increase logarithmically with t (since we would not expect zero validation error), which might cause us to think we are over-ﬁtting or otherwise encourage us to stop the optimization.
But this increase does not actually represent the model getting worse, merely (cid:107) w ( t ) (cid:107) getting larger, and in fact the model might be getting better (with larger margin and possibly smaller error rate).
[Figure]
E XTENSIONS
We discuss several possible extensions of our results.
4. M ULTI -C LASS C LASSIFICATION WITH C ROSS -E NTROPY L OSS
So far, we have discussed the problem of binary classiﬁcation.
For multi-class problems commonly encountered, we frequently learn a predictor w k for each class, and use the cross-entropy loss with a softmax output, which is a generalization of the logistic loss.
What do the linear predictors w k ( t ) converge to if we minimize the cross-entropy loss by gradient descent on the predictors?
In Appendix
C we analyze this problem for separable data, and show that again, the predictors diverge to inﬁnity and the loss converges to zero.
Furthermore, we show that, generically, the loss converges to a logistic loss for transformed data, for which our theorems hold.
This strongly suggests that gradient descent converges to a scaling of the K -class SVM solution
[Equation]
We believe this can also be established rigorously and for generic exponential tailed multi-class loss.
4. O THER OPTIMIZATION METHODS
In this paper we examined the implicit bias of gradient descent.
Different optimization algorithms exhibit different biases, and understanding these biases and how they differ is crucial to understanding and constructing learning methods attuned to the inductive biases we expect.
Can we characterize the implicit bias and convergence rate in other optimization methods?
In Figure 1 we see that adding momentum does not qualitatively affects the bias induced by gradient descent.
In Figure 4 in Appendix E we also repeat the experiment using stochastic gradient descent, and observe a similar bias.
This is consistent with the fact that momentum, acceleration and stochasticity do not change the bias when using gradient descent to optimize an under determined least squares problems.
It would be beneﬁcial, though, to rigorously understand how much we can generalize our result to gradient descent variants, and how the convergence rates might change in these cases.
Employing adaptive methods, such as AdaGrad (Duchi et al., 2011) and ADAM (Kingma & Ba, 2015), does signiﬁcantly affect the bias.
In Figure 2 we show the predictors obtained by ADAM and by gradient descent on a simple data set.
Both methods converge to zero training error solutions.
But although gradient descent converges to the L 2 max margin predictor, as predicted by our theory,
ADAM does not.
The implicit bias of adaptive method has been a recent topic of interest, with
Hoffer et al.
(2017) and Wilson et al.
(2017) suggesting they lead to worse generalization.
Wilson et al.
discuss the limit of AdaGrad on lest square problems, but fall short of providing an actual characterization of the limit.
This is not surprising, as the limit of AdaGrad on least square problems is fragile and depends on the choice of stepsize and other parameters, and thus complicated to characterize.
We expect our methodology could be used to precisely characterize the implicit bias of such methods on logistic regression problems.
The asymptotic nature of the analysis is appealing here, as it is insensitive to the initial point, initial conditioning matrix, and large initial steps.
More broadly, it would be interesting to study the behavior of mirror descent and natural gradient descent, and relate the bias they induce to the potential function or divergence underlying them.
A reasonable conjecture, which we have not yet investigated, is that for any potential function Ψ( w ) , these methods converge to the maximum Ψ -margin solution arg min w Ψ( w ) s.t. ∀ n : w (cid:62) x n ≥ 1 .
Since mirror descent can be viewed as regularizing progress using Ψ( w ) , it is worth noting the results of Rosset et al.
(2004b): they considered the regularization path w λ =
arg min L ( w ) + λ (cid:107) w (cid:107) pp for similar loss function as we do, and showed that lim λ
→ 0 w λ / (cid:107) w λ (cid:107) p is proportional to the maximum
L p margin solution.
Rosset et al.
do not consider the effect of the optimization algorithm, and instead add explicit regularization—here we are speciﬁcally interested in the bias implied by the algorithm not by adding (even
inﬁnitesimal) explicit regularization
Our analysis also covers the exp-loss used in boosting, as its tail is similar to that of the logistic loss.
However, boosting is a coordinate descent procedure, and not a gradient descent procedure.
Indeed, the coordinate descent interpretation of AdaBoost shows that coordinate descent on the exp-loss for a linearly separable problem is related to ﬁnding the maximum L 1 margin solution (Schapire et al., 1998; Rosset et al., 2004a; Shalev-Shwartz & Singer, 2010).
4. D EEP NETWORKS
In this paper, we only consider linear prediction.
Naturally, it is desirable to generalize our results also to non-linear models and especially multi-layer neural networks.
Even without a formal extension and description of the precise bias, our results already shed light on how minimizing the cross-entropy loss with gradient descent can have a margin maximizing effect, how the margin might improve only logarithmically slow, and why it might continue improving even as the validation loss increases.
These effects are demonstrated in Figure 3 and Table 1 which portray typical training of a convolutional neural network using unregularized gradient descent 2 .
As can be seen, the norm of the weight increases, but the validation error continues decreasing, albeit very slowly (as predicted by the theory), even after the training error is zero and the training loss is extremely small.
We can now understand how even though the loss is already extremely small, some sort of margin might be gradually improving as we continue optimizing.
We can also observe how the validation loss increases despite the validation error decreasing, as discussed in Section 3.
As an initial advance toward tackling deep network, we can point out that for two special cases, our results may be directly applied to multi-layered networks.
First, our results may be applied exactly,
[Table]
as we show in Appendix D, if only a single weight layer is being optimized, and furthermore, after a sufﬁcient number of iterations, the activation units stop switching and the training error goes to zero.
Second, our results may also be applied directly to the last weight layer if the last hidden layer becomes ﬁxed and linearly separable after a certain number of iterations.
This can become true, either approximately, if the input to the last hidden layer is normalized ( e.g. , using batch norm), or exactly, if the last hidden layer is quantized (Hubara et al., 2016).
4. M ATRIX F ACTORIZATION
With multi-layered neural networks in mind, Gunasekar et al.
(2017) recently embarked on a study of the implicit bias of under-determined matrix factorization problems, where we minimize the squared loss of linear observation of a matrix by gradient descent on its factorization.
Since a matrix factorization can be viewed as a two-layer network with linear activations, this is perhaps the simplest deep model one can study in full, and can thus provide insight and direction to studying more complex neural networks.
Gunasekar et al.
conjectured, and provided theoretical and empirical evidence, that gradient descent on the factorization for an under-determined problem converges to the minimum nuclear norm solution, but only if the initialization is inﬁnitesimally close to zero and the step-sizes are inﬁnitesimally small.
With ﬁnite step-sizes or ﬁnite initialization, Gunasekar et al.
could not characterize the bias.
It would be interesting to study the same problem with a logistic loss instead of squared loss.
Beyond the practical relevance of the logistic loss, taking our approach has the advantage that because of its asymptotic nature, it does not depend on the initialization and step-size.
It thus might prove easier to analyze logistic regression on a matrix factorization instead of the least square problem, providing signiﬁcant insight into the implicit biases of gradient descent on non-convex multi-layered optimization.
S UMMARY
We characterized the implicit bias induced by gradient descent when minimizing smooth monotone loss functions with an exponential tail.
This is the type of loss commonly being minimized in deep learning.
We can now rigorously understand:
How gradient descent, without early stopping, induces implicit L 2 regularization and converges to the maximum L 2 margin solution, when minimizing the logistic loss, or exploss, or any other monotone decreasing loss with appropriate tail.
In particular, the non-tail part does not affect the bias and so the logistic loss and the exp-loss, although very different on non-separable problems, behave the same for separable problems.
The bias is also independent of the step-size used (as long as it is small enough to ensure convergence) and (unlike for least square problem) is also independent on the initialization.
This convergence is very slow.
This explains why it is worthwhile continuing to optimize long after we have zero training error, and even when the loss itself is already extremely small.
We should not rely on slow decrease of the training loss, or on no decrease of the validation loss, to decide when to stop.
We might improve the validation, and test, errors even when the validation loss increases and even when the decrease in the training loss is tiny.
Perhaps that gradient descent leads to a max L 2 margin solution is not a big surprise to those for whom the connection between L 2 regularization and gradient descent is natural.
Nevertheless, we are not familiar with any prior study or mention of this fact, let alone a rigorous analysis and study of how this bias is exact and independent of the initial point and the step-size.
Furthermore, we also analyze the rate at which this happens, leading to the novel observations discussed above.
Perhaps even more importantly, we hope that our analysis can open the door to further analysis of different optimization methods or in different models, including deep networks, where implicit regularization is not well understood even for least square problems, or where we do not have such a natural guess as for gradient descent on linear problems.
Analyzing gradient descent on logistic/cross-entropy loss is not only arguably more relevant than the least square loss, but might also be technically easier.
A CKNOWLEDGMENTS
The authors are grateful to S. Gunasekar and C. Zeno for helpful comments on the manuscript.
The research of DS was supported by the Taub foundation.
Appendix A P ROOF OF MAIN RESULTS
In the following proofs, for any solution w ( t ) , we deﬁne
[Equation]
where ˆ w and ˜ w follow the conditions of Theorems 3 and 4, that is ˆ w is the L 2 is the max margin vector, which satisﬁes eq. 2.4:
[Equation]
and ˜ w is a vector which satisﬁes eq. [Equation]
[Equation]
where we recall that we denoted
X S ∈ R
d ×|S| as the matrix whose columns are the support vectors, a subset S ⊂ { 1 , . . .
, N } of the columns of X =
[ x 1 , . .
, x N ] ∈ R d × N .
In Lemma 8 (Appendix F) we prove that for almost every dataset α is uniquely deﬁned, there are no more then d support vectors and α n (cid:54)
= 0 , ∀ n ∈ S .
Therefore, eq.
A.1 is well-deﬁned in those cases.
If the support vectors do not span the data, then the solution ˜ w to eq.
A.1 might not be unique.
In this case, we can use any such solution in the proof.
We furthermore denote
[Equation]
and by C i , (cid:15) i , t i ( i ∈ N ) various positive constants which are independent of t .
Lastly, we deﬁne
P 1 ∈ R d × d as the orthogonal projection matrix 3 to the subspace spanned by the support vectors (the columns of X S ), and P 2 =
I − P 1 as the complementary projection (to the left nullspace of X S ).
A. S IMPLE PROOF OF T HEOREM 3 FOR SPECIAL CASE
In this section we ﬁrst examine the special case that (cid:96) ( u ) = e − u and take the continuous time limit of gradient descent: η → 0 , so [Equation]
The proof in this case is rather simple and self-contained
[Equation]
and so it helps clarify the main ideas of the general (more complicated) proof which we will give in the next sections.
Recall we deﬁned
[Equation]
Our goal is to show that (cid:107) r ( t ) (cid:107) is bounded
[Equation]
and therefore ρ ( t ) =
r ( t ) + ˜ w is bounded.
Eq.
A. implies that
[Equation]
and therefore
[Equation]
where in the last equality we used eq.
A.3 and decomposed the sum over support vectors S and non-support vectors.
We examine both bracketed terms.
Recall that ˆ w (cid:62) x n = 1 for n ∈ S , and that we deﬁned (in eq.
A.1)
˜ w so that (cid:80)
[Equation]
=
[Equation]
w , the ﬁrst bracketed term in eq.
A.5 can be written as
[Equation]
where the last line is non-positive since z ( e − z − 1) ≤ 0 .
Furthermore, since exp ( − z ) z ≤ 1 and θ = argmin n/ ∈S x (cid:62) n ˆ w > 1 , the second bracketed term in eq.
A.5 can be upper bounded by
[Equation]
Substituting eq.
A.6 and A.7 into eq.
A.5, we obtain, since θ > 1 , that ∀ t 1 , ∀ t > t 1 :
[Equation]
Thus, we showed that r ( t ) is bounded, which completes the proof for the special case (cid:4) .
Next, we give the proof for the general case, of discrete time, and exponentially-tailed functions.
It is somewhat more involved then the special case we considered here, but is based on similar ideas.
A. A UXILIARY L EMMATA
In the following proof, we use the following Lemmata:
Lemma 5.
Let L ( w ) be a β -smooth non-negative objective.
If η < 2 β − 1 , then, for any w (0) , with the GD sequence
[Equation]
we have that (cid:80) ∞ u =
(cid:107)∇L ( w ( u )) (cid:107) 2 < ∞ and therefore lim t →∞ (cid:107)∇L ( w ( t )) (cid:107) 2 = 0 .
Lemma 6.
We have
[Equation]
Additionally, ∀ (cid:15) 1 > 0 , ∃ C 2 , t 2 , such that ∀ t > t 2 , if
[Equation]
then the following improved bound holds
[Equation]
We prove both Lemmata below, in appendix sections [-A.
-]A.5 and A.6, after we prove both theorems.
A. P ROOF OF T HEOREM
Our goal is to show that (cid:107) r ( t ) (cid:107) is bounded, and therefore ρ ( t ) =
r ( t ) + ˜ w is bounded.
To show this, we will upper bound the following equation
[Equation]
First, we note that ﬁrst term in this equation can be upper-bounded by
[Equation]
A.
[Equation]
where in (1) we used eq. 2.3, in (2) we used eq. 2.2, and in (3) we used ∀ x > 0 : x ≥ log (1 + x ) > 0 , and also that
[Equation]
since ˆ w (cid:62) x n ≥ 1 (from the deﬁnition of ˆ w ) and (cid:96) (cid:48) ( u ) ≤ 0 .
Also, from Lemma 5 we know that
[Equation]
Substituting eq.
A.15 into eq.
A.13, and recalling that a t − ν power series converges for any ν > [-
-]1 , we can ﬁnd C 0 such that
[Equation]
Note that this equation also implies that ∀
[Equation]
Next, we would like to bound the second term in eq.
A.12.
From eq.
A.9 in Lemma 6, we can ﬁnd t 1 , C 1 such that ∀ t > t 1 :
[Equation]
Thus, by combining eqs.
A.18 and A.16 into eq.
A.12, we ﬁnd
[Equation]
[-u + 1) (cid:107) 2 − (cid:107) r ( u ) (cid:107) 
-]
which is a bounded, since θ > 1 (eq. A.2).
Therefore, (cid:107) r ( t ) (cid:107) is bounded.
(cid:4)
A. P ROOF OF T HEOREM
All that remains now is to show that (cid:107) r ( t ) (cid:107) → 0 if rank ( X S )
= rank ( X ) , and that ˜ w is unique given w (0) .
To do so, this proof will continue where the proof of Theorem 3 stopped, using notations and equations from that proof.
Since r ( t ) has a bounded norm, its two orthogonal components r ( t )
= P 1 r ( t ) + P 2 r ( t ) also have bounded norms (recall that P 1 , P 2 were deﬁned in the beginning of appendix section A).
From eq.
2.2, ∇L ( w ) is spanned by the columns of X .
If rank ( X S ) = rank ( X ) , then it is also spanned by the columns of X S , and so P 2 ∇L ( w ) = 0 .
Therefore, P 2 r ( t ) is not updated during GD, and remains constant.
Since ˜ w in eq.
2.3 is also bounded, we can absorb this constant P 2 r ( t ) into ˜ w without affecting eq.
2.7 (since ∀ n ∈ S : x (cid:62) n P 2 r ( t ) = 0 ).
Thus, without loss of generality, we can assume that r ( t ) =
P 1 r ( t ) .
Now, recall eq.
A.11 in Lemma
[Equation]
Combining this with eqs.
A.12 and A.16, implies that ∃ t 3 > max [ t 2 , t 0 ] such that ∀ t > t 3 such that (cid:107) r
( t ) (cid:107) > (cid:15) {+
+} , we have that (cid:107) r
( t + 1) (cid:107) 2 is a decreasing function since then
[Equation]
A.
[Equation]
Additionally, this result also implies that we cannot have (cid:107) r ( t ) (cid:107) > (cid:15) 1 ∀ t > t 3 , since then we arrive to the contradiction.
[Equation]
Therefore, ∃ t 4 > t 3 such that (cid:107) r ( t 4 ) (cid:107) ≤ (cid:15) 1 .
Recall also that (cid:107) r ( t ) (cid:107) is a decreasing function whenever (cid:107) r ( t ) (cid:107) ≥ (cid:15) 1 (eq.
A.19).
Also, recall that t 4 > t 0 , so from eq.
A.17, we have that ∀ t > t 4 , |(cid:107) r ( t + 1) (cid:107) − (cid:107) r ( t ) (cid:107)| < (cid:15) 0 .
Combining these three facts we conclude that ∀ t > t 4 : (cid:107) r ( t ) (cid:107) ≤ (cid:15) 1 + (cid:15) 0 .
Since this reasoning holds ∀ (cid:15) 1 , (cid:15) 0 , this implies that (cid:107) r ( t ) (cid:107) → 0 .
Lastly, we note that since P 2 r ( t ) is not updated during GD, we have that P 2 ( ˜ w − w (0))
= 0 .
This sets ˜ w uniquely, together with eq.
2.7. (cid:4)
A. P ROOF OF L EMMA
Lemma 5.
Let L ( w ) be a β -smooth non-negative objective.
If η < 2 β − 1 , then, for any w (0) , with the GD sequence
[Equation]
we have that (cid:80) ∞ u =
(cid:107)∇L ( w ( u )) (cid:107) 2 < ∞ and therefore lim t →∞ (cid:107)∇L ( w ( t )) (cid:107) 2 = 0 .
This proof is a slightly modiﬁed version of the proof of Theorem 2 in (Ganti, 2015).
Recall a well-known property of β -smooth functions:
[Equation]
From the β -smoothness of L ( w )
[Equation]
Thus, we have
[Equation]
which implies t
[Equation]
The right hand side is upper bounded by a ﬁnite constant, since L ( w (0)) < ∞ and 0 ≤
L ( w ( t + 1)) .
This implies ∞ (cid:88) u
= (cid:107)∇L ( w ( u ))
(cid:107) 2 < ∞ , and therefore (cid:107)∇L ( w ( t )) (cid:107) 2 → 0 .
A. P ROOF OF L EMMA
Recall that we deﬁned r ( t ) =
w ( t ) −
ˆ w log t − ˜ w , with ˆ w and ˜ w follow the conditions of the
Theorems 3 and 4, i.e. ˆ w is the L 2 max margin vector and (eq. 2.4), and eq.
2.7 holds
[Equation]
Lemma 6.
We have
[Equation]
Additionally, ∀ (cid:15) 1 > 0 , ∃ C 2 , t 2 , such that ∀ t > t 2 , if
[Equation]
then the following improved bound holds
[Equation]
From Lemma 1, ∀ n : lim t →∞ w ( t ) (cid:62) x n = ∞ .
In addition, from assumption 3 the negative loss derivative − (cid:96) (cid:48) ( u ) has an exponential tail e − u
(recall we assume a = c = 1 without loss of generality).
Combining both facts, we have positive constants µ − , µ + , t − and t + such that ∀ n
[Equation]
Next, we examine the expression we wish to bound, recalling that r ( t ) =
w ( t ) − ˆ w log t − ˜ w :
[Equation]
where in last line we used eqs.
2.6 and 2.7 to obtain
[Equation]
We examine the three terms in eq.
A.23.
The ﬁrst term can be upper bounded by
[Equation]
≤
[Equation]
if [Equation] P [Equation]
≤
[Equation]
o
[Equation]
if (cid:107) P [Equation] r ( t [Equation]
(cid:15)
[Equation]
(A.24) where in (1) we used that P 2 ˆ w = P 2 X S α = 0 from eq. 2.6, and in (2) we used that ˆ w (cid:62) r ( t ) =
o ( t ) , since ˆ w
[Equation]
where in the last line we used that ∇L ( w ( t ))
=
o (1) , from Lemma 5.
[Table]
[Equation]
where in (1) we used eq.
A.21, in (2) we used w ( t ) =
ˆ w log t + ˜
w
+ r ( t ) , in (3) we used xe − x ≤ 1 and x (cid:62) n r ( t ) ≥ 0 , in (4) we used θ > 1 , from eq.
A.2 and in (5) we deﬁned t (cid:48)
+ = max (cid:2) t + , exp (cid:0) [Equation] n ˜ [Equation]
(cid:62) [Equation] (cid:1)(cid:3) .
Lastly, we will aim to bound the sum in the third term in eq.
A.
[Equation]
We examine each term k in this sum, and divide into two cases, depending on the sign of x (cid:62) k r ( t ) .
First, if x (cid:62) k r ( t ) ≥ 0 , then term k in eq.
A.26 can be upper bounded ∀ t > t + , using eq.
A.21, by
[Equation]
We further divide into cases:
If (cid:12)(cid:12) x (cid:62) k r ( t ) (cid:12)(cid:12) ≤ C 0 t − 0 .
5 µ + , then we can upper bound eq.
A.27 with
[Equation]
If (cid:12)(cid:12) x (cid:62) k r ( t ) (cid:12)(cid:12) > C 0 t − 0 .
5 µ + , then we can ﬁnd t (cid:48)(cid:48)
+ > t (cid:48)
+ to upper bound eq
[Equation]
A.27 ∀ t > t (cid:48)(cid:48) + : [Equation] [Equation] µ
[Equation]
+
[Equation]
t −
[Equation]
k r ( t )
[Equation]
≤ 0 , ∀ t > t (cid:48)(cid:48) + (A.29) where in (1) we used the fact that e − x ≤ 1 − x + x 2 for x ≥ 0 and in (2) we deﬁned t (cid:48)(cid:48)
+ so that the previous expression is negative — this is possible since t − 0 .
5 µ + decreases slower then t − µ + .
If (cid:12) (cid:12) x (cid:62) k r ( t ) (cid:12) (cid:12) ≥ (cid:15) 2 , then we deﬁne t (cid:48)(cid:48)(cid:48) + > t (cid:48)(cid:48) + such that t (cid:48)(cid:48)(cid:48) + > exp (cid:0) min n ˜ w (cid:62) x n (cid:1) (cid:2) e 0 .
5 (cid:15) 2 − 1 (cid:3) − 1 /µ + , and
therefore ∀ t > t (cid:48)(cid:48)(cid:48)
+ ,
we have
(cid:0) 1 + t − µ + exp (cid:0) − µ + ˜ w (cid:62) x n (cid:1)(cid:1) e − (cid:15) 2 < e − 0 .
5 (cid:15) 2 .
This implies that ∀ t > t (cid:48)(cid:48)(cid:48)
+ we can upper bound eq.
A.27 by
[Equation]
[Table]
[Equation]
(A.30) Second, if x (cid:62) k r ( t ) < 0 , we again further divide into cases: If (cid:12)(cid:12) x (cid:62) k r ( t ) (cid:12)(cid:12) ≤ C 0 t − 0 .
5 µ − , then, since − (cid:96) (cid:48) (cid:16) w ( t ) (cid:62) x n (cid:17) > 0 , we can upper bound term k in eq.
A.26 with
[Equation]
If (cid:12)(cid:12) x (cid:62) k r ( t ) (cid:12)(cid:12) > C 0 t − 0 .
5 µ − , then, using eq.
A.22 we upper bound term k in eq.
A.26 with
[Equation]
Next, we will show that ∃ t (cid:48)− > t − such that the last expression is strictly negative ∀ t > t (cid:48)− .
Let M > 1 be some arbitrary constant.
Then, since (cid:104) t − 1 e − ˜ w
[Equation]
x k exp
[Equation]
−
[Equation]
t ) x k
[Equation]
µ − = exp (cid:16) − µ − w ( t ) (cid:62) x k (cid:17) → 0 from Lemma 1, ∃ t M > t − such that ∀ t > t M , t > Me − ˜ w (cid:62) x k , and if exp (cid:0) − r (cid:62) ( t ) x k (cid:1) ≥ M > 1 then
[Equation]
> 1 .
[Equation]
[Equation]
Furthermore, if ∃ t > t M such that exp (cid:0) − r (cid:62) ( t ) x k (cid:1) < M , then exp
[Equation]
which is lower bounded by
[Equation]
since (cid:12)(cid:12) x (cid:62) k r ( t ) (cid:12)(cid:12) > C 0 t − 0 .
5 µ − , x (cid:62) k r ( t ) < 0 and e x ≥ 1 + x .
In this case last line is strictly larger then 1 for sufﬁciently large t .
Therefore, after we substitute eqs.
A.33 and A.34 into
A.32, we ﬁnd that ∃ t (cid:48)− > t M > t − such that ∀ t > t (cid:48)− , term k in eq.
A.26 is strictly negative
[Equation]
If (cid:12)(cid:12) x (cid:62) k
[Equation]
( t ) (cid:12)(cid:12) ≥ [Equation] 2 , which is a special case of the previous case ( (cid:12)(cid:12) x (cid:62) k r ( t ) (cid:12)(cid:12) > C 0 t − 0 .
5 µ − ) then ∀ t > t (cid:48)− , either eq.
A.33 or A.34 holds.
Furthermore, in this case, ∃ t (cid:48)(cid:48)− > t (cid:48)− and
M (cid:48)(cid:48) > 1 such that ∀ t > t (cid:48)(cid:48)− eq.
A.34 can be lower bounded by exp
[Equation]
M (cid:48)(cid:48) > 1 .
Substituting this, together with eq.
A.33, into eq.
A.32, we can ﬁnd C (cid:48) [-
-]0 > 0 such we can upper bound term k in eq.
A.26 with − C
[Equation]
0 t − 1 , ∀ t > t (cid:48)(cid:48)− . (A.36) To conclude, we choose t 0 = max (cid:2) t (cid:48)(cid:48)(cid:48) + , t (cid:48)(cid:48) − (cid:3) : 1.
If (cid:107) P 1 r ( t ) (cid:107) ≥ (cid:15) 1 (as in Eq.
A.10), we have that max n
[Equation]
A. [Equation] where in (1) we used P (cid:62) 1 x n = x n ∀ n ∈ S , in (2) we denoted by σ min ( X S ) , the minimal non-zero singular value of X S and used eq.
A.10.
Therefore, for some k , (cid:12)(cid:12) x (cid:62) k r (cid:12)(cid:12) ≥ (cid:15) 2 (cid:44) |S| − 1 σ 2min ( X S ) (cid:15) 21 .
In this case, we denote C (cid:48)(cid:48)
as the minimum between C (cid:48)
(eq.
A.36) and η exp (cid:0) − max n ˜ w (cid:62) x n (cid:1) (cid:0) 1 − e − 0 .
5 (cid:15) 2 (cid:1) (cid:15) 2 (eq. A.30).
Then we ﬁnd that eq.
A.26 can be upper bounded by − C (cid:48)(cid:48)
t − 1 + o (cid:0) t − 1 (cid:1) , ∀ t > t 0 , given eq.
A.10.
Substituting this result, together with eqs.
A.24 and A.25 into eq.
A.23, we obtain ∀ t > t( r ( t + 1) − r ( t ))
{+(cid:62) r ( t ) ≤ − C (cid:48)(cid:48) 
 t − 1 + o (cid:0) t − 1 (cid:1) .+}
This implies that ∃ C 2 < C (cid:48)(cid:48)
and ∃ t 2 > t 0 such that eq.
A.11 holds.
This implies also that eq.
A.9 holds for (cid:107) P 1 r ( t ) (cid:107) ≥ (cid:15) 1 .
Otherwise, if (cid:107) P 1 r ( t ) (cid:107) < (cid:15) 1 , we ﬁnd that ∀ t > t 0 , each term in eq.
A.26 can be upper bounded by either zero (eqs.
A.29 and A.35), or terms proportional to t − 1 − 1 .
5 µ + (eq. A.28) or t − 1 − 0 .
5 µ − , (eq. A.31).
Combining this together with eqs.
A.24, A.25 into eq.
A.23 we obtain (for some positive constants C 3 , C 4 , C 5 , and C 6 )
[Equation]
Therefore, ∃ t 1 > t 0 and C 1 such that eq.
A.9 holds.
(cid:3)
B C ALCULATION OF CONVERGENCE RATES
From Theorem 3, we can write w ( t ) =
ˆ w log t + ρ ( t ) , where ρ ( t ) has bounded norm.
Calculation of normalized weight vector (eq. 3.1):
[Equation]
ˆ w (cid:107) 2 log t +   3(cid:32) 2 ρ ( t ) ˆ w (cid:107) ˆ w (cid:107)(cid:33)− (cid:107) ρ ( t ) (cid:107)2 (cid:107) ˆ w (cid:107)  log 2 t + O (cid:18) log 3 t (cid:19) (B.1)
=
ˆ w (cid:107) ˆ w (cid:107) + (cid:32) ρ
( t ) (cid:107) ˆ w (cid:107) − ˆ w (cid:107) ˆ w (cid:107) ρ ( t ) (cid:62) ˆ w (cid:107) ˆ w (cid:107)(cid:33)log t + O (cid:18) log 2 t (cid:19)
=
ˆ w (cid:107) ˆ w (cid:107) + (cid:32) I − ˆ w ˆ w (cid:62) (cid:107) ˆ w (cid:107)(cid:33) ρ ( t ) (cid:107) ˆ w (cid:107) log t + O (cid:18) log 2 t (cid:19) where to obtain eq. B.1 we used 1 √ 1+ x = 1 − 12 x + 34 x 2 + O (cid:0) x 3 (cid:1) , and in the last line we used the fact that ρ ( t ) has a bounded norm.
We use eq.
B.1 to calculate the angle (eq. 3.2): w ( t ) (cid:62) ˆ w (cid:107) w ( t ) (cid:107) (cid:107) ˆ w (cid:107) = ˆ w (cid:62) (cid:107) ˆ w (cid:107)(cid:18) ρ ( t ) 1 log t + ˆ w (cid:19)   1 − 1 log t ρ ( t ) (cid:62) ˆ w (cid:107) ˆ w cid:107)+   3(cid:32) 2 ρ ( t ) (cid:62) ˆ w (cid:107) ˆ w (cid:107)(cid:33)− (cid:107) ρ ( t ) (cid:107)2 (cid:107) ˆ w (cid:107)  log 2 t + O (cid:18) log 3 t (cid:19) 
 =1 + 2 (cid:107) ρ ( t ) (cid:107)(cid:107) ˆ w (cid:107)  (cid:32) ρ ( t ) (cid:62) ˆ w (cid:107) ˆ w (cid:107) (cid:107) ρ ( t ) (cid:107) (cid:33)−   log 2 t
+ O (cid:18) log 3 t (cid:19)
[Equation]
where in eq.
B.2 we used eq.
A.2.
Calculation
(eq. 3.4):
[Equation]
Next, we give an example demonstrating the bounds above are strict.
Consider optimization with and exponential loss (cid:96) ( u ) =
e − u , and a single data point x = (1 , 0) .
In this case ˆ w = (1 , 0) and (cid:107) ˆ w (cid:107) =
We take the limit η → 0 , and obtain the continuous time version of GD:
[Equation]
We can analytically integrate these equations to obtain
[Equation]
Using this example with w 2 (0) > 0 , it is easy to see that the above upper bounds are strict.
Lastly, recall that V is a set of indices for validation set samples.
We calculate of the validation loss for logistic loss, if the error of the L 2 max margin vector has some classiﬁcation errors on the validation, i.e., ∃ k ∈ V : ˆ w (cid:62) x k < 0 :
[Equation]
C S OFTMAX OUTPUT WITH CROSS - ENTROPY LOSS
We examine multiclass classiﬁcation.
In the case the labels are the class index y n ∈ { 1 , . . .
, K }
and we have a weight matrix W ∈ R K × d with w k being the k -th row of W .
Furthermore, we deﬁne w = vec (cid:0) W (cid:62) (cid:1) , a basis vector e k ∈ R K so that ( e k ) i = δ ki , and the matrix
A k ∈ R
dK ×
d so that A k = e k ⊗ I d , where ⊗ is the Kronecker product and I d is the d -dimension identity matrix.
Note that A (cid:62) k w = w k .
Consider the cross entropy loss with softmax output
[Equation]
Using our notation, this loss can be re-written as
[Equation]
Therefore ∇L ( w ) =
[Equation]
If, again, make the assumption that the data is strictly linearly separable, i.e. , in our notation
Assumption 4.
∃ w ∗ such that w (cid:62)∗ ( A k − A y n ) x n < 0 ∀ k (cid:54) =
y n .
then the expression
[Equation]
is strictly negative for any ﬁnite w .
However, from Lemma 5, in gradient descent with learning rate η > 2 β − 1 , we have that ∇ L ( w ( t ))
→ 0 .
This implies that: (cid:107) w ( t ) (cid:107) → ∞ , and ∀ k (cid:54) =
y n , ∃ r : w ( t ) (cid:62) ( A r − A k ) x n → ∞ , which implies ∀ k (cid:54) =
y n , max k w ( t ) (cid:62) ( A k − A y n ) x n → −∞ .
Examining the loss (eq.
C.1)
we ﬁnd that L ( w ( t ))
→ 0 in this case.
Thus, we arrive to an equivalent Lemma to Lemma 1, for this case:
Lemma 7.
Let w ( t ) be the iterates of gradient descent (eq. 2.2) with η < 2 β − 1 , for crossentropy loss operating on a softmax output, under the assumption of strict linear separability (Assumption 4), then: (1)
lim t →∞ L ( w ( t ))
= 0 , (2) lim t →∞ (cid:107) w ( t ) (cid:107) = ∞ , and (3) ∀ n, k (cid:54) =
y n : lim t →∞ w ( t ) (cid:62) ( A y n − A k ) x n = ∞ .
Therefore, since
[Equation]
w [Equation] where in the last line we assumed for simplicity that w ( t ) (cid:62) ( A k − A y n ) x n has a unique minimum in k , since then the other exponential terms inside the log become negligible.
If
[Equation]
has a limit k n , then we deﬁne ˜ x n =
( A y n − A k n ) x n , so eq.
C.2 is transformed to the standard logistic regression loss
[Equation]
to which our Theorems directly apply.
Therefore, w ( t ) / (cid:107) w ( t ) (cid:107) → ˆ w where
[Equation]
Recalling that A (cid:62) k w = w k , we can re-write this as
[Equation]
D D EEP NETWORKS , IF ONLY A SINGLE LAYER IS OPTIMIZED
We examine a deep neural network (DNN) with m = 1 , . . .
, L layers, piecewise linear activation functions f l and loss function (cid:96) following assumption 3, parameterized by weights matrices W l .
Since f l are piecewise linear, we can write for almost every u : f l ( u ) = ∇ f l ( u ) (cid:12) u (an element-wise product).
Given an input sample x n , for each layer l the input u n,l and output v n,l are calculated sequentially in a “forward propagation”
[Equation]
initialized by v n, 0 = x n .
Then, given the DNN output u n,L and target y n ∈ {− 1 , 1 } the loss (cid:96) ( y n u n,L ) can be calculated.
During training, the gradients of the loss are calculated using the chain rule in a “back-propagation”
[Equation]
initialized by δ n,L = 1 .
Finally, the weights are updated with GD.
The basic update (without weight sharing) is
[Equation]
where in the last line we used
[Equation]
Denoting ˜ x n,l = y n δ n,l ⊗ v n,l − 1 and w l = vec (cid:0) W (cid:62) l (cid:1) we obtain
[Equation]
We got the same update as in eq.
Thus, if ˜ x n,l does not change between iterations and becomes linearly separable so the training error can go to zero, we can apply Theorem 3.
This can happen if we only optimize W l , and the activation units stop crossing their thresholds, after a sufﬁcient number of iterations.
E A N EXPERIMENT WITH STOCHASTIC GRADIENT DESCENT
[Figure]
F G ENERIC SOLUTIONS OF THE KKT CONDITIONS IN EQ . 2.
Lemma 8.
For almost all datasets there is a unique α which satisﬁes the KKT conditions (eq. 2.6): ˆ w =
N (cid:88) n = α n x n ∀ n (cid:0) α n ≥ 0 and ˆ w (cid:62) x n = 1 (cid:1) OR (cid:0) α n = 0 and ˆ w (cid:62) x n > 1 (cid:1)
Furthermore, in this solution α n (cid:54) =
if ˆ w (cid:62) x n = 1 , i.e. , x n is a support vector ( n ∈ S ), and there are at most d such support vectors.
Proof.
For almost every set X , no more than d points x n can be on the same hyperplane.
Therefore, since all support vectors must lie on the same hyperplane, there can be at most d support vectors, for almost every X .
Given the set of support vectors, S , the KKT conditions of eq.
2.6 entail that α n = 0 if n / ∈ S and
[Equation]
where we denoted α S as α restricted to the support vector components.
For almost every set X , since d ≥ |S| , X (cid:62)S X S ∈ R |S|×|S| is invertible.
Therefore, α S has the unique solution
[Equation]
This implies that ∀ n ∈ S , α n is equal to a rational function in the components of X S , i.e. , α n = p n ( X S ) /q
n ( X S ) , where p n and q n are polynomials in the components of X S .
Therefore, if α n = 0 , then p n ( X S ) = 0 , so the components of X S must be at a root of the polynomial p n .
The roots of the polynomial p n have measure zero, unless ∀ X S : p n ( X S ) = 0 .
However, p n cannot be identically equal to zero, since, for example, if X (cid:62)S = (cid:2) I |S|×|S| ,
|S|× ( d −|S| ) (cid:3) , then X (cid:62)S X S = I |S|×|S| , and so in this case ∀ n ∈ S , α n = 1 (cid:54) = 0 , from eq.
F.2.
Therefore, for a given S , the event that "eq. F.1 has a solution with a zero component" has a zero measure.
Moreover, the union of these events, for all possible S , also has zero measure, as a ﬁnite union of zero measures sets (there are only ﬁnitely many possible sets S ⊂ { 1 , . . . , N } ).
This implies that, for almost all datasets X , α n = 0 only if n / ∈ S .
Furthermore, for almost all datasets the solution α is