C ROSS -E NTROPY L OSS L EADS T O P OOR M ARGINS
Anonymous authors
Paper under double-blind review
A BSTRACT
Neural networks could misclassify inputs that are slightly different from their training data, which indicates a small margin between their decision boundaries and the training dataset.
In this work, we study the binary classiﬁcation of linearly separable datasets and show that linear classiﬁers could also have decision boundaries that lie close to their training dataset if cross-entropy loss is used for training.
In particular, we show that if the features of the training dataset lie in a low-dimensional afﬁne subspace and the cross-entropy loss is minimized by using a gradient method, the margin between the training points and the decision boundary could be much smaller than the optimal value.
This result is contrary to the conclusions of recent related works such as (Soudry et al., 2018), and we identify the reason for this contradiction.
In order to improve the margin, we introduce differential training, which is a training paradigm that uses a loss function deﬁned on pairs of points from each class.
We show that the decision boundary of a linear classiﬁer trained with differential training indeed achieves the maximum margin.
The results reveal the use of cross-entropy loss as one of the hidden culprits of adversarial examples and introduces a new direction to make neural networks robust against them.
I NTRODUCTION
Training neural networks is challenging and involves making several design choices.
Among these are the architecture of the network, the training loss function, the optimization algorithm used for training, and their hyperparameters, such as the learning rate and the batch size.
Most of these design choices inﬂuence the solution obtained by the training procedure and have been studied in detail (Kingma & Ba, 2014;
Hardt et al., 2015; He et al., 2016; Wilson et al., 2017; Nar & Sastry, 2018; Smith et al., 2018).
Nevertheless, one choice has been mostly taken for granted when the network is trained for a classiﬁcation task: the training loss function.
Cross-entropy loss function is almost the sole choice for classiﬁcation tasks in practice.
Its prevalent use is backed theoretically by its association with the minimization of the Kullback-Leibler divergence between the empirical distribution of a dataset and the conﬁdence of the classiﬁer for that dataset.
Given the particular success of neural networks for classiﬁcation tasks (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014; He et al., 2016), there seems to be little motivation to search for alternatives for this loss function, and most of the software developed for neural networks incorporates an efﬁcient implementation for it, thereby facilitating its use.
Recently there has been a line of work analyzing the dynamics of training a linear classiﬁer with the cross-entropy loss function (Soudry et al., 2018; Nacson et al., 2018a;b; Ji & Telgarsky, 2018).
They speciﬁed the decision boundary that the gradient descent algorithm yields on linearly separable datasets and claimed that this solution achieves the maximum margin.
However, these claims were observed not to hold in the simple experiments we ran.
For example, Figure 1 displays a case where the cross-entropy minimization for a linear classiﬁer leads to a decision boundary which attains an extremely poor margin and is nearly orthogonal to the solution given by the hard-margin support vector machine (SVM).
We set out to understand this discrepancy between the claims of the previous works and our observations on the simple experiments.
We can summarize our contributions as follows.
[Figure]
We analyze the minimization of the cross-entropy loss for a linear classiﬁer by using only two training points,
i
e., only one point from each of the two classes, and we show that the dynamics of the gradient descent algorithm could yield a poor decision boundary, which could be almost orthogonal to the boundary with the maximum margin.
We identify the source of discrepancy between our observations and the claims of the recent works as the misleading abbreviation of notation in the previous works.
We clarify why the solution obtained with cross-entropy minimization is different from the SVM solution.
We show that for linearly separable datasets, if the features of the training points lie in an afﬁne subspace, and if the cross-entropy loss is minimized by a gradient method with no regularization to train a linear classiﬁer, the margin between the decision boundary of the classiﬁer and the training points could be much smaller than the optimal value.
We verify that when a neural network is trained with the cross-entropy loss to classify two classes from the CIFAR-10 dataset, the output of the penultimate layer of the network indeed produces points that lie on an afﬁne subspace.
We show that if there is no explicit and effective regularization, the weights of the last layer of a neural
network
could grow to inﬁnity during
training
with a gradient
method
Even
though
this
has been
observed in recent works as well,
we
are the
ﬁrst to point out
that
this
divergence
drives the conﬁdence of the neural network to 100% at almost every point in the input space if the network
is
trained for long.
In other words,
the conﬁdence
depends
heavily
on
the training duration and its exact
value might
be of little
signiﬁcance as long
as it is
above 50%.
We introduce differential training, which is a training paradigm that uses a loss function deﬁned on pairs of points from each class – instead of only one point from any class.
We show that the decision boundary of a linear classiﬁer trained with differential training indeed produces the SVM solution with the maximum hard margin
C LASSIFICATION OF T WO P OINTS M ANIFESTS P OOR M ARGIN
We start with a simple binary classiﬁcation problem.
Given two points x ∈ R d and − y ∈ R d from two different classes, we can ﬁnd a linear classiﬁer by minimizing the cross-entropy loss function
[Equation]
or equivalently, by solving
[Equation]
where ˜ x
= [ x (cid:62) 1] (cid:62) , − ˜ y
=
[ − y (cid:62) 1] (cid:62) and ˜ w =
[ w (cid:62) b ] (cid:62) .
Unless the two points x and − y are equal, the function (1) does not attain its minimum at a ﬁnite value of ˜ w .
Consequently, if the gradient descent algorithm is used to minimize (1), the iterate at time k , ˜ w [ k ] , diverges as k increases.
The following theorem characterizes the growth rate of ˜ w [ k ] and its direction in the limit by using a continuous-time approximation to the gradient descent algorithm.
Theorem 1.
Given two points x ∈ R d and − y ∈ R d , let ˜ x and − ˜
y denote [ x (cid:62) 1] (cid:62) and [ − y (cid:62) 1] , respectively.
Without loss of generality, assume (cid:107) x (cid:107) ≤ (cid:107)
y (cid:107) .
If the two points are in different classes and we minimize the cross-entropy loss
[Equation]
by using the continuous-time approximation to the gradient descent algorithm
[Equation]
with the initialization ˜ w (0)
=
and the learning rate δ , then
[Equation]
where σ x = (cid:107) ˜ x (cid:107) 2 , σ xy = ˜ x (cid:62) ˜
y and σ y = (cid:107) ˜
y
(cid:107) 2 .
Note that ﬁrst d coordinates of (2) represent the normal vector of the decision boundary obtained by minimizing the cross-entropy loss (1).
This vector is different from x + y , which is the direction of the maximum-margin solution given by the SVM.
In fact, the direction in (2) could be almost orthogonal to the SVM solution in certain cases, which implies that the margin between the points and the decision boundary could be much smaller than the optimal value.
Corollary 1 describes a subset of these cases.
Corollary 1.
Given two points x and − y in R d , let ψ denote the angle between the solution given by (2) and the solution given by the SVM, i.e., ( x + y ) .
If x (cid:62) y = 1 , then
[Equation]
, where σ x = (cid:107) x (cid:107) 2 +1 and σ y = (cid:107) y (cid:107) 2 +1 .
Consequently, as (cid:107) x (cid:107) / (cid:107) y (cid:107) approaches 0 while maintaining the condition x (cid:62) y = 1 , the angle ψ converges to π/ 2 .
Remark 1.
Corollary 1 shows that if x and − y have disparate norms, the minimization of the crossentropy loss with gradient descent algorithm could lead to a direction which is almost orthogonal to the maximum-margin solution.
It may seem like this problem could be avoided with preprocessing the data so as to normalize the data points.
However, this approach will not be effective for neural networks: if we consider an L -layer neural network, w (cid:62)
φ L − 1 ( x ) , and regard the ﬁrst L − 1 layers, φ L − 1 ( · ) , as a feature mapping, preprocessing a dataset { x i } i ∈ I will not produce a normalized set of features { φ L − 1 ( x i ) } i ∈ I .
Note that we could not normalize { φ L − 1 ( x i ) } i ∈ I directly either, since the mapping
φ L − 1 ( · ) evolves during training.
Remark 2.
Theorem 1 shows that the norm of w keeps growing unboundedly as the training continues.
The same behavior will be observed for larger datasets in the next sections as well.
Since the “conﬁdence” of the classiﬁer for its prediction at a point x is given by
[Equation]
this unbounded growth of (cid:107) w (cid:107) drives the conﬁdence of the classiﬁer to 100% at every point in the input space, except at the points on the decision boundary, if the algorithm is run for long.
Given the lack of effective regularization for neural networks, a similar unbounded growth is expected to be observed in neural network training as well, which is mentioned in (Bartlett et al., 2017).
As a result, the conﬁdence of a neural network might be highly correlated with the training duration, and whether a neural network gives 99% or 51% conﬁdence for a prediction might be of little importance as long as it is above 50%.
In other words, regarding this conﬁdence value as a measure of similarity between an input and the training dataset from the most-likely class should be reconsidered.
D ATA WITH L OW -D IMENSIONAL F EATURES A TTAIN P OOR M ARGIN
In this section, we examine the binary classiﬁcation of a linearly separable dataset by minimizing the cross-entropy loss function.
Recently, this problem has also been studied in (Soudry et al., 2018;
Nacson et al., 2018b;a; Ji & Telgarsky, 2018).
We restate an edited version of the main theorem of (Soudry et al., 2018), followed by the reason of the edition.
Theorem 2 (Adapted from Theorem 3 of (Soudry et al., 2018)) .
Given two sets of points { x i } i ∈ I and {− y j } j ∈ J that are linearly separable in R d , let ˜ x
i and − ˜ y
j denote [ x (cid:62) i 1] (cid:62) and [ − y (cid:62) j 1] (cid:62) , respectively, for all i ∈ I , j ∈ J .
Then the iterate of the gradient descent algorithm, ˜ w ( t ) , on the cross-entropy loss function
[Equation]
with a sufﬁciently small step size will converge in direction:
[Equation]
where w is the solution to
[Equation]
The solution (4) given in Theorem 2 was referred in (Soudry et al., 2018), and consequently in the other works, as the maximum-margin solution.
However, due to the misleading absence of the bias term in the notation, this is incorrect.
Given the linearly separable sets of points { x i } i ∈ I and {− y j } j ∈ J , the maximum-margin solution given by the SVM solves minimize w,b (cid:107) w
[Equation]
subject to (cid:104) w, x i (cid:105) + b ≥∀ i ∈
I, (cid:104) w, − y j (cid:105) + b ≤ − 1 ∀ j ∈ J.
(P1)
On the other hand, the solution given by Theorem 2 corresponds to minimize w,b (cid:107) w (cid:107) 22 + bsubject to (cid:104) w, x i (cid:105)
+ b = (cid:104)
˜ w, ˜ x i (cid:105) ≥∀ i ∈
I, (cid:104) w, − y j (cid:105) + b = (cid:104) ˜ w, − ˜ y
j (cid:105) ≤ − 1 ∀
j ∈ J, (P2) where we deﬁne ˜ w
= [ w (cid:62) b ] (cid:62) , ˜ x i =
[ x (cid:62) i 1] (cid:62) and ˜ y j
= [ y (cid:62) j − 1] (cid:62) for all i ∈ I, j ∈ J .
Even though the sets of constraints for both problems are identical, their objective functions are different, and consequently, the solutions are different.
As a result, the decision boundary obtained by crossentropy minimization does not necessarily attain the maximum hard margin.
In fact, as the following theorem shows, its margin could be arbitrarily worse than the maximum margin.
Theorem 3.
Assume that the points { x i } i ∈ I and {− y j } j ∈ J are linearly separable and lie in an afﬁne subspace; that is, there exist a set of orthonormal vectors { r k } k ∈ K and a set of scalars { ∆ k } k ∈ K such that
[Equation]
Let (cid:104) w, ·(cid:105) + B = 0 denote the decision boundary obtained by minimizing the cross-entropy loss, i.e. the pair ( w, B ) solves
[Equation]
Then the minimization of the cross-entropy loss (3) yields a margin smaller than or equal to
[Equation]
where γ denotes the optimal hard margin given by the SVM solution.
Remark 3.
Theorem 3 shows that if the training points lie in an afﬁne subspace, the margin obtained by the cross-entropy minimization will be smaller than the optimal margin value.
As the dimension of this afﬁne subspace decreases, the cardinality of the set K increases and the term (cid:80)
k ∈ K ∆ 2 k could become much larger than 1 /γ 2 .
Therefore, as the dimension of the subspace containing the training points gets smaller compared to the dimension of the input space, cross-entropy minimization with a gradient method becomes more likely to yield a poor margin.
Note that this argument also holds for classiﬁers of the form w (cid:62) φ ( x ) with the ﬁxed feature mapping φ ( · ) .
The next theorem relaxes the condition of Theorem 3 and allows the training points to be near an afﬁne subspace instead of being exactly on it.
Note that the ability to compare the margin obtained by cross-entropy minimization with the optimal value is lost.
Nevertheless, it highlights the fact that same set of points could be assigned a different margin by cross-entropy minimization if all of them are shifted away from the origin by the same amount in the same direction.
Theorem 4.
Assume that the points { x i } i ∈ I and {− y j } j ∈ J in R d are linearly separable and there exist a set of orthonormal vectors { r k } k ∈ K and a set of scalars { ∆ k } k ∈ K such that
[Equation]
Let (cid:104) w, ·(cid:105) + B = 0 denote the decision boundary obtained by minimizing the cross-entropy loss, i.e. the pair ( w, B ) solves
[Equation]
Then the minimization of the cross-entropy loss (3) yields a margin smaller than or equal to
[Equation]
Remark 4.
Both Theorem 3 and Theorem 4 consider linearly separable datasets.
If the dataset is not linearly separable, (Ji & Telgarsky, 2018) predicts that the normal vector of the decision boundary, w , will have two components, one of which converges to a ﬁnite vector and the other diverges.
The diverging component still has the potential to drive the decision boundary to a direction with a poor margin.
In fact, the margin is expected to be small especially if the points intruding into the opposite class lie in the same subspace as the optimal normal vector for the decision boundary.
In this work, we focus on the case of separable datasets as this case provides critical insight into the issues of state-of-the-art neural networks, given they can easily attain zero training error even on randomly generated datasets, which indicates the linear separability of the features obtained at their penultimate layers (Zhang et al., 2017).
D IFFERENTIAL T RAINING I MPROVES M ARGIN
In previous sections, we saw that the cross-entropy minimization could lead to poor margins, and the main reason for this was the appearance of the bias term in the objective function of (P2).
In order to remove the effect of the bias term, consider the SVM problem (P1) and note that this problem could be equivalently written as
[Equation]
if we only care about the weight parameter w .
This gives the hint that if we use the set of differences { x i + y j : i ∈ I, j ∈ J } instead of the individual sets { x i } i ∈ I and {− y j } j ∈ J , the bias term could be excluded from the problem.
This was also noted in (Keerthi et al., 2000; Ishibashi et al., 2008) previously.
Indeed, this approach allows obtaining the SVM solution with a loss function similar to the cross-entropy loss, as the following theorem shows.
Theorem 5.
Given two sets of points { x i } i ∈ I and {− y j } j ∈ J that are linearly separable in R d , if we solve
[Equation]
by using the gradient descent algorithm with a sufﬁciently small learning rate, the direction of w converges to the direction of maximum-margin solution, i.e.
[Equation]
where w SVM is the solution of (P3).
Proof.
Apply Theorem 2 by replacing the sets { x i } i ∈
I and {− y j } j ∈ J with { x i + y j } i ∈
I,j ∈ J and the empty set, respectively.
Then the minimization of the loss function (5) with the gradient descent algorithm leads to
[Equation]
where w satisﬁes
[Equation]
Since w SVM is the solution of (P3), we obtain w = 12 w SVM , and the claim of the theorem holds.
(cid:4)
Remark 5.
Theorem 5 is stated for the gradient descent algorithm, but the identical statement could be made for the stochastic gradient method as well by invoking the main theorem of (Nacson et al., 2018).
Minimization of the cost function (5) yields the weight parameter
ˆ w of the decision boundary.
The bias parameter, b , could be chosen by plotting the histogram of the inner products {(cid:104) ˆ w, x i (cid:105)} i ∈ I and {(cid:104) ˆ w, − y j (cid:105)} j ∈ J and ﬁxing a value for ˆ b such that
[Equation]
7b)
The largest hard margin is achieved by
[Equation]
[Equation]
However, by choosing a larger or smaller value for ˆ b , it is possible to make a tradeoff between the
Type-I and Type-II errors.
The cost function (5) includes a loss deﬁned on every pair of data points from the two classes.
This cost function can be considered as the cross-entropy loss on a new dataset which contains | I | × | J | points.
There are two aspects of this fact:
When standard loss functions are used for classiﬁcation tasks, we need to oversample or undersample either of the classes if the training dataset contains different number of points from different classes.
This problem does not arise when we use the cost function (5).
Number of pairs in the new dataset, | I | × | J | , will usually be much larger than the original dataset, which contains | I | + | J | points.
Therefore, the minimization of (5) might appear more expensive than the minimization of the standard cross-entropy loss computationally.
However, if the points in different classes are well separated and the stochastic gradient method is used to minimize (5), the algorithm achieves zero training error after using only a few pairs, which is formalized in Theorem 6.
Further computation is needed only to improve the margin of the classiﬁer.
In addition, in our experiments to train a neural network to classify two classes from the CIFAR-10 dataset, only a few percent of | I | × | J | points were observed to be sufﬁcient to reach a high accuracy on the training dataset.
Theorem 6.
Given two sets of points { x i } i ∈ I and {− y j } j ∈ J that are linearly separable in R d , assume the cost function (5) is minimized with the stochastic gradient method.
Deﬁne
[Equation]
and let γ denote the hard margin that would be obtained with the SVM:
[Equation]
If 2 γ ≥ 5 max( R x , R y ) , then the stochastic gradient algorithm produces a weight parameter, ˆ w , only in one iteration which satisﬁes the inequalities (7a)-(7b) along with the bias, ˆ b , given by (8).
N UMERICAL E XPERIMENTS
In this section, we present numerical experiments supporting our claims.
Differential training. In Figure 2, we show the decision boundaries of two linear classiﬁers, where one of them is trained by minimizing the cross-entropy loss , and the other through differential training .
Unlike the example shown in Figure 1, here the data do not exactly lie in an afﬁne subspace.
In particular, one of the classes is composed of 10 samples from a normal distribution with mean (2 , 12) and variance 25 , and the other class is composed of 10 samples from a normal distribution with mean (40 , 50) and variance 25 .
As can be seen from the ﬁgure, the cross-entropy minimization yields a margin that is smaller than differential training, even though when the training dataset is not low-dimensional, which is predicted by Theorem 4.
[Figure]
Low-dimensionality. We empirically evaluated if the features obtained at the penultimate layer of a neural network indeed lie in a low-dimensional afﬁne subspace.
For this purpose, we trained a convolutional neural network architecture to classify horses and planes from the CIFARdataset (Krizhevsky & Hinton, 2009).
Figure 3 shows the cumulative variance explained for the features that feed into the soft-max layer as a function of the number of principle components used.
We observe that the features, which are the outputs of the penultimate layer of the network, lie in a low-dimensional afﬁne subspace, and this holds for a variety of training modalities for the network.
This observation is relevant to Remark 3.
The dimension of the subspace containing the training points is at most 20 , which is much smaller than the dimension of the feature space, 84 .
Consequently, cross-entropy minimization with a gradient method is expected to yield a poor margin on these features.
D ISCUSSION
We compare our results with related works and discuss their implications for the following subjects.
Adversarial examples.
State-of-the-art neural networks have been observed to misclassify inputs that are slightly different from their training data, which indicates a small margin between their decision boundaries and the training dataset (Szegedy et al., 2013;
Goodfellow et al., 2015; MoosaviDezfooli et al., 2017; Fawzi et al., 2017).
Our results reveal that the combination of gradient methods, cross-entropy loss function and the low-dimensionality of the training dataset (at least in some domain) has a responsibility for this problem.
Note that SVM with the radial basis function was shown to be robust against adversarial examples, and this was attributed to the high nonlinearity of the radial basis function in (Goodfellow et al., 2015).
Given that the SVM uses neither the cross entropy loss function nor the gradient descent algorithm for training, we argue that the robustness of SVM is no surprise – independent of its nonlinearity.
Lastly, effectiveness of differential training for neural networks against adversarial examples is our ongoing work.
[Figure]
i
[Figure]
n
[Figure]
e
[Figure]
i
[Figure]
Low-dimensionality of the training dataset.
As stated in Remark 3, as the dimension of the afﬁne subspace containing the training dataset gets very small compared to the dimension of the input space, the training algorithm will become more likely to yield a small margin for the classiﬁer.
This observation conﬁrms the results of (Marzi et al., 2018), which showed that if the set of training data is projected onto a low-dimensional subspace before feeding into a neural network, the performance of the network against adversarial examples is improved – since projecting the inputs onto a low-dimensional domain corresponds to decreasing the dimension of the input space.
Even though this method is effective, it requires the knowledge of the domain in which the training points are low-dimensional.
Because this knowledge will not always be available, ﬁnding alternative training algorithms and loss functions that are suited for low-dimensional data is still an important direction for future research.
Robust optimization.
Using robust optimization techniques to train neural networks has been shown to be effective against adversarial examples (Madry et al., 2018; Athalye et al., 2018).
Note that these techniques could be considered as inﬂating the training points by a presumed amount and training the classiﬁer with these inﬂated points.
Consequently, as long as the cross-entropy loss is involved, the decision boundaries of the neural network will still be in the vicinity of the inﬂated points.
Therefore, even though the classiﬁer is robust against the disturbances of the presumed magnitude, the margin of the classiﬁer could still be much smaller than what it could potentially be.
Differential training.
We introduced differential training, which allows the feature mapping to remain trainable while ensuring a large margin between different classes of points.
Therefore, this method combines the beneﬁts of neural networks with those of support vector machines.
Even though moving from 2 N training points to N 2 seems prohibitive, it points out that a true classiﬁcation should in fact be able to differentiate between the pairs that are hardest to differentiate, and this search will necessarily require an N 2 term.
Some heuristic methods are likely to be effective, such as considering only a smaller subset of points closer to the boundary and updating this set of points as needed during training.
If a neural network is trained with this procedure, the network will be forced to ﬁnd features that are able to tell apart between the hardest pairs.
Nonseparable data.
What happens when the training data is not linearly separable is an open direction for future work.
However, as stated in Remark 4, this case is not expected to arise for the state-of-the-art networks, since they have been shown to achieve zero training error even on randomly generated datasets (Zhang et al., 2017), which implies that the features represented by the output of their penultimate layer eventually become linearly separable.
A P ROOF OF T HEOREM
Theorem 1 could be proved by using Theorem 2, but we provide an independent proof here.
Gradient descent algorithm with learning rate δ on the cross-entropy loss (1) yields
[Equation]
If ˜ w (0) = 0 , then ˜ w ( t ) =
p ( t )˜ x
+ q ( t )˜
y for all t ≥ 0 ,
where ˙ p =
δ
[Equation]
Deﬁne
[Equation]
xy .
Then we can write
[Equation]
+ e − α .
Finally, deﬁne z = e α and v = e β so that ˙
z =
z z +(cid:18)
[Equation]
+
[Equation]
(cid:19)
[Equation]
˙ v = v
[Equation]
+(cid:18) c
[Equation]
bv
[Equation]
z +(cid:19) .
Without loss of generality, assume c ≥ a .
Before we proceed, note that d
[Equation]
Let u and w denote zv and z +1 v +1 , respectively.
Then,
[Equation]
Lemma 1.
If b = 0 , then lim t →∞ ˜ w ( t ) log
[Equation]
t
[Equation] [Equation]
˜ x (cid:107) ˜ x
[Equation]
(cid:107) ˜ y (cid:107) ˜
y.
Proof.
Note that
[Equation]
Then, lim t →∞
[Equation]
and lim
[Equation]
Lemma 2.
If b < 0 , then there exists t 0 ∈ (0 , ∞ ) such that
[Equation]
Proof.
Note that − bc ≤ a − b c − b ≤ a − b because b < 0 .
First assume z 0 +1 v 0 +1 ≥ a − b .
Then, ˙ z ≤ 0 and ˙ v = v v +(cid:18) c + bv +z +(cid:19) ≥ v v +
[Equation]
a
[Equation]
a
[Equation]
which implies that z
+
[Equation]
a
[Equation]
(cid:19)
[Equation]
as long as z +1 v +1 ≥ a − b , and this can be satisﬁed only for a ﬁnite time.
Now assume z 0 +1 v 0 +1 ≤ − bc .
Then, ˙ v ≤ 0 and ˙ z =
z z +(cid:18) a + bz +v +(cid:19) ≥ zz 0 +ac − bc , which implies z +v + 1 ≥ (cid:18)
zz
+ac − bc t +
[Equation]
as long as z +1 v +1 ≤ − bc , and this can be satisﬁed only for a ﬁnite time as well.
[Equation]
Lemma 3.
If b < 0 , then
[Equation]
where t 0 is given by Lemma 2.
Proof.
[Equation]
Lemma 4.
If b < 0 , then lim
[Equation]
Proof.
From Lemma 3,
[Equation]
and
[Equation]
Combining these two inequalities, we have
[Equation]
As a result,
[Equation]
By using Lemma
[Equation]
Lemma 5.
If b < 0 , then
[Equation]
Proof.
Solving the set of equations
[Equation]
we obtain
[Equation]
Lemma 6.
If b > 0 , then
[Equation]
Proof.
Note that ˙
z ≥ a/ 2 and ˙ v ≥ c/ 2 ; therefore,
[Equation]
if either side exists.
Remember that
[Equation]
We can compute
[Equation]
The function f is strictly increasing and convex for w > 0 .
We have
[Equation]
Therefore, when b ≥ a , the only ﬁxed point of f over [0 , ∞ ) is the origin, and when a > b , 0 and ( a − b ) / ( c − b ) are the only ﬁxed points of f over [0 , ∞ ) .
Figure 4 shows the curves over which ˙ u = 0 and ˙ w = 0 .
Since lim t →∞ u = lim t →∞ w , the only points ( u, w ) can converge to are the ﬁxed points of f .
Remember that
[Equation]
so when a > b , the origin (0 , 0) is unstable in the sense of Lyapunov, and ( u, w ) cannot converge to it.
Otherwise, (0 , 0) is the only ﬁxed point, and it is stable.
As a result,
[Equation]
[Figure]
Lemma 7.
If a > b > 0 , then
[Equation]
Proof.
From Lemma 6,
[Equation]
Consequently, lim
[Equation]
which gives the same solution as Lemma
: lim
[Equation]
Lemma 8.
If b ≥ a
[Equation]
then lim
[Equation]
Proof.
[Equation]
Proof of Theorem 1.
Lemma 1, Lemma 5, Lemma 7 and Lemma 8 prove all cases of Theorem 1. (cid:4)
B P ROOF OF C OROLLARY
Since x (cid:62) y =
[Equation]
we have σ xy = ˜
x (cid:62) ˜
y
= x (cid:62) y − 1 = 0 < a .
Then the normal vector of the decision boundary is proportional to 1 σ x x
+ 1 σ y y .
For the angle between this vector and the solution of the
SVM
[Equation]
we can write
[Equation]
We can obtain a lower bound for square of the denominator as
[Equation]
As a result
[Equation]
C P ROOF OF T HEOREM
Let (cid:104) w SVM , ·(cid:105) +
b SVM =
denote the hyperplane obtained as the solution of SVM, i.e., ( w SVM , b SVM ) is the solution to the problem
[Equation]
Assume that w = u
+ (cid:80) mk =
α k r k , where u ∈ R d and (cid:104) u, r k (cid:105) = 0 for all k ∈ K .
The Lagrangian of the problem (4) can be written as
[Equation]
where µ i ≥ 0 for all i ∈ I and ν j ≥ 0 for all j ∈ J .
KKT conditions for the optimality of w and B requires that
[Equation]
and consequently, for each k ∈ K ,
[Equation] [Equation]
Then
[Equation]
we can write w as
[Equation]
Remember, by deﬁnition,
[Equation]
J.
Since the vector u also satisﬁes (cid:104) u, x i + y j (cid:105) =
(cid:104) w, x
i + y j (cid:105) ≥ 2 for all i ∈ I, j ∈ J , we have (cid:107) u (cid:107) ≥ (cid:107) w SVM (cid:107) = 1 γ .
As a result, the margin obtained by minimizing the cross-entropy loss is
[Equation]
D P ROOF OF T HEOREM
If B < 0 , we could consider the hyperplane (cid:104) w, ·(cid:105) − B = 0 for the points {− x i } i ∈ I and { y j } j ∈ J , which would have the identical margin due to symmetry.
Therefore, without loss of generality, assume B ≥ 0 .
As in the proof of Theorem 3, KKT conditions for the optimality of w and B requires
[Equation]
where µ i ≥ 0 and ν j ≥ 0 for all i ∈ I, j ∈ J .
Note that for each k ∈ K ,
[Equation]
Since { r k } k ∈ K is an orthonormal set of vectors,
[Equation]
The result follows from the fact that (cid:107) w (cid:107) − 1 is an upper bound on the margin.
[Equation]
E P ROOF OF T HEOREM
In order to achieve zero training error in one iteration of the stochastic gradient algorithm, it is sufﬁcient to have
[Equation]
or equivalently,
[Equation]
By deﬁnition of the margin
[Equation]
there exists a vector w SVM ∈ R d with unit norm which satisﬁes
[Equation]
i ∈
[Equation]
Note that w SVM is orthogonal to the decision boundary given by the SVM.
Then we can write every x i + y j as
[Equation]
where δ xi , δ yj ∈ R d and (cid:107) δ xi (cid:107) ≤ R x and (cid:107) δ yj (cid:107) ≤ R y .
Then, condition (9) is satisﬁed if
[Equation]
γw
[Equation]
, or equivalently
if
[Equation]
If we choose γ > 52 max( R x , R y ) , we have
[Equation]
R y ) 2 > 0 , which guarantees (10) and completes the proof.
(cid:4)