Understanding Square Loss in Training Overparameterized Neural Network Classiﬁers
Author(s
Abstract
Deep learning has achieved many breakthroughs in modern classiﬁcation tasks.
Numerous architectures have been proposed for different data structures but whenit comes to the loss function, the cross-entropy loss is the predominant choice.
Recently, several alternative losses have seen revived interests for deep classiﬁers.
In particular, empirical evidence seems to promote square loss but a theoreticaljustiﬁcation is still lacking.
In this work, we contribute to the theoretical understanding of square loss in classiﬁcation by systematically investigating how itperforms for overparameterized neural networks in the neural tangent kernel (NTK)regime.
Interesting properties regarding the generalization error, robustness, andcalibration error are revealed.
We consider two cases, according to whether classesare separable or not.
In the general non-separable case, fast convergence rate isestablished for both misclassiﬁcation rate and calibration error.
When classes areseparable, the misclassiﬁcation rate improves to be exponentially fast.
Further,the resulting margin is proven to be lower bounded away from zero, providingtheoretical guarantees for robustness.
We expect our ﬁndings to hold beyond the
NTK regime and translate to practical settings.
To this end, we conduct extensiveempirical studies on practical neural networks, demonstrating the effectiveness ofsquare loss in both synthetic low-dimensional data and real image data.
Comparingto cross-entropy, square loss has comparable generalization error but noticeableadvantages in robustness and model calibration.
Introduction
The pursuit of better classiﬁers has fueled the progress of machine learning and deep learning research.
The abundance of benchmark image datasets, e.g., MNIST, CIFAR, ImageNet, etc., provide test ﬁeldsfor all kinds of new classiﬁcation models, especially those based on deep neural networks (DNN).
With the introduction of CNN, ResNets and transformers, DNN classiﬁers are constantly improvingand catching up to the human-level performance.
In contrast to the active innovations in modelarchitecture, the training objective remains largely stagnant, with cross-entropy loss being the defaultchoice.
Despite its popularity, cross-entropy has been shown to be problematic in some applications.
Among others, [1] argued that features learned from cross-entropy lack interpretability and proposed anew loss aiming for maximum coding rate reduction.
[2] linked the use of cross-entropy to adversarialvulnerability and proposed a new classiﬁcation loss based on latent space matching.
[3] discoveredthat the conﬁdence of most DNN classiﬁers trained with cross-entropy is not well-calibrated.
Recently, several alternative losses have seen revived interests for deep classiﬁers.
In particular,many existing works have presented empirical evidence promoting the use of square loss over crossentropy.
[4] conducted large-scale experiments comparing the two and found that square loss tendsto perform better in natural language processing related tasks while cross-entropy usually yields slightly better accuracy in image classiﬁcation.
Similar comparisons are also made in [5, 6, 7, 8].[9] compared a variety of loss functions and output layer regularization strategies on the accuracyand out-of-distribution robustness, and found that square loss has greater class separation and betterperformance.
In comparison to the empirical investigation, theoretical understanding of square loss in training deeplearning classiﬁers is still lacking.
Through our lens, square loss has its uniqueness among classicclassiﬁcation losses, and we argue that it has great potentials for modern classiﬁcation tasks.
Belowwe list our motivations and reasons why.
Explicit feature modeling Deep learning’s success can be largely attributed to its superior abilityas feature extractors.
For classiﬁcation, the ideal features should be separated between classes andconcentrated within classes.
However, when optimizing cross-entropy loss, it is not obvious whatthe learned features should look like [1].
In the terminal stage of training, [10] proved that whencross-entropy is sufﬁciently minimized, the penultimate layers features will collapse to the scaledsimplex structure.
Such phenomenon is referred to as “neural collapse".
Knowing this, would it bebetter to directly enforce the terminal solution by using square loss with the simplex coding [11],as Euclidean distance is probably most natural to measure the distance between samples’ featuresand class-means?
Unlike cross-entropy, square loss uses the label codings (one-hot, simplex etc.) asfeatures, which can explicitly control class separations.
Model calibration An ideal classiﬁer should not only give the correct class prediction, but alsowith the correct conﬁdence.
Calibration error measures the closeness of the predicted conﬁdenceto the underlying conditional probability ⌘ .
Using square loss in classiﬁcation can be essentiallyviewed as regression where it treats discrete labels as continuous code vectors.
It can be shown thatthe optimal classiﬁer under square loss is 2 ⌘  1 , linear with the ground truth.
This distinguishingproperty allows it to easily recover ⌘ .
In comparison, the optimal classiﬁers under the hinge loss andcross-entropy are sign (2 ⌘  1) and log( ⌘ 1  ⌘ ) , respectively [12].
Therefore, hinge loss doesn’t providereliable information on the prediction conﬁdence, and cross-entropy can be problematic when ⌘ isclose to 0 or 1, as stated in [12].
Hence, in terms of model calibration, square loss is a natural choice.
Connections to popular approaches Mixup [13] is a data augmentation technique where augmented data are constructed via convex combinations of inputs and their labels.
Like in square loss,mixup treats labels as continuous and is shown to improve the generalization of DNN classiﬁers.
Inknowledge distillation [14], where a student classiﬁer is trying to learn from a trained teacher, [15]proved that the “best" teacher with the ground truth conditional probabilities provides the lowestvariance in student learning.
Since classiﬁers trained using square loss is a consistent estimator of ⌘ ,one can argue that it is a better teacher.
In supervised contrastive learning [16], the optimal featuresare the same
as those from square loss with simplex label coding [17] (details in Section 3.4).
Despite its lack of popularity in practice, square loss has many advantages that can be easilyoverlooked.
In this work, we systematically investigate from a statistical estimation perspective, theproperties of deep learning classiﬁers trained using square loss.
Comparing to cross entropy, thesquare loss is much more theoretically tractable, which allows us to develop sharper results on thetraining process and generalization performance.
The neural networks in our analysis are requiredto be sufﬁciently overparameterized in the neural tangent kernel (NTK) regime.
Even though thisrestricts the implication of our results, it is a necessary ﬁrst step towards a deeper understanding.
Insummary, our main contributions are:
• Generalization error bound : We consider two cases, according to whether classes are separableor not.
In the general non-separable case, we adopt the classical binary classiﬁcation setting withsmooth conditional probability.
Fast convergence rate is established for overparameterized neuralnetwork classiﬁers with Tsybakov’s noise condition.
If two classes are separable with positive margin,we show that overparameterized neural network classiﬁers can provably reach zero misclassiﬁcationerror with probability exponentially tending to one.
To the best of our knowledge, this is the ﬁrst suchresult for separable but not linear separable classes.
Furthermore, we bridge these two cases and offera uniﬁed view by considering auxiliary random noise injection.•
Adversarial robustness (margin property) :
When two classes are separable, the decision boundary is not unique and large-margin classiﬁers are preferred.
The margin is naturally connected toadversarial robustness [18].
In the separable case, we show that the decision boundary of overparameterized neural network classiﬁers trained by square loss cannot be too close to the data support and the
resulting margin is lower bounded away from zero, providing theoretical guarantees for adversarialrobustness.• Calibration error :
We show that classiﬁers trained using square loss are inherently well-calibrated,i.e., the trained classiﬁer provides consistent estimation of the ground-truth conditional probability inL 1 norm.
Such property doesn’t hold for cross-entropy.• Empirical evaluation : We corroborate our theoretical ﬁndings with empirical experiments in bothsynthetic low-dimensional data and real image data.
Comparing to cross-entropy, square loss hascomparable generalization error but noticeable advantages in robustness and model calibration.
This work contributes to the theoretical understanding of deep classiﬁers from a nonparametricestimation point of view, which has been a classic topic in statistics literature.
Among others, [19]established the optimal convergence rate for 0-1 loss excess risk when the decision boundary issmooth.
[12, 20] extended the analysis to various surrogate losses.
[21, 22] studied the convergencerates for plug-in classiﬁers from local averaging estimators.
[23] investigated the convergence ratefor support vector machine using Gaussian kernels.
We build on and extend classic results to neuralnetworks in the NTK regime.
There exist nonparametric results on deep classiﬁers, e.g., [24] derivedfast convergence rates of DNN classiﬁers that minimize the empirical hinge loss, [25] considereda family of CNN-inspired classiﬁers and developed similar results for the empirical cross-entropyminimizer, etc.
Unlike aforementioned works that only concern the existence of a good classiﬁer(with theoretical worst-case guarantee), in ignorance of the tremendous difﬁculty of neural networkoptimization, our results further incorporate the training algorithm and apply to trained classiﬁers,which relates better to practice.
To the best of the authors’ knowledge, similar attainable fast rates(faster than n  12 ) have never been established for neural network classiﬁers.
We require the neural network to be overparameterized, which has been extensively studied recentlyvia NTK.
Most such results are in the regression setting with a handful of exceptions.
[26] showedthat only polylogarithmic width is sufﬁcient for gradient descent to overﬁt the training data usinglogistic loss.
[27] proved generalization error bound for regularized NTK in classiﬁcation.
[28, 29]provided optimization and generalization guarantees for overparameterized network trained withcross-entropy.
In comparison, our results are sharper in the sense that we take the ground truth dataassumptions into consideration.
This allows a faster convergence rate, especially when the classes areseparable, where the exponential convergence rate is attainable.
The NTK framework greatly reducesthe technical difﬁculty for our theoretical analysis.
However, our results are mainly due to propertiesof the square loss itself and we expect them to hold for a wide range of classiﬁers.
There are other works investigating the use of square loss for training (deep) classiﬁers.
[30]uncovered that the “neural collapse" phenomenon also occurs under the square loss.
[31] comparedclassiﬁcation and regression tasks in the overparameterized linear model with Gaussian features,illustrating different roles and properties of loss functions used at the training and testing phases.[32] made interesting observations on effects of popular regularization techniques such as batchnormalization and weight decay on the gradient ﬂow dynamics under square loss.
These ﬁndingssupport our theoretical results’ implication, which further strengthens our beliefs that the essencecomes from the square loss and our analysis can go beyond NTK regime.
The rest of this paper is arranged as follows.
Section 2 presents some preliminaries.
Main theoreticalresults are in Section 3.
The simplex label coding is discussed in Section 3.4 followed by numericalstudies in Section 4 and conclusions in Section 5.
Technical proofs and details of the numericalstudies can be found in the Appendix.
Preliminaries
Notation For a function f : ⌦ !
R , let k f k
=
sup
x
2 ⌦ | f ( x ) | and k f k p = (
R ⌦ | f ( x ) | p d x ) 1 /p .
For a vector x , k x k p denotes its p -norm, for 1  p  1 .
L p and l p are used to distinguish functionnorms and vector norms.
For two positive sequences { a n } n 2 N and { b n } n 2 N , we write a n .
b n if 139 there exists a constant C > 0 such that a n  Cb n for all sufﬁciently large n.
We write a n ⇣ b n if 140 a n .
b n and b n .
a n .
Let [ N ] = { 1 , . .
. , N } for N 2 N , I be the indicator function, and I d be the 141 d ⇥ d identity matrix.
N ( µ, ⌃ ) represents Gaussian distribution with mean µ and covariance ⌃ .
Classiﬁcation problem settings Let P be an underlying probability measure on ⌦ ⇥ Y , where⌦ ⇢ R d is compact and Y = { 1 ,  1 } .
Let ( X, Y ) be a random variable with respect to P .
Suppose we have observations { ( x i , y i ) } ni =1 ⇢ ( ⌦ ⇥ Y ) n i.i.d.
sampled according to P .
The classiﬁcationtask is to predict the unobserved label y given a new input x 2 ⌦ .
Let ⌘ deﬁned on ⌦ denote theconditional probability, i.e., ⌘ ( x ) =
P ( y = 1 | x ) .
Let P X be the marginal distribution of P on X .
Remark 1.
In this work, we consider the ﬁxed dimension settings, i.e., d is ﬁxed.
As pointed by areviewer, [33, 34] discussed overparameterized neural networks in high dimensional settings underregression.
The study in the high dimensional settings will be pursued in the future.
According to whether labels are deterministic, there are two scenarios of interest.
If ⌘ only takesvalues from { 0 , 1 } , i.e., labels are deterministic, we call this case the separable case .
Note that inthe separable case we consider, the classes are not limited to linearly separable but can be arbitrarilycomplicated, as long as the label is deterministic.
Let ⌦ 1 = { x | ⌘ ( x ) = 1 } , ⌦ 2 = { x | ⌘ ( x ) = 0 } 154 and ⌦ =
⌦ 1 [ ⌦ 2 .
If the probability measure of { x | ⌘ ( x ) 2 (0 , 1) } is non-zero, i.e., the labels 155 contain randomness, we call this case the non-separable case .
In the separable case, we furtherassume that there exists a positive margin, i.e., dist ( ⌦ 1 , ⌦ 2 )  2  > 0 , where  is a constant, anddist ( ⌦ 1 , ⌦ 2 ) =
inf x 2 ⌦ 1 , x 0 2 ⌦ 2 k x  x 0 k 2 .
In the non-separable case, to quantify the difﬁculty of 158 classiﬁcation, we adopt the well-established Tsybakov’s noise condition [21], which measures howlarge the “difﬁcult region” is where ⌘ ( x ) ⇡ 1 / 2 .
Deﬁnition 2.1 (Tsybakov’s noise condition) .
Let  2 [0 , 1 ] .
We say P has Tsybakov noise exponent if there exists a constant C, T > 0 such that for all 0 < t < T , P X ( | 2 ⌘ ( X )  1 | < t )  C · t  .
A large value of  implies the difﬁcult region to be small.
It is expected that a larger  leads to a fasterconvergence rate of a neural network classiﬁer.
This intuition is veriﬁed for the overparameterizedneural network classiﬁer trained by square loss and ` 2 regularization.
See Section 3 for details.
The key quantity of interest is the misclassiﬁcation error, i.e., 0-1 loss.
In the population level, the 0-1loss can be written as
[Equation]
One optimal minimizer of L ( f ) is 2 ⌘  1 , and we deﬁne L ⇤ =
L (2 ⌘  1) .
In fact, as long as theclassiﬁer has the same sign with 2 ⌘ ( x )  1 for all x 2 ⌦ , it is a optimal minimizer and achieves theminimum 0-1 loss.
Neural network setup We mainly focus on the one-hidden-layer ReLU neural network family Fwith m nodes in the hidden layer, denoted by f W , a ( x ) = m  1 /
P mr =1 a r  ( W > r x ) , where x 2 ⌦ ,
W =
( W 1 , · · · , W m ) 2 R d ⇥ m is the weight matrix in the hidden layer, a = ( a 1 , · · · , a m ) > 2 R m 173 is the weight vector in the output layer,  ( z )
=
max { 0 , z } is the rectiﬁed linear unit (ReLU).
The initial values of the weights are independently generated from W r (0) ⇠ N ( 0 , ⇠ 2 I m ) , a r ⇠unif {  1 , 1 } , 8 r 2 [ m ] .
Based on the observations { ( x i , y i ) } ni =1 , the goal of training a neural network 176 is to ﬁnd a solution to
[Equation]
where l is the loss function, R is the regularization, and µ  0 is the regularization parameter.
Note in
Equation 2.2 that we only consider training the weights W .
This is because a ·  ( z ) = sign ( a ) ·  ( | a | z ) ,which allows us to reparametrize the network to have all a i ’s to be either 1 or  1 .
In this work, weconsider square loss associated with ` 2 regularization, i.e., l ( f W , a ( x i ) , y i ) =
( f W , a ( x i )  y i ) 2 and
R
[Equation]
A popular way to train the neural network is via gradient based methods.
It has been shown thatthe training process of DNNs can be characterized by the neural tangent kernel (NTK)
Asis usually assumed in the NTK literature [36, 27, 37, 38], we consider data on the unit sphere
S d  1 , i.e., k x i k 2 = 1 , 8 i 2 [ n ] , and the neural network is highly overparameterized ( m  n ) andtrained by gradient descent (GD).
For details about NTK and GD in one-hidden-layer ReLU neuralnetworks, we refer to Appendix A.
In the rest of this work, f W ( k ) , a denotes the GD-trained neuralnetwork classiﬁer under square loss associated with ` 2 regularization, where k is the iteration numbersatisfying Assumption D.1 and W ( k ) is the weight matrix after k -th iteration.
Theoretical results
In this section, we present our main theoretical results, which consist of three parts: generalizationerror, robustness, and calibration error.
Throughout the analysis, we make the following assumptionson the data and the estimation model.
Due to the page limit, we move the technical speciﬁcation ofthe assumptions to Appendix D, with detailed discussions.
Data assumptions We assume the ground-truth ⌘ ( x ) to be well-behaved (Assumption D.2).
Optionally, the marginal density X is assumed to be upper bounded (Assumptions D.4) or both upperand lower bounded (Assumptions D.5).
Assumptions D.4 and D.5 are standard in classical analysisof classiﬁcation in statistics literature [21, 22], which covers a large class of density functions.
Model assumptions We require the ReLU neural network to be sufﬁciently overparameterized(with a ﬁnite width), and imposes conditions on the learning rate and iteration number (Assumption
D.1)
; similar settings have been adopted by [36, 38].
We also assume that the solution to (2.2) iswell-behaved, i.e., the complexity of the neural network estimator generated by the GD training iscontrolled (Assumption D.3).
3.1 Generalization error bound
In classiﬁcation, the generalization error is typically referred to as the misclassiﬁcation error, whichcan be quantiﬁed by L ( f ) deﬁned in Equation 2.1.
In the non-separable case, the excess risk, deﬁnedby L ( f )  L ⇤ , is used to evaluate the quality of a classiﬁer f , where L ⇤ =
L (2 ⌘  1) , which is theminimum 0-1 loss, as stated after Equation 2.1.
Theorem 3.1 states that the overparameterized neuralnetwork with GD and ` 2 regularization can achieve a small excess risk in the non-separable case.
Theorem 3.1 (Excess risk in the non-separable case) .
Suppose Assumptions D.1, D.2, and D.4 hold.
Assume the conditional probability ⌘ ( x ) satisﬁes Tsybakov’s noise condition with component  .
Letµ ⇣ n d  1 2 d  1 .
Then
[Equation]
From Theorem 3.1, we can see that as  becomes larger, the convergence rate becomes faster, whichis intuitively true.
Generalization error bounds in this setting is scarce.
To the best of the authors’knowledge, [27] is the closest work (the labels are randomly ﬂipped), where the bound is in the orderof O P (1 / p n ) .
Our bound is faster, especially with larger  .
It is known that the optimal convergencerate under Assumptions D.2 and D.4 is O P ( n  d (  +1)
d  +4 d  2 ) [21].
The differences between (3.1) and theoptimal convergence rate is the extra ( d  1)  in the denominator of the convergence rate in (3.1)(since n  d (  +1) (2 d  1)(  +2) =
n  d (  +1)
( d  1)  + d  +4 d  2 ).
If the conditional probability ⌘ has a bounded Lipschitzconstant, [22] showed that the convergence rate based on the plug-in kernel estimate is O P ( n
 +1  +3+ d ) ,which is slower than the rate in Equation 3.1 if d is large.
Now we turn to the separable case.
Since ⌘ only takes value from { 0 , 1 } in the separable case, ⌘ isbounded away from 1/2.
Therefore, one can trivially take  ! 1 in Equation 3.1 and obtain theconvergence rate O P ( n  d/ (2 d  1) ) .
However, this rate can be signiﬁcantly improved in the separablecase, as stated in the following theorem.
Theorem 3.2 (Generalization error in the separable case) .
Suppose Assumptions D.1, D.3, and D.5hold.
Let µ = o (1) .
There exist positive constants C 1 , C 2 such that the misclassiﬁcation rate is 0%with probability at least 1    C 1 exp(  C 2 n ) , and  can be arbitrarily small 1 by enlarging theneural network’s width.
In Theorem 3.2, the regularization parameter can take any rate that converges to zero.
In particular, µcan be zero, and the corresponding classiﬁer overﬁts the training data.
Theorem 3.2 states that theconvergence rate in the separable case is exponential, if a sufﬁciently wide neural network is applied.
This is because the observed labels are not corrupted by noise, i.e., P ( y = 1 | x ) is either one or zero.
Therefore, it is easier to classify separable data, which is intuitively true.
3.2 Robustness and calibration error
If two classes are separable with positive margin, the decision boundary is not unique.
Practitionersoften prefer the decision boundary with large margins, which are robust against possible perturbationon input points [39, 40].
The following theorem states that the square loss trained margin can belower bounded by a positive constant.
Recall that
in the separable case, ⌦ = ⌦ 1 [ ⌦ 2 , where⌦ 1 = { x | ⌘ ( x ) = 1 } and ⌦ 2 = { x | ⌘ ( x ) = 0 } .
Theorem 3.3 (Robustness in the separable case) .
Suppose the assumptions of Theorem 3.2are satisﬁed.
Let µ = o (1) .
Then there exist positive constants C, C 1 , C 2 such thatmin x 2 D T , x 0 2 ⌦ 1 [ ⌦ 2 k x  x 0 k 2  C, and the misclassiﬁcation rate is 0% with probability at least 244   C 1 exp(  C 2 n ) for all n , where D T is the decision boundary, and  is as in Theorem 3.2.
Theorem 3.3 states that the square loss trained margin is robust in the sense that the predicted label willnot change in case of any noise whose l 2 norm is smaller than C. Since k x  x 0 k 1  p d k x  x 0 k 2 , 247 Theorem 3.3 also indicates l 1 robustness.
To the best of our knowledge, similar theoretical robustnessguarantee has not been provided for any other loss functions.
The most relevant work is [41], buttheir result is not on the population and doesn’t apply to ReLU networks trained via GD.
In the non-separable case, ⌘ ( x ) varies within (0,1) and practitioners may not only want a classiﬁer with a small excess risk, but also want to recover the underlying conditional probability ⌘ .
Therefore, square loss is naturally preferred since it treats the classiﬁcation problem as a regressionproblem.
The following theorem states that, one can recover the conditional probability ⌘ by using anoverparameterized neural network with ` 2 regularization and GD training.
Theorem 3.4 (Calibration error) .
Suppose Assumptions D.1-D.4 are fulﬁlled.
Let µ ⇣ n d  1 2 d  1 .
Then
[Equation]
Theorem 3.4 states that the underlying conditional probability in the non-separable case can berecovered by ( f W ( k ) , a + 1) / 2 .
The form ( f W ( k ) , a + 1) / 2 is to account for the {  1 , 1 } label coding.
258 Under {0,1} coding, the estimator would be f
W ( k ) , a itself.
The L 1 consistency doesn’t hold forcross-entropy trained neural networks, due to the form of the optimal solution log( ⌘ 1  ⌘ )
Withlimited capacity, the network’s conﬁdence prediction is bounded away from 0 and 1.
In practice, wewant to control the complexity of the neural network thus it is usually the case that  f W ( k ) , a  1 < C 262 for some constant C .
Hence, following the results in [12], direct calculation shows that such aneural network with limited complexity cannot accurately estimate ⌘ ( x ) when ⌘ ( x ) >
e C 1+ e C or⌘ ( x ) <
1 1+ e C , which makes the calibration error under the cross-entropy loss always bounded awayfrom zero.
However, square loss does not have such a problem.
Notice that the calibration error bound in Theorem 3.4 does not depend on the Tsybakov’s noisecondition, and is slower than the excess risk.
This is because a small calibration error is muchstronger than a small excess risk, since the former requires the conditional probability estimation tobe uniformly accurate, not just matching the sign of ⌘  1 / 2 .
To be more speciﬁc, a good estimated b ⌘can always lead to a low risk plug-in classiﬁer b f =b ⌘  1 , but not vice versa.
Remark 2 (Technical challenge) .
Despite the similar forms of regression and classiﬁcation usingsquare loss, most of the regression analysis techniques cannot be directly applied to the classiﬁcationproblem, even if the supports of two classes are non-separable.
Moreover, it is clear that classiﬁcationproblems in the separable case are completely different with regression problems.
3.3 Transition from separable to non-separable
The general non-separable case and the special separable case can be connected via Gaussian noiseinjection.
In practice, data augmentation is an effective way to improve robustness and the simplestway is Gaussian noise injection [42].
In this section, we only consider it as an auxiliary tool fortheoretical analysis purpose and not for actual robust training.
Injecting Gaussian noise amountsto convoluting a Gaussian distribution N (0 ,  2 I d ) to the marginal distribution P X , which enlargesboth ⌦ 1 and ⌦ 2 to R d and a unique decision boundary D  can be induced.
Correspondingly, the“noisy" conditional probability, denoted as e ⌘  , is also smoothed to be continuous on R d .
As  ! 0 ,
k e ⌘   ⌘ k 1 ! 0 on ⌦ 1 and ⌦ 2 and the limiting e ⌘ 0 is a piecewise constant function with discontinuityat the induced decision boundary.
Lemma 3.5 (Tsybakov’s noise condition under Gaussian noises) .
Let the margin be 2  > 0 , thenoise be N (0 ,  2 I d ) .
Then there exist some constants T, C > 0 such that
[Equation]
Theorem 3.6 (Exponential convergence rate) .
Suppose the classes are separable with margin 2  > 0 .
No matter how complicated ⌦ 1 [ ⌦ 2 are, the excess risk of the over parameterized neural networkclassiﬁer satisfying Assumptions D.1 and D.4 has the rate O P ( e  n  / 7 ) .
Although Theorem 3.6 is similar to that in [43], the techniques are different.
Our analysis is directlyfrom the general non-separable case (Theorem 3.1), by the addition of auxiliary noises (Lemma 3.5).
The proof of Theorem 3.6 involves taking the auxiliary noise to zero, e.g., v = v n ⇣ 1 / p n .
Theexponential convergence rate is a direct outcome of Lemma 3.5 and Theorem 3.1.
Note that ourexponential convergence rate is much faster than existing ones under the similar separable setting[26, 28, 29], which are all polynomial with n , e.g., O P (1 / p n ) .
Remark 3 (Extension on NTK) .
Although our analysis only concerns overparameterized one-hiddenlayer ReLU neural networks, it can potentially apply to other types of neural networks.
Recently,it has been shown that overparameterized multi-layer networks correspond to the Laplace kernel[44, 45].
As long as the trained neural networks can approximate the classiﬁer induced by the NTK,our results can be naturally extended.
Remark 4.
Theorems 3.4 and 3.6 share the same gist: the overparameterized neural networkclassiﬁers can have exponential convergence rate when data are separable with positive margin.
Theresult of Theorem 3.6 is weaker than that of Theorem 3.4, but with milder conditions.
Technically,
Theorem 3.6 also bridges the non-separable case and separable case through auxiliary noise injection.
3.4 Multiclass Classiﬁcation
In binary classiﬁcation, the labels are usually encoded as  1 and 1 .
When there are K > 2 classes,the default label coding is one-hot.
However, it is empirically observed that this vanilla square lossstruggles when the number of classes are large, for which scaling tricks have been proposed [4, 7].
Another popular coding scheme is the simplex coding [11], which takes maximally separated Kpoints on the sphere as label features.
When K = 2 , this reduces to the typical  1 , 1 coding.
Manyadvantages of the simplex coding have been discussed, including its relationship with cross-entropyloss and supervised contrastive learning [10, 30, 17, 46].
Even though the choice of label coding mayhave signiﬁcant impact on optimization, it does not signiﬁcantly affect our analysis of square lossfrom a theoretical perspective.
In this work, we adopt the simplex coding 2 .
Given the label coding, one can easily generalize the theoretical development in Section 3 byemploying the following objective function
[Equation]
where f W , a : ⌦ 7!
R K , and y i =
( y i, 1 , ..., y i,K ) > is the label of i -th observation.
Next proposition states a relationship between the simplex coding and the conditional probability,which is a generalization of the binary classiﬁcation case.
Proposition 3.7 (Conditional probability) .
Let f
⇤ : ⌦ !
R K minimize the mean square error
E X ( f ⇤ ( X )  v y ) 2 , where v y is the simplex coding vector of label y .
Then
[Equation]
Based on the relationship established by Proposition 3.7, one can show that the calibration error canbe small in the multi-class classiﬁcation case.
Unlike the softmax function when using cross entropy,the estimated conditional probability using square loss is not guaranteed to be within 0 and 1.
Thiswill cause issues for adversarial attacks; see more discussions in Appendix G.2.
Numerical experiments
Although our theoretical results are for overparameterized neural network in the NTK regime, weexpect our conclusions to generalize to practical network architectures.
The focus of this section isnot on improving the state-of-the-art performance for deep classiﬁers, but to illustrate the differencebetween cross-entropy and square loss.
We provide experiment results on both synthetic and realdata, to support our theoretical ﬁndings and illustrate the practical beneﬁts of square loss in trainingoverparameterized DNN classiﬁers.
Compared with cross-entropy, the square loss has comparablegeneralization performance, but with stronger robustness and smaller calibration error.
4.1 Synthetic data
We consider the square loss based, cross-entropy based overparameterized neural networks (ONN)with ` 2 regularization, and the cross-entropy based ONN without ` 2 regularization, denoted as
SL-ONN + ` 2 , CE-ONN + ` 2 , and CE-ONN
respectively.
For the separable case, we considertwo separated classes with spiral curve like supports.
Figure 1 shows one instance of the testmisclassiﬁcation rate and decision boundaries attained by three methods, where we can see that
SL-ONN + ` 2 has a smaller test misclassiﬁcation rate and a much smoother decision boundary.
Inparticular, in the red region, where the training data are sparse, SL-ONN + ` 2 ﬁts the correct datadistribution best.
For the non-separable case, we consider the calibration performance of SL-ONN+ ` 2 and CE-ONN + ` 2 , where the classiﬁers are denoted by b f l 2 and b f ce , respectively.
The resultspresented in Figure G.9 in the Appendix shows that b f l 2 has the smaller mean and standard deviationthan b f ce .
More implementation details are in Appendix G.1.
4.2 Real data
To make a fair comparison, we adopt popular architectures, ResNet [47] and Wide ResNet [48] andevaluate them on the CIFAR image classiﬁcation datasets [49], with only the training loss functionchanged, from cross-entropy (CE) to square loss with simplex coding (SL).
Further, we don’t employany large scale hyper-parameter tuning and all the parameters are kept as default except for thelearning rate (lr) and batch size (bs), where we are choosing from the better of (lr=0.01, bs=32) and(lr=0.1, bs=128).
Each experiment setting is replicated 5 times and we report the average performancefollowed by its standard deviation in the parenthesis.
(lr=0.01, bs=32) works better for the most casesexcept for square loss trained WRN-16-10 on CIFAR-100.
More experiment details and additionalresults can be found in Appendix G.2.
Generalization In both CIFAR-10 and CIFAR-100, the performance of cross-entropy and squareloss with simplex coding are quite comparable, as observed in [4].
Cross-entropy tends to performslightly better for ResNet, especially on CIFAR-100 with an advantage of less than 1%.
There is amore signiﬁcant gap with Wide ResNet where square loss outperforms cross-entropy by more than1% on both CIFAR-10 and CIFAR-100.
The details can be found in Table 1.
Adversarial robustness Naturally trained deep classiﬁers are found to be adversarially vulnerable and adversarial attacks provide a powerful tool to evaluate classiﬁcation robustness.
For ourexperiment, we consider the black-box Gaussian noise attack, the classic white-box PGD attack[50] and the state-of-the-art AutoAttack [51], with attack strength level 2/255, 4/255, 8/255 in l
[Table]
norm.
AutoAttack contains both white-box and black-box attacks and offers a more comprehensiveevaluation of adversarial robustness.
The Gaussian noises results are presented in Table G.3 in the
Appendix.
At different noise levels, square loss consistently outperforms cross-entropy, especiallyfor WRN-16-10, with around 2-4% accuracy improvement.
More details can be found in Appendix
G.2.
The PGD and AutoAttack results are reported in Table 1.
Even though classiﬁers trained withsquare loss is far away from adversarially robust, it consistently gives signiﬁcantly higher adversarialaccuracy.
The same margin can be carried over to standard adversarial training as well.
Table 2 listsresults from standard PGD adversarial training with CE and SL.
By substituting cross-entropy loss tosquare loss, the robust accuracy increased around 3% while maintaining higher clean accuracy.
One thing to notice is that when constructing white-box attacks, square loss will not work well sinceit doesn’t directly reﬂect the classiﬁcation accuracy.
More speciﬁcally, for a correctly classiﬁed image( x , y ) , maximizing the square loss may result in linear scaling of the classiﬁer f ( x ) , which doesn’tchange the predicted class (see Appendix G.2 for more discussion).
To this end, we consider a specialattack for classiﬁers trained by square loss by maximizing the cosine similarity between f ( x ) andv y .
We call this angle attack and also utilize it for the PGD adversarial training paired with squareloss in Table 2.
In our experiments, this special attack rarely outperforms the standard PGD withcross-entropy and the reported PGD accuracy are from the latter settings.
This property of squareloss may be an advantage in defending adversarial attacks.
[Table]
Model calibration
The predicted class probabilities for square loss can be obtained from Equation 3.2.
Expected calibration error (ECE) measures the absolute difference between predictedconﬁdence and actual accuracy.
Deep classiﬁers are usually over-conﬁdent [52].
Using ResNet asan example, we report the typical reliability diagram in Figure 2.
On CIFAR-10 with ResNet-18,the average ECE for cross-entropy is 0.028 (0.002) while that for square loss is 0.0097 (0.001).
On
CIFAR-100 with ResNet-50, the average ECE for cross-entropy is 0.094 (0.005) while that for squareloss is 0.068 (0.005).
Square loss results are much more calibrated with signiﬁcantly smaller ECE.
Conclusions
Classiﬁcation problems are ubiquitous in deep learning.
As a fundamental problem, any progressin classiﬁcation can potentially beneﬁt numerous relevant tasks.
Despite its lack of popularity inpractice, square loss has many advantages that can be easily overlooked.
Through both theoreticalanalysis and empirical studies, we identify several ideal properties of using square loss in trainingneural network classiﬁers, including provable fast convergence rates, strong robustness, and smallcalibration error.
We encourage readers to try square loss in your own application scenarios.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes] See Section 3.
(b) Did you describe the limitations of your work?
[Yes] Our theoretical results apply
to shallow overparameterized neural networks in the NTK regime.
However, our
ﬁndings can still hold beyond the assumptions as we conducted extensive experiments
on practical neural networks in Section 4.
(c) Did you discuss any potential negative societal impacts of your work?
[No] We do not
foresee any potential negative societal impact.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Appendix
D.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix E.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[No] The real data
we use is the publicly available.
The code is proprietary.
However, we have provided
implementation details of our experiments in Appendix G.1 and Appendix G.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] They are all discussed in detail either in the main submission or in
appendix.
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] See Fig.
G.3 and Fig.
G.
(d) Did you include the total amount of compute and the type of resources used (e.g.,
type of GPUs, internal cluster, or cloud provider)?
[No] Our experiments are not
computationally intensive.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 4.
(b) Did you mention the license of the assets?
[Yes] See Section 4.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
Does not apply
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A] Does not apply
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A] Does not apply
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A] Does not apply
(b) Did you describe any potential participant risks
, with links to Institutional Review
Board
[Table]
IRB) approvals, if applicable
[Table]
[N/
[Table]
] Does not apply
[Table]
Did you
[Table]
participants
[Table]
spent
[Table]
participant compensation
[Table]
not apply