U NDERSTANDING S QUARE L OSS IN T RAINING O VERPARAMETRIZED N EURAL N ETWORK C LASSIFIERS
Anonymous authors
Paper under double-blind review
A BSTRACT
Deep learning has achieved many breakthroughs in modern classiﬁcation tasks.
Numerous architectures have been proposed for different data structures but when it comes to the loss function, the cross-entropy loss is the predominant choice.
Recently, several alternative losses have seen revived interests for deep classiﬁers.
In particular, empirical evidence seems to promote square loss but a theoretical justiﬁcation is still lacking.
In this work, we contribute to the theoretical understanding of square loss in classiﬁcation by systematically investigating how it performs for overparametrized neural networks in the neural tangent kernel (NTK) regime.
Interesting properties regarding the generalization error, robustness, and calibration error are revealed.
We consider two cases, according to whether classes are separable or not.
In the general non-separable case, fast convergence rate is established for both misclassiﬁcation rate and calibration error.
When classes are separable, the misclassiﬁcation rate improves to be exponentially fast.
Further, the resulting margin is proven to be lower bounded away from zero, providing theoretical guarantees for robustness.
We expect our ﬁndings to hold beyond the NTK regime and translate to practical settings.
To this end, we conduct extensive empirical studies on practical neural networks, demonstrating the effectiveness of square loss in both synthetic low-dimensional data and real image data.
Comparing to crossentropy, square loss has comparable generalization error but noticeable advantages in robustness and model calibration.
INTRODUCTION
The pursuit of better classiﬁers has fueled the progress of machine learning and deep learning research.
The abundance of benchmark image datasets, e.g., MNIST, CIFAR, ImageNet, etc., provides test ﬁelds for all kinds of new classiﬁcation models, especially those based on deep neural networks (DNN).
With the introduction of CNN, ResNets, and transformers, DNN classiﬁers are constantly improving and catching up to the human-level performance.
In contrast to the active innovations in model architecture, the training objective remains largely stagnant, with cross-entropy loss being the default choice.
Despite its popularity, cross-entropy has been shown to be problematic in some applications.
Among others, Yu et al.
(2020) argued that features learned from cross-entropy lack interpretability and proposed a new loss aiming for maximum coding rate reduction.
Pang et al.
(2019) linked the use of cross-entropy to adversarial vulnerability and proposed a new classiﬁcation loss based on latent space matching.
Guo et al.
(2017) discovered that the conﬁdence of most DNN classiﬁers trained with cross-entropy is not well-calibrated.
Recently, several alternative losses have seen revived interests for deep classiﬁers.
In particular, many existing works have presented empirical evidence promoting the use of square loss over crossentropy.
Hui & Belkin (2020) conducted large-scale experiments comparing the two and found that square loss tends to perform better in natural language processing related tasks while cross-entropy usually yields slightly better accuracy in image classiﬁcation.
Similar comparisons are also made in Demirkaya et al.
Kornblith et al.
compared a variety of loss functions and output layer regularization strategies on the accuracy and out-of-distribution robustness, and found that square loss has greater class separation and better out-of-distribution robustness.
In comparison to the empirical investigation, theoretical understanding of square loss in training deep learning classiﬁers is still lacking.
Through our lens, square loss has its uniqueness among classic classiﬁcation losses, and we argue that it has great potentials for modern classiﬁcation tasks.
Below we list our motivations and reasons why.
Explicit feature modeling Deep learning’s success can be largely attributed to its superior ability as feature extractors.
For classiﬁcation, the ideal features should be separated between classes and concentrated within classes.
However, when optimizing cross-entropy loss, it’s not clear what the learned features should look like (Yu et al., 2020).
In comparison, square loss uses the label codings (one-hot, simplex etc.) as features, which can be modeled explicitly to control class separations.
Model Calibration
An ideal classiﬁer should not only give the correct class prediction, but also with the correct conﬁdence.
Calibration error measures the closeness of the predicted conﬁdence to the underlying conditional probability ⌘ .
Using square loss in classiﬁcation can be essentially viewed as regression where it treats discrete labels as continuous code vectors.
It can be shown that the optimal classiﬁer under square loss is 2 ⌘  1 , linear with the ground truth.
This distinguishing property allows it to easily recover ⌘ .
In comparison, the optimal classiﬁers under the hinge loss and cross-entropy are sign (2 ⌘  1) and log( ⌘ 1  ⌘ ) , respectively.
Therefore, hinge loss doesn’t provide reliable information on the prediction conﬁdence, and cross-entropy can be problematic when ⌘ is close to 0 or 1 (Zhang, 2004).
Hence, in terms of model calibration, square loss is a natural choice.
Connections to popular approaches Mixup (Zhang et al., 2017) is a popular data augmentation technique where augmented data are constructed via convex combinations of inputs and their labels.
Like in square loss, mixup treats labels as continuous and is shown to improve the generalization of DNN classiﬁers.
In knowledge distillation (Hinton et al., 2015), where a student classiﬁer is trying to learn from a trained teacher, Menon et al. (2021) proved that the “optimal” teacher with the ground truth conditional probabilities provides the lowest variance in student learning.
Since classiﬁers trained using square loss is a natural consistent estimator of ⌘ , one can argue that it is a better teacher.
In supervised contrastive learning (Khosla et al., 2020), the optimal features are the same as those from square loss with simplex label coding (Graf et al., 2021)
(details in Section 4).
Despite its lack of popularity in practice, square loss has many advantages that can be easily overlooked.
In this work, we systematically investigate from a statistical estimation perspective, the properties of deep learning classiﬁers trained using square loss.
The neural networks in our analysis are required to be sufﬁciently overparametrized in the neural tangent kernel (NTK) regime.
Even though this restricts the implication of our results, it is a necessary ﬁrst step towards a deeper understanding.
In summary, our main contributions are:
• Generalization error bound: We consider two cases, according to whether classes are separable or not.
In the general non-separable case, we adopt the classical binary classiﬁcation setting with smooth conditional probability.
Fast rate of convergence is established for overparametrized neural network classiﬁers with Tsybakov’s noise condition.
If two classes are separable with positive margins, we show that overparametrized neural network classiﬁers can provably reach zero misclassiﬁcation error with probability exponentially tending to one.
To the best of our knowledge, this is the ﬁrst such result for separable but not linear separable classes.
Furthermore, we bridge these two cases and offer a uniﬁed view by considering auxiliary random noise injection.
• Robustness (margin property): When two classes are separable, the decision boundary is not unique and large-margin classiﬁers are preferred.
In the separable case, we further show that the decision boundary of overparametrized neural network classiﬁers trained by square loss cannot be too close to the data support and the resulting margin is lower bounded away from zero, providing theoretical guarantees for robustness.
• Calibration error:
We show that classiﬁers trained using square loss are inherently well-calibrated, i.e., the trained classiﬁer provides consistent estimation of the ground-truth conditional probability in L 1 norm.
Such property doesn’t hold for cross-entropy.
• Empirical evaluation: We corroborate our theoretical ﬁndings with empirical experiments in both synthetic low-dimensional data and real image data.
Comparing to cross-entropy, square loss has comparable generalization error but noticeable advantages in robustness and model calibration.
This work contributes towards the theoretical understanding of deep classiﬁers, from an estimation point of view, which has been a classic topic in statistics literature.
Among others, Mammen &
Tsybakov (1999) established the optimal convergence rate for 0-1 loss excess risk when the decision boundary is smooth.
Zhang (2004); Bartlett et al.
(2006) extended the analysis to various surrogate losses.
Audibert & Tsybakov (2007); Kohler & Krzyzak (2007) studied the convergence rates for plug-in classiﬁers from local averaging estimators.
Steinwart et al.
(2007) investigated the convergence rate for support vector machine using Gaussian kernels.
We build on and extend classic results to neural networks in the NTK regime.
Comparing to existing works on deep learning classiﬁcation, e.g., Kim et al.
(2018) derived fast convergence rates of ReLU DNN classiﬁers that minimize the empirical hinge loss, our results incorporate the training algorithm and apply to trained classiﬁers.
We require the neural network to be overparametrized, which has been extensively studied recently, under the umbrella term NTK.
Most such results are in the regression setting with a handful of exceptions.
Ji & Telgarsky (2019) showed that only polylogarithmic width is sufﬁcient for gradient descent to overﬁt the training data using logistic loss.
Hu et al.
(2020) proved generalization error bound for regularized NTK in classiﬁcation.
Cao & Gu (2019; 2020) provided optimization and generalization guarantees for overparametrized network trained with cross-entropy.
In comparison, our results are sharper in the sense that we take the ground truth data assumptions into consideration.
This allows a faster convergence rate, especially when the classes are separable, where the exponential convergence rate is attainable.
The NTK framework greatly reduces the technical difﬁculty for our theoretical analysis.
However, our results are mainly due to properties of the square loss itself and we expect them to hold for a wide range of classiﬁers.
There are other works investigating the use of square loss for training (deep) classiﬁers.
Han et al.
(2021) uncovered that the “neural collapse” phenomenon also occurs under square loss where the last-layer features eventually collapse to their simplex-style class-means.
Muthukumar et al.
compared classiﬁcation and regression tasks in the overparameterized linear model with Gaussian features, illustrating different roles and properties of loss functions used at the training and testing phases.
Poggio & Liao (2019) made interesting observations on effects of popular regularization techniques such as batch normalization and weight decay on the gradient ﬂow dynamics under square loss.
These ﬁndings support our theoretical results’ implication, which further strengthens our beliefs that the essence comes from the square loss and our analysis can go beyond NTK regime.
The rest of this paper is arranged as follows.
Section 2 presents some preliminaries.
Main theoretical results are in Section 3.
The simplex label coding is discussed in Section 4 followed by numerical studies in Section 5 and conclusions in Section 6.
Technical proofs and details of the numerical studies can be found in the Appendix.
P RELIMINARIES
Notation For a function f : ⌦ !
R , let k f k
=
sup
x
2 ⌦ | f ( x ) | and k f k p = (
R ⌦ | f ( x ) | p d x ) 1 /p .
For a vector x , k x k p denotes its p -norm, for 1  p  1 .
L p and l p are used to distinguish function norms and vector norms.
For two positive sequences { a n } n 2 N and { b n } n 2 N , we write a n .
b n if there exists a constant C > 0 such that a n  Cb n for all sufﬁciently large n .
We write a n ⇣ b n if a n .
b n and b n .
a n .
Let [ N ] = { 1 , . .
. , N } for N 2 N , I be the indicator function, and I d be the d ⇥ d identity matrix.
N ( µ, ⌃ ) represents Gaussian distribution with mean µ and covariance ⌃ .
Classiﬁcation problem settings Let P be an underlying probability measure on ⌦ ⇥ Y , where ⌦ ⇢ R d is compact and Y = { 1 ,  1 } .
Let ( X, Y ) be a random variable with respect to P .
Suppose we have observations { ( x i , y i ) } ni =1 ⇢ ( ⌦ ⇥ Y ) n i.i.d.
sampled according to P .
The classiﬁcation task is to predict the unobserved label y given a new input x 2 ⌦ .
Let ⌘ deﬁned on ⌦ denote the conditional probability, i.e., ⌘ ( x ) =
P ( y = 1 | x ) .
Let P X be the marginal distribution of P on X .
The key quantity of interest is the misclassiﬁcation error, i.e., 0-1 loss. In the population level, the 0-1 loss can be written as
[Equation]
where the expectation is taken with respect to the probability measure P .
Clearly, an optimal classiﬁer with the minimal 0-1 loss is 2 ⌘  1 .
According to whether labels are deterministic, there are two scenarios of interest.
If ⌘ only takes values from { 0 , 1 } , i.e., labels are deterministic, we call this case the separable case 1 . Let ⌦ 1 = { x | ⌘ ( x ) = 1 } , ⌦ 2 = { x | ⌘ ( x ) = 0 } and ⌦ =
⌦ 1 [ ⌦ 2 .
If the probability measure of { x | ⌘ ( x ) 2 (0 , 1) } is non-zero, i.e., the labels contain randomness, we call this case the non-separable case .
In the separable case, we further assume that there exists a positive margin, i.e., dist ( ⌦ 1 , ⌦ 2 )  2  > 0 , where  is a constant, and dist ( ⌦ 1 , ⌦ 2 ) =
inf x 2 ⌦ 1 , x 0 2 ⌦ 2 k x  x 0 k 2 .
In the non-separable case, to quantify the difﬁculty of classiﬁcation, we adopt the well-established Tsybakov’s noise condition (Audibert & Tsybakov, 2007), which measures how large the “difﬁcult region” is where ⌘ ( x ) ⇡ 1 / 2 .
Deﬁnition 2.1 (Tsybakov’s noise condition) .
Let  2 [0 , 1 ] .
We say P has Tsybakov noise exponent  if there exists a constant C, T > 0 such that for all 0 < t < T , P X ( | 2 ⌘ ( X )  1 | < t )  C · t  .
A large value of  implies the difﬁcult region to be small.
It is expected that a larger  leads to a faster convergence rate of a neural network classiﬁer.
This intuition is veriﬁed for the overparametrized neural network classiﬁer trained by square loss and ` 2 regularization.
See Section 3 for more details.
Neural network setup We mainly focus on the one-hidden-layer ReLU neural network family F with m nodes in the hidden layer, denoted by
[Equation]
where x 2 ⌦ , W =
( W 1 , · · · , W m ) 2 R d ⇥ m is the weight matrix in the hidden layer, a = ( a 1 , · · · , a m ) > 2 R m is the weight vector in the output layer,  ( z ) =
max { 0 , z } is the rectiﬁed linear unit (ReLU).
The initial values of the weights are independently generated from
[Equation]
Based on the observations { ( x i , y i ) } ni =1 , the goal of training a neural network is to ﬁnd a solution to
[Equation]
where l is the loss function, R is the regularization, and µ  0 is the regularization parameter.
Note in Equation 2.2 that we only consider training the weights W .
This is because a ·  ( z ) = sign ( a ) ·  ( | a | z ) , which allows us to reparametrize the network to have all a i ’s to be either 1 or  1 .
In this work, we consider square loss associated with ` 2 regularization, i.e., l ( f W , a ( x i ) , y i ) =
( f W , a ( x i )  y i ) 2 and R ( W , a )
=
k W k 22 .
A popular way to train the neural network is via gradient based methods.
It has been shown that the training process of DNNs can be characterized by the neural tangent kernel (NTK)
(Jacot et al., 2018).
As is usually assumed in the NTK literature (Arora et al., 2019;
Hu et al., 2020; Bietti &
Mairal, 2019; Hu et al., 2021), we consider data on the unit sphere S d  1 , i.e., k x i k 2 = 1 , 8 i 2 [ n ] , and the neural network is highly overparametrized ( m  n ) and trained by gradient descent (GD).
For details about NTK and GD in one-hidden-layer ReLU neural networks, we refer to Appendix A.
In the rest of this work, we use f W ( k ) , a to denote the GD-trained neural network classiﬁer under square loss associated with ` 2 regularization, where k is the iteration number satisfying Assumption
D.1 and W ( k ) is the weight matrix after k -th iteration.
T HEORETICAL R ESULTS
In this section, we present our main theoretical results.
Throughout the analysis, we assume that the overparametrized neural network f W , a and the training process via GD satisfy Assumption D.1 (see
Appendix D), which essentially requires the neural network to be sufﬁciently overparametrized (with a ﬁnite width), and imposes conditions on the learning rate and iteration number.
Our theoretical results consist of three parts: generalization error, robustness, and calibration error.
3. G ENERALIZATION ERROR BOUND
In classiﬁcation, the generalization error is typically referred to as the misclassiﬁcation error, which can be quantiﬁed by L ( f ) deﬁned in Equation 2.1.
In the non-separable case, the excess risk, deﬁned by L ( f )  L ⇤ , is used to evaluate the quality of a classiﬁer f , where L ⇤ =
L (2 ⌘  1) , which minimizes the 0-1 loss.
The following theorem states that the overparametrized neural network with
GD and ` 2 regularization can achieve a small excess risk in the non-separable case.
Theorem 3.1 (Excess risk in the non-separable case) .
Suppose Assumptions D.1, D.2, and D. hold.
Assume the conditional probability ⌘ ( x ) satisﬁes Tsybakov’s noise condition with component  .
Let µ ⇣ n d  1 2 d  1 .
Then
[Equation]
From Theorem 3.1, we can see that as  becomes larger, the convergence rate becomes faster, which is intuitively true.
Generalization error bounds in this setting is scarce.
To the best of the authors’ knowledge, Hu et al.
(2020) is the closest work (the labels are randomly ﬂipped), where the bound is in the order of O P (1 / p n ) .
Our bound is faster, especially with larger  .
It is known that the optimal convergence rate under Assumptions D.2 and D.4 is O P ( n  d (  +1)
d  +4 d  2 ) (Audibert & Tsybakov, 2007).
The differences between Equation 3.1 and the optimal convergence rate is that there is an extra ( d  1)  in the denominator of the convergence rate in Equation 3.1 (since n  d (  +1) (2 d  1)(  +2) = n  d (  +1) ( d  1)  + d  +4 d  2 ).
If the conditional probability ⌘ has a bounded Lipschitz constant, then Kohler & Krzyzak (2007) showed that the convergence rate based on the plug-in kernel estimate is O P ( n
 +1  +3+ d ) , which is slower than the rate in Equation 3.1 if d is large.
Now we turn to the separable case.
Since ⌘ only takes value from { 0 , 1 } in the separable case, ⌘ is bounded away from 1/2.
Therefore, one can trivially take  ! 1 in Equation 3.1 and obtain the convergence rate O P ( n  d/ (2 d  1) ) .
However, this rate can be signiﬁcantly improved in the separable case, as stated in the following theorem.
Theorem 3.2 (Generalization error in the separable case) .
Suppose Assumptions D.1, D.3, and D. hold.
Let µ = o (1) .
There exist positive constants C 1 , C 2 such that the misclassiﬁcation rate is 0% with probability at least 1    C 1 exp(  C 2 n ) , and  can be arbitrarily small 2 by enlarging the neural network’s width.
Note that in Theorem 3.2, the regularization parameter can take any rate that converges to zero.
In particular, µ can be zero, and the corresponding classiﬁer overﬁts the training data.
Theorem 3.
states that the convergence rate in the separable case is exponential, if a sufﬁciently wide neural network is applied.
This is because the observed labels are not corrupted by noise, i.e., P ( y = 1 | x ) is either one or zero.
Therefore, it is easier to classify separable data, which is intuitively true.
3. R OBUSTNESS AND CALIBRATION ERROR
If two classes are separable with positive margin, the decision boundary is not unique.
Practitioners often prefer the decision boundary with large margins, which are robust against possible perturbation on input points (Elsayed et al., 2018;
Ding et al., 2018).
The following theorem states that the square loss trained margin can be lower bounded by a positive constant. Recall that
in the separable case, ⌦ = ⌦ 1 [ ⌦ 2 , where ⌦ 1 = { x | ⌘ ( x ) = 1 } and ⌦ 2 = { x | ⌘ ( x ) = 0 } .
Theorem 3.3 (Robustness in the separable case) .
Suppose the assumptions of Theorem 3.2 are satisﬁed.
Let µ = o (1) .
Then there exist positive constants C, C 1 , C 2 such that for all n ,
[Equation]
and the misclassiﬁcation rate is 0% with probability at least 1    C 1 exp(  C [Equation] n ) , where D T is the decision boundary, and  is as in Theorem 3.2.
Remark 1. Note that k x  x 0 k 1  p d k x  x 0 k 2 , thus Theorem 3.3 also indicates l 1 robustness.
In the non-separable case, ⌘ ( x ) varies within (0,1) and practitioners may not only want a classiﬁer with a small excess risk, but also want to recover the underlying conditional probability ⌘ .
Therefore, square loss is naturally preferred since it treats the classiﬁcation problem as a regression problem.
The following theorem states that, one can recover the conditional probability ⌘ by using an overparametrized neural network with ` 2 regularization and GD training.
Theorem 3.4 (Calibration error) .
Suppose the conditions in Theorem 3.1, Assumption D.3 and D. are fulﬁlled.
Let µ ⇣ n d  1 2 d  1 .
Then
[Equation]
Theorem 3.4 states that the underlying conditional probability in the non-separable case can be recovered by ( f W ( k ) , a + 1) / 2 .
The form ( f W ( k ) , a + 1) / 2 is to account for the {  1 , 1 } label coding.
Under { 0,1 } coding, the estimator would be f W ( k ) , a itself.
The L 1 consistency doesn’t hold for cross-entropy trained neural networks, due to the form of the optimal solution log( ⌘ 1  ⌘ ) .
With limited capacity, the network’s conﬁdence prediction is bounded away from 0 and 1 (Zhang, 2004).
In practice, we want to control the complexity of the neural network thus it is usually the case that k f W ( k ) , a k 1 < C for some constant C .
Hence, it cannot accurately estimate ⌘ ( x ) when ⌘ ( x ) >
e C 1+ e C or ⌘ ( x ) <
1 1+ e C , which makes the calibration error under the cross-entropy loss always bounded away from zero.
However, square loss does not have such a problem. Notice that the calibration error bound in Theorem 3.4 does not depend on the Tsybakov’s noise condition, and is slower than the excess risk. This is because, a small calibration error is much stronger than a small excess risk, since the former requires the conditional probability estimation to be uniformly accurate, not just matching the sign of ⌘ ( x )  1 / 2 . To be more speciﬁc, a good estimated b ⌘ can always result in a low risk plug-in classiﬁer b f ( x )
= b ⌘ ( x )  1 , but not vice versa. Remark 2 (Technical challenge) . Despite the similar forms of regression and classiﬁcation using square loss, most of the regression analysis techniques cannot be directly applied to the classiﬁcation problem, even if the supports of two classes are non-separable.
Moreover, it is clear that classiﬁcation problems in the separable case are completely different with regression problems.
Remark 3 (Extension on NTK) .
Although our analysis only concerns overparametrized onehidden-layer ReLU neural networks, it can potentially apply to other types of neural networks in the NTK regime.
Recently, it has been shown that overparametrized multi-layer networks correspond to the Laplace kernel (Geifman et al., 2020; Chen & Xu, 2020).
As long as the trained neural networks can approximate the classiﬁer induced by the NTK, our results can be naturally extended.
3. T RANSITION FROM SEPARABLE TO NON - SEPARABLE
The general non-separable case and the special separable case can be connected via Gaussian noise injection.
In practice, data augmentation is an effective way to improve robustness and the simplest way is Gaussian noise injection (He et al., 2019).
In this section, we only consider it as an auxiliary tool for theoretical analysis purpose and not for actual robust training.
Injecting Gaussian noise amounts to convoluting a Gaussian distribution N (0 ,  2 I d ) to the marginal distribution P X , which enlarges both ⌦ 1 and ⌦ 2 to R d and a unique decision boundary D  can be induced.
Correspondingly, the “noisy” conditional probability, denoted as e ⌘  , is also smoothed to be continuous on R d .
As  ! 0 ,
k e ⌘   ⌘ k 1 ! 0 on ⌦ 1 and ⌦ 2 and the limiting e ⌘ 0 is a piecewise constant function with discontinuity at the induced decision boundary.
Lemma 3.5 (Tsybakov’s noise condition under Gaussian noises) .
Let the margin be 2  > 0 , the noise be N (0 ,  2 I d ) .
Then there exist some constants T, C > 0 such that for any 0 < t < T ,
[Equation]
Theorem 3.6 (Exponential convergence rate) .
Suppose the classes are separable with margin 2  > 0 .
No matter how complicated ⌦ 1 [ ⌦ 2 are, the excess risk of the over parameterized neural network classiﬁer satisfying Assumptions D.1 and D.4 has the rate O P ( e  n  / 7 ) .
The proof of Theorem 3.6 involves taking the auxiliary noise to zero, e.g., v = v n ⇣ 1 / p n .
The exponential convergence rate is a direct outcome of Lemma 3.5 and Theorem 3.1.
Note that our exponential convergence rate is much faster than existing ones under the similar separable setting (Ji & Telgarsky, 2019; Cao & Gu, 2019; 2020), which are all polynomial with n , e.g., O P (1 / p n ) .
Remark 4.
Theorems 3.4 and 3.6 share the same gist that the over parameterized neural network classiﬁers can have exponential convergence rate when data are separable with positive margin, while the result of Theorem 3.6 is weaker than that of Theorem 3.4, but with milder conditions.
Nevertheless, Theorem 3.6 bridges the non-separable case and separable case.
M ULTICLASS C LASSIFICATION
In binary classiﬁcation, the labels are usually encoded as  1 and 1 .
When there are K > 2 classes, the default label coding is one-hot.
However, it is empirically observed that this vanilla square loss struggles when the number of classes are large, for which scaling tricks have been proposed (Hui & Belkin, 2020; Demirkaya et al., 2020).
Another popular coding scheme is the simplex coding (Mroueh et al., 2012), which takes maximally separated K points on the sphere as label features. When K = 2 , this reduces to the typical  1 , 1 coding. Many advantages of the simplex coding have been discussed, including its relationship with cross-entropy loss and supervised contrastive learning (Papyan et al., 2020; Han et al., 2021; Graf et al., 2021; Fang et al., 2021). In this work, we adopt the simplex coding.
More discussion and empirical comparison about the coding choices can be found in Appendix G.2.
Given the label coding, one can easily generalize the theoretical development in Section 3 by employing the following objective function
[Equation]
where f W , a : ⌦ 7!
R K , and y i =
( y i, 1 , ..., y i,K ) > is the label of i -th observation.
The following proposition states a relationship between the simplex coding scheme and the conditional probability.
Proposition 4.1 (Conditional probability) .
Let f
⇤ : ⌦ !
R K minimize the mean square error
E X ( f ⇤ ( X )  v y ) 2 , where v y is the simplex coding vector of label y .
Then we have
[Equation]
Unlike the softmax function when using cross entropy, the estimated conditional probability using square loss is not guaranteed to be within 0 and 1.
This will cause issues for adversarial attacks, which will be discussed in detail in Appendix G.2.
NUMERICAL STUDIES
Although our theoretical results are for overparametrized neural network in the NTK regime, we expect our conclusions to generalize to practical network architectures.
The focus of this section is not on improving the state-of-the-art performance for deep classiﬁers, but to illustrate the difference between cross-entropy and square loss.
We provide experiment results on both synthetic and real data, to support our theoretical ﬁndings and illustrate the practical beneﬁts of square loss in training overparametrized DNN classiﬁers.
Compared with cross-entropy, the square loss has comparable generalization performance, but with stronger robustness and smaller calibration error.
5. S YNTHETIC D ATA
We consider the square loss based and cross-entropy based overparametrized neural networks (ONN) with ` 2 regularization, denoted as SL-ONN + ` 2 and CE-ONN + ` 2 , respectively.
The chosen ONNs are two-hidden-layer ReLU neural networks with 500 neurons for each layer, and the parameter µ is selected via a validation set.
More implementation details are in Appendix G.1.
Separable case We consider two separated classes with spiral curve like supports.
We also present the performance of the cross-entropy based ONN without ` 2 regularization (CE-ONN).
Figureshows one instance of the test misclassiﬁcation rate and decision boundaries attained by SL-ONN + ` 2 (Left), CE-ONN + ` 2 (Center), and CE-ONN (Right).
From this example and other examples in
Appendix G.1, it can be seen that SL-ONN + ` 2 has a smaller test misclassiﬁcation rate and a much smoother decision boundary.
In particular, in the red region, where the training data are sparse,
SL-ONN + ` 2 ﬁts the correct data distribution best.
Non-separable case We consider the conditional probability ⌘ ( x ) = sin( p 2 ⇡ || x || 2 ) ,
x 2 [  1 , 1] 2 , and the calibration performance of SL-ONN + ` 2 and CE-ONN + ` 2 , where the classiﬁers are denoted by b f l 2 and b f ce , respectively.
The results are presented in Figure G.8 in the Appendix.
The error bar plot of the test calibration error shows that b f l 2 has the smaller mean and standard deviation than b f ce .
It suggests that square loss generally outperforms cross entropy in calibration.
The histogram and kernel density estimation of the test calibration errors for one case show that the pointwise calibration errors on the test points of b f l 2 are more concentrated around zero than those of b f ce .
Moreover, despite a comparable misclassiﬁcation rate with b f ce , b f l 2 has a smaller calibration error.
Figure G.8 demonstrates that SL-ONN + ` 2 recovers ⌘ much better than CE-ONN + ` 2 .
5. R EAL D ATA
To make a fair comparison, we adopt popular architectures, ResNet (He et al., 2016) and Wide
ResNet (Zagoruyko & Komodakis, 2016) and evaluate them on the CIFAR image classiﬁcation datasets, with only the training loss function changed, from cross-entropy (CE) to square loss with simplex coding (SL).
Further, we don’t employ any large scale hyper-parameter tuning and all the parameters are kept as default except for the learning rate (lr) and batch size (bs), where we are choosing from the better of (lr=0.01, bs=32) and (lr=0.1, bs=128).
Each experiment setting is replicated 5 times and we report the average performance followed by its standard deviation in the parenthesis.
(lr=0.01, bs=32) works better for the most cases except for square loss trained WRN-16on CIFAR-100.
More experiment details and additional results can be found in Appendix G.2.
Generalization In both CIFAR-10 and CIFAR-100, the performance of cross-entropy and square loss with simplex coding are quite comparable, as observed in Hui & Belkin (2020).
Cross-entropy tends to perform slightly better for ResNet, especially on CIFAR-100 with an advantage of less than 1%.
There is a more signiﬁcant gap with Wide ResNet where square loss outperforms cross-entropy by more than 1% on both CIFAR-10 and CIFAR-100.
The details can be found in Table [Table] [Table]
Adversarial robustness Normally trained deep classiﬁers are found to be adversarially vulnerable and adversarial attacks provide a powerful tool to evaluate classiﬁcation robustness.
For our experiment, we consider the black-box Gaussian noise attack, the classic white-box PGD attack (Madry et al., 2017) and the state-of-the-art AutoAttack (Croce & Hein, 2020), with attack strength level 2/255, 4/255, 8/255 in l 1 norm.
AutoAttack contains both white-box and black-box attacks and offers a more comprehensive evaluation of adversarial robustness.
The Gaussian noises results are [Table] presented in Table G.3 in the Appendix.
At different noise levels, square loss consistently outperforms cross-entropy, especially for WRN-16-10, with around 2-4% accuracy improvement.
More details can be found in Appendix G.2.
The PGD and AutoAttack results are reported in Table 1.
Even though classiﬁers trained with square loss is far away from adversarially robust, it consistently gives signiﬁcantly higher adversarial accuracy.
The same margin can be carried over to standard adversarial training as well.
Table 2 lists results from standard PGD adversarial training with CE and SL.
By substituting cross-entropy loss to square loss, the robust accuracy increased around 3% while maintaining higher clean accuracy.
One thing to notice is that when constructing white-box attacks, square loss will not work well since it doesn’t directly reﬂect the classiﬁcation accuracy.
More speciﬁcally, for a correctly classiﬁed image ( x , y ) , maximizing the square loss may result in linear scaling of the classiﬁer f ( x ) , which doesn’t change the predicted class (see Appendix G.2 for more discussion).
To this end, we consider a special attack for classiﬁers trained by square loss by maximizing the cosine similarity between f ( x ) and v y .
We call this angle attack and also utilize it for the PGD adversarial training paired with square loss in Table 2.
In our experiments, this special attack rarely outperforms the standard PGD with cross-entropy and the reported PGD accuracy are from the latter settings.
This property of square loss may be an advantage in defending adversarial attacks.
Model calibration
The predicted class probabilities for square loss can be obtained from Equation 4.1.
Expected calibration error (ECE) measures the absolute difference between predicted conﬁdence and the actual accuracy.
Deep classiﬁers are usually found to be over-conﬁdent (Vaicenavicius et al., 2019).
Using ResNet as an example, we report the typical reliability diagram in Figure 2.
On
CIFAR-10 with ResNet-18, the average ECE for cross-entropy is 0.028 (0.002) while that for square loss is 0.0097 (0.001).
On CIFAR-100 with ResNet-50, the average ECE for cross-entropy is 0.
(0.005) while that for square loss is 0.068 (0.005).
Square loss results are much more calibrated with signiﬁcantly smaller ECE.
C ONCLUSIONS
Classiﬁcation problems are ubiquitous in deep learning.
As a fundamental problem, any progress in classiﬁcation can potentially beneﬁt numerous relevant tasks.
Despite its lack of popularity in practice, square loss has many advantages that can be easily overlooked.
Through both theoretical analysis and empirical studies, we identify several ideal properties of using square loss in training neural network classiﬁers, including provable fast convergence rates, strong robustness, and small calibration error.
We encourage readers to try square loss in your own application scenarios.
Ethnics Statement We acknowledge the ICLR Code of Ethics.
This submission is mostly theoretical and the authors could not think of any potential violations of them in this submission.
Reproducibility Statement For our theoretical results, explanations of assumptions can be found in Appendix D and a complete proof of the claims can be found in the Appendix E and Appendix
F. Our experiment details can be found in Appendix G, with both data and training descriptions