Are All Losses Created Equal: A Neural Collapse Perspective
Abstract
While cross entropy (CE) is the most commonly used loss function to train deep neural networks for classiﬁcation tasks, many alternative losses have been developed to obtain better empirical performance.
Among them, which one is the best to use is still a mystery, because there seem to be multiple factors affecting the answer, such as properties of the dataset, the choice of network architecture, and so on.
This paper studies the choice of loss function by examining the last-layer features of deep networks, drawing inspiration from a recent line work showing that the global optimal solution of CE and mean-square-error (MSE) losses exhibits a Neural Collapse ( NC ) phenomenon.
That is, for sufﬁciently large networks trained until convergence, (i) all features of the same class collapse to the corresponding class mean and (ii) the means associated with different classes are in a conﬁguration where their pairwise distances are all equal and maximized.
We extend such results and show through global solution and landscape analyses that a broad family of loss functions including commonly used label smoothing (LS) and focal loss (FL) exhibits NC .
Hence, all relevant losses (i.e., CE, LS,
FL, MSE) produce equivalent features on training data.
In particular, based on the unconstrained feature model assumption, we provide either the global landscape analysis for LS loss or the local landscape analysis for FL loss and show that the (only!)
global minimizers are NC solutions, while all other critical points are strict saddles whose Hessian exhibit negative curvature directions either in the global scope for LS loss or in the local scope for FL loss near the optimal solution.
The experiments further show that NC features obtained from all relevant losses (i e., CE, LS, FL, MSE) lead to largely identical performance on test data as well, provided that the network is sufﬁciently large and trained until convergence.
The source code is available at https://github.com/jinxinzhou/nc_loss .
Introduction
Loss function is an indispensable component in the training of deep neural networks (DNNs).
While cross-entropy (CE) loss is one of the most popular choices for classiﬁcation tasks, studies over the past few years have suggested many improved versions of CE that bring better empirical performance.
Some notable examples include label smoothing (LS) [1] where one-hot label is replaced by a smoothed label, focal loss (FL) [2] which puts more emphasis on hard misclassiﬁed samples and reduces the relative loss on the already well-classiﬁed samples, and so on.
Aside from CE and its variants, the mean squared error (MSE) loss which was typically used for regression tasks is recently demonstrated to have a competitive performance when compared to CE for classiﬁcation tasks [3].
Despite the existence of many loss functions there is however a lack of consensus as to which one is the best to use, and the answer seems to depend on multiple factors such as properties of the dataset, choice of network architecture, and so on [4].
In this work, we aim to understand the effect of loss function in classiﬁcation tasks from the perspective of characterizing the last-layer features and classiﬁer of a DNN trained under different losses.
Our study is motivated by a sequence of recent work that identify an intriguing Neural Collapse ( NC ) phenomenon in trained networks, which refers to the following properties of the last-layer features and classiﬁer:
(i) Variability Collapse : all features of the same class collapse to the corresponding class mean.
(ii) Convergence to Simplex ETF : the means associated with different classes are in a Simplex Equiangular Tight Frame (ETF) conﬁguration where their pairwise distances are all equal and maximized.
(iii) Convergence to Self-duality : the class means are ideally aligned with the last-layer linear classiﬁers.
(iv) Simple Decision Rule : the last-layer classiﬁer is equivalent to a Nearest Class-Center decision rule.
This NC phenomena is ﬁrst discovered by Papyan et al.
[5, 6] under canonical classiﬁcation problems trained with the CE loss.
Following with the CE loss, Han et al.
[7] recently reported that DNNs trained with MSE loss for classiﬁcation problems also exhibit similar NC phenomena.
These results imply that deep networks are essentially learning maximally separable features between classes, and a max-margin classiﬁer in the last layer upon these learned features.
The intriguing empirical observation motivated a surge of theoretical investigation [7–22], mostly under a simpliﬁed unconstrained feature model [10] or layer-peeled model [12] that treats the last-layer features of each samples before the ﬁnal classiﬁer as free optimization variables.
Under the simpliﬁed unconstrained feature model, it has been proved that the NC solution is the only global optimal solution for the CE and
MSE losses which are also proved to have benign global landscape, explaining why the global NC solution can be obtained.
Contributions.
While previous work provide thorough analysis for NC under CE and MSE losses, the theoretical analysis beyond CE and MSE losses is still limited, and their work only focus on one speciﬁc loss without a general format.
In this paper, we consider a broad family of loss functions that includes CE and some other popular loss functions such as LS and FL as special cases.
Under the unconstrained feature model , we theoretically demonstrate in Section 3 that the NC solution is the only global optimal solution to the family of loss functions.
Moreover, we provide a global landscape analysis, showing that the LS loss function is a strict saddle function and FL loss function is a local strict saddle function [23–25].
A (local) strict saddle function is a function for which every critical point is either a global solution or a strict saddle point with negative curvature (locally).
Hence, our result suggests that any optimizer can escape strict saddle points and converge to the global solution responding to NC for LS and FL.
As far as we know, this paper is the ﬁrst work that conducts global optimal solution and benign optimization landscape analysis beyond the scope of
CE and MSE losses.
Our theoretical results explained above have important implications for understanding the role of loss function in training DNNs for classiﬁcation tasks.
Because all losses lead to NC solutions, their corresponding features are equivalent up to a rotation of the feature space.
In other words, our analysis provides a theoretical justiﬁcation for the following claim:
All losses (i.e., CE, LS, FL, MSE) lead to largely identical features on training data by large DNNs and sufﬁciently many iterations.
We also provide an experimental veriﬁcation of this claim through experiments in Section 4.1.
While NC reveals that all losses are equivalent at training time, it does not have a direct implication for the features associated with test data as well as the generalization performance [26].
In particular, a recent work [27] shows empirically that NC does not occur for the features associated with test data.
Nonetheless, we show through empirical evidence that for large DNNs, NC on training data well predicts the test performance.
In particular, our empirical study in Section 4.2 shows the following:
All losses (CE, LS, FL, MSE) lead to largely identical performance on test data by large DNNsand
sufﬁciently
many iterations.
Our conclusion that all losses are created equal appears to go against existing evidence on the advantages of some losses over the others.
Here we emphasize that our conclusion has an important premise, namely the neural network has sufﬁcient approximation power and the training is performed for sufﬁciently many iterations.
Hence, our conclusion implies that the better performance with particular choices of loss functions comes as a result that the training does not produce a globally optimal (i.e., NC ) solution.
In such cases different losses lead to different solutions on the training data, and correspondingly different performance on test data.
Such an understanding may provide important practical guidance on what loss to choose in different cases (e.g., different model sizes and different training time budgets), as well as for the design of new and better losses in the future.
We note that our conclusion is based on natural accuracy, rather than model transferability or robustness, which is worth additional efforts to exploit and is left as future work.
The Problem Setup
A typical deep neural network Ψ( · ) : R D (cid:55)→ R K consists of a multi-layer nonlinear compositional feature mapping Φ( · ) : R D (cid:55)→ R d and a linear classiﬁer ( W , b ) , which can be generally expressed as
[Equation]
where we use θ to represent the network parameters in the feature mapping and W ∈ R K × d and b ∈ R K to represent the linear classiﬁer’s weight and bias, respectively.
Therefore, all the network parameters are the set of Θ = { θ , W , b } .
For the input x , the output of the feature mapping Φ θ ( x ) is usually termed as the representation or feature learned from the network.
With an appropriate loss function, the parameters Θ of the whole network are optimized to learn the underlying relation from the input sample x to their corresponding target y so that the output of the network Ψ Θ ( x ) approximates the corresponding target, i.e. Ψ Θ ( x ) ≈ y in term of the expectation over a distribution D of input-output data pairs ( x , y ) .
While it is hard to get access to the groundtruth distribution D in most cases, one can approximate the distribution D through sampling enough data pairs i.i.d.
from D .
In this paper, we study the multi-class balanced classiﬁcation tasks with K class and n samples per class, where we use the one-hot vector y k ∈ R K with unity only in k -th entry (1 ≤ k ≤ K ) to denote the label of the i -th sample x k,i ∈
R D in the k -th class.
We then learn the parameters Θ via minimizing the following empirical risk over the total N = nK training samples
[Equation]
where λ > 0 is the regularization parameter (a.k.a., the weight decay parameter 2 ) and
L (Ψ Θ ( x k,i ) , y k ) is a predeﬁned loss function that appropriately measures the difference between the output Ψ Θ ( x k,i ) and the target y k .
Some common loss functions used for training deep neural networks will be speciﬁed in the next section.
2.1 Commonly Used Training Losses
In this subsection, we ﬁrst present four common loss functions for classiﬁcation task.
To simplify the notation, let z
= W Φ θ ( x ) + b denote the network’s output (“logit”)
vector for the input x .
Assume z belongs to the k -th class.
Also let y smooth k
=
(1 − α )
y k
+ α K 1 K denote the smoothed targets of k -th class, where 0 ≤ α < 1 and 1 K ∈ R K is a vector with all entries equal to one.
We will use z (cid:96) , y k,(cid:96) and y smooth (cid:96) to denote the (cid:96) -th entry of z , y k and y smooth k , respectively, where y smooth k,
[Equation]
Cross entropy (CE) is perhaps the most common loss for multi-class classiﬁcation in deep learning.
It measures the distance between the target distribution y k and the network output distribution obtained by applying the softmax function on z , resulting in the following expression
[Equation]
Focal loss (FL) [2] is ﬁrst proposed to deal with the extreme foreground-background class imbalance in dense object detection, which adaptively focuses less on the well-classiﬁed samples.
Recent work [28, 29] reports that focal loss also improves calibration and automatically forms curriculum learning in multi-class classiﬁcation setting.
Letting γ ≥ 0 denote the tunable focusing parameter, the focal loss can be expressed as:
[Equation]
Label smoothing (LS) [1] replaces the hard targets in CE with smoothed targets y smooth k obtained from mixing the original targets y k with a uniform distribution over all entries 1 K .
Experiments in [30, 31] ﬁnd that classiﬁcation models trained with label smoothing have better calibration and generalization.
Denoting by 0 ≤ α ≤ 1 the tunable smoothing parameter, the label smoothing loss function can be formulated as:
[Equation]
When α = 0 , the above label smoothing loss reduces to the CE loss.
Mean square error (MSE) is often used for regression but not classiﬁcation task.
The recent work [3] shows that classiﬁcation networks trained with MSE loss achieve on par performance compared to those trained with the CE loss.
Throughout our paper, we use the rescaled MSE version [3]:
[Equation]
where κ > 0 and β > 0 are hyperparameters.
2.2 Problem Formulation Based on Unconstrained Feature Models
Because of the interaction between a large number of nonlinear layers in the feature mapping Φ θ , it is tremendously challenging to analyze the optimization of deep neural networks.
To simplify the difﬁculty of deep neural network analysis, a series of recent works of theoretically studying NC phenomenon use a so-called unconstrained feature model (or layer-peeled model in [12]) which treats the last-layer features as free optimization variables h
= Φ( x ) ∈ R d .
The reason behind the unconstrained feature model is that modern highly overparameterized deep networks are able to approximate any continuous functions [32–35] and the characterization of NC are only related with the last layer features.
We adopt the same approach and study the effects of different training losses on the last-layer representations of the network under the unconstrained feature model.
For convenient, let us denote
[Equation]
H
[Equation]
H n ] ∈ R
[Equation]
N
[Equation]
and
Y
[Equation]
where w k is the k -th row vector of W , all the features in the k -th class are denoted as H i :=
[ h 1 ,i · · · h K,i ] ∈ R d × K and h k,i is the feature of the i -th sample in the k -th class, and Y k :=
[ y k · · · y k ] ∈ R K × n for all k = 1 , 2 , · · · , K and i = 1 , 2 , · · · , n .
Based on the unconstrained feature model, we consider a slight variant of (2), given by
[Equation]
where λ W , λ H , λ b > 0 are the penalty parameters for W , H , and b , respectively.
By viewing the last-layer feature H as a free optimization variable, the simpliﬁed objective function (7) consider the weight decay about W and H , which is slightly different from practice that the weight decay is imposed on all the network parameters Θ as shown in (2).
Nonetheless, the underlying rationale is that the weight decay on Θ implicitly penalizes the energy of the features (i.e., (cid:107) H (cid:107) F ) [16].
As NC phenomena for the learned features and classiﬁers is ﬁrst discovered for neural networks trained with the CE loss [5], the CE loss has been mostly studied through the above simpliﬁed unconstrained feature model [8,9,11,12,16] to understand the NC phenomena.
The work [7,10,14, 22] also studied the MSE loss, but the analysis there shows the solutions of the learned features and classiﬁers depend crucially on the bias term, while for CE loss with or without the bias term have no effect on the learned features and classiﬁers under the unconstrained feature model.
The other losses such as focal loss and label smoothing have been less studied, though they are widely employed in practice to obtain better performance.
This will be the subject of next section.
Understanding Loss Functions Through Unconstrained Features Model
In this section, we study the effect of different loss functions through the unconstrained features model.
We will ﬁrst present a contrastive property for general loss function L GL in Deﬁnition 1.
We will then study the global optimality conditions in terms of the learned features and classiﬁers as well as geometric properties for (7) with such a general loss function L GL .
3.1 A Contrastive Property for the Loss Functions
In this paper, we aim to provide a uniﬁed analysis for different loss functions.
Towards that goal, we ﬁrst present some common properties behind the CE, FL and FL to motivate the discussion.
Taking
CE as an example, we can lower bound it by
[Equation]
1 + ( K − 1) exp (cid:18)(cid:80) j
(cid:54) =
k ( z j − z k )
K −(cid:19)(cid:19) =
φ CE  (cid:88)
j (cid:54) =
k ( z j − z k ) 
 (8) where φ CE ( t ) =
log (cid:16)
+ ( K − 1) exp (cid:16) tK − 1 (cid:17)(cid:17) , and the inequality achieves equality when z j =
z j (cid:48) for all j, j (cid:48)
(cid:54) =
k .
This requirement is reasonable because the commonly used losses treat all the outputs except for the k -th output z identically.
Since φ CE is an increasing function, minimizing the CE loss L CE ( z , y k ) is equivalent to maximizing ( K − 1)
z k − (cid:80) j
(cid:54) =
K z j , which contrasts the k -th output z k simultaneously to all the other outputs z j for all j (cid:54) =
k .
Thus, we call (8) as a contrastive property .
Maximizing ( K − 1)
z k − (cid:80) j
(cid:54) =
K z j would lead to a positive (and relatively large) z k and negative (and relatively small) z j .
In particular, within the unit sphere (cid:107) z (cid:107)
=
1 , ( K − 1)
z k − (cid:80) j
(cid:54) =
K z j achieves its maximizer when z k = (cid:113) K − 1 K and z j = − (cid:113) 1 K ( K − 1) for all j (cid:54)
=
k , which satisﬁes the requirement z j = z
j (cid:48) for all j, j (cid:48)
(cid:54) =
k .
Thus, z k = (cid:113) K − 1 K and z j = − (cid:113) 1 K ( K − 1) is also the global minimizer for φ CE within the unit sphere (cid:107) z (cid:107) 2 = 1 .
As the global minimizer is unique for each class, it encourages intra-class compactness.
On the other hand, the minimizers to different classes are maximally distant, promoting inter-class separability.
Motivated by the above discussion, we now introduce the following properties for a general loss function L GL ( z , y k ) .
Deﬁnition 1 (Contrastive property) .
We say a loss function L GL ( z , y k ) satisﬁes the contrastive property if there exists a function φ such that L GL ( z , y k ) can be lower bounded by
[Equation]
where the equality holds only when z j = z (cid:48)
j for all j, j (cid:48) (cid:54) =
k .
Moreover, φ ( t ) satisﬁes t ∗ =
arg min t φ ( t ) + c | t | is unique for any c > 0 , and t ∗
[Equation]
[Equation]
In the appendix, we show that CE, FL and LS all satisfy this property.
The motivation for (9) follows from the above discussion.
In particular, (9) achieves equality when all the outputs except for the k -th one are identical, which holds for common loss functions since those outputs are treated identically.
In (10), c is a constant related with the weight decay penalty parameters.
By (9), we can ﬁnd the global minimizer for L GL ( z , y k ) by minimizing the right hand side since the equality in (9) is achievable.
Thus, the requirement of a unique minimizer (10) ensures a unique minimizer for the regularized L GL ( z , y k ) .
This condition can be easily satisﬁed.
For example, φ CE ( t ) deﬁned in (8) for the CE loss is an increasing and strictly convex function and thus has unique minimizer for φ CE ( t )
+ c | t | .
Along the same line, we require a negative minimizer t (cid:63) to ensure that the minimizer for the regularized L GL ( z , y k ) has k -th entry being its largest entry, which is required to ensure correct prediction since the largest entry predicts the class membership.
Therefore, such a condition is generally satisﬁed by the common losses.
For example, φ CE ( t ) is an increasing function and thus must have a non-positive minimizer for φ CE ( t )
+ c | t | .
Finally, we note that the MSE loss is not included since it has different form than others and thus the analysis will be different.
But as mentioned above, the MSE loss has been studied in [7,10,14,22].
3.2 Landscape Analysis for the Unconstrained Features Model
We now study the global optimality conditions in terms of the learned features and classiﬁers as well as geometric properties for the training problem (7) with the general loss function L GL satisfying the above contrastive property.
Theorem 1 (Global Optimality Condition) .
Assume that the number of classes K is smaller than the feature dimension d , i.e., K < d , and the dataset is balanced for each class, n = n 1 = · · · = n K .
Then any global minimizer ( W (cid:63) , H (cid:63) , b (cid:63) ) of f in (7) with a loss function L GL satisfying the contrastive property in Deﬁnition 1 has following properties:
[Equation]
and b (cid:63) =
b (cid:63) 1 ,
[Equation] [Equation]
, i
[Equation] [Equation]
and h (cid:63)i := 1 K
K (cid:88)
j = h (cid:63)j,i = 0 , ∀ i ∈ [ n ] , where either b (cid:63) = 0 or λ b = 0 , and the matrix W (cid:63) (cid:62) is in the form of K -simplex ETF structure (see appendix for the formal deﬁnition) in the sense that
[Equation]
Its proof is given in Appendix C. At a high level, we lower bound the general loss function based on the contrastive property (9) and then check the equality conditions hold for the lower bounds.
While similar strategy has been used for CE loss [12,13,16], our proof is different from previous work in terms of dealing with the nuclear norm and checking the structures of the minimizer per sample and class, enabling the global optimality analysis for general loss functions.
Theorem 1 implies that for all the loss functions (e.g., CE, LS, and FL) satisfying the contrastive property, they share similar global solutions with NC property in the learned features and classiﬁers.
While Theorem 1 shows that NC features and classiﬁers are the only global minimizers to (7), it is not obvious whether local search algorithms (such as gradient descent) can efﬁciently ﬁnd these benign global solutions.
The reason is that the training problem (7) is nonconvex due to the bilinear form between W and H .
To address this challenge, we use the recent advances on the geometric analysis for nonconvex optimization [23–25,36] to guarantee that the global solutions of (7) can be efﬁciently achieved by iterative algorithms.
Towards that goal, we ﬁrst present the following general results concerning the global landscape for (7).
Theorem 2 (Benign Landscape) .
Assume that the feature dimension d is larger than the number of classes K , i.e., d > K .
Also assume L ( z , y ) is a convex function in terms of z .
Then the objective function f in (7) is a strict saddle function with no spurious local minimum.
That is, any of its critical point is either a global minimizer, or it is a strict saddle point whose Hessian has a strictly negative eigenvalue.
This result is similar to [16, Theorem 3.2] which studies the particular CE loss.
Though the result in [16] is about the CE loss, we checked its proof and it only uses convexity and smoothness and thus the result can be applied more generally for any smooth convex loss function L ( · , y ) .
So we omit the proof of Theorem 2.
We note that the geometric analysis is also closed related to nonconvex low-rank matrix problems [37–43] with the Burer-Moneirto factorization approach [44] if one views
W and H as two factors of a matrix Z = W H .
We refer to [16] for more discussions about the connections and differences.
We now exploit Theorem 2 for the label smoothing and focal loss.
In the supplementary material, we show that LS is a convex function.
Thus, the following result establishes global optimization landscape for the training problem (7) with such a loss.
Corollary 1 (Benign Landscape with LS) .
Assume that the feature dimension d is larger than the number of classes K , i.e., d > K .
Then the objective function f in (7) with LS loss L LS is a strict saddle function with no spurious local minimum.
Unlike LS, focal loss L FL ( z , y k ) is convex only in a local region rather than the entire space.
For example, we can show that L FL ( z , y k ) is convex within the region Ω = { z ∈ R K : exp( z k ) / (cid:80)
Kj =1 exp( z j ) ≥ 0 . 21 } .
The set Ω contains a relative large region including the global minimizer which has the value exp( z k ) (cid:80)
Kj =1 exp( z j ) approaching 1.
Thus, we obtain a benign local landscape for the training problem (7) with FL.
Corollary 2 (Benign Landscape with FL) .
Assume that the feature dimension d is larger than the number of classes K , i.e., d > K .
Then the objective function f in (7) with FL loss L FL has a benign local landscape: f is a strict saddle function with no spurious local minimum within the region { ( W , H , b ) :
W h
[Equation]
While Corollary 2 only provides a local benign landscape for the FL, we observe from experiments that gradient descent with random initialization always converges to a global solution with NC properties for (7).
So we expect the training problem in (7) with FL loss has benign landscape in a much larger region.
One direction is to show L FL ( · , y k ) is locally convex in a much larger region Ω , but we leave the thorough investigation to future work.
Noting that the CE and MSE losses are also convex, these results imply that (stochastic) gradient descent with random initialization [23,36] almost surely ﬁnds the global solutions of the training problem in (7) with different training losses.
This together with Theorem 1 implies that for different losses, gradient descent will always learn similar features and classiﬁers—those that exhibit the NC phenomenon.
Experiments
We conduct experiments with practical network architectures on standard image classiﬁcation datasets to study the effect of different loss functions.
First, Section 4.1 provides results to show that the NC phenomena are not restricted to networks trained via the CE and MSE losses.
Rather, there is a family of loss functions, and for the purpose of illustration we pick FL and LS as two prominent special cases, that exhibit the same NC phenomena.
Such results verify our theoretical results in Section 3.
To demonstrate the implication of NC for test performance, we present experimental results in Section 4.2 with a varying number of training iterations and a varying width of networks, showing that all losses with NC global optimality have similar performance on the test dataset when the network is sufﬁciently large and trained long enough.
Before presenting the experiment results, we ﬁrst introduce our experimental setup, including datasets, network architectures, training procedure, and metrics for measuring NC .
Setup of Loss Function, Network Architecture, Dataset, and Training We focus on the CE, FL,
LS and MSE loss functions for which we use γ = 3 for FL, α = 0 .
1 for LS, and κ = 1 and β =for MSE, except otherwise speciﬁced.
We train a WideResNet50 network [45] on CIFAR10 and
CIFAR100 datasets [46] and a WideResNet18 network on miniImageNet [47] with various widths and number of iterations for image classiﬁcation using these four different losses.
3 To examine the effect of model size, we experiment with four versions of WideResNet, denoted as WideResNet- X , where X ∈ { 0 . 25 , 0 . 5 , 1 , 2 } is a multiplier on the width of its corresponding standard WideResNet.
Due to the page limit, we put all results on CIFAR100 and miniImageNet in the Appendix.
We use standard preprocessing such that images are normalized (channel-wise) by their mean and standard deviation, as well as standard data augmentation.
For optimization, we use SGD with momentum 0 .
9 and an initial learning rate 0 .
1 decayed by a factor of 0.1 at 37 and 57 of the total number of iterations.
Following [28], the norm of gradient is clipped at 2 which can improve performance for all losses.
For CIFAR10 and miniImageNet, the weight decay is set to 5 × 10 4 for all conﬁgurations with all losses.
For CIFAR100, the weight decay is ﬁne-tuned to achieve best accuracy for every conﬁguration and loss.
Three NC Metrics NC 1 - NC 3 during Network Training
We use the same three metrics NC 1NC 3 for the last-layer features and classiﬁer as in [5,16,22] to measure the ﬁrst three NC properties in Section 1.
Before we describe these three metrics, let us denote the global mean h G and k -th class mean h k of last-layer features { h k,i } as
[Equation]
Within-class variability collapse is measured by N C 1 which depicts the relative magnitude of the within-class covariance Σ
[Equation]
t. the between-class covariance Σ B = 1 K (cid:80)
Kk =1 (cid:0)
h k − h G (cid:1) (cid:0)
[Equation]
k − h G (cid:1) (cid:62) ∈ R
d × d of the last-layer features as following:
[Equation]
where Σ † B is the pseudo inverse of Σ B .
Convergence to simplex ETF is measured by N C 2 which reﬂects the (cid:96) 2 distance between the normalized simplex ETF and the normalized W W (cid:62) as following:
[Equation]
where W ∈ R K × d is the weight matrix of learned classiﬁer.
Convergence to self-duality is measured by N C 3 which calculates the (cid:96) 2 distance between the normalized simplex ETF and the normalized W H as following:
[Equation]
where H = (cid:2) h 1 − h G · · · h K − h G (cid:3)
∈ R d × K is the centered class-mean matrix.
4.1 Prevalence of NC Across Varying Training Losses
We show that all loss functions lead to NC solutions during the terminal phase of training.
The results on CIFAR10 using WideResNet50-2 and different loss functions is provided in Figure 1.
We consistently observe that all three NC metrics across different losses converge to a small value as training progresses.
This supports our theoretical results in Section 3 that the last-layer features learned under different losses are always maximally linearly separable and perfectly aligned with the linear classiﬁer, and the features and the weight of linear classiﬁer learned by different losses are almost equivalent up to a rotation and a scale of the feature space.
The evolution of three NC metrics across different losses on CIFAR100 is in Appendix A.2.
4.2 All Losses Lead to Largely Identical Performance
We show that all loss functions have largely identical performance once the training procedure converges to the NC global optimality.
In Figure 2, we plot the evolution of the training accuracy,
[Figure] [Figure]
validation accuracy and test accuracy as training progresses, where all losses are optimized on the same WideResNet50-2 architecture and CIFAR10 for 800 epochs.
To reduce the randomness, we average the results from 3 different random seeds per width-iteration conﬁguration, and the test accuracy is reported based on the model with best accuracy on validation set, where we organize the validation set by holding out 10 percent data from the training set.
The results consistently show that for all cases the training accuracy converges to one hundred percent (reaching to terminal phase), and the validation accuracy and test accuracy are largely the same, as long as the network is trained longer enough and converges to the NC global solution.
While previous work advocates the advantage of some losses over other others, our experiments show that when conditions between dataset and model allow for SGD to ﬁnd an NC solution, all losses we tested produced indistinguishable results.
In Figure 3, we plot the average test accuracy of different losses under different pairs of width and iterations.
We consistently observe three phenomenon.
First, with a ﬁxed number of iterations, increasing the width of network improves the test accuracy for all losses.
This is because the wider networks (more over-parameterized) are more powerful to ﬁt the underlying mapping from input data to the targets.
Second, with a ﬁxed width of network, increasing the number of iterations improves the test accuracy for all losses.
This is because the longer optimization leads the last-layer features and the linear classifer closer to the
NC global solutions.
Finally, while there are some unignorable difference between different losses in some width-iteration conﬁgurations, the results consistently show that all losses lead to largely identical performance when the network is sufﬁciently large and trained long enough to achieve a global NC solution (e.g. width=2 and epochs=800).
Conclusion
In this work we provided a theoretical study to extend the scope of NC , a curious phenomenon associated with last-layer features and classiﬁer weight of a classiﬁcation network, from networks trained with particular losses (i.e., CE and MSE) to those trained via a broad family of loss functions including the popular LS and FL as special cases.
Our theory not only establishes NC as the only global solutions, but also shows a benign optimization landscape that explains why NC solutions
[Figure]
are easy to obtain in practice.
Such results readily suggest that all relevant losses (i.e., CE, MSE, LS, and FL) produce entirely equivalent features on the training data.
Although NC is an optimization phenomenon pertaining to training data only, we found through experiments that all relevant losses (i.e., CE, MSE, LS, and FL) lead to very similar test performance as well.
Such a result may come as a surprise to the common belief that some losses are intrinsically better than the others, and clarify some mystery on how different losses affect the performance.
The family of loss functions considered in this paper by no means is inclusive of all possible loss functions that lead to NC .
There are many other popular loss functions, such as center loss [48], large-margin softmax (L-Softmax) loss [49] and many of its variants [50–52], which are all designed with the intuition of encouraging intra-class compactness and inter-class separability between learned features.
In addition, many generalized versions of the cross-entropy loss such as those for robust learning under label noise [53–55] and long-tail distribution [56,57] may have similar property as the vanilla cross-entropy loss.
We conjecture that many of them provably produce NC solutions under unconstrained feature models, while leave a formal justiﬁcation to future work.
Beyond losses for classiﬁcation task, NC may also arise with popular losses used in metric learning [58,59] evidenced by recent study [60].
This means that the observations from this paper, namely all losses lead to largely the same test performance, may apply for all such losses as well.
Loss functions that do not lead to NC .
While the study in this paper covers many of the most commonly used loss functions for classiﬁcation tasks, we note that there are alternative choices in the literature which do not induce NC features.
Many of such losses such as [60–63] are particularly designed to discourage variability collapse and learn diverse features, which are shown to beneﬁt model transferability [64] and robustness
Acknowledgment
JZ and ZZ acknowledge support from NSF grants CCF-2240708 and CCF-2241298.
XL and QQ acknowledge support from U-M START & PODS grants, NSF CAREER CCF 2143904, NSF CCF 2212066, NSF CCF and ONR N00014-22-1-2529 grants.
[-• Case 1: If d 1 > d 2 > · · · > d n and z i (cid:54) = 
-]
[-z (cid:48) (cid:62) , where D (cid:48) ∈ R ( n − 1) × ( n − 1) and z (cid:
-]
SL acknowledges support from
NSF NRT-HDR Award 1922658 and Alzheimer’s Association grant AARG- NTF-21-848627.
[-Therefore, the global minimizer is achieved when x = 
-]
[-Therefore, the global minimizer is achieved when x = 
-]
[-(a) (cid:107) w (cid:63) (cid:107) 2 = (cid:13)(cid:13) w (cid:63) 
-]
[-(cid:13)(cid:13) 
-]
[-(cid:13)(cid:13) w (cid:63) 
-]
[-From the (35), we know that ¯ Z (cid:63) 
-]
[-=
-]
[-Thus, η (cid:48) ( p i ) ≥ 
-]