Are All Losses Created Equal: A Neural Collapse Perspective
Author
Abstract
While cross entropy (CE) is the most commonly used loss function to train deepneural networks for classification tasks, many alternative losses have been developed to obtain better empirical performance.
Among them, which one is the bestto use is still a mystery, because there seem to be multiple factors affecting theanswer, such as properties of the dataset, the choice of network architecture, andso on.
This paper studies the choice of loss function by examining the last-layerfeatures of deep networks, drawing inspiration from a recent line work showing thatthe global optimal solution of CE and mean-square-error (MSE) losses exhibits a
Neural Collapse phenomenon.
That is, for sufficiently large networks trained untilconvergence, (i) all features of the same class collapse to the corresponding classmean and (ii) the means associated with different classes are in a configurationwhere their pairwise distances are all equal and maximized.
We extend such resultsand show through global solution and landscape analyses that a broad family ofloss functions including commonly used label smoothing (LS) and focal loss (FL)exhibits Neural Collapse.
Hence, all relevant losses (i.e., CE, LS, FL, MSE) produce equivalent features on training data.
In particular, based on the unconstrainedfeature model assumption, we provide either the global landscape analysis for LSloss or the local landscape analysis for FL loss and show that the (only!)
globalminimizers are neural collapse solutions, while all other critical points are strictsaddles whose Hessian exhibit negative curvature directions either in the globalscope for LS loss or in the local scope for FL loss near the optimal solution.
Theexperiments further show that Neural Collapse features obtained from all relevantlosses (i.e., CE, LS, FL, MSE) lead to largely identical performance on test data aswell, provided that the network is sufficiently large and trained until convergence.
Introduction
Loss function is an indispensable component in the training of deep neural networks (DNNs).
Whilecross-entropy (CE) loss is one of the most popular choices for classification tasks, studies over thepast few years have suggested many improved versions of CE that bring better empirical performance.
Some notable examples include label smoothing (LS) [1] where one-hot label is replaced by asmoothed label, focal loss (FL) [2] which puts more emphasis on hard misclassified samples andreduces the relative loss on the already well-classified samples, and so on.
Aside from CE and itsvariants, the mean squared error (MSE) loss which was typically used for regression tasks is recentlydemonstrated to have a competitive performance when compared to CE for classification tasks [3].
Despite the existence of many loss functions there is however a lack of consensus as to which oneis the best to use, and the answer seems to depend on multiple factors such as properties of thedataset, choice of network architecture, and so on [4].
In this work, we aim to understand the effect of loss function in classification tasks from the perspective of characterizing the last-layer featuresand classifier of a DNN trained under different losses.
Our study is motivated by a sequence of recentwork that identify an intriguing Neural Collapse ( NC ) phenomenon in trained networks, which refersto the following properties of the last-layer features and classifier:
(i) Variability Collapse : all features of the same class collapse to the corresponding class mean.(ii) Convergence to Simplex ETF : the means associated with different classes are in a Simplex Equiangular Tight Frame (ETF) configuration where their pairwise distances are all equal and maximized.(iii) Convergence to Self-duality : the class means are ideally aligned with the last-layer linear classifiers.(iv)
Simple Decision Rule : the last-layer classifier is equivalent to a Nearest Class-Center decision rule.
This NC phenomena is first discovered by Papyan et al.
[5, 6] under canonical classification problemstrained with the CE loss.
Following with the CE loss, Han et al.
[7] recently reported that DNNstrained with MSE loss for classification problems also exhibit similar NC phenomena.
Theseresults imply that deep networks are essentially learning maximally separable features betweenclasses, and a max-margin classifier in the last layer upon these learned features.
The intriguingempirical observation motivated a surge of theoretical investigation [7–22], mostly under a simplifiedunconstrained feature model [10] or layer-peeled model [12] that treats the last-layer features of eachsamples before the final classifier as free optimization variables.
Under the simplified unconstrainedfeature model, it has been proved that the NC solution is the only global optimal solution for the CEand MSE losses which are also proved to have benign global landscape, explaining why the global
NC solution can be obtained.
Contributions.
While previous work provide thorough analysis for NC under CE and MSE losses,the theoretical analysis beyond CE and MSE losses is still limited, and their work only focus on onespecific loss without a general format.
In this paper, we consider a broad family of loss functions thatincludes CE and some other popular loss functions such as LS and FL as special cases.
Under theunconstrained feature model , we theoretically demonstrate in Section 3 that the NC solution is theonly global optimal solution to the family of loss functions.
Moreover, we provide a global landscapeanalysis, showing that the LS loss function is a strict saddle function and FL loss function is a localstrict saddle function [23–25].
A (local) strict saddle function is a function for which every criticalpoint is either a global solution or a strict saddle point with negative curvature (locally).
Hence, ourresult suggests that any optimizer can escape strict saddle points and converge to the global solutionresponding to NC for LS and FL.
As far as we know, this paper is the first work that conducts globaloptimal solution and benign optimization landscape analysis beyond the scope of CE and MSE losses.
Our theoretical results explained above have important implications for understanding the role ofloss function in training DNNs for classification tasks.
Because all losses lead to NC solutions,their corresponding features are equivalent up to a rotation of the feature space.
In other words, ouranalysis provides a theoretical justification for the following claim:
All losses (i.e., CE, LS, FL, MSE) lead to largely identical features on training data by large DNNs.
We also provide an experimental verification of this claim through experiments in Section 4.1.
While NC reveals that all losses are equivalent at training time, it does not have a direct implicationfor the features associated with test data as well as the generalization performance [26].
In particular,a recent work [27] shows empirically that NC does not occur for the features associated with test data.
Nonetheless, we show through empirical evidence that for large DNNs, NC on training data wellpredicts the test performance.
In particular, our empirical study in Section 4.2 shows the following:
All losses (CE, LS, FL, MSE) lead to largely identical performance on test data by large DNNs.
Our conclusion that all losses are created equal appears to go against existing evidence on theadvantages of some losses over the others.
Here we emphasize that our conclusion has an importantpremise, namely the neural network has sufficient approximation power and the training is performedfor sufficiently many iterations.
Hence, our conclusion implies that the better performance withparticular choices of loss functions comes as a result that the training does not produce a globallyoptimal (i.e., NC ) solution.
In such cases different losses lead to different solutions on the trainingdata, and correspondingly different performance on test data.
Such an understanding may provideimportant practical guidance on what loss to choose in different cases (e.g., different model sizes and different training time budgets), as well as for the design of new and better losses in the future.
Wenote that our conclusion is based on natural accuracy, rather than model transferability or robustness,which is worth additional efforts to exploit and is left as future work.
The Problem Setup
A typical deep neural network Ψ( · ) : R D (cid:55)→ R K consists of a multi-layer nonlinear compositionalfeature mapping Φ( · ) : R D (cid:55)→ R d and a linear classifier ( W , b ) , which can be generally expressed as
[Equation]
(1) where we use θ to represent the network parameters in the feature mapping and W ∈ R K × d andb ∈ R K to represent the linear classifier’s weight and bias, respectively.
Therefore, All the networkparameters are the set of Θ = { θ , W , b } .
For the input x , the output of the feature mapping Φ θ ( x )is usually termed as the representation or feature learned from the network.
With an appropriate loss function, the parameters Θ of the whole network are optimized to learnthe underlying relation from the input samples x to their corresponding target y so that the outputsof the network Ψ Θ ( x ) approximates the corresponding target, i.e. Ψ Θ ( x ) ≈ y in term of theexpectation over a distribution D of input-output data pairs ( x , y ) .
While it is hard to get access to theground-truth distribution D in most cases, one can approximate the distribution D through samplingenough data pairs i.i.d.
from D .
In this paper, we study the multi-class balanced classification taskswith K class and n samples per class, where we use the one-hot vector y k ∈ R K with unity only ink -th entry (1 ≤ k ≤ K ) to denote the label of the i -th sample x k,i ∈
R D in the k -th class.
We thenlearn the parameters via minimizing the following empirical risk over the total N = nK trainingsamplesmin Θ N K (cid:88) k =
[Equation]
n (cid:88)
[Equation]
where λ > 0 is the regularization parameter (a.k.a., the weight decay parameter 1 ) and
L (Ψ Θ ( x k,i ) , y k ) is a predefined loss function that appropriately measures the difference betweenthe output Ψ Θ ( x k,i ) and the target y k .
Some common loss functions used for training deep neuralnetworks will be specified in the next section.
2.1 Commonly Used Training Losses
In this subsection, we first present four common loss functions for classification task.
To simplify thenotation, let z
= W Φ θ ( x ) + b denote the network’s output (“logit") vector for the input x .
Assumez belongs to the k -th class.
Also let y smooth k
=
(1 − α )
y k + αK 1 K denote the smoothed targets of k -thclass, where 0 ≤ α < 1 and 1 K ∈ R K is a vector with all entries equal to one.
We will use z ℓ , y k,ℓand y smooth ℓ to denote the ℓ -th entry of z , y k and y smooth k , respectively, where y smooth k,k = 1 − K − 1 K α andy smooth k,ℓ =
αK for k ̸ = ℓ .
Cross entropy (CE) is perhaps the most common loss for multi-class classification in deep learning.
It measures the distance between the target distribution y k and the network output distributionobtained by applying the softmax function on z , resulting in the following expression
[Equation]
Focal loss (FL) [2] is first proposed to deal with the extreme foreground-background class imbalancein dense object detection, which adaptively focuses less on the well-classified samples.
Recent work[28, 29] reports that focal loss also improves calibration and automatically forms curriculum learningin multi-class classification setting.
Letting γ ≥ 0 denote the tunable focusing parameter, the focalloss can be expressed as:
[Equation]
Label smoothing (LS) [1] replaces the hard targets in CE with smoothed targets y smooth k obtainedfrom mixing the original targets y k with a uniform distribution over all entries 1 K .
Experimentsin[30, 31] find that classification models trained with label smoothing have better calibration andgeneralization.
Denoting by 0 ≤ α ≤ 1 the tunable smoothing parameter, the label smoothing lossfunction can be formulated as:
[Equation]
When α = 0 , the above label smoothing loss reduces to the CE loss.
Mean square error (MSE) is often used for regression but not for classification task.
The recent work[3] shows that classification networks trained with MSE loss achieve on par performance comparedto those trained with the CE loss.
Through our paper, we use the rescaled MSE version [3]:
[Equation]
where κ > 0 and β > 0 are hyperparameters.
2.2 Problem Formulation Based on Unconstrained Feature Models
Because of the interaction between a large number of nonlinear layers in the feature mapping Φ θ , itis tremendously challenging to analyze the optimization of deep neural networks.
To simplify thedifficulty of deep neural network analysis, a series of recent works of theoretically studying NCphenomenon use a so-called unconstrained feature model (or layer-peeled model in [12]) whichtreats the last-layer features as free optimization variables h
= Φ( x ) ∈ R d .
The reason behindthe unconstrained feature model is that modern highly overparameterized deep networks are ableto approximate any continuous functions [32–35] and the characterization of NC are only relatedwith the last layer features.
We adopt the same approach and study the effects of different traininglosses on the last-layer representations of the network under the unconstrained feature model.
Forconvenient, let us denote
[Equation]
, and [Equation] where w k is the k -th row vector of W , all the features in the k -th class are denoted as H i :=[
h 1 ,i · · · h K,i ] ∈ R d × K and h k,i is the feature of the i -th sample in the k -th class, and Y k :
=[
y k · · · y k ] ∈ R K × n for all k = 1 , 2 , · · · , K and i = 1 , 2 , · · · , n .
Based on the unconstrainedfeature model, we consider a slight variant of (2), given bymin W , H , b
[Equation]
W , [Equation] [Equation] where λ W , λ H , λ b > 0 are the penalty parameters for W , H , and b , respectively.
By viewing the last-layer feature H as a free optimization variable, the simplified objective function(7) consider the weight decay about W and H , which is slightly different from practice that theweight decay is imposed on all the network parameters Θ as shown in (2).
Nonetheless, theunderlying rationale is that the weight decay on Θ implicitly penalizes the energy of the features (i.e.,∥ H ∥ F ) [16].
As NC phenomena for the learned features and classifiers is first discovered for neural networkstrained with the CE loss [5], the CE loss has been mostly studied through the above simplifiedunconstrained feature model [8, 9, 11, 12, 16] to understand the NC phenomena.
The work [10, 7,14, 22] also studied the MSE loss, but the analysis there shows the solutions of the learned featuresand classifiers depends crucially on the bias term, while for CE loss with or without the bias termhave no effect on the learned features and classifiers under the unconstrained feature model.
Theother losses such as focal loss and label smoothing have been less studied, though they are widelyemployed in practice to obtain better performance.
This will be the subject of next section.
Understanding Loss Functions Through Unconstrained Features Model
In this section, we study the effect of different loss functions through the unconstrained featuresmodel.
We will first present a contrastive property for general loss function L GL in Definition 1.
Wewill then study the global optimality conditions in terms of the learned features and classifiers as wellas geometric properties for (7) with such a general loss function L GL .
3.1 A Contrastive Property for the Loss Functions
In this paper, we aim to provide a unified analysis for different loss functions.
Towards that goal, wefirst present some common properties behind the CE, FL and FL to motivate the discussion.
Taking
CE as an example, we can lower bound it by
[Equation]
where ϕ CE ( t ) = log (cid:16) 1 + ( K − 1) exp (cid:16) tK − 1 (cid:17)(cid:17) , and the inequality achieves equality when z j
= z j ′for all j, j ′ ̸ = k .
This requirement is reasonable because the common used losses treat all the outputsexcept for the k -th output z identically.
Since ϕ CE is an increasing function, minimizing the CE loss
L CE ( z , y k ) is equivalent to maximizing ( K − 1)
z k − (cid:80)
j ̸ = K z j , which contrasts the k -th output z ksimultaneously to all the other outputs z j for all j ̸ = k .
Thus, we call (8) as a contrastive property .
Maximizing ( K − 1)
z k − (cid:80)
j ̸ =
K z j would lead to a positive (and relatively large) z k and negative(and relatively small) z j .
In particular, within the unit sphere ∥ z ∥ 2 = 1 , ( K − 1)
z k − (cid:80)
j ̸ = K z jachieves its maximizer when z k = (cid:113) K − 1 K and z j = − (cid:113) 1 K ( K − 1) for all j ̸ = k , which satisfies therequirement z j = z j ′ for all j, j ′ ̸
= k .
Thus, z k = (cid:113) K − 1 K and z j = − (cid:113) 1 K ( K − 1) is also the globalminimizer for ϕ CE within the unit sphere ∥
z ∥ 2 = 1 .
As the global minimizer is unique for each class,it encourages intra-class compactness.
On the other hand, the minimizers to different classes aremaximally distant, promoting inter-class separability.
Motivated by the above discussion, we now introduce the following properties for a general lossfunction L GL ( z , y k ) .
Definition 1 (Contrastive property)
We say a loss function L GL ( z , y k ) satisfies the contrastiveproperty if there exists a function ϕ such that L GL ( z , y k ) can be lower bounded by
[Equation]
where the equality holds only when z j = z ′ j for all j, j ′ ̸
= k .
Moreover, ϕ ( t ) satisfiest ∗ = arg min t ϕ ( t ) + c | t | is unique for any c > 0 , and t ∗ ≤ 0 .
In the appendix, we show that CE, FL and LS all satisfy this property.
The motivation for (9) followsfrom the above discussion.
In particular, (9) achieves equality when all the outputs except for the k -thone are identical, which holds for common loss functions since those outputs are treated identically.
In(10), c is a constant related with the weight decay penalty parameters.
By (9), we can find the globalminimizer for L GL ( z , y k ) by minimizing the right hand side since the equality in (9) is achievable.
Thus, the requirement of a unique minimizer (10) ensures a unique minimizer for the regularized
L GL ( z , y k ) .
This condition can be easily satisfied.
For example, ϕ CE ( t ) defined in (8) for the CE lossis an increasing and strictly convex function and thus has unique minimizer for ϕ CE ( t )
+ c | t | .
Alongthe same line, we require a negative minimizer t ⋆ to ensure that the minimizer for the regularized
L GL ( z , y k ) has k -th entry being its largest entry, which is required to ensure correct prediction sincethe largest entry predicts the class membership.
Therefore, such a condition is generally satisfied bythe common losses.
For example, ϕ CE ( t ) is an increasing function and thus must have a non-positiveminimizer for ϕ CE ( t )
+ c | t | .
Finally, we note that the MSE loss is not included since it has differentform than others and thus the analysis will be different.
But as mentioned above, the MSE loss hasbeen studied in [10, 7, 14, 22].
3.2 Landscape Analysis for the Unconstrained Features Model
We now study the global optimality conditions in terms of the learned features and classifiers as wellas geometric properties for the training problem (7) with the general loss function L GL satisfying theabove contrastive property.
Theorem 1 (Global Optimality Condition) Assume that the number of classes K is smaller thanthe feature dimension d , i.e., K < d , and the dataset is balanced for each class, n = n 1 = · · · =n K .
Then any global minimizer ( W ⋆ , H ⋆ , b ⋆ ) of f in (7) with a loss function L GL satisfying thecontrastive property in Definition 1 has following properties:
[Equation]
and
[Equation]
b
[Equation]
[Equation]
and h ⋆i := 1 K
K (cid:88)
j
[Equation]
h ⋆j,
i
[Equation]
[Equation] n ] [Equation] where either b ⋆ = 0 or λ b = 0 , and the matrix W ⋆ ⊤ is in the form of K -simplex ETF structure (seeappendix for the formal definition) in the sense that
[Equation]
Its proof is given in Appendix C. At a high level, we lower bound the general loss function based onthe contrastive property (9) and then check the equality conditions hold for the lower bounds.
Whilesimilar strategy has been used for CE loss [12, 13, 16], our proof is different from previous work interms of dealing with the nuclear norm and checking the structures of the minimizer per sample andclass, enabling the global optimality analysis for general loss functions.
Theorem 1 implies that forall the loss functions (e.g., CE, LS, and FL) satisfying the contrastive property, they share similarglobal solutions with NC property in the learned features and classifiers.
While Theorem 1 shows that NC features and classifiers are the only global minimizers to (7), itis not obvious whether local search algorithms (such as gradient descent) can efficiently find thesebenign global solutions.
The reason is that the training problem (7) is nonconvex due to the bilinearform between W and H .
To address this challenge, we use the recent advances on the geometricanalysis for nonconvex optimization [24, 23, 36, 25] to guarantee that the global solutions of (7)can be efficiently achieved by iterative algorithms.
Towards that goal, we first present the followinggeneral results concerning the global landscape for (7).
Theorem 2 (Benign landscape) Assume that the feature dimension d is larger than the number ofclasses K , i.e., d > K .
Also assume L ( z , y ) is a convex function in terms of z .
Then the objectivefunction f in (7) is a strict saddle function with no spurious local minimum.
That is, any of its criticalpoint is either a global minimizer, or it is a strict saddle point whose Hessian has a strictly negativeeigenvalue.
This result is similar to [16, Theorem 3.2] which studies the particular CE loss.
Though the resultin [16] is about the CE loss, we checked its proof and it only uses convexity and smoothness andthus the result can be applied more generally for any smooth convex loss function L ( · , y ) .
So weomit the proof of Theorem 2.
We note that the geometric analysis is also closed related to nonconvexlow-rank matrix problems [37–43] with the Burer-Moneirto factorization approach [44] if one views W and H as two factors of a matrix Z = W H .
We refer to [16] for more discussions about theconnections and differences.
We now exploit Theorem 2 for the label smoothing and focal loss.
In the supplementary material,we show that label smoothing is a convex function.
Thus, the following result establishes globaloptimization landscape for the training problem (7) with such a loss.
Corollary 1 (Benign landscape with LS) Assume that the feature dimension d is larger than thenumber of classes K , i.e., d > K .
Then the objective function f in (7) with LS loss L LS is a strictsaddle function with no spurious local minimum.
Unlike LS, focal loss L FL ( z , y k ) is convex only in a local region rather than the entire space.
For example, we can show that L FL ( z , y k ) is convex within the region Ω = { z ∈ R K : exp( z k ) / (cid:80)
Kj =1 exp( z j ) ≥ 0 . 21 } .
The set Ω contains a relative large region including the globalminimizer which has the value
[Equation]
approaching 1.
Thus, we obtain a benign local landscapefor the training problem (7) with focal loss.
Corollary 2 (Benign landscape with FL) Assume that the feature dimension d is larger than thenumber of classes K , i.e., d > K .
Then the objective function f in (7) with FL loss L FL has a benignlocal landscape: f is a strict saddle function with no spurious local minimum within the region{ ( W , H , b ) :
W h k,i + b ∈ Ω , 1 ≤ k ≤ K, 1 ≤ i ≤ n } .
While Corollary 2 only provides a local benign landscape for the focal loss, we observe fromexperiments that gradient descent with random initialization always converges to a global solutionwith NC properties for (7).
So we expect the training problem in (7) with FL loss has benignlandscape in a much larger region.
One direction is to show L FL ( · , y k ) is locally convex in amuch larger region Ω , but leave the thorough investigation as future work.
Noting that the CE and
MSE losses are also convex, these results implies that (stochastic) gradient descent with randominitialization [23, 36] almost surely find the global solutions of the training problem in (7) withdifferent training losses.
This together with Theorem 1 implies that for different losses, gradientdescent will always learn similar features and classifiers—exhibit the NC phenomenon.
Experiments
We conduct experiments with practical network architectures on standard image classification datasetsto study the effect of different loss functions.
First, Section 4.1 provides results to show that
the
NC phenomena is not restricted to networks trained via the CE and MSE losses.
Rather, there is afamily of loss functions, and for the purpose of illustration we pick FL and LS as two prominentspecial cases, that exhibits the same NC phenomena.
Such results verify our theoretical results in
Section 3.
To demonstrate the implication of NC for test performance, we present experimentalresults in Section 4.2 with a varying number of training iterations and a varying width of networks,showing that all losses with NC global optimality have similar performance on the test dataset whenthe network is sufficiently large and trained long enough to achieve a global solution.
Before presenting the experiment results, we first introduces our experiment setup, including theevaluate datasets, network architecture and training procedure.
Setup of Loss Function, Network Architecture, Dataset, and Training We focus on the CE, FL,
LS and MSE loss functions for which we use γ = 3 for FL, α = 0 .
1 for LS, and κ = 1 and β = 15for MSE, except otherwise specificed.
We train a WideResNet50 network [45] on CIFAR10 and CIFAR100 datasets [46] and a WideResNet18 network on miniImageNet [47] with various widthsand number of iterations for image classification using these 4 different losses.
2 To examine theeffect of model size, we experiment with four versions of WideResNet, denoted as WideResNet- X ,where X ∈ { 0 . 25 , 0 . 5 , 1 , 2 } is a multiplier on the width of its corresponding standard WideResNet.
Due to the page limit, we put all results on CIFAR100 and miniImageNet in the Appendix.
Weuse the standard preprocessing where images are normalized (channel-wise) by their mean andstandard deviation, as well as the standard data augmentation.
For optimization, we use the SGDwith momentum 0 .
9 and an initial learning rate 0 .
1 decayed by a factor of 0.1 at 37 and 57 of thetotal number of iterations.
Following [28], the norm of gradient is clipped at 2 which we observeto improve performance for all losses.
For CIFAR10 and miniImageNet, the weight decay is setto 5 × 10 4 for all configurations with all losses.
For CIFAR100, the weight decay is fine-tuned toachieve best accuracy for every configuration and loss.
4.1 Prevalence of NC Across Varying Training Losses
We show that all loss functions lead to NC solutions during the terminal phase of training.
To evaluatehow well the learned last-layer classifier and features satisfy the NC properties, we use the N C 1 ,
N C 2 and N C 3 metrics from [5, 16, 22], which measures the variability collapse , convergence to [Figure] [Figure] simplex ETF , and convergence to self-duality properties, respectively.
For completeness, we providethe definitions of N C 1 , N C 2 and N C 3 in Appendix A.1.
The results on CIFAR10 using WideResNet50-2 and different loss functions is provided in Figure 1.
We consistently observe that all three NC metrics across different losses converge to a small valueas training progresses.
These results imply that all losses exhibit NC at the end, regardless of thechoice of loss functions.
The evolution of three NC metrics across different losses on CIFAR100 isin Appendix A.3.
4.2 All Losses Lead to Largely Identical Performance
We show that all loss functions have largely identical performance once the training procedureconverges to the NC global optimality.
In Figure 2, we plot the evolution of the training accuracy,validation accuracy and test accuracy with training progressing, where all losses are optimized onthe same WideResNet50-2 architecture and CIFAR10 for 800 epochs.
To reduce the randomness,we average the results from 3 different random seeds per iteration-width configuration, and the testaccuracy is reported based on the model with best accuracy on validation set, where we organize thevalidation set by holding out 10 percent data from the training set.
The results consistently showsthat the training accuracy trained by different losses all converge to one hundred percent (reachingto terminal phase), and the validation accuracy and test accuracy across different losses are largelysame, as long as the optimization procedure converges to the NC global solution.
Combined withthe results in Figure 2, we can verify our theoretical results in Section 3 that the last-layer featureslearned under different losses are always maximally linearly separable and perfectly aligned withthe linear classifier, and the features and the weight of linear classifier learned by different losses arealmost equivalent up to a rotation and a scale of the feature space.
While previous work advocates the advantage of some losses over other others, our experiments showthat when conditions between dataset and model allow for SGD to find an NC solution, all losses wetested produced indistinguishable results.
In Figure 3, we plot the average test accuracy of differentlosses under different pairs of width and iterations.
We consistently observe three phenomenon.
[Figure]
First, with a fixed number of iterations, increasing the width of network improves the test accuracyfor all losses.
This is because the wider networks (more over-parameterized) are more powerful tofit the underlying mapping from input data to the targets.
Second, with a fixed width of network,increasing the number of iterations improves the test accuracy for all losses.
This is because the longeroptimization leads the last-layer features and the linear classifer closer to the NC global solution.
Finally, while there are some unignorable difference between different losses in some width-iterationconfigurations, the results consistently show that all losses lead to largely identical performance whenthe network is sufficiently large and trained long enough to achieve a global solution (e.g. width=2and epochs=800).
Conclusion
In this work we provided a theoretical study to extend the scope of NC , a curious phenomenonassociated with last-layer features and classifier weight of a classification network, from networkstrained with particular losses (i.e., CE and MSE) to those trained via a broad family of loss functionsincluding the popular LS and FL as special cases.
Our theory not only establishes NC as the onlyglobal solutions, but also shows a benign optimization landscape that explains why NC solutions areeasy to obtain in practice.
Such results readily suggest that all relevant losses (i.e., CE, MSE, LS,and FL) produce entirely equivalent features on the training data.
Although NC is an optimizationphenomenon pertaining to training data only, we found through experiments that all relevant losses(i.e.
, CE, MSE, LS, and FL) lead to very similar test performance as well.
Such a result may come asa surprise to the common belief that some losses are intrinsically better than the others, and clarifysome mystery on how different losses affect the performance.
The family of loss functions considered in this paper by no means is inclusive of all possible lossfunctions that lead to NC .
There are many other popular loss functions, such as center loss [48],large-margin softmax (L-Softmax) loss [49] and many of its variants [50–52], which are all designedwith the intuition of encouraging intra-class compactness and inter-class separability between learnedfeatures.
In addition, many generalized versions of the cross-entropy loss such as those for robustlearning under label noise [53–55] and long-tail distribution [56, 57] may have similar property as thevanilla cross-entropy loss.
We conjecture that many of them provably produce NC solutions underunconstrained feature models, while leave a formal justification to future work.
Beyond losses forclassification task, NC may also arise with popular losses used in metric learning [58, 59] evidencedby recent study [60].
This means that the observations from this paper, namely all losses lead tolargely the same test performance, may apply for all such losses as well.
Loss functions that do not lead to NC .
While the study in this paper covers many of the mostcommonly used loss functions for classification tasks, we note that there are alternative choices in theliterature which do not induce NC features.
Many of such losses such as [61–63, 60] are particularlydesigned to discourage variability collapse and learn diverse features, which are shown to benefitmodel transferability [64] and robustness.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
In the abstract, introduction, main results, and conclusion, we explicitly stat that our results are about unconstrainedfeature model.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
This
paper mainly focuses on understanding the neural collapse phenomena observed in
practical neural networks.
Based on this understanding, we propose to fix the last layerclassifier as a Simplex ETF.
So no potential negative societal impact is expected of this
work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] We explicitly
mention the assumptions in Theorem 1 and Theorem 2.
(b) Did you include complete proofs of all theoretical results?
[Yes] We include all theproofs in the Appendix.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] See Section (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] All the results are obtained by averaging the resluts from 3 different random seeds.
See Section
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix A.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We cite them in
Appendix A.
(b) Did you mention the license of the assets?
[Yes] This is mentioned in Appendix A.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] This is discussed in Appendix A.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
The CIFAR datasets do not contain personally
identifiable information or offensive content
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
Our research does not involve with participants and screenshots
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
Our research does not include any potential
participant risks, with links to Institutional Review Board (IRB) approvals
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
Our work does not require participants.