Are All Losses Created Equal: A Neural Collapse Perspective
Abstract
While cross entropy (CE) is the most commonly used loss function to train deep neural networks for classiﬁcation tasks, many alternative losses have been developed to obtain better empirical performance.
Among them, which one is the best to use is still a mystery, because there seem to be multiple factors affecting the answer, such as properties of the dataset, the choice of network architecture, and so on.
This paper studies the choice of loss function by examining the last-layer features of deep networks, drawing inspiration from a recent line work showing that the global optimal solution of CE and mean-square-error (MSE) losses exhibits a Neural Collapse ( NC ) phenomenon.
That is, for sufﬁciently large networks trained until convergence, (i) all features of the same class collapse to the corresponding class mean and (ii) the means associated with different classes are in a conﬁguration where their pairwise distances are all equal and maximized.
We extend such results and show through global solution and landscape analyses that a broad family of loss functions including commonly used label smoothing (LS) and focal loss (FL) exhibits NC .
Hence, all relevant losses (i.e., CE, LS,
FL, MSE) produce equivalent features on training data.
In particular, based on the unconstrained feature model assumption, we provide either the global landscape analysis for LS loss or the local landscape analysis for FL loss and show that the (only!)
global minimizers are NC solutions, while all other critical points are strict saddles whose Hessian exhibit negative curvature directions either in the global scope for LS loss or in the local scope for FL loss near the optimal solution.
The experiments further show that NC features obtained from all relevant losses (i e., CE, LS, FL, MSE) lead to largely identical performance on test data as well, provided that the network is sufﬁciently large and trained until convergence.
The source code is available at https://github.com/jinxinzhou/nc_loss .
Introduction
Loss function is an indispensable component in the training of deep neural networks (DNNs).
While cross-entropy (CE) loss is one of the most popular choices for classiﬁcation tasks, studies over the past few years have suggested many improved versions of CE that bring better empirical performance.
Some notable examples include label smoothing (LS) [1] where one-hot label is replaced by a smoothed label, focal loss (FL) [2] which puts more emphasis on hard misclassiﬁed samples and reduces the relative loss on the already well-classiﬁed samples, and so on.
Aside from CE and its variants, the mean squared error (MSE) loss which was typically used for regression tasks is recently demonstrated to have a competitive performance when compared to CE for classiﬁcation tasks [3].
Despite the existence of many loss functions there is however a lack of consensus as to which one is the best to use, and the answer seems to depend on multiple factors such as properties of the dataset, choice of network architecture, and so on [4].
In this work, we aim to understand the effect of loss function in classiﬁcation tasks from the perspective of characterizing the last-layer features and classiﬁer of a DNN trained under different losses.
Our study is motivated by a sequence of recent work that identify an intriguing Neural Collapse ( NC ) phenomenon in trained networks, which refers to the following properties of the last-layer features and classiﬁer:
(i) Variability Collapse : all features of the same class collapse to the corresponding class mean.
(ii) Convergence to Simplex ETF : the means associated with different classes are in a Simplex Equiangular Tight Frame (ETF) conﬁguration where their pairwise distances are all equal and maximized.
(iii) Convergence to Self-duality : the class means are ideally aligned with the last-layer linear classiﬁers.
(iv) Simple Decision Rule : the last-layer classiﬁer is equivalent to a Nearest Class-Center decision rule.
This NC phenomena is ﬁrst discovered by Papyan et al.
[5, 6] under canonical classiﬁcation problems trained with the CE loss.
Following with the CE loss, Han et al.
[7] recently reported that DNNs trained with MSE loss for classiﬁcation problems also exhibit similar NC phenomena.
These results imply that deep networks are essentially learning maximally separable features between classes, and a max-margin classiﬁer in the last layer upon these learned features.
The intriguing empirical observation motivated a surge of theoretical investigation [7–22], mostly under a simpliﬁed unconstrained feature model [10] or layer-peeled model [12] that treats the last-layer features of each samples before the ﬁnal classiﬁer as free optimization variables.
Under the simpliﬁed unconstrained feature model, it has been proved that the NC solution is the only global optimal solution for the CE and
MSE losses which are also proved to have benign global landscape, explaining why the global NC solution can be obtained.
Contributions.
While previous work provide thorough analysis for NC under CE and MSE losses, the theoretical analysis beyond CE and MSE losses is still limited, and their work only focus on one speciﬁc loss without a general format.
In this paper, we consider a broad family of loss functions that includes CE and some other popular loss functions such as LS and FL as special cases.
Under the unconstrained feature model , we theoretically demonstrate in Section 3 that the NC solution is the only global optimal solution to the family of loss functions.
Moreover, we provide a global landscape analysis, showing that the LS loss function is a strict saddle function and FL loss function is a local strict saddle function [23–25].
A (local) strict saddle function is a function for which every critical point is either a global solution or a strict saddle point with negative curvature (locally).
Hence, our result suggests that any optimizer can escape strict saddle points and converge to the global solution responding to NC for LS and FL.
As far as we know, this paper is the ﬁrst work that conducts global optimal solution and benign optimization landscape analysis beyond the scope of
CE and MSE losses.
Our theoretical results explained above have important implications for understanding the role of loss function in training DNNs for classiﬁcation tasks.
Because all losses lead to NC solutions, their corresponding features are equivalent up to a rotation of the feature space.
In other words, our analysis provides a theoretical justiﬁcation for the following claim:
All losses (i.e., CE, LS, FL, MSE) lead to largely identical features on training data by large DNNs and sufﬁciently many iterations.
We also provide an experimental veriﬁcation of this claim through experiments in Section 4.1.
While NC reveals that all losses are equivalent at training time, it does not have a direct implication for the features associated with test data as well as the generalization performance [26].
In particular, a recent work [27] shows empirically that NC does not occur for the features associated with test data.
Nonetheless, we show through empirical evidence that for large DNNs, NC on training data well predicts the test performance.
In particular, our empirical study in Section 4.2 shows the following:
All
losses (CE, LS, FL,
MSE) lead to largely identical performance on
test
data by large DNNs.
Our conclusion that all losses are created equal appears to go against existing evidence on the advantages of some losses over the others.
Here we emphasize that our conclusion has an important premise, namely the neural network has sufﬁcient approximation power and the training is performed for sufﬁciently many iterations.
Hence, our conclusion implies that the better performance with particular choices of loss functions comes as a result that the training does not produce a globally optimal (i.e., NC ) solution.
In such cases different losses lead to different solutions on the training data, and correspondingly different performance on test data.
Such an understanding may provide important practical guidance on what loss to choose in different cases (e.g., different model sizes and different training time budgets), as well as for the design of new and better losses in the future.
We note that our conclusion is based on natural accuracy, rather than model transferability or robustness, which is worth additional efforts to exploit and is left as future work.
The Problem Setup
A typical deep neural network Ψ( · ) : R D (cid:55)→ R K consists of a multi-layer nonlinear compositional feature mapping Φ( · ) : R D (cid:55)→ R d and a linear classiﬁer ( W , b ) , which can be generally expressed as
[Equation]
where we use θ to represent the network parameters in the feature mapping and W ∈ R K × d and b ∈ R K to represent the linear classiﬁer’s weight and bias, respectively.
Therefore, all the network parameters are the set of Θ = { θ , W , b } .
For the input x , the output of the feature mapping Φ θ ( x ) is usually termed as the representation or feature learned from the network.
With an appropriate loss function, the parameters Θ of the whole network are optimized to learn the underlying relation from the input sample x to their corresponding target y so that the output of the network Ψ Θ ( x ) approximates the corresponding target, i.e. Ψ Θ ( x ) ≈ y in term of the expectation over a distribution D of input-output data pairs ( x , y ) .
While it is hard to get access to the groundtruth distribution D in most cases, one can approximate the distribution D through sampling enough data pairs i.i.d.
from D .
In this paper, we study the multi-class balanced classiﬁcation tasks with K class and n samples per class, where we use the one-hot vector y k ∈ R K with unity only in k -th entry (1 ≤ k ≤ K ) to denote the label of the i -th sample x k,i ∈
R D in the k -th class.
We then learn the parameters Θ via minimizing the following empirical risk over the total N = nK training samples
[Equation]
where λ > 0 is the regularization parameter (a.k.a., the weight decay parameter 2 ) and
L (Ψ Θ ( x k,i ) , y k ) is a predeﬁned loss function that appropriately measures the difference between the output Ψ Θ ( x k,i ) and the target y k .
Some common loss functions used for training deep neural networks will be speciﬁed in the next section.
2.1 Commonly Used Training Losses
In this subsection, we ﬁrst present four common loss functions for classiﬁcation task.
To simplify the notation, let z
= W Φ θ ( x ) + b denote the network’s output (“logit”)
vector for the input x .
Assume z belongs to the k -th class.
Also let y smooth k
=
(1 − α )
y k
+ α K 1 K denote the smoothed targets of k -th class, where 0 ≤ α < 1 and 1 K ∈ R K is a vector with all entries equal to one.
We will use z (cid:96) , y k,(cid:96) and y smooth (cid:96) to denote the (cid:96) -th entry of z , y k and y smooth k , respectively, where
[Equation]
[Equation]
Cross entropy (CE) is perhaps the most common loss for multi-class classiﬁcation in deep learning.
It measures the distance between the target distribution y k and the network output distribution obtained by applying the softmax function on z , resulting in the following expression
[Equation]
Focal loss (FL) [2] is ﬁrst proposed to deal with the extreme foreground-background class imbalance in dense object detection, which adaptively focuses less on the well-classiﬁed samples.
Recent work [28, 29] reports that focal loss also improves calibration and automatically forms curriculum learning in multi-class classiﬁcation setting.
Letting γ ≥ 0 denote the tunable focusing parameter, the focal loss can be expressed as:
[Equation]
Label smoothing (LS) [1] replaces the hard targets in CE with smoothed targets y smooth k obtained from mixing the original targets y k with a uniform distribution over all entries 1 K .
Experiments in [30, 31] ﬁnd that classiﬁcation models trained with label smoothing have better calibration and generalization.
Denoting by 0 ≤ α ≤ 1 the tunable smoothing parameter, the label smoothing loss function can be formulated as:
[Equation]
When α = 0 , the above label smoothing loss reduces to the CE loss.
Mean square error (MSE) is often used for regression but not classiﬁcation task.
The recent work [3] shows that classiﬁcation networks trained with MSE loss achieve on par performance compared to those trained with the CE loss.
Throughout our paper, we use the rescaled MSE version [3]:
[Equation]
where κ > 0 and β > 0 are hyperparameters.
2.2 Problem Formulation Based on Unconstrained Feature Models
Because of the interaction between a large number of nonlinear layers in the feature mapping Φ θ , it is tremendously challenging to analyze the optimization of deep neural networks.
To simplify the difﬁculty of deep neural network analysis, a series of recent works of theoretically studying NC phenomenon use a so-called unconstrained feature model (or layer-peeled model in [12]) which treats the last-layer features as free optimization variables h
= Φ( x ) ∈ R d .
The reason behind the unconstrained feature model is that modern highly overparameterized deep networks are able to approximate any continuous functions [32–35] and the characterization of NC are only related with the last layer features.
We adopt the same approach and study the effects of different training losses on the last-layer representations of the network under the unconstrained feature model.
For convenient, let us denote
[Equation]
H
[Equation]
H n ] ∈ R
[Equation]
N
[Equation]
and
Y
[Equation]
where w k is the k -th row vector of W , all the features in the k -th class are denoted as H i :=
[ h 1 ,i · · · h K,i ] ∈ R d × K and h k,i is the feature of the i -th sample in the k -th class, and Y k :=
[ y k · · · y k ] ∈ R K × n for all k = 1 , 2 , · · · , K and i = 1 , 2 , · · · , n .
Based on the unconstrained feature model, we consider a slight variant of (2), given by
[Equation]
where λ W , λ H , λ b > 0 are the penalty parameters for W , H , and b , respectively.
By viewing the last-layer feature H as a free optimization variable, the simpliﬁed objective function (7) consider the weight decay about W and H , which is slightly different from practice that the weight decay is imposed on all the network parameters Θ as shown in (2).
Nonetheless, the underlying rationale is that the weight decay on Θ implicitly penalizes the energy of the features (i.e., (cid:107) H (cid:107) F ) [16].
As NC phenomena for the learned features and classiﬁers is ﬁrst discovered for neural networks trained with the CE loss [5], the CE loss has been mostly studied through the above simpliﬁed unconstrained feature model [8,9,11,12,16] to understand the NC phenomena.
The work [7,10,14, 22] also studied the MSE loss, but the analysis there shows the solutions of the learned features and classiﬁers depend crucially on the bias term, while for CE loss with or without the bias term have no effect on the learned features and classiﬁers under the unconstrained feature model.
The other losses such as focal loss and label smoothing have been less studied, though they are widely employed in practice to obtain better performance.
This will be the subject of next section.
Understanding Loss Functions Through Unconstrained Features Model
In this section, we study the effect of different loss functions through the unconstrained features model.
We will ﬁrst present a contrastive property for general loss function L GL in Deﬁnition 1.
We will then study the global optimality conditions in terms of the learned features and classiﬁers as well as geometric properties for (7) with such a general loss function L GL .
3.1 A Contrastive Property for the Loss Functions
In this paper, we aim to provide a uniﬁed analysis for different loss functions.
Towards that goal, we ﬁrst present some common properties behind the CE, FL and FL to motivate the discussion.
Taking
CE as an example, we can lower bound it by
[Equation]
1 + ( K − 1) exp (cid:18)(cid:80) j
(cid:54) =
k ( z j − z k )
K −(cid:19)(cid:19) =
φ CE  (cid:88)
j (cid:54) =
k ( z j − z k ) 
 (8) where φ CE ( t ) =
log (cid:16)
+ ( K − 1) exp (cid:16) tK − 1 (cid:17)(cid:17) , and the inequality achieves equality when z j =
z j (cid:48) for all j, j (cid:48)
(cid:54) =
k .
This requirement is reasonable because the commonly used losses treat all the outputs except for the k -th output z identically.
Since φ CE is an increasing function, minimizing the CE loss L CE ( z , y k ) is equivalent to maximizing ( K − 1)
z k − (cid:80) j
(cid:54) =
K z j , which contrasts the k -th output z k simultaneously to all the other outputs z j for all j (cid:54) =
k .
Thus, we call (8) as a contrastive property .
Maximizing ( K − 1)
z k − (cid:80) j
(cid:54) =
K z j would lead to a positive (and relatively large) z k and negative (and relatively small) z j .
In particular, within the unit sphere (cid:107) z (cid:107)
=
1 , ( K − 1)
z k − (cid:80) j
(cid:54) =
K z j achieves its maximizer when z k = (cid:113) K − 1 K and z j = − (cid:113) 1 K ( K − 1) for all j (cid:54)
=
k , which satisﬁes the requirement z j = z
j (cid:48) for all j, j (cid:48)
(cid:54) =
k .
Thus, z k = (cid:113) K − 1 K and z j = − (cid:113) 1 K ( K − 1) is also the global minimizer for φ CE within the unit sphere (cid:107) z (cid:107) 2 = 1 .
As the global minimizer is unique for each class, it encourages intra-class compactness.
On the other hand, the minimizers to different classes are maximally distant, promoting inter-class separability.
Motivated by the above discussion, we now introduce the following properties for a general loss function L GL ( z , y k ) .
Deﬁnition 1 (Contrastive property) .
We say a loss function L GL ( z , y k ) satisﬁes the contrastive property if there exists a function φ such that L GL ( z , y k ) can be lower bounded by
[Equation]
where the equality holds only when z j = z (cid:48)
j for all j, j (cid:48) (cid:54) =
k .
Moreover, φ ( t ) satisﬁes t ∗ =
arg min t φ ( t ) + c | t | is unique for any c > 0 , and t ∗
[Equation]
[Equation]
In the appendix, we show that CE, FL and LS all satisfy this property.
The motivation for (9) follows from the above discussion.
In particular, (9) achieves equality when all the outputs except for the k -th one are identical, which holds for common loss functions since those outputs are treated identically.
In (10), c is a constant related with the weight decay penalty parameters.
By (9), we can ﬁnd the global minimizer for L GL ( z , y k ) by minimizing the right hand side since the equality in (9) is achievable.
Thus, the requirement of a unique minimizer (10) ensures a unique minimizer for the regularized L GL ( z , y k ) .
This condition can be easily satisﬁed.
For example, φ CE ( t ) deﬁned in (8) for the CE loss is an increasing and strictly convex function and thus has unique minimizer for φ CE ( t )
+ c | t | .
Along the same line, we require a negative minimizer t (cid:63) to ensure that the minimizer for the regularized L GL ( z , y k ) has k -th entry being its largest entry, which is required to ensure correct prediction since the largest entry predicts the class membership.
Therefore, such a condition is generally satisﬁed by the common losses.
For example, φ CE ( t ) is an increasing function and thus must have a non-positive minimizer for φ CE ( t )
+ c | t | .
Finally, we note that the MSE loss is not included since it has different form than others and thus the analysis will be different.
But as mentioned above, the MSE loss has been studied in [7,10,14,22].
3.2 Landscape Analysis for the Unconstrained Features Model
We now study the global optimality conditions in terms of the learned features and classiﬁers as well as geometric properties for the training problem (7) with the general loss function L GL satisfying the above contrastive property.
Theorem 1 (Global Optimality Condition) .
Assume that the number of classes K is smaller than the feature dimension d , i.e., K < d , and the dataset is balanced for each class, n = n 1 = · · · = n K .
Then any global minimizer ( W (cid:63) , H (cid:63) , b (cid:63) ) of f in (7) with a loss function L GL satisfying the contrastive property in Deﬁnition 1 has following properties:
[Equation]
and b (cid:63) =
b (cid:63) 1 ,
[Equation] [Equation]
, i
[Equation] [Equation]
and h (cid:63)i := 1 K
K (cid:88)
j = h (cid:63)j,i = 0 , ∀ i ∈ [ n ] , where either b (cid:63) = 0 or λ b = 0 , and the matrix W (cid:63) (cid:62) is in the form of K -simplex ETF structure (see appendix for the formal deﬁnition) in the sense that
[Equation]
Its proof is given in Appendix C. At a high level, we lower bound the general loss function based on the contrastive property (9) and then check the equality conditions hold for the lower bounds.
While similar strategy has been used for CE loss [12,13,16], our proof is different from previous work in terms of dealing with the nuclear norm and checking the structures of the minimizer per sample and class, enabling the global optimality analysis for general loss functions.
Theorem 1 implies that for all the loss functions (e.g., CE, LS, and FL) satisfying the contrastive property, they share similar global solutions with NC property in the learned features and classiﬁers.
While Theorem 1 shows that NC features and classiﬁers are the only global minimizers to (7), it is not obvious whether local search algorithms (such as gradient descent) can efﬁciently ﬁnd these benign global solutions.
The reason is that the training problem (7) is nonconvex due to the bilinear form between W and H .
To address this challenge, we use the recent advances on the geometric analysis for nonconvex optimization [23–25,36] to guarantee that the global solutions of (7) can be efﬁciently achieved by iterative algorithms.
Towards that goal, we ﬁrst present the following general results concerning the global landscape for (7).
Theorem 2 (Benign Landscape) .
Assume that the feature dimension d is larger than the number of classes K , i.e., d > K .
Also assume L ( z , y ) is a convex function in terms of z .
Then the objective function f in (7) is a strict saddle function with no spurious local minimum.
That is, any of its critical point is either a global minimizer, or it is a strict saddle point whose Hessian has a strictly negative eigenvalue.
This result is similar to [16, Theorem 3.2] which studies the particular CE loss.
Though the result in [16] is about the CE loss, we checked its proof and it only uses convexity and smoothness and thus the result can be applied more generally for any smooth convex loss function L ( · , y ) .
So we omit the proof of Theorem 2.
We note that the geometric analysis is also closed related to nonconvex low-rank matrix problems [37–43] with the Burer-Moneirto factorization approach [44] if one views
W and H as two factors of a matrix Z = W H .
We refer to [16] for more discussions about the connections and differences.
We now exploit Theorem 2 for the label smoothing and focal loss.
In the supplementary material, we show that LS is a convex function.
Thus, the following result establishes global optimization landscape for the training problem (7) with such a loss.
Corollary 1 (Benign Landscape with LS) .
Assume that the feature dimension d is larger than the number of classes K , i.e., d > K .
Then the objective function f in (7) with LS loss L LS is a strict saddle function with no spurious local minimum.
Unlike LS, focal loss L FL ( z , y k ) is convex only in a local region rather than the entire space.
For example, we can show that L FL ( z , y k ) is convex within the region Ω = { z ∈ R K : exp( z k ) / (cid:80)
Kj =1 exp( z j ) ≥ 0 . 21 } .
The set Ω contains a relative large region including the global minimizer which has the value exp( z k ) (cid:80)
Kj =1 exp( z j ) approaching 1.
Thus, we obtain a benign local landscape for the training problem (7) with FL.
Corollary 2 (Benign Landscape with FL) .
Assume that the feature dimension d is larger than the number of classes K , i.e., d > K .
Then the objective function f in (7) with FL loss L FL has a benign local landscape: f is a strict saddle function with no spurious local minimum within the region { ( W , H , b ) :
W h
[Equation]
While Corollary 2 only provides a local benign landscape for the FL, we observe from experiments that gradient descent with random initialization always converges to a global solution with NC properties for (7).
So we expect the training problem in (7) with FL loss has benign landscape in a much larger region.
One direction is to show L FL ( · , y k ) is locally convex in a much larger region Ω , but we leave the thorough investigation to future work.
Noting that the CE and MSE losses are also convex, these results imply that (stochastic) gradient descent with random initialization [23,36] almost surely ﬁnds the global solutions of the training problem in (7) with different training losses.
This together with Theorem 1 implies that for different losses, gradient descent will always learn similar features and classiﬁers—those that exhibit the NC phenomenon.
Experiments
We conduct experiments with practical network architectures on standard image classiﬁcation datasets to study the effect of different loss functions.
First, Section 4.1 provides results to show that the NC phenomena are not restricted to networks trained via the CE and MSE losses.
Rather, there is a family of loss functions, and for the purpose of illustration we pick FL and LS as two prominent special cases, that exhibit the same NC phenomena.
Such results verify our theoretical results in Section 3.
To demonstrate the implication of NC for test performance, we present experimental results in Section 4.2 with a varying number of training iterations and a varying width of networks, showing that all losses with NC global optimality have similar performance on the test dataset when the network is sufﬁciently large and trained long enough.
Before presenting the experiment results, we ﬁrst introduce our experimental setup, including datasets, network architectures, training procedure, and metrics for measuring NC .
Setup of Loss Function, Network Architecture, Dataset, and Training We focus on the CE, FL,
LS and MSE loss functions for which we use γ = 3 for FL, α = 0 .
1 for LS, and κ = 1 and β =for MSE, except otherwise speciﬁced.
We train a WideResNet50 network [45] on CIFAR10 and
CIFAR100 datasets [46] and a WideResNet18 network on miniImageNet [47] with various widths and number of iterations for image classiﬁcation using these four different losses.
3 To examine the effect of model size, we experiment with four versions of WideResNet, denoted as WideResNet- X , where X ∈ { 0 . 25 , 0 . 5 , 1 , 2 } is a multiplier on the width of its corresponding standard WideResNet.
Due to the page limit, we put all results on CIFAR100 and miniImageNet in the Appendix.
We use standard preprocessing such that images are normalized (channel-wise) by their mean and standard deviation, as well as standard data augmentation.
For optimization, we use SGD with momentum 0 .
9 and an initial learning rate 0 .
1 decayed by a factor of 0.1 at 37 and 57 of the total number of iterations.
Following [28], the norm of gradient is clipped at 2 which can improve performance for all losses.
For CIFAR10 and miniImageNet, the weight decay is set to 5 × 10 4 for all conﬁgurations with all losses.
For CIFAR100, the weight decay is ﬁne-tuned to achieve best accuracy for every conﬁguration and loss.
Three NC Metrics NC 1 - NC 3 during Network Training
We use the same three metrics NC 1NC 3 for the last-layer features and classiﬁer as in [5,16,22] to measure the ﬁrst three NC properties in Section 1.
Before we describe these three metrics, let us denote the global mean h G and k -th class mean h k of last-layer features { h k,i } as
[Equation]
Within-class variability collapse is measured by N C 1 which depicts the relative magnitude of the within-class covariance Σ
[Equation]
t. the between-class covariance Σ B = 1 K (cid:80)
Kk =1 (cid:0)
h k − h G (cid:1) (cid:0)
[Equation]
k − h G (cid:1) (cid:62) ∈ R
d × d of the last-layer features as following:
[Equation]
where Σ † B is the pseudo inverse of Σ B .
Convergence to simplex ETF is measured by N C 2 which reﬂects the (cid:96) 2 distance between the normalized simplex ETF and the normalized W W (cid:62) as following:
[Equation]
where W ∈ R K × d is the weight matrix of learned classiﬁer.
Convergence to self-duality is measured by N C 3 which calculates the (cid:96) 2 distance between the normalized simplex ETF and the normalized W H as following:
[Equation]
where H = (cid:2) h 1 − h G · · · h K − h G (cid:3)
∈ R d × K is the centered class-mean matrix.
4.1 Prevalence of NC Across Varying Training Losses
We show that all loss functions lead to NC solutions during the terminal phase of training.
The results on CIFAR10 using WideResNet50-2 and different loss functions is provided in Figure 1.
We consistently observe that all three NC metrics across different losses converge to a small value as training progresses.
This supports our theoretical results in Section 3 that the last-layer features learned under different losses are always maximally linearly separable and perfectly aligned with the linear classiﬁer, and the features and the weight of linear classiﬁer learned by different losses are almost equivalent up to a rotation and a scale of the feature space.
The evolution of three NC metrics across different losses on CIFAR100 is in Appendix A.2.
4.2 All Losses Lead to Largely Identical Performance
We show that all loss functions have largely identical performance once the training procedure converges to the NC global optimality.
In Figure 2, we plot the evolution of the training accuracy,
[Figure] [Figure]
validation accuracy and test accuracy as training progresses, where all losses are optimized on the same WideResNet50-2 architecture and CIFAR10 for 800 epochs.
To reduce the randomness, we average the results from 3 different random seeds per width-iteration conﬁguration, and the test accuracy is reported based on the model with best accuracy on validation set, where we organize the validation set by holding out 10 percent data from the training set.
The results consistently show that for all cases the training accuracy converges to one hundred percent (reaching to terminal phase), and the validation accuracy and test accuracy are largely the same, as long as the network is trained longer enough and converges to the NC global solution.
While previous work advocates the advantage of some losses over other others, our experiments show that when conditions between dataset and model allow for SGD to ﬁnd an NC solution, all losses we tested produced indistinguishable results.
In Figure 3, we plot the average test accuracy of different losses under different pairs of width and iterations.
We consistently observe three phenomenon.
First, with a ﬁxed number of iterations, increasing the width of network improves the test accuracy for all losses.
This is because the wider networks (more over-parameterized) are more powerful to ﬁt the underlying mapping from input data to the targets.
Second, with a ﬁxed width of network, increasing the number of iterations improves the test accuracy for all losses.
This is because the longer optimization leads the last-layer features and the linear classifer closer to the
NC global solutions.
Finally, while there are some unignorable difference between different losses in some width-iteration conﬁgurations, the results consistently show that all losses lead to largely identical performance when the network is sufﬁciently large and trained long enough to achieve a global NC solution (e.g. width=2 and epochs=800).
Conclusion
In this work we provided a theoretical study to extend the scope of NC , a curious phenomenon associated with last-layer features and classiﬁer weight of a classiﬁcation network, from networks trained with particular losses (i.e., CE and MSE) to those trained via a broad family of loss functions including the popular LS and FL as special cases.
Our theory not only establishes NC as the only global solutions, but also shows a benign optimization landscape that explains why NC solutions
[Figure]
are easy to obtain in practice.
Such results readily suggest that all relevant losses (i.e., CE, MSE, LS, and FL) produce entirely equivalent features on the training data.
Although NC is an optimization phenomenon pertaining to training data only, we found through experiments that all relevant losses (i.e., CE, MSE, LS, and FL) lead to very similar test performance as well.
Such a result may come as a surprise to the common belief that some losses are intrinsically better than the others, and clarify some mystery on how different losses affect the performance.
The family of loss functions considered in this paper by no means is inclusive of all possible loss functions that lead to NC .
There are many other popular loss functions, such as center loss [48], large-margin softmax (L-Softmax) loss [49] and many of its variants [50–52], which are all designed with the intuition of encouraging intra-class compactness and inter-class separability between learned features.
In addition, many generalized versions of the cross-entropy loss such as those for robust learning under label noise [53–55] and long-tail distribution [56,57] may have similar property as the vanilla cross-entropy loss.
We conjecture that many of them provably produce NC solutions under unconstrained feature models, while leave a formal justiﬁcation to future work.
Beyond losses for classiﬁcation task, NC may also arise with popular losses used in metric learning [58,59] evidenced by recent study [60].
This means that the observations from this paper, namely all losses lead to largely the same test performance, may apply for all such losses as well.
Loss functions that do not lead to NC .
While the study in this paper covers many of the most commonly used loss functions for classiﬁcation tasks, we note that there are alternative choices in the literature which do not induce NC features.
Many of such losses such as [60–63] are particularly designed to discourage variability collapse and learn diverse features, which are shown to beneﬁt model transferability [64] and robustness
Acknowledgment
JZ and ZZ acknowledge support from NSF grants CCF-2008460 and CCF-2106881.
XL and QQ acknowledge support from U-M START & PODS grants, NSF CAREER CCF 2143904, NSF CCF 2212066, NSF CCF and ONR N00014-22-1-2529 grants.
SL acknowledges support from
NSF NRT-HDR Award 1922658 and Alzheimer’s Association grant AARG- NTF-21-848627.
Appendices
Organizations and Basic.
The appendix is organized as follows.
We ﬁrst introduce the basic deﬁnitions and inequalities used throughout the appendices.
In Appendix A, we provide more details about the datasets, computational resources, and more experiment results on CIFAR10, CIFAR and miniImageNet datasets.
In Appendix B, we prove that CE, FL and LS satisfy the contrastive property in Deﬁnition 1.
In Appendix C, we provide a detailed proof for Theorem 1, showing that
the
Simplex ETFs are the only global minimizers, as long as the loss function satisﬁes the Deﬁnition 1.
Finally, in Appendix D, we present the whole proof for Theorem 2 that the FL function is a locally strict saddle function with no spurious local minimizers existing locally and LS function is a globally strict saddle function with no spurious local minimizers existing globally.
Deﬁnition 2 ( K -Simplex ETF) .
A standard Simplex ETF is a collection of points in R K speciﬁed by the columns of
[Equation]
where I K ∈
R K × K is the identity matrix, and 1 K ∈ R K is the all ones vector.
In the other words, we also have
[Equation]
As in [5,12], in this paper we consider general Simplex ETF as a collection of points in R d speciﬁed by the columns of (cid:113) KK − 1 P (cid:0)
I K − 1 K 1 K 1 (cid:62) K (cid:1) , where P ∈ R d × K ( d ≥ K ) is an orthonormal matrix, i.e., P (cid:62) P
= I K .
Lemma 1 (Young’s Inequality) .
Let p, q be positive real numbers satisfying 1 p + 1 q = 1 .
Then for any a, b ∈ R , we have
[Equation]
where the equality holds if and only if | a | p
= | b | q .
The case p =
q = 2 is just the AM-GM inequality for a 2 , b 2 : | ab | ≤ 12 (cid:0) a 2 + b 2 (cid:1) , where the equality holds if and only if | a | = | b | .
The following Lemma extends the standard variational form of the nuclear norm.
Lemma 2.
For any ﬁxed W ∈ R K × d , H i ∈ R d × K , ¯ Z i = W H i ∈
R K × K and α > 0 , we have
[Equation]
Here, (cid:13)(cid:13)
¯ Z i (cid:13)(cid:13) ∗ denotes the nuclear norm of ¯ Z i : (cid:13)
[Equation]
∗ :=
K
[Equation]
k = σ k ( ¯ Z i ) =
trace ( Σ ) , with ¯ Z i =
U Σ V (cid:62) , where { σ k } Kk =1 denotes the singular values of ¯ Z i , and ¯ Z i =
U Σ V (cid:62) is the singular value decomposition (SVD) of ¯ Z i .
Proof of Lemma 2.
Let ¯ Z i =
U Σ V (cid:62) be the SVD of ¯ Z i .
For any W H i =
¯ Z i , we have
[Equation]
where the ﬁrst inequality utilize the Young’s inequality in Lemma 1 that | trace( AB ) | ≤ 12 c (cid:107) A (cid:107) 2 F + c 2 (cid:107) B (cid:107) 2 F for any c > 0 and A , B of appropriate dimensions, and the last inequality follows because (cid:107) U (cid:107) = 1 and (cid:107) V (cid:107) = 1 .
Therefore, we have
[Equation]
We complete the proof.
Lemma 3 (Eigenvalues of Diagonal-Plus-Rank-One Matrices) .
Let τ < 0 , z ∈ R n , and D be an n × n diagonal matrix with diagonals d 1 , . . .
, d n .
Let λ 1 , . . .
λ n be the eigenvalues of the diagonalplus-rank-one matrix D + τ zz (cid:62) .
• Case 1: If d 1 > d 2 > · · · > d n and z i (cid:54) =
for all i = 1 , · · · , n , then the eigenvalues { λ i } are equal to the n roots of the rational function [65,66]
[Equation]
and the diagonals { d i } strictly separate the eigenvalues as following:
[Equation]
• Case 2:If z i = 0 for some i , then d i is an eigenvalue of D + τ zz (cid:62) with corresponding eigenvector e i since
[Equation]
The remaining n − 1 eigenvalues of D + τ zz (cid:62) are equal to the eigenvalues of the smaller matrix D (cid:48)
+ τ z (cid:48)
z (cid:48) (cid:62) , where D (cid:48) ∈ R ( n − 1) × ( n − 1) and z (cid:
) ∈ R n − 1 are obtained by removing the i -th rows and columns from D and the i -th element from z , respectively.
One can repeat this process if z (cid:48) still has zero element.
• Case 3: If there are m mutually equal diagonal elements, say d i +1 = · · · = d i + m =
d , then for any orthogonal m × m matrix P , D + τ zz (cid:62)
has the same eigenvalues as
T DT (cid:62) + τ ( T z )
( T z ) (cid:62)
=
D + τ (cid:98) z (cid:98) z
(cid:62) ,
where T = (cid:34) I
i
P
I n − i − m (cid:35) , (cid:98) z
= T (cid:98) z .
We can then choose P as a Householder transformation such that
[Equation]
Thus, according to Case 2, d is an eigenvalue of D + τ (cid:98)
z (cid:98) z (cid:62) repeated m − 1 times and the remaining eigenvalues can be computed by checking the smaller matrix.
Based on Lemma 3, we can prove the following Lemma.
Lemma 4.
Let K ≥ 3 and Z = − (cid:0)
I K − 1 K 11 (cid:62) (cid:1) diag ( ρ 1 ,
ρ 2 , · · · , ρ K ) with | ρ 1 | ≥ | ρ 2 | ≥ ·
· · ≥ | ρ K | and | ρ 1 | > 0 .
Also let σ
i ≥ 0 be the i -th largest singular value of Z .
Suppose there exists k with 1 ≤ k ≤ K − 1 such that
[Equation]
Then | ρ 1 | , · · · , | ρ K | must satisfy either
[Equation]
, with σ max = | ρ 1 | , or ρ 2 = · · · = ρ K = 0 , with σ max
= (cid:114) K −
K | ρ 1 | .
Proof of Lemma 4.
Because
[Equation]
where ρ = [ ρρ· · · ρ K ] (cid:62) , Z (cid:62) Z satisﬁes the form of Diagonal-Plus-Rank-One in Lemmawith D = diag (cid:0) ρ 21 , ρ 22 , · · · , ρ 2 K (cid:1) , z = ρ and τ = − 1 K .
Let λ 1 ≥ λ 2 ≥ · · · λ K ≥ 0 denote the n eigenvalues of Z (cid:62) Z .
Due to 1 (cid:62) Z = 0 (cid:62) , we can have λ K = 0 .
• If | ρ 1 | = | ρ 2 | = · · · = | ρ K | : we have ρ 21 = λ 1 = · · · = λ K − 1 = ρ 2 K > λ K = 0 .
Thus, σ max = √ λ 1 = | ρ 1 | .
• If | ρ 1 | > | ρ 2 | = · · · = | ρ K | = 0 : according to Case 2 in Lemma 3, we have
[Equation] [Equation]
K = 0 .
Thus, σ max = (cid:112) (1 − 1 /K )
ρ 21 = (cid:112) ( K − 1) /K
| ρ 1 | .
• If | ρ 1 | > | ρ 2 | = · · · = | ρ K | (cid:54) = 0 : according to Case 3 in Lemma 3, we have
[Equation]
and the remaining two eigenvalues are the same to those of
[Equation] [Equation]
[Equation]
(cid:21) (cid:2)
[Equation]
(cid:3) .
According to (12) in Lemma 3, we can obtain
[Equation] [Equation]
> ρ 2 K > λ K = 0 .
Combing them together, we can have
[Equation]
thus, 0 = λ K < λ 2 < λ 1 = λ max , which violates the assumption (13).
• If | ρ 1 | = · · · = | ρ i | > | ρ i +1 | = · · · = | ρ K | = 0 and 1 <
i < K : according to the Caseand Case 3 in Lemma 3, we can have
[Equation]
and 0 < λ i = ρ 21 − iK ρ 21 < ρ 21 = λ max , which violates the assumption (13).
• If | ρ 1 | = · · · = | ρ i | > | ρ i +1 | = · · · = | ρ K | (cid:54) = 0 and 1 <
i < K : according to Case 3 in Lemma 3, we have
[Equation]
and the remaining two eigenvalues are the same to those of D =
(cid:20) ρ
[Equation]
K
[Equation]
( − 1 K ) (cid:20)
[Equation]
√
[Equation]
iρ K
[Equation]
iρ
[Equation]
iρ K (cid:3) .
According to (12) in Lemma 3, we can obtain
[Equation]
Combing them together, we can have
[Equation]
thus, 0 = λ K <
λ
i < λ 1 = λ max , which violates the assumption (13).
• If | ρ 1 | > | ρ i | > | ρ K | for some 1 <
i < K : Suppose | ρ 1 | = · · · = | ρ m | , | ρ i | = · · · = | ρ i + n − 1 | and | ρ K − t +1 | = · · · = | ρ K | , where m <
i , i + n − 1 < K − t +1 and m, n, t ≥ 1 .
According to the (12), Case 2 and Case 3 in Lemma 3, we can ﬁnd ρ
[Equation]
λ K
=thus, 0 = λ
K <
λ
i + n − 1 < λ m ≤ λ max , which violates the assumption (13).
We complete the proof
[Figure]
A Experiments
In this section, we ﬁrst describe more details about the datasets and the computational resource used in the paper.
Particularly, all CIFAR10, CIFAR100 and miniImageNet are publicly available for academic purpose under the MIT license, and we run all experiments on a single RTX
GPU with 24GB memory.
Moreover, additional experimental results on CIFAR10, CIFAR100 and miniImageNet are presented in Section A.1, Section A.2, and Section A.3, respectively.
A.1 Additional experimental results on CIFAR
In Section 4, we present the test accuracy for different losses function across various different iteration-width conﬁgurations.
Moreover, we further show the N C 1 for different loss functions across different iteration-width conﬁgurations , and we reuse the results of test accuracy in Figurefor better investigation.
The experiment results in Figure 4 consistently show that the value of N Cof training WideResNet50-0.25 for 100 epochs is around three orders of magnitude larger than it of training WideResNet50-2 for 800 epochs, which indicates that the previous conﬁguration setting is much less collapsed than the latter one.
In terms of test accuracy, the maximal difference across different losses for width = 0 . 25 and epochs = 100 conﬁguration is 1 .
037% , which is larger than 0 .
36% for width = 2 and epochs = 800 conﬁguration.
These results support our claim that all losses lead to identical performance, as long as the network has sufﬁcient approximation power and the number of optimization is enough for the convergence to the NC global optimality.
A.2 Additional experimental results on CIFAR
In this parts, we show the additional results on CIFAR100 dataset.
Prevalence of NC Across Varying Training Losses
We show that all loss functions lead to NC solutions during the terminal phase of training on CIFAR100 dataset.
The results on CIFAR using WideResNet50-2 and different loss functions is provided in Figure 5.
We consistently observe that all three NC metrics of FL and MSE converge to a small value as training progresses, and metrics of CE and FL still continue to decrease at the last iteration, because CIFAR100 is more difﬁcult than CIFAR10 and requires networks to be optimized longer.
The decreasing speed of FL is slowest, which is consistent with our global landscape analysis that FL has benign landscape in
[Figure]
[Figure]
the local region near optimality.
These results imply that all losses exhibit NC at the end, regardless of the choice of loss functions.
All Losses Lead to Largely Identical
Performance Same as the results on CIFAR10 dataset, the conclusion on CIFAR100 also holds that all loss functions have largely identical performance once the training procedure converges to the NC global optimality.
In Figure 6, we plot the evolution of the training accuracy, validation accuracy and test accuracy with training progressing, where all losses are optimized on the same WideResNet50-2 architecture and CIFAR100 for 800 epochs.
To reduce the randomness, we average the results from 3 different random seeds per iteration-width conﬁguration, and the test accuracy is reported based on the model with best accuracy on validation set, where we organize the validation set by holding out 10 percent data from the training set.
The results consistently shows that the training accuracy trained by different losses all converge to one hundred percent (reaching to terminal phase), and the validation accuracy and test accuracy across different losses are largely same, as long as the optimization procedure converges to the NC global solution.
In Figure 7, we plot the average N C 1 and test accuracy of different losses under different pairs of width and iterations for CIFAR100 dataset.
The three phenomenon mentioned in
Section 4.2 also exist on CIFAR100 in most cases.
Moreover, the values of N C 1 for width=0.
and epochs=100 conﬁguration are also around three orders magnitude larger than them for width= and epochs=800 conﬁguration and the less collapsed conﬁguration leads to larger difference gap across different loss functions.
While there are some small difference between different losses in width = 2 and epochs = 800 conﬁgurations, We guess that it is because CIFAR100 is much harder than CIFAR10 datasets, and network is not sufﬁciently large and trained not long enough for all losses to achieve a global solution.
A.3 Additional experimental results on miniImageNet
In this parts, we show the additional results on miniImageNet dataset.
We trained WideResNet180.25 and WideResNet18-2 on miniImageNet for 100 epochs and 800 epochs, respectively.
To reduce the randomness, we average the results from 3 different random trials.
The N C 1 and test accu-
[Figure] [Figure]
racy of different loss functions are provided in Figure 8 for comparison.
We consistently observe that the N C 1 metric of all losses converges to a small value as training progress, when the neural network has sufﬁcient approximation power and the training is performed for sufﬁciently many iterations, such as WideResNet18-2 for 800 epochs.
Additionally, the conclusion on miniImageNet also holds that all loss functions have largely identical performance once the training procedure converges to the NC global optimality.
Speciﬁcally, while the last-iteration test accuracy of training
WideResNet18-0.25 for 100 epochs is 0 .
7020 and 0 .
7040 , respectively, the lastiteration test accuracy of training WideResNet18-2 for 800 epochs is 0 .
7932 and 0 .
8020 for CE, MSE, FL and LS, respectively.
The experiment results on miniImageNet also support our claim that ( i ) the test performance may be different across different loss functions when the network is not large enough and is optimized with limited number of iterations, but ( ii ) the test accuracy across different loss are largely identical, once the networks has sufﬁcient capacity and the training is optimized to converge to the NC global solution.
B Proof of CE, FL and LS included in GL
In this section, we prove that CE, FL and LS belong to GL in Section B.1, Section B.2 and Section B.3, respectively.
Before starting the proof for each loss, let us restate the deﬁnition of the GL in Deﬁnition 1:
Deﬁnition 3 (Contrastive property) .
We say a loss function L GL ( z , y k ) satisﬁes the contrastive property if there exists a function φ such that L GL ( z , y k ) can be lower bounded by
[Equation]
where the equality holds only when z j = z (cid:48)
j for all j, j (cid:48) (cid:54) =
k .
Moreover, φ ( t ) satisﬁes t ∗ =
arg min t φ ( t ) + c | t | is unique for any c > 0 , and t ∗ ≤ 0 .
[Equation]
B.1 CE is in GL
In this section, we will show that the CE deﬁned in (3) belongs to the GL deﬁned in Deﬁnition 3.
First, let us rewrite the CE deﬁnition in GL form as following:
[Equation]
where the inequality is due to the log is an increasing and function and exp is a strictly convex function, and it achieves equality only when z j = z j (cid:48) for all j, j (cid:48) (cid:54)
=
k .
Therefore, there exists such a function φ CE to lower bound original CE loss L CE ( z , y k ) as following:
[Equation]
which satisﬁes the condition of (14).
Next, we will show φ CE ( t ) satisﬁes the condition (15).
The ﬁrst-order gradient of φ CE ( t ) is following:
[Equation]
which is an increasing function and greater than 0 for t ∈ R .
Let denote ψ CE ( t ) =
φ CE ( t ) + c | t | , then
• When t ≥ 0 : ∇ ψ CE ( t ) = ∇ φ CE ( t ) + c > 0 , thus the ψ CE ( t ) is an increasing function w.r.t.
t , and the minimizer is achieved when t = 0 .
• When t ≤ 0 : ∇ ψ CE ( t ) = ∇ φ CE ( t ) − c , and ∇ φ CE ( t ) is an increasing function, which achieves minimizer when t = 0 such that ∇ φ CE ( t )
= 1 K .
– if c ≥ 1 K , ∇ ψ CE ( t ) < 0 , and ψ ( t ) is a decreasing function for t ≤ 0 , and the minimizer is achieved when t = 0 ; – if 0 < c ≤ 1 K , there exist such t ∗ such that ∇ ψ CE ( t ) = 0 .
When t < t ∗ , φ CE ( t ) is a decreasing function; and when t ∗ < t ≤ 0 , φ CE ( t ) is an increasing function.
Therefore
, the
minimizer
is
achieved
when
t
=
t
∗
<
Combing them together, we can prove that φ CE satisﬁes the condition of (15).
B.2 FL is in GL
In this section
[Equation]
we will show that the FL deﬁned in (4) belongs to the GL deﬁned in Deﬁnition 3.
let us rewrite the FL deﬁnition in GL form as following:
[Equation]
where the function η ( t ) =
(1 − 1 t ) γ log ( t ) is an increasing function for t ≥ 1 because
[Equation]
Thus
[Equation]
we can ﬁnd the lower bound function by
[Equation]
−
[Equation]
=
[Equation]

[Equation]
 where φ FL ( t ) =
η ( ξ ( t )) and ξ ( t ) =
+ ( K − 1) exp tK − 1 ∈ [1 , K ] , which satisﬁes the condition of (14).
Next
[Equation]
we will show φ FL ( t ) satisﬁes the condition (15).
The ﬁrst-order gradient of φ FL ( t ) is following:
[Equation]
)) ≥
[Equation]
Similarly, by chain rule, the second-order derivation is:
[Equation]

• When t ≥ 0 : ∇ t ψ FL ( t )
= 1 K − 1 ξ ( t )+ c ≥ 0 , thus the ψ CE ( t ) is an increasing function w.r.t.
t , and the minimizer is achieved when x = 0 .
• When t ≤ 0 : ∇ t ψ FL ( t ) =
[Equation]
K − 1 ξ ( t ) − c ≥ 0 .
Moreover, we can ﬁnd ϑ ( ξ ( t )) is a decreasing function w.r.t.
ξ ( t ) and ξ ( t ) is an increasing function w.r.t.
t , therefore, ϑ ( ξ ( t )) is a decreasing function w.r.t.
t .
– If ϑ ( ξ (0))
=
ϑ ( K ) ≥ 0 , then ∇ 2 x ψ ( x ) > 0 for x ≤ 0 , which means that ∇ x ξ ( t ) is an increasing function.
Because ς ( ξ ( −∞ ))
=
ς (1) = 0 , here we need to consider two cases(Please refer to Figure 9)
* if ς ( ξ (0) =
ς ( K ) ≤ c ( K − 1) , then ∇ t ψ FL ( t ) ≥ 0 , that is, ψ FL ( t ) is a decreasing function.
Therefore, the global minimizer is achieved when x =
(the blue curve in Figure 9).
* if ς ( ξ (0) =
ς ( K ) ≥ c ( K − 1) , so ψ FL ( x ) will ﬁrst decrease and then increase.
Therefore the global minimizer is unique (the red curve in Figure 9).
– If ϑ ( ξ (0)) =
ϑ ( K ) < 0 , then for t ∈ [ −∞ , t (cid:48) ] , ∇ t ψ FL ( x ) is an increasing function w.r.t.
t ; for t ∈ [ t (cid:48) , 0) , ∇ t Φ FL ( t ) is a decreasing function w.r.t.
t .
Here we need to consider three cases(please refer to Figure 10): * if ς ( ξ ( t (cid:48) ))
≤ c ( K − 1) , then ∇ t ψ FL ( t ) ≤ 0 , that is, ψ FL ( t ) is a decreasing function.
Therefore, the global minimizer is achieved when x =
(the green curve in Figure 10).
* if ς ( ξ (0))
=
ς ( K ) ≥ c ( K − 1) , so ψ FL ( x ) will ﬁrst decrease and then increase.
Therefore the global minimizer is unique (the red curve in Figure 10).
* if ς ( ξ ( t (cid:48) )) ≥ c ( K − 1) and ς ( ξ (0))
=
ς ( K ) ≤ c ( K − 1) , then ∇ t ψ FL ( t ) = 0 has two solutions t 1 and t 2 .
For t ∈ [ −∞ , t 1 ] , ψ FL ( t ) is an decreasing function w.r.t.
t ; for t ∈ [ t 1 , t 2 ] , Φ FL ( t ) is an increasing function w.r.t.
t ; and for t ∈ [ t 2 , 0) , ψ FL ( t ) is a decreasing function w.r.t.
t .
The unique minimizer is achieved when either t = 0 or t =
t 1 , as long as ψ FL (0) (cid:54) =
ψ
FL ( t 1 ) .
As for the minor case ψ FL (0)
=
ψ FL ( t 1 ) , it requires carefully chosen penalized parameters, which can be omitted (the blue curve in Figure 10).
In conclusion, for focal loss, ψ FL ( t ) has a unique minimum in terms of t ≤ 0 , which satisﬁes the condition of (15).
[Figure] [Figure]
B.3 LS is in GL
In this section, we will show that the LS deﬁned in (5) belongs to the GL deﬁned in Deﬁnition 3.
First, let us rewrite the LS deﬁnition in GL form as following:
[Equation]
log (cid:32)(cid:80)
[Equation]
 
K
[Equation]
j =
[Equation]
−
[Equation]
where the inequality is due to the log is an increasing and function and exp is a strictly convex function, and it achieves equality only when z j = z j (cid:48) for all j, j (cid:48) (cid:54)
=
k .
Therefore, there exists such a function φ LS to lower bound original LS loss L LS ( z , y k ) as following:
[Equation]
which satisﬁes the condition of (14).
Next, we will show φ LS ( t ) satisﬁes the condition (15).
The ﬁrst-order gradient of φ LS ( t ) is following:
[Equation]
Let denote ψ LS ( t ) =
φ LS ( t ) + c | t | , then
• When t ≥ 0 : ∇ ψ LS ( t )
= ∇ φ LS ( t ) + c > 0 due to ∇ φ LS ( t ) ≥ 0 for t > 0 , thus the ψ LS ( t ) is an increasing function w.r.t.
t , and the minimizer is achieved when x = 0 .
• When t ≤ 0 : ∇ ψ LS ( t ) =
∇ φ LS ( t ) − c , and ∇ φ LS ( t ) is an increasing function, which achieves minimizer when t = 0 such that φ LS ( t ) = 1 − αK .
– if c ≥ 1 − αK , ∇ ψ LS ( t ) < 0 , and ψ ( t ) is a decreasing function for t ≤ 0 , and the minimizer is achieved when t = 0 ; – if 0 < c ≤ 1 − αK , there exist such t ∗ such that ∇ ψ LS ( t ) = 0 .
When t < t ∗ , φ LS ( t ) is a decreasing function; and when t ∗ < t ≤ 0 , φ LS ( t ) is an increasing function.
Therefore, the minimizer is achieved when t = t ∗ <
Combing them together, we can prove that φ LS satisﬁes the condition of (14).
C Proof of Theorem 1 for GL
In this part of appendices, we prove Theorem 1 in Section 3 that we restate as follows.
Theorem 3 (Global Optimality Condition of GL) .
Assume that the number of classes K is smaller than feature dimension d , i.e., K < d , and the dataset is balanced for each class, n = n 1 = · · · = n K .
Then any global minimizer ( W (cid:63) , H (cid:63) , b (cid:63) ) of min W ,
[Equation] [Equation]
16) with
[Equation]
L
[Equation]
) satisfying the the Contrastive property in Deﬁnition 3 ; (18) obeys the following
[Equation]
and b (cid:63) =
b (cid:63)
[Equation]
and h (cid:63)i
[Equation]
∀
[Equation]
[ n
[Equation]
, where either b (cid:63) = 0 or λ b = 0 , and the matrix W (cid:63) (cid:62) is in the form of K -simplex ETF structure deﬁned in Deﬁnition 2 in the sense that
[Equation]
C.1 Main Proof
At a high level, we lower bound the general loss function based on the contrastive property (14), then check the equality conditions hold for the lower bounds and these equality conditions ensure that the global solutions ( W (cid:63) , H (cid:63) , b (cid:63) ) are in the form as shown in Theorem 3.
Proof of Theorem 3.
First by Lemma 5, Lemma 6 and Lemma 7, we know that any critical point ( W , H , b ) of f in (16) satisﬁes
[Equation]
For the rest of the proof, let G i =
∇
Z
i =
WH
i g ( W H i + b 1 (cid:62) ) and τ = − ∇ g ( WH + b 1 (cid:62) )
λ b to simplify the notations, and thus (cid:107) H (cid:107) 2 F = λ
[Equation]
(cid:107) W (cid:107)
[Equation]
[Equation]
G i and b = τ 1 .
We will ﬁrst provide a lower bound for the general loss term g ( W H + b 1 (cid:62) ) according to the
Deﬁnition 3, and then show that the lower bound is attained if and only if the parameters are in the form described in Theorem 3.
By Lemma 8, we have
[Equation]
where φ is lower bound function satisfying the Deﬁnition 3, ρ (cid:63)
[Equation]
K
[Equation]
Furthermore, by Lemma 8, we know that ¯ Z (cid:63)i
[Equation]
− ρ (cid:63)
[Equation]
I K − 1 K 1 K 1 (cid:62) K (cid:1) , which satisﬁes the K -simplex ETF structure deﬁned in Deﬁnition 2.
In
Lemma 9, we show the any minimizer ( W (cid:63) , H (cid:63) , b (cid:63) ) of f ( W , H , b ) has following properties via check the equality conditions hold for the lower bounds in Lemma 8:
(a) (cid:107) w (cid:63) (cid:107) 2 = (cid:13)(cid:13) w (cid:63)
(cid:13)(cid:13)
=
(cid:13)(cid:13) w (cid:63)
(cid:13)(cid:13) 2 = · · ·
[Equation]
(cid:13)(cid:13)
[Equation]
(cid:63)K (cid:13)(cid:13) 2 ; (b) b (cid:63) =
b (cid:63) 1 , where either b (cid:63) = 0 or λ b = 0 ;
(c) h (cid:63)i :=
1 K (cid:80) Kj =1 h (cid:63)j,i = 0 , ∀ i ∈ [ n ] , and (cid:113) λ W λ H n w k(cid:63)
=
h (cid:63)k,i , ∀ k ∈
[ K ] , i ∈ [ n ] ; (d) W W (cid:62) =
(cid:107) w (cid:63) (cid:107) 22 K − 1 K (cid:0)
I K − 1 K 1 K 1 (cid:62) K (cid:1) ;
The proof is complete.
C.2 Supporting Lemmas
We ﬁrst characterize the following balance property between W and H for any critical point ( W , H , b ) of our loss function:
Lemma 5.
Let ρ = (cid:107) W (cid:107) 2 F .
Any critical point ( W , H , b ) of (16) obeys
[Equation]
and ρ = (cid:107)
[Equation]
Proof of Lemma 5.
By deﬁnition, any critical point ( W , H , b ) of (16) satisﬁes the following:
[Equation]
Left multiply the ﬁrst equation by W (cid:62) on both sides and then right multiply second equation by
H (cid:62) on both sides, it gives
[Equation]
Therefore, combining the equations above, we obtain
[Equation]
Moreover, we have
[Equation]
as desired.
Next, we characterize the following relationship per group between W and H i for i ∈ [ n ] for any critical ( W , H , b ) of (16) satisﬁes the following:
Lemma 6.
Let G i = ∇ Z i
=
WH
i g ( W H i + b 1 (cid:62) ) .
Any critical point ( W , H , b ) of (16) obeys
[Equation]
Proof of Lemma 5.
By deﬁnition, any critical point ( W , H , b ) of (16) satisﬁes the following:
[Equation]
as desired.
We then characterize the following isotropic property of b for any critical point ( W , H , b ) of our loss function:
Lemma 7.
Let τ = − ∇ g ( WH + b 1 (cid:62) )
λ b .
Any critical point ( W , H , b ) of (16) obeys b
= τ 1 .
Proof of Lemma 7.
By deﬁnition, any critical point ( W , H , b ) of (16) satisﬁes the following:
[Equation]
26) as desired.
Lemma 8.
Let W =
[Equation]
N ,
[Equation]
[ h 1 ,i · · · h K,i ] ∈ R d × K , ¯ Z = W H ∈ R d × N , N = nK , and b = τ 1 .
Given g
( W H + b 1 (cid:62) K ) deﬁned in (17) , for any critical point ( W , H , b ) of (16) , it satisﬁes
[Equation]
where φ is lower bound function satisfying the Deﬁnition 3, ρ (cid:63)
= arg min ρ φ ( ρ )+ K √ nλ W λ H | ρ | , and ¯ Z (cid:63)
= W (cid:63) H (cid:63) .
Proof of Lemma 8.
With ¯ Z i =
W H
i , and (cid:13)(cid:13)
¯ Z i (cid:13)(cid:13) 2 = σ max i , we have the following lower bound for f ( W , H , b ) as
[Equation]
F + λ H 2 (cid:107) H i (cid:107) 2 F (cid:19) + λ b 2 (cid:107) b (cid:107)≥ n (cid:88) i = (cid:16) g ( ¯ Z i + b 1 (cid:62) )
+ (cid:112) λ W λ H /n
(cid:13)(cid:13)
¯ Z i (cid:13)(cid:13) ∗ (cid:17) + λ b 2 (cid:107) b (cid:107)≥ n (cid:88) i = (cid:32) g
( ¯ Z i + b 1 (cid:62) ) + (cid:112) λ W λ H /n
(cid:13) (cid:13) ¯ Z (cid:13) (cid:13) 2 F (cid:13)(cid:13)
¯ Z i (cid:13)(cid:13)(cid:33) + λ b 2 (cid:107) b (cid:107)= n (cid:88) i = (cid:32) g
( ¯ Z i + b 1 (cid:62) ) + (cid:112) λ W λ H /n
σ max i (cid:13)(cid:13)
¯ Z i (cid:13)(cid:13) 2 F (cid:33) + λ b 2 (cid:107) b (cid:107) 22 , where the ﬁrst inequality is from Lemma
[Equation]
and the second inequality becomes equality only when ¯
Z i (cid:54) = 0 and
[Equation]
[Equation]
where σ k ( ¯ Z i ) is the k -th singular value of ¯ Z i .
While we only consider ¯ Z i (cid:54) =
[Equation]
we will show the ¯
Z i = 0 can be included in an uniform form as following proof.
We can further bound f ( W , H , b ) by
[Equation]
where the ﬁrst inequality is from the ﬁrst condition (14) of loss function L and the equality achieves only when ¯ z k,i,j = ¯ z k,i,j (cid:48) for j (cid:54)
=
k, j (cid:48) (cid:54) =
k , and b j − b k = 0 is due to Lemma 7.
If we denote by ρ k,i = (cid:80) Kj
[Equation]
=
k
[Equation]
¯ z k,i,j − ¯ z k,i,k ) /
( K − 1) , then
[Equation]
where the ﬁrst inequality achieves equality only when ¯ z k,i,j = ¯ z k,i,j (cid:48) for j (cid:54)
=
k, j (cid:48) (cid:54)
=
k , and the last line achieves equality only when ¯ z k,i,k
= − K − 1 K ρ k,i , thus ¯ z k,i,j = 1 K ρ k,i for j (cid:54) =
k .
Denoting ρ i
=
[ ρ i,ρ i,· · · ρ i,K ] and diag ( ρ i ) is a diagonal matrix using ρ i as diagonal entries, and supposing | ρ 1 | ≥ | ρ 2 | > · · · > | ρ K | , we can express ¯ Z i as:
[Equation]
and we can extend the expression of (30) as following
[Equation]
which is decouplable if we treat the i -th samples per class as a group, thus we only consider the i -th samples per class.
In the next part, denote ρ (cid:63) =
arg min ρ φ ( ρ ) + ( K − 1) √ nλ W λ H | ρ | .
When K ≥ 3 , according to the Z = − ( I K − 1 K 1 K 1 (cid:62) K ) diag ( ρ i ) , the condition of (29) and
Lemma 4, we know Z has only two possible forms corresponding to two different objective value of (cid:80) Kk =1 ψ ( ρ k ) such that
• | ρ 1 | = | ρ 2 | = · · · = | ρ K | : we can have σ max = | ρ 1 |
and
[Equation]
where the last line holds equality only when | ρ 1 | = | ρ 2 | = · · · = | ρ K | = ρ ∗ .
• | ρ 2 | = · · · = | ρ K | = 0 : we can have σ max =
(cid:112) ( K − 1) /K
| ρ 1 | and K
[Equation]
where the last line holds equality only when | ρ 1 | = · · · = | ρ K | = | ρ (cid:63) | = 0 .
When K = according to the Lemma we can calculate σ max =
[Equation]
21 +
[Equation] [Equation]
, then
[Equation]
where the last line holds equality only when | ρ 1 | = | ρ 2 | = | ρ (cid:63) | .
Combining them together
[Equation]
for K ≥
[Equation]
we can further extend the expression of (32) as following
[Equation]
33) where the last equation is achieved when b = 0 or λ b = 0 .
According to the condition (15) of loss function L that the minimizer ρ (cid:63) of φ ( ρ ) + c | ρ | is unique for any c > 0 , and by denoting
I nK =
[ I K · · ·
I K ] ∈ R K × nK , we have
[Equation] [Equation]
(35) as desired.
Next, we show that the lower bound in (27) is attained if and only if ( W , H , b ) satisﬁes the following conditions.
Lemma 9.
Under the same assumptions of Lemma 8, the lower bound in (27) is attained for any minimizer ( W (cid:63) , H (cid:63) , b (cid:63) ) of (16) if and only if the following hold
[Equation]
and b (cid:63) =
b (cid:63) 1 ,
[Equation]
∀
[Equation]
] , i ∈ [ n ] , and h (cid:63)i := 1 K
K (cid:88)
j = h (cid:63)j,i = 0 , ∀ i ∈ [ n ] , where either b (cid:63) = 0 or λ b = 0 , and the matrix W (cid:63) (cid:62) is in the form of K -simplex ETF structure (see appendix for the formal deﬁnition) in the sense that
[Equation]
The proof of Lemma 9 utilizes the Lemma Lemma 5, Lemma 6 and Lemma 7, and the conditions (33) and the structure of ¯ Z (cid:63) (35) during the proof of Lemma 8.
Proof of Lemma 9.
From the (35), we know that ¯ Z (cid:63)
= ¯ Z (cid:63) 2 = · · · = ¯ Z (cid:63)n and then G (cid:63)i = ∇
¯ Z (cid:63)i = W (cid:63) H
(cid:63)i g ( W (cid:63) H (cid:63)i + b 1 (cid:62) ) is equivalent for i ∈ [ n ] .
Let denote G (cid:63) =
G (cid:63) 1 = G (cid:63) 2 = · · · = G (cid:63)n , the (22) in Lemma 6 can be expressed as:
[Equation]
Therefore, ˜ H (cid:63) =
H (cid:63) 1 = H (cid:63) 2 = · · · = H (cid:63)n , which means the last-layer features from different classes are collapsed to their corresponding class-mean h (cid:63)k, 1 = h
(cid:63)k, 2 = · · · = h (cid:63)k,n , for k ∈ [ K ] .
Furthermore, H (cid:63) H
[Equation]
= n ˜ H
[Equation]
˜ H (cid:63) (cid:62) , combining this with (19) in Lemma 5, we know that
[Equation]
By denoting W (cid:63) =
U W Σ W V (cid:62) W and ˜ H (cid:63) =
U ˜ H Σ
˜ H V (cid:62) ˜ H , where U W , Σ W , V (cid:62) W are the left singular vector matrix, singular value matrix, and right singular vector matrix of W (cid:63) , respectively; and U ˜ H (cid:63) , Σ ˜ H (cid:63) , V (cid:62)
˜ H (cid:63) are the left singular vector matrix, singular value matrix, and right singular vector matrix of ˜ H , respectively, we can get
[Equation]
Therefore, Z (cid:63)i = W (cid:63)
˜ H (cid:63) = (cid:113) λ
W nλ H U W Σ 2 W V (cid:62)
˜ H .
According to the Z i = − ρ (cid:63) ( I K − 1 K 1 K 1 (cid:62) K ) in (34) and ρ (cid:63) ≤ 0 , which is symmetric, thus, U W = V ˜ H , W (cid:63) =
(cid:113) nλ H λ W ˜ H (cid:63) (cid:62) , that is, w (cid:63)k = (cid:113) nλ H λ W h
(cid:63)k,i , ∀ k ∈
[ K ] , i ∈ [ n ] and
Z (cid:63)i
[Equation]
Therefore,
[Equation]
h
[Equation]
∀ i ∈ [ n ] where h (cid:63)i =
(cid:80)
Kk =1 ( h (cid:63)k,i ) and according to the condition of (33) and Lemma 7, b (cid:63) = 0 or λ b = 0 .
D Proof of Corollary 1 and Corollary
Following Theorem 2, we only need to prove convexity for label smoothing and local convexity for focal loss.
For any output (logit) z ∈ R K , deﬁne
[Equation]
Let y smooth ∈ R K be the label vector with 0 ≤
y smooth i ≤ 1 and (cid:80) i
y smooth i = 1 .
The three loss functions can be written as
[Equation]
Some useful properties:
[Equation]
Therefore
[Equation]
the gradient and Hessian of f ( z ) are given by ∇ f ( z ) =
[Equation]
[Equation]
(cid:125) (cid:23) 
Thus, ∇ 2 f ( z ) is PSD when η ( p i ) ≤ 0 and η (cid:48)
( p i ) ≥ 0 for all i , i.e., ξ (cid:48)
( p i ) ≤ 0 , ξ (cid:48)(cid:48)
[Equation]
p i
[Equation]
Now we consider the following cases:
• CE loss with y smooth =
e k and ξ ( t ) =
− log( t ) .
In this case, ξ (cid:48) ( p i ) = − 1 p i and η ( p i ) =
ξ (cid:48) ( p i ) p i = − 1 , and thus
[Equation]
where the inequality can be obtained by the Gershgorin circle theorem.
• Label smoothing with y smooth
=
(1 − α )
e k + α K 1 and ξ ( t ) =
− log( t ) .
In this case, ξ (cid:48) ( p i ) = − 1 p i and η ( p i ) =
ξ (cid:48) ( p i ) p i = − 1 , and thus ∇
[Equation]
y
[Equation]
(cid:0) diag( p ) − pp
[Equation]
= diag( p ) − pp
[Equation]
since (cid:80) Ki
=
y smooth i = 1 .
• Focal loss with y smooth =
e k and ξ ( t ) =
− (1 − t ) β log( t ) .
In this case, ξ (cid:48) ( p i ) =
β (1 − p i ) β − 1 log( p i ) − (
1 − p i )
β p
i , η ( p i ) = ξ (cid:48)
( p i ) p i
=
βp
i (1 − p i ) β − 1 log( p i ) − (1 − p i ) β ≤ 0 , ∀ β ≥ 0 , p i ∈ [0 , 1] , η (cid:48) ( p i ) =
β (1 − p i ) β − 1 log( p i ) − β ( β − 1) p
i (1 − p i ) β − 2 log( p i ) + β (1 − p i ) β − 1 + β (1 − p i )
β −= β (1 − p i )
β − 2 ((1 − βp i ) log( p i )
+ 2(1 − p i ))
≥ β
1 − p i )
β − 2 (log( p i )
+ 2(1 − p i )) .
Thus, η (cid:48) ( p i ) ≥
whenever 0 .
21 ≤ p i ≤ 1 .
The Hessian becomes
[Equation]
which is PSD when 0 .
21 ≤ p k ≤ 1 .