Inducing Neural Collapse in Imbalanced Learning: Do We Really Need a Learnable Classiﬁer at the End of Deep Neural Network?
Abstract
Modern deep neural networks for classiﬁcation usually jointly learn a backbone for representation and a linear classiﬁer to output the logit of each class.
A recent study has shown a phenomenon called neural collapse that the within-class means of features and the classiﬁer vectors converge to the vertices of a simplex equiangular tight frame (ETF) at the terminal phase of training on a balanced dataset.
Since the
ETF geometric structure maximally separates the pair-wise angles of all classes in the classiﬁer, it is natural to raise the question, why do we spend an effort to learn a classiﬁer when we know its optimal geometric structure?
In this paper, we study the potential of learning a neural network for classiﬁcation with the classiﬁer randomly initialized as an ETF and ﬁxed during training.
Our analytical work based on the layer-peeled model indicates that the feature learning with a ﬁxed
ETF classiﬁer naturally leads to the neural collapse state even when the dataset is imbalanced among classes.
We further show that in this case the cross entropy (CE) loss is not necessary and can be replaced by a simple squared loss that shares the same global optimality but enjoys a better convergence property.
Our experimental results show that our method is able to bring signiﬁcant improvements with faster convergence on multiple imbalanced datasets.
Introduction
Modern deep neural networks for classiﬁcation are composed of a backbone network to extract features, and a linear classiﬁer in the last layer to predict the logit of each class.
As widely adopted in various deep learning ﬁelds, the linear classiﬁer has been learnable jointly with the backbone network using the cross entropy (CE) loss function for classiﬁcation problems [19, 10, 13].
A recent study reveals a very symmetric phenomenon named neural collapse, that the last-layer features of the same class will collapse to their within-class mean, and the within-class means of all classes and their corresponding classiﬁer vectors will converge to the vertices of a simplex equiangular tight frame (ETF) at the terminal phase of training on a balanced dataset [28].
A simplex ETF describes a geometric structure that maximally separates the pair-wise angles of K vectors in R d , d ≥ K − 1 , and all vectors have an equal (cid:96) 2 norm.
As shown in Figure 1, when d = K − 1 , the ETF reduces to a regular simplex.
Following studies focus on unveiling the physics of such a phenomenon based on the layer-peeled model (LPM) [5] or unconstrained feature model [24].
They peel off the topmost layer, so the features are independent variables to optimize [41].
Although this toy model is impracticable for application, it inherits the nature of feature and classiﬁer learning in real deep networks.
𝑂
The black spheres are the vertices of the ETF
The “+” and “
” signs in different colors refer to features and classiﬁer vectors of different classes, respectively
Neural collapse
- dicates that the features and vectors are aligned
the
It has been shown that the optimality of LPM satisﬁes under the CE loss with constraints [22, 5, 41, 8], regularization [50], or even no explicit constraint [15].
The other studies turn to analyze the mean squared error (MSE) loss and also derive the neural collapse solution as global optimality [24, 9, 31, 38].
However, all these theoretical results are only valid for balanced training.
Albeit neural collapse is an observed empirical result and has not been entirely understood from a theoretical point, it is intuitive and sensible.
Features collapsing to their means minimize the withinclass variability, while the ETF geometric structure maximizes the between-class variability, so the Fisher discriminant ratio [6, 33] is maximized, which corresponds to an optimal linearly separable state for classiﬁcation.
Several works have shown that neural networks progressively increase the linear separability during training [26, 27, 46].
Therefore, a classiﬁer that satisﬁes the ETF structure should be a “correct” answer for network training.
However, as pointed out by [5], in the training on an imbalanced dataset, the classiﬁer vectors of minor classes will be merged, termed as minority collapse , which breaks up the ETF structure and deteriorates the performance on test data.
So a learnable classiﬁer does not always lead to neural collapse when training on imbalanced data.
With the spirit of Occam’s Razor [36], we raise and study the question:
Do we really need to learn a linear classiﬁer at the end of deep neural network?
Since a backbone network is usually over-parameterized and can align the output feature with any direction, what makes classiﬁcation effective should be the feature-classiﬁer geometric structure, instead of their speciﬁc directions.
In this paper, we propose to initialize the classiﬁer using a randomly generated ETF and ﬁx it during training, i.e., only the backbone is learnable.
It turns out that this practice makes the LPM more mathematically tractable.
Our analytical work indicates that even in the training on an imbalanced dataset, the features will converge to the vertices of the same
ETF formed by the ﬁxed classiﬁer, i.e., neural collapse inherently emerges regardless of class balance.
We further analyze the cross entropy loss function.
We point out the reason for minority collapse from the perspective of imbalanced gradients with respect to a learnable classiﬁer.
As a comparison, our ﬁxed ETF classiﬁer does not suffer from this dilemma.
We also show that the gradients with respect to the last-layer feature are composed of a “pull” term that pulls the feature into the direction of its corresponding classiﬁer vector of the same class, and a “push” term that pushes it away from the other classiﬁer vectors.
It is the pull-push mechanism in the CE loss that makes the features collapsed and separated.
However, in our case, the classiﬁer has been ﬁxed as a “correct” answer.
As a result, the “pull” term is always an accurate gradient towards the optimality, and we no longer need the “push” term that may cause inaccurate gradients.
Inspired by the analyses, we further propose a simple squared loss, named dot-regression (DR) loss, which shares a similar “pull” gradient with the
CE loss, but does not have any “push” term.
It has the same neural collapse global optimality, but is proved to enjoy a better convergence property than the CE loss.
The contributions of this study can be listed as follows:
• We theoretically show that the neural collapse optimality can be induced even in imbalanced learning as long as the learnable classiﬁer is ﬁxed as a simplex ETF.
• Our analysis indicates that the broken neural collapse in imbalanced learning is caused by the imbalanced gradients w.r.t.
a learnable classiﬁer.
• We point out that our ﬁxed ETF classiﬁer no longer relies on the “push” gradient w.r.t feature that is crucial for the CE loss but may be inaccurate.
We further propose a new loss function with only “pull” gradient.
Its better convergence property is theoretically proved.
• Statistical results show that the model with our method converges more closely to neural collapse.
In experiments, our method is able to improve the performance of classiﬁcation on multiple imbalanced datasets by a signiﬁcant margin.
Our method enjoys a faster convergence compared with the traditional learnable classiﬁer with the CE loss.
As an extension, we ﬁnd that our method also works for ﬁne-grained classiﬁcation.
Related Work
Neural Network for Classiﬁcation.
Despite the success of deep learning for classiﬁcation [18, 35, 10, 13, 45, 11], a theoretical foundation that can guide the design of neural architecture still remains an open problem and inspires many studies from different perspectives [7, 23, 20, 44, 1, 14, 42].
The last-layer linear classiﬁer has been relatively transparent.
In most cases, it is jointly optimized with the backbone network.
In long-tailed classiﬁcation, a two-stage method of training backbone and classiﬁer successively is preferred [2, 16, 48].
Some prior studies have shown that ﬁxing the classiﬁer as regular polytopes [29, 30] and Hadamard matrix [12] does not harm the classiﬁcation performance.
Zhu et al.
[50] tried the practice of ﬁxing the classiﬁer as a simplex ETF in experiment, but does not provide any beneﬁt except for the saved computation cost.
We also study the potential of using a ﬁxed classiﬁer throughout the training.
But different from these studies, our proposed practice is proved to be beneﬁcial to imbalanced learning by both theoretical and experimental results.
Neural Collapse.
In [28], neural collapse was observed at the terminal phase of training on a balanced dataset.
Albeit the phenomenon is intuitive, its reason has not been entirely understood, which inspires several lines of theoretical work on it.
Papyan et al.
[28] proved that if features satisfy neural collapse, the optimal classiﬁer vectors under the MSE loss will also converge to neural collapse based on [40].
Some studies turn to a simpliﬁed model that only considers the last-layer features and classiﬁer as independent variables.
They prove that neural collapse emerges under the CE loss with proper constraints or regularizations [41, 5, 22, 50, 15, 8].
Other studies focus on the neural collapse under the MSE loss [24, 9, 31, 38, 49, 32].
In [4], a convex formulation is proposed for a particular network to explain neural collapse.
However, current results are only valid for balanced training.
Inspired by neural collapse, the study [43] modiﬁes the CE loss for imbalanced learning, but still cannot rigorously induce neural collapse.
Our work differs from these studies in that we theoretically show that neural collapse can inherently happen even in imbalanced learning .
We also derive a new loss function that theoretically enjoys a better convergence property than the CE loss.
Preliminaries 3.1 Neural Collapse
Papyan et al. [28] revealed the phenomenon that the last-layer features will converge to their withinclass means, and the within-class means together with the classiﬁer vectors will collapse to the vertices of a simplex equiangular tight frame at the terminal phase of training on a balanced dataset.
Deﬁnition 1 (Simplex Equiangular Tight Frame)
A collection of vectors m i ∈
R
d , i = 1 , 2 , · · · , K , d ≥ K − 1 , is said to be a simplex equiangular tight frame if:
[Equation]
where M = [ m 1 , · · · , m K ] ∈ R d × K , U ∈ R d × K allows a rotation and satisﬁes U T U =
I
K , I K is the identity matrix, and 1 K is an all-ones vector.
All vectors in a simplex ETF have an equal (cid:96) 2 norm and the same pair-wise angle, i.e.,
[Equation]
where δ i,j equals to 1 when i = j and 0 otherwise.
The pair-wise angle − 1 K − 1 is the maximal equiangular separation of K vectors in R d [28].
Then the neural collapse (NC) phenomenon can be formally described as: (NC1) Within-class variability of the last-layer features collapse: Σ W → 0 , and Σ W :=
Avg i,k { ( h k,i − h k )( h k,i − h k ) T } , where h k,i is the last-layer feature of the i -th sample in the k -th class, and h k =
Avg i { h k,i } is the within-class mean of the last-layer features in the k -th class; (NC2) Convergence to a simplex ETF: ˜
h
k
=
( h k − h G ) /
||
h k − h G || , k ∈ [1 , K ] , satisﬁes Eq.
(2), where h G is the global mean of the last-layer features,
i
e., h G =
Avg i,k { h k,i } ; (NC3) Self duality: ˜ h k = w k / || w k || , where w k is the classiﬁer vector of the k -th class; (NC4) Simpliﬁcation to the nearest class center prediction: argmax k (cid:104) h , w k (cid:105) =
argmin k
||
h − h k
|| ,
where h is the last-layer feature of a sample to predict for classiﬁcation.
3.2 Layer-peeled Model
Neural collapse has attracted a lot of researchers to unveil the physics of such an elegant phenomenon.
Currently most studies target the cross entropy loss function that is widely used in deep learning for classiﬁcation.
It is deﬁned as:
[Equation]
where h ∈ R d is the feature output by a backbone network with input x , W =
[ w 1 , · · · , w K ] ∈ R d × K is a learnable classiﬁer, and c is the class label of x .
However, deep neural network as a highly interacted function is difﬁcult to analyze due to its nonconvexity.
A simpliﬁcation is always necessary to make tractable analysis.
For neural collapse, current studies often consider the case where only the last-layer features and classiﬁer are learnable without considering the layers in the backbone network.
It is termed as layer-peeled model (LPM) [5], and can be formulated as 1 :
[Equation]
where h k,i is the feature of the i -th sample in the k -th class, W is a learnable classiﬁer, L CE is the cross entropy loss function deﬁned in Eq.
(3), N is the number of samples, and E H and E W are the (cid:96) 2 norm constraints for feature h and classiﬁer vector w , respectively.
Albeit the LPM cannot be applied to real application problems, it serves as an analytical tool and inherits the learning behaviors of the last-layer features and classiﬁer in deep neural network.
Actually, the learning of a backbone network is through the multiplication between the Jacobian and the gradient with respect to the last-layer features, i.e. , ∂ H ∂ W 1: L − 1 ∂ L ∂ H , where W 1: L − 1 denotes the parameters in the backbone network, and H is the collection of the last-layer features.
It has been shown that the global optimality for the LPM in Eq.
(4) satisﬁes neural collapse in the balanced case [5, 8].
The CE loss with regularization also has a similar conclusion [50].
However, the results in current studies are only valid for training on a balanced dataset , i.e., n k = n, ∀ 1 ≤ k ≤ K .
Main Results 4.1 ETF Classiﬁer
From the neural collapse solution, NC1 minimizes the within-class covariance Σ W , and NC maximizes the between-class covariance Σ B by the ETF structure.
So the Fisher discriminant ratio, deﬁned as Σ − 1 W Σ B , is maximized, which can measure the linear separability and has been used to extract features to replace the CE loss [37].
So we deem an ETF as the optimal geometric structure for the linear classiﬁer.
Considering that a backbone network is usually over-parameterized and can produce features aligned with any direction, in this paper, we study the potential of learning a network with the linear classiﬁer ﬁxed as an ETF, named ETF classiﬁer.
Concretely, we initialize the linear classiﬁer W as a random simplex ETF by Eq.
(1) with a scaling of √ E W as the ﬁxed length for each classiﬁer vector, and only optimize the features H .
In this case, the layer-peeled model (LPM) in Eq.
(4) reduces to the following problem:
[Equation]
where W ∗ is the ﬁxed classiﬁer as a simplex ETF and satisﬁes:
[Equation]
where δ k,k (cid:48) equals to 1 when k = k (cid:48) and 0 otherwise.
We observe that this practice decouples the multiplied learnable variables h k,i and w k of LPM in Eq.
(4), and makes the model in Eq.
(5) a convex problem that is more mathematically tractable.
We term the decoupled LPM in Eq.
(5) as DLPM for short.
We have the global optimality for DLPM in the imbalanced case with the ETF classiﬁer in the following theorem.
Theorem 1 No matter the data distribution is balanced or not among classes (it is allowed that ∃ k, k (cid:48) ∈
[1 , K ] , k (cid:54) =
k (cid:48) , such that n k (cid:29) n k (cid:48) ), any global minimizer H ∗ =
[ h ∗ k,i : 1 ≤ k ≤ K, 1 ≤ i ≤ n k ] of Eq.
(5) converges to a simplex ETF with the same direction as W ∗ and a length of √ E H , i.e.,
[Equation]
which means that the neural collapse phenomenon emerges regardless of class balance.
Proof 1 Please refer to Appendix A for our proof.
[Equation]
Remark 1 As observed in [5], LPM in the extreme imbalance case would suffer from “ minority collapse ”, where the classiﬁer vectors of minor classes are close or even merged into the same vector, which explains the deteriorated classiﬁcation performance of imbalanced training.
As a comparison,
Theorem 1 shows that DLPM with our ETF classiﬁer can inherently produce the neural collapse solution even in the training on imbalanced data.
Although our practice of using a ﬁxed ETF classiﬁer simpliﬁes the problem, it actually brings theoretical merits that also get validated in our long-tailed classiﬁcation experiments.
4.2 Rethinking the Cross Entropy Loss
In this subsection, we rethink the CE loss L CE from the perspective of gradients with respect to both feature and classiﬁer to analyze its learning behaviors.
4.2.1 Gradient w.r.t Classiﬁer
We ﬁrst analyze the gradient of L CE w.r.t a learnable classiﬁer W =
[ w 1 , · · · , w K ] ∈ R d × K :
[Equation]
where p k ( h ) is the predicted probability that h belongs to the k -th class.
It is calculated by the softmax function and takes the following form in the CE loss:
[Equation]
We make the following deﬁnitions:
[Equation]
where
[Equation]
It reveals that the negative gradient w.r.t w k is decomposed into two terms.
The “pull” term F ( w ) pull pulls w k towards the directions of the features of the same class, i.e. , h k,i , while the “push” term
F ( w ) push pushes w k away from the directions of the features of the other classes, i.e. , h k (cid:48) ,i , ∀ k (cid:48) (cid:54) =
k .
Remark 2 Note that F ( w ) pull and F ( w ) push have different magnitudes.
When the dataset is in extreme imbalance, the direction of − ∂ L CE ∂ w k for a minor class is dominated by the push term F ( w ) push because n k is small, while N − n k is large.
In this case, the classiﬁer vectors of two minor classes are optimized towards nearly the same direction and would be close or merged after training.
So the deteriorated performance of classiﬁcation with imbalanced training data results from the imbalanced gradient w.r.t a learnable classiﬁer in the CE loss.
As a comparison, our proposed practice of ﬁxing the classiﬁer as an ETF does not suffer from this dilemma.
4.2.2 Gradient w.r.t Feature
The gradient of CE loss in Eq.
(3) with respect to h is:
[Equation]
where c is the class label of h , and p k ( h ) is the probability that h belongs to the k -th class as deﬁned in Eq.
It is shown that Eq.
(12) has a similar form to that of Eq.
The negative gradient − ∂ L CE ∂ h can also be decomposed as the addition of “pull” and “push” terms deﬁned as:
[Equation]
The “pull” term F ( h ) pull pulls h towards the classiﬁer vector of the same class, i.e., w c , while the “push” term F ( h ) push pushes h away from the other classiﬁer vectors, i.e., w k , ∀ k (cid:54)
=
c .
Remark
We note that Eq.
(11) and Eq.
(13) have a similar form and are very symmetric.
The “pull” terms F ( w ) pull and F ( h ) pull force w and h of the same class to converge to the same direction, which is in line with (NC1) and (NC3).
The “push” terms F ( w ) push and F ( h ) push make them of different classes separated, which is in line with (NC2).
Generally, (NC1)-(NC3) can lead to (NC4).
We remark that it is the “pull-push” mechanism in the CE loss that leads to the neural collapse solution in the case of training on balanced data.
The “push” gradient F ( h ) push in Eq.
(13) is crucial for a learnable classiﬁer.
However, in our case where the classiﬁer has been ﬁxed as an ETF, the “push” gradient F ( h ) push is unnecessary.
As shown in
Figure 2, the “pull” gradient F ( h ) pull in our case is always accurate towards the optimality w ∗ c , which corresponds to the neural collapse solution.
But the “push” gradient does not necessarily direct to the optimality.
So, we no longer rely on the “push” term F ( h ) push that may cause deviation.
It inspires us to develop a new loss function speciﬁed for our ETF classiﬁer.
4.3 Dot-Regression Loss
We consider the following squared loss function:
[Equation] [Equation]
where c is the class label of h , W ∗ is a ﬁxed ETF classiﬁer, and E W and E H are the (cid:96) 2 -norm constraints (predeﬁned and not learnable ) given in Eq.
It performs regression between the dot product of h and w ∗ c and the multiplication of their lengths.
We term this simple loss function as dot-regression (DR) loss.
Its gradient with respect to h takes the form:
[Equation]
15) where cos ∠ ( h , w ∗ c ) denotes the cosine similarity between h and w ∗ c .
w !
∗ h −𝜕ℒ𝜕h !
= (1−𝑝 !
(h))w !
∗ −𝜕ℒ𝜕h
# = −𝑝 # (h)w #∗ −𝜕ℒ𝜕h $
= −𝑝 $ (h)w $∗ w #∗ w $∗ w !
∗ h w #∗ w $∗ −𝜕ℒ𝜕h =
(1−cos∠(h,w !∗ ))
w !
∗ (CE) (DR) (pull) (push) (push) (pull)
We see that the gradient has a similar form to the ﬁrst term in Eq.
(12), which plays the role of “pull”, but has no “push” term.
It is easy to identify that if we replace the CE loss in the decoupled layer-peeled model (DLPM) deﬁned in Eq.
(5) with the DR loss deﬁned in Eq.
(14), the same global minimizer Eq.
(7) still holds.
The global optimality happens when L DR ( h ∗ , W ∗ )
=
and cos ∠ ( h ∗ , w ∗ c ) = 1 accordingly.
Then we give a formal analysis of the convergence properties of both CE and DR loss functions in the DLPM.
Deﬁnition
Given δ > 0 , for any h satisfying ||
h − h ∗ || ≤ δ and ||
h
|| 2 =
E H , the η − regularity number of function L ( h ) is deﬁned by the convergence rate of the projected gradient method.
That is, there exists η h ∈ [0 , 1] such that:
[Equation]
where γ is the learning rate such that η h is as small as possible.
Note that η h is decided by h .
For many problems, we cannot ﬁnd its uniform upper bound η h ≤ ¯ η < 1 .
The smaller η h is, the better property the loss function has.
Theorem 2 Assume that given a small δ > 0 , when || h − h ∗ || ≤ δ , p ( h ) deﬁned in Eq.
(9) satisﬁes that 2 p k ( h ) =
(1 − p c ( h )) /
( K − 1) , ∀ k (cid:54) =
c , where c is the label of h .
When optimizing the DLPM deﬁned in Eq.
(5) with the CE and DR loss funcitons, for any ﬁxed learning rate γ , we have:
[Equation]
where η ( CE ) h and η ( DR ) h are the η -regularity numbers of the CE and DR loss functions, respectively.
Proof 2 Please refer to Appendix B for our proof.
[Equation]
Eq.
(16) indicates that the DR loss has a better convergence property when h is close to h ∗ .
In implementations, we train a backbone network with our ETF classiﬁer and DR loss.
To induce balanced gradients w.r.t the backbone network parameters, we deﬁne the length of each classiﬁer vector according to class distribution.
Please refer to Appendix C for the complete implementation details.
In experiments, we also compare our method with the CE loss weighted by class distribution.
Experiments
In experiments, we ﬁrst make empirical observations of neural collapse convergence in the imbalanced training with and without our method, and then compare the performances on long-tailed classiﬁcation.
As an extension, we surprisingly ﬁnd that our method is also able to improve the performance of ﬁne-grained classiﬁcation, which can be deemed as another imbalanced problem where a majority of features are close to each other.
The datasets and training details are described in Appendix D.
[Figure]
[Table]
5.1 Empirical Results
Following [28], we calculate statistics during training to show the neural collapse convergence.
We ﬁrst compare the averages and standard deviations of two cosine similarities, cos ∠ ( h c − h g , h k − h g ) and cos ∠ ( h c − h g , w k ) , where h g is the global mean, for all pairs of different classes ( c, k ) , c (cid:54) =
k , with and without our method.
As shown in Figure 3, the averages of both cos ∠ ( h c − h g , h k − h g ) and cos ∠ ( h c − h g , w k ) converge to a negative value near zero.
It is consistent with neural collapse that the feature means or classiﬁer vectors of different classes should have a cosine similarity of − 1 K − 1 .
However, their standard deviations are much smaller when our method is used.
It indicates that the models with our method converge to neural collapse more closely.
We further calculate the averages of cos ∠ ( h c − h g , w c ) , ∀ 1 ≤ c ≤ K , and || ˜ W − ˜ H
|| 2 F in
Figure 4 and 5 in Appendix
E.
It reveals that the model using our method generally has a higher cos ∠ ( h c − h g , w c ) and a lower || ˜ W −
˜
H
|| 2 F , which indicates that the feature means and classiﬁer vectors of the same class are better aligned.
We observe no advantage of ResNet on STL-10 and
DenseNet on CIFAR-100 in Figure 4 and 5.
In Table 2, we see that the two cases
5.2 Performances on Long-tailed Classiﬁcation
We conduct an ablation study with ResNet on CIFAR-10.
As shown in Table 1, when we replace the learnable classiﬁer with our ﬁxed ETF classiﬁer, the performances get improved for the imbalance ratio τ of 0.01 and 0.02 without Mixup [47].
They also achieve comparable results for τ = 0 .and
the balanced setting ( τ = 1 ).
However, only using the ETF classiﬁer does not work for the extreme imbalance case where τ = 0 .
Besides, it is not compatible with Mixup, which is a
[Table] [Table]
[Table]
strong augmentation tool to alleviate the bias brought by adversarial training samples.
When the DR loss that is speciﬁcally designed for the ETF classiﬁer is used, we achieve signiﬁcant performance improvements for τ of 0.005, 0.01, and 0.02 both with and without Mixup.
Generally our proposed
ETF classiﬁer with the DR loss has better performances on multiple long-tailed cases than the learnable classiﬁer with the original or weighted CE loss.
In Table 2, the advantage of our method is further veriﬁed on more datasets with ResNet and DenseNet.
Concretely, we achieve an average improvement of 2.0% for ResNet and 1.1% for DenseNet.
We also test our method on the large-scale dataset, ImageNet-LT [21], which is an imbalanced version of the
ImageNet dataset [34].
As shown in Table 3, we compare our method with baseline by training the models for different epochs.
We observe that the superiority of our method is more remarkable when training for less epochs.
It can be explained by the fact that our method directly has the classiﬁer in its optimality and optimizes the features towards the neural collapse solution, while the learnable classiﬁer with the CE loss needs a sufﬁcient training process to separate classiﬁer vectors of different classes.
So our method can be preferred when fast convergence or limited training time is required.
The accuracy curves in training for the results in Table 3 are shown in Figure 6 in Appendix E.
5.3 Performances on Fine-grained Classiﬁcation
Fine-grained classiﬁcation can also be deemed as an imbalanced problem as the features of multiple classes are close to each other.
We surprisingly ﬁnd that our method is also helpful for ﬁne-grained classiﬁcation even though most of the analytical work is conducted under the case of class imbalance.
As shown in Table 4, our method achieves 0.7%-0.8% accuracy improvements on CUB-200-2011.
Conclusion
In this paper, we study the potential of training a network with the last-layer linear classiﬁer randomly initialized as a simplex ETF and ﬁxed during training.
This practice enjoys theoretical merits under the layer-peeled analytical framework.
We further develop a simple loss function speciﬁcally for the
ETF classiﬁer.
Its advantage gets veriﬁed by both theoretical and experimental results.
We conclude that it is not necessary to learn a linear classiﬁer for classiﬁcation networks, and our simpliﬁed practice even helps to improve the long-tailed and ﬁne-grained classiﬁcation performances with no cost.
Our work may help to further understand neural collapse and the neural network architecture for classiﬁcation.
Code address: https://github.com/NeuralCollapseApplications/ImbalancedLearning.
Limitations and societal impacts are discussed in Appendix F. As suggested by a reviewer of this paper, we compare with [5, 30, 50] in more details in Appendix G.
Acknowledgments and Disclosure of Funding
Z. Lin was supported by the NSF China (No.s 62276004 and 61731018), NSFC Tianyuan Fund for
Mathematics (No. 12026606), and Project 2020BD006 supported by PKU-Baidu Fund.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] See the last paragraph in Section 1.
(b) Did you describe the limitations of your work?
[Yes] See Appendix F. (c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Appendix F. (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] We have read the guidelines and ensured that our paper conforms to them.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Theoremand 2.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix A and B. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See the supplementray material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Appendix D. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Tabel 1.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix D. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] We cite their papers, as shown in Table 2, 3, and 4.
(b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Inducing Neural Collapse in Imbalanced Learning: Do We Really Need a Learnable Classiﬁer at the End of Deep Neural Network?
(Appendix)
A Proof for Theorem
Note that the constraint constrained optimization problem of Eq.
(5) in our case is separable.
We consider the k -th ( 1 ≤ k ≤ K and K ≥ 2 ) problem as:
[Equation]
where W ∗ is the ﬁxed ETF classiﬁer.
The problem above is convex as the objective is a sum of afﬁne functions and log-sum-exp functions with convex constraints.
We have the Lagrange function as:
[Equation]
where µ i is the Lagrange multiplier.
We have its gradient with respect to h k,i as:
[Equation]
First we consider the case when µ i = 0 .
∂
˜ L/∂ h k,i = 0 gives the following equation:
[Equation]
Multiplying w ∗ k by both sides of the equation, we should have:
[Equation] [Equation]
which contradicts with p j > 0 , ∀ 1 ≤ j ≤ K when the (cid:96) 2 norm of h k,i is constrained and W ∗ has a ﬁxed (cid:96) 2 norm.
So we have µ i > 0 and according to the KKT condition:
[Equation]
Then we have the equation:
[Equation]
where h ∗ k,i is the optimal solution of h k,i .
Multiplying w ∗ j (cid:48)
( j (cid:48) (cid:54) =
k ) by both sides of Eq.
(23), we get:
[Equation]
Since p j (cid:48) > 0 and K − 1 > 0 , we have (cid:104) h ∗ k,i , w ∗ j (cid:48) (cid:105) < 0 .
Then for any pair j, j (cid:48) (cid:54) =
k , we have:
[Equation] [Equation]
Considering that the function f ( x ) = exp( x ) /x is monotonically increasing when x < 0 , we have :
[Equation]
where C and p are constants.
From Eq.
(24), we have:
[Equation] [Equation]
and
[Equation] [Equation]
From Eq.
(23), we have:
[Equation]
k,i
[Equation] [Equation]
The ETF classiﬁer deﬁned in Eq.
(1) satisﬁes that (cid:80)
Ki w ∗
i = 0 .
Given that p j = p, ∀ j (cid:54) =
k and Eq.
(29), we have:
[Equation] [Equation]
which indicates that h ∗ k,i has the same direction as w ∗ k .
Its length has been given in Eq.
Then we have:
[Equation]
and
[Equation]
which is equivalent to Eq.
(7) and concludes the proof.
[Equation]
B Proof for Theorem
We would like to show that the η h of L DR is always smaller than that of L CE given h t is close to h ∗ .
For the DR loss:
Since || h tk,i − h ∗ k,i || ≤ δ and h ∗ k,i is aligned with w ∗ k , we let || h
k,i ||
= E H and cos ∠ ( h 0 k,i , w ∗ k )
≥for
any k ∈ [1 , K ] , i ∈ [1 , n k ] .
For the DR loss in (14), the projected SGD takes the following step at time t + 1 with the sample i in the class k :
[Equation]
34) where Proj E H is the orthogonal projection onto the ball { h : ||
h
|| 2 ≤ E H } .
Suppose that at the t -th iteration ||
h
tk,i || 2 =
E H and cos ∠ ( h tk,i , w ∗ k ) ≥ 0 , and one has:
[Equation]
which means || h t +1 k,i || 2 = E H .
It is also easy to identify cos ∠ ( h t +1 k,i , w ∗ k ) ≥ 0 .
So for all time t ≥in the sequence from h 0 k,i to h ∗ k,i , we have ||
h tk,i ||
= E H and cos ∠ ( h tk,i , w ∗ k ) ≥ 0 .
By the non-expansiveness of projection, one has the following convergence:
[Equation]
where a = holds because || h tk,i −
h
[Equation]
k,i || 2 = 2 E H (1 − cos ∠ ( h tk,i , w ∗ k )) .
When γ = √ E H √ E W , we have:
[Equation]
Then we get that the η h -regularity number of the DR loss is:
[Equation]
For the CE loss:
On the other hand, for the CE loss in (3), the projected SGD takes the following step at time t +with the sample i in the class k :
[Equation]
By the non-expansiveness of projection, one has the following convergence:
[Equation]
where we have:
[Equation]
(cid:11) − 2 γ (cid:42) (1 − p k ) γ w ∗ k ,
K (cid:88)
j
(cid:54) =
k p j w ∗ j (cid:43) + 2 γ (cid:42) h ∗ k,i − h
tk,i ,
K (cid:88)
j
(cid:54) =
k p j w ∗ j (cid:43) =
2 γ (1 − p k )  (cid:10) h
tk,i − h ∗ k,i , w ∗ k (cid:11) + γE W K −
K (cid:88)
j
(cid:54) =
k p j 

+ 2 γ (cid:42) h ∗ k,i − h
tk,i ,
K (cid:88)
j
(cid:54) =
k p j w ∗ j (cid:43) =
2 γ (1 − p k ) (cid:18)(cid:10)
h t k,i , w ∗ k (cid:11) − (cid:112) E H E W + γE W (1 − p k ) K −(cid:19) − 2 γ h t k,i
K (cid:88)
j
(cid:54) =
k p j w ∗ j − 2 γ (1 − p k )
K −(cid:112) E H E W = 2 γ (1 − p k ) (cid:10) h
tk,i , w ∗ k (cid:11) −
γ h
tk,i
K (cid:88)
j
(cid:54) =
k p j w ∗ j − 2 γ (1 − p k )
K
K −(cid:112) E H E W + 2 γ 2 (1 − p k ) 2 E W
K −.
Since (cid:80) Ki w ∗
i
= 0 for an ETF classiﬁer,
and we have assumed in Theorem 2 that p i = p
j = 1 − p k
K − 1 , ∀ i, j (cid:54)
=
k , then we have:
[Equation]
where a = follows from the assumption that p i =
p
j = 1 − p k K −
[Equation]
and b = holds because (cid:68) h
tk,i , (cid:80)
Ki w ∗
i (cid:69) = 0 .
Then we have :
[Equation]
E W
K −= − 2 (
[Equation]
p k )
K K
[Equation]
1 γ (cid:112) E
[Equation] [Equation]
, (40) and (cid:13)(cid:13)(cid:13) h t +1 k,i − h ∗ k,i (cid:13)(cid:13)(cid:13)≤ (cid:13)(cid:13)
h
tk,i − h ∗ k,i (cid:13)(cid:13)
[Equation]
2 K √ E
[Equation]
E
[Equation] [Equation]
(cid:0)
[Equation]
(cid:0)
[Equation]
(cid:1)(cid:1)
+ γ
[Equation] [Equation]
[Equation]
+ (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) γ
K (cid:88)
[Equation]
(cid:54)
[Equation]
k p
[Equation]
w
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
[Equation] [Equation]
W
K
[Equation]
=2 E H (cid:0) 1 − cos ∠ ( h tk,i , w ∗ k ) (cid:1) − E H (cid:0) 1 − cos ∠ ( h tk,i
[Equation]
w ∗ k )
(cid:1)+ E H (cid:0) 1 − cos ∠ ( h tk,i , w ∗ k ) (cid:1)
[Equation]
2 K √
[Equation]
K
[Equation]
[Equation] [Equation]
(cid:0)
[Equation]
i
[Equation]
(cid:1)(cid:1)
+ γ 2 ( p k − 1) 2 E W + γ
(1 − p
[Equation]
Let γ (1 − p k ) KK − 1 =
s k
[Equation]
and we consider the problem:
[Equation] [Equation]
When
[Equation]
, the objective in
Eq.
(42) is minimized.
Following Eq.
(41) we get the optimal bound for CE as:
[Equation]
which is the same as DR.
However, in this case γ varies with h tk,i and cannot be a constant as deﬁned in Deﬁnition 2.
For any ﬁxed learning rate γ , the objective in Eq.
(42) is larger than 0.
So we have the η h -regularity number of the CE loss:
[Equation]
and conclude the proof.
[Equation]
C Implementation Details
In implementations, we train a backbone network with our proposed ETF classiﬁer and DR loss.
For small datasets such as CIFAR, SVHN, and STL, we simply perform an (cid:96) 2 normalization for the features output from the backbone network, which means √ E H = 1 .
For large datasets, such as
ImageNet, (cid:96) 2 normalization will induce training instability due to the large dimensionality.
We add an (cid:96) 2 regularization term of the output features instead, to ensure a constrained feature length.
Our analytical work has shown that using a ﬁxed ETF classiﬁer does not suffer from the imbalanced gradient w.r.t classiﬁer.
In implementations, we also consider the gradient w.r.t the backbone network parameters W 1: L − 1 :
[Equation]
Then we have:
[Equation]
where σ max denotes the largest singular value of the Jacobian ∂ h k,i ∂ W 1: L − 1 , ∀ 1 ≤ k ≤ K, 1 ≤ i ≤ n k , (cid:112) E w k is the length of the k -th classiﬁer vector, and (cid:13)(cid:13)(cid:13)
∂ L DR ∂ h k,i (cid:13)(cid:13)(cid:13) 2 ≤ 2 (cid:112) E w k by Eq.
Although we cannot realize a balanced gradient w.r.t W 1: L − 1 among classes, we can balance the upper bound of its gradient norm of each class by controlling (cid:112) E w k .
In implementations, we set (cid:112) E w k = NKn k , which is equivalent to performing a weighted loss function on different classes.
In experiments, we also compare our method with the weighted CE loss for fair comparison.
D Datasets and Training Details
We conduct long-tailed classiﬁcation experiments on the four datasets, CIFAR-10, CIFAR-100,
SVHN, and STL-10, with two architectures, ResNet-32 and DenseNet with a depth of 150, a growth rate of 12, and a reduction of 0.5.
All models are trained with the same training setting.
Concretely, we train for 200 epochs, with an initial learning rate of 0.1, a batchsize of 128, a momentum of
[Figure] [Figure]
We also conduct long-tailed classiﬁcation experiments on ImageNet-LT with ResNet-50.
We train all models for 90, 120, 150, and 180 epochs, with a batchsize of 1024 among 8 NVIDIA A-100 GPUs.
Following [48] we use the SGD optimizer with a momentum of 0.9 and a weight decay of 5 e − 4 .
The initial learning is 0.1 and decays following the cosine annealing schedule.
We conduct ﬁne-grained classiﬁcation experiments on CUB-200-2011 with ResNet-34, ResNet-50, and ResNet-101.
The ResNet backbone networks are pre-trained on ImageNet.
We train forepochs on CUB-200-2011 with a batchsize of 64 and an initial learning rate of 0.04, which is dropped by 0.1 at epoch 90, 180, and 270.
The standard data normalization and augmentation are adopted.
Other training settings are the same as the long-tailed experiments.
E Additional Empirical and Experimental Results
Empirical Results.
We provide more empirical results that have been discussed in Section 5.1.
We calculate the averages of cos ∠ ( h c − h g , w c ) , ∀ 1 ≤ c ≤ K , and || ˜ W − ˜ H
|| 2 F in Figure 4 and 5.
It reveals that the model using our method generally has a higher cos ∠ ( h c − h g , w c ) and a lower
[Figure]
||
˜ W − ˜ H
|| 2 F , which indicates that the feature means and classiﬁer vectors of the same class are better aligned.
We observe no advantage of ResNet on STL-10 and DenseNet on CIFAR-100 in Figure 4 and 5.
In Table 2, we see that the two cases are right the failure cases, which shows consistency between neural collapse convergence and classiﬁcation performance.
Accuracy Curves of Long-tailed Classiﬁcation on ImageNet-LT.
As shown in Figure 6, we depict the accuracy curves in training of the long-tailed classiﬁcation results in Table 3.
It reveals that the model using our proposed ETF classiﬁer and DR loss converges faster than the traditional learnable classiﬁer with the CE loss.
Especially when we train for less epochs, the accuracy curves of our “ETF classiﬁer + DR loss” are less affected, while those of “learnable classiﬁer + CE loss” are deteriorated and converge slowly.
Thus the superiority in performance of our method is more remarkable when training for limited epochs.
It can be explained by the fact that our method directly has the classiﬁer in its optimality and optimizes the features towards the neural collapse solution, while the learnable classiﬁer with the CE loss needs a sufﬁcient training process to separate classiﬁer vectors of different classes.
So our method can be preferred when fast convergence or limited training time is required.
F Limitations and Societal Impacts
Limitations.
The limitations of this study may include: (1) The beneﬁts of our proposed ETF classiﬁer and DR loss are mainly analyzed for the case of imbalanced training.
But our methods are general and applicable to all classiﬁcation problems.
The advantages and disadvantages of our methods for other machine learning areas, such as label noise learning and few-shot learning, are not discussed in this study, and deserve our future work.
In a neural network with a learnable classiﬁer, the lengths, i.e., (cid:96) 2 -norms, of both features and classiﬁer vectors are usually increasing as training.
But for our ﬁxed classiﬁer, the lengths of the classiﬁer vectors are ﬁxed during training.
As shown in Eq.
(12) (for CE loss) and Eq.
(15) (for DR loss), the gradient norm with respect to feature is decided by the lengths of classiﬁer vectors.
When feature is in a large length, its gradient using our ﬁxed classiﬁer may have a limited step size.
So an adaptive mechanism to adjust the length of the
ETF classiﬁer may be more favorable and further improve the performance.
Societal Impacts.
Our study proposes a new paradigm for neural network classiﬁcation.
It enjoys some theoretical beneﬁts for imbalanced training, which is an important topic in machine learning.
Our method is a general technique for network training, but is not related to any potential impact on privacy, public health, fairness, and other societal issues.
Besides, our proposed ETF classiﬁer and
DR loss actually reduce the computation cost compared with the traditional learnable classiﬁer with the CE loss, so will not introduce extra environment burden.
G Detailed Comparison with Some Studies
As suggested by a reviewer of this paper, we compare with [5, 30, 50] in more details.
The objective of LPM studied in [5] is also the CE loss with constraints of feature and classiﬁer.
They prove that (1) neural collapse is the global optimality of this objective when training on a balanced dataset; (2) in imbalanced training, neural collapse will be broken, and the prototypes of minor classes will be merged, which explains the difﬁculty of imbalanced training.
We also study the objective of CE loss with feature and classiﬁer constraints.
As a comparison, (1)
We prove that neural collapse can also be the global optimality for imbalanced training as long as the classiﬁer is ﬁxed as an ETF (Theorem 1); (2) We analyze from the gradient perspective and show that the broken neural collapse in imbalanced training is caused by the imbalanced magnitude of gradients of the CE loss (Remark 2).
We also show that the “pull-push” mechanism is crucial for the emergence of neural collapse in the CE loss in balanced training (Remark 3); (3) Inspired by the analyses, we propose a new loss function with a provable advantage over the CE loss (Theorem 2).
The objective of LPM studied in [50] is the CE loss with regularizations of feature and classiﬁer.
They prove that: (1) neural collapse is the global optimality of this objective when training on a balanced dataset; (2) despite being nonconvex, the landscape of the objective in this case is benign, which means that there is no spurious local minimum, so gradient-based optimization method can easily escape from the strict saddle points to look for the global minimizer.
In contrast, the objective we consider is the CE loss with constraints of feature and classiﬁer, instead of regularizations.
We mainly consider neural collapse in imbalanced learning, and accordingly propose a new loss function, which are different from the two results in [50].
[29, 30] show that ﬁxing a learnable classiﬁer as the vertices of regular polytopes, including d-simplex, d-cube, and d-orthoplex, helps to learn stationary and maximally separated features.
It does not harm the performance, and in many cases improves the performance.
It also brings faster speed of convergence and reduces the model parameters.
Their spirit of learning maximally separated features is very similar to neural collapse.
Compared with [29, 30], we prove that neural collapse can be the global optimality of the CE loss even in imbalanced learning based on the LPM analytical tool.
We also propose a new loss function with a provable advantage over the CE loss.