Neural Collapse with Normalized Features: A Geometric Analysis over the Riemannian Manifold
Abstract
When training overparameterized deep networks for classiﬁcation tasks, it hasbeen widely observed that the learned features exhibit a so-called “neural collapse”phenomenon.
More speciﬁcally, for the output features of the penultimate layer,for each class the within-class features converge to their means, and the meansof different classes exhibit a certain tight frame structure, which is also alignedwith the last layer’s classiﬁer.
As feature normalization in the last layer becomes acommon practice in modern representation learning, in this work we theoreticallyjustify the neural collapse phenomenon for normalized features.
Based on anunconstrained feature model, we simplify the empirical loss function in a multiclass classiﬁcation task and obtain a nonconvex optimization problem over the
Riemannian manifold by constraining all features and classiﬁers over the sphere.
Inthis context, we analyze the nonconvex landscape of the Riemannian optimizationproblem over the product of spheres, showing a benign global landscape in thesense that the only global minimizers are the neural collapse solutions while allother critical points are strict saddles with negative curvature.
Experimental resultson practical deep networks corroborate our theory and demonstrate that betterrepresentations can be learned faster via feature normalization.
Introduction
Depsite the tremendous success of deep learning in engineering and scientiﬁc applications overthe past decades, the underlying mechanism of deep neural networks (DNNs) still largely remainsmysterious.
Towards the goal of understanding the learned deep representations, a recent seminalline of works [1–5] presents an intriguing phenomenon that persists across a range of canonicalclassiﬁcation problems during the terminal phase of training.
Speciﬁcally, it has been widely observedthat last-layer features (i e., the output of the penultimate layer) and last-layer linear classiﬁers of atrained DNN exhibit simple but elegant mathematical structures, in the sense that
• (NC1) Variability Collapse: the individual features
of each class concentrate to their
class-means.• (NC2) Convergence to Simplex ETF: the class-means have the same length and are maximallydistant.
In other words, they form a Simplex Equiangular Tight Frame (ETF).
• (NC3) Convergence to Self-Duality: the last-layer linear classiﬁers perfectly match their classmeans.
Such a phenomenon is referred to as Neural Collapse ( NC ) [1], which has been shown empiricallyto persist across a broad range of canonical classiﬁcation problems, on different loss functions(e.g.
, cross-entropy (CE) [1,3,4], mean-squared error (MSE)
[5,6], and supervised contrasive (SC)losses [7]), different neural network architectures (e.g., VGG [8], ResNet [9], and DenseNet [10]),and on a variety of standard datasets (such as MNIST [11], CIFAR [12], and ImageNet [13], etc).
[Figure]
[Table]
Recently, in independent lines of research, many works are devoted to learning maximally compactand separated features; see, e.g., [14–22].
This has also been widely demonstrated in a number ofrecent works [23–29], including state-of-the-art natural language models (such as BERT, RoBERTa,and GPT) [24].
Motivations & contributions.
In this work, we further demystify why NC happens in networktraining with a common practice of feature normalization (i.e., normalizing the last-layer features onthe unit hypersphere), mainly motivated by the following reasons:
• Feature normalization is a common practice in training model deep networks.
Recently, manyexisting results demonstrated that training with feature normalization often improves the quality
of learned representation with better class separation
Such a representationis closely related to the discriminative representation in literature; see, e.g., [16, 18, 19, 33].
As
illustrated in Fig.
1 and Table 1, experimental results visualized in low-dimensional space show
that features learned with normalization are more uniformly distributed over the sphere and henceare more linearly separable than those learned without normalization.
In particular, it has been
shown that the learned representations with larger class separation usually lead to improvedtest performances; see, e.g., [7, 34].
Moreover, it has been demonstrated that discriminativerepresentations can also improve robustness to mislabeled data [30,31], and has become a commonpractice in recent advances on (self-supervised) pretrained models [32,35].• A common practice of theoretically studying NC with norm constraints.
Due to these practicalreasons, many existing theoretical studies on NC consider formulations with both the norms offeatures and classiﬁers constrained [3, 7, 36–38].
Based upon assumptions of simpliﬁed unconstrained feature models [3,4,39], these works show that the only global solutions are NC for avariety of loss functions (e.g., MSE, CE, SC losses, etc).
Nonetheless, they only focused on theglobal optimality conditions without looking into their nonconvex landscapes, and therefore fail toexplain why global NC solutions can be efﬁciently reached by classical training algorithms suchas stochastic gradient descent (SGD).
In this work we study the global nonconvex landscape of training deep networks with norm constraintson the features and classiﬁers.
We consider the commonly used CE loss and formulate the problemas a Riemannian optimization problem over products of unit spheres (i e., the oblique manifold).
Our study is also based upon the assumption of the so-called unconstrained feature model (UFM) [4,5,39]or layer-peeled model
[40], where the last-layer features of the deep network are treated as freeoptimization variables to simplify the nonlinear interactions across layers.
The underlying reasoningis that modern deep networks are often highly overparameterized with the capacity of learning anyrepresentations [41–43], so that the last-layer features can approximate, or interpolate, any point inthe feature space.
Assuming the UFM, we show that the Riemannian optimization problem has a benign global landscape, in the sense that the loss with respect to (w.r.t.)
the features and classifers is a strict saddlefunction [44,45] over the Riemannian manifold.
More speciﬁcally, we prove that every local minimizer is a global solution satisfying the NC properties, and all the other critical points exhibitdirections with negative curvature.
Our analysis for the manifold setting is based upon a nontrivialextension of recent studies for the NC with penalized formulations [2,4–6], which could be of independent interest.
Our work brings new tools from Riemannian optimization for analyzing optimizationlandscapes of training deep networks with an increasingly common practice of feature normalization.
At the same time, we empirically demonstrate the advantages of the Riemannian/constrained formulation over its penalized counterpart for training deep networks – faster training and higher qualityrepresentations.
Lastly, under the UFM we believe the benign landscape over the manifold couldhold for many other popular training losses beyond CE, such as the (supervised) contrastive loss [34].
We leave this for future exploration.
Prior arts and related works on NC .
The empirical NC phenomenon has inspired a recent lineof theoretical studies on understanding why it occurs [4–7, 36, 39, 40].
Like ours, most of theseworks studied the problem under the UFM.
In particular, despite of the nonconvexity, recent worksshowed that the only global solutions are NC solutions for a variety of nonconvex training losses(e.g., CE [4,36,40], MSE [5,6], SC losses [7]) and different problem formulations (e.g., penalized,constrained, and unconstrained) [2,4–6,36].
Recently, this study has been extended to deeper modelswith the MSE training loss [6].
More surprisingly, it has been further shown that the nonconvex lossesunder the UFM have benign global optimization landscapes, in the sense that every local minimizersatisﬁes NC properties and the remaining critical points are strict saddles with negative curvature.
Such results have been established for both CE and MSE losses [4,5], where they considered theunconstrained formulations with regularization on both features and classiﬁers.
Moreover, there is a line of recent works investigating the beneﬁts of NC on generalizations of deepnetworks.
The work [46] shows that NC also happens on test data drawn from the same distributionasymptotically, but less collapse for ﬁnite samples [47].
Other works [47, 48] demonstrated thatthe variability collapse of features is actually happening progressively from shallow to deep layers,and [49] showed that test performance can be improved when enforcing variability collapse onfeatures of intermediate layers.
The works [50,51] showed that ﬁxing the classiﬁer as a simplex ETFimproves test performance on imbalanced training data and long-tailed classiﬁcation problems.
Formore details on related works, we refer interested readers to the Appendix A.
Notation.
Let R n be the n -dimensional Euclidean space and (cid:107) · (cid:107) 2 be the Euclidean norm.
Wewrite matrices in bold capital letters such as A , vectors in bold lower-case such as a , and scalars inplain letters such as a .
Given a matrix A ∈ R d × K , we denote its k -th column by a k , its i -th row bya i , its ( i, j ) -th element by a ij , and let (cid:107) A (cid:107) be its spectral norm.
We use diag( A ) to denote a vectorthat consists of diagonal elements of A , and we use ddiag( A ) to denote a diagonal matrix composedby only the diagonal entries of A .
Given a positive integer n , we denote the set { 1 , . . . , n } by [ n ] .
We denote the unit hypersphere in R d by S d − 1 :=
{ x ∈ R d : (cid:107) x (cid:107) 2 = 1 } .
Nonconvex Formulation with Spherical Constraints
In this section, we review the basic concepts of deep neural networks and introduce notation that willbe used throughout the paper.
Based upon this, we formally introduce the problem formulation overthe Riemannian manifold under the assumption of the UFM.
2.1 Basics of Deep Neural Networks
In this work, we focus on the multi-class (e.g., K class) classiﬁcation problem.
Given input datax ∈ R D , the goal of deep learning is to learn a deep hierarchical representation (or feature) h ( x ) =
φ θ ( x ) ∈ R
d of the input along with a linear classiﬁer 1 W ∈ R
d
× K such that the output ψ Θ ( x ) =
W (cid:62) h ( x ) of the network ﬁts the input x to an one-hot training label y ∈ R K .
More precisely, in thevanilla form an L -layer fully connected deep neural network can be written as
[Equation]
where each layer is composed of an afﬁne transformation, represented by some weight matrix W k ,and bias b k , followed by a nonlinear activation σ ( · ) , and Θ = { W k ,
b k } Lk
=1 and θ = { W k , b k } L − 1 k
=1denote the weights for all the network parameters and those up to the last layer, respectively.
Giventraining samples { x k,i ,
y k } ∈ R D
× R K drawn from the same data distribution D , we learn thenetwork parameters Θ via minimizing the empirical risk over these samples,
[Equation]
where y k ∈ R K is a one-hot vector with only the k th entry equal to unity and the remaining onesequal to zero for all k ∈
[ K ] , x k,i ∈
R D is the i -th sample in the k -th class, { n k } Kk =1 denotes thenumber of training samples in each class, and the set C denotes the constraint set of the networkparameters Θ that we will specify later.
Here, we study the most widely used CE loss of the form
[Equation]
2.2 Riemannian Optimization over the Product of Spheres
For the K -class classiﬁcation problem, let us consider a simple case where the number of trainingsamples in each class is balanced (i.e., n = n 1 = n 2 = · · · = n K ) and let N =
Kn , and weassume that all the biases { b k } Lk =1 are zero with the last activation function σ ( · ) before the outputto be linear.
Analyzing deep networks ψ Θ ( x ) is a tremendously difﬁcult task mainly due to thenonlinear interactions across a large number of layers.
To simplify the analysis, we assume theso-called unconstrained feature models (UFM) following the previous works [4, 7, 38, 39].
Morespeciﬁcally, we simplify the nonlinear interactions across layers by treating the last-layer featuresh k,i =
φ θ ( x k,i ) ∈ R d as free optimization variables, where the underlying reasoning is that moderndeep networks are often highly overparameterized to approximate any continuous function [41–43].
Concisely, we rewrite all the features in a matrix form as
[Equation]
and correspondingly denote the classiﬁer W by
[Equation]
Based upon the discussion in Section 1, we assume that both the features H and the classiﬁers Ware normalized, 2 in the sense that (cid:107) h k,i (cid:107) 2 = τ , (cid:107) w k (cid:107)
= 1 , for all k ∈
[ K ] and all i ∈ [ n ] , whereτ > 0 is a temperature parameter.
As a result, we study NC with a constrained formulation over a
Riemannian manifold
[Equation]
Since the temperature parameter τ can be absorbed into the loss function, we replace h k,i by τ h k,iby a change of variable, so that the original constraint becomes (cid:107) h k,i (cid:107) 2 = 1 for all k ∈
[ K ] and alli ∈ [ n ] .
Moreover, the product of spherical constraints forms an oblique manifold [52] embedded in
Euclidean space,
[Equation]
Consequently, we can rewrite the constrained problem as a Riemannian optimization problem overthe oblique manifold w.r.t.
W and H as
[Equation]
In Section 3, we will show that all global solutions satisfy NC properties, and the function (3) is astrict saddle function [53,54] of ( W , H ) over the oblique manifold so that the NC solution can beefﬁciently achieved.
[Equation]
Riemannian derivatives over the oblique manifold.
In
Section 3, we will use tools from Riemannian optimization tocharacterize the global optimality condition and the geometric properties of the optimization landscape of Problem (3).
Before that, let us ﬁrst brieﬂy introduce some basic derivations of the Riemannian gradient and Hessian, deﬁned on thetangent space of the oblique manifold.
For more technicaldetails, we refer the readers to Appendix B.1.
Accordingto [55, Chapter 3 & 5] and [56, 57], we can calculate the
Riemannian gradients and Hessian of Problem (3) as follows.
Since those quantities are deﬁned on the tangent space, according to [56, Section 3.1] and the illustration in Figure 2,we ﬁrst have the tangent space to OB ( d, K ) at W as
[Equation]
That is, the tangent space contains all Z such that z k is orthogonal to w k for all k ; when K = 1 , itreduces to the tangent space to the unit sphere S d − 1 .
Analogously, we can derive the tangent space for H with a similar form.
Let us deﬁne
[Equation]
First, the Riemannian gradient of f ( W , H ) of Problem (3) is basically the projection ofthe ordinary Euclidean gradient ∇ f
( W , H ) onto its tangent space, i.e., grad W f ( W , H ) =
P T W OB ( d,K ) ( ∇ W f
( W , H ))
and grad H f ( W , H )
= P T H OB ( d,N ) ( ∇ H f ( W , H )) .
More specifically, we have
[Equation]
Second, for any ∆ = ( ∆ W , ∆ H ) ∈ R d × K × R d × N , we compute the Hessian bilinear form off ( W , H ) along the direction ∆ by
[Equation]
We can derive the Riemannian Hessian bilinear form of f ( W , H ) along any direction ∆ ∈
T W OB ( d, K ) × T H OB ( d, N ) as
[Equation]
where the extra terms besides ∇ 2 f ( W , H )[ ∆ , ∆ ] compensate for the curvatures on the obliquemanifold.
For derivations of (6) and (7), see Appendix B.2.
In the following section, we will use the
Riemannian gradient and Hessian to characterize the optimization landscape of Problem (3).
Main Theoretical Analysis
In this section, we ﬁrst characterize the structure of the global solution set of Problem (3).
Based uponthis, we analyze the global landscape of Problem (3) via characterizing its Riemannian derivatives.
[Figure]
m a l V a l u e
Theoretical
Empirical
3.1 Global Optimality Condition
For the feature matrix H , let us denote the class mean for each class as
[Equation]
h
[Equation]
n
[Equation] [Equation]
Based upon this, we show any global solution of Problem (3) exhibits NC properties in the sense thatit satisﬁes ( NC1 ) variability collapse, ( NC2 ) convergence to simplex ETF, and ( NC3 ) convergence toself-duality.
Theorem 1 (Global Optimality Condition) .
Suppose that the feature dimension is no smaller thanthe number of classes (i.e., d ≥ K ) and the training labels are balanced in each class (i.e., n = n 1 =· · · = n K ).
Then for the CE loss f ( W , H ) in Problem (3) , it holds that
[Equation]
for all W =
[ w 1 , . .
. , w K ] ∈ OB ( d, K ) and H =
[ h 1 , 1 , . . .
, h K,n ] ∈ OB ( d, N ) .
In particular,equality holds if and only if there are h 1 , h 2 , · · · , h K ∈ S d − 1 such that
• ( NC1 ) Variability collapse: h k,i = h k , ∀ i ∈ [ n ] ;
• ( NC2 )
Simplex ETF structure:
{ h k } Kk =1 form a sphere-inscribed simplex ETF
in the sense that
[Equation]
• ( NC3 ) Self-duality: w k = h k , ∀ k ∈
[ K ]
[Equation]
Compared to the unconstrained regularized problems in [4,5], it is worth noting that the regularizationparameters there inﬂuence the structure of global solutions, while the temperature parameter τ onlyaffects the optimization landscape but not the global solutions.
On the other hand, the result of ourproblem (3) is closely related to [7, Theorem 1] (i.e., spherical constraints vs. that of ball constraints).
In fact, our problem and that in [7] share the same global solution set.
As such, the proof followssimilar ideas of a line of recent works [4–7,36,40], and we refer the readers to Appendix C for theproof.
It should be noted that we do not claim originality of this result compared to previous works;instead our major contribution lies in the following global landscape analysis.
3.2 Global Landscape Analysis
Due to the nonconvex nature of Problem (3), the characterization of global optimality alone in
Theorem 1 is not sufﬁcient for guaranteeing efﬁcient optimization to those desired global solutions.
Thus, we further study the global landscape of Problem (3) by characterizing all the Riemanniancritical points (
W , H ) ∈
OB ( d, K ) × OB ( d, N ) withgrad H f
( W , H )
= 0 , grad W f
( W , H )
= 0 .
As such, we state our major result below.
Theorem 2 (Global Landscape Analysis) .
Assume that the number of training samples in each classis balanced n = n 1 = · · · = n K .
If the feature dimension is larger than the number of classes d > Kand the temperature parameter satisﬁes τ < 2( d − 2)(1 + ( K mod 2) /K )
− 1 , then the functionf ( W , H ) is a strict saddle function that has no spurious local minimum, in the sense that
• Any local minimizer of Problem (3) is a global minimizer of the form shown in Theorem 1.
• Any Riemannian critical point ( W , H ) of Problem (3) that is not a local minimizer is a Riemannian strict saddle point with negative curvatures, in the sense that the Riemannian Hes-
sian Hess f ( W , H )
at the critical point ( W , H ) is non-degenerate, and there exists a direction
∆ =
( ∆ W , ∆ H ) ∈ T W OB ( d, K ) × T H OB
( d, N ) such that
Hess f ( W , H )
[ ∆
∆ ] <
In other words, λ min (Hess f ( W , H )) < 0 at the corresponding Riemannian critical point.
For the details of the proof, we refer readers to Appendix D. The ﬁrst bullet point naturally followsfrom Theorem 1.
The major challenge of our analysis is showing the second bullet, i.e., to ﬁnd anegative curvature direction ∆ for Hess f ( W , H ) .
Our key observation is that the set of non-globalcritical points can be partitioned into two separate cases.
In the ﬁrst case, the last two terms of(7) vanish, and we show that the second term of (6) is negative and dominates the ﬁrst term for anappropriate direction.
We require τ to not be too large, since the ﬁrst term is O ( τ 2 ) , whereas thesecond term is O ( τ ) .
In the second case, using the assumption that d > K we can ﬁnd a rank-onedirection that makes the ﬁrst term of (6) vanishing.
In this case, we similarly show that the secondterm of (6) is negative but instead dominates the last two terms of (7).
In the following, we discussthe implications, relationship, and limitations of our results in Theorem 2.
• Efﬁcient global optimization to NC solutions.
Our theorem implies that, for optimizing ( W , H )via Problem (3), the NC solutions can be efﬁciently reached by Riemannian ﬁrst-order methods(e
g., Riemannian stochastic gradient descent) with random initialization [53,58,59]; see Figure
for a demonstration.
For training practical deep networks, this can be efﬁciently implemented bynormalizing last-layer features when running SGD.
• Relation to existing works on NC .
Most existing results have only studied the global minimizers
under the UFM [6,7,36,40], which has limited implication for optimization.
On the other hand,our landscape analysis is based upon a nontrivial extension of that with the unconstrained problemformulation [4,5].
Compared to those works, Problem (3) is much more challenging for analysis,due to the fact that the set of critical points of our problem is essentially much larger than thatof [4,5].
Moreover, we empirically demonstrate the advantages of the manifold formulation over
its regularized counterpart, in terms of representation quality and training speed.
• Assumptions on the feature dimension d and temperature parameter τ .
Our current result requiresthat d > K , which is the same requirement in [4,5].
Furthermore, through numerical simulationswe conjecture that the global landscape also holds
even when d (cid:28) K , while the global solutionsare uniform over the sphere [36] rather than being simplex ETFs (see Figure 1).
The analysis ond (cid:28) K is left for future work.
On the other hand, the required upper bound on τ is for the easeof analysis and it holds generally in practice,
but we conjecture that the benign landscape holdswithout it.
[Figure]
T r a i n i n g A cc u r a c y
Width= [Figure] Width=
[Figure]
Width [Figure] 16 Width [Figure]
Width
[Figure]
• Relation to other Riemannian nonconvex problems.
Our result joins a recent line of work onthe study of global nonconvex landscapes over Riemannian manifolds, such as orthogonal tensordecomposition [44], dictionary learning [60–64], and sparse blind deconvolution [65–68].
Forall these problems constrained over a Riemannian manifold, it can be shown that they exhibit“equivalently good” global minimizers due to symmetries and intrinsic low-dimensional structures,and the loss functions are usually strict saddles [44,45,69].
As we can see, the global minimizers(i.e., simplex ETFs) of our problem here also exhibit a similar rotational symmetry, in the sensethat W (cid:62) H =
( QW ) (cid:62) ( QH ) for any orthogonal matrix Q .
Additionally, our result show howtools from Riemmanian optimization can be powerful for the study of deep learning.
Experiments
In this section, we support our theoretical results in previous sections and provide further motivationwith experimental results on practical deep network training.
In the ﬁrst experiment, we validate theassumption of UFM introduced in Section 2 for analyzing NC , by demonstrating that NC occurs forincreasingly overparameterized deep networks.
In the second experiment, we further motivate featurenormalization with empirical results showing that feature normalization can lead to faster trainingand better collapse than the unconstrained counterpart with regularization.
This occurs not only withthe UFM but also with practical overparameterized networks.
We detail the network architectures,datasets, training details, and metrics used in these experiments, as well as additional experiments in
Appendix E.
4.1 Validation of the UFM for training networks with feature normalization
In Section 2, our study of the Riemannian optimization problem (3) is based upon the UFM, wherewe assume H is a free optimization variable.
Here, we justify this assumption by showing that
NC happens for training overparameterized networks even when the training labels are completelyrandom.
By using random labels, we disassociate the input from their class labels, by which wecan characterize the approximation power of the features of overparmeterized models.
To show this,we train ResNet-18 with varying widths (i.e., the number of feature maps resulting from the ﬁrstconvolutional layer) on CIFAR10 with random labels, with normalized features and classiﬁers.
As shown in Figure 4, we observe that increasing the width of the network allows for perfect classiﬁcation on the training data even when the labels are random.
Furthermore, increasing the networkwidth also leads to better NC , measured by the decrease in each NC metric.
This corroborates that( i ) our assumption of UFM is reasonable given that NC seems to be independent of the input data,and ( ii ) NC happens under the our constraint formulation (3) on practical networks.
4.2 Improved training speed and better representation quality with feature normalization
We now investigate the beneﬁts of using feature normalization, more speciﬁcally for improvedtraining speed and better representation quality.
First, we consider the UFM formulation, wherewe optimize Problem (3) and compare to the regularized UFM in [4].
The results are shown in
Figure 5.
We can see that normalizing features over the sphere consistently results in reaching perfectclassiﬁcation and greater feature collapse (i.e., smaller NC 1 ) quicker than penalizing the features.
To demonstrate that these behaviors are reﬂected in training practical deep networks, we train both
ResNet-18 and ResNet-50 architectures on a reduced CIFAR100 [12] dataset with N = 3000 total
[Figure]
d =
[Figure]
ResNet- and ResNet-
on
CIFAR
with n = 30 overepochs
samples, comparing the training accuracy and metrics of NC with and without feature normalization.
The results are shown in Figure 6.
From Figure 6 (left), we can see that for the ResNet-18 network, we reach perfect classiﬁcation ofthe training data about 10-20 epochs sooner by using feature normalization compared to that of theunconstrained formulation.
From Figure 6 (right), training the ResNet-50 network without feature normalization for 100 epochs shows slow convergence with poor training accuracy, whereas using featurenormalization arrives at above 90% training accuracy in the same number of epochs.
By keeping thesize of the dataset the same and increasing the number of parameters, it is reasonable that optimizingthe ResNet-50 network is more challenging due to the higher degree of overparameterization, yet thiseffect is mitigated by using feature normalization.
At the same time, for both architectures, using feature normalization leads to greater feature collapse(i.e.
, smaller NC 1 ) compared to that of the unconstrained counterpart.
As shown in recent work [1,5,46] , better NC often leads to better generalization performance.
Experimental results demonstratingthat feature normalization generalizes better than regularization are provided in Appendix E.2.
Lastbut not least, we conjecture that the benign landscape holds beyond the CE loss (e.g., SC [7], see
Appendix E.4)
; and we believe the beneﬁts of feature normalization are not limited to the evidencethat we showed here, as it could also lead to better robustness [30,31] that is worth further exploration.
Conclusion & Discussion
In this work, we study the prevalence of the NC phenomenon with normalized features.
Basedupon the assumption of UFM, we formulate the problem as a Riemannian optimization problemover the product of spheres.
We showed that the loss function is a strict saddle function overthe manifold with respect to the last-layer features and classiﬁers, with no other spurious localminimizers.
We demonstrate this on practical deep network training, and show practical beneﬁts offeature normalization in terms of training speed and learned representation quality.
As future work,we would like to expand the study to other popular loss functions such as contrastive loss and studythe settings d (cid:28) K , which could be of great importance for studying self-supervised learning.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the
paper’scontributions and scope?
Yes
(b) Did you describe the limitations of your work?
[Yes] We state in Section 2 that
our
analysis uses
the unconstrained feature model, and in Section 3 we discuss therequirement of
d
> K and upper
bound on τ for Theorem
(c) Did you discuss any potential negative
societal impacts of
your
work
N/A
] Thispaper mainly focuses on understanding the neural collapse phenomena observed inpractical neural networks.
We believe there is no potential negative societal impact inthis work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] We explicitlymention the assumptions in Section 3.
(b) Did you include complete proofs of all theoretical results?
[Yes] We include all theproofs in the Appendix.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] The CIFAR
datasets used here are cited and easily accessible.
The GitHub code repository will be
made available after review.
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Section 4.
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] Figure 3 does not show error bars since the attained value
is always the same independent of initialization due to the benign global landscape.
In Section 4, we used shaded area as error bars for Figure 5, but regardless we haveobserved that the results are similar and reproducible independent of the random seed.
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Section 4.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We cite them in
Section 4.
(b) Did you mention the license of the assets?
[Yes] This is mentioned in Section 4.
(c) Did you include any new assets either in the supplemental material or as a URL?
N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
[Table]
using/curating?
[Yes] This is mentioned in Section 4.
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
[Table]
information or offensive content?
[N/A]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A] [Table]