Neural Collapse with Normalized Features: A Geometric Analysis over the Riemannian Manifold
Abstract
When training overparameterized deep networks for classiﬁcation tasks, it has been widely observed that the learned features exhibit a so-called “neural collapse” phenomenon.
More speciﬁcally, for the output features of the penultimate layer, for each class the within-class features converge to their means, and the means of different classes exhibit a certain tight frame structure, which is also aligned with the last layer’s classiﬁer.
As feature normalization in the last layer becomes a common practice in modern representation learning, in this work we theoretically justify the neural collapse phenomenon under normalized features.
Based on an unconstrained feature model, we simplify the empirical loss function in a multi-class classiﬁcation task into a nonconvex optimization problem over the Riemannian manifold by constraining all features and classiﬁers over the sphere.
In this context, we analyze the nonconvex landscape of the Riemannian optimization problem over the product of spheres, showing a benign global landscape in the sense that the only global minimizers are the neural collapse solutions while all other critical points are strict saddle points with negative curvature.
Experimental results on practical deep networks corroborate our theory and demonstrate that better representations can be learned faster via feature normalization.
Code for our experiments can be found at https://github.com/cjyaras/normalized-neural-collapse .
Introduction
Depsite the tremendous success of deep learning in engineering and scientiﬁc applications over the past decades, the underlying mechanism of deep neural networks (DNNs) still largely remains mysterious.
Towards the goal of understanding the learned deep representations, a recent line of seminal works [1–5] presents an intriguing phenomenon that persists across a range of canonical classiﬁcation problems during the terminal phase of training.
Speciﬁcally, it has been widely observed that last-layer features (i.e., the output of the penultimate layer) and last-layer linear classiﬁers of a trained DNN exhibit simple but elegant mathematical structures, in the sense that
• (NC1) Variability Collapse: the individual features of each class concentrate to their class-means.
• (NC2) Convergence to Simplex ETF: the class-means have the same length and are maximally distant.
In other words, they form a Simplex Equiangular Tight Frame (ETF). Equal
[Figure] [Table] • (NC3) Convergence to Self-Duality: the last-layer linear classiﬁers perfectly match their classmeans.
Such a phenomenon is referred to as Neural Collapse ( NC ) [1], which has been shown empirically to persist across a broad range of canonical classiﬁcation problems, on different loss functions (e.g., cross-entropy (CE)
[1,3,4], mean-squared error (MSE)
[5,6], and supervised contrasive (SC) losses [7], etc.), different neural network architectures (e.g., VGG [8], ResNet [9], and DenseNet [10], etc.), and on a variety of standard datasets (e.g., MNIST [11], CIFAR [12], ImageNet [13], etc.).
Recently, in independent lines of research, many works are devoted to learning maximally compact and separated features; see, e.g., [14–22].
This has also been widely demonstrated in a number of recent works [23–29], including state-of-the-art natural language models (such as BERT, RoBERTa, and GPT) [24].
Motivations & contributions.
In this work, we further demystify why NC happens in network training with feature normalization (i.e., normalizing the last-layer features on the unit hypersphere), mainly motivated by the following reasons:
• Feature normalization is a common practice in training deep networks.
Recently, many existing results demonstrated that training with feature normalization often improves the quality of learned representation with better class separation [7,16,18–20,30–32].
Such a representation is closely related to the discriminative representation in literature; see, e.g., [16,18,19,33].
As illustrated in Fig.
1 and Table 1, experimental results visualized in low-dimensional space show that features learned with normalization are more uniformly distributed over the sphere and hence are more linearly separable than those learned without normalization.
In particular, it has been shown that the learned representations with larger class separation usually lead to improved test performances; see, e.g., [7,34].
Moreover, it has been demonstrated that discriminative representations can also improve robustness to mislabeled data [30, 31], and has become a common practice in recent advances on (self-supervised) pretrained models [32,35]. • A common practice of theoretically studying NC with norm constraints.
Due to these practical reasons, many existing theoretical studies on NC consider formulations with both the norms of features and classiﬁers constrained [3,7,36–38].
Based upon assumptions of unconstrained feature models [3, 4, 39], these works show that the only global solutions satisfy NC properties for a variety of loss functions (e.g., MSE, CE, SC losses, etc).
Nonetheless, they only focused on the global optimality conditions without looking into their nonconvex landscapes, and therefore fail to explain why global NC solutions can be efﬁciently reached by classical training algorithms such as stochastic gradient descent (SGD).
In this work , we study the global optimization landscape of the training loss with norm constraints on the features and classiﬁers.
We consider the commonly used CE loss and formulate the problem as a Riemannian optimization problem over products of unit spheres (i e., the oblique manifold).
Our study is also based upon the assumption of the so-called unconstrained feature model (UFM) [4,5,39] or layer-peeled model [40] , where
the last-layer features of the deep network are treated as free optimization
variables to simplify the nonlinear interactions across layers. The underlying reasoning is that modern deep networks are often highly overparameterized with the capacity of learning any representations [41–43] so that the last-layer features can approximate, or interpolate, any point
in the feature space Assuming the UFM we show that the Riemannian optimization problem has a benign global landscape in the sense that the loss with respect to (w.r t.) the features and classifers is a strict saddle function [44,45] over the Riemannian manifold.
More speciﬁcally we prove that every local minimizer is a global solution satisfying the NC properties and all the other critical points exhibit directions with negative curvature.
Our analysis for the manifold setting is based upon a nontrivial extension of recent studies for the NC with penalized formulations [2, 4–6], which could be of independent interest.
Our work brings new tools from Riemannian optimization for analyzing optimization landscapes of training deep networks with an increasingly common practice of feature normalization.
At the same time, we empirically demonstrate the advantages of the constrained formulation on the manifold over its penalized counterpart for training deep networks – faster training and higher quality representations.
Lastly, under the UFM we believe the benign landscape over the manifold could hold for many other popular training losses beyond CE, such as the (supervised) contrastive loss [34].
We leave this for future exploration.
Prior arts and related works on NC .
The empirical NC phenomenon has inspired a recent line of theoretical studies on understanding why it occurs [4–7, 36, 39, 40].
Like ours, most of these works studied the problem under the UFM.
In particular, despite the nonconvexity, recent works showed that the only global solutions are NC solutions for a variety of nonconvex training losses (e.g., CE [4,36,40], MSE [5,6], SC losses [7]) and different problem formulations (e.g., penalized, constrained, and unconstrained) [2,4–6,36].
Recently, this study has been extended to deeper models with the MSE training loss [6].
More surprisingly, it has been further shown that the nonconvex losses under the UFM have benign global optimization landscapes, in the sense that every local minimizer satisﬁes NC properties and the remaining critical points are strict saddles with negative curvature.
Such results have been established for both CE and MSE losses [4,5], where they considered the unconstrained formulations with regularization on both features and classiﬁers.
We should also mention that the benign global optimization landscapes of many other problems in neural networks have been widely found in the literature; see, e.g., [46–50].
Moreover, there is a line of recent works investigating the beneﬁts of NC on generalization of deep networks.
The work [51] showed that NC also happens on test data drawn from the same distribution asymptotically, but with less collapse for ﬁnite samples [52].
Other works [52, 53] demonstrated that the variability collapse of features is actually happening progressively from shallow to deep layers, and [54] showed that test performance can be improved when enforcing variability collapse on features of intermediate layers.
The works [55,56] showed that ﬁxing the classiﬁer as a simplex ETF improves test performance on imbalanced training data and long-tailed classiﬁcation problems.
For more details on related works, we refer the readers to the Appendix A.
Notation.
Let R n be the n -dimensional Euclidean space and (cid:107) · (cid:107) 2 be the Euclidean norm.
We write matrices in bold capital letters such as A , vectors in bold lower-case such as a , and scalars in plain letters such as a .
Given a matrix A ∈ R d × K , we denote its k -th column by a k , its i -th row by a i , its ( i, j ) -th element by a ij , and its spectral norm by (cid:107) A (cid:107) .
We use diag( A ) to denote a vector that consists of diagonal elements of A , and we use ddiag( A ) to denote a diagonal matrix composed by only the diagonal entries of A .
Given a positive integer n , we denote the set { 1 , . . . , n } by [ n ] .
We denote the unit hypersphere in R d by S d − 1 :=
{ x ∈ R d : (cid:107) x (cid:107) 2 = 1 } .
Nonconvex Formulation with Spherical Constraints
In this section, we review the basic concepts of deep neural networks and introduce notation that will be used throughout the paper.
Based upon this, we formally introduce the problem formulation over the Riemannian manifold under the assumption of the UFM.
2.1 Basics of Deep Neural Networks
In this work, we focus on the multi-class (e.g., K class) classiﬁcation problem.
Given input data x ∈ R D , the goal of deep learning is to learn a deep hierarchical representation (or feature) h ( x ) =
φ θ ( x ) ∈ R
d of the input along with a linear classiﬁer 1 W ∈ R
d
× K such that the output ψ Θ ( x ) =
W (cid:62) h ( x ) of the network ﬁts the input x to an one-hot training label y ∈ R K .
More precisely, in vanilla form an L -layer fully connected deep neural network can be written as
[Equation]
where each layer is composed of an afﬁne transformation, represented by some weight matrix W k , and bias b k , followed by a nonlinear activation σ ( · ) , and Θ = { W k ,
b k } Lk
=1 and θ = { W k , b k } L − 1 k = denote the weights for all the network parameters and those up to the last layer, respectively.
Given training samples { ( x k,i , y k ) } ⊂ R D × R K drawn from the same data distribution D , we learn the network parameters Θ via minimizing the empirical risk over these samples,
[Equation]
where y k ∈ R K is a one-hot vector with only the k th entry being 1 and the remaining ones being 0 for all k ∈
[ K ] , x k,i ∈
R D is the i -th sample in the k -th class, { n k } Kk =1 denotes the number of training samples in each class, and the set C denotes the constraint set of the network parameters Θ that we will specify later.
Here, we study the most widely used CE loss of the form
[Equation]
2.2 Riemannian Optimization over the Product of Spheres
For the K -class classiﬁcation problem, let us consider a simple case where the number of training samples in each class is balanced (i.e., n = n 1 = n 2 = · · · = n K ) and N =
Kn . We assume that all the biases { b k } Lk =1 are zero with the last activation function σ ( · ) before the output being linear.
Analyzing deep networks ψ Θ ( x ) is a tremendously difﬁcult task mainly due to the nonlinear interactions across a large number of layers.
To simplify the analysis, we assume the so-called unconstrained feature model (UFM) following the previous works [4,7,38,39].
More speciﬁcally, we simplify the nonlinear interactions across layers by treating the last-layer features h k,i =
φ θ ( x k,i ) ∈ R d as free optimization variables, where the underlying reasoning is that modern deep networks are often highly overparameterized to approximate any continuous function [41–43].
Concisely, we write all the features in a matrix form as
[Equation]
and correspondingly write the classiﬁer W as
[Equation]
Based upon the discussion in Section 1, we assume that both the features H and the classiﬁers W are normalized, 2 i.e., (cid:107) h k,i (cid:107) 2 = 1 and (cid:107) w k (cid:107)
= τ for all k ∈
[ K ] and all i ∈ [ n ] , where τ > 0 is a temperature parameter.
As a result, we obtain a constrained formulation of the NC problem over a
Riemannian manifold
[Equation]
W [Equation] Since the temperature parameter τ can be absorbed into the loss function, we replace w k by τ w k and change the original constraint into (cid:107) w k (cid:107) 2 = 1 for all k ∈
[ K ] .
Moreover, the product of spherical constraints forms an oblique manifold [57] embedded in Euclidean space,
[Equation]
Consequently, we can rewrite Problem (3) as a Riemannian optimization problem over the oblique manifold w.r.t.
W and H , i.e.,
[Equation]
In Section 3, we will show that all global solutions of this problem satisfy NC properties, and its objective function is a strict saddle function [58,59] of ( W , H ) over the oblique manifold so that the NC solution can be efﬁciently achieved.
[Equation]
h
Riemannian derivatives over the oblique manifold.
In
Section 3, we will use tools from Riemannian optimization to characterize the global optimality condition and the geometric properties of the optimization landscape of Problem (4).
To proceed, let us ﬁrst brieﬂy introduce some basic derivations of the Riemannian gradient and Hessian, deﬁned on the tangent space of the oblique manifold.
For more technical details, we refer the readers to Appendix B.1.
According to [57, Chapter 3 & 5] and [60,61], we can calculate the Riemannian gradients and Hessian of Problem (4) as follows.
Since those quantities are deﬁned on the tangent space, according to [60, Section 3.1] and the illustration in Figure 2, we ﬁrst have the tangent space to OB ( d, K ) at W as
[Equation]
This indicates that the tangent space contains all Z such that z k is orthogonal to w k for all k ; when K = 1 , it reduces to the tangent space to the unit sphere S d − 1 .
Analogously, we can derive the tangent space for H with a similar form.
Let us deﬁne
[Equation]
First, the Riemannian gradient of f ( W , H ) of Problem (4) is basically the projection of the ordinary Euclidean gradient ∇ f ( W , H ) onto its tangent space, i.e., grad W f ( W , H ) =
P T W OB ( d,K ) ( ∇ W f
( W , H ))
and grad H f ( W , H )
= P T H OB ( d,N ) ( ∇ H f ( W , H )) .
More specifically, we have
[Equation]
Second, for any ∆ = ( ∆ W , ∆ H ) ∈ R d × K × R d × N , we compute the Hessian bilinear form of f ( W , H ) along the direction ∆ by
[Equation]
We compute the Riemannian Hessian bilinear form of f ( W , H ) along any direction ∆ ∈
T W OB ( d, K ) × T H OB ( d, N ) by
[Equation]
where the extra terms besides ∇ 2 f ( W , H )[ ∆ , ∆ ] compensate for the curvature on the oblique manifold.
For derivations of (7) and (8), see Appendix B.2.
In the following section, we will use the
Riemannian gradient and Hessian to characterize the optimization landscape of Problem (4).
Main Theoretical Analysis
In this section, we ﬁrst characterize the structure of the global solution set of Problem (4).
Based upon this, we analyze the global landscape of Problem (4) via characterizing its Riemannian derivatives.
3.1 Global Optimality Condition
For the feature matrix H , let us denote the class mean for each class as
[Equation]
where h k := 1 n n (cid:88)
i
[Equation]
Based upon this, we show any global solution of Problem (4) exhibits NC properties in the sense that it satisﬁes ( NC1 ) variability collapse, ( NC2 ) convergence to simplex ETF, and ( NC3 ) convergence to self-duality.
Theorem 1 (Global Optimality Condition) .
Suppose that the feature dimension is no smaller than the number of classes, i.e., d ≥ K , and the training labels are balanced in each class, i.e., n = n 1 = · · · = n K .
Then for the CE loss f ( W , H ) in Problem (4) , it holds that
[Equation]
for all W =
[ w 1 , . .
. , w K ] ∈ OB ( d, K ) and H =
[ h 1 , 1 , . . .
, h K,n ] ∈ OB ( d, N ) .
In particular, equality holds if and only if
• ( NC1 ) Variability collapse: h k,i = h k , ∀ i ∈ [ n ] ; • ( NC2 ) Convergence to Simplex ETF: { h k }
Kk =1 form a sphere-inscribed simplex ETF in the sense
that
[Equation]
• ( NC3 ) Convergence to Self-duality: w k = h k , ∀ k ∈
[ K ] .
Compared to the unconstrained regularized problems in [4,5], it is worth noting that the regularization parameters there inﬂuence the structure of global solutions, while the temperature parameter τ only affects the optimization landscape but not the global solutions.
On the other hand, the result of our problem (4) is closely related to [7, Theorem 1] (i.e., spherical constraints vs. ball constraints).
In fact, our problem and that in [7] share the same global solution set.
As such, the proof follows similar ideas of a line of recent works [4–7,36,40], and we refer the readers to Appendix C for the proof.
It should be noted that we do not claim originality of this result compared to previous works. Instead, our major contribution lies in the following global landscape analysis.
3.2 Global Landscape Analysis
Due to the nonconvex nature of Problem (4), the characterization of global optimality alone in
Theorem 1 is not sufﬁcient for guaranteeing efﬁcient optimization to those desired global solutions.
Thus, we further study the global landscape of Problem (4) by characterizing all the Riemannian critical points (
W , H ) ∈
OB ( d, K ) × OB ( d, N ) satisfying
[Equation]
We now state our major result below.
[Figure] a l V a l u e
Theoretical
Empirical
Theorem 2 (Global Landscape Analysis) .
Assume that the number of training samples in each class is balanced, i.e., n = n 1 = · · · = n K .
If the feature dimension is larger than the number of classes, i.e., d > K , and the temperature parameter satisﬁes τ < 2( d − 2)(1 + ( K mod 2) /K ) − 1 , then the function f
( W , H ) is a strict saddle function that has no spurious local minimum, in the sense that
• Any Riemannian critical point ( W , H ) of Problem (4) that is not a local minimizer is a Riemannian strict saddle point with negative curvatures, in the sense that the Riemannian Hessian Hess f ( W , H ) at the critical point ( W , H ) is non-degenerate, and there exists a direction ∆ = ( ∆ W , ∆ H ) ∈ T W OB
( d, K ) × T H OB ( d, N )
such that
[Equation]
In other words, λ min (Hess f ( W , H )) < 0 at the corresponding Riemannian critical point.
• Any local minimizer of Problem (4) is a global minimizer of the form
shown in Theorem 1. For the details of the proof, we refer readers to Appendix D. The second bullet point naturally follows from Theorem 1 and the ﬁrst bullet point.
The major challenge of our analysis is showing the ﬁrst bullet, i.e., to ﬁnd a negative curvature direction ∆ for Hess f ( W , H ) .
Our key observation is that the set of non-global critical points can be partitioned into two separate cases.
In the ﬁrst case, the last two terms of (8) vanish, and we show that the second term of (7) is negative and dominates the ﬁrst term for an appropriate direction.
We require τ to not be too large, since the ﬁrst term is O ( τ 2 ) , whereas the second term is O ( τ ) .
In the second case, using the assumption that d > K we can ﬁnd a rank-one direction that makes the ﬁrst term of (7) vanishing.
In this case, we similarly show that the second term of (7) is negative but instead dominates the last two terms of (8).
In the following, we discuss the implications, relationship, and limitations of our results in Theorem 2.
• Efﬁcient global optimization to NC solutions.
Our theorem implies that the NC solutions can be efﬁciently reached by Riemannian ﬁrst-order methods (e
g., Riemannian stochastic gradient descent) with random initialization [58, 62, 63]; see Figure 3 for a demonstration.
For training practical deep networks, this can be efﬁciently implemented by normalizing last-layer features when running SGD.
• Relation to existing works on NC .
Most existing results have only studied the global minimizers under the UFM [6,7,36,40], which has limited implications for optimization.
On the other hand, our landscape analysis is based upon a nontrivial extension of that with the unconstrained problem formulation [4,5].
Compared to those works, Problem (4) is much more challenging for analysis, due to the fact that the set of critical points of our problem is essentially much larger than that of [4,5].
Moreover, we empirically demonstrate the advantages of the manifold formulation over its regularized counterpart, in terms of representation quality and training speed.
• Assumptions on the feature dimension d and temperature parameter τ .
Our current result requires that d > K , which is the same as that in [4,5].
Furthermore, through numerical simulations we conjecture that the global landscape also holds even when d (cid:28) K , while the global solutions are uniform over the sphere [36] rather than being simplex ETFs (see Figure 1).
The analysis on d (cid:28) K is left for future work.
On the other hand, the required upper bound on τ is for the ease of analysis and it holds generally in practice,
but we conjecture that the benign landscape holds without it.
• Relation to other Riemannian nonconvex problems.
Our result joins a recent line of work on the study of global nonconvex landscapes over Riemannian manifolds, such as orthogonal tensor decomposition [44], dictionary learning [64–68], subspace clustering [69], and sparse blind deconvolution [70–73].
For all these problems constrained over a Riemannian manifold, it can be shown that they exhibit “equivalently good” global minimizers due to symmetries and intrinsic low-dimensional structures, and the loss functions are usually strict saddles [44,45,74].
As we can see, the global minimizers (i.e., simplex ETFs) of our problem here also exhibit a similar rotational symmetry, in the sense that W (cid:62) H =
( QW ) (cid:62) ( QH ) for any orthogonal matrix Q .
Additionally, our result shows that tools from Riemmanian optimization can be powerful for the study of deep learning.
Experiments
In this section we support our theoretical results in previous sections and provide further motivation with experimental results on practical deep network training.
In the ﬁrst experiment, we validate the assumption of UFM introduced in Section 2 for analyzing NC , by demonstrating that NC occurs for increasingly overparameterized deep networks.
In the second experiment, we further motivate feature normalization with empirical results, showing that feature normalization can lead to faster training and better collapse than the unconstrained counterpart with regularization.
This occurs not only with the UFM but also with practical overparameterized networks.
We detail the network architectures, datasets, training details, and metrics used in these experiments, as well as additional experiments in
Appendix E.
4.1 Validation of the UFM for training networks with feature normalization
In Section 2, our study of the Riemannian optimization problem (4) is based upon the UFM, where we assume H is a free optimization variable.
Here, we justify this assumption by showing that
NC happens for training overparameterized networks even when the training labels are completely random.
By using random labels, we disassociate the input from their class labels, by which we can characterize the approximation power of the features of overparameterized models.
To show this, we train ResNet-18 with varying widths (i.e., the number of feature maps resulting from the ﬁrst convolutional layer) on CIFAR10 with random labels, with normalized features and classiﬁers.
As shown in Figure 4, we observe that increasing the width of the network allows for perfect classiﬁcation on the training data even when the labels are random.
Furthermore, increasing the network width also leads to better NC , measured by the decrease in each NC metric.
This validates that ( i ) our assumption of UFM is reasonable given that NC seems to be independent of the input data, and ( ii ) NC happens under the constrained formulation (4) on practical networks.
4.2 Improved training speed and better representation quality with feature normalization
We now investigate the beneﬁts of using feature normalization, more speciﬁcally for improved training speed and better representation quality.
First, we consider the UFM formulation, where we optimize Problem (4) and compare to the regularized UFM in [4].
The results are shown in
Figure 5.
We can see that normalizing features over the sphere consistently results in reaching perfect classiﬁcation and greater feature collapse (i.e., smaller NC 1 ) quicker than penalizing the features.
To demonstrate that these behaviors are reﬂected in training practical deep networks, we train both
ResNet-18 and ResNet-50 architectures on a reduced CIFAR100 [12] dataset with N = 3000 total samples, comparing the training accuracy and metrics of NC with and without feature normalization.
The results are shown in Figure 6.
[Figure]
[Figure]
From Figure 6 (left), we can see that for the ResNet-18 network, we reach perfect classiﬁcation of the training data about 10-20 epochs sooner by using feature normalization compared to that of the unconstrained formulation.
From Figure 6 (right), training the ResNet-50 network without feature normalization for 100 epochs shows slow convergence with poor training accuracy, whereas using feature normalization arrives at above 90% training accuracy in the same number of epochs.
By keeping the size of the dataset the same and increasing the number of parameters, it is reasonable that optimizing the ResNet-50 network is more challenging due to the higher degree of overparameterization, yet this effect is mitigated by using feature normalization.
At the same time, for both architectures, using feature normalization leads to greater feature collapse (i.e., smaller NC 1 ) compared to that of the unconstrained counterpart.
As shown in recent work [1,5, 51] , better NC often leads to better generalization performance.
Experimental results demonstrating that feature normalization generalizes better than regularization are provided in Appendix E.2.
Last but not least, we conjecture that the benign landscape holds beyond the CE loss (e.g., SC [7], see
Appendix E.4)
; and we believe the beneﬁts of feature normalization are not limited to the evidence that we showed here, as it could also lead to better robustness [30,31] that is worth further exploration.
Conclusion & Discussion
In this work, we study the prevalence of the NC phenomenon with normalized features.
Based upon the assumption of UFM, we formulate the problem as a Riemannian optimization problem over the product of spheres.
We showed that the loss function is a strict saddle function over the manifold with respect to the last-layer features and classiﬁers, with no other spurious local minimizers.
We demonstrate this on practical deep network training, and show practical beneﬁts of feature normalization in terms of training speed and learned representation quality.
As future work, we would like to expand the study to other popular loss functions such as contrastive loss and study the settings d (cid:28) K , which could be of great importance for studying self-supervised learning.
Acknowledgement
Can Yaras and Qing Qu acknowledge support from U-M START & PODS grants, NSF CAREER
CCF 2143904, NSF CCF 2212066, NSF CCF 2212326, and ONR N00014-22-1-2529 grants. Peng Wang and Laura Balzano acknowledge support from ARO YIP W911NF1910027, AFOSR YIP [Figure]
FA9550-19-1-0026, and NSF CAREER CCF-1845076. Zhihui Zhu acknowledges support from
NSF grants CCF-2008460 and CCF-2106881.
We would like to thank Zhexin Wu (ETH), Yan Wen (Tsinghua), and Pengru Huang (UMich) for fruitful discussion through various stages of the work. Checklist For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] We state in Section 2 that our analysis uses the unconstrained feature model, and in Section 3 we discuss the requirement of d > K and upper bound on τ for Theorem 2.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
This paper mainly focuses on understanding the neural collapse phenomena observed in practical neural networks.
We believe there is no potential negative societal impact in this work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] We explicitly mention the assumptions in Section 3.
(b) Did you include complete proofs of all theoretical results?
[Yes] We include all the proofs in the Appendix. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] The CIFAR datasets used here are cited and easily accessible.
The GitHub code repository will be made available after review.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section 4.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] Figure 3 does not show error bars since the attained value is always the same independent of initialization due to the benign global landscape.
In Section 4, we used shaded area as error bars for Figure 5, but regardless we have observed that the results are similar and reproducible independent of the random seed.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Section 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] We cite them in Section 4.
(b) Did you mention the license of the assets?
[Yes] This is mentioned in Section 4.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[Yes] This is mentioned in Section 4.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]