Neural Collapse with Normalized Features: A Geometric Analysis over the Riemannian Manifold
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Recently, it has been empirically observed that optimization of overparameterizeddeep networks for classiﬁcation tasks often leads to a so-called “neural collapse”phenomenon.
Speciﬁcally, it has been observed that the learned features (theoutput of the penultimate layer) of within-class samples converge to their means,and the means of different classes exhibit a certain tight frame structure, whichis also aligned with the last layer’s classiﬁer.
As feature normalization in thelast layer becomes a common practice in modern representation learning, wetheoretically justify the neural collapse phenomenon for normalized features in thiswork.
Based on an unconstrained feature model, we simplify the empirical lossfunction in a multi-class classiﬁcation task and obtain a nonconvex optimizationproblem over the Riemannian manifold by constraining all features and classiﬁersover the sphere.
In this context, we analyze the nonconvex landscape of the Riemannian optimization problem over the product of spheres, showing a benignglobal landscape in the sense that the only global minimizers are the neural collapsesolutions while all other critical points are strict saddles with negative curvature.
Experimental results on practical deep networks are reported to corroborate ourtheory and demonstrate that better representations can be learned faster via featurenormalization.
Introduction
Depsite the tremendous success of deep learning in engineering and scientiﬁc applications overthe past decades, the underlying mechanism of deep neural networks (DNNs) still largely remainsmysterious.
Towards the goal of understanding the learned deep representations, a recent seminalline of works [1, 2, 3] presents an intriguing phenomenon that persists across a range of canonicalclassiﬁcation problems during the terminal phase of training.
Speciﬁcally, last-layer features andlast-layer linear classiﬁers of a trained DNN exhibit simple but elegant mathematical structures, inthe sense that
• (NC1) Variability Collapse: the individual features of each class concentrate to their class-means.• (NC2) Convergence to Simplex ETF: the class-means have the same length and are maximallydistant.
In other words, they form a Simplex Equiangular Tight Frame (ETF).
• (NC3) Convergence to Self-duality: the last-layer linear classiﬁers perfectly match their classmeans.
Such a phenomenon is referred to as Neural Collapse ( NC ) [1], which has been shown empirically topersist across a broad range of canonical classiﬁcation problems, on different loss functions (e.g.,cross-entropy (CE) [1, 4, 3], mean-squared error (MSE) [5, 6], and supervised contrasive (SC) losses
[Figure]
[Table]
[7]), different neural network architectures (e.g., VGG [8], ResNet [9], and DenseNet [10]), and ona variety of standard datasets (such as MNIST [11], CIFAR [12], and ImageNet [13], etc).
Relatedphenomena have further been observed in a number of recent works [14, 15, 16, 17, 18, 19, 20, 21],including state-of-the-art natural language models (such as BERT, RoBERTa, and GPT) [15].
Motivations & contributions.
In this work, we try to understand why NC happens with a commonpractice that the last-layer features are normalized. Our work has the following motivations.
• Feature normalization is a common practice in training model deep networks.
Recently, many existing results demonstrated that training with feature normalization often improves the quality
of learned representation with better class separation [22, 23, 24, 7].
As illustrated in Fig.
and Table 1, experimental results visualized in low-dimensional space show that features learned
with normalization are more uniformly distributed over the sphere and hence are more linearly separable than those learned without normalization.
As a result, the better learned representations lead to improved test performances as shown in [25, 7].
Additionally, recent works [22, 23] also demonstrate improved robustness when learning with feature normalization. • Many existing works study NC with norm constraints.
As shown in [26, 27, 3, 7, 28], many existingtheoretical studies for NC consider the formulations with the norms of both features and classiﬁersconstrained.
Based upon unconstrained feature models [29], it has been shown that the only global solutions are NC for a variety of loss functions (e g., MSE, CE, SC losses). Nonetheless, these
works only focused on the global optimality conditions without looking into their nonconvex landscapes and therefore failed to explain why global NC solutions can be efﬁciently reached by a training algorithm such as stochastic gradient descent (SGD)
Thus, in this work we study the global nonconvex landscape of the training deep networks with normconstraints on the features and classiﬁers. We consider the commonly used
cross -entropy (CE) lossand formulate the problem as a Riemannian optimization problem over the product of unit spheres(i.e., oblique manifold).
Our study is also based upon the assumption of the so-called unconstrained feature model (UFM) [29] or layer-peeled model [30], where the
last -layer features of the deep network are treated as free optimization variables to simplify the nonlinear interactions across layers.
The underlying reasoning is that modern deep networks are often highly overparameterized with thecapacity of learning any representations [31, 32, 33], so that the last-layer features can approximate,or interpolate, any point in the feature space.
Assuming the UFM, we show that the Riemannian optimization problem has a benign global landscape, in the sense that the loss with respect to (w.r.t.)
the features and classifers is a strict saddlefunction [34, 35] over the manifold.
We show that every local minimizer is a NC solution, and all theother critical points exhibit directions with negative curvature.
Our analysis for the manifold settingis based upon a nontrivial extension of recent studies for the unconstrained formulation, which couldbe of independent interest.
Our work brings new tools from Riemannian optimization for analyzingdeep learning.
At the same time, we empirically show the advantages of our constrained formulationover its penalized counterpart for solving the training problems, and also demonstrate its practicalbeneﬁts for training deep networks.
Lastly, we believe the benign landscape over the manifold isnot limited only to CE loss. It could potentially hold for other popular training losses such as thecontrastive loss [7], which is an exciting direction for future work.
Prior art and related works on NC .
The empirical NC phenomenon has inspired a recent lineof theoretical studies on understanding why it happens [7, 30, 26, 29, 5, 6, 4].
Like ours, most ofthese works studied the problem under the UFM.
In particular, recent works showed that the onlyglobal solutions are NC solutions for a variety of nonconvex training losses (e.g., crosss-entropy (CE)[4, 30, 26], mean squared error (MSE) [5, 6], supervised contrastive (SC) [7]) and different problemformulations (e.g., penalized unconstrained, constrained).
Recently, this study has been extended todeeper models under the MSE training loss [5].
More surprisingly, it has been further shown thatthe nonconvex losses under the UFM have benign global optimization landscapes, in the sense thatevery local minimizer satisﬁes NC properties and the remaining critical points are strict saddles withnegative curvature.
Such results have been established for both CE and MSE losses [4, 6], wherethe studies in [4, 6] are limited to unconstrained versions of the problem with regularization on bothfeatures and classiﬁers.
Additionally, there is a line of recent works deciphering generalization of deep networks in termsof NC .
The work [36] shows that NC also happens on test data drawn from the same distributionasymptotically, but not for ﬁnite samples [37].
Other works [37, 38] demonstrated that the variabilitycollapse of features is actually happening progressively from shallow to deep layers, and [39]showed that test performance can be improved when enforcing variability collapse on features ofintermediate layers.
The works [40, 41] showed that ﬁxing the classiﬁer as a simplex ETF improvestest performance on imbalanced training data and long-tailed classiﬁcation problems.
Notation.
Let R n be the n -dimensional Euclidean space and (cid:107) · (cid:107) 2 be the Euclidean norm.
Wewrite matrices in bold capital letters such as A , vectors in bold lower-case such as a , and scalarsin plain letters such as a .
For a matrix A ∈ R d × K , we denote by a k its k -th column and by a i itsi -th row.
We use diag( A ) to denote a vector that consists of diagonal elements of A and ddiag( A )to denote a diagonal matrix whose diagonal elements are the diagonal ones of A .
Given a positiveinteger n , we denote by [ n ] the set { 1 , . . . , n } .
Nonconvex Formulation with Spherical Constraints
In this section, we review the basic concepts of deep neural networks and introduce notation that willbe used throughout the paper.
Based upon this, we formally introduce the problem formulation overthe Riemannian manifold with the unconstrained feature model.
2.1 Basics of Deep Neural Networks
In this work, we focus on the multi-class ( K class) classiﬁcation problem.
Given input data x ∈ R D ,the goal of deep learning is to learn a deep hierarchical representation (or feature) h ( x ) =
φ θ ( x ) ∈ 107 R d of the input along with a linear classiﬁer 1 W ∈ R
d
× K such that the output ψ Θ ( x ) =
W (cid:62) h ( x )of the network ﬁts the input x to an one-hot training label y ∈ R K .
More precisely, in the vanilla form an L -layer deep network can be written as
[Equation]
where each layer is composed of an afﬁne transformation, represented by some weight matrix W k ,and bias b k , followed by a nonlinear activation σ ( · ) , and Θ = { W k ,
b k } Lk
=1 and θ = { W k , b k } L − 1 k
=1denote the weights for all the network parameters and those up to the last layer, respectively.
Giventraining samples { x k,i , y k } ∈ R d × R K drawn from the same data distribution, we learn the networkparameters Θ via minimizing the empirical risk over these samples,
[Equation]
where y k ∈ R K is a one-hot vector with only the k th entry equal to unity and the remaining onesequal to zero for all 1 ≤ k ≤ K , x k
,i ∈
R D is the i -th sample in the k -th class, { n k } Kk =1 denotesthe number of training samples in each class, and the set C denotes the constraint set of the networkparameters Θ that we will specify later.
Here, we study the cross-entropy loss of the form
[Equation]
where π k ( z ) denotes that z belong to the k -the class.
2.2 Riemannian Optimization over the Product of Spheres
For the K -class classiﬁcation problem, let us consider a simple case where the number of trainingsamples in each class is balanced (i.e., n = n 1 = n 2 = · · · = n K ) and let N = nK , and we assumethat all the biases { b k } Lk =1 are zero with the last activation function σ ( · ) before the output to belinear.
Analyzing deep networks ψ Θ ( x ) is a tremendously difﬁcult task mainly due to the nonlinearinteractions across a large number of layers.
Fortunately, based upon previous study on unconstrainedfeature models [28, 29, 7, 4], we can simplify the nonlinear interactions by treating the last-layerfeatures h k,i =
φ θ ( x k,i ) ∈ R d as free optimization variables, where the underlying reasoning is thatmodern deep networks are often highly overparameterized to approximate any continuous function[31, 32, 33].
Speciﬁcally, we rewrite all the features in a matrix form
[Equation]
=
[Equation]
and correspondingly denote the classiﬁer W by
[Equation]
We assume that both the features H and the classiﬁers W are normalized, in the sense that (cid:107) h k,i (cid:107) 2 =1 , (cid:107) w k (cid:107)
= τ , for all k ∈ [ K ] and all i ∈ [ n ] , where τ > 0 is the temperature parameter.
As a result,we study NC with a constrained formulation over a Riemannian manifoldmin W , H
[Equation]
Since the temperature parameter τ can be absorbed into the loss function, we can replace w k by τ w kand change the original constraint into (cid:107) w k (cid:107) 2 = 1 .
Moreover, the product of spherical constraintsforms an oblique manifold [42] embedded in Euclidean space, i.e.,
[Equation]
Consequently, we can rewrite the constrained problem as a Riemannian optimization problem overthe oblique manifold w.r.t.
W and H as
[Equation]
We will show that all global solutions satisfy NC properties, and the function (3) is a strict saddlefunction [43, 44] of ( W , H ) over the oblique manifold so that the NC solution can be efﬁcientlyachieved.
Riemannian derivatives over the oblique manifold.
In the following, we will use tools from Riemannian optimization to characterize the global optimality condition and the geometric propertiesof the optimization landscape of the problem (3).
Thus, let us introduce some basic derivations of the Riemannian gradient and Hessian, deﬁned on the tangent space of the oblique manifold.
For moredetails, we refer the readers to the appendices.
According to [45, Chapter 3 & 5] and [46, 47], we can calculate the Riemannian gradients and
Hessian of Problem (3) as follows.
According to [45, Example 3.18], we have the tangent space to OB ( d, K ) at W as
[Equation]
We can similarly derive the tangent space for H .
Let us deﬁne
[Equation]
First, we can derive the Riemannian gradient of f ( W , H ) of Problem (3) as
[Equation]
Second, for any ∆ = ( ∆ W , ∆ H ) ∈ T W OB ( d, K ) × T H OB ( d, N ) , we compute the Hessianbilinear form of f ( W , H ) along the direction ∆ by
[Equation]
Based on this, according to [45, Example 5.32], we can derive the Riemannian Hessian bilinear formof f
( W , H ) along the direction ∆ as
[Equation]
In the following, we will use the Riemannian gradient and Hessian to characterize the optimizationlandscape of (3).
Main Theoretical Analysis
In this section, we ﬁrst characterize the structure of the global solution set of Problem (3).
Based uponthis, we analyze the global landscape of Problem (3) via characterizing its Riemannian derivatives.
3.1 Global Optimality Condition
For the feature H , let us denote the class mean for each class as
[Equation]
[Equation]
Based upon this, we show any global solution of the problem (3) exhibits NC properties in thesense that it satisﬁes ( NC1 ) variability collapse, ( NC2 ) convergence to simplex ETF, and ( NC3 )convergence to self-duality.
Theorem 1 (Global Optimality Condition) .
Suppose that the feature dimension is no smaller thanthe number of classes (i.e., d ≥ K ) and the training labels are balanced in each class (i.e., n = n 1 =· · · = n K ).
Then, it holds for the CE loss f ( W , H ) in problem (3) that
[Equation]
for all W =
[ w 1 , . .
. , w K ] ∈ R d × K and H =
[ h 1 , 1 , . . .
, h K,n ] ∈ R d × N .
In particular, equalityholds if and only if there are h 1 , h 2 , · · · , h K ∈ R d such that [Figure] Empirical
[Figure]
a l V a l u e Theoretical Empirical • ( NC1 ) Variability collapse: h k,i = h k , ∀ i ∈ [ n ] ;
• ( NC2 )
Simplex ETF structure: { h k } Kk =1 form a simplex ETF in the sense that
[Equation]
• ( NC3 ) Self-duality: w k = h k , ∀ k ∈
[ K ] .
The proof follows similar ideas of a line of recent works [7, 30, 26, 5, 6, 4], and we refer the readersto the appendices for the proof.
The result of our problem (3) is closely related to [7, Theorem 1] (i.e.,spherical constraints vs. that of ball constraints).
In fact, our problem and that in [7] share the sameglobal solution set.
Additionally, compared to the unconstrained regularized problems in [6, 4], it isworth noting that the regularization parameters there inﬂuence the structure of global solutions, whilethe temperature parameter τ only affects the optimization landscape but not the global solutions.
3.2 Global Landscape Analysis
Due to the nonconvex nature of Problem (3), the characterization of global optimality alone in
Theorem 1 is not sufﬁcient for guaranteeing efﬁcient optimization to those desired global solutions.
Thus, we further study the global landscape of Problem (3) by characterizing all the Riemanniancritical points ( W , H ) with
[Equation]
As such, we state our major result below.
Theorem 2 (Global Landscape Analysis) .
Assume that the number of training samples in each classis balanced n = n 1 = · · · = n K .
If the feature dimension is larger than the number of samplesd > N , the function f ( W , H ) is a strict saddle function that has no spurious local minimum, in thesense that
• Any local minimizer of Problem (3) is a global minimizer of the form shown in Theorem 1.•
Any Riemannian critical point ( W , H ) of Problem (3) that is not a local minimizer is a Riemannianstrict saddle point with negative curvature, in the sense that the Riemannian Hessian Hess f ( W , H )at the critical point ( W , H ) is non-degenerate, and there
exists
a direction ∆ = ( ∆ W , ∆ H ) ∈ T W OB ( d, K ) × T H OB ( d, N ) such that
[Equation]
In other words, λ min (Hess f ( W , H )) < 0 at the corresponding Riemannian critical point.
The major challenge of our analysis is to ﬁnd the negative curvature for Hess f ( W , H ) . Our keyobservation is that, because d > N and H ∈ R d × N , there always exists a non-zero a ∈ R d such thatit lies the null space of H , i.e., H (cid:62)
a = 0 .
According to this, (4), (5), and the properties of ∇ g ( M ) ,we can show that the same a ∈ R d is also in the null space of W (cid:62) , i.e., W (cid:62) a = 0 . This allows usto construct a direction along which the leading bilinear form of ∇ 2 g ( M ) in (6) vanishes.
Basedon this and the optimality condition for Problem (3), we can construct a rank-1 negative curvaturedirection ( ∆ W , ∆ H ) for W and H , respectively. For more details of the proof, we refer readers tothe appendices.
In the following, we discuss the implications, relationship, and limitations of ourresults.
• Efﬁcient global optimization to NC solutions.
Our theorem implies that, for optimizing ( W , H )via Problem (3), the NC solutions can be efﬁciently reached by Riemannian ﬁrst-order methods(e.g.
, Riemannian stochastic gradient descent) with random initialization [43, 48, 49]; see Figure
for a demonstration.
For training practical deep networks, this can be efﬁciently implemented by normalizing last-layer features when running SGD.
• Relation to existing works on NC .
Most existing results have only studied the global minimizersunder UFM [7, 30, 26, 5], which has limited implication for optimization.
On the other hand, ourlandscape analysis is based upon a nontrivial extension of that with the unconstrained problem formulation [6, 4].
Compared to those works, Problem (3) is much more challenging for analysis, due to the fact that the set of critical points of our problem is essentially much larger than that of[6, 4].
Moreover, we empirically demonstrate the advantages of the manifold formulation over its
regularized counterpart, in terms of representation quality and training speed.
• Limitations on the assumptions of the feature dimension d .
The major limitation of our current result is that we require relatively large d > N , 2 which could hold for modern highly overparameterized
[-even when d (cid:28) K , while the global solutionsare uniform over the sphere [36] rather than being simplex ETFs (see Figure 1). The analysis ond (cid:28) K is left for future work. On the other hand, the required upper bound on τ is for the easeof analysis and it holds generally in practice, 
 but we conjecture that the benign landscape holdswithout it.-]networks [50, 31, 32, 51, 33].
We believe that this bound can be improved to d > K + 1 by a moredelicate analysis; this is also corroborated by our experimental results (see Figure 2).
Furthermore,through numerical simulations we conjecture the global landscape also holds even when d (cid:28) K ,while the global solutions are uniform over the sphere [26] rather than being simplex ETFs (seeFigure 1).
The analysis on d (cid:28) K is left for future work.
• Relation to other Riemannian nonconvex problems.
Our result joins a recent line of work on the study of global nonconvex landscapes over Riemannian manifolds, such as orthogonal tensor decomposition [34], dictionary learning [52, 53, 54, 55, 56], and sparse blind deconvolution
For all these problems constrained over a Riemannian manifold, it can be shown that they exhibit “equivalently good” global minimizers due to symmetries and intrinsic low-dimensional structures, and the loss functions are usually strict saddles [34, 35, 61].
As we can see, the global minimizers (i.e., simplex ETFs) of our problem here also exhibit a similar
rotational symmetry, in the sense that W (cid:62) H = ( QW ) (cid:62) ( QH ) for any orthogonal matrix Q .
Additionally, by adapting tools from Riemmanian optimization, our result showed how these tools can be powerful for the study of deep learning.
Experiments
In this section, we support our theoretical results in previous sections and provide further motivationwith experimental results on practical deep network training.
In the ﬁrst experiment, we validate theassumption of UFM introduced in Section 2 for analyzing NC , by demonstrating that NC occursfor increasingly overparameterized deep networks.
In the second experiment, we further motivatefeature normalization with empirical results showing that feature normalization can lead to fastertraining and better collapse than the unconstrained counterpart with regularization.
This occurs notonly with the UFM but also with practical overparameterized networks.
Before that, let us introducesome basics of the experimental setup and metrics for evaluating NC . Network architectures, datasets, and training details.
In our experiments, we use ResNet [9]architectures for the feature encoder.
For the normalized network, we project the output of theencoder onto the sphere of radius τ (as done in [7]) and also project the weight classiﬁers to the unitsphere after each optimization step to maintain constraints.
In all experiments, we set τ = 1 .
For thenon-normalized UFM and network, we use a weight and feature decay of 10 − 4 (using the loss in [4]).
[Figure] [Figure] [Figure] [Figure]
[Figure]
Width=
[Figure]
Width=
Width=16 Width=
Width=     200 Epoch 0.
Width=4 Width=
Width=16 Width=
Width=     200 Epoch 0.
Width=4 Width=
Width=16 Width=
Width=     200 Epoch 0.
T r a i n i n g A cc u r a c y
Width=4 Width=
Width=16 Width=
Width=
We do not use a bias term for the classiﬁer for either architecture.
For both experiments, we use the
CIFAR dataset 3 [12], where we use CIFAR10 and CIFAR100 for the ﬁrst and second experiment,respectively.
In both experiments, we train the networks using SGD with a batch size of 128 andmomentum 0 .
9 with an initial learning rate of 0 . 05 , and we decay the learning rate by a factor of0 .
1 after every 40 epochs - these hyperparameters are chosen to be the same as those in [4] for faircomparisons.
All networks are trained on Nvidia Tesla V100 GPUs with 16G of memory.
Neural collapse metrics.
For measuring different aspects of neural collapse, we adopt similar NCmetrics from [1, 4, 6], given by
[Equation]
where Σ W and Σ B are the within-class and between-class covariance [1], Σ † B denotes pseudo inverseof Σ B , and H is the centered class mean matrix in (7).
More speciﬁcally, NC 1 measures NC1 withinclass variability collapse, NC 2 measures NC2 convergence to the simplex ETF, and NC 3 measures
NC3 duality collapse.
4.1 Validation of the UFM for training networks with feature normalization
In Section 2, our study of the Riemannian optimization problem (3) is based upon the UFM, wherewe assume H is a free optimization variable.
Here, we justify this assumption by showing that
NC happens for training overparameterized networks even when the training labels are completelyrandom.
By using random labels, we disassociate the input from their class labels, by which we cancharacterize the approximation power of the features of overparmeterized models.
To show this, wetrain ResNet-18 on CIFAR10 dataset with random labels, with normalized features and classiﬁerswith varying widths (i.e., the number of feature maps resulting from the ﬁrst convolutional layer).
As shown in Figure 3, we observe that increasing the width of the network allows for perfect classiﬁcation on the training data even when the labels are random.
Furthermore, increasing the networkwidth also leads to better NC , measured by the decrease in each NC metric.
This corroborates that( i ) our assumption of UFM is reasonable given that NC seems to be independent of the input data,and ( ii ) NC happens under the our constraint formulation (3) on practical networks.
4.2 Improved training speed and better representation quality with feature normalization
We now investigate the beneﬁts of using feature normalization, more speciﬁcally for improvedtraining speed and better representation quality.
First, we consider the UFM formulation, wherewe optimize Problem (3) and compare to the regularized UFM in [4].
The results are shown in
Figure 4.
We can see that normalizing features over the sphere consistently results in reaching perfectclassiﬁcation and greater feature collapse (i.e., smaller NC 1 ) quicker than penalizing the features.
To demonstrate that these behaviors are reﬂected in training practical deep networks, we train both
ResNet-18 and ResNet-50 architectures on a reduced CIFAR100 [12] dataset with N = 3000 totalsamples, comparing the training accuracy and neural collapse with and without feature normalization. The results are shown in Figure 5.
[Figure]
[Figure]
100 Epoch 0.
T r a i n i n g A cc u r a c y Normalized=
{+False Normalized= True    100 Epoch 
+}
Normalized= False Normalized= True
ResNet- From Figure 5 (left), we can see that for the ResNet-18 network, we reach perfect classiﬁcation ofthe training data about 10-20 epochs sooner by using feature normalization compared to that of theunconstrained formulation.
From Figure 5 (right), training the ResNet-50 network without feature normalization for 100 epochs shows slow convergence with poor training accuracy, whereas using featurenormalization arrives at above 90% training accuracy in the same number of epochs.
By keeping thesize of the dataset the same and increasing the number of parameters, it is reasonable that optimizingthe ResNet-50 network is more challenging due to the higher degree of overparameterization, yet thiseffect is mitigated by using feature normalization.
At the same time, for both architectures, using feature normalization leads to greater feature collapse(i.e.
, smaller NC 1 ) compared to that of the unconstrained counterpart.
As shown in recent work[1, 6, 36] , better NC often leads to better generalization performance.
Last but not least, we believethe beneﬁts of feature normalization are not limited to the evidence that we showed here, it couldalso lead to better generalization [25, 7] and robustness [22, 23] that are worth further exploration.
Conclusion & Discussion
In this work, we study the prevalence of the NC phenomenon with normalized features.
Basedupon the assumption of UFM, we formulate the problem as a Riemannian optimization problemover the product of spheres.
We showed that the loss function is a strict saddle function over the manifold with respect to the last-layer features and classiﬁers, with no other spurious local minimizers.
We demonstrate this on practical deep network training and show practical beneﬁts of feature normalization in terms of training speed and learned representation quality.
As future work, we would like to expand the study to other popular loss functions such as contrastive loss and study the settings d (cid:28) K , which could be of great importance for studying self-supervised learning
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] We state in Section {+
+}
that our
analysis uses the unconstrained feature model, and in Section 3 we discuss the strongrequirement of d > N for Theorem 2.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] Thispaper mainly focuses on understanding the neural collapse phenomena observed inpractical neural networks.
We believe there is no potential negative societal impact inthis work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] We explicitlymention the assumptions in Section 3.
(b) Did you include complete proofs of all theoretical results?
[Yes] We include all theproofs in the Appendix.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] The CIFAR
datasets used here are cited and easily accessible.
The GitHub code repository will be
made available after review.
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Section 4.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] Figure 2 does not show error bars since the attained value
is always the same independent of initialization due to the benign global landscape.
In Section 4, we used shaded area as error bars for Figure 4, but regardless we haveobserved that the results are similar and reproducible independent of the random seed.
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes] See Section 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We cite them in
Section 4.
(b) Did you mention the license of the assets?
[Yes] This is mentioned in Section 4.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] This is mentioned in Section 4.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]