Paper: Towards Understanding Neural Collapse: The Effects of Batch Normalization and Weight Decay (ICLR-2024)

Reviewer comment (Theory):
For the theoretical analysis, the layer-peeled model is a very specific setting. I understand that narrow settings may be required to obtain rigorous results. I would like to understand, however, the rationale behind this choice and how the authors believe that the results can generalize (also theoretically) to more general settings.

Author response:
As with prior works, we adapt the layer-peeled model due to the difficulty of analyzing the effect of multiple layers of non-linear activation functions and the variable neural network architecture possibilities of the lower layers. Suppose the activations functions of the Neural Network are linear (e.g. the identity function). In that case, our results can be relatively easily generalized to features of lower layers since the layers can be combined into a single linear transformation. This generalization to lower layers is much more difficult for non-linear activation functions and would likely require more intricate mathematical tools. This direction has recently been pursued by [1][2] for the MSE loss in the optimal case. However, the corresponding generalization for CE loss in the near-optimal case is more involved and likely to require significant future work.

Reviewer comment (Theory):
At the beginning of Section 2.2, the authors claim that quantities proposed in the past to quantify NC fail to deliver insights whenever the "values are non-zero". Why is that the case? And how does cosine similarity excel in this regard? To my basic understanding, a measure such as SNR (which I take to be the ratio between inter-class distances over intra-class distances) seems quite intuitive (and also accounts for vector lengths, which the proposed cosine similarity does not). To some extent, I would even believe that SNR allows a similar theoretical analysis as cosine similarity (assuming that SNR also depends on Euclidean distances).

Author response:
We thank the reviewer for pointing out this point of potential misunderstanding. We would like to clarify our choice of using cosine similarity as a measure:
    1. The cosine similarity measure is a widely used adopted similarity measure to compare between features of different samples in both practical feature learning and machine learning theory for its geometric interpretability. Quantitatively analyzing this measure allows our theorems to have more direct implications for researchers in such relevant fields
    2. The cosine similarity measure allows direct value comparison between NC1 and NC2, corresponding to inter-class and intra-class cosine similarity respectively, which is a benefit over prior works. For example, in the original NC paper [1], the authors adopted 2 different measures, SNR ratio and feature mean cosine similarity, for NC1 and NC2, which cannot be directly compared since they are 2 different measures.
    3. Connected to the previous point, while the SNR ratio compares intra-class variability and inter-class distance, a small SNR ratio does not distinguish whether both are small or both are large. In particular, the SNR ratio alone does not ensure proximity to the ETF structure. Explicitly measuring the cosine similarity for both intra-class and inter-class allows separate guarantees using the same measure that implies NC1 and NC2 (without norm equality) when combined.
    4. Cosine similarity also allows relatively simple quantitative analysis.
    
    We would like to note that we do not intend to decrease the value of the SNR ratio as a measure, which we’re convinced to have significant intuition and insights in other fields that are not captured by cosine similarity. However, the above reasons make us believe that cosine similarity is an appropriate measure for our purpose of quantitative analysis of Neural Collapse.

Reviewer comment (Evaluation):
Some experimental details are not clear (see questions below), I would appreciate clarifications. Most importantly, what are the training and test errors for the settings in Fig. 1 and 2. I would appreciate if these curves could be plotted on top of the existing ones, to get a better understanding of the training success.

Author response:
We thank multiple reviewers for making this suggestion and will adjust our plots accordingly. The questions are answered in the Questions section

Reviewer comment (Evaluation):
Another shortcoming of the paper is that the results for the larger architectures in Fig. 2 are not as clear; VGG11, for example, seems not to exhibit NC, at least not using the measures that are proposed. This limits the contribution of the paper slightly, as the phenomenon of NC is either not as prevalent as claimed, or the proposed measures are inadequate of capturing this phenomenon in interesting settings (i.e. for large architectures).

Author response:
Due to the intricacy of deep neural network training, it is hard to make certain assertions on the result of training complex networks. However, we think that the following reasons may be influential in the current results for the real-world networks and datasets:
    1. Achieving near-1 intra-class cosine similarity is a strong requirement on the features within the same class when the vector dimension is large.
    2. We used the *minimum* intra-class cosine similarity and *maximum* inter-class cosine similarity, which again is a strong requirement for the structure of feature vectors. This measure can be affected by outliers since there are $C(C-1)$ pairs of classes for inter-class cosine similarity
    
    To address the second point, we’re currently running experiments to also record the average intra-class and inter-class cosine similarity for the real-world models and dataset combinations to resolve the sensitivity to outliers. We will update the paper when they’re completed.

Reviewer comment (Novelty):
The use of cosine similarity is not novel. Cosine similarity has long been used in previous works (e.g. [1]) to characterize the inter-class and intra-class feature variations. The authors should avoid saying that they "propose the cosine similarity measure", and should provide a more thorough discussion of related literatures.

Author response:
We thank multiple reviewers for pointing out the potentially misleading expression of our contributions on the cosine similarity. We are aware that cosine similarity has been widely used in previous literature. Our main intent is to claim that we are novel in applying cosine similarity to the specific phenomenon of analyzing Neural Collapse, which has not been done before to the best of our knowledge at the time of writing. We will clarify our contributions with regard to this statement and discuss the related use of the cosine similarity measure for similar purposes in previous literature.

Reviewer comment (Evaluation):
The use of cosine similarity is not well justified. Cosine similarity can only guarantee the directions of the feature vectors are similar, but has nothing to do with the norm of the feature vector. Using it as a metric for NC is too weak.

Author response:
As mentioned in section 2.2, we adapt cosine similarity as a quantitative measure for NC due to the straightforward geometric meaning it provides compared to previous measures. In particular, cosine similarity is a widely adapted measure for similarity between features of different inputs, both in practical feature learning and in machine learning theory. Therefore, we believe that our measures already capture the majority of the implications of NC. The simplistic mathematical definition of the cosine similarity measure also allows for relatively straightforward quantitative analysis in the near-optimal regime that can be adopted by future works. Our main novelty lies in the quantitative analysis of the influence of various factors on the guaranteed proximity to the NC structure in the near-optimal regime. The extension of this analysis to the full predictions of NC, including norm equality, requires future work.

Reviewer comment (Theory):
The theoretical contributions are limited. The major theoretical contribution of this paper, is to establish a *near-optimal* NC configuration for a *near-optimal* CE loss. There are already various previous works proving this relationship for the exact setting (e.g. [2] ), and the paper mainly follow their proof outline. The novel component is Lemma 2.1, where the authors prove an approximate version of Jenson's inequality for strongly convex objectives. Therefore, I believe the major theoretical contribution is to directly extend previous guarantees for NC to the non-asymptotic cases. Furthermore, the exponential terms in the non-asymptotic bound implies that the result has a very high dependency on the parameter norm, which again questions whether such an extension is necessary or meaningful.

Author response:
We think that the main doubts of the reviewer on the contribution of our theorem can be summarized as a) the asymptotic nature of our theorems, b) the exponential dependency of the theorem, and c) limited theoretical contributions. We will respond to these points separately:
    1. We would like to note that our big-O notation is not intended to indicate that our bound only works when $\epsilon\rightarrow 0$. We addressed this in detail in the global response.
    2. For theorem 3.1, we believe that the exponential dependence on $\alpha\beta$ does not make the theorem vacuous or meaningless. Under the presence of batch normalization and high weight decay, the value of $\alpha\beta$ are not typically large. In fact, the dependence of the bound on the weight and feature norms is the exact point made by the theorem.
        
        For theorem 3.2, the exponential dependence on $\frac{1}{\lambda}$ may indeed be vacuous in a large number of settings. We have carefully examined our proof and found that our results can be improved to remove this exponential dependence. The newly derived bound has been noted in the global response.
        
    3. In analyzing the near-optimal scenario, we quantitatively investigate the effects of weight decay parameters and the dependence on the loss value, offering a novel perspective on the understanding of weight decay and training loss value. Moreover, our result—focusing on the near-optimal regime with batch normalization and weight decay—more closely mirrors realistic settings, in contrast to the less realistic assumptions made in other papers analyzing NC under CE loss (e.g. [1][2]) such as exact optimality, strictly normalized feature vectors, or direct optimization of last-layer features.

Reviewer comment (Writing):
Some terminologies are not accurate. For example, the authors repeatedly use "unbiased neural network" to refer to network architectures without bias, which causes ambiguity due to its statistical meaning.

Author response:
We thank the reviewer for pointing out this potentially misleading expression. To avoid confusion, we will change the expression to “neural networks without the bias term”.

Reviewer comment (Experiments):
The original NC phenomenon contains four aspects (conditions) but this paper only analyzes the NC1 and NC2.

Author response:
- It is true that our current theorems only provide guarantees on NC1 and NC2. Compared with the full NC phenomenon, the original NC paper showed that NC4 is implied by NC1, NC2 and NC3, therefore the main missing part in our work is NC3. We did not officially include the bounds for NC3 in our main theorem since properties of class-feature vectors are typically of more practical significance than last-layer weight vectors. However, the corresponding bound for NC3 is available as an intermediate result of our proof (page 23, bottom) with the same asymptotic bound as inter-class cosine similarity. We will gladly include this as a separate result if it strengthens the contribution of the paper.

Reviewer comment (Novelty):
As previous works on NC, this paper is also limited to the last-layer feature vectors.

Author response:
- Indeed, the scope of our work, like previous studies on NC, is confined to the last-layer feature vectors. This approach aligns with the original NC claim in [1].

Reviewer comment (Theory):
The affine parameter $\gamma$ for BN is not included in theorem 3.2, it is not clear that how BN influences the inter- and intra-class cosine similarities.

Author response:
- We believe the reference is to theorem 2.2. Here, the the affine parameter $\gamma$ is regularized by weight decay, as shown in $L_{\text{reg}} = L + \frac{\lambda}{2}(\|\gamma\|^2 + \|\boldsymbol{W}\|^2_F)$ in section 2.1. The theorem's conclusions are a result of the combined effect of batch normalization and weight decay, with the influence of batch normalization being implicit in the $\gamma$ parameter's effect. We will clarify this in the revised manuscript.  

The purpose of theorem 2.2 is to establish a bound that is not dependent on the feature and weight norms and only on the weight decay parameter and the loss gap. A theorem directly incorporating the $\gamma$ parameter can be obtained by directly substituting the $\beta$ parameter with $\|\gamma\|$ (The correctness of this substitution is established by proposition 2.1). We will clarify this in future versions of the paper.

Reviewer comment (Theory):
The bias term of BN is ignored in this paper, which could be important as it would greatly influence the feature vectors' norms and further influence the bound over the cosine similarity.

Author response:
- We appreciate the reviewer highlighting the significance of the bias term in BN, which can indeed impact vector norms and consequently the cosine similarity measures. Our current definitions of intra-class and inter-class cosine similarity assume globally centered feature vectors. To address networks with biased batch normalization, a more general cosine similarity measure for these similarities where the feature vectors are globally centered should be used which centers the feature vectors by subtracting the mean. The globally centered feature vectors is also used in the original proposal of NC in [1]. This revised definition will be included in the updated version of our paper.

Reviewer comment (Evaluation):
For Sec 3., only minimum intra-class and maximum inter-class cosine similarities are analyzed. I thought the authors might want to show the hardest case for both intra-class and maximum inter-class cosine similarities. But, to be pointed, the average value should be also measured as it could capture some general information of the cosine similarities across each pair of classes.

Author response:
- We thank the reviewer for suggesting the inclusion of average values for intra-class and inter-class cosine similarities. This will be incorporated in the revised manuscript  

There is a very significant gap between models with and without BN for the worst-class intra-class and inter-class cosine similarity (i.e. the second and fourth column in Fig 3). The gap in the average case is less pronounced but is still a considerable gap, especially at high weight decay values. We believe that these are sufficient justification for our empirical conclusions.

 This is because achieving high average intra-class cosine similarity and low average inter-class cosine similarity is a much more lenient requirement. Achieving a high value for the average measure still allows individual classes/class pairs to significantly diverge from the Neural Collapse requirement.

Reviewer comment (Theory):
In Fig 2., some experimental results cannot be predicted by their theorems. For example, for conic hull dataset and 3-layer MLP, at the same level of weight decay (e.g., $10^{-4}$), the model without BN layers holds better intra-class cosine similarities.

Author response:
- The phenomenon observed by the authors with the conic hull dataset and 3-layer MLP is indeed interesting. However, our theoretical results suggest that higher weight decay, coupled with batch normalization, leads to a greater degree of Neural Collapse at near-optimal loss. The experiments aim to demonstrate this relationship, particularly under the influence of BN and show that BN models have higher neural collapse than non-BN models at high weight decay. The comparison between models with and without BN across smaller weight decay values is beyond the scope of our current theorems since the theoretical guarantees provided by BN under very small weight decay are mostly vacuous as suggested by our theorems.  

We do not claim that the experimental results are strictly predicted by our theorems, which provide a worst-case bound on the inter-class and intra-class cosine similarity. The exact values in the experiments are highly dependent on the training dynamics and the neural network architecture. 
The main purpose of the experiments is to establish that BN and high weight decay achieve high levels of NC, which is verified by our experiments.

Reviewer comment (Experiments):
In Fig 3., I don't see the value of including the results of ResNet. There is no clear baseline for ResNet (with BN) for comparison.

Author response:
- Our objective was to illustrate that Neural Collapse levels increase with weight decay in the presence of BN, as is shown and justified in the ResNet figures even without a non-BN baseline. Training a ResNet model without BN layers to achieve near-optimal loss is challenging, which is why a direct baseline comparison was not included.

Reviewer comment (Presentation):
The curves of Model (w/o BN) should be put into one single figure for better comparison.

Author response:
- We acknowledge the feedback regarding the figures. We will address these points by generating higher-resolution images and reorganizing the figures for clearer comparison in the revised manuscript.

Reviewer comment (Theory):
Is there any benefit to analyze the near-optimal regime over the optimal one?

Author response:
There are 2 main benefits of analyzing the near-optimal regime

1) The exact optimal loss is rarely achieved in practical training. Therefore, while prior works provide insights into the ideal case, our theorems extend their results into more practical final losses.

2) Another more important reason is that analyzing the near-optimal regime allows quantitative analysis of the effect of different parameters on the proximity to NC. For example, while prior works have shown that feature normalization/regularization and optimal loss guarantee the exact NC structure, our theoretical results show the quantitative relationship between the guaranteed proximity to the NC structure and weight decay, which is not possible by only analyzing the optimal case.

Reviewer comment (Theory):
The result in Theorems 2.1 and 2.2 is only asymptotic, and thus it cannot be used to estimate proximity to NC from a given loss value.

Author response:
1. While the bound in theorems is formulated using asymptotic notation, the use of this notation is mainly intended to remove the constants and higher-order terms to simplify the formula in the theorem. It is possible to use our result to generate predictions for given values of $\epsilon$ as long as they are small enough for higher order terms of $\epsilon$ (e.g. $\epsilon^2$) to be safely ignored, which is the case for most successful training of over-parameterized neural networks. We have placed the formula for the asymptotic bound in the global response
    
    We would like to further note that, the main insight gained from the bounds in the theoretical result is its dependency relationship on the parameters rather than the exact value. Therefore, we believe that using the asymptotic notation is sufficient in the main paper as all the dependency relationships are captured with such notation.

Reviewer comment (Theory):
The bound in Theorem 2.2 is used as the basis to argue that under the presence of batch normalization and weight decay of the final layer, larger values of weight decay provide stronger NC guarantees because the intra-class cosine similarity of most classes is nearer to 1 and the inter-class cosine similarity of most pairs of classes is nearer to -1/(C-1). However, this argument is flawed because the loss function itself depends on the regularization parameter λ, making it unclear whether ε-values are comparable for different λ. The provided counter-example with L(x,λ) = λx² illustrates that this reasoning may lead to incorrect conclusions.

Author response:
2. We thank the reviewer for the meticulous review of the robustness of our second theorem. We acknowledge that the loss function of our second theorem is dependent on $\lambda$. However, we’re currently not convinced that the reviewer’s argument challenges the robustness of our theorem and would appreciate further elaborations on the argument, as outlined below:
    1. In the example given by the authors, the objective function is strongly convex with respect to the bounded variable (i.e. x), and the convexity of the function with respect to x is directly dependent on $\lambda$. This resulted in the trivial conclusion that a larger $\lambda$ value provides stronger guarantees on the value of $x$. However, in our theorems, $\lambda$ is the coefficient of the norm of the weights and the feature vectors, and thus the loss function is strongly convex with respect to the weight and feature norms. However, the intra-class and inter-class cosine similarity that our theorem provides guarantees on is not directly affected by the norm of the feature vectors since proportionally scaling all features does not affect the cosine similarity values. Therefore, the claim that “larger weight decay results in stronger guarantees under the same $\epsilon$ value from the optimal loss” is non-trivial in our theorems, unlike the example provided by the authors to illustrate their point.
    2. If $\epsilon$ is replaced with $\epsilon\lambda$ as the reviewers suggested, the main conclusions of our work are still upheld. Notably, the bounds in theorem 3.2 are still inversely correlated to $\lambda$ if $\epsilon$ is replaced with $\epsilon\lambda$

Reviewer comment (Theory):
There are doubts about how batch normalization is handled. In the proofs, batch normalization enters only through the condition Σᵢ ||hᵢ||² = ||hᵢ||² (Proposition 2.1), which is only an implication and does not capture stronger constraints induced by batch normalization. The theorems assume that the loss minimizer is a simplex ETF in the presence of batch normalization, but this is neither proven nor discussed, and not accounted for in deriving the loss minimum m_reg in Theorem 2.2.

Author response:
**Reply to Concern on Batch Norm:** It is true that Batch Normalization provides stronger guarantees than what is used in theorems and may lead to tighter guarantees. However, the constraint we used is a necessary implication of Batch Normalization and additional constraints of BN thus do not influence the validity of our theorem. It may be possible that additional constraints by BN may lead to better results, but this is not within the scope of this paper. On your second point where “the theorems assume that the loss minimizer is a simplex ETF in the presence of batch normalization”, our theorems do not rely on this particular assumption. Instead, this statement can be derived from our theorem by setting $\epsilon=0$. Could you explain more on where in our proof is this assumption necessary so that we can either provide more explanation or clarification in our proof?

Reviewer comment (Experiments):
Theorems 2.1 and 2.2 are not evaluated empirically. It is not tested whether the average intra-class and inter-class cosine similarities of near optimal solutions follow the exponential dependency in λ and the square (or sixth) root dependency on ε as suggested by the theorems.

Author response:
- Our bounds provide the worst-case scenario bound on the intra-class and inter-class cosine similarity. Since we proved our theorems in the appendix, it is not necessary to verify our theorems experimentally. I believe the reviewer is referring to verifying how close the actual cosine similarity is to the predicted bound. However, we would like to note that the exact intra-class and inter-class cosine similarity resulting in a training process is heavily dependent on the training dynamics and the underlying model architecture, therefore predicting the exact cosine similarity value or its scaling is much more involved and is not within the scope of the paper. The main insights from the theorems are that 1) Batch normalization and high weight decay imply neural collapse under near-optimal loss and 2) higher weight decay has stronger guarantees of proximity to the Neural Collapse structure. The purpose of our experiments is to investigate these two insights to see whether they transfer into practice.

Reviewer comment (Experiments):
Instead of evaluating the theorem predictions, the experiments measure the dependency of cosine similarities at the end of training (200 epochs) on weight decay strength. While intra-class cosine similarities improve with higher weight decay strength, there are problems with this experimental setup, as it is inconsistent with the theoretical setting and thus provides only limited insight into whether the theoretical results transfer to practice.

Author response:
- As clarified in the previous point, the purpose of the experiments is not to verify whether the worst-case bounds in our theorems provide a good prediction of the actual intra-class and inter-class cosine similarities, which is heavily dependent on the training dynamics and architecture. Rather, the experiments substantiate the insights gained from the theoretical results, namely batch normalization is an important factor in Neural Collapse Emergence, and higher weight decay results in stronger guarantees of Neural Collapse.

Reviewer comment (Experiments):
Inconsistency between theory and experiments — the theory depends only on weight decay strength on the last layer parameters, but in the experiments, weight decay is applied to all layers, with its strength varied across experiments, instead of varying only the last layer’s weight decay.

Author response:
1. We thank the reviewer for pointing out that applying batch normalization for every layer is suboptimal. We have rerun our experiments with only last-layer BN along with other changes and the results have indeed shown improved Neural Collapse under our measures.

Reviewer comment (Theory):
The theorems assume near optimal training loss, but training losses are not reported. The reported cosine similarities are far from optimal (e.g., intra-class is around 0.2 instead of 1), suggesting the training loss is also far from optimal and that model capacity may be too small to justify the ‘unconstrained-features’ assumption.

Author response:
2. We thank the reviewer for pointing out that including training losses will make our plots more informative. We would first like to note that, our theorems do not rely on the unconstrained features assumption so long as near-optimal loss is obtained. On the doubts on the reported intra-class and inter-class results, we think that the following reasons may be influential in the current results for the real-world networks and datasets:
        1. Achieving near-1 intra-class cosine similarity is a strong requirement on the features within the same class when the vector dimension is large.
        2. We used the *minimum* intra-class cosine similarity and *maximum* inter-class cosine similarity, which again is a strong requirement for the structure of feature vectors. This measure can be affected by outliers since there are $C(C-1)$ pairs of classes for inter-class cosine similarity
        
        To address the second point, we would report the average intra-class and inter-class cosine similarity to provide additional measures that are less sensitive to outliers. For the synthetic data experiments, we also adjusted the model architecture to only include batch normalization in the final layer and this has indeed improved the measured results.

Reviewer comment (Theory):
On real-world datasets, the inter-class cosine similarity increases with weight decay (even for batch normalization models like VGG11), which contradicts the theoretical prediction. This contradiction is insufficiently acknowledged.

Author response:
4. We thank the reviewer for pointing out this result that seems contradictory to the insights from our theorems. We would acknowledge this phenomenon in our updated version.

Reviewer comment (Novelty):
The central question of the work — “What is a minimal set of conditions that would guarantee the emergence of NC?” — has already been answered in prior work, showing that minimal loss plus a norm constraint on features (via feature normalization or weight decay) implies neural collapse. Adding batch normalization contradicts minimality.

Author response:
1. **Minimal set of conditions for NC.**
    1. **Motivation.** Past work has demonstrated the emergence of Neural Collapse under various assumptions, such as minimal loss combined with regularization[1], and minimal loss with feature normalization [2][3]. However, there is a significant gap between theory and practice. Empirically, we rarely achieve the optimal loss exactly, nor do we apply regularization and normalization precisely as assumed in these papers. Therefore, the motivation of our paper is to bridge this gap. Specifically, we aim to theoretically identify a minimal set of conditions closely related to practical applications that guarantee the emergence of Neural Collapse.
    2. **Batch normalization.** The reviewer’s claim that 'The authors argue to add batch normalization to this list but that contradicts minimality' does not accurately reflect our position. Firstly, previous studies have also incorporated various normalization assumptions to facilitate Neural Collapse. Compared to these normalizations, our use of batch normalization and weight decay is more prevalent in practical applications. Therefore, employing batch normalization should not be viewed as an 'additional' assumption, but rather as an alternative or practical implementation of the norm constraint assumption. Secondly, our experiments demonstrate that without batch normalization, Neural Collapse is not significantly observed in some datasets (see Figures 2 and 3). This underscores the necessity of batch normalization from another perspective. Below we outline the role of BN and WD in our theorems in detail:
        1. To illustrate the above point, our first theorem shows the case where “norm constraint on the features” and near-optimal loss implies Neural Collapse. In this theorem, our theoretical result does not rely on Batch Normalization and only requires the feature norm to be bounded.
        2. Our second theorem replaces the feature and weight norm constraint with a batch-normalization layer and weight decay. Our results show that replacing the strict norm constraint with a norm penalty is also sufficient for Neural Collapse. Therefore, the BN assumption is not added as an additional assumption to the bounded norm constraint assumption but as a replacement that better resembles practical training.
    3. **Sub-optimal Loss.** Another area where our work contributes is in generalizing from the optimal scenario to the sub-optimal one. The theorems in previous works assume a strictly optimal loss, which is a much more stringent requirement than a near-optimal loss, the latter being more representative of practical scenarios. Therefore, in this regard, our assumptions are more 'minimal' compared to those in prior theorems. As we clarified previously, our big-O notation is used to simplify the formula by removing constants and higher order terms does not imply that our theorem only holds when $\epsilon\rightarrow 0$.
    4. **Weight decay.** Furthermore, we have provided a quantitative analysis of how various parameters influence the bound, specifically the gap from optimal loss and the weight decay parameter, aspects not explicitly analyzed in prior works. Weight decay, while widely used in practice, has a sparse theoretical understanding. Our work contributes to understanding weight decay from the perspective of NC.

Reviewer comment (Novelty):
The first listed contribution, proposing intra-class and inter-class cosine similarity measures to quantify proximity to NC, is not a genuine novelty. Cosine similarity is a well-known metric, and its use for measuring closeness to NC was already established in Papyan et al. (2020) and other works.

Author response:
2. We thank multiple reviewers for pointing out the potentially misleading expression of our contributions on the cosine similarity. We are aware that cosine similarity has been widely used in previous literature. Our main claim is that we’re novel in applying quantitative analysis of the cosine similarity measure in the theoretical analysis of Neural Collapse. We will clarify our contributions with regard to this statement and discuss the related use of the cosine similarity measure for similar purposes in previous literature.

Reviewer comment (Writing):
Minor issues — grammatical error in the second sentence of the second paragraph; missing punctuation after formulas; inconsistent formatting of ‘intra / inter’ (italics vs. upright); symbol β used with different meanings; inconsistency in N definition between Proposition 2.1 (batch size) and Theorem 2.2 (number of samples per class); γ may need to be rescaled to account for number of batches.

Author response:
We also thank the reviewer for pointing out minor errors in our manuscript and will fix them in our updated version.

Reviewer comment (Theory):
Question — Is it possible to arrange features in a simplex ETF such that batch normalization is satisfied? Is it possible if γ=1 (batch normalization without affine parameters)? If not, does this affect the validity of Theorem 2.2?

Author response:
1. A simplex ETF structure can be arranged to satisfy batch normalization regardless of the $\gamma>0$ value. To avoid confusion, we would like to clarify that we consider “scaled simplex ETF”, which can have vector norms to be any value (that is the same across different classes) and is also the case in the original NC literature. Therefore, we do not believe that this affects the validity of our theorems

Reviewer comment (Theory):
Question — Why is ε/δ ≪ 1 listed as an assumption for Theorems 2.1 and 2.2, given that big-O notation already implies the result holds for ε small enough?

Author response:
2. As we mentioned in the global response, the big-O notation is mainly intended to omit constants and higher-order terms instead of implying that our bounds only hold when $\epsilon\rightarrow 0$. However, this assumption is still required for our theorem to hold true.