Paper: Symmetric Neural-Collapse Representations with Supervised Contrastive Loss: The Impact of ReLU and Batching (ICLR-2024)

Reviewer comment (Experiments):
It seems that batch size is an important argument when applying sample selections according to the theoretical analysis. It would be better to add the ablation study on that.

Author response:
**Batch size:**
Thanks for highlighting the role of batch size. Indeed, for contrastive loss and in the analysis of our model, batch size plays an important role in defining the set of global solutions. This is the case as the batching scheme specifies the interactions among samples during training. Our theoretical framework outlines the necessary and sufficient conditions for the batching to achieve a unique minimizer for contrastive loss. Following your suggestion, we conducted additional experiments for varying batch sizes to empirically explore its impact on the final learned embeddings. 
The detailed results of these new experiments are presented in Section E.4.3 in the appendix of the revised draft. In these experiments, we examine the effect of batch size in two scenarios, depending on whether ReLU is applied on the final embedding layer or not. We compare the learned features with OF when ReLU is present at the final layer, and with ETF (as predicted by Graf et. al (2021)) when ReLU is removed. Our observations indicate that the presence of ReLU reduces the reliance on batch size, leading to consistent convergence to OF. In contrast, in the absence of ReLU, large batches are necessary to achieve the global optimizer ETF. We also note that Khosla et.al (2020) argue that increased batch sizes lead to improved performance. In particular, they typically use batch sizes extending up to 6144. Our findings suggest that with the addition of ReLU, one can potentially achieve comparable performance with smaller batch sizes, which offers venues for reducing the training computational cost.

Reviewer comment (Evaluation):
Based on the analysis, it seems that various activation functions can achieve the similar effect as ReLU. Is it possible to add a discussion about that?

Author response:
**Activation functions:**

Thank you for bringing light to this subtle point. We considered ReLU as the primary focus of our study in terms of the activation function due to its simplicity and frequent use in practice. While it is interesting to study different activation functions, our motivation was to identify that a simple and commonly used activation function can significantly alter the implicit geometry of features.

That said, we agree that this is indeed an interesting question. The theoretical analysis only assumes that the post-activation features are non-negative and span the region $[0, \infty]$. It is reasonable to anticipate that activation functions satisfying this requirement lead to the same result. However, the non-linearity of the activation function is likely to play a role in the optimization dynamics and needs further investigation. We have planned to conduct specific experiments and report our findings in the paper.

Reviewer comment (Experiments):
The minimum of the loss is not zero and varies with the number of samples in different classes.

Author response:
That is correct. Due to fixing the norm of the features to a constant value (say 1), the loss does not go to 0. The minimum value is exactly given by the lower bounds in Eq (3) and (4), respectively, for full-batch and mini-batch optimizations. For full batch SCL, we show the loss evolution in a DNN experiment, where the loss is seen to converge exactly to our lower bounds in Theorem 1. This prediction is made possible due to our theoretical analysis.

To demonstrate the effectiveness of our Theorem 1, in Fig 3, we show how closely the empirical loss in a full-batch DNN experiment converges to the predicted lower bound.

Reviewer comment (Theory):
While the stated objective of the paper, i.e., embeddings that satisfy NC & OF properties, is clear, it is not at all clear that it is a justified goal. For example, consider the simple case of binary classification. The optimal solution without ReLU will map two classes to some anti-parallel vectors \(u\) and \(v = -u\). This means that the hidden representations for one class will have the maximum degree of separation \(\| u - v\| = 2\). Now, if we pass them through ReLU, forcing them to be element-wise positive, the solutions will instead be some \(u\) and \(v\) such that \(u^\top v = 0\). Thus, their mean-class distance will be \(\| u - v\| = \sqrt{2}\). Without any formalities, what doesn't seem to be a benefit in terms of distance, doesn't seem to be a good approach for generalization either. Even if we make the two classes here very unbalanced, still the anti-parallel geometry seems to be better than the orthogonal approach. Since the authors do not make any formal/theoretical comments on this topic, the only relevant evidence they provide in this direction is Table 1 test accuracy results. Because of the aforementioned reasons, I consider the results of Table 1 to be particularly surprising. Thus, if the main evidence to support utility of the approach is empirical, I would like much more substantive experiments. Some elaborate explanations or at least speculative comments by the authors will also go a long way to explain the benefits of the orthogonal mean-classes.

Author response:
**For example, consider the simple case of binary classification...**:
Thank you for prompting this discussion. The statement regarding antipodal features for binary SCL without ReLU is exactly correct. In fact, this holds in general for balanced multi-class setup as well: When the dataset is balanced Graf et al. (2020) shows that the embedding geometry will form an ETF in the absence of ReLU. In an ETF geometry, we have $\frac{\mu_c^\top\mu_{c'}}{\|\mu_c\|\|\mu_{c'}\|}=\frac{-1}{k-1}$ for the mean-embeddings of any two classes. As opposed to OF where $\frac{\mu_c^\top\mu_{c'}}{\|\mu_c\|\|\mu_{c'}\|}=0$.

Therefore, for ETF, $\|\mu_c-\mu_{c'}\|^2 = \frac{2k}{\tau(k-1)}$ and in the case of OF $\|\mu_c-\mu_{c'}\|^2 = \frac{2}{\tau}$. $\tau$ is the temperature of the SCL loss. While the distance in the ETF geometry is larger for a fixed temperature parameter, we note that by simply scaling the temperature parameter $\tau$ by $\frac{k-1}{k}$, we can achieve an equivalent absolute distance in the learned geometry in the presence of ReLU. Thus, in terms of this measure, the addition of ReLU does not compromise the trained model.

On the contrary, as shown in Fig. 1, in the imbalanced setup, without ReLU, the minority classes are drawn closer to each other, yielding a reduced absolute distance between classes. The introduction of ReLU, however, maintains an equal pairwise distance between classes. Hence, in terms of this measure, the addition of ReLU is in fact helpful.

Finally, as noted in Section D.1.2, centering the OF by $\frac{1}{k}\sum_c\mu_c$ results in an ETF. So, in summary, the geometries learned in balanced or binary scenarios, with or without ReLU, can be considered equivalent, accounting for a shift and scaling.

Reviewer comment (Writing):
I think there is a slight overuse of abbreviations in this paper which makes it slightly hard to read. For example, I had to go back several times to read what ETF is, or other notations. If possible, please keep the shorthands to a minimum.

Author response:
**overuse of abbreviations:** Thanks for the feedback. We have added a table defining all the important abbreviations used in the paper in one place at the beginning of the appendix.

Reviewer comment (Theory):
There is a small discrepancy between eq(1) and eq(2) in the presence of \((1+ ... )\) within the logarithm and the summation is over \(\ell\neq i, j\) while another is over \(\ell \neq i\). Perhaps authors can comment on this?

Author response:
**eq(1) and eq(2):** Eq (1) and (2) are equivalent, in terms of the loss function. The apparent difference is only in presentation, in that in Eq (2), the sum over $l$ includes $j$, so that when $l = j$, the innermost exponential term becomes $\exp(\mathbf{h}_i^\top\mathbf{h}_l - \mathbf{h}_i^\top\mathbf{h}_j) = \exp(0) = 1$.

Reviewer comment (Theory):
Significance: The work appears to be written on the premise that achieving a symmetric (orthogonal or simplex) feature geometry is beneficial. For example, a batching strategy is devised that guarantees such a geometry. While symmetry seems reasonable for class-balanced data, for imbalanced data this is not so clear. I did not see an argument supporting this premise, and thus also not for why using the proposed batching strategy is a good idea. Moreover, there is few empirical support, as the use of ReLUs does not have any effect on classification accuracy, cf. Table 1.

Author response:
Thank you for your time and effort in reviewing our paper. On the motivation, we would kindly refer you to our comments in the global response. At the outset, the explicit goal was not to symmetrize the feature geometry, but to **uncover and understand** the same. This is in line with the research goal of the rapidly growing interest in neural collapse and its implications for DNN training with various objective functions. Please note that there is **no prior work** that characterizes the feature geometry and thus the behavior of the loss function in inducing the same, for the SCL under imbalances. Prior to our work, the understanding of the **intricate role of mini-batch optimization** was limited as well. We have identified a setting where the geometry is predictable, incidentally observing that it is symmetric. Based on the works in literature on related loss functions under class imbalance, this result remains surprising. Additionally, we show that the convergence quality is excellent, in practical experiments, as compared to experimental results on geometry under imbalances in literature. Thus, we believe that our discovery, supported by concrete theoretical results, paves the way to important future questions in understanding the loss functions that are used across datasets and model architectures, and identifying characteristic properties of various loss objectives.

Reviewer comment (Experiments):
The empirical part of this work would be stronger, if the theoretical findings are verified on a large scale imbalanced dataset (compared to the mid-scale datasets CIFAR10, CIFAR100 and TinyImagenet).

Author response:
**Empirical:**
We appreciate the reviewer's recommendation regarding experiments on large-scale datasets and consider adding these experiments to the final version of the paper. Along these lines, we are excited to report new experimental findings in support of our hypothesis of OF geometry on transformer architecture. We have included additional experiments on vision transformers (see Fig.22). Based on our current observations, our hypothesis of OF geometry holds here as well, thus solidifying the claim of the generality of our results across various architectures.

In summary, we would like to note that our current findings are backed up by experiments spanning various architectures and datasets, including MLP, ResNet, and DenseNet applied to MNIST, CIFAR10, CIFAR100, and TinyImageNet. Across this range of tasks, from simpler and smaller ones like MNIST (10 classes) to more complex and larger tasks like TinyImageNet (200 classes), we have not observed significant changes in the convergence behavior or the communication of our key message (see Fig 2) as long as the network and dimension of output features are large enough.
The observations from all of the above experiments provides us with the confidence to believe that the core message of our paper remains largely unchanged. This is the case for both our main findings: convergence to OF and the efficacy of the new batch-binding scheme.

Reviewer comment (Experiments):
Are there empirical differences when training models with and without ReLU activations after the last layer. If so, can such differences be explained by the targeted feature geometry? Are such differences more pronounced when using the batching strategy?

Author response:
**Empirical differences:** At the moment the main empirical differences observed are the following : the test accuracies (Tab. 1) and new analysis on worst-class accuracies between with and without ReLU. We discuss this in the global response as well. We are not aware of differences in the batching aspects. In fact, the observation in Fig. 21 suggests that the batch-binding scheme is beneficial even when training without ReLU, for convergence to the final feature geometry.

## Rebuttal 2 (ID: aMs8n2ar1C)
Please note that there was an unfortunate typo that may have led to a misunderstanding. We did not mean to claim "a major difference". Instead, it was meant to read "the major differences at the moment are the accuracy noted in Tab 1 **and** worst class accuracy". In other words, these are the two empirical differences that we have observed.

Please note that the typo has been corrected.