Paper: NECO: NEural Collapse Based Out-of-distribution detection (ICLR-2024)

Reviewer comment (Experiments):
The benchmark on ImageNet is missing results on other popular architectures, such as ResNet, that are present only on the CIFAR benchmark. Results on ResNet-50 are highly appreciated and common in a plethora of previous work, and would strengthen the empirical benchmark of the paper.

Author response:
**Additional Experiments on ResNet-50:** 
Thank you for your suggestion. We have conducted the necessary experiments using ResNet-50 and present the results in the table below. 
However, it's worth noting that the effectiveness of certain techniques may vary when applied to transformers.

| **Model** | **Method** | **ImageNet-o** | **Texture** | **iNaturalist** | **SUN** | **Places365** | **Average** |
|-----------|-------------------|------------|---------|-------------|-----|-----------|---------|
| **ResNet-50** |**Softmax score**| 53.13 95.65 | 77.12 71.28 | 84.40 60.95 |78.74 **71.03** | 76.66 **73.41** | 77.01 74.46 |
| | **Maxlogit** | 51.19 94.95 | 71.26 74.40 | 79.88 68.52 | 74.05 75.47 | 71.30 77.42 | 69.54 78.15 |
| | **Energy** |48.23 92.80 | 49.29 95.64 | 50.85 98.14 | 50.13 97.93 | 48.90 97.76 |49.48 96.45 |
| | **Energy+React** | 36.17 99.05 | 40.14 97.67 | 30.09 99.98 | 28.53 99.79 | 27.81 99.80| 32.55 99.26|
| |**ViM** | **72.34 81.75** | 84.00 58.57 | 84.20 61.42 | 66.23 90.24 | 63.92 90.96 | 74.31 76.59|
| |**Residual** |71.90 82.55|83.31 61.92 | 83.13 65.02 |65.54 91.64 | 63.54 92.37 | 73.48 78.70 |
| |**GradNorm** | 39.03 99.00 | 37.64 98.22 | 30.40 99.98 | 29.08 99.79 | 28.53 99.73 | 32.94 99.34 |
| |**Mahalanobis** | 73.35 85.40 | 87.54 53.74 | **90.73 49.18** | 76.60 83.34 | 74.84 84.15 | 80.61 71.16 |
| |**KL-Matching** | 67.55 90.45 | 86.11 63.93 | 89.51 50.04 | **81.23** 73.89 | **79.28** 75.98 | **80.74** **70.86** |
| |**ASH-B** | 43.14 99.40 |46.57 98.37 | 36.24 99.99 | 32.01 99.55| 29.79 99.75| 37.55 99.41 |
| |**ASH-P** |52.93 95.10 | 50.40 96.38 | 57.54 95.64 | 58.91 95.30 | 59.59 93.66 | 55.87 95.22 |
| |**ASH-S** |51.08 93.50 | 70.78 73.35 | 79.57 68.11 | 73.26 75.44 | 70.21 78.27 | 68.98 77.73|
| | **NuSA** | 66.06 89.45 | 80.86 64.17| 44.60 98.24 |52.66 97.97 | 51.59 98.44 |59.15 89.65 |
| | **NECO (ours)** | 69.80 86.43 | **88.09 51.53**| 87.94 62.69| 75.56 77.55 | 73.07 78.62 | 78.89 71.36|

Reviewer comment (Writing):
Some passages of the paper are hard to follow. The writing and especially the notation can be polished.

Author response:
**Clarity in Writing:**
 Thank you for your feedback. We employed the official notations as per the ICLR template. However, to enhance clarity, we will introduce these notations in the supplementary material. If there are other clarity issues please warn us. We have replaced the word operators by operation in section D.

Reviewer comment (Writing):
Some notations are not clear to me. What does the operator $avg_{c,c^\prime}$ mean? $\frac{1}{c}\frac{1}{c^\prime}\sum_{i=1}^c\sum_{j=1}^{c^\prime}(\cdot)_{ij}$?

Author response:
**Operator meaning:**  It is true that the operator $Avg_{c,c'}$ can be ambiguous.
 In our case, this operator computes the equation  inside the $\left|...\right|$ over all possible combinations of the couple $(c , c’)$,
  where $c,c’        \\in  \\{  1 \\ldots C\\}$, and then averages the result on these cases.

Reviewer comment (Experiments):
I would suggest the authors run experiments on ResNet-50 on the ImageNet benchmark to strengthen their empirical results.

Author response:
**Additional Experiments on ResNet-50:** 
Thank you for your suggestion. We have conducted the necessary experiments using ResNet-50 and present the results in the table below. 
However, it's worth noting that the effectiveness of certain techniques may vary when applied to transformers.

| **Model** | **Method** | **ImageNet-o** | **Texture** | **iNaturalist** | **SUN** | **Places365** | **Average** |
|-----------|-------------------|------------|---------|-------------|-----|-----------|---------|
| **ResNet-50** |**Softmax score**| 53.13 95.65 | 77.12 71.28 | 84.40 60.95 |78.74 **71.03** | 76.66 **73.41** | 77.01 74.46 |
| | **Maxlogit** | 51.19 94.95 | 71.26 74.40 | 79.88 68.52 | 74.05 75.47 | 71.30 77.42 | 69.54 78.15 |
| | **Energy** |48.23 92.80 | 49.29 95.64 | 50.85 98.14 | 50.13 97.93 | 48.90 97.76 |49.48 96.45 |
| | **Energy+React** | 36.17 99.05 | 40.14 97.67 | 30.09 99.98 | 28.53 99.79 | 27.81 99.80| 32.55 99.26|
| |**ViM** | **72.34 81.75** | 84.00 58.57 | 84.20 61.42 | 66.23 90.24 | 63.92 90.96 | 74.31 76.59|
| |**Residual** |71.90 82.55|83.31 61.92 | 83.13 65.02 |65.54 91.64 | 63.54 92.37 | 73.48 78.70 |
| |**GradNorm** | 39.03 99.00 | 37.64 98.22 | 30.40 99.98 | 29.08 99.79 | 28.53 99.73 | 32.94 99.34 |
| |**Mahalanobis** | 73.35 85.40 | 87.54 53.74 | **90.73 49.18** | 76.60 83.34 | 74.84 84.15 | 80.61 71.16 |
| |**KL-Matching** | 67.55 90.45 | 86.11 63.93 | 89.51 50.04 | **81.23** 73.89 | **79.28** 75.98 | **80.74** **70.86** |
| |**ASH-B** | 43.14 99.40 |46.57 98.37 | 36.24 99.99 | 32.01 99.55| 29.79 99.75| 37.55 99.41 |
| |**ASH-P** |52.93 95.10 | 50.40 96.38 | 57.54 95.64 | 58.91 95.30 | 59.59 93.66 | 55.87 95.22 |
| |**ASH-S** |51.08 93.50 | 70.78 73.35 | 79.57 68.11 | 73.26 75.44 | 70.21 78.27 | 68.98 77.73|
| | **NuSA** | 66.06 89.45 | 80.86 64.17| 44.60 98.24 |52.66 97.97 | 51.59 98.44 |59.15 89.65 |
| | **NECO (ours)** | 69.80 86.43 | **88.09 51.53**| 87.94 62.69| 75.56 77.55 | 73.07 78.62 | 78.89 71.36|

Reviewer comment (Experiments):
What is the detection performance of only equation (8)? I suggest the authors ablate on the influence of the MaxLogit multiplied to it.

Author response:
**Isolating Equation (8) Performance:**
We conducted experiments specifically using only Equation 8 across various architectures and datasets. Here are the results obtained when the MaxLogit is not utilized:

| **Model** | **Method** | **ImageNet-o** | **Texture** | **iNaturalist** | **SUN** | **Places365** | **Average** |
|-----------|-------------------|-------------|-------------|-------------|-------------|-------------|-------------|
| **ViT** | **NECO w/o Maxlogit** | 93.27 30.05 | 89.02 46.96 | **99.48 1.47** | 89.90 38.24 | 85.89 52.15 | 91.51 33.77 |
|  | **NECO** | **94.53 25.20** | **92.86 32.44** | 99.34 3.26 | **93.15 33.98** | **90.38 42.66** | **94.05 27.51** |
| **Swin** | **NECO w/o Maxlogit** | 63.17 87.95 | 75.52 69.63 | 90.72 36.37 | 81.59 **57.19** | 80.52 **60.77** | 78.31 62.38 |
|  | **NECO** | **65.03 80.55** | **82.27 54.67** | **91.89 34.41** | **82.13** 62.26 | **81.46** 64.04 | **80.56 59.19** |
| **DeiT** | NECO w/o Maxlogit | 56.21 95.80 | 60.02 96.68 | 71.41 84.49 | 57.29 84.10 | 55.54 82.74 | 60.10 88.76 |
|  | **NECO** | **62.72 84.10** | **80.82 59.47** | **89.54 42.26** | **77.64 65.55** | **76.48 68.13** | **77.44 63.90** |
| **ResNet-50** | **NECO w/o Maxlogit** | **71.01 85.44** | 84.86 60.79 | 81.04 84.85 | 60.42 96.77 | 61.92 96.34 | 71.85 84.83 |
|  | **NECO** | 69.80 86.43 | **88.09 51.53** | **87.94 62.69** | **75.56 77.55** | **73.07 78.62** | **78.89 71.36** |

Reviewer comment (Theory):
In equation 6, I'm assuming the numerator goes to zero, and the denominator is always bounded. Is this correct?

Author response:
**Understanding Equation 6:**  
I confirm to you that the denominator is bounded. According the NC1 property that guarantees that all ID class clusters must be far away from each other i.e maximal inter-class variance guarantees that  $||\mu_c||_2>\epsilon$.  As for the numerator, if the denominator is a finite value, the numerator converges to zero.

Reviewer comment (Theory):
On the proof, it states that $C\leq d \leq D$, but what if $C> D$? Which is usually the case for ImageNet-1K.

Author response:
**Handling Cases with less dimension than classes:**
The scenario where C>D poses a considerable challenge, as the vertices of the simplex ETF (i.e., the means of class clusters) cannot be perfectly orthogonal. Despite this, the robustness of NECO is maintained due to the fulfillment of NC1, NC2-equiangularity, and NC5 even in such challenging cases. These properties play a crucial role in preserving the separability of ID and OOD data, even when the perfect adherence to the simplex ETF structure definition is not feasible.
Additionally, NC3 asserts the geometric similarity between the DNN classifier (with dimension D=C) and the penultimate layer representation, where NECO is defined. Furthermore, NC4 indicates that the classifier, with a dimension of D=C, chooses the class with the closest class mean as the prediction. In essence, the maximum value of the classifier logits is expected to be inversely proportional to the distance to the nearest class mean, with a proportional rescaling. These insights prompted the introduction of the Maximum Logit multiplication to recalibrate NECO, addressing challenges in scenarios with a small feature layer dimension.

Reviewer comment (Experiments):
I may have missed it in the text, but how do the authors select the parameter $d$ for the PCA projection?

Author response:
**Selection of PCA Parameter d:**
Regarding sensitivity analysis, we have already investigated the metric's behavior with changes in PCA dimensionality, as outlined in Section C.5. Figures C.4-C.6 illustrate the robustness of NECO to the chosen dimension. Once the selected number of components sufficiently covers the entire ETF structure, adding more components has negligible impact on the AUROC. As might be expected, this dimensionality is model-dependent and requires re-evaluation with any DNN architecture changes.  
R3: 1.A   We empirically observed that the performances are stable across different values of $d$, by performing an exhaustive evaluation for all $d \in \]1, D\]$. 
In the case of the evaluation phase, we fixed $d$ by selecting the first $d$ principal components that explain at least 90% of the ID variance from the train dataset.
In the paper, we updated Figure C4 and Figure C5  with the explained variance curve.
1.B Thank you for noticing the issue on Figure C5.b and Figure C6.b, we have updated the figures in the final manuscript. Now, the behavior is coherent with respect to other architectures.

Reviewer comment (Experiments):
The manuscript lacks critical baseline comparisons, such as [1]. It is recommended that the authors conduct a comparison with it to provide a more comprehensive evaluation.

Author response:
**Lack of Baseline Comparisons:**

Thank you for your suggestion. We will incorporate [1] into the manuscript in the coming days.

Reviewer comment (Experiments):
Why not put NuSA in the baseline? Can NECO outperform NuSA?

Author response:
**Absence of NuSA in Baseline:** In response to your recommendation and the insights from Review of Ek3F, we have now implemented NuSA using the same dimension as in ViM. The corresponding results are available in the table below, and we will incorporate these results into the paper.

  
| **Model** | **Method** | **ImageNet-o** | **Texture** | **iNaturalist** | **SUN** | **Places365** | **Average** |
|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
| **ViT** | 92.48 34.30 | 88.28 51.24 | 99.30 3.12 | 89.26 40.08 | 85.28 54.51 | 90.92 36.65 | 90.92 36.65 |
| **Swin** | 56.50 91.95 | 62.72 83.14 | 64.01 83.58 | 55.97 91.28 | 54.44 92.71 | 58.73 88.53 | 58.72 88.53 |
| **DieT** | 56.39 93.47 | 54.65 94.85 | 53.96 95.46 | 72.47 85.31 | 50.12 96.28 | 50.74 95.47 | 56.39 93.47 |
| **ResNet-50** | 66.06 89.45 | 80.86 64.17 | 44.60 98.24 | 52.66 97.97 | 51.59 98.44 | 59.15 89.65 | 59.15 89.65 |

Reviewer comment (Theory):
The article does not discuss the conditions and scope of the neural network collapse phenomenon, such as whether specific loss functions, activation functions, optimizers, etc.

Author response:
We acknowledge this concern and plan to include a discussion on the conditions that lead to Neural Collapse (NC) in deep neural networks (DNNs). We recognize the need for a comprehensive exploration of NC conditions to enhance the understanding of our approach.

Reviewer comment (Evaluation):
The article does not analyze the sensitivity of anomaly detection metrics.

Author response:
We applied classic anomaly detection and OOD detection metrics. In response to the reviewer's suggestion, we will expand the discussion in Section 5 and include additional details in the supplementary section to describe these metrics more comprehensively. Regarding sensitivity analysis, we have already investigated the metric's behavior with changes in PCA dimensionality, as outlined in Section C.5.

If you have particular experiments in mind for further evaluation, please provide details. Note that we currently have a single parameter, the PCA dimension, and we have already examined its sensitivity.

Reviewer comment (Evaluation):
The article does not explore the interpretability of anomaly detection metrics, such as whether it can give the causes or characteristics of anomalous data.

Author response:
We're uncertain about the question's intent. If you're inquiring why we haven't delved into the interpretability of anomaly detection, it's important to note that it's not the primary focus of our method. Our objective is to present an OOD detection algorithm and elucidate its effectiveness. Our approach calculates the relative norm of the data within the Simplex ETF structure to derive the OOD-ness score. Therefore, a misclassification of OOD sample indicates that the data sample shares some similarity with some ID classes.

Reviewer comment (Presentation):
In section 3.2, "Nervous Breakdown", could more visual explanations or visual diagrams of the nature of NC1 and NC2 be provided?

Author response:
We are uncertain about the specific requirements, as we have already included figures 1, 2, D7, D9, D10, D11, D12, D13. Therefore, it's not entirely clear what the reviewer is requesting. If the suggestion is to provide visual explanations for the definitions in Section 3, we will incorporate them accordingly. We will also add the pseudo code in the supplementary section C.2.

Reviewer comment (Reproducibility):
Is the observation of Out of Distribution Neural Collapse mentioned in Section 4 based on some existing theoretical or experimental evidence? If it is based on experimental evidence, please provide more detailed experimental settings and results.

Author response:
Our primary contribution lies in demonstrating experimental results, specifically illustrating NC1 to NC4, as shown in section 4 and section D. We have conducted a qualitative evaluation similar to [1], presenting the results in figures 1, 2, D7, D9, D10, D11, D12, D13. If additional evaluations are needed, please feel free to request them.

Reviewer comment (Theory):
Why was this particular form chosen to define NECO(x) in Eq. (8)? What is the intuition or theoretical basis behind it?

Author response:
The motivation behind NECO is interesting. We observed that performing PCA on the latent space of the prelogit of pretrained DNN reveals the emergence of clusters in both the ID and OOD data when the dimension is reduced. Remarkably, the OOD cluster consistently appears centered around the null vector, as depicted in figures 2 and D13. Consequently, NECO(x) is small for OOD data. This phenomenon is consistent across different architectures. While we recognized a potential link to NC, we acknowledge that current literature in NC does not explain OOD, prompting us to propose a new NC property on OOD scenarios (NC5).

Reviewer comment (Reproducibility):
Throughout the article, is it possible to provide a complete flowchart or pseudo-code of the algorithm to help readers better understand the whole process of NECO?

Author response:
We are uncertain about the specific requirements, as we have already included figures 1, 2, D7, D9, D10, D11, D12, D13. Therefore, it's not entirely clear what the reviewer is requesting. If the suggestion is to provide visual explanations for the definitions in Section 3, we will incorporate them accordingly. We will also add the pseudo code in the supplementary section C.2.

Reviewer comment (Writing):
At times, the manuscript is a bit misleading. For instance, the sentence "Intuitively, these properties depicts the tendency of the network to maximally separate class features while minimizing the separation within them" is quite counter-intuitive. I'd suggest the authors to proofread the manuscript again to make it easier to digest.

Author response:
**Clarity Concerns:**

We acknowledge the need for clarity improvements in the manuscript. A proofread is scheduled to rectify misleading formulations and correct any typos by next Friday.

Reviewer comment (Evaluation):
Although NECO shows strong ID classification and OOD detection capabilities across aggregated benchmarks, it undeperforms on standalone datasets such as ImageNet-O, Textures, and iNaturalist depending on the encoding architecture that is used. An explanation for this behavior could make a stronger case for the paper.

Author response:
**Performance Variability:**

The performance of out-of-distribution (OOD) detection is intricately tied to both the specific OOD dataset used and the underlying architecture, a characteristic common to all techniques.

Take, for instance, the ImageNet-O dataset, which consists of adversarially filtered images specifically designed to challenge OOD detectors.

The adversarial selection for this dataset was based on ResNet-50, explaining why ViT performs significantly better in this case compared to DeiT, which employs ResNet as a teacher, and Swin, which emulates ResNets in its architecture.

In essence, ViT demonstrates superior performance on average, and this can be attributed to its tendency to exhibit more collapse due to its classical style of training. This is in contrast to other transformer models that modify their training procedures in diverse way: DeIT utilizes a distillation strategy while SwinV2 utilizes a shifted window mechanism.

Exploring the impact of training and pre-training on the neural collapse phenomenon could provide valuable insights and serve to validate our assumptions. This requires a specific experimental setting, hence we suggest exploring this topic in future works.

Reviewer comment (Experiments):
Since NECO is a post-hoc method, it is not obvious from the manuscript at what point in the training process does the neural collapse occurs for NECO to kick in. In other words, a model could still be well-trained without necessarily reaching the neural collapse phase. In that specific case, would NECO require training the model a bit more, or how would NECO's performance compare with say NuSA or ViM?

Author response:
**Activation Point of NECO:**

All models utilized in our experiments were trained until convergence, without any over-training, to ensure a fair comparison across all methods, including ViM and NuSA. While over-training has the potential to enhance neural collapse (NC), the observed improvement is marginal, as evident in figures 1, D.7-D.12. It's important to note that the models depicted in the CIFAR10 figures are not the ones used for testing; additional steps were added for figure generation to illustrate clear convergence.

Enhancing NC metrics is likely to benefit out-of-distribution (OOD) detection, as it simplifies the model representation. This improvement would consequently enhance all methods leveraging NC properties, even indirectly. Self-supervised methods, such as those used in Foundation models or transformers, present promising avenues to increase the NC, as it inherently aims at clustering the data in the latent space.

Regarding ViM/NuSa, while increased collapse might enhance performance, there are no guarantees, as components outside the ETF structure are not governed by the NC phenomenon. These components may or may not exhibit larger values for OOD.

Reviewer comment (Theory):
NECO makes a somewhat stringent assumption that the subspace of the ID (and OOD?) is linear by making use of PCA to learn the matrix $P$ spanned by the d eigenvectors corresponding to the largest d eigenvalues of the covariance matrix $H$, where $H$ represents the feature space of the ID data. I think a justification as to why this feature space is linear is important. A non-linear dimensionality reduction, or a manifold-based learning method, would be more appropriate given the fact that neural nets are non-linear in nature.

Author response:
**Linearity Assumption:**

Despite the inherent non-linearity of DNNs, the occurrence of the NC phenomenon simplifies this property, particularly at the penultimate layer/DNN-classifier level, where the ETF structure becomes linear by definition [see Introduction section: Convergence to Simplex ETF (NC2), and the figure D.7.

This structure defines the feature representation and geometry. It's worth noting that alternative techniques like kernel PCA and manifold learning were tested, but no significant improvement in results was observed. If needed we can add a discussion on that.