Paper: Neuron Activation Coverage: Rethinking Out-of-distribution Detection and Generalization (ICLR-2024)

Reviewer comment (Evaluation):
The introduction contains some issues: The methods like ASH, React do not negatively impact the ID accuracy. The model can employ an unchanged classifier for the ID task, which only introduces a negligible computational cost.

Author response:
**A1:** We apologize for the confusion. In practice, our claim that "ASH, ReAct decrease the model classification ability" is referred from the ASH paper [R1], where they clearly state that ReAct and their ASH methods lead to InD accuracy drops due to pruning. We would like to invite the reviewer to check **Figure 2 in ASH paper** [R1] for the details.

[R1] Extremely Simple Activation Shaping for Out-of-Distribution Detection. ICLR, 2023. https://openreview.net/pdf?id=ndYXTEL6cZz 

Despite the above clarification, we do agree with the reviewer that using an unchanged classifier can be an efficient solution. However, it is also important to recognize that the success of **this solution hinges on the strict assumption that only later layers are utilized by neuron pruning**. This restricted perspective may largely limit the potential of neuron-based methods, since shallow layers also offer valuable information, as demonstrated in Table 4 of our paper. Therefore, it is imperative to approach this solution with caution and careful consideration. 

To further address your concern, **we have added a footnote in our Introduction** to clarify this point: "While it may be argued that maintaining neuron outputs for double-propagation preserves InD accuracy with low computational cost,  it relies on the assumption that only later layers are utilized in neuron pruning, thus undermining the potential of these neuron-based methods."

Lastly, we would also like to remind the reviewer that our **NAC-UE significantly outperforms ReAct and ASH on three benchmarks**. For your convenience, we provide their averaged AUROC results below. The scores are sourced from our Table 1 and 2, where ReAct and ASH are officially implemented by OpenOOD.

|| ReAct     | ASH   | NAC-UE | 
|----|---|----|---|
| CIFAR-10  | 90.42 | 78.49  | **94.60** |
| CIFAR-100 | 80.39 | 80.58  | **86.98** |
| ImageNet  | 89.68 | 73.65  | **94.22** |

Reviewer comment (Experiments):
The motivation is somewhat perplexing: The author suggests that OOD is more prone to activate neurons with lower activation frequency. However, this paper fails to provide empirical evidence to back up this assertion.

Author response:
**A2:** We are sorry for the confusion. In practice, **our paper has already provided both qualitative and quantitative results** to support our motivation:

1) For qualitative results, we kindly request the reviewer to refer to our **Figure 5 (rightmost column)**. This figure effectively illustrates that when considering our defined neuron states, InD and OOD data elicit distinct activation patterns in neurons. Specifically, OOD data tends to activate neurons with states around 0.5, representing the "center" of the activation space. Instead, InD data primarily activates neurons located at the "corners" of the space. This contrasting behavior shows that OOD data potentially activate neuron states that are less frequently observed with InD data.

2) For quantitative results, we would invite the reviewer to refer to **Figure 8 in Appendix G.2**. In this figure, we present the distribution of averaged coverage scores w.r.t all neurons, which reveal that **OOD samples generally present lower coverage scores compared to InD samples**. Since a low coverage score signifies a high inconsistency in neuron behavior between test samples and InD training data (See Eq. (5)), we can further confirm that OOD data tend to provoke abnormal neuron behaviors in comparison to InD data.

Finally, it is worth noting that there also are many existing studies revealing that InD and OOD trigger distinct activation patterns (e.g., [R2, R3]), which is consistent with our findings and supports the validity of our results.

[R2] ReAct: Out-of-distribution Detection With Rectified Activations. NeurIPS, 2021\
[R3] Line: Out-of-distribution detection by leveraging important neurons. CVPR, 2023

Reviewer comment (Experiments):
The confusion between OOD and ID images may stem from the fact that OOD images can potentially activate the same neurons as ID images. How does the author's approach work in such a scenario? It would be beneficial for the author to include near-ood experiments [1] to enhance the comprehensiveness of the study.

Author response:
**A3:** We appreciate this valuable comment, and agree with the reviewer that it is possible that OOD images trigger the same neuron states as InD images. However, as a network often comprises numerous neurons, it is also often the case that some neuron activations are triggered differently by OOD data, as evidenced by our qualitative and quantitative results (explained in **A2**).

To further address your concern, **we have conducted a series of experiments on near-ood situations**, similar to [1]. Specifically, we utilized a model trained on CIFAR-10 as the foundation, and evaluated OOD detection methods on near-ood datasets: CIFAR-100 and Tiny ImageNet. We carefully follow the evaluation protocol of OpenOOD, and present the results below. For the sake of brevity, we merely report the top-5 performed methods as well as ReAct and ASH. The detailed experiments and full results for 21 baselines will be included in Appendix G.


| Method | CIFAR-100 || TIN-IMGNET ||Average||
|---|---|---|---|---|---|---|
| | FPR↓  | AUROC↑  | FPR↓ | AUROC↑  | FPR↓   | AUROC↑  |
| ReAct | 67.40 ± 7.34 | 85.93 ± 0.83 | 59.71 ± 7.31 | 88.29 ± 0.44 | 63.56 ± 7.33 | 87.11 ± 0.61 |
| ASH  | 87.31 ± 2.06 | 74.11 ± 1.55 | 86.25 ± 1.58 | 76.44 ± 0.61 | 86.78 ± 1.82 | 75.27 ± 1.04 |
| TempScale | 55.81 ± 5.07 | 87.17 ± 0.40 | 46.11 ± 3.63 | 89.00 ± 0.23 | 50.96 ± 4.32 | 88.09 ± 0.31 |
| RMDS | 43.86 ± 3.49 | 88.83 ± 0.35 | 33.91 ± 1.39 | 90.76 ± 0.27 | 38.89 ± 2.39 | 89.80 ± 0.28 |
| VIM  | 49.19 ± 3.15 | 87.75 ± 0.28 | 40.49 ± 1.55 | 89.62 ± 0.33 | 44.84 ± 2.31 | 88.68 ± 0.28 |
| KNN  | 37.64 ± 0.31 | 89.73 ± 0.14 | 30.37 ± 0.65 | 91.56 ± 0.26 | 34.01 ± 0.38 | 90.64 ± 0.20 |
| GEN| 58.75 ± 3.97 | 87.21 ± 0.36 | 48.59 ± 2.34 | 89.20 ± 0.25 | 53.67 ± 3.14 | 88.20 ± 0.30 |
| NAC-UE  | **35.06** ± 0.30 | **89.78** ± 0.31 | **26.53** ± 0.21 | **91.98** ± 0.24 | **30.80** ± 0.13 | **90.88** ± 0.25 |

From the above results, we can see that **NAC-UE still outperforms existing SoTA methods on the near-ood benchmark**. This further confirms the effectiveness and robustness of our proposed approach.

[1] Exploring the limits of out-of-distribution detection. NeurIPS, 2021.

Reviewer comment (Evaluation):
The performance of NAC-UE, as depicted in Tables 1 and 2, is not consistent and competitive across all OOD datasets, indicating a considerable bias in its effectiveness on different OOD datasets. For example, when CIFAR100 is used as ID and Places365 as OOD, NAC-UE only yields a 73.05% AUROC and 73.57 FPR95.

Author response:
**A4:** 
**There is no single winner on all OOD datasets**, as evidenced by [R8]. More crucially, most previously published methods (e.g., [R4-R7]) do exhibit limitations on one or several OOD datasets as well. 

We want to emphasize again that our NAC method is effective by surpassing 21 OOD detection baselines across three benchmarks. This has showcased its stronger ability compared to existing OOD detection methods, not to mention its potential in evaluating model robustness for OOD generalization. By considering a broader range of previously published works, we hope the reviewer could recognize the superiority of our work.

[R4] GEN: Pushing the Limits of Softmax-Based Out-of-Distribution Detection. CVPR, 2023\
[R5] MOS: Towards Scaling Out-of-Distribution Detection for Large Semantic Space. CVPR, 2021\
[R6] How to Exploit Hyperspherical Embeddings for Out-of-Distribution Detection? ICLR, 2023\
[R7] ViM: Out-Of-Distribution with Virtual-logit Matching. CVPR, 2022\
[R8] OpenOOD v1.5: Enhanced Benchmark for Out-of-Distribution Detection. Arxiv, 2023

Reviewer comment (Novelty):
The author has overlooked a discussion on a comparable study [2].

Author response:
**A5:** We thank the reviewer for this valuable comment. We will cite the paper and add a discussion section in Appendix H. 

While our NAC and [2] both focus on OOD detection and generalization, **they are actually different in their targets, design choices, and experimental settings**.

1) **Target**: Our NAC aims to provide an off-the-shelf/post-hoc tool that efficiently detects OOD data and evaluates model robustness. In contrast, [2] targets an effective learning strategy, which trains the network to overcome OOD scenarios.

2) **Design**: NAC directly leverages neuron distributions to reflect model status under OOD scenarios, while [2] enforces energy margin during the training phase. 

3) **Experimental setup**:  Our paper focuses on the prevalent OOD detection and generalization setup, where the InD and OOD data are clearly separated. Instead, [2] centers on the wild scenarios, where data distribution is a mixed version of InD and OOD, turning the OOD into valuable learning resources.

[2] Feed two birds with one scone: Exploiting wild data for both out-of-distribution generalization and detection. ICML, 2023.

Reviewer comment (Reproducibility):
The absence of open-source code is a setback as it impedes the reproducibility and validation of the results presented in the study.

Author response:
**A6:** We are sorry for the confusion. As illustrated in Appendix F, **we will release our code with detailed instructions**. We also have outlined the hyperparameter choices under different datasets to promote reproducibility (Appendix D.4 and E.4).

Reviewer comment (Evaluation):
Given the time-consuming process of using KL gradient, computational cost of proposed method with other SOTA should be reported.

Author response:
Thanks for your valuable comment! Following your suggestion, we have conducted a series of experiments to analyze the efficiency of our method. To provide a comprehensive comparison, we selected the top-3 performed methods from Table 2 as baselines, and compared them with our NAC-UE in terms of preprocessing and inference time on ImageNet. For consistency, we used the same software and hardware configurations as in our previous experiments, which are detailed in Appendix F.

|| Preprocessing Time (s)| Total Inference Time (s) | AUROC↑| 
|---|---|---|---|
| GEN *(CVPR' 2023)* | 0.00 ± 0.0 | 43.33 ± 0.3	| 89.76 |
| ViM *(CVPR' 2022)*  | 1087.82 ± 9.0 | 48.10 ± 0.4 | 92.68 |
| SHE *(ICLR' 2023)* | 1019.34 ± 2.2 | 41.85 ± 0.5 | 90.92 |
| NAC-UE (layer4) | 5.43 ± 0.3 | **39.63** ± 0.2 | 94.57 |
| NAC-UE (layer4+layer3)  | 6.75 ± 0.3  | 46.09 ± 0.7 | 95.05 |
| NAC-UE (layer4+layer3+layer2) | 7.75 ± 0.2  | 69.73 ± 0.4 | **95.23** |


From the above results, the following observations can be made:

1) While NAC-UE requires more inference time as more layers are considered, **it significantly reduces preprocessing time compared to ViM and SHE**, e.g., 7.75s (NAC-UE) vs. 1019.34s (ViM). This finding coincides with our previous experiments (Appendix G.1), where we show that NAC-UE achieves favorable performance despite utilizing only 1% of the training data for NAC modeling.

2) Remarkably, even when utilizing just a single layer (layer4), **NAC-UE outperforms SoTA methods in terms of both inference time and detection performance**. For example, NAC-UE achieves an AUROC of 94.57 (39.63s inference time), whereas GEN achieves only 89.76 on AUROC with 43.33s inference time. This highlights the efficiency of our approach.

Additionally, it is also worth noting that there are numerous ongoing research efforts dedicated to facilitating gradient calculation [R1, R2], which could potentially complement our proposed method. We will include the above experiments in Appendix G.

[R1] Low Complexity Gradient Computation Techniques to Accelerate Deep Neural Network Training, TNNLS, 2021.\
[R2] Acceleration of DNN Backward Propagation by Selective Computation of Gradients, DAC, 2019.

Reviewer comment (Evaluation):
The proposed metric is extended for uncertainty estimation, while no quantification or evaluation (calibration of uncertainty) on this dimension.

Author response:
We are so sorry for the confusion. In line with many prevalent OOD uncertainty estimators (e.g., Gram [R3], ReAct [R4], MOS [R5]), our uncertainty estimation mainly serves for OOD detection problems: _predicting whether a test sample is from OOD or not_. Thus, our experiments primarily emphasize OOD detection performance rather than calibration errors, similar to the aforementioned studies.

In response to your suggestions, **we have conducted additional experiments to further explore the potential of our NAC-UE**. Our experiments carefully align with [R6], where two calibration error measures (RMS and MAD) are employed. For the calibration evaluation, we utilized a pretrained model on the CIFAR-10 dataset as the foundation, and assessed the calibration errors on both InD and OOD test data. We mainly compare NAC-UE with two simple baselines: MSP (Hendrycks & Gimpel, 2017) and Temperature (Guo et al., 2017). 

|OOD Dataset| | RMS↓ ||| MAD↓ ||
|---|---|---|---|---|---|---|
|  | MSP| Temp | NAC-UE | MSP | Temp | NAC-UE |
| CIFAR100| 50.62 | 43.01 | **33.04** | 42.56| 36.64  | **26.92** |
| Tiny ImageNet | 48.01 | 40.25 | **31.99** | 38.86| 32.88  | **26.25** |
| MNIST | 71.74  | 60.91| **51.30** | 67.81 | 57.16  | **49.45** |
| SVHN  | 65.82 | 56.41| **45.32** | 59.57 | 51.05  | **41.60** |
| Texture | 42.65 | 35.19 | **28.74** | 32.37| 26.90  | **23.72** |
| Places365 | 68.85 | 59.67  | **48.65** | 64.65  | 56.02  | **45.33** |

From the above results, we can see that our NAC-UE significantly outperforms two baseline approaches, which demonstrates its potential in prediction calibration. We will include the above experiments in Appendix G.

[R3] On the Importance of Gradients for Detecting Distributional Shifts in the Wild. NeurIPS, 2021\
[R4] ReAct: Out-of-distribution Detection With Rectified Activations. NeurIPS, 2021\
[R5] MOS: Towards Scaling Out-of-Distribution Detection for Large Semantic Space. CVPR, 2021\
[R6] Deep Anomaly Detection with Outlier Exposure. ICLR, 2019

Reviewer comment (Evaluation):
How does this metric generalized to regression problems?

Author response:
Thank you for this insightful question. Our proposed metric NAC, which focuses on neuron distributions, can indeed be generalized to regression problems without any changes to its definition. The only update required is about the formulation of neuron states. Since the current formulation involves KL divergence which is not suitable for regression problems, we need to simplify the neuron states from $ z \odot g'(z) (p-u)$ to $ z \odot g'(z) $. This simplified version corresponds to the Input $\times$ Gradient explanation, which is also meaningful in explaining neuron behaviors and provides insights into the model status.

Reviewer comment (Experiments):
For NAC-UE uncertainty estimation, the paper averages coverage scores across layers when applying to multiple layers. Is it possible to weight the contribution of different layers especially given that earlier layers tend to be more general?

Author response:
**A1:** Nice suggestion! Inspired by your question, we have conducted a series of experiments on the CIFAR-10 benchmark, where we randomly searched the weight for each layer within the same space: [0.2, 0.4, 0.6, 0.8, 1.0]. We found that our **NAC-UE can be further improved in this weighted version** (e.g., 2.47% gain on average FPR):

|  |   MNIST  || SVHN         || Textures     || Places365    || Average  ||              
|------|---|--------|------|------|--------|-----|-----|-------|----|-----|
|                       | FPR↓               | AUROC↑          | FPR↓          | AUROC↑          | FPR↓         | AUROC↑          | FPR↓          | AUROC↑          | FPR↓          | AUROC↑          | 
| NAC-UE            | 15.14 ± 2.60 | 94.86 ± 1.36 | 14.33 ± 1.24 | 96.05 ± 0.47 | 17.03 ± 0.59 | 95.64 ± 0.44 | 26.73 ± 0.80 | 91.85 ± 0.28 | 18.31 ± 0.92 | 94.60 ± 0.50 |
| NAC-UE (weighted) | 13.94 ± 2.42 | 95.55 ± 1.08 | 9.90 ± 1.09  | 98.10 ± 0.18 | 13.36 ± 0.65 | 97.25 ± 0.21 | 26.16 ± 0.79 | 92.31 ± 0.26 | 15.84 ± 0.65 | 95.80 ± 0.24 |

What's interesting is that we noticed assigning larger weights to the deeper layers often results in better performance. We conjecture this is due to that deeper layers often encode richer semantic information than shallow layers, making them crucial in detection problems. We will include such experiments in Appendix G.

Reviewer comment (Novelty):
Missing a related work that shares a similar motivation of leveraging neuron behaviors. [1] uses neural mean discrepancy of neuron activations for OOD detection. Discussing relations to such works could provide additional insights.

Author response:
**A2:** Thank you for pointing out the missed related work. We will cite the paper and add a discussion section in Appendix H. Here are the main differences outlined:
1)  [1] primarily investigates the raw neuron output, while **our paper centers on a new formulation of neuron states**. This formulation can be decoupled as the neuron gradients, neuron output, and model prediction deviations, thus offering a fresh interpretation of neurons in OOD scenarios.
2) Our NAC specifically focuses on the **distribution** of neuron states, while [1] mainly examines the **mean** of neuron output. This distinctive perspective makes our NAC more comprehensive and superior in understanding neuron behaviors. 
3) While Neural Mean Discrepancy [1] may effectively detect OOD samples, it requires an additional classifier during the inference phase. Instead, NAC directly calculates the coverage scores in a **parameter-free** manner, serving as an efficient measure for both OOD detection and generalization. 

[1] Dong, Xin, et al. "Neural mean discrepancy for efficient out-of-distribution detection." CVPR, 2022

Reviewer comment (Experiments):
Is it possible to directly train or regularize models based on NAC to improve robustness. For example, one could minimize the entropy of NAC distributions to encourage balanced coverage. Evaluating such NAC-driven training schemes could further validate its usefulness.

Author response:
**A3:** Yes, it is possible! Based on your suggestion, we have tried to regularize the model using our NAC entropy loss: 
$H(\mathbf{z})= -\sum_{i=1}^{N}p_i(z_i) \log{p_i(z_i)}$, where $p_i$ associates with the NAC distribution of $i$-th neuron.  We conducted experiments on the PACS dataset, and the results are shown below:

|| Art      | Cartoon     | Photo       | Sketch      | Average     |
|----|---|----|-----|----|----|
| ERM| 77.32 ± 0.7 | 71.91 ± 0.7 | 72.36 ± 1.1 | 94.44 ± 0.2 | 79.01 |
| ERM+NAC (Minimizing Entropy)| 77.28 ± 0.2 | 69.17 ± 0.2 | 66.73 ± 1.2 | 93.21 ± 0.2 | 76.60 |
| ERM+NAC (Maximizing Entropy)| 78.64 ± 0.5 | 72.97 ± 0.3 | 72.39 ± 0.3 | 95.09 ± 0.1 | 79.77 |


Interestingly, **we find that maximizing NAC entropy leads to improved performance**. This finding also aligns with the intuitive understanding presented in [2]. By maximizing NAC entropy, we encourage the activation of neurons in unexplored regions over NAC distribution, thus diversifying the neuron activities and improving the model robustness. Conversely, minimizing entropy may result in collapsed neuron behavior. We will include the above experiments in Appendix G. 

[2] Dubey, Abhimanyu, et al. "Maximum-Entropy Fine-Grained Classification". NeurIPS 2018.