Paper: Scaling for Training Time and Post-hoc Out-of-distribution Detection Enhancement (ICLR-2024)

Reviewer comment (Novelty):
Incorrect Motivation: The author seems to have not conducted sufficient research in the field of OOD detection. Generally, OOD detection requires the model to identify OOD data without affecting the accuracy of ID classification. Hence, there should not be a trade-off between ID classification accuracy and OOD detection performance. Post-hoc OOD detection refers to performing OOD detection through post-processing algorithms without altering the model parameters. Consequently, React, Dice, and ASH utilize the original classifier for ID classification tasks and a modified classifier for OOD detection tasks, introducing a very minimal computational overhead. Therefore, I believe there is a problem with the motivation presented in the introduction section of the paper.

Author response:
Thanks for highlighting this confusion. Indeed the ID accuracy drop can be avoided through a separate classifiers with minimal computational cost. We will emphasize this point more in the introduction. We note that ID-OOD accuracies trade-offs are also reported in ReAct Table 2 and DICE Table 5, and further explored in ASH where they argue that an ideal OOD detection pipeline "should not deteriorate ID task performance, nor should it require a cumbersome parallel setup that handles the ID task and OOD detection separately". We will clarify this point.
We note that our motivation is not only achieving a better tradeoff between OOD detection and ID classification but also achieving higher OOD detection performance than others. We do this in a principled way through theoretical analysis, giving explanations and insights into the working mechanisms of existing works before making our proposal.

Reviewer comment (Novelty):
Limited Novelty: SCALE and ISH are incremental works of ASH. Thus, I attribute the commendable performance of ISH and SCALE mainly to the superior performance of ASH. The performance improvement over ASH is quite minimal.

Author response:
Thank you for your comments. The main contribution is providing a theoretical understanding of the current state-of-the-art method ASH. Our development of SCALE is directly born out of this analysis and verifies that it is only the scaling operation which benefits OOD detection (as pointed out by reviewer AeGx). 
Furthermore, we propose to leverage scores as ID-ness in extra training, which further improves OOD detection without the need for extra OOD validation data.
For experimental results. Compared to ASH-S, we achieved 1.73+ on near-OOD, +0.06 on far-OOD, and +0.67 on ID accuracy for SCALE, and achieved 4.83+ on near-OOD, 0.32+ on far-OOD, and 0.56+ on ID accuracy for ISH+SCALE. We note that current methods are already good for easier far-OOD detection so the improvement room is smaller.

Reviewer comment (Experiments):
Unsubstantiated Claim on React: The author claims that React “hinders the OOD detection process.” However, this analysis is solely theoretical and lacks experimental validation. I think the results from SCALE+React or ISH+React could potentially validate this point.

Author response:
Thanks for your suggestions. Our statement in the introduction is that ”the lower-part pruning approach, in contrast to ReAct, hinders the OOD detection process”. ReAct is a higher-part pruning. In our original paper, we provide an analysis that ASH pruning impedes the OOD process because lower-part pruning brings ID and OOD data closer in the logits space. For ReAct, higher-part pruning enlarges the distance between ID and OOD and this provides an explanation for its effectiveness.

Reviewer comment (Experiments):
Minimal Performance Gain: As observed from Table 5, the performance improvement brought by combining ISH and SCALE compared to using SCALE alone is extremely minimal. I am curious to know if combining ISH with other OOD detectors (e.g., ISH+MSP/ODIN/Energy/React/ASH) could yield any performance improvement.

Author response:
Thanks for your suggestions. For the experiments results, ISH + SCALE improves an additional notable 1.34 on SCALE, which has 21% improvement than AugMix with less training efforts. We provide results for ISH + MSP/ODIN/Energy/ReAct/ASH in the following table and show that ISH increases all results. 

|            | Near-OOD AUROC | Far-OOD AUROC |
|------------|----------------|---------------|
| MSP        |    76.03       |     85.23     |
| MSP+ISH    |    **76.87**       |     **85.86**     |
| ODIN       |     74.75      |     89.47     |
| ODIN+ISH   |     **76.51**          |    **90.18**        |
| Energy     |    75.89       |     89.47     |
| Energy+ISH |     **77.62**      |     **90.06**     |
| ReAct      |     77.38      |     93.67     |
| ReAct+ISH  |     **78.99**      |     **93.96**     |
| ASH-S        |     79.63      |     96.47     |
| ASH-S+ISH    |     **82.26**      |     **96.61**     |

Reviewer comment (Experiments):
Lack of Ablation Study: The paper is missing an ablation study related to the placement of SCALE, specifically, how its performance varies when placed after different stages of ResNet50.

Author response:
We appreciate your valuable suggestions and provide the ablation study on the following table:


|             | Near-OOD AUROC | Far-OOD AUROC |
|-------------|---------------|---------------|
| Block1      |  48.62  |  64.09  |
| Block2      |  62.79  |    62.06 |
| Block3      | 70.08   |  62.27  |
| Block4      | 78.86   |  90.65  |
| Penultimate | **81.36** | **96.53**   |

As the stage becomes shallower, the performance decreases. This is because the early stages tend to learn low-level features that are common to both the ID and OOD datasets.

Reviewer comment (Experiments):
For CIFAR experiments, I strongly encourage the authors to refrain from using LSUN-Crop and LSUN-Resize as OOD datasets, if they are directly taken from the ODIN paper. LSUN-Resize has been shown to exhibit obvious resizing artifacts [1], which make the detection trivially easy. LSUN-Crop is 32x32 crops from images with larger resolution, and arguably the resulting samples will have unnaturally different distribution compared to CIFAR's 32x32 natural images, which again makes the evaluation less meaningful.

Author response:
- **LSUN-Crop and LSUN-Resize results**: 
Thanks for pointing this out. In our revision, we will remove LSUN-Crop and LSUN-Resize from the main text and place the original table in the Supplementary for consistent comparison to ODIN; and provide discussion to give awareness to the reader.

Reviewer comment (Evaluation):
I was wondering if the analysis provided in this work can also help explain ReAct's effect. Some elaboration or discussion on this can help provide even a more complete picture.

Author response:
- **Explanation of ReAct**:
Our analysis is relevant to ReAct, a higher-part pruning approach. In our original paper, we provide an analysis that ASH pruning impedes the OOD process because lower-part pruning brings ID and OOD data closer in the logits space. For ReAct, higher-part pruning enlarges the distance between ID and OOD and this provides an explanation for its effectiveness.

Reviewer comment (Experiments):
The method is tested only in Imagenet. The authors should use it for OOD detection in other kinds of datasets to ensure that the method is applicable in various domains.

Author response:
We thank reviewer DQbC for your insightful review.
- **The method is tested only in Imagenet:** 
Table 3 shows experiments on CIFAR-10 and CIFAR-100.

Reviewer comment (Experiments):
How does the method generalize to other types of data e.g. tabular, time-series etc.?

Author response:
We thank reviewer DQbC for your insightful review.
- **Other Data Domains:**  
Our method, like prevailing ad-hoc methods for OOD (EBO, ReAct, ASH) are intended for already-trained networks by design. The readily available off-the-shelf image backbones is why we and others work primarily in the image domain, extending it to other domains like time series and tabular data which do not have analogous backbones will likely need fundamental changes in the assumptions and approach beyond our current scope; we leave this for future work.
We strongly support the point that OOD for other domains is a worthwhile focus and we leave it for future research.

EBO: Liu, Weitang, Xiaoyun Wang, John Owens, and Yixuan Li. "Energy-based out-of-distribution detection." _Advances in neural information processing systems_ 33 (2020): 21464-21475.

ReAct: Sun, Yiyou, Chuan Guo, and Yixuan Li. "React: Out-of-distribution detection with rectified activations." _Advances in Neural Information Processing Systems_ 34 (2021): 144-157.

ASH: Djurisic, Andrija, Nebojsa Bozanic, Arjun Ashok, and Rosanne Liu. "Extremely simple activation shaping for out-of-distribution detection." _arXiv preprint arXiv:2209.09858_ (2022).

Reviewer comment (Reproducibility):
Could the authors provide an algorithm to demonstrate the overall pipeline of the proposed method? I am confused about when we should invoke the method in Section 3.4.

Author response:
We extend our gratitude to RAx2 for the insightful review.
- **Algorithm for ISH pipeline**: 
Thanks for your suggestion. We provide pseudo-code of the algorithm here and will add it to Sec. 4.3 in the revision.
```
function ISH(pretrained_model M, data D, learning_rate L, epochs I)
	for input, label in D: 
		# forward pass get predicted results and intermediate activations.
		activation_1, ...,  activations_p, predicted = forward_pass(M, input)
		gradient_z = loss(predicted-input)
		
		# get scaling s for the activations of for penultimate layer:
		s = SCALE(activations_p) # as defined in equation (7)
		
		# for penultimate layer:
		# scale activations_p by s for gradient computation
		gradient_weights = dot_product(s  * activations_p^T, gradient_z)
		gradient_biases = gradient_z
		
		update_parameters(gradient_weights, gradient_biases, learning_rate) 
		
		update other parameters
		
		return weights, biases
```