Paper: ReweightOOD: Loss Reweighting for Distance-based OOD Detection (ICLR-2024)

Reviewer comment (Evaluation):
In the abstract, the author emphasizes that their method outperforms the baseline by 38%, which seems impressive. However, this improvement is mainly due to the low performance of the baseline method itself. The proposed method does not actually achieve such a significant improvement compared to the state-of-the-art (SOTA). In my opinion, this exaggerates the contribution of this paper. It would be better to clarify the improvement relative to SOTA in the beginning to avoid misleading the readers.

Author response:
We will clarify the improvement relative to the SoTA approaches by being more specific in the abstract in the upcoming versions. We believe the performance improvement compared to the current competitive approaches such as SupCon and CIDER is still significant.

Reviewer comment (Novelty):
The simplicity of the method itself can be considered an advantage. However, based on the experimental results, the improvement of this paper compared to the state-of-the-art is limited. Therefore, I hope the authors can further explore this direction and achieve more substantial advancements.

Author response:
We will clarify the improvement relative to the SoTA approaches by being more specific in the abstract in the upcoming versions. We believe the performance improvement compared to the current competitive approaches such as SupCon and CIDER is still significant.

Reviewer comment (Experiments):
The experiments conducted in this paper are not comprehensive enough. Firstly, experiments on complete ImageNet-1K were not conducted. Additionally, the comparison with the latest methods, such as those presented in recent conferences like CVPR 2023, was not included.

Author response:
Orthogonal to the papers presented in CVPR2023, our work has the distinction in that it adopts a reweighting approach which is novel to the best of our knowledge.

The ImageNet-1k experiments adopting the ImageNet100 setup is presented below:

  | Method                 | iNaturalist | SUN  | Textures | Places | SSB Hard | Ninco | Openimage | Average **FPR** $\downarrow$ |
  |------------------------|-------------|------|----------|--------|----------|-------|-----------|--------------------------------|
  | SupCon                 | 61.93        | 68.85 | 25.43     | 75.28   | 90.79    | 78.82 | 55.64      | 65.25                          |
  | CIDER                   | 53.54        | 61.27 | 23.95     | 68.76   | 91.52    | 79.15 | 54.22      | 61.77                          |
  | **(ReweightOOD) *Ours***| 50.70        | 59.46 | 24.27     | 66.27   | 90.18   | 76.76 | 50.97      | **59.80**                      |
  
  It can be observed that the proposed approach has noticeable gains over SupCon and CIDER demonstrating its superiority.

Reviewer comment (Experiments):
**There are many recent OOD baselines but they are not taken into consideration for comparison.** There are many recent OOD baselines but the authors either do not cite them [1,2,3] or do not add them in the comparison (e.g., ASH, GradNorm). In particular, these are very strong baselines and I see the performance of this method at the same level as ReAct. It is a bit strange that the evaluation does not involve any activation-clipping baselines such as ReAct and ASH (they are clear state-of-the-art methods and do not require any training).

Author response:
We thank the reviewer for the concrete suggestions. We address the presented concerns below:

- The updated manuscript now contains the suggested comparisons and we do cite all the mentioned works in the manuscript.

Reviewer comment (Experiments):
**The experiments are not sufficient only on CIFAR/ImageNet100.** The authors only validate the proposed approach on CIFAR100 and ImageNet100, which is far from sufficient and comprehensive. Usually, people evaluate OOD methods on CIFAR10, CIFAR100, and ImageNet-1k benchmarks. I would be convinced if the method also works for ImageNet-1k.

Author response:
We thank the reviewer for the concrete suggestions. We address the presented concerns below:

- The experiments with the Imagenet-1k dataset following the setup of the ImageNet-100 experiment are shown below.
  | Method                 | iNaturalist | SUN  | Textures | Places | SSB Hard | Ninco | Openimage | Average **FPR** $\downarrow$ |
  |------------------------|-------------|------|----------|--------|----------|-------|-----------|--------------------------------|
  | SupCon                 | 61.93        | 68.85 | 25.43     | 75.28   | 90.79    | 78.82 | 55.64      | 65.25                          |
  | CIDER                   | 53.54        | 61.27 | 23.95     | 68.76   | 91.52    | 79.15 | 54.22      | 61.77                          |
  | **(ReweightOOD) *Ours***| 50.70        | 59.46 | 24.27     | 66.27   | 90.18   | 76.76 | 50.97      | **59.80**                      |
  
  It can be observed that the proposed approach has noticeable gains over SupCon and CIDER demonstrating its superiority.

Reviewer comment (Experiments):
**The instance weight can be replaced with learnable temperatures.** Adding weight significance to the samples is one way to obtain a higher inter-class separation, but I am wondering if it would also make sense to make the temperature learnable. Compared with weight significance, it would bring another benefit: learnable distribution shaping for each category.

Author response:
We thank the reviewer for the concrete suggestions. We address the presented concerns below:

- We thank the reviewer for the possible further extensions. We aim to undertake such an approach in future work.

Reviewer comment (Experiments):
**How does the method perform when scaling to more/fewer categories?** I would suspect the performance is highly related to the number of categories. If the value is too large (e.g., 1000) or too small (e.g., 5), the performance might deteriorate. Can authors provide some ablation studies on subsets of the used datasets?

Author response:
We thank the reviewer for the concrete suggestions. We address the presented concerns below:

- As can be observed from the above ImageNet-1k experiments, our approach demonstrates performance gain even when scaled to an even more challenging (higher number of categories) setting.

Reviewer comment (Presentation):
The figures 3 and 4 are not mentioned in the article.

Author response:
We have resolved the issue in the latest manuscript by referencing Figure 3 and Figure 4.

Reviewer comment (Presentation):
I find Figure 5 visually unappealing, and I wonder why there is only one category in the right figure.

Author response:
Apologies for the overleaf rendering issues, the correctly rendered visualization is updated in the manuscript.

Reviewer comment (Experiments):
More importantly, why not conduct experiments on the ImageNet dataset, like existing works [1] have done?

Author response:
Thanks for pointing it out. Adopting the training setup of ImageNet100, we present the requested comparison with ImageNet-1k below: 
             
  | Method                 | iNaturalist | SUN  | Textures | Places | SSB Hard | Ninco | Openimage | Average **FPR** $\downarrow$ |
  |------------------------|-------------|------|----------|--------|----------|-------|-----------|--------------------------------|
  | SupCon                 | 61.93        | 68.85 | 25.43     | 75.28   | 90.79    | 78.82 | 55.64      | 65.25                          |
  | CIDER                   | 53.54        | 61.27 | 23.95     | 68.76   | 91.52    | 79.15 | 54.22      | 61.77                          |
  | **(ReweightOOD) *Ours***| 50.70        | 59.46 | 24.27     | 66.27   | 90.18   | 76.76 | 50.97      | **59.80**                      |

  It can be observed that the proposed approach has noticeable gains over SupCon and CIDER demonstrating its superiority.

Reviewer comment (Novelty):
Although the reweighting on constrative learning has been under-explored in OOD detection community, focusing more on hard postivie/negatives are widely applied in deep metric learning (especially in face recognition). E,g [1][2]. I think the novelty is sort of limited as the paper adopt hard sample mining/weighting into OOD detection task. In rebuttal, can you explain the difference bewteen your work and [1,2] in terms of hard samples re-weighting?

Author response:
A detailed clarification of the distinction between our proposed approach against CurricularFace and AdaFace is presented below.

-  CurricularFace adopts a curriculum-style learning approach, wherein early/late training dynamics are taken into account. It emphasizes easy samples at the earlier stage while prioritizing the hard samples at the later stage. ReweightOOD does not impose curriculum-style learning at all, as it does not distinguish between early and late training phases. The reweighting function of ReweightOOD, in the form of a sigmoid, strategically suppresses easy samples from the outset, directing attention toward challenging samples. AdaFace, another proxy-based approach, adopts the quality of the image -- approximated by feature norm -- into the weighting of the sample's importance. In AdaFace, hard samples from higher-quality images receive increased weighting, while easy samples from lower-quality images are weighted higher. In contrast, our approach focuses solely on the cosine similarity when reweighting pair instances. Moreover, ReweightOOD facilitates distinct treatment of positive and negative pairs due to the presence of a transformation function with separate hyperparameters. The utilization of two sets of linear transformations, specific to positive and negative pairs, enables flexible weighting tailored to achieving optimal OOD detection. Additionally, both curricularFace and adaFace adopt cross-entropy-based optimization while ReweightOOD uses similarity maximization/minimization optimization as shown in equation (3).

Reviewer comment (Evaluation):
For Table 1,2, I am interested to see how the MES score and Average centroid dispersion for CIDER.

Author response:
The updated MES Table with the results of CIDER is given below:
   | Method                 | Apples | Aquarium Fish | Baby | Bear | Beaver | Bed  | Bee | Beetle | Bicycle | Bottles | ... | Mean  |
   |------------------------|--------|---------------|------|------|--------|------|-----|--------|---------|---------|-----|-------|
   | Baseline               | 1.09   | 1.11          | 1.07 | 0.93 | 0.97   | 1.07 | 1.03| 1.07   | 1.10    | 1.07    | ... | 1.05  |
   | SupCon                 | 0.97   | 1.01          | 0.97 | 1.00 | 1.00   | 0.98 | 0.99| 0.97   | 1.04    | 1.03    | ... | 1.01  |
   |  *CIDER*              | 0.96   | 0.94          | 0.89 | 0.91 | 0.86   | 0.88 | 0.87| 0.88   | 1.04    | 0.99    | ... | 0.92  |
   | **(ReweightOOD) *Ours***| 0.95   | 0.90          | 0.90 | 0.89 | 0.84   | 0.90 | 0.89| 0.91   | 0.99    | 0.96    | ... | **0.90** |

   As evidenced from the above Table, the average MES of ReweightOOD is minimum among all. Moreover, the class centroid dispersion of ReweightOOD (0.63) is comparable to that of CIDER (0.64) despite not explicitly encouraging the inter-class dispersion.

Reviewer comment (Experiments):
Hyperparameters test on linear transformation are encouraged. It is said that ResNet18 network are set to (5, −2, 2, 1) and (5, −4, 2, 1). Do you have to set differently for other backbones or datasets?

Author response:
The random hyperparameter optimization is employed with respect to the validation set (gaussian noise). The hyperparameter set to (5, −4, 2, 1) consistently demonstrates strong performance in experiments conducted on both CIFAR100 and ImageNet100 datasets, across all architectures used in the paper.