Paper: Rotation Has Two Sides: Evaluating Data Augmentation for Deep One-class Classification (ICLR-2024)

Reviewer comment (Theory):
In the first stage of the proposed method, there is no reason presented for the choice of contrastive pre-training.

Author response:
Contrastive pre-training enables the model to extract non-trivial image representations, allowing us to measure the distribution discrepancy in the representation space instead of low-level raw pixel space. The first stage of contrastive pre-training is crucial to make the entire pipeline unsupervised, although the method is not limited to self-supervised feature extractors. Alternatively, using a more powerful feature extractor pre-trained on large-scale datasets in a supervised manner yields improved recall and precision scores. See **Q2** for experimental details.

Reviewer comment (Experiments):
There are no large-scale dataset evaluations, such as imagenet.

Author response:
We conduct experiments on the TinyImageNet which has 100,000 images of 200 classes. As the two-stage pipeline in our manuscript, the first stage pre-trains the encoder by using SimCLR and the second stage learns rotation distribution. Likewise, we use the prediction (Eq.(2)) to identify RAI and non-RAI. Given the partition of RAI and non-RAI, we again pre-train the model by using SimCLR, following the practice [5] to deal with RAIs and non-RAIs, i.e., RAIs and their rotations are postive while non-RAIs and their rotations are negative. For fair comparison, we keep the same hyper-parameters in the pre-training except for the different separations of RAIs and non-RAIs. We adopt liear probing and report the top-1 classification accuracy. We improve the baseline by +2.59% and we attribute the gain to the enhanced distinction between RAI and non-RAI.

| TinyImageNet | RAI ratio (%) |  top-1 accuracy (%) |
|--------------|---------------|---------------------|
|     PNDA     |      31.3     |        37.17        |
|     Ours     |      28.9     |     39.76(+2.59)    |

Reviewer comment (Evaluation):
There is no direct quantitative evaluation of the RAI predictions. For instance, can the authors annotate a test set with binary labels of whether or not a sample is RAI?

Author response:
For quantitative evaluation, binary labels of RAI and non-RAI for the CIFAR-10 training set are manually annotated. The rotation prediction model [3] is initially employed to identify RAI samples incorrectly predicted at 0 degrees, which are then corrected manually as ground-truth RAIs. The remaining samples are designated as non-RAI. The prediction precision and recall of RAI, organized by class, are reported in the table. The table shows that our method can accurately identify RAIs in the training set. 

|Class             | plane |  car  | bird |  cat | deer |  dog | frog | horse | ship  | truck | Mean |
|------------------|-------|-------|------|------|------|------|------|-------|-------|-------|------|
|RAI ratios (%)    |  15.3 |  0.9  | 17.0 | 18.5 |  7.8 |  8.5 | 14.9 |  2.5  |   2.5 |  2.9  |  9.1 |
|   Recall         |  91.1 |  97.7 | 92.6 | 89.2 | 81.3 | 88.3 | 77.8 |  93.6 |  89.6 |  87.8 | 84.7 |
|   Precision      |  88.3 |  89.6 | 89.8 | 84.3 | 84.8 | 71.4 | 88.6 |  80.7 |  88.2 |  81.1 | 88.9 |

Reviewer comment (Theory):
In Page 1 "In cases where real-world outliers are lacking, one typical solution is to generate negative samples by applying geometric transformations, such as rotation, to the training samples.", why are the augmented samples negative?

Author response:
The augmented samples are considered negative based on the widely-adopted hypothesis in previous works [1, 2, 3, 4]. While acknowledging this hypothesis, our work emphasizes that it does not always hold, depending on the given training dataset. We propose a novel unsupervised approach to identify RAIs and non-RAIs for effective representation learning.

Reviewer comment (Experiments):
In Sec.3.1, are other pre-trained models suitable for stage 2? Why?

Author response:
The model used in the second stage requires two key attributes: 1) the ability to extract meaningful image representations, and 2) distinct representations for images and their rotated counterparts. Therefore, alternative models pre-trained in either a self-supervised or supervised manner can be considered, as long as they satisfy these conditions. The table below shows the results on one-class CIFAR-10 and suggests that our method is applicable for different pre-training approaches. Besides, we also examine a series of transformations on the SVHN dataset using a supervised model (See the Section 4.3 in the updated manuscript). The table shows the results on the training set of CIFAR-10.


|  pre-training dataset  |  pre-training way |  Recall |  Precision |
|------------------------|-------------------|---------|------------|
|   one-class CIFAR-10   |  self-supervised  |  84.7   |    88.9    |
|        CIFAR-10        |     supervised    |  93.6   |    90.3    |
|      ImageNet-1k       |     supervised    |  95.5   |    93.5    |

Reviewer comment (Reproducibility):
In Sec.3.2, how to ensure there is one sample in p_i belonging to the input domain after xy-shuffling?

Author response:
We apologize for the confusion. We first perform y-shuffle to ensure each pool $p_i$ has exactly four rotation angles but contains different image instances. Subsequently, x-shuffle is independently performed within each pool. The figure for method has been revised to display xy-shuffle sequentially for clarity.

Reviewer comment (Reproducibility):
How to pick the RAI samples according to Eq. (2）？

Author response:
Images with a prediction of 0 are deemed non-RAI, i.e., $argmax_r{p_r(x))}$ $r \in $ {0,90,180,270} equals 0, while those with any other predictions are considered RAI.

Reviewer comment (Writing):
There is no definition of p(R_{r}(x)) before it is been used in Eq.(2). And the end of Eq.(2) should be a period.

Author response:
We have revised them in the manuscript.

Reviewer comment (Experiments):
While the analysis is intriguing, its applicability remains limited to rotation-related datasets and methods. For instance, it may not be suitable for numerous real-world anomaly detection tasks, such as MVTec and VisA.

Author response:
Our work extends beyond rotation-related datasets and methods. We additionally conduct experiments on translation distribution prediction in the SVHN dataset (See the Section 4.3 in the updated manuscript). Our experimental evaluation on MvTec-AD, adopting patch representations of 32x32 built upon UniCon-HA, demonstrates our method's superiority compared to counterparts that incorporate rotation augmentation. Though our method lags behind state-of-the-art anomaly detectors, we underline the need for tailored pretext tasks [1] or leveraging pre-trained models [2] in the context of industrial anomaly detection. 

The reported Image/pixel-level AUROC scores are as follows:

|         | RotNet | DROC [3] | UniCon-HA [4]| Cutpaste [1] | Ours | 
|---------|--------|----------|--------------|--------------|------|
|  Image  |   71.0 |   86.5   |      89.8    |     95.2     | 90.6 |
|  Pixel  |   92.6 |   90.2   |      94.3    |     96.0     | 95.2 |

Reviewer comment (Evaluation):
The improvements, as depicted in Table 1, are somewhat modest compared to existing methods.

Author response:
Our approach, altering loss functions to handle semantics-preserving/-shifting samples separately, consistently achieves improvements on powerful baselines. While the average improvement over ten classes on one-class CIFAR-10 (Table 1) is not significantly high, it is noteworthy that different classes contain varying RAI ratios. Larger improvements are achieved for classes with a higher proportion of RAIs. For example, for the cat class, which has the largest number of RAIs (see Figure 4(c)), we obtain +1.0% based on CSI and +1.2% based on UniCon.

Reviewer comment (Experiments):
An essential evaluation is missing. This paper identifies RAI images and treats them differently from the original method. However, it is unclear whether the observed improvement stems from these RAI images alone. Assessing the performance of RAI images separately might lead to a more substantial improvement.

Author response:
Based on CSI, we present separate results for RAI and non-RAI on one-class CIFAR-10. The table below displays the results for the classes which include a relatively high proportion of RAI samples. Our method significantly enhances OCC performance on RAI, especially for classes with a larger proportion of RAI samples, such as plane and cat. 

|              |         |     plane    |    bird    |     cat    |    deer    |     dog    |     frog   |     Mean   |
|--------------|---------|--------------|------------|------------|------------|------------|------------|------------|
|RAI ratios (%)|         |  20.2        | 18.6       | 24.4       |  8.1       |  9.7       | 12.9       | 14.74      |  
|   CSI        |  RAI    |  73.1        | 90.2       | 79.4       | 92.0       | 91.4       | 92.4       | 86.4       |  
|   CSI + Ours |  RAI    |  84.2(+11.1) | 93.1(+2.9) | 85.3(+5.9) | 92.8(+0.8) | 92.5(+1.1) | 92.8(+0.4) | 90.1(+3.7) |
|   CSI        | non-RAI |  93.2        | 96.5       | 90.8       | 95.2       | 94.9       | 97.4       | 94.7       | 
|   CSI + Ours | non-RAI |  93.4(+0.2)  | 96.5(+0.0) | 91.9(+1.1) | 95.6(+0.4) | 95.2(+0.3) | 97.7(+0.3) | 95.1(+0.4) |

Reviewer comment (Presentation):
The paper's structure could be improved. There is significant overlap in the information presented in Figures 1, 2, 4, 5, 6, and 7. It may be advisable to move figures like 4-7 to the supplementary section.

Author response:
Figure 1 and Figure 2 illustrate the correlation between rotation prediction and OCC. Figures 4-7 provide detailed analyses of the effects of RAIs and non-RAIs. We reorganize the structure in our final version.

Reviewer comment (Experiments):
The rationale behind the authors' decision to use the version of UniCon without soft aggregation and hierarchical augmentation remains unclear. Since hierarchical augmentation is an integral module within UniCon, it is advisable for the authors to use the full version of UniCon as the baseline for their study.

Author response:
Built upon the complete version of UniCon-HA, which is a very strong OCC detector, it is noted that we also achieve the consistent improvement (+0.2%) on one-class CIFAR-10.

Reviewer comment (Evaluation):
There is not a quantifiable way to measure the accuracy in RAI vs. non-RAI determination. One reason for this is that RAI images may be classified as 0, making it hard to separate these images from the truly non-RAI images.

Author response:
**W1(Q1)**: Quantitative accuracy in RAI vs. non-RAI determination.

For quantitative evaluation, binary labels of RAI and non-RAI for the CIFAR-10 training set are manually annotated. The rotation prediction model [3] is initially employed to identify RAI samples incorrectly predicted at 0 degrees, which are then corrected manually as ground-truth RAIs. The remaining samples are designated as non-RAI. The prediction precision and recall of RAI, organized by class, are reported in the table. The table shows that our method can accurately identify RAIs. The results can be further improved by leveraging a more powerful pre-trained feature extractor, as demonstrated in **W2**. 

 
Trainig set:
|Class             | plane |  car  | bird |  cat | deer |  dog | frog | horse | ship  | truck | Mean |
|------------------|-------|-------|------|------|------|------|------|-------|-------|-------|------|
|RAI ratios (%)    |  15.3 |  0.9  | 17.0 | 18.5 |  7.8 |  8.5 | 14.9 |  2.5  |   2.5 |  2.9  |  9.1 |
|   Recall         |  91.1 |  97.7 | 92.6 | 89.2 | 81.3 | 88.3 | 77.8 |  93.6 |  89.6 |  87.8 | 84.7 |
|   Precision      |  88.3 |  89.6 | 89.8 | 84.3 | 84.8 | 71.4 | 88.6 |  80.7 |  88.2 |  81.1 | 88.9 |

Reviewer comment (Evaluation):
There is no discussion or inclusion of failure modes to understand where the model may succeed vs. fail.

Author response:
**W2(Q2)**: Failure modes.

In fact, the last table shows that our model has misclassified images and these incorrect predictions stem from imperfect representations learned in contrastive pre-training, which are contextually biased towards spurious scene correlations [1, 2]. To address this, we suggest using a more powerful feature extractor pre-trained on large-scale datasets. For instance, leveraging an encoder pre-trained on ImageNet-1k has resulted in improved outcomes in the identification of both RAI and non-RAI. The table shows the results on the training set of CIFAR-10.

|  pre-training dataset  |  pre-training way |  Recall |  Precision |
|------------------------|-------------------|---------|------------|
|   one-class CIFAR-10   |  self-supervised  |  84.7   |    88.9    |
|        CIFAR-10        |     supervised    |  93.6   |    90.3    |
|      ImageNet-1k       |     supervised    |  95.5   |    93.5    |

Reviewer comment (Experiments):
The utility of the transformation distribution may be larger but is focused on OCC, and datasets used to evaluate OCC are from CIFAR-10, which may have different characteristics than some of the anomaly detection settings where labeled data of out-of-distribution samples could be limited.

Author response:
**W3**: Transformation distribution vs. OCC.

While rotation prediction as a pretext task has been explored in the self-supervised literature and facilitates bundles of downstream tasks, its significance becomes more pronounced in OCC, where the effectiveness relies solely on the design of the pretext task to capture normal patterns. Furthermore, our empirical experiments reveal a strong linear relationship between the accuracy of rotation prediction and the performance of OCC. This observation motivates us to delve into investigating how the separation between RAI and non-RAI impacts OCC. While we have achieved improved results in multi-class classification on TinyImageNet, evaluated by linear probing. We have revised the manuscript to provide additional clarity on this aspect.

| TinyImageNet | RAI ratio (%) |  top-1 accuracy (%) |
|--------------|---------------|---------------------|
|     PNDA     |      31.3     |        37.17        |
|     Ours     |      28.9     |     39.76(+2.59)    |

Reviewer comment (Experiments):
A big weakness right now is the lack of extensive empirical validation. The authors currently only perform experiments on CIFAR-10 and their experiments on other kinds of transformations are also very limited. Though the authors show interesting results for another transformation, it is immensely difficult with this set of results to comment on how well their approach could work.

Author response:
**W1**: Lack of empirical validation on datasets other than CIFAR-10.

We additionally conduct experiments on the TinyImageNet which has 100,000 images of 200 classes. As the two-stage pipeline in our manuscript, the first stage pre-trains the encoder by using SimCLR and the second stage learns rotation distribution. Likewise, we use the prediction (Eq.(2)) to identify RAI and non-RAI. Given the partition of RAI and non-RAI, we again pre-train the model by using SimCLR, following the practice [5] to deal with RAIs and non-RAIs, i.e., RAIs and their rotations are postive while non-RAIs and their rotations are negative. For fair comparison, we keep the same hyper-parameters in the pre-training except for the different separations of RAIs and non-RAIs. We adopt liear probing and report the top-1 classification accuracy. We improve the baseline by +2.59% and we attribute the gain to the enhanced distinction between RAI and non-RAI.

| TinyImageNet | RAI ratio (%) |  top-1 accuracy (%) |
|--------------|---------------|---------------------|
|     PNDA     |      31.3     |        37.17        |
|     Ours     |      28.9     |     39.76(+2.59)    |

For the experiments on SVHN, refer to responses to W2 for details.

Reviewer comment (Experiments):
One of the really interesting findings from this paper is about transformation bias, and representations learned from transformations already present tend to be less effective. I believe this would be well-shown by experiments across multiple datasets and multiple kinds of models.

Author response:
**W2 (Q1 and Q2)**: Other transformations.

To address the need for exploring other transformations, we have extended our evaluation to include translation on the Street View House Numbers dataset (SVHN). The goal is to classify the central digit in each image into one of the categories (0 through 9). We consider five translation directions (original, up, down, left, right) with an 8-pixel shift. Notably, our approach is not confined to self-supervised pre-training. The first stage trains the ResNet-18 from scratch in a supervised way, achieving 96.24% accuracy on the test set. The second stage predicts the 5-way translation distribution. As shown in the updated manuscript, it showcases the model’s ability to correctly identify off-centered digits. See Section 4.3 in the updated manuscript for more discussion on the experiments on SVHN.

Reviewer comment (Experiments):
How does the analysis across other kinds of popular transformations look like and does this approach still hold, I would encourage the authors to include talking about other transforms even if they do not seems to work well.

Author response:
**W3**: Experiments across other models.

We have explored ViT-based models, specifically ViT-B/16, for rotation distribution prediction in CIFAR-10. Additionally, we considered variations in architecture types and pre-training strategies (self-supervised vs. supervised). For quantitative evaluation, binary labels of RAI and non-RAI for the CIFAR-10 training set are manually annotated. The provided table demonstrates the architecture-agnostic nature of our method, with ResNet-18 leading to a larger improvement than ViT-B/16. This is attributed to the lack of inductive bias in ViT, which struggles to learn local relations well with a small amount of data, such as CIFAR-10. The results also highlight our method's preference for the supervised training strategy, emphasizing the significance of well-trained image representations, as indicated by average recall and precision values across the 10 classes in CIFAR-10.


| Architecture | pre-train (1st stage) |   Recall  |  Precision  |
|--------------|-----------------------|-----------|-------------|
|    PNDA[1]   |           /           |   77.1    |    62.0     |
|   ResNet-18  |    self-supervised    |   84.7    |    88.9     |
|   ResNet-18  |      supervised       |   93.6    |    90.3     |
|   ViT-B/16   |    self-supervised    |   82.6    |    83.6     |
|   ViT-B/16   |      supervised       |   90.2    |    85.4     |




   
[1] Miyai et al. Rethinking rotation in self-supervised contrastive learning: Adaptive positive or negative data augmentation. In WACV, 2023.

Reviewer comment (Presentation):
Related works section of the paper is difficult to follow and seems incomplete.

Author response:
We have incorporated more discussion on relevant OCC literature in the revised manuscript. Anomaly detection (AD) and OCC share similarities, aiming to identify instances deviating from normal training data and providing a binary classifier to determine normality or anomaly. The key distinction lies in the training data characteristics. AD considers both normal and anomalous instances during training, while OCC is confined to only normal instances in training. Therefore, OCC can be regarded as a special case of AD. When all training data belongs to one class, AD and OCC methods are expected to yield similar results.

Reviewer comment (Writing):
There exists a confusion regarding anomaly detection and OCC cited in related work, in which cases are they both considered the same?

Author response:
We have incorporated more discussion on relevant OCC literature in the revised manuscript. Anomaly detection (AD) and OCC share similarities, aiming to identify instances deviating from normal training data and providing a binary classifier to determine normality or anomaly. The key distinction lies in the training data characteristics. AD considers both normal and anomalous instances during training, while OCC is confined to only normal instances in training. Therefore, OCC can be regarded as a special case of AD. When all training data belongs to one class, AD and OCC methods are expected to yield similar results.