Paper: Unified Anomaly Detection via Multi-Scale Contrasted Memory (ICLR-2024)

Reviewer comment (Experiments):
While the authors assert that their model outperforms state-of-the-art approaches across a wide range of anomalies, it is notable that the paper lacks evaluation on widely-accepted benchmark datasets such as MVTec [1] and VisA [2]. These datasets contain texture anomalies, which are crucial for a comprehensive evaluation. Although CIFAR-10/100 and CUB are important, their evaluation should be complemented by assessments on these texture-oriented datasets.

Author response:
Our extensive tests cover a wide range of anomalies, including coarse object anomalies (CIFAR), subtle style anomalies (CUB200, a challenging fine-grained dataset for anomaly detectors with random performance of Sota method like CSI [1]; while recent method MSC [2] does not report their performance on this dataset), and a real-world dataset of face presentation attack WMCA with local anomalies. We would like to highlight that the WMCA conveys different texture anomalies with different forms of attacks, i.e., paper print (PP), screen recording (SR), paper mask (PM) and flexible mask (PM).

With respect to datasets such as MVTec and VisA, only a few studies have employed them for evaluating anomaly detection. Notably, these datasets are primarily designed for anomaly localization purposes. We intend to consider the localization task in future work, as mentioned at the end of the paper.

[1] Tack et al., “CSI: Novelty detection via contrastive learning on distributionally shifted instances”, NeurIPS 2020.

[2] Tal Reiss and Yedid Hoshen, “Mean-shifted contrastive loss for anomaly detection”. AAAI 2023

Reviewer comment (Evaluation):
The paper mentions superior performance over existing methods, but Table 2 reveals that AnoMem, at best, achieves comparable results in out-of-distribution (OOD) detection. It's important to ensure that the claims made align with the empirical findings.

Author response:
Thank you for your effort for the detailed review. Indeed, we were already aware of the positioning of our results in the literature for OOD detection.

We have thoroughly reviewed the initial submission and found that we did not explicitly claim superior performance over existing methods for OOD detection. For example, in the abstract, we stated, 'Our model outperforms the state-of-the-art on a wide range of anomalies, including object, style, and local anomalies, as well as face presentation attacks' without specifically mentioning superior performance for OOD. In Section 4.2.2, we began by stating, 'AnoMem performs similarly well as the state-of-the-art baseline CSI.' To eliminate any possible confusion, we have now explicitly mentioned the performance of our method for OOD in the abstract.
It is worth noting that for OOD detection with CIFAR10 as ID, CSI and pretrained models need to store 60000 samples, whereas ours only requires 768 samples.

Reviewer comment (Experiments):
Several crucial components of the proposed method are not examined by the authors. For example, while the authors briefly mention the potential substitution of the NTX objective with alternative contrastive frameworks like Barlow-Twins, which are known for their efficiency and ability to operate effectively with smaller batch sizes, this proposition lacks empirical demonstration. The absence of a comparative analysis or experimental results to support this claim leaves a gap in the evaluation of the method's adaptability and robustness across different contrastive learning frameworks.

Author response:
We do recognize that our initial statement might have been misinterpreted as an affirmation of our framework working with all contrastive methods. We only wanted to mention the fact that our contrastive memory learning can easily be integrated into a two-branched contrastive approach such as Barlow twins, BYOL or SimSiam.

Due to the limited time of the rebuttal phase, we removed our potentially misleading statement from the method presentation, and reframed it in future work as a direction to improve the efficiency of multiscale contrastive learning.

Reviewer comment (Experiments):
In FPAD, the CSI method is not included in performance metrics. What are the CSI results?

Author response:
We have now incorporated the results of CSI for FPAD using its official code in Table 3. Once again, AnoMem demonstrates an important performance advantage over CS in this challenging task. It is also worth noting that in CUB, AnoMem significantly outperforms CSI, showing improvements of nearly 30 points.

Reviewer comment (Evaluation):
The explanation of the linear evaluation protocol, while briefly described, lacks depth. It would be valuable to have a more detailed elaboration on the rationale behind employing this protocol and why it is considered an important metric in the context of anomaly detection. Moreover, given that this protocol may not be standard within the AD community, it raises curiosity about how other existing methods perform under this evaluation criterion. Could the authors shed light on the performance of competing methods using this protocol for comparison and context? This information would further enrich the assessment of the proposed approach.

Author response:
Thank you for your question. 

In fact, the linear evaluation experiment focuses on multi-class image classification rather than anomaly detection. The objective is to validate the efficacy of the first step in our overall approach. In more detail, once the representation 'z' is learned, we freeze it and train an additional linear classifier. We then evaluate the entire model (frozen encoder + linear classifier) for image classification tasks using CIFAR-10 and CIFAR-100. The initial submission may have contained some ambiguities due to the merging of results from two distinct tasks. To address this, we have now separated the results for different tasks into two distinct Tables 4 and 5.

Reviewer comment (Novelty):
The motivation behind the paper is very direct, but the authors do not further discuss the benefits of unifying the one-class problem and out-of-distribution problem.

Author response:
Thank you for the question. We made this point clearer in the revised paper.
We would like to emphasize that AnoMem stands out as the first to be both unified and generic.
1.  In terms of its unified property, specifically regarding the availability of anomaly samples during training, AnoMem excels in both one-class and imbalanced outlier-exposure settings—a crucial practical feature in real-world scenarios where normal samples are initially available, and anomaly samples are acquired later, a scenario not adequately addressed by existing literature.
2. Moving on to its generic property, AnoMem demonstrates effectiveness in object-/semantic-level anomaly detection, out-of-distribution (OOD) detection, and also in industrial applications like fine-grained anomaly detection (FPAD). 
3. While not the primary focus of our paper, we further demonstrate that our learning method can enhance the more general task of image classification. This aspect might have been overlooked in the initial submission, where different results for distinct tasks were presented in the same Table 4. To provide a clearer depiction of its capabilities, we have now separated the results for different tasks into Tables 4 and 5.

Reviewer comment (Evaluation):
Although the framework of the paper for the one-class problem and out-of-distribution problem is consistent, the paper adds a classifier for the out-of-distribution problem. This may be due to the different evaluation metrics of the two problems, but it makes the two problems slightly disconnected.

Author response:
Thank you for your question. 
We did NOT include a classifier specifically for out-of-distribution detection. The additional linear classifier is solely trained to evaluate our learned representation z on image classification tasks using CIFAR-10 and CIFAR-100. The initial submission may have contained some ambiguities due to the merging of results from two distinct tasks, as mentioned earlier. To address this, we have now separated the results for different tasks into two distinct tables (please refer to Tables 4 and 5).

Reviewer comment (Experiments):
Essentially, the authors weight different scales of features to unify the one-class problem and out-of-distribution problem. However, the parameter selection for weighting seems somewhat arbitrary.

Author response:
For the choice of weights for each scale, we considered two types of monotonically increasing functions to give more weight to the latter scales in which we have more confidence. The results for the two weight functions linear and exponential have been included in Table 7 in Appendix (due to the space limit). As we can see, we obtained better results with the exponential function justifying our choice.

Reviewer comment (Experiments):
The authors mention at the end of the paper that their method can be used for anomaly localization, and the model design does seem to support this. It is possible that the authors did not consider further experiments on this due to unsatisfactory experimental results.

Author response:
Thank you for your remark.
We do believe that our model could be successfully used for anomaly localization, however not in its current form. Indeed, using the two scales in the latter layers as in our proposed model yields rather low resolution anomaly maps and would have a low AUPRO segmentation metric on datasets such as MVTec. However, as discussed in the paper, adding more scales in earlier layers would result in a substantially slower and larger model. Therefore, the main issue to tackle for an efficient anomaly localization in future work would be to optimize the memory mechanism or use an alternative contrastive scheme that takes less memory per memorized pattern.

Reviewer comment (Experiments):
It is unclear how much difference it makes by bringing multi-scale learning into the memory learning-based anomaly detection approaches. No appropriate ablation study or empirical comparison is presented there. The baseline in table 4 may be changed to a memory learning method that involves multiple normal prototypes to serve this purpose.

Author response:
**Reply:**
We include in Tab.5 an additional ablation study on the multi-scale component alone to better measure its impact on anomaly detection. As we can see, the multiple scales by themself improve notably the AD performances on CIFAR-10, CIFAR-100 and WMCA.

Reviewer comment (Evaluation):
As demonstrated in table 4, the multi-scale learning has very limited contribution to the overall detection performance.

Author response:
**Reply:**
The limited impact on the overall detection performances of CIFAR-10 and CIFAR-100 can be explained by the low amount of local anomalies in these datasets. By including representations from earlier layers in the network we can help our model to better encompass smaller and finer anomalies that are more represented in WMCA with partial attacks, as can be seen in the **additional ablation results** on WMCA in Table 5.

Reviewer comment (Experiments):
It mentions at page 4 about the issue of overfitting in existing methods that use a pre-trained encoder on large-scale image datasets, but this issue should be easily fixed by tuning on the target normal data. I cannot find convincing reasons for not using such well pre-trained encoders, and not comparing with such methods. Additionally, in this work, there is the extensive fitting of the multi-scale memory units to the limited normal data in the target data, which could easily lead to overfitting to the normal data, and so the proposed method could perform poorly on datasets with distribution shift. I wonder whether the authors could perform experiments on datasets with distribution shift to justify their argument.

Author response:
**Reply:**
In Table 3, we have included additional results for the state-of-the-art pretrained method MSC [Reiss AAAI’23] on WMCA. As can be seen, this pretrained model encounters challenges in generalizing to the specific FPAD task, given the substantial distribution shift between the WMCA domain and the ImageNet images that MSC uses to pretrain its model.

The table also demonstrates the advantage of our simple method (without additional large-scale dataset) in the case of unseen attacks, which can be considered a form of distribution shift.


Moreover, one motivation that can favor models without large-scale dataset for pretraining is dataset licensing, especially in the context of industrial applications like FPAD. Indeed, some collected data might not be licensed to be used in a commercial application (e.g. ImageNet21K).

Reviewer comment (Writing):
The clarity is bad in several aspects. **1)** The memory learning in eqs. 2-4 involves ground truth $y_k$ that could be either $0$ for anomaly or $1$ for normal sample, but in sec. 3.2 there is a one-class AD objective, i.e., eq. 6, where no training anomaly examples are supposed to be available, so I'm confused that how the one-class AD model can be trained together with eqs. 2-4. **2)** The concept of outlier exposure (OE) is defined and used in OOD detection for using external datasets as pseudo OOD examples to train OOD detection models, but in this work it seems it treats real anomaly examples as OE examples. This is very confusing. Rather than using the so-called AD-OE concept, it may be clearer to use semi-supervised AD or open-set AD as in *"Deep semi-supervised anomaly detection. arXiv preprint arXiv:1906.02694."*, *"Catching both gray and black swans: Open-set supervised anomaly detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 7388-7398)."*, or *"Ubnormal: New benchmark for supervised open-set video anomaly detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 20143-20153)."* **3)** It is also unclear why we need a method that works well for both anomaly detection and OOD detection, given the fact that the two tasks have quite different application settings.

Author response:
**Reply:**
First, we note that in the first learning stage (Fig. 2a), anomalies are only used to further enrich the contrastive learning scheme (learning the representation z) but not for the memory learning (Mem). Indeed, at any time only normal prototypes are learned.

The provided equations 2-4 present our framework of representation learning for unified anomaly detection, meaning it can work with or without (special case where =0) anomalies. Nothing circumvents the omission of anomalous samples in the first stage since it is driven by **unsupervised** contrastive learning. We can notice that the only use of the label y_k during the first stage is inside the memory layer Mem(x,y) that act as a pass-through if y=0, i.e. an anomaly.

We made this point more clear in Sec.3.1 of the revised paper.

Reviewer comment (Experiments):
Following up the above point, since anomaly examples are used in the training stage, recent SOTA semi-supervised/open-set anomaly detection methods should be used in the experiment comparison to justify the advantages the work has, e.g., see *"Catching both gray and black swans: Open-set supervised anomaly detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 7388-7398)."* for some of such methods.

Author response:
**Reply:**
First, as mentioned earlier, our setting is different from your mentioned ones (i.e., not open-set supervised). Second, several recent methods including your mentioned model aims to solve anomaly localization rather than object-/semantic- level anomaly detection as in our work. 
Finally, we note that the datasets considered in those anomaly localization papers are generally much smaller than our considered datasets (~5K images compared to ~100K with our datasets).

Reviewer comment (Experiments):
Experiments on large-scale high-resolution image datasets, e.g., the popular setting that uses ImageNet-1k as the ID dataset, are missing for the OOD detection task.

Author response:
**Reply:**
Thank you for your question. 

To the best of our knowledge, ImageNet-1k has not been utilized as the ID dataset in popular detectors, possibly due to its size, which may render it impractical for such purposes. For example, detectors like CSI [1] and MSC [2] have presented their results using CIFAR-10 as the ID dataset. CSI used a significantly smaller subset of ImageNet-1K, namely ImageNet-30, as their ID dataset. It is important to note that in the original paper, CSI did not provide results on CUB, and our results outperform theirs by nearly 30 points (the results of CSI on CUB comes from [5]).

We would like to emphasize that we have already presented high-quality results for anomaly detection (in two settings), out-of-distribution detection, and fine-grained anomaly detection. We have chosen to leave the evaluation on other datasets/tasks for future work, as indicated in the paper.