Paper: MuSc: Zero-Shot Industrial Anomaly Classification and Segmentation with Mutual Scoring of the Unlabeled Images (ICLR-2024)

Reviewer comment (Evaluation):
This paper employs a methodology that utilizes unlabeled test images to collectively measure anomaly scores, differing somewhat from the traditional zero-shot recognition setting. In the conventional approach, each image is independently evaluated, such as WinCLIP. This difference in evaluation methodologies can result in a somewhat unfair comparison.

Author response:
We agree that our approach is different from several existing approaches, e.g., winCLIP, in using the unlabeled test images, which may lead to a potentially unfair comparison. Therefore we have added the comparison to ACR [b]\[c] in Tab.1.  Similar to our approach, ACR assumes access to a set of unlabeled test images together during inference, the experimental results demonstrate that MuSc outperforms ACR by a large margin. We hope such a comparison is more fair.

Reviewer comment (Novelty):
The underlying concept is reminiscent of [a], which presumes homogeneous input texture and identifies image regions that disrupt this homogeneity as anomalies. In other words, if a patch significantly differs from its neighboring areas, it's deemed an anomaly.

Author response:
Thanks for your valuable suggestion! We have carefully studied [a] and added the discussion in the Related Work section. [a] is based on the assumption that the input texture is homogeneous, and the image regions that break the homogeneity are detected as anomalies. [a] explores the relationship between the patches inside one test texture image, while our method leverages the relationship in a set of unlabeled images.

Reviewer comment (Experiments):
The method heavily relies on many heuristic design choices. It consists of a three-step pipeline which depends on many hyperparameters for feature representation, mutual scoring estimation, and classification rescoring. The paper poorly presents sensitivity of performance to optimizing all these parameters.

Author response:
A2: Our method contains three steps, and each step has only one hyperparameter, i.e., the aggregation degree $r$ in LNAMD, Interval Average (IA) on the minimum $X\%$, and window mask size in RsCIN. 

The setting of aggregation degree $r$ indeed impacts the result as it is related to the size of anomalies. As shown in Tab. 3, in multiple settings, the model's classification AUROC on the MVTec AD dataset is reduced by up to 3.6%, and segmentation AUROC is reduced by up to 3%. On the VisA dataset, the use of $r$={5} results in a 10.4% reduction in classification results compared to $r$={1,3,5}, due to more small defects on VisA, making it unsuitable to use only the large aggregation degree. For better practicality, we use the same setting {1,3,5} in all the datasets to obtain a good segmentation of anomalies with various sizes. 

The $X\%$ in Interval Average is an insensitive hyperparameter. As shown in Fig. 7, we compare the models using different percentage ranges of the Interval Average in the MSM module. The change of this parameter has a maximum effect of 3% on the image-AUROC and 1% on the pixel-AUROC.

The window mask size in RsCIN is also an insensitive hyperparameter, which is proved by the experiments in Fig. 8. Observably, on the MVTec AD data set, F1-max-score fluctuates between 97.1 and 97.5, with the amplitude not exceeding 0.4. On the VisA data set, F1-max-score floats between 87.8 and 89.5, and the amplitude does not exceed 1.7. Furthermore, we also verify the sensitivity of the RsCIN module to different methods, as shown in Tab. 11 of the appendix. All methods maintain the same parameter settings of the RsCIN module as ours. Almost all methods achieve performance improvements after applying the RsCIN module, except DRAEM only. This experiment proves that the window mask size in RsCIN is insensitive for different methods.

In summary, the aggregation degree $r$ is a sensitive parameter, while the other two hyperparameters are insensitive.

In addition, we show the performance of our method on the BTAD dataset, and the parameter settings are consistent with those in our experiments. Specific settings include aggregation degree $r$ as {1, 3, 5},  $X$ as 30, and window mask size as {2, 3}. The following table shows that the classification and segmentation performance of our method on the BTAD dataset still exceeds that of zero/few-shot methods and even some full-shot methods. Such an experimental result shows that although the performance of MuSc is sensitive to the setting of aggregation degree $r$, an appropriate choice of $r$ can achieve good results in different datasets simultaneously, which proves its universality.

|   Method    |  Setting  | Image-AUROC | Pixel-AUROC |
| :---------: | :-------: | :---------: | :---------: |
|   VT-ADL    | full-shot |    83.7     |    90.0     |
|   P-SVDD    | full-shot |    83.3     |    92.1     |
|    SPADE    | full-shot |    87.6     |    96.9     |
|    PaDiM    | full-shot |    93.7     |    97.3     |
| PyramidFlow | full-shot |    95.8     |    97.7     |
|  PatchCore  |  4-shot   |    91.4     |    96.3     |
|    RegAD    |  4-shot   |    91.0     |    97.5     |
|  APRIL-GAN  |  4-shot   |    91.3     |    92.4     |
|  APRIL-GAN  |  0-shot   |    72.3     |    89.6     |
| MuSc(ours)  |  0-shot   |    94.8     |    97.3     |

Reviewer comment (Novelty):
Overall technical novelty seems incremental, since the method incorporates well-established multiscale features (Sec 3.1), norm-2 distance (Sec 3.2), and clustering (Sec 3.3).

Author response:
A3: Thanks for your comment! We clarify our technique novelty as follows.

Firstly, we find that the patch-level features extracted by ViT have the limitation of detecting industrial anomalies of varying sizes. Swin transformer has large patches in some stages leading to the patch-level features extracted by it not working well. The proposed LNAMD module addresses this issue in ViT, as far as we know, such a design is a new technique to optimize the patch-level features of ViT in industrial vision.

Secondly, in MSM, the mutual scoring operation in Eq. (1) is inspired by the observation of normal and abnormal patches. It uses the minimum similarity between the patch $m$ of image $I_i$ and all the patches of image $I_j$ to score the patch $m$ of image $I_i$. This is a new technique to score the patch by unlabeled images. The interval average operation in Eq. (2) is based on our observation of the false positives, so we designed this operation to alleviate these in scoring. To the best of our knowledge, the Mutual Scoring Mechanism is a new mechanism for the zero-shot AC/AS community.

Thirdly, in RsCIN, we observe that due to the high smoothness of $\textbf{C}$, the traditional manifold learning approach may be influenced by a large number of images. Therefore, we propose the Multi-window Mask Operation (MMO) to constrain the image number, which makes each image only influenced by a small number of neighborhood images. RsCIN was first proposed in the field of anomaly classification and segmentation. Meanwhile, we experimentally proved that RsCIN can be further employed to improve the AC accuracy of existing approaches in Appendix A.2.4.

Reviewer comment (Experiments):
Since the method lacks a training phase and directly generates results based on estimating the test-set statistics, inference is likely to be much slower than in previous approaches. I was not able to find a report/comparison of the inference times.

Author response:
A4: Thanks for your insightful comment. Driven by your suggestion, we have examined the MuSc's code implementation thoroughly and found two issues that slowed down the model and increased the number of parameters. For now, both issues have been solved. After the paper is accepted, we will release the source code to prove that both issues are indeed addressed and the parameter amount and speed are optimized. We describe these two issues and the corresponding countermeasures in the response to Q1 of reviewer NLA8.

After modifying the implementation of LNAMD, we have conducted a **comprehensive efficiency comparison**, which is listed in the following table. Since there is no official source code of WinCLIP, its GPU cost is unknown (marked by “-”). In addition, the author of APRIL-GAN does not provide the pre-trained model using ViT-B-16-plus-240 (our backbone), so its AUROC and AUPRO are unknown (marked by “-”).

|      Method       |     Backbone      | Training | Inference time (ms) | GPU (MB) | Image-AUROC | AUPRO |
| :---------------: | :---------------: | :------: | :-----------------: | :------: | :---------: | :---: |
|   RegAD(4-shot)   |     ResNet18      |   yes    |        309.4        |   8920   |    89.1     | 88.0  |
| APRIL-GAN(0-shot) |   ViT-L-14-336    |   yes    |        100.2        |   4996   |    86.1     | 44.0  |
|    MuSc($s=1$)    |   ViT-L-14-336    |    no    |        998.8        |   7168   |    97.8     | 93.8  |
|    MuSc($s=2$)    |   ViT-L-14-336    |    no    |        605.8        |   5666   |    97.1     | 93.8  |
|    MuSc($s=3$)    |   ViT-L-14-336    |    no    |        513.5        |   5026   |    96.7     | 93.7  |
| APRIL-GAN(0-shot) | ViT-B-16-plus-240 |   yes    |        64.7         |   3226   |      -      |   -   |
|  WinCLIP(0-shot)  | ViT-B-16-plus-240 |    no    |         389         |    -     |    91.8     | 64.6  |
|    MuSc($s=1$)    | ViT-B-16-plus-240 |    no    |        455.3        |   3002   |    95.4     | 91.9  |
|    MuSc($s=2$)    | ViT-B-16-plus-240 |    no    |        285.7        |   2740   |    95.4     | 91.5  |
|    MuSc($s=3$)    | ViT-B-16-plus-240 |    no    |        238.3        |   2648   |    95.2     | 91.4  |

In terms of inference time, when using the same backbone as WinCLIP and setting $s$ to 3, our inference time is 238.3ms per image, which is 150.7ms shorter than WinCLIP, and 71.1ms shorter than RegAD. Additionally, APRIL-GAN has the shortest inference time (64.7ms) among these methods. In terms of memory cost, when using the same backbone as WinCLIP, our GPU memory cost is 3002MB, significantly lower than RegAD (8920MB) and 224MB less than APRIL-GAN (3226MB). The detailed comparisons of inference time and memory cost have been added to Appendix A.4.

We will continue to speed up MuSc and reduce its memory cost in our future works.

Reviewer comment (Experiments):
Could your approach handle a setting when a test dataset lacks ground truth? What is performance when a test dataset lacks ground truth for optimizing the proposed heuristic procedure?

Author response:
A5: Thanks for your comment. As we mentioned in the response to Q2 of reviewer dCfU, our approach contains three hyperparameters. Our performance is sensitive to the aggregation degree $r$ but insensitive to the other two hyperparameters. In addition, the experimental results on the new BTAD dataset demonstrate that the fixed hyperparameter setting could achieve good experimental results in different datasets. Therefore, we do not need to use the ground truth to optimize our approach. It is clarified that all the steps of our approach do not use the ground truth actually.

Reviewer comment (Experiments):
One of the claimed contributions, the mutual scoring mechanism, appears to depend significantly on the relative positions of feature patches. This limits the application of this approach in real settings where orientations and scales are not consistent. What is performance when test images exhibit inconsistent orientations or scales?

Author response:
A6: Thanks for your comment! In the mutual scoring mechanism, we leverage each image in {$D_{test}$ \\ $I_i$} to assign an anomaly score for each aggregated patch token of a test image $I_i$ in stage one as follows，

$$
a_{i,l}^{m,r}(I_j) = \\min_{n} \\Vert \\hat{p}_{i,l}^{m,r}-\\hat{p}^{n,r}\_{j,l} \\Vert_2
$$

where $m$ and $n$ are the index of the image patches, $r$ is the aggregation degree, $i$ and $j$ are the index of the unlabeled images, and $l$ indicates the stage $l$ of ViT. We clarify that $m$ is irrelevant to the position of $n$,  since Eq.1 first computes the distance between the  $m$-th patch (in image $I_i$) and all the patches in image $I_j$, and then performs min operation to select the most similar image patch from image  $I_j$. Accordingly, the anomaly score only depends on the aggregated patch token and is irrelevant to the relative positions of feature patches. 

In addition, according to your insightful comment, we select the categories with inconsistent orientations or scales from MVTec AD and VisA datasets for the additional experimental evaluation, which is given in Tab. 12 in the paper and also given in the following table. We compare our method with WinCLIP and APRIL-GAN and show the image-AUROC (AC)/pixel-AUROC (AS) in this table. We observe that our approach and WinCLIP are both influenced by inconsistent orientations or scales. The reason is that both of us use the fixed pre-trained vision transformer as the feature extractor. As introduced in [c], during the pre-training process, the vision transformer only performs minor data augmentation on orientations and scales, and hence both of us cannot well address the inconsistent orientations or scales. APRIL-GAN also achieves a decreased AC score, while it obtains a better AS score. We guess that it is due to its additional training set for optimizing AS. In addition, we should emphasize that our approach still outperforms WinCLIP and APRIL-GAN in inconsistent orientations or scale cases.

In our future works, we plan to design rotation and scale invariance features to alleviate such a limitation.

| Category  |    WinCLIP    |   APRIL-GAN   |  MuSc(ours)   |
| :-------: | :-----------: | :-----------: | :-----------: |
|           |     AC/AS     |     AC/AS     |     AC/AS     |
|   screw   |   83.3/89.6   |   84.9/97.8   |   83.5/98.9   |
| hazelnut  |   93.9/94.3   |   89.6/96.1   |   99.6/99.4   |
| metal_nut |   97.1/61.0   |   68.4/65.4   |   96.3/86.0   |
| capsules  |   85.0/81.6   |   61.2/97.5   |   88.8/98.8   |
| macaroni2 |   63.7/59.3   |   64.6/97.8   |   69.9/97.2   |
| **mean**  | **84.6/77.2** | **73.7/90.9** | **87.6/96.1** |
| mean-ALL  |   91.8/85.1   |   86.1/87.6   |   97.8/97.3   |

Reviewer comment (Experiments):
Could your method be extended to (or readily address) the few-shot setting?

Author response:
A7: Thanks for your valuable comment. We provided two simple few-shot setting extensions based on MuSc in Appendix A.6. In the first extension, we added a few normal reference images directly to the test images that assign scores to image $I_i$ in the MSM module. In the second extension method, we designed additional scoring strategies for these normal reference images. In detail, we modified the original Eq. 2 to replace the interval average operation with the minimum operation. Since our zero-shot method uses rich implicit normal prior information in the unlabeled dataset, adding a few reference images does not greatly improve the results. The results of the two extended few-shot settings are in Tab. 14 and Tab. 15. In addition, as shown in Tab. 1, our zero-shot approach outperforms several few-shot approaches.

Reviewer comment (Experiments):
Technical soundness: The method relies heavily on availability of a set of images in order to compute the anomaly scores. I dont think this emulates the practical scenario of anomaly detection in industry. The more realistic scenario is a method is required to classify and segment the anomaly given one image. Using test set images to produce the output on the same set of images also does not conform with the scientific procedure. I dont think this is the definition of a zero shot method either. The setting sounds completely unreasonable to me.

Author response:
**Q1: The method relies heavily on availability of a set of images in order to compute the anomaly scores. I dont think this emulates the practical scenario of anomaly detection in industry.**

A1: Thanks for your comment. Since we have long-term collaboration with several well-known companies, our approach is designed according to the practical industry requirements.  Let us take a famous anonymous company as an example. In general, a production line (AMOLED/LCD) has a daily output of 1500 pieces. Typically, a production line is equipped with at least 3 parallel quality inspection equipment, and each furnished with at least 2 cameras. The image resolution of each camera exceeds 4000×5000, and a typical solution is to crop a series of 512×512 images from high-resolution images by sliding windows with 10% overlap. The above production line needs to inspect nearly **a million product images per day**. As far as we know, such a production quality inspection requirement exists in a large number of companies. This example illustrates that unlabeled images are abundant in real industrial scenarios and easy to collect.

In addition, it needs to be clarified that MuSc did not rely heavily on a large number of unlabeled images, which can be demonstrated by Tab. 7 of the paper. MuSc obtains decent AUROCs under the test sets of different amounts, even when the test set only has 14-56 images. Referring to the example above, we are confident that such a number of images can be easily achieved in practical scenarios. 

**Q2：The more realistic scenario is a method is required to classify and segment the anomaly given one image. Using test set images to produce the output on the same set of images also does not conform with the scientific procedure. I dont think this is the definition of a zero shot method either.**

A2: Thanks for your comment. We agree that it is a widely-used practice to classify and segment anomalies in one image. However, this does not negate the scientific validity of using a set of test images to produce the output on the same set of images. As mentioned by Reviewer H5pV, our approach aligns closely with ACR [b]\[c], an excellent zero-shot AD approach published in NeurIPS2023. Both of us jointly measure the anomaly scores across either the entire test dataset or a sub-set of unlabeled test images. Therefore, we hope the comparison with ACR makes our approach more fair and dispels your doubts.

Reviewer comment (Novelty):
Contribution: The algorithm appears to be a set of heuristic tricks applied in a sequence. There is not a solid technique that is novel, elegant, theoretically justified and of broad interest.

Author response:
**Q3: The algorithm appears to be a set of heuristic tricks applied in a sequence. There is not a solid technique that is novel, elegant, theoretically justified and of broad interest.**

A3: Thanks for your comment! We clarify our technique novel as follows.

Firstly, we find that the patch-level features extracted by ViT have the limitation of detecting industrial anomalies of varying sizes. Swin transformer has large patches in some stages leading to the patch-level features extracted by it not working well. The proposed LNAMD module addresses this issue in ViT, as far as we know, such a design is a new technique to optimize the patch-level features of ViT in industrial vision.

Secondly, in MSM, the mutual scoring operation in Eq. (1) is inspired by the observation of normal and abnormal patches. It uses the minimum similarity between the patch $m$ of image $I_i$ and all the patches of image $I_j$ to score the patch $m$ of image $I_i$. This is a new technique to score the patch by unlabeled images. The interval average operation in Eq. (2) is based on our observation of the false positives, so we designed this operation to reduce the false positives. To the best of our knowledge, the Mutual Scoring Mechanism is a new mechanism for the zero-shot AC/AS community.

Thirdly, in RsCIN, we observe that due to the high smoothness of $\textbf{C}$, the traditional manifold learning approach may be influenced by a large number of images. Therefore, we propose the Multi-window Mask Operation (MMO) to constrain the image number, which makes each image only influenced by a small number of neighborhood images. RsCIN was first proposed in the field of anomaly classification and segmentation. Meanwhile, we experimentally proved that RsCIN can be further employed to improve the AC accuracy of existing approaches in Appendix A.2.4.

Reviewer comment (Writing):
Clarity: None of the techniques were adequately explained as to why they are being performed and why it makes sense (intuitively or conceptually) to apply them. Just stating the process does not qualify as a good scientific exposition.

Author response:
**Q4: None of the techniques were adequately explained as to why they are being performed and why it makes sense (intuitively or conceptually) to apply them. Just stating the process does not qualify as a good scientific exposition.**

A4: Thanks for your comment, we clarify the motivations as follows.

The LNAMD module is motivated by the observation that *the large aggregation degree $r​$ is suitable for detecting large anomaly regions, and a small aggregation degree $r​$ is suitable for detecting small abnormal regions*, which are described in the second paragraph of Section 3.1.

The MSM contains three steps, and the motivations are given as follows. The motivation of the first step is that *the normal patch in image $I_i$ can find similar patches in most other images, while the abnormal patch is difficult to find similar patches* (line 9 to line 13 in the first paragraph of Section 3.2). The second step is inspired by our observation of the anomaly scores histogram. We find that *some normal patches with appearance variations of different normal images may cause false positives*, so we take the Interval Average operation to reduce them in scoring (line 2 to line 6 in the second paragraph). The third step, which aims to *achieve the fine-grained anomaly score for each image patch*, is described in the first line of the third paragraph.

For the motivation of RsCIN, as we described in the first paragraph of Section 3.3. We observed that *images similar to normal images have smaller classification anomaly scores, while those similar to abnormal images have larger classification anomaly scores*, which inspired us to use these similar images to optimize classification scores and design this module.