Paper: AnomalyCLIP: Object-agnostic Prompt Learning for Zero-shot Anomaly Detection (ICLR-2024)

Reviewer comment (Evaluation):
Little/no discussion about the additional cost of training

Author response:
We have supplemented the analysis about the additional cost of training by measuring the time taken during training (training time per epoch) in Table 8 of Appendix D. For a fair comparison, all experiments are conducted in a single NVIDIA RTX 3090 24GB GPU, and the GPU is kept free before evaluation. In Table 8, AnomalyCLIP takes 13.71 min per epoch on MVTec AD (The total number of samples is 1725) and only requires a total of 15 epochs for the whole fine-tuning. Once AnomalyCLIP finishes fine-tuning, AnomalyCLIP can be applied to different datasets and domains without additional training. We also compare AnomalyCLIP with other baselines that need auxiliary data (i.e., CoOp and VAND). For example, the minimum training time per epoch is 12.25 min of CoOp, and hence, AnomalyCLIP has a similar training computation cost as CoOp.

Reviewer comment (Experiments):
No comparison of different ViT backbones of CLIP

Author response:
We have added the backbone ablation study in Table 13 in Appendix D. We originally used ViT-L-14 as the backbones of CLIP, which is a representative backbone in CLIP. To fully explore the effect of different sizes of backbones, we use ViT-B-16-plus and ViT-H-14, which is a smaller and larger model than ViT-L-14, respectively. Compared to pixel-level performance 90.5% AUROC and 90.8% AUROC from ViT-B-16-plus and ViT-H-14, ViT-L-14 achieves the best performance 91.1% AUROC. The results indicate that AnomalyCLIP maintains promising performance across various ViT backbones. Meanwhile, it also reveals that increasing the model size does not necessarily result in improved performance. The larger model may introduce a strong bias towards object semantics, hindering AnomalyCLIP's ability to learn object-agnostic text prompts capturing generic normality and abnormality in an image. However, a smaller backbone like ViT-B-16-plus could cause AnomalyCLIP to struggle with generalization to unseen anomalies.

Reviewer comment (Theory):
It would be better to provide more intuition why the Focal and Dice loss are necessary

Author response:
We thank the reviewer for pointing this out. Although local features are refined by DPAM, there still exists misalignment among textual semantic and local visual semantics, especially for unseen objects. Focal and Dice loss play a crucial role in optimizing local context. They are introduced to empower object-agnostic text prompts to focus on fine-grained, local abnormal regions from intermediate layers of the visual encoder. As mentioned in Section 3.2, Focal loss addresses the imbalance between anomaly and normal pixels, typically caused by the smaller size of anomalous regions. Meanwhile, Dice loss aims to precisely constrain the anomaly boundary by measuring the overlap between the predicted segmentation ($S_n$/$S_a$) and the ground truth mask. To provide a more comprehensive analysis, we have included an ablation study on Focal and Dice loss in Table 14 of Appendix D. Compared to scenarios without local context optimization, Dice loss improves the pixel-level and image-level performance from 80.3%AUROC to 87.2%AUROC and 86.6%AUROC to 90.1%AUROC in pixel level on MVTec AD and VisA. Focal loss also brings the performance gain of 10.3%AUROC and 8.3%AUROC. Combining Focal and Dice loss, AnomalyCLIP achieves the best results (i.e., 91.1%AUROC and 95.5%AUROC). Note that the global context optimization is always used during the ablation, since we need at least one loss function to drive the optimization.

Reviewer comment (Presentation):
Figure 2 caption should be more explanatory

Author response:
Thank you for your effort on helping improve the paper. We have added more illustrations in the revised version: "To adapt CLIP to ZSAD, AnomalyCLIP introduces object-agnostic text prompt templates to capture generic normality and abnormality regardless of the object semantics. Then, we introduce glocal context optimization to incorporate global and fine-grained anomaly semantics into object-agnostic textual embedding learning. Finally, textual prompt tuning and DPAM are used to support the learning in the textual and local visual spaces of CLIP.\"

Reviewer comment (Evaluation):
Besides all of the other modifications such as DPAM, why are object-agnostic prompts better than object-aware prompts for a specific object class?

Author response:
Thank you for raising this question. The object semantics in the object-aware text prompt can be regarded as the condition information. During fine-tuning, anomalies across different objects are used to ensure that the prompt captures generic anomaly patterns. In such a process, the object-aware prompt actually learns the condition information of specific objects, hindering the prompt from learning the generic abnormality for unseen objects. Different from the object-aware text prompt, the object-agnostic text prompt excludes the object semantics from text prompt templates, allowing learnable text prompt templates to focus on capturing the characteristics of anomalies themselves, rather than the objects. Therefore, when detecting anomalies in unseen objects, the object-agnostic text prompt can better capture the underlying anomaly patterns than the object-aware text prompt.

Reviewer comment (Reproducibility):
Is it true that hidden layer visual embeddings are only used for the segmentation map, while the final output embedding is only used for the image-level score?

Author response:
Thank you for your detailed feedback. Besides the intermediate local visual embeddings used for segmentation, we also include the local visual embeddings of the final output. As for image classification, we only use the global visual embedding of the final output for image-level score. For clear clarification, the $k$-th visual representation of the visual encoder is denoted as $F_k\in R ^ {(1 + H\times W) \times D}$, where the class token $f_k \in R ^ D$ is treated as its global visual embedding, and patch tokens $f_k^m \in R ^ {H\times W\times D}$ are referred to as local visual embeddings. In AnomalyCLIP, $k$ is set to {6, 12, 18, 24}, and the 24-th layer is the top layer of the visual encoder. To combine more local visual semantics, we integrate $f_{6}^m$, $f_{12}^m$, $f_{18}^m$, and $f_{24}^m$ to perform anomaly segmentation. Since anomaly classification requires accurate global representation, we only use the class token of the top layer, $f_{24}$, to perform anomaly classification. We hope this helps clarify the confusion.

Reviewer comment (Experiments):
The motivation behind DPAM appears to be relatively weak. It is not entirely clear why employing V-V self-attention would preserve more localized information compared to the original Q-K attention. Relying solely on a single instance shown in Figure 3 may not furnish sufficient evidence to support this assertion. Furthermore, the rationale for treating these features separately remains unclear. What might be the outcome if we were to combine Q-Q, K-K, V-V, and Q-K features into an ensemble?

Author response:
**Q1: The motivation behind DPAM appears to be relatively weak. It is not entirely clear why employing V-V self-attention would preserve more localized information compared to the original Q-K attention.**

Thank you for raising this concern. Please refer to our **General Response** above for an intuitive explanation of DPAM strategy from the attention mechanism.

**Q2: Relying solely on a single instance shown in Figure 3 may not furnish sufficient evidence to support this assertion.**

We have added experiments to study the effect of DPAM in Table 7 of Appendix D. Without our proposed DPAM module, AnomalyCLIP shows a decrease from 91.1% AUROC to 87.9% AUROC in pixel-level performance and from 91.5% AUROC to 80.7% AUROC in image-level performance on MVTec AD. Additionally, there is a decrease from 95.5% AUROC to 91.9% AUROC in pixel-level performance and from 82.1% AUROC to 73.0% AUROC in image-level performance on VisA. The results demonstrate the superiority of $V$-$V$ attention over $Q$-$K$ attention. However, the performance decreases at the image level is more pronounced than that at the pixel level. This discrepancy is attributed to the fact that the total loss places greater emphasis on local context optimization, driven by a larger local loss compared to the case with DPAM. As a result, AnomalyCLIP lacks sufficient optimization for global alignment, leading to a significant decline in image-level performance. Note that the original CLIP (no DPAM) exploits $Q$-$K$ features and suffers from weak segmentation performance.

**Q3: What might be the outcome if we were to combine Q-Q, K-K, V-V, and Q-K features into an ensemble?**

Thank you for your helpful comments. Ensembling different features is an excellent idea. We have added two experiments to explore two ensemble methods, namely AnomalyCLIP$\_{ensemble1}$ and AnomalyCLIP$\_{ensemble2}$, involving the ensemble of **Q-Q, K-K, and V-V** and the ensemble of **Q-K, Q-Q, K-K, and V-V**, respectively in Table 12 in Appendix D. We average the logit output of different features for the ensemble. In Table 12, AnomalyCLIP$\_{ensemble1}$ shows performance improvement by leveraging the advantages of three DPAM features. However, while AnomalyCLIP$\_{ensemble2}$ outperforms the $Q$-$K$ feature version, it experiences a performance decrease compared to $V$-$V$ from 91.1%AUROC to 90.7%AUROC in pixel level and 91.5%AUROC to 90.8%AUROC in image level on MVTec AD. There is also a decline from 95.5%AUROC to 94.9%AUROC in pixel level and 82.1%AUROC to 80.7%AUROC in image level on VisA. The decline in performance upon adding $Q$-$K$ features to AnomalyCLIP$_{ensemble1}$ suggests that the $Q$-$K$ feature fails to provide valid local visual semantics to facilitate ZSAD. Note that the original CLIP exploits $Q$-$K$ features and gets weak segmentation performance. The good pixel-level performance of $Q$-$K$ in AnomalyCLIP is attributed to local optimization, where the object-agnostic prompt helps alleviate the disrupted local visual semantics of $Q$-$K$.

Reviewer comment (Experiments):
The ablation studies lack clarity regarding the specific details of each component, making it challenging to assess the actual contributions of the proposed method. The experiments in Table 4 involve the incremental addition of modules (T1-T4), but a more informative approach would be to examine the performance when each of them is removed. For instance, it is unclear how DPAM (T1) performs when used in conjunction with both T2, T3, and T4. An important setting should be removing T1 but keeping T2-T4. Additionally, the increase in AUC from 47.9 to 54.8 by introducing T1 does not signify a significant improvement (both values are close to random guessing) for a two-class classification problem.

Author response:
**Q4: A more informative approach would be to examine the performance when each of them is removed.**

Thank you for detailed feedback. To demonstrate the effectiveness of each module we propose, we have supplemented the additional module ablation in Table 7 of Appendix D. We test the contribution of one module by removing one module and maintaining the rest module.

1. The effectiveness of DPAM ($T_1$). When we remove DPAM, the results show a decrease from 91.1% AUROC to 87.9% AUROC in pixel-level performance and from 91.5% AUROC to 80.7% AUROC in image-level performance. This performance decline indicates the importance of DPAM, which enhances local visual semantics by modifying the attention mechanism. However, the decrease in performance at the image level is more pronounced than that at the pixel level. This discrepancy is attributed to the fact that the total loss places greater emphasis on local context optimization, driven by a larger local loss compared to the case with DPAM.
2. The effectiveness of object-agnostic prompt learning ($T_2$). Excluding object-agnostic prompt learning makes AnomalyCLIP suffer from the huge performance gap (i.e., 91.1% AUROC to 84.3% AUROC in pixel-level and 91.5% AUROC to 65.6% AUROC in image-level). This performance decline illustrates that the object-agnostic text prompt template plays a significant role in improving the performance of AnomalyCLIP at both pixel and image levels.
3. The effectiveness of textual prompt tuning ($T_3$). When removing textual prompt tuning, the performance of AnomalyCLIP declines from 91.1% AUROC to 90.6% AUROC in pixel-level performance and from 91.5% AUROC to 90.4% AUROC in image-level performance. This demonstrates the importance of adapting original textual space by adding learnable textual tokens in the text encoder.
4. The effectiveness of the integration of multi-layer local visual semantics ($T_4$). When removing multi-layer local visual semantics, the outcomes reveal a decrease from 91.1% AUROC to 90.0% AUROC in pixel-level performance and from 91.5% AUROC to 91.0% AUROC in image-level performance. This performance decline indicates the importance of incorporating multi-layer local visual semantics.

Reviewer comment (Experiments):
In Table 5, it remains unclear how the models are optimized when both the global and local losses are absent.

Author response:
**Q5: In Table 5, it remains unclear how the models are optimized when both the global and local losses are absent.**

Thank you for pointing this out. When both the global and local losses are absent, the model is not optimized. Therefore, we report the results of the base method, where there is no $T_1$, $T_2$, $T_3$, and $T_4$. However, it would be better to contain $T_1$ and $T_4$ for appropriate analysis. This is because $T_1$ and $T_4$ do not require optimization, while $T_2$ and $T_3$ involve the optimization process. We have updated the results in the revised paper.

Reviewer comment (Theory):
My largest worry is the paper relies on an optimistic assumption of abnormality, namely, the abnormality is object-agnostic. The assumption might hold in some cases. The learned model is able to detect the abnormality that has similar abnormal patterns to the features in the auxiliary training set, e.g., detecting the defects of a manufactured part. However, the assumption can also fail in some cases. For example, the model may misclassify a product with texture patterns similar to scratches as an abnormal one, even though the scratch-like textures are normal texture patterns of the product. It would be better to explicitly discuss the successful cases and the potential failures of the proposed method.

Author response:
Thank you for raising this concern. We agree that in a minority of cases, AnomalyCLIP exhibits false detection due to the object-agnostic assumption. However, ZSAD requires detection models to detect anomalies without any training sample in a target dataset, and capturing the object-agnostic anomaly patterns is a promising solution because anomalies from different application scenarios typically have substantial variations in their visual appearance, foreground objects, and background features. As suggested by the reviewer, we have supplemented the experiment to explicitly discuss the successful cases and the potential failures in Figure 7 in Appendix D. In Figure 7(a), AnomalyCLIP accurately detects scratch-like patterns on the product, even when they typically appear in the texture. However, false detection occurs when scratch-like patterns are situated in the background, as depicted in Figure 7(b). Meanwhile, we also show the color stain patterns. As shown in Figure 7(c), AnomalyCLIP successfully detects the color stain, which exhibits subtle visual differences from the detected entities. On the other hand, AnomalyCLIP may face challenges when the normal region displays patterns that are almost indistinguishable to the naked eye from anomalies. For instance, in skin cancer detection, the normal regions are falsely detected as anomalies are visually similar to the disease region in Figure 7(d). Also, the stain interference in the background is a problem. These failure cases illustrate that the importance of mitigating background interference and achieving fine-grained discrimination, especially in cases of visually similar abnormalities. Exploring these challenges for enhancing ZSAD is a valuable direction for future research.

Reviewer comment (Writing):
Minor comment: $S_n$ and $S_a$ are first-time used in $L_{local}$ on page 5, but are clearly defined on page 6. Having the definitions and equations of $S_n$ and $S_a$ before $L_{local}$ would be good.

Author response:
Thank you for your effort on helping improve our paper. We have updated the definitions of $S_n$ and $S_a$ before $L_{local}$.

Reviewer comment (Novelty):
The discussion of some related work about zero-shot anomaly detection, e.g., [1,2,3], is missing.

Author response:
We thank the reviewer for pointing us towards these related works. We have updated our related work section and added the works suggested by the reviewer. CLIP-AD \[1\] and ZOC \[2\] are early works in the utilization of CLIP for anomaly/out-of-distribution detection task, and they primarily focus on anomaly classification tasks, whereas AnomalyCLIP jointly optimizes anomaly detection and segmentation tasks, enabling both tasks within a single forward pass during inference. Despite ACR \[3\] addressing detection and segmentation tasks, it requires distinct auxiliary data for specific zero-shot detection tasks. AnomalyCLIP can adapt to various datasets and domains for zero-shot detection by learning object-agnostic prompts only once.  

>**References:**  
>1. Philipp Liznerski, Lukas Ruff, Robert A Vandermeulen, Billy Joe Franks, Klaus Robert Muller, and Marius Kloft. Exposing outlier exposure: What can be learned from few, one, and zero outlier images. Transactions on Machine Learning Research, 2022.  
>2. Sepideh Esmaeilpour, Bing Liu, Eric Robertson, and Lei Shu. Zero-shot out-of-distribution detection based on the pretrained model clip. In Proceedings of the AAAI conference on artificial intelligence, 2022  
>3. Aodong Li, Chen Qiu, Marius Kloft, Padhraic Smyth, Maja Rudolph, Stephan Mandt. Zero-Shot Batch-Level Anomaly Detection. Thirty-seventh Conference on Neural Information Processing Systems, 2023.

Reviewer comment (Experiments):
Why is learning two text prompts necessary? Do you have comparisons to $g_n = [V_1][V_2]...[V_E][object]$ and $g_a = [V_1][V_2]...[V_E][damaged][object]$?

Author response:
We have supplemented experiments to investigate the effect of shared prompts in Table 11 in Appendix D. When sharing the learnable word embeddings of $g_n$ and $g_a$, AnomalyCLIP achieves 90.5% AUROC in pixel level and 90.9% in image-level on MVTec AD and 95.0% AUROC in pixel level and 81.5% AUROC in image level on VisA. The results show that AnomalyCLIP without sharing also works well for ZSAD and the efficiency of our object-agnostic prompt learning. However, the shared prompt performs slightly worse than the unshared prompts (used in the original paper). The performance decrease is 0.6%AUROC and 0.6%AUROC in image level on MVTec AD and Visa, and 0.5%AUROC and 0.6%AUROC in pixel level. We believe that the separate learning for these two prompts helps discriminate the generic normality and abnormality because when we share the parameters of $V_i$ and $W_i$, the learned semantics of normal and anomaly may be confused.

Reviewer comment (Experiments):
How is the performance of the variant with all components except DPAM?

Author response:
We have added the experiment in Table 7 of Appendix D. When we remove DPAM, the results show a decrease from 91.1% AUROC to 87.9% AUROC in pixel-level performance and from 91.5% AUROC to 80.7% AUROC in image-level performance. This highlights the crucial role played by DPAM in improving AnomalyCLIP's anomaly detection performance. DPAM enhances local visual semantics by modifying the attention mechanism. However, the results indicate that the performance decrease at the image level is more pronounced than that at the pixel level. This discrepancy is attributed to the fact that the total loss places greater emphasis on local context optimization, driven by a larger local loss compared to the case with DPAM. As a result, AnomalyCLIP lacks sufficient optimization for global alignment, leading to a significant decline in image-level performance. Note that the original CLIP (no DPAM) exploits $Q$-$K$ features and suffers from weak segmentation performance.

Reviewer comment (Novelty):
The idea and some techniques, like glocal context optimization, are similar to a concurrent work [1]. The author may compare with highly related works. [1] Gu, Zhaopeng, et al. "AnomalyGPT: Detecting Industrial Anomalies using Large Vision-Language Models." arXiv preprint arXiv:2308.15366 (2023).

Author response:
**Q1: The idea and some techniques, like glocal context optimization, are similar to a concurrent work \[1\].**

Thank you for pointing us toward this concurrent work. AnomalyGPT \[1\] is a great work for anomaly detection. Below we summarize the difference between AnomalyGPT and AnomalyCLIP as follows:

1. The scope is different. AnomalyGPT **eliminates the need for manual threshold adjustments** via LLM when detecting the seen objects during training. However, AnomalyCLIP aims to **improve the model generalization to unseen objects across diverse datasets and domains.**
2. We focus on different settings. AnomalyGPT focuses on **unsupervised anomaly detection** and **few-shot anomaly detection**. However, our work aims at **zero-shot anomaly detection**.
3. Although both AnomalyGPT and AnomalyCLIP propose to combine local and global loss, the purpose is different. AnomalyGPT uses global loss to **quantify the disparity between the text sequence generated by the model and the target text sequence**, ensuring LLM produces corresponding textual descriptions. However, our proposed global context optimization serves to **align the global visual semantics for anomaly classification.**
4. The supervision information is different. Besides the images and text prompts, AnomalyGPT needs **human-crafted text descriptions for different objects to query the LLM during fine-tuning**. AnomalyCLIP just **uses the images and text prompts.**
5. We introduce object-agnostic prompt learning to capture the generic normality and abnormality. AnomalyGPT uses human-crafted text prompts to produce textual embeddings.
6. We propose DPAM to persevere local visual semantics of the visual encoder without additional training. AnomalyGPT introduces a lightweight image decoder to learn the adaption of local features.

In summary, since these two works have different scopes and focus on different aspects of anomaly detection, and the required supervision information is also different. We can not provide a comparison of the quantitative results between AnomalyGPT and AnomalyCLIP. Instead, we include and discuss AnomalyGPT in Related Work in the revised manuscript, which is marked in orange.

>**Reference:**
>1. Zhaopeng Gu, Bingke Zhu, Guibo Zhu, Yingying Chen, Ming Tang, and Jinqiao Wang. Anomalygpt: Detecting industrial anomalies using large vision-language models, 2023.

Reviewer comment (Experiments):
The DPAM strategy is confusing. The author claims that the Q-Q, K-K, V-V self-attention suffers from different issues, and V-V self-attention derives the best result. However, these three variants are very similar in the DPAM mechanism. It is better to explain it in the paper and show the results of the original Q-K attention.

Author response:
**Q2: The DPAM strategy is confusing. These three variants are very similar in the DPAM mechanism. It is better to explain it in the paper and show the results of the original Q-K attention.**

Thank you for raising this concern. Please refer to **General Response** for a detailed explanation of the DPAM strategy from the attention mechanism. We have also explained the reason why the $V$-$V$ achieves the best performance among DPAM variants there. We have added the response in **Refinement of the local visual space** in Section 3.3 in the updated paper, as well as a more detailed analysis in Appendix C.

Reviewer comment (Experiments):
I doubt the contribution of some modules. The module ablation results indicate that the T2 object-agnostic text prompts module contributes the most, similar to CoOp method. However, the author employed a different prompt for the CoOp experiment. For instance, the text prompt template for normality is defined as [V1][V2]...[VN][normal][cls], while that for abnormality is [V1][V2]...[VN][anomalous][cls]. This variation in prompt settings doesn't allow for a fair comparison.

Author response:
**Q3: I doubt the contribution of some modules.**

Thank you for detailed feedback. To demonstrate the effectiveness of each module we propose, we have supplemented the additional module ablation in Table 7 of Appendix D. We test the contribution of one module by removing one module and maintaining the rest module.

1. The effectiveness of DPAM ($T_1$). When we remove DPAM, the results show a decrease from 91.1% AUROC to 87.9% AUROC in pixel-level performance and from 91.5% AUROC to 80.7% AUROC in image-level performance. This performance decline indicates the importance of DPAM, which enhances local visual semantics by modifying the attention mechanism. However, the decrease in performance at the image level is more pronounced than that at the pixel level. This discrepancy is attributed to the fact that the total loss places greater emphasis on local context optimization, driven by a larger local loss compared to the case with DPAM.
2. The effectiveness of object-agnostic prompt learning ($T_2$). Excluding object-agnostic prompt learning makes AnomalyCLIP suffer from the huge performance gap (i.e., 91.1% AUROC to 84.3% AUROC in pixel-level and 91.5% AUROC to 65.6% AUROC in image-level). This performance decline illustrates that the object-agnostic text prompt template plays a significant role in improving the performance of AnomalyCLIP at both pixel and image levels.
3. The effectiveness of textual prompt tuning ($T_3$). When removing textual prompt tuning, the performance of AnomalyCLIP declines from 91.1% AUROC to 90.6% AUROC in pixel-level performance and from 91.5% AUROC to 90.4% AUROC in image-level performance. This demonstrates the importance of adapting original textual space by adding learnable textual tokens in the text encoder.
4. The effectiveness of the integration of multi-layer local visual semantics ($T_4$). When removing multi-layer local visual semantics, the outcomes reveal a decrease from 91.1% AUROC to 90.0% AUROC in pixel-level performance and from 91.5% AUROC to 91.0% AUROC in image-level performance. This performance decline indicates the importance of incorporating multi-layer local visual semantics.

**Q4: Similar to CoOp method. However, the author employed a different prompt for the CoOp experiment.**

Thank you for your comments. Firstly, the reason for adopting the text prompt templates $[V_1][V_2]...[V_N][\textbf{normal}][\verb'cls']$ and $[V_1][V_2]...[V_N][\textbf{anomalous}][\verb'cls']$ in CoOp is because the common text prompt template used in CLIP for anomaly detection is `A photo of a `**`normal`**` [cls]` and `A photo of an `**`anomalous`**` [cls]`. The distinction between CLIP and CoOp lies in the use of learning prompts. In the CLIP vs. CoOp comparison, we aim to demonstrate the performance difference between fixed and learnable text prompts. Secondly, we have introduced an experiment in Appendix D, where CoOp utilizes the text prompt template suggested by the reviewer, denoted as CoOp$_2$, with the template $[V_1][V_2]...[V_N][\verb'cls']$ and $[V_1][V_2]...[V_N][\textbf{damaged}][\verb'cls']$ representing normality and abnormality, respectively. The original CoOp version is referred to as CoOp$_1$. We compare the performance of CoOp$_1$ and CoOp$_2$ across different datasets and domains, presenting the results in Table 9 and Table 10. CoOp$_1$ and CoOp$_2$ have very similar performance, though the text prompt varies. Additionally, we explore $[V_1][V_2]...[V_N][\textbf{anomalous}][\verb'object']$ for AnomalyCLIP in Table 16 and Table 17 of Appendix D. These results suggest that whether using the same prompt template or not, we observe a similar superiority of AnomalyCLIP over CoOp.

Reviewer comment (Writing):
A weakness in the paper is the unexplained initial use of the term "glocal." Clarifying this key term when first mentioned would improve understanding and clarity.

Author response:
We thank the reviewer for the positive and encouraging review. When initially using the term 'glocal' in Section Introduction, we have added the explanation in the revised manuscript: "We then introduce a novel ZSAD approach, called AnomalyCLIP, in which we utilize an object-agnostic prompt template and a glocal abnormality loss function (**i.e., a combination of global and local loss**) to learn the generic abnormality and normality prompts using auxiliary data."

Reviewer comment (Experiments):
The prompt template might be too restrictive. "damaged [cls]" may not well represent all types of anomalies. For example, if a component is missing or applying the method to other domains than defect identification the proposed prompt template may not work well.

Author response:
**Q1: The prompt template might be too restrictive. "damaged \[cls\]\" may not well represent all types of anomalies.**

We are not sure about whether the reviewer refers to the object-agnostic text prompt "damaged **\[object\]**\" (where `object` is a fixed token like `damaged`) or the object-aware text prompt "damaged **\[cls\]**\" (where `cls` is a variable token that will be replaced with a specific class name in the algorithm implementation).In our submission, AnomalyCLIP uses the former one. Nevertheless, we posit that the reviewer's main concern is on the limited scope of the token "damaged\" may have for anomaly detection.

Since the pattern of anomaly is typically unknown and diverse, it is practically difficult to list all possible anomaly types. Thus, we agree that the prompt "`damaged [object]`\" can be restrictive. To remedy this problem, during the fine-tuning process, we provide pixel-level anomaly ground truth with diverse anomaly types for training our ZDAD. This way helps substantially extend the scope of the prompt template "`damaged [object]`\". It indicates in the linguistic semantic of the token "`damaged`\", incorporating fine-grained anomaly semantics of diverse anomaly types. This intuition is verified by our ablation study for the prompt template in Table 16 and Table 17 of Appendix D, in which AnomalyCLIP shows stable performance across different datasets when replacing "`damaged`\" with other similar tokens: "`anomalous`\", "`flawed`\", "`defective`\", and "`blemished`\".

As for the reason why such a prompt template can be applied to other domains, we believe that the learnable part of the prompt template is optimized to focus on the anomaly regions regardless of its foreground object, which allows our model to capture generic object-agnostic, or even domain-agnostic normality and abnormality. Further, compared to object-aware text prompt templates (i.e., "`damaged [cls]`\"), excluding the object semantics from text prompt templates allows learnable text prompt templates to focus on capturing the characteristics of anomalies themselves, rather than the objects. Additionally, the strong generalization of the visual encoder is also important because it can project images from different domains into the same embedding space. Therefore, AnomalyCLIP can detect medical datasets even when fine-tuned on the industrial defect dataset, as shown in Table 2.

Reviewer comment (Experiments):
According to Figure 2, the pipeline needs to feed the same images into two visual encoders, this would introduce additional computation overhead.

Author response:
**Q2: Explanations on the effectiveness of DPAM and computation overhead are necessary.**

We provide a detailed explanation of DPAM and attempt to analyze the effectiveness of the attention mechanism in **General Response**. **We would like to clarify that AnomalyCLIP does not need to feed the images into two visual encoders**. In Figure 2, we plot two paths to intuitively provide the computation difference between global and local visual embedding. **Without introducing additional parameters, we just create two paths when computing the attention map (DPAM), and the rest process shares the same visual encoder.** As for computation overhead, we have supplemented the assessment of the time taken during training (training time per epoch) and the inference speed (frames per second, FPS). For a fair comparison, all experiments are conducted in a single 3090 NVIDIA RTX 3090 24GB GPU, and the GPU is kept free before evaluation. In Table 8, AnomalyCLIP takes 13.71 min per epoch on MVTec AD (The total number of samples is 1725) and only requires a total of 15 epochs for the whole fine-tuning. Once AnomalyCLIP finishes fine-tuning, AnomalyCLIP can be applied to different datasets and domains without additional training. We also compare AnomalyCLIP with other baselines that need auxiliary data (i.e., CoOp and VAND). The minimum training time per epoch is 12.25 min of CoOp, and hence the training time taken is similar for fine-tuning methods. As for inference speed, CLIP achieves the 13.23 FPS. However, it suffers from weak detection performance. Although WinCLIP achieves better performance, WinCLIP has only 1.2 FPS because it needs multiple forward image patches to derive the segmentation. AnomalyCLIP outperforms WinCLIP and obtains an FPS of 8.92.

We also evaluated the computation overhead of DPAM separately. In Table 8, without DPAM, AnomalyCLIP takes 12.98 min to train per epoch. Compared to the 13.71 min for AnomalyCLIP with DPAM, we observe that introducing DPAM does not significantly increase the time complexity. This is attributed to the fact that DPAM only creates the two paths during the computation of the attention map and is frozen during fine-tuning, thereby avoiding the computationally expensive process of gradient computation. Meanwhile, DPAM also does not result in large computation overhead during inference: AnomalyCLIP w/ DPAM gets 8.92 FPS vs. 10.21 FPS for w/o using DPAM.

Reviewer comment (Theory):
The paper has not discussed why the proposed approach is able to learning class-agnostic feature from the pretrained semantic CLIP as the CLIP is trained to be sensitive to semantic classes.

Author response:
**Q3: Why the proposed approach is able to learn class-agnostic features from the pretrained semantic CLIP as the CLIP is trained to be sensitive to semantic classes.**

Thank you for your insightful comments. To mitigate the sensitivity to object semantics in CLIP, AnomalyCLIP proposes three modules to adapt the original textual and visual embeddings to object-agnostic anomaly semantics. Firstly, we propose object-agnostic prompt learning to exclude the object semantics from text prompt templates and allow learnable text prompt templates to focus on capturing the characteristics of anomalies themselves, rather than the objects. This empowers AnomalyCLIP to generate the appropriate embeddings in the textual space. However, the object-agnostic embedding is not only decided by the text prompt but also depends on the appropriate textual space. Therefore, we also introduce learnable textual tokens in the textual encoder to adapt the textual space. Besides the textual encoder, we also propose DPAM to improve the local visual embeddings to facilitate the learning of object-agnostic features. Finally, during fine-tuning, AnomalyCLIP is optimized by glocal context optimization to focus on image-level and pixel-level anomalies across diverse anomaly types regardless of object semantics. This process further enables textual embeddings to be more generic and capable of identifying anomalies across diverse objects and different domains.

Reviewer comment (Evaluation):
It is important to discuss why the "damaged [cls]" is suitable for more diverse anomaly detection scenarios.

Author response:
**Q1: The prompt template might be too restrictive. "damaged \[cls\]\" may not well represent all types of anomalies.**

We are not sure about whether the reviewer refers to the object-agnostic text prompt "damaged **\[object\]**\" (where `object` is a fixed token like `damaged`) or the object-aware text prompt "damaged **\[cls\]**\" (where `cls` is a variable token that will be replaced with a specific class name in the algorithm implementation).In our submission, AnomalyCLIP uses the former one. Nevertheless, we posit that the reviewer's main concern is on the limited scope of the token "damaged\" may have for anomaly detection.

Since the pattern of anomaly is typically unknown and diverse, it is practically difficult to list all possible anomaly types. Thus, we agree that the prompt "`damaged [object]`\" can be restrictive. To remedy this problem, during the fine-tuning process, we provide pixel-level anomaly ground truth with diverse anomaly types for training our ZDAD. This way helps substantially extend the scope of the prompt template "`damaged [object]`\". It indicates in the linguistic semantic of the token "`damaged`\", incorporating fine-grained anomaly semantics of diverse anomaly types. This intuition is verified by our ablation study for the prompt template in Table 16 and Table 17 of Appendix D, in which AnomalyCLIP shows stable performance across different datasets when replacing "`damaged`\" with other similar tokens: "`anomalous`\", "`flawed`\", "`defective`\", and "`blemished`\".

As for the reason why such a prompt template can be applied to other domains, we believe that the learnable part of the prompt template is optimized to focus on the anomaly regions regardless of its foreground object, which allows our model to capture generic object-agnostic, or even domain-agnostic normality and abnormality. Further, compared to object-aware text prompt templates (i.e., "`damaged [cls]`\"), excluding the object semantics from text prompt templates allows learnable text prompt templates to focus on capturing the characteristics of anomalies themselves, rather than the objects. Additionally, the strong generalization of the visual encoder is also important because it can project images from different domains into the same embedding space. Therefore, AnomalyCLIP can detect medical datasets even when fine-tuned on the industrial defect dataset, as shown in Table 2.

Reviewer comment (Evaluation):
Explanations on the effectiveness of DPAM and computation overhead are necessary.

Author response:
**Q2: Explanations on the effectiveness of DPAM and computation overhead are necessary.**

We provide a detailed explanation of DPAM and attempt to analyze the effectiveness of the attention mechanism in **General Response**. **We would like to clarify that AnomalyCLIP does not need to feed the images into two visual encoders**. In Figure 2, we plot two paths to intuitively provide the computation difference between global and local visual embedding. **Without introducing additional parameters, we just create two paths when computing the attention map (DPAM), and the rest process shares the same visual encoder.** As for computation overhead, we have supplemented the assessment of the time taken during training (training time per epoch) and the inference speed (frames per second, FPS). For a fair comparison, all experiments are conducted in a single 3090 NVIDIA RTX 3090 24GB GPU, and the GPU is kept free before evaluation. In Table 8, AnomalyCLIP takes 13.71 min per epoch on MVTec AD (The total number of samples is 1725) and only requires a total of 15 epochs for the whole fine-tuning. Once AnomalyCLIP finishes fine-tuning, AnomalyCLIP can be applied to different datasets and domains without additional training. We also compare AnomalyCLIP with other baselines that need auxiliary data (i.e., CoOp and VAND). The minimum training time per epoch is 12.25 min of CoOp, and hence the training time taken is similar for fine-tuning methods. As for inference speed, CLIP achieves the 13.23 FPS. However, it suffers from weak detection performance. Although WinCLIP achieves better performance, WinCLIP has only 1.2 FPS because it needs multiple forward image patches to derive the segmentation. AnomalyCLIP outperforms WinCLIP and obtains an FPS of 8.92.

We also evaluated the computation overhead of DPAM separately. In Table 8, without DPAM, AnomalyCLIP takes 12.98 min to train per epoch. Compared to the 13.71 min for AnomalyCLIP with DPAM, we observe that introducing DPAM does not significantly increase the time complexity. This is attributed to the fact that DPAM only creates the two paths during the computation of the attention map and is frozen during fine-tuning, thereby avoiding the computationally expensive process of gradient computation. Meanwhile, DPAM also does not result in large computation overhead during inference: AnomalyCLIP w/ DPAM gets 8.92 FPS vs. 10.21 FPS for w/o using DPAM.