Paper: Hierarchical Gaussian Mixture Normalizing Flows Modeling for Multi-Class Anomaly Detection (ICLR-2024)

Reviewer comment (Presentation):
Figure 2(b)/(d): The authors need to clarify the colors associated with the classes.

Author response:
**[To W2].** We are sorry about this confusion. Due to page limitation, we move the overall loss function to the Appendix. This results in the absence of definitions of hyperparameters $\lambda_1$ and $\lambda_2$. We really appreciate that you point out this problem. We will revise the layout of our paper to ensure that such a problem won't appear in our paper.

Reviewer comment (Reproducibility):
What are the $\lambda_1, \lambda_2$ in Section 4.2? It is confusing that the authors list several separate loss functions in Section 3.3 without any articulation about how to deal with these equations to achieve the goal(s). The reviewer finally found the objective function of the target goal after checking with the appendix, but the authors didn't mention anything in the main paper.

Author response:
**[To W3].** The Feature Extractor will extract multi-scale feature maps, level-k means the kth feature map level. Using multi-scale features is common in anomaly detection, and our method also builds a normalizing flow model at each feature level. We appreciate your suggestion and will add explanations for level-k in the revision.

Reviewer comment (Theory):
In Section 3.2, since the authors pointed out that Eq. (2) is used to maximize the log-likelihood of normal features, why do the normalizing flows present a large log-likelihood for both normal and abnormal features? In other words, are both normal and abnormal/anomaly observations used in this loss function?

Author response:
**[To W5].** In anomaly detection, we only use normal samples for training as our goal is to detect anomalies. In our paper, we have clearly explained that normalizing flow (NF) may have a ”homogeneous mapping'' problem when used for multi-class anomaly detection, where NF may take a bias to map different input features to similar latent features (i.e., this means anomalies will have similar log-likelihoods to normal). Moreover, the goal of training is to maximize the log-likelihoods of normal features. Thus, normalizing flows will represent large likelihoods for both normal and abnormal features. Please see the sec. 3.2 in our paper, we have provided detailed explanations for this.

Reviewer comment (Experiments):
In Section 4, what is the partition for the data in experiments? What are the normal classes? What are the anomaly classes? If label information (including anomalies) is used in the training, why do we call this multi-class anomaly detection? What is the difference between this and regular multi-class classification?

Author response:
**[To W6].** Firstly, we will indicate the difference between industrial anomaly detection and semantic anomaly detection. Our paper focuses on industrial anomaly detection rather than semantic anomaly detection. In industrial anomaly detection, all classes are normal, the anomalies are defective areas that exist in the image and don't have classes. Only in semantic anomaly detection (e.g., CIFAR10, CIFAR100, and ImageNet-30 are usually the datasets), we will select some classes as normal and the other classes are used as abnormal.

The partition follows the standard way of these industrial AD datasets, MVTecAD, BTAD, MVTec3D-RGB, and VisA. Generally speaking, most industrial AD papers will not specifically elaborate on the data partition, as the training and test sets are fixed. 

We have clearly defined the multi-class AD in the second paragraph of Introduction: one unified model is trained with normal samples from multiple classes, and the objective is to detect anomalies in these classes. In training, we only use normal samples from multiple product classes without any anomaly, the label information is used to indicate one normal sample belongs to which class. For the last question, please see our response to W3.

Reviewer comment (Reproducibility):
Since there are multiple goals contained in the objective function and different training strategies in the experiments, to clearly summarize the work, it would be better to use pseudocode to outline the algorithm.

Author response:
**[To W7].** Thanks for your suggestion, we will add the pseudocodes of our algorithm in the revision.

Reviewer comment (Evaluation):
There is weak support of the necessity of the intra-class centers: From Figure 3(b), the reviewer cannot see a significant difference among different numbers of intra-centers.

Author response:
**[To W8].** We validate the effectiveness of the intra-class centers in ablation studies (see Tab. 3(a)). The number of intra-class centers is the same for each class (we state this in our paper, please see sec. 4.2, Setup). Figure 3 is just a schematic diagram of our model. Moreover, we can see that the number of intra-class centers in the two classes is the same, but the intra-class distributions are different (this is our purpose of introducing intra-class centers).

Reviewer comment (Experiments):
Figure 3: Why is the positional encoding added to the normalizing flow? Is this necessary? Did the authors conduct the ablation study of this design?

Author response:
**[To Q2].** It's necessary. With positional embeddings, we can achieve better results. We don't conduct an ablation study for this design, as the effectiveness of positional embeddings is already validated in the previous work CFLOW [1]. In our paper, we also state that the normalizing flow used in our model is the same as the one in CFLOW (please see sec. 4.2, Quantitative Results).

[1] Denis Gudovskiy, Shun Ishizaka, and Kazuki Kozuka. Cflow-ad: Real-time unsupervised anomaly
detection with localization via conditional normalizing flows. In IEEE Winter Conference on
Application of Computer Vision, 2022.

Reviewer comment (Reproducibility):
Bottom of Page 5: Why not just use sample class priors to estimate $p(Y)$? Which part of the architecture in Figure 3 is used to estimate $p(Y)$? Could the authors explain in detail?

Author response:
**[To Q3].** Because we think that parameterization $p(Y)$ can enable the network to adaptively learn the class weights, and parameterization $p(Y)$ only introduces a small number of parameters. The parameter to control $p(Y)$ is $\psi$, and $\psi$ is learned by optimizing the E.q(6). Then $p(y)$ is estimated by ${\rm softmax}_y(\psi)$. No module in Figure 3 is dedicated to estimating $p(Y)$, the parameter $\psi$ belongs to the normalizing flow model.

Reviewer comment (Experiments):
The notations in (8) and (9) are bad. What is $\mu\_{y^\prime}$? Do you mean the center vector? Is loss (9) necessary? Why is there no penalty cost before this loss in the final objective function? Did the authors conduct the ablation study for this loss function?

Author response:
**[To Q4].** We respectfully disagree with this comment. $\mu_{y^\prime}$ means all other class centers except the class center $\mu_y$ corresponding to $y$. Because our method is used for multi-class anomaly detection, there will be many class centers $\{\mu_y\}, y \in \{1,\dots,N\}$. In Eq.(8) and (9), when we employ the softmax function to calculate the value for a special class $y$, all the other classes are naturally represented as $y^\prime$. We think that such notations are commonly used in the softmax function. So, we don't specifically explain $y^\prime$. We will add corresponding explanations to make E.q (8) and (9) easier to understand in the revision.

Loss (9) is necessary. In Tab 3(c), we show that using Entropy is beneficial to achieve better results. And loss (9) is used to optimize entropy during training. The total loss is $\mathcal{L} = \lambda _1\mathcal{L} _g + \lambda _2\mathcal{L} _{mi} + \mathcal{L} _e + \mathcal{L} _{in}$. Our design is to use $ \mathcal{L} _e $ and $ \mathcal{L} _{in} $ as auxiliary losses. As there are four optimization objectives, adding a weighting factor and conducting ablation studies for each loss item will result in a large number of combinational experiments, which will bring us too much burden.

Reviewer comment (Experiments):
In Section 3.3, the reviewer sees one loss function is designed to maximize the log-likelihood of normal observations. Why did the authors claim that using a label should not be a strict condition? Did the authors conduct the experiment to support this conclusion?

Author response:
**[To Q6].** As we mentioned above, the label is used to distinguish which class each sample belongs to. In a stricter condition, we may hope the multi-class AD models even don't need labels, which means that we have many normal samples from different classes but don't know which class they belong to. Because the AD datasets themselves separate samples according to classes, we can easily get the class label from these datasets. Therefore, we claim that using class labels should not be a strict condition (or constraint) when designing multi-class AD methods. We think that we explain this clearly, this conclusion seems not to require experimental validation.

Reviewer comment (Writing):
The abstraction is somewhat lengthy. Please polish the abstraction and make it concise.

Author response:
**[To W1].** We are very grateful for this suggestion, we will polish the abstract to make it more concise in the reavision.

Reviewer comment (Writing):
The representation should be improved to be more professional. The explanations of some equations (eg. Eq6 and Eq9 ) are not easy-understood.

Author response:
**[To W3].** Thanks for your suggestion. E.q(6) is a derived loss (the detailed derivation is in Appendix E), we will include some key intermediate steps for assisting understanding. E.q(9) is actually the entropy formula, where we use $-||\varphi_\theta(x)-\mu_y||^2_2/2$ as class logits. $\mu_{y^\prime}$ means all the other class centers except for $\mu_y$. We don't specifically explain $\mu_{y^\prime}$ as we think this notation is commonly used in the softmax function. We will add more explanations to make E.q(9) easier to understand. For other equations, we will also carefully check and explain these equations more clearly in the revision.

Reviewer comment (Novelty):
Is the homogeneous mapping issue intrinsically equal to the well-known identical shortcut problem?

Author response:
**[To Q1].** Thank you for the comment, but we cannot fully agree with this comment. The identical shortcut is essentially caused by the leakage of abnormal information. The process of reconstruction is to remove abnormal information in the input, resulting in the failure of reconstruction in abnormal regions. But if the reconstruction network is overfitted, the abnormal features may be leaked into the output, resulting in the reconstruction network directly returning a copy of the input as output. So, the reconstruction errors in abnormal regions will be small, leading to abnormal missing detection. This issue usually can be addressed by masking, such as the neighbor masking mechanism in UniAD. 

Homogeneous mapping is a specific issue in normalizing flow (NF) based AD methods. In previous NF-based AD methods, the latent feature space has a single center. When used for multi-class AD, we need to map different class features to the single latent center, this may cause the model more prone to take a bias to map different input features to similar latent features. Thus, with the bias, the log-likelihoods of abnormal features will become closer to the log-likelihoods of normal features, causing normal misdetection or abnormal missing detection. To address this issue, we propose hierarchical Gaussian Mixture normalizing flow modeling. Because there are significant differences in the causes and solutions of the two issues, we think that the two issues are not intrinsically equal. In Appendix A.2, we have provided a thorough discussion for this question.

Reviewer comment (Presentation):
The citations might be wrong. Many citations should be placed in the brackets. Please pay attention to the difference between '\citep' and '\citet'.

Author response:
**[To Q2].** We are very grateful for this comment, it's really helpful for us to improve our paper. We will revise this issue in the revision.

Reviewer comment (Writing):
The full name of HGAD should be listed.

Author response:
**[To Q3].** HGAD is taken from our paper title: **H**ierarchical **G**aussian Mixture Normalizing Flows Modeling for Multi-class **A**nomaly **D**etection. As this is too long, we didn't list the full name in the main text (only list “Hierarchical Gaussian mixture” in Abstract). In the revision, we will explain the HGAD naming in the main text.

Reviewer comment (Evaluation):
Why the performance of multi-class case is lower than the unified case, as shown in Table 1.

Author response:
**[To Q4].** In our paper, multi-class and unified actually have the same meaning. In the caption of Tab 1, we use “unified/multi-class” to express that multi-class is an alias for unified. The meaning of results is explained by the following sentence: “$\cdot$/$\cdot$ means the image-level and pixel-level AUROCs''. We are sorry for this misunderstanding. We will revise this misleading caption in the revision.

Reviewer comment (Evaluation):
The best performance on MVTec in Table 1 are 98.4/97.9, but 97.7/97.6 in Table 3.

Author response:
**[To Q5].** In Tab 3(a), the hyperparameters $\lambda_1$ and $\lambda_2$ are set as 1 and 10. However, we later find that setting $\lambda_1$ and $\lambda_2$ to 1 and 100 can get better results (see Tab 3(d)). In Tab 1, we update the better results, but most results in Tab 3 are obtained with the $\lambda_1$ and $\lambda_2$ set as 1 and 10.

Reviewer comment (Experiments):
One of my main concerns is that most of the methods compared to (e.g. UniAD, FastFlow, etc) are *unsupervised* whereas the proposed method is a *supervised* approach explicitly requiring class labels to be provided (see e.g. discussion in Appendix A.1). On the face of it, this does not seem like a fair comparison to make. It is important that the authors explicitly summarize what supervision each method uses and justify why theirs is a better approach despite requiring explicit label information to be provided during training.

Author response:
**[To W1].** In the anomaly detection field, we think that our method should not be seen as supervised. Because anomaly detection focuses on distinguishing anomalies from normals, methods that use abnormal samples during training should be seen as supervised. For example, BGAD is supervised as it uses few-shot anomalies during training. However, our method only uses normal samples during training. 

As we have explained in Appendix A.1, our method has the same data organization format as the one-for-one AD methods (e.g., PaDiM, MKD, DRAEM, FastFlow, and CFLOW in our paper) and doesn't introduce any extra data collection cost. The existing AD datasets are collected for one-for-one anomaly detection (i.e., we need to train a model for each class). Thus, the existing AD datasets need to be separated according to classes, with each class as a subdataset. Therefore, one-for-one AD methods also need class labels, as they require normal samples from the same class to train. If the classes are not separated (in other words, without class labels), these one-for-one AD methods cannot be used either. Our method actually has the same supervision as these methods. The difference is that we explicitly use the class labels but they don't explicitly use the class labels. However, our advantage is that we can achieve only training a unified model for all classes, this is a significant innovation compared to the one-for-one AD methods. The unified AD method, PMAD, also explicitly uses class labels when accomplishing multi-class anomaly detection. The UniAD and OmniAL don't use class labels, but as we mentioned above, using class labels doesn't introduce any extra data collection cost. However, our method can achieve better multi-class detection results than UniAD and OmniAL.

In a stricter condition, we may hope the multi-class AD models even don’t need labels, which means that we have many normal samples from different classes but don’t know which class
they belong to. Because the AD datasets themselves separate samples according to classes, we can easily get the class labels from these datasets. Therefore, we think that using class labels should not be a strict condition (or constraint) when designing multi-class AD methods.

Reviewer comment (Presentation):
The writing and presentation is at places hard to follow. The authors are urged to present the high-level approach first before dwelling into the details of the individual loss components. Having an explicit pseudo-code stating what the supervision is for the algorithm, and how the overall optimization objective looks like would be very helpful.

Author response:
**[To W2].** Thanks for your suggestion. In the revision, we will provide an explicit pseudo-code of our algorithm to more clearly describe the whole algorithm process and the overall optimization object to make readers more easier to follow our paper.

Reviewer comment (Theory):
The proposed approach appears to have a lot of moving parts: there are four loss components (one for a inter-class Gaussian mixture, one for an intra-class Gaussian mixture, a mutual information based and an entropy-based loss for class diversity), with two hyper-parameters for weighting them (Appendix C). Although the authors do conduct some analysis of different hyper-parameter combinations, one if left with a feeling that the approach is highly heuristic in nature, with the gains coming largely from heavy engineering effort. Improving the writing and presentation may help boost the reader's confidence in the proposed method.

Author response:
**[To W3].** Thanks for this comment, it’s really helpful for us to improve our paper. We will take your suggestions and other reviewers' suggestions seriously to improve the presentation and organization order of our paper. We would be very grateful if you could provide us with more suggestions on further improving our paper.