Paper: Self-Tuning Self-Supervised Anomaly Detection (ICLR-2024)

Reviewer comment (Experiments):
The authors evaluate MV Tech AD with the data being collected according to given abnormal patterns. I wonder if the proposed framework can simultaneously deal with multiple different augmentations, since this case is more general in real-world anomaly detection scenarios and is more likely to avoid overfitting to the optimal **a**. It will be helpful to show the results with **UN**split original classes in MV Tech AD.

Author response:
Our current approach can be subpar on test data that contains anomalies of multiple types, for which tuning the discrete choice of augmentation functions becomes a new challenge.  Our work presents a framework to tune the *continuous* hyperparameters of an augmentation function, provided with the discrete choice. Please refer to **C2** of our common responses for possible hyperparameter optimization (HPO) approaches we laid out as future work.

Reviewer comment (Evaluation):
The assumption that the validation and test distributions are consistent may be too strong for the setting of anomaly detection task (*e.g.* new anomaly pattern or even mixing anomalies makes **a** not optimal). It may not be reasonable to omit test images drawn from unseen distributions in real scenarios.

Author response:
We believe that transductive learning is common in real-world anomaly detection tasks. Please  refer to **C1** of our common responses for an extended discussion on transductive vs. inductive learning in the context of anomaly detection. Nevertheless, we also conducted additional experiments dividing unlabeled test data into validation and (new) test data, and then using only the validation data for computing the alignment loss. Please refer to **C3 #2** of our common responses for the results.

Reviewer comment (Theory):
The author believes that the heuristic function $S$ will be high (higher variance of anomaly scores) *only if* augmentation parameters are initialized better (more separable distributions). However, this may not always hold, as the optimization of a mismatched *strong/hard* augmentation (as long as the decision boundary divides testing distribution into any two parts) may also lead to high variance of anomaly scores. I wonder if there are any theoretical or intuitive explanations about this issue.

Author response:
We remark that this heuristic is employed only after optimizing for the alignment first. In other words, we use it only to choose from among models that are already optimized to align with the test data as well as possible. Since a decision boundary dividing the test data would not yield a low validation loss in the first place, the heuristic is not likely to be exposed to such scenarios in practice.

Reviewer comment (Experiments):
The authors use CutDiff and Rot for non-semantic and semantic shift detection respectively. But I would like to know could ST-SSAD learn zero-angle Rot for non-semantic defects and zero-size CutDiff for semantic shifts in end-to-end optimization when using both augmentations together.

Author response:
As discussed in detail in **C2** of our common responses, our current model is designed to tune the *continuous* hyperparameters of a given augmentation function. Given a set of augmentation functions, making a discrete choice between the different functions or optimizing a mixture of functions together poses a new challenge. For example, with both CutDiff and Rotate included, the ratio of augmented samples to generate with the two functions becomes another hyperparameter. Being the first solution to the automatic model selection for SSAD, we expect our work will trigger several follow-up works.

Reviewer comment (Theory):
The term ``Ratio invariance’’ may be imprecise. With the augmentation quantity changed, $L_{val}$ could be changed together because the included $\frac{\sqrt{N}}{{\Vert {Z}^{c}\Vert}_{F}}$ is different (I notice that this have no negative impact on optimization target).

Author response:
That is correct. The total distance normalization is affected by the size of augmented data, and thus our loss function is not exactly invariant to it. However, the effect is negligible, since that mainly changes the scale of embeddings, rather than their distributions. For preciseness, we have changed the term into "ratio robustness" in the revised manuscript.

Reviewer comment (Writing):
There are some writing issues, *e.g.* superscript $^{n}$ should be $^{a}$ in **Lemma 2**.

Author response:
Thank you for finding out the typo. We have updated the revised manuscript accordingly.

Reviewer comment (Experiments):
During evaluation, each anomaly type is separated. The proposed ST-SSAD seems to assume only one anomaly type exists in the test set. That is, the user might need to use ST-SSAD for each anomaly type. How to handle multiple anomaly types in the same test set is not clear.

Author response:
As the reviewer accurately pointed out, our approach does not currently address test data that contain anomalies of multiple types, for which tuning the discrete choice of augmentation functions becomes a new challenge. Our work presents a framework to tune the *continuous* hyperparameters of an augmentation function, provided with the discrete choice. We refer to **C2** of our common responses for possible hyperparameter optimization (HPO) approaches we laid out as future work toward tackling a mixture of anomalies in test data.

Reviewer comment (Experiments):
In the experiments, anomaly types are separately evaluated. That seems to mean that the anomaly type is known, and the augmentation parameters are learned to match the anomaly type. This seems to be assumed in Equation 3 because the second term calculates the mean of the augmented instances. That is, Eq. 3 seems to be finding an augmentation and its parameters to match the anomaly type. If that is correct, how can multiple types of anomalies in the same test set be handled?

Author response:
As the reviewer accurately pointed out, our approach does not currently address test data that contain anomalies of multiple types, for which tuning the discrete choice of augmentation functions becomes a new challenge. Our work presents a framework to tune the *continuous* hyperparameters of an augmentation function, provided with the discrete choice. We refer to **C2** of our common responses for possible hyperparameter optimization (HPO) approaches we laid out as future work toward tackling a mixture of anomalies in test data.

Reviewer comment (Experiments):
While the augmentation parameters are learnable, the augmentation types such as cut and rotation need to be specified. Also, to be detected, seemingly an anomaly type in the test set needs to match one of the augmentation types. Could the matching be relaxed? That is, the user potentially does not need to know what the anomaly types are.

Author response:
As we show in Section 4.4, our approach succeeds if the augmentation function is aligned with the true anomalies at the functional level. As discussed in **C2** of our common responses, the *discrete* choice of the augmentation from a *set* of possible functional forms can be addressed via black-box hyperparameter optimization (HPO). Our work presents a framework to tune the *continuous* hyperparameters provided with the discrete choice. We expect several follow-up works, not only proposing new differentiable augmentation functions but also more elaborate HPO techniques to select from this set, as laid out in **C2.**

Reviewer comment (Experiments):
What is the size of \(D_{aug}\)?

Author response:
We set the size of $D_{aug}$ to be the same as that of $D_{trn}$ in all experiments. This was to avoid introducing an additional hyperparameter. To show robustness, we conducted additional experiments varying the ratio of augmented data over the size of training data. Please refer to **C3 #1** of our common responses for the results.

Reviewer comment (Writing):
In Eq 2, equation on the right for \(z_i^c\): should \(z\) be \(z_i\) and the summation is over another index such as \(j\) to not confuse with \(i\)?

Author response:
That’s correct. We have changed Eq. (2) accordingly in the revised manuscript. Thank you for pointing it out.

Reviewer comment (Writing):
In Eq 3, mean(.) seems to be similar to \(1/N\sum_{i=1}^{N}z_{i}\) in Eq 2. If so, using mean(.) in Eq 2 would make the presentation consistent. If not, what is the difference?

Author response:
Both expressions are almost the same: mean(.) in Eq. (3) and $1/N \sum_{i=1}^N z_i$ in Eq. (2). The only difference is that mean(.) in Eq. (3) is an operator on a set of vectors, while in Eq. (2) we computed the mean of rows in a matrix. It would make them consistent if we treat both as sets or both as matrices, but we believe the current way is more intuitive and easier to understand.

Reviewer comment (Experiments):
A major concern is that ST-SSAD replies on the entire test dataset during training and tuning. While the authors mention transductive learning, this approach is not quite realistic, particularly in the context of anomaly detection. The tuning result will be overly sensitive to the specific anomaly types in the test data and may not generalize well. The paper lacks clarification, experimental results, or in-depth discussion on this issue, which significantly limits the applicability and advantages of the proposed method.

Author response:
We believe that transductive learning is common in real-world anomaly detection tasks, whereas generalization is a matter for inductive learning. Please refer to **C1** of our common responses.

Reviewer comment (Experiments):
The mean distance loss is proposed for ratio invariance with theoretical properties, but no experimental result validating this invariance is provided.

Author response:
We have conducted additional experiments varying the ratio of augmented data over the size of training data. Please refer to **C3 #1** of our common responses for the result.

Reviewer comment (Theory):
The method still requires prior knowledge about anomalies and heavily depends on it. For example, the augmentation functions of either local (CutDiff) or global (rotation) augmentations are considered and therefore, it works well only when anomalies closely resemble specific shapes that these functions can reflect. The method will also fail in the case where rotated samples are considered normal.

Author response:
As we discussed in Section 4.4, our approach succeeds if the augmentation function is aligned with the true anomalies at the functional level. Nevertheless, as the first systematic approach to the automatic model selection for self-supervised anomaly detection, we believe our work can lead to numerous future works that extend our approach to new differentiable augmentations that capture other types of real-world anomalies. Please refer to **C2** of our common responses for more discussion on the problem setup and suggested directions of future works.

Reviewer comment (Experiments):
The authors state that 'we focus on the performance of each anomaly type rather than overall accuracy.' However, in real-world scenarios, it is common to encounter various types of anomalies. Therefore, it will be more crucial to investigate such practical scenarios.

Author response:
Our current approach does not address test data that contain anomalies of multiple types, for which tuning the discrete choice of augmentation functions becomes a new challenge. Our work presents a framework to tune the *continuous* hyperparameters of an augmentation function, provided with the discrete choice. As we answered to W3, please refer to **C2** of our common responses for possible hyperparameter optimization (HPO) approaches we laid out as future work toward tackling a mixture of anomalies in test data.

Reviewer comment (Experiments):
It was mentioned that "there are no direct competitors on end-to-end augmentation hyperparameter tuning..."; however, it is essential to include performance comparison with the latest models that clearly distinguish train and test data. The results in Tables 1 and 2 appear to be more like an ablation study, so it is difficult to assess whether the proposed method truly outperforms the latest models in a meaningful way.

Author response:
Please note that it is difficult to conduct a fair comparison between our approach and other works on anomaly detection, since the extent of “fair” hyperparameter tuning is hard to define between different lines of works. A vast majority of work on unsupervised anomaly detection does not address the unsupervised hyperparameter (HP) tuning problem at all.

Our results show significant performance boost over two most prominent unsupervised approaches; DeepSVDD (one-class) and DeepAE (reconstruction-based). On the self-supervised side, we show that tuning augmentation significantly outperforms arbitrarily choosing the augmentation, including the CutOut and CutPaste approaches (CO and CP in Table 1) proposed by Devries & Taylor (2017) and Li et al. (2021), respectively.  Since we used the detector network and the score function used in (Li et al., 2021) throughout all experiments, we can consider the RS-CP and RD-CP baselines in Table 1 as "untuned" versions of the CutPaste approach (Li et al., 2021).

Reviewer comment (Experiments):
Can you provide results using a validation set that is disjoint from the test set?

Author response:
Yes, we have conducted additional experiments dividing unlabeled test data into validation and (new) test data, and then using only the validation data for computing the alignment loss. Please refer to **C3. #2** of our common responses for the results.

Reviewer comment (Experiments):
I guess the proposed method may be quite sensitive to the proportion of abnormal samples in the test set. Can you provide experimental results or a discussion addressing this issue? And please provide the ratio between normal and abnormal samples in the presented results of the current manuscript.

Author response:
Our datasets already vary in the proportion of anomalies they contain in the test data. The following table shows the percentage of anomalies across different tasks in the MVTec dataset, where the minimum and maximum are 0.147 and 0.404, respectively. Similarly in the SVHN dataset, the fraction is between 0.279 and 0.722, since the different classes contain different numbers of examples. Thus, we believe that our approach is not sensitive to the ratio of anomalies in test data, as long as the anomalies are separable from the normal data and thus the alignment can be measured by our validation loss.

obj_type | ano_type | fraction
--- | --- | ---
cable | bent_wire | 0.183 
cable | cable_swap | 0.171 
cable | combined | 0.159 
cable | cut_inner_insulation | 0.194 
cable | cut_outer_insulation | 0.147 
cable | missing_cable | 0.171 
cable | missing_wire | 0.147 
cable | poke_insulation | 0.147 
carpet | color | 0.404 
carpet | cut | 0.378 
carpet | hole | 0.378 
carpet | metal_contamination | 0.378 
carpet | thread | 0.404 
grid | bent | 0.364 
grid | broken | 0.364 
grid | glue | 0.344 
grid | metal_contamination | 0.344 
grid | thread | 0.344 
tile | crack | 0.340 
tile | glue_strip | 0.353 
tile | gray_stroke | 0.327 
tile | oil | 0.353 
tile | rough | 0.312

Reviewer comment (Experiments):
Is it possible for the model to learn effectively in a scenario where abnormal samples are inherently present in the training set but remain unlabeled?

Author response:
Yes, this refers to the setting where training data is not “clean” (i.e. anomaly-free) but is contaminated with some anomalies. We have conducted additional experiments by varying the degree of contamination in the training data. Please refer to **C3 #3** of our common responses for the results.