Paper: Model Selection of Anomaly Detectors in the Absence of Labeled Validation Data (ICLR-2024)

Reviewer comment (Experiments):
The paper's abnormal synthesis function is entirely based on DiffStyle, and it remains to be verified whether the method of interpolating abnormal images from different normal images is reasonable. Perhaps perturbing features in different dimensions in the latent space may have better results. The paper should consider different designs for high-level semantic anomalies and low-level semantic anomalies in this regard.

Author response:
There are many options for modifying how the diffusion process works. The first, and simplest ideas would be to change (i) what images are used as “style” and “content” for DiffStyle and (ii) to modify the value of gamma in the DiffStyle generation process (which modifies the style-content proportions). The original DiffStyle paper (Jeong et al. 2023a) evaluates both of these parameters. Rather than search over the parameter space for optimal results, our paper shows that, even with a simple combination of images and default DiffStyle values, our method is effective on some (i.e., CUB and Flowers) datasets for model selection! This is a valuable result on its own and follow-up experiments would be interesting, but are beyond the scope of the current paper.

Reviewer comment (Experiments):
It is unclear what considerations the authors had in comparing the Flowers, CUB, and MVTec datasets. There are significant differences between these datasets.

Author response:
We selected the CUB, Flowers, and MVTec-AD datasets as they were the baseline datasets used in Mirzaei et al (2023), see Table 1 in their work. Mirzaei et al. also use CIFAR-10 and CIFAR-100, but we found that the images in these datasets were of too low resolution for the ImageNet-pretrained diffusion model, and upsampled versions of these images did not produce good results with DiffStyle.

Reviewer comment (Evaluation):
The authors could elaborate more on the limitations of the approach. For instance, one problem I see is that the synthetically generated anomalies do not necessarily resemble the ground truth distribution of anomalies. In particular, the question for the image classification datasets is indeed what would an actual anomaly look like? In particular, to me, it is questionable whether the generated images make sense at all as the observations would probably never be made in the real world.

Author response:
Regarding the nature of anomalies, we do not claim that our synthetic anomalies are indicative of real anomalies; we only claim that they can be used to estimate the performance one would get with real validation data. If our method were able to produce realistic anomalies, then these examples could be added to real validation datasets. 

This highlights the importance of our technique; since we assume a setting where real anomalies are difficult to generate and cannot be reproduced, there is a strong need for synthetic data that can be used to estimate performance, even if they do not visually represent the distribution of anomalies.

Reviewer comment (Theory):
How are anomalies even defined in the image domain? Is it just out of distribution data? Unrealistic images? If the latter, why would one even expect to observe such images?

Author response:
Regarding the nature of anomalies, we do not claim that our synthetic anomalies are indicative of real anomalies; we only claim that they can be used to estimate the performance one would get with real validation data. If our method were able to produce realistic anomalies, then these examples could be added to real validation datasets. 

This highlights the importance of our technique; since we assume a setting where real anomalies are difficult to generate and cannot be reproduced, there is a strong need for synthetic data that can be used to estimate performance, even if they do not visually represent the distribution of anomalies.