Paper: Going beyond familiar features for deep anomaly detection (ICLR-2024)

Reviewer comment (Theory):
The manuscript claims a formal definition of familiar and novel features as one of the main contributions. However, the presentation in Sec.3 is rather poor - definitions of familiar/unfamiliar features are bundled together, the notations are non-standard, and the whole section is hard to comprehend. Claimed definitions of familiar and novel features should be neatly formulated with appropriate definitions, lemmas, and theorems as in [a].

Author response:
We recognize the importance of formalizing the definition of these concepts. This can not only help AD but also tasks like Open set recognition, anomaly detection and novel class detection. We revised Section 3 to highlight the definitions from other discussions. Our primary aim is to define the idea of familiar and novel features in a test input in the context of anomaly detection and illustrate the practical implications. Taking forward the hypothesis in [2], a future work that approaches the theoretical foundation can help the field.

Reviewer comment (Writing):
The definition of the final anomaly score should be clearly stated together with the corresponding equation.

Author response:
Thank you for the suggestion, have added that to the section. The anomaly score computed as, novelty and familiarity score combined is highlighted.

Reviewer comment (Novelty):
Missing relevant related work which introduces hybrid formulation for anomaly detection [b].

Author response:
Thank you for the reference. We have added it to the related work as other notions of hybrid anomaly detection. [b] proposes a method to unify the approaches of generative modelling for regular training data and discriminating with respect to negative training data for anomaly detection. This is an interesting idea yet different from the notion of familiarity hypothesis explored in our work.

Reviewer comment (Experiments):
Section 5.2 is not well argued as it is not clear that this methodology has reduced the role of the background class, as there are no experiments that measure performance with and without training with the background class. Instead, this paper simply changes the way the background class is generated.

Author response:
The key motivation here is to compare the method's performance between the SOTA background class (diffusion model to approximate normal semantic distribution) and simpler approximation (the distribution statistics are the same as the normal class with no guarantees on the semantics). There is a correlation between AD performance and the FID distance between the normal and the background used to train the model.   The two are negatively correlated until the FID is too low (normal~background). The intuition here is that the closer the background is to the normal (or more challenging the discriminative task for the encoder ), the better suited the learned feature is for AD. In the paper, we say that we hypothesize that reduced reliance on encoder derived features in our method can be the cause of this reduced reliance on the choice of background class.  

    CIFAR10 - 100 (Near ND trained with SDE)
***
    FID          400             300               200            100              50   
    AUC         82.3             83.4              86.0           88.5            89.2
Using a normal approximation as background is a simpler, and comes at much lesser computation (~zero training) giving an FID of : 523.7 across classes. For familiarity methods, using high quality approximations gives an improvement of more than 7% on the near-ND benchmark[12]. Proposed method shows the performance is consistent across the 2 evaluations showing the reduced reliance on background for AD by incorporating novelty.

The table below shows the AD performance with other standard generators for generating background for FFS. We picked the best method from this for our experiment. 

                          Baseline-VAE    Baseline-Gan       BigGAN    StyleGAN-ADA  StyleGAN-XL  DenseFlow     SDE    
    CIFAR10                 93.8             94.5             96.9         97.8         96.5        96.4       99.1    
    MyTecAD                 68.8            71.8              76.4         80.5         73.2        78.93      86.4

[11] PANDA: Adapting Pretrained Features for Anomaly Detection and Segmentation  
[12]  Fake it until you make it : towards accurate near-distribution novelty detection  
[13] Opengan: open-set recognition via open data generation

Reviewer comment (Writing):
Bad academic terminology: “the complementary set is the background B…”, it’s weird to call anomalies as background set. In the defect detection (or sensory anomalies in paper), background generally refers to normal regions. Even for semantic anomaly, it’s also weird to call anomalies as background. The description of familiar and novel features is also hard to follow.

Author response:
Thank you for highlighting the potential confusion in the definition. We have included clarification in the definition to avoid confusing background samples with background pixels within images. What we call background is different from what previous methods have called ‘training outliers’ [2], ‘outliers’ [5] [1],  irregular samples[4], or just anomalies[1]. In prior art, background has been used to denote novel distributions/classes [2][3]. We use background for denoting complimentary set not in the normal distribution, including novel semantic classes, abnormal images, and random pixel configurations. 
Furthermore, Appendix 1 shows how training on a subset of background actually can reduce test AD performance in some cases (even on SOTA-generated background), motivating not just calling them anomalies.

Reviewer comment (Evaluation):
Evaluation on sensory anomaly dataset is not enough, only MVTec. Moreover, your method performs poorly on sensory anomalies and cannot compared with the anomaly localization methods. So, just focus on the semantic anomaly, your method visual inspectable explanations are not very valuable.

Author response:
Explainability is a desirable property in AD[6], and having a method that reliably explains anomalies can be a helpful addition. ML explainability has focused on reliability of the explanation methods[11], and the need to going beyond saliency explanation[12]. A limitation of feature-KNN methods is the lack of explanation [7]. Explanation research has application beyond localizing anomalies as well. Our evaluation of sensory anomaly is the same as the benchmarks used in [1] the SOTA familiar feature-based AD method. The paper states that we do not match the sensory anomaly SOTA with dense feature matching and shows a comparison with the familiar feature KNN methods. MVTec is one of the most popular and widely used industrial anomaly datasets suited for sensory anomaly detection [8][9]. Localization papers have also used it as the only sensory AD benchmark [8].

Reviewer comment (Theory):
According to the assumption in the paper, untrained anomalies don’t belong to the familiar space, so these anomalies will be determined as false negatives? Moreover, anomaly appearances are usually different from normal, although the network don’t generate representations in the familiar space, the network will generate representations that are different from normal. So, anomalies can also be determined through familiarity branch.

Author response:
The paper does not assume that ‘untrained anomalies’ will be false negatives. [10] Present Familiarity Hypothesis: Existing AD methods succeed by detecting the presence and absence of familiar features (features the encoder has learned to represent) rather than the presence of novelty. The paper shows how this is an essential shortcoming of feature based anomaly detection. 
Anomalies represented differently from normal samples in the feature space can be captured using the familiarity branch. The novelty branch helps capture the features the encoder is not trained to represent meaningfully in the feature space.  Our experiment shows how incorporating novel features into computing anomaly scores improves AD performance, especially reducing false negatives.

Reviewer comment (Experiments):
Thus, I think you should demonstrate that kind of anomalies will be determined as false negatives by the familiarity branch, and the novelty branch can solve this. Although Figure 6 shows that the novelty branch can reduce false negative rate, this figure is likely unfair, as we don’t know what the familiarity-based method specifically refers to. It needs to your familiarity branch to be fair.

Author response:
Evaluation on different types of benchmarks show the AD performance improvement on different  'kind of anomalies'. We compare AD performance using the familiarity branch alone and after incorporating novelty across 3 AD types to show the effect of incorporating novel features into AD computation.
'Likely unfair': We have now clarified the description in section 5.2. Please note that, the figure shows how incorporating novelty to the familiarity branch reduces FNR. We use a mechanism similar to SOTA to detect the lack of familiar features[1].

Reviewer comment (Evaluation):
Figure 5 still cannot explain the effectiveness of your method. Adding novelty only increases the distance of anomalies that could be classified in the first figure, but the confusion regions in the first image still exists in the third figure.

Author response:
The figure shows two dominant axes of the PCA plot. It is shown to visualize the inner workings of incorporating novelty. The improvement in performance and the reduction in FN is the quantitative proof that incorporating novelty is effective in improving the anomaly score shown as distance in the plot.