Paper: Continual Supervised Anomaly Detection (ICLR-2024)

Reviewer comment (Evaluation):
On tabular datasets such as UNSW, bank, and credit, the model does not significantly outperform the other baselines. In many cases, the performance is similar to that of the binary classifier, suggesting that the performance is due to the binary classifier included in the method rather than the proposed method.

Author response:
As you mentioned, the difference between the performances of our method and the binary classifier is very small in Table 1. However, the performance of the binary classifier should decrease when the number of anomalies available for training becomes small because the binary classifiers are not suitable for detecting unseen anomalies. To confirm this, we conducted some experiments when the number of anomalous instances available for training was reduced to 1/5. The following are the results.

|                       | FMNIST          | MNIST           | UNSW            | bank           | credit          |
|:----------------------|:----------------|:----------------|:----------------|:---------------|:----------------|
| BC, A-GEM       | 66.28$\pm$12.41 | 64.92$\pm$7.39  | **96.69$\pm$0.53**  | 79.51$\pm$3.02 | 89.70$\pm$1.84  |
| BC, EWC         | 63.48$\pm$10.12 | 64.25$\pm$8.78  | **90.11$\pm$10.01** | **80.92$\pm$2.14** | 89.70$\pm$1.84  |
| BC, LB          | 58.63$\pm$10.12 | 52.66$\pm$9.88  | 67.60$\pm$12.09 | 74.07$\pm$1.71 | 89.64$\pm$1.75  |
| Ours           | **83.61$\pm$3.79**  | **89.37$\pm$6.09**  | **96.56$\pm$0.83**  | **82.36$\pm$0.97** | **94.75$\pm$0.69** |

We can see that while the performance of the binary classifier has dropped significantly, the performance of the proposed method has not dropped that much. The difference is especially obvious for the credit dataset, which is tabular data.

Reviewer comment (Experiments):
The experimental baselines are too simple. BC and VAE are components of the proposed method, and there are many methods that might outperform DevNet and Deep SAD, at least in the image domain (Liu et al., 2023). Many anomaly detection methods in the image domain are not designed for continual learning, but since EWC and A-GEM can be applied, it would be meaningful if the proposed method outperforms in this setting.

Author response:
To compare our method with the latest supervised anomaly detection method, we have implemented BGAD[3], mentioned by Reviewer QxJr, and conducted experiments. The experimental results are shown below.
[3] Yao X, Li R, Zhang J, et al. Explicit Boundary Guided Semi-Push-Pull Contrastive Learning for Supervised Anomaly Detection. CVPR, 2023.

|                       | FMNIST          | MNIST           | UNSW            | bank            | credit         |
|:----------------------|:----------------|:----------------|:----------------|:----------------|:---------------|
| BGAD, A-GEM     | 78.01$\pm$4.98  | 77.85$\pm$6.15  | 96.52$\pm$2.22  | **84.41$\pm$2.05** | **95.52$\pm$0.89** |
| BGAD, EWC       | 69.98$\pm$11.63 | 70.91$\pm$8.40  | 95.83$\pm$1.75  | **86.40$\pm$1.80** | **95.66$\pm$0.97** |
| BGAD, LB        | 65.91$\pm$7.16  | 63.85$\pm$7.76  | 91.33$\pm$6.33  | 72.74$\pm$4.00 | 94.39$\pm$1.93 |
| Ours        | **88.23$\pm$2.56**  | **95.46$\pm$1.64**  | **98.15$\pm$2.21**  | **86.75$\pm$2.93** | **96.12$\pm$1.22** |

Although the method uses the trained backbone model to extract the feature maps from the inputs, it was removed in the experiment above because it does not fit our paper's problem settings. This is because one of the goals of this paper is how to continuously train models from scratch from multiple tasks consisting of a small number of data, and the availability of models trained on large data sets in advance, such as backbone models, is inconsistent with this goal. The hyperparameters for BGAD basically followed its paper and GitHub implementation. We have added the details of the experiments to our manuscript. From the above results, we can see that our method outperforms the latest supervised anomaly detection methods.

Reviewer comment (Experiments):
In the image domain, the proposed method shows better performance than other baselines, but it seems that experiments on larger datasets are needed to show the practicality of the proposed method. The method was only tested on FMNIST and MNIST with MLP structure, but it would be useful to test it on larger datasets such as CIFAR10 and CelebA.

Author response:
As we mentioned in the replay to Reviewer vozw, our method is not domain-specific and is applicable to other domains, including tabular data. We conducted experiments on MNIST and FMNIST to show that the proposed method can detect anomalies even in image domains, and we believe these results are helpful enough to show that. Of course, using the proposed method on difficult image datasets would also be possible by using powerful generative models (VQ-VAE, GAN, diffusion models, etc.) instead of VAE.  A more in-depth investigation of proposed methods for specific domains is a topic for future work.

Reviewer comment (Experiments):
Similar to other continuous-learning papers, it would be nice to be able to see how performance changes with additional training on each task.

Author response:
Thank you for your advice. We will add to the paper the performance changes with additional training on each task.

Reviewer comment (Novelty):
The novelty of the proposed framework is limited. The overall network architecture consists of a VAE and a classifier without any particularly unique components.

Author response:
As you mentioned, our method consists of a binary classifier and a VAE, which are very simple and basic modules. However, this does not mean that the novelty of the proposed method is limited. In fact, it is not obvious how to combine VAE with a binary classifier and how to perform generative replay from them. Our method's novelty is not in using VAEs or binary classifiers but in finding a way to use them to solve the problem of continual supervised anomaly detection. For example, in Eqs. (7)-(9), the formulation of generating anomalies is naturally derived by appropriately combining VAE and a binary classifier. In addition, its simplicity has the advantage that it can be easily extended if necessary (e.g., by changing the generative model or the classifier).

Reviewer comment (Experiments):
This paper does not include a comparison with some of the latest supervised anomaly detection methods such as DRA[1], PRN[2], BGAD[3], which might be relevant for a more comprehensive evaluation.

Author response:
Thank you for your comment. For a more comprehensive evaluation, we have conducted the experiments with BGAD[3]. The following are the results.

|                       | FMNIST          | MNIST           | UNSW            | bank            | credit         |
|:----------------------|:----------------|:----------------|:----------------|:----------------|:---------------|
| BGAD, A-GEM     | 78.01$\pm$4.98  | 77.85$\pm$6.15  | 96.52$\pm$2.22  | **84.41$\pm$2.05** | **95.52$\pm$0.89** |
| BGAD, EWC       | 69.98$\pm$11.63 | 70.91$\pm$8.40  | 95.83$\pm$1.75  | **86.40$\pm$1.80** | **95.66$\pm$0.97** |
| BGAD, LB        | 65.91$\pm$7.16  | 63.85$\pm$7.76  | 91.33$\pm$6.33  | 72.74$\pm$4.00 | 94.39$\pm$1.93 |
| Ours        | **88.23$\pm$2.56**  | **95.46$\pm$1.64**  | **98.15$\pm$2.21**  | **86.75$\pm$2.93** | **96.12$\pm$1.22** |

Although the methods you mentioned [1-3] rely on the trained backbone model to extract the feature maps from the inputs, it was removed in the experiment above because it does not fit our paper's problem settings. This is because one of the goals of this paper is how to continually train models from scratch using multiple tasks consisting of a small number of data, and the availability of models trained on large datasets in advance, such as backbone models, is inconsistent with this goal. The hyperparameters for BGAD basically followed its paper and GitHub implementation. We have added the details of the experiments to our manuscript. From the above results, we can see that our method outperforms the latest supervised anomaly detection methods.