Paper: ImAD: An End-to-End Method for Unsupervised Anomaly Detection in the Presence of Missing Values (ICLR-2024)

Reviewer comment (Experiments):
Section 2.2 introduces a few related studies on anomaly detection with incomplete data, but these methods are not included in the experimental evaluation. The results primarily rely on conventional approaches that do not explicitly account for missing data, such as DeepSVDD.

Author response:
Thanks for your comments. Actually, the related methods ([Zemicheal & Dietterich 2019], [Fan et al. 2022], and [Sarda et al. 2023]) discussed in Section 2.2 are all two-stage methods: perform missing data imputation first and then conduct anomaly detection. A few two-stage methods are considered in our paper. We used two strong missing data imputation methods including MissForest [1] and GAIN [2] and three strong anomaly detection methods including IForest [3], Deep SVDD [4] and NeutraL AD [5]. Our end-to-end method outperformed all two-stage methods.

References:

[1] Stekhoven, Daniel J., and Peter Bühlmann. "MissForest—non-parametric missing value imputation for mixed-type data." Bioinformatics 28.1 (2012): 112-118.

[2] Yoon, Jinsung, James Jordon, and Mihaela Schaar. "Gain: Missing data imputation using generative adversarial nets." International conference on machine learning. PMLR, 2018.

[3] Liu, Fei Tony, Kai Ming Ting, and Zhi-Hua Zhou. "Isolation forest." 2008 eighth ieee international conference on data mining. IEEE, 2008.

[4] Ruff, Lukas, et al. "Deep one-class classification." International conference on machine learning. PMLR, 2018.

[5] Qiu, Chen, et al. "Neural transformation learning for deep anomaly detection beyond images." International Conference on Machine Learning. PMLR, 2021.

Reviewer comment (Experiments):
Experimental scenarios are too limited. Only two datasets are tested, while there exist many other benchmark datasets available for anomaly detection. The choice of two specific missing rates (0.2 and 0.5) may not realistically represent the practical scenarios. Only AUROC results are presented, omitting other essential metrics.

Author response:
We conducted experiment on **five** more publicly-available real-world datasets and added AUPRC as a new evaluation metric. For the selection of missing rates (0.2 and 0.5), we followed the previous data imputation works [1,2,3].

References:

[1] Yoon, Jinsung, James Jordon, and Mihaela Schaar. "Gain: Missing data imputation using generative adversarial nets." International conference on machine learning. PMLR, 2018.

[2] Camino, Ramiro D., Christian A. Hammerschmidt, and Radu State. "Improving missing data imputation with deep generative models." arXiv preprint arXiv:1902.10666 (2019).

[3] Li, Yuxuan, Ayse Dogan, and Chenang Liu. "Ensemble Generative Adversarial Imputation Network with Selective Multi-Generator (ESM-GAIN) for Missing Data Imputation." 2022 IEEE 18th International Conference on Automation Science and Engineering (CASE). IEEE, 2022.

Reviewer comment (Novelty):
Although this paper includes some theoretical analysis in section 3.6, the derivations of the inequalities appear to be straightforward in the context of plain neural nets. It remains unclear whether these inequalities provide more valuable insights than those derived from other related methods, especially on practical utility.

Author response:
Please take a look at Figure 4 of our latest manuscript. In Figure 4, we visualize the projection results of Botnet dataset in 2-D space. It can be observed that the majority of normal training and testing samples are mapped into the target distribution while most abnormal samples fall outside of the decision boundary. This demonstrates that our method is practical and Theorem 3.1 is effective for real-world scenarios.

Reviewer comment (Experiments):
In Figure 1, OC-SVM outperforms IForest across different missing rates, but OC-SVM is excluded in other experiments. Can you provide a rationale for this selectiveness?

Author response:
First of all, in our main experiments, our method achieves better detection performance than the strong baselines Deep SVDD [1] and NeutraL AD [2] and both of them outperformed OC-SVM in their previous works. Therefore, OC-SVM is not a necessary baseline for our work. The second reason is that, our experiments include diverse experimental settings including different missing rates, three missing mechanisms, different data splitting, and different combinations of imputation and detection methods and OC-SVM has quadratic time and space complexities (very slow on large datasets such as KDD). Therefore, we do not include OC-SVM in our experiments.

References: 

[1] Ruff, Lukas, et al. "Deep one-class classification." International conference on machine learning. PMLR, 2018.

[2] Qiu, Chen, et al. "Neural transformation learning for deep anomaly detection beyond images." International Conference on Machine Learning. PMLR, 2021.

Reviewer comment (Evaluation):
Can you provide results using additional performance metrics such as AUPRC?

Author response:
Yes, we add AUPRC as new metric on all seven datasets and related results are provided in our overall response.

Reviewer comment (Experiments):
It would be interesting to see some results on the robustness in relation to the values of r1 and r2.

Author response:
For your concern on radii $r_1, r_2$, we conduct related experiments on Arrhythmia dataset by changing $r_1, r_2$ and experimental details and results are provided in the following tables.

As showed in the following table, we change the dimension $d$ of latent space and then get $r = \sigma \sqrt{F^{-1}_{d}(p)}$ (See Proposition A.1).

Simultaneously we keep consistent with the experiment setting in our manuscripts (See Appendix F) and get 
\begin{equation}
r_1 = 0.5 \sqrt{F^{-1}_{d}(0.9)}
\end{equation} 

\begin{equation}
r_2 = \sqrt{F^{-1}_{d}(0.9)}
\end{equation} 
\begin{matrix}
    \hline
    Radius & & & Latent Dimension (d) & &  \\\\
\hline
    & 16 & 32 & 64 & 128 & 256 \\\\
    \hline
  r_1 & 2.42 & 3.26 & 4.44 & 6.10 & 8.45 \\\\
    \hline
    r_2 & 4.85 & 6.52 & 8.88 & 12.20 & 16.90 \\\\
\hline
\end{matrix}

The following table shows the average AUROC(\%) and AUPRC(\%) on Arrhythmia dataset when changing the radii $r_1, r_2$. It can be observed that our method is not quite sensitive to the changes of radii $r_1, r_2$ and the performance declines with the decrease of the latent dimension, which is reasonable since there will be more information loss when the latent dimension becomes smaller. 
\begin{matrix}
    \hline
    \text{Radius} & \text{Radius}  & \text{AUROC(\\%)} & \text{AUROC(\\%)} & \text{AUPRC(\\%)} & \text{AUPRC(\\%)}\\\\
    \hline
    r_1 & r_2& p=0.2 & p=0.5 & p=0.2 & p=0.5 \\\\
    \hline
    2.42 & 4.85 & 80.83 & 78.59 & 79.75 & 77.51 \\\\
    & & (1.91) & (1.34) & (1.74) & (2.39) \\\\
    \hline
    3.26 & 6.52 & 80.06 & 79.37 & 81.06 & 79.19 \\\\
    & & (2.00) & (0.56) & (0.87) & (2.12) \\\\
    \hline
    4.44 & 8.88 & 81.61 & 80.30 & 80.81 & 80.29\\\\
    & & (1.05) & (1.21) & (2.06) & (1.09) \\\\
    \hline
    6.10 & 12.20 & 82.24 & 81.76 & 83.74 & 83.37\\\\
    & & (1.76) & (1.19) & (1.85) & (1.36) \\\\
    \hline
    8.45 & 16.90 & 81.74 & 79.53 & 81.68 & 80.17\\\\
    & & (2.05) & (0.73) & (2.80) & (1.05) \\\\
    \hline
\end{matrix}

Reviewer comment (Evaluation):
Figure 1 uses AUROC on y-axis whereas for anomaly detection I think AUPR is better to motivate as most anomaly detection techniques have poor precision. Also Adult and KDD are UCI datasets which do not have a lot of relevance today as most techniques perform superlatively on these datsets. Given that the framework is applicable for missing values, it makes more sense to benchmark/motivate on real world use cases for anomaly detection (IoT sensor data, etc)

Author response:
Following your concern, we conduct experiments on **five** new real-world datasets and add AUPRC as a new evaluation metric. Detailed data information and experimental results can be found in our overall response.  The KDD and Adult datasets are important benchmarks frequently used in previous anomaly detection works [1,2,3,4,5] and we hence included them to improve the persuasiveness.

References:

[1] Zong, Bo, et al. "Deep autoencoding gaussian mixture model for unsupervised anomaly detection." International conference on learning representations. 2018.

[2] Lahoti, Preethi, et al. "Fairness without demographics through adversarially reweighted learning." Advances in neural information processing systems 33 (2020): 728-740.

[3] Qiu, Chen, et al. "Neural transformation learning for deep anomaly detection beyond images." International Conference on Machine Learning. PMLR, 2021.

[4] Buyl, Maarten, and Tijl De Bie. "Optimal transport of classifiers to fairness." Advances in Neural Information Processing Systems 35 (2022): 33728-33740.

[5] Han, Xiao, et al. "Achieving Counterfactual Fairness for Anomaly Detection." Pacific-Asia Conference on Knowledge Discovery and Data Mining. Cham: Springer Nature Switzerland, 2023.

Reviewer comment (Presentation):
Section 3.5 seems way too short and not very insightful with the purpose of just being added in the paper to have such a section in the main paper.

Author response:
For target distribution 
$
\mathcal{D}_{\mathbf{z}} \sim \mathcal{N}(\mathbf{0}, \sigma^2\mathbf{I}_d) 
$,
we need to a constrained sampling radius $r$ to get corresponding truncated Gaussian distribution. Section 3.5 gives the lower bound of the constrained sampling radius when given a sampling probability $p$.

Reviewer comment (Evaluation):
Also on a more real-world applicability note, imputation is not often used within industry and missing data if any is more often discarded. I would encourage authors to think of very strong real-world insights of when imputation should be used and when should it be avoided.

Author response:
In fact, in many real-world scenarios, data imputation is necessary and crucial. For instance, in rare disease diagnosis and fault detection of sophisticated equipments, we cannot discard the incomplete samples and we have to make decisions based on the incomplete samples in time. In recommendation systems especially collaborative filtering problems, samples (e.g. users) with missing values commonly cannot be thrown away. In chemical processes (e.g. [1]), we need to conduct fault detection for each sample to ensure product quality and system safety. We appreciate your suggestion and enhanced the motivation in the section of Introduction.

Reference:

[1] Fan et al. Kernel-based statistical process monitoring and
fault detection in the presence of missing data. IEEE Transactions on Industrial Informatics, 18
(7):4477–4487, 2022.

Reviewer comment (Experiments):
I would like to see stronger results on AUPR for high dimensional real-world datasets which actually have a lot of missing values. Some real-world datasets from IoT sensor domains or others will definitely help in improving contributions of this paper. Some other questions are mentioned in the weakness section above.

Author response:
Yes, we have added more datasets to the experiments and included AUPRC. Our method outperformed all baselines in almost all cases in terms of both AUPRC and AUROC.

Reviewer comment (Experiments):
The current scope of experiments is somewhat limited, and it is strongly recommended to expand the experimental evaluation to include a broader range of datasets. This will facilitate a more comprehensive validation of the efficacy of the proposed method.

Author response:
Thanks for your well-intentioned suggestion. We added five new real-world datasets to the experiments and one more evaluation metric AUPRC. All experimental results are provided in our overall response and the new experimental results are complemented to section of Experiments in our latest manuscript.

Reviewer comment (Theory):
The assumption of a Gaussian distribution in the latent space could potentially impose limitations on the practical applicability of the proposed method. Real-world datasets with inherent complexity may not fit into a single Gaussian distribution, raising concerns about the generalizability of the approach to such intricate data scenarios.

Author response:
This is really an insightful comment and we absolutely agree with you. Indeed, the real-world data scenarios could be more complex and we may consider multiple Gaussians. 

We also thought about projecting the normal data into multiple truncated Gaussian distributions but there are two difficulties. First, the number of classes in normal data is usually unknown, which makes it difficult to determine the number of Gaussians. Second, the spatial locations of multiple Gaussians or hyperballs are not easy to determine.

On the other hand, if the structure of real data is not very complex, a sufficiently large neural network is able to transform the unknown data distribution to any target distribution such as a single Gaussian, which is similar to the feasibility of GAN (Goodfellow et al. 2014) converting a single Gaussian distribution to any data distribution.
In unsupervised anomaly detection, the most important thing is to find a reliable decision boundary using the normal samples. Therefore, the target distribution is supposed to be compact, which ensures that normal data would lie in high-density regions in the latent space and we can obtain a reliable decision boundary easily. 

We appreciate your comments.

Reviewer comment (Theory):
How can we ensure that the generated pseudo abnormal samples are similar to real abnormal samples? Because the distribution of pseudo abnormal samples in latent space is assumed, it can differ significantly from that of real abnormal samples. This difference can lead to strange imputation.

Author response:
This is a great question. Actually, there is no need to ensure that the generated pseudo-abnormal samples are very similar to real abnormal samples. We only want to ensure that the generated pseudo-abnormal samples closely surround the normal samples in the original data space, which means the generated pseudo-abnormal samples interpolate between the normal region and the abnormal region in the original data space and hence form a meaningful decision boundary. This goal can be achieved by our design of the distributions in the latent space and the Lipschitz-continuity of the neural network. Specifically, as shown by Figure 2 of our paper, in the latent space, the pseudo-abnormal samples surround the normal samples. According to our Theorem 3.1(a), in the original data space, the pseudo-abnormal samples still surround the normal samples.

Please also take a look at Figure 4 of our latest manuscript. We visualize the projection results of Botnet dataset in 2-D space. It can be observed that the majority of normal training and testing samples are mapped into the target distribution while most abnormal samples fall outside of the decision boundary. This demonstrates that our method is practical and Theorem 3.1 is effective for real-world scenarios.

Reviewer comment (Experiments):
When dealing with a dataset of very high dimensionality, is the proposed method scalable? Will the assumption in generating pseudo-abnormal samples still hold?

Author response:
Our method projects the original high-dimensional data into the target distribution, which can reduce the dimensionality significantly, and the anomaly score is defined in the low-dimensional latent space. Therefore, our method is scalable to high-dimensional data. In this revision, the new datasets Segerstolpe and Usoskin especially have high-dimension and our method still outperformed all baselines significantly.

Reviewer comment (Writing):
There is an excessive use of similar symbols placed above characters, which can impede readability.

Author response:
Thanks for pointing it out. We enhanced the related descriptions and made them easy to read and understand.