Paper: Deep Orthogonal Hypersphere Compression for Anomaly Detection (ICLR-2024)

Reviewer comment (Experiments):
Can the authors offer some details about the averaging of scores (table 1)? Did they conduct repeatedly the same experiment like in tables 2,3?

Author response:
**Response:** Thank you for your inquiry. Similar to the approach detailed in Tables 2 and 3, we conducted each experiment 10 times to report the experimental results (the averages and standard deviations) in Table 1.

Reviewer comment (Experiments):
For the visualization of the data, the paper pictures it 'by setting the projection dimension to 3'. Thus, was any further processing of the output applied? E.g. a dimensionality reduction technique? If yes, the AUC ROC score after this outcome may have changed from the table 3 score.

Author response:
**Response:** Thanks for the comment. To clarify, when visualizing the data by projecting it into a 3-dimensional space, we did not employ any additional dimensionality reduction techniques beyond setting the neuron number of the projection layer directly to 3. This means that the representation you see in the visualizations is a direct outcome of the neural network's projection layer, **without any further processing or transformation**. 

**Hope this response can solve your concerns. We thank the reviewer again for recognizing our work.**

Reviewer comment (Theory):
Why can the orthogonal projection layer ensure a standard hypersphere?

Author response:
Thank you for the comment. The orthogonal projection layer is designed to ensure that the latent representations are orthogonal, similar to the operation of Principal Component Analysis (PCA). By doing so, it reduces the correlation between different variables in the latent space and the variables have the same (unit) variance, which results in a more spherical distribution of samples. It was also demonstrated by our visualization experiment in the paper (see Fig. 12-15 in Appendix H). Note that ensuring uncorrelated (or even independent) and unit-variance variables (e.g. PCA, KPCA, and ICA) has shown effectiveness in novelty detection and statistical process control.

Reviewer comment (Writing):
The occurrence of the "soap bubble phenomenon" needs further clarification. Does it mean incompletely optimized?

Author response:
Thanks for reminding us to emphasize this aspect. The `soap bubble phenomenon' is not related to incomplete optimization. It is a common phenomenon of high-dimensional data. When we optimize the model, the sum of (squared) distances between all samples and the center is minimized, which is equivalent to maximum likelihood estimation (MLE), where the projected data points obey a Gaussian distribution and the mean vector is the center of the hypersphere. Even when the Gaussian assumption is realistic and the optimization is sufficient, the soap bubble phenomenon still exists due to our Proposition 1. Additionally, hypersphere compression can only ensure the sum of these distances is as small as possible, but it is challenging to guarantee that all distances are close to 0 according to the high-dimensional statistics. Thus, the 'soap bubble' is an actual issue that has been neglected by previous state-of-the-arts. We hope this explanation could address your concerns.

Reviewer comment (Evaluation):
We know that Deep SVDD compels normal data close to the center of the decision boundary, why do anomalies appear within this decision boundary? What are the main differences and similarities between normal data and these anomaly data?

Author response:
Thank you for highlighting this aspect. Deep SVDD assumes that the majority of the training data are normal instances and thus tries to minimize the volume of the hypersphere that encloses these normal data. Despite its design, anomalies can still appear within the decision boundary of Deep SVDD for several reasons: 1) Training data is insufficient or biased, so that the decision region is not compact enough for all normal data; 2) Anomalies that are similar to the normal data in certain dimensions might fall within the decision boundary. Consequently, an insufficiently compact training data distribution results in more empty holes in the decision region. These empty holes further pose a higher risk of anomalous data falling within them. The most remarkable difference is the different distributions followed by normal data and anomalous data. Even if precisely finding a perfect hypersphere decision region filled with normal data is challenging. However, the hypersphere contraction can isolate most of the abnormal data outside the decision boundary, and furthermore, the bi-hypersphere compression further enhances the compactness of the region. This is one of the motivations for proposing DO2HSC. 

In our paper, we provided an example of the anomalous data by Example 1. Suppose the input data is zero or very close to zero (or has many zero attributes), then after the projection of Deep SVDD, the data is at or close to the center of the hypersphere (origin). Although it is in the hypersphere, it is very different from the normal training data and should be treated as abnormal data. Deep SVDD failed to detect it. Real examples can be found in Figure 4.

Reviewer comment (Presentation):
Some details need to be double-checked, such as the bolding of three results in Table 13, Class 1 of MUTAG result, while the caption specifies marking only the best two results.

Author response:
Thank you for pointing this out. We have revised these points and ensured consistency. We appreciate your attention to detail and your help in improving the accuracy and clarity of our research. The corrected version of Table 13 is updated in the revised manuscript.

**Hope this response can solve your concerns. We thank the reviewer again for recognizing our work.**

Reviewer comment (Theory):
Figure 2 shows that the orthogonal projection improves the performance of anomaly detection. What is the fundamental reason? I suggest the authors provide further analysis as well as some references if possible.

Author response:
First, if there is no orthogonal project, we may encounter an inconsistency between the hypersphere assumption and the actual (optimal) decision boundary (e.g., an ellipsoid), which stems from the following two points: 1) the learned features have different variances and 2) the learned features are correlated. For instance, in the left plot of Fig. 2, the assumption and the anomaly score are based on the hypersphere but the actual decision boundary is an ellipsoid. Comparing the right plot with the left one reveals that the left plot has a lower true positive (TP) ratio  but a larger false positive (FP) ratio, which leads to a lower detection precision:

$\frac{TP\downarrow}{TP\downarrow+FP\uparrow}$. Here we regarded normal data as the negative class and abnormal data as the positive class. Therefore, we propose to use the orthonormal projection. This refinement contributes to an improved performance. The idea is similar to that of applying PCA to fault detection in industrial processes [1].

[1] T. Kourti and J. F. MacGregor, “Process analysis, monitoring and diagnosis, using multivariate projection methods,” Chemometrics Intell. Lab. Syst., vol. 28, no. 1, pp. 3–21, 1995.

Reviewer comment (Writing):
A typo or grammar issue in the first paragraph of Section 2.1.2: ‘cannot be avoided by solving equation 1’, it is not an equation. It is an optimization problem.

Author response:
Thank you for pointing out this typo. We have revised it.

Reviewer comment (Theory):
Does Proposition 2 mean the distance (to the original or hypersphere center) based anomaly score in high-dimensional space are not reliable? If yes, the authors may add a few words to the last paragraph in Section 2.2.1 to provide a hint or motivation for the new anomaly score defined by (9).

Author response:
We admit that the distance-based measures to the center (be it the original or hypersphere center) can be less reliable for anomaly detection. This is a consequence of the high-dimensional geometry and the behavior of distances in such spaces.
We would add a statement that acknowledges the limitations of traditional distance-based anomaly scores in high-dimensional spaces, as highlighted by Proposition 2, and motivates the need for the new anomaly score. The added contents are shown as follows:
"Given the implications of Proposition 2, we recognize that in high-dimensional spaces, traditional distance-to-center based anomaly scores may lose their reliability due to the concentration of measure phenomenon." This insight motivates our proposal of a new anomaly score as defined in Equation (9), which aims to address these high-dimensional challenges more effectively.

Reviewer comment (Theory):
Given Proposition 2, in the high-dimensional space, the normal data are already far away from the origin. Why do we need to further push them to the outer hypersphere, namely, performing the hypersphere compression to reduce the thickness of the shell?

Author response:
We appreciate your concern. The primary goal of hypersphere compression is not just to push normal data further away from the center, but rather to reduce the thickness of the shell in which the normal data reside. This is particularly crucial in high-dimensional spaces where data points, including both normal and anomalous, tend to be equidistant from the center due to the curse of dimensionality.

As Proposition 2 suggests, normal data are indeed farther away from the center as the dimensionality grows. However, the challenge lies in the relative positioning of normal and anomalous data. Without hypersphere compression, both normal and anomalous data might occupy a broad shell-like region, making it difficult to distinguish between them effectively.

By employing hypersphere compression, we effectively reduce the volume of the space where normal data reside. This compression increases the contrast between normal and anomalous data, as anomalous data will now be more likely to fall outside this compressed hypersphere. This enhanced separation improves the sensitivity and specificity of the anomaly detection process.

Reviewer comment (Reproducibility):
Are \(r_{max}\) and \(r_{min}\) fixed or adjusted adaptively?

Author response:
$r_{\text{max}}$ and $r_{\text{min}}$ remain fixed throughout the entire training process after they are initialized by formula (7). This approach ensures that all normal data are optimized to lie within the interval area bounded by the bi-hyperspheres.

Reviewer comment (Theory):
In Proposition 3, when \(r_{min} = r_{max}\), \(\kappa\) is infinity. Does this still make sense?

Author response:
Thanks for mentioning this point. In the case of $r_{\text{min}}=r_{\text{max}}$, the volume of DO2HSC is close to an infinitely thin shell, essentially transforming into the surface of a hypersphere. In this scenario, the data density of DO2HSC is obviously higher compared with that of DOHSC. However, it's important to note that this situation is quite rare, particularly in high-dimensional space. We have supplemented this special situation in Appendix D.

Reviewer comment (Novelty):
What make graph anomaly detection special compared to image and tabular data anomaly detection?

Author response:
Graph data, compared to other types of data, is inherently complex and rich in structural and relational information. Achieving a global representation for a graph involves harmonizing information across all nodes, while node representation further necessitates aggregating information from connected neighbors through edges. Consequently, it encompasses both independent node details and structural information. The crucial consideration is preserving a meaningful graph representation that is conducive to the anomaly detection task. This motivates our choice to maximize the mutual information among all nodes while simultaneously detecting anomalies.

Reviewer comment (Experiments):
In Section 3.1, the authors mentioned a comparison method FCDD, but Table 2 doesn’t include the corresponding result.

Author response:
Thanks for your careful review. We originally wanted to include FCDD as a baseline but found that FCDD uses additional data in the training, which means FCDD cannot be compared with our methods as well as the baselines considered in our paper. So we didn't include the results in Table 2. This is our neglect and we have corrected it in the revised paper.

Reviewer comment (Experiments):
In Appendix K, the authors showed the results of imbalanced experiments of graph data. Does it mean the results on graph data in the main paper are from balanced experiments? What is the difference between these two settings?

Author response:
The results on graph data in the main paper adhere to balanced settings. The data split involves allocating 80\% of the data from the normal class for training. The testing data is then constructed by combining the retained normal data with an equal or smaller number of anomalous data samples. Recognizing the significance of imbalance settings in practical anomaly detection scenarios, we follow the same data split for training and adjust the test dataset to a 10:1 ratio.

**Hope this response can solve your concerns. We thank the reviewer again for recognizing our work.**

Reviewer comment (Theory):
This article mentions the concept of hyperspheres but doesn't provide a more rigorous theoretical explanation for why standard hyperspheres are superior to boundaries formed by ellipsoids. Adding mathematical proofs or a deeper theoretical foundation would strengthen the paper.

Author response:
Thank you for your insightful feedback. Actually, our paper does not involve the comparison between the standard hypersphere decision boundary and the ellipsoid decision boundary. We aim to address the inconsistency between the hypersphere assumption and the actual decision boundary of classical methods such as Deep SVDD. To be more precise, Deep SVDD assumes that, after the neural network transformation, the normal data are included in a standard hypersphere and use the distance of a data point to the center of the hypersphere as the anomaly score. However, the optimal decision boundary given by the neural network is often not a standard hypersphere, and can be an ellipsoid. The reason is that the optimization of Deep SVDD cannot ensure that the variables in the latent space are uncorrelated and have unit variance. The inconsistency between the computation of the anomaly score (assumption) and the actual optimal decision boundary (reality) degrades the detection accuracy. For a concrete example, please refer to Figure 2 in our paper. In the left plot, the black ellipse denotes the data distribution learnt by the encoder. However, hypersphere-based anomaly detection methods adopt the blue circle as the final evaluated decision boundary. This leads to more false positive samples (FP) and fewer true positive samples (TP), compared to the right plot, where we regarded normal data (purple) as the negative class and abnormal data (red) as the positive class. The detection precision, which is calculated as $\frac{TP\downarrow}{TP\downarrow+FP\uparrow}$, eventually decreases. We have incorporated these explanations into the revised manuscript to provide a more comprehensive understanding of our approach and its theoretical underpinnings.

Reviewer comment (Theory):
The proof of Proposition 2 in section C of supplementary materials should be made more clear.

Author response:
Thanks for your interest in the proof of Proposition 2. In the proof for Proposition 2, we start by leveraging the result from Proposition 1 and the properties of the function $f$, which is $\eta$-Lipschitz. The central idea is to connect the bounds on the norm of $\mathbf{s}-\bar{\mathbf{c}}$ with those on $\mathbf{z}-\mathbf{c}$, using the Lipschitz condition. We have added a step-by-step explanation to the manuscript, which aids in making the logical flow more apparent.

Reviewer comment (Theory):
In equation 3, how to guarantee the projected embeddings is orthogonal via singular value decomposition?

Author response:
Thanks for mentioning this point. The singular value decomposition can be formulated 
$$\mathbf{U}\mathbf{\Lambda}\mathbf{V}^\top =\mathbf{Z},$$
where $\mathbf{U}$ contains a set of vectors orthogonal to each other. In our paper, the projection matrix is $\mathbf{W}=\mathbf{V}\_{k'} \mathbf{\Lambda}\_{k'}^{-1}$, where $\mathbf{V}\_{k'}$ consists of the first $k'$ columns of $\mathbf{V}$. It follows that $\tilde{\mathbf{Z}}=\mathbf{Z}\mathbf{W}=\mathbf{U\Lambda V}^\top\mathbf{V}\_{k'} \mathbf{\Lambda}\_{k'}^{-1}=\mathbf{U}\_{k'}$, where $\mathbf{U}\_{k'}$ is composed of the first $k'$ columns of $\mathbf{U}$. Therefore, the columns of the embeddings $\tilde{\mathbf{Z}}$ are orthogonal naturally, namely $\tilde{\mathbf{Z}}^\top\tilde{\mathbf{Z}}=\mathbf{U}\_{k'}^\top\mathbf{U}\_{k'}=\mathbf{I}\_{k'}$.