Original: D EEP A UTOENCODING G AUSSIAN M IXTURE M ODEL FOR U NSUPERVISED A NOMALY D ETECTION
Revised: D EEP A UTOENCODING G AUSSIAN M IXTURE M ODEL FOR U NSUPERVISED A NOMALY D ETECTION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Unsupervised anomaly detection on multi- or high-dimensional data is of great importance in both fundamental machine learning research and industrial applications, for which density estimation lies at the core.
Revised: Unsupervised anomaly detection on multi- or high-dimensional data is of great importance in both fundamental machine learning research and industrial applications, for which density estimation lies at the core.

Original: Although previous approaches based on dimensionality reduction followed by density estimation have made fruitful progress, they mainly suffer from decoupled model learning with inconsistent optimization goals and incapability of preserving essential information in the low-dimensional space.
Revised: Although previous approaches based on dimensionality reduction followed by density estimation have made fruitful progress, they mainly suffer from decoupled model learning with inconsistent optimization goals and incapability of preserving essential information in the low-dimensional space.

Original: In this paper, we present a Deep Autoencoding
Revised: In this paper, we present a Deep Autoencoding

Original: Gaussian Mixture Model (DAGMM) for unsupervised anomaly detection.
Revised: Gaussian Mixture Model (DAGMM) for unsupervised anomaly detection.

Original: Our model utilizes a deep autoencoder to generate a low-dimensional representation and reconstruction error for each input data point, which is further fed into a Gaussian
Revised: Our model utilizes a deep autoencoder to generate a low-dimensional representation and reconstruction error for each input data point, which is further fed into a Gaussian

Original: Mixture Model (GMM).
Revised: Mixture Model (GMM).

Original: Instead of using decoupled two-stage training and the standard Expectation-Maximization (EM) algorithm, DAGMM jointly optimizes the parameters of the deep autoencoder and the mixture model simultaneously in an end-to-end fashion, leveraging a separate estimation network to facilitate the parameter learning of the mixture model.
Revised: Instead of using decoupled two-stage training and the standard Expectation-Maximization (EM) algorithm, DAGMM jointly optimizes the parameters of the deep autoencoder and the mixture model simultaneously in an end-to-end fashion, leveraging a separate estimation network to facilitate the parameter learning of the mixture model.

Original: The joint optimization, which well balances autoencoding reconstruction, density estimation of latent representation, and regularization, helps the autoencoder escape from less attractive local optima and further reduce reconstruction errors, avoiding the need of pre-training.
Revised: The joint optimization, which well balances autoencoding reconstruction, density estimation of latent representation, and regularization, helps the autoencoder escape from less attractive local optima and further reduce reconstruction errors, avoiding the need of pre-training.

Original: Experimental results on several public benchmark datasets show that, DAGMM signiﬁcantly outperforms state-of-the-art anomaly detection techniques, and achieves up to 14% improvement based on the standard F 1 score.
Revised: Experimental results on several public benchmark datasets show that, DAGMM signiﬁcantly outperforms state-of-the-art anomaly detection techniques, and achieves up to 14% improvement based on the standard F 1 score.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Unsupervised anomaly detection is a fundamental problem in machine learning, with critical applications in many areas, such as cybersecurity (Tan et al.
Revised: Unsupervised anomaly detection is a fundamental problem in machine learning, with critical applications in many areas, such as cybersecurity (Tan et al.

Original: (2011)), complex system management (Liu et al.
Revised: (2011)), complex system management (Liu et al.

Original: (2008)), medical care (Keller et al.
Revised: (2008)), medical care (Keller et al.

Original: (2012)), and so on.
Revised: (2012)), and so on.

Original: At the core of anomaly detection is density estimation: given a lot of input samples, anomalies are those ones residing in low probability density areas.
Revised: At the core of anomaly detection is density estimation: given a lot of input samples, anomalies are those ones residing in low probability density areas.

Original: Although fruitful progress has been made in the last several years, conducting robust anomaly detection on multi- or high-dimensional data without human supervision remains a challenging task.
Revised: Although fruitful progress has been made in the last several years, conducting robust anomaly detection on multi- or high-dimensional data without human supervision remains a challenging task.

Original: Especially, when the dimensionality of input data becomes higher, it is more difﬁcult to perform density estimation in the original feature space, as any input sample could be a rare event with low probability to observe (Chandola et al.
Revised: Especially, when the dimensionality of input data becomes higher, it is more difﬁcult to perform density estimation in the original feature space, as any input sample could be a rare event with low probability to observe (Chandola et al.

Original: To address this issue caused by the curse of dimensionality, two-step approaches are widely adopted (Cand`es et al.
Revised: To address this issue caused by the curse of dimensionality, two-step approaches are widely adopted (Cand`es et al.

Original: (2011)), in which dimensionality reduction is ﬁrst conducted, and then density estimation is performed in the latent low-dimensional space.
Revised: (2011)), in which dimensionality reduction is ﬁrst conducted, and then density estimation is performed in the latent low-dimensional space.

Original: However, these approaches could easily lead to suboptimal performance, because dimensionality reduction in the ﬁrst step is unaware of the subsequent density estimation task, and the key information for anomaly detection could be removed in the ﬁrst place.
Revised: However, these approaches could easily lead to suboptimal performance, because dimensionality reduction in the ﬁrst step is unaware of the subsequent density estimation task, and the key information for anomaly detection could be removed in the ﬁrst place.

Original: Therefore, it is desirable to combine the force of dimensionality reduction and density estimation, although a joint optimization accounting for these two components is usually computationally difﬁcult.
Revised: Therefore, it is desirable to combine the force of dimensionality reduction and density estimation, although a joint optimization accounting for these two components is usually computationally difﬁcult.

Original: Several recent works (Zhai et al.
Revised: Several recent works (Zhai et al.

Original: ; Yang et al.
Revised: ; Yang et al.

Original:  ; Xie et al. (2016)) explored this direction by utilizing the strong modeling capacity of deep networks, but the resulting performance is limited either by a reduced low-dimensional space that is unable to preserve essential information of input samples, an over-simpliﬁed density estimation model without enough capacity, or a training strategy that does not ﬁt density estimation tasks.
Revised: (2017a) ; Xie et al. (2016)) explored this direction by utilizing the strong modeling capacity of deep networks, but the resulting performance is limited either by a reduced low-dimensional space that is unable to preserve essential information of input samples, an over-simpliﬁed density estimation model without enough capacity, or a training strategy that does not ﬁt density estimation tasks.

Original: In this paper, we propose Deep Autoencoding Gaussian Mixture Model (DAGMM), a deep learning framework that addresses the aforementioned challenges in unsupervised anomaly detection from several aspects.
Revised: In this paper, we propose Deep Autoencoding Gaussian Mixture Model (DAGMM), a deep learning framework that addresses the aforementioned challenges in unsupervised anomaly detection from several aspects.

Original: First, DAGMM preserves the key information of an input sample in a low-dimensional space that includes features from both the reduced dimensions discovered by dimensionality reduction and the induced reconstruction error.
Revised: First, DAGMM preserves the key information of an input sample in a low-dimensional space that includes features from both the reduced dimensions discovered by dimensionality reduction and the induced reconstruction error.

Original: From the example shown in Figure 1, we can see that anomalies differ from normal samples in two aspects: (1) anomalies can be signiﬁcantly deviated in the reduced dimensions where their features are correlated in a different way; and (2) anomalies are harder to reconstruct, compared with normal samples.
Revised: From the example shown in Figure 1, we can see that anomalies differ from normal samples in two aspects: (1) anomalies can be signiﬁcantly deviated in the reduced dimensions where their features are correlated in a different way; and (2) anomalies are harder to reconstruct, compared with normal samples.

Original: Unlike existing methods that only involve one of the aspects (Zimek et al.
Revised: Unlike existing methods that only involve one of the aspects (Zimek et al.

Original: (2012); Zhai et al.
Revised: (2012); Zhai et al.

Original: (2016)) with sub-optimal performance
Revised: (2016)) with sub-optimal performance

Original: , DAGMM utilizes a sub-network called compression network to perform dimensionality reduction by an autoencoder, which prepares a low-dimensional representation for an input sample by concatenating reduced low-dimensional features from encoding and the reconstruction error from decoding.
Revised: , DAGMM utilizes a sub-network called compression network to perform dimensionality reduction by an autoencoder, which prepares a low-dimensional representation for an input sample by concatenating reduced low-dimensional features from encoding and the reconstruction error from decoding.

Original: Second, DAGMM leverages a Gaussian Mixture Model (GMM) over the learned low-dimensional space to deal with density estimation tasks for input data with complex structures, which are yet rather difﬁcult for simple models used in existing works (Zhai et al.
Revised: Second, DAGMM leverages a Gaussian Mixture Model (GMM) over the learned low-dimensional space to deal with density estimation tasks for input data with complex structures, which are yet rather difﬁcult for simple models used in existing works (Zhai et al.

Original: While GMM has strong capability, it also introduces new challenges in model learning.
Revised: While GMM has strong capability, it also introduces new challenges in model learning.

Original: As GMM is usually learned by alternating algorithms such as Expectation-Maximization (EM) (Huber (2011)), it is hard to perform joint optimization of dimensionality reduction and density estimation favoring GMM learning, which is often degenerated into a conventional two-step approach.
Revised: As GMM is usually learned by alternating algorithms such as Expectation-Maximization (EM) (Huber (2011)), it is hard to perform joint optimization of dimensionality reduction and density estimation favoring GMM learning, which is often degenerated into a conventional two-step approach.

Original: To address this training challenge,
Revised: To address this training challenge,

Original: DAGMM utilizes a sub-network called estimation network that takes the low-dimensional input from the compression network and outputs mixture membership prediction for each sample.
Revised: DAGMM utilizes a sub-network called estimation network that takes the low-dimensional input from the compression network and outputs mixture membership prediction for each sample.

Original: With the predicted sample membership, we can directly estimate the parameters of GMM, facilitating the evaluation of the energy/likelihood of input samples.
Revised: With the predicted sample membership, we can directly estimate the parameters of GMM, facilitating the evaluation of the energy/likelihood of input samples.

Original: By simultaneously minimizing reconstruction error from compression network and sample energy from estimation network, we can jointly train a dimensionality reduction component that directly helps the targeted density estimation task.
Revised: By simultaneously minimizing reconstruction error from compression network and sample energy from estimation network, we can jointly train a dimensionality reduction component that directly helps the targeted density estimation task.

Original: Finally, DAGMM is friendly to end-to-end training.
Revised: Finally, DAGMM is friendly to end-to-end training.

Original: Usually, it is hard to learn deep autoencoders by end-to-end training, as they can be easily stuck in less attractive local optima, so pre-training is widely adopted (Vincent et al.
Revised: Usually, it is hard to learn deep autoencoders by end-to-end training, as they can be easily stuck in less attractive local optima, so pre-training is widely adopted (Vincent et al.

Original: ; Yang et al.
Revised: ; Yang et al.

Original:  ; Xie et al. (2016)).
Revised: (2017a) ; Xie et al. (2016)).

Original: However, pre-training limits the potential to adjust the dimensionality reduction behavior because it is hard to make any signiﬁcant change to a well-trained autoencoder via ﬁne-tuning.
Revised: However, pre-training limits the potential to adjust the dimensionality reduction behavior because it is hard to make any signiﬁcant change to a well-trained autoencoder via ﬁne-tuning.

Original: Our empirical study demonstrates that, DAGMM is well-learned by the end-to-end training, as the regularization introduced by the estimation network greatly helps the autoencoder in the compression network escape from less attractive local optima.
Revised: Our empirical study demonstrates that, DAGMM is well-learned by the end-to-end training, as the regularization introduced by the estimation network greatly helps the autoencoder in the compression network escape from less attractive local optima.

Original: Experiments on several public benchmark datasets demonstrate that, DAGMM has superior performance over state-of-the-art techniques, with up to 14% improvement of F1 score for anomaly detection.
Revised: Experiments on several public benchmark datasets demonstrate that, DAGMM has superior performance over state-of-the-art techniques, with up to 14% improvement of F1 score for anomaly detection.

Original: Moreover, we observe that the reconstruction error from the autoencoder in DAGMM by the end-to-end training is as low as the one made by its pre-trained counterpart, while the reconstruc- tion error from an autoencoder without the regularization from the estimation network stays high.
Revised: Moreover, we observe that the reconstruction error from the autoencoder in DAGMM by the end-to-end training is as low as the one made by its pre-trained counterpart, while the reconstruction error from an autoencoder without the regularization from the estimation network stays high.

Original: In addition, the end-to-end trained DAGMM signiﬁcantly outperforms all the baseline methods that rely on pre-trained autoencoders.
Revised: In addition, the end-to-end trained DAGMM signiﬁcantly outperforms all the baseline methods that rely on pre-trained autoencoders.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Tremendous effort has been devoted to unsupervised anomaly detection (Chandola et al.
Revised: Tremendous effort has been devoted to unsupervised anomaly detection (Chandola et al.

Original: (2009)), and the existing methods can be grouped into three categories.
Revised: (2009)), and the existing methods can be grouped into three categories.

Original: Reconstruction based methods assume that anomalies are incompressible and thus cannot be effectively reconstructed from low-dimensional projections.
Revised: Reconstruction based methods assume that anomalies are incompressible and thus cannot be effectively reconstructed from low-dimensional projections.

Original: Conventional methods in this category include Principal Component Analysis (PCA) (Jolliffe (1986)) with explicit linear projections, kernel
Revised: Conventional methods in this category include Principal Component Analysis (PCA) (Jolliffe (1986)) with explicit linear projections, kernel

Original: PCA with implicit non-linear projections induced by speciﬁc kernels (G¨unter et al.), and Robust PCA (RPCA) (Huber (2011); Cand ` es et al.
Revised: PCA with implicit non-linear projections induced by speciﬁc kernels (G¨unter et al.), and Robust PCA (RPCA) (Huber (2011); Cand ` es et al.

Original: (2011)) that makes PCA less sensitive to noise by enforcing sparse structures.
Revised: (2011)) that makes PCA less sensitive to noise by enforcing sparse structures.

Original: In addition, multiple recent works propose to analyze the reconstruction error induced by deep autoencoders, and demonstrate promising results (Zhou & Paffenroth (2017); Zhai et al.
Revised: In addition, multiple recent works propose to analyze the reconstruction error induced by deep autoencoders, and demonstrate promising results (Zhou & Paffenroth (2017); Zhai et al.

Original: However, the performance of reconstruction based methods is limited by the fact that they only conduct anomaly analysis from a single aspect, that is, reconstruction error.
Revised: However, the performance of reconstruction based methods is limited by the fact that they only conduct anomaly analysis from a single aspect, that is, reconstruction error.

Original: Although the compression on anomalous samples could be different from the compression on normal samples and some of them do demonstrate unusually high reconstruction errors, a signiﬁcant amount of anomalous samples could also lurk with a normal level of error, which usually happens when the underlying dimensionality reduction methods have high model complexity or the samples of interest are noisy with complex structures.
Revised: Although the compression on anomalous samples could be different from the compression on normal samples and some of them do demonstrate unusually high reconstruction errors, a signiﬁcant amount of anomalous samples could also lurk with a normal level of error, which usually happens when the underlying dimensionality reduction methods have high model complexity or the samples of interest are noisy with complex structures.

Original: Even in these cases, we still have the hope to detect such “lurking” anomalies, as they still reside in low-density areas in the reduced low-dimensional space.
Revised: Even in these cases, we still have the hope to detect such “lurking” anomalies, as they still reside in low-density areas in the reduced low-dimensional space.

Original: Unlike the existing reconstruction based methods, DAGMM considers the both aspects, and performs density estimation in a low-dimensional space derived from the reduced representation and the reconstruction error caused by the dimensionality reduction, for a comprehensive view.
Revised: Unlike the existing reconstruction based methods, DAGMM considers the both aspects, and performs density estimation in a low-dimensional space derived from the reduced representation and the reconstruction error caused by the dimensionality reduction, for a comprehensive view.

Original: Clustering analysis is another popular category of methods used for density estimation and anomaly detection, such as multivariate Gaussian Models, Gaussian Mixture Models, k -means, and so on (Barnett & Lewis (1984); Zimek et al.
Revised: Clustering analysis is another popular category of methods used for density estimation and anomaly detection, such as multivariate Gaussian Models, Gaussian Mixture Models, k -means, and so on (Barnett & Lewis (1984); Zimek et al.

Original: ; Kim & Scott (2011);
Revised: ; Kim & Scott (2011);

Original: Xiong et al.
Revised: Xiong et al.

Original: Because of the curse of dimensionality, it is difﬁcult to directly apply such methods to multi- or high- dimensional data.
Revised: Because of the curse of dimensionality, it is difﬁcult to directly apply such methods to multi- or high- dimensional data.

Original: Traditional techniques adopt a two-step approach (Chandola et al.
Revised: Traditional techniques adopt a two-step approach (Chandola et al.

Original: (2009)), where dimensionality reduction is conducted ﬁrst, then clustering analysis is performed, and the two steps are separately learned.
Revised: (2009)), where dimensionality reduction is conducted ﬁrst, then clustering analysis is performed, and the two steps are separately learned.

Original: One of the drawbacks in the two-step approach is that dimensionality reduction is trained without the guidance from the subsequent clustering analysis, thus the key information for clustering analysis could be lost during dimensionality reduction.
Revised: One of the drawbacks in the two-step approach is that dimensionality reduction is trained without the guidance from the subsequent clustering analysis, thus the key information for clustering analysis could be lost during dimensionality reduction.

Original: To address this issue, recent works propose deep autoencoder based methods in order to jointly learn dimensionality reduction and clustering components.
Revised: To address this issue, recent works propose deep autoencoder based methods in order to jointly learn dimensionality reduction and clustering components.

Original: However, the performance of the state-of-the-art methods is limited by over-simpliﬁed clustering models that are unable to handle clustering or density estimation tasks for data of complex structures, or the pre-trained dimensionality reduction component ( i.e., autoencoder) has little potential to accommodate further adjustment by the subsequent ﬁne-tuning for anomaly detection.
Revised: However, the performance of the state-of-the-art methods is limited by over-simpliﬁed clustering models that are unable to handle clustering or density estimation tasks for data of complex structures, or the pre-trained dimensionality reduction component ( i.e., autoencoder) has little potential to accommodate further adjustment by the subsequent ﬁne-tuning for anomaly detection.

Original: DAGMM explicitly addresses these issues by a sub-network called estimation network that evaluates sample density in the low-dimensional space produced by its compression network.
Revised: DAGMM explicitly addresses these issues by a sub-network called estimation network that evaluates sample density in the low-dimensional space produced by its compression network.

Original: By predicting sample mixture membership, we are able to estimate the parameters of GMM without
Revised: By predicting sample mixture membership, we are able to estimate the parameters of GMM without

Original: EM-like alternating procedures.
Revised: EM-like alternating procedures.

Original: Moreover, DAGMM is friendly to end-to-end training so that we can unleash the full potential of adjusting dimensionality reduction components and jointly improve the quality of clustering analysis/density estimation.
Revised: Moreover, DAGMM is friendly to end-to-end training so that we can unleash the full potential of adjusting dimensionality reduction components and jointly improve the quality of clustering analysis/density estimation.

Original: In addition, one-class classiﬁcation approaches are also widely used for anomaly detection.
Revised: In addition, one-class classiﬁcation approaches are also widely used for anomaly detection.

Original: Under this framework, a discriminative boundary surrounding the normal instances is learned by algorithms, such as one-class SVM (Chen et al.
Revised: Under this framework, a discriminative boundary surrounding the normal instances is learned by algorithms, such as one-class SVM (Chen et al.

Original: ; Song et al.
Revised: ; Song et al.

Original: ; Williams et al.
Revised: ; Williams et al.

Original: When the number of dimensions grows higher, such techniques usually suffer from suboptimal performance due to the curse of dimensionality.
Revised: When the number of dimensions grows higher, such techniques usually suffer from suboptimal performance due to the curse of dimensionality.

Original: Unlike these methods, DAGMM estimates data density in a jointly learned low-dimensional space for more robust anomaly detection. 
Revised: Unlike these methods, DAGMM estimates data density in a jointly learned low-dimensional space for more robust anomaly detection. There has been growing interest in joint learning of dimensionality reduction (feature selection) and Gaussian mixture modeling.

Original: [Equation]
Revised: Yang et al. (2014; 2017b) propose a method that jointly learns linear dimensionality reduction and GMM. Paulik (2013) studies how to perform better feature selection with a pre-trained GMM as a regularizer. Variani et al.

Original: 3. O VERVIEW
Revised: and Zhang & Woodland (2017) propose joint learning frameworks, where the parameters of GMM are directly estimated through supervision information in speech recognition applications. T¨uske et al. (2015a;b) investigate how to use log-linear mixture models to approximate GMM posterior under the conditions that a class/mixture prior distribution is given and a covariance matrix is globally shared. Unlike the existing works, we focus on unsupervised settings: DAGMM extracts useful features for anomaly detection through linear/non-linear dimensionality reduction realized by a deep autoencoder, and jointly learns their density under the GMM framework by mixture membership estimation, for which DAGMM can be viewed as a complex deep version of adaptive mixture of experts (Jacobs et al.

Original: Deep Autoencoding Gaussian Mixture Model (DAGMM) consists of two major components: a compression network and an estimation network.
Revised: in combination with a deep autoencoder. More importantly, DAGMM combines induced reconstruction error and learned latent representation for unsupervised anomaly detection. [Figure]

Original:  As shown in Figure 2, DAGMM works as follows: (1) the compression network performs dimensionality reduction for input samples by a deep autoencoder, prepares their low-dimensional representations from both the reduced space and the reconstruction error features, and feeds the representations to the subsequent estimation network; (2) the estimation network takes the feed, and predicts their likelihood/energy in the framework of Gaussian Mixture
Revised: [Equation] 3. O VERVIEW Deep Autoencoding Gaussian Mixture Model (DAGMM) consists of two major components: a compression network and an estimation network. As shown in Figure 2, DAGMM works as follows: (1) the compression network performs dimensionality reduction for input samples by a deep autoencoder, prepares their low-dimensional representations from both the reduced space and the reconstruction error features, and feeds the representations to the subsequent estimation network; (2) the estimation network takes the feed, and predicts their likelihood/energy in the framework of Gaussian Mixture

Original: Model (GMM).
Revised: Model (GMM).

Original: 3. C OMPRESSION N ETWORK
Revised: 3. C OMPRESSION N ETWORK

Original: The low-dimensional representations provided by the compression network contains two sources of features: (1) the reduced low-dimensional representations learned by a deep autoencoder; and (2) the features derived from reconstruction error.
Revised: The low-dimensional representations provided by the compression network contains two sources of features: (1) the reduced low-dimensional representations learned by a deep autoencoder; and (2) the features derived from reconstruction error.

Original: Given a sample x , the compression network computes its low-dimensional representation z as follows.
Revised: Given a sample x , the compression network computes its low-dimensional representation z as follows.

Original: [Equation]
Revised: [Equation]

Original: where z c is the reduced low-dimensional representation learned by the deep autoencoder, z r includes the features derived from the reconstruction error, θ e and θ d are the parameters of the deep autoencoder, x (cid:48) is the reconstructed counterpart of x , h ( · ) denotes the encoding function, g ( · ) denotes the decoding function, and f ( · ) denotes the function of calculating reconstruction error features.
Revised: where z c is the reduced low-dimensional representation learned by the deep autoencoder, z r includes the features derived from the reconstruction error, θ e and θ d are the parameters of the deep autoencoder, x (cid:48) is the reconstructed counterpart of x , h ( · ) denotes the encoding function, g ( · ) denotes the decoding function, and f ( · ) denotes the function of calculating reconstruction error features.

Original: In particular, z r can be multi-dimensional, considering multiple distance metrics such as absolute Euclidean distance, relative Euclidean distance, cosine similarity, and so on.
Revised: In particular, z r can be multi-dimensional, considering multiple distance metrics such as absolute Euclidean distance, relative Euclidean distance, cosine similarity, and so on.

Original: In the end, the compression network feeds z to the subsequent estimation network.
Revised: In the end, the compression network feeds z to the subsequent estimation network.

Original: 3. E STIMATION N ETWORK
Revised: 3. E STIMATION N ETWORK

Original: Given the low-dimensional representations for input samples, the estimation network performs density estimation under the framework of GMM.
Revised: Given the low-dimensional representations for input samples, the estimation network performs density estimation under the framework of GMM.

Original: In the training phase with unknown mixture-component distribution φ , mixture means µ , and mixture covariance Σ , the estimation network estimates the parameters of GMM and evaluates the likelihood/energy for samples without alternating procedures such as EM (Zimek et al.
Revised: In the training phase with unknown mixture-component distribution φ , mixture means µ , and mixture covariance Σ , the estimation network estimates the parameters of GMM and evaluates the likelihood/energy for samples without alternating procedures such as EM (Zimek et al.

Original: The estimation network achieves this by utilizing a multi-layer neural network to predict the mixture membership for each sample.
Revised: The estimation network achieves this by utilizing a multi-layer neural network to predict the mixture membership for each sample.

Original: Given the low-dimensional representations z and an integer K as the number of mixture components, the estimation network makes membership prediction as follows.
Revised: Given the low-dimensional representations z and an integer K as the number of mixture components, the estimation network makes membership prediction as follows.

Original: [Equation]
Revised: [Equation]

Original: where ˆ γ is a K -dimensional vector for the soft mixture-component membership prediction, and p is the output of a multi-layer network parameterized by θ m .
Revised: where ˆ γ is a K -dimensional vector for the soft mixture-component membership prediction, and p is the output of a multi-layer network parameterized by θ m .

Original: Given a batch of N samples and their membership prediction, ∀ 1 ≤ k ≤ K , we can further estimate the parameters in GMM as follows.
Revised: Given a batch of N samples and their membership prediction, ∀ 1 ≤ k ≤ K , we can further estimate the parameters in GMM as follows.

Original: [Equation]
Revised: [Equation]

Original: where ˆ γ i is the membership prediction for the low-dimensional representation z i , and ˆ φ k , ˆ µ k , ˆΣ k are mixture probability, mean, covariance for component k in GMM, respectively.
Revised: where ˆ γ i is the membership prediction for the low-dimensional representation z i , and ˆ φ k , ˆ µ k , ˆΣ k are mixture probability, mean, covariance for component k in GMM, respectively.

Original: With the estimated parameters, sample energy can be further inferred by
Revised: With the estimated parameters, sample energy can be further inferred by

Original: [Equation]
Revised: [Equation]

Original: where | · | denotes the determinant of a matrix.
Revised: where | · | denotes the determinant of a matrix.

Original: In addition, during the testing phase with the learned GMM parameters, it is straightforward to estimate sample energy, and predict samples of high energy as anomalies by a pre-chosen threshold.
Revised: In addition, during the testing phase with the learned GMM parameters, it is straightforward to estimate sample energy, and predict samples of high energy as anomalies by a pre-chosen threshold.

Original: 3. O BJECTIVE FUNCTION
Revised: 3. O BJECTIVE FUNCTION

Original: Given a dataset of N samples, the objective function that guides DAGMM training is constructed as follows.
Revised: Given a dataset of N samples [Equation] the objective function that guides DAGMM training is constructed as follows.

Original: [Equation]
Revised: [Equation]

Original: This objective function includes three components.
Revised: This objective function includes three components.

Original: • L ( x i , x (cid:48) i ) is the loss function that characterizes the reconstruction error caused by the deep autoencoder in the compression network.
Revised: • L ( x i , x (cid:48) i ) is the loss function that characterizes the reconstruction error caused by the deep autoencoder in the compression network.

Original: Intuitively, if the compression network could make the reconstruction error low, the low-dimensional representation could better preserve the key information of input samples.
Revised: Intuitively, if the compression network could make the reconstruction error low, the low-dimensional representation could better preserve the key information of input samples.

Original: Therefore, a compression network of lower reconstruction error is always desired.
Revised: Therefore, a compression network of lower reconstruction error is always desired.

Original: In practice, L 2 -norm usually gives desirable results, as L( x i , x (cid:48) i )
Revised: In practice, L 2 -norm usually gives desirable results, as L

Original: = (cid:107) x
Revised: [Equation]

Original:  i − x(cid:48)
Revised: x i [Equation] x

Original: i (cid:107) 22 .
Revised: [Equation]

Original: • E ( z i ) models the probabilities that we could observe the input samples.
Revised: • E ( z i ) models the probabilities that we could observe the input samples.

Original: By minimizing the sample energy, we look for the best combination of compression and estimation networks that maximize the likelihood to observe input samples.
Revised: By minimizing the sample energy, we look for the best combination of compression and estimation networks that maximize the likelihood to observe input samples.

Original: • DAGMM also has the singularity problem as in GMM: trivial solutions are triggered when the diagonal entries in covariance matrices degenerate to 0.
Revised: • DAGMM also has the singularity problem as in GMM: trivial solutions are triggered when the diagonal entries in covariance matrices degenerate to 0.

Original: To avoid this issue, we penalize small values on the diagonal entries by P ( ˆΣ ) =
Revised: To avoid this issue, we penalize small values on the diagonal entries by P ( ˆΣ ) =

Original: (cid:80)
Revised: (cid:80)

Original: Kk =1 (cid:80) dj
Revised: Kk =1 (cid:80) dj

Original: = 1 ˆΣ kjj , where d is the number of dimensions in the low-dimensional representations provided by the compression network.
Revised: = 1 ˆΣ kjj , where d is the number of dimensions in the low-dimensional representations provided by the compression network.

Original: • λ 1 and λ 2 are the meta parameters in DAGMM.
Revised: • λ 1 and λ 2 are the meta parameters in DAGMM.

Original: In practice, λ 1 = 0 .
Revised: In practice, λ 1 = 0 .

Original: 1 and λ 2 = 

Revised: 1 and λ 2 =

Original: .usually render desirable results.
Revised: .usually render desirable results.

Original: 3. T RAINING S TRATEGY
Revised: 3. R ELATION TO V ARIATIONAL I NFERENCE

Revised: In DAGMM, we leverage the estimation network to make membership prediction for each sample.

Revised: From the view of probabilistic graphical models, the estimation network plays an analogous role of latent variable ( i.e., sample membership) inference.

Revised: Recently, neural variational inference (Mnih &

Revised: Gregor (2014)) has been proposed to employ deep neural networks to tackle difﬁcult latent variable inference problems, where exact model inference is intractable and conventional approximate methods cannot scale well.

Revised: Theoretically, we can also adapt the membership prediction task of DAGMM into the framework of neural variational inference.

Revised: For sample x i , the contribution of its compressed representation z i to the energy function can be upper-bounded as follows (Jordan et al.

Revised: [Equation]

Revised: where Q θ m ( k | z i ) is the estimation network that predicts the membership of z i , KL ( ·||· ) is the

Revised: Kullback-Leibler divergence between two input distributions, and p ( k |

Revised: z i ) is the true posterior probability distribution of mixture component k given z i .

Revised: By minimizing the negative evidence lower bound in Equation (8), we can make the estimation network approximate the true posterior and tighten the bound of energy function.

Revised: In DAGMM, we use Equation (6) as a part of the objective function instead of its upper bound in Equation (9) simply because the energy function of DAGMM is tractable and efﬁcient to evaluate.

Revised: Unlike neural variational inference that uses the deep estimation network to deﬁne a variational posterior distribution as described above, DAGMM explicitly employs the deep estimation network to parametrize a sampledependent prior distribution.

Revised: In the history of machine learning research, there were research efforts towards utilizing neural networks to calculate sample membership in mixture models, such as adaptive mixture of experts (Jacobs et al.

Revised: From this perspective, DAGMM can be viewed as a powerful deep version of adaptive mixture of experts in combination with a deep autoencoder.

Revised: 3. T RAINING S TRATEGY

Original: Unlike existing deep autoencoder based methods (Yang et al.
Revised: Unlike existing deep autoencoder based methods (Yang et al.

Original:  ; Xie et al. (2016)) that rely on pre-training, DAGMM employs end-to-end training.
Revised: (2017a) ; Xie et al. (2016)) that rely on pre-training, DAGMM employs end-to-end training.

Original: First, in our study, we ﬁnd that pre-trained compression networks suffer from limited anomaly detection performance, as it is difﬁcult to make signiﬁcant changes in the well-trained deep autoencoder to favor the subsequent density estimation tasks.
Revised: First, in our study, we ﬁnd that pre-trained compression networks suffer from limited anomaly detection performance, as it is difﬁcult to make signiﬁcant changes in the well-trained deep autoencoder to favor the subsequent density estimation tasks.

Original: Second, we also ﬁnd that the compression network and estimation network could mutually boost each others’ performance.
Revised: Second, we also ﬁnd that the compression network and estimation network could mutually boost each others’ performance.

Original: On one hand, with the regularization introduced by the estimation network, the deep autoencoder in the compression network learned by end-to-end training can reduce reconstruction error as low as the error from its pre-trained counterpart, which meanwhile cannot be achieved by simply performing end-to-end training with the deep autoencoder alone.
Revised: On one hand, with the regularization introduced by the estimation network, the deep autoencoder in the compression network learned by end-to-end training can reduce reconstruction error as low as the error from its pre-trained counterpart, which meanwhile cannot be achieved by simply performing end-to-end training with the deep autoencoder alone.

Original: On the other hand, with the well-learned low-dimensional representations from the compression network, the estimation network is able to make meaningful density estimations.
Revised: On the other hand, with the well-learned low-dimensional representations from the compression network, the estimation network is able to make meaningful density estimations.

Original: In Section 4.5, we employ an example from a public benchmark dataset to discuss the choice between pre-training and end-to-end training in DAGMM.
Revised: In Section 4.5, we employ an example from a public benchmark dataset to discuss the choice between pre-training and end-to-end training in DAGMM.

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: In this section, we use public benchmark datasets to demonstrate the effectiveness of DAGMM in unsupervised anomaly detection.
Revised: In this section, we use public benchmark datasets to demonstrate the effectiveness of DAGMM in unsupervised anomaly detection.

Original: 4. D ATASET
Revised: 4. D ATASET

Original: [Table]
Revised: [Table]

Original: In this study, we employ three benchmark datasets: KDDCUP, Thyroid,  and Arrhythmia.
Revised: We employ four benchmark datasets: KDDCUP, Thyroid, Arrhythmia, and KDDCUP-Rev.

Original: • KDDCUP .
Revised: • KDDCUP .

Original: The KDDCUP99 10 percent dataset from the UCI repository (Lichman (2013)) originally contains samples of 41 dimensions, where 34 of them are continuous and 7 are categorical.
Revised: The KDDCUP99 10 percent dataset from the UCI repository (Lichman (2013)) originally contains samples of 41 dimensions, where 34 of them are continuous and 7 are categorical.

Original: For categorical features, we further use one-hot representation to encode them, and eventually we obtain a dataset of 120 dimensions.
Revised: For categorical features, we further use one-hot representation to encode them, and eventually we obtain a dataset of 120 dimensions.

Original: As 20 % of data samples are labeled as “normal” and the rest are labeled as attacks, “normal” samples are in a minority group; therefore, “normal” ones are treated as anomalies in this task.
Revised: As 20 % of data samples are labeled as “normal” and the rest are labeled as “attack”, “normal” samples are in a minority group; therefore, “normal” ones are treated as anomalies in this task.

Original: • Thyroid .
Revised: • Thyroid .

Original: The Thyroid (Lichman (2013)) dataset is obtained from the ODDS repository 1 .
Revised: The Thyroid (Lichman (2013)) dataset is obtained from the ODDS repository 1 .

Original: There are 3 classes in the original dataset.
Revised: There are 3 classes in the original dataset.

Original: In this task, the hyperfunction class is treated as anomaly class and the other two classes are treated as normal class, because hyperfunction is a clear minority class.
Revised: In this task, the hyperfunction class is treated as anomaly class and the other two classes are treated as normal class, because hyperfunction is a clear minority class.

Original: • Arrhythmia .
Revised: • Arrhythmia .

Original: The Arrhythmia (Lichman (2013)) dataset is also obtained from the ODDS repository.
Revised: The Arrhythmia (Lichman (2013)) dataset is also obtained from the ODDS repository.

Original: The smallest classes, including 3, 4, 5, 7, 8, 9, 14, and 15, are combined to form the anomaly class, and the rest of the classes are combined to form the normal class.
Revised: The smallest classes, including 3, 4, 5, 7, 8, 9, 14, and 15, are combined to form the anomaly class, and the rest of the classes are combined to form the normal class.

Original: More detailed information about the datasets is shown in Table 1.
Revised: • KDDCUP-Rev . This dataset is derived from KDDCUP.

Original: Following the setting in (Zhai et al.
Revised: We keep all the data samples labeled as “normal” and randomly draw samples labeled as “attack” so that the ratio between “normal” and “attack” is 4 : 1 . In this way, we obtain a dataset with anomaly ratio 0 .

Original:  ,  in each run, we take 50 % of data by random sampling for training with the rest 50 % reserved for testing, and only data samplesfrom the normal class are used for training models.
Revised: 2 , where “attack” samples are in a minority group and treated as anomalies. Note that “attack” samples are not ﬁxed, and we randomly draw “attack” samples in every single run. Detailed information about the datasets is shown in Table 1.

Original: 4. B ASELINE M ETHODS
Revised: 4. B ASELINE M ETHODS

Original: We consider both traditional and state-of-the-art deep learning methods as baselines.
Revised: We consider both traditional and state-of-the-art deep learning methods as baselines.

Original: • OC-SVM.
Revised: • OC-SVM.

Original: One-class support vector machine (Chen et al.
Revised: One-class support vector machine (Chen et al.

Original: (2001)) is a popular kernel-based method used in anomaly detection.
Revised: (2001)) is a popular kernel-based method used in anomaly detection.

Original: In the experiment, we employ the widely adopted radial basis function (RBF) kernel in all the tasks.
Revised: In the experiment, we employ the widely adopted radial basis function (RBF) kernel in all the tasks.

Original: • DSEBM-e. Deep structured energy based model (DSEBM)
Revised: • DSEBM-e. Deep structured energy based model (DSEBM)

Original: (Zhai et al.
Revised: (Zhai et al.

Original: (2016)) is a state-ofthe-art deep learning method for unsupervised anomaly detection.
Revised: (2016)) is a state-ofthe-art deep learning method for unsupervised anomaly detection.

Original: In DSEBM-e, sample energy is leveraged as the criterion to detect anomalies.
Revised: In DSEBM-e, sample energy is leveraged as the criterion to detect anomalies.

Original: • DSEBM-r. DSEBM-e and DSEBM-r (Zhai et al.
Revised: • DSEBM-r. DSEBM-e and DSEBM-r (Zhai et al.

Original: (2016)) share the same core technique, but reconstruction error is used as the criterion in DSEBM-r for anomaly detection.
Revised: (2016)) share the same core technique, but reconstruction error is used as the criterion in DSEBM-r for anomaly detection.

Original: • DCN.
Revised: • DCN.

Original: Deep clustering network (DCN) (Yang et al.
Revised: Deep clustering network (DCN) (Yang et al.

Original: (2017)) is a state-of-the-art clustering algorithm that regulates autoencoder performance by k-means.
Revised: (2017a)) is a state-of-the-art clustering algorithm that regulates autoencoder performance by k-means.

Original: We adapt this technique to anomaly detection tasks.
Revised: We adapt this technique to anomaly detection tasks.

Original: In particular, the distance between a sample and its cluster center is taken as the criterion for anomaly detection: samples that are farther from their cluster centers are more likely to be anomalies.
Revised: In particular, the distance between a sample and its cluster center is taken as the criterion for anomaly detection: samples that are farther from their cluster centers are more likely to be anomalies.

Original: Moreover we include the following DAGMM variants as baselines to demonstrate the importance of individual components in DAGMM.
Revised: Moreover, we include the following DAGMM variants as baselines to demonstrate the importance of individual components in DAGMM.

Original: • GMM-EN.
Revised: • GMM-EN.

Original: In this variant, we remove the reconstruction error component from the objective function of DAGMM.
Revised: In this variant, we remove the reconstruction error component from the objective function of DAGMM.

Original: In other words, the estimation network in DAGMM performs membership estimation without the constraints from the compression network.
Revised: In other words, the estimation network in DAGMM performs membership estimation without the constraints from the compression network.

Original: With the learned membership estimation, we infer sample energy by Equation (5) and (6) under the GMM framework.
Revised: With the learned membership estimation, we infer sample energy by Equation (5) and (6) under the GMM framework.

Original: Sample energy is used as the criterion for anomaly detection.
Revised: Sample energy is used as the criterion for anomaly detection.

Original: • PAE.
Revised: • PAE.

Original: We obtain this variant by removing the energy function from the objective function of DAGMM, and this DAGMM variant is equivalent to a deep autoenoder.
Revised: We obtain this variant by removing the energy function from the objective function of DAGMM, and this DAGMM variant is equivalent to a deep autoenoder.

Original: To ensure the compression network is well trained, we adopt the pre-training strategy (Vincent et al.
Revised: To ensure the compression network is well trained, we adopt the pre-training strategy (Vincent et al.

Original: Sample reconstruction error is the criterion for anomaly detection.
Revised: Sample reconstruction error is the criterion for anomaly detection.

Original: • E2E-AE.
Revised: • E2E-AE.

Original: This variant shares the same setting with PAE, but the deep autoencoder is learned by end-to-end training.
Revised: This variant shares the same setting with PAE, but the deep autoencoder is learned by end-to-end training.

Original: Sample reconstruction error is the criterion for anomaly detection • PAE-GMM.
Revised: Sample reconstruction error is the criterion for anomaly detection • PAE-GMM-EM.

Original: This variant adopts a two-step approach.
Revised: This variant adopts a two-step approach.

Original: At step one, we learn the compression network by pre-training deep autoencoder.
Revised: At step one, we learn the compression network by pre-training deep autoencoder.

Original: At step two, we use the output from the compression network to train the estimation network.
Revised: At step two, we use the output from the compression network to train the GMM by a traditional EM algorithm. The training procedures in the two steps are separated. Sample energy is used as the criterion for anomaly detection. • PAE-GMM.

Revised: This variant also adopts a two-step approach.

Revised: At step one, we learn the compression network by pre-training deep autoencoder.

Revised: At step two, we use the output from the compression network to train the estimation network.

Original: The training procedures in the two steps are separated.
Revised: The training procedures in the two steps are separated.

Original: Sample energy is used as the criterion for anomaly detection.
Revised: Sample energy is used as the criterion for anomaly detection.

Original: • DAGMM-p.
Revised: • DAGMM-p.

Original: This variant is a compromise between DAGMM and PAE-GMM: we ﬁrst train the compression network by pre-training, and then ﬁne-tune DAGMM by end-to-end training.
Revised: This variant is a compromise between DAGMM and PAE-GMM: we ﬁrst train the compression network by pre-training, and then ﬁne-tune DAGMM by end-to-end training.

Original: Sample energy is the criterion for anomaly detection.
Revised: Sample energy is the criterion for anomaly detection.

Revised: • DAGMM-NVI.

Revised: The only difference between this variant and DAGMM is that this variant adopts the framework of neural variational inference (Mnih & Gregor (2014)) and replaces Equation (6) with the upper bound in Equation (9) as a part of the objective function

Original: 4. DAGMM C ONFIGURATION
Revised: 4. DAGMM C ONFIGURATION

Original: In all the experiment, we consider two reconstruction features from the compression network: relative
Revised: In all the experiment, we consider two reconstruction features from the compression network: relative

Original: Euclidean distance and cosine similarity.
Revised: Euclidean distance and cosine similarity.

Original: Given a sample x and its reconstructed counterpart x (cid:48) , their relative Euclidean distance is deﬁned as (cid:107) x − x (cid:48) (cid:107) 2 (cid:107) x (cid:107) 2 , and the cosine similarity is derived by x · x (cid:48) (cid:107) x (cid:107) 2 (cid:107) x (cid:48) (cid:107) 2 . 
Revised: Given a sample x and its reconstructed counterpart x (cid:48) , their relative Euclidean distance is deﬁned as (cid:107) x − x (cid:48) (cid:107) 2 (cid:107) x (cid:107) 2 , and the cosine similarity is derived by x · x (cid:48) (cid:107) x (cid:107) 2 (cid:107) x (cid:48) (cid:107) 2 . In Appendix D, for readers of interest, we discuss why reconstruction features are important to

Original:  The network structures of DAGMM used on individual datasets are summarized as follows.
Revised: DAGMM and how to select reconstruction features in practice. The network structures of DAGMM used on individual datasets are summarized as follows

Original: • KDDCUP.
Revised: • KDDCUP.

Original: For this dataset, its compression network provides 3 dimensional input to the estimation network, where one is the reduced dimension and the other two are from the reconstruction error.
Revised: For this dataset, its compression network provides 3 dimensional input to the estimation network, where one is the reduced dimension and the other two are from the reconstruction error.

Original: The estimation network considers a GMM with 4 mixture components for the best performance.
Revised: The estimation network considers a GMM with 4 mixture components for the best performance.

Original: In particular, the compression network runs with FC(120, 60, tanh )-FC(60, 30, tanh )-FC(30, 10, tanh )-FC(10, 1, none)-FC(1, 10, tanh )-FC(10, 30, tanh )-FC(30, 60, tanh )-FC(60, 120, none), and the estimation network performs with FC(3, 10, tanh )-Drop(0.5)-FC(10, 4, softmax).
Revised: In particular, the compression network runs with FC(120, 60, tanh )-FC(60, 30, tanh )-FC(30, 10, tanh )-FC(10, 1, none)-FC(1, 10, tanh )-FC(10, 30, tanh )-FC(30, 60, tanh )-FC(60, 120, none), and the estimation network performs with FC(3, 10, tanh )-Drop(0.5)-FC(10, 4, softmax).

Original: • Thyroid.
Revised: • Thyroid.

Original: The compression network for this dataset also provides 3 dimensional input to the estimation network, and the estimation network employs 2 mixture components for the best performance.
Revised: The compression network for this dataset also provides 3 dimensional input to the estimation network, and the estimation network employs 2 mixture components for the best performance.

Original: In particular, the compression network runs with FC(6, 12, tanh )-FC(12, 4, tanh )
Revised: In particular, the compression network runs with FC(6, 12, tanh )-FC(12, 4, tanh )-FC(

Original: -FC(4, 1, none)-FC(1, 4, tanh ) -FC(4, 12, tanh )-FC(12, 6, none), and the estimation network performs with FC(3, 10, tanh )-Drop(0.5)-FC(10, 2, softmax).
Revised: , 1, none)-FC(1, 4, tanh )-FC(4, 12, tanh )-FC(12, 6, none), and the estimation network performs with FC(3, 10, tanh )-Drop(0.5)-FC(10, 2, softmax).

Original: • Arrhythmia.
Revised: • Arrhythmia.

Original: The compression network for this dataset provides 4 dimensional input, where two of them are the reduced dimensions, and the estimation network adopts a setting of 2 mixture components 
Revised: The compression network for this dataset provides 4 dimensional input, where two of them are the reduced dimensions, and the estimation network adopts a setting of 2 mixture components for the best performance.

Original:  for the best performance.
Revised: In particular, the compression network runs with FC(274, 10, tanh )-FC(10, 2, none)-FC(2, 10, tanh )-FC(10, 274, none), and the estimation network performs with FC(4, 10, tanh )-Drop(0.5)-FC(10, 2, softmax). • KDDCUP-Rev. For this dataset, its compression network provides 3 dimensional input to the estimation network, where one is the reduced dimension and the other two are from the reconstruction error. The estimation network considers a GMM with 2 mixture components for the best performance.

Original: In particular, the compression network runs with FC(274, 10, tanh )-FC(10, 2, none)-FC(2, 10, tanh )-FC(10, 274, none), and the estimation network performs with FC(4, 10, tanh )-Drop(0.5)-FC(10, 2, softmax).
Revised: In particular, the compression network runs with FC(120, 60, tanh )-FC(60, 30, tanh )-FC(30, 10, tanh )-FC(10, 1, none)-FC(1, 10, tanh )-FC(10, 30, tanh )-FC(30, 60, tanh )-FC(60, 120, none), and the estimation network performs with FC(3, 10, tanh )-Drop(0.5)-FC(10, 2, softmax).

Original: where FC( a , b , f ) means a fully-connected layer with a input neurons and b output neurons activated by function f (none means no activation function is used), and Drop( p ) denotes a dropout layer with keep probability p during training.
Revised: where FC( a , b , f ) means a fully-connected layer with a input neurons and b output neurons activated by function f (none means no activation function is used), and Drop( p ) denotes a dropout layer with keep probability p during training.

Original: All the DAGMM instances are implemented by tensorﬂow (Abadi et al.
Revised: All the DAGMM instances are implemented by tensorﬂow (Abadi et al.

Original: and trained by
Revised: and trained by

Original: Adam (Kingma & Ba (2015)) algorithm with learning rate 0 . 0001 .
Revised: Adam (Kingma & Ba (2015)) algorithm with learning rate 0 . 0001 .

Original: For KDDCUP, Thyroid,and Arrhythmia,  the number of training epochs are 200, 20000,and 10000,  respectively.
Revised: For KDDCUP, Thyroid, Arrhythmia, and KDDCUP-Rev, the number of training epochs are 200, 20000, 10000, and 400, respectively.

Original: For the sizes of mini-batches, they are set as 1024, 1024,and 128,  respectively.
Revised: For the sizes of mini-batches, they are set as 1024, 1024, 128, and 1024, respectively.

Original: Moreover, in all the DAGMM instances, we set λ 1 as 0 .
Revised: Moreover, in all the DAGMM instances, we set λ 1 as 0 .

Original: 1 and λ 2 as 0 .
Revised: 1 and λ 2 as 0 .

Revised: For readers of interest, we discuss how λ 1 and λ 2 impact DAGMM in Appendix F.

Original: For the baseline methods, we conduct exhaustive search to ﬁnd the optimal meta parameters for them in order to achieve the best performance.
Revised: For the baseline methods, we conduct exhaustive search to ﬁnd the optimal meta parameters for them in order to achieve the best performance.

Original: We detail their exact conﬁguration in Appendix A.
Revised: We detail their exact conﬁguration in Appendix A.

Original: 4. A CCURACY
Revised: 4. A CCURACY

Original: Metric .
Revised: Metric .

Original: We consider average precision, recall, and F 1 score as intuitive ways to compare anomaly detection performance.
Revised: We consider average precision, recall, and F 1 score as intuitive ways to compare anomaly detection performance.

Original: In particular, based on the anomaly ratio suggested in Table 1, we select the threshold to identify anomalous samples.
Revised: In particular, based on the anomaly ratio suggested in Table 1, we select the threshold to identify anomalous samples.

Original: For example, when DAGMM performs on KDDCUP, the top 20 % samples of the highest energy will be marked as anomalies.
Revised: For example, when DAGMM performs on KDDCUP, the top 20 % samples of the highest energy will be marked as anomalies.

Original: We take anomaly class as positive, and deﬁne precision, recall, and F 1 score accordingly.
Revised: We take anomaly class as positive, and deﬁne precision, recall, and F 1 score accordingly.

Original: [Table]
Revised: In the ﬁrst set of experiment, we follow the setting in (Zhai et al. with completely clean training data: in each run, we take 50 % of data by random sampling for training with the rest 50 % reserved for testing, and only data samples from the normal class are used for training models.

Original: Table 2 reports the average precision, recall, and F 1 score after 20 runs for DAGMM and its baselines.
Revised: Table 2 reports the average precision, recall, and F 1 score after 20 runs for DAGMM and its baselines.

Original: In general, DAGMM demonstrates superior performance over the baseline methods at almost all the ends.
Revised: In general, DAGMM demonstrates superior performance over the baseline methods in terms of

Original:  Especially on KDDCUP dataset, DAGMM achieves 14%  improvement at F 1 score, compared with the closest baseline.
Revised: F 1 score on all the datasets. Especially on KDDCUP and KDDCUP-Rev, DAGMM achieves 14% and 10% improvement at F 1 score, compared with the existing methods.

Original: For OC-SVM, the curse of dimensionality could be the main reason that limits its performance.
Revised: For OC-SVM, the curse of dimensionality could be the main reason that limits its performance.

Original: For DSEBM,  it performs reasonably well on Arrhythmia, but it is unable to handle data of more complex structures, such as KDDCUP and Thyroid.
Revised: For DSEBM, while it works reasonably well on multiple datasets, DAGMM outperforms as both latent representation and reconstruction error are jointly considered in energy modeling.

Original: For DCN, PAE-GMM, and
Revised: For DCN, PAE-GMM, and

Original: DAGMM-p, their performance could be limited by the pre-trained deep autoencoders.
Revised: DAGMM-p, their performance could be limited by the pre-trained deep autoencoders.

Original: When a deep autoencoder is well-trained, it is hard to make any signiﬁcant change on the reduced dimensions and favor the subsequent density estimation tasks.
Revised: When a deep autoencoder is well-trained, it is hard to make any signiﬁcant change on the reduced dimensions and favor the subsequent density estimation tasks.

Original: For GMM-EN, without the reconstruction constraints, it seems difﬁcult to perform reasonable density estimation.
Revised: For GMM-EN, without the reconstruction constraints, it seems difﬁcult to perform reasonable density estimation.

Original: In terms of PAE, the single view of reconstruction error may not be sufﬁcient for anomaly detection tasks.
Revised: In terms of PAE, the single view of reconstruction error may not be sufﬁcient for anomaly detection tasks.

Original: For E2E-AE, we observe that it is unable to reduce reconstruction error as low as PAE and DAGMM do on KDDCUP and Thyroid.
Revised: For E2E-AE, we observe that it is unable to reduce reconstruction error as low as PAE and DAGMM do on KDDCUP, KDDCUP-Rev, and Thyroid.

Original: As the key information of data could be lost during dimensionality reduction, E2E-AE suffers poor performance on KDDCUP and Thyroid. 
Revised: As the key information of data could be lost during dimensionality reduction, E2E-AE suffers poor performance on KDDCUP and Thyroid. In addition, the performance of DAGMM and DAGMM-NVI is quite similar.

Revised: As GMM is a fairly simple graphical model, we cannot spot signiﬁcant improvement brought by neural variational inference in DAGMM.

Revised: In Appendix B, for readers of interest, we show the cumulative distribution functions of the energy function learned by

Revised: DAGMM for all the datasets under the setting of clean training data.

Revised: [Table]

Original: Insum, the  DAGMM learned by end-to-end training achieves the best accuracy on thepublic benchmark datasets, and provides a promising alternative for unsupervised anomaly detection tasks.
Revised: In the second set of experiment, we investigate how DAGMM responds to contaminated training data. In each run, we reserve 50 % of data by random sampling for testing. For the rest 50 % , we take all samples from the normal class mixed with c % of samples from the anomaly class for model [Table]

Revised: Table 3 reports the average precision, recall

Revised: [Table]

Revised: and F 1 score after 20 runs of DAGMM, DCN, DSEBM-e, and OC-SVM on the KDDCUP dataset, respectively.

Revised: Note that we only show the results of DSEBMe, as DSEBM-e outperforms DSEBM-r in most cases.

Revised: As expected, contaminated training data negatively affect detection accuracy.

Revised: When contamination ratio c increases from 1 to 5 , average precision, recall, and F 1 score decrease at all the methods.

Revised: Meanwhile, we notice that DAGMM is able to maintain good detection accuracy with 5% contaminated data.

Revised: For OC-SVM, we adopt the same parameter setting used in the experiment with clean training data, and observe that OC-SVM is more sensitive to contamination ratio.

Revised: In order to receive better detection accuracy, it is important to train a model with high-quality data ( i.e., clean or keeping contamination ratio as low as possible).

Revised: In sum, the DAGMM learned by end-to-end training achieves the state-of-the-art accuracy on the public benchmark datasets, and provides a promising alternative for unsupervised anomaly detection.

Original: 4. V ISUALIZATION ON THE L EARNED L OW -D IMENSIONAL R EPRESENTATION
Revised: 4. V ISUALIZATION ON THE L EARNED L OW -D IMENSIONAL R EPRESENTATION

Original: In this section, we use an example to demonstrate the advantage of DAGMM learned by end-to-end training, compared with the baselines that rely on pre-trained deep autoencoders.
Revised: In this section, we use an example to demonstrate the advantage of DAGMM learned by end-to-end training, compared with the baselines that rely on pre-trained deep autoencoders.

Original:  Figure 3 shows the low-dimensional representation learned by DAGMM, PAE, DAGMM-p, and
Revised: [Figure] Figure 3 shows the low-dimensional representation learned by DAGMM, PAE, DAGMM-p, and

Original: DCN, from one of the experiment runs on the KDDCUP dataset.
Revised: DCN, from one of the experiment runs on the KDDCUP dataset.

Original: First, we canclearly see from 
Revised: First, we can see from Figure 3a that

Original: Figure 3a that DAGMM is able to well separate anomalous samples from normal samples in the [Figure]
Revised: DAGMM can better separate anomalous samples from normal samples in the learned low-dimensional space, while anomalies overlap more with normal samples in the low-dimensional space learned by PAE, DAGMM-p, or DCN.

Original: Second, Even if DAGMM-p and DCN take effort to ﬁne-tune the -trained deep  by its estimation network or k-means regularization, one could barely see signiﬁcant change among Figure 3b, Figure 3c, and Figure 3d, where many anomalous samples are still mixed with normal samples.
Revised: Second, Even if DAGMM-p and DCN take effort to ﬁne-tune the pre-trained deep autoencoder by its estimation network or k-means regularization, one could barely see signiﬁcant change among Figure 3b, Figure 3c, and Figure 3d, where many anomalous samples are still mixed with normal samples.

Original: Indeed, when a deep autoencoder is pretrained, it tends to be stuck in a good local optima for the purpose of reconstruction only, but it could be suboptimal for the subsequent density estimation tasks.
Revised: Indeed, when a deep autoencoder is pre-trained, it tends to be stuck in a good local optima for the purpose of reconstruction only, but it could be suboptimal for the subsequent density estimation tasks.

Original: In addition, in our study, we ﬁnd that the reconstruction error in a trained DAGMM is as low as the error received from a pre-trained deep autoencoder ( e.g., around 0 .
Revised: In addition, in our study, we ﬁnd that the reconstruction error in a trained DAGMM is as low as the error received from a pre-trained deep autoencoder ( e.g., around 0 .

Original: 26 in terms of per-sample reconstruction error for KDDCUP).
Revised: 26 in terms of per-sample reconstruction error for KDDCUP).

Original: Meanwhile, we also observe that it is difﬁcult to reduce the reconstruction error for a deep autoencoder of the identical structure by end-to-end training ( e.g., around 1 . 13 in terms of per-sample reconstruction error for KDDCUP).
Revised: Meanwhile, we also observe that it is difﬁcult to reduce the reconstruction error for a deep autoencoder of the identical structure by endto-end training ( e.g., around 1 . 13 in terms of per-sample reconstruction error for KDDCUP).

Original: In other words, the compression network and estimation network mutually boost each others’ performance during end-to-end training: the regularization introduced by the estimation network helps the deep autoencoder escape from less attractive local optima for better compression, while the compression network feeds  meaningful low-dimensional representations to estimation network for robust density estimation. 
Revised: In other words, the compression network and estimation network mutually boost each others’ performance during end-to-end training: the regularization introduced by the estimation network helps the deep autoencoder escape from less attractive local optima for better compression, while the compression network feeds more meaningful low-dimensional representations to estimation network for robust density estimation. In Appendix C, for readers of interest, we show the visualization of the latent representation learned by DSEBM.

Original: In summary, our experimental results show that DAGMM suggests a promising direction for density estimation and anomaly detection, where one can combine the forces of dimensionality reduction and density estimation by end-to-end training. 
Revised: In summary, our experimental results show that DAGMM suggests a promising direction for density estimation and anomaly detection, where one can combine the forces of dimensionality reduction and density estimation by end-to-end training. In Appendix E, we provide another case study to discuss which kind of samples beneﬁt more from joint training in DAGMM for readers of interest.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we propose the Deep Autoencoding Gaussian Mixture Model (DAGMM) for unsupervised anomaly detection.
Revised: In this paper, we propose the Deep Autoencoding Gaussian Mixture Model (DAGMM) for unsupervised anomaly detection.

Original: DAGMM consists of two major components: compression network and estimation network, where the compression network projects samples into a low-dimensional space that preserves the key information for anomaly detection, and the estimation network evaluates sample energy in the low-dimensional space under the framework of Gaussian Mixture Modeling.
Revised: DAGMM consists of two major components: compression network and estimation network, where the compression network projects samples into a low-dimensional space that preserves the key information for anomaly detection, and the estimation network evaluates sample energy in the low-dimensional space under the framework of Gaussian Mixture Modeling.

Original: DAGMM is friendly to end-to-end training: (1) the estimation network predicts sample mixture membership so that the parameters in GMM can be estimated without alternating procedures; and (2) the regularization introduced by the estimation network helps the compression network escape from less attractive local optima and achieve low reconstruction error by end-to-end training.
Revised: DAGMM is friendly to end-to-end training: (1) the estimation network predicts sample mixture membership so that the parameters in GMM can be estimated without alternating procedures; and (2) the regularization introduced by the estimation network helps the compression network escape from less attractive local optima and achieve low reconstruction error by end-to-end training.

Original: Compared with the pre-training strategy, the end-to-end training could be more beneﬁcial for density estimation tasks, as we can have more freedom to adjust dimensionality reduction processes to favor the subsequent density estimation tasks.
Revised: Compared with the pre-training strategy, the end-to-end training could be more beneﬁcial for density estimation tasks, as we can have more freedom to adjust dimensionality reduction processes to favor the subsequent density estimation tasks.

Original: In the experimental study, DAGMM demonstrates superior performance over state-of-the-art techniques on public benchmark datasets with up to 14% improvement on the standard F 1 score, and suggests a promising direction for unsupervised anomaly detection on multior high-dimensional data.
Revised: In the experimental study, DAGMM demonstrates superior performance over state-of-the-art techniques on public benchmark datasets with up to 14% improvement on the standard F 1 score, and suggests a promising direction for unsupervised anomaly detection on multior high-dimensional data.

Original: A B ASELINE CONFIGURATION
Revised: A B ASELINE CONFIGURATION

Original: OC-SVM .
Revised: OC-SVM .

Original: Unlike other baselines that only need decision thresholds in the testing phase, OC-SVM needs parameter ν be set in the training phase.
Revised: Unlike other baselines that only need decision thresholds in the testing phase, OC-SVM needs parameter ν be set in the training phase.

Original: Although ν intuitively means anomaly ratio in training data, it is non-trivial to set a reasonable ν in the case where training data are all normal samples and anomaly ratio in the testing phase could be arbitrary.
Revised: Although ν intuitively means anomaly ratio in training data, it is non-trivial to set a reasonable ν in the case where training data are all normal samples and anomaly ratio in the testing phase could be arbitrary.

Original: In this study, we simply perform exhaustive search to ﬁnd the optimal ν that renders the highest F 1 score on individual datasets.
Revised: In this study, we simply perform exhaustive search to ﬁnd the optimal ν that renders the highest F 1 score on individual datasets.

Original: In particular, ν is set to be 0 . 1 , 0 . 02 ,  and 0 .
Revised: In particular, ν is set to be 0 . 1 , 0 . 02 , 0 . 04 , and 0 .

Original: 04 for KDDCUP, Thyroid,and Arrhythmia,  respectively.
Revised: 1 for KDDCUP, Thyroid, Arrhythmia, and KDDCUP-Rev, respectively.

Original: DSEBM .
Revised: DSEBM .

Original: We use the network structure for the encoding in DAGMM as guidelines to set up DSEBM instances.
Revised: We use the network structure for the encoding in DAGMM as guidelines to set up DSEBM instances.

Original: For KDDCUP, it is conﬁgured as FC(120, 60, softplus)-FC(60, 30, softplus)-FC(30, 10, softplus)-FC(10, 1, softplus).
Revised: For KDDCUP and KDDCUP-Rev, it is conﬁgured as FC(120, 60, softplus)-FC(60, 30, softplus)-FC(30, 10, softplus)-FC(10, 1, softplus).

Original: For Thyroid, it is FC(6, 12, softplus)-FC(12, 4, softplus)-FC(4, 1, softplus).
Revised: For Thyroid, it is FC(6, 12, softplus)-FC(12, 4, softplus)-FC(4, 1, softplus).

Original: For Arrhythmia, it is FC(274, 10, softplus)-FC(10, 2, softplus).
Revised: For Arrhythmia, it is FC(274, 10, softplus)-FC(10, 2, softplus).

Original: Moreover, for KDDCUP, Thyroid,and Arrhythmia,  the numbers of epochs are 200, 20000,and 10000,  respectively, and the sizes of mini-batches are 1024, 1024,and 128,  respectively.
Revised: Moreover, for KDDCUP, Thyroid, Arrhythmia, and KDDCUP-Rev, the numbers of epochs are 200, 20000, 10000, and 400, respectively, and the sizes of mini-batches are 1024, 1024, 128, and 1024, respectively.

Original: DCN .
Revised: DCN .

Original: We use the network conﬁguration for the autoencoder in DAGMM as guidelines to set up autoencoders in DCN.
Revised: We use the network conﬁguration for the autoencoder in DAGMM as guidelines to set up autoencoders in DCN.

Original: For KDDCUP, the structure is FC(120, 60, tanh )-FC(60, 30, tanh )-FC(30, 10, tanh )-FC(10, 1, none)-FC(1, 10, tanh )-FC(10, 30, tanh )-FC(30, 60, tanh )-FC(60, 120, none).
Revised: For KDDCUP and KDDCUP-Rev, the structure is FC(120, 60, tanh )-FC(60, 30, tanh )-FC(30, 10, tanh )-FC(10, 1, none)-FC(1, 10, tanh )-FC(10, 30, tanh )-FC(30, 60, tanh )- FC(60, 120, none).

Original: For Thyroid, it is FC(6, 12, tanh )-FC(12, 4, tanh )
Revised: For Thyroid, it is FC(6, 12, tanh )-FC(12, 4, tanh )

Original: -FC(4, 1, none)-FC(1, 4, tanh )
Revised: -FC(4, 1, none)-FC(1, 4, tanh )

Original: -FC(4, 12, tanh )-FC(12, 6, none).
Revised: -FC(4, 12, tanh )-FC(12, 6, none).

Original: For Arrhythmia, it is FC(274, 10, tanh )-FC(10, 2, none)-FC(2, 10, tanh )-FC(10, 274, none).
Revised: For Arrhythmia, it is FC(274, 10, tanh )-FC(10, 2, none)-FC(2, 10, tanh )-FC(10, 274, none).

Original: Moreover, for KDDCUP, Thyroid,and Arrhythmia,  the numbers of epochs for per-layer pre-training are 200, 20000,and 10000,  respectively, the numbers of epochs for ﬁne tuning are 200, 20000,and 10000,  respectively, and the sizes of mini-batches in all the training phases are 1024, 1024,and 128,  respectively.
Revised: Moreover, for KDDCUP, Thyroid, Arrhythmia, and KDDCUP-Rev, the numbers of epochs for per-layer pre-training are 200, 20000, 10000, and 400, respectively, the numbers of epochs for ﬁne tuning are 200, 20000, 10000, and 400, respectively, and the sizes of mini-batches in all the training phases are 1024, 1024, 128, and 1024, respectively.

Original: GMM-EN .
Revised: GMM-EN .

Original: GMM-EN also borrows the wisdom from the network conﬁgurations in DAGMM.
Revised: GMM-EN also borrows the wisdom from the network conﬁgurations in DAGMM.

Original: For KDD, it is FC(120, 60, tanh )-FC(60, 30, tanh )-FC(30, 10, tanh )-FC(10, 1, none)-FC(1, 10, tanh )-Drop(0.5)-FC(10, 4, softmax).
Revised: For KDDCUP, it is FC(120, 60, tanh )-FC(60, 30, tanh )-FC(30, 10, tanh )-FC(10, 1, none)-FC(1, 10, tanh )-Drop(0.5)-FC(10, 4, softmax).

Original: For Thyroid, it is FC(6, 12, tanh )-FC(12, 4, tanh )
Revised: For Thyroid, it is FC(6, 12, tanh )-FC(12, 4, tanh )

Original: -FC(4, 1, none)-FC(1, 10, tanh )-Drop(0.5)-FC(10, 2, softmax).
Revised: -FC(4, 1, none)-FC(1, 10, tanh )-Drop(0.5)-FC(10, 2, softmax).

Original: For Arrhythmia, it is FC(274, 10, tanh )-FC(10, 2, none)-FC(2, 10, tanh )-Drop(0.5)-FC(10, 2, softmax). 
Revised: For Arrhythmia, it is FC(274, 10, tanh )- FC(10, 2, none)-FC(2, 10, tanh )-Drop(0.5)-FC(10, 2, softmax). For KDDCUP-Rev, it is FC(120, 60, tanh )-FC(60, 30, tanh )-FC(30, 10, tanh )-FC(10, 1, none)-FC(1, 10, tanh )-Drop(0.5)-FC(10, 2, softmax).

Original: For KDDCUP, Thyroid,and Arrhythmia,  the numbers of epochs for training are 200, 20000,and 10000,  respectively, and the sizes of mini-batches are 1024, 1024,and 128,  respectively.
Revised: For KDDCUP, Thyroid, Arrhythmia, and KDDCUP-Rev, the numbers of epochs for training are 200, 20000, 10000, and 400, respectively, and the sizes of mini-batches are 1024, 1024, 128, and 1024, respectively.

Original: PAE .
Revised: PAE .

Original: PAE share identical network structures with the autoencoder in DAGMM.
Revised: PAE shares identical network structures with the autoencoder in DAGMM.

Original: For KDDCUP,
Revised: For KDDCUP,

Original: Thyroid,and Arrhythmia,  the numbers of epochs for per-layer pre-training are 200, 20000,and 10000,  respectively, the numbers of epochs for ﬁne tuning are 200, 20000,and 10000,  respectively, and the sizes of mini-batches in all the training phases are 1024, 1024,and 128,  respectively.
Revised: Thyroid, Arrhythmia, and KDDCUP-Rev, the numbers of epochs for per-layer pre-training are 200, 20000, 10000, and 400, respectively, the numbers of epochs for ﬁne tuning are 200, 20000, 10000, and 400, respectively, and the sizes of mini-batches in all the training phases are 1024, 1024, 128, and 1024, respectively.

Original: E2E-AE .
Revised: E2E-AE .

Original: E2E-AE share identical network structures with the autoencoder in DAGMM.
Revised: E2E-AE shares identical network structures with the autoencoder in DAGMM.

Original: For KDDCUP, Thyroid,and Arrhythmia,  the numbers of epochs for end-to-end training are 200, 20000,and 10000,  respectively, and the sizes of mini-batches are 1024, 1024,and 128,  respectively.
Revised: For KDDCUP, Thyroid, Arrhythmia, and KDDCUP-Rev, the numbers of epochs for end-to-end training are 200, 20000, 10000, and 400, respectively, and the sizes of mini-batches are 1024, 1024, 128, and 1024, respectively.

Original: PAE-GMM .
Revised: PAE-GMM-EM .

Original: PAE-GMM and DAGMM share identical network conﬁgurations, but they differ in training strategies.
Revised: PAE-GMM and DAGMM share identical network conﬁgurations. For KDDCUP,

Revised: Thyroid, Arrhythmia, and KDDCUP-Rev, the numbers of epochs for per-layer pre-training are 200, 20000, 10000, and 400, respectively, the numbers of epochs for ﬁne tuning are 200, 20000, 10000, and 400, respectively, and the sizes of mini-batches in all the training phases are 1024, 1024, 128, and 1024, respectively.

Revised: For GMM learning, the EM algorithm stops when the maximum difference of the parameters between current iteration and its previous iteration is smaller than 10 − 6 .

Original:  For KDDCUP, Thyroid,and Arrhythmia,  the numbers of epochs for per-layer pre-training are 200, 20000,and 10000,  respectively, the numbers of epochs for ﬁne tuning or GMM training are 200, 20000,and 10000,  respectively, and the sizes of mini-batches in all the training phases are 1024, 1024,and 128,  respectively.
Revised: PAE-GMM . PAE-GMM and DAGMM share identical network conﬁgurations. For KDDCUP, Thyroid, Arrhythmia, and KDDCUP-Rev, the numbers of epochs for per-layer pre-training are 200, 20000, 10000, and 400, respectively, the numbers of epochs for ﬁne tuning or GMM training are 200, 20000, 10000, and 400, respectively, and the sizes of mini-batches in all the training phases are 1024, 1024, 128, and 1024, respectively.

Original: DAGMM-p .
Revised: DAGMM-p .

Original: DAGMM-p and DAGMM share identical network conﬁgurations, but they are only different in training strategies: DAGMM adopts the strategy of end-to-end training, while DAGMM-p relies on pre-training to compression network and then joint ﬁne-tuning.
Revised: DAGMM-p and DAGMM share identical network conﬁgurations, but they are only different in training strategies: DAGMM adopts the strategy of end-to-end training, while DAGMM-p relies on pre-training to compression network and then joint ﬁne-tuning.

Original: For KDDCUP, Thyroid,and
Revised: For KDDCUP, Thyroid,

Original: Arrhythmia,  the numbers of epochs for per-layer pre-training are 200, 20000,and 10000,  respectively, the numbers of epochs for ﬁne tuning are 200, 20000,and 10000,  respectively, and the sizes of mini-batches in all the training phases are 1024, 1024,and 128,  respectively. 
Revised: Arrhythmia, and KDDCUP-Rev, the numbers of epochs for per-layer pre-training are 200, 20000, 10000, and 400, respectively, the numbers of epochs for ﬁne tuning are 200, 20000, 10000, and 400, respectively, and the sizes of mini-batches in all the training phases are 1024, 1024, 128, and 1024, respectively. DAGMM-NVI . DAGMM and DAGMM-NVI share identical network conﬁgurations and training strategies as discussed in Section 4. B C UMULATIVE DISTRIBUTION FUNCTION OF THE ENERGY FUNCTION LEARNED BY DAGMM

Revised: [Figure]

Revised: Figure 4 shows the cumulative distribution function (cdf) of the energy function learned by DAGMM on KDDCUP, Arrhythmia, Thyroid, and KDDCUP-Rev, respectively.

Revised: In particular, on KDDCUP and

Revised: KDDCUP-Rev, we observe rapid energy increase at around 80% , and most samples whose energy is beyond the 80 th percentile are true anomalous samples.

Revised: C L OW -

Revised: [Figure]

Revised: Figure 5 demonstrates the reduced 1-dimensional representation for KDDCUP samples learned by

Revised: DSEBM, where Figure 5a includes all the samples and Figure 5b includes the normal samples only.

Revised: As shown above, normal and anomalous samples are mixed in the range of [ − 8 , − 7 . 8] .

Revised: For samples in this range, it is difﬁcult to use the energy derived from the latent representation to separate them.

Revised: D R ECONSTRUCTION FEATURES IN DAGMM

Revised: In this section, we detail the discussion on reconstruction features.

Revised: Why reconstruction features are important?

Revised: We realize the importance of reconstruction features from our investigation on a private network security dataset.

Revised: In this dataset, normal samples are normal network ﬂows, and anomalies are network ﬂows with spooﬁng attack.

Revised: As it is difﬁcult to analyze the samples from their original space with 20 dimensions, we utilize deep autoencoders to perform dimension reduction.

Revised: In this case, we are a little bit ambitious, and reduce dimensions from 20 to 1.

Revised: In the reduced 1-dimensional space, for some of the anomalies, we are able to easily separate them from normal samples.

Revised: However, for the rest, their latent representations are quite similar to the representations of the normal samples.

Revised: Meanwhile, in the original space, they are actually quite different from the normal ones.

Revised: Inspired by this observation, we investigate their L 2 reconstruction error, and obtain the plot shown in Figure 1.

Revised: In Figure 1, the red points in the top-right corner are the anomalies sharing similar representations with the normal samples in the reduced space.

Revised: With the additional view from reconstruction error, it becomes easier to separate these anomalies from the normal samples.

Revised: In our study, this concrete example motivates us to include reconstruction features into DAGMM.

Revised: What are the guidelines for reconstruction feature selection?

Revised: In practice, one can select reconstruction features by the following rules.

Revised: First, for an error metric used to derive a reconstruction feature, its analytical form should be continuous and differentiable.

Revised: Second, the output of an error metric should be in a range of relatively small values for the ease of training the estimation network in DAGMM.

Revised: In the experiment of this paper, we select cosine similarity and relative Euclidean distance based on these two rules.

Revised: For cosine similarity, it is continuous and differentiable, and the range of its output is [ − 1 , 1] .

Revised: For relative Euclidean distance, it is also continuous and differentiable.

Revised: Theoretically, the range of its output is [0 , + ∞ ) .

Revised: On the datasets considered in the experiment, we observe that its output is usually a small positive value; therefore, we include this metric as one of the reconstruction features.

Revised: In sum, as long as an error metric meets the above two rules, it could serve as a candidate metric to derive a reconstruction feature for DAGMM.

Revised: E C ASE STUDY : W HEN JOINT TRAINING OUTPERFORMS DECOUPLED TRAINING ?

Revised: In this section, we perform a case study to investigate what kind of samples beneﬁt more from the joint training applied in DAGMM over decoupled training.

Revised: In the evaluation, we employ PAE-GMM as a representative for the methods that leverage decoupled training, and the following results are generated from one run on the KDDCUP dataset.

Revised: [Table]

Revised: Table 4 presents the comparison between DAGMM and PAE-GMM in terms of their anomaly detection results.

Revised: In the testing data of this run, there are 48 , 889 anomalies in total, where 34 , 285 of them are detected by both techniques, 926 anomalies can be detected by neither of them, 1 , 640 can only be detected by PAE-GMM, and 12 , 038 of them can only be detected by DAGMM.

Revised: Next, we drill deeper and investigate the commonalities of these 12 , 038 anomalies that can only be detected by DAGMM.

Revised: Figure 6 illustrates sample distribution in the low-dimensional spaces learned by DAGMM and

Revised: PAE-GMM, where Figure 6a (6b) includes all the normal samples and anomalies, Figure 6c (6d) includes all the normal samples and the anomalies detected by both techniques, and Figure 6e (6f) includes all the normal samples and the anomalies detected by DAGMM only.

Revised: From Figure 6c and 6d, we observe that the anomalies of low cosine similarity and high relative Euclidean distance could be the easy ones that are captured by both techniques.

Revised: For the difﬁcult ones shown in Figure 6e and 6f, we observe that they usually have medium level of relative Euclidean distance (in the range of [1 . 0 , 1 . 2] for both cases) with larger than 0 .

Revised: 6 cosine similarity.

Revised: For such anomalous samples, the model learned by PAE-GMM has difﬁcult time to separate them from the normal samples.

Revised: In addition, we also observe that the model learned by DAGMM tends to assign lower cosine similarity to such anomalies than PAE-GMM does, which also makes it easier to differentiate the anomalies from the normal samples.

Revised: F H OW THE HYPERPARAMETERS IN THE OBJECTIVE FUNCTION IMPACT DAGMM

Revised: As shown in Equation (7), the objective function of DAGMM includes three components: the loss function from deep autoencoder, the energy function from estimation network, and the penalty function for covariance matrices.

Revised: The coefﬁcient ratio among the three components can be characterized as 1 : λ 1 : λ 2 .

Revised: In terms of λ 1 , a large value could make the loss function of deep autoencoder play little role in optimization so that we are unable to obtain a good reduced representation for input samples, while a small value could lead to ineffective estimation network so that GMM is not well trained.

Revised: For λ 2 of a large value, DAGMM tends to ﬁnd GMM with large covariance, which is less desirable as many samples will have high energy as rare events.

Revised: For λ 2 of a small value, the regularization may not be strong enough to counter the singularity effect.

Revised: In our exploration, we ﬁnd the ratio 1 : 0 . 1 : 0 .

Revised: 005 consistently delivers expected results across all the datasets in the experiment.

Revised: To investigate the sensitivity of this ratio, we vary its base and see how different bases affect anomaly detection accuracy.

Revised: For example, when the base is set to 2 , λand λ 2 are adjusted to 0 .

Revised: 2 and 0 .

Revised: 01 , respectively.

Revised: Table 5 shows the average precision, recall, and F 1 score after 20 runs of DAGMM on the KDDCUP dataset.

Revised: As we vary the base from 1 to 9 with step 2 , DAGMM performs in a consistent way, and λ 1 , λ 2 are not sensitive to the changes on the base.

Revised: [Figure]

Revised: [Figure]