Original: V ERSATILE A NOMALY D ETECTION WITH O UTLIER P RESERVING D ISTRIBUTION M APPING A UTOENCODERS
Revised: V ERSATILE O UTLIER D ETECTION WITH O UTLIER P RESERVING D ISTRIBUTION M APPING A UTOENCODERS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: State-of-the-art deep learning methods for outlier detection make the assumption that anomalies will appear far away from inlier data in the latent space produced by distribution mapping deep networks.
Revised: State -of the-art deep learning methods for outlier detection make the assumption that outliers will appear far away from inlier data in the latent space produced by distribution mapping deep networks.

Original: However, this assumption fails in practice, because the divergence penalty adopted for this purpose encourages mapping outliers into the same high-probability regions as inliers.
Revised: However, this assumption fails in practice, because the divergence penalty adopted for this purpose encourages mapping outliers into the same high-probability regions as inliers.

Original: To overcome this shortcoming, we introduce a novel deep learning outlier detection method, called Outlier Preserving
Revised: To overcome this shortcoming, we introduce a novel deep learning outlier detection method, called Outlier Preserving

Original: Distribution Mapping Autoencoder (OP-DMA), which succeeds to map outliers to low probability regions in the latent space of an autoencoder.
Revised: Distribution Mapping Autoencoder (OP-DMA) which succeeds to map outliers to low probability regions in the latent space of an autoencoder.

Original: For this we leverage the insight that outliers are likely to have a higher reconstruction error than inliers.
Revised: For this we leverage the insight that outliers are likely to have a higher reconstruction error than inliers.

Original: We thus achieve outlier-preserving distribution mapping through weighting the reconstruction error of individual points by the value of a multivariate Gaussian probability density function evaluated at those points.
Revised: We thus achieve outlier-preserving distribution mapping through weighting the reconstruction error of individual points by the value of a multivariate Gaussian probability density function evaluated at those points.

Original: This weighting implies that outliers will result  overall penalty if they are mapped to low-probability regions.
Revised: This weighting implies that outliers will result in an overall penalty if they are mapped to low-probability regions.

Original: We show that if the global minimum of our newly proposed loss function is achieved, then our OP-DMA maps inliers to regions with a Mahalanobis distance less than δ , and outliers to regions past this δ , δ being the inverse Chi Squared CDF evaluated at 1 − α with α the percentage of outliers in the dataset.
Revised: We show that if the global minimum of our newly proposed loss function is achieved, then our OP-DMA maps inliers to regions with a Mahalanobis distance less than δ , and outliers to regions past this δ , δ being the inverse Chi Squared CDF evaluated at 1 − α with α the percentage of outliers in the dataset.

Original: Our experiments conﬁrm that OP-DMA consistently outperforms the state-of-art methods on a rich variety of outlier detection benchmark datasets.
Revised: Our experiments conﬁrm that OP-DMA consistently outperforms the state-of-art methods on a rich variety of outlier detection benchmark datasets

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Background and Motivation.
Revised: Background and Motivation.

Original: Anomaly detection, the task of discovering rare or abnormal instances in a dataset, is critical for many applications from fraud detection, error identiﬁcation in measurements, to fault detection in systems (2).
Revised: Outlier detection, the task of discovering rare or abnormal instances in a dataset, is critical for many applications from fraud detection, error identiﬁcation in measurements, to fault detection in systems (2).

Original: Which points are anomalous is often unknown in practice, and thus labeling for which points are outliers versus inliers is scarce or even completely unavailable.
Revised: Which points are outliers is often unknown in practice, and thus labeling for which points are outliers versus inliers is scarce or even completely unavailable.

Original: For this reason, in this work we focus on unsupervised methods for anomaly detection (1).
Revised: For this reason, in this work we focus on unsupervised methods for outlier detection (1).

Original: In the ideal case in which inliers follow a simple unimodal distribution and outliers occur “far away” from the mean compared to the inliers, anomaly detection has a simple solution.
Revised: In the ideal case in which inliers follow a simple unimodal distribution and outliers occur “far away” from the mean compared to the inliers, outlier detection has a simple solution.

Original: In this special case, anomalies can be found by simply hardcoding some ﬁxed distance as the cutoff between which points are classiﬁed as inlier or as outlier, such as is done by EllipticEnvelope (7).
Revised: In this special case, outliers can be found by simply hardcoding some ﬁxed distance as the cutoff between which points are classiﬁed as inlier or as outlier, such as is done by EllipticEnvelope (7).

Original: Unfortunately, in practice data is often not distributed in such a convenient manner (13).
Revised: Unfortunately, in practice data is often not distributed in such a convenient manner (13).

Original: Rather, inliers are distributed in a complex, unknown fashion potentially with multiple clusters of inliers in the feature space (6).
Revised: Rather, inliers are distributed in a complex, unknown fashion potentially with multiple clusters of inliers in the feature space (6).

Original: Other methods with not quite as strict assumptions on the distribution of the data can work on arbitrary distributions, such as One-Class methods (4) and IsolationForest (3) to name just a few.
Revised: Other methods with not quite as strict assumptions on the distribution of the data can work on arbitrary distributions, such as One-Class methods (4) and IsolationForest (3) to name just a few.

Original: These methods typically use a density-based approach (3; 4; 23).
Revised: These methods typically use a density-based approach (3; 4; 23).

Original: However, these methods are known to become infeasible in high dimensional data sets, because in sufﬁciently high dimensional spaces, all data points are known to be roughly equidistant and the space around all points will be sparse (5).
Revised: However, these methods are known to become infeasible in high dimensional data sets, because in sufﬁciently high dimensional spaces, all data points are known to be roughly equidistant and the space around all points will be sparse (5).

Original: State-of-Art Deep Learning Methods for Outlier Detection.
Revised: State-of-Art Deep Learning Methods for Outlier Detection.

Original: Deep learning methods for outlier detection not only overcome the limitations of density and distance based methods, but they can also leverage the inherent featurization capabilities of deep networks.
Revised: Deep learning methods for outlier detection not only overcome the limitations of density and distance based methods, but they can also leverage the inherent featurization capabilities of deep networks.

Original: Recently, deep learning methods use the reconstruction error of an autoencoder-based model for anomaly detection, either by considering the reconstruction error itself as an anomaly score (18; 20) or by leveraging the reconstruction error in a more sophisticated way (10; 19; 9).
Revised: Recently, deep learning methods use the reconstruction error of an autoencoder-based model for outlier detection, either by considering the reconstruction error itself as an outlier score (18; 20) or by leveraging the reconstruction error in a more sophisticated way (10; 19; 9).

Original: However, it has been shown that the usage of the mere reconstruction error can be a suboptimal choice for an anomaly score because anomalies tend to converge to an average reconstruction error that is indistinguishable from that of inliers as the network converges (19).
Revised: However, it has been shown that the usage of the mere reconstruction error can be a suboptimal choice for an outlier score because outliers tend to converge to an average reconstruction error that is indistinguishable from that of inliers as the network converges (19).

Original: This is a greater issue for the case where outliers occur in clusters as opposed to uniform noise, as shown in Figure 1.
Revised: This is a greater issue for the case where outliers occur in clusters as opposed to uniform noise, as shown in Figure 1.

Original: The cluster outlier case is is likely to arise when the outliers represent a small minority class or when they are caused by a systematic error.
Revised: The cluster outlier case is is likely to arise when the outliers represent a small minority class or when they are caused by a systematic error.

Original: [Figure]
Revised: [Figure]

Original:  erroneous assumption of many methods is that outliers are mapped to regions far from the high probability region of the prior distribution
Revised: The erroneous assumption of many methods is that outliers are mapped to regions far from the high probability region of the prior distribution.

Original: To go beyond this problem of the pure reconstruction error, recent works have leveraged the distribution-mapping capabilities of generative models such as Variational Autoencdoers (VAEs) that encourage the data in the latent space to follow a prior distribution (19; 21).
Revised: To go beyond this problem of the pure reconstruction error, recent works have leveraged the distribution-mapping capabilities of generative models such as Variational Autoencdoers (VAEs) that encourage the data in the latent space to follow a prior distribution (19; 21).

Original: These methods assume that due to their anomalous nature outlier points will not be mapped to the encouraged prior distribution while the inliers will be, and consequently outliers will occur in low-probability regions of the prior distribution (19; 21)
Revised: These methods assume that due to their anomalous nature outlier points will not be mapped to the encouraged prior distribution while the inliers will be, and consequently outliers will occur in low-probability regions of the prior distribution (19; 21)

Original: [Figure 2 (b)].
Revised: [Figure 2 (b)].

Original: However, it was observed that this widely held assumption is faulty in practice (21), with the distribution mapping continuing to map both inliers and outliers to high probability regions of the prior distribution for a wide variety of applications [Figure 2 (a)].
Revised: However, it was observed that this widely held assumption is faulty in practice (21), with the distribution mapping continuing to map both inliers and outliers to high probability regions of the prior distribution for a wide variety of applications [Figure 2 (a)].

Original: OP-DMA: Our Approach.
Revised: OP-DMA: Our Approach.

Original: In this work, we propose a method which leverages the distribution mapping capabilities of generative models , but unlike other distribution mapping outlier detection methods we no longer conveniently assume that outliers are not well mapped to the enforced prior distribution.
Revised: In this work, we propose a method which leverages the distribution mapping capabilities of generative models , but unlike other distribution mapping outlier detection methods we no longer conveniently assume that outliers are not well mapped to the enforced prior distribution.

Original: Our core idea instead is to design a novel prior probability-weighted loss function that actively encourages outliers to be mapped to low probability regions in the latent space.
Revised: Our core idea instead is to design a novel prior probability-weighted loss function that actively encourages outliers to be mapped to low probability regions in the latent space.

Original: Our method is an autoencoder architecture that we henceforth refer to as the Outlier Preserving Distribution
Revised: Our method is an autoencoder architecture that we henceforth refer to as the Outlier Preserving Distribution

Original: Mapping Autoencoder (OP-DMA).
Revised: Mapping Autoencoder (OP-DMA).

Original: The network is trained to minimize both the Maximum Mean
Revised: The network is trained to minimize both the Maximum Mean

Original: Discrepancy (MMD) distance between the distribution of the data in the latent space and a multivariate
Revised: Discrepancy (MMD) distance between the distribution of the data in the latent space and a multivariate

Original: Gaussian distribution, and what we call a prior-weighted L2 distance between the input and the reconstruction of the input.
Revised: Gaussian distribution, and what we call a prior-weighted L2 distance between the input and the reconstruction of the input.

Original: This prior-weighted distance weights each input datapoint point by the corresponding value of the multivariate Gaussian PDF evaluated at that point’s latent representation.
Revised: This prior-weighted distance weights each input datapoint point by the corresponding value of the multivariate Gaussian PDF evaluated at that point’s latent representation.

Original: The prior-weighted L2 distance encourages points with a high reconstruction error, as we assume anomalies have, to be placed in low-probability regions in the latent space.
Revised: The prior-weighted L2 distance encourages points with a high reconstruction error, as we assume outliers have, to be placed in low-probability regions in the latent space.

Original: This corresponds to the important observation that anomalies will be mapped far from the mean of the data.
Revised: This corresponds to the important observation that outlier will be mapped far from the mean of the data.

Original: The beauty of this mapping is that it thus allows us to use a simple off-the-shelf distance-based metric to identify outliers post-transformation.
Revised: The beauty of this mapping is that it thus allows us to use a simple off-the-shelf distance-based metric to identify outliers post-transformation.

Original: As the original distribution of the data in the feature space is arbitrary and no distance or density assumptions are placed on the data, OP-DMA thus corresponds to a highly versatile anomaly detection strategy.
Revised: As the original distribution of the data in the feature space is arbitrary and no distance or density assumptions are placed on the data, OP-DMA thus corresponds to a highly versatile outlier detection strategy.

Original: Our key contributions are summarized as follows:
Revised: Our key contributions are summarized as follows:

Original: Introduce OP-DMA, a novel distribution-mapping autoencoder that preserves ourliers in the latent space without making asumptions on the distribution of the data in the feature space.
Revised: Introduce OP-DMA, a novel distribution-mapping autoencoder that preserves ourliers in the latent space without making asumptions on the distribution of the data in the feature space.

Original: Prove that the optimal solution for OP-DMA places outliers further than (cid:114)(cid:82) 1 − αt − n/ 2 − 1 e 12 t 2 n 2 Γ( n 2 ) dt
Revised: Prove that the optimal solution for OP-DMA places outliers further than (cid:114)(cid:82) 1 − αt − n/ 2 − 1 e 12 t 2 n 2 Γ( n 2 ) dt

Original: according to the Mahalanobis distance, where α is the probability that draw from a given dataset is an outlier.
Revised: according to the Mahalanobis distance, where α is the probability that draw from a given dataset is an outlier.

Original: This allows us to use a distance-based anomaly detection classiﬁer 3.
Revised: This allows us to use a distance-based outlier detection classiﬁer 3.

Original: Show experimentally that OP-DMA routinely outperforms standard distribution-mapping autoencoders for anomaly detection on a variety of real-world benchmark datasets
Revised: Show experimentally that OP-DMA routinely outperforms standard distribution-mapping autoencoders for outlier detection on a variety of real-world benchmark datasets

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Outlier Detection Using Autoencoders Coupled with Classical Anomaly Detectors.
Revised: Outlier Detection Using Autoencoders Coupled with Classical Outlier Detectors.

Original: Erfani et al.
Revised: Erfani et al.

Original: (11) used autoencoders for dimension reduction before classifying the projected data with a One
Revised: (11) used autoencoders for dimension reduction before classifying the projected data with a One

Original: Class SVM.
Revised: Class SVM.

Original: Raghavendra et al.
Revised: Raghavendra et al.

Original: (12) introduce a One Class Neural Network (OC-NN), a feed-forward neural network inspired by One Class SVMs for the anomaly-detection.
Revised: (12) introduce a One Class Neural Network (OC-NN), a feed-forward neural network inspired by One Class SVMs for the outlier-detection.

Original: The OC-NN method ﬁrst pretrains an autoencoder before introducing a OC-NN layer after the encoder.
Revised: The OC-NN method ﬁrst pretrains an autoencoder before introducing a OC-NN layer after the encoder.

Original: Unlike our method (OP-DMA) these methods do not encourage outliers in the feature space to remain outliers in the latent space, making the strong and in practice often incorrect assumption that outliers in the feature space remain outliers in the latent space.
Revised: Unlike our method (OP-DMA) these methods do not encourage outliers in the feature space to remain outliers in the latent space, making the strong and in practice often incorrect assumption that outliers in the feature space remain outliers in the latent space.

Original: Recreation Error Methods.
Revised: Recreation Error Methods.

Original: Zhou and Paffenroth (10) introduced Robust Deep Autoencoders (RDA), a deep network version of Robust Principal Component Analysis.
Revised: Zhou and Paffenroth (10) introduced Robust Deep Autoencoders (RDA), a deep network version of Robust Principal Component Analysis.

Original: It separates a given data matrix D into a low-rank matrix L and a sparse matrix S , where anomalous datapoints are isolated in S .
Revised: It separates a given data matrix D into a low-rank matrix L and a sparse matrix S , where anomalous datapoints are isolated in S .

Original: This forces the network to reconstruct the instances in L , and punishes the network for putting instances into S .
Revised: This forces the network to reconstruct the instances in L , and punishes the network for putting instances into S .

Original: Chen et al.
Revised: Chen et al.

Original: (9) introduced RandNet , an ensemble of autoencoders for anomaly detection with anomaly score corresponding to the average recreation error among the networks.
Revised: (9) introduced RandNet , an ensemble of autoencoders for outlier detection with outlier score corresponding to the average recreation error among the networks.

Original: However, as shown in (19), for autoencoders the reconstruction error of outliers often converges to that of inliers.
Revised: However, as shown in (19), for autoencoders the reconstruction error of outliers often converges to that of inliers.

Original: This negatively impacts the performance of such reconstruction error methods.
Revised: This negatively impacts the performance of such reconstruction error methods.

Original: Generative Outlier Detection Methods.
Revised: Generative Outlier Detection Methods.

Original: Perera et.
Revised: Perera et.

Original: al.
Revised: al.

Original: (21) proposed adversarial autoencoders with two discriminator networks for anomaly detection.
Revised: (21) proposed adversarial autoencoders with two discriminator networks for outlier detection.

Original: The ﬁrst discriminator encourages the representation of the data in the latent space to be uniform, while the second encourages the output of the decoder conditioned on random samples from the uniform distribution to match real data samples.
Revised: The ﬁrst discriminator encourages the representation of the data in the latent space to be uniform, while the second encourages the output of the decoder conditioned on random samples from the uniform distribution to match real data samples.

Original: Vu et.
Revised: Vu et.

Original: al.
Revised: al.

Original: (20) used an adversarial autoencoder to map the latent representation of their data to a standard normal distribution.
Revised: (20) used an adversarial autoencoder to map the latent representation of their data to a standard normal distribution.

Original: They then combine the anomaly score from a One Class SVM on the latent data with the One Class score on the reconstructed data as the ﬁnal anomaly score.
Revised: They then combine the outlier score from a One Class SVM on the latent data with the One Class score on the reconstructed data as the ﬁnal outlier score.

Original: These distribution mapping methods rely on the assumption that unlike the inliers, the outliers will be mapped to low-orobability regions of the prior distribution.
Revised: These distribution mapping methods rely on the assumption that unlike the inliers, the outliers will be mapped to low-orobability regions of the prior distribution.

Original: However, as we discussed in Section 1, in practice both inliers and outliers are often well mapped to the prior distribution.
Revised: However, as we discussed in Section 1, in practice both inliers and outliers are often well mapped to the prior distribution.

Original: This is because outliers that are mapped to low-probability regions will generally incur a higher cost from the divergence term which matches the latent distribution to the prior, and unlike in our OP-DMA approach there is not a corresponding term to counteract this cost.
Revised: This is because outliers that are mapped to low-probability regions will generally incur a higher cost from the divergence term which matches the latent distribution to the prior, and unlike in our OP-DMA approach there is not a corresponding term to counteract this cost.

Original: Liu et.
Revised: Liu et.

Original: al. proposed MO-GAAL (26), a deep generative model which uses multiple generator networks to create outlier datapoints near subsets of the input data.
Revised: al. proposed MO-GAAL (26), a deep generative model which uses multiple generator networks to create outlier datapoints near subsets of the input data.

Original: A dsicriminator network is used to identify boundaries around the subsets of real data such that the generated outliers are rejected.
Revised: A dsicriminator network is used to identify boundaries around the subsets of real data such that the generated outliers are rejected.

Original:  P ROPOSED M ETHOD : O UTLIER P RESERVING D ISTRIBUTION M APPING
Revised:  P ROPOSED M ETHOD : O UTLIER P RESERVING D ISTRIBUTION M APPING

Original: The goal of OP-DMA is to ﬁnd a latent representation of a dataset X such that anomalies in X can be easily isolated from inliers in the latent representation of X. That is, we ﬁnd a transformation f that we can apply to a given dataset X with a distribution P X such that f ( X ) ∼ P Z , where P X is an arbitrary and unknown distribution of our input dataset X and P Z is the target distribution.
Revised: The goal of OP-DMA is to ﬁnd a latent representation of a dataset X such that outlier in X can be easily isolated from inliers in the latent representation of X. That is, we ﬁnd a transformation f that we can apply to a given dataset X with a distribution P X such that f ( X ) ∼ P Z , where P X is an arbitrary and unknown distribution of our input dataset X and P Z is the target distribution.

Original: Further, if a ∈ X is an outlier, then f ( a ) should be an outlier in f ( X ) as well.
Revised: Further, if a ∈ X is an outlier, then f ( a ) should be an outlier in f ( X ) as well.

Original: That is, p Z ( f ( a )) should be small.
Revised: That is, p Z ( f ( a )) should be small.

Original: For the purpose of anomaly detection, we chose P Z to be a Gaussian distribution as low probability regions would then simply be values far from the mean.
Revised: For the purpose of outlier detection, we chose P Z to be a Gaussian distribution as low probability regions would then simply be values far from the mean.

Original: We chose an autoencoder architecture to accomplish this distribution mapping, where f is selected to be the encoder Q parameterized by weights θ that are learned through stochastic backpropagation.
Revised: We chose an autoencoder architecture to accomplish this distribution mapping, where f is selected to be the encoder Q parameterized by weights θ that are learned through stochastic backpropagation.

Original: OP-DMA accomplishes the goal of mapping outliers to low probability regions on P Z by weighing the autoencoder’s reconstruction error by the likelihood of the latent data in the latent space with distribution P Z .
Revised: OP-DMA accomplishes the goal of mapping outliers to low probability regions on P Z by weighing the autoencoder’s reconstruction error by the likelihood of the latent data in the latent space with distribution P Z .

Original: In order to accomplish this, we need a one-to-one correspondence between each original datapoint, the latent representation of that datapoint, and the reconstructed point.
Revised: In order to accomplish this, we need a one-to-one correspondence between each original datapoint, the latent representation of that datapoint, and the reconstructed point.

Original: Additionally, while we want the distribution of the latent data to follow P Z on the whole, we do not want all points to be mapped to high-probability regions of the prior.
Revised: Additionally, while we want the distribution of the latent data to follow P Z on the whole, we do not want all points to be mapped to high-probability regions of the prior.

Original: Rather, we must develop a solution that encourages some datapoints (i.e. outliers) to have a low likelihood in the prior distribution.
Revised: Rather, we must develop a solution that encourages some datapoints (i.e. outliers) to have a low likelihood in the prior distribution.

Original: For these reasons, a Wasserstein Autoencoder (WAE) is a more appropriate choice than the more common
Revised: For these reasons, a Wasserstein Autoencoder (WAE) is a more appropriate choice than the more common

Original: Variational Autoencoder (VAE).
Revised: Variational Autoencoder (VAE).

Original: We chose WAEs, instead of the more commonly used VAEs, because the WAEs encourages the latent representations as a whole to match the prior, whereas the loss function of a VAE encourages each individual latent point to represent a distribution that matches the prior (15).
Revised: We chose WAEs, instead of the more commonly used VAEs, because the WAEs encourages the latent representations as a whole to match the prior, whereas the loss function of a VAE encourages each individual latent point to represent a distribution that matches the prior (15).

Original: This is counter to the goal of OP-DMA, which is to have some points (i.e., the outliers) mapped away from the prior distribution.
Revised: This is counter to the goal of OP-DMA, which is to have some points (i.e., the outliers) mapped away from the prior distribution.

Original: Additionally, by formulating the reconstruction process as a Wassertein distance we can leverage the distribution mapping capabilities of the generative model while still operating with both a deterministic encoder and decoder.
Revised: Additionally, by formulating the reconstruction process as a Wassertein distance we can leverage the distribution mapping capabilities of the generative model while still operating with both a deterministic encoder and decoder.

Original: This is important as this allows us to maintain a one-to-one correspondence between a given point in the original space and its corresponding mapped point in the latent space so we can weight the reconstruction error by the value of the prior PDF evaluated in the latent space.
Revised: This is important as this allows us to maintain a one-to-one correspondence between a given point in the original space and its corresponding mapped point in the latent space so we can weight the reconstruction error by the value of the prior PDF evaluated in the latent space.

Original: 3. W ASSERSTEIN A UTOENCODERS
Revised: 3. W ASSERSTEIN A UTOENCODERS

Original: As OP-DMA is an extension of WAE, we ﬁrst brieﬂy describe the structure of WAEs.
Revised: As OP-DMA is an extension of WAE, we ﬁrst brieﬂy describe the structure of WAEs.

Original: WAEs are distribution-mapping autoencoders which minimize the Wasserstein distance between original data and its reconstruction.
Revised: WAEs are distribution-mapping autoencoders which minimize the Wasserstein distance between original data and its reconstruction.

Original: For original data X and reconstruction Y, it is deﬁned as:
Revised: For original data X and reconstruction Y, it is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where Γ denotes the set of all joint distributions of X and Y , such that the marginal distributions Γ( X | Y ) and Γ( Y | X ) are distributed according to P X and P Y respectively, and c is a Wassersteindivergence cost function.
Revised: where Γ denotes the set of all joint distributions of X and Y , such that the marginal distributions Γ( X | Y ) and Γ( Y | X ) are distributed according to P X and P Y respectively, and c is a Wassersteindivergence cost function.

Original: That is, the c-Wasserstein distance is the expectation of the cost function c taken with respect to the joint distribution γ with marginals equal to the two priors, such that γ is a joint distribution that produces the minimum expectation of Equation 1.
Revised: That is, the c-Wasserstein distance is the expectation of the cost function c taken with respect to the joint distribution γ with marginals equal to the two priors, such that γ is a joint distribution that produces the minimum expectation of Equation 1.

Original: Simply ﬁnding the Wasserstein distance, which is the distance that a WAE is trained to minimize, is itself an optimization problem.
Revised: Simply ﬁnding the Wasserstein distance, which is the distance that a WAE is trained to minimize, is itself an optimization problem.

Original: However, it is shown in (17) that when the encoded data Q ( X ) ∼ P Z , where Q is the encoder network and P Z is the prior, then the search over all joint distributions previously required to compute the Wasserstein distance can be replaced with a search over all random encoders, inf γ ∈ Γ E ( X,Y ) ∼ γ [ c ( X, G ( Z ))]
Revised: However, it is shown in (17) that when the encoded data Q ( X ) ∼ P Z , where Q is the encoder network and P Z is the prior, then the search over all joint distributions previously required to compute the Wasserstein distance can be replaced with a search over all random encoders, inf γ ∈ Γ E ( X,Y ) ∼ γ [ c ( X, G ( Z ))]

Original: = inf Q : P Q = P Z E P X E Q
Revised: = inf Q : P Q = P Z E P X E Q

Original: ( Z | X )
Revised: ( Z | X )

Original: [ c ( X, G ( Z ))] .
Revised: [ c ( X, G ( Z ))] .

Original: This means we can minimize the c-Wasserstein distance by ﬁnding the weights of the encoder that produce the minimum expectation of cost c .
Revised: This means we can minimize the c-Wasserstein distance by ﬁnding the weights of the encoder that produce the minimum expectation of cost c .

Original: As is done in (16) where WAE is deﬁned, we can relax the constraint that P Q ∼ P Z by adding a penalty term D
Revised: As is done in (16) where WAE is deﬁned, we can relax the constraint that P Q ∼ P Z by adding a penalty term D

Original: ( P Q , P Z ) , where D is some function that measures the divergence between P Q and P Z .
Revised: ( P Q , P Z ) , where D is some function that measures the divergence between P Q and P Z .

Original: Thus,
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: (2) where λ is a constant weight term that determines how much the divergence of P Q from the prior
Revised: where λ is a constant weight term that determines how much the divergence of P Q from the prior

Original: P Z is penalized.
Revised: P Z is penalized.

Original: In the case of the optimal solution for Q , W λc ( X, Y )
Revised: In the case of the optimal solution for Q , W λc ( X, Y )

Original: =
Revised: =

Original: W c ( X, Y ) .
Revised: W c ( X, Y ) .

Original: In that case the
Revised: In that case the

Original: Wasserstein objective will equal the true c-Wasserstein distance.
Revised: Wasserstein objective will equal the true c-Wasserstein distance.

Original: While the WAE objective function encourages the distribution of the latent data as a whole to match the prior distribution, WAEs do not encourage outliers in the feature space to remain outliers in the latent space.
Revised: While the WAE objective function encourages the distribution of the latent data as a whole to match the prior distribution, WAEs do not encourage outliers in the feature space to remain outliers in the latent space.

Original: Even though the divergence term does not explicitly encourage every point to occur in a high probability region, in practice this is what happens.
Revised: Even though the divergence term does not explicitly encourage every point to occur in a high probability region, in practice this is what happens.

Original: Consider when D is a discriminator network.
Revised: Consider when D is a discriminator network.

Original: The discriminator is likely to learn a boundary around the high probably region of the prior distribution, and the encoder network will be penalized for mapping an outlier to a low probablity region outside of the boundary as the discriminator would correctly identify it as a generated point.
Revised: The discriminator is likely to learn a boundary around the high probably region of the prior distribution, and the encoder network will be penalized for mapping an outlier to a low probablity region outside of the boundary as the discriminator would correctly identify it as a generated point.

Original: On the other hand, as described below, our proposed Outlier-Perserving Distribution Mapping Autoencoders actively encourages outliers to be mapped to low probability region in the latent space.
Revised: On the other hand, as described below, our proposed Outlier-Perserving Distribution Mapping Autoencoders actively encourages outliers to be mapped to low probability region in the latent space.

Original: 3. O UTLIER -P RESERVING D ISTRIBUTION M APPING A UTOENCODERS (OP-DMA)
Revised: 3. O UTLIER -P RESERVING D ISTRIBUTION M APPING A UTOENCODERS (OP-DMA)

Original: So far, we have formulated the distribution mapping process in terms of random encoders and decoders Q and G .
Revised: So far, we have formulated the distribution mapping process in terms of random encoders and decoders Q and G .

Original: However, we can deterministically produce Q ( X ) and G ( Q ( X ) | X ) by requiring Q ( X ) =
Revised: However, we can deterministically produce Q ( X ) and G ( Q ( X ) | X ) by requiring Q ( X ) =

Original: δ µ ( X ) , where µ is some function mapping X to Q ( X ) .
Revised: δ µ ( X ) , where µ is some function mapping X to Q ( X ) .

Original: This allows us to have a one-to-one correspondence between input points X and output G ( Q ( X )) points, so that E P X E Q ( X )
Revised: This allows us to have a one-to-one correspondence between input points X and output G ( Q ( X )) points, so that E P X E Q ( X )

Original: [ c ( X, G ( Q ( X )))] can deﬁne a proper reconstruction error between each point x ∈ X and its corresponding output y ∈ Y .
Revised: [ c ( X, G ( Q ( X )))] can deﬁne a proper reconstruction error between each point x ∈ X and its corresponding output y ∈ Y .

Original: This reconstruction error can be used to determine which points should be mapped to low probability regions in the latent space.
Revised: This reconstruction error can be used to determine which points should be mapped to low probability regions in the latent space.

Original: Let us deﬁne c (cid:48) :
Revised: Let us deﬁne c (cid:48) :

Original: = c ( x, G ( Q ( x ))) · p Z ( Q ( X )) .
Revised: = c ( x, G ( Q ( x ))) · p Z ( Q ( X )) .

Original: We can thus deﬁne the OP-DMA objective function, W λc (cid:48) , as:
Revised: We can thus deﬁne the OP-DMA objective function, W λc (cid:48) , as:

Original: [Equation]
Revised: [Equation]

Original: We can think of p Z ( Q ( X )) as a weight on the reconstruction term.
Revised: We can think of p Z ( Q ( X )) as a weight on the reconstruction term.

Original: Thus, high-reconstruction error points that are mapped to high-probability regions will be penalized more than high-reconstruction error points which are mapped to low probability regions.
Revised: Thus, high-reconstruction error points that are mapped to high-probability regions will be penalized more than high-reconstruction error points which are mapped to low probability regions.

Original: Since anomalies are assumed to result in a high reconstruction error (at least during early epochs of training), by lessening the penalty to the network for poorly reconstructed points that have been mapped to low-probability regions of the prior we encourage the network to map anomalies to these low-probability regions.
Revised: Since outliers are assumed to result in a high reconstruction error (at least during early epochs of training), by lessening the penalty to the network for poorly reconstructed points that have been mapped to low-probability regions of the prior we encourage the network to map outlier to these low-probability regions.

Original: However, the encoder isn’t encouraged to map every point (that is, both inliers and outliers) to low probability regions, as the D term makes it so that the distribution of all the encoded points on the whole should well match the prior.
Revised: However, the encoder isn’t encouraged to map every point (that is, both inliers and outliers) to low probability regions, as the D term makes it so that the distribution of all the encoded points on the whole should well match the prior.

Original: Although we have made a signiﬁcant modiﬁcation to the Wasserstein loss function, as Theorem 1 states minimizing this cost function also corresponds to minimizing a lower bound on the nonprior-weighted Wasserstein divergence:.
Revised: Although we have made a signiﬁcant modiﬁcation to the Wasserstein loss function, as Theorem 1 states minimizing this cost function also corresponds to minimizing a lower bound on the nonprior-weighted Wasserstein divergence:.

Original: The proofs of all theorems below are provided in the appendix for space reasons.
Revised: The proofs of all theorems below are provided in the appendix for space reasons.

Original: Theorem 1.
Revised: Theorem 1.

Original: Let P Z = N ( µ, Σ) .
Revised: Let P Z = N ( µ, Σ) .

Original: Then W c (cid:48)
Revised: Then W c (cid:48)

Original: ( X, G ( Q ( X )))
Revised: ( X, G ( Q ( X )))

Original: ≤ W c ( X, G ( Q ( X )))
Revised: ≤ W c ( X, G ( Q ( X )))

Original: [Equation]
Revised: [Equation]

Original: π [Equation] | Σ | 
Revised: Additionally, if | Σ | ≤

Original: [Equation]
Revised: [Equation]

Original: Additionally, if | Σ | ≤ 1 (4 π n ) thenW λc (cid:48) ( X, G ( Q ( X ))) ≤ W λc ( X, G ( Q ( X )))
Revised: then

Original: [Equation]
Revised: [Equation]

Original: Additionally, not only is the cost function of OP-DMA a lower bound on the c-Wasserstein divergence, but also itself is a Wasserstein divergence (the c (cid:48)
Revised: Additionally, not only is the cost function of OP-DMA a lower bound on the c-Wasserstein divergence, but also itself is a Wasserstein divergence (the c (cid:48)

Original: -Wasserstein divergence):
Revised: -Wasserstein divergence):

Original: Theorem 2.
Revised: Theorem 2.

Original: Let W c be a Wasserstein divergence.
Revised: Let W c be a Wasserstein divergence.

Original: Then W c (cid:48) is a Wasserstein divergence, with c’ the prior-weighted c.
Revised: Then W c (cid:48) is a Wasserstein divergence, with c’ the prior-weighted c.

Original: As stated previously, our ideal mapping would place all inliers within regions where the probability was greater than some value, and all outliers into some regions where the probability is less than that value.
Revised: As stated previously, our ideal mapping would place all inliers within regions where the probability was greater than some value, and all outliers into some regions where the probability is less than that value.

Original: Theorem 3 shows that this scenario is the optimal solution for the loss function of OP-DMA.
Revised: Theorem 3 shows that this scenario is the optimal solution for the loss function of OP-DMA.

Original: Theorem 3.
Revised: Theorem 3.

Original: Let Q be an encoder such that D ( P Q , P Z , F ) = 0, where D ( A, B, F ) is the Maximum
Revised: Let Q be an encoder such that D ( P Q , P Z , F ) = 0, where D ( A, B, F ) is the Maximum

Original: Mean Discrepancy between A an B , F is the set of unbounded continuous functions and P Z =
Revised: Mean Discrepancy between A an B , F is the set of unbounded continuous functions and P Z =

Original: N ( 0 , Σ ) .
Revised: N ( 0 , Σ ) .

Original: Let X : Ω → R n be a centered random variable, X ∼ P X .
Revised: Let X : Ω → R n be a centered random variable, X ∼ P X .

Original: Let X ( A ) , A ⊂ Ω , be outliers and let H
Revised: Let X ( A ) , A ⊂ Ω , be outliers and let H

Original: =
Revised: =

Original: Ω − A be the inliers, where (cid:82) X ( A )
Revised: Ω − A be the inliers, where (cid:82) X ( A )

Original: p X ( x ) dx = α .
Revised: p X ( x ) dx = α .

Original: Furthermore, let c (cid:48)
Revised: Furthermore, let c (cid:48)

Original: ( a, G ( Q ( a )) >
Revised: ( a, G ( Q ( a )) >

Original: c (cid:48)
Revised: c (cid:48)

Original: ( h, G ( Q ( h )) ∀ a ∈ X ( A ) , h ∈ X ( H ) .
Revised: ( h, G ( Q ( h )) ∀ a ∈ X ( A ) , h ∈ X ( H ) .

Original: Then, the optimal solution of OP-DMA is to map such that (cid:107) Q ( X ( A ))
Revised: Then, the optimal solution of OP-DMA is to map such that (cid:107) Q ( X ( A ))

Original: (cid:107) m ≥ δ and (cid:107) Q ( X ( H ))
Revised: (cid:107) m ≥ δ and (cid:107) Q ( X ( H ))

Original: (cid:107) m < δ , where δ = (cid:114) (cid:82) 1 − α t − n/ 2 − 1 e 1 2 t 2 n 2 Γ( n 2 ) dt
Revised: (cid:107) m < δ , where δ = (cid:114)(cid:82) 1 − αt − n/ 2 − 1 e 12 t 2 n 2 Γ( n 2 ) dt

Original: and (cid:107) · (cid:107) m is the Mahalanobis distance.
Revised: and (cid:107) · (cid:107) m is the Mahalanobis distance.

Original: Thus, the optimal solution for OP-DMA’s cost function is one that maps outliers to regions with a larger Mahalanobis distance than that of inliers.
Revised: Thus, the optimal solution for OP-DMA’s cost function is one that maps outliers to regions with a larger Mahalanobis distance than that of inliers.

Original: This has the important implication that after transformation with OP-DMA, outliers can be separated from inliers with a simple distance metric.
Revised: This has the important implication that after transformation with OP-DMA, outliers can be separated from inliers with a simple distance metric.

Revised: This motivates out outlier detection scheme outlined in Section 3.3, where due to Theorem 1 we use the EllipticEnvelope method to identify outliers.

Revised: 3. U NSUPERVISED O UTLIER D ETECTION WITH OP-DMA

Revised: Lastly, we describe our end-to-end solution of outlier detection using OP-DMA.

Revised: First, we transform the distribution of the dataset X to match a Gaussian distribution using OP-DMA.

Revised: That is, the

Original: Encoder
Revised: Encoder

Original: Latent
Revised: Latent

Original: Data
Revised: Data

Original: Decoder
Revised: Decoder

Original: Reconstructed
Revised: Reconstructed

Original: Data
Revised: Data

Original: Original
Revised: Original

Original: Data
Revised: Data

Original: Latent Dist.
Revised: Latent Dist.

Original: Approximate Latent Dist
Revised: Approximate [Figure]

Original: Prior Dist.
Revised: Prior [Figure]

Original: Evaluate prior PDF at each latent point
Revised: [Figure] point

Original: Original
Revised: Original

Original: Data
Revised: Data

Original: OP-DMA Training Process Anomaly
Revised: OP-DMA Training Process Outlier

Original: Detection
Revised: Detection

Original: Procedure
Revised: Procedure

Original: [Figure]

Original: Outlier

Original: Inlier Procedure

Original: [Figure]

Original: [Figure]

Original: Calculate Divergence

Original: Reconstruction Error

Original: Prior-Weighted

Original: [Figure]
Revised: [Figure]

Original: Error

Original: Loss
Revised: Loss

Original: [Figure]
Revised: [Figure]

Original: This motivates out outlier detection scheme outlined in Section 3.3, where due to Theorem 1 we use the EllipticEnvelope method to identify outliers.

Original: 3. U NSUPERVISED A NOMALY D ETECTION WITH OP-DMA

Original: Lastly, we describe our end-to-end solution of anomaly detection using OP-DMA.

Original: First, we transform the distribution of the dataset X to match a Gaussian distribution using OP-DMA.

Original: That is, the input dataset is transformed to match a prior distribution in the latent space of our autoencoder by a divergence term in the network’s loss function that penalizes the divergence between the latent distribution and the prior distribution.
Revised: input dataset is transformed to match a prior distribution in the latent space of our autoencoder by a divergence term in the network’s loss function that penalizes the divergence between the latent distribution and the prior distribution.

Original: For the purpose of anomaly detection, we chose this prior to be a multivariate Gaussian distribution with 0 mean and identity covariance.
Revised: For the purpose of outlier detection, we chose this prior to be a multivariate Gaussian distribution with 0 mean and identity covariance.

Original: Then, EllipticEnvelope (7) is used to determine outliers by ﬁtting a Gaussian to the transformed data.
Revised: Then, EllipticEnvelope (7) is used to determine outliers by ﬁtting a Gaussian to the transformed data.

Original: Theorem 3 tells us that EllipticEnvelope is an appropriate choice for the anomaly detection step, as it determines anomalies based on Mahalanobis distance.
Revised: Theorem 3 tells us that EllipticEnvelope is an appropriate choice for the outlier detection step, as it determines outliers based on Mahalanobis distance.

Original: However, other anomaly detection methods such as IsolationForest (3) or OneClassSVM (4) could equally be leveraged.
Revised: However, other outlier detection methods such as IsolationForest (3) or OneClassSVM (4) could equally be leveraged.

Original: We use the empirical rather than the theoretical mean and covariance for the EllipticEnvelope as it is unrealistic to expect any such deep distribution mapping model to exactly match the prior.
Revised: We use the empirical rather than the theoretical mean and covariance for the EllipticEnvelope as it is unrealistic to expect any such deep distribution mapping model to exactly match the prior.

Original: The anomaly detection process is outlined in Algorithm 1 and in Figure 2.
Revised: The outlier detection process is outlined in Algorithmand in Figure 2.

Original: As seen in Algorithm 1, we use the unbiased estimator of the MMD from Gretton et.
Revised: As seen in Algorithm 1, we use the unbiased estimator of the MMD from Gretton et.

Original: al.
Revised: al.

Original: For the kernel k of the MMD we use the inverse multiquadratics kernel as is used in (16), and Mean Squared Error (MSE) for c .
Revised: For the kernel k of the MMD we use the inverse multiquadratics kernel as is used in (16), and Mean Squared Error (MSE) for c .

Original: A high-level overview of the training and outlier detection process of OP-DMA is given in Figure 3.
Revised: A high-level overview of the training and outlier detection process of OP-DMA is given in Figure 3.

Original:  E XPERIMENTAL E VALUATION 4. E XPERIMENTAL M ETHODOLOGY
Revised:  E XPERIMENTAL E VALUATION 4. E XPERIMENTAL M ETHODOLOGY

Original: Synthetic Data Sets.
Revised: Synthetic Data Sets.

Original: We determine possible failure cases for our method OP-DMA versus a standard autoencoder (AE).
Revised: We determine possible failure cases for our method OP-DMA versus a standard autoencoder (AE).

Original: We use two synthetic datasets: (D1)
Revised: We use two synthetic datasets: (D1)

Original: Multimodal distribution in R 4 with one cluster mean at (0,0,0,0) and another at (5,5,5,5) both with standard covariance and uniform noise added as outliers, and (D2) same multimodal distribution for inliers with a small cluster of outliers with a uniform probability in a small region between (-5,-5,-5,-5) and (-3,-3, -3, -3).
Revised: Multimodal distribution in R 4 with one cluster mean at (0,0,0,0) and another at (5,5,5,5) both with standard covariance and uniform noise added as outliers, and (D2) same multimodal distribution for inliers with a small cluster of outliers with a uniform probability in a small region between (-5,-5,-5,-5) and (-3,-3, -3, -3).

Original: In both cases, inliers account for 2.4% of the data.
Revised: In both cases, inliers account for 2.4% of the data.

Original: Algorithm
Revised: Benchmark Data Sets. We compared OP-DMA to state-of-the-art methods from the ODDs database (8), a commonly used benchmark database of outlier detection datasets. Since we perform unsupervised outlier detection, we do not split each dataset into a training set and testing set. Instead, outlier detection is performed in an unsupervised manner on the entire dataset.

Revised: In each dataset all points are labeled as either inlier or outlier.

Revised: We use these labels to calculate the weighted F1 score, but no method we test is trained on the labels.

Revised: The datasets were chosen such that we could test our methods on datasets with a wide range of dimensionality in the feature space and with a wide range

Original: Require : Regularization coefﬁcient λ
Revised: Require : Regularization coefﬁcient λ

Original: Contamination parameter α
Revised: Contamination parameter α

Original: Initialized encoder network Q Φ and decoder network G Θ with random weights Φ and Θ
Revised: Initialized encoder network Q Φ and decoder network G Θ with random weights Φ and Θ

Original: Dataset X while Θ , Φ not converged do
Revised: Dataset X while Θ , Φ not converged do

Original: Sample { x 1 , x 1 , ..., x n } from X
Revised: Sample { x 1 , x 1 , ..., x n } from X

Original: Sample { z 1 , z 1 , ..., z n } from N ( 0 , I )
Revised: Sample { z 1 , z 1 , ..., z n } from N ( 0 , I )

Original: Sample { ˜ z 1 , ˜ z 1 , ..., ˜ z n } from Q Φ ( Z | X )
Revised: Sample { ˜ z 1 , ˜ z 1 , ..., ˜ z n } from Q Φ ( Z | X )

Original: Update weights Φ and Θ by descending n n (cid:88) 
Revised: Update weights Φ and Θ by descending n n (cid:88) i

Original: [Equation]
Revised: = c ( x i , G Θ (˜ z i )) · λ · p z (˜ z i )+ n ( n − 1) (cid:88) h (cid:54) = j k ( z h , z j )+ n ( n − 1) (cid:88) h (cid:54) = j k (˜ z h , ˜ z

Revised: j ) − 2 n 2 (cid:88) h,j k ( z h , ˜ z

Original:  end
Revised: j ) end

Original: Find D min = { Q Φ ( x i ) , Q Φ ( x j ) , ..., Q Φ ( x k ) } , (cid:107) D min (cid:107) =
Revised: Find D min = { Q Φ ( x i ) , Q Φ ( x j ) , ..., Q Φ ( x k ) } , (cid:107) D min (cid:107) =

Original: (1 − α )
Revised: (1 − α )

Original: (cid:107) D (cid:107) with Minimum
Revised: (cid:107) D (cid:107) with Minimum

Original: Covariance Determinant estimator, inf ˜Σ Det { ˜Σ } .
Revised: Covariance Determinant estimator, inf ˜Σ Det { ˜Σ } .

Original: Find estimated mean ˜ µ from D min return (cid:107) Q Φ ( x i ) (cid:107) m =
Revised: Find estimated mean ˜ µ from D min return (cid:107) Q Φ ( x i ) (cid:107) m =

Original: ( Q Φ ( x i ) − ˜ µ ) (cid:48) ˜Σ( Q Φ ( x i ) − ˜ µ ) for x i ∈ D as anomaly scores Benchmark Data Sets.
Revised: ( Q Φ ( x i ) − ˜ µ ) (cid:48) ˜Σ( Q Φ ( x i ) − ˜ µ ) for x i ∈ D as outlier scores of outlier contamination percentage.

Original: We compared OP-DMA to state-of-the-art methods from the ODDs database (8), a commonly used benchmark database of anomaly detection datasets.

Original: Since we perform unsupervised anomaly detection, we do not split each dataset into a training set and testing set.

Original: Instead, anomaly detection is performed in an unsupervised manner on the entire dataset.

Original: In each dataset all points are labeled as either inlier or outlier.

Original: We use these labels to calculate the weighted

Original: F1 score, but no method we test is trained on the labels.

Original: The datasets were chosen such that we could test our methods on datasets with a wide range of dimensionality in the feature space and with a wide range of outlier contamination percentage.

Original: Table 3 (in Appendix B) breaks doen the number of featurs, datapoints, and percentage of outliers in each dataset.
Revised: Table 3 (in Appendix B) breaks doen the number of featurs, datapoints, and percentage of outliers in each dataset.

Original: Alternative Methods.
Revised: Alternative Methods.

Original: In order to determine the validity of our OP-DMA, we compare it to several state-of-the-art distribution mapping anomaly detection methods.
Revised: In order to determine the validity of our OP-DMA, we compare it to several state-of-the-art distribution mapping outlier detection methods.

Original: These include methods that perform anomaly detection on the latent space of a Wasserstein , a Variational Autoencoder , and an Adversarial
Revised: These include methods that perform outlier detection on the latent space of a Wasserstein , a Variational Autoencoder , and an Adversarial

Original: Autoencoder (25) all with a Gaussian prior, here denoted as WAE, VAE, and AAE respectively.
Revised: Autoencoder (25) all with a Gaussian prior, here denoted as WAE, VAE, and AAE respectively.

Original: Additionally, we test against MO-GAAL (26), a state-of-the-art deep generative outlier detection model.
Revised: Additionally, we test against MO-GAAL (26), a state-of-the-art deep generative outlier detection model.

Original: We also test against LOF (23) and OC-SVM (4), two state-of-the-art non-deep learning outlier detection method.
Revised: We also test against LOF (23) and OC-SVM (4), two state-of-the-art non-deep learning outlier detection method.

Original: Parameter Conﬁgurations of Methods.
Revised: Parameter Conﬁgurations of Methods.

Original: The encoders and decoders of all methods consist of 3layer neural networks, where the decoder in each encoder-decoder pair mirrors the structure of the encoder.
Revised: The encoders and decoders of all methods consist of 3layer neural networks, where the decoder in each encoder-decoder pair mirrors the structure of the encoder.

Original: The number of nodes in the hidden layer of each network is a hyperparameter chosen from {5, 6, 9, 15, 18, 100}.
Revised: The number of nodes in the hidden layer of each network is a hyperparameter chosen from {5, 6, 9, 15, 18, 100}.

Original: The number of nodes in the output/latent layer is chosen from {2, 3, 6, 9, 15}.
Revised: The number of nodes in the output/latent layer is chosen from {2, 3, 6, 9, 15}.

Original: The value of the regularization parameter λ is chosen such that the reconstruction error is on the same order of magnitude as the MMD error for the ﬁrst epoch.
Revised: The value of the regularization parameter λ is chosen such that the reconstruction error is on the same order of magnitude as the MMD error for the ﬁrst epoch.

Original: We use the standard parameters of MO-GAAL from the authors’ publically released code.
Revised: We use the standard parameters of MO-GAAL from the authors’ publically released code.

Original: We use the standard parameters from
Revised: We use the standard parameters from

Original: Scikit-Learn for LOF and OC-SVM.
Revised: Scikit-Learn for LOF and OC-SVM.

Original: 4. E XPERIMENTAL R ESULTS
Revised: 4. E XPERIMENTAL R ESULTS

Original: Synthetic Experiments.
Revised: Synthetic Experiments.

Original: Figure 4 A shows the ratio of the recreation error for outliers/inliers for
Revised: Figure 4 A shows the ratio of the recreation error for outliers/inliers for

Original: OP-DMA versus an autoencoder (AE) for each synthetic dataset.
Revised: OP-DMA versus an autoencoder (AE) for each synthetic dataset.

Original: For uniform noise (D1), both
Revised: For uniform noise (D1), both

Original: OP-DMA and AE feature a high ratio for the error of outlier/inliers.
Revised: OP-DMA and AE feature a high ratio for the error of outlier/inliers.

Original: In this case, a standard AE would be adequate to ﬁnd the outliers.
Revised: In this case, a standard AE would be adequate to ﬁnd the outliers.

Original: However, for data where outliers are a small cluster (D2), the ratio between outliers to inliers goes quickly to 1 for AE whereas it levels off to ~10 for OP-DMA.
Revised: However, for data where outliers are a small cluster (D2), the ratio between outliers to inliers goes quickly to 1 for AE whereas it levels off to ~10 for OP-DMA.

Original: In this case, our OP-DMA succeeds to identify the outliers, while AE cannot.
Revised: In this case, our OP-DMA succeeds to identify the outliers, while AE cannot.

Original: We also verify that OP-DMA maps outliers to low-probability regions of the latent space.
Revised: We also verify that OP-DMA maps outliers to low-probability regions of the latent space.

Original: Figure 4 B shows the latent space of
Revised: Figure 4 B shows the latent space of

Original: OP-DMA for the uniform outlier dataset.
Revised: OP-DMA for the uniform outlier dataset.

Original: The outliers occur far from the mean in low-probability[Table] regions.
Revised: The outliers occur far from the mean in low-probability regions.

Original: The average probability in the prior for outliers after transformation was 0.02, whereas the average probability of the inliers according to the prior was 0.
Revised: The average probability in the prior for outliers after transformation was 0.02, whereas the average probability of the inliers according to the prior was 0.

Original: Outlier Detection Accuracy.
Revised: Outlier Detection Accuracy.

Original: The results reported in Table 2 are the maximum weighted F1-scores of each method over all contamination rates from 0.1 to 1.0 with a step size of 0.
Revised: The results reported in Table 2 are the maximum weighted F1-scores of each method over all contamination rates from 0.1 to 1.0 with a step size of 0.1. [Figure] [Table] [Figure]

Original: [Table]
Revised: [Table]

Revised: [Figure]

Original: Running Time.
Revised: [Table] [Figure] [Table] [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Original:  In order to determine the feasibility of using OP-DMA instead of one of the leading methods in practice, we ran an experiment to determine the running times of each method.
Revised: [Table] [Table] Running Time. In order to determine the feasibility of using OP-DMA instead of one of the leading methods in practice, we ran an experiment to determine the running times of each method.

Original: This consisted of performing 5 runs of OP-DMA, WAE, VAE, and AAE on the Satellite dataset (24).
Revised: This consisted of performing 5 runs of OP-DMA, WAE, VAE, and AAE on the Satellite dataset (24).

Original: The average running time of each method over the 5 iterations is shown in the following table.
Revised: The average running time of each method over the 5 iterations is shown in the following table.

Original: Each method was run on a machine with 32GB of DDR4 RAM and a Intel Xeon Platinum 8160 CPU.
Revised: Each method was run on a machine with 32GB of DDR4 RAM and a Intel Xeon Platinum 8160 CPU.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: OP-DMA outperforms the state-of-the-art methods on the majority of the outlier detection datasets.
Revised: OP-DMA outperforms the state-of-the-art methods on the majority of the outlier detection datasets.

Original: This is because unlike the state-of-the-art methods, OP-DMA actively encourages outliers to be mapped to low probability regions instead of relying on outliers to be poorly mapped to the prior.
Revised: This is because unlike the state-of-the-art methods, OP-DMA actively encourages outliers to be mapped to low probability regions instead of relying on outliers to be poorly mapped to the prior.

Original: While in this work, we focused on unsupervised anomaly detection, OP-DMA could be extended to supervised anomaly detection by making the following change to the objective function:
Revised: While in this work, we focused on unsupervised outlier detection, OP-DMA could be extended to supervised outlier detection by making the following change to the objective function:

Original: [Equation]
Revised: [Equation]

Original: where X in corresponds to the set of labeled inlier points and X out is the set of outliers.
Revised: where X in corresponds to the set of labeled inlier points and X out is the set of outliers.

Original: OP-DMA can also be modiﬁed for semisupervised OP-DMA by pretraining on labeled inliers without weighting the reconstruction error by the likelihood, and then performing OP-DMA as is done in the unsupervised version on the remaining unlabeled datapoints.
Revised: OP-DMA can also be modiﬁed for semisupervised OP-DMA by pretraining on labeled inliers without weighting the reconstruction error by the likelihood, and then performing OP-DMA as is done in the unsupervised version on the remaining unlabeled datapoints.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have introduced OP-DMA, an autoencoder-based distribution mapping method for anomaly detection that maps outliers in the feature space to low probability regions in the latent space in which a multivariate standard normal Gaussian prior distribution is enforced.
Revised: We have introduced OP-DMA, an autoencoder-based distribution mapping method for outlier detection that maps outliers in the feature space to low probability regions in the latent space in which a multivariate standard normal Gaussian prior distribution is enforced.

Original: Anomalies are consequently easily identiﬁable in the latent space.
Revised: Outliers are consequently easily identiﬁable in the latent space.

Original: Our experimental study comparing OP-DMA to state-of-the-art methods on a collection of different benchmark outlier detection datasets shows that it outperforms
Revised: Our experimental study comparing OP-DMA to state-of-the-art methods on a collection of different benchmark outlier detection datasets shows that it outperforms

Original: WAE, VAE, and AAE on the majority of the datasets.
Revised: WAE, VAE and AAE on the majority of the datasets.

Original: We have also demonstrated that there is not a signiﬁcant increase in running time between our method and state-of-the-art methods.
Revised: We have also demonstrated that there is not a signiﬁcant increase in running time between our method and state-of-the-art methods

Original: Appendices
Revised: Appendices

Original: AP ROOFS
Revised: AP ROOFS

Original: Theorem 1.
Revised: Theorem 1.

Original: Let P Z = N ( µ, Σ) .
Revised: Let P Z = N ( µ, Σ) .

Original: Then W c (cid:48)
Revised: Then W c (cid:48)

Original: ( X, G ( Q ( X )))
Revised: ( X, G ( Q ( X )))

Original: ≤ W c ( X, G ( Q ( X ))) 1 (2 π n/ 2 ) | Σ | 1 / 2 .
Revised: ≤ W c ( X, G ( Q ( X ))) 1 (2 π n/ 2 ) | Σ | 1 / 2 .

Original: Additionally, if | Σ | ≤ 1 (4 π n ) then W λc (cid:48) ( X, G ( Q ( X ))) ≤ W λc ( X, G ( Q ( X ))) 1 (2 π n/ 2 ) | Σ | 1 / 2 .
Revised: Additionally, if | Σ | ≤ 1 (4 π n ) then W λc (cid:48) ( X, G ( Q ( X ))) ≤ W λc ( X, G ( Q ( X ))) 1 (2 π n/ 2 ) | Σ | 1 / 2 .

Original: Proof.
Revised: Proof.

Original: We will ﬁrst show that W c (cid:48)
Revised: We will ﬁrst show that W c (cid:48)

Original: ( X, G ( Q ( X )))
Revised: ( X, G ( Q ( X )))

Original: ≤ W c ( X, G ( Q ( X ))) 1 (2 π n/ 2 ) | Σ | 1 / 2 , and then show that W λc (cid:48)
Revised: ≤ W c ( X, G ( Q ( X ))) 1 (2 π n/ 2 ) | Σ | 1 / 2 , and then show that W λc (cid:48)

Original: ( X, G ( Q ( X )))
Revised: ( X, G ( Q ( X )))

Original: ≤ W λc ( X, G ( Q ( X ))) 1 (2 π n/ 2 ) | Σ | 1 / 2 .
Revised: ≤ W λc ( X, G ( Q ( X ))) 1 (2 π n/ 2 ) | Σ | 1 / 2 .

Original: The maximum value of p Z ( z ) will occur at z = µ .
Revised: The maximum value of p Z ( z ) will occur at z = µ .

Original: Then, sup p Z = 1 (2 π n/ 2 ) | Σ | 1 / 2 .
Revised: Then, sup p Z = 1 (2 π n/ 2 ) | Σ | 1 / 2 .

Original: Thus
Revised: Thus

Original: [Equation]
Revised: [Equation]

Original: Similarly, to show that the prior-weighted c-Wasserstein divergence is a constant multiple of the true
Revised: Similarly, to show that the prior-weighted c-Wasserstein divergence is a constant multiple of the true

Original: Wasserstein divergence:
Revised: Wasserstein divergence:

Original: [Equation]
Revised: [Equation]

Original: Theorem 2.
Revised: Theorem 2.

Original: Let W c be a Wasserstein divergence.
Revised: Let W c be a Wasserstein divergence.

Original: Then W c (cid:48) is a Wasserstein divergence, with c’ the prior-weighted c.
Revised: Then W c (cid:48) is a Wasserstein divergence, with c’ the prior-weighted c.

Original: Proof.
Revised: Proof.

Original: Since c is a Wasserstein divergence, we know that c ( x 1 , x 2 ) ≥ 0 ∀ x 1 , x 2 ∈ supp ( P ) , c ( x, x ) = 

Revised: Since c is a Wasserstein divergence, we know that c ( x 1 , x 2 ) ≥ 0 ∀ x 1 , x 2 ∈ supp ( P ) , c ( x, x ) =

Original: ∀ x ∈ supp ( P ) , and E γ [ c ( x 1 , x 2 )] ≥ 0 ∀ γ ∈ Γ
Revised: ∀ x ∈ supp ( P ) , and E γ [ c ( x 1 , x 2 )] ≥ 0 ∀ γ ∈ Γ

Original: [ P, P z ] .
Revised: [ P, P z ] .

Original: Since P Z ( z ) ≥ 0 ∀ z , c (cid:48) will also fulﬁll the three aforementioned properties of c .
Revised: Since P Z ( z ) ≥ 0 ∀ z , c (cid:48) will also fulﬁll the three aforementioned properties of c .

Original: Thus, W c (cid:48) is a Wasserstein divergence.
Revised: Thus, W c (cid:48) is a Wasserstein divergence.

Original: Theorem 3.
Revised: Theorem 3.

Original: Let Q be an encoder such that D ( P Q , P Z , F ) = 0, where D ( A, B, F ) is the Maximum
Revised: Let Q be an encoder such that D ( P Q , P Z , F ) = 0, where D ( A, B, F ) is the Maximum

Original: Mean Discrepancy between A an B , F is the set of unbounded continuous functions and P Z =
Revised: Mean Discrepancy between A an B , F is the set of unbounded continuous functions and P Z =

Original: N ( 0 , Σ ) .
Revised: N ( 0 , Σ ) .

Original: Let X : Ω → R n be a centered random variable, X ∼ P X .
Revised: Let X : Ω → R n be a centered random variable, X ∼ P X .

Original: Let X ( A ) , A ⊂ Ω , be outliers and let H
Revised: Let X ( A ) , A ⊂ Ω , be outliers and let H

Original: =
Revised: =

Original: Ω − A be the inliers, where (cid:82) X ( A )
Revised: Ω − A be the inliers, where (cid:82) X ( A )

Original: p X ( x ) dx = α .
Revised: p X ( x ) dx = α .

Original: Furthermore, let c (cid:48)
Revised: Furthermore, let c (cid:48)

Original: ( a, G ( Q ( a )) >
Revised: ( a, G ( Q ( a )) >

Original: c (cid:48)
Revised: c (cid:48)

Original: ( h, G ( Q ( h )) ∀ a ∈ X ( A ) , h ∈ X ( H ) .
Revised: ( h, G ( Q ( h )) ∀ a ∈ X ( A ) , h ∈ X ( H ) .

Original: Then, the optimal solution of OP-DMA is to map such that (cid:107) Q ( X ( A ))
Revised: Then, the optimal solution of OP-DMA is to map such that (cid:107) Q ( X ( A ))

Original: (cid:107) m ≥ δ and (cid:107) Q ( X ( H ))
Revised: (cid:107) m ≥ δ and (cid:107) Q ( X ( H ))

Original: (cid:107) m < δ , where δ = (cid:114)(cid:82) 1 − αt − n/ 2 − 1 e 12 t 2 n 2 Γ( n 2 )
Revised: (cid:107) m < δ , where δ = (cid:114)(cid:82) 1 − αt − n/ 2 − 1 e 12 t 2 n 2 Γ( n 2 )

Original: dt
Revised: dt

Original: and (cid:107) · (cid:107) m is the Mahalanobis distance.
Revised: and (cid:107) · (cid:107) m is the Mahalanobis distance.

Original: Proof.
Revised: Proof.

Original: The Mahalanobis distance of Q ( X ) can itself be expressed as a random variable, δ = (cid:112)
Revised: The Mahalanobis distance of Q ( X ) can itself be expressed as a random variable, δ = (cid:112)

Original: Q ( X )Σ − 1 Q ( X ) T .
Revised: Q ( X )Σ − 1 Q ( X ) T .

Original: Let Φ δ be the CDF of δ .
Revised: Let Φ δ be the CDF of δ .

Original: Then, Φ δ (1 − α )
Revised: Then, Φ δ (1 − α )

Original: =
Revised: =

Original: P ( δ ≤ 1 − α ) =
Revised: P ( δ ≤ 1 − α ) =

Original: P ( δ 2 ≤ (1 − α ) 2 ) =
Revised: P ( δ 2 ≤ (1 − α ) 2 ) =

Original: Φ δ 2 ((1 − α ) 2 ) .
Revised: Φ δ 2 ((1 − α ) 2 ) .

Original: Let Y = Q ( X ) M − 1 , where M T M = Σ is the Choleski decomposition of the covariance Σ .
Revised: Let Y = Q ( X ) M − 1 , where M T M = Σ is the Choleski decomposition of the covariance Σ .

Original: Since
Revised: Since

Original: D ( P Q , P Z , F ) = 0, and D ( A, B, F ) = 

Revised: D ( P Q , P Z , F ) = 0, and D ( A, B, F ) =

Original: iff A = B , we thus know that Q ( X ) ∼ N ( 0 , Σ ) .
Revised: iff A = B , we thus know that Q ( X ) ∼ N ( 0 , Σ ) .

Original: Thus since
Revised: Thus since

Original: Q ( X ) is normally distributed and centered, Y thus is normally distributed with identity covariance.
Revised: Q ( X ) is normally distributed and centered, Y thus is normally distributed with identity covariance.

Original: Since δ 2 = Q ( X )Σ − 1 Q ( X ) T =
Revised: Since δ 2 = Q ( X )Σ − 1 Q ( X ) T =

Original: Y Y T , Φ δ 2 is the CDF of of the sum of squares of n normally distributed variables with mean 0 and σ = 1 .
Revised: Y Y T , Φ δ 2 is the CDF of of the sum of squares of n normally distributed variables with mean 0 and σ = 1 .

Original: Thus, Φ δ 2 is the Chi Squared distribution.
Revised: Thus, Φ δ 2 is the Chi Squared distribution.

Original: The inverse
Revised: The inverse

Original: Chi Squared CDF will therefore give us the distance δ such that 1 − α percent of the points are within δ :
Revised: Chi Squared CDF will therefore give us the distance δ such that 1 − α percent of the points are within δ :

Original: [Equation]
Revised: [Equation]

Original: Now, let us assume that for some parameter choice Θ (cid:48) for Q that αP ( Q ( X ( A ) | Θ (cid:48) ) ≤ δ )
Revised: Now, let us assume that for some parameter choice Θ (cid:48) for Q that αP ( Q ( X ( A ) | Θ (cid:48) ) ≤ δ )

Original: =
Revised: =

Original: β , β > 0 .
Revised: β , β > 0 .

Original: Consequently, (1 − α ) P ( Q ( X ( H ) | Θ (cid:48) )
Revised: Consequently, (1 − α ) P ( Q ( X ( H ) | Θ (cid:48) )

Original: > δ )
Revised: > δ )

Original: =
Revised: =

Original: β , since P ( Q ( X ) > δ ) =
Revised: β , since P ( Q ( X ) > δ ) =

Original: α and (cid:82)
Revised: α and (cid:82)

Original: X ( A ) p X
Revised: X ( A ) p X

Original: ( x ) dx =
Revised: ( x ) dx =

Original: α .
Revised: α .

Original: Conversely, let us assume that there is a parameter conﬁguration Θ such
Revised: Conversely, let us assume that there is a parameter conﬁguration Θ such

Original: that αP ( Q ( X ( A ) | Θ) ≤ δ ) = 

Revised: that αP ( Q ( X ( A ) | Θ) ≤ δ ) =

Original: and so (1 − α )
Revised: and so (1 − α )

Original: P ( Q ( X ( H ) | Θ) > δ )
Revised: P ( Q ( X ( H ) | Θ) > δ )

Original: = 0 .
Revised: = 0 .

Original: Since P Z ∼ N ( 0 , Σ ) , p Z ( d 1 ) <
Revised: Since P Z ∼ N ( 0 , Σ ) , p Z ( d 1 ) <

Original: p Z ( d 2 ) for (cid:107) d 1 (cid:107) m > (cid:107) d 2 (cid:107) m .
Revised: p Z ( d 2 ) for (cid:107) d 1 (cid:107) m > (cid:107) d 2 (cid:107) m .

Original: Thus, since we assume c ( a, G ( Q ( a )) > c ( h, G ( Q ( h )) ∀ a ∈ X ( A ) , h ∈ X ( H ) , then E P X E Q
Revised: Thus, since we assume c ( a, G ( Q ( a )) > c ( h, G ( Q ( h )) ∀ a ∈ X ( A ) , h ∈ X ( H ) , then E P X E Q

Original: ( Z | X ) c (cid:48) ( x p , G ( Q ( x p | Θ (cid:48) ))) =
Revised: ( Z | X ) c (cid:48) ( x p , G ( Q ( x p | Θ (cid:48) ))) =

Original: E P X E Q ( Z | X ) c ( x p , G ( Q ( x p | Θ (cid:48) )))
Revised: E P X E Q ( Z | X ) c ( x p , G ( Q ( x p | Θ (cid:48) )))

Original: p z ( x p ) >
Revised: p z ( x p ) >

Original: E P
Revised: E P

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: E P X E Q (
Revised: E P X E Q (

Original: [Equation]
Revised: [Equation]

Original: c (cid:48) (
Revised: c (cid:48) (

Original: [Equation]
Revised: [Equation]

Original: G ( Q
Revised: G ( Q

Original: [Equation]
Revised: [Equation]

Original: x p
Revised: x p

Original: [Equation]
Revised: [Equation]

Original: Θ))) .
Revised: Θ))) .

Original: Thus, a mapping that places outliers δ away from the mean will have a lower loss than one that places some percentage of outliers less than δ from the mean
Revised: Thus, a mapping that places outliers δ away from the mean will have a lower loss than one that places some percentage of outliers less than δ from the mean

Original: [Table]
Revised: [Table]

Original: C S ENSITIVITY TO C ONTAMINATION P ARAMETER
Revised: C S ENSITIVITY TO C ONTAMINATION P ARAMETER

Original: While the contamination parameter α is not used during the training of OP-DMA, it is used to ﬁt the Elliptic Envelope on the encoded data after training.
Revised: While the contamination parameter α is not used during the training of OP-DMA, it is used to ﬁt the Elliptic Envelope on the encoded data after training.

Original: We test the sensitivity of this method to the value of the contamination parameter by evaluating the F1 score of outlier detection on the Satellite dataset after encoding the data with OP-DMA.
Revised: We test the sensitivity of this method to the value of the contamination parameter by evaluating the F1 score of outlier detection on the Satellite dataset after encoding the data with OP-DMA.

Original: The results are shown in the ﬁgure below.
Revised: The results are shown in the ﬁgure below.

Original: This analysis shows that as long as the contamination parameter is not signiﬁcantly less than the true contamination percentage the F1-score is robust to different values of the contamination parameter.
Revised: This analysis shows that as long as the contamination parameter is not signiﬁcantly less than the true contamination percentage the F1-score is robust to different values of the contamination parameter.