U AI N ETS : F ROM U NSUPERVISED TO A CTIVE D EEP A NOMALY D ETECTION
Anonymous authors
Paper under double-blind review
A BSTRACT
This work presents a new method for active anomaly detection which can be built upon existing deep learning solutions for unsupervised anomaly detection.
We show that a prior needs to be assumed on what the anomalies are, in order to have performance guarantees in unsupervised anomaly detection.
We argue that active anomaly detection has, in practice, the same cost of unsupervised anomaly detection but with the possibility of much better results.
To solve this problem, we present a new layer that can be attached to any deep learning model designed for unsupervised anomaly detection to transform it into an active method, presenting results on both synthetic and real anomaly detection datasets
I NTRODUCTION
Anomaly detection (a.k.a. outlier detection) (Hodge & Austin, 2004; Chandola et al., 2009; Aggarwal, 2015) aims to discover rare instances that do not conform to the patterns of majority.
From a business perspective, though, we are not only interested in ﬁnding rare instances, but “interesting anomalies”.
This problem has been amply studied recently (Liu et al., 2017; Li et al., 2017; Zong et al., 2018;
Maurus & Plant, 2017; Zheng et al., 2017), with solutions inspired by extreme value theory (Siffer et al., 2017), robust statistics (Zhou & Paffenroth, 2017) and graph theory (Perozzi et al., 2014).
Unsupervised anomaly detection is a sub-area of outlier detection which aims to discover these rare instances in already ‘contaminated’ datasets.
It is a specially hard task, where there is usually no information on what these rare instances are and most works use models with implicit priors or heuristics to discover these anomalies, providing an anomaly score s ( x ) for each instance in this dataset.
Active anomaly detection is an interesting alternative approach to this problem, which has presented good results in recent works such as (Veeramachaneni et al., 2016; Das et al., 2016; 2017).
In this work, we ﬁrst show that unsupervised anomaly detection requires priors to be assumed on the anomaly distribution; we then argue in favor of approaching it with active anomaly detection, an important, but under-explored approach (Section 2).
We propose a new layer, called here Universal Anomaly Inference (UAI), which can be applied on top of any unsupervised anomaly detection model based on deep learning to transform it into an active model (Section 3).
This layer uses the strongest assets of deep anomaly detection models, i.e. its learned latent representations ( l ) and anomaly score ( s ), to train a classiﬁer on the few already labeled instances.
An example of such an application can be seen in Figure 1, where an UAI layer is built upon a Deanoising AutoEncoder (DAE).
We then present extensive experiments, analyzing the performance of our systems vs unsupervised, semi-supervised and active ones under similar budgets in both synthetic and real data, showing our algorithm improves state of the art results in several datasets, with no hyperparameter tuning (Section 4).
Finally, we visualize our models learned latent representations, comparing them to unsupervised models’ ones and analyze our model’s performance for different numbers of labels (Appendix C).
P ROBLEM D EFINITION
Grubbs (1969) deﬁnes an outlying observation, or outlier, as one that appears to deviate markedly from other members of the sample in which it occurs.
Hawkins (1980) states that an outlier is an observation that deviates so much from other observations as to arouse suspicion that it was generated by a different mechanism.
While Chandola et al.
(2009) says that normal data instances occur in high probability regions of a stochastic model, while anomalies occur in the low probability ones.
Following these deﬁnitions, specially the one from (Hawkins, 1980), we assume there is a probability density function
[Figure]
from which our ‘normal’ data instances are generated: X normal ∼ p normal ( x )
= p ( x | y = 0) , where x is an instance’s available information 1 and y is a label saying if the point is anomalous or not.
There is also a different probability density function from which anomalous data instances are sampled:
X anom ∼ p anom ( x ) = p ( x | y = 1) .
In this problem, a dataset would be composed of both normal and anomalous instance, being sampled from a probability distribution that follows:
[Equation]
where λ is an usually small constant representing the probability of a random data point being anomalous ( λ = p ( y = 1) ), this constant can be either known a priori or not.
Chandola et al.
(2009) divides anomaly detection learning systems in three different types:
• Supervised: A training and a test set are available with curated labels for non-anomalous and anomalous instances.
This case is similar to an unbalanced supervised classiﬁcation setting
[Equation]
• Semi-Supervised: A training set is available containing only non-anomalous instances and the challenge
is to
identify anomalous instances in a test set
This is also called novelty detection:
[Equation]
Unsupervised: A dataset containing both non-anomalous and anomalous instance is available and the challenge is to identify anomalous instances in it.
There is no concept of a test set since anomalous instances must be sorted in the dataset itself:
[Equation]
2. U NSUPERVISED A NOMALY D ETECTION
In this work we will focus on unsupervised anomaly detection .
Here, in possession of the full set of points X ∼ p full ( x ) , we want to ﬁnd a subset X anom ⊂ X which is composed of the anomalous instances.
The full distribution p full is a mixture of distributions and if these distributions overlap very closely, it may be impossible to learn the individual distributions beyond a certain accuracy threshold (Dasgupta et al., 2005). It is a well-known result that general mixture models are unidentiﬁable (Aragam et al., 2018; Bordes et al., 2006).
In the sequence, we further show that we gain no information on p anom from p full for any small λ without a prior on the anomalies’ probability distribution.
This differs from the usual unidentiﬁability of mixture models result in that we make no assumptions on the prior for p normal , showing all valid distributions of p anom are equally probable.
Theorem 1.
No free anomaly theorem.
Consider two independent arbitrary probability distributions p normal and p anom .
For a small number of anomalies λ ≈ 0 , p full = p gives us no further knowledge on the distribution of p anom :
[Equation]
where
[Equation]
From Theorem 1 we can conclude that unsupervised anomaly detection requires a prior on the anomalies distribution.
A more tangible example of this can be seen in Figure 2, where we present a synthetic data distribution composed of three classes of data clustered in four visibly separable clusters.
Anomaly detection is an undecidable problem in this setting without further information, since it is impossible to know if the low density cluster is composed of anomalies or the anomalies are the unclustered low density points (or a combination of both).
If we used a high capacity model to model the data distribution in Figure 2, the low density points (Right) would be detected as anomalous.
If we used a low capacity model, the cluster (Center) would probably present a higher anomaly score.
Our choice of algorithm implicitly imposes a prior on the detected anomalies.
Theorem 1 highlights this and makes the need to consider priors explicit.
In a more practicle example, assume we are working with clinical data.
In this setting, some low density clusters may indicate diseases (anomalies), while other low density clusters may be caused by uncontrolled factors in the data, such as high performance athletes.
At the same time, rare diseases might seem like scattered (low density) points.
We want to be able to distinguish between anomalies and ‘uninteresting’ low probability points. [Figure]
M ODEL
The usual strategy when working with unsupervised anomaly detection problems is training a parameterized model p θ ( x ) to capture the full data distribution p full ( x ) (e.g. a PCA, or AutoEncoder), and, since λ is, by deﬁnition, a small constant, assuming p full ( x ) ≈ p normal ( x ) and assuming points with low probability are anomalous (Zhou & Paffenroth, 2017).
An anomaly score s ( x ) is then deﬁned as s ( x ) = 1 p ( x ) .
There are three main problems with this strategy: (1) if anomalous items are more common than expected, p full might be a poor approximation of p normal ; (2) if anomalous items are tightly clustered in some way, high capacity models may learn to identify that cluster as a high probability region; (3) if anomalous items are as rare as expected, since we only have access to p full , Theorem 1 states we have no information about p anom without further assumptions on its probability distribution.
Most unsupervised anomaly detection systems also already rely on further veriﬁcation of the results by human experts, due to their uncertain performance.
Being mostly used as a ranking system to get high probability instances in the top of a ‘list’ to be further audited by these experts.
From Theorem 1, we conclude it is impossible to have an universal and reliable unsupervised anomaly detection system, while we know that most such systems already rely on the data being later audited by human experts.
These arguments together argue in favor of an active learning strategy for anomaly detection, including the auditor experts in the system’s training loop.
Thus, anticipating feedback and beneﬁting from it to ﬁnd further anomalous instances, which results in a more robust system.
Having an extremely unbalanced dataset in this problem ( λ ≈ 0 ) is also another justiﬁcation for an active learning setting, which has the potential of requiring exponentially less labeled data than supervised settings (Settles, 2012).
3. A CTIVE A NOMALY D ETECTION
With these motivations, we argue in favor of active anomaly detection methods, which despite its many advantages remains an under-explored approach to this problem.
Nonetheless, recent work has shown promising results (Veeramachaneni et al., 2016; Das et al., 2016; 2017).
In unsupervised anomaly detection, we start with a dataset D = { x | x ∼ p full ( x ) } and want to rank elements in this dataset so that we have the highest possible recall/precision for a certain budget b , which is the number of elements selected to be audited by an expert, with no prior information on anomaly labels.
In active anomaly detection, we also start with a completely unlabeled anomaly detection dataset
D = { x | x ∼ p full ( x ) } , but instead of ranking anomalies and sending them all to be audited at once by our expert, we select them in small parts, waiting for the experts feedback before continuing.
We iteratively select the most probable k (cid:28) b elements to be audited 2 , wait for the expert to select their label, and continue training our system using this information, as shown in Algorithm 1.
This requires the same budget b as an unsupervised anomaly detection system, while having the potential of achieving a much better performance. labels ) 6: labels
[Equation]
[Equation]
With this in mind, we develop the Universal Anomaly Inference (UAI) layer.
This layer can be incorporated on top of any deep learning based white box anomaly detection system which provides an anomaly score for ranking anomalies.
It takes as input both a latent representation layer ( l ( x ) ), created by the model, and its output anomaly score ( s ( x ) ), and passes it through a classiﬁer to ﬁnd an item’s anomaly probability.
[Equation]
This is motivated by recent works stating learned representations have a simpler statistical structure (Bengio et al., 2013), which makes the task of modeling this manifold and detecting unnatural points much simpler (Lamb et al., 2018).
In this work, we model the UAI layer using a simple logistic regression as our classiﬁer, but any architecture could be used here:
[Equation]
where W act ∈ R 1 ,d +1 is a linear transformation, b act ∈ R is a bias term and σ ( · ) is the sigmoid function.
We learn the values of W and b using back-propagation with a cross entropy loss function, while allowing the gradients to ﬂow through l , but not through s , since s might be non-differentiable.
For the rest of this document, we will refer to the networks with a UAI layer as UaiNets.
An example of this architecture is shown in Figure 1.
E XPERIMENTS
In this section, we test our new UAI layer on top of two distinct architectures: a Denoising
AutoEncoder (DAE, with s dae ( x ) =
|| x − ˆ x || 2 2 ) and a Classiﬁer ( Class , with s class ( x )
= want to have the most anomalies for any given budget. Besides, because anomaly detection is already a highly imbalanced problem, we might not get anomalous instances even when picking only the top anomalous results, so actively searching for them is usually a good choice. This approach follows recent work in active anomaly detection (Veeramachaneni et al., 2016; Das et al., 2016; 2017) cross _ entropy ( x y , (cid:99) x y ) ), which use standard multi layer perceptrons.
Both architectures are described in details in Appendix A.1.
To test our algorithm we start by analyzing its performance on synthetic data created with different properties (Section 4.1).
We then present results using UaiNets on real anomaly detection datasets (Section 4.2) and in a semi-supervised setting (Section 4.3).
S YNTHETIC D ATA
When designing experiments, we had the objective of showing that our model can work with different deﬁnitions of anomaly, while completely unsupervised models will need, by deﬁnition, to trade-off accuracy in one setting for accuracy in the other.
While this may seem straight forward, these results can show how robust our approach is to the choice of underlying architecture, analyzing how well they do when their underlying architecture has a bad prior for that speciﬁc “type” of anomaly.
With this in mind, we used the MNIST dataset and deﬁned four sets of experiments:
MNIST
: For the ﬁrst set of experiments, we reduced the presence of the 0 digit class to only 10% of its original number of samples, making it only 1 / 91 ≈ 1 .
1% of the dataset samples.
The 0 s still present in the dataset had its class randomly changed to x y ∼ Uniform ([1; 9]) and were deﬁned as anomalies.
MNIST 0-2 : The second set of experiments follows the same dataset construction, but we reduce the number of instances of numbers 0 , 1 and 2 , changing the labels of the remaining items in these categories to x y ∼ Uniform ([3; 9]) , and again deﬁning them as anomalous.
In this dataset anomalies composed 3 / 73 ≈ 4 .
1% of the dataset.
MNIST hard : The third set of experiments aims to test a different type of anomaly.
In order to create this dataset, we ﬁrst trained a weak one hidden layer MLP classiﬁer on MNIST and selected all misclassiﬁed instances as anomalous, keeping them in the dataset with their original properties ( x x and x y ).
In this dataset anomalies composed ≈ 3 .
3% of the dataset. 4.
MNIST pca :
In this set of experiments, for each image class ( x y ), we used a PCA to reduce the dimensionality of MNIST images ( x x ) to 2 and selected the 5% instances with the largest reconstruction error as anomalies.
We kept all 60 , 000 instances in the dataset with their original properties ( x x and x y ) and in this dataset anomalies composed 5% of the dataset.
Results for these experiments are shown in Figure 3 and the main conclusion taken from them is that, even though our algorithm might not get better results than its underlying model for every budget-dataset pair, it is robust to different types of anomalies, which is not the case for the underlying completely unsupervised models.
While Class gives really good results in MNIST 0 and MNIST 0datasets, it does not achieve the same performance in MNIST hard and MNIST pca , which might indicate it is better at ﬁnding clustered anomalies than low density ones.
At the same time, DAE has good results for MNIST pca and MNIST hard , but bad ones for MNIST 0 and MNIST 0-2 , which indicates it is better at ﬁnding low density anomalies than clustered ones.
Nevertheless, both UaiNets are robust in all four datasets, being able to learn even on datasets which are hard for their underlying models, although they might have a cold start to produce results.
4. R EAL D ATA
Here we analyze our model’s performance on public benchmarks composed of real anomaly detection datasets.
We employ 11 datasets in our analysis: KDDCUP; Thyroid; Arrhythmia; KDDCUP-Rev;
Yeast; Abalone; Cardiotocography (CTG); Credit Card; Covtype; Mammography (MMG); Shuttle (Lichman, 2013; Dheeru & Taniskidou, 2017; Pozzolo et al., 2015; Woods et al., 1993).
We compare our algorithm against: DAE (Vincent et al., 2008); DAGMM (Zong et al., 2018); LODA-AAD ( et al and
[Figure]
Table 1 presents results for these real datasets.
In these experiments, DAGMM (clean) was trained on a semi-supervised anomaly detection setting, using clean datasets during training, DAGMM (dirty) and DAE were trained in an unsupervised setting, while LODA-AAD, Tree-AAD and DAE uai were trained in an active anomaly detection setting.
We can clearly see from these results that DAE produces fairly bad results for all datasets analyzed here, nevertheless, even using a simple architecture as its underlying model, DAE uai produces similar or better results to the best baselines on the 11 datasets, even when the baselines were trained in completely clean training sets.
DAE uai also usually presents better results than LODA-AAD and Tree-AAD, which are similarly trained in an active setting.
One possible criticism to our method is that the importance of the proposed approach becomes more relevant the fewer the proportion of anomalous instances, which seems self-defeating.
But we see that the largest difference from the active methods to the other algorithms was in Covtype, which has less than 1% anomalies but 286 , 048 instances.
When working with large datasets (>1M instances), even if only 0.1% of the dataset is contaminated there is still the chance to beneﬁt from this feedback to improve performance.
The active algorithms are also more robust than the others, DAGMM used different hyperparameters for each experiment, while DAE uai and AAD use the same for all (except for k which was reduced from 10 to 3 for the datasets with less than 100 anomalies).
4. A M ORE P RACTICAL A NOMALY D ETECTION S ETTING
Another practical scenario where our model could be applied is a mixture of semi-supervised and unsupervised anomaly detection.
In this case, we have a dataset which contains anomalies that we want to ﬁnd and audit.
At the same time, new data instances, which may include new types of anomalies not seen before, can be added to the dataset at any time and we would like to detect anomalies in this dataset as well.
[Equation]
With this in mind, we ran an experiment training DAE uai and LODA-AAD on KDDCUP-Rev in the same way as in Section 4.2, while evaluating it on its test set for different budgets.
This test set [Table] contains 20 new types of anomalies (the train set contains 16 types of anomalies and the test set 36).
The evaluation was done by selecting the most anomalous instances found by each model on the test set and calculating the recall for both seen and unseen anomalies in that group.
Results for this experiment can be seen in Figure 4.
In this ﬁgure, the right y axis shows the number of anomalies detected in the training set for a certain budget and corresponds to the light blue lines.
The left y axis present the recall for the test dataset.
We see that DAGMM is not so effective on this test set, while
DAE is able to detect well novelty (new classes).
We also see that DAE uai is signiﬁcantly better at detecting known types of anomalies, while it maintains a recall close to the best on new unseen classes, giving better results than LODA-AAD for both seen and unseen classes of anomalies.
[Figure]
R ELATED W ORKS
Anomaly Detection
This ﬁeld has been amply studied and good overviews can be found in (Hodge & Austin, 2004; Chandola et al., 2009).
Although many algorithms have been recently proposed, classical methods for outlier detection, like LOF Breunig et al.
and OC-SVM (Schölkopf et al., 2001), are still used and produce good results.
Recent work on anomaly detection has focused on statistical properties of “normal” data to identify these anomalies, such as Maurus & Plant (2017), which uses Benford’s Law to identify anomalies in social networks, and (Siffer et al., 2017), which uses Extreme Value Theory to detect anomalies.
Other works focus on speciﬁc types of data, (Zheng et al., 2017) focuses on spatially contextualized data, while (Perozzi et al., 2014; Perozzi & Akoglu, 2016; Li et al., 2017; Liu et al., 2017) focus on graph data.
Recently, energy based models (Zhai et al., 2016) and GANs (Schlegl et al., 2017) have been successfully used to detect anomalies, but autoencoders are still more popular in this ﬁeld.
Zhou & Paffenroth (2017) propose a method to train robust autoencoders, drawing inspiration from robust statistics (Huber, 2011) and more speciﬁcally robust PCAs, (Yang et al., 2017) focuses on clustering, and trains autoencoders that generate latent representations which are friendly for k-means.
The work most similar to ours is DAGMM (Zong et al., 2018), where they train a deep autoencoder and use its latent representations, together with its reconstruction error, as input to a second network, which they use to predict the membership of each data instance to a mixture of gaussian models, training the whole model end-to-end in an semi-supervised manner for novelty detection.
Active Anomaly Detection Despite its many advantages, active anomaly detection remains an under-explored approach to this problem, nevertheless, over the years some really interesting work has been developed in this topic.
In (Pelleg & Moore, 2005), the authors solve the rare-category detection problem by proposing an active learning strategy to datasets with extremely skewed distributions of class sizes.
Abe et al.
(2006) reduces outlier detection to classiﬁcation using artiﬁcially generated examples that play the role of potential outliers and then applies a selective sampling mechanism based on active learning to the reduced classiﬁcation problem.
In (Görnitz et al., 2013), the authors proposed a Semi-Supervised Anomaly Detection (SSAD) method based in Support Vector Data
Description (SVDD) (Tax & Duin, 2004), which he expanded to a semi-supervised setting, where he accounts for the presence of labels for some anomalous instances, and with an active learning approach to select these instances to label. Veeramachaneni et al.
(2016) propose an active approach that combines unsupervised and supervised learning to select items to be labeled by experts, with each approach selecting k 2 instances at a time.
The most similar prior works to ours in this setting are (Das et al., 2016), which proposed an algorithm that can be employed on top of any ensemble methods based on random projections, and (Das et al., 2017), which expands Isolation Forests to work in an active setting.
Our work differs from these prior works mainly in that we prove the necessity of priors for unsupervised anomaly detection, further motivating the Active Anomaly Detection framework, and in our proposed model.
UAI layers can be assembled on top of any Deep Learning based anomaly detection architecture, which is the state of the art for unsupervised anomaly detection, to make it work in an active anomaly detection setting.
Besides, after each iteration with experts both
LODA-AAD and Tree-AAD have a time
complexity O ( t ) , where t is the number of already labeled instances, while each iteration of UaiNets runs in constant time O (1) with respect to t .
D ISCUSSIONS AND F UTURE W ORK
We proposed here a new architecture, Universal Anomaly Inference (UAI), which can be applied on top of any deep learning based anomaly detection architecture.
We show that, even on top of very simple architectures, like a DAE, UaiNets can produce similar/better results to state-of-the-art unsupervised/semi-supervised anomaly detection methods.
We also give both theoretical and practical arguments motivating active anomaly detection , arguing that, in most practical settings, there would be no detriment to using this instead of a fully unsupervised approach.
We further want to make clear that we are not stating our method is better than our semi-supervised baselines (DAGMM, DCN, DSEBM-e).
Our contributions are orthogonal to theirs.
We propose a new approach to this hard problem which can be built on top of them, this being our main contribution in this work.
To the best of our knowledge, this is the ﬁrst work which applies deep learning to active anomaly detection. We use the strongest points of these deep learning algorithms (their learned representations and anomaly scores) to build an active algorithm, presenting an end-to-end architecture which learns representations by leveraging both the full dataset and the already labeled instances.
Important future directions for this work are using the UAI layers conﬁdence in its output to dynamically choose between either directly using its scores, or using the underlying unsupervised model’s anomaly score to choose which instances to audit next.
Another future direction would be testing new architectures for UAI layers, in this work we restricted all our analysis to simple logistic regression.
A third important future work would be analyzing the robustness of UaiNets to mistakes being made by the labeling experts.
Finally, making this model more interpretable, so that auditors could focus on a few “important” features when labeling anomalous instances, could increase labeling speed and make their work easier.
A E XPERIMENTS D ESCRIPTIONS
In this section we give detailed descriptions of the experiments.
Section A.1 presents the used model architectures for both DAE and Class models, as well as DAE uai
and Class uai .
Section
A.2 presents details on the synthetic MNIST datasets and on the hyper-parameters used for the experiments.
Finally, Section A.3 contains detailed descriptions on the used datasets, baselines and experimental settings for the experiments on real anomaly detection datasets.
A. M ODEL A RCHITECTURES
To show our algorithm can be assembled on top of any deep learning model, we tested it using two simple but very different anomaly detection models.
The ﬁrst model we test it on top of is a normal
Denoising AutoEncoder (DAE).
A DAE is a neural network mainly composed by an encoder, which transforms the input into a latent space, and a decoder, which reconstructs the input using this latent representation, typically having a loss function that minimizes the reconstruction error L 2 norm:
[Equation]
where both f enc and f dec are usually feed forward networks with the same number of layers, l ∈ R d is a d -dimensional latent representation and (cid:15) is a zero mean noise, sampled from a Gaussian distribution with a ϕ standard deviation.
When used in anomaly detection, the reconstruction error is usually used as an approximation of the inverse of an item’s probability, and as its anomaly score:
[Equation]
We then create a DAE uai network by assembling the proposed UAI layer on top of the DAE:
[Equation]
where uai ( · ) is the classiﬁer chosen for the UAI layer.
This architecture can be seen in Figure 1.
Another typical approach to unsupervised anomaly detection is, when given a dataset with labeled data X = ( x x , x y ) , training a classiﬁer ( Class ) to predict x y from x x 5 and using the cross-entropy of an item as an approximation to the inverse of its probability distribution:
[Equation]
where f class ( · ) is typically a feed forward neural network with p layers, from which we can use its last hidden layer ( h p − 1 ) as the data’s latent representation to be used in the Class uai .
[Equation]
This architecture can be seen in Figure 5.
For all experiments in this work, unless otherwise stated, the DAE’s encoder and decoder had independent weights and we used both the DAE and Class models with 3 hidden layers and hidden sizes [256 , 64 , 8] .
This means the latent representations provided to the UAI layers are l ∈ R 8 .
We implemented all experiments using TensorFlow (Abadi et al., 2016), and used a learning rate of 0 .
01 , batch size of 256 and the RMSprop optimizer with the default hyper-parameters.
For the active learning models, we pre-train the DAE/Class model foroptimization steps, select k = 10 items to be labeled at a time, and further train for 100 iterations after each labeling call.
To deal with the cold start problem, for the ﬁrst 10 calls of select_top, we use the base anomaly score ( s ) of the DAE / Class model to make this selection, using the UAI one for all later labeling decisions.
X
S(x)
=
cross_entropy(Y,Ŷ)
L
Class uai Ŷ
Y
X uai
S uai (x) = σ(W act [X uai ] + b act )
Uai Layer
[Table]
A. S YNTHETIC D ATA
Detailed statistics on the synthetic MNIST datasets can be seen in Table 2.
MNIST 0 and MNIST 0were mainly generated with the purpose of simulating the situation in Figure 2 (Center), where anomalies were present in sparse clusters.
At the same time, MNIST hard and MNIST pca were designed to present similar characteristics to the situation in Figure 2 (Right), where anomalous instances are in sparse regions of the data space.
A. R EAL D ATA
For these experiments
[Table]
most datasets were used as suggested in (Dheeru & Taniskidou, 2017)
[Table]
but we processed the KDDCUP, Thyroid, Arrhythmia and KDDCUP-Rev datasets in the same manner as (Zong et al., 2018) to be able to better compare with their results:
• KDDCUP (Lichman, 2013) :
The KDDCUP99 10 percent dataset from the UCI repository.
Since it contains only 20% of instances labeled as “normal” and the rest as “attacks”, “normal” instances are used as anomalies, since they are in a minority group.
This dataset contains 34 continuous features and 7 categorical ones.
We transform these 7 categorical features into their one hot representations, and obtain a dataset with 120 features.
• Thyroid (Lichman, 2013) :
A dataset containing data from patients which can be divided in three classes: normal (not hypothyroid), hyperfunction and subnormal functioning.
In this dataset, we treat the hyperfunction class as an anomaly, with the other two being treated as normal.
It can be obtained from the ODDS repository.•
Arrhythmia (Lichman, 2013) :
This dataset was designed to create classiﬁcation algorithms to distinguish between the presence and absence of cardiac arrhythmia.
In it, we use the
smallest classes (3, 4, 5, 7, 8, 9, 14, and 15) as anomalies and the others are treated as normal.
This dataset can also be obtained from the ODDS repository.
• KDDCUP-Rev (Lichman, 2013) : Since “normal” instances are a minority in the KDDCUP dataset, we keep all “normal” instances and randomly draw “attack” instances so that they compose 20% of the dataset.
We compare our algorithm against:
• DAE (Vincent et al., 2008):
Denoising Autoencoders are autoencoder architectures which are trained to reconstruct instances from noisy inputs.
• DAGMM (Zong et al., 2018):
Deep Autoencoding Gaussian Mixture Model is a stateof-the-art model for semi-supervised anomaly detection which simultaneously learns a latent representation, using deep autoencoders, and uses both this latent representation and the autoencoder’s reconstruction error to learn a Gaussian Mixture Model for the data distribution.
• LODA-AAD
(Das et al., 2016):
Lightweight on-line detector of anomalies (LODA) Active Anomaly Discovery (AAD) is a work which uses the active anomaly detection framework on top of LODA (Pevn ` y, 2016), which is a method based on ensembles of weak anomaly detection models.
• Tree-AAD
(Das et al., 2017): This work learns weights for each node in an Isolation Forest anomaly detection model, by incorporating knowledge gained through active anomaly detection.
Since there is no validation/test set in unsupervised anomaly detection, we cannot tune hyperparameters on a validation set.
Because of this, to make the DAE baselines more competitive, we got the results for several different hyper-parameter conﬁgurations and present only the best among them.
This is not a realistic approach, but we only do it to our baselines, while for our proposed algorithm we keep hyper-parameters ﬁxed for all experiments.
We even keep our hidden sizes ﬁxed to [256 , 64 , 8] on thyroid, which only contains 6 features per instance, since our objective here is not getting the best possible results, but showing the robustness of our approach.
The only hyper-parameter change we make in UAI networks is that, since there are fewer anomalies in some datasets, we set our active learning approach to choose k = 3 instances at a time, instead of 10, for datasets with less thananomalies.
Results for DAGMM are from our implementation of this model and follow the same procedures, architectures and hyper-parameters as described in (Zong et al., 2018), being trained in a semisupervised setting.
The results for LODA-AAD and Tree-AAD were run using the code made available by the authors and with the same steps as DAE uai .
For all experiments, results for
LODA-AAD, Tree-AAD, DAE and DAE uai used the number of anomalies in the dataset as the budget b .
B D ETAILED R ESULTS
In this section, we present more detailed results for both the synthetic (Section B.1) and real (Section
B.2) anomaly detection datasets, which couldn’t ﬁt on the main paper due to lack of space.
We also present results for synthetic anomaly detection experiments on Fashion-MNIST (Section B.3).
B. D ETAILED R ESULTS ON MNIST
We present here detailed results for small budgets ( b ≤ 5000 ) on the MNIST experiments, with graphs zoomed in for these budget values.
Analyzing Figure 6 we see that for some of these datasets
UaiNets present a cold start, producing worse results for small budgets.
Nonetheless, after this cold start, they produce better results in all MNIST experiments.
An interesting future work would be to measure the conﬁdence in the UaiNet’s prediction to dynamically choose between using its anomaly score or the underlying network’s one, which could solve/reduce this cold start problem. [Figure]
B. D ETAILED R ESULTS ON R EAL D ATA
Table 3 presents a detailed comparison for experiments ran on KDDCUP, Thyroid, Arrhythmia and KDDCUP-Rev datasets with other baselines, also showing precision, recall and their standard deviations.
In this table we also compare our results to:
• OC-SVM (Chen et al., 2001):
One-class support vector machines are a popular kernel based anomaly detection method.
In this work, we employ it with a Radial Basis Function (RBF) kernel.
• DCN (Yang et al., 2017):
Deep Clustering Network is a state-of-the-art clustering algorithm.
Its architecture is designed to learn a latent representation using deep autoencoders which is easily separable when using k-means.
• PAE (Vincent et al., 2008): Denoising AutoEncoders pretrained as suggested in (Vincent et al., 2010).
• DSEBM-e (Zhai et al., 2016):
Deep Structured Energy Based Models are anomaly detection systems based on energy based models (LeCun et al., 2006), which are a powerful tool for density estimation.
We compare here against DSEBM-e, which uses a data instance’s energy as the criterion to detect anomalies.
• DSEBM-r
(Zhai et al., 2016):
Deep Structured Energy Based Model with the same architecture and training procedures as DSEBM-e, but using an instance’s reconstruction error as the criterion for anomaly detection.
The results presented here are averages of ﬁve runs, with standard deviations in parenthesis.
In this table, results for OC-SVM, PAE, DSEBM-r, DSEBM-e, DCN and DAGMM were taken from (Zong [Table] et al., 2018) [Table]
while DAGMM ∗ are results from our implementation of DAGMM.
Unfortunately [Table] we were not able to reproduce their results in the Thyroid dataset, getting a high variance in the results.
LODA-AAD does not scale well to large datasets, so to run it on KDDCUP and KDDCUP-Rev we needed to limit its memory about the anomalies it had already learned, forgetting the oldest ones.
This reduced its runtime complexity from O ( b 2 ) to O ( b ) in our tests, where b is the budget limit for the anomaly detection task.
We did the same (limit memory) for Tree-AAD on KDDCUP.
On this table we can see that DAE uai produces better results than LODA-AAD on all analyzed datasets and than Tree-AAD on three out of four.
Our proposed method also, besides presenting results comparable to state-of-the-art DAGMM trained on a clean dataset, is much more stable, having a lower standard deviation than the baselines in almost all datasets.
B. E XPERIMENTS ON F ASHION -MNIST
In this Section, we present results for experiments on synthetic anomaly detection datasets based on Fashion-MNIST (Xiao et al., 2017).
To create these datasets we follow the same procedures as done for MNIST in Section 4.1, generating four datasets: Fashion-MNIST 0 ; Fashion-MNIST 0-2 ;
Fashion-MNIST hard ; Fashion-MNIST pca .
Detailed statistics of these datasets can be seen in Table 4.
[Table]
We run experiments on these datasets following the exact same procedures as in Section 4.1.
Figureshows the results for Fashion-MNIST 0 and Fashion-MNIST 0-2 , while Figure 8 show the results for Fashion-MNIST hard and Fashion-MNIST pca .
These ﬁgures show similar trends to the ones for
MNIST, although algorithms ﬁnd anomalies in these datasets harder to identify.
In one run of
Fashion-MNIST 0 , DAE uai needed several examples to start learning and for Fashion-MNIST hard ,
Class uai takes a long time to start producing better results than Class .
Nevertheless, UaiNets are still much more robust than the underlying networks to different types of anomalies, producing good results in all four datasets, even when its underlying network gives weak results on that dataset.
C F URTHER A NALYSIS
In this section we further study UaiNets, analyzing the evolution of hidden representations and anomaly scores through training (Section C.1), and the dependence of results on the number of audited anomalies (Section C.2).
C. L EARNED R EPRESENTATIONS AND A NOMALY S CORES
In this section, we show visualizations of the learned representations ( l dae/class ) and anomaly scores ( s dae/class ) of UaiNets’ underlying networks, presenting their evolution as more labels are fed into the network through the active learning process.
With this purpose, we retrain UaiNets on both
MNIST 0-2 and MNIST hard , with a hidden size of [256 , 64 , 1] , so that its latent representation is one dimensional ( l ( x ) ∈ R 1 ), and plot these representations vs the anomaly scores ( s ) of the base network (either DAE or Class ) for different budgets ( b ).
Figure 9 shows the evolution of DAE uai ’s underlying l dae ( x ) and s dae ( x ) .
In it, we can see that initially (Figures 9 (a, d))
anomalies
and normal data instances are not separable in this space.
Nevertheless, with only a few labeled instances ( b = 250 ) the space becomes much easier to separate, while for b = 2000 the space is almost perfectly linearly separable.
Figure 10 shows the same evolution for Class uai ’s underlying l class ( x ) and s class ( x ) .
In it, we can also see the same patterns, as initially anomalies and normal data instances are not separable, but with a few labeled instances anomalies become much more identiﬁable.
The main conclusion taken from these visualizations is how the gradient ﬂow through l is important, since it helps the network better separate data in these spaces, allowing good performance even when the underlying networks are not good at identifying a speciﬁc type of anomaly.
[Figure]
C. A NOMALY C HOICES E VOLUTION THROUGH T RAINING
This experiments aim at showing how the networks choice quality evolves with the access to more labels.
Here, we present the choices DAE uai network would make having access to a ﬁxed number of expert labels.
With this in mind, we train the networks in the same way as in Section 4.
[Figure]
but stop after reaching a speciﬁc budget ( b ), showing the choices made up to that point
[Figure]
and after that with no further training.
Figure 11 shows the evolution of DAE uai anomaly choices as it is fed more expert knowledge.
We can see that with only a few labels it already fairs a lot better than its underlying network.
In
KDDCUP with only 3 , 000 labeled instances, which is less than 1% of the dataset, it can correctly ﬁnd 80 , 000 anomalies with a high precision, while the DAE with no expert knowledge does a lot worse.
On Thyroid and KDDCUP-Rev, with ≈ 10% of the dataset labeled ( b = 531 and b = 4000 , respectively) it ﬁnds all or almost all anomalies in the dataset correctly.
The Arrhythmia dataset is a lot smaller and with few anomalies, so DAE uai improves on DAE in a smaller scale here, but it still does fairly better than the underlying network.
[Figure]
D
P ROOFS
D. L EMMA 1. M IXTURE PROBABILITY LEMMA
Lemma 1.
Mixture probability lemma.
Consider two independent arbitrary probability distributions p 1 and p 2 .
Given only a third distribution p + = p composed of the weighted average of the two:
[Equation]
and considering P i as the residual probability distribution hyperplanes:
[Equation]
Without further assumptions on p 2 (without a prior on its probability distribution), we only know that p ( p 1 | p + = p ) =
p ( p 1 | p 1 ∈ P 1 ) and p ( p 2 | p + = p α ) = p ( p 1 = p
α − λ α · p 2 | p 2 ∈ P 2 ) .
Proof.
Given p + = p we know that:
[Equation]
with λ α = λ 1 − λ and p α = p 1 − λ .
Assuming the distribution of p 2 is independent of p 1 , and with no further assumptions on it, p 2 is random and uniform on the set of all possible probability distributions, [Figure] [Figure] so its probability distribution is:
[Equation]
[Figure]
where P is the hyperspace containing all probability distributions, with an hyper-volume m .
Now we can try to ﬁnd p ( p 1 | p + = p ) :
[Equation]
= [Equation] The equality in (1) comes from the deﬁnition of the space P 1 , which is the space of all possible values of p 1 that could result in p + = p , so if p 1 / ∈ P 1 , then p ( p + = p | p 1 ) = 0 .
Equality (2) is a simple variable substitution where p ( p + = p ) =
p (cid:16) p 1 = x, p 2 = p
α − x λ α (cid:17) .
(3) comes from the assumption that p 2 and p 1 are independent.
Equality (4) results from p 2 ∼ Uniform ( P ) and P having a volume m .
Finally, Equality (5) is a result from the fact that p α − x λ α ∈ P ⇔ x ∈ P 1 .
With a similar strategy we can ﬁnd p ( p 2 | p + = p ) : p
[Equation]
p
[Equation]
where Equality (1) and (2) result from the fact that p 1 ∈ P 1 ⇔ p 2 ∈ P 2 , given a speciﬁc value of p + = p .
This completes this proof.
D. L EMMA 2. E XTREME MIXTURES LEMMA
Lemma 2.
Extreme mixtures lemma.
Consider two independent arbitrary probability distributions p 1 and p 2 .
Given only a third probability distribution p + = p composed of the weighted mixture of the two, and for a small λ ≈ 0 , we can ﬁnd a small residual hyperplane P 1 , which tends to { p } .
[Equation]
We can also ﬁnd a very large residual hyperplane P 2 for p 2 , which tends to:
[Equation]
where supp ( · ) is the support of a probability distribution.
Proof.
In this proof, we start with the arbitrary residual hyperplanes P r and ﬁnd restrictions in the limits of λ → 0 and λ → 1 .
For a β ≈ 0 :
[Equation]
P r ≈ { p r = p − β · p, ∀ p ∈ P | β · p ≤ p } β ≈
P≈ { p r = p − λ · p, ∀ p ∈ P | λ · p ≤ p } λ ≈ P≈ { p r = p − (1 − λ ) · p, ∀ p ∈ P | (1 − λ ) · p ≤ p } λ ≈ 1 ∴ β ≈ For a β ≈ 1 we start with the other deﬁnition of P r : lim β → 1 P r = lim β
→ 1 { p r , ∀ p r ∈ P | (1 − β ) · p r ≤ p } =
lim β
→ 1 { p r , ∀ p r ∈ P | supp (
p r ) ⊆ supp ( p ) , (1 − β ) · p r ≤ p } = { p r , ∀ p r ∈ P | supp (
p r ) ⊆ supp ( p ) }
P r ≈ { p r , ∀ p r ∈ P | supp ( p r ) ⊆ supp ( p ) } β ≈
P≈ { p r , ∀ p r ∈ P | supp (
p r ) ⊆ supp ( p ) } λ ≈
P≈ { p r , ∀ p r ∈ P | supp (
p r ) ⊆ supp ( p ) } λ ≈ 0 ∴ β
≈
This ﬁnishes this proof.
D. T HEOREM 1. N O FREE ANOMALY THEOREM
Theorem 1.
Consider two independent arbitrary probability distributions p normal and p anom .
For a small number of anomalies λ ≈ 0 , the knowledge that p full = p gives us no further knowledge on the distribution of p anom :
[Equation]
Proof.
Consider in Lemmas 1 and 2 that p 2 = p anom ∼ Uniform ( P ) .
We then have that, for a small value of λ ≈ 0 :
[Equation]
This ﬁnishes this proof.
E F URTHER P ROOFS
In this section, we prove upper and lower bounds on the maximum distance a probability distribution p 1 can be from p + , based on the value of λ .
This can be directly applied to p normal for small values of λ and to p anom for large ones.
Theorem 2.
Upper Bound on Mixture Probability Distance For two independent arbitrary probability distributions p 1 and p 2 , given only a third probability distribution p + composed of the weighted mixture of the two:
[Equation]
We have an upper bound on the distance measures δ ( p + , p 1 ) and || p + − p 1 || given by:
[Equation]
which is a tight bound for λ ≈ 0 .
In this equation δ ( · ) is the total variation distance between two probability distributions and || · || is the L 1 norm.
Proof.
Pinsker’s inequality states that if p and q are two probability distributions on a common measurable space ( A , F ):
[Equation]
where D KL ( p || q ) is the Kullback–Leibler divergence in nats.
So we have that:
[Equation]
and this Kullback–Leibler divergence is itself upper-bounded by:
[Equation]
where this maximum Kullback–Leibler divergence is achieved when p 1 and p 2 are disjoint probability distributions:
[Equation]
which concludes the proof that:
[Equation]
Theorem 3.
Lower Bound on Maximum Mixture Probability Distance For two independent arbitrary probability distributions p 1 and p 2 , given only a third probability distribution p + composed of the weighted mixture of the two:
[Equation]
We have a lower bound on the maximum possible distance measures δ ( p + , p 1 ) and || p + − p 1 || for a chosen maximizing p 1 given by:
[Equation]
which is a tight bound for λ ≈ 1 , considering the maximum L 1 distance between two probability distributions is 2 .
Proof.
We can prove a lower bound on the maximized distance of a probability distribution p 1 from p + by expanding the distance equations:
[Equation]
where in (a) we lower bound based on the probability distribution that would have the smallest possible superior distance to a later maximized probability distribution p 1 .
This probability distribution pcan always maximize its superior distance to p 2 by:
[Equation]
else In (b) we choose the uniform distribution as the one that would reduce this superior distance and in (c) we set p 1 ( a )
= 1 for a random a , since p 2 is uniform.
With a similar strategy we ﬁnd:
[Equation]
This concludes this proof.