A NOMALY D ETECTION B ASED ON U NSUPERVISED D ISENTANGLED R EPRESENTATION L EARNING IN C OMBINATION WITH M ANIFOLD L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Identifying anomalous samples from highly complex and unstructured data is a crucial but challenging task in a variety of intelligent systems.
In this paper, we present a novel deep anomaly detection framework named AnoDM (standing for
Ano maly detection based on unsupervised D isentangled representation learning and M anifold learning).
The disentanglement learning is currently implemented by β -VAE for automatically discovering interpretable factorized latent representations in a completely unsupervised manner.
The manifold learning is realized by t-SNE for projecting the latent representations to a 2D map.
We deﬁne a new anomaly score function by combining β -VAE’s reconstruction error in the raw feature space and local density estimation in the t-SNE space.
AnoDM was evaluated on both image and time-series data and achieved better results than models that use just one of the two measures and other deep learning methods.
I NTRODUCTION
Detecting anomalies in data ﬂow of modern intelligent systems is an important but challenging problem.
Formally speaking, anomaly detection problems can be statistically viewed as identifying outliers having low probabilities from the modelling of data distribution p ( x ) .
Practically, since statistical modelling of the data is often difﬁcult, it degenerates to domain description (Tax & Duin, 1999) or supervised prediction (Gornitz et al., 2013) problems in some cases.
The exact explanation of an anomalous data point depends on the speciﬁc domain of focus.
In data centers, it probably indicates an attempt of cyber intrusion.
In recognition systems, it could be an adversarial attack.
In biomedical information systems, it means possible onset of certain diseases.
In Internet of
Things (IoT) systems, it may represent a hardware failure or alarming event captured by sensors.
An anomalous sample is not always associated with negativity.
Sometimes, it leads to novel discoveries in scientiﬁc explorations.
However, from the data analytics perspective, anomaly detection is a difﬁcult task due to the following reasons.
(1) Many forms of data, e.g., images, text, and other types of sequences, are often highly unstructured and complex.
How can these data be well represented and high-level information be extracted by an algorithm?
The sample sizes of modern data sets are often extremely large and most of them are unlabelled.
Unfortunately, traditional methods do not scale and perform well on these data.
(3) When data of multiple modalities are naturally available for same events in a system, a robust and precise algorithm needs to be designed to integrate these information for system diagnosis or decision making.
(4) Many intelligent systems, such as IoTs, require real-time detection and reaction of abnormal events to avoid costly and irrevocable damages.
Thus, anomaly monitoring algorithms to be designed in these platforms must be highly efﬁcient.
In summary, anomaly detection raises challenges in representability, scalability, multimodality, and time complexity.
Deep learning (LeCun et al., 2015) offers great potentials to overcome these challenges.
(1) Representation learning mechanisms (such as convolution for images, embedding for discrete symbols, and recurrence for time-series) have been developed in supervised and unsupervised deep models to consider the nature of speciﬁc types of input samples and encode them into vectors of continuous values as corresponding latent representations.
(2) Most deep learning models are trained using stochastic gradient descent that splits a giant training set into mini-batches.
Thus, learning be- comes unrestricted and blessed by a large sample size.
Particularly, stochastic variational inference (Hoffman et al., 2013; Zhang et al., 2018) has successfully enabled scalable learning and inference for deep generative models (DGMs) on a vast amount of unlablled data.
The development of deep learning programming packages, such as PyTorch (Paszke et al., 2017) and TensorFlow (Abadi et al., 2016), greatly eases the assembly of multiple network components (corresponding to different modalities) together for multimodal representation learning (Li et al., 2018b).
(4) Once a deep model is learned, the inference or encoding step is very efﬁcient, thanks to the highly parallel computing architectures and techniques.
In some applications, if the domain of anomalous and normal samples is well deﬁned, anomaly detection can be reduced to binary classiﬁcation problems.
However, in many situations, either the domain of anomalous samples cannot be fully understood or modelled, or the domain of the normal samples is too complicated to be modelled in one class.
DGMs are more suitable than supervised methods in such cases.
DGMs are concerned with the joint distribution of visible and latent variables with a hierarchy of stochastic (and deterministic) layers.
With proper emphasis on disentanglement of latent representations, DGMs have the potential of dissecting hidden factors that are key to sample generation.
Unsupervised disentangled representation learning (Bengio et al., 2013) renders several beneﬁts.
(1) It helps better understand our data, providing a path towards explainable AI.
(2) It gives a better control on the generation process of novel samples.
The disentanglement of latent factors may provide an opportunity to distinguish anomalies based on the landscape of latent space, which is our interest in this paper.
It has been shown that the likelihood of a data point p ( x ) estimated in DGM is not a reliable measure for detecting abnormal samples (Nalisnick et al., 2019).
Instead, reconstruction error is widely used as an anomaly score function (An & Cho, 2015).
As a variant of variational autoencoder (VAE) (Kingma & Welling, 2014), β -VAE (Higgins et al., 2017) is designed for unsupervised discovery of interpretable factorized latent representations from raw image data.
An adjustable hyperparameter β is introduced to balance the extent of learning constraints (a limit on the capacity of the latent information channel and an emphasis on learning statistically independent latent factors) and reconstruction accuracy.
It was demonstrated that βVAE with appropriately tuned value of β (when β > 1 ) qualitatively outperforms VAE (when β = 1 , β -VAE is exactly VAE).
Burgess et al.
(2018) proposed a modiﬁcation to the training regime of βVAE by progressively increasing the information capacity of the latent code during training.
This modiﬁcation facilitates the robust learning of disentangled representations in β -VAE, without the previous trade-off in the reconstruction accuracy.
Hoffman et al.
(2017) introduced a reformulation of β -VAE for 0 < β < 1 .
They argued that, within in this range, training β -VAE is equivalent to optimizing an approximate log-marginal likelihood bound of VAE under an implicit prior.
Manifold learning is a family of nonlinear dimensionality reduction techniques.
The t-distributed stochastic neighbor embedding (t-SNE) (van der Maaten & Hinton, 2008) is an unsupervised manifold learning method primarily used for data exploration and visualization by approximating highdimensional data distribution using a two or three-dimensional map that could preserve local and certain global structures of the data.
The use of t-SNE for anomaly detection has been sceptical (van der Maaten & Hinton, 2008).
However, no comprehensive investigation has been made in this topic.
Taking advantages of both disentangled representation learning (using β -VAE as an implementation) and low-dimensional manifold learning (using t-SNE as an implementation), we propose a novel anomaly detection approach named AnoDM , standing for Ano maly detection based on unsupervised D isentangled representation learning and M anifold learning .
We introduce a new anomaly score function by combining: (1) β -VAE’s reconstruction error, and (2) distances between latent representations of test points and training points in t-SNE map.
AnoDM is a general framework, thus any disentangled representation learning and manifold learning techniques can be applied.
The choice of a lower-level encoding scheme in β -VAE depends on data type of interest.
For image data, deterministic convolutional network (CNN) is used in the encoder.
In case of time series (sequence) data, we design an improved version of β -VAE by replacing CNN with temporal convolutional network (TCN) (Bai et al., 2018), a generic architecture for convolutional sequence prediction, in the encoder.
We incorporate TCN as part of the encoder, because Bai et al.
(2018) have shown that
TCN outperforms canonical recurrent networks such as LSTMs (Hochreiter & Schmidhuber, 1997) across a range of supervised learning tasks and recommended that CNN should be regarded as the ﬁrst method to try for sequence modeling tasks.
Regarding the decoding architecture, we simply choose CNN, because by choosing a simpler CNN architecture as a part of the decoder, the model can achieve a comparable even better performance but take much less running time.
The contributions of this paper are summarized as follows.
(1) We comprehensively explore the capacity of unsupervised disentangled representation learning, using β -VAE as an implementation, for anomaly detection.
(2) We thoroughly investigate the potential of manifold learning for outlier identiﬁcation by taking the disentangled latent representations from β -VAE as input to t-SNE.
To the best of our knowledge, this is the ﬁrst attempt to explore t-SNE for anomaly detection.
For sequence anomaly detection, instead of using prevailed recurrent networks (such as LSTM), as a practical contribution, we adopt an improved convolution architecture (TCN) to capture the temporal dependency in the encoder in unsupervised way.
R ELATED W ORK
In the big data era, the development of deep learning models, especially DGMs, ﬂourishes due to the need of modelling and analyzing massive amount of unstructured data (such as images, timeseries, graphs, text, etc.)
generated in many application domains.
Designing DGM-based solutions for anomaly detection becomes an important topic.
Since DGMs, such as VAE (Kingma & Welling, 2014), and deep belief net (DBN) (Hinton et al., 2006; Li & Zhu, 2018a), aim at modelling the joint distribution of visible and latent variables (that is p ( x , h ) ), their likelihood p ( x ) by marginalizing out h may serve as an abnormality indicator.
However, unlike exponential family restricted Boltzmann machines (exp-RBMs) (Li & Zhu, 2018b), exact likelihood is unavailable for most DGMs.
Alternatively, reconstruction error serves as an abnormality measure based on the intuition that outof-distribution samples can be reconstructed badly (An & Cho, 2015).
Some deep hybrid methods (e.g. VAE+OCSVM (Andrews et al., 2016) and DBN+OCSVM (Erfani et al., 2016)), successfully combine classical one-class support vector machine (OCSVM; or kernel-based support vector domain description (SVDD)) with DGMs by using DGMs to learn latent representations of samples and using OCSVM to detect abnormal data points.
However, these methods face the challenge of scalability, because the size of kernel matrices in dual form of SVDD is quadratic of sample size.
Generative adversarial net (GAN) has also been applied for anomaly detection (Schlegl et al., 2017).
Since there is no encoder in GAN, Deecke et al.
(2019) presented the ADGAN algorithm based on the availability of a good representation of a sample in latent space of its generator by assuming that the generator is able to effectively capture the distribution of the training data.
Li et al.
(2018a) proposed the GAN-AD method for cyber-physical systems (CPSs).
It distinguishes fake data from actual data by taking into consideration of both discrimination loss calculated by the trained discriminator and residual loss between reconstructed and actual test data.
Furthermore, DGM-based algorithms are also devised to detect anomaly problem on sequence data (e.g. LSTM-VAE (Park et al., 2017) and GAN-AD (Li et al., 2018a)).
Conventionally, canonical recurrent networks (such as LSTM and GRUs (Cho et al., 2014)) are considered as the dedicated methods for sequence modeling.
Some recent studies have also claimed that there was no architecture that could consistently beat LSTM in some typical sequence modelling tasks (Jozefowicz et al., 2015; Greff et al., 2015; Melis et al., 2017).
On the other hand, some other researchers insist that
CNN (LeCun et al., 1989) should be considered as more appropriate choice for sequences.
Inspired by more recent CNN-based sequence modelling (such as machine translation (Gehring et al., 2016; 2017) and language modeling (Dauphin et al., 2016)), Bai et al.
(2018) conducted a systematic evaluation of generic convolutional and recurrent architectures for sequence modelling across a broad range of tasks that are commonly used to benchmark recurrent networks, and concluded that convolutional networks, rather than recurrent networks, should be respected as a “natural starting point for sequence modelling tasks”.
M ETHOD
In this paper, we propose a novel generic anomaly detection framework named AnoDM, which the ﬁrst time combines unsupervised disentangled representation learning (implemented using β -VAE as an example) and low-dimensional manifold learning (currently using t-SNE as implementation) together to detect outliers via effectively taking the advantages of reconstruction at raw feature space and disentangled latent distribution in t-SNE map.
Figure 1 shows the architecture of AnoDM which includes two main phases: (1) unsupervised disentangled representation learning and (2) anomaly detector.
After β -VAE is learned using unlablled training normal samples, it then can be employed by the anomaly detector to efﬁciently obtain latent encoding and reconstructed version of a sample.
Once latent embeddings of both training samples (or a representative subset from the training data) and a test sample are obtained, t-SNE is used to map the latent representations of these samples further to the 2-dimensional space (called t-SNE space or map), such that the average distance between the 2D representation of the test sample and its k nearest neighbors from the 2D representations of training samples is calculated.
Finally, this distance is combined with the reconstruction error of the test sample to deﬁne its anomaly score.
The essential parts of this framework are discussed below in details.
Full AnoDM approach is given in Algorithm 1 in Appendix E.
3. U NSUPERVISED D ISENTANGLED R EPRESENTATION L EARNING
The unsupervised disentangled representation learning component in our architecture is implemented but not limited by β -VAE (Higgins et al., 2017; Burgess et al., 2018; Hoffman et al., 2017;
Mathieu et al., 2019).
A self-inclusive description of β -VAE is provided in
Appendix B.
The objective function to be maximized for β -VAE is deﬁned as (Burgess et al., 2018):
[Equation]
The ﬁrst term of this objective function corresponds to reconstruction error in the raw feature space.
The KL divergence characterizes the discrepancy between approximate posterior and isotropic prior of latent representations.
A small discrepancy between them indicates high disentanglement of latent representations in independent variables.
C is a hyperparameter which is used to improve the quality of reconstructed images.
The loss function of the original β -VAE proposed in (Higgins et al., 2017) does not have this hyperparameter.
The value of β trades off reconstruction error and disentanglement.
Unlike (Higgins et al., 2017) and (Burgess et al., 2018), we consider β > 0 rather than just β > 1 , because it is unnecessary to bound the value of β by 1, β > 0 allows us search for a more appropriate disentanglement.
The special case β = 0 could make the model learning very unstable, because the variance of inference distribution loses control.
It worth highlighting that other unsupervised disentanglement models can be used as well in AnoDM.
For example, Mathieu et al.
(2019) interpreted disentanglement as decomposition instead of independence by adding an additional regularization term to reduce the discrepancy between the aggregate posterior and a desired structured prior.
However, designing a properly structured prior could be practically challenging.
The adoption of β -VAE in our framework is sufﬁcient to prove the concept that unsupervised disentanglement helps anomaly detection.
3. E FFECTIVITY OF T -SNE A LGORITHM
In addition to β -VAE’s reconstruction error, we use the average distance between a test sample and its k -nearest neighbors from the collection of training samples in the t-SNE map to score the outlierness of a test sample.
As discussed in Appendix C, t-SNE is signiﬁcantly inﬂuenced by perplexity.
The nature of complexity in data distributions makes it impossible to utilize a uniform criteria to deﬁne optimal perplexity for all data.
Moreover, Wattenberg et al.
(2016) mentioned several weaknesses of t-SNE, for examples, (1) it naturally expands dense clusters and contracts sparse ones, evening out cluster sizes, and (2) distances between clusters might not reﬂect global geometry.
However, it is likely that k -nearest neighbors still work for local clumps, because, with proper value of perplexity, local topological information of latent distributions can be preserved by the t-SNE plot.
Thus, in t-SNE space, the measure of k -nearest neighbors is more suitable than full density estimation which is very sensitive with the sizes of clusters.
Furthermore, as to be shown in
Section 4, distancing in the 2D t-SNE map could be more robust than distancing in β -VAE’s latent space where many non-determinative factors may inﬂuence the calculation of distances.
Finally, it worth clarifying that we do not directly learn a 2D latent representation from β -VAE, because it will bottleneck too much the information ﬂow for reconstruction.
Instead, a lower-dimensional representation is learned through t-SNE for density estimation only.
3. D ETERMINISTIC OR S TOCHASTIC L ATENT R EPRESENTATIONS FOR T -SNE
Either the mean µ or a sample z from the approximate inference distribution q ( z | x ) can be passed to t-SNE to calculate the k -NN distance of a test sample.
There is trivial difference between performances achieved by these two methods in our framework.
Generally, the µ -based method achieved slight better performance.
The comparison of these two methods can be found in Table 2 in appendices.
Furthermore, from the latent representations’ t-SNE maps (see Figure 9 in appendices), one can interestingly see that when β is small (not overly large), the t-SNE maps for both methods are quite similar.
As β becomes overly large, some normal classes can still form their own clusters (even though some similar classes, such as classes 3 and 8 in MNIST, tend to mingle together) in µ -based method, but in z -based method all classes are entangled with each other.
Same phenomena can be observed on the other datasets (see Figures 10 and 11 in appendices).
Therefore, the µ -based method is used in current design of AnoDM.
3. TCN E NCODER FOR U NSUPERVISED S EQUENCE M ODELLING
Bai et al.
(2018) distilled superior design in convolutional network into a simple architecture and referred it as a temporal convolutional network (TCN) with two distinctive characteristics: (1) the convolutions in the architecture are causal, and (2) the architecture can take a sequence of any length and map it to an output vector of ﬁxed length, just as with an RNN.
Bai et al.
(2018) also explained that TCNs capture signiﬁcantly longer history than recurrent networks.
Inspired by Bai et al. (2018), we replace CNN with TCN in the encoder of β -VAE when evaluating the proposed AnoDM framework on time-series data, while we still use CNN in the decoder, because our preliminary experiments demonstrated that keeping decoder as simpler CNN can help achieve comparable even better results, and take much less computing time.
The architecture of TCN used in this paper is depicted in Figure 7 of appendices.
In this TCN, we set kernel size to 4 and dilation factors to [1 , 2 , 4 , 8 , 16 , 32] .
In Section 4, the comparison among TCN, CNN, and LSTM encoders in our framework also shows that TCN outperforms CNN and particularly LSTM to a great extent for ECG signal anomaly detection.
3. A NOMALY S CORE F UNCTION IN A NO DM
In the anomaly detector, the reconstruction error of a test sample in the original feature space and the average distance from its k -nearest-neighbors in training samples within the 2D t-SNE map are combined as a anomaly function:
[Equation]
where the ﬁrst term is deﬁned using normalized squared error (NSE):
[Equation]
where x te is a test sample, and x (cid:48) te is its reconstructed version by sending the stochastic latent encoding through the decoder of β -VAE.
The second term in Equation (2) is deﬁned using β -VAE’s deterministic latent encoding (mean from the encoder of learned β -VAE) as input to t -SNE:
[Equation]
where l ( i ) te is the 2D representation of the i -th test sample in t-SNE map, N ( i, k ) is the set of indices of l ( i ) te ’s k nearest neighbors from training samples’ 2D representations l tr in t-SNE map.
In Equation (2), α ∈ [0 , 1] is the combination hyperparameter such that the two terms can effectively complement each other.
To allow the anomaly score function to achieve its full potential, α value should be sensitively searched, because the values of D RE and D k tSNE can stay at different magnitudes, a very small change of α value may dramatically alter the contributions of these two terms.
Alternatively, the distance score in Equation 4 can be normalized by average distance of training samples in t-SNE map, which may alleviate the magnitude difference, thus ease search of optimal value of α .
The use of this normalized distance is investigated in Appendix F.
E XPERIMENTS
We evaluated our framework on four public image datasets, including MNIST (LeCun et al., 1998),
Fashion-MNIST (Xiao et al., 2017), Small-Norb (LeCun et al., 2004) and CIFAR-10 (Krizhevsky, 2009), as well as one collection of ECG heartbeat categorization time-series data named Arrhythmia (Fazeli, 2018).
Brief descriptions of these data sets can be found in Appendix G.
The detail of β -VAE architecture is given in Table 3 in appendices.
The number of epochs was set to 20 for experiments on
MNIST and Fashion-MNIST datasets, and 50 for CIFAR-10, Small-Norb, and Arrhythmia datasets; batch size was set to 100 for experiments on all these datasets.
When using t-SNE, the dimension of t-SNE map was set as 2 for all datasets, perplexity 30 , the learning rate 200 , and maximum number of iterations 1000 .
The value of α in the anomaly score function is searched from set { 0 . 0 , 0 . 005 , 0 . 01 , 0 . 05 , 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 , 0 . 6 , 0 . 7 , 0 . 8 , 0 . 9 , 0 .
We used k = 1 for calculating k -nearest-neighbors distance in t-SNE map (we also tried to set k to 3 or 5, the results were similar).
4. C OMPARISON WITH C APS N ET , GAN S , AND VAE
We compared AnoDM with state-of-the-art algorithms including a supervised method – CapsNet (Li et al., 2019), and two types of generative models – GANs (including AnoGAN and ADGAN)
(Deecke et al., 2019) and β -VAE (implemented by setting α = 1 thus using only reconstruction error as anomaly score).
As shown in Table 1, on average, AnoDM achieved either comparable (on
MNIST) or better (on Fashion-MNIST and CIFAR-10) performance in terms of receiver operating characteristic curve (auROC).
On Fashion-MNIST, CapsNet (prediction-probability-based), as the best benchmark method, obtained an average auROC of 0.765, while AnoDM achieved 0.883.
On
MNIST, both AnoDM and CapsNet obtained the highest performance.
However, CapsNet is a supervised method that takes advantages of class information, while ours is completely unsupervised that is more suitable in many practices as class information is either incomplete or unavailable.
Furthermore, by comparing AnoDM with β -VAE that only considers reconstruction error as anomaly score, AnoDM dramatically improved the performance in all cases.
In other words, t-SNE makes a prominent contribution to improve β -VAE for anomaly detection problems.
However, all generative models did not work well on Small-Norb, mainly because these models used convolution to extract features from image, but convolution is only able to capture translation but not other afﬁne transformations.
Although CapsNet learns these transformations as a supervised method, it worth exploring unsupervised learning of afﬁne transformations as a future topic.
[Table]
CIFAR-
[Table]
4. I MPACT OF B ETA TO P ERFORMANCE
As discussed in Higgins et al.
(2017), β ( > 1 ) functions as a controller to encourage most efﬁcient latent representation learning via limiting the capacity of latent information channel.
Mathieu et al.
(2019) however interpreted the objective of β as tuning a proper level of overlap of encodings by working with another term that regularizes the divergence of the aggregate posterior q φ ( z ) and the desired prior p ( z ) .
The main intuition is that purely increasing β induces too much overlap which actually discourages the disentanglement of data (information which is necessary for expressing desired structure is lost).
Higgins et al.
(2017) demonstrated that β -VAE with β > 1 leads to interesting results when learning interpretable factorized latent representations on a variety of datasets.
Surprisingly, our investigation demonstrates that by setting 0 < β < 1 , it actually achieved the state-of-theart results for anomaly detection problems on a range of datasets, such as MNIST, Fashion-MNIST and Arrhythmia.
Figure 2 illustrates the impact of values of β and α in our anomaly score function.
Interestingly, best performances were achieved when β < 1 and the performances generally degrade as β increases, resonating with Mathieu et al.
(2019) that overly large values of β actually causes a mismatch between q φ ( z ) and p ( z ) (resulting in inappropriate level of overlap in the latent space).
This phenomenon can be further seen in the t-SNE maps of latent embeddings in Figure 3.
When β becomes extremely larger than the appropriate value, the anomalous class becomes entangled with its normal neighboring classes and the boundaries between normal classes become unclear.
Theoretically, adding the divergence of the aggregate posterior q φ ( z ) and the desired structured prior p ( z ) is an effective way to limit the level of overlap when β is too large.
However, it is practically challenging to design an appropriate structured prior.
Therefore, in our investigation, we focused on exploring the full range of β ’s value in β -VAE for the impact of disentanglement to anomaly
[Figure]
[Figure]
detection.
Since our framework is quite general, it can be easily extended to other unsupervised disentangled representation learning models for anomaly detection.
4. I MPACT OF B ETA TO T -SNE R EPRESENTATIONS
The t-SNE plots in Figure 3 reﬂect the impact of β ’s value on latent representations in the case of identifying anomalous digit 7 from MNIST.
In this example, the best performance (auROC = 0.975) was achieved when β = 0 .
Clearly, as β increases, all latent clusters become less dense, and more anomalous latent data points move to neighboring clusters.
Furthermore, Figure 3 also corroborates that, even though in t-SNE maps distances between clusters might not reﬂect global geometry and cluster sizes might not mirror the true sizes (Wattenberg et al., 2016), using averaged distance from a test sample to its k nearest normal data points represented in t-SNE space to qualify outlierness still is a very effective way for distinguishing anomalous samples when β is tuned properly.
4. E VALUATION OF A NOMALY S CORE F UNCTION
In order to better evaluate our anomaly score function, as formulated in Equation (2), we conducted a comprehensive comparison with methods only based on either distance in t-SNE map ( D k tSNE ) or reconstruction-error in raw feature space ( D RE ).
To see the contribution of t-SNE, it is also compared with the method that calculates nearest neighbor distance directly in latent space of β -VAE.
Figuredisplays the ROC curves of these four approaches when assuming anomalous classes are respectively 1 (“Trouser/pants”), 3 (“Dress”), 5 (“Sandal”), and 7 (“Sneaker”) on Fashion-MNIST.
It is obvious that AnoDM achieves best results among them by taking advantages of both β -VAE reconstruction and t-SNE embedding.
The β -VAE reconstruction reﬂects whether useful information is captured by the model through recovering the input x ; the t-SNE embedding indicates the disentanglement of latent representations z .
Both measures effectively complement each other.
Besides, comparing the auROCs between t-SNE-based and latent-distance-based score functions, one can clearly see that the former dramatically outperforms the latter.
Same conclusion can be drawn for MNIST, CIFAR-10, and Arrhythmia, as displayed in Figures 12, 14, 16, 17, and 18 in appendices.
To further show that the optimal values of α are close to 1 (see Figure 2) is due to magnitude difference rather than less usefulness of t-SNE, we replaced the distance score (Equation (4)) with normalized distance score in the weighted ﬁnal anomaly score function (Equation (2)).
We found that the optimal values of α shift to the lower end of the spectrum (see Figure 8).
It implies that t-SNE does play a critical role in our framework.
[Figure]
4. A NO DM FOR T IME -S ERIES
As mentioned in Section 3, our method uses a TCN encoder in β -VAE for time-series anomaly detection.
Figure 5 displays the comparison among TCN, CNN and LSTM encoders in the AnoDM framework on Arrhythmia.
As a special case of LSTM- β -VAE, LSTM-VAE was presented in (Park et al., 2017) for state-of-the-art sequence modelling.
For the ﬁve classes in Arrhythmia, iteratively one class was treated as anomalous class, while the other classes were used as normal classes.
The TCN-encoder-based method outperforms the other two methods signiﬁcantly in all ﬁve cases.
Even though the CNN encoder achieved impressive results when detecting anomalous class “S”, “V”, “F” and “Q” respectively, it did not work quite well when class “N” was treated as anomaly.
One possible reason might be that comparing with TCN and LSTM, the performance of CNN is more sensitive on the training sample size.
Taking the above case as an example, as class 0 (“N”) accounts for over 80% of training data, when considering it as anomaly, normal training data hence become less sufﬁcient for learning β -VAE.
Nevertheless, in TCN-based β -VAE, each hidden unit of the last deterministic hidden layer before latent encoding at the bottleneck is calculated based on much longer sequence dependency, such that it is less sensitive to the limitation of small sample size.
Conclusively, as mentioned in (Bai et al., 2018), TCN should be regarded as a natural starting point for sequence modeling tasks.
As a case study, Figure 6a shows the original EGG signals and reconstructed signals by TCN-based β -VAE when considering class “Q” as anomaly.
Normal samples can be reconstructed very well, whereas abnormal samples suffer from larger reconstruction errors.
Meanwhile, the corresponding t-SNE plot in Figure 6b displays two distinctive clusters of abnormal samples.
The combination of these two measures thus leads to the best performance as seen in Figure 6c.
[Figure]
C ONCLUSIONS
We propose a new methodology which successfully integrates t-SNE with disentangled representation learning for anomaly detection.
This approach achieved state-of-the-art performances on both image data (MNIST, Fashion-MNIST and CIFAR-10) and Arrhythmia time-series data.
Speciﬁcally, best performance is accomplished when 0 < β < 1 for almost all cases involving β -VAE.
We also deﬁned an anomaly score function by effectively taking the advantages of both low-dimensional t-SNE embedding and β -VAE reconstruction.
Our algorithm demonstrated that t-SNE plays an essential role for measuring abnormality.
This research initiates the research on anomaly detection using unsupervised disentangled representation learning and lower-dimensional manifold learning.
Besides, our model uses TCN network as encoding architecture for detecting anomalous time-series data and the experimental results convince us that TCN consistently outperforms CNN and LSTM.
As a proof of concept, our current framework automatically inheres advantages of deep learning to address anomaly detection’s issues in representability and scalability as discussed in the beginning of this paper.
The extension of our framework to multimodal data is straightforward.
It is also possible that a neural t-SNE component could be designed and integrated into the learning of β -VAE to achieve real-time efﬁciency.
Other new well-performing manifold learning methods, such as UMAP (McInnes et al., 2018) which is faster and keeps global topologies, could be employed as replacement of t -SNE.
A V ARIATIONAL L OWER B OUND AND V ARIATIONAL A UTOENCODER
A directed generative model concerns the joint distribution p θ ( x , z ) where x is a random vector variable representing an observation in d -dimension space, z is the vector of latent variables in kdimensional space, and θ is the set of model hyperparameters.
Inference p θ ( z | x ) = p ( x | z ) p ( z ) p ( x ) is generally intractable, with few exceptions such as exponential family RBM (Li & Zhu, 2018b).
An easy parametric distribution q φ ( z | x ) is introduced to approximate the intractable posterior p θ ( z | x ) .
Distribution q φ ( z | x ) should be as close to p θ ( z | x ) as possible by estimating inference parameters φ .
Based on mean-ﬁeld variational Bayes, reverse Kullback-Leibler divergence is employed to measure the distance between these two distributions, D KL (cid:0) q φ ( z | x ) || p θ ( z | x ) (cid:1) , with respect to φ .
After being expanded, the following formula is obtained ( p θ is ﬁxed with respect to q φ ),
[Equation]
[Equation]
After being rearranged, we can obtain an objective with respect to both generative θ and inference parameter φ :
[Equation]
Since the KL divergence in Equation (6) is always non-negative and thus L is the varitional lower bound (or evidence lower bound, ELBO) of log p θ ( x ) :
[Equation]
Equation (8) implies that maximizing the ELBO is to maximize the reconstruction error and also minimize the difference between the inference distribution and generative prior distribution.
From the variational Bayes perspective, the aim is to obtain the optimal generative and inference parameters through maximizing the ELBO:
[Equation]
The main challenge in learning of directed generative models is to derive the gradient of Equation (7) with respect to the inference parameter.
Therefore, the reparameterization trick is introduced in VAE (Kingma & Welling, 2014; Rezende et al., 2014) such that backpropagation can be used through expressing the random variable z as a deterministic variable z
= τ φ ( x , (cid:15) ) , where (cid:15) is an auxiliary independent random variable, and the transformation function τ φ parameterized by φ converts (cid:15) to z .
For example, when q φ ( z | x ) is a multivariate Gaussian with a diagonal covariance structure ( q φ ( z | x ) = N ( z
[Equation]
µ , σ 2 I ) ), sampling from this distribution is equivalent to:
[Equation]
[Equation]
where [ µ ( x ) , σ ( x )]
= f φ ( x ) is a deterministic two-head multilayer perceptrons parameterized by φ , and (cid:12) denotes element-wise product.
The reparametrization trick enables E z ∼ q ( z | x ) [log p θ ( x , z ) − log q φ
[Equation]
z | x )]
[Equation]
E
[Equation]
[log p θ
[Equation]
x
[Equation]
z
[Equation]
log q
[Equation]
z | x )] , such that the gradient of Equation (7) with respect to φ can be reformulated to
[Equation]
The gradient with respect to generative parameters θ is easier, because it does not rely on the reparameterization trick:
[Equation]
B B ETA -VAE AND B EYOND FOR D ISENTANGLED R EPRESENTATION L EARNING
Higgins et al.
(2017) proposed a novel deep generative model, named β -VAE, a modiﬁcation of VAE by introducing an adjustable hyperparameter β to learn an interpretable disentangled representation of the data generative latent factors.
Speciﬁcally, β functions as a controller to trade off between the extent of learning constraints and reconstruction accuracy.
The constraints impose a limit on the capacity of the latent information channel and an emphasis on learning statistically independent latent factors.
Higgins et al.
(2017) demonstrated that β -VAE with appropriately tuned β ( β > 1 ) qualitatively outperforms VAE ( β = 1 ) as well as state of the art unsupervised (InfoGAN) and semisupervised (DC-IGN) approaches to disentangled factor learning on a variety of datasets (celebA, faces and chairs).
Higgins et al.
(2017) assumed that an image x is generated by the true world simulator using ground truth data generative factors: p ( x | v , w ) =
Sim ( v , w ) , where v is set of conditionally independent factors and w is set of conditionally dependent factors.
Therefore, the joint distribution of the data x and a set of generative latent factors z is: p ( x | z )
≈ p ( x | v , w ) =
Sim ( v , w ) .
The aim of this generative model is then to ensure that the inferred latent factors from q φ ( z | x ) capture the generative factors v in a disentangled manner.
The conditionally dependent data generative factors w can remain entangled in a separate subset of z that is not entangled with v .
Considering the prior p ( z ) is set to be an isotropic unit Gaussian p ( z ) =
N ( 0 , I ) , a constraint δ is introduced to encourage the matching between q φ ( z | x ) and p ( z ) such that the disentangling property in the inferred q φ ( z | x ) can be realized.
Following the same incentive as in VAE: maximizing the probability of generating real data, while minimizing the distance between the generative and approximate posterior distributions, as formulated below
[Equation]
where X = { x 1 , x 2 , .
. . , x n } is the training data set.
Rewriting this equation as a Lagrangian under the KKT conditions (Kuhn & Tucke, 1951; Karush, 1939), the following equation F ( θ, φ, β ) is obtained: [Equation]
[Equation]
The objective function to be maximized in β -VAE is thus deﬁned as:
[Equation]
[Equation] where the Lagrangian multiplier β is the regularisation coefﬁcient that constrains the capacity of the latent information channel z and puts implicit independence pressure on the learnt posterior due to the isotropic nature of the Gaussian prior p ( z ) .
When β = 1 , β -VAE corresponds to the original
VAE formulation of Kingma & Welling (2014).
When β > 1 , it applies stronger constraint which limits the capacity of z and encourages the model to learn the most efﬁcient representation of the data.
Theoretically, a higher β encourages more efﬁcient latent encoding and further encourages the disentanglement.
However, a higher β may lead to poorer reconstructions due to the loss of high frequency details when passing through a constrained latent bottleneck.
Burgess et al.
(2018) proposed an improvement to β -VAE, by progressively increasing the information capacity of the latent code during training.
It facilitates the robust learning of disentangled representations in β -VAE without the previous trade-off in reconstruction accuracy.
The objective function to be maximized in β -VAE is redeﬁned as:
[Equation]
where the hyperparameter β controls how heavily to penalise the deviation of D KL (cid:0) q φ
[Equation]
z | x
[Equation]
|
[Equation]
p
[Equation]
z
[Equation]
and a controllable value C .
By gradually increasing C from zero to a large value, good-quality reconstruction can be obtained.
Hoffman et al.
(2017) had also made a deeper research on β -VAE considering β < 1 .
They argued that optimizing this partially regularized ELBO is equivalent to performing variational expectation maximization (EM) with an implicit prior r ( z ) ( r ( z ) ∝ q φ ( z ) (1 − β ) p ( z ) β ) that depends on the marginal (aggregate) posterior q ( z ) (cid:44) 1 N (cid:80)
Ni =1 q ( z | x ( i ) ) , and further derived some approximations to examine this prior.
Mathieu et al.
(2019) explained that most recent work for learning disentangled representations of data with deep generative models has focused on capturing purely independent factors of variation by employing regularizers explicitly encouraging independence in the representations.
They argued that such an approach is not generalisable, and quite restrictive for complex models where the true generative factors are not independent, very large in number, or where a set of true generative factors cannot be well deﬁned.
Overly large β is not universally beneﬁcial for disentanglement, Since this in turn causes a mismatch between marginal posterior q φ ( z ) and the prior p ( z ) .
Thus they proposed a generalization of disentanglement in VAE by explicitly separating such a decomposition as two tasks: a) the latent encoding of data should achieved an appropriate level of non-negligible overlap in aggregate encoding q φ ( z ) , and b) the aggregate encoding of data q φ ( z ) should match the prior p ( z ) which demonstrates the desired dependency structure between latent variables.
By improving the match of q φ ( z ) and p ( z ) , the overlap is only up to an appropriate level.
Mathieu et al.
(2019) developed a new objective that incorporates both a) and b) by introducing an additional divergence term D (cid:0) q φ ( z ) , p ( z ) (cid:1) .
The objective of this improved β -VAE is deﬁned as below:
[Equation]
By appropriately setting β and α , it allows direct control over the level of overlap and the regularization between the marginal posterior and the prior.
However, a practical challenge of this method is how to deﬁne a proper structured prior when the structure of real hidden factors is poorly known.
For this reason, our computational experiment in this paper is based on Burgess et al.
(2018)’s
β -VAE.
Instead of employing the regularization controller β to disentangle the independent data factors by putting implicit independence pressure in the representation, Hamaguchi et al.
(2018) presented a new method to learn disentangled representation by introducing two additional loss functions L sim (a similarity loss function) and L act (an activation function).
This technique aims to detect trivial events in an image resulting from environmental changes (such as illumination changes, background motions and shadows) by disentangling each image into two kinds of features, speciﬁc and common.
The functionality of L sim constrains common features to represent invariant factors between two paired images.
L act encourages activation of common features to avoid a trivial solution.
After common features of paired images are separated, the means of common features from two images are fed into event detector (a classiﬁer) for training.
As mentioned in Hamaguchi et al. (2018), this method cannot achieve good disentanglement when dealing with complicated scenes, since the activation of the units in the common features are degenerated a certain value.
C T -SNE FOR M ANIFOLD L EARNING
The t-distributed stochastic neighbor embedding (t-SNE), introduced by van der Maaten & Hinton (2008), is a popular technique for nonlinear dimensionality reduction that is particularly well suited for visualizing similarity of high-dimensional data that lie on several different, but related, lowdimensional manifolds.
It is an improvement of stochastic neighbor embedding (SNE) presented by Hinton & Roweis (2002) with easier optimization and better visualization via replacing the SNE cost function with a symmetrized version and using a Student-t distribution instead of a Gaussian to compute the similarity between two points in the low-dimensional space.
C. S TOCHASTIC N EIGHBOR E MBEDDING (SNE)
SNE measures similarities (in both original high-dimensional space and mapped low-dimensional space) between points by converting Euclidean distances between data points into conditional probabilities under Gaussian distributions.
In original high-dimensional space, for example, the similarity of a pair of data points x i and x j is represented by p j | i under a Gaussian centered at x i .
High p
j | i means x i and x j are close to each other, and vice versa.
p
j | i is deﬁned as:
[Equation]
26) where σ i is the variance of the Gaussian that is centered on data point x i .
It is inﬂuenced by the perplexity
Perp ( P i ) , which is deﬁned as follows,
[Equation]
(27) where P i corresponds particular σ i and H ( P i ) is the Shannon entropy of P i measured in bits.
For the low-dimensional counterparts y i and y j of the high-dimensional data points x i and x j , the similarity of these two points is denoted as q j | i , and similarly deﬁned as:
[Equation]
[Equation]
where the variance of the Gaussian is set to 1 √ 2 .
Since the focus is modeling pairwise similarities, both p i | i and q i | i are set to zero.
C. C OST F UNCTION OF T -SNE
Kullback-Leibler divergence is utilized to measure the difference between the low-dimensional data representation distribution q j | i and high-dimensional data distribution p j | i .
The cost function of
SNE, which is sum of Kullback-Leibler divergences over all data points, to be optimized by a gradient descent method, is given by
[Equation]
Aiming to alleviate the difﬁcult optimization problem of the cost function above and “crowding problem” (comparing nearby data points, moderate distant data points will not occupy reasonably large area in the low-dimensional map), van der Maaten & Hinton (2008) introduced symmetric
SNE and employed a Student t-distribution with one degree of freedom (which is the same as a
Cauchy distribution) as the heavy-tailed distribution in the low-dimensional map.
The cost function is redeﬁned as:
[Equation]
where again, p ii and q ii are also set to zero, and
[Equation]
The gradient of the Kullback-Leibler divergence between P and the Student-t based joint probability distribution Q is given by
[Equation]
i
[Equation]
D T EMPORAL C ONVOLUTIONAL N ETWORKS
Bai et al.
(2018) employed a basic architecture which is essentially same as the time delay neural network proposed by Waibel et al.
(1990) to ensure outputs of same length as inputs and no leakage from the future into the past.
[Equation]
However, since a simple causal convolutions is not able to achieve a long effective history size, Bai et al.
(2018) employed dilated causal convolutions (same architecture as WaveNet (van den Oord et al., 2016), please refer to Figure 7a) that enable an exponentially large receptive ﬁeld.
The dilated convolution operation F on element s of the sequence is deﬁned as:
[Equation]
where x ∈ R n is a 1-D sequence input, f : 0 , . . .
, k − 1 → R is a ﬁlter, d is the dilation factor, k is the ﬁlter size, and s − d · i accounts for the direction of the past.
When d = 1 , a dilated convolution reduces to a regular convolution.
By choosing larger ﬁlter sizes k and increasing the dilation factor d , the receptive ﬁeld of TCN can be enlarged.
Bai et al.
(2018) also utilized a generic residual module (He et al., 2015) in place of a convolutional layer.
A residual block is deﬁned as:
[Equation]
The outputs of a series of transformations F are added to the input x of the block.
To tackle with discrepancy of input and output widths in a standard residual block, an additional 1 × 1 convolution is used to ensure that element wise addition ⊕ receives tensors of the same shape (see Figure 7b).
Bai et al.
(2018) further discussed the advantages of TCN (including parallelism, ﬂexible receptive ﬁeld size, stable gradients, low memory requirement for training and variable length inputs) and its disadvantages (including possibly high memory requirement for evaluation and potential parameter change for a transfer of domain).
E T HE F ULL A NO DM A LGORITHM
Algorithm
Result: Anomaly scores of test samples
Inputs
1 while epoch no more than training iterations do
Encoder net maps X tr into µ tr and σ tr ;
Z tr = µ tr + σ tr (cid:12) (cid:15) , (cid:15) ∼ N (0 , I ) ;
Decoder net reconstructs X tr to X (cid:48) tr using tr ;
; 6 end tr
2D representations tr i ) te within X te do
D RE ( x ( i ) te ) (cid:44) NSE ( x ( i ) te , x (cid:48)
( i ) te ) =
(cid:107) x
( i ) te − x (cid:48)
( i ) te (cid:107) 22 (cid:107) x
( i ) te (cid:107); // reconstruction error
D k tSNE ( x ( i ) te ) (cid:44) 1 k (cid:80) j ∈ N ( i,k ) (cid:107) l
( i ) te − l ( j ) tr (cid:107) 2 ; //
N ( i, k ) is the set of indices of l ( i ) te ’s k nearest neighbors from l tr
S β VAE+tSNE ( x ( i ) te ) =
α D RE + (1 − α )
D k tSNE ; // α ∈ [0 , 1] 13 end
[Figure]
F A NOMALY SCORES WITH NORMALIZED K -NN DISTANCE IN T -SNE MAPS
Since the normalised reconstruction error in input space and the k -NN distance in t-SNE maps may have very different magnitudes (as mentioned in Section 3.5), some values of α in Equation (2) are close to (but not exactly equal to) 1 .
The reader may have the intuition that t-SNE is not useful in
AnoDM.
A simple way, to ﬁnd out whether large α value is due to magnitude difference or useless of t-SNE, is to replace k -NN distance of a test sample in t-SNE map (Equation (4)) with a k -NN distance normalized by average distance among training samples in the t-SNE map as deﬁned below:
[Equation]
[Equation]
36) where c > 0 is normalization hyperparameter (we set it to 0.5 in our experiment); x tr is a training sample; n is the total number of training samples in a t-SNE map.
The heatmaps in Figuredepict performances of AnoDM on Fashion-MNIST (“Dress”class is considered as anomaly) and
Arrhythmia (“F” class is treated as anomaly) using normalized k -NN distance in t-SNE maps in combination reconstruction error.
By comparing Figure 8 with Figure 2, one can see that the optimal values of α short right upper corner area to the left upper corner area.
It thus implies that, the optimal value of α is affected by the magnitude difference, and t-SNE indeed plays an essential role in AnoDM.
[Figure]
G D ESCRIPTIONS OF D ATA S ETS
• MNIST :
It contains a training set of 60000 gray scale digit images of 28 × 28 and a test set of 10000 same resolution gray scale examples from approximately 500 different writers (LeCun et al., 1998).
• Fashion-MNIST :
It is a dataset of Zalando’s article images, comprising 70000 MNISTlike labeled fashion images of 28 × 28 , with 7000 images per category (Xiao et al., 2017).
The training set has 60000 images and the test set has 10000 images.
The samples come from 10 classes: T-shirt/top, trouser, pullover, dress, coat, sandal, shirt, sneaker, bag, and ankle boot.
• Small-Norb : It contains 24300 96 × 96 grayscale images pairs of 50 toys belonging togeneric categories: four-legged animals, human ﬁgures, airplanes, trucks, and cars (LeCun et al., 2004).
The objects were imaged by two cameras under 6 lighting conditions, 9 elevations , and 18 azimuths.
As in (Sabour et al., 2017), the images were resized to 48 × 48; random 32 × 32 crops of them were obtained during training process.
Central 32 × 32 patches of test images were used during test.
• CIFAR-10 :
It consists of 60000 32 × 32 colour images in 10 classes, with 6000 images per class.
There are 10000 test images which include exactly 1000 randomly-selected images from each class and 50000 training images (with 5000 images from each class) are randomly grouped into 5 batches (Krizhevsky, 2009).
Arrhythmia :
It is derived from Physionet’s MIT-BIH Arrhythmia Datase which consists of ECG recordings from 47 different subjects recorded at the sampling rate of 360 H z .
This dataset is created by ﬁve different heart beat categories: “N”, “S”, “V”, “F” and “Q”, in accordance with Association for the Advancement of Medical Instrumentation (AAMI) EC57 standard
(Kachuee et al., 2018).
The meanings of these categories are explained below.
“N”: normal, left or right bundle branch block, atrial escape and nodal escape.
“S”: atrial premature, aberrant atrial premature, nodal premature and supra-ventricular premature.
“V”: premature ventricular contraction and ventricular escape. 4.
“F”: fusion of ventricular and normal.
“Q”: paced, fusion of paced and normal and unclassiﬁable.
[Table] [Table]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Table]