Self-Supervised Anomaly Detection via Neural Autoregressive Flows with Active Learning
Abstract
Many self-supervised methods have been proposed with the target of image anomaly detection.
These methods often rely on the paradigm of data augmentation with predeﬁned transformations such as ﬂipping, cropping, and rotations.
However, it is not straightforward to apply these techniques for non-image data, such as time series or tabular data, while the performance of the existing deep approaches has been under our expectation on tasks beyond images.
In this work, we propose a novel active learning (AL) scheme that relied on neural autoregressive ﬂows (NAF) for self-supervised anomaly detection, speciﬁcally on small-scale data.
Unlike other generative models such as GANs or VAEs, ﬂow-based models allow to explicitly learn the probability density and thus can assign accurate likelihoods to normal data which makes it usable to detect anomalies.
The proposed NAF-AL method is achieved by efﬁciently generating random samples from latent space and transforming them into feature space along with likelihoods via invertible mapping.
The samples with lower likelihoods are selected and further checked by outlier detection using Mahalanobis distance.
The augmented samples incorporating with normal samples are used for training a better detector so as to approach decision boundaries.
Compared with random transformations, NAF-AL can be interpreted as a likelihood-oriented data augmentation that is more efﬁcient and robust.
Extensive experiments show that our approach outperforms existing baselines on multiple time series and tabular datasets, and a real-world application in advanced manufacturing, with signiﬁcant improvement on anomaly detection accuracy and robustness over the state-of-the-art.
Introduction
Anomaly detection, ﬁnding rare data that substantially differs from the majority of the data, is one of the essential problems in artiﬁcial intelligence.
One typical anomaly detection setting is a one class classiﬁcation, where the target is to detect samples as normal or anomalous.
Many deep anomaly detection methods are recently proposed to solve one class classiﬁcation tasks, speciﬁcally on image benchmarks, with different scenarios, including supervised anomaly detection, unsupervised anomaly detection, and self-supervised anomaly detection [Ruff et al., 2021].
Here, we focus on the selfsupervised setting where we have a training set of normal samples without anomalies and detect anomalies in the testing set which contains both normal and anomalous samples.
However, for data beyond images, such as tabular or time series data, we have several challenges in pursuing accurate and robust detection of anomalies.
First, many recent advances in anomaly detection rely on data augmentation.
Typical transformations, such as translation, rotation and reﬂection, are designed for images so that a strong detector is obtained based on the transformation predictions.
Unfortunately, it is less well known which transformations are useful and hand-crafted transformation is not a straightforward task for non-image data [Bergman and Hoshen, 2020, Qiu et al., 2021].
Second, many tabular and time series data are from medical and healthcare.
Small dataset size with sparse labels gives rise to unique difﬁculties which result that the anomaly detection performance is always under our expectation [Zong et al., 2018].
Third, although many deep anomaly detection methods show exceptional performance on large-scale image benchmarks, it is still a non-trivial task to handle small-scale tabular and time series data with high reliability and robustness [Pang et al., 2021].
This work aims at addressing these challenges in the scenario of self-supervised anomaly detection for data types beyond images.
We develop a novel active learning scheme for effective data augmentation, which is a simple end-to-end procedure built upon a likelihood-based anomaly detection.
The key idea is to leverage the advantages of neural autoregressive ﬂows to assign likelihoods to normal data which enables to detect anomalies.
Augmented samples with explicit likelihoods, drawn from the learned ﬂow models can incorporate with original small data to improve the detector accuracy with high robustness.
[Figure]
Augmentedsamples
I n v e r t i b l e m a
pp i n g
Outlier
N A F A n o m a l y D e t e c t i o n F r a m e w o r k N A F A c t i v e L e a r n i n g ( N A F - A L )
M a r g i n a l s t r a t e g y
Speciﬁcally, our proposed method consists of two core components: NAF anomaly detection framework (NAF-AD) and NAF-based active learning scheme (NAF-AL).
Figure 1 visualizes the core idea behind our method.
NAFAD ﬁrst performs data via dom afﬁne transformations and by utilizing
[Huang et al., 2018].
Unlike GANs or VAEs, ﬂow-based models enable a bijective mapping between feature space and latent space in which each sample is assigned to a likelihood, which is used to derive a score function to decide a sample is normal or anomalous.
We propose
NAF-AL by efﬁciently generating random samples from latent space and transforming them into feature space via bijective mapping
The samples with lower likelihoods are selected and further checked by outlier detection using Mahalanobis distance.
The left effective samples are merged into normal data to approach decision boundaries for better detection.
Compared with random transformations, NAF-AL can be interpreted as a likelihood-oriented data augmentation that is more active and efﬁcient.
As a result, we achieve superior performance in deep anomaly detection beyond images, speciﬁcally on small-scale tabular and time series data, with signiﬁcant improvement on anomaly detection accuracy and robustness over the state-of-the-art.
Related Work
Deep Anomaly Detection.
Many recent advances have been proposed to use deep learning for anomaly detection.
Ruff et al.
[2021], Pang et al.
[2021] provided a thorough survey and review on the recent development of deep anomaly detection approaches.
Related work on deep anomaly detection include one class classiﬁcation [Ruff et al., 2018, Liznerski et al., 2020, Ruff et al., 2019], outlier exposure [Hendrycks et al., 2019a, Goyal et al., 2020], and out-of-distribution (OOD) detection [Ren et al. , Hendrycks and Gimpel, 2017, Kirichenko et al., 2020].
There has been an increasingly growing interest in self-supervised scenarios since this supervision is easy to obtain in practical settings and also shows promising accuracy in detecting anomalies [Pang et al., 2019, Hendrycks et al., 2019b, Sohn et al., 2020, Tack et al., 2020, Li et al., 2021, Sehwag et al., 2020].
Self-supervised methods solve one or more classiﬁcation-based auxiliary tasks (e.g., data transformations [Golan and El-Yaniv, 2018, Wang et al., 2019]), using normal data for training and the learned classiﬁer is useful to detect anomalies.
Bergman and Hoshen [2020] extended the work from Golan and El-Yaniv [2018], Wang et al.
[2019] to investigate self-supervised anomaly detection for general data.
This approach is established based on the open-set setting with afﬁne transformations for tabular datasets.
Qiu et al.
[2021] followed a similar scope for anomaly detection but with learnable transformations, and demonstrated a higher performance.
Likelihood (Density)-based Anomaly Detection.
Differing from the classiﬁcation-based methods [Ruff et al., 2018, 2019, Bergman and Hoshen, 2020], likelihood-based methods offer a probabilistic view for anomaly detection.
In this scenario, a ﬂow-based model, learning a bijective mapping between data distributions and latent distributions via invertible neural networks, is an ideal candidate because it has signiﬁcant advantages in explicit likelihood calculation and efﬁcient sample generation [Kobyzev et al., 2020, Papamakarios et al., 2021].
Much recent effort has been made to improve model expressivity and computational efﬁciency that allow more accurate likelihood calculation and enable faster sampling [Rezende and Mohamed, 2015, Dinh et al., 2017, Kingma et al., 2016,
Papamakarios et al., 2017, Kingma and Dhariwal, 2018, Grathwohl et al., 2018, Ho et al., 2019].
Huang et al.
[2018] proposed a neural autoregressive ﬂow (NAF) which is a universal approximator for density functions, and addresses the challenges in inverse AFs [Kingma et al., 2016].
Although the properties of normalizing ﬂows are promising, ﬂow-based models for anomaly detection have not raised much attention yet, although some works presented promising results using RealNVP [Rudolph et al., 2021], residual ﬂows [Zisselman and Tamar, 2020] and conditional normalizing ﬂows [Gudovskiy et al., 2021].
However, all of these works deal with large-scale image datasets.
It remains an open, and sometimes contentious, debate as to why and why not ﬂow-based models can guarantee anomaly detection [Kirichenko et al., 2020, Schirrmeister et al., 2020, Lan and Dinh, 2020].
Active Learning for Anomaly Detection.
Deep anomaly detection tends to require immense amounts of computational and human resources for training and labeling.
The design of effective training methods that require small labeled training sets is a fundamental research challenge [Tran et al., 2019].
To address this issue, two are particularly interesting: data augmentation , which artiﬁcially generates new samples for training, while active learning selects the most informative subset of unlabeled samples to be labeled.
Although successful in image data, data augmentation does not utilize computational resources since the generated samples are not guaranteed to be informative [Shorten and Khoshgoftaar, 2019].
Active learning deals with this limitation through an iterative selection of small subsets while assessing how informative those subsets are for the training process.
Recent advances on active learning rely on the incorporation of Bayesian approach [Gal et al., 2017,
Tran et al., 2019] and deep generative models [Sinha et al., 2019, Liu et al., 2019].
Active learning strategies for anomaly detection [Stokes et al., 2008, Görnitz et al., 2009, Pelleg and
Moore, 2004] which identify informative instances for labeling, have primarily only been explored for shallow detectors and could be extended to deep learning approaches [Pimentel et al., 2020,
Trittenbach and Böhm, 2019].
Our goal is to integrate a likelihood-based detector with active learning, which leads to a more effective data augmentation scheme for designing anomaly detection that continuously improves via likelihood feedback loops, see Figure 1.
This idea has not yet been explored for deep self-supervised anomaly detection.
NAF-AL Method
We develop a novel approach that relies on neural autoregressive ﬂows with active learning (NAF-AL), which is designed for self-supervised anomaly detection beyond images.
3.1 Data Transformations in Self-Supervised Setting
Our method is built upon the self-supervised scenario which can be typically deﬁned by
Deﬁnition 1 Assume all data X lies in space S d , where d is the data dimension.
Normal data X lie in subspace X ⊂ S d
but all anomalies X ∗ lie outside X .
The task of self-supervised anomaly detection is to build a classiﬁer C based on completely normal data,
such that C ( x ) =
if x ∈ X and C ( x ) = [-
-]0 if x ∈ S d \ X .
In the self-supervised setting, data transformations T
= { T 1 , ..., T k | T k : X → X} (e.g., translation, rotation and reﬂection), are often used to generate K different views, which leads to a strong anomaly detector based on the transformation prediction or representations learned using these views.
However, these transformations are not applicable to non-image data.
To this end, we generalize the set of transformations to random afﬁne transformations:
[Equation]
where A k and b k are afﬁne matrix and coefﬁcient respectively, deﬁned by random Gaussian distributions.
The random afﬁne transformation is a more general class that works for general data type with an unlimited number of transformations.
Since only normal data are used for training, we ﬁrst transform the normal data X into K subspaces
X 1 , ..., X k , and then learn a feature extractor f θ ( x ) using a neural network parametrized by θ , which maps the original normal data space X into a feature representation space ˜ X .
The probability of data point x after transformation k is denoted by p ( T k ( x )
∈ X k ) .
By assuming independence between different transformations T k , the probability that x is normal p ( x ∈ X ) is the product of the probabilities that all transformed samples are in their respective subspace.
[Equation]
where P ( x ) computes the degree of anomaly of each data.
Lower probabilities (likelihoods) indicate a more anomalous data.
We will introduce how to explicitly calculate these probabilities (likelihoods) using ﬂow-based models below.
3.2 Learning Likelihood by Autoregressive Flows
Normalizing ﬂows (NF) are a ﬂexible class of generative models that map a target distribution p X ( x ) into a base distribution in the latent space
p Z ( z ) via an invertible transformation f ψ : Z → X where f ψ is an invertible neural network parametrized by ψ .
Based on the change of variable theorem, the likelihood for an input x is
[Equation]
Flow-based models are typically trained by minimizing the negative log-likelihood of the training data D with respect to the parameters ψ of the invertible transformation f ψ .
[Equation]
Much effort in NFs focuses on designing expressive transformations while retaining efﬁcient computing the determinant of the Jacobian | det J | .
In particular, autoregressive ﬂows (AFs) decompose a joint distribution p X ( x ) into a product of m univartiate conditional densities:
[Equation]
where each univariate density is parametrized by a NF.
In particular, the transformation f − 1 , ( i ) ψ can be decomposed via invertible transformers t ( i ) ψ and conditioners c ( i ) ψ :
[Equation]
( x ≤ i ) = t ( i ) ψ ( x i , c ( i ) ψ ( x <i )) .
The resulting ﬂows have a lower triangular Jacobian and the invertibility of the ﬂows as a whole depends on each t ( i ) ψ being an invertible function of x i and each c ( i ) is an unrestricted function.
RealNVP [Dinh et al., 2017] model each t ( i ) ψ by using an afﬁne transformation whose parameters are predicted by c ( i ) .
However, these models require complex conditioners and a composition of multiple ﬂows due to their simplicity which leads to a limitation on expressiveness of f ψ .
Neural autoregressive ﬂow (NAF) [Huang et al., 2018] was proposed by learning a complex bijection using a neural network monotonic in x i .
NAF is a universal approximator for explicitly learning likelihood with greater expressivity that allows it to better capture multimodal target distributions.
The NAF architecture is illustrated by Figure 2.
We propose to utilize NAF to learn the distribution of feature space and train NAF using a maximum likelihood objective, which is equivalent to minimizing loss deﬁned by
[Equation]
where n is the size of training data D .
During training, L ( D n ) is optimized for feature space x of different transformations T of an input data.
After training, the learned NAF model can be used to evaluate the log-likelihood of the testing dataset D t that contains normal and anomalous samples.
[Equation]
[Table]
[Equation]
) predicts pseudo-parameters for t ( i ) , which in turns processes x i .
We use the calculated likelihoods as a criterion to classify a sample as normal or anomalous.
To pursue a robust anomaly score function S ( x ) , we concatenate all the variable z from multiple transformations T k ( x ) ∈ T and average the negative log-likelihood as
[Equation]
where
Υ = f θ ( T k ( x )) represents the feature space extracted by a neural network f θ .
If the anomaly score S ( x ) is above the threshold value ξ , the sample is identiﬁed as anomalous, otherwise as normal, which is given by
[Equation]
where Q ( x ) = 1 indicates an anomaly.
The threshold value ξ is varied to calculate the AUC and F score in experiments.
Figure 3 shows empirical evidence for NAF-AD method.
While the histogram of anomaly scores (computed using Eq.
(7)) is similar for inliers and anomalies before training, this changes drastically after training, and held-out inliers and anomalies become easily distinguishable.
3.3 Active Learning with Marginal Strategy
Formally, active learning is used to automatically select the most informative subset of unlabeled training samples and label them by an oracle.
One of the key enabling techniques is uncertainty sampling, which uses one classiﬁer to identify unlabeled samples with least conﬁdence [Zhu et al., 2009].
However, active learning for self-supervised anomaly detection is different from the original scope.
To this end, we devise a novel active learning scheme to query low-conﬁdence decisions, hence guiding the detector with augmented normal samples in the training process.
Such a marginal strategy can be expressed by adding the sample x ∗ that is close to the likelihood-based decision boundary:
[Equation]
This is achieved by generating samples x augj from the learned NAF model, calculating the loglikelihood L ( x augj ) and retaining the samples with lower likelihoods if L ( x augj ) <
Q α ( L ( x i ))
where
Q α is referred to as the α -quantile ( α = 0 .
9 ∼ 0 .
95 is a hyperparameter) and L ( x i ) is the loglikelihood of current training samples.
It is critical to design an appropriate likelihood level set trade-off between aggressive boundary and conservative boundary, see Figure 6 for more discussion.
The sample with lower likelihoods are desired but it is probably an outlier (anomalous sample) if the likelihood is too small.
Thus we have an additional step to check the sample by outlier detection, which is done by Mahalanobis distance M ( x ) [Lee et al., 2018, Ren et al., 2021]
[Equation]
We deﬁne a threshold δ M that is Mahalanobias distance at χ 20 . 05 .
The samples with M ( x ) > δ M are rejected.
We eventually determine the augmented samples x ∗ by solving the optimization problem in
Eq.
(9) subject to two constrains:
[Equation]
[Figure] [Equation] [Figure] [Equation]
[Figure]
These samples x ∗ are selected as normal samples and added to update the training process
Appropriate stopping criterion for active learning is a trade-off issue between training cost and effectiveness of the detector.
We set up a criterion to stop if the number of training iteration increase to ﬁve due to time cost limitation or the augmented samples size reaches 50 % of the original training size given the concern of sample efﬁciency.
In summary, we propose an active data augmentation by leveraging these synthesized normal samples based on meaningful likelihoods, without changing the original training, testing, and proportion of anomalies.
Note that the augmented samples generated from the NAF are located on the feature space rather than the original data space.
The details of the NAF-AL are provided in Algorithm 1.
Add
Experiments
We provide several experiments to demonstrate the effectiveness of our NAF-AL approach on deep anomaly detection beyond images.
Most image datasets are large-scale and have existing strong baselines so we do not expect signiﬁcant improvement via our NAF-AL method.
Instead, our focus is on small-scale tabular data and time series data.
4.1 Anomaly Detection Baseline Methods
We compare our NAF-AL with a couple of anomaly detection methods, including:
• Shallow AD baselines : Isolation Forest (IForest)
[Liu et al., 2008] uses a tree-based model to isolate anomalies.
Local Outlier Factor (LOF)
[Breunig et al., 2000] utilizes density estimation with k -nearest neighbors.
One-Class SVM (OC-SVM)
[Schölkopf et al., 1999] is a kernel-based approach for one-class classiﬁcation.
• Deep AD baselines : Deep Autoencoding Gaussian Mixture Model (DAGMM)
[Zong et al., 2018] uses latent space to estimate density.
Deep Support Vector Data Description (DSVDD) [Ruff et al., 2018] is a distance-based method with one-class SVM in the feature space.
Feature Bagging Autoencoder (FB-AE)
[Chen et al., 2017] is an ensemble method with autoencoders as the base classiﬁer.
GOAD [Bergman and Hoshen, 2020] is a self-supervised classiﬁcation-based method.
Neural Transformation Learning for Anomaly Detection (NeuTral) [Qiu et al., 2021], is a self-supervised method with learned transformations.
For time series data, we also include LSTM-ED [Malhotra et al., 2016] which is an encoder-decoder model to detect anomalies based on reconstruction error.
4.2 Tabular Data Experiments
Tabular data plays a crucial role in anomaly detection applications since many medical, health, and cybersecurity data come in this format.
However, many important areas, e g., medical, only have small-scale data because the data collection is time-consuming, while labeling relied on expert opinion is expensive.
Datasets.
We focus on six tabular datasets, including four small-scale medical datasets, Arrhythmia, Cardiotocograph, Lymphography and Thyroid from the Outlier Detection Datasets (ODDs) repository 2 , and two cybersecurity datasets, KDD and KDDRev from the empirical studies of Zong et al.
[2018], Bergman and Hoshen [2020], Qiu et al.
[2021] which are used to show our potential to large-scale datasets.
Table 4 shows the key statistics (data size, dimension, and anomaly ratio) of the tabular datasets, and all relevant details of the datasets can be found in the Appendix.
Following the setting of Zong et al.
[2018], we train all models on 50% of the normal data and evaluate the performance on testing data containing the rest of normal data as well as all the anomalies.
[Figure]
Implementation Details.
We use a standard normal distribution to generate random afﬁne transformation matrices for each case.
Similar as the setting in Bergman and Hoshen [2020], we use 256 transformations for small-scale medical datasets and 64 for large-scale datasets (KDD and KDDRev).
For the feature extractor, we used fully-connected hidden layers ( layer with 8 hidden nodes for the small-scale datasets and 5 layers with 128 hidden nodes for large-scale datasets) with leaky-ReLU activations, as well as one 1d convolutional layer on the top.
The NAF model consists of 4 ﬂow blocks with 2 layers (128 hidden units) for small data and 8 ﬂow blocks with 3 layers (1024 hidden units).
We optimized the network parameters using Adam with a learning rate of 0.001.
Considering the training cost limit, we set up a stop criterion by using a maximum number of iteration ( N max = 5 ) for active learning.
For each iteration, we draw a branch of samples where the sample size equals the training sample size, then rank the samples based on their likelihoods and ﬁnally reject the larger 90% samples (retain 10% samples with lower likelihoods near the decision boundaries).
These samples are further checked by Mahalanobis distance criterion if it is an outlier.
After that, we combine these augmented samples with the existing normal samples to update training.
[Table] The implementation details of the baseline methods are replicated from the existing studies [Zong et al., 2018, Bergman and Hoshen, 2020], as we report their results with mean and standard deviation (if they provide).
We also implement these baselines for two additional small-scale datasets (Cardio. and Lympho.) using their ofﬁcial code (if they have, otherwise keep the relevant cell blank).
Results.
The results of NAF-AL in comparison to all baseline methods on tabular data are shown in Table 1.
We follow the conﬁguration of previous work [Zong et al., 2018, Bergman and Hoshen, 2020, Qiu et al., 2021] to report results in terms of F1-scores.
• Small-scale datasets: all medical datasets are small with a low anomaly to normal ratio.
Our NAF-AD performs reasonably well as similar to most baselines.
Our NAF-AL outperforms all baselines on these small-scale datasets thanks to the beneﬁts from active learning.
Compared with GOAD which is a classiﬁcation-based method, our probabilistic ﬂow-based model is competitive even without the help of active learning.
NeuTral beats our NAF-AD in the Arrhythmia dataset but underperforms our NAF-AL method.
Since our NAF-AL is ﬂexible to incorporate any number of transformations such that our robustness (with a smaller variance) is better than NeuTral. • Large-scale datasets : The deep baselines show superior performance than the shallow methods in this case.
NAF-AD is slightly lower than NeuTral, GOAD, and DSVDD but NAF-AL is still competitive.
One explanation is that the performance improvement in such a large dataset is not signiﬁcant as the small-scale case discussed above.
The large datasets, having different dynamics from very small datasets found by Bergman and Hoshen [2020], are probably not well-suited to the probabilistic methods.
4.3 Time Series Data Experiments
Differing from novelty detection within time series (point or group anomalies), we aim to detect abnormal time series on a whole time sequence .
In other words, the whole time series data is labeled by normal or anomalies.
This scenario is also important in practice.
For example, we identify abnormal facility operations by detecting abnormal sensor measurements over the whole time-series signals in scientiﬁc applications.
Anomalies in medical, health, and sport monitoring may indicate injury, disease, or more serious issues.
Datasets.
We focus on ﬁve multivariate time series datasets from the UEA multivariate time series classiﬁcation archive 3 which has been widely used for anomaly detection tasks [Zhang et al., 2020,
Ruiz et al., 2021, Jiao et al., 2020, Zerveas et al., 2021, Qiu et al., 2021].
The datasets include two relatively large cases, Character Trajectories (CT) and Spoken Arabic Digits (SAD), and three small-scale cases, Epilepsy (EPSY), NATOPS, and Racket Sports (RS).
Table 5 provides the necessary information, e.g., data size, dimension, data length, and the number of classes, and more details are offered in the Appendix.
Evaluation Protocol.
We evaluate NAF on these benchmark datasets based on two protocols:
• one-vs-rest: The goal is to create N one class classiﬁcation tasks by splitting the dataset by N class labels.
The anomaly detection models are trained on data from one class and tested on data from the rest of classes.
The class used for training is labeled as normal data, while the other classes are labeled as anomalies.
• m-vs-rest: This protocol is more challenging because multiple classes m ( 1 < m < N ) are labeled as normal and the rest of classes are treated as anomalies.
In this case, the normal data is no longer from one class such that the variability increases signiﬁcantly.
[Table]
Implementation Details.
The implementation details of most baselines are replicated from Qiu et al.
and we implemented the FB-AE method using their ofﬁcial code.
For the time series datasets in Table 5, the ﬁrst four are small-scale while the SAD dataset is slightly large.
We therefore use a similar setting in small-scale tabular datasets for the time series AD tasks.
For the m -vs-rest tasks, we use the same setting m =
N − 1 , which makes the task more challenging.
Results.
Table 2 shows the results of NAF-AD and NAF-AL in comparison to the shallow and deep
AD baselines on multiple time series experiments summarized in Table 5.
NAF-AL outperforms all baselines in CT, EPSY, NATOPS, and RS experiments.
In most cases, the performance from
NAF-AD is already competitive and further improved by augmented samples from active learning.
Only on the SAD dataset, our NAF-AL is outperformed by NeuTral with learned transformations which have an advantage over the random transformations, while our active learning improvement looks marginal because its dataset size is larger than the other experiments.
Our NAF-AL shows a superior performance close to LOF but still better than the other deep baselines, like GOAD and
FB-AE.
The shallow baselines perform worse on the small-scale datasets, like EPSY, NATOPS, and
RS, but show better on CT and SAD.
Our NAF-AL can well handle both scenarios although it is designed for addressing the speciﬁc challenges from small data.
The results of the m -vs-rest tasks are shown in Table 3.
In this case, NAF-AL outperforms all baselines on EPSY, NATOPS, and RS experiments.
LOF performs best on CT and SAD and is also competitive in one-vs-rest tasks in Table 2.
It is interesting to see this KNN-based method that outperforms all deep baselines.
Compared with the deep baseline, our NAF-AL shows superior performance on 4 out of 5 experiments.
On SAD, NAF-AL is only slightly lower than NeuTral but still very competitive.
Although the m -vs-rest is more challenging, the results are consistent with the performance under one-vs-rest tasks in Table 2.
4.4 A real-world application to advanced manufacturing
Our NAF-AL method is naturally generalized to solve anomaly detection problems on general data.
In this section, we demonstrate our capability to study self-supervised anomaly detection on images.
This problem is a challenging real-world application in advanced manufacturing where only sparse labeled data is provided because the labeling process is time-consuming and needs expert’s help.
Datasets.
We collected data from an Okuma MU-8000V Laser EX hybrid manufacturing system using a coaxial meltpool camera, as shown in Figure 5.
This camera is used to monitor the shape and size of the meltpool during the directed energy deposition process.
A total number of 7913 meltpool images are collected but we only have 104 labeled normal data.
There are three subsets: Cube T4,
[Table]
T5, and T
[Table] and we have 2450 total with 31 normal data for T4, 2323 total with 40 normal data for T [Table] and 3140 total with 33 normal data for T6.
The image resolution is 576 × 704.
The normal data is labeled by domain experts from the Manufacturing Demonstration Facility at Oak Ridge
National Laboratory.
This manufacturing data can be used as a benchmark dataset for future testing and comparison in AI/ML community.
Results.
Our proposed NAF-AL method is well-suited to such problems with sparse labels.
We use the limited normal data for training and sequentially identify more normal data with higher likelihoods via an active learning scheme.
Using this way, our prediction will be gradually improved as we update our training models with more normal data.
After ﬁve active learning iterations, the histogram of anomaly scores is shown in Figure 5.
From the distribution, we can easily identify normal and anomalous data since our training model has clearly separated the normal and anomalies.
The predicted normal data with lower scores show a very consistent pattern with the data identiﬁed by experts and the anomalies with higher scores show a substantially different pattern from the normal data.
Our method also enables automatically labeling based on limited initial sparse labels, which will signiﬁcantly improve the working efﬁciency in anomaly detection for advanced manufacturing.
Conclusion
We propose a likelihood-based active learning method for self-supervised anomaly detection on small data beyond images.
The key contribution is to develop a new active learning strategy that beneﬁts from efﬁcient sampling and explicit likelihoods from neural autoregressive ﬂows.
We demonstrate the novel method on several tabular, time series benchmarks and real-world application in advanced manufacturing with superior performance over the state of the art.
We plan to generalize NAF-AL to further improve the anomaly detection performance on general large-scale datasets.
This work was supported by the U.S. Department of Energy, Ofﬁce of Science, Ofﬁce of Advanced Scientiﬁc Computing Research (ASCR), Applied Mathematics program; and by the Artiﬁcial Intelligence Initiative at the Oak Ridge National Laboratory (ORNL). This work used resources of the
Oak Ridge Leadership Computing Facility, which is supported by the Ofﬁce of Science of the U.S.
Department of Energy under Contract
No. DE-AC05-00OR22725.
A Discussion
[Figure]
How does the active learning work with NAF?
Figure 6 shows the data augmentation scheme via active learning in NAF.
Ideally, we expect the augmented samples are near the decision boundary as close as possible.
However, the decision boundary is typically unknown and difﬁcult to determine.
Instead, we pursue an ideal level set of the data likelihood (black dash line), which enables us to detect normal and anomalies.
For each iteration in active learning, an aggressive strategy is to only retain the samples with very low likelihoods but this way will expand the likelihood boundary (red dash line) across the decision boundary.
Under this way, the likelihood of anomalous data (red dots) in testing may lie at the same level set as the normal data (green dots), which confuses the detector (anomalous → normal) and hurt the detection performance.
On the contrary, one can choose a conservative strategy by rejecting the samples with relatively low likelihoods but this way will shrink the likelihood boundary which is far away from the decision boundary.
The potential issue is that the likelihood of normal samples in testing tends to be smaller and these samples are probably labeled as anomalous data (normal → anomalous).
To deal with this trade-off issue, We propose a marginal strategy that sequentially augments samples with a small proportion in each iteration and adaptively pushes them to the boundary, while controlling the likelihood level set by detecting outliers with Mahalanobis distance.
This scheme effectively avoids active learning from being too aggressive or too conservative.
How does NAF-AL improve the AD performance?
Figure 4 shows the improvement of F1-score with respect to the proportion of augmented samples ( λ = N aug /N train ) .
We choose the NAF-AD results as the base for four small-scale experiments in tabular datasets and compare the increment of
F1-score via ﬁve active learning iterations.
All experiments show a consistent trend as augmented samples are gradually added to the training.
Arrhythmia and Lympho.
experiments show better improvement since their original training data is very small.
The improvement tends to converge if more iterations are used but we choose 50 % as a threshold given training cost limitation.
Figure 4 (right) shows the effect of transformations on active learning.
Although a smaller number of transformations increases the classiﬁcation error (also reported by Bergman and Hoshen [2020]), our NAF-AL can decrease the error via sample augmentation and achieve an equivalent accuracy by using fewer transformations and thus reduce the computational cost.
B Tabular and time series data information and statistics
The statistical information of the tabular datasets and time series dataset are provided in Table 4 and 5 respectively.
[Table]
[Table]