E NERGY -B ASED A NOMALY D ETECTION AND L OCALIZATION
A BSTRACT
This brief sketches initial progress towards a uniﬁed energy-based solution for the semi-supervised visual anomaly detection and localization problem.
In this setup, we have access to only anomaly-free training data and want to detect and identify anomalies of an arbitrary nature on test data.
We employ the density estimates from the energy-based model (EBM) as normalcy scores that can be used to discriminate normal images from anomalous ones.
Further, we back-propagate the gradients of the energy score with respect to the image in order to generate a gradient map that provides pixel-level spatial localization of the anomalies in the image.
In addition to the spatial localization, we show that simple processing of the gradient map can also provide alternative normalcy scores that either match or surpass the detection performance obtained with the energy value.
To quantitatively validate the performance of the proposed method, we conduct experiments on the MVTec industrial dataset.
Though still preliminary, our results are very promising and reveal the potential of EBMs for simultaneously detecting and localizing unforeseen anomalies in images.
I NTRODUCTION
The advent of smart manufacturing and the explosion of data collection in factories and other industrial setups have brought considerable attention to the topic of anomaly detection.
Its goal is to identify rare, abnormal events from the observation of data.
This research area has already inspired a vast body of literature whose central idea is to learn a model of normalcy from the positive data and subsequently ﬂag samples that deviate from the learned model as anomalous.
Classic approaches include methods based on distances and nearest-neighbor Knorr et al.
; Breunig et al.
(2000), clustering He et al.
(2003), principal component analysis and its kernel variants Hoffmann (2007), and one-class support vector machines.
Recently, deep learning has also been used for (deep) anomaly detection, and existing out-of-distribution (OOD) detection methods were successfully applied to this problem Lee et al.
; Ahuja et al.
; Hendrycks et al.
Ren et al.
In addition, deep generative models such as variational autoencoders (VAE) and generative adversarial networks (GAN) have also been used to characterize abnormality in a variety of manners, based on gradient Kwon et al.
(2020), reconstruction error Chen & Konukoglu (2018), or density estimation An & Cho (2015).
We refer to Ruff et al.
; Goldstein & Uchida (2016) for an exhaustive review of anomaly detection methods.
The overwhelming majority of these methods deal with the detection problem, where one simply needs to discriminate normal samples from anomalous ones.
There are numerous usages, medical imaging being one example, wherein it is very important not only to identify abnormal images, but also to localize the abnormality in such images for diagnosis purposes (e.g. areas with brain lesions in CT images).
The literature on anomaly localization is relatively more scant, but includes recent methods based on deep generative models that provide spatial localization by comparing the reconstructed image with the original input Bergmann et al.
(2019b)
; Schlegl et al.
; Chen & Konukoglu (2018).
While still budding, there is also a new trend of emerging works that focus on anomaly localization
Yi & Yoon (2020); Defard et al.
Cohen & Hoshen (2021).
Energy-based models for visual anomaly detection and localization.
The principal idea behind energy-based models (EBM) is to learn an energy function E θ ( x ) :
R D
→ R that parametrizes a density p θ ( x ) as (LeCun et al., 2006):
[Equation]
where θ are parameters of the model.
An interesting property of EBMs is that, unlike probabilistic models, they don’t necessitate normalization nor a tractable likelihood.
Hence, any nonlinear regression function can act as an energy function.
These attributes provide great ﬂexibility to EBMs, and we have seen many recent works successfully take advantage of it across different data domains and a variety of energy functions and deep networks.
However, being devoid of a tractable likelihood requirement comes with its own challenges.
Stable and resource-efﬁcient training of EBMs remains an open problem and recent research has focused on training on popular datasets that were compiled for small-scale classiﬁcation problems (e.g. CIFAR10).
In this regard, the application of EBMs to new problems and different data domains is an opportunity to further develop and popularize EBMs.
Of special importance to this work, EBMs have been shown to have better performance in OOD detection than other likelihood-based generative models (Zhai et al., 2016; Grathwohl et al., 2019; Liu et al., 2020; Du & Mordatch, 2019).
This makes EBMs a promising alternative for semi-supervised anomaly detection problems, where only positive samples (i.e. normal) are available during training.
This is especially true for real-world industrial settings where defect data is limited or even unavailable, and the nature of anomalies and defects is not known a priori .
In those cases, training a model with an explicit discriminative loss term is impractical.
Contributions.
This paper proposes a uniﬁed EBM method that simultaneously addresses imagelevel anomaly detection and pixel-level anomaly localization in the semi-supervised setting.
Our main contributions can be listed as:
• An EBM method for visual anomaly detection that jointly tackles detection and localization.
• The back-propagation of gradients of the EBM energy w.r.t.
the image, resulting in a (gradient) heat map from which we can derive both an innovative detection score and pixel-level localization of anomalies in the input image.
To our knowledge, this is the ﬁrst time that gradients of energy score w.r.t input images are interpreted as spatial anomaly information.
• The application of the method to a real-world industrial case and its quantitative validation with objective metrics.
Industrial defect detection stresses the capabilities of generative models as it comes with challenges that can be hardly approximated with synthetic or research datasets.
We provide detailed detection and localization
results on
the
MVTec AD dataset, and compare against reported bechmarks.
M ETHOD
We present the details on our method in this section.
Details about the training procedure are provided in Section 3.
During inference, an input x ∈ R w × h × c is ﬁrst processed by the network in a typical forward pass to generate an energy scalar.
w and h are the width and height, respectively, of the input, and c is the number of channels ( c = 3 for RGB images, c
= 1 for grayscale images).
The gradient of this scalar w.r.t to the input (not the weights) is calculated by back-propagation to [Figure] Figure produce a gradient-map, g ( x ) , which has the same shape as that of the input,
[Equation]
Let g k denote the k th channel of g .
A visualization of these gradient-maps shows that although the gradient value is high in the anomalous regions in the image (see Fig.
2a), it is also high in other non-defective regions of the image.
Hence, distinguishing between good and defective pixels based on these raw scores may result in signiﬁcant false positives.
We can signiﬁcantly improve on this by modeling a probability density p x ( g ) over the gradient-map’s values for each pixel location individually.
Note that this density is modelled over values of g at a particular pixel location x .
The modeling is performed using the set of gradient-maps, g train ( x ) , derived from the set of training images.
During testing, the log-likelihood score of the observed gradient value, ˆ g , at x is calculated as l = log p x (ˆ g ) and this score is used as an anomaly score.
Hence, the determination of whether a pixel is good or defective is based on how much its gradient value deviates from the typical values observed on the training set images.
If the distribution used is a normal distribution, then the loglikelihood, l , is equivalent to standardizing the raw anomaly score by its mean, µ k ( x ) , and standard deviation, σ k ( x ) , as follows
[Equation]
Note that µ k ( x ) and σ k ( x ) are calculated for each per-pixel location, x , in each channel k separately, from g train values at x .
The ﬁnal anomaly score, a ( x ) at x is the L 1 or L 2 norm of the corresponding raw or standardized gradient scores from each channel, i.e.
[Equation]
As can be seen in Fig.
2b, standardized scores show much better separation between good and defective pixels.
Finally, we aggregate the per-pixel anomaly scores to obtain a single score A ( x ) for an entire image, again by taking the L 1 or L 2 norm as follows
[Equation]
This score can be used to for the classiﬁcation of the image as defective or good.
Note that A raw is essentially the same as approximate mass score proposed in (Grathwohl et al., 2019).
[Table]
E XPERIMENTS 3. T RAINING
In this work, we take an MCMC-based maximum likelihood (ML) approach to training our model.
As our energy function approximator, we use an all convolutional neural network similar to the one
Nijkamp et al.
(2019) used (see Table 3 in the Appendix), but instead of Leaky-ReLU , we use ELU (Clevert et al., 2016) activation functions between all layers to stabilize training.
To optimize for likelihood, we use Contrastive Divergence (CD) (Hinton, 2002).
To synthesize the negative samples needed by the CD objective, we employ a Stochastic Gradient
Langevin Dynamics (SGLD) MCMC sampler (Welling & Teh, 2011).
We initialize MCMC chains from a uniform distribution and run a ﬁxed number of MCMC steps ( 100 ).
For a more resourceefﬁcient training, we experimented with reducing the number of steps and using a replay buffer to persist MCMC chains (Tieleman, 2008) instead of initalizing from noise at each training iteration, but we could not manage to ﬁnd a stable training regime on all categories and had sporadic energy spikes.
We leave the solution of these problems to later work and share only the results of non-persistent trainings for consistency over all dataset categories.
During sampling, we don’t accumulate gradients w.r.t.
model parameters for later use at model parameter update step.
During training, we only use nondefective images and we don’t explicitly optimize for any discriminative objective.
In that sense, training is unsupervised.
But, as we know that the training data is not contaminated with defective samples, we prefer to call our approach semi-supervised .
We believe that a fully unsupervised method should tackle the consequences of possible training data contamination.
3. R ESULTS
We test our approach on the problem of visual anomaly detection on the recently introduced MVTec dataset (Bergmann et al., 2019a), which is a dataset for benchmarking anomaly detection methods with a focus on industrial inspection.
There are ﬁfteen different object and texture categories in the dataset, and we train a separate EBM for each category.
The training is performed using defect-free training images only and the model is not exposed to any defective images during training.
Training of EBMs is known to be computationally demanding and hence the input images are downsized, to 128 × 128 to keep the training time manageable.
For a test input, we generate both the per-pixel anomaly scores, a ( x ) , and the per-frame anomaly scores, A ( x ) as described in Section 2.
These scores can be used to distinguish between good and defective data, at both the pixel level and the frame level.
This effectively creates a binary classiﬁer, whose performance is characterized by the receiver operating characteristics (ROC) curve.
We tested performance with both the raw and standardized scores, and found that the standardized scores generally outperform the raw scores and hence, we report results using standardized scores.
For the detection task, we report the area under the ROC curve (AUROC) in Table 1, and for the localization task, we report the AUROC in Table 2.
For a comparison between raw and standardized scores, please refer to section A.3 in the Appendix.
We note that although the EBM achieves respectable scores on both tasks, there is room for improvement.
C ONCLUSION
This work presented an energy-based method for the semi-supervised anomaly detection and localization problem.
We propose an alternative scoring function for image-level anomaly detection and spatial anomaly maps for pixel-level localization; both are derived from back-propagating gradients [Table] of the energy score w.r.t.
the test image.
We experiment our approach on the MVTec AD industrial dataset and benchmark against the state of the art.
While the method is still in the early stages of development, initial results are encouraging with indications of how to obtain further improvements.
These include using larger models with higher capacity to model the energy function, and exploring regimens to perform training in a more stable and less resource-intensive manner
[Table]
A. CNN T OPOLOGY
[Table]
A. R AW VS S TANDARDIZED SCORES
We compare AUROC scores for both detection and localization using raw gradient-map scores and standardized scores.
In general, we observe that standardized scores perform better than raw scores.
For a few categories, the raw scores do provide better AUROC.
[Table] [Table]