Few-Shot Fast-Adaptive Anomaly Detection
Anonymous Author(s)
AffiliationAddress email
Abstract
The ability to detect anomaly has long been recognized as an inherent humanability, yet to date, practical AI solutions to mimic such capability have beenlacking.
This lack of progress can be attributed to several factors.
To begin with,the distribution of “abnormalities” is intractable.
Anything outside of a givennormal population is by definition an anomaly.
This explains why a large volumeof work in this area has been dedicated to modeling the normal distribution ofa given task followed by detecting deviations from it.
This direction is howeverunsatisfying as it would require modeling the normal distribution of every task thatcomes along, which includes tedious data collection.
In this paper, we report ourwork aiming to handle these issues.
To deal with the intractability of abnormaldistribution, we leverage Energy Based Model (EBM).
EBMs learn to associateslow energies to correct values and higher energies to incorrect values.
At its core,the EBM employs Langevin Dynamics (LD) in generating these incorrect samplesbased on an iterative optimization procedure, alleviating the intractable problemof modeling the world of anomalies.
Then, in order to avoid training an anomalydetector for every task, we utilize an adaptive sparse coding layer.
Our intention isto design a plug and play feature that can be used to quickly update what is normalduring inference time.
Lastly, to avoid tedious data collection, this mentionedupdate of the sparse coding layer needs to be achievable with just a few shots.
Here,we employ a meta learning scheme that simulates such a few shot setting duringtraining.
We support our findings with strong empirical evidence.
Introduction
Anomaly detection is an important area of study in the field of artificial intelligence.
It has found utilityin computer vision applications such as industrial inspection [4] and video surveillance [22, 50, 32],in the context of abuse prevention such as misinformation, fraud and network intrusion detection[49, 6, 29] and others such as system health monitoring and fault detection [3, 33].
In this paper, wepropose an approach for detecting anomaly in images where we have carefully designed steps tohandle some of the bigger issues that have prevented the deployment of image anomaly detection inthe real-world.
Image anomaly detection can generally be defined as the identification of abnormalities in a givenimage.
An exact definition of abnormality in this case is elusive because abnormality can be derivedfrom any unknown distribution outside of a normal population.
Many studies have hence focused onmodeling the normal population instead of learning irregularities, where the goal is to capture theshared concept among all of the normal data as one or several reference models.
This process usuallyrequires investing significant efforts in curating a large set of normal samples for each task , afterwhich anomaly is detected as deviations from the reference model(s) [1, 47].
Recent work from [40]provides algorithms that utilize only a few normal samples to train models from scratch.
However,the models still have to be provisioned for each new task, which requires considerable human effortsand expertise, and thus lack the fast deployment criterion that is often time critical for real-world applications.
In view of these challenges, our goals for this work are threefold.
We are interestedin designing an anomaly detection system that is capable of: (G1) modeling the normal populationwhile at the same time has a principled approach towards modeling the abnormalities; (G2) quicklyadapting to a new task at inference time; and (G3) requiring only a few normal shots to update itselfto the new task at hand.
For (G1), we introduce the class of Energy Based Model (EBM), which is an important familyof generative models [51, 12, 46].
EBMs have been shown to demonstrate superior capability onmodeling data density and localizing anomaly [15].
For our purpose, the EBM we adopted learn toassign low energy to normal samples but high energy to abnormal samples.
More importantly, theabnormal samples are generated with a procedure known as Langevin Dynamics (LD) [44], which,in its original form, starts with a noise image (see App.
Fig F) and gradually samples from thedistribution along the direction of lower energy.
This lends itself gracefully to utilizing the generatedintermediate samples as negative/abnormal.
The LD procedure is then coupled with maximumlikelihood loss [19] that aims to maximize the energy differences between the normal and abnormalsamples.
To achieve (G2), we propose an adaptive sparse coding layer that is attached to the deep featureextractor in the EBM as Figure 1 shows.
The extracted deep feature is forwarded to the sparse codinglayer, where the dictionary is constructed with the features of a few normal samples of the giventask.
In essence, the input representation has been decomposed into a linear combination of normalfeatures with the sparsity constraint imposed.
The final energy score is measured by the distancebetween the original and the reconstructed features (after the sparse coding layer).
Under this scheme,the dictionary for a particular task is not obtained by learning, but instead is constructed by the featurerepresentations of a few normal samples during inference.
As a result, this simple “plug-and-play”trick allows the model to be adapted to novel tasks promptly without re-training.
Further, we expectthat the dictionary, which is formed by normal features, will not be able to explain the abnormalsamples well, causing relatively high reconstruction error that lends itself for subsequent detection.
As a bonus, a backward pass of energy score minimization can be used for localizing abnormalregions.
We show that using gradient to localize anomalies yields superior robustness.
Towards (G3), we utilize meta learning [42, 13] to simulate the scenario of being given a new taskwith a few normal shots to update the dictionary, followed by training the EBM.
This is accomplishedby episodic training, where in each episode the model is adapted to a held back task that is givena few normal samples.
To accelerate the EBM training, we introduce “learning from inpainting”,a simple yet effective strategy for synthesizing hard abnormal samples quicker by starting the LDprocedure with a synthesized image that is simply a normal sample with a noise patch injected asopposed to a noise image that is traditionally what is used.
We show the proposed few-shot fast-adaptive anomaly detection and localization framework is ableto efficiently adapt to a novel task (e.g., a new object category or scenes from a new camera) with afew normal samples without training on both industrial inspection and video surveillance.
Comparedwith previous methods that adapts to new task through either from scratch training in few shots[40, 45] or few-shot with few steps of gradient descent [26], the proposed framework is the first thatperforms task adaptation with a single forward pass and without any gradient descent.
Despite thefast adaptation, we provide both qualitative and quantitative results to demonstrate that our methodoutperforms other adaptive frameworks and is comparable to methods that rely on large amount ofnormal samples.
Backgrounds
We briefly introduce two key ingredients of the proposed method: EBMs and sparse coding.
Energy-based Model In EBMs
, the goal is to learn an energy function E θ ( x ) : R d → R whichparametrizes the data density p θ ( x ) as:
[Equation]
where θ is the parameter of the energy function and Z θ = (cid:82) x exp( − E θ ( x )) is the partition function.
Approximating the true data distribution p data ( x ) is equivalent to minimizing the expected negativelog-likelihood function over the data distribution, defined by the loss function:
[Equation]
[Figure]
As the computation of L ML involves an intractable term Z θ , the common practice is to represent thegradient of L ML as,
[Equation]
This objective decreases the energy of positive data samples x + from the true distribution (normalsamples in our use case) and increases the energy of negative samples x − from the model p θ(synthesized abnormal samples).
In practice, the synthesized negative samples are achieved through
Langevin dynamics [44], which a J -steps sampling along the direction of energy minimization isgiven by:
[Equation]
where β is the step size, and the initialization x 0 is sampled from a predefined prior distribution.
The synthesizing ability of EBMs enables generating abnormal samples to help in learning a moreaccurate data density, and is often touted as the one of the advantages of using an EBM.
Sparse coding.
Approximating a signal z ∈ R d with the sparse linear combination over a dictionary
D ∈ R d × k can be expressed as:
[Equation]
where α is the sparse coefficients, with its sparsity ( l 1 norm) and λ is the weight of the sparsityconstraint.
D α is a sparse approximation to the original signal z .
In practice, finding the dictionaryatoms and the sparse coefficients is usually formulated as an optimization problem.
In this paper, we adopt Iterative Soft Thresholding Continuation (ISTC)
[20] to convert this optimization problem into linear operations with a non-linear shrinkage function, which allows sparse codingto be seamlessly integrated into the deep neural networks.
To compute a sparse coefficient α , ISTCperforms iterations of gradient steps on reconstruction ||
z − D
α
|| 2 and a proximal projection step toincrease coefficient sparsity.
Formally, initializing the coefficients at the first step α 0 with all zeros, each step of ISTC refines thesparse code with descending values of λ from λ max to λ
⋆ : each step of ISTC is expressed as:
[Equation]
where σ ( · , · ) here is a shrinkage function that truncates small values (lower than λ ) of the coefficientsto 0 to enforce sparsity, and can be easily implemented by a customized ReLU activation function:
[Equation]
Proposed Method
In this section, we describe the proposed fast adaptive anomaly detection framework in details.
In
Section 3.1, we introduce the adaptive EBM which consists of a deep feature extractor followed by an adaptive sparse coding layer.
From there, we further show that utilizing larger receptivefield in the sparse coding could improve training robustness (Section 3.1.1), and applying smoothedshrinkage functions could help speed up convergence (Section 3.1.2).
In Section 3.2, we describethe episodic training regime on various anomaly detection tasks that mimics few-shot adaptationin the meta-testing stage while learning common knowledge across tasks.
Instead of synthesizingnegative samples (anomaly) directly from noise, we introduce a simple but effective “learning frominpainting” operation to accelerate the training in Section 3.3.
Finally, we summarize training stepsof the proposed method in Algorithm 1.
3.1 Adaptive Energy-based Model
An EBM is a form of generative model and it is widely used for modeling data density and sampling.
While there has been recent work [15] applying EBM to anomaly detection, it still requires retraining for each new task.
To efficiently adapt the EBM to novel tasks, we introduce an adaptivesparse coding layer which is conditioned on the dictionary constructed by the features of normalsamples.
Specifically, as illustrated in Fig 1, given an input image, x ∈ R 3 × h × w , we first obtain thecorresponding feature z ∈ R d × h ′ × w ′ from the deep feature extractor Ψ with parameters θ , so thatz =
Ψ ( x ; θ ) .
All feature vectors along spatial axes of z are then sparsely decomposed through thesparse coding layer over a task-specific dictionary D ∈ R d × Kh ′ w ′ , which contains the features of
K normal samples of the current task as shown in the Fig 1(a).
Each feature vector of the normalsample feature is then directly used as an atom in the task dictionary.
The decomposed coefficientsare α = S ( z ; D ) , where α ∈ R Kh ′
w ′
×
h ′ × w ′ and S denotes the iterative sparse decompositionprocess of (6).
By multiplying the coefficient α with the dictionary D , we obtain the reconstructedfeatures z ′ =
D α .
The sparsity regularization to α is important, as it encourages input features tobe reconstructed by simple combinations of dictionary atoms (normal features), so that it wouldbe difficult for features of abnormal samples to be well-approximated, therefore producing higherreconstruction errors that make it conducive for detecting anomalies.
From here, the final energyscore is formulated as the mean squared error (MSE) between the original and the reconstructedfeatures:
[Equation]
In effect, Eqn. 8 depicts a conditional EBM, which is conditioned on the task-specific D formed bynormal features.
With the energy score, we can obtain pixe-wise anomaly localization maps through∇ x − E θ ( x ; D ) , i.e., the gradients of pixels along the direction of minimization.
High gradientmagnitudes indicate regions that cannot be well explained by the dictionary D .
Modifications tothese regions can potentially remove the anomaly and reduce the energy as in Eqn. 4.
In Section 4.1and Appendix Section B.5, we show that using the gradient (as a natural ingredient of EBMs with
LD) is more robust compared with auto-encoder and reconstruction based methods to generalize wellto unseen tasks (Appendix Figure C).
In the following sections, we will discuss how to make thetraining of this adaptive structure more robust.
3.1.1 Sparse Coding with Receptive Field.
As discussed in Section 3.1, the input feature z is represented as h ′ × w ′ of d -dim feature vectorsand they are treated independently while passing through the sparse coding layer.
The region ofthe input image that affects one feature vector is determined by the receptive field of the featureextractor.
The trade-off is that a small receptive field may not capture enough contextual information,while applying a large receptive field would make feature maps spatially coarse and make it hard tospot small anomaly regions.
To solve this dilemma, instead of carefully tuning the receptive fieldof each layer of the feature extractor, we introduce a simple yet effective technique of applying thereceptive field on the sparse coding layer.
Specifically, as illustrated in Appendix Fig A, rather thanperforming sparse coding to each individual d-dim feature vectors, we apply it on d × l × l volumescentered around each feature vector, where l is the receptive field.
This is equivalent to applying al × l , sliding window on spatial axes of the feature map and can be easily implemented by imageto column ( Im2Col ) operation.
Then we flatten the feature volumes into dl 2 -dim vectors and adjustthe shape of the dictionary accordingly.
In this way, we are able to capture contextual informationwithout needing to carefully tune the architecture of the feature extractor and we show in the laterexperiments that this technique improves the robustness of the network on different types of objects.
56789:5i … 56789:5 n !<
669=> 85>
Adaptation Dictionary ‘plug and play’ ! ?
<5=@ 85>
Normalsamples " (Evaluation)
Synthesized abnormalsamples
Step
Step Step
Step Step
Process of Langevin dynamics 56789:5 n- "
3.1.2 Shrinkage Function
The effectiveness of training the EBM for localizing anomaly regions heavily depends on the gradientpropagation from later to earlier layers.
It is shown in [11] that smooth activation functions like
Swish [35] could be beneficial here.
Notably, the gradients of the dictionary D are determined by thesparse coding coefficients α as shown in Eqn. 6.
However, the sparsity constraint of α would turn offthe gradient computation of many elements in D and this could be detrimental during the early stageof the training.
To alleviate the sparse gradient issue, we replace the RELU-like shrinkage functionin Eqn.
7 with its smoothed counterparts by introducing the Sigmoid based shrinkage functions(SigShrink).
The SigShink is originally proposed for non-parametric signal estimation in [2], and canbe defined as:
[Equation]
(9) where τ is the hyperparameter of smoothness.
We present visualizations of the hard shrinkagefunction Eqn. 7 and SigShrink with different values of τ in Fig B.
Comparing to the hard shrinkagefunction which truncates small values into zeros, the SigShrink with a large τ can sharply force smallvalues to near-zeros.
Therefore, the SigShrink will guarantee non-zero gradients everywhere.
3.2 Episodic Training
To train the proposed adaptive EBM, we perform episodic training that is widely adopted by metalearning few-shot learners [13, 41].
Following the terminology of few-shot learning, in each trainingepisode, the model is adapted and tested with a task sampled from the underlying task distribution.
Specifically, the model is adapted to a support set of the given task, then a query set with groundtruth labels is applied to evaluate the adaptation, which is used to update the model parameters.
Asshown in Fig 2, the support set of the i -th episode task contains a small number of K normal samples{ s ik } Kk =1 .
The features z ik = Ψ ( x ik ; θ ) of these normal samples are plugged into the dictionary
D i ∈ R
d × Kh ′
w ′ corresponding to the i -th task to adapt the dictionary.
After that, the adapted modelis measured by a query set consisting of M normal samples { q im } Mm =1 and M abnormal samples{ ˆ q im } Mm =1 .
Note that there is no actual abnormal samples given during training, instead, they areiteratively sampled from the EBM and the sampling will be discussed in details in Section 3.3.
Recallthat the training of EBM with contrastive divergence as in Eqn.
3 requires the estimation of energyscores of both positive samples from the true data distribution and negative samples from the modeleddistribution.
The positive energy can be estimated empirically with normal query set samples.
Thenegative energy can be estimated by performing the MCMC-based sampling technique [31, 44],typically Langevin Dynamics as described in Eqn. 4.
Denoting the output of Langevin dynamics(sampled abnormal samples) initialized with ˆ q
im as LD (ˆ q im ) , we have the empirical estimation of thecontrastive divergence of the i -th episode as:
[Equation]
With the energy score equivalent to the feature reconstruction error in Eqn. 8, minimizing L cdencourages normal features to be well-reconstructed by a sparse linear combination of dictionaryatoms while the features from abnormal samples tend to produce relatively higher reconstructionerrors so that they can be easily spotted.
3.3 Synthesizing Negative Samples
Typical EBM training with contrastive divergence conducts negative sampling from the modeleddensity using techniques such as Langevin Dynamics, which applies gradient descent to a noiseinitialization (App. Fig F) with small step size and large number of steps [12].
Such negative samplingsteps can be costly and we argue that it is unnecessary in our case.
Instead, we introduce a newstrategy of “learning by inpainting”.
Starting from a positive query sample q im , we synthesize thecorresponding negative sample
ˆ q im by randomly placing a small uniform noise patch on the image.
The Langevin Dynamics procedure is then initialized with the resulting image instead of a noiseimage.
As the Langevin Dynamics proceeds, synthesized abnormal samples LD (ˆ q im ) are inpaintedalong the direction of “normal”, q im , and we introduce the following reconstruction loss:
[Equation]
We show in Fig 2(a) that, starting from a synthesized abnormal sample, only 5 steps of Langevindynamic would be sufficient to make it visually close to the corresponding normal sample duringtraining, serving as “hard negatives” that further facilitates the learning.
The final loss of the episodictraining is simply:
[Equation]
where η 0 and η 1 are balance two loss terms.
We summarize the overall training of the proposed few-shot adaptive anomaly detection method in
Alg. 1.
Experiments
In this section, we conduct evaluation on the industrial inspection task with the MVTec-AD dataset[4] (Section 4.1).
Even though our proposed framework is image-based, we further demonstrate it’sefficacy on the video anomaly detection task in Section 4.2.
In Section 4.3, we show ablations andinsights relating to the adaptive sparse coding components.
We show additional ablations includingthe superiority of using gradient of EBMs over pixel-wise reconstruction to localize anomalies in
App.
B and we provide implementation details in App. A.
4.1 Industrial Inspection
The goal of this anomaly detection task is to predict whether a manufactured component contains anydefects.
The MVTec-AD dataset includes 15 categories of object.
To demonstrate the fast adaptation
[Figure]
Category
[Figure]
Carpet
[Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
Grid
[Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table]
Leather
[Table]
[Table]
[Table]
[Table] [Table] [Table] Tile
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
Wood
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
Bottle
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
Cable
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
Capsule
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
Hazelnut
[Table]
capability of the proposed method, we adopt a leave-one-out training strategy.
Specifically, samplesof each target category are reserved for testing only, and the episodic training is performed on theremaining categories.
During the training stage, the model will not see any samples from the targetcategory.
During testing, we first adapt the model to the target category with 10 randomly selectednormal samples , then measure the performance with the entire testing set.
We run the test 5 times,each time the model is adapted to random sets of 10 normal samples from the target category.
Thefinal result is the average of the 5 runs.
In Table 1, we first show performance of “upper-bound” methods, which train each category fromscratch with massive normal samples.
Specifically, [5, 4] train auto-encoders (AE) with normalsamples and measure the reconstruction errors during the inference; AnoGAN [38] adopts a generativeadversarial network (GAN) to learn the manifold of normal; VE-VAE [23] presents a visuallyexplainable variational auto-encode through gradient-based attention.
For apple-to-apple comparison,we create a strong baseline by applying model-agnostic meta-learning [13] on an AE (denoted as
MAML-AE, more details in App.
Sec. A.3).
Hierarchical transformation-discriminating generativemodels (HTDGM) [40] trains a GAN-based anomaly detector in few shots.
Although HTDGMallows adapt to a new task with few normal samples, the model still needs to be trained from scratch,which takes 4000 iterations in the 10 shot setting based on their official implementation.
To thebest of our knowledge, the proposed method is the first to allow adapt to new tasks with a singleforward pass without any training , therefore the learned parameters are shared across tasks whichgreatly accelerate the model deployment.
All results from our methods are obtained without any dataaugmentation .
The proposed method outperforms MAML-AE by a large margin, and performs betterthan HTDGM [40] in majority of tasks.
Our results are even competitive with the “upper-bounds”in some categories.
We show the localized anomaly regions from our method in Fig 3.
Additionalvisualizations are in the App.
Fig D.
4.2 Video Surveillance
In video anomaly detection, a common goal is to detect abnormal events captured by surveillancecameras (e.g., a motorcycle on the sidewalk).
A model trained on videos from one camera might not
[Figure]
[Table]
[Figure] [Figure]
[Table] [Table]
[Table]
generalize well on other cameras due to different locations / mounting heights / lightning conditions,and it is not feasible to train one model for every new camera in practice.
The ability to quickly adaptto new scenes is a significant contribution to the task of video surveillance.
We are only aware of thework in [26] (r-GAN) that has such adaptation capability.
Specifically, the model adapts to a newscene using gradient descent with several beginning frames of a query video, after which a GAN isapplied to generate future frames.
Anomaly is then detected via the discrepancy between predictedfuture frames and the original frames.
Note that the MAML-AE baseline we conducted in Section 4.1can be seen as an ablation of r-GAN on the single-frame without temporal information.
We follow the same evaluation regime as r-GAN by training with normal samples in all 13 scenesfrom SH-Tech [22] and testing on UCSD Pedestrian 1, UCSD Pedestrian 2 [28], and CUHK Avenue[24].
Note that since our method is image-based, it predicts the video frames independently withoutleveraging any temporal information as in r-GAN.
In each episode, we adapt our model with a supportset containing a few normal frames randomly sampled from the target scenes.
In Table 2, we compareour method against r-GAN pre-trained on SH-Tech only (r-GAN Pre-train), fine-tuned on targetdatasets (r-GAN Fine-tune), and with one step gradient descent with meta-learning (r-GAN MAML).
We also show the performance of MAML-AE as a baseline for image-based meta-learning method.
Inthe last section of Table 2, we present intra-dataset results as well by training with 6 scenes of SH-Techand testing on remaining 7.
We follow common evaluation protocol and measure the frame-level
AUC-ROC.
Without leveraging temporal information and re-training (gradient descent), our method achieves comparable results to r-GAN MAML and outperforms image-based meta-learning methodby a large margin.
In App B.4, we show that incorporating simple temporal information can furtherimprove the performance.
4.3 Ablation Studies
Sparse coding receptive fields.
To evaluate the effectiveness of using large receptive fields in thesparse coding layer, we conduct additional experiments on the MVTec-AD dataset, and select 5representative categories with different levels of difficulties to present the comparisons with l = 1 andl = 3 (Sec. 3.1.1) in Table 3.
Sparse coding with large receptive field clearly benefits more complexstructural objects (hazelnut, cable, and capsule), while the improvements are limited for the textureobjects (leather and grid), where contextual regularization is intuitively less important.
Shrinkage functions.
To show the benefits of smooth shrinkage function, we plot the loss curvesof models trained with smooth SigShrink (Eqn. 9) and non-smooth RELU-like shrinkage (Eqn.
7)functions in Fig 5.
The model with smooth shrinkage function converges notably faster in the earlytraining stage and achieves lower loss.
Sparsity constraint.
As discussed in Section 3.1, we impose sparsity constraint to the featuredecomposition in the adaptive sparse coding layer, in order to prevent abnormal features from beingwell-approximated by the linear combinations of normal features, so that the reconstruction errorsare effective for detecting anomaly.
To validate this, we conduct experiments by removing theshrinkage function σ in the sparse coding stage (Eqn. 6).
We show comparison in Table 4 withmIoU, AUC-ROC, and the difference of averaged reconstruction errors between abnormal and normalsamples.
Without sparsity, the performance drops dramatically, and reconstruction errors of normaland abnormal samples become closer.
Related Work
Anomaly detection with sparse coding.
Early efforts on adopting sparse coding in anomaly detectionare based on optimization (with L1 penalty)
Recent advances on iterative sparse thresholdingalgorithms [9, 20] allow seamless integration of online sparse coding with deep neural networks, and[27] formulates the sparse coding as stack RNNs for video anomaly detection.
Anomaly detection with generative models.
Generative models are widely utilized in anomalydetection due to the capability in modeling the density of desired data distribution.
Early effortson variational autoencoders (VAE) based methods [1, 47] are arguably having hard time calibratinguncertainties in novel samples [30], accurately localizing abnormal regions through reconstructionerrors [10].
Recent efforts have explored variant generative architectures like energy-based models(EBM)
[15], GANs [40], and combining VAE with EBM [10].
Various methods also exploit intraimage structures [8, 5], cross-frame consistency [25], and motion-appearance consistency in videos[32] while detecting anomaly.
Few-shot learning.
Few-shot learning is extensively explored in classification tasks.
Proposedmethods are based on optimization [13, 37, 14, 48, 36], learning metric [41, 43] and parameterprediction [17, 34, 16].
These technologies are further applied in other tasks like image generation[7, 21] and out-of-distribution detection [39].
Conclusion
In this paper, we introduced few-shot fast-adaptive anomaly detection.
We formulated our model as anenergy-based model with an adaptive sparse coding layer, of which the dictionary is directly formedby normal features of a target task.
We adopted episodic meta-learning to learn common knowledgeacross tasks, which enables few shots adaptation.
We further introduced smooth shrinkage functions,sparse coding with large receptive fields, and learning by inpainting to improve and accelerate thetraining.
Notably, when evaluating our method’s performance on industrial inspection and videoanomaly detection, our method is comparable and even boasts better performance than methodstrained with a large amount of normal samples.
Through this work, we hope to have made a significantcontribution to the important problem of anomaly detection by shedding light on our findings thatanomaly detection can indeed be generalized to new tasks with a few normal samples only.
Social Impact and Ethics.
As a general framework for few-shot anomaly detection, the proposedmethod does not suffer from particular ethical concerns or negative social impacts.
All datasets usedare public, and we have blurred all human faces in the qualitative visualizations.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contribu-
tions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[No]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental
results (either in the supplemental material or as a URL)?
[No] Code will be released upon
acceptance.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experimentsmultiple times)?
[No](d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs,internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If
[Table]
[Table]
[Table] [Table]
[Table]
[Table]
[Table]
[Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table]
[Table] [Table]
[Table]
Appendix
A Implementation Details
We adopt a residual network (ResNet) [18] based feature extractor, with ELU as the activation function.
Following 493 [11], we adopt Group Normalization for better stability of the network.
The detailed network construction isshown in Table A.
[Table]
A.1 Industrial Inspection
We adopt the “leave-one-out” training strategy for obtaining the results on each of the categories of MVTec-AD.
497 All experiments are performed with the same settings and hyperparameters.
We resize all images to 128 × 128 ,and do not perform any data augmentation.
We adopt a simple reduced-sized ResNet as the feature extractor 499 as shown in Table A.
Following [11], we adopt group normalization (denoted as GroupNorm) instead of batchnormalization, and use Exponential Linear Unit (ELU) as the activation function.
We empirically observed that 501 using Tanh as the final activation function can remarkably improve the numerical stability of the sparse codingstage as the magnitude of the feature values is effectively bounded by the final activation function.
We adopt Adam as the optimizer, with a consistent learning rate of 1e-4.
We do not apply any net regularization 504 methods like dropout or weight decay in training.
The weights of reconstruction η 0 and contrastive divergenceη 1 are set to 1.0 and 0.25, respectively.
Each training batch contains 4 randomly sampled training tasks with 10 506 query ( M = 10 ) for each task.
All training can be conducted on a single NVIDIA Tesla A100 GPU.
We perform 8 steps of sparse coding in the adaptive sparse coding layer, with an initial λ max = 0 .
3 and a final 508 λ ⋆ = 0 .
We perform 5 steps of Langevin dynamics with a step size of β = 1 . 0 to synthesize negative 509 samples, which we show in the examples of Figure 2 and Figure F to be sufficient for producing hard negativesamples.
A.2 Video Anomaly Detection
We resize each frame to 240 × 320 .
No data augmentation is performed.
All other hyperparameters equal to 513 those applied in industrial inspection experiments.
A.3 MAML-AE
We adopt a full-convolutional auto-encoder network to construct the MAML-AE.
A 10 layer ResNet [18] as the 516 encoder, and consecutive transpose-convolutional layers with batch normalization and RELU activation functionas the decoder to recover the feature resolution.
The hyperparameters of the episodic training of MAML AE are 518 exactly the same with those of our methods.
We perform 5 steps of gradient descent as the inner-loop adaption.
[Figure]
𝑑 𝑙
[Figure]
[Figure]
We directly use the energy score in (Eqn.
8) as the anomaly score.
When evaluating AUC-ROC and mIoU, we 520 obtain the normalized energy score for each sample by uniformly normalize to the scores of all samples in testset.
A.4 Initializing Negative Sample
In the synthesis of negative samples, we randomly place at most three random patches to each normal image.
Each patch is created by one of the following:
• Random uniform: A random patch with the values of each pixel sampled from a uniform distribution.
The minimum and maximum values of the uniform distribution is equal to those of the pixel values ofthe images
• Random consistent: A random patch with consistent pixel values sampled from a uniform distribution.
The minimum and maximum values of the uniform distribution is equal to those of
the
pixel values ofthe
images.
• Random copy-paste: A random patch randomly cropped from the same image.
• Random blurring: Applying Gaussian blurring to a random patch of the normal image.
Denoting U ( a, b ) as an uniform distribution with minimum and maximum values of a and b , respectively, the 534 relative size of the random patch w.r.t.
the original image is randomly sampled from U (0 . 0025 , 0 . 025) , andthe aspect ratio is randomly sampled from U (0 . 01 , 100 . 0) .
See Fig.
E for illustrations of synthesized negative 536 samples and the corresponding generative sequences of Langevin dynamics.
A.5 Illustrations
Sparse coding layer with receptive field.
We illustrate sparse coding with receptive field in Fig.
A.
An l × l 539 sliding window is applied spatially to the feature map, and the feature volume in each window is flatten as afeature vector to perform sparse coding.
In this way, the context information of each position is effectively 541 considered in the adaptive sparse coding layer.
Shrinkage functions.
We plot different shrinkage function in Figure B. Beside the desired property of sparse 543 projection, the SigShrink we use enjoys the advantages of being differential and having non-zero gradientseverywhere.
B Additional ablationsB.1 Robustness to Pose Variations
To validate the robustness of our method to pose variations, we perform additional experiments with pose 548 variations and provide the quantitative comparisons in Table
B. We report results by performing random ± 90 orotation to samples during testing stage when adapting the trained model to novel categories.
We observe that 550 the textures categories (leather and grid) are robust to any pose variations in both the query and support samples.
For the hazelnut category, where there are intrinsically significant pose variations across samples, performing 552 any new rotations does not influence the results.
On the other hand, for the pill category, where all samples arealigned horizontally (see examples in Appendix, Figure D), performing rotation to only query samples results in 554 performance drop.
Performing rotation to both query and support samples help to recover the performance.
Theabove results suggest the robustness of our methods under the condition of sufficient sample diversity in the 556 support set.
[Table]
B.2 Comparison to Image Differencing
[Table]
Image differencing and its variances are an intuitive approach for detecting image anomaly.
We perform [Table] image differencing and background subtraction on the industrial inspection and video anomaly detection task,respectively, and report performance in Table C.
Our method demonstrates clear advantages.
[Table]
B.3 Robustness Against Contaminated Training Data
While it is a common practice among machine learning practitioners to assume clean training data, this may 563 not be true in real world applications.
In this section, we evaluate the robustness of our method againstcontaminated training data by inserting certain amount of abnormal samples.
As shown in Table D [Table] we 565 progressively contaminate the normal training data by increasing the amount of abnormal data from 1% to10% .
Our proposed method is in general robust to data contamination, where contamination under 5 % only 567 decreases the performance slightly.
The network is still able to perform decently in the extreme case of 10%contamination.
[Table]
B.4 Leveraging Temporal Information
An EBM is agnostic to the underlying backbone network architecture, therefore it is straightforward to incorporate 571 temporal information into the image-based anomaly detection framework described in Section 5.2.
We replacethe 2D convolutions in the first four layers with 3D convolutions, and have the model accepts a 5-frame input 573 instead.
Without heavy tuning, we report 5-shot performance on both ‘1 frame’ (original, without temporal) and‘5 frames’ (with temporal) in Table
E.
The results show that when temporal information is added, it leads to 575 further improvements.
B.5 Adaptive Sparse Coding Layer with Auto-encoder
Auto-encoders are widely used in anomaly detection with the localization performed by measuring the recon- 578 struction error.
An straightforward baseline of few-shot anomaly detection can be constructed by plugging theproposed adaptive sparse coding layer as the bottleneck of an autoencoder.
However, we empirically find it 580 very difficult to train a decoder that can generalize well to unseen classes.
As visualized in Fig.
C, the poorgeneralization of the decoder consistently results in drastically high reconstruction error when applied to novel 582 test tasks.
Quantitatively, this simple baseline consistently delivers nearly 0 mIoU, therefore is not included inthe comparisons.
On the other hand, utilizing only an encoder and using the backward gradient of the EBM to 584 localize the anomaly demonstrate outstanding robustness on novel tasks that are unseen during the training stageas we discuss in Sec. 4. [Table]
C Qualitative Results
We present in Figure D additional anomaly localization results of categories with different anomalies in the 588 MVTec-AD dataset.
D Additional Figures
Common practice of sampling from EBMs is computationally costly, requiring as many as 50 steps of Langevin 591 Dynamics when initialized with full noise, instead of a synthesized negative sample as shown in Fig. E. Someexamples are provided in Fig. F.
[Figure]
GT
Prediction
Input
GT
Prediction
[Figure]