Re Learning Memory Guided Normality for Anomaly Detection
Anonymous Author(s)
AfﬁliationAddress email
The authors have introduced a novel method for unsupervised anomaly detection that utilises a newly introduced
“Memory Module” in their paper [1].
We validate the authors’ claim that this helps improve performance by helping the
network learn prototypical patterns and uses the learnt memory to reduce the representation capacity of Convolutional
Neural Networks
Further we validate the efﬁcacy of two losses introduced by the authors,
“Separateness
Loss” and
“Compactness
Loss” presented to increase the discriminative power of the memory items and the deeply learned features
We test the efﬁcacy with the help of t-SNE plots of the memory items.
Methodology
The authors provide a codebase available at https://github.com/cvlab-yonsei/MNAD with scripts for the Prediction task.
We reused the available code to build scripts for the Reconstruction task and variants with and without memory.
The completed codebase for all tasks available at https://anonymous.4open.science/r/40136d86-adcb-44cf-8f71-b70ad55a1bde/.
Results
We obtain results that are within a 3% range of the reported results for all datasets other than on the ShanghaiTechdataset[2].
The anomalous run stuck out since the "Non Memory" of the same run could score markedly better than the"Memory" variant.
This led us to investigate the behaviour of the memory module and found valuable insights whichare presented in Section 5.
What was easy
• The authors provide codes to run the Prediction task with memory.
• The ideas presented in the paper were clear and easy to understand.
• The datasets were easily available, with the exception of the ShanghaiTech dataset[2].
What
was
difﬁcult
• After training the model on the ShanghaiTech dataset[2], we found that the “Memory” variants of the modelhad issues as described in Sections 4 and 5.
We investigated the behaviour of the memory module and
introduced additional supervision to solve the problem.
• We required hyperparameter optimization to hit the reported scores.
• The paper also lacked information as to which parts of the pipeline were actually to be credited for the
improvement in performance, as discussed in Section 5.
Communication with original authors
The codebase has scripts only for the Prediction task "Memory" variant of the benchmarks.
We reused portions of thecodebase to create the other variants.
We then validated our modiﬁcations with the authors.
We also communicated the discrepancies we saw in parts of the results and the authors asked us to wait until they update their repository with themissing scripts.
Submitted to
Introduction
The paper that we have tried to reproduce attempts to detect anomalous events in a video sequence.
Anomaly detectionbased on Convolutional Neural Networks (CNNs) usually leverage proxy tasks like reconstruction of input framesor prediction of future frames, i.e., trained only on the "Normal" scenes thereby having a hard time reconstructinganomalies.
Conventionally, Peak Signal-To-Noise ratio (PSNR) is used to gauge if the input batch of images hasanomalous frames.
The authors have introduced a hybrid metric for Abnormality Score calculation, comprising aweighted sum of the PSNR value and L2 loss between the Queries and the Memory items.
We utilise the term "Reconstruction" to refer to the model and the term "reconstruction" to refer to the output generatedby the model (s).
The author’s primary contribution is the introduction of a "Memory Module" which is claimed to memorize prototypicalpatterns of normal data, and these memory items are used to reduce the representation capacity of the CNN which, intheory, decreases the quality of the output (Reconstruction or Prediction) when the input batch has anomalous frames,thereby making it easier to classify input batches as anomaly or not.
The authors also introduce two losses, namely,
Compactness loss and Separateness loss which help increase the diversity and discriminative power of the memoryitems.
The ﬁnal contribution is the introduction of Test-Time Updation of memory to help tune the memory itemsaccording to the normal scenes during inference.
Scope of reproducibility
We have attempted to reproduce the reported scores of the "Memory" variant and "Non Memory" variant on all datasetsfor both the Prediction and Reconstruction tasks, alongside reproducing the scores for the ablation studies as well.
Theauthor’s central claim is that the "memory module" helps the network learn prototypical patterns of normal scenes, andusing the learnt memory items as "noise" reduces the representation capacity of the CNN and makes it harder for thenetwork to reconstruct/predict anomalous output thereby making it increasingly easier to classify anomalous inputbatch of images.
We ran experiments to compare the performance of the Reconstruction and Prediction models with and without memorymodules.
While benchmarking these models we observed a few deviations in the reported results and reproduced resultswhich lead us to experiments to get more clarity on the behaviour of the memory module, which we shall discuss in
Section 4.
We provide the hyperparameters with which we got the best results on the provided datasets.
Their secondary claims revolve around the Compactness loss and Separateness loss, which they claim helps the networkspace out and the memory items and decreases the spread between similar deeply learned features.
They also claimthat the Test-Time updation of the memory items helps tune the memory items to the normal scenes during inferencetime, subsequently classifying anomalies more accurately.
We attempted to reproduce the ablation studies results whichcovers both the secondary claim and the Test-Time updation scheme claim.
The claims we are testing:
• "Memory module" helps the network to learn prototypical patterns of normal scenes, and using learnt memoryitems to lessen the capacity of CNNs.
• feature compactness and separateness losses to train the memory, ensuring the diversity and discriminativepower of the memory items.
improves performance.• update scheme of the memory, when both normal and abnormal samples exist at test time, improves perfor-
mance.
Methodology3.1 Model descriptions
The authors present a novel algorithm for unsupervised anomaly detection that makes use of a memory module.
Themodel architecture comprises of three parts: The encoder, the memory module and the decoder.
The model eitherreconstructs or predicts a video frame with a combination of features from the encoder and the memory items ratherthan just the features from the encoder.
The memory module consists of 10 items of 512 each.
A “read” operation generates a tensor of size 32*32*512 fromthe memory module which is then concatenated to the tensor obtained from the encoder.
This obtained matrix is concatenated with the query features to generate a tensor of depth 1024, then passed to the decoder.
The key idea hereis that the memory item is used to generate a tensor using a weighted average, thus utilising all the memory items.
Thisallows the model to learn diverse patterns.
The network and memory module is trained only on normal frames.
As aresult, the cosine distance between the query and the relevant memory item is less and they are close to each other.
Onthe contrary, since the network has not seen the anomalous frames during the training phase, the memory item will befar from the queries obtained.
Since they are far, the Query-Memory pair will be sub-optimal thus resulting in poorreconstruction of the input frames.
To read the items, the cosine similarity score is computed between each query q kt and all memory items p m , resulting ina 2-dimensional correlation map of size M × K , A softmax function is then applied along a vertical direction, andmatching probabilities w k,mt are obtained as shown in Equation 1.
[Equation]
For each query q kt , the memory items are read by a weighted average of the items p m with corresponding weights w k,mtand obtain ˆ p kt as shown in Equation 2.
The ˆ p kt values computed from each query makes up the feature map which isthen concatenated to the features obtained from the Encoder depth wise.
[Equation]
The "update" operation updates the memory items as it iterates through the dataset and learns the pattern of normalfeatures.
To update the memory, for each item, all queries that are closest to the memory item are chosen using thematching probabilities in Equation 1.
The set of indices for the corresponding queries for m -th item in the memory aredenoted by U mt .
The item using the queries indexed by U mt is updated as shown in Equation 3.
f ( · ) is the L2 norm.
[Equation]
A weighted average of the queries is used.
The matching probabilities are computed by applying the softmax functionto the correlation map of M × K along a horizontal direction as shown in Equation 4.
[Equation]
The authors choose to use PSNR as a part of the metric to check the output quality from the decoder.
Along with this,the authors utilise an L2 distance loss calculated between the queries and its corresponding nearest memory item.
The
PSNR and L2 distance is coupled together as shown in Equation 5.
[Equation]
The authors also introduce a new pair of losses, Compactness loss and Separateness loss.
The Separateness lossminimizes the distance between each feature and its nearest item, while maximising the discrepancy between the featureand the second nearest one, similar to Triplet loss [3].
The feature Compactness loss maps the features of a normalvideo frame to the nearest item in the memory and forces them to be close.
These losses coupled, spread the individualitems in the memory and as a result enhance the discriminative power of the features and the memory items.
The architecture used in Reconstruction task "Memory" variant is similar to that of the Prediction task "Memory"variant with the exception that it does not have skip connections.
The Prediction "Non Memory" variant resemble a
U-Net [4] and the Reconstruction models resemble a simple Autoencoder [5].
The authors have provided a codebase available at https://github.com/cvlab-yonsei/MNAD.
However, the codebasehas only the codes for the "Memory" variant of the Prediction task 1 .
We reused a major portion of the available code to build scripts for the Reconstruction task and variants with and without memory.
We validated the new scripts andchanges by communicating with the authors over email to ensure there is no gap between their intention and ourdeveloped codebase.
The completed codebase for all tasks available at https://bit.ly/3s3nTIR.
The available code for the“Prediction” task could be run on the Ped2[6] and CUHK[7] datasets without any changes.
However, the ShanghaiTechdataset[2] required us to generate the frames from the provided videos.
We utilise the VideoCapture class available onthe OpenCV library to generate 2,74,515 frames.
We make the following changes in the available code to create the scripts for the Reconstruction task:
• We tweak the code to accept one input and generate one output of the same frame by changing the time step
parameter. • We remove the skip connections present in the architecture for the Prediction task.
We make the following changes for the " Non Memory" variants:
• The decoder needs only the query features and hence we change the ﬁrst layer of the decoder to accept andcreate a 512 size vector.• For the "Non Memory" variants of the models, we cannot calculate the L2 distance and only the PSNR valueis to be used for Abnormality Score calculation in the evaluation script.
3.2 Datasets
We test and report results on all datasets that the authors have reported scores on.
The three datasets used are UCSD
Ped2[6], CUHK Avenue[7] and the ShanghaiTech[2] anomaly detection dataset.
All three datasets are readily availablein the author’s repository.
However, only the UCSD Ped2 dataset[6] and CUHK Avenue dataset [7] are available directlyin the format required by the authors’ code.
The ShanghaiTech dataset[2] contains only videos which we had to splitinto frames using OpenCV to generate 274,515 frames.
We have provided the script used to save the frames on the
GitHub repository.
3.3 Hyperparameters
We experimented heavily with the λ hyperparameter to achieve the reported scores.
We could not ﬁnd a ﬁxed trend inits behaviour across the three datasets.
We have provided the best results after tweaking λ and the results obtained asper the author’s recommendation of hyperparameters as discussed in Section 4.1.
As for the other hyperparameters,we found that the recommended values worked well and gave us similar results to the paper with the exception of the
ShanghaiTech dataset[2].
3.4 Experimental setup
We use multiple hardware set-ups for our experiments, namely, vast.ai instances, Google Colab and our personalsystems.
We used NVidia Tesla T4 and P100 from Google Colab, along with 1080 Ti, 2070, 2070 SUPER, 2080 Tirented on vast.ai.
To maintain uniformity, we benchmark all our scores on a 2080 Ti.
We recorded all our trainingmetrics on “wandb” 2 .
3.5 Computational requirements
For our experiments and benchmarking, we use a rented GPU from vast.ai.
All our scores are reported on a 2080Ti with8 GB vRAM.
The training times are as shown in Table 1.
The inference times and model parameters for the Prediction task "Memory" variant and Reconstruction task "Memory"variant are as shown in Table 2 and Table 3 respectively.
Results
The paper has two major tasks, “Prediction” and “Reconstruction”, each of which has a “Memory” and “Non Memory”variant.
We test these 4 models on all three datasets that the authors have evaluated on, namely Ped2[6], CUHK Avenue
[Table] [Table]
[7] and ShanghaiTech[2].
We obtain results that are within a 3% range of the reported results for all datasets other thanon the ShanghaiTech dataset[2].
However, we observed that hyperparameter optimization has to be done to obtainthe best results possible.
The weighted average of the PSNR and L2 distance between query and memory item, λ in
Equation 5, was the primary hyperparameter we had to tune for optimal results.
We have reported the results obtainedon the hyperparameter values chosen by the authors and the best obtained results, post hyperparameter optimisation,discussed in Section 4.1.
We further validate the efﬁcacy of the introduced Separateness Loss, Compactness Loss and Test-Time updation thatthe authors claim improves performance.
These results are discussed in Section 4.2.
We ﬁnd that the obtained scores donot follow the trend shown in the original paper.
In our experiments, we found that the results majorly supported claim 1.
The t-SNE plots also show that the supervisionproposed by the authors work well for ensuring diversity and increasing the discriminative power of the memory items,i.e., supporting the second claim.
The results we observed for complex datasets like ShanghaiTech[2] clearly do notsupport the idea behind Test-Time Updation.
Training the proposed Prediction task "Memory" variant on the ShanghaiTech dataset[2] posed problems, which otherdevelopers have also faced (https://github.com/cvlab-yonsei/MNAD/issues/6).
The "Memory" variants also performedworse than the "Non-Memory" variants on this dataset.
This led us to study the behaviour of the memory module.
We checked the distribution of the memory items vs deeply learned features and observed that the distribution wascompletely lopsided and only a part of the memory items were being used.
This was consistent throughout the datasetsbut the skew in the distribution of the memory items linked to the deeply learned features in the ShanghaiTech dataset[2]was extreme, speciﬁcally, only one memory item was linked to all the features.
Ped2[6] and CUHK Avenue[7] had 6and 9 memory items linked to the deeply learned features, respectively.
We introduced a new supervision (discussedin Section 5) to force a uniform distribution on the memory distribution to help with this problem and we were ableto reproduce the reported scores on the "Memory" variants for both the Prediction and Reconstruction tasks on the
ShanghaiTech dataset[2], which previously failed.
Our proposed solution is not ideal, but was enough to ﬁx these issuesto be able to reproduce the reported scores.
4.1 Benchmarking Results
The results obtained on the hyperparameter values provided by the authors are provided in Table 4.
The best obtained results are provided along with the hyperparameter values in Table 5.
We evaluated the models savedat every 5th epoch and tweaked values of λ and Batch Size.
4.2 Ablation Results
In our ablation study experiments, we validate the following:
• We test the efﬁcacy of the Separateness Loss, Compactness Loss and Test-Time updation for both theReconstruction and Prediction tasks on the UCSD Ped2 dataset[6].
The results are as shown in Table 6 and
Table
[Table] [Table]
• We compare the distribution of query features, learnt with and without the Separateness Loss on the UCSD
Ped2 dataset[6] as shown in Figure 1.
Additional experiments and inferences
• Reconstruction with Skip-Connections : We tried the Prediction model without skip connection and noticed
a drop in performance.
To check if skip connections were to be credited for the performance boost, we tried
to introduce skip connections to the Reconstruction task.
The problem with adding skip connections to the
Reconstruction task is that, since the input and output are the same, the network quickly learns to copy the
input to the output which makes the reconstruction near perfect for any input, be it normal or anomalous.
To
prevent the network from copying the input, we added salt and pepper noise to the input and the output was
the denoised input.
This allowed us to train models on the Reconstruction task with skip connections and we
straight away saw a performance gain of around 4%.
The results from this model do not outperform the results
from the Prediction task, but it is important to note that the Reconstruction task only accounts for Spatial
anomalies and not Spatio-Temporal or Temporal anomalies.
This clearly shows that the skip connections are
to be credited heavily for how well the Prediction models perform.
Along with skip connections, the model
also gets Temporal as well as Spatial information for the Prediction task.
To add to all these advantages, in the
Prediction task, the input is a batch of 4 consecutive frames and the output is the 5th frame.
Since the output is
not a subset of
the input, this facilitates the use of skip connections without having to worry about the networkcopying the input.
The input to output translation is also more consistent than the salt and pepper noise weused in the "Reconstruction with skip connections" experiments.
We conclude that the skip connections in the Prediction model are to be credited more than the “frameprediction” task itself.
A more ideal method of benchmarking would be to instead use the difference inarchitecture, that is the skip connections vs no-skip connections and with Temporal information vs without
Temporal information rather than just the task itself, i.e Prediction vs Reconstruction.25% SALT and PEPPER noise gave us a best result of 92.23% AUC, a near 4% jump on the Reconstructiontask.
Dataset
[Table]
CUHK Avenue[7]
[Table]
ShanghaiTech[2]
[Table]
[Figure] [Figure]
[Table]
• Memory Distribution plots for better understanding the utilisation of memory :
The model architectureoffers 10 memory items of size 512 each.
During our evaluation of the ShanghaiTech dataset[2], we found thatthe Abnormality Score vector had a large number of NaNs, due to which we could not evaluate the model.
Wefurther investigated the Prediction "Memory" variant of the model in order to ﬁx this bottleneck.
To validateour doubts and concerns, we plotted the distribution of the memory items vs the deeply learned features andobserved that the distribution was lopsided and only some of the items were being utilised.
We saw this behaviour repeat across the three datasets however the worst affected was ShanghaiTech[2] whereonly one memory item was being utilised (See Figure 4a).
Ped2[6]
and CUHK[7] utilised 6 and 9 memoryitems respectively but the distribution was still lopsided as it seems that some memory items are utilised morethan the others as shown in Figure 2.
The abnormality score is a proportional summation of the PSNR and L2 distance between the Deeply learnedfeature and the closest memory item.
In case of ShanghaiTech[2], one memory item was responsible for all ofthe features and due to min-max normalisation used in the abnormality score, the minimum and maximum
L2 would be computed to the exact same value which meant that the normalization process would have todivide by 0, which resulted in NaNs.
We found
that
other
developers
had similar issues as shown in this issue.https:
This problem resulted in Evaluation failing on the ShanghaiTech dataset[2] and returning NaNs in the ﬁnalanomaly score.
A temporary work around was setting the λ value to 1, as a result weighting only thereconstruction or PSNR scores and not using the L2 loss between the query and memory items for theAbnormality Score calculation.
As a solution
we added a new supervision to force a uniform distribution across the memory items.
We wereable to reproduce the reported results on ShanghaiTech[2] without any issues, which previously failed.
The
methodology used and results are further discussed ahead.
• Proposed memory distribution Supervision :
We introduce a new supervision to ensure that all the memory items are utilised uniformly.
We create auniformly distributed correlation map of size 10 × 1024 which is used to supervise the distribution of Queryfeatures across the memory items using Mean Squared Error (MSE) Loss as shown in following the codesnippet.
loss_
[Equation]
)softmax_score_query , softmax_score_memory = s e l f .
g e t _ s c o r e ( keys , query )
Separateness Loss
Compactness Loss
Test
[Table]
Updation λ
AUC
N N
[Table]
N
Y
[Table] [Table]
Y
N
[Table] [Table]
Y
[Table]
N
[Table] [Table]
Y
[Table] [Table]
[Figure]
mem_dist =
t
o r c h .
argmax ( softmax_score_memory , dim = 1)mem_dist = F . one_hot ( mem_dist , num_classes =
10)mem_dist = t
o r c h .
sum ( mem_dist , dim = 0)query_reshape = query .
contiguous ( ) .
view ( b a t c h _ s i
z e *h*w, dims )mem_dist_loss
= loss_mse ( softmax_score_memory , s e l f .
i
d e a l .
f l o a t ( ) )
Figure 3 is only a representation of the Proposed Memory distribution supervision.
This gives a more uniformdistribution of query features across the memory items as shown in Figure 4b.
• t-SNE plots to check clusters :
We ran t-SNE plots on the query features obtained from the encoder for the Ped2[6] and CUHK[7] datasets tovalidate the reported ablation study (Section 4.3 in the original paper).
Further, we also obtain t-SNE plotson our introduced supervision.
We also compare these with the t-SNE plots obtained from the model trainedwithout Separateness loss discussed in Section 4.2.
[Figure]
The t-SNE plots for Ped2[6] are shown in Figure 5.
We observe that the model trained with the settings provided by the author has 6 and 9 clusters for the Ped2[6] and CUHK[7] datasets respectively.
These areequal to the number of memory items utilised by the model.
We also observe that the clusters are spaced out and are thus discriminative with Separateness Loss.
[Figure] [Figure]
[Figure]
• Results on our newly introduced supervision : We trained the "Memory" variant of the Prediction and Re-
construction tasks again using our proposed supervision.
We were able to evaluate the trained models on the
ShanghaiTech dataset[2] with a value of λ lesser than 1, which was previously not possible.
We were able tobeat the scores reported by the authors on both the Prediction and Reconstruction tasks.
The obtained resultsare as shown in Table 8.
We observed that
our obtained scores on the Reconstruction task was more than the Prediction task.
The
fact that the Reconstruction model performs better than the Prediction model here is deeply concerning as
the
Prediction model also has Temporal information which helps recognise Temporal and Spatio
Temporalanomalies while the Reconstruction model does not.
Thus, the Reconstruction model completely misses out on
Temporal anomalies but is still able to outperform the Prediction model.
A possible explanation for the samecould be the fact that the ShanghaiTech dataset[2] is too complex for the memory mechanism to be utilisedas intended.
However, we can say with surety that the "Memory" module does not seem to be helping theperformance of the model on the ShanghaiTech dataset[2].
While forcing the model to use each memory item equally is not an
ideal solution, it helps the model workwithout errors and shows the need for a better and dynamic learning scheme for the memory
items.•
Results with and without Test-Time updation : The authors claim that Test-Time updation of the memory
helps improve the performance of the model.
However, we found contrary results for the Reconstruction withMemory task on ShanghaiTech[2] and Ped2[6].
For the Prediction task, we empirically found that the 1stskip connection is heavily weighted and is highly responsible for the reconstruction quality.
As a result, the
[Table]
intermediate and memory items are not involved in the Prediction task.
We passed zeros, ones and randomtensors for skip 2,3 and the deeply learnt features from the encoder and the output was optically unchangedfor this experiment.
As a result, the memory does not seem to affect the output quality much for complexdatasets like ShanghaiTech[2].
For Reconstruction task, we can clearly see the drop in performance “with
Test-Time-Updation” in ShanghaiTech[2] as there are no skip connections to overcompensate.
The results on
ShanghaiTech[2] are as shown in Table 9.
[Table]
• Synthetic Dataset for Reconstruction vs Prediction : The Reconstruction task, by virtue of the fact that it
takes in only one input and reconstructs one output, is able to only identify only Spatial anomalies.
The
Prediction
task reigns
superior in this
regard
and identiﬁes Spatial
Temporal
and Spatio-Temporal anomalies
To check and visualise the reconstruction quality of Spatial and Temporal anomalies, we created a Syntheticdataset comprising Spatial, Temporal and Spatio-Temporal anomalies as shown below.
We create a Synthetic dataset where the normal instances include circles of radius 10 pixels moving across ata speed of 5 pixels per frame.
Spatial anomalies involve squares moving at the same speed while Temporalanomalies involve circles moving at a speed of 10 pixels per frame.
Spatio-Temporal anomalies are squaresmoving at the faster speed.
We clearly see in Figure 6 that both Reconstruction and Prediction models can reconstruct the normal framesperfectly.
Figure 7 shows both Reconstruction and Prediction models have trouble reconstructing the inputframe, making it possible to classify these frames as anomalous.
Figure 8 shows how the Reconstructionmodel can easily reconstruct the Temporal anomalies whereas the Prediction model reconstructs these frameswith artifacts.
Only the Prediction model’s outputs would allow for classifying these frames as anomalies,and the Reconstruction model’s outputs would classify these as normal frames.
Finally, Figure 9 shows bothmodels having trouble reconstructing the input frames, but the Prediction model reconstructs the image withconsiderably more artifacts making it easier to classify correctly.
We use Mean Squared Error (MSE) as ametric to measure the reconstruction quality for the Synthetic Dataset.
The MSE value has been provided alongwith the reconstruction itself in Figures 6, 7, 8 and 9.
The MSE has been calculated between the reconstructionand the input frame.
We observe that the Prediction model sees a marked difference in reconstruction quality for entities both faster,slower and stationary objects where the normal scenario is an object moving at a constant speed.
[Figure] [Figure]
Discussion
Our results as shown in Section 4.1, show that the memory models do perform better than the ones without for Ped2[6]and CUHK[7] datasets.
However, the "Memory" variants achieve lower AUC scores than the "Non-Memory" variantson the ShanghaiTech dataset[2] (Table 4).
Our Ablation study experiments on Separateness Loss, Compactness Loss and
Test-Time updation do not follow the same trend as the authors, shown in Table 6 and Table 7.
However, our generatedt-SNE plots (Figure 5) show that the Separateness Loss and Compactness Loss help increase the discriminative powerof the Memory items by spacing the Query Features out.
Our results in Table 9 show that Test-Time updation fails oncomplex datasets like ShanghaiTech[2].
As shown in Section 5, the memory distribution is lopsided.
Utilising only one memory item results in similarreconstruction for both anomalous and normal frames.
We provide a temporary solution to ensure that the modelworks smoothly.
Our newly introduced Supervision is described in Section 5 and is backed with results (Table 8) andhistograms to show the uniform distribution of features among the memory items as shown in Figure 4b .
While most of our results corroborate with the claims made by the authors, we believe there are a number of changesthat can be brought about to improve performance.
As shown in Figure 4a, in Section 5, for a complex dataset like
ShanghaiTech[2], only one memory item is being utilised.
This is against the core idea of the paper, that is to reduce therepresentative power of CNNs.
The utilisation of just one memory item means that irrespective of the features being close or far, the same tensorobtained after reading the memory is concatenated to the features obtained from the encoder.
As a result, there isno difference in reconstruction quality for anomalous features as compared to the normal features.
In Figure 10, weillustrate how a number of query features correspond to a particular Memory item and how it is concatenated dependingon the cosine distance from the query features.
[Figure] [Figure]
While our proposed supervision is not ideal, we were able to get this script running and were able to achieve the reportedscores by the author as shown in Table 8 without any issues described in Section 5.
Further, for ShanghaiTech[2], we found that the Reconstruction models with our new Supervision seem to work betterthan the Prediction with Memory models.
Given the fact that the Prediction task has Temporal information as well, itshould be outperforming the Reconstruction task due to the correct classiﬁcation of Temporal anomalies.
Thus, the
Reconstruction task outperforming the Prediction task shows that the model does not work as intended.
Finally, we show the difference between the reconstruction of Spatial, Temporal and Spatio-Temporal anomalies fromboth the Prediction and Reconstruction models as depicted using the Synthetic Dataset discussed in Section 5 and asshown in Figures 6, 7, 8 and 9.
6.1 What was easy
• Training and evaluation of the Prediction task "Memory" variant is straightforward with the codebase that theauthors provide.•
The codebase was clear enough for us to easily reuse it for implementing the missing variants discussed in the
paper, namely the Prediction "Non Memory" variant and both the "Memory" and "Non Memory" variants ofthe Reconstruction task.
• The ideas presented in the paper were clear and easy to understand.• All benchmarking datasets were readily available in the format required by the codebase, with the exception of
the ShanghaiTech dataset[2].
[Figure]
6.2 What was difﬁcult
• Training on the ShanghaiTech dataset[2] took 25 hrs, and once trained, we observed the "Memory" variants had
clear issues (discussed in Section 5) which made it difﬁcult to hit the reported scores.
We further investigated
the behaviour of the memory module and had to introduce additional supervision to help with the problems(Section 5).• We found that the hyperparameters speciﬁed by the author were not suitable for a number of tasks and thusneeded hyperparameter optimization to hit the reported scores.
• There was also a want of information on which parts of the pipeline were to be credited for the results.
Theheavy emphasis on benchmarking on the Reconstruction vs Prediction tasks misdirects the reader from why
certain inclusions and modiﬁcations in the network architecture is the reason for reported state of the artperformance (discussed in Section 5).
6.3 Communication with original authors
The codebase has scripts only for the Prediction task "Memory" variant of the benchmarks.
We reused the majority ofthe codebase to create the other variants so as to have minimum variation from what the authors intended.
We thenvalidated our modiﬁcations with the authors through e-mail communication and will have all correspondence availablein the code repository.
We also communicated the discrepancies we saw in parts of the results and the authors told usthey would update their repository with the missing scripts soon to help match the reported scores.
Reproducibility Recommendations
We have the following recommendations:
• We believe that the ideal value of the hyperparameter λ is 0.7-0.8 for most datasets as this seems to yield thebest results for the "Memory" variant of the Prediction task as compared to 0.6 suggested by the authors.•
There also appears to be a problem with PyTorch versioning where the best results are obtained with PyTorch1.1.0.
This is an issue that other developers noticed as well (
https://github.com/cvlab-yonsei/MNAD/issues/1).
The reason for this behaviour is unclear.