Reproducibility Report: Towards Visually Explaining Variational Autoencoders
Anonymous Author(s)
AfﬁliationAddress email
Reproducibility Summary
Scope of Reproducibility
Using a modiﬁcation of Grad-CAM, attention maps can be created for Variational Autoencoders, resulting in explainable generations.
Using these attention maps, state-of-the-art anomaly detection and latent space disentanglement is reached.
Methodology
We started the challenge using the author’s code, but this only covered one experiment of the paper, namely anomalydetection for the MNIST dataset.
Therefore we added models, training and testing code for all other anomaly detectionexperiments, those on the UCSD-Ped1 and MVTec dataset, and also for the latent space disentanglement experiments.
Some of these implementations were based on other existing repositories, whereas some were implemented completelyby ourselves.
We worked for four weeks full-time on reproducing the results with two GPUs available to use.
Results
We were able to successfully generate attention maps using the method described by Liu et al.
and could apply them toanomaly detection as well.
For the MNIST experiments, this led to results that were similar to the paper.
However, forthe UCSD-Ped1 experiments, the author’s explainable VAE model actually performed worse than our own baseline.
Moreover, we were not able to support the author’s claim that they achieve state-of-the-art on the MVTec dataset.
Finally, for the latent space disentanglement, our found results were not as good as claimed by Liu et al., but they stillout-performed the set baseline, as was also claimed by the authors.
What was easy
Running the initial implementation of the authors, since their code was relatively straightforward.
We were able togenerate attention maps and anomaly detections for the MNIST dataset using a Variational Autoencoder without toomany difﬁculties.
What was difﬁcult
The code of the authors covered only a small portion of the paper and extending this to the whole paper was verydifﬁcult, as the paper was not often
Communication with original authors
Introduction
Recently there has been an increasing interest in model explainability within artiﬁcial intelligence research.
One branchof study concerns itself with generating visual explanations, or visual attention.
These visual explanations highlight theareas in images or other visual data that the model deems most important in making correct predictions, thus essentiallyexplaining how the model reasons.
This makes the inner workings of AI models more transparent.
Visual attentiontechniques have so far mainly been applied to Convolutional Neural Networks (CNN), to visualize the regions of animage that are most important for making a classiﬁcation.
However, visual explanations have not yet been applied tomany generative models.
Liu et al.
attempt to bridge this gap with their paper "Towards Visually Explaining Variational Autoencoders".
Theydescribe a technique for generating visual attention for Variational Autoencoders (VAE)
[Kingma and Welling, 2014],a type of generative model.
With their visual attention method for VAEs, Liu et al.
take a step towards making AImodels more transparent.
We ﬁnd this to be important, because as AI grows more prominent, so does the desire for themodels to be explainable.
If AI is to take a central spot in our lives and society, then the decisions it makes need to betransparent.
This makes the models safer, less subjective to manipulation and makes people more likely to trust thedecisions made by the model.
As generative models have mostly been black-box type up until now, the claims made inthe paper by Liu et al.
are very promising for the ﬁeld of transparent AI.
We will therefore attempt to reproduce thispaper, giving more insight in the validity of the made claims.
The three major claims are made in the paper are as follows:
• It is possible to generate visual attention maps conditioned on the latent space of a VAE using a method basedon Grad-CAM.
• Using these attention maps, it becomes possible to achieve state-of-the-art performance for an anomalylocalization task on the MVTec-AD dataset.•
The attention maps can also be incorporated into a new learning objective called attention disentanglementloss, which improves upon the state-of-the-art in latent space disentanglement
for VAEs.
In this report, we will attempt to validate all three claims by reproducing the described experiments.
The choice toreproduce all three is made because the ﬁrst claim contains the core implementation and the second and third claimstate-of-the-art results.
Thus by evaluating all three claims we aim to ascertain the strengths and weaknesses of thepaper and underlying code.
Our results show that the ﬁrst claim can be reproduced, as we got results that were similarto the paper.
As for the two other claims, we were able to implement all experiments, but were not able to reach thesame state-of-the-art results as the authors.
This was the case for both the MVTec-AD dataset and the latent spacedisentanglement experiments, therefore leading us to not being able to fully support the second and third claim.
Methodology
2.1 Concepts and Models
The below subsections will further elaborate on the theory behind each of the three claims and the model implementationswe used to run the corresponding experiments.
2.1.1 VAE Attention
The ﬁrst claim the paper makes is that they can generate and visualize attention for VAEs, which was previously onlyapplicable to CNNs.
The method they describe for this is based on a technique called Grad-CAM, which stands for
Gradient-weighted Class Activation Mapping [Selvaraju et al., 2017].
Grad-CAM uses gradients for visualizing theregions in an image that are most important for classiﬁcation.
It does this by computing gradients backpropagated fromthe classiﬁer unit to a target convolutional layer, thus generating a feature map M .
In Liu et al., this method is extendedto work on VAEs.
The key difference is that, for the method proposed by Liu et al., the gradients are not backpropagatedfrom a CNN’s classiﬁcation unit, but from a latent vector z of the VAE.
For each z i in z , the corresponding attentionmap M i is computed by backpropagating gradients to the target layer feature maps A ∈ R n × h × w .
Details of thismethod can be found in equations 2 and 3 in the original paper by Liu et al..
Via the website of Papers with Code 1 , we found a GitHub repository from one of the authors that corresponded with thepaper 2 .
However, the majority of the required code to reproduce all three claims was absent.
For example, there was no code available for the particular attention generation method described above.
Therefore, we wrote the implementationourselves.
This allowed us to access the individual feature maps M i for each z i , which were also later needed for theimplementation of the AD loss function.
2.1.2 Anomaly Detection
The second claim the authors make is that they can reach state-of-the-art performance for anomaly detection using their
VAE attention method.
To implement the attention generating mechanism for anomaly detection, a slightly differentmethod than the one described above was used.
Instead of computing individual attention maps M i for each element inthe latent vector z , they take the inferred mean vector of the latent space and sum it to compute the score s , which isthen backpropagated to the target layer.
The GitHub repository of the paper contained an implementation of the aforementioned method.
This code however didnot include any documentation or comments and there were also blocks of code present that were never called.
Weremoved these parts from the code, slightly restructured it to be more efﬁcient and added documentation to each ofthe implemented functions.
Another notable feature in the authors their code, was the absence of a ReLU function forthe attention map, as was originally mentioned in the paper.
In its place was an absolute operation that serves as analternative to the ReLU to eliminate negative values for the attention maps.
However, we think the ReLU only makessense for the original Grad-CAM paper Selvaraju et al.
[2017], and that only the magnitude of the attention and not thesign is important for the VAE.
After this inspection, we also moved the absolute operation inside the summation of theattention maps, which improved our results.
Two different models were implemented for the anomaly detection task.
The ﬁrst model the authors implement, called
Vanilla expVAE, applies the attention generation technique described above to a relatively simple one-class VAE.
Theexact details of this model’s architecture were not mentioned in the paper.
However, via the FACT-AI course where thisreport is part of, we received a supplementary document originally created by the authors which gave more details intothe exact architectures and some hyperparameters.
We used the same architecture for the Vanilla expVAE as describedin the document, which can be found in Appendix A.
The second model implemented by the paper applies the VAEattention mechanism to a VAE model with Resnet18 CNN architecture as an encoder.
A precise implementation of the
Resnet18 expVAE as described in the supplemented materials turned out to be unattainable since the authors mention a512 × 8 × 8 output size at two layers before the output, however the Resnet18 output has a size of 512 × 16 × 16 atthis point.
To this end, we instead used the original Resnet18 implementation as the encoder but kept the decoder thesame as the authors described.
Exact details of this architecture can also be found in appendix A.
2.1.3 Attention Disentanglement
The third claim made by the authors is that they can reach state-of-the-art performance in latent space disentanglementby incorporating their attention maps in an already existing disentanglement model called FactorVAE
[Kim and Mnih,2018].
The original FactorVAE improved upon the β -VAE [Higgins et al., 2016] by overcoming the trade-off betweendisentanglement and reconstruction quality inherent to the β -VAE, reaching state-of-the-art performance on latent spacedisentanglement whilst not hurting the reconstructions.
The assumption is that, if a VAE is completely disentangled,one latent dimension will correspond to one latent factor in the data and when a latent traversal is done over this latentdimension, the transformation will be similar to traversing the corresponding latent factor.
Liu et al.
claim that they can improve on this by adding an additional loss module based on the generated attentionmaps, which they call Attention Disentanglement (AD) loss.
The AD loss uses two attention maps computed fromdifferent latent dimensions and increases based on the overlap between both maps.
This corresponds to equation 5 in[Liu et al.].
Note that the A for this equation does not represent the same thing as the one in equation 2 of their paper.
As Liu et al.
use the original FactorVAE architecture, our code builds upon an open-source implementation by
WonKwang Lee 3 .
This implementation missed two parts which are required for reproducing the results: (1) thedisentanglement metric proposed by Kim and Mnih was not implemented and (2) the AD-loss and thus also a versionof Grad-CAM for generating attention maps per latent dimension needed to be implemented.
For (1), the disentanglement metric is based on a majority vote classiﬁer.
A vote corresponds to which factor, a variableattribute of the dataset, is aligned with which latent dimension of the VAE, see [Kim and Mnih, 2018] for a moredetailed explanation.
An almost complete implementation of the metric was found online 4 .
It was slightly modiﬁed,mainly in terms of efﬁciency, and ported to PyTorch from TensorFlow.
Note that we, following the example of this implementation, only use the mean outputs of the encoder to retrieve the variance instead of reparameterizing theencoder outputs, which allowed for the baseline FactorVAE to get similar results as achieved by Liu et al..
This isbecause using the means will create a more stable representation and thus also a more stable/higher score, whilst stillbeing faithful to the output latents, as these will be centred around the means.
For (2), the modiﬁed implementation of Grad-CAM by Liu et al.
was used to create attention maps for two latentdimensions.
As mentioned in section 2.1.1, for this purpose too, alterations had to be made to the original code to allowfor the creation of attention maps of individual latent dimensions.
The ReLU activation (from equation 2 of [Liu et al.])was still used here, as the focus here is on positive attention (the loss should not be negative).
In the paper it is neverexactly stated how the two latent dimensions for the attention maps were chosen, we chose those randomly at eachiteration during training.
Retrieving the attention map for each latent dimension to allow for a different implementationseemed unwise: calculating the loss for two maps already resulted in an efﬁciency drop of around 35 per cent (from70 iterations per second to 45).
This drop is likely due to the creation of the attention maps being computationallyexpensive, as it requires an extra backward step through the encoder network per latent dimension.
2.2 Datasets and hyperparameters
Using these models, Liu et al.
ran experiments on multiple datasets: anomaly detection experiments were run for the
MNIST, UCSD-Ped1 and MVTec datasets and latent space disentanglement on the dSprites dataset.
This section givesan overview of these datasets, as well as the hyperparameters we used for reproducing the experiments during trainingand testing.
2.2.1 MNIST
The MNIST dataset is a dataset containing 60,000 training and 10,000 testing samples of black-and-white images ofhandwritten digits, ranging from zero to nine [LeCun et al., 2010].
A dataloader script, which downloads the requireddata and divides it into training and testing sets, was present in the author’s GitHub repository.
Therefore no furtheradditions were required to process this dataset.
In the supplementary document provided by the authors, it was mentioned that the images were resized to 28 × 28pixels.
Furthermore, the learning rate was set to 0.001, the latent size to 32 and the batch size to 128.
As for the numberof epochs, no information was provided in the paper or the supplementary document.
Therefore, we trained the networkfor a number of different epochs and determined from the resulting loss graphs that performance did not improve after100 epochs anymore, so we used that as our number of training epochs.
Moreover, the code by the author’s used the
Adam optimizer [Kingma and Ba, 2014] and a Binary Cross-Entropy (BCE) loss module, which made us decide to alsouse this for our experiments.
2.2.2 UCSD-Ped
The USCD-Ped1 Anomaly Detection Dataset [Chan and Vasconcelos, 2008] is an open-source dataset containing34 training and 36 testing samples of videos of a pedestrian walkway.
Each video consists of 200 frames.
In thetraining videos, pedestrians can be seen walking towards and away from the stationary camera overlooking the walkway.
However, the testing videos also contain some anomalies such as bikers, skaters, small carts or pedestrians walkingoff the walkway.
These anomalies are indicated using a mask.
To train the pedestrian dataset all images were, like inthe original paper, resized to 100 × 100 pixels and the latent size was set to 32.
After consultancy with [Liu et al.] weused the VAE architecture as provided by the in appendix A ﬁgure 5 and a batch size of 32.
We decided to train thisarchitecture for 512 epochs and used BCE loss and the Adam optimizer.
To evaluate this dataset, the Area Under the Receiver Operating Characteristic Curve (AUROC) was computed.
In order to do so, the scikit-learn[Pedregosa et al. ,2011] library was used.
In addition, we recreated the baseline provided by the paper by computing the differencesbetween the input images and their reconstructions.
At last, we tried a new baseline by computing the differencebetween the average VAE reconstruction and the input image.
2.2.3 MVTec AD
MVTec AD
[Bergmann et al., 2019] is a dataset that contains over 3929 training and 1725 testing samples of highresolution images, divided into ﬁfteen different object and texture categories.
Speciﬁcally, these classes include bottles,cables, capsules, carpets, grids, hazelnuts, leather, pills, screws, tiles, toothbrushes, transistors, wood and zippers.
Foreach of these classes, the training set contains anomaly-free image samples and the test set contains anomaly-free andanomalous images.
Each class contains a variety of defects in the anomalous images, that are common for that object.
On average 5 different types of defects per object.
For all the anomaly images a binary mask image is present whichmarks the defected area of the object.
For the MVTec AD experiments, we again based most of our hyperparameters on the supplementary document.
First ofall, all images were resized to 256 × 256 pixels.
Moreover, we applied data augmentation during training with randomrotations between of0.5.
We set the learning rate 0.0001 and batch size to 8 for training and used a latent size of 32.
Additional inputprocessing was implemented by normalizing the input images by the mean and standard deviation of the entire dataset.
This normalization technique is known to be a common approach when working with image data and is applied in manyscientiﬁc papers to improve the performance of the network.
Finally, the Adam optimizer was used and both Mean
Squared Error (MSE) loss and BCE loss were compared.
As BCE gave a slightly higher performance, we decided touse this as the loss module.
2.2.4 AD-FactorVAE
The dSprites dataset [Matthey et al., 2017] is a dataset developed by Deepmind to enable ﬁnding correlations
betweenlatent factors of the dataset and the dimensions of the latent dimensions of the VAE.
The dataset is in a black and whitecolour space and consists of six latent factors: colour, shape (square, ellipse, heart), scale, orientation, x position and yposition.
Each factor has multiple classes (except for colour, which can only be white) which can be changed withoutinﬂuencing the other factors.
Liu et al.
mention they build upon the original FactorVAE, of which the hyperparameters and architectures can be foundin Appendix A of Kim and Mnih [2018].
We use the same hyperparameters and model architectures as Kim and Mnihas they are more efﬁcient, but with latent dimensionality of 32, as used by Liu et al.
(see ﬁgure 7), which allowed forthe baseline FactorVAE to reach similar results as theirs.
For adding the AD loss, a λ term is mentioned, but it is neverspeciﬁed which value the authors use for this hyperparameter, thus we run experiments with λ set at 1, 20, 40 and 80.
In addition, the AD loss can be calculated for different convolutional layers in the FactorVAE, but the authors neverexplicitly mention which one is used.
Therefor, we perform all experiments using the ﬁrst convolutional layer and oneextra experiment using the third convolutional layer with λ = [-
-]1 .
2.3 Setup and Computational Requirements
The code for our experiments is made publicly available on GitHub 5 , which includes python ﬁles and a notebook ﬁleto run all the experiments.
The major part of the code was run using one GPU of the Lisa Cluster from SURFSARA(GeForce 1080Ti) 6 , however the last part of our experiments concerning the attention disentanglement were run on ourown GeForce 1080 GPU.
Using this hardware, one of the major computational costs arose from training the VAE on the MVTec AD dataset.
Onthe Lisa Cluster, this took us around 45 minutes each, so for ﬁve classes, this meant up to four hours for 200 iterations.
Furthermore, training the USCD-Ped1 dataset could take up to 4 hours for 512 iterations and training the FactorVAEtook around 1 hour and 15 minutes per run (300000 iterations).
Additionally, when adding the attention disentanglementloss, it took around 1 hour and 50 minutes per run, without the disentanglement metric being used.
Adding this metricresulted in approximately twenty extra minutes per run.
Results
In order to validate the three claims made by Liu et al.
, we reproduced the experiments described in the paper.
Intheir paper, after Liu et al.
introduce their VAE attention mechanism, they apply the technique to anomaly detection,therefore showing that they are in fact able to produce VAE attention maps.
For this reason, we have followed the sameformat and reproduced all of their anomaly detection experiments, thereby evaluating the validity of both claims 1 and 2.
We then proceeded to evaluate the validity of their third claim, by also reproducing their latent space disentanglementexperiments.
The results of all of these experiments are discussed below.
3.1.1 Evaluation on MNIST Dataset
In their second claim, Liu et al.
state that their attention generation technique can be used for anomaly detection and iseven able to achieve state-of-the-art on the MVTec AD dataset.
The ﬁrst anomaly detection experiment they describe is
[Table]
[Table]
[Table]
[Table]
a qualitative evaluation on the MNIST dataset, in which they train their model on the digit "1" and test on a variety ofother digits. Below ﬁgure shows a selection of results for our reproduction of the experiment.
As can be seen in ﬁgure 1, there is some variation in the performance of the model.
Some of the anomaly detectionmaps correctly highlight all parts of the digit that are anomalous to a "1".
These results look very similar to the results
Liu et al.
shown in their paper in Figure 4.
However, there are also some results where the whole digit, or no areas inthe digit at all are highlighted.
3.1.2 Evaluation on USCD-Ped1 Dataset
Figure 2 shows qualitative anomaly detection results for different methods (c, d, e) compared to the input image andground truth mask (a, b).
Qualitative results are shown in table 1a, where our reproduction and best model are comparedto the results of Liu et al..
Our results are substantially lower. Even though we only care about the highest performinglayer, it is visible that the results are less consistent per layer. Even the obtained baseline score is lower.
After addingbatch normalization, a learning rate scheduler and using convolutional layers of depth [192, 144, 96] we achieve aminor increase in the obtained AUROC score as shown in the column "Our best".
Note that our new suggested baselinescore is as high as the best layer of Liu et al..
[Figure]
3.1.3 Evaluation on MVTec AD
To evaluate the second claim, the author’s results were reproduced for ﬁve different object classes: two of their besttexture classes (leather and tile), two of their best object classes (hazelnut and metal nut) and their worst-performingclass (capsule).
An example of a generated attention map for the hazelnut class is shown in Appendix B. Table 1b showsthe AUROC score and the best Intersection over Union (IoU).
In addition, to make the results in , for transparency, weincluded the target layer that was used to generate the best score for each class.
The AUROC scores show comparableresults for hazelnut and leather, lacking results for metal nut, but much higher AUROC score for the capsule.
3.2 Latent Space Disentanglement
[Figure]
[Figure]
[Figure] [Figure] [Figure] For the results corresponding to the reproduction of the third claim about state-of-the-art disentanglement, see ﬁgure3.
The left ﬁgure shows the quantitative results, this is the reproduction of ﬁgure 8 from Liu et al..
The ﬁgure on theright shows the reproduction of the qualitative results of the attention maps generated for both a FactorVAE and an
AD-FactorVAE model
, it corresponds to ﬁgure 9 from Liu et al..
3.3 Discussion of Results
First of all, for the MNIST experiments, we were able to replicate the results shown in table 4.
However, there existedsome variation in the quality of the generated anomaly detection maps that was not present in the results of the authors.
Image samples resembling the ones used by the authors in table 4 generally show good performance, but for many otherimage samples, the model did not perform as well.
Because the authors mostly show a homogeneous collection of digitswith little variation in table 4, it is difﬁcult to determine whether our results are lacking or if the authors just decidedto show only the images with the best performance.
Our best performing results do, however, closely resemble theirresults, which substantiates this claim.
Secondly, for the UCSD Ped1 dataset, Liu et al.
score a high AUROC score for all layers, where we only score high forone layer.
Moreover, their highest score of 0.92 is substantially higher than our score of 0.802.
After hyperparametersearch, we were able to improve our best results to 0.858 but were unable to match their result.
Several explanations arepossible: On the one hand, their training setup can differ from ours.
They might use another data augmentation pipelineor it is possible that other training hyperparameters are used since it is unclear which optimizer and loss function theyuse and for how many epochs they train.
On the other hand, their calculation of the AUROC score could differ fromours.
This can be substantiated by the fact that when we tried to reproduce their baseline of 0.86 by taking the differencebetween input and the VAE its reconstruction, we obtained a substantially lower baseline of 0.701.
We were not able toﬁnd out how Liu et al.
computed the AUROC score, however, we used the scikit-learn implementation, which uses astandard method.
At last, we suggest using a new baseline for evaluating this dataset by taking the difference betweenthe input and the average VAE reconstruction of the latent space, which is essentially an image of an empty pedestrianwalkway.
Taking this difference resulted in an AUROC score of 0.921, which demonstrates that adding the authorsexplainable VAE model does not improve anomaly detection for this dataset.
For anomaly detection on the MvTec dataset, most of the reproduced experiments achieved lower scores than the paper.
It is noteworthy, however, that the classes leather, capsule and hazelnut all showed decent AUROC scores, similar tothe paper or in the case of the capsule even higher results.
From these three classes, the signiﬁcance of the anomalydetection as claim 2 described is at least established.
Although the divergent results of the capsule and metal nutclass indicate our implementation is different from the one by Liu et al..
Potentially the slight difference in networkarchitecture that resulted from their infeasible network description, in combination with longer training time, couldincrease the performance to match the state-of-the-art results.
Lastly, the results from the capsule class indicate thattheir implementation is capable of improvements for at least some classes.
Finally, for the latent space disentanglement experiments, the results show that, just as with the results of Liu et al., ourreproduction of the AD-FactorVAE with lambda =
40 outperforms the standard AD-FactorVAE for the disentanglementmetric, without increasing the reconstruction loss.
However, it does not do so by as great a margin as claimed in theoriginal paper: we get an improvement of around 0.05, whilst they state an improvement of around 0.09.
This could bedue to us using a slightly different model, but we found this to be unlikely as we ﬁrst used a model more similar totheirs but later on switched to the current model.
Both setups gave similar results, but ours was slightly faster.
Liu
et al.also show qualitative results to indicate that the latent space disentanglement is visibly better in the resulting attentionmaps.
We found this to be untrue, as from ﬁgure 3b it is not clearly visible that the two latent dimensions are visiblymore dissimilar for the AD-FactorVAE than for the FactorVAE.
Qualitative analysis is, however, not representative asonly a very small part of the dataset was checked, but this is also the case for Liu et al.  Discussion
4.1 Reproducibility
For this reproducibility challenge, there were some difﬁculties in reproducing the paper by Liu et al.
although therewere also a few parts that proceeded more smoothly.
Reproducing the MNIST experiments, for example, was relativelyeasy, as the was code available on GitHub repository of the authors.
In order to get results that were more consistentwith the ones shown in their paper, only minimal hyperparameter tweaking was required, but in general, the experimentcould be reproduced by simply running the provided code.
Furthermore, implementing the FactorVAE with the dSpritesdataset was also not too difﬁcult, as the found implementation was very well structured and documented.
The most difﬁcult part of reproducing the results of the paper arose from the brief and often partial descriptions of theauthor’s implementations.
An important example is the ReLU activation that appeared in the paper but was missingin the code.
We ran various preliminary experiments comparing the performance of ReLU and absolute, where theabsolute operation showed much better performance.
For this reason, we suspect that the ReLU in the paper might havebeen a mistake and the absolute operation is the appropriate function they use.
In addition, even though a supplementarydocument was given, training speciﬁcs like the number of epochs or the use of a learning rate scheduler were notmentioned anywhere.
Another important detail that remained unclear was which target layers were used for creatingthe attention maps for different models.
At certain points in the paper, for example the UCSD-Ped1 experiments, theauthors clearly state which layers were used, but for all other experiments this detail remained unclear or was nevermentioned at all.
Furthermore, while the supplementary document gave some insight into the scaling and rotatingof input images, no mention was given whether the input was normalized.
Although it is often considered commonpractice to normalize input images before training, the lack of mention in the paper makes it difﬁcult to assume anything.
In addition, implementing the disentanglement metric proposed by Kim and Mnih [2018] was difﬁcult, as it was bothhard to interpret the implementation stated in the paper and the results were not similar to those for the original modelat ﬁrst.
4.2 Conclusion
From the results, we can conclude that the ﬁrst claim Liu et al.
make, that they can generate visual attention maps for
VAEs, is true.
We were successfully able to generate these attention maps and could apply them to anomaly detectionas
well.
For the MNIST experiments, this too led to results that were similar to the paper.
However, for the UCSD-Ped1experiments, the explainable VAE model of the authors actually performed worse than our own baseline.
Moreover, wewere not able to support the authors their claim that they achieve state-of-the-art on the MVTec dataset.
It is worthnoting that we may have arrived at these different results due to the unclarity of the authors their implementation details.
Therefore, we conclude that their second claim partially holds, as our experiments show that it is possible to use theirattention maps for anomaly detection, but we cannot always match their performance.
Finally, the third claim, thatattention can help with latent space disentanglement, also holds.
Although our found results are not as good as claimedby Liu et al.
, they are better than those for the set baseline that does not use the attention maps.
B Qualitative Results on MVTec Dataset