Reproducing Visual Explanations of Variational Autoencoders
Anonymous Author(s)
AfﬁliationAddress email
Reproducibility Summary Scope of Reproducibility
In this work we perform a replication study of the paper “Towards Visually Explaining Variational Autoencoders".
This paper claims to have found a method to provide visual explanations of Variational Autoencoders (VAEs). Thepaper’s primary claim is that their proposed method can generate gradient-based attention maps from the latent space ofa VAEs. This is visually demonstrated on the MNIST dataset.
Moreover, these attention maps are claimed to be usefulfor anomaly detection, which is demonstrated on the UCSD Ped1 and MVTec-AD datasets.
Finally, this method isintegrated into a loss function to obtain the attention disentanglement loss. This loss is shown to improve latent spacedisentanglement when integrated into a FactorVAE model, which is demonstrated on the dSprites dataset.
This paperaims to reproduce all of the claims stated above.
Methodology
In order to produce attention maps which can localize anomalies for the MNIST dataset the code from the repository ofthe authors could be reused.
Additional models were implemented for the UCSD Ped1 and MVTec-AD datasets basedon supplementary material provided for the original paper.
To reproduce the latent space disentanglement results, a AD-FactorVAE model – which combines the attention disentanglement loss and FactorVAE model – was implementedbased on the original paper, the original paper’s supplement, the FactorVAE paper and external code sources.
Results
The attention maps for the MNNIST dataset were successfully replicated.
Nevertheless, we failed to reproduce theresults for the UCSD Ped1 and MVTec-AD dataset.
Furthermore, we were unable to reproduce the results for the
AD-FactorVAE.
Potential explanations for this could be the incorrect aggregation and/or weighting of the attentiondisentanglement loss, not training the models for enough epochs, or not using the correct method to produce theattention maps.
What was easy
Reproducing the attention maps and anomaly localization results for the MNIST dataset was relatively easy
What
MVTec
AD datasets was reproducing the disentanglement proved
Communication with
Introduction
This paper describes our attempt to reproduce the paper “Towards Visually Explaining Variational Autoencoders" by Liuet al [1].
Their paper introduces a method to provide visual explanations for variational autoencoders (VAEs) [2], whichis inspired by the recent development of gradient-based attention maps for Convolutional Neural Networks (CNNs) thataid in visualizing and understanding these models [3].
The attention maps highlight areas of an image that are importantfor the classiﬁcation (in the CNN case) or reconstruction (in the VAE case) of the respective image.
It is importantto search for these explanations since deep learning models tend to be difﬁcult to interpret due to their complicatedstructure [4].
This complexity allows for superior performance but also prevents creators from understanding the reasonsbehind their models’ outputs.
If the models’ underlying assumptions cannot be made explicit, there is a possibilitythat the predictions of the models rely on false assumptions.
This could cause failure in later stages of deployment.
Furthermore, adequate explanations for predictions can improve consumers’ trust in the abilities and fairness of a model[5].
Additionally, we could learn new properties of the data with these explanations, and improve our own decisionmaking abilities [6].
Deep models have been especially successful in the ﬁeld of computer vision and have been widelyadopted in real-life tasks which makes the demand for proper understanding of their performance all the more urgent[7, 8, 9].
The performance of techniques to produce explanations can generally only be determined by qualitativeanalysis, which is highly subjective and selective since not all train or test instances can be evaluated.
Thus, we decidedto reproduce a paper that not only promises to provide a method that can generate visual explanations for VAEs but alsostates that these attention maps produce state-of-the-art anomaly localization results and can be used to improve latentspace disentanglement, which can both be quantitatively measured.
Scope of reproducibility
The original paper proposes a gradient-based technique to produce visual attention maps for VAEs [1].
It claimsthat these attention maps are not only useful as explanations of VAE predictions but can also be used to performanomaly localization, which is demonstrated on the MNIST, UCSD Ped1 and MVTec-AD datasets, and latent spacedisentanglement, which is demonstrated on the dSprites dataset.
In this paper we try to reproduce the attention maps andthe results for these two practical use-cases.
In order to determine whether the paper can be successfully reproduced,the following claims were constructed that capture the most important statements and which can be either conﬁrmed byour research (in which case the reproduction was successful) or rejected (in which case our reproduction has failed)
• The method introduced in [1] is able to generate gradient-based attention maps from the latent space learnedby a VAE model.
These attention maps denote characteristics of a set of images that are important to the modeland intuitively make sense, which is veriﬁed by qualitative analysis on the MNIST dataset.
• The attention maps are useful for anomaly localization, which is demonstrated on the MNIST data set, forwhich only qualitative analysis is provided.
Additionally, it is shown on the UCSD Ped1 dataset where itobtains a better AUROC score than the Vanilla-VAE method that simply computes the difference between the original and reconstructed image.
Finally, the performance for anomaly localization is state-of-the-art, which is shown on the MVTec-AD dataset where it obtains AUROC and IOU scores that are better or similar to thoseobtained by the models compared in the MVTec-AD paper.
• The attention maps can be used to improve latent space disentanglement by combining them into a loss function that can be added to the VAE loss during training.
This is proven by incorporating the loss with the FactorVAE which results in a higher disentanglement score for the dSprites data set than the baseline FactorVAE model, while keeping a similar reconstruction loss.
Methodology
The following section discusses the implementation of the necessary functions to reproduce the claims stated in theprevious section.
3.1 Attention maps
The ﬁrst claim of the original paper is that the proposed gradient-based attention maps can explain the inner workingsof a VAE model.
Attention maps can be generated for each latent dimension separately.
First, the image is forwardedthrough the model to obtain its latent representation.
Subsequently, the activation of each of the dimensions of thisrepresentation is backpropagated to one of the last convolutional layers.
The resulting gradients are aggregated perchannel of the feature maps of the convolutional layer to obtain the weight given to each channel of the feature map.
This weighted sum of channels of the respective feature maps can then be used to retrieve the attention map of a singlelatent dimension.
The method to obtain attention maps for anomaly detection is also closely related to this technique.
The authors showcase how these attention maps explain the model by training models on a single class of images fromthe MNIST data set [10] and then showing attention maps generated by the model for images of other classes.
Theseattention maps highlight areas which are not characteristic of the originally learned class.
The training of the VAE model was done by following the instructions in the repository with the code made available bythe authors which included the main training procedure and the experimental VAE architecture.
Small adjustmentswere implemented to make the core run on CPU as well.
Acquiring the dataset, and transforming this to the right datastructure was already done in the provided code as well.
In addition to training the models ourselves, we had access tothe pre-trained models which could be downloaded via a link in the authors’ GitHub repository.
The VAE structure –two convolutional layers followed by a fully connected layer in the encoder and a mirrored structure in the decoder –and the hyperparameters – image_size =
| batch_size = 128 | latent_space = 32 | learning_rate 0 .
001 – were describedin the supplementary paper, which is available online 1 , as well as provided in the code base.
The amount of epochs wasunclear, but we trained the model for 100 epochs since this was the default provided in the code.
The provided code already contained an implementation for attention map generation, which we could run afterremoving an erroneous parameter (gradient=one_hot) in the backpropagation function 2 .
However, when consulting thepaper it seemed that some parts of the code did not match the original method.
By following the paper more closely wediscovered that Liu et al.
distinguish between two different methods, (1) the ﬁrst is proposed to visualise explanationsfor VAEs by calculating the attention maps per latent dimension and aggregating these, and (2) the second is proposedspeciﬁcally for anomaly detection where attention maps are generated from the sum of the inferred mean vector.
In theprovided code base only the (2) second method is implemented and so, in addition, we implemented the (1) ﬁrst methodourselves.
Besides implementing both methods, we also chose to test various functions in place of the ReLU in function (2) ofthe paper.
This was done as the provided code implemented the absolute value instead of the ReLU, which seemed togive similar results and we were curious to see whether the sigmoid could be used as well, since this is also a commonfunction used to scale values between 0 and 1 .
To conclude, we tested both the pre-trained models and the modelstrained from scratch with the two different methods and the three different functions.
3.2 Anomaly detection
The second claim by Liu et al.
is that the attention maps are not only useful for anomaly localization but can producestate-of-the-art results.
They show this by conducting qualitative and quantitative experiments on two different datasetsfor anomaly detection: the UCSD Ped1 dataset [11] and the MVTec-AD dataset [12].
3.2.1 UCSD Ped1 set up
The UCSD Ped1 dataset contains video frames of a pedestrian walkway where all non-pedestrains are considered to beanomalies.
The implementation for the anomaly detection on the UCSD Ped1 dataset was not provided in the authors’code.
For the qualitative research we could use the same approach as in the previous section, namely training a VAEmodel on the UCSD Ped1 training set – which does not contain any anomalies – and afterwards generate attention mapsof the test set – which does contain anomalies – for each of the convolutional layers of the VAE using the (2) secondmethod.
The changes we had to make for these experiments consisted of implementing a different architecture for the
VAE model containing three convolutional layers instead of two and adding functions to properly read in the UCSD
Ped1 training and test set, which included resizing the images to 100 by 100 pixels.
Table 2 in the addendum papershows the architecture of the VAE model that was used for these experiments and speciﬁes a learning rate of 0 . 0001with a batch size of 32 frames for training.
The amount of epochs was unclear, but we ended up training the model for1000 epochs, which took around 4 hours on one NVIDIA 1080 GPU.
For replicating the quantitative results, we implemented functions to evaluate the produced attention maps.
The UCSD Ped1 test set contains both video frames containing anomalies as well as ground truth masks denoting the location of apossible anomaly.
Given the anomaly attention maps, binary anomaly localization maps were generated using a varietyof thresholds whose respective overlap with these ground truth masks is encapsulated in a pixel-level true positive rate(TPR), false positive rate (FPR) and ROC curve.
Subsequently, these could be used to calculate the area under the ROCcurve (ROC AUC) score using the sklearn metric ‘roc_auc_score‘.
Furthermore, with the trained model we tried toreproduce the baseline score presented in the paper, which involved computing the difference between the input imageand its reconstruction and similarly comparing the result with the ground truth masks to obtain the ROC AUC score.
3.2.2 MVTec-AD set up
In the paper, the anomaly detection is further tested on the in 2019 released MVTec Anomaly Detection (MVTec-AD)dataset [12].
This dataset consists of images of 15 natural objects and textures with different defects and pixel-levelground truth masks.
Again the repository did not contain any code to run experiments speciﬁcally for this dataset, butthe functions to obtain the attention maps using method (2) could be used.
Additionally, we implemented the VAEas stated in Table 3 of the addendum paper by the authors.
The encoder consists of a ResNet18 (except for its lasttwo layers) followed by two linear layers and the decoder consists of two linear layers, six blocks containing a 2D
Convolutional layer, batch normalization and a ReLU, and ﬁnally a sigmoid layer.
During training, Adam was usedfor optimization with a learning rate of 1 e − 4 and batch size of 8 .
We trained a separate model for each object/texturecategory.
Again, the number of epochs was undeclared, but we trained each category for 100 epochs due to timeconstraints.
Before training, the data was augmented similarly to the MVTec-AD paper, of which the speciﬁcations are as follows.
All images were resized to 256 × 256 pixels.
Additionally, they were randomly rotated between and/orhorizontally mirrored to create an augmented training set of 10.000 images per object/texture.
The probability for thehorizontal mirroring was not speciﬁed but chosen to be 0.5 because it is the default value for this method.
The paddingused after rotation was also unclear.
We therefore decided to use black pixels since that is the standard parameter forpadding.
For comparison, the leather category was padded differently, namely with the colour of the original image’supper-left pixel.
After training, the attention maps for the images containing anomalies could be generated for the last convolutionallayer of the VAE model.
These attention maps could then be compared to the ground truth masks to obtain the ROC
AUC score.
Moreover, the intersection-over-union (IOU) score was also calculated from the ROC curve.
The experiments were run on the same GPU node as the USCD data set.
Creation of the augmented dataset took around1,5 hours per object, while training of one model for 100 epochs took around 2,5 hours.
The MVTec-AD dataset has asize of 5GB and training requires around 11GB of memory
3.3 Latent space disentanglement
The authors’ code did not contain any implementation speciﬁc to the proposed attention disentanglement (AD) loss.
Nonetheless, the code to generate attention maps applying the (2) method proved to be useful.
Consequently, we hadto implement the FactorVAE, disentanglement metric, and calculation of the AD loss ourselves.
The authors use the FactorVAE [13] to showcase how the proposed loss can be integrated into other loss terms.
They call this new versionof the model AD-FactorVAE.
To implement FactorVAE, we adapted an external GitHub repository [14] which wasrecommended to us by the original authors.
From this repository we also adopted code to read in the dSprites dataset,which is used for the evaluation of both the FactorVAE and AD-FactorVAE [15].
We integrated the proposed lossinto this implementation according to Equation 6 in the paper.
Besides the model, the paper also makes use of thedisentanglement metric proposed in the FactorVAE paper.
The underlying assumption of this metric is that if the latentspace is perfectly disentangled, then each latent dimension of the representation should correspond to a single latentfactor used to generate the data.
Consequently, if that latent factor is kept ﬁxed, the variance of the correspondinglatent dimensions should be 0 .
A majority vote classiﬁer is trained to predict factors corresponding to speciﬁc latentdimensions by ﬁxing single factors, passing data through the trained VAE model and checking for the dimension withthe lowest variance.
The metric is the accuracy of this classiﬁer.
The implementation of this metric was adopted froman external GitHub repository [16].
The hyperparameters used for these experiments are shown in Table 1.
The original paper states that they did not modifyany parameters of the model compared the ones mentioned in the FactorVAE paper [13], other then increasing thenumber of latent dimensions from 10 to 32 .
The negative slope of the LeakyReLU in the Discriminator is not mentionedin the FactorVAE paper, however the adapted implementation assumes it to be 0 . 2 .
Moreover the authors also did notspecify the size of the test set used to obtain the disentanglement score.
We chose to use 200 votes, because this did notslow down the score calculation too much while still providing us with a robust indication of the accuracy.
Additionaly,some hyperparameters of AD-FactorVAE were unclear from the paper.
There is no optimal value stated for λ , theweight of the AD loss.
It was also not clear how to aggregate AD losses of several latent dimension pairs or how topick one or more pairs for the AD loss in general.
However due to time constraints, we could only experiment withthe different aggregation of AD losses for all possible dimension pairings.
The experiments were again run on one NVIDIA {+
 GPU node.+}
Training requires around 16GB memory and 24 hours to ﬁnish, while testing requires thesame amount of memory and at most 30 minutes.
Optimizer:
Adam
Learning rate: 10 −
Batch size:
Latent dimensions:
VAE beta1: 0 .
VAE beta2: 0 .
LeakyReLU slope : 0 .
Discriminator beta1: 0 .
Discriminator beta2: 0 .
Metric batch size:
Metric training size:
Metric evaluation size : γ value: { 20 , 40 , 100 } λ value :
Aggregation method : {sum, mean}
Results
This section states the results of the experiments described in the previous section.
4.1 Attention maps
Replicating the results for the MNIST dataset was an involved process – as described in the methodology – leaving uswith various versions of replications.
In Figure 1 we present the two different methods based on the pre-trained modeltrained on handwritten digit 1’s and tested on handwritten digit 7’s with three different functions (ReLU, absolute valueand sigmoid).
Most of these versions would be able to produce the images presented in the paper, except for thosegenerated using the sigmoid which is why this function was excluded from further experiments.
The only differencebetween the decent versions is the level of cherry picking Liu et al did.
We are assuming a minimal amount, andtherefore conclude that the examples in Figure 4 in the original paper were most likely produced with method (1) andthe ReLU function, even though the paper seems to suggest they were produced using method (2).
We chose to proceedwith method (1) as the results resembled those presented in the paper much better.
The attention maps for the modeltrained on digit 1 and tested on the digits 9, 2 and 4, as well as the model trained on digit 3 and tested on 8 and 5 areproduced with those settings and the pre-trained models.
These can be found in Figure 2.
We are fairly certain that the differences between the pre-trained model and the models trained from scratch are not dueto differences in architecture or hyperparameters as they were explicitly stated in the addendum paper, which means theonly remaining inﬂuential factor is the amount of epochs for which the model was trained.
As we reached out to theauthors about the amount of epochs but did not get a response we could not conﬁrm this suspicion.
4.2 UCSD Ped1 Results
We were able to generate qualitative results which match those presented in the paper to a certain degree, however, theyare not as good in quality.
These results can be seen on Figure 3.
The frames in the paper could be matched in quality ifwe were cherry picking results, which could be one of the explanations for the mismatch, but a more likely explanationwould be that we were simply not able to replicate the results properly.
Quantitative results are shown in Table 4.
These results show that we were unable to recreate the exact scores presentedby Liu et al.
It is worth noting that we were also not able to exactly reproduce the baseline scores presented by Liu et al.for the VAE model (also Vanilla-VAE ).
The baseline score in the paper reached 0 .
86 while ours got stuck at 0 .
Inconclusion, all our ROC AUC scores were signiﬁcantly lower compared to the scores from the paper.
The scores were highly dependent on the method, and while method (2) was presented as the anomaly localisation method, method (1)seems to consistently perform better.
An additional topic of uncertainty was how masks without anomalies were handled.
As it was not touched upon in thepaper or the addendum paper we explored two options, (1) including all images whether they had anomalies or notand (2) excluding those frames without any anomalies.
Furthermore, because we created this code from scratch weexperimented with various small adjustments to the code, one of which was whether we should upsample or downscalethe images, depending on the convolutional layer.
Where the second (downscaling) led to higher scores, we decided tostick with the ﬁrst approach as we felt it was more representative of the actual performance of the model (downscalingincreases the probability of getting the right pixel value, so as we apply more convolutions the image gets smaller whichleads to higher scores).
Another question left unanswered was whether to do any processing on the generated attentionmaps, such as normalization etc.
We decided against this, as it did not improve the scores and the normalization in thecode base seemed to only have the goal of displaying the images correctly.
Paper (
[Table]
baseline
[Table]
[Table]
[Table]
[Table]
[Table]
conv
[Table]
[Table]
[Table]
[Table]
[Table]
conv
[Table]
[Table]
[Table]
[Table]
conv
[Table]
4.3 MVTec-AD Results
The MVTec-AD results are shown in Table 2.
Due to time constraints we were not able to produce scores for allobjects and textures and could only use the (2) second method for attention map generation.
We see that our scoresare signiﬁcantly lower compared to the scores stated in the paper and presented in the second column of Table 2.
Theresults are also lower compared the scores presented in the original work of Bergmann et al.
[12] which are presentedin Table 2 of the original paper by Liu et al.
Figure 5a shows some qualitative results from three categories.
Theseattention maps were one of the best that were produced during our experiments.
One can note that these attention mapsare far from accurate in selecting the anomalous regions.
Therefore both the quantitative and the qualitative results weproduced are quite different from the results in the original paper.
However, in contrast to the UCSD Ped1 experiments,there is no baseline score we can use to put the scores into perspective. Figure
4.4 Anomaly detection
[Table]
[Table]
Bottle
[Table]
[Table]
[Table]
[Table]
Cable
[Table]
[Table] [Table] [Table] Capsule
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
Hazelnut
[Table] [Table] [Table] [Table]
[Table]
[Table]
Metal Nut
[Table]
[Table] [Table] [Table]
[Table]
Pill
[Table]
[Table] [Table]
[Table] [Table]
Screw
[Table]
[Table]
[Table]
Toothbrush
[Table]
[Table]
[Table]
[Table]
Transistor
[Table] [Table] [Table] from MVTec-AD dataset.
egory
In conclusion, we can state that we were not able to replicate the results foranomaly detection proposed by the authors. We therefore cannot support theclaim that the performance of their proposed anomaly localization methodis state-of-the-art by obtaining ROCAUC and IOU scores that are better orsimilar to those obtained by the models compared in the original work of Bergmann et al. Moreover, the attention maps did not exceed the baselinescore for the UCSD Ped1 experiments.
There might, however, be room for improvement within our implementationswhich could lead to state-of-the-art performance, as stated by Liu et al.
Possibilities for improvement could be training the models for more epochs,different weight initialization, or changing the way we generate the attentionmaps, for which there is some confusion about the usage of method (1) ormethod (2), and using ReLU or absolute value.
Striking is the fact that method (1) – which was originally just proposedfor visualising the attention maps – consistently outperforms method (2) –which was speciﬁcally proposed for anomaly localization – on the anomalylocalization task and is able to generate results much closer related to thosepresented in the paper.
The trade-off between the two functions seems to bequality vs. time.
Method (1) has to loop over the entire latent space, makingit much slower, where method (2) just has to propagate the mean vector.
Weare unsure which one of these methods was actually used to obtain the ﬁnalresults presented in the paper.
4.5 Attention Disentanglement Results
As the FactorVAE paper does not mention the weight initialisation used,we relied on the external code which provides options for both normal andkaiming normal initialization.
However after being unable to reproduce thebaseline results with either of these methods, explicit weight initialization wasremoved causing the pytorch default, kaiming uniform, to be applied.
Thisallowed us to achieve the desired results, but it also shows how sensitive thesemethods are to small technical details.
Additionally, we had to ignore thecolor factor to reach the correct disentanglement scores.
Object or backgroundcolor cannot be varied randomly in the dSprites dataset hence it always actsas a ﬁxed factor for the metric.
Setups where loss values were aggregated using the mean and a γ value of20 or 40 , converged successfully to the expected reconstruction loss.
Resultsof their corresponding scores can be seen on Figure 5b.
We can clearly seethat using ReLU achieves signiﬁcantly superior results compared to usingabsolute value.
However, in general we can see that adding the proposedloss did not increase the disentanglement score compared to the baseline FactorVAE.
Setups where the loss values were aggregated using a sum or a γ of 100 , converged to reconstruction loss values wellover 100 .
A possible reason to why summing does not work is that then the AD loss gets too large and inﬂuences theoverall loss too much.
The same could apply to the issues with a large γ parameter and the Total Correlation loss of FactorVAE, however as this was part of the baseline, this rather suggests that we might have missed key informationabout training models with a parameter of this magnitude.
Because the authors claim the addition of the AD loss doesnot decrease the reconstruction loss, we only determined a model’s disentanglement score if it reached a reconstructionloss similar to the baseline FactorVAE model.
Discussion
In this paper, we have provided insights in the reproducibility attempt of the paper “Towards Visually Explaining
Variational Autoencoders" by Liu et al.
Our results show that, with the provided tools and within a time-frame of fourweeks, we have been able to reproduce similar attention maps for the MNIST dataset.
The gradient-based attention maps clearly denoted characteristics that intuitively made sense.
We can therefore conﬁrm the ﬁrst claim by the authors.
Besides that, we have not been able to produce any results that were comparable to the ones from paper.
This leads tothe conclusion that we cannot successfully conﬁrm the second and third claim made by the authors.
However, someresults and extra experiments indicated that there might be room for improvement within the used methods throughhyperparameter tuning or small model changes.
Our approach has several weaknesses which will be discussed in thenext section.
5.1 Strengths and weaknesses
Our implementation has several strong points, including:
• The authors’ code was partly available on GitHub containing method (2) for the generation of the attentionmaps (as explained in Section 3.1) as well as the set-up for the MNIST experiments.
This means that for theseparts, we are quite certain that our code matches the authors’ code.
• All datasets were publicly available and well documented, causing us to be fairly conﬁdent that we used thesame data as the authors.
• For the models that were not implemented in the authors’ code, the architectural details were publicly availablein the authors’ addendum paper.
We were therefore able to implement the same models as the authors used.• For the FactorVAE model, the authors referred us to a repository which contained an implementation thatonly needed some small adjustments to give us the correct baseline results.
Thus, it seems unlikely that ourimplementation of the FactorVAE model is incorrect.• For the FactorVAE disentanglement metric a Google repository was found that contained a proper implementation, which gives a high probability that our implementation of this metric is correct.•
In many of our experiments, we have tried several different approaches to obtain the same results as the authors.
For example, because we did not obtain correct results for the MNIST experiments, we also implemented theﬁrst approach for the attention maps generation even though the paper mentioned the results were obtainedusing the second approach.
For the attention disentanglement we experimented with several attention mapgeneration and loss aggregation methods.
However, there are also some weaknesses in our approach:
• The authors’ code and the paper were sometimes inconsistent with each other.
Especially for the generation of the attention maps. It was unclear if the results were obtained using ReLU (as stated in the paper) or usingabsolute value (as stated in the code).
We tried to overcome this weakness by implementing both approaches.
Furthermore, it was unclear which method to use for which part.
The misalignment between the fact thatmethod (2) seems to be presented as the method to use for all experiments in the original paper, while method(1) consistently brings us results that are closer to the results presented is vexing to us and might account for some of the discrepancies between the original results and ours.
• For each experiment the amount of training epochs was unclear causing us to make guesses due to limited time.
• It showed to be very difﬁcult to reproduce the qualitative and quantitative results for anomaly localization on both the UCSD Ped1 dataset and the MVTec-AD data even though the models’ architectures were speciﬁed inthe addendum paper.
Additional research is needed to discover why our experiments failed to produce thecorrect scores.
• It was not possible to implement the exact same data augmentation for the MVTec-AD dataset as the paper didnot provide enough information on this topic.•
The entire implementation for the attention disentanglement was missing in the authors’ code.
We thereforehad to make many guesses about the implementation of this part, which made it difﬁcult to reproduce thequalitative and quantitative results for latent space disentanglement on the dSprites dataset.
5.2 Communication with original authors
We have communicated with the authors on two occasions.
The ﬁrst time we e-mailed them with a request for thecomplete code as the public repository had many missing parts.
They replied that they were not able to give ustheir full code as they were planning to patent their implementation in the future.
We sent them another e-mail withsome follow-up questions about the number of epochs, data-augmentation for the MVTec-AD dataset and method ofaggregation of the AD loss.
Unfortunately, we did not receive a reply to that last e-mail.