Original: Towards Visually Plausible Explanations
Revised: Towards Visually Plausible Explanations

Original: Introduction
Revised: SummaryIntroduction

Original: The goal of this assignment is to ﬁrstly reimplement the work from the paper Liu et al.
Revised: The goal of this assignment is to ﬁrstly reimplement the work from the paper Liu et al.

Original: (2020) and secondly extendtheir work.
Revised: (2020) and secondly extendtheir work.

Original: Liu et al.
Revised: Liu et al.

Original: develops a new technique which visually explains VAE by generating attention maps from thelearned latent space.
Revised: develops a new technique which visually explains VAE by generating attention maps from thelearned latent space.

Original: Then the paper applies the VAE into two applications: anomaly localisation and latent spacedisentanglement.
Revised: Then the paper applies the VAE into two applications: anomaly localisation and latent spacedisentanglement.

Original: This paper reimplements Liu’s experiments and compares our results with the results from the paperunder restricted computational resources.
Revised: This paper reimplements Liu’s experiments and compares our results with the results from the paperunder restricted computational resources.

Original: The acquired results will be analyzed and compared to the original paper.
Revised: The acquired results will be analyzed and compared to the original paper.

Original:  Scope of Reproducibility
Revised:  Scope of Reproducibility

Original: The paper Liu et al.
Revised: The paper Liu et al.

Original: (2020) claims that their research takes steps towards visually explaining generative models by usinga new method visual attention maps conditioned on the latent space of a variational autoencoder (VAE).
Revised: (2020) claims that their research takes steps towards visually explaining generative models by usinga new method visual attention maps conditioned on the latent space of a variational autoencoder (VAE).

Original: Furthermore,the attention maps can be used to demonstrate the localisation of anomalies in images.
Revised: Furthermore,the attention maps can be used to demonstrate the localisation of anomalies in images.

Original: This localisation method foranomaly detection, resulted in state-of-the-art performance in the MVTec-AD dataset.
Revised: This localisation method foranomaly detection, resulted in state-of-the-art performance in the MVTec-AD dataset.

Original: Moreover, a new learningobjective was formed: attention disentanglement loss.
Revised: Moreover, a new learningobjective was formed: attention disentanglement loss.

Original: This resulted in better performance on the Dpsrites datasetcompared to state-of-the-arts methods.
Revised: This resulted in better performance on the Dpsrites datasetcompared to state-of-the-arts methods.

Original: Methodology
Revised: Methodology

Original: For the ﬁrst part of the experiment the author’s code was used, and for the second experiment, the proposed method from(Kim and Mnih, 2019) was used with the addition of the disentanglement metric mentioned in the original paper.
Revised: For the ﬁrst part of the experiment the author’s code was used, and for the second experiment, the proposed method from(Kim and Mnih, 2019) was used with the addition of the disentanglement metric mentioned in the original paper.

Original: Thetotal training lasted around six hours for the anomaly localisation and for the disentanglement it lasted approximately40 to 80 hours.
Revised: Thetotal training lasted around six hours for the anomaly localisation and for the disentanglement it lasted approximately40 to 80 hours.

Original: Slight hyperparameter tuning was needed for some of the datasets.
Revised: Slight hyperparameter tuning was needed for some of the datasets.

Original: Results
Revised: Results

Original: The results showed that for the most part the anomaly localisation claims seem to be reproducible (except for the UCSDpedestrians dataset).
Revised: The results showed that for the most part the anomaly localisation claims seem to be reproducible (except for the UCSDpedestrians dataset).

Original: But the disentanglement was not reproducible.
Revised: But the disentanglement was not reproducible.

Original: What was easy
Revised: What was easy

Original: The easiest reproducible parts for this experiment was the part for which the code and documentation was available.
Revised: The easiest reproducible parts for this experiment was the part for which the code and documentation was available.

Original: What was difﬁcult
Revised: What was difﬁcult

Original: There was a wide range of obstacles, when reproducing this code.
Revised: There was a wide range of obstacles, when reproducing this code.

Original: Much of the information was dependent on the interpretation of the reader.
Revised: Much of the information was dependent on theinterpretation of the reader.

Original: There is no explicit documentation about the implementation in the paper; hyper-parameters,experiment set-up and model descriptions are not present.
Revised: There is no explicit documentation about the implementation in the paper; hyper-parameters,experiment set-up and model descriptions are not present.

Original: Also, the full second part of the implementation was missingfrom the code base.
Revised: Also, the full second part of the implementation was missingfrom the code base.

Original: And many equations or formulations were not sufﬁciently explained.
Revised: And many equations or formulations were not sufﬁciently explained.

Original: Communication with original authors
Revised: Communication with original authors

Original: There was little communication, only an opened issue on their Github repository, giving a solution to a known issue
Revised: There was little communication, only an opened issue on their Github repository, giving a solution to a known issue

Original:  Introduction
Revised:  Introduction

Original: The goal of this assignment is to ﬁrstly reimplement the work from the paper Liu et al.
Revised: The goal of this assignment is to ﬁrstly reimplement the work from the paper Liu et al.

Original: (2020) and secondly extendtheir work.
Revised: (2020) and secondly extendtheir work.

Original: Liu et al.
Revised: Liu et al.

Original: (2020) state that their research takes steps towards visually explaining generative models.
Revised: (2020) state that their research takes steps towards visually explaining generative models.

Original: This isdone by using a new method; visual attention maps conditioned on the latent space of a variational autoencoder (VAE).
Revised: This isdone by using a new method; visual attention maps conditioned on the latent space of a variational autoencoder (VAE).

Original: Attention maps aim at intuitively representing different regions of normal and abnormal images.
Revised: Attention maps aim at intuitively representing different regions of normal and abnormal images.

Original: The paper applies the
Revised: The paper applies the

Original: VAE in two experiments: anomaly localisation and latent space disentanglement.
Revised: VAE in two experiments: anomaly localisation and latent space disentanglement.

Original: Anomaly localisation means to detectand localise the abnormal phenomena in an image.
Revised: Anomaly localisation means to detectand localise the abnormal phenomena in an image.

Original: This is performed through the cues in the attention maps.
Revised: This is performed through the cues in the attention maps.

Original: Latentspace disentanglement focuses on learning a decomposed representation, meaning it describes each piece of data x by aset of statistically independent factors z .
Revised: Latentspace disentanglement focuses on learning a decomposed representation, meaning it describes each piece of data x by aset of statistically independent factors z .

Original: This has several beneﬁts, such as explaining the underlying decision makingprocess in unsupervised learning (Nitzan et al., 2020).
Revised: This has several beneﬁts, such as explaining the underlying decision makingprocess in unsupervised learning (Nitzan et al., 2020).

Original: The novel disentanglement loss (mentioned in the original paper)helps utilize the information from the attention maps to improve the performance of Factor VAE (Liu et al., 2020).
Revised: The novel disentanglement loss (mentioned in the original paper)helps utilize the information from the attention maps to improve the performance of Factor VAE (Liu et al., 2020).

Original: Wereimplemented their experiments and compare our results with the result from the paper.
Revised: Wereimplemented their experiments and compare our results with the result from the paper.

Original: Moreover, we analyze theclaims of the paper and assess their validity.
Revised: Moreover, we analyze theclaims of the paper and assess their validity.

Original:  Scope of reproducibility
Revised:  Scope of reproducibility

Original: The work presented in this report aligns with our views of having visual systems that are more explainable.
Revised: The work presented in this report aligns with our views of having visual systems that are more explainable.

Original: The abilityto extract insights about the decisions made by Generative models is very important given that they are targets for thenew trend of deepFakes (Nguyen et al., 2020).
Revised: The abilityto extract insights about the decisions made by Generative models is very important given that they are targets for thenew trend of deepFakes (Nguyen et al., 2020).

Original: Hence, we think investigation in this area should be developed andapplied in the industry and academia.
Revised: Hence, we think investigation in this area should be developed andapplied in the industry and academia.

Original: According to the study from Liu et al.
Revised: According to the study from Liu et al.

Original: (2020), the proposed attention mechanism is able to highlight anomalous areas inthe images, and consequently identify the features in the underlying latent space that cause the abnormality.
Revised: (2020), the proposed attention mechanism is able to highlight anomalous areas inthe images, and consequently identify the features in the underlying latent space that cause the abnormality.

Original: Regarding the Anomaly detection, (i) good qualitative results are found for the MNIST dataset, meaning that theattention maps’ highlighted regions correlate with high precision to the localisation of a single anomaly, (ii) thequalitative and quantitative results for the UCSD Pedestrians dataset perform better than the baseline.
Revised: Regarding the Anomaly detection, (i) good qualitative results are found for the MNIST dataset, meaning that theattention maps’ highlighted regions correlate with high precision to the localisation of a single anomaly, (ii) thequalitative and quantitative results for the UCSD Pedestrians dataset perform better than the baseline.

Original: (iii)
Revised: (iii)

Original: And, the
Revised: And, the

Original: MVTec-AD dataset out-performed other state-of-the-arts (Bergmann et al., 2019)
Revised: MVTec-AD dataset out-performed other state-of-the-arts (Bergmann et al., 2019)

Original: (for more information on evaluationsee  3.5) Regarding, the attention disentanglement (iv) a new learning objective is proposed that improves the trade-offbetween reconstruction loss and disentanglement metric over existing methods.
Revised: (for more information on evaluationsee section 3.5) Regarding, the attention disentanglement (iv) a new learning objective is proposed that improves thetrade-off between reconstruction loss and disentanglement metric over existing methods.

Original: Additionally, (v) the paper assertsthat the attention disentanglement loss helps to separate the high-response pixel regions which promise better resultsquantitatively and qualitatively in comparison to the state-of-the-arts (Liu et al., 2020).
Revised: Additionally, (v) the paperasserts that the attention disentanglement loss helps to separate the high-response pixel regions which promise betterresults quantitatively and qualitatively in comparison to the state-of-the-arts (Liu et al., 2020).

Original: To push the boundaries of the proposed visual explanations we extend their approach to Restricted Boltzmann Machines,hypothesising similar claims to the original paper; expecting signiﬁcant performance qualitative and quantitatively .
Revised: To push the boundaries of the proposed visual explanations we extend their approach to Restricted Boltzmann Machines,hypothesising similar claims to the original paper; expecting signiﬁcant performance qualitative and quantitatively .

Original:  Methodology
Revised:  Methodology

Original: The research is divided into three different part: the anomaly detection, the latent space disentanglement and the ablationstudy.
Revised: The research is divided into three different part: the anomaly detection, the latent space disentanglement and the ablationstudy.

Original: 3.1 Datasets
Revised: 3.1 Datasets

Original: The datasets used in this research are the MNIST, UCSD Ped 1, MVTec-AD and Dsprites datasets.
Revised: The datasets used in this research are the MNIST, UCSD Ped 1, MVTec-AD and Dsprites datasets.

Original: The MNISTdataset contains 70,000 grey level images of handwritten digits (60,000 in training and 10,000 in test set) that aresize-normalized and centered in a ﬁxed-size image of 28x28 (LeCun et al., 1998).
Revised: The MNISTdataset contains 70,000 grey level images of handwritten digits (60,000 in training and 10,000 in test set) that aresize-normalized and centered in a ﬁxed-size image of 28x28 (LeCun et al., 1998).

Original: The MVTec-AD Dataset contains 5354 high-resolution images.
Revised: The MVTec-AD Dataset contains 5354 high-resolution images.

Original: These images are divided into ﬁfteen different objectand texture various defects.
Revised: These images are divided into ﬁfteen different objectsand textures with various defects.

Original: It includes pixel-wise annotations.
Revised: It includes pixel-wise annotations.

Original: (Bergmann et al., 2019).
Revised: (Bergmann et al., 2019).

Original: The images are resized to256x256{+resizedto 256x
+} (as is done in the original paper).
Revised: The images are {+resizedto 256x
+} (as is done in the original paper).

Original: The data is also augmented by performing random ﬂips and rotations.
Revised: The data is also augmented by performing random ﬂips and rotations.

Original: Formore information on the distribution for this dataset and the MNIST dataset see ﬁgure 9 & 8 in the Appendix.
Revised: For more information on the distribution for this dataset and the MNIST dataset see ﬁgure 9 & 8 in the supplementarymaterials.

Original: The UCSD Ped 1 anomaly detection dataset is compromised of images (frames from videos) from a stationary cameraoverlooking a pedestrian walkway.
Revised: The UCSD Ped 1 anomaly detection dataset is compromised of images (frames from videos) from a stationary cameraoverlooking a pedestrian walkway

Original: The anomalies are in the form of non-pedestrian entities (e.g. cart, wheelchair).

Original: Peds1 consists of 5500 frames training test and 3400 anomalous test frames.

Original: The frames are resized to 100x100 pixels(same as the original paper).

Original: Some manually generated pixel-level binary anomaly masks are available.

Original: (Li et al., 2013).

Original: The Dsprites datasets consists of 737280 64x64 black and white images of 2D shapes.
Revised: The Dsprites datasets consists of 737280 64x64 black and white images of 2D shapes.

Original: The images are augmented byvarying six ground truth independent latent factors (color, shape, scale, rotation and x and y positions).(Higgins et al.,2016)
Revised: The images are augmented byvarying six ground truth independent latent factors (color, shape, scale, rotation and x and y positions).(Higgins et al.,2016)

Original: (Matthey et al., 2017).
Revised: (Matthey et al., 2017).

Original: 3.2 Anomaly detection
Revised: 3.2 Anomaly detection

Original: The anomaly localisation with attention maps applies a vanilla VAE (one-class VAE).
Revised: The anomaly localisation with attention maps applies a vanilla VAE (one-class VAE).

Original: This essentially is an autoencodertrained with a reconstruction objective and a variational term objective that attempts learning a standard normal latentspace distribution.
Revised: This essentially is an autoencodertrained with a reconstruction objective and a variational term objective that attempts learning a standard normal latentspace distribution.

Original: The variational term objective is generally implemented with a Kullback-Leibler distribution metric,which is based on a computation featuring the latent space distribution and a standard Gaussian.
Revised: The variational term objective is generally implemented with a Kullback-Leibler distribution metric,which is based on a computation featuring the latent space distribution and a standard Gaussian.

Original: Additionally you canrefer to (Liu et al., 2020)(Kingma and Welling, 2014).
Revised: Additionally you canrefer to (Liu et al., 2020)(Kingma and Welling, 2014).

Original: The attention maps are generated by computing a score from thelatent space and back-propagating on this score to a particular convolutional layer to get the gradients of the score withrespect to the activations of that layer.
Revised: The attention maps are generated by computing a score from thelatent space and back-propagating on this score to a particular convolutional layer to get the gradients of the score withrespect to the activations of that layer.

Original: The paper introduced a way to generate attention maps for anomaly localisation,by computing a score through taking the element-wise sum of the mean vector, µ , from the encoder (for more detailedinformation see (Liu et al., 2020)).
Revised: The paper introduced a way to generate attention maps for anomaly localisation,by computing a score through taking the element-wise sum of the mean vector, µ , from the encoder (for more detailedinformation see (Liu et al., 2020)).

Original: There are no speciﬁcs on the model architectures for the anomaly localisation.
Revised: There are no speciﬁcs on the model architectures for the anomaly localisation.

Original: However, the Github repository from theauthors provides a simple VAE architecture for MNIST, which provides decent qualitative results.
Revised: However, the Github repository from theauthors provides a simple VAE architecture for MNIST, which provides decent qualitative results.

Original: This model wasadapted to ﬁt the resized UCSD images, by adding extra convolutional layers.
Revised: This model wasadapted to ﬁt the resized UCSD images, by adding extra convolutional layers.

Original: For the MVTec-AD dataset accordingto the original paper a ResNet-18 is used.
Revised: For the MVTec-AD dataset accordingto the original paper a ResNet-18 is used.

Original: To build the architecture for this dataset, a ResNet-18 VAE was used byreplacing the last two layers of the standard ResNet-18 architecture to the latent dimension space, in this case 32,to build a ResNet-18 encoder.
Revised: To build the architecture for this dataset, a ResNet-18 VAE was used byreplacing the last two layers of the standard ResNet-18 architecture to the latent dimension space, in this case 32,to build a ResNet-18 encoder.

Original: For the decoder, transpose convolutions and upsampling layers are used instead ofconvolutional layers (for more information on the architectures appendix B.1).
Revised: For the decoder, transpose convolutions and upsampling layers are used instead ofconvolutional layers (for more information on the architectures supplementary material B.1).

Original: The hyperparameter settings can be found in the Table 1, the hyperparameters for the MNIST dataset were alreadypredeﬁned and sufﬁcient to achieve comparable results to the original paper, so these functions were used.
Revised: The hyperparameter settings can be found in the Table 1, the hyperparameters for the MNIST dataset were alreadypredeﬁned and sufﬁcient to achieve comparable results to the original paper, so these functions were used.

Original: These alsoserved as a starting point for (manual) tuning the other two models on the UCSD and MVTec-AD dataset.
Revised: These alsoserved as a starting point for (manual) tuning the other two models on the UCSD and MVTec-AD dataset.

Original: However, forthe MVTec-AD it was changed slightly to accommodate our computational power.
Revised: However, forthe MVTec-AD it was changed slightly to accommodate our computational power.

Original: 3.3 Latent space disentanglement
Revised: 3.3 Latent space disentanglement

Original: The factor disentanglement adapts the Factor VAE which attempts to learn a factorial distribution for the latent variablesdistributions by using adiscriminator namely a Multilayer Perceptron.
Revised: The factor disentanglement adapts the Factor VAE which attempts to learn a factorial distribution for the latent variablesdistributions by using a Multilayer Perceptron.

Original: The Factor VAE loss ( L FV ) component takesinto account the batch size N , the reconstruction loss, the Kullback-Leibler metric and the total correlation penalty.
Revised: The Factor VAE loss ( L FV ) component takes into account the batchsize N , the reconstruction loss, the Kullback-Leibler metric and the total correlation penalty.

Original: Theγ hyperparameter controls the importance of the discriminator cross-entropy loss.
Revised: The γ hyperparametercontrols the importance of the discriminator cross-entropy loss.

Original: For details of the Factor VAE refer to(Kim and Mnih, 2019).
Revised: For details of the Factor VAE refer to (Kim and Mnih,2019).

Original: The innovative contribution in the disentanglement combines the information from the attentionmaps to create the disentanglement loss component
Revised: The innovative contribution in the disentanglement combines the information from the attention maps to createthe disentanglement loss component

Original: [Equation]
Revised: [Equation]

Original: where the model loss is L =
Revised: where the model loss is L =

Original: L FV + λL AD and λ is a hyperparameter.
Revised: L FV + λL AD and λ is a hyperparameter.

Original: The last loss component promotes the divergencebetween the two attention maps and takes into account their lowest response pixel regions.
Revised: The last loss component promotes the divergencebetween the two attention maps and takes into account their lowest response pixel regions.

Original: The attention maps selectionmechanism used in all experiments pairs the attention maps and averages the disentanglement loss from equation 1 overall pairs.
Revised: The attention maps selectionmechanism used in all experiments pairs the attention maps and averages the disentanglement loss from equation 1 overall pairs.

Original: For example, in the case of 4 attention maps a, b, c, d we average the loss over the pairs ( a, b ) , ( b, c ) , ( c, d ) .
Revised: For example, in the case of 4 attention maps a, b, c, d we average the loss over the pairs ( a, b ) , ( b, c ) , ( c, d ) .

Original: One drawback is that by forcing the attention between consecutive maps to diverge it might lead to one of thembecoming closer to another map, however these trade-offs are beyond the scope of the current experiments and futurework could explore different attention maps selection mechanisms.
Revised: One drawback is that by forcing the attention between consecutive maps to diverge it might lead to one of thembecoming closer to another map, however these trade-offs are beyond the scope of the current experiments and futurework could explore different attention maps selection mechanisms.

Original: The disentanglement metric employed was proposed by (Kim and Mnih, 2019).
Revised: The disentanglement metric employed was proposed by (Kim and Mnih, 2019).

Original: It is a majority vote linear classiﬁerthat aims at being robust over hyperparameter changes and different training schedules.
Revised: It is a majority vote linear classiﬁerthat aims at being robust over hyperparameter changes and different training schedules.

Original: Since the attention disentanglement implementation was missing from the author’s codebase, we used this Githubrepository for the implementation.
Revised: Since the attention disentanglement implementation was missing from the author’s codebase, we used this Githubrepository for the implementation.

Original: Moreover, the disentanglement metric used in the experiments by Kim and Mnih(2019) was adapted from the following Github repository.
Revised: Moreover, the disentanglement metric used in the experiments by Kim and Mnih(2019) was adapted from the following Github repository.

Original: The architecture for the experiments is described in theappendix C.1.
Revised: The architecture for the experiments is described in thesupplementary material C.1.

Original: The inﬂuence of the attention disentanglement loss (deﬁned by equation 1) was compared throughvarying the values of gamma in the Factor VAE and the AD-Factor VAE (Table 1).
Revised: The inﬂuence of the attention disentanglement loss (deﬁned by equation 1) was comparedthrough varying the values of gamma in the Factor VAE and the AD-Factor VAE (Table 1).

Original: Moreover, the latent dimensionsthat collapsed to the prior were removed by using a threshold of 0.01 (in the metric score calculation).
Revised: Moreover, the latentdimensions that collapsed to the prior were removed by using a threshold of 0.01 (in the metric score calculation).

Original: The amount ofepochs and values of gamma are based on the FactorVAE original paper (the exact values of the hyperparameters areshown in Table 1).
Revised: The amount of epochs and values of gamma are based on the FactorVAE original paper (the exact values of thehyperparameters are shown in Table 1).

Original: Across the experiments for each model we use 10 latent variables and the number of iterations washalf of the iterations mentioned in Kim and Mnih (2019).
Revised: Across the experiments for each model we use 10 latent variables and thenumber of iterations was half of the iterations mentioned in Kim and Mnih (2019).

Original: For more details on this decision refer to appendix C.2.
Revised: For more details on this decisionrefer to supplementary material C.2.

Revised: Hyperparameters

Revised: [Table]

Original: 3.4 Ablation study
Revised: 3.4 Ablation study

Original: Two distinct types of ablation studies were performed.
Revised: Two distinct types of ablation studies were performed.

Original: The ﬁrst type is to analyse discrepancies and ambiguities in thecode.
Revised: The ﬁrst type is to analyse discrepancies and ambiguities in thecode.

Original: The second type is to extend the approach of the original paper to a different type of Generative model.
Revised: The second type is to extend the approach of the original paper to a different type of Generative model.

Original: Hyperparameters

Original: [Table]

Original: 3.4.1 Hyper parameters and discrepancies
Revised: 3.4.1 Hyper parameters and discrepancies

Original: The aim was to reproduce the paper from the available code and description in the original paper.
Revised: The aim was to reproduce the paper from the available code and description in the original paper.

Original: However, due todiscrepancies between the author’s codebase and paper, additional experiments were performed.
Revised: However, due todiscrepancies between the author’s codebase and paper, additional experiments were performed.

Original: First of all, in equation2 in the original paper there is a ReLU activation on the attention maps, which is absent from the codebase.
Revised: First of all, in equation2 in the original paper there is a ReLU activation on the attention maps, which is absent from the codebase.

Original: But, thisbarely affected any of the attention maps and quantitative results, so this was deemed unnecessary.
Revised: But, thisbarely affected any of the attention maps and quantitative results, so this was deemed unnecessary.

Original: Moreover, thecodebase uses L2 normalization on the gradients, dzdA , before computing the weights α for generating the attention maps.
Revised: Moreover, thecodebase uses L2 normalization on the gradients, dzdA , before computing the weights α for generating the attention maps.

Original: This was not mentioned in the paper, but also hardly affected the generated attention maps and AUROC scores.
Revised: This was not mentioned in the paper, but also hardly affected the generated attention maps and AUROC scores.

Original: Lastly,the visualization of the attention maps as shown in the paper was accomplished by interpolating the attention maps tothe input size and overlaying them on top of the input image (also not explained in the paper).
Revised: Lastly,the visualization of the attention maps as shown in the paper was accomplished by interpolating the attention maps tothe input size and overlaying them on top of the input image (also not explained in the paper).

Original: On the Attention Disentanglement section there were also some lacking documentations.
Revised: On the Attention Disentanglement section there were also some lacking documentations.

Original: On formula 6 there is noinformation regarding the λ hyperparameter, so the most suitable value was searched in an ablation study.
Revised: On formula 6 there is noinformation regarding the λ hyperparameter, so the most suitable value was searched in an ablation study.

Original: 3.4.2 Applying Stacked RBM to Anomaly Detection
Revised: 3.4.2 Applying Stacked RBM to Anomaly Detection

Original: To extend the work of the original paper, the novelty of this paper is applied to the stacked restricted boltzman machine(stacked RBM).
Revised: To extend the work of the original paper, the novelty of this paper is applied to the stacked restricted boltzman machine(stacked RBM) (Van et al., 2017).

Original: In order to perform anomaly detection, an attention map is generated for the RBM.
Revised: In order to perform anomaly detection, an attention map is generated for the RBM.

Original: A RBM is composedof visible and hidden units.
Revised: A RBM is composed of visible and hidden units.

Original: It is parameterized by a bias vector in the visible layer, a bias vector in the hidden layerand a weight matrix that does transformations between these two layers.
Revised: It is parameterized by a bias vector in the visible layer, a bias vector inthe hidden layer and a weight matrix that does transformations between these two layers.

Original: A RBM is an energy-based model that aimsat minimizing the energy score of the network and maximizing the likelihood of observed data.
Revised: A RBM is an energy-basedmodel that aims at minimizing the energy score of the network and maximizing the likelihood of observed data.

Original: The optimal weightsare found by applying contrastive divergence (CD) with Gibbs Sampling.
Revised: Theoptimal weights are found by applying contrastive divergence (CD) with Gibbs Sampling.

Original: This means that a RBM draws samplesalternatively from two conditional probabilities until the CD converges.
Revised: This means that a RBMdraws samples alternatively from two conditional probabilities until the CD converges.

Original: We stack multiple RBMs together to form adeep RBM network.
Revised: We stack multiple RBMstogether to form a deep RBM network.

Original: The deep RBM network is trained from bottom RBM to top RBM.
Revised: The deep RBM network is trained from bottom RBM to top RBM.

Original: To generate the attention map,we take the output of the layer that we specify, and compute the attention map as is explained in section 3.2.
Revised: To generatethe attention map, we take the output of the layer that we specify, and compute the attention map as is explainedin section 3.2.

Original: Moreover,the hidden units are constraint to either 1 or -1, which makes it impossible to apply the reparametarization trick used inthe original paper to generate the attention maps.
Revised: Moreover, the hidden units are constraint to either 1 or -1, which makes it impossible to apply thereparametarization trick used in the original paper to generate the attention maps.

Original: In order to approximate an equivalent continuous distribution we usedthe Gumbel-Max Trick (Maddison et al., 2017).
Revised: In order to approximate an equivalentcontinuous distribution we used the Gumbel-Max Trick (Maddison et al., 2017).

Original: 3.5 Evaluation
Revised: 3.5 Evaluation

Original: The evaluation is performed in the same fashion as in the original paper.
Revised: The evaluation is performed in the same fashion as in the original paper.

Original: For anomaly detection, the performance of allquantitative results are evaluated pixel-level segmentation through the area under the receiver operating characteristiccurve score(ROC AUC or AUROC).
Revised: For anomaly detection, the performance of allquantitative results are evaluated pixel-level segmentation through the area under the receiver operating characteristiccurve score(ROC AUC or AUROC).

Original: The true positive rate (TPR) is deﬁned as the amount of pixels across the testingset correctly classiﬁed as anomalous pixels.
Revised: The true positive rate (TPR) is deﬁned as the amount of pixels across the testingset correctly classiﬁed as anomalous pixels.

Original: On the other hand, the False positive rate (FPR) is the amount of pixelswrongly classiﬁed as anomalous across the testing set.
Revised: On the other hand, the False positive rate (FPR) is the amount of pixelswrongly classiﬁed as anomalous across the testing set.

Original: Moreover, the best threshold is picked based on the thresholdsreturned in the ROC curve and picking the best one based off the intersection-over-union (IoU).
Revised: Moreover, the best threshold is picked based on the thresholdsreturned in the ROC curve and picking the best one based off the intersection-over-union (IoU).

Original: Since the UCSD and MVTec-AD datasets include target masks, these are evaluated quantitatively and qualitatively,wheareas the MNIST datset is only evaluated qualitatively.
Revised: Since the UCSD and MVTec-AD datasets include target masks, these are evaluated quantitatively and qualitatively,wheareas the MNIST datset is only evaluated qualitatively.

Original: For the UCSD, the attention mechanism is also tested withdifferent spatial solutions by back-propagating the three penultimate convolutional layers in the model. 
Revised: For the UCSD, the attention mechanism is also tested withdifferent spatial solutions by back-propagating the three penultimate convolutional layers in the model. [Figure]

Original: For the disentanglement experiment, two different random seeds were used to smooth the results for all the experimentsand reduce instabilities.
Revised: For the disentanglement experiment, two different random seeds were used to smooth the results for all the experimentsand reduce instabilities.

Original: Quantitative results were obtained by evaluating the performance through the training metricsof reconstruction error, true total correlation (from VAE) and the test metric of disentanglement.
Revised: Quantitative results were obtained by evaluating the performance through the training metricsof reconstruction error, true total correlation (from VAE) and the test metric of disentanglement.

Original: Here the baselineswere the Factor VAE and β -VAE.
Revised: Here the baselineswere the Factor VAE and β -VAE.

Original: [Figure]

Original: 3.6 Computational requirements
Revised: 3.6 Computational requirements

Original: [Figure]
Revised: [Figure]

Original: The experiments were run on two different systems.
Revised: The experiments were run on two different systems.

Original: The anomaly localisation was performed on Nvidia RTX 2070 Super(8GB VRAM).
Revised: The anomaly localisation was performed on Nvidia RTX 2070 Super(8GB VRAM).

Original: The total amount of time for running the model constituted around six hours and 18 minutes.
Revised: The total amount of time for running the model constituted around six hours and 18 minutes.

Original: However,this varied between datasets, since they are of different lengths.
Revised: However,this varied between datasets, since they are of different lengths.

Original: On the other hand, the attention disentanglement wasperformed with a Nvidia GTX 1080 Ti GPU (11GB VRAM).
Revised: On the other hand, the attention disentanglement wasperformed with a Nvidia GTX 1080 Ti GPU (11GB VRAM).

Original: The experiment 1 and 2 took around 40 hours each to becompleted, while the third was about 75 hours.
Revised: The experiment 1 and 2 took around 40 hours each to becompleted, while the third took about 75 hours.

Original:  Results
Revised:  Results

Revised: [Figure]

Original: This section describes the results that were obtained from the experiments.
Revised: This section describes the results that were obtained from the experiments.

Original: 4.1 shows the results regarding the claimsmade by the original paper.
Revised: 4.1 shows the results regarding the claimsmade by the original paper.

Original: Andthe section 4.2 shows the results from the ablation studies performed on the originalpaper.
Revised: And section 4.2 shows the results from the ablation studies performed on the original paper.

Original: For visualizing the ﬁgures with more detail, please refer to the following Github repository. 
Revised: For visualizing the ﬁgures with more detail, please refer to the following Github repository. [Figure]

Original: 4.1 Results reproducing original paper
Revised: 4.1 Results reproducing original paper

Original: [Figure]
Revised: [Figure]

Original: 4.1.1 Anomaly detection
Revised: 4.1.1 Anomaly detection

Original: [Figure]
Revised: [Figure]

Original: Figure 1 illustrates qualitative results from the MNIST dataset.
Revised: Figure 1 illustrates qualitative results from the MNIST dataset.

Original: Figure 1a show attention maps for a model trained oninlier digit 1, and evaluated on digit 7, 4, 9 and 2, as done in the original paper.
Revised: Figure 1a show attention maps for a model trained oninlier digit 1, and evaluated on digit 7, 4, 9 and 2, as done in the original paper.

Original: Figure 1b shows the attention mapsfrom a model trained on digits 3, and evaluated on digit 5 and 8.
Revised: Figure 1b shows the attention mapsfrom a model trained on digits 3, and evaluated on digit 5 and 8.

Original: We achieve similar anomaly maps to the original paper,for instance horizontal bars are highlighted as anomalous regions when trained on digits 1.
Revised: We achieve similar anomaly maps to the original paper,for instance horizontal bars are highlighted as anomalous regions when trained on digits 1.

Original: Figure 2 compares the qualitative results from the original paper with our results, with similar examples shown for theanomaly attention maps and target masks.
Revised: Figure 2 compares the qualitative results from the original paper with our results, with similar examples shown for theanomaly attention maps and target masks.

Original: We observe that though very similar, our attention maps are less preciselylocalised around the anomaly.
Revised: We observe that though very similar, our attention maps are less preciselylocalised around the anomaly.

Original: Additionally, the AUROC our model achieved was lower (0.8731 as opposed to 0.92)than reported in the paper.
Revised: Additionally, the AUROC our model achieved was lower (0.8731 as opposed to 0.92)than reported in the paper.

Original: Furthermore, for all convolutional layers to which we back-propagate, we achieved the exactsame AUROC.
Revised: Furthermore, for all convolutional layers to which we back-propagate, we achieved the exactsame AUROC.

Original: The MVTec-AD AUROC and IoU scores are similar to that of the paper (see  2).
Revised: The MVTec-AD AUROC and IoU scores are similar to that of the paper (see Table 2).

Original: In ﬁgure 3 various sample comparisonsare shown side-by-side with samples from the original paper.
Revised: In ﬁgure 3 various samplecomparisons are shown side-by-side with samples from the original paper.

Original: We achieved AUROC scores that were close enough,however, the IoU scores on average were much higher.
Revised: We achieved AUROC scores that were closeenough, however, the IoU scores on average were much higher.

Original: Perhaps this is due to a different reduction method used tocalculate the overall IoU of the test set, or different test sets that were used.
Revised: Perhaps this is due to a different reduction method usedto calculate the overall IoU of the test set, or different test sets that were used.

Original: Qualitatively, the anomaly maps look verysimilar to the original results for most categories, especially as hazelnut, leather and wood.[Table] [Figure]
Revised: Qualitatively, the anomaly maps lookvery similar to the original results for most categories, especially as hazelnut, leather and wood.

Original: 4.1.2 Latent space disentanglement
Revised: 4.1.2 Latent space disentanglement

Original: [Figure] The ﬁrst attention disentanglement experiment results are depicted in ﬁgure 4.
Revised: The ﬁrst attention disentanglement experiment results are depicted in ﬁgure 4.

Original: The trade-off between the reconstructionerror and disentanglement metric for the FactorVAE and the AD-FactorVAE is assessed.
Revised: The trade-off between the reconstructionerror and disentanglement metric for the FactorVAE and the AD-FactorVAE is assessed.

Original: Given our assumption about theattention map selection mechanism, the author’s claim regarding the improvement of the disentanglement metric overstate-of-the-arts is not reproducible.
Revised: Given our assumption about theattention map selection mechanism, the author’s claim regarding the improvement of the disentanglement metric overstate-of-the-arts is not reproducible.

Original: Over training iterations, the disentanglement metric does not offer much insightssince in both approaches the ﬂuctuations are minimal after 100,000 iterations.
Revised: Over training iterations, the disentanglement metric does not offer much insightssince in both approaches the ﬂuctuations are minimal after 100,000 iterations.

Original: In ﬁgure 5 the lowest reconstructionlosses are observed in AD-FactorVAE with γ = 30 and FactorVAE with γ = 20 . 
Revised: In ﬁgure 5 the lowest reconstructionlosses are observed in AD-FactorVAE with γ = 30 and FactorVAE with γ = 20 . [Table] Wood Bottle

Revised: Cable

Original:  The qualitative results present the attention maps of the above highlighted models conﬁgurations with the lowestreconstruction loss and disentanglement metric in ﬁgure 4 (left).
Revised: Capsule [Table] [Figure] The qualitative results present the attention maps of the above highlighted models conﬁgurations with the lowestreconstruction loss and disentanglement metric in ﬁgure 4 (left).

Original: The ﬁrst row shows three input images and the nextfour rows describe the ﬁrst and second highest response attentions maps for FactorVAE and AD-FactorVAE.
Revised: The ﬁrst row shows three input images and the nextfour rows describe the ﬁrst and second highest response attentions maps for FactorVAE and AD-FactorVAE.

Original: The
Revised: The

Original: FactorVAE attention maps show the highest response in similar regions where the sprite is visible.
Revised: FactorVAE attention maps show the highest response in similar regions where the sprite is visible.

Original: However, in twosituation this is not the case and the attention is spread.
Revised: However, in twosituation this is not the case and the attention is spread.

Original: On the AD-FactorVAE attention maps, the attention is mostly inthe same region indicating that they did not disentangle.
Revised: On the AD-FactorVAE attention maps, the attention is mostly inthe same region indicating that they did not disentangle.

Original: Given the insights from the anomaly experiments, we knowthat the layer selected for computing the attention maps has a considerable inﬂuence.
Revised: Given the insights from the anomaly experiments, we knowthat the layer selected for computing the attention maps has a considerable inﬂuence.

Original: In this case we used the lastconvolutional layer of the encoder, which turned out to be detrimental.
Revised: In this case we used the lastconvolutional layer of the encoder, which turned out to be detrimental.

Original: 4.2 Results beyond original paper
Revised: 4.2 Results beyond original paper

Original: 4.2.1 Ablation study
Revised: 4.2.1 Ablation study

Original: The results for the second disentanglement experiment are shown in ﬁgure  are very similar in performance.
Revised: The results for the second disentanglement experiment are shown in ﬁgure 6. The disentanglement metric depictedon the left shows that all the gamma values are very similar in performance.

Original:  observing the reconstruction loss  and 
Revised: By observing the reconstruction loss itbecomes visible that γ = 25 and γ = 30 have lower reconstruction errors, at least with λ = 1 . 0 . Additionally, the truetotal correlation loss ﬂuctuates close to 0.

Original: Figure 7 shows the results for experiment 3.
Revised: Figure 7 shows the results for experiment 3.

Original: On the disentanglement metric there is a separation where the γ = 1 .
Revised: On the disentanglement metric there is a separation where the γ = 1 .

Original: 5shows higher disentanglement.
Revised: 5shows higher disentanglement.

Original: This trend is veriﬁed by the lower reconstruction loss for γ = 20 , λ = 1 . 5 andγ = 30 , λ = 1 . 5 .
Revised: This trend is veriﬁed by the lower reconstruction loss for γ = 20 , λ = 1 . 5 andγ = 30 , λ = 1 . 5 .

Original: There appears to be a smaller total correlation ﬂuctuation compared to the λ = 1 . 0 counterparts.
Revised: There appears to be a smaller total correlation ﬂuctuation compared to the λ = 1 . 0 counterparts.

Original: These are no deﬁnitive results and further experiments with higher λ values should be performed to build strongerpremises.
Revised: These are no deﬁnitive results and further experiments with higher λ values should be performed to build strongerpremises.

Original: 4.2.2 Anomaly attention maps with Stacked RBMs on MNIST
Revised: 4.2.2 Anomaly attention maps with Stacked RBMs on MNIST

Original: The results from the stacked RBM do not seem to show meaningful information regarding potential anomalous regionsin the image as with the VAE anomaly attention maps.
Revised: The results from the stacked RBM do not seem to show meaningful information regarding potential anomalous regionsin the image as with the VAE anomaly attention maps.

Original: We suspect this is caused by the use of fully connected layers inwhich the weights are not spatially correlated, as opposed to convolutional layer.
Revised: We suspect this is caused by the use of fully connected layers inwhich the weights are not spatially correlated, as opposed to convolutional layer.

Original: Therefore, the results might not be asinterpretable.
Revised: Therefore, the results might not be asinterpretable.

Original: For more information and some example results see Appendix D ﬁgure 12.
Revised: For more information and some example results see supplementary material D ﬁgure 12.

Original:  Discussion
Revised:  Discussion

Original: Reproducing the results over all went seemingly well.
Revised: Reproducing the results over all went seemingly well.

Original: Most of the qualitative results for the experiment on the anomalydetection experiments were in line of the results from the original paper.
Revised: Most of the qualitative results for the experiment on the anomalydetection experiments were in line with the results from the original paper.

Original: Thus, for anomaly detection we conﬁrm claim(i) the results from the MNIST dataset are quantitatively good.
Revised: Thus, for anomaly detection we conﬁrmclaim (i) the results from the MNIST dataset are quantitatively good.

Original: Furthermore, claim (iii) is also conﬁrmed, since our MVTec-AD results were similar to the results in the original paper.
Revised: Furthermore, claim (iii) is also conﬁrmed, sinceour MVTec-AD results were similar to the results in the original paper.

Original: However, claim (ii) for the UCSD dataset, was not reproducible, since the attention maps were qualitatively not localisedenough and the quantitative re AUROC scores were more in line with the baseline.
Revised: However, claim (ii) for the UCSD dataset, was not reproducible, since the attention maps were qualitatively not localisedenough and the quantitative re AUROC scores were more in line with the baseline.

Original: Regarding disentanglement claim (iv) the results appear to be irreproducible given our assumption about the attentionmap selection strategy, since our results deviate strongly from the results in the original paper.
Revised: Regarding disentanglement claim (iv) the results appear to be irreproducible given our assumption about the attentionmap selection strategy, since our results deviate strongly from the results in the original paper.

Original: For claim (v) the resultscannot be validated, since the convolutional layer used for generating the attention maps in the encoder was not properlyselected.
Revised: For claim (v) the resultscannot be validated, since the convolutional layer used for generating the attention maps in the encoder was not properlyselected.

Original: In the next sections, the limits of the original paper and the difﬁculties as well as ease will be discussed.
Revised: In the next sections, the limits of the original paper and the difﬁculties as well as ease will be discussed.

Original: 5.1 What was easy
Revised: 5.1 What was easy

Original: The easier part of the experiment was the ﬁrst part, since the code was readily available and similar results were quicklyreproduced.
Revised: The easier part of the experiment was the ﬁrst part, since the code was readily available and similar results were quicklyreproduced.

Original: 5.2 What was difﬁcult
Revised: 5.2 What was difﬁcult

Original: The most difﬁcult part of reproducing this paper was that much of the information was left up to the interpretation of thereader.
Revised: The most difﬁcult part of reproducing this paper was that much of the information was left up to the interpretation of thereader.

Original: For example, there is no explicit documentation about the implementation in the paper.
Revised: For example, there is no explicit documentation about the implementation in the paper.

Original: Hyperparameters (e.g.value of λ , number of epochs and the learning rate), experiment set-up, dataset augmentation and model descriptionsare not present.
Revised: Hyperparameters (e.g.value of λ , number of epochs and the learning rate), experiment set-up, dataset augmentation and model descriptionsare not present.

Original: The only presence of this information is in a referral to the gradcam paper (Selvaraju et al., 2017), inwhich the implementation is explained in more detail.
Revised: The only presence of this information is in a referral to the gradcam paper (Selvaraju et al., 2017), inwhich the implementation is explained in more detail.

Original: Additionally, there were many discrepancies between the paperand the codebase.
Revised: Additionally, there were many discrepancies between the paperand the codebase.

Original: For example, in the original paper, equation 5 for the attention disentanglement loss, is not clear onthe selection mechanism for the attention maps A 1 ij and A 2 ij in equation 1 (our solution for this is explained in section3.3).
Revised: For example, in the original paper, equation 5 for the attention disentanglement loss, is not clear onthe selection mechanism for the attention maps A 1 ij and A 2 ij in equation 1 (our solution for this is explained in section3.3).

Original: For more information and solutions on this see section 3.4.
Revised: For more information and solutions on this see section 3.4.

Original: Furthermore, in the paper a second approach for computing the score to backpropagate on for the attention maps inanomaly detection is presented.
Revised: Furthermore, in the paper a second approach for computing the score to backpropagate on for the attention maps inanomaly detection is presented.

Original: However, this method was not used in their paper and the equation (like many others)was not sufﬁciently explained; a u is mentioned, but there is no mention of what this u might refer to.
Revised: However, this method was not used in their paper and the equation (like many others)was not sufﬁciently explained; a u is mentioned, but there is no mention of what this u might refer to.

Original: 5.3 Communication with original authors
Revised: 5.3 Communication with original authors

Original: Some communication with the author was had through the use of issues in the author’s Github repository.
Revised: Some communication with the author was had through the use of issues in the author’s Github repository.

Original: Some of thecode was not running correctly due to different version of pytorch conﬂicting with each other hence the communicationwith the authors was made only in that occasion. 
Revised: Some of thecode was not running correctly due to different version of pytorch conﬂicting with each other hence the communicationwith the authors was made only in that occasion. AppendixA Dataset information

Revised: The detailed labels distribution for some of the anomaly detection datasets is described in ﬁgure 10.

Revised: B Details of Anomaly localisation experimentsB.1 Architecture

Revised: For the models used on MNIST and UCSD pedestrians, no explicit model architecture was speciﬁed.

Revised: However, thecodebase from the authors did provide a simple VAE architecture to be used on MNIST, which provided decentqualitative results.

Revised: We adapted this model to ﬁt the resized UCSD pedestrian images, by adding an extra convolutionallayer.

Revised: Initially, adding two additional convolutional layers to the existing two convolutional layers seemed logical,as to help reduce the number of parameters.

Revised: This resulted in poor performance relative to the results reported in thepaper, with the attention maps highlighting large regions of the pedestrian paths, instead of localizing its attentionon smaller regions.

Revised: Instead, the better architecture seemed to be 3 total convolutional layers for the encoder, ratherthan 4.

Revised: Although the exact model used on the UCSD dataset is unclear, we could infer that parts of the model werecorrect, as the ﬁrst 3 convolutional layers are supposed to output shapes of 50x50, 25x25 and 12x12, which was thecase after the addition of the third convolutional layer.

Revised: The model used on the MVTec-AD dataset was said to be a

Revised: ResNet-18 based VAE.

Revised: Even though the ResNet-18 architecture is a well deﬁned model, this is usually the case forclassiﬁcation problems, whereas in the case of attention generation, we require a ResNet-18 based encoder and decodermodel.

Revised: To build the ResNet-18 Encoder/Decoder, we used a ResNet-18 VAE implementation from PyTorch Lightning

Revised: Bolts, which replaces the last two layers in the encoder to map to our latent space dimension of 32.

Revised: For the decoder, it isthe same model, but using transpose convolutions and upsampling layers instead.

Revised: C Details of Disentanglement experimentsC.1 Architecture

Revised: The Variational Autoencoder architecture used in all the experiments is depicted in table 5.

Revised: Additionally, the discriminator is a 6 layers Multilayer Perceptron with leaky ReLU (lReLU) activation that outputs 2 values.

Revised: The lReLU is appliedbetween every two FC layers with a negative slope of 0.2.

Revised: C.2 About training time

Revised: As mentioned previously, there are several motives that led to reduce the training computation time.

Revised: Similarly to (Kimand Mnih, 2019) after 150000 iterations, the variations in the training losses are smaller and the results tend to stabilize(see ﬁgure 11).

Revised: This might be related to the fact with the batch size of 64 images, this training schedule consisted

Revised: MNIST

Revised: UCSD

Revised: Encoder

Revised: Tensor Shape

Revised: Encoder

Revised: Tensor Shape

Revised: Input 28 × 28 binary image

Revised: Input 100 × 100 binary image 4 × 4 conv.

Revised: ReLU 64 , 2 ,14 × 14 ×4 × 4 conv.

Revised: ReLU 64 , 2 ,50 × 50 ×4 × 4 conv.

Revised: ReLU 128 , 2 ,7 × 7 ×4 × 4 conv.

Revised: ReLU 128 , 2 ,25 × 25 ×

Revised: Flatten 5 × 5 conv.

Revised: ReLU 128 , 2 ,12 × 12 ×

Revised: Linear 4 × 4 conv.

Revised: ReLU 128 , 2 ,6 × 6 ×

Revised: ReLU

Revised: Flatten

Revised: Linear

Revised: Linear

Revised: ReLU

Revised: Linear

Revised: Decoder

Revised: Tensor Shape

Revised: Decoder

Revised: Tensor Shape

Revised: Input latent variables ∈ R z

Revised: Input latent variables ∈

Revised: R

Revised: z linear linear

Revised: ReLU

Revised: ReLU linear linear

Revised: ReLU

Revised: ReLU

Revised: Reshape 7 × 7 ×

Revised: Reshape 6 × 6 ×

Revised: ReLU 7 × 7 ×

Revised: ReLU 6 × 6 ×4 × 4 upconv.

Revised: ReLU 64 , 2 , 1 14 × 14 ×4 × 4 upconv.

Revised: ReLU 128 , 2 , 1 12 × 12 ×

Revised: ReLU 14 × 14 ×5 × 5 upconv.

Revised: ReLU 128 , 2 , 1 25 × 25 ×4 × 4 upconv.

Revised: ReLU 1 , 2 ,28 × 28 ×4 × 4 upconv.

Revised: ReLU 64 , 2 ,50 × 50 ×

Revised: Sigmoid 28 × 28 ×4 × 4 upconv.

Revised: 1 , 2 ,100 × 100 ×

Revised: Sigmoid 100 × 100 ×

Revised: , activation function, output shape, stride, padding.

Revised: Moreover, upconv represents the transpose convolution.

Revised: of 13 epochs over the 737280 images which is a considerable training schedule given the dimensions of the dataset.

Revised: Additionally, for better results, given the computational resources available, we decided to prioritize training with morethan one random seed over training longer.

Revised: Hence we decided to train the models with 150000 iterations.

Revised: D Results for Stacked Restricted Boltzmann Machine extension

Revised: Encoder

Revised: Tensor Shape

Revised: Decoder

Revised: Tensor Shape

Revised: Layers

Revised: Input 100 × 100 binary image

Revised: Input latent variables ∈ R z

Revised: Layer7 × 7 conv.

Revised: ReLU 64 , 2 , 3 128 × × 128 ×

Revised: Linear

Revised: BatchNorm 128 × × 128 ×

Revised: Interpolate 8 × 8 ×

Revised: ReLU 128 × × 128 ×3 × 3 MaxPool 64 , 2 ,64 × 64 ×

Revised: Layer 1, Encoder block3 × 3 conv.

Revised: 64 , 1 ,64 × 64 ×3 × 3 conv.

Revised: 512 , 1 , 1 8 × 8 ×

Revised: BatchNorm 64 × 64 ×

Revised: BatchNorm 8 × 8 ×

Revised: ReLU 64 × 64 ×

Revised: Interpolate 8 × 8 ×3 × 3 conv.

Revised: 64 , 1 ,64 × 64 ×3 × 3 conv.

Revised: 256 , 1 , 1 16 × 16 ×

Revised: Batchnorm 64 × 64 ×

Revised: Batchnorm 16 × 16 ×

Revised: Layer 1, Encoder block3 × 3 conv.

Revised: 64 , 1 ,64 × 64 ×3 × 3 conv.

Revised: 256 , 1 , 1 16 × 16 ×

Revised: BatchNorm 64 × 64 ×

Revised: BatchNorm 16 × 16 ×

Revised: ReLU 64 × 64 ×

Revised: ReLU 16 × 16 ×3 × 3 conv.

Revised: 64 , 1 ,64 × 64 ×3 × 3 conv.

Revised: 256 , 1 , 1 16 × 16 ×

Revised: Batchnorm 64 × 64 ×

Revised: BatchNorm 16 × 16 ×

Original: {+Layer 2, Encoder
+}
Revised: {+Layer 2, Encoder
+}

Revised: × 3 conv.

Revised: 128 , 2 ,32 × 32 ×3 × 3 conv.

Revised: 256 , 1 , 1 16 × 16 ×

Revised: BatchNorm 32 × 32 ×

Revised: BatchNorm 16 × 16 ×

Revised: ReLU 32 × 32 ×

Revised: ReLU 16 × 16 ×3 × 3 conv.

Revised: 128 , 1 ,32 × 32 ×

Revised: Interpolate 16 × 16 ×

Revised: Batchnorm 32 × 32 ×3 × 3 conv.

Revised: 128 , 1 , 1 32 × 32 ×

Revised: Downsample 1 × 1 conv.

Revised: 128 , 2 ,32 × 32 ×

Revised: BatchNorm 16 × 16 ×

Revised: BatchNorm 32 × 32 ×

Revised: Interpolate 16 × 16 ×3 × 3 conv.

Revised: 128 , 1 , 1 32 × 32 ×

Revised: Layer 2, Encoder Block 1 3 × 3 conv.

Revised: 128 , 2 ,32 × 32 ×3 × 3 conv.

Revised: 128 , 1 , 1 32 × 32 ×

Revised: BatchNorm 32 × 32 ×

Revised: BatchNorm 32 × 32 ×

Revised: ReLU 32 × 32 ×

Revised: ReLU 32 × 32 ×3 × 3 conv.

Revised: 128 , 1 ,32 × 32 ×

Revised: Interpolate 32 × 32 ×

Revised: Batchnorm 32 × 32 ×3 × 3 conv.

Revised: 128 , 1 , 1 32 × 32 ×

Revised: BatchNorm 32 × 32 ×

Revised: Interpolate 32 × 32 ×3 × 3 conv.

Revised: 128 , 1 , 1 32 × 32 ×

Revised: Layer 3, Encoder Block 0 3 × 3 conv.

Revised: 256 , 2 ,16 × 16 ×3 × 3 conv.

Revised: 128 , 1 , 1 32 × 32 ×

Revised: BatchNorm 16 × 16 ×

Revised: BatchNorm 32 × 32 ×

Revised: ReLU 16 × 16 ×

Revised: ReLU 32 × 32 ×3 × 3 conv.

Revised: 256 , 1 ,32 × 32 ×

Revised: Interpolate 32 × 32 ×

Revised: Batchnorm 16 × 16 ×3 × 3 conv.

Revised: 64 , 1 ,64 × 64 ×

Revised: Downsample 1 × 1 conv.

Revised: 256 , 2 ,16 × 16 ×

Revised: BatchNorm 64 × 64 ×

Revised: BatchNorm 16 × 16 ×

Revised: Layer 3, Encoder Block 1 3 × 3 conv.

Revised: 256 , 2 ,16 × 16 ×3 × 3 conv.

Revised: 64 , 1 ,64 × 64 ×

Revised: BatchNorm 16 × 16 ×

Revised: BatchNorm 64 × 64 ×

Revised: ReLU 16 × 16 ×

Revised: ReLU 64 × 64 ×3 × 3 conv.

Revised: 256 , 1 ,16 × 16 ×3 × 3 conv.

Revised: 64 , 1 ,64 × 64 ×

Revised: Batchnorm 16 × 16 ×

Revised: BatchNorm 64 × 64 ×

Original: {+Layer 4, Encoder
+}
Revised: {+Layer 4, Encoder
+}

Revised: × 3 conv. 512 , 2 ,8 × 8 ×3 × 3 conv.

Revised: 64 , 1 ,32 × 32 ×

Revised: BatchNorm 8 × 8 ×

Revised: BatchNorm 32 × 32 ×

Revised: ReLU 8 × 8 ×

Revised: ReLU 32 × 32 ×3 × 3 conv. 512 , 1 ,8 × 8 ×

Revised: Interpolate 32 × 32 ×

Revised: Batchnorm 8 × 8 ×3 × 3 conv.

Revised: 64 , 1 ,32 × 32 ×

Revised: Downsample 1 × 1 conv.

Revised: 512 , 2 ,8 × 8 ×

Revised: BatchNorm 32 × 32 ×

Revised: BatchNorm 8 × 8 ×

Revised: Interpolate 32 × 32 ×3 × 3 conv.

Revised: 64 , 1 ,64 × 64 ×

Revised: BatchNorm 64 × 64 ×

Original: {+Layer 4, Encoder
+}
Revised: {+Layer 4, Encoder
+}

Revised: × 3 conv. 512 , 1 ,8 × 8 ×3 × 3 conv.

Revised: 64 , 1 ,64 × 64 ×

Revised: BatchNorm 8 × 8 ×

Revised: BatchNorm 64 × 64 ×

Revised: ReLU 8 × 8 ×

Revised: ReLU 64 × 64 ×3 × 3 conv. 512 , 1 ,8 × 8 ×3 × 3 conv.

Revised: 64 , 1 ,64 × 64 ×

Revised: Batchnorm 8 × 8 ×

Revised: BatchNorm 128 × 128 ×

Revised: [Table]

Revised: [Figure]