Original: E XPLAINING THE D ECISIONS OF N EURAL N ETWORKS WITH L ATENT S YMPATHETIC E XAMPLES
Revised: E XPLAINING THE M ISTAKES OF N EURAL N ETWORKS U SING L ATENT S YMPATHETIC E XAMPLES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Neural networks make mistakes.
Revised: Neural networks make mistakes.

Original: The reason why a mistake is made often remains a mystery.
Revised: The reason why a mistake is made often remains a mystery.

Original: As such neural networks often are considered a black box.

Original: It would be useful to have a method that can give an explanation that is intuitive to a user as to why an image is misclassiﬁed.
Revised: It would be useful to have a method that can give an explanation that is intuitive to a user as to why an image is misclassiﬁed.

Original: In this paper we develop a method for explaining the mistakes of a classiﬁer model by visually showing what must be added to an image such that it is correctly classiﬁed
Revised: In this paper we develop a method for explaining the mistakes of an image classiﬁcation model by visually showing what must be added to an image such that it is correctly classiﬁed.

Original: Our work combines the ﬁelds of adversarial examples generative modeling and a correction technique based on difference target propagation to create an technique that creates explanations of why an image is misclassiﬁed
Revised: Our work combines the ﬁelds of adversarial examples, generative modeling and a correction technique based on difference target propagation to create an technique that creates explanations of why an image is misclassiﬁed.

Original: In this paper we explain our method and demonstrate it on MNIST and CelebA. This approach could aid in demystifying neural networks for a user.
Revised: In this paper we explain our method and demonstrate it on MNIST and CelebA. This approach could aid in demystifying neural networks for a user.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Given the increasingly widespread use of deep learning in real-world applications, it is becoming important to be able to understand why a neural network makes a mistake when it happens.
Revised: Given the increasingly widespread use of deep learning in real-world applications, it has become increasingly important to give explanations for speciﬁc images that are misclassﬁed.

Original: For example, suppose a self-driving car, controlled by a neural network, makes a sudden decision to stop in the middle of the road, because it mis-identiﬁed a tree as a pedestrian who was about to cross.
Revised: For example, suppose a self-driving car, controlled by a neural network, makes a sudden decision to stop in the middle of the road, because it mis-identiﬁed a tree as a pedestrian who was about to cross.

Original: How could we understand that the reason for this decision was a poorly-trained pedestrian detector?
Revised: How could we understand that the reason for this decision was a poorly-trained pedestrian detector?

Original: One approach we could take to“explain” these errors would be to use Gradient Descent learn a perturbation to the misclassiﬁed image such that it is correctly classiﬁed.
Revised: One approach we could take to“explain” these errors would be to use Gradient Descent learn a perturbation to the misclassiﬁed image such that it is correctly classiﬁed.

Original: We could then observe the perturbation and see what needed to be modiﬁed in the input to produce the correct class.
Revised: We could then observe the perturbation and see what needed to be modiﬁed in the input to produce the correct class.

Original: In our example this might correspond to perturbing the car’s camera image at the time of the stop, to minimize the strength of the “stop” output.
Revised: In our example this might correspond to perturbing the car’s camera image at the time of the stop, to minimize the strength of the “stop” output.

Original: We could then observe what change to the image was needed to avoid the sudden stop.
Revised: We could then observe what change to the image was needed to avoid the sudden stop.

Original: We might observe that the tree which was mis-identiﬁed as a person becomes less person-like and conclude that a faulty pedestrian-detector was to blame.
Revised: We might observe that the tree which was mis-identiﬁed as a person becomes less person-like and conclude that a faulty pedestrian-detector was to blame.

Original: This  “Gradient Descent on the input” approach taken by (Szegedy et al., 2013), in which they introduced the notion of adversarial examples.
Revised: This is the “Gradient Descent on the input” approach taken by (Szegedy et al., 2013), in which they introduced the notion of adversarial examples.

Original: Adversarial examples are created by perturbing an image so that it is misclassiﬁed.
Revised: Adversarial examples are created by perturbing an image so that it is misclassiﬁed.

Original: The surprising ﬁnding of this paper was that it is almost always possible to create an imperceptibly small perturbation to an image that changes the output class to any target value.
Revised: The surprising ﬁnding of this paper was that it is almost always possible to create an imperceptibly small perturbation to an image that changes the output class to any target value.

Original: These perturbations tend to look like white noise to the naked eye, and tell us nothing about what caused the image to be classiﬁed as it was.
Revised: These perturbations tend to look like white noise to the naked eye, and tell us nothing about what caused the image to be classiﬁed as it was.

Original: Ideally, for the user to understand why a certain image is misclassiﬁed, the perturbations should be constrained to only the parts of an image relevant to the class and must be interpretable to the user.
Revised: Ideally, for the user to understand why a certain image is misclassiﬁed, the perturbations should be constrained to only the parts of an image relevant to the class and must be interpretable to the user.

Original: In this paper, we generate explanations that align with human perception and are meaningful using generative models that perturb the features of a misclassiﬁed image such that it is correctly classiﬁed.
Revised: In this paper, we generate explanations that align with human perception and are meaningful using generative models that perturb the features of a misclassiﬁed image such that it is correctly classiﬁed.

Original: The paper is structured in the following manner.
Revised: The paper is structured in the following manner.

Original: In section 2 we introduce our proposed method.
Revised: In section 2 we introduce our proposed method.

Original: In section 3 the experiments are discussed.
Revised: In section 3 the experiments are discussed.

Original: Finally, in section 4 we discuss the related work and in section 5 we will conclude.
Revised: Finally, in section 4 we discuss the related work and in section 5 we will conclude.

Original:  M ETHODN EW 2. T ERMINOLOGY
Revised:  M ETHOD 2. T ERMINOLOGY

Original: Our method (described visually in Figure 1), consists of a classiﬁer ( c ( · ) ) and a generative model, both trained separately on the same dataset.
Revised: Our method (described visually in Figure 1), consists of a classiﬁer ( c ( · ) ) and a generative model, both trained separately on the same dataset.

Revised: We refer to the discrete classes predicted by the classiﬁer as c ( · ) and the class probabilities as p c ( · ) .

Original: We use the cross entropy error for our classiﬁer denoted as CE ( · , y ) .
Revised: We use the cross entropy error for our classiﬁer denoted as CE ( · , y ) .

Original: The generative model consists of an encoder ( e ( · )) and a generator ( g ( · )) .
Revised: The generative model consists of an encoder ( e ( · )) and a generator ( g ( · )) .

Original: We refer to the original image as x and the correct class for the original image as y. Once the original image has been encoded to the latent space of the generative model ( e ( x )) , we refer to it as z .
Revised: We refer to the original image as x and the correct class for the original image as y. Once the original image has been encoded to the latent space of the generative model ( e ( x )) , we refer to it as z .

Original: The reconstruction of z we call the reconstructed image and refer to as g ( z ) .
Revised: The reconstruction of z we call the reconstructed image and refer to as g ( z ) .

Original: Furthermore, we call z + (cid:15) the perturbation and g ( z + (cid:15) )
Revised: Furthermore, we call z + (cid:15) the perturbation and g ( z + (cid:15) )

Original: the perturbed reconstruction .
Revised: the perturbed reconstruction .

Original: Lastly, we call the result of our method x (cid:48) the perturbed image.
Revised: Lastly, we call the result of our method x (cid:48) the perturbed image .

Original: 2. O UR M ETHOD
Revised: 2. O UR M ETHOD

Original: Formally, the problem we try to solve can be deﬁned in the following manner.
Revised: Formally, the problem we try to solve can be deﬁned in the following manner.

Original: We have a misclassiﬁed image x
Revised: We have a misclassiﬁed image x

Original: (i.e.. c ( x ) (cid:54) =
Revised: (i.e.. c ( x ) (cid:54) =

Original: y ).
Revised: y ).

Original: We wish to generate x (cid:48) , a perturbed version of x , such that c ( x (cid:48) ) =
Revised: We wish to generate x (cid:48) , a perturbed version of x , such that c ( x (cid:48) ) =

Original: y .
Revised: y .

Original: We deﬁne the perturbed reconstruction according to equation 1.
Revised: We deﬁne the perturbed reconstruction according to equation 1.

Original: [Equation]
Revised: [Equation]

Original: We can see that if (cid:15) is 0, we are left with x .
Revised: We can see that if (cid:15) is 0, we are left with x .

Original: As such, so long as g is smooth, the condition lim (cid:15) → 0 x (cid:48) = x holds.
Revised: As such, so long as g is smooth, the condition lim (cid:15) → 0 x (cid:48) = x holds.

Original: This equation is inspired by Difference Target Propagation, proposed by (Lee et al., 2015), where they used it to propagate a perturbation backwards through a network.
Revised: This equation is inspired by Difference Target Propagation, proposed by (Lee et al., 2015), where they used it to propagate a perturbation backwards through a network.

Original: The goal of our method is to ﬁrstly, minimize the Cross Entropy: CE (  c ( x (cid:48) ) , y ) and secondly to keep the euclidean norm of (cid:15) as small as possible to limit the amount of features added to the original image.
Revised: The goal of our method is to ﬁrstly, minimize the Cross Entropy: CE ( p c ( x (cid:48) ) , y ) and secondly to keep the euclidean norm of (cid:15) as small as possible to limit the amount of features added to the original image.

Original: We accomplish the ﬁrst goal by using gradient descent to learn the perturbation (cid:15) .
Revised: We accomplish the ﬁrst goal by using gradient descent to learn the perturbation (cid:15) .

Original: As such epsilon is updated iteratively by formula 2.
Revised: As such epsilon is updated iteratively by formula 2.

Original: Where λ is a small constant learning rate.
Revised: Where λ is a small constant learning rate.

Original: [Equation]
Revised: [Equation]

Original: The second goal is accomplished by stopping the perturbation of (cid:15) once the condition c ( x (cid:48) )
Revised: The second goal is accomplished by stopping the perturbation of (cid:15) once the condition c ( x (cid:48) )

Original: =
Revised: =

Original: y is met.
Revised: y is met.

Original: A visual representation of our method is shown in ﬁgure 1.
Revised: A visual representation of our method is shown in ﬁgure 1.

Original: Because we are minimizing the CE (  x (cid:48)  , y ) using a generative model as opposed to maximizing it, we say that our method generates Latent Sympathetic Examples (LSE).
Revised: Because we are minimizing the CE ( p c ( x (cid:48) ) , y ) using a generative model as opposed to maximizing it, we say that our method generates Latent Sympathetic Examples (LSE).

Original: For this research we use a variational auto encoder (Kingma & Welling, 2013) as a generative model for MNIST and a combination of a variational auto encoder with a Generative Adversarial Network (Larsen et al., 2015) as a generative model for CelebA.
Revised: For this research we use a variational auto encoder (Kingma & Welling, 2013) as a generative model for MNIST and a combination of a variational auto encoder with a Generative Adversarial Network (Larsen et al., 2015) as a generative model for CelebA.

Original: 2. D EFINING WHAT IT MEANS TO EXPLAIN A CLASSIFICATION
Revised: 2. D EFINING WHAT IT MEANS TO EXPLAIN A CLASSIFICATION

Original: Various deﬁnitions for explaining classiﬁer decisions exist.
Revised: Various deﬁnitions for explaining classiﬁer decisions exist.

Original: We deﬁne an explanation for a classiﬁer decision as; the perturbation that must be made to an image such that it is correctly classiﬁed.
Revised: We deﬁne an explanation for a classiﬁer decision as; the perturbation that must be made to an image such that it is correctly classiﬁed.

Original: Subject to the conditions that:
Revised: Subject to the conditions that:

Original: The added parts are interpretable and  visible by the naked eye.
Revised: The added parts are interpretable and preferably visible by the naked eye.

Original: The perturbation is  close  to the original encoded point in the latent space of the generative model.
Revised: The perturbation is as close as possible to the original encoded point in the latent space of the generative model.

Original: This is measured as the euclidean norm of (cid:15) .
Revised: This is measured as the euclidean norm of (cid:15) .

Revised: Our method is encouraged to keep the euclidean norm of (cid:15) to a minimum.

Original: As such our methodtries to add only  those parts of the reconstruction that are most relevant for the original image to be perturbed into the correct class.
Revised: This is done such that method only adds those parts of the reconstruction that are most relevant for the original image to be perturbed into the correct class.

Original: [Figure]
Revised: This is relevant because otherwise the method could project a prototypical class example over the original image.

Original: [Figure]
Revised: [Figure]

Original: reconstruction error ε z=e(x) z+ε g(z+ε)
Revised: reconstruction error ε z=e(x) z+ε g(z+ε)

Original: x’ = g(z+ε)
Revised: x’ = g(z+ε)

Original: + x - g(z) x - s pa c e z - s pa c e y c(x’)
Revised: + x - g(z) x - s pa c e z - s pa c e y c(x’)

Original: (cid:7040) y - s pa c e
Revised: (cid:7040) y - s pa c e

Original:  E XPERIMENTS 3. MNIST
Revised:  E XPERIMENTS 3. MNIST

Original: We trained two different Variational Auto Encoders on MNIST.
Revised: We trained two different Variational Auto Encoders on MNIST.

Original: The ﬁrst had a 2 dimensional latent space, the second a 10 dimensional latent space.
Revised: The ﬁrst had a 2 dimensional latent space, the second a 10 dimensional latent space.

Original: We used the former in order to create a latent space that was easy to display visually, alongside the CE (  c ( x (cid:48) ) , y ) errors.
Revised: We used the former in order to create a latent space that was easy to display visually, alongside the CE ( p c ( x (cid:48) ) , y ) errors.

Original: We used the latter to determine the impact of the dimensionality of the latent space on the rate of success of our method to push images into another class .
Revised: We used the latter to determine the impact of the dimensionality of the latent space on the rate of success of our method to push images into another class.

Original: We attached a classiﬁer to the variational auto encoders and trained it without updating the weights of the VAE’s.
Revised: We attached a classiﬁer to the variational auto encoders and trained it without updating the weights of the VAE’s.

Original: In order to get an intuition of how our method works, we discuss the results created using the 2-D VAE and their respective conclusions.
Revised: In order to get an intuition of how our method works, we discuss the results created using the 2-D VAE and their respective conclusions.

Original: Firstly, the predictions made by the classiﬁer from the g ( z ) may differ from the predictions made using x (cid:48) .
Revised: Firstly, the predictions made by the classiﬁer from the g ( z ) may differ from the predictions made using x (cid:48) .

Original: In order to illustrate this we have plotted the predictions c ( g ( z )) and the predictions of c ( x (cid:48) ) for various values of z and (cid:15) , respectively.
Revised: In order to illustrate this we have plotted the predictions c ( g ( z )) and the predictions of c ( x (cid:48) ) for various values of z and (cid:15) , respectively.

Original: c ( g ( z )) is shown in ﬁgure 2 and c ( x (cid:48) ) is shown in ﬁgure 3.
Revised: c ( g ( z )) is shown in ﬁgure 2 and c ( x (cid:48) ) is shown in ﬁgure 3.

Original: As can be seen the predicted values are quite different.
Revised: As can be seen the predicted values are quite different.

Original: Secondly, our method may not be able to always perturb to the target class.
Revised: Secondly, our method may not be able to always perturb to the target class.

Original: Figure 3 shows that our method may not work for all cases.
Revised: Figure 3 shows that our method may not work for all cases.

Original: The class ‘1‘ is missing for this image.
Revised: The class ‘1‘ is missing for this image.

Original: If this is the target class, then our method will not be able to generate a LSE for it.
Revised: If this is the target class, then our method will not be able to generate a LSE for it.

Original: Furthermore, our method sometimes gets stuck in a local minimum.
Revised: Furthermore, our method sometimes gets stuck in a local minimum.

Original: In that case it may be possible to create an LSE, however the method is unable to reach it.
Revised: In that case it may be possible to create an LSE, however the method is unable to reach it.

Original: An example of this happening is shown in appendix B.
Revised: An example of this happening is shown in appendix B.

Original: In order to illustrate how our our method works part of the latent space of a successfully perturbed image has been plotted in ﬁgure 4.
Revised: In order to illustrate how our our method works part of the latent space of a successfully perturbed image has been plotted in ﬁgure 4.

Original: This ﬁgure also demonstrates how the perturbation is visible in the image space.
Revised: This ﬁgure also demonstrates how the perturbation is visible in the image space.

Original: As can be seen the image quality is quite coarse.
Revised: As can be seen the image quality is quite coarse.

Original: The  10-D latent space yields qualitatively better results.
Revised: The VAE with the 10-D latent space yields qualitatively better results.

Original: These can be found in appendix A.
Revised: Appendix A plots the perturbed image at each gradient step for a VAE with a 10-D latent space.

Original:  E FFECTS OF THE DIMENSIONALITY OF THE LATENT SPACE
Revised: 3. E FFECTS OF THE DIMENSIONALITY OF THE LATENT SPACE

Original: Another goal of our research is to explore what the relation between the dimensionality of the latent space of the VAE is and effectiveness of our method to generate LSE‘s.
Revised: Another goal of our research is to explore what the relation between the dimensionality of the latent space of the VAE is and effectiveness of our method to generate LSE‘s.

Original: We conduct 2 experiments in order to investigate this.
Revised: We conduct 2 experiments in order to investigate this.

Original: Firstly, we perturb a set of misclassiﬁed examples and calculate the success rate.
Revised: Firstly, we perturb a set of misclassiﬁed examples and calculate the success rate.

Original: We deﬁne the success rate as what percentage of the time our method succeeds in ﬁnding a perturbation which pushes the perturbed image to the correct class.
Revised: We deﬁne the success rate as what percentage of the time our method succeeds in ﬁnding a perturbation which pushes the perturbed image to the correct class.

Original: Secondly, we try to use our method to generate latent adversarial examples (LAE’s), we take correctly classiﬁed images and try to perturb them to another class.
Revised: Secondly, we try to use our method to generate latent adversarial examples (LAE’s), we take correctly classiﬁed images and try to perturb them to another class.

Original: We verify that our method should have more difﬁculty generating
Revised: We verify that our method should have more difﬁculty generating

Original: LAE’s.
Revised: LAE’s.

Original: However, it is only ineffective at generating LAE’s using the 2D latent space, not the 10D latent space.
Revised: However, it is only ineffective at generating LAE’s using the 2D latent space, not the 10D latent space.

Revised: Finally we qualitatively review successfully generated LSE’s when the VAE has aand 10 dimensional latent space.

Original: 3.2. E FFECT OF  DIMENSIONALITY  VAE ON  SUCCESS RATE  LSE
Revised: 3.2. E FFECT OF THE DIMENSIONALITY OF THE LATENT SPACE OF THE VAE ON THE SUCCESS RATE OF FINDING A LSE

Original: In order to explore what the impact of the dimensionality of the latent space is on the success rate of LSE generation
Revised: In order to explore what the impact of the dimensionality of the latent space is on the success rate of LSE generation

Original: 2 experiments are run.
Revised: 2 experiments are run.

Original: We attempt to perturb 87 misclassiﬁed examples for both experiments to their correct class.
Revised: We attempt to perturb 87 misclassiﬁed examples for both experiments to their correct class.

Original: We used a  λ of 0.1 .
Revised: We used a step size λ of 0.1 .

Original: The method was successful in generating an LSE in 100% of the cases for the 10D VAE.
Revised: The method was successful in generating an LSE in 95.40% of the cases for the 10D VAE.

Original: The average euclidean distance from the original e ( x ) points was 0.22.
Revised: The average euclidean distance from the original e ( x ) points was 0.217.

Original: The 2D VAE successfully perturbed 78.16 % of the misclassiﬁed examples to their target class with an average euclidean distance of 0.10.
Revised: The 2D VAE successfully perturbed 78.16 % of the misclassiﬁed examples to their target class with an average euclidean distance of 0.10.

Original: This is evidence that the dimensionality of the latent space directly impacts the success rate positively.
Revised: This is evidence that the dimensionality of the latent space directly impacts the success rate of ﬁnding an LSE positively

Original:  3.2. E FFECT OF  DIMENSIONALITY  VAE ON  SUCCESS RATE  LAE
Revised: . 3.2. E FFECT OF THE DIMENSIONALITY OF THE LATENT SPACE OF THE VAE ON THE SUCCESS RATE OF FINDING AN LAE

Original: In this experiment we want to check whether this method has more success in creating an LSE then creating a LAE.
Revised: In this experiment we want to check whether this method has more success in creating an LSE then creating a LAE.

Original: Therefore, an experiment is run where the goal is to perturb 100 correctly classiﬁed examples to a class different from their target class.
Revised: Therefore, an experiment is run where the goal is to perturb 100 correctly classiﬁed examples to a class different from their target class.

Original: Before generating the LAE’s the images which already had the label that was being targeted were removed.
Revised: Before generating the LAE’s the images which already had the label that was being targeted were removed.

Original: As such each of theresulting experiments (each of the MNIST labels was targeted), had a different number of images.
Revised: As such each of theresulting experiments (each of the MNIST labels was targeted), had a different number of images.

Original: The weighted average success rate of the 2-D and 10-D VAE‘s are 22.34 % and 91.11% respectively, the weighted average L-2 distances are 0.52 and 1.53, respectively.
Revised: The weighted average success rate of the 2-D and 10-D VAE‘s are 22.34 % and 91.11% respectively, the weighted average L-2 distances are 0.52 and 1.53, respectively.

Original: It can be seen that the average success rates for generating latent adversarial examples for the 2-D and 10-D VAE differ by quite a large margin.
Revised: It can be seen that the average success rates for generating latent adversarial examples for the 2-D and 10-D VAE differ by quite a large margin.

Original: An explanation for this difference is that it is harder to get stuck in a 10-D latent space as compared to a 2-D latent  space.
Revised: An explanation for this difference is that it is harder to get stuck in a 10-D latent space as compared to a 2-D latent space, with their being more ways “down” in the error space.

Original: Furthermore, it can be seen that the LSE generation is more effective compared to the LAE generation.
Revised: Furthermore, it can be seen that the LSE generation is more effective compared to the LAE generation.

Original: This may be due to the distance being traveled in the latent space.
Revised: This may be due to the distance being traveled in the latent space.

Original: Because the target class in the LAE case is farther away then it is for LSEs on average, the chances of getting stuck in a local minimum are higher. 
Revised: Because the target class in the LAE case is farther away then it is for LSEs on average, the chances of getting stuck in a local minimum are higher. [Figure]

Original: 3. C ELEB A
Revised: 3.2. Q UALITATIVE IMPACT OF THE DIMENSIONALITY OF THE LATENT VARIABLE MODEL ON THE PERTURBED RECONSTRUCTION

Revised: In order for our method to succeed perturbations to the original image must ﬁrstly, be meaningful to a person and secondly be close to the original encoded point. The second condition is there to encourage that the perturbations added to the original image are limited to only those that are necessary. Other factors that can inﬂuence the size of the perturbations made in the image space are the quality of the reconstructions.

Revised: A sample of perturbed images using a VAE with a latent space of 2-D and 10-D is shown in ﬁgure 5.

Revised: These perturbations are visible by the naked eye.

Original:  In order to test whether this method can be applied to get meaningful results on  a more complicated dataset we applied this method on CelebA.
Revised: This image shows that the perturbations done by the latent variable model which used a 2-D latent space are much more coarse then its 10-D counterpart. Furthermore, the perturbations that were done using the latent variable model with the 10-D latent space are more precise. In the case of the four in the second row, the lower part of the long stroke on the left is dimmed a bit. The six in the third row has part of its circle dimmed and the top stroke is elongated and the eight in the ﬁnal row that was misclassiﬁed as a zero was made less round by dimming some pixels in the middle and on the sides.

Original: We trained a VAEGAN with a 1024-D latent space on the 
Revised: The perturbations done by the model with the 2-D latent space sometimes are less precise. For example, the image in the top row has a six projected over it and the six in the third row is ﬁlled. These images are nonetheless, after this perturbation, correctly classiﬁed.

Original: [Figure]
Revised: [Figure]

Original:  dataset.
Revised: In order to test whether this method can be applied to get meaningful results on a more complicated dataset we applied this method on CelebA. We trained a VAEGAN with a 1024-D latent space on the dataset.

Original: We trained several classiﬁers on the traits; “Receding Hairline”, “Big Lips”,  “Wearing Earrings” and “Eyeglasses”.
Revised: We trained several classiﬁers on the traits; “Male”, “Big Lips”, “Receding Hairline”, “Wearing Earrings”, “Eyeglasses”, “Wearing Necktie”, “Mustache”, “Sideburns”, “Smiling” and “Rosy Cheeks”.

Original: Several examples of successfully explained examples are shown in ﬁgure 5.
Revised: Several examples of successfully explained examples are shown in ﬁgure 6.

Original: It can be seen that the method gives plausible explanations as to why an image is misclassiﬁed.
Revised: It can be seen that the method gives plausible explanations as to why an image is misclassiﬁed.

Revised: Some are visible only when looking at the perturbation ( x (cid:48) − x ) .

Revised: Such explanations also have the potential to be useful in ﬁnding potential biases in the dataset.

Revised: For example in image 10 in appendix B, images of males with long hair 1 are shown that were misclassiﬁed as female.

Revised: By looking at a set of perturbations it becomes possible to ﬁnd traits, which datapoints that are added to a dataset must have in order to remove bias.

Revised: In this case that trait should be males with long hair.

Original: We want to see whether the success depends on the trait.
Revised: We want to see whether the success depends on the trait.

Original: To this end we again calculated the success rates.
Revised: To this end we again calculated the success rates.

Original: These are shown in ﬁgure 1.
Revised: These are shown in ﬁgure 1.

Original: As can be seen the success rate varies per trait, with the trait “ Big Lips” performing signiﬁcantly worse compared to the other traits.
Revised: As can be seen the success rate varies per trait, with the trait “Mustache” performing signiﬁcantly worse compared to the other traits.

Original: Furthermore, it is also interesting to see how the imagechanged as it was being perturbed to its correct class, this is shown in ﬁgure 6.
Revised: We also show an image being perturbed to its correct class, this is shown in ﬁgure 8, appendix A.

Original: [Table]
Revised: [Table]

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: The work done in our research is related to adversarial examples.
Revised: The work done in our research is related to adversarial examples.

Original: Adversarial examples were introduced by (Szegedy et al., 2013).
Revised: Adversarial examples were introduced by (Szegedy et al., 2013).

Original: These examples take advantage of ﬁnding low probability pockets in the manifold on which the images are located which fool the neural network into classifying the image in another class.
Revised: These examples take advantage of ﬁnding low probability pockets in the manifold on which the images are located, which fool the neural network into classifying the image in another class.

Original: As such an optimization problem can be formulated and solved to ﬁnd adversarial examples.
Revised: As such an optimization problem can be formulated and solved to ﬁnd adversarial examples.

Original: Such a problem can be optimized by means of various optimizers.
Revised: Such a problem can be optimized by means of various optimizers.

Original: For example, (Szegedy et al., 2013) used L-BFGS and (Goodfellow et al., 2014) took the sign of the gradients and used those to update the image until the prediction error was maximized.
Revised: For example, (Szegedy et al., 2013) used L-BFGS and (Goodfellow et al., 2014) took the sign of the gradients and used those to update the image until the prediction error was maximized.

Original: The prior two methods generate adversarial attacks in an untargeted fashion, meaning that they simply perturb to another unspeciﬁed class.
Revised: The prior two methods generate adversarial attacks in an untargeted fashion, meaning that they simply perturb to another unspeciﬁed class.

Original: Generating adversarial examples of a particular class is also possible.
Revised: Generating adversarial examples of a particular class is also possible.

Original: An example of this is the Deepfool algorithm (Moosavi-Dezfooli et al., 2016).
Revised: An example of this is the Deepfool algorithm (Moosavi-Dezfooli et al., 2016).

Original: More closely related to our work is the work done by (Kos et al., 2017).
Revised: More closely related to our work is the work done by (Kos et al., 2017).

Original: The authors devise several different adversarial attacks to change the encoding of an image in the latent space of a VAE or VAE-GAN such that the reconstructed image is different in a way that is advantageous to the attacker.
Revised: The authors devise several different adversarial attacks to change the encoding of an image in the latent space of a VAE or VAE-GAN such that the reconstructed image is different in a way that is advantageous to the attacker.

Original: Our work extends the prior work in the sense that it uses gradient descent to create an adversarial example and attempts to create an explanation of why a certain classiﬁcation was made .
Revised: Our work extends the prior work in the sense that it uses gradient descent to create an adversarial example and attempts to create an explanation of why a certain classiﬁcation was made .

Original: However there are two key differences.
Revised: However there are two key differences.

Original: Firstly, as was said before, our method attempts to help the trained classiﬁer to minimize the prediction error, in contrast to adversarial examples.
Revised: Firstly, as was said before, our method attempts to help the trained classiﬁer to minimize the prediction error, in contrast to adversarial examples.

Original: As such it can be said such examples are not adversarial at all but sympathetic to the classiﬁer.
Revised: As such it can be said such examples are not adversarial at all but sympathetic to the classiﬁer.

Original: Secondly, despite one of the goals of adversarial attacks being to better understand the input-output mapping of a neural network, classical adversar- ial examples are non-intuitive to any user.
Revised: Secondly, despite one of the goals of adversarial attacks being to better understand the input-output mapping of a neural network, classical adversarial examples are non-intuitive to any user.

Original: As such our method constrains the search for adversarial examples by means of a trained generative model.
Revised: As such our method constrains the search for adversarial examples by means of a trained generative model.

Original: The intuition being that the search is constrained to features that should be interpretable to a user.
Revised: The intuition being that the search is constrained to features that should be interpretable to a user.

Original: Our work was in part inspired by the Difference Target Propagation technique developed by (Lee et al., 2015).
Revised: Our work was in part inspired by the Difference Target Propagation technique developed by (Lee et al., 2015).

Original: They propose an alternative to backpropagation for assigning credit in deep networks, called “target propagation”.
Revised: They propose an alternative to backpropagation for assigning credit in deep networks, called “target propagation”.

Original: During training, target propagation sends targets, instead of gradients, backwards through the network, where the target is a small perturbation on the activation of the layer from the forward pass.
Revised: During training, target propagation sends targets, instead of gradients, backwards through the network, where the target is a small perturbation on the activation of the layer from the forward pass.

Original: The authors show that so long as layers are invertable, this will minimize the ﬁnal loss.
Revised: The authors show that so long as layers are invertable, this will minimize the ﬁnal loss.

Original: Because layers are in general not invertable, they apply a linear correction (which we borrow in Equation 1) to adjust shift the backpropagating target so that it remains close to the forward activation value.
Revised: Because layers are in general not invertable, they apply a linear correction (which we borrow in equation 1) to adjust shift the backpropagating target so that it remains close to the forward activation value.

Original: We use the same correction in our paper to ensure that as our latent perturbation becomes small, so does our perturbation in image space.
Revised: We use the same correction in our paper to ensure that as our latent perturbation becomes small, so does our perturbation in image space.

Original: i.e. lim (cid:15) →
Revised: i.e. lim (cid:15) →

Original: 0 x (cid:48) = 
Revised: 0 x (cid:48) = x .

Revised: The work in this paper is also related to heatmapping methods.

Original: x
Revised: Heatmapping methods indicate which pixels contributed to a speciﬁc classiﬁcation decision. Speciﬁcally, it is related to the image speciﬁc salieny map technique introduced by Simonyan et al. A downside of this technique is that the heatmaps are noisy due to them providing local explanations (Samek et al., 2017). This means that the irrelevant background parts of an image can also be changed in order to increase the classiﬁcation score for the target class.

Revised: For example, an image of a tree on an empty ﬁeld of grass can be made more “tree” like for the classiﬁer by putting tree like structures on the empty ﬁeld.

Revised: Our method often produces more localized perturbations.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: In this paper we introduced a new method to construct an explanation for why a neural network has misclassiﬁed an image that is intuitive for a user to understand.
Revised: In this paper we introduced a new method to construct an explanation for why a neural network has misclassiﬁed an image that is intuitive for a user to understand.

Original: We have demonstrated that the method gives meaningful results on both simple and more complicated datasets.
Revised: We have demonstrated that the method gives meaningful results on both simple and more complicated datasets.

Original: Though the method looks promising, there is still a potential problem with applying it in the real world.
Revised: Though the method looks promising, there is still a potential problem with applying it in the real world.

Original: A problem may be that the method does not work well on all annotated traits that images may have.
Revised: A problem may be that the method does not work well on all annotated traits that images may have.

Original: We already saw in the experiments on CelebA that the success rate depends on the trait.
Revised: We already saw in the experiments on CelebA that the success rate depends on the trait.

Original: An explanation for this may be that the opinion of when an image has a certain trait may be ambiguous for annotators.
Revised: An explanation for this may be that the opinion of when an image has a certain trait may be ambiguous for annotators.

Original: For example, what constitutes somebody having big lips may vary among individuals to such an extent that only the extremes can be classiﬁed with certainty, making them hard to reach using gradient descent. 
Revised: For example, what constitutes somebody having big lips may vary among individuals to such an extent that only the extremes can be classiﬁed with certainty, making them hard to reach using gradient descent. An application for our method, is to ﬁnd class imbalances in data sets.

Revised: An example was already given in the paper, however another method may also be possible.

Revised: By averaging the difference between the perturbed reconstruction and the reconstructed image on a set of misclassiﬁed images.

Revised: A heatmap can be constructed to ﬁnd which area is most often perturbed.

Original:  Another application for our method, besides providing explanations, is improving classiﬁer performance.
Revised: This area can point to a potential bias in the dataset. Another application for our method is improving classiﬁer performance.

Original: We can use our method to generate edge cases, by ﬁnding a class boundary and jumping in and out of it on various points.
Revised: We can use our method to generate edge cases, by ﬁnding a class boundary and jumping in and out of it on various points.

Original: These edge cases can then be annotated by a human and used for training.
Revised: These edge cases can then be annotated by a human and used for training.

Original: A hurdle to widespread use of neural networks is that they can seem like black-boxes, when they feel it is difﬁcult to understand why.
Revised: A hurdle to widespread use of neural networks is that they can seem like black-boxes, when they feel it is difﬁcult to understand why.

Original: We believe that this method will help overcome this problem by making the decisions explainable. 
Revised: We believe that this method will help overcome this problem by making the decisions explainable. A P ERTURBATION PROCESS E XAMPLES

Original:  B L OCAL M INIMUM 
Revised: qualitatively much B L OCAL M INIMUM C A PPLICATION FOR DETECTING FEATURE IMBALANCES IN CLASSES

Original: [Figure] 
Revised: [Figure] [Figure]