Original: Diffusion Visual Counterfactual Explanations
Revised: Diffusion Visual Counterfactual Explanations

Original: Abstract
Revised: Abstract

Original: Visual Counterfactual Explanations (VCEs) are an important tool to understand the decisions of an image classiﬁer.
Revised: Visual Counterfactual Explanations (VCEs) are an important tool to understand the decisions of an image classiﬁer.

Original: They are “small” but “realistic” semantic changes of the image changing the classiﬁer decision.
Revised: They are “small” but “realistic” semantic changes of the image changing the classiﬁer decision.

Original: Current approaches for the generation of VCEs are restricted to adversarially robust models and often contain non-realistic artefacts, or are limited to image classiﬁcation problems with few classes.
Revised: Current approaches for the generation of VCEs are restricted to adversarially robust models and often contain non-realistic artefacts, or are limited to image classiﬁcation problems with few classes.

Original: In this paper, we overcome this by generating Diffusion Visual
Revised: In this paper, we overcome this by generating Diffusion Visual

Original: Counterfactual Explanations (DVCEs) for arbitrary ImageNet classiﬁers via a diffusion process.
Revised: Counterfactual Explanations (DVCEs) for arbitrary ImageNet classiﬁers via a diffusion process.

Original: Two modiﬁcations to the diffusion process are key for our
Revised: Two modiﬁcations to the diffusion process are key for our

Original: DVCEs: ﬁrst, an adaptive parameterization, whose hyperparameters generalize across images and models, together with distance regularization and late start of the diffusion process, allow us to generate images with minimal semantic changes to the original ones but different classiﬁcation.
Revised: DVCEs: ﬁrst, an adaptive parameterization, whose hyperparameters generalize across images and models, together with distance regularization and late start of the diffusion process, allow us to generate images with minimal semantic changes to the original ones but different classiﬁcation.

Original: Second, our cone regularization via an adversarially robust model ensures that the diffusion process does not converge to trivial non-semantic changes, but instead produces realistic images of the target class which achieve high conﬁdence by the classiﬁer.
Revised: Second, our cone regularization via an adversarially robust model ensures that the diffusion process does not converge to trivial non-semantic changes, but instead produces realistic images of the target class which achieve high conﬁdence by the classiﬁer.

Original: Code is available under https://github.com/valentyn1boreiko/DVCEs .
Revised: Code is available under https://github.com/valentyn1boreiko/DVCEs .

Original:  Introduction
Revised:  Introduction

Original: It can be argued that one of the main problems hindering the widespread use of machine learning and image classiﬁcation in particular, is the missing possibility to explain the decisions of black-box models such as neural networks.
Revised: It can be argued that one of the main problems hindering the widespread use of machine learning and image classiﬁcation in particular, is the missing possibility to explain the decisions of black-box models such as neural networks.

Original: This is not only a problem for decisions affecting humans where the current draft for AI regulation in Europe [10] requires “transparency”, but it is a pressing problem in all applications of machine learning.
Revised: This is not only a problem for decisions affecting humans where the current draft for AI regulation in Europe [10] requires “transparency”, but it is a pressing problem in all applications of machine learning.

Original: The reason, to some extent, is that humans would like to understand but also control if the learning algorithm has captured the “concepts” of the underlying classes or if it just predicts well using spurious features, artefacts in the data set or other sources of error.
Revised: The reason, to some extent, is that humans would like to understand but also control if the learning algorithm has captured the “concepts” of the underlying classes or if it just predicts well using spurious features, artefacts in the data set or other sources of error.

Original: In this paper we focus on model-agnostic explanations which can, in principle, be applied to any image classiﬁer and do not rely on the speciﬁc structure of the classiﬁer such as decision trees or linear classiﬁers.
Revised: In this paper we focus on model-agnostic explanations which can, in principle, be applied to any image classiﬁer and do not rely on the speciﬁc structure of the classiﬁer such as decision trees or linear classiﬁers.

Original: In this area, in particular for image classiﬁcation, sensitivity based explanations [4], explanations based on feature attributions [3], saliency maps [47, 46, 17, 56, 51], Shapley additive explanations [31], and local ﬁts of interpretable models [39] have been proposed.
Revised: In this area, in particular for image classiﬁcation, sensitivity based explanations [4], explanations based on feature attributions [3], saliency maps [47, 46, 17, 56, 51], Shapley additive explanations [31], and local ﬁts of interpretable models [39] have been proposed.

Original: Moreover, [55] proposed counterfactual explanations (CEs), which are instance-speciﬁc explanations.
Revised: Moreover, [55] proposed counterfactual explanations (CEs), which are instance-speciﬁc explanations.

Original: They can be applied to any classiﬁer and it has been argued that they are close to the human justiﬁcation of decisions [34] using counterfactual reasoning: “I would recognize it as zebra (instead of horse) if it had black and white stripes.”
Revised: They can be applied to any classiﬁer and it has been argued that they are close to the human justiﬁcation of decisions [34] using counterfactual reasoning: “I would recognize it as zebra (instead of horse) if it had black and white stripes.”

Original: For a given classiﬁer almost all methods for the generation of CEs [55, 15, 35, 6, 38, 54, 44] try to solve the following problem: “Given a target class c what is the minimal change  of input x , such that x +  is classiﬁed as class c with high probability and is a realistic instance of my data generating distribution?”
Revised: For a given classiﬁer almost all methods for the generation of CEs [55, 15, 35, 6, 38, 54, 44] try to solve the following problem: “Given a target class c what is the minimal change  of input x , such that x +  is classiﬁed as class c with high probability and is a realistic instance of my data generating distribution?”

Original: From the perspective of “debugging” existing machine learning models, CEs are interesting as they construct a concrete input x +  with a different classiﬁcation that allows the developer to test if the model has learned the correct features.
Revised: From the perspective of “debugging” existing machine learning models, CEs are interesting as they construct a concrete input x +  with a different classiﬁcation that allows the developer to test if the model has learned the correct features.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: The main reason why visual counterfactual explanations (VCEs), that is CEs for image classiﬁcation, are not widely used is that the tasks of generating CEs and adversarial examples [52] are very related.
Revised: The main reason why visual counterfactual explanations (VCEs), that is CEs for image classiﬁcation, are not widely used is that the tasks of generating CEs and adversarial examples [52] are very related.

Original: Even imperceivable changes of the image can already change the prediction of the classiﬁer, however, the resulting noise patterns do not show the user if the classiﬁer has picked up the right class-speciﬁc features.
Revised: Even imperceivable changes of the image can already change the prediction of the classiﬁer, however, the resulting noise patterns do not show the user if the classiﬁer has picked up the right class-speciﬁc features.

Original: One possible solution is using adversarially robust models [43, 1, 7], which have been shown to produce semantically meaningful VCEs by directly maximizing the probability of the target class in image space.
Revised: One possible solution is using adversarially robust models [43, 1, 7], which have been shown to produce semantically meaningful VCEs by directly maximizing the probability of the target class in image space.

Original: These approaches have the downside that they only can generate VCEs for robust models which is a signiﬁcant restriction as these models are not competitive in terms of prediction accuracy.
Revised: These approaches have the downside that they only can generate VCEs for robust models which is a signiﬁcant restriction as these models are not competitive in terms of prediction accuracy.

Original: The second approach is to restrict the generation of VCEs using a generative model or constraining the set of potential image manipulations [19, 20, 42, 8, 18, 45, 27].
Revised: The second approach is to restrict the generation of VCEs using a generative model or constraining the set of potential image manipulations [19, 20, 42, 8, 18, 45, 27].

Original: However, these approaches are either restricted to datasets with a small number of classes, cannot provide explanations for arbitrary classiﬁers, or generate VCEs that look realistic but have so little in common with the original image that not much insight can be gained.
Revised: However, these approaches are either restricted to datasets with a small number of classes, cannot provide explanations for arbitrary classiﬁers, or generate VCEs that look realistic but have so little in common with the original image that not much insight can be gained.

Original: Recently, [26] trained a StyleGAN2 [24] model to discover and manipulate class attributes.
Revised: Recently, [26] trained a StyleGAN2 [24] model to discover and manipulate class attributes.

Original: While their approach yields impressive results on smaller-scale datasets with few similar classes (for example, different bird species), the authors did not demonstrate that the method scales to complex tasks such as ImageNet with hundreds of classes where different classes require different sets of attributes.
Revised: While their approach yields impressive results on smaller-scale datasets with few similar classes (for example, different bird species), the authors did not demonstrate that the method scales to complex tasks such as ImageNet with hundreds of classes where different classes require different sets of attributes.

Original: Another disadvantage is that the StyleGAN model needs to be retrained for every classiﬁer, making it prohibitively expensive to explain multiple large models.
Revised: Another disadvantage is that the StyleGAN model needs to be retrained for every classiﬁer, making it prohibitively expensive to explain multiple large models.

Original: Moreover, [25] have proposed a loss to do VCEs in the latent space of a GAN/VAE.
Revised: Moreover, [25] have proposed a loss to do VCEs in the latent space of a GAN/VAE.

Original: They show promising results for MNIST/FMNIST, but no code is available for ImageNet.
Revised: They show promising results for MNIST/FMNIST, but no code is available for ImageNet.

Original: Furthermore, to explain a classiﬁer, the conditional information during the generation of explanations should ideally come only from the classiﬁer itself, but [25] rely on a conditional GAN which might introduce a bias.
Revised: Furthermore, to explain a classiﬁer, the conditional information during the generation of explanations should ideally come only from the classiﬁer itself, but [25] rely on a conditional GAN which might introduce a bias.

Original: In this paper, we overcome the aforementioned challenges and generate our Diffusion Visual Counterfactual Explanations (DVCEs) for arbitrary ImageNet classiﬁers (see Fig 1).
Revised: In this paper, we overcome the aforementioned challenges and generate our Diffusion Visual Counterfactual Explanations (DVCEs) for arbitrary ImageNet classiﬁers (see Fig 1).

Original: We use the progress in the generation of realistic images using diffusion processes [48, 23, 49, 50], which recently were able to outperform GANs [14].
Revised: We use the progress in the generation of realistic images using diffusion processes [48, 23, 49, 50], which recently were able to outperform GANs [14].

Original: Similar to [37, 2], we use a classiﬁer and a distance-type regularization to guide the generation of the images.
Revised: Similar to [37, 2], we use a classiﬁer and a distance-type regularization to guide the generation of the images.

Original: Two modiﬁcations to the diffusion process are key elements for our DVCEs: i) a combination of distance regularization and starting point of the diffusion process together with an adaptive reparameterization lets us generate VCEs visually close to the original image in a controlled way so that hyperparameters can be ﬁxed across images and even across models, ii) our cone regularization of the gradient of the classiﬁer via an adversarially robust model ensures that the diffusion process does not converge to trivial non-semantic changes but instead produces realistic images of the target class which achieve high conﬁdence by the classiﬁer.
Revised: Two modiﬁcations to the diffusion process are key elements for our DVCEs: i) a combination of distance regularization and starting point of the diffusion process together with an adaptive reparameterization lets us generate VCEs visually close to the original image in a controlled way so that hyperparameters can be ﬁxed across images and even across models, ii) our cone regularization of the gradient of the classiﬁer via an adversarially robust model ensures that the diffusion process does not converge to trivial non-semantic changes but instead produces realistic images of the target class which achieve high conﬁdence by the classiﬁer.

Original: Our approach can be employed for any dataset where a generative diffusion model and an adversarially robust model are available.
Revised: Our approach can be employed for any dataset where a generative diffusion model and an adversarially robust model are available.

Original: In a qualitative comparison, user study, and a quantitative evaluation (Sec. 4.1), we show that our DVCEs achieve higher realism (according to FID) and have more meaningful features (according to the user study) than both the recent methods of [7] and [2].
Revised: In a qualitative comparison, user study, and a quantitative evaluation (Sec. 4.1), we show that our DVCEs achieve higher realism (according to FID) and have more meaningful features (according to the user study) than both the recent methods of [7] and [2].

Original:  Diffusion models
Revised:  Diffusion models

Original: Diffusion models [14, 48, 49, 23, 36] are generative models that consist of two steps: a forward diffusion process (and thus a Markov process) that transforms the data distribution to a prior distribution (which is usually assumed to be a standard normal distribution) and the reverse diffusion process that transforms the prior distribution back to the data distribution.
Revised: Diffusion models [14, 48, 49, 23, 36] are generative models that consist of two steps: a forward diffusion process (and thus a Markov process) that transforms the data distribution to a prior distribution (which is usually assumed to be a standard normal distribution) and the reverse diffusion process that transforms the prior distribution back to the data distribution.

Original: The existence of the (continuous-time) reverse diffusion process was closer investigated in [50].
Revised: The existence of the (continuous-time) reverse diffusion process was closer investigated in [50].

Original: In the discrete-time setting, a Markov chain { x 1 , ..., x T } for any data point x 0 in the forward direction is deﬁned via a Markov process that adds noise to the data point at each timestep:
Revised: In the discrete-time setting, a Markov chain { x 1 , ..., x T } for any data point x 0 in the forward direction is deﬁned via a Markov process that adds noise to the data point at each timestep:

Original: [Equation]
Revised: [Equation]

Original: where {  1 , ...,  T } is some variance schedule, chosen such that q ( x T ) ⇠ N (0 , I ) .
Revised: where {  1 , ...,  T } is some variance schedule, chosen such that q ( x T ) ⇠ N (0 , I ) .

Original: Note that given x 0 , it is possible to sample from q ( x t | x 0 ) in closed form instead of applying noise t times using:
Revised: Note that given x 0 , it is possible to sample from q ( x t | x 0 ) in closed form instead of applying noise t times using:

Original: [Equation]
Revised: [Equation]

Original: where ↵ t :=
Revised: where ↵ t :=

Original: Q tk =1 (1   k ) .
Revised: Q tk =1 (1   k ) .

Original: In [48], the authors have shown that the reverse transitions q ( x t  1 | x t ) approach diagonal Gaussian distributions as T ! 1 and thus one can use a DNN with parameters ✓ to approximate q ( x t  1 | x t ) as p ✓ ( x t  1 | x t ) by predicting the mean and the diagonal covariance matrix µ ✓ ( x t , t ) and ⌃ ✓ ( x t , t ) :
Revised: In [48], the authors have shown that the reverse transitions q ( x t  1 | x t ) approach diagonal Gaussian distributions as T ! 1 and thus one can use a DNN with parameters ✓ to approximate q ( x t  1 | x t ) as p ✓ ( x t  1 | x t ) by predicting the mean and the diagonal covariance matrix µ ✓ ( x t , t ) and ⌃ ✓ ( x t , t ) :

Original: [Equation]
Revised: [Equation]

Original: To sample from p ✓ ( x 0 ) , one samples x T from N (0 , I ) and then follows the reverse process by repeatedly sampling from the transition probabilities p ✓ ( x t  1 | x t ) .
Revised: To sample from p ✓ ( x 0 ) , one samples x T from N (0 , I ) and then follows the reverse process by repeatedly sampling from the transition probabilities p ✓ ( x t  1 | x t ) .

Original: Instead of predicting µ ✓ ( x t , t ) directly, it has been shown in [23] that the best performing parameterization uses a neural network ✏ ✓ ( x t , t ) to approximate the source noise ✏ in (3) and the loss used for training resembles that of a denoising model:
Revised: Instead of predicting µ ✓ ( x t , t ) directly, it has been shown in [23] that the best performing parameterization uses a neural network ✏ ✓ ( x t , t ) to approximate the source noise ✏ in (3) and the loss used for training resembles that of a denoising model:

Original: [Equation]
Revised: [Equation]

Original: The mean of the reverse step µ ✓ in (4) can be derived [32] using Bayes theorem as:
Revised: The mean of the reverse step µ ✓ in (4) can be derived [32] using Bayes theorem as:

Original: [Equation]
Revised: [Equation]

Original: This is, however, only one of the possible equivalent parameterization of the learning objective as shown in [32].
Revised: This is, however, only one of the possible equivalent parameterization of the learning objective as shown in [32].

Original: Because the objective L simple does not give a learning signal for ⌃ ✓ ( x t , t ) , in practice one combines L simple with another loss based on a variational lower bound of the data likelihood [48], which unlike L simple allows us to learn the diagonal covariance matrix ⌃ ✓ ( x t , t ) .
Revised: Because the objective L simple does not give a learning signal for ⌃ ✓ ( x t , t ) , in practice one combines L simple with another loss based on a variational lower bound of the data likelihood [48], which unlike L simple allows us to learn the diagonal covariance matrix ⌃ ✓ ( x t , t ) .

Original: Concretely, the network ✏ ✓ ( x t , t ) outputs additionally a vector that is used to parametrize ⌃ ✓ ( x t , t ) .
Revised: Concretely, the network ✏ ✓ ( x t , t ) outputs additionally a vector that is used to parametrize ⌃ ✓ ( x t , t ) .

Original: 2.1 Class conditional sampling
Revised: 2.1 Class conditional sampling

Original: There exist diffusion models trained with and without knowledge about classes in the dataset [14, 22].
Revised: There exist diffusion models trained with and without knowledge about classes in the dataset [14, 22].

Original: For our experiments, we only use the class-unconditional diffusion model, such that all classconditional features are introduced by the classiﬁer that we want to explain.
Revised: For our experiments, we only use the class-unconditional diffusion model, such that all classconditional features are introduced by the classiﬁer that we want to explain.

Original: For the noise-aware classiﬁer p  ( y |· , · ) : R d ⇥ { 1 , ..., T } !
Revised: For the noise-aware classiﬁer p  ( y |· , · ) : R d ⇥ { 1 , ..., T } !

Original: [0 , 1] , with parameters  , that is trained on noisy images corresponding to the various timesteps [14], the reverse process transitions are then of the form:
Revised: [0 , 1] , with parameters  , that is trained on noisy images corresponding to the various timesteps [14], the reverse process transitions are then of the form:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: for a normalization constant Z .
Revised: for a normalization constant Z .

Original: As we would like to explain any classiﬁer and not only noise-aware ones, we follow the approach of [2], where a classiﬁer p  ( y |· ): R d !
Revised: As we would like to explain any classiﬁer and not only noise-aware ones, we follow the approach of [2], where a classiﬁer p  ( y |· ): R d !

Original: [0 , 1] is given as input the denoised sample ˆ x
Revised: [0 , 1] is given as input the denoised sample ˆ x

Original: 0 = f dn ( x t ) of x t , using the mapping:
Revised: 0 = f dn ( x t ) of x t , using the mapping:

Original: [Equation]
Revised: [Equation]

Original: This mapping, derived from (3) estimates the noise-free image x 0 using the noise approximated by the model ✏ ✓ ( x t , t ) for a given timestep t .
Revised: This mapping, derived from (3) estimates the noise-free image x 0 using the noise approximated by the model ✏ ✓ ( x t , t ) for a given timestep t .

Original: With this, we can deﬁne a timestep-aware posterior p  ( y | x t , t ) for any classiﬁer p  ( y |· ) as p  ( y | x t , t ):= p  ( y | f dn ( x t , t )) .
Revised: With this, we can deﬁne a timestep-aware posterior p  ( y | x t , t ) for any classiﬁer p  ( y |· ) as p  ( y | x t , t ):= p  ( y | f dn ( x t , t )) .

Original: The issue is that efﬁcient sampling from the original diffusion model is possible only because the reverse process is made of normal distributions.
Revised: The issue is that efﬁcient sampling from the original diffusion model is possible only because the reverse process is made of normal distributions.

Original: As we need to sample from p ✓ ,  ( x t  1 | x t , y ) hundreds of times to obtain a single sample from the data distribution, it is not possible to use MCMC-samplers with high complexity to sample from each of the individual transitions.
Revised: As we need to sample from p ✓ ,  ( x t  1 | x t , y ) hundreds of times to obtain a single sample from the data distribution, it is not possible to use MCMC-samplers with high complexity to sample from each of the individual transitions.

Original: In [14], they proposed to solve it by approximating p ✓ ,  ( x t  1 | x t , y ) with slightly shifted versions of p ✓ ( x t  1 | x t ) to make closed-form sampling possible.
Revised: In [14], they proposed to solve it by approximating p ✓ ,  ( x t  1 | x t , y ) with slightly shifted versions of p ✓ ( x t  1 | x t ) to make closed-form sampling possible.

Original: Such transition kernels are given by:
Revised: Such transition kernels are given by:

Original: [Equation]
Revised: [Equation]

Original: which we further adapt for the goal of generating VCEs and use in our experiments.
Revised: which we further adapt for the goal of generating VCEs and use in our experiments.

Original:  Diffusion Visual Counterfactual Explanations
Revised:  Diffusion Visual Counterfactual Explanations

Original: A VCE x for a chosen target class y , a given classiﬁer p  ( c |· ) , and an input ˆ x should satisfy the following criteria: i) validity: the VCE x should be classiﬁed by p  ( y |· ) as the desired target class y with high predicted probability, ii) realism: the VCE should be as close as possible to a natural image, iii) minimality/closeness: the difference between the VCE x and the original image ˆ x should be the minimal semantic modiﬁcation necessary to change the class, in particular, the generated image x should be close to ˆ x while being valid and realistic, e.g. by changing the object in the image and leaving the background unchanged.
Revised: A VCE x for a chosen target class y , a given classiﬁer p  ( y |· ) , and an input ˆ x should satisfy the following criteria: i) validity: the VCE x should be classiﬁed by p  ( y |· ) as the desired target class y with high predicted probability, ii) realism: the VCE should be as close as possible to a natural image, iii) minimality/closeness: the difference between the VCE x and the original image ˆ x should be the minimal semantic modiﬁcation necessary to change the class, in particular, the generated image x should be close to ˆ x while being valid and realistic, e.g. by changing the object in the image and leaving the background unchanged.

Original: Note that targeted adversarial examples are valid but do not show meaningful semantic changes in the target class for a non-robust model and are not realistic.
Revised: Note that targeted adversarial examples are valid but do not show meaningful semantic changes in the target class for a non-robust model and are not realistic.

Original: The l 1 . 5 -SVCEs of [7] change the image in order to maximize the predicted probability of the classiﬁer into the target class inside an l 1 .
Revised: The l 1 . 5 -SVCEs of [7] change the image in order to maximize the predicted probability of the classiﬁer into the target class inside an l 1 .

Original: 5 -ball around the image which is a targeted adversarial example.
Revised: 5 -ball around the image which is a targeted adversarial example.

Original: Thus this only works for robust classiﬁers and they use an ImageNet classiﬁer that was trained to be multiple-norm robust (MNR), which we denote in this paper as MNR-RN50 (see Sec. 4.2).
Revised: Thus this only works for robust classiﬁers and they use an ImageNet classiﬁer that was trained to be multiple-norm robust (MNR), which we denote in this paper as MNR-RN50 (see Sec. 4.2).

Original: The realism of the l 1 . 5 -SVCE comes purely from the generative properties of robust classiﬁers [43], which can lead to artefacts.
Revised: The realism of the l 1 . 5 -SVCEs comes purely from the generative properties of robust classiﬁers [43], which can lead to artefacts.

Original: In contrast, our Diffusion Visual Counterfactual Explanations (DVCEs) work for any classiﬁer and our DVCEs are more realistic due the better generative properties of diffusion models.
Revised: In contrast, our Diffusion Visual Counterfactual Explanations (DVCEs) work for any classiﬁer and our DVCEs are more realistic due the better generative properties of diffusion models.

Original: An approach similar to our DVCE framework is Blended Diffusion (BD) [2] which manipulates the image inside a masked region.
Revised: An approach similar to our DVCE framework is Blended Diffusion (BD) [2] which manipulates the image inside a masked region.

Original: One can adapt BD for the generation of VCEs by using as mask the whole image.
Revised: One can adapt BD for the generation of VCEs by using as mask the whole image.

Original: DVCE and BD share the same diffusion model, but BD cannot be applied to arbitrary classiﬁers and requires image-speciﬁc hyperparameter tuning, see Fig.
Revised: DVCE and BD share the same diffusion model, but BD cannot be applied to arbitrary classiﬁers and requires image-speciﬁc hyperparameter tuning, see Fig.

Original: 3.1 Adaptive Parameterization
Revised: 3.1 Adaptive Parameterization

Original: For generating the DVCE of the original image ˆ x , we do not want to sample just any image x from p ( x | y )
Revised: For generating the DVCE of the original image ˆ x , we do not want to sample just any image x from p ( x | y )

Original: (high realism and validity), but also to make sure that d ( x, ˆ x ) is small.
Revised: (high realism and validity), but also to make sure that d ( x, ˆ x ) is small.

Original: For this, we have to condition our diffusion process on ˆ x .
Revised: For this, we have to condition our diffusion process on ˆ x .

Original: Using the denoising function f dn and analogously to the derivation of (9) and its applications in [14, 27], the mean of the transition kernel becomes:
Revised: Using the denoising function f dn and analogously to the derivation of (9) and its applications in [14, 27], the mean of the transition kernel becomes:

Original: [Equation]
Revised: [Equation]

Original: where C c is the coefﬁcient of the classiﬁer, and C d the one of the distance guiding loss.
Revised: where C c is the coefﬁcient of the classiﬁer, and C d the one of the distance guiding loss.

Original: Intuitively, we take a step in the direction that increases the classiﬁer score while staying close to ˆ x .
Revised: Intuitively, we take a step in the direction that increases the classiﬁer score while staying close to ˆ x .

Original: Note that the last term can be interpreted as the log gradient of a distribution with density exp(  C d d t (ˆ x, · )) , where d t (ˆ x, · ):= d (ˆ x, f dn ( · , t )) .
Revised: Note that the last term can be interpreted as the log gradient of a distribution with density exp(  C d d t (ˆ x, · )) , where d t (ˆ x, · ):= d (ˆ x, f dn ( · , t )) .

Original: Thus we introduce a timestep-aware prior distribution that enforces our output to be similar to ˆ x .
Revised: Thus we introduce a timestep-aware prior distribution that enforces our output to be similar to ˆ x .

Original: In our work, we use the l 1 -distance as it produces sparse changes.
Revised: In our work, we use the l 1 -distance as it produces sparse changes.

Original: Several works have tried to minimize some distance during the diffusion process, implicitly in [9] or explicitly, but for the background, in BD.
Revised: Several works have tried to minimize some distance during the diffusion process, implicitly in [9] or explicitly, but for the background, in BD.

Original: However, there is no principled way to choose the coefﬁcient for such a regularization term.
Revised: However, there is no principled way to choose the coefﬁcient for such a regularization term.

Original: It turns out that it is impossible to ﬁnd a parameter setting for BD that works across images and classiﬁers.
Revised: It turns out that it is impossible to ﬁnd a parameter setting for BD that works across images and classiﬁers.

Original: Thus, we propose a parameterization
Revised: Thus, we propose a parameterization

Original: [Equation]
Revised: [Equation]

Original: which adapts to the predicted mean of the diffusion model that we use to change µ t in (10) to
Revised: which adapts to the predicted mean of the diffusion model that we use to change µ t in (10) to

Original: [Equation]
Revised: [Equation]

Original: This adaptive parameterization allows for ﬁne-grained control of the inﬂuence of the classiﬁer and distance regularization so that now the hyperparameters C c and C d have the same inﬂuence across
Revised: This adaptive parameterization allows for ﬁne-grained control of the inﬂuence of the classiﬁer and distance regularization so that now the hyperparameters C c and C d have the same inﬂuence across

Original: [Figure]
Revised: [Figure]

Original: images and even classiﬁers.
Revised: images and even classiﬁers.

Original: It facilitates the generation of DVCEs as otherwise hyperparameter ﬁnetuning would be necessary for each image as in BD, see Fig.
Revised: It facilitates the generation of DVCEs as otherwise hyperparameter ﬁnetuning would be necessary for each image as in BD, see Fig.

Original: 2 for a comparison.
Revised: 2 for a comparison.

Original: However, even with our adaptive parameterization, it is still not easy to produce semantically meaningful changes close to ˆ x as can be seen in App.
Revised: However, even with our adaptive parameterization, it is still not easy to produce semantically meaningful changes close to ˆ x as can be seen in App.

Original: B.2.
Revised: B.2.

Original: Thus, as in [2], we vary the starting point of the diffusion process and observe in App.
Revised: Thus, as in [2], we vary the starting point of the diffusion process and observe in App.

Original: B.2 that starting from step T 2 of the forward diffusion process, together with the adaptive parameterization and using as the distance the l 1 -distance, provides us with sparse but semantically meaningful changes.
Revised: B.2 that starting from step T 2 of the forward diffusion process, together with the adaptive parameterization and using as the distance the l 1 -distance, provides us with sparse but semantically meaningful changes.

Original: In our experiments, we set T = 200 .
Revised: In our experiments, we set T = 200 .

Original: 3.2 Cone Projection for Classiﬁer Guidance
Revised: 3.2 Cone Projection for Classiﬁer Guidance

Original: A key objective for our DVCEs is that they can be applied to any image classiﬁer regardless of whether it is adversarially robust or not.
Revised: A key objective for our DVCEs is that they can be applied to any image classiﬁer regardless of whether it is adversarially robust or not.

Original: Using diffusion with the new mean as in (13) does not work with a non-robust classiﬁer and leads to very small modiﬁcations of the image, which are similar to adversarial examples without meaningful changes.
Revised: Using diffusion with the new mean as in (13) does not work with a non-robust classiﬁer and leads to very small modiﬁcations of the image, which are similar to adversarial examples without meaningful changes.

Original: The reason is that the gradients of non-robust classiﬁers are noisier and less semantically meaningful than those of robust classiﬁers.
Revised: The reason is that the gradients of non-robust classiﬁers are noisier and less semantically meaningful than those of robust classiﬁers.

Original: We illustrate this for a non-robust Swin-TF [28] in Fig.
Revised: We illustrate this for a non-robust Swin-TF [28] in Fig.

Original: 3 where hardly any changes are generated.
Revised: 3 where hardly any changes are generated.

Original: Note that this is even the case if we ﬁrst denoise the sample using the denoising function f dn (8).
Revised: Note that this is even the case if we ﬁrst denoise the sample using the denoising function f dn (8).

Original: As a solution, we suggest projecting the gradient of an adversarially robust classiﬁer with parameters , r x t log p robust , ( y | f dn ( x t , t )) , onto a cone centered at the gradient of the classiﬁer, r x t log p  ( y | f dn ( x t , t )) .
Revised: As a solution, we suggest projecting the gradient of an adversarially robust classiﬁer with parameters , r x t log p robust , ( y | f dn ( x t , t )) , onto a cone centered at the gradient of the classiﬁer, r x t log p  ( y | f dn ( x t , t )) .

Original: More precisely, we deﬁne
Revised: More precisely, we deﬁne

Original: [Equation]
Revised: [Equation]

Original: where cone ( ↵ , v ) := { w 2 R d : \ ( v, w )  ↵ } is the cone of angle ↵ around vector v , and the projection P cone ( ↵ ,v ) onto cone ( ↵ , v ) is given as:
Revised: where cone ( ↵ , v ) := { w 2 R d : \ ( v, w )  ↵ } is the cone of angle ↵ around vector v , and the projection P cone ( ↵ ,v ) onto cone ( ↵ , v ) is given as:

Original: [Equation]
Revised: [Equation]

Original:  where P v ? ( w ) :=
Revised: else , where P v ? ( w ) :

Original:  w  h w,v i h v,v i v and u = sin( ↵ ) P v ?
Revised: = w  h w,v i h v,v i v and u = sin( ↵ ) P v ?

Original: ( w )
Revised: ( w )

Original: [Equation]
Revised: [Equation]

Original: k {+
 + cos( ↵ ) v k+} v k 2 (note k u k 2 = 1 ).
Revised: k {+
 + cos( ↵ ) v k+} v k 2 (note k u k 2 = 1 ).

Original: The motivation for this projection is to reduce the noise in the gradient of the non-robust classiﬁer.
Revised: The motivation for this projection is to reduce the noise in the gradient of the non-robust classiﬁer.

Original: Note,
Revised: Note,

Original: [Figure]
Revised: [Figure]

Original: that the projection of the gradient of the robust classiﬁer onto the cone generated by the non-robust classiﬁer with angle ↵ < 90  is always an ascent direction for log p  ( y | f dn ( x t , t )) , which we would like to maximize (note that g proj is not necessarily an ascent direction for log p robust , ( y | f dn ( x t , t )) ).
Revised: that the projection of the gradient of the robust classiﬁer onto the cone generated by the non-robust classiﬁer with angle ↵ < 90  is always an ascent direction for log p  ( y | f dn ( x t , t )) , which we would like to maximize (note that g proj is not necessarily an ascent direction for log p robust , ( y | f dn ( x t , t )) ).

Original: Thus, the cone projection is a form of regularization of r x t log p  ( y | f dn ( x t , t )) , which guides the diffusion process to semantically meaningful changes of the image.
Revised: Thus, the cone projection is a form of regularization of r x t log p  ( y | f dn ( x t , t )) , which guides the diffusion process to semantically meaningful changes of the image.

Original: 3.3 Final Scheme for Diffusion Visual Counterfactuals
Revised: 3.3 Final Scheme for Diffusion Visual Counterfactuals

Original: Our solution for a non-adversarially robust classiﬁer p  ( y |· ) is to use Algorithm 1 of [14] by replacing the update step with:
Revised: Our solution for a non-adversarially robust classiﬁer p  ( y |· ) is to use Algorithm 1 of [14] by replacing the update step with:

Original: [Equation]
Revised: [Equation]

Original: t , t ) + ⌃ ✓ ( x t , t ) k µ ✓ ( x t , t ) k 2 g update , p ( x t  1 | x t , ˆ x, y ) =
Revised: t , t ) + ⌃ ✓ ( x t , t ) k µ ✓ ( x t , t ) k 2 g update , p ( x t  1 | x t , ˆ x, y ) =

Original: N ( µ t , ⌃ ✓ ( x t , t )) .
Revised: N ( µ t , ⌃ ✓ ( x t , t )) .

Original: For an adversarially robust classiﬁer the cone projection is omitted and one uses g proj =
Revised: For an adversarially robust classiﬁer the cone projection is omitted and one uses g proj =

Original: r x t log p robust ,  y | f dn ( x t , t )  .
Revised: r x t log p robust ,  y | f dn ( x t , t )  .

Original: In all our experiments we use C c = 0 .
Revised: In all our experiments we use C c = 0 .

Original: 1 , and C d = 0 .
Revised: 1 , and C d = 0 .

Original: 15 unless we show ablations for one of the parameters.
Revised: 15 unless we show ablations for one of the parameters.

Original: The angle ↵ for the cone projection is ﬁxed to 30  .
Revised: The angle ↵ for the cone projection is ﬁxed to 30  .

Original: In strong contrast to BD, these parameters generalize across images and classiﬁers.
Revised: In strong contrast to BD, these parameters generalize across images and classiﬁers.

Original:  Experiments
Revised:  Experiments

Original: In this section, we evaluate the quality of the DVCE.
Revised: In this section, we evaluate the quality of the DVCE.

Original: We compare DVCE to existing works in Sec.
Revised: We compare DVCE to existing works in Sec.

Original: In Sec. 4.2, we compare DVCEs for various state-of-the-art ImageNet models and show how
Revised: In Sec. 4.2, we compare DVCEs for various state-of-the-art ImageNet models and show how

Original: DVCEs can be used to interpret differences between classiﬁers.
Revised: DVCEs can be used to interpret differences between classiﬁers.

Original: For our DVCEs, we use for all experiments the ﬁxed hyperparameters given in Sec. 3.3.
Revised: For our DVCEs, we use for all experiments the ﬁxed hyperparameters given in Sec. 3.3.

Original: The diffusion model used for DVCE is taken from [14] and has been trained class-unconditionally on 256x256 ImageNet images using a modiﬁed UNet[40] architecture.
Revised: The diffusion model used for DVCE is taken from [14] and has been trained class-unconditionally on 256x256 ImageNet images using a modiﬁed UNet[40] architecture.

Original: The user study, discovery of spurious features using VCEs, further experiments, and ablations are in the appendix.
Revised: The user study, discovery of spurious features using VCEs, further experiments, and ablations are in the appendix.

Original: 4.1 Comparison of Methods for VCE Generation
Revised: 4.1 Comparison of Methods for VCE Generation

Original: We compare DVCEs with VCEs produced by BD (BDVCEs) and l 1 .
Revised: We compare DVCEs with VCEs produced by BD (BDVCEs) and l 1 .

Original: 5 -SVCEs.
Revised: 5 -SVCEs.

Original: As the latter only works for adversarially robust classiﬁers, we use the multiple-norm robust ResNet50 from [7], MNRRN50, as the classiﬁer to create VCEs for all three methods.
Revised: As the latter only works for adversarially robust classiﬁers, we use the multiple-norm robust ResNet50 from [7], MNRRN50, as the classiﬁer to create VCEs for all three methods.

Original: As this model is robust on its own, we do not use the cone projection for its DVCEs.
Revised: As this model is robust on its own, we do not use the cone projection for its DVCEs.

Original: First, in Fig.
Revised: First, in Fig.

Original: 4 we present a qualitative evaluation, where we transform one image into two different classes that are close to the true one in the WordNet hierarchy[33].
Revised: 4 we present a qualitative evaluation, where we transform one image into two different classes that are close to the true one in the WordNet hierarchy[33].

Original: The radius of the l 1 . 5 -ball of [7] is chosen as the smallest r 2 { 50 , 75 , 100 , 150 } such that the conﬁdence in the target class is larger than 0 .
Revised: The radius of the l 1 . 5 -ball of [7] is chosen as the smallest r 2 { 50 , 75 , 100 , 150 } such that the conﬁdence in the target class is larger than 0 .

Original: 90 per image.
Revised: 90 per image.

Original: For BD, we select the image with the smallest classiﬁer and regularization weight that reaches conﬁdence larger than 0.9 from the set of parameters discussed in Sec. 3.3.
Revised: For BD, we select the image with the smallest classiﬁer and regularization weight that reaches conﬁdence larger than 0.9 from the set of parameters discussed in Sec. 3.3.

Original: If 0.9 is not reached by any setting for one of the two baselines, we show the image that achieves the
Revised: If 0.9 is not reached by any setting for one of the two baselines, we show the image that achieves the

Original: [Figure]
Revised: [Figure]

Original: white wolf, volcano, night snake) and have often lower image quality.
Revised: white wolf, volcano, night snake) and have often lower image quality.

Original: highest conﬁdence.
Revised: highest conﬁdence.

Original: As Fig.
Revised: As Fig.

Original: 4 shows, DVCE is the only method that satisﬁes all desired properties of
Revised: 4 shows, DVCE is the only method that satisﬁes all desired properties of

Original: VCEs.
Revised: VCEs.

Original: For example, for “mashed potato”, DVCE preserves the bowl and only changes the content into either guacamole or carbonara.
Revised: For example, for “mashed potato”, DVCE preserves the bowl and only changes the content into either guacamole or carbonara.

Original: Our qualitative comparison shows that the same hyperparameter setting of DVCE can handle different classes and transfer between similar classes, such as different snakes or wolf types, as well as different object sizes, e.g. cheetah and snake.
Revised: Our qualitative comparison shows that the same hyperparameter setting of DVCE can handle different classes and transfer between similar classes, such as different snakes or wolf types, as well as different object sizes, e.g. cheetah and snake.

Original: In contrast, both l 1 . 5 -SVCEs and BDVCEs require different hyperparameters for different images to achieve high conﬁdence in the target class.
Revised: In contrast, both l 1 . 5 -SVCEs and BDVCEs require different hyperparameters for different images to achieve high conﬁdence in the target class.

Original: Even with the six parameter conﬁgurations, BD is not able to always produce images with high conﬁdence in the target class.
Revised: Even with the six parameter conﬁgurations, BD is not able to always produce images with high conﬁdence in the target class.

Original: More problematic for
Revised: More problematic for

Original: VCEs is that the resulting images can have high conﬁdence but are neither realistic (cheetah ! tiger) nor resemble the original image at all (dingo !
Revised: VCEs is that the resulting images can have high conﬁdence but are neither realistic (cheetah ! tiger) nor resemble the original image at all (dingo !

Original: timber wolf/white wolf).
Revised: timber wolf/white wolf).

Original: Even if the method works with the given parameters, for example, mashed potato !
Revised: Even if the method works with the given parameters, for example, mashed potato !

Original: guacamole or carbonara, the overall image quality cannot match that of DVCE as often the images contain overly bright colors.
Revised: guacamole or carbonara, the overall image quality cannot match that of DVCE as often the images contain overly bright colors.

Original: In the case of the volcano VCE, DVCE shows class features like lava whereas BDVCE can not clearly be labeled as a volcano.
Revised: In the case of the volcano VCE, DVCE shows class features like lava whereas BDVCE can not clearly be labeled as a volcano.

Original: For the l 1 . 5 -SVCEs, one often needs large radii to achieve the desired conﬁdence of 0.90, which often results in images that do not look realistic.
Revised: For the l 1 . 5 -SVCEs, one often needs large radii to achieve the desired conﬁdence of 0.90, which often results in images that do not look realistic.

Original: As noted in [7], a quantitative analysis of VCEs using FID scores is difﬁcult as methods not changing the original image have low FID score.
Revised: As noted in [7], a quantitative analysis of VCEs using FID scores is difﬁcult as methods not changing the original image have low FID score.

Original: Thus, we have developed a cross-over evaluation scheme, where one partitions the classes into two sets and only analyzes cross-over VCEs (more details are in
Revised: Thus, we have developed a cross-over evaluation scheme, where one partitions the classes into two sets and only analyzes cross-over VCEs (more details are in

Original: App. E).
Revised: App. E).

Original: We show the results in Tab. 1.
Revised: We show the results in Tab. 1.

Original: In terms of closeness, DVCEs are worse than l 1 .
Revised: In terms of closeness, DVCEs are worse than l 1 .

Original: 5 -SVCEs, which can be expected as they optimize inside an l 1 . 5 -ball.
Revised: 5 -SVCEs, which can be expected as they optimize inside an l 1 . 5 -ball.

Original: However, DVCE are the most realistic ones (FID score) and have similar validity as l 1 . 5 -SVCEs.
Revised: However, DVCE are the most realistic ones (FID score) and have similar validity as l 1 . 5 -SVCEs.

Original: BDVCEs are the worst in all categories.
Revised: BDVCEs are the worst in all categories.

Original: Moreover, we have conducted a user study ( 20 users), in which participants decided if the changes of the VCE are meaningful or subtle and if the generated image is realistic , see App.
Revised: Moreover, we have conducted a user study ( 20 users), in which participants decided if the changes of the VCE are meaningful or subtle and if the generated image is realistic , see App.

Original: C for details.
Revised: C for details.

Original: The percentage of total images having the three different properties is (order: DVCE/ l 1 .
Revised: The percentage of total images having the three different properties is (order: DVCE/ l 1 .

Original: 5 -SVCE/BD): meaningful - 62.0% , 48.4%, 38.7%; realism - 34.7%, 24.6%, 52.2% ; subtle - 45.0%, 50.6% , 31.0%.
Revised: 5 -SVCE/BD): meaningful - 62.0% , 48.4%, 38.7%; realism - 34.7%, 24.6%, 52.2% ; subtle - 45.0%, 50.6% , 31.0%.

Original: This conﬁrms that DVCEs generate more meaningful features in the target classes.
Revised: This conﬁrms that DVCEs generate more meaningful features in the target classes.

Original: While the result regarding realism seems to contradict the quantitative evaluation, this is due to fact that realism means that the user considered the image realistic irrespectively if it shows the target class or not.
Revised: While the result regarding realism seems to contradict the quantitative evaluation, this is due to fact that realism means that the user considered the image realistic irrespectively if it shows the target class or not.

Original: [Table]
Revised: [Table]

Original: 4.2 Model comparison
Revised: 4.2 Model comparison

Original: Non-robust models.
Revised: Non-robust models.

Original: In Fig.
Revised: In Fig.

Original: 5 we show that DVCEs can be generated for various state-of-the-art
Revised: 5 we show that DVCEs can be generated for various state-of-the-art

Original: ImageNet [41] models.
Revised: ImageNet [41] models.

Original: We use a Swin-TF[28], a ConvNeXt [29] and a Noisy-Student EfﬁcientNet [57, 53].
Revised: We use a Swin-TF[28], a ConvNeXt [29] and a Noisy-Student EfﬁcientNet [57, 53].

Original: Both the Swin-TF and the ConvNeXt are pretrained on ImageNet21k [13] whereas the
Revised: Both the Swin-TF and the ConvNeXt are pretrained on ImageNet21k [13] whereas the

Original: EfﬁcientNet uses noisy-student self-training on a large unlabeled pool.
Revised: EfﬁcientNet uses noisy-student self-training on a large unlabeled pool.

Original: As all models do not yield perceptually aligned gradients, we use the cone projection with 30  angle described in Sec. 3.
Revised: As all models do not yield perceptually aligned gradients, we use the cone projection with 30  angle described in Sec. 3.

Original: with the robust model from the previous section.
Revised: with the robust model from the previous section.

Original: All other parameters are identical to the previous experiment.
Revised: All other parameters are identical to the previous experiment.

Original: We highlight that DVCE is the ﬁrst method capable of explaining (in the sense of
Revised: We highlight that DVCE is the ﬁrst method capable of explaining (in the sense of

Original: VCEs) arbitrary classiﬁers on a task as challenging as ImageNet.
Revised: VCEs) arbitrary classiﬁers on a task as challenging as ImageNet.

Original: Overall, DVCEs satisfy all desired properties of VCEs.
Revised: Overall, DVCEs satisfy all desired properties of VCEs.

Original: It also allows us to inspect the most important features of each model and class.
Revised: It also allows us to inspect the most important features of each model and class.

Original: For the stupa and church classes, for example, it seems like all models use different roof and tower structures as the most prominent feature as they spent most of their budget on changing those.
Revised: For the stupa and church classes, for example, it seems like all models use different roof and tower structures as the most prominent feature as they spent most of their budget on changing those.

Original: Robust models.
Revised: Robust models.

Original: Here, we evaluate the DVCEs of different robust models trained to be multiple-norm adversarially robust.
Revised: Here, we evaluate the DVCEs of different robust models trained to be multiple-norm adversarially robust.

Original: They are generated by multiple norm-ﬁnetuning [11] an initially l p -robust model to become robust with respect to l 1 -, l 2 - and l 1 -threat model, speciﬁcally
Revised: They are generated by multiple norm-ﬁnetuning [11] an initially l p -robust model to become robust with respect to l 1 -, l 2 - and l 1 -threat model, speciﬁcally

Original: an l 2 -robust ResNet [16] resulting in MNR-RN50 used in the revious sections, an l 1 -robust XCiT-transformer model [12] called MNR-XCiT in the following and an l 1 -robust DeiT-transformer [5] called MNR-DeiT.
Revised: an l 2 -robust ResNet [16] resulting in MNR-RN50 used in the revious sections, an l 1 -robust XCiT-transformer model [12] called MNR-XCiT in the following and an l 1 -robust DeiT-transformer [5] called MNR-DeiT.

Original: Their multiple-norm robust accuracies can be found in [11].
Revised: Their multiple-norm robust accuracies can be found in [11].

Original: In [7], different l p -ball adversarially robust models were compared for the generation of VCEs and they showed that for their l 1 . 5 -SVCEs the multiple norm robust model was better (both in terms of FID and qualitatively) than individual l p -norm robust classiﬁers.
Revised: In [7], different l p -ball adversarially robust models were compared for the generation of VCEs and they showed that for their l 1 . 5 -SVCEs the multiple norm robust model was better (both in terms of FID and qualitatively) than individual l p -norm robust classiﬁers.

Original: This is the reason why we also use a multiple-norm robust model for the cone-projection of a non-robust classiﬁer.
Revised: This is the reason why we also use a multiple-norm robust model for the cone-projection of a non-robust classiﬁer.

Original: In Fig.
Revised: In Fig.

Original: 6, we show DVCEs of the three different classes for two examples images and two target classes each.
Revised: 6, we show DVCEs of the three different classes for two examples images and two target classes each.

Original: All of the multiple-norm robust models have similarly good DVCEs showing classiﬁer-speciﬁc variations in the semantic changes.
Revised: All of the multiple-norm robust models have similarly good DVCEs showing classiﬁer-speciﬁc variations in the semantic changes.

Original: This proves again the generative properties of adversarially robust models, in particular of robust transformers.
Revised: This proves again the generative properties of adversarially robust models, in particular of robust transformers.

Original: More examples and further experiments are in App.
Revised: More examples and further experiments are in App.

Original: B.4.
Revised: B.4.

Original: [Figure]
Revised: [Figure]

Original: [Table] 
Revised: [Table] [Figure] [Table] [Figure]

Original:  [Figure]
Revised: [Table] [Figure] [Table] [Figure]

Original: [Table] 
Revised: [Table] [Figure] [Table] [Figure]

Original:  [Figure]
Revised: [Table] [Figure] [Table] [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure] [Table] 
Revised: [Figure] [Table] [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table][Figure] [Table] [Figure]
Revised: [Table]

Original: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original:  Limitations, Future Work and Societal impact
Revised:  Limitations, Future Work and Societal impact

Original: In comparison to GANs, diffusion-based approaches can be expensive to evaluate as they require multiple iterations of the reverse process to create one sample.
Revised: In comparison to GANs, diffusion-based approaches can be expensive to evaluate as they require multiple iterations of the reverse process to create one sample.

Original: This can be an issue when creating a large amount of VCEs and makes deployment in a time-sensitive setting challenging.
Revised: This can be an issue when creating a large amount of VCEs and makes deployment in a time-sensitive setting challenging.

Original: We also rely on a robust model during the creation of VCEs for the cone projection.
Revised: We also rely on a robust model during the creation of VCEs for the cone projection.

Original: While we show that standard classiﬁers do not yield the desired gradients, training robust models can be challenging.
Revised: While we show that standard classiﬁers do not yield the desired gradients, training robust models can be challenging.

Original: An interesting direction for future research is thus to replace the cone projection with another “denoising” procedure for the gradient of a non-robust model.
Revised: An interesting direction for future research is thus to replace the cone projection with another “denoising” procedure for the gradient of a non-robust model.

Original: From a theoretical standpoint, using conditional sampling with reverse transitions of the form (7) is justiﬁed, in practice, however, we have to approximate the reverse transitions with shifted normal distributions of the form (9).
Revised: From a theoretical standpoint, using conditional sampling with reverse transitions of the form (7) is justiﬁed, in practice, however, we have to approximate the reverse transitions with shifted normal distributions of the form (9).

Original: This approximation relies on the conditioning function having low curvature, and it can be hard to verify this once we start adding a classiﬁer, distance and other possible terms and it is unclear how this inﬂuences the outcome of the diffusion process.
Revised: This approximation relies on the conditioning function having low curvature, and it can be hard to verify this once we start adding a classiﬁer, distance and other possible terms and it is unclear how this inﬂuences the outcome of the diffusion process.

Original: The quantitative evaluation of VCEs is difﬁcult, as the standard FID metric compares the distribution of features of a classiﬁer over a test and a generated dataset.
Revised: The quantitative evaluation of VCEs is difﬁcult, as the standard FID metric compares the distribution of features of a classiﬁer over a test and a generated dataset.

Original: However, for VCEs it is not only important to generate realistic images, but also to achieve high conﬁdence and to create meaningful changes.
Revised: However, for VCEs it is not only important to generate realistic images, but also to achieve high conﬁdence and to create meaningful changes.

Original: Moreover, metrics such as IM1, IM2 [30] for VCEs rely on a well-trained (V)AE for every class, which is difﬁcult to achieve for a dataset with 1000 classes and high-resolution images.
Revised: Moreover, metrics such as IM1, IM2 [30] for VCEs rely on a well-trained (V)AE for every class, which is difﬁcult to achieve for a dataset with 1000 classes and high-resolution images.

Original: Future research should therefore try to develop metrics for the quantitative evaluation of VCEs and we think that the evaluation in Tab. 1 is the ﬁrst step in that direction.
Revised: Future research should therefore try to develop metrics for the quantitative evaluation of VCEs and we think that the evaluation in Tab. 1 is the ﬁrst step in that direction.

Original: DVCEs and VCEs in general help to discover biases of the classiﬁers and thus have a positive societal impact, however one can abuse them for unintended purposes as any conditional generative model.
Revised: DVCEs and VCEs in general help to discover biases of the classiﬁers and thus have a positive societal impact, however one can abuse them for unintended purposes as any conditional generative model.

Original:  Conclusion
Revised:  Conclusion

Original: We have proposed DVCEs, a novel way to create VCEs for any state-of-the-art ImageNet classiﬁer using diffusion models and our cone projection.
Revised: We have proposed DVCEs, a novel way to create VCEs for any state-of-the-art ImageNet classiﬁer using diffusion models and our cone projection.

Original: DVCEs can handle vastly different image conﬁgurations, object sizes, and classes and satisfy all desired properties of VCEs.
Revised: DVCEs can handle vastly different image conﬁgurations, object sizes, and classes and satisfy all desired properties of VCEs.

Original: Acknowledgement
Revised: Acknowledgement

Original: The authors acknowledge support by the DFG Excellence Cluster Machine Learning - New Perspectives for Science, EXC 2064/1, Project number 390727645 and the German Federal Ministry of
Revised: The authors acknowledge support by the DFG Excellence Cluster Machine Learning - New Perspectives for Science, EXC 2064/1, Project number 390727645 and the German Federal Ministry of

Original: Education and Research (BMBF):
Revised: Education and Research (BMBF):

Original: Tübingen AI Center, FKZ: 01IS18039A as well as the DFG grant 389792660 as part of TRR 248.
Revised: Tübingen AI Center, FKZ: 01IS18039A as well as the DFG grant 389792660 as part of TRR 248.