Original: Diffusion Visual Counterfactual Explanations
Revised: Diffusion Visual Counterfactual Explanations

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Visual Counterfactual Explanations (VCEs) are an important tool to understandthe decisions of an image classiﬁer.
Revised: Visual Counterfactual Explanations (VCEs) are an important tool to understandthe decisions of an image classiﬁer.

Original: They are “small” but “realistic” semanticchanges of the image changing the classiﬁer decision.
Revised: They are “small” but “realistic” semanticchanges of the image changing the classiﬁer decision.

Original: Current approaches forthe generation of VCEs are restricted to adversarially robust models and oftencontain non-realistic artefacts or are restricted to image classiﬁcation problemswith few classes.
Revised: Current approaches forthe generation of VCEs are restricted to adversarially robust models and oftencontain non-realistic artefacts or are restricted to image classiﬁcation problemswith few classes.

Original: In this paper, we overcome this by generating Diffusion Visual
Revised: In this paper, we overcome this by generating Diffusion Visual

Original: Counterfactual Explanations (DVCEs) for arbitrary ImageNet classiﬁers via adiffusion process.
Revised: Counterfactual Explanations (DVCEs) for arbitrary ImageNet classiﬁers via adiffusion process.

Original: Two modiﬁcations to the diffusion process are key for our
Revised: Two modiﬁcations to the diffusion process are key for our

Original: DVCEs: i) an adaptive parameterization allows us to have hyperparameters thatgeneralize across images and models and together with distance regularization andlate start of the diffusion process allow us to generate close images and ii) our coneregularization via an adversarially robust model ensures that the diffusion processdoes not converge to trivial non-semantic changes but instead produces realisticimages of the target class which achieve high conﬁdence by the classiﬁer.
Revised: DVCEs: i) an adaptive parameterization allows us to have hyperparameters thatgeneralize across images and models and together with distance regularization andlate start of the diffusion process allow us to generate close images and ii) our coneregularization via an adversarially robust model ensures that the diffusion processdoes not converge to trivial non-semantic changes but instead produces realisticimages of the target class which achieve high conﬁdence by the classiﬁer.

Original:  Introduction
Revised:  Introduction

Original: It can be argued that one of the main problems hindering the widespread use of machine learningis the missing possibility to explain the decisions of black-box models such as neural networks.
Revised: It can be argued that one of the main problems hindering the widespread use of machine learningis the missing possibility to explain the decisions of black-box models such as neural networks.

Original: This is not only a problem for decisions affecting humans where the current draft for AI regulationin Europe [9] requires “transparency”, but it is a pressing problem in all applications of machinelearning.
Revised: This is not only a problem for decisions affecting humans where the current draft for AI regulationin Europe [10] requires “transparency”, but it is a pressing problem in all applications of machinelearning.

Original: The reason, to some extent, is that humans would like to understand but also control if thelearning algorithm has captured the “concepts” of the underlying classes or if it just predicts wellusing spurious features, artefacts in the data set, or other sources of error.
Revised: The reason, to some extent, is that humans would like to understand but also control if thelearning algorithm has captured the “concepts” of the underlying classes or if it just predicts wellusing spurious features, artefacts in the data set, or other sources of error.

Original: In this paper, we focus onmodel-agnostic explanations which can be applied to any classiﬁer and do not rely on the speciﬁcstructure of the classiﬁer such as decision trees or linear classiﬁers.
Revised: In this paper, we focus onmodel-agnostic explanations which can be applied to any classiﬁer and do not rely on the speciﬁcstructure of the classiﬁer such as decision trees or linear classiﬁers.

Original: In this area, in particular forimage classiﬁcation, sensitivity based explanations [4], explanations based on feature attributions [3],saliency maps [40, 39, 14, 50, 45], Shapley additive explanations [26], and local ﬁts of interpretablemodels [33] have been proposed.
Revised: In this area, in particular forimage classiﬁcation, sensitivity based explanations [4], explanations based on feature attributions [3],saliency maps [47, 46, 19, 57, 52], Shapley additive explanations [33], and local ﬁts of interpretablemodels [40] have been proposed.

Original: Moreover, [49] proposed counterfactual explanations (CEs), which are instance-speciﬁc explanations.
Revised: Moreover, [56] proposed counterfactual explanations (CEs), which are instance-speciﬁc explanations.

Original: They can be applied to any classiﬁer and it has been argued that they are close to human justiﬁcationof decisions [28] using counterfactual reasoning: “I would have recognized it as zebra (instead ofhorse), if it would have had black and white stripes.”
Revised: They can be applied to any classiﬁer and it has been argued that they are close to the humanjustiﬁcation of decisions [35] using counterfactual reasoning: “I would have recognized it as zebra(instead of horse), if it would have had black and white stripes.”

Original: For a given classiﬁer almost all methods for thegeneration of CEs [49, 12, 29, 5, 32, 48, 37] try to solve the following problem: “Given a target classc what is the minimal change  of input x , such that x +  is classiﬁed as class c with high probabilityand is  realistic instance of my data generating distribution?”
Revised: For a given classiﬁer almost allmethods for the generation of CEs [56, 16, 36, 6, 39, 55, 44] try to solve the following problem:“Given a target class c what is the minimal change  of input x , such that x +  is classiﬁed as classc with high probability and is a realistic instance of my data generating distribution?”

Original: From the perspective of “debugging”existing machine learning models, CEs are interesting as they construct a concrete input x +  withdifferent classiﬁcation which allows the developer to test if the model has learned the correct features.
Revised: From theperspective of “debugging” existing machine learning models, CEs are interesting as they construct aconcrete input x +  with a different classiﬁcation that allows the developer to test if the model haslearned the correct features.

Original: The main problem why visual counterfactual explanations (VCEs), that is CEs for image classiﬁcation,are not widely used is that the problem of generating CEs is very related to the problem of adversarial

Original: [Figure]
Revised: [Figure]

Original: examples [46] and thus even imperceivable changes of the image can already change the prediction,which clearly does not show to the user if the classiﬁer has picked up the right class-speciﬁc features.
Revised: The main problem why visual counterfactual explanations (VCEs), that is CEs for image classiﬁcation,are not widely used is that the problem of generating CEs is very related to the problem of adversarialexamples [53] and thus even imperceivable changes of the image can already change the prediction,which clearly does not show to the user if the classiﬁer has picked up the right class-speciﬁc features.

Original: One possible solution are adversarially robust models [36, 1, 6], which have been shown to producesemantically meaningful VCEs by directly maximizing the probability of the target class in imagespace.
Revised: One possible solution is using adversarially robust models [43, 1, 7], which have been shown toproduce semantically meaningful VCEs by directly maximizing the probability of the target class inimage space.

Original: These approaches have the downside that they only can generate VCEs for robust modelswhich is a signiﬁcant restriction as these models are not competitive in terms of prediction accuracy.
Revised: These approaches have the downside that they only can generate VCEs for robust modelswhich is a signiﬁcant restriction as these models are not competitive in terms of prediction accuracy.

Original: The second approach is to restrict the generation of VCEs using a generative model or constrainingthe set of potential image manipulations [16, 17, 35, 7, 15, 38, 22].
Revised: The second approach is to restrict the generation of VCEs using a generative model or constrainingthe set of potential image manipulations [21, 22, 42, 8, 20, 45, 29].

Original: However, these approaches areeither restricted to datasets with a small number of classes, cannot provide explanations for arbitraryclassiﬁers, or the generated images look realistic but are far away from the original one, so that the
Revised: However, these approaches areeither restricted to datasets with a small number of classes, cannot provide explanations for arbitraryclassiﬁers, or the generated images look realistic but are far away from the original one, so that the

Original: VCE and the original image have so little in common that not much insight can be gained.
Revised: VCE and the original image have so little in common that not much insight can be gained.

Original: Recently, [21] trained a StyleGAN model to discover and manipulate class attributes.
Revised: Recently, [28] trained a StyleGAN model to discover and manipulate class attributes.

Original: While theirapproach yields impressive results on smaller-scale datasets with few similar classes (for example,different bird species), the authors did not demonstrate that the method scales to complex tasks suchas ImageNet with hundreds of classes where different classes require different sets of attributes.
Revised: While theirapproach yields impressive results on smaller-scale datasets with few similar classes (for example,different bird species), the authors did not demonstrate that the method scales to complex tasks suchas ImageNet with hundreds of classes where different classes require different sets of attributes.

Original: Another disadvantage is that the StyleGAN model needs to be retrained for every classiﬁer, making itprohibitively expensive to explain multiple large models.
Revised: Another disadvantage is that the StyleGAN model needs to be retrained for every classiﬁer, making itprohibitively expensive to explain multiple large models.

Revised: Moreover, [27] have proposed a loss to do

Revised: VCEs in the latent space of a GAN/VAE.

Revised: While they show promising results for MNIST/FMNIST,there is no code available for generating VCEs on ImageNet.

Revised: Furthermore, they rely on a conditional

Revised: GAN, which might introduce a bias as it interferes with the classiﬁer, and it is unclear, how well anunconditional generative model will work.

Original: In this paper, we overcome this by generating our Diffusion Visual Counterfactual Explanations(DVCEs) for arbitrary ImageNet classiﬁers.
Revised: In this paper, we overcome this by generating our Diffusion Visual Counterfactual Explanations(DVCEs) for arbitrary ImageNet classiﬁers.

Original: We use the progress in the generation of realistic imagesusing diffusion processes [41, 20, 42, 44] reaching recently the level of GANs [11].
Revised: We use the progress in the generation of realistic imagesusing diffusion processes [48, 26, 49, 51] reaching recently the level of GANs [15].

Original: Similarly to[31, 2], we use a classiﬁer and a distance-type regularization to guide the generation of the images.
Revised: Similarly to[38, 2], we use a classiﬁer and a distance-type regularization to guide the generation of the images.

Original: Two modiﬁcations to the diffusion process are key elements for our DVCEs: i) a combinationof distance regularization and starting point of the diffusion process together with an adaptivereparameterization lets us generate VCEs visually close to the original image in a controlled way sothat hyperparameters can be ﬁxed across images and even across models, ii) our cone regularizationof the gradient of the classiﬁer via an adversarially robust model ensures that the diffusion processdoes not converge to trivial non-semantic changes but instead produces realistic images of the targetclass which achieve high conﬁdence by the classiﬁer.
Revised: Two modiﬁcations to the diffusion process are key elements for our DVCEs: i) a combinationof distance regularization and starting point of the diffusion process together with an adaptivereparameterization lets us generate VCEs visually close to the original image in a controlled way sothat hyperparameters can be ﬁxed across images and even across models, ii) our cone regularizationof the gradient of the classiﬁer via an adversarially robust model ensures that the diffusion processdoes not converge to trivial non-semantic changes but instead produces realistic images of the targetclass which achieve high conﬁdence by the classiﬁer.

Original: Our approach can be employed to any datasetwhere a generative diffusion model and an adversarially robust model are available.
Revised: Our approach can be employed for any datasetwhere a generative diffusion model and an adversarially robust model are available.

Original: In a qualitativecomparison and a user study in App.
Revised: In a qualitativecomparison and a user study in App.

Original: C we show that our DVCEs outperform the recent method of[6] as well as an adaptation of [2] for the generation of VCEs.
Revised: C we show that our DVCEs outperform the recent method of[7] as well as an adaptation of [2] for the generation of VCEs.

Original:  Diffusion models
Revised:  Diffusion models

Original: Diffusion models [11, 41, 42, 20, 30] are generative models that consist of two steps: forward diffusionprocess (and thus a Markov process) that transforms the data distribution to a prior distribution (whichis usually assumed to be a standard normal distribution) and the reverse diffusion processthattransforms the prior distribution back to the data distribution. The existence of the (continuous-time)reverse diffusion process was closer investigated in [44].
Revised: Diffusion models [15, 48, 49, 26, 37] are generative models that consist of two steps reverse diffusion process was closer investigated in [51].

Original: In the discrete-time setting, a Markov chain { x 1 , ..., x T } for any data point x 0 in the forward directionis deﬁned via a Markovian process that adds noise to the data point at each timestep:q ( x t | x t  1 ) = N ( p
Revised: In the discrete-time setting, a Markov chain { x 1 , ..., x T } for any data point x 0 in the forward directionis deﬁned via a Markovian process that adds noise to the data point at each timestep:

Original: [Equation]
Revised: [Equation]

Original: where {  1 , ...,  T } is some variance schedule, chosen such that q ( x T ) ⇠ N (0 , I ) .
Revised: where {  1 , ...,  T } is some variance schedule, chosen such that q ( x T ) ⇠ N (0 , I ) .

Original: Note that given 78 x 0 , it is possible to sample from q ( x t | x 0 ) in closed form instead of applying noise t times using:
Revised: Note that given 83 x 0 , it is possible to sample from q ( x t | x 0 ) in closed form instead of applying noise t times using:

Original: [Equation]
Revised: [Equation]

Original: where ↵ t :=
Revised: where ↵ t :=

Original: Q tk =1 (1   k ) .
Revised: Q tk =1 (1   k ) .

Original: As the number of total time steps T approaches inﬁnity,  the reversetransitions q ( x t  1 | x t ) approach diagonal Gaussian distributions.
Revised: As the number of total time steps, T approaches inﬁnity, and the reversetransitions q ( x t  1 | x t ) approach diagonal Gaussian distributions.

Original: Then one uses a DNN to predictmean µ ✓ and covariance ⌃ ✓ for this reverse process with transition probabilities:
Revised: Then one uses a DNN to predictmean µ ✓ and covariance ⌃ ✓ for this reverse process with transition probabilities:

Original: [Equation]
Revised: [Equation]

Original: To sample from p ✓ ( x 0 ) , one samples x T from N (0 , I ) and then follows the reverse process byrepeatedly sampling from the transition probabilities p ✓ ( x t  1 | x t ) .
Revised: To sample from p ✓ ( x 0 ) , one samples x T from N (0 , I ) and then follows the reverse process byrepeatedly sampling from the transition probabilities p ✓ ( x t  1 | x t ) .

Original: Originally, diffusion models weretrained by optimizing a variational lower bound of the data likelihood [41].
Revised: Originally, diffusion models weretrained by optimizing a variational lower bound of the data likelihood [48].

Original: Later, [20] proposed asimpliﬁed loss L simple that resembles a denoising auto-encoder:[Equation]
Revised: Later, [26] proposed asimpliﬁed loss L simple that resembles a denoising auto-encoder:

Original: [Equation]
Revised: [Equation]

Original: where µ can be recovered as: µ ✓ ( x t, t )
Revised: where µ can be recovered as: µ ✓ ( x t

Revised: [Equation]

Original:  = 1 p ↵ t ( x t  1  ↵ t p 1  ↵ t ✏ ✓ ( x t , t )) .
Revised: t ) = 1 p ↵ t ( x t  1  ↵ t p 1  ↵ t ✏ ✓ ( x t , t )) .

Original: [30, 11] combined both lossesto learn ⌃ and µ .
Revised: [37, 15] combined both lossesto learn ⌃ and µ .

Original: In the continuous-time setting, [43, 19] have derived the respective lower bounds.
Revised: In the continuous-time setting, [50, 25] have derived the respective lower bounds.

Original: 2.1 Class conditional sampling
Revised: 2.1 Class conditional sampling

Original: There exists diffusion models trained with and without knowledge about classes in the dataset[11, 18].
Revised: There exist diffusion models trained with and without knowledge about classes in the dataset [15, 24].

Original: For our experiments, we only use the class-unconditional diffusion model, such that all classconditional features are introduced by the classiﬁer which we want to explain.
Revised: For our experiments, we only use the class-unconditional diffusion model, such that all classconditional features are introduced by the classiﬁer which we want to explain.

Original: For the noise-awareclassiﬁer, the reverse process transitions are then of the form (Proof appendix H in [11]):
Revised: For the noise-awareclassiﬁer, the reverse process transitions are then of the form (Proof Appendix H in [15]):

Original: [Equation]
Revised: [Equation]

Original:  for a normalization constant Z .
Revised: (5) for a normalization constant Z .

Original: Typically p ✓ ( y |· ) is a time-step aware classiﬁer speciﬁcally trainedon noisy images belonging to time-step t [11].
Revised: Typically p ✓ ( y |· ) is a time-step aware classiﬁer speciﬁcally trainedon noisy images belonging to time-step t [15].

Original: As further we would like to explain any classiﬁer andnot only noise-aware ones, we follow the approach of [2], where a classiﬁer p ✓ ( y |· ) is given as aninput the denoised sample ˆ x 0 = f dn ( x t ) of x t , using the mapping:
Revised: As further we would like to explain any classiﬁer andnot only noise-aware ones, we follow the approach of [2], where a classiﬁer p ✓ ( y |· ) is given as aninput the denoised sample ˆ x 0 = f dn ( x t ) of x t , using the mapping:

Original: [Equation]
Revised: [Equation]

Original: This mapping is derived from eq. (2), from which we have that
Revised: This mapping is derived from Eq. 2, from which we have that

Original: [Equation]
Revised: [Equation]

Original:  and because we use our diffusion model ✏ ✓ ( x t , t ) to predict ✏ .
Revised: (7) and because we use our diffusion model ✏ ✓ ( x t , t ) to predict ✏ .

Original: With this, we can deﬁne a time-stepaware posterior p ✓ ( y | x t , t ) for any classiﬁer p ✓ ( y |· ) as p ✓ ( y | x t , t ) =
Revised: With this, we can deﬁne a time-stepaware posterior p ✓ ( y | x t , t ) for any classiﬁer p ✓ ( y |· ) as p ✓ ( y | x t , t ) =

Original: p ✓ ( y 
Revised: p ✓ ( y | f

Original: [Equation]
Revised: [Equation]

Original: f dn ( x t , t )) .
Revised: ( x t , t )) .

Original: The issue is that efﬁcient sampling from the original diffusion model is possible only because thereverse process is made of normal distributions.
Revised: The issue is that efﬁcient sampling from the original diffusion model is possible only because thereverse process is made of normal distributions.

Original: As we need to sample from p ✓ ( x t  1 | x t , y ) thousandsof times to obtain a single sample from the data distribution, it is not possible to use MCMC-samplerswith high complexity to sample from each of the individual transitions.
Revised: As we need to sample from p ✓ ( x t  1 | x t , y ) thousandsof times to obtain a single sample from the data distribution, it is not possible to use MCMC-samplerswith high complexity to sample from each of the individual transitions.

Original: In [11], they proposed to solveit by approximating p ✓ ( x t  1 | x t , y ) with slightly shifted versions of p ✓ ( x t  1 | x t ) to make closed-form 105 sampling possible.
Revised: In [15], they proposed to solveit by approximating p ✓ ( x t  1 | x t , y ) with slightly shifted versions of p ✓ ( x t  1 | x t ) to make closed-form 110 sampling possible.

Original: Such transition kernels are given by:
Revised: Such transition kernels are given by:

Original: [Equation]
Revised: [Equation]

Original:  where
Revised: 8) where

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original:  Diffusion Visual Counterfactual Explanations
Revised:  Diffusion Visual Counterfactual Explanations

Original: A VCE z for a chosen target class c and a given classiﬁer f and input x should satisfy the followingcriteria: i) validity: the VCE z should be classiﬁed by f as the desired target class c with highprobability, ii) realism: the VCE z should be indistinguishable or at least close to a natural image, iii)minimality/closeness: the difference of the VCE z and the original image x should be the minimalsemantic modiﬁcation necessary to change the class, in particular the generated image z should beclose to x while being valid and realistic, e.g. by changing the object in the image and leaving thebackground unchanged.
Revised: A VCE z for a chosen target class c and a given classiﬁer f and input x should satisfy the followingcriteria: i) validity: the VCE z should be classiﬁed by f as the desired target class c with highprobability, ii) realism: the VCE z should be indistinguishable or at least close to a natural image, iii)minimality/closeness: the difference of the VCE z and the original image x should be the minimalsemantic modiﬁcation necessary to change the class, in particular, the generated image z should beclose to x while being valid and realistic, e.g. by changing the object in the image and leaving thebackground unchanged.

Original: Note that while targeted adversarial examples are valid, they do not showmeaningful semantic changes into the target class for a non-robust model and are also not realistic.
Revised: Note that while targeted adversarial examples are valid, they do not showmeaningful semantic changes into the target class for a non-robust model and are also not realistic.

Original: Although adversarial l 1 perturbations are (almost) imperceivable for humans, adversarial images arevery unlikely to occur in nature.
Revised: Although adversarial l 1 perturbations are (almost) imperceivable for humans, adversarial images arevery unlikely to occur in nature.

Original: The l 1 . 5 -SVCE method of [6] optimizes the predicted probability of classiﬁer into the target class bymanipulating the image and is very close to the generation of adversarial examples and thus worksonly for adversarially robust classiﬁers.
Revised: The l 1 . 5 -SVCE method of [7] optimizes the predicted probability of classiﬁer into the target class bymanipulating the image and is very close to the generation of adversarial examples and thus worksonly for adversarially robust classiﬁers.

Original: The realism of the image comes purely from the generativeproperties of robust classiﬁers [36] which can lead to artefacts.
Revised: The realism of the image comes purely from the generativeproperties of robust classiﬁers [43] which can lead to artefacts.

Original: In contrast, our Diffusion Visual
Revised: In contrast, our Diffusion Visual

Original: Counterfactal Explanations (DVCEs) works for any classiﬁer and our generated images are morerealistic due to the integration of diffusion models.
Revised: Counterfactual Explanations (DVCEs) work for any classiﬁer and our generated images are morerealistic due to the integration of diffusion models.

Original: An approach similar to our DVCE framework isblended diffusion of [2] which manipulates the image inside a masked region.
Revised: An approach similar to our DVCE framework isblended diffusion of [2] which manipulates the image inside a masked region.

Original: However, it is easy toadapt it to the task of VCEs by allowing the whole image to be manipulated.
Revised: However, it is easy toadapt it to the task of VCEs by allowing the whole image to be manipulated.

Original: DVCE and blendeddiffusion share the same diffusion model, but, as we show in the appendix, blended diffusion cannotbe applied to arbitrary classiﬁers and requires image-speciﬁc hyperparameter tuning, see Fig.2,which is infeasible in practice.
Revised: DVCE and blendeddiffusion share the same diffusion model, but, as we show in the Appendix, blended diffusion cannotbe applied to arbitrary classiﬁers and requires image-speciﬁc hyperparameter tuning, see Fig.

Original: 3.1 Adaptive Parameterization
Revised: 3.1 Adaptive Parameterization

Original: For the DVCE of the original image ˆ x , we don’t want to sample just any image x from p ( x | y ) (highrealism and validity) but also to make sure that d ( x, ˆ x ) is small in order to enforce a minimal change.
Revised: For the DVCE of the original image ˆ x , we don’t want to sample just any image x from p ( x | y ) (highrealism and validity) but also to make sure that d ( x, ˆ x ) is small.

Original: To do this, we have to condition our diffusion process on ˆ x .
Revised: To do this, we have to condition ourdiffusion process on ˆ x .

Original: Using the introduced denoising functionf dn and analogously to the derivation in [11] and following application in [22], the mean of the transition kernel becomes:
Revised: Using the introduced denoising function f dn and analogously to the derivation in [15] and following application in [29], the mean of the transition kernel becomes:

Original: [Equation]
Revised: [Equation]

Original: where C c is the coefﬁcient of the classiﬁer, and C d the one of the distance guiding loss.
Revised: where C c is the coefﬁcient of the classiﬁer, and C d the one of the distance guiding loss.

Revised: Intuitively,we take a step in the direction that increases the classiﬁer score while staying close to ˆ x .

Original: Note that thelast term can be interpreted as the log gradient of a distribution with density exp(  d t (ˆ x, · )  ) , where = 1 C d and d t (ˆ x, · ) =
Revised: Note that thelast term can be interpreted as the log gradient of a distribution with density exp(  d t (ˆ x, · )  ) , where = 1 C d and d t (ˆ x, · ) =

Original: d (ˆ x, f dn ( · , t )) , thus we introduce a time-aware prior distribution that enforcesour output to be similar to ˆ x .
Revised: d (ˆ x, f dn ( · , t )) , thus we introduce a time-aware prior distribution that enforcesour output to be similar to ˆ x .

Original: In our work, we use the l 1 -distance as it produces sparser changes.
Revised: In our work, we use the l 1 -distance as it produces sparser changes.

Original: Several works have tried to minimize some distance during the diffusion process, implicitly in [8] orexplicitly, but for the background in [2].
Revised: Several works have tried to minimize some distance during the diffusion process, implicitly in [9] orexplicitly, but for the background in [2].

Original: However, there is no principled way to choose the coefﬁcientfor such a regularization term.
Revised: However, there is no principled way to choose the coefﬁcientfor such a regularization term.

Original: In fact it turns out that it is impossible to ﬁnd a parameter setting forblended diffusion [2] that works across images and classiﬁers.
Revised: It turns out that it is impossible to ﬁnd a parameter setting for blendeddiffusion [2] that works across images and classiﬁers.

Original: Thus,in this paper we propose anadaptive parametrizationg update
Revised: Thus, we propose an adaptive parametrization

Original: [Equation]
Revised: [Equation]

Original: relative to the predicted mean of the diffusion model that we use to change µ t in Eq.
Revised: relative to the predicted mean of the diffusion model that we use to change µ t in Eq.

Original: 9 to
Revised: 9 to

Original: [Equation]
Revised: [Equation]

Original: The adaptive parameterization allows for a ﬁnegrained control of the inﬂucence of the classiﬁer anddistance regularization so that now the hyperparameters C c and C d have the same inﬂuence acrossimages and even classiﬁers which facilitates the generation of DVCEs as otherwise hyperparameterﬁnetuning would be necessary for each image as in blended diffusion, see Fig.
Revised: The adaptive parameterization allows for ﬁne-grained control of the inﬂucence of the classiﬁer anddistance regularization so that now the hyperparameters C c and C d have the same inﬂuence acrossimages and even classiﬁers which facilitates the generation of DVCEs as otherwise hyperparameterﬁnetuning would be necessary for each image as in blended diffusion, see Fig.

Original: 2 for a comparison.
Revised: 2 for a comparison.

Original: However, even with our adaptive parameterization, it is still not easy to produce semanticallymeaningful changes close to ˆ x as can be seen in App.
Revised: However, even with our adaptive parameterization, it is still not easy to produce semanticallymeaningful changes close to ˆ x as can be seen in App.

Original: B.2.
Revised: B.2.

Original: Thus, as in [2], we vary the starting pointin the diffusion process and observe in App.
Revised: Thus, as in [2], we vary the starting pointin the diffusion process and observe in App.

Original: B.2 that starting from step T 2 of the forward diffusionprocess, together with the adaptive parameterization and using as  distance the l 1 -distance provides uswith sparse but semantically meaningful changes.
Revised: B.2 that starting from step T 2 of the forward diffusionprocess, together with the adaptive parameterization and using as the distance the l 1 -distance providesus with sparse but semantically meaningful changes.

Revised: In our experiments, we set T = 200 .

Original: 3.2 Cone-Projection for Classiﬁer Guidance
Revised: 3.2 Cone-Projection for Classiﬁer Guidance

Original: A major component of our DVCEs is that they can be applied to any image classiﬁer regardless ofwhether it is adversarially robust or not.
Revised: A major component of our DVCEs is that they can be applied to any image classiﬁer regardless ofwhether it is adversarially robust or not.

Original: However, it turns out that naively applying diffusion with thenew mean (Eq. 12) to a non-robust classiﬁer does not work and leads to very small modiﬁcationsof the image, which are similar to adversarial examples without meaningful changes.
Revised: However, it turns out that naively applying diffusion with thenew mean (Eq. 12) to a non-robust classiﬁer does not work and leads to very small modiﬁcations ofthe image, which are similar to adversarial examples without meaningful changes.

Original: The reasonis that gradients of non-robust classiﬁers are much more noisy and less semantically meaningful.
Revised: The reason is thatgradients of non-robust classiﬁers are much noisier and less semantically meaningful.

Original: We illustrate this for a non-robust Swin-Transformer[23] in Fig.
Revised: We illustratethis for a non-robust Swin-Transformer[30] in Fig.

Original: 3 where hardly any changes aregenerated.
Revised: 3 where hardly any changes are generated.

Original: Note that this is even the case if we denoise the sample ﬁrst using the denoising functionf
Revised: Notethat this is even the case if we denoise the sample ﬁrst using the denoising function

Original:  dn (Eq. 6).
Revised: f dn (Eq. 6).

Original: As a solution to this problem we suggest to project the gradient of a robust classiﬁer, r x t log p robust , ✓ ( y | f dn ( x t , t )) , onto a cone centered at the gradient of the classiﬁer,r x t log p ✓ ( y | f dn ( x t , t )) , (we use ↵ = 30  in all experiments for non-robust classiﬁers).
Revised: As a solution, we suggest projecting the gradient of a robust classiﬁer, r x t log p robust , ✓ ( y | f dn ( x t , t )) ,onto a cone centered at the gradient of the classiﬁer, r x t log p ✓ ( y | f dn ( x t , t )) , (we use ↵ = 30  in allexperiments for non-robust classiﬁers).

Original: Moreprecisely, we deﬁne
Revised: More precisely, we deﬁne

Original: [Equation]
Revised: [Equation]

Original:  where cone ( ↵ , v ) := { w 2 R d : \ ( v, w )  ↵ } is the cone of angle ↵ around vector v , and theprojection P cone ( ↵ ,v ) onto cone ( ↵ , v ) is given as:
Revised: i , where cone ( ↵ , v ) := { w 2 R d : \ ( v, w )  ↵ } is the cone of angle ↵ around vector v , and theprojection P cone ( ↵ ,v ) onto cone ( ↵ , v ) is given as:

Original: [Equation]
Revised: [Equation]

Original: else , where P v ? ( w ) :
Revised: else , where P v ? ( w ) :

Original: = w  h w,v i h v,v i v
Revised: = w  h w,v i h v,v i v

Original: and we deﬁne u = sin( ↵ ) P v ?
Revised: and we deﬁne u = sin( ↵ ) P v ?

Original: ( w ) k P v ?
Revised: ( w ) k P v ?

Original: ( w ) k 2 +cos( ↵ ) v k v k 2 (note that k u k 2 = {+
+}
Revised: ( w ) k 2 + cos( ↵ ) v k v k 2 (note thatk u k 2 = {+
+}

Original: 1and h u, v i = cos( ↵ ) ).
Revised: and h u, v i = cos( ↵ ) ).

Original: The idea of this projection is that in this way we reduce drastically the noisein the gradient of the non-robust classiﬁer.
Revised: The idea of this projection is that in this way we reduce drasticallythe noise in the gradient of the non-robust classiﬁer.

Original: Note, however, that the projection of the gradient of therobust classiﬁer onto the cone generated by the non-robust classiﬁer is always an ascent directionfor log p ✓ ( y | f dn ( x t , t )) and we still have the goal of maximizing this objective (note that g proj is notnecessarily an ascent direction for log p robust , ✓ ( y | f dn ( x t , t )) ).
Revised: Note, however, that the projection of the gradientof the robust classiﬁer onto the cone generated by the non-robust classiﬁer is always an ascentdirection for log p ✓ ( y | f dn ( x t , t )) and we still have the goal of maximizing this objective (note thatg proj is not necessarily an ascent direction for log p robust , ✓ ( y | f dn ( x t , t )) ).

Original: Thus, one can see the cone projectionas a form of regularization of r x t log p ✓ ( y | f dn ( x t , t )) which guides the diffusion process intosemantically meaningful changes of the classiﬁer.
Revised: Thus, one can see the coneprojection as a form of regularization of r x t log p ✓ ( y | f dn ( x t , t )) which guides the diffusion processinto semantically meaningful changes of the classiﬁer.

Original: [Figure]
Revised: [Figure]

Original: While 3.3 Final Scheme for Diffusion Visual Counterfactuals
Revised: 3.3 Final Scheme for Diffusion Visual Counterfactuals

Original: For the convenience of the reader, we summarize here the full scheme of DVCEs for a nonadversarially robust classiﬁer p ✓ ( y | x ) where we assume that we are given additionally the adversarially robust model p robust , ✓ ( y | x ) :
Revised: We summarize here the full scheme of DVCEs for a non-adversarially robust classiﬁer p ✓ ( y | x ) wherewe assume that we are given additionally the adversarially robust model p robust , ✓ ( y | x ) :

Original: [Equation]
Revised: [Equation]

Original: For a robust model the only difference is that the cone projection is omitted and one uses g proj =
Revised: For a robust model, the only difference is that the cone projection is omitted and one uses g proj =

Original: r x t log p robust , ✓  y | f dn ( x t , t )  .
Revised: r x t log p robust , ✓ [Equation] )  .

Original: We use in all our experiments C c = 0 .
Revised: We use in all our experiments C c = 0 .

Original: 1 , and C d 2 0 .
Revised: 1 , and C d 2 0 .

Original: 15 unless weshow ablations for one of the parameters.
Revised: 15 unless weshow ablations for one of the parameters.

Original: The angle ↵ for the cone projection is ﬁxed to 30  .
Revised: The angle ↵ for the cone projection is ﬁxed to 30  .

Original: Instrong contrast to blended diffusion, these parameters generalize across images and even classiﬁersand allow us to generate VCEs for arbitrary classiﬁers for which a diffusion model is available.
Revised: Instrong contrast to blended diffusion, these parameters generalize across images and even classiﬁersand allow us to generate VCEs for arbitrary classiﬁers for which a diffusion model is available.

Original:  Experiments
Revised:  Experiments

Original: In this section we evaluate the quality of the visual counterfactual explanations of DVCE.
Revised: In this section, we evaluate the quality of the visual counterfactual explanations of DVCE.

Original: We start bycomparing DVCE to existing works in Sec. 4.1.
Revised: We start bycomparing DVCE to existing works in Sec. 4.1.

Original: In Sec. 4.2, we compare counterfactuals of DVCEfor various state-of-the-art ImageNet models and show, how our method can be used to interpretdifferences between classiﬁers.
Revised: In Sec. 4.2, we compare counterfactuals of DVCEfor various state-of-the-art ImageNet models and show, how our method can be used to interpretdifferences between classiﬁers.

Original: For all experiments, we use the same set of hyperparameters for
Revised: For all experiments, we use the same set of hyperparameters for

Original: DVCE given in the last section.
Revised: DVCE given in the last section.

Original: The diffusion model used for DVCE is taken from [11] and is trainedclass-unconditionally on ImageNet images of size 256x256 and uses a modiﬁed UNet
Revised: The diffusion model used for DVCE is taken from [15] and is trainedclass-unconditionally on ImageNet images of size 256x256 and uses a modiﬁed UNet

Original: architecture(more details in [11]).
Revised: architecture(more details in [15]).

Original: User study, further experiments, ablations and details can be found in the Appendix.
Revised: User study, further experiments, ablations, and details are in the Appendix.

Original: 4.1 Method comparison
Revised: 4.1 Method comparison

Original: Creating VCEs for ImageNet with 1000 vastly different classes is a challenging task and there onlyexist a small number of methods that try to solve it without reducing dataset size or image resolution.
Revised: Creating VCEs for ImageNet with 1000 vastly different classes is a challenging task and there onlyexist a small number of methods that try to solve it without reducing dataset size or image resolution.

Original: For evaluation purposes, we compare our method to blended diffusion [2] and a recent method for
Revised: For evaluation purposes, we compare our method to blended diffusion [2] and a recent method for

Original: VCEs based purely on optimization [6].
Revised: VCEs based purely on optimization [7].

Original: As the latter only works for adversarially robust classiﬁers,we use the multiple-norm robust model from [6], which they call Madry [13] + FT, as the classiﬁer tocreate VCEs for all three methods.
Revised: As the latter only works for adversarially robust classiﬁers,we use the multiple-norm robust model from [7], which they call Madry [17] + FT, as the classiﬁer tocreate VCEs for all three methods.

Original: Results can be found in Fig.
Revised: Results can be found in Fig.

Original: 4, where we transform one imageinto two different classes that are close in the WordNet hierarchy[27].
Revised: 4, where we transform one imageinto two different classes that are close in the WordNet hierarchy[34].

Original: The radius of the l 1 . 5 -ball of [6] is chosen as the smallest r 2 { 50 , 75 , 100 , 150 } such that theconﬁdence in the target class is larger than 0 .
Revised: The radius of the l 1 . 5 -ball of [7] is chosen as the smallest r 2 { 50 , 75 , 100 , 150 } such that theconﬁdence in the target class is larger than 0 .

Original: 90 per image - if this is not reached, we show the onewith highest conﬁdence.
Revised: 90 per image - if this is not reached, we show the onewith highest conﬁdence.

Original: For blended diffusion, we select the image with the smallest classiﬁer andregularization weight that reachesa conﬁdence larger than 0.9 from the set of parameters discussedin Sec. 3.3.
Revised: For blended diffusion, we select the image with the smallest classiﬁer andregularization weight that reaches conﬁdence larger than 0.9 from the set of parameters discussed in Sec. 3.3.

Original: If 0.9 is not reached by any setting, we again show the one with highest conﬁdence.
Revised: If 0.9 is not reached by any setting, we again show the one with highest conﬁdence.

Original: For
Revised: For

Revised: DVCEs, we only use the single set of parameters provided in the previous section.

Revised: As the model isrobust on its own, we do not use the cone projection in DVCE.

Original: [Figure]
Revised: [Figure]

Original: timber wolf, white wolf and alp).
Revised: We show our DVCEs (left column), the l 1 . 5 -SVCEs of [7] ( ), and the adaptation of blended diffusion [2] for VCEs (right). For each original image (outer left column) we show the VCEs for two target classes. See the text for the parameter selection of all methods.

Revised: Only DVCEs satisfy all desired properties of VCEs.

Revised: Blended diffusion fails for leopard and tiger and often produces images far away from the original one (pizza, potpie, timber wolf, white wolf, and alp).

Original: l 1 . 5 -SVCEs show artefacts (white wolf, volcano, night snake) and have often lower image quality.
Revised: l 1 . 5 -SVCEs show artefacts (white wolf, volcano, night snake) and have often lower image quality.

Original: DVCEs, we only use the single set of parameters provided in the previous section. As the model isrobust its own, we do not use the cone projection in DVCE.
Revised: As Fig.

Original: As Figure 4 shows, DVCEs is the only method that satisﬁes all desired properties of VCEs: i) theproduced VCEs have high conﬁdence in the target class, ii) are close to natural images and iii) showminimal semantic changes into the target class.
Revised: 4 shows, DVCEs is the only method that satisﬁes all desired properties of VCEs: i) theproduced VCEs have high conﬁdence in the target class, ii) are close to natural images, and iii)show minimal semantic changes in the target class.

Original: For example, for “mashed potato”, DVCE preservesthe bowl and only changes the content into either guacamole or carbonara.
Revised: For example, for “mashed potato”, DVCEpreserves the bowl and only changes the content into either guacamole or carbonara.

Original: Our qualitative comparisonshows that the same hyperparameter setting of DVCE can handle a variety of classes and transferbetween similar classes, such as different snakes or wolf types, as well as different object sizes.
Revised: Our qualitativecomparison shows that the same hyperparameter setting of DVCE can handle different classes andtransfer between similar classes, such as different snakes or wolf types, as well as different objectsizes.

Original: Forexample, it works for  “cheetah” which covers the entire image, but also for the “snake” which only
Revised: For example, it works for the “cheetah” which covers the entire image, but also for the “snake”which only

Original: [Figure]
Revised: [Figure]

Original: [Table] 
Revised: [Table] [Figure] [Table] [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Original:  [Figure]
Revised: [Table] [Figure] [Table] [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure] [Table] 
Revised: [Figure] [Table] [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: we show DVCEs into two target classes.

Original: Please zoom into the images to see more ﬁnegrained details and differences between the generated images.

Original: covers a fraction of the input.
Revised: covers a fraction of the input.

Original: In contrast, both l 1 . 5 -SVCEs and blended diffusion VCEs requiredifferent hyperparameters for different images to achieve high conﬁdence into the target class.
Revised: In contrast, both l 1 . 5 -SVCEs and blended diffusion VCEs requiredifferent hyperparameters for different images to achieve high conﬁdence in the target class.

Original: Evenwith the six parameter conﬁgurations, blended diffusion is not able to produce always images withhigh conﬁdence in the target class.
Revised: Evenwith the six parameter conﬁgurations, blended diffusion is not able to produce always images withhigh conﬁdence in the target class.

Original: Even more problematic for VCEs: sometimes the resulting imageshave high conﬁdence but do not look realistic
Revised: Even more problematic for VCEs: sometimes the resulting imageshave high conﬁdence but do not look realistic

Original: (cheetah ! tiger) or do not resemble the original imageat all (dingo !
Revised: (cheetah ! tiger) or do not resemble the original imageat all (dingo !

Original: timber wolf/white wolf).
Revised: timber wolf/white wolf).

Original: Even if the method works with the given parameters, forexample mashed potato ! guacamole or carbonara, the overall image quality cannot match that of
Revised: Even if the method works with the given parameters, forexample mashed potato ! guacamole or carbonara, the overall image quality cannot match that of

Original: DVCE as often the images contain overly bright colors.
Revised: DVCE as often the images contain overly bright colors.

Original: In the case of the volcano VCE, DVCEshows class features like lava whereas blended diffusion VCE can not clearly be labeled as a volcano.
Revised: In the case of the volcano VCE, DVCEshows class features like lava whereas blended diffusion VCE can not clearly be labeled as a volcano.

Original: For the l 1 . 5 -SVCEs, one often needs large radii to achieve the desired conﬁdence of 0.90, which oftenresults in color artefacts and images that generally do not look realistic.
Revised: For the l 1 . 5 -SVCEs, one often needs large radii to achieve the desired conﬁdence of 0.90, which oftenresults in color artefacts and images that generally do not look realistic.

Original: 4.2 Model comparison
Revised: 4.2 Model comparison

Original: In Fig.
Revised: In Fig.

Original: 5 we show that DVCEs can be generated for various state-of-the-art ImageNet [34] models.
Revised: 5 we show that DVCEs can be generated for various state-of-the-art ImageNet [41] models.

Original: Weuse a Swin-Transformer[23], a ConvNeXt [24] and a Noisy-Student EfﬁcientNet [51, 47].
Revised: Weuse a Swin-Transformer[30], a ConvNeXt [31] and a Noisy-Student EfﬁcientNet [58, 54].

Original: Both the
Revised: Both the

Original: Swin-Tranformer and the ConvNeXt are pretrained on ImageNet21k [10] wheareas the EfﬁcientNetuses noisy-student self-training on a large unlabeled pool.
Revised: Swin-Transformer and the ConvNeXt are pretrained on ImageNet21k [14] whereas the EfﬁcientNetuses noisy-student self-training on a large unlabeled pool.

Original: As all models do not yield perceptuallyaligned gradients, we use the cone projection with 30  angle described in Sec.
Revised: As all models do not yield perceptuallyaligned gradients, we use the cone projection with 30  angle described in Sec.

Original: 3.2 with the robustmodel from the previous section.
Revised: 3.2 with the robustmodel from the previous section.

Original: All other parameters are identical to the previous experiment.
Revised: All other parameters are identical to the previous experiment.

Original: We highlight that DVCE is the ﬁrst method capable of visualizing arbitrary classiﬁers on a task aschallenging as ImageNet [34].
Revised: We highlight that DVCE is the ﬁrst method capable of visualizing arbitrary classiﬁers on a task aschallenging as ImageNet [41].

Original: Overall, DVCEs satisfy all desired properties of VCEs.
Revised: Overall, DVCEs satisfy all desired properties of VCEs.

Original: It also allowsus to inspect the most important features for each model and class.
Revised: It also allowsus to inspect the most important features of each model and class.

Original: For example, all three modelshave learnt to associate the speciﬁc texture pattern of the box turtle with that particular class.
Revised: For example, all three models havelearned to associate the speciﬁc texture pattern of the box turtle with that particular class.

Original: For thestupa and church classes, it seems like all models use different roof and tower structures as the mostprominent feature as they spent most of their budget on changing those.
Revised: For thestupa and church classes, it seems like all models use different roof and tower structures as the mostprominent feature as they spent most of their budget on changing those.

Original:  Limitations, Future Work and Societal impact
Revised:  Limitations, Future Work and Societal impact

Original: In comparison to GANs, diffusion-based approaches can be expensive to evaluate as they requiremultiple iterations of the reverse process to create one sample.
Revised: In comparison to GANs, diffusion-based approaches can be expensive to evaluate as they requiremultiple iterations of the reverse process to create one sample.

Original: This can be an issue when creating alarge amount of VCEs and makes deployment in a time-sensitive setting challenging.
Revised: This can be an issue when creating alarge amount of VCEs and makes deployment in a time-sensitive setting challenging.

Original: We also rely on a robust model during the creation of VCEs for the cone projection.
Revised: We also rely on a robust model during the creation of VCEs for the cone projection.

Original: While weshow that standard classiﬁers do not yield the desired gradients, training robust models can bechallenging.
Revised: While weshow that standard classiﬁers do not yield the desired gradients, training robust models can bechallenging.

Original: An interesting direction for future research is thus to replace the cone projection withanother “denoising” procedures for the gradient of a non-robust model.
Revised: An interesting direction for future research is thus to replace the cone projection withanother “denoising” procedure for the gradient of a non-robust model.

Original: From a theoretical standpoint, using conditional sampling with reverse transitions of the form Eq.
Revised: From a theoretical standpoint, using conditional sampling with reverse transitions of the form Eq.

Original: 5is justiﬁed, in practice, however, we have to approximate the reverse transitions with shifted normaldistributions of the form Eq. 8.
Revised: 5is justiﬁed, in practice, however, we have to approximate the reverse transitions with shifted normaldistributions of the form Eq. 8.

Original: This approximation relies on the conditioning function having lowcurvature, and it can be hard to verify this once we start adding a classiﬁer, distance and other possibleterms and it is unclear how this inﬂuences the outcome of the diffusion process.
Revised: This approximation relies on the conditioning function having lowcurvature, and it can be hard to verify this once we start adding a classiﬁer, distance and other possibleterms and it is unclear how this inﬂuences the outcome of the diffusion process.

Original: The quantitative evaluation of VCEs is difﬁcult, as the standard FID metric compares the distributionof features of a classiﬁer over a test and a generated dataset.
Revised: The quantitative evaluation of VCEs is difﬁcult, as the standard FID metric compares the distributionof features of a classiﬁer over a test and a generated dataset.

Original: However, for VCEs it is not onlyimportant to generate realistic images, but also  achievea high conﬁdence and to create meaningfulchanges.
Revised: However, for VCEs it is not onlyimportant to generate realistic images, but also to achieve high conﬁdence and to create meaningfulchanges.

Original: Moreover, metrics such as IM1, IM2 [25] for VCEs rely on a well-trained (V)AE for everyclass, which is difﬁcult to achieve for a dataset with 1000 classes and high-resolution images.
Revised: Moreover, metrics such as IM1, IM2 [32] for VCEs rely on a well-trained (V)AE for everyclass, which is difﬁcult to achieve for a dataset with 1000 classes and high-resolution images.

Original: Futureresearch should therefore try to develop metrics for the quantitative evaluation of VCEs.
Revised: Futureresearch should therefore try to develop metrics for the quantitative evaluation of VCEs.

Original: DVCEs, and VCEs in general help to discover biases of the classiﬁers and thus have a positive societalimpact, however one can abuse them for unintended purposes as any conditional generative model.
Revised: DVCEs and VCEs in general help to discover biases of the classiﬁers and thus have a positive societalimpact, however one can abuse them for unintended purposes as any conditional generative model.

Original:  Conclusion
Revised:  Conclusion

Original: We have proposed DVCEs, a novel way to create VCEs for any state-of-the-art ImageNet classiﬁerusing diffusion models and our cone projection.
Revised: We have proposed DVCEs, a novel way to create VCEs for any state-of-the-art ImageNet classiﬁerusing diffusion models and our cone projection.

Original: Our DVCEs satisfy all desired properties of VCEs.
Revised: Our DVCEs satisfy all desired properties of VCEs.

Original: Throughout the paper, we show that DVCEs can handle vastly different image conﬁgurations, objectsizes and classes.Checklist For all authors...
Revised: Throughout the paper, we show that DVCEs can handle vastly different image conﬁgurations, objectsizes, and classes.

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s

Original: contributions and scope?

Original: [Yes]

Original: (b) Did you describe the limitations of your work?

Original: [Yes] See Sec. 5.

Original: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] See Sec.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?

Original: [Yes]

Original: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]

Original: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]

Original: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?

Original: [Yes] See supplemental material.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?

Original: [N/A]

Original: We didn’t train models.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [N/A] We have conducted qualitative experiments in the mainpaper for the same seed and show the diversity of VCEs across seeds in App.

Original: B.3.(d)

Original: Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] See App. D4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a)

Original: If your work uses existing assets, did you cite the creators?

Original: [Yes](b) Did you mention the license of the assets?

Original: [Yes] Yes, directly in citation, whenapplicable.(c)

Original: Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]We include our code in supplemental material.

Original: [Table]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?

Original: [N/A]

Original: ImangeNet and ImageNet21k are public datasets.(e)

Original: Did you discuss whether the data you are using/curating contains personally identiﬁableinformation or offensive content?

Original: [N/A]5.

Original: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots,if applicable?

Original: [Yes] We include the results from the user study together with thescreenshots of instructions in App. C.(b)

Original: Did you describe any potential participant risks, with links to Institutional ReviewBoard (IRB) approvals, if applicable?

Original: [N/A] Not applicable.(c)

Original: Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?

Original: [Yes] See App.

Original: C.