DeDUCE: Generating Counterfactual Explanations At Scale
Abstract
When an image classiﬁer outputs a wrong class label, it can be helpful to see what changes in the image would lead to a correct classiﬁcation.
This is the aim of algorithms generating counterfactual explanations.
However, there is no easily scalable method to generate such counterfactuals.
We develop a new algorithm providing counterfactual explanations for large image classiﬁers trained with spectral normalisation at low computational cost.
We empirically compare this algorithm against baselines from the literature; our novel algorithm consistently ﬁnds counterfactuals that are much closer to the original inputs.
At the same time, the realism of these counterfactuals is comparable to the baselines.
The code for all experiments is available at https://github.com/benedikthoeltgen/DeDUCE .
Introduction
Figure 1:
In many use cases, counterfactuals should lie just across the decision boundary as seen from the original x and lie on the data manifold, which is here satisﬁed by CF2 but not by CF1 (taken from [30]).
Much of the recent staggering success of machine learning is due to large and complex models whose inner workings are in many cases elusive.
The prime examples of such ’black box’ models are deep neural networks.
Despite the drawbacks that come with a poor understanding of their inner workings, such models are already widely deployed in practice due to their state-of-the-art performance in many areas.
But this poor understanding can make it difﬁcult to interpret the decision making process of such models, and thus to identify the problem when a model makes an error.
To explain erroneous classiﬁcations, it can therefore be helpful to investigate the relevant decision boundary.
An intuitive way to do so is to look for alternative inputs that are similar but classiﬁed differently; this is the realm of counterfactual explanations.
For debugging, we usually want to generate realistic counterfactuals that are just across the model’s classiﬁcation decision boundary (ﬁg. 1).
We want a counterfactual that stays as close to the original input as possible but results in a correct classiﬁcation, in order to understand the system’s decision making.
At the same time, the counterfactual should stay on the manifold of realistic images (CF2 in ﬁg. 1) since we are interested in inputs that can be interpreted.
Being a nascent area of research, there is so far no algorithm providing such counterfactual explanations for large models at low cost.
In this work, we introduce a fast novel algorithm using speciﬁc properties of residual networks (ResNets), a widely used model, to ﬁnd counterfactuals which are very close to the original input.
The contributions presented in this work are the following:
• We introduce and implement a new, scalable algorithm for generating counterfactual explanations in residual networks trained with spectral normalisation.
In comparison to previous approaches, this algorithm has a low engineering overhead as well as low computational requirements; it does not rely on auxiliary generative models and can run on single forward pass residual networks rather than on ensembles of models.
• We propose and implement a way to assess realism in counterfactual explanations, drawing on the literature on anomaly detection.
• We evaluate the novel algorithm on the MNIST dataset and compare its performance against three baselines from the literature.
Background 2.1 Counterfactual explanations
Counterfactual explanations for machine learning models have been introduced in [31].
They are used as post-hoc explanations for individual decisions.
In general, a counterfactual explanation is understood as the presentation of an alternative input x (cid:48) , the counterfactual, which is in some way similar to the original x yet leads to a different prediction.
Due to their similarity (both conceptually and w.r.t.
the algorithms that generate them), counterfactuals are often introduced in juxtaposition with adversarial examples (AEs).
3 Freiesleben [5] provide an illuminating discussion of various deﬁnitions in the literature and come to the conclusion that while AEs are necessarily misclassiﬁed, counterfactuals need not be (and often should not be).
Furthermore, in agreement with [21], they deﬁne counterfactuals as the closest alternative input (on some suitable metric) that changes the prediction (to a pre-deﬁned target, if applicable).
This seems to be too restrictive in general as there are applications that do not require such a strong focus on proximity.
Furthermore, the deﬁnition then depends too much on the choice of the similarity metric.
Consequently, we will generally call an alternative input x (cid:48) a counterfactual to an input x under model f if it is similar to x on a suitable similarity relation but changes the prediction of the model.
Counterfactual explanations can be useful for debugging: They can be used to answer questions like ‘Why did the self-driving car misidentify the ﬁre hydrant as a stop sign?’
More precisely, they can answer the questions ’What would the image need to look like in order to be classiﬁed as a ﬁre hydrant?’
and ’What changes would need to happen in other ﬁre hydrant (stop sign) images in order to be classiﬁed as a stop sign (ﬁre hydrant)?’.
Depending on the model and the application, we might only be interested in counterfactuals that look realistic.
In particular, we learn more about the decision boundary when we understand the changes made to the image.
If, on the other hand, the classiﬁcation is changed by adding unrealistic noise, then we only learn that the classiﬁer is not robust to this.
In terms of Lipton [18], such counterfactuals are less informative.
Counterfactual explanations are often required to be realistic [26], likely [21], or plausible [11].
Perhaps the most useful way to formulate it is to require counterfactuals to be ‘likely states in the (empirical) distribution of features’ which are ‘close to the data manifold’ [11].
Another often-mentioned desideratum for counterfactual explanations is sparsity: in general, the less features are changed in input space, the better.
Sparse perturbations are usually more interpretable as the change in classiﬁcation can be attributed to a smaller part of the input which is easier to grasp both in terms of the model behaviour and in terms by imperceptible perturbations.
Another, less operational deﬁnition could be that they are not detected to be outside the distribution of realistic datapoints – although the notion of ‘misclassiﬁcation’ could already be interpreted as requiring the datapoint to be in-distribution as it is a correct classiﬁcation of the input itself. If all parts of the input change a little, it might be harder to understand what the perturbation means and how it affects the model.
2.2 Epistemic uncertainty
Previous work has shown that epistemic uncertainty can be used as a proxy for realism [27, 26].
There are two kinds of uncertainty relevant to machine learning models.
Aleatoric uncertainty, on the one hand, is inherent in the data distribution and cannot be reduced.
It is high when there is no clear ground-truth label and maximised in the extreme case of random labels.
Epistemic uncertainty, on the other hand, is due to a lack of knowledge on the part of the model.
It is a quantity that can be reduced for a given input by including it in the training set.
Given this characterisation, epistemic uncertainty estimates are used for active learning (selecting samples that are particularly useful to train on) as well as for detecting out-of-distribution inputs.
It has also been observed that neuron activations in late layers of a neural network, which we call features , can be used to estimate epistemic uncertainty when two properties called sensitivity and smoothness are satisﬁed by the mapping into the feature space [28].
Sensitivity can be seen as a lower Lipschitz bound, ensuring that the features remain sensitive to differences between inputs, thereby preventing ‘feature collapse’:
Otherwise, out-of-distribution (OoD) inputs might not be distinguishable from in-distribution (iD) inputs as they could be mapped to the same area in feature space.
Conversely, smoothness can be seen as an upper Lipschitz bound: Similar inputs are guaranteed not to be too far from each other in feature space, such that distances in this space remain meaningful.
Building on [2], Liu et al.
[19] show that applying spectral normalisation (SN, [20]) with a coefﬁcient c ≤ 1 to ResNets [9] is enough to enforce both sensitivity and smoothness.
Using such models,
Mukhoti et al.
[22] ﬁt a probability distribution to the feature space after the last ResNet block, using the feature representations of the training data.
To estimate the epistemic uncertainty of an input, they then calculate the negative log-likelihood of its feature representation under the learned distribution.
For the probability distribution, they use a Gaussian mixture model (GMM).
We will make use of this approach, called Deep Deterministic Uncertainty (DDU), in the algorithm we propose below.
2.3 Related work
In this section, we provide a (non-exhaustive) survey of algorithms that provide counterfactual explanations.
In the initial paper, Wachter et al.
[31] propose to simply optimise the objective
[Equation]
(1) where f denotes the model, y (cid:48)
the desired model prediction, d a pre-deﬁned distance metric, and λ is a hyper-parameter.
They suggest to iterate through increasing values of λ , always solving for x (cid:48) for ﬁxed λ , until a counterfactual sufﬁciently close to the original input is found.
This is quite a minimal approach, which comes at the cost of x (cid:48) not being constrained to lie on the data manifold (which might be required, depending on the task).
Van Looveren and Klaise [29] build on this (and on [3]), but focus on generating more interpretable counterfactuals by optimising a more complex objective function.
Their prototype-guided approach minimises the loss
[Equation]
Here, L pred =
f ( x (cid:48) )
l − max i (cid:54) =
l f ( x (cid:48) )
i where f ( x (cid:48) )
l is the softmax output of the classiﬁer f on the counterfactual x (cid:48) for the original class l , i.e. its conﬁdence that x (cid:48) belongs to class l .
L 1 and Lrefer to the corresponding distances between x and x (cid:48) in input space.
L AE =
γ (cid:107) x (cid:48) − AE ( x (cid:48) )
(cid:107)where AE is an autoencoder trained on the training data.
Lastly, L proto =
θ (cid:107) ENC ( x (cid:48) )
− proto t (cid:107)where proto t = 1 K (cid:80)
Kk =1 ENC ( x tk ) is the latent prototype deﬁned by the K nearest neighbours x tk of x (cid:48) in target class t and ENC is an encoder.
For untargeted counterfactuals, t is chosen as arg min t (cid:54) =
l (cid:107) ENC ( x (cid:48) )
− proto t (cid:107) 2 .
While L pred enforces a change of classiﬁcation, L AE and L proto are included to encourage realism, measured by a low reconstruction loss under the autoencoder and similarity to similar training samples in the encoded latent space.
The additional use of an autoencoder and the additional loss terms generate a computational overhead that slows down the approach.
Furthermore, the approach comes with many hyperparameters which might require a lot of tuning when applying it to a new task.
The interplay of the different loss terms is not straightforward to analyse either formally or conceptually, so it is hard to predict and improve the performance on new applications.
A very different route to provide interpretable counterfactuals is taken by Schut et al.
[26], based on the notion of uncertainty [6].
They suggest that counterfactuals are realistic if the model classiﬁes them with low epistemic uncertainty and unambiguous when the model has low aleatoric uncertainty.
Consequently, they propose to minimise overall uncertainty in models that provide accurate uncertainty estimates through their softmax output, such as deep ensembles [16].
They show that maximising the ensemble’s target class prediction f
( x (cid:48) ) t is sufﬁcient to minimise its predictive entropy and hence both epistemic and aleatoric uncertainty.
Rather than using an off-the-shelf optimisation algorithm as the two previously mentioned approaches, Schut and colleagues compute the gradient of the classiﬁcation loss in input space and identify the most salient pixel for reducing this loss.
Similar to JSMA [23], they iteratively change the most salient pixel until the target prediction is above 99%, when the uncertainty is sufﬁciently low.
This works well in practice but using an ensemble of models (for their MNIST experiments they use 50 models) is computationally expensive, which might hinder its deployment in practice.
Another approach that has been suggested both for providing counterfactual explanations and for algorithmic recourse is REVISE [10].
This algorithm requires the availability of a generative model, such as the decoder of a VAE trained on the training data.
Similar to [31], the overall idea is to minimise the function
[Equation]
where f is the classiﬁer, t is the target, (cid:96) is some loss function, and G is the generative model.
To ﬁnd a z (cid:48) that minimises the loss, z is initialised to the encoding of the original input x ; then the gradient of the loss in the latent space is computed and the algorithm iteratively takes small steps in latent space until the prediction changes to the target.
Since the resulting counterfactual x (cid:48)
=
G ( z (cid:48) ) is produced by the generative model, it can be dissimilar to x : Although the L 1 norm is known to encourage sparsity, the algorithm cannot be expected to provide sparse solutions, as the changes are not taken in the input space.
Another disadvantage of using a generative model is, of course, the need to train it beforehand which can pose difﬁculties of varying degree depending on the data domain.
Several recent works utilise GANs for generating counterfactuals, such as [13].
Novel algorithm: DeDUCE
For large image datasets, ResNets are often the model of choice.
When ResNets are trained with spectral normalisation, we can use DDU (section 2.2) to estimate their epistemic uncertainty.
DDU achieves stateof-the-art results in OoD detection such as MNIST vs. FashionMNIST.
The authors also demonstrate that it can be used for active learning; this implies that it is particularly useful to train on inputs whose representations have low likelihood under the GMM, i.e. such inputs are substantially different from the previous training data.
This suggests that DDU’s measure of epistemic uncertainty can be a useful target when aiming to generate counterfactuals that are similar to the original training data.
The idea of the novel algorithm presented here is to cross the model’s decision boundary while keeping the epistemic uncertainty as low as possible, using DDU.
Instead of maximising the whole GMM density for minimising epistemic uncertainty (i.e. maximising feature-space density), we propose to only maximise the target class density (ﬁg. 2).
Otherwise, one would also maximise the original label’s class density, which could lead to unstable behaviour.
As we also want to cross the decision boundary quickly, we suggest to change the pixels that are most salient for the gradient of the loss
[Equation]
where f is the model, t the target class, (cid:96) c the cross-entropy, p t the target class density, and f Z the feature extractor, i.e. the part of the model that maps to the feature space after the last ResNet block.
This means we are trying to change the input in a way that quickly changes the classiﬁcation (ﬁrst term) and makes it more similar to the target class in feature space (second term).
The ﬁrst term typically has values in [0 , 100] , while the second term can have values as low as 10 − 6 .
Instead of changing the pixels that minimise the weighted loss for some λ , we select the pixels which make the largest relative difference to either of the two loss terms.
Therefore, we use the alternative gradient
[Equation]
instead of the gradient of the weighted loss given in equation (4).
Note that the cross-entropy loss (cid:96) c ( f ( x (cid:48) ) , t ) is non-negative whereas log p t ( f Z ( x (cid:48) )) can be negative or positive, depending on whether the density is above or below one.
Despite their similarity, including both terms in the objective indeed improves the quality of generated counterfactuals; we also ﬁnd that using the alternative gradient without further weighting ( µ = 1 ) works better than the gradient of the loss for any value of λ (see appendix A).
The novel approach that we call DeDUCE
(Deep Deterministic Uncertainty-based Counterfactual
Explanations) is described in algorithm 1:
In order to only make small and sparse changes to the original input, DeDUCE iteratively perturbs only few pixels at a time.
At each iteration, it determines the most salient pixels for maximising the objective, by computing the gradient in input space, and then perturbs them by a ﬁxed step size δ .
This is similar to how the Jacobian-based
Saliency Map Attack (JSMA) [23] generates adversarial examples and the approach of [26] generates counterfactuals.
Thereby, DeDUCE iteratively perturbs the input in small steps in a way that makes it more and more similar to the target class until it crosses the decision boundary.
The algorithm stops when the softmax output for the target class is above 50% as this corresponds to the model choosing ‘in target class’ over ‘not in target class’.
Following [26], DeDUCE limits the number of times each pixel can be updated 5 and clips x (cid:48) to the input domain bounds.
Similar to some work on adversarial examples [4], we also add momentum to the gradient, replacing the expression for g by
[Equation]
with x k referring to the state of the input after k > 0 iterations.
For the experiments reported below, we change one pixel at a time and use a momentum of 0.6.
Adding momentum of this size often does not make a difference; in our experiments, it only affected around 1.3% of the generated counterfactuals.
Algorithm 1 DeDUCE
Inputs: original input x ∈ X , target class t , trained model f with feature extractor f Z , training data
X tr ∈ X N , coefﬁcient λ , step size δ , max iterations max _ iter , max pixel changes p , number of pixels m , target conﬁdence γ .
Output
while f ( x (cid:48) ) t ≤ γ and k < max _ iter do 6: compute gradient g in input space 7: select m most salient pixels: I = select _ q _ largest _ masked ( | g | , P < p ) 8: update these pixels: ∀ i ∈ I : x (cid:48) [ i ] ← x (cid:48)
[ i ] + sign ( g [ i ]) · δ 9: clip to input domain: x (cid:48) ← clip ( x (cid:48) ) 10: ∀ i ∈ I : P [ i ] ← P [ i ] +11: k ← k +12: return x (cid:48)
Experiments 4.1 Dataset and metrics
We perform experiments on the MNIST dataset [17], so far the only widely used image dataset in the literature on counterfactual explanations.
We use the L 1 and L 0 distances in input space to assess similarity and sparsity, respectively.
We also want to measure how realistic the generated counterfactuals are, as this is usually helpful (cf. section 2.1).
There is no consensus in the ﬁeld on how to measure realism (or ’plausibility’ [12]), and for images, this proves to be quite difﬁcult.
Therefore, we turn to the literature on anomaly detection and use the approach that performed best on an anomaly detection task for MNIST in a recent study [24].
This approach, called AnoGAN [25], uses a pre-trained generative adversarial network (GAN, [7]) to compare the similarity of a given input x with the closest image G ( z ) that the GAN can generate.
AnoGAN uses gradient descent in latent space to minimise the loss
[Equation]
where G is the generator and f D is a mapping to a later layer of the discriminator.
The ﬁrst term gives the L 1 distance between the generated sample and the input whereas the second term measures how similar their feature representations are in the discriminator model.
We use a Wasserstein GAN [1] trained on MNIST.
To reduce the dependence on the initial z , we perform gradient descent three times from different, randomly sampled starting points.
We use λ = 0 .
1 , an initial learning rate of 0 . 03 , and bound the number of iterations to 4000.
We tested how well the resulting metric allows to distinguish actual MNIST images from EMNIST character as well as Fashion-MNIST images.
Our
AnoGAN method achieves an AUROC of 0.913 and 0.998, respectively.
Note that these comparisons are quite different to the generated counterfactuals and thus only provide general sanity checks rather than actual performance tests.
4.2 Baselines
To assess the performance of the novel algorithm, we compare it with three baselines applied to
ResNets without spectral normalisation.
The prototype-guided approach that we shall call ‘VLK’ [29] as well as REVISE [10] were discussed in section 2.3.
Their selection is largely based on a general scarcity of algorithms that provide counterfactual explanations, were demonstrated on image data, and are applicable to ResNets.
Although also demonstrated on MNIST, the mentioned approach of [26] is not included since it cannot be applied to single ResNets.
In order to get the required calibrated uncertainty outputs, one would need an ensemble of around ten models [16], which would make the results much less comparable.
In addition to VLK and REVISE, we include JSMA [23] as a third baseline.
It should be noted that JSMA was introduced to generate (perceptible) adversarial examples rather than counterfactual explanations, so the comparison with regard to realism is not a fair one.
However, since DeDUCE is loosely based on JSMA, the comparison is interesting as it shows how the modiﬁcations affect the results.
While the original paper recommends changing two pixels at a time, we change one as this makes it perform better in our setting and more comparable to the used
DeDUCE algorithm.
To generate counterfactuals with VLK , we use the authors’ implementation in the alibi package [14] in order to be as faithful as possible.
As the algorithm is already tuned to MNIST, we only make one change to the default setting, namely setting k for k -nearest (encoded) neighbours in the L proto term to 5.
This is recommended in the paper and generally improves the quality of the generated counterfactuals.
Note that VLK uses an optimisation algorithm that includes the model’s target conﬁdence in the L pred loss term.
This means that, contrary to the other three algorithms, we cannot prescribe the generated counterfactuals to have a target conﬁdence just above 50%.
In fact, they have a mean conﬁdence of 0.42 and standard deviation of 0.44, with many values being close to 0 or 1.
The third baseline REVISE requires more tuning, as it has not been demonstrated on MNIST before.
REVISE is designed to be applicable to image data, with the authors providing a demonstration on the
CelebA dataset.
Sample reconstructions of the used VAE are shown in appendix B. Note that a more powerful generative model than the one employed here could improve the quality of the generated counterfactuals; this might, however, come with even higher engineering efforts and computational costs.
Appendix B provides more details on the implementation of REVISE.
4.3 Results
With the four algorithms tuned to the dataset and base model [Table] we can look at their performance.
We use ﬁve sets of 100 original MNIST images from the test set and try to ﬁnd counterfactuals for all 9 potential target classes, resulting in 5 × 900 runs overall for each algorithm.
The quantitative metrics are the L 0 and L 1 distance to the original, and the rate at which the algorithms fails to output a candidate counterfactual.
For the evaluation of how realistic/anomalous the images are, we use the anomaly detection metric ‘AGAN’ using AnoGAN.
The results are reported in table 1.
In order to ensure a fair comparison, only image-target pairs are included for which all algorithms found a counterfactual.
[Table]
REVISE achieves the best results on the AnoGAN metric, but their scores are not as good as for the original images, which get a score of 15 . 44 ± 2 . 04 .
VLK and DeDUCE do not show signiﬁcant differences on this metric.
We note that the directly reconstructed images under the REVISE VAE (without applying REVISE) are judged to be more realistic than the original images, achieving a score of 14 .
64 ± 0 .
This shows that the metric has a bias towards VAE-generated images, which beneﬁts REVISE and perhaps also VLK, as the latter minimises an autoencoder loss.
DeDUCE achieves better results than JSMA on all four metrics, with these differences are all being highly signiﬁcant ( p < 10 − 7 on paired t-tests).
Both VLK and REVISE perform much worse than the other two on the sparsity as well as the similarity metric.
This is also expected since both DeDUCE and
JSMA take pixel-wise steps in the input space.
The standard deviations on all metrics are generally fairly low, especially for DeDUCE and JSMA.
There are great differences with respect to the required computation times, not only because the provided implementation of VLK does not support the generation of counterfactuals in batches.
Even when all approaches are used to generate counterfactuals individually, VLK is the slowest approach (table 2).
JSMA is slightly faster than DeDUCE, which is still signiﬁcantly faster than REVISE.
REVISE has a large standard deviation because some runs take particularly long.
Note that REVISE could work with a slightly larger step size and with a lower number of iterations, at the expense of quality.
All computations were performed on NVIDIA Tesla T4 GPUs.
[Table]
As pointed out above, the GAN-based realism metric is only partly reliable.
This can already be seen from the observation that VAE reconstructions are generally judged to be more realistic than the original image.
Therefore, a qualitative examination of the generated counterfactuals is also warranted, although a conclusive verdict would require a comprehensive human evaluation study.
Individual examples generated by the different algorithms for randomly drawn images and targets are presented in ﬁgure 3.
6 Note that REVISE failed to output a counterfactual in the fourth column.
[Figure] [Table] Column 2 and 4 seem generally difﬁcult, while all approaches arguably ﬁnd good counterfactuals for column 1 and 5, with the exception of JSMA on the former.
Column 3 and 6 are met with varying success.
Overall, DeDUCE and VLK might be seen to provide the best counterfactuals.
The individual metric scores are presented in table 3.
Despite the lack of a user study, it seems clear that the NLL and AGAN scores do not always reﬂect human judgement.
Because of this and the much higher L 0 and L 1 scores, the low realism scores of counterfactuals generated by VLK and REVISE clearly do not imply that they are more interpretable than the ones generated by DeDUCE.
Discussion
As demonstrated in the experiments, DeDUCE is an efﬁcient algorithm performing small and sparse perturbations that often result in realistic counterfactuals.
In particular, it provides counterfactuals that are much more similar to the original image than the other considered approaches.
This allows to give more precise explanations for the model’s decision making.
As discussed, DeDUCE is only applicable to classiﬁers that satisfy sensitivity and smoothness assumptions.
It is sufﬁcient to have a
ResNets with some loose spectral normalisation, which might be desired anyway.
Still, this clearly restricts the applicability of DeDUCE.
Overall, DeDUCE could prove to be a viable technique for a considerable number of use cases, namely image classiﬁcation tasks that require the deployment of large neural networks.
One limitation of this work is the lack of human evaluation studies.
In order to conclusively assess how interpretable the generated counterfactuals are and how helpful the explanations are for debugging, such studies will eventually be necessary.
We also plan to include more complex datasets in the future.
While MNIST allows a ﬁrst proof of concept, the algorithm is designed to be scalable to larger datasets and should also be assessed there.
We plan to address this is future work.
Lastly, future work could also try to improve the algorithm itself.
In particular, it might be possible to use a different latent density model instead of the one used here.
For example, the GMM could be ﬁtted to ambiguous data (using multiple labels and a probabilistic ﬁt) to improve density estimation on such inputs.
This could make it necessary to also train the classiﬁcation model on ambiguous data.
Another option could be to use conﬁdence-weighting of the datapoints for ﬁtting the class-wise
Gaussians.
We also hope to investigate these ideas systematically in future work.
Acknowledgements
We would like to thank Andreas Kirsch for helpful discussions.
B.H. was supported through a DAAD scholarship.
L.S. and J.M.B. were supported by DeepMind and Cancer Research UK, respectively.
Both L.S. and J.M.B. were also supported by the EPSRC Centre for Doctoral Training in Autonomous
Intelligent Machines and Systems (EP/S024050/1). A Effect of different gradient expressions
[Table]
B REVISE implementation
Recall that REVISE takes small steps in the latent space of the VAE, guided by the gradient of the loss function (cid:96) ( f ( G ( z (cid:48) )) , t ) + λ · (cid:107) G
( z (cid:48) )
− x (cid:107) 1 where f is the classiﬁer, t is the target, G is the generative model, and x is the original image.
As in the original paper, we use the cross-entropy loss function for (cid:96) .
In addition to the VAE, it is therefore necessary to tune λ as well as the gradient step size that we denote by δ .
I perform a grid search on λ ∈ { 0 . 1 , 1 , 10 } and δ ∈ { 10 − 3 , 10 − 4 , 10 − 5 } , considering both qualitative and quantitative performance.
For the three setting of λ , we limit the number of iterations to 50,000, 10,000, and 5,000, respectively.
For λ = 10 , the algorithm hardly ever terminates: in all settings for δ , it has a failure rate of over 75% .
Comparing all settings on the same image-target pairs would then mean to leave out the vast majority of counterfactuals, so I report quantitative evaluations only for the other six settings in table 5.
Most notably, for λ = 0 . 1 , the Lvalues are very high; a look at the generated images conﬁrms that these are too far from the original inputs to be useful.
Given λ = 1 , the setting with δ = 10 − 5 performs clearly the best overall, so we adopt this for the evaluation on the testset.
[Table]
C Additional DeDUCE outputs