E XPLANATION BY P ROGRESSIVE E XAGGERATION
A BSTRACT
As machine learning methods see greater adoption and implementation in high stakes applications such as medical image diagnosis, the need for model interpretability and explanation has become more critical.
Classical approaches that assess feature importance ( e.g. , saliency maps) do not explain how and why a particular region of an image is relevant to the prediction.
We propose a method that explains the outcome of a classiﬁcation black-box by gradually exaggerating the semantic effect of a given class.
Given a query input to a classiﬁer, our method produces a progressive set of plausible variations of that query, which gradually changes the posterior probability from its original class to its negation.
These counter-factually generated samples preserve features unrelated to the classiﬁcation decision, such that a user can employ our method as a “tuning knob” to traverse a data manifold while crossing the decision boundary.
Our method is model agnostic and only requires the output value and gradient of the predictor with respect to its input.
I NTRODUCTION
With the explosive adoption of deep learning for real-world applications, explanation and model interpretability have received substantial attention from the research community (Kim, 2015; DoshiVelez & Kim, 2017; Molnar, 2019;
Guidotti et al., 2019).
Explaining an outcome of a model in high stake applications, such as medical diagnosis from radiology images, is of paramount importance to detect hidden biases in data (Cramer et al., 2018), evaluate the fairness of the model (Doshi-Velez & Kim, 2017), and build trust in the system (Glass et al., 2008).
For example, consider evaluating a computer-aided diagnosis of Alzheimer’s disease from medical images.
The physician should be able to assess whether or not the model pays attention to age-related or disease-related variations in an image in order to trust the system.
Given a query, our model provides an explanation that gradually exaggerates the semantic effect of one class, which is equivalent to traversing the decision boundary from side to another.
Although not always clear, there are subtle differences between interpretability and explanation (Turner, 2016).
While the former mainly focuses on building or approximating models that are locally or globally interpretable (Ribeiro et al., 2016), the latter aims at explaining a predictor aposteriori.
The explanation approach does not compromise the prediction performance.
However, a rigorous deﬁnition for what is a good explanation is elusive.
Some researchers focused on providing feature importance ( e.g. , in the form of a heatmap (Selvaraju et al., 2017)) that inﬂuence the outcome of the predictor.
In some applications ( e.g. , diagnosis with medical images) the causal changes are spread out across a large number of features ( i.e. , large portions of the image are impacted by a disease).
Therefore, a heatmap may not be informative or useful, as almost all image features are highlighted.
Furthermore, those methods do not explain why a predictor returns an outcome.
Others have introduced local occlusion or perturbations to the input (Zhou et al., 2014; Fong & Vedaldi, 2017) by assessing which manipulations have the largest impact on the predictors.
There is also recent interest in generating counterfactual inputs that would change the black box classiﬁcation decision with respect to the query inputs (Goyal et al., 2019; Liu et al., 2019).
Local perturbations of a query are not guaranteed to generate realistic or plausible inputs, which diminishes the usefulness of the explanation, especially for end users ( e.g. , physicians).
We argue that the explanation should depend not only on the predictor function but also on the data.
Therefore, it is reasonable to train a model that learns from data as well as the black-box classiﬁer ( e.g. , (Chang et al., 2019; Dabkowski & Gal, 2017; Fong & Vedaldi, 2017)).
Our proposed method falls into the local explanation paradigm.
Our approach is model agnostic and only requires access to the predictor values and its gradient with respect to the input.
Given a query input to a black-box, we aim at explaining the outcome by providing plausible and progressive variations to the query that can result in a change to the output.
The plausibility property ensures that perturbation is natural-looking.
A user can employ our method as a “tuning knob” to progressively transform inputs, traverse the decision boundary from one side to the other, and gain understanding about how the predictor makes a decision.
We introduce three principles for an explanation function that can be used beyond our application of interest.
We evaluate our method on a set of benchmarks as well as real medical imaging data.
Our experiments show that the counterfactually generated samples are realistic-looking and in the real medical application, satisfy the external evaluation.
We also show that the method can be used to detect bias in training of the predictor.
M ETHOD
Consider a black box classiﬁer that maps an input space X ( e.g. , images) to an output space Y ( e.g. , labels).
In this paper, we consider binary classiﬁcation problems where Y = {− 1 , +1 } .
To model the black-box, we use f ( x ) =
P ( y | x ) to denote the posterior probability of the classiﬁcation.
We assume that f is a differentiable function and we have access to its value as well as its gradient with respect to the input ∇ x f ( x )
[Equation]
[Figure]
[Equation]
[Table]
[Equation]
[Table]
<
[Table]
ˆ x <latexit sha1_base64="2aKXJmi+n/KddbUoGh83z2f6zsY=">AAADAHicbVLLbtQwFHXCq4RHpyCxYWMxipQRZZQUJFiWlwqiQkVi2kqTUeQ4Tseq85B9g2bkZsOvsGEBQmz5DHb8Dc5M2plpeyVLx+ece32v7bgUXIHv/
[Table]
+xdtO5dfvO3fXOxr19VVSSsgEtRCEPY6KY4DkbAAfBDkvJSBYLdhAfv270gy9MKl7kn2FaslFGjnKeckrAUNGG9cANgU1Af9itvdQLMwLjONWTOtJhwgSQuodPTvCy0sOP8VzrOe6Cdty3yyazddwdL6RJAZs0Sr3JZpvUW/CXlInm3Gq16Mx4mvnkjHmzIP3TSpQI/
[Table]
+WZz052XH+vaCccE9MJdR52u3/
[Table]
+CoAVd1MZe1PkbJgWtMpYDFUSpYeCXMNJEAqeCmQMqxUpCj8kRGxqYk4ypkZ49YI1dwyQ4LaRZOeAZu5yhSabUNIuNs2lRndca8jJtWEH6YqR5XlbAcjo/KK0EhgI3vwEnXDIKYmoAoZKbXjEdE0komD/jmEsIzo98Eexv9YOn/a1Pz7rbr9rrWEMP0SPkoQA9R9voHdpDA0St2vpm/bB+2l/ [Table] [Table] +/
[Table]
[Table] =</latexit> E <latexit sha1_base64="OXViHdoc30LV6voycsN98t9f9S [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
<
[Equation]
>
G  f
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
</latexit>
[Table]
[Equation]
<latexit sha1_base64="70D0v550HvxBWi/8Z6lQlbjZpB8=">AAADXHicbZLfa9swEMcVp9tSd13TDfayF7FgSFgbnK7QPXa/yMbK6GBpC3FmZFluROUfWOeRoPif3Ftf9q9scn7ZaXsgOH2+d6e7Q14iuATbvq0Z9a1Hj580ts2dp7vP9pr7zy9knKWUDWgs4vTKI5IJHrEBcBDsKkkZCT3BLr2bj4V
[Table]
[Table]
[Table]
[Table]
/ZV9k8Hbr7kW706wVKvwgkjGfarWoaYDpIL852I4y3tvCVbjLEUthwXdOaxIIWRh9+q3O0dOiacFKrRM8y1YyVng3f8KNVt23y4PvG04tdEC9TlHzb+uH5MspJEkHAsxcuxEjhVOJSOc5pabCZpgcoOv6UibEQ6pGKty73LY1sSHQZzqJ5KwpOsZCodCzEJPRxZaxbavgLt8o0wGx2PFoiSTNCLVh4KMQxnDYomhz1JKJJ9pA5OUaa2QTHCKidSrbukhONst3zYuDvvO6/7ht6PW6bt6HHvgBXgJOsABb8Ap+ATOwRAQ48j4YRDDN09MZAbmpAo1GnXOc7BxzF//AX7VK0g=</latexit> (b)
We view the (visual) explanation of the black-box as a generative process that produces an input for the black-box that slightly perturbs current prediction ( f ( x ) + δ ) while remaining plausible and realistic.
By repeating this process towards each end of the binary classiﬁcation spectrum, we can traverse the prediction space from one end to the other and exaggerate the underlying effect.
We conceptualize the traversal from one side of the decision boundary to the other as walking across a data manifold, M x .
We assume the walk has a ﬁxed step size and each step of the walk makes δ change to the posterior probability of the the classiﬁer, f .
Since the output of f is bounded between [0 , 1] , we can take at-most (cid:98) 1 δ (cid:99) steps.
Each positive (negative) step increases (decreases) the posterior probability of the previous step.
We assume that there is a low-dimensional embedding space ( M z ) that encodes the walk.
An encoder, E : M x → M z , maps an input, x , from the data manifold,
M x , to the embedding space.
A generator, G : M z → M y , takes both the embedding coordinate and the number of steps and maps it back to the data manifold (see Figure1).
We use I f ( · , · ) to denote the explainer function.
Formally, I f ( x , δ ) :
( X , R ) → X is a function that takes two arguments: a query image x and the desired perturbation δ .
This function generates a perturbed image which is then passed through function f .
The difference between the outputs of f given the original image and the perturbed image should be the desired change i.e. , f ( x δ ) − f ( x ) = δ .
We use x δ to denote I f ( x , δ ) .
This formulation enables us to use δ as a knob to exaggerate the visual explanations of the query sample while it is crossing the decision boundary given by function f .
Our proposed interpretability function I f should satisfy the following properties:
Data Consistency: perturbed samples generated by I f should lie on the data manifold, M x , to be consistent with real data.
In other words, the generated samples should look realistic when compared to other samples.
Compatibility with f : changing the second argument in I f ( x , · ) should produce the desired outcome from classiﬁer f , i.e. , f ( I f ( x , δ ))
≈ f
( x ) + δ . 3.
Self Consistency: Applying reverse perturbation should bring x back to its original form i.e. , I f
( I f ( x , δ ) , − δ )
=
x .
Also, applying setting δ to zero should return the query, i.e. , I f ( x , 0) = x .
Each criterion is enforced via a loss function which are discussed in the following sections.
2. D ATA C ONSISTENCY
We adopt the Generative Adversarial Networks (GANs) framework for our model (Goodfellow et al., 2014).
The GANs implicitly model the underlying data distribution by setting up a min-max game between generative ( G ) and discriminative ( D ) networks:
[Equation]
where z and P z are the noise distribution and the corresponding canonical distribution.
There has been signiﬁcant progress toward improving GANs stability as well as sample quality
(Brock et al., 2019; Karras et al., 2019).
The advantage of GANs is that they produce realistic-looking samples without an explicit likelihood assumption about the underlying probability distribution.
This property is appealing for our application.
Furthermore, we need to provide the desired amount of perturbation to the black-box, f .
Hence, we use a Conditional GAN (cGAN) that allows the incorporation of a context as a condition to the
GAN (Mirza & Osindero, 2014; Miyato & Koyama, 2018).
To deﬁne the condition, we ﬁx the step size, δ , and descritize the walk which effectively cuts the posterior probability range of the predictor ( i.e. , [0 , 1] ) into (cid:98) 1 δ (cid:99) equally-sized bins.
Hence, one can view the perturbation from f ( x ) to f ( x )+ δ as changing the bin index from the current value
c f ( x , 0) to c f ( x , δ ) where c f ( x , δ ) returns the bin index of f ( x ) + δ .
We use c f ( x , δ ) as a condition to the cGAN.
The cGAN optimizes the following loss function:
[Equation]
where c denotes a condition.
Instead of generating random samples from P z , we use the output of an encoder, E ( x ) , as input to the generator.
Finally, the explainer function is deﬁned as:
[Equation]
Our architecture is based on Projection GAN (Miyato & Koyama, 2018), a modiﬁcation of cGAN.
An advantage of the Projection GAN is that it scales well with the number of classes allowing δ → 0 .
The Projection GAN imposes the following structure on the discriminator loss function:
[Equation]
where L cGAN ( D, ˆ G ) indicates the loss function in Eq.
1 when ˆ G is ﬁxed, φ ( · ) and ψ ( · ) are networks producing vector (feature) and scalar outputs respectively.
The r ( c | x ) is a conditional ratio function which will be discussed in Section 2.2.
2. C OMPATIBILITY WITH THE BLACK BOX
In our model, the condition c is an ordered variable i.e. , c f ( x , δ 1 ) <
c f ( x , δ 2 ) when δ 1 < δ 2 .
Therefore, we adapt the ﬁrst term in Eq.
3 to account for ordinal multi-class regression by transforming c f ( x , δ ) into (cid:98) 1 δ (cid:99) − 1 binary classiﬁcation terms (Frank & Hall, 2001):
[Equation]
where φ ( · ) is the feature network in Eq.
3 and v i ’s are parameters.
We also need to ensure that plugging x δ into f ( · ) yields f ( x ) + δ ( i.e. , compatible with f ).
This condition is enforce by a
KullbackLeibler (KL) divergence loss term.
Adding the KL loss and the conditional ratio function we arrive at the following loss:
[Equation]
While the ﬁrst term is a function of both G and D , the second term inﬂuences only the generator G .
2. S ELF C ONSISTENCY
We use a reconstruction loss term to enforce encoder-decoder consistency and satisfy the identity constraint of x = I f
( x , 0) ,
[Equation]
[Equation]
We also require that the perturbation is reversible ( i.e. , I f
( I f ( x , δ ) , − δ )
= x ).
We use a cycleconsistency (Zhu et al., 2017) loss to reconstruct the input from its corresponding perturbed image,
[Equation]
Note that the conditions for the generators in Eq. 5 and 6 are the same.
However, in the former, we are reconstructing the input x from its latent space, but in the latter, we perturb x δ from the bin index
c f ( x , δ ) back to original bin index c f ( x , 0) .
2. O BJECTIVE F UNCTIONS
We adapted the hinge version of the adversarial loss for L cGAN ( G, D ) .
[Equation]
The overall objective function is
[Equation]
where λ cGAN , λ f , λ rec are the hyper-parameters that balance the importance of the loss terms.
R ELATED W ORK
Our work broadly relates to literature in interpretation methods that are designed to provide a visual explanation of the decisions made by a black-box function f , for a given query sample x .
Perturbation-based methods: These methods provide interpretation by showing what minimal changes are required in x to induce a desirable output of f .
Some methods employed image manipulation via the removal of image patches (Zhou et al., 2014) or the occlusion of image regions (Zhou et al., 2014) to change the classiﬁcation score.
Recently, the use of inﬂuence function, as proposed by (Koh & Liang, 2017) are applied as a form of data perturbation to modify a classiﬁer’s response.
The authors in (Fong & Vedaldi, 2017) proposed the use of optimal perturbation, deﬁned as removing the smallest possible image region in x that results in the maximum drop in classiﬁcation score.
In another approach, (Chang et al., 2019) proposed a generative process to ﬁnd and ﬁll the image regions that correspond to the largest change in the decision output of a classiﬁer.
To switch the decision of a classiﬁer, (Goyal et al., 2019) suggested generating counterfactuals by replacing the regions of x with patches from images with a different class label.
All of the aforementioned works perform pixel- or patch-level manipulation to x , which may not result in natural-looking images.
In contrast, our model enforces that the perturbed data be consistent with the unperturbed data to ensure that the perturbation is plausible.
Furthermore, our method can be applied to general data and is not restricted to the imaging domain.
Saliency map-based methods: Saliency maps explains the decision of f on x by highlighting the relevant regions of x .
Some earlier work in this direction(Simonyan et al., 2013; Springenberg et al., 2015; Bach et al., 2015) focuses on computing the gradient of the target class with respect to x and considers the image regions with large gradients as most informative.
Building on this work, the class activation map (CAM)
(Zhou et al., 2016) and its generalized version Grad-CAM Selvaraju et al.
(2017) and other variants such as LPR (Bach et al., 2015) use a linear or non-linear combination of the activation layers to derive relevance score for every pixel in an image.
These gradient-based methods are not model-agnostic and require access to intermediate layers.
Recently, Adebayo et al.
(2018) have shown that some saliency methods are independent both of the model and of the data generating process.
We used their propose evaluation to validate our interpretation model.
The saliency maps are also prone to adversarial attacks as shown by Ghorbani et al.
and Kindermans et al.
Furthermore, if the causal effect of a class is distributed across an image, which is the case in radiology images, the saliency approaches highlight large sections of the image, which greatly reduce the usefulness of the interpretation.
Generative explanation-based methods: These are interpretation models that uses a generative process to produce visual explanations.
The contrastive explanations method (CEM) (Dhurandhar et al., 2018) generates explanations that show minimum regions in x which must be present/absent for a particular classiﬁcation decision.
In another work, (Liu et al., 2019; Joshi et al., 2019; Samangouei et al., 2018) generates explanations that highlight what features should be changed in x so that the classiﬁer conﬁdence in the prediction is strengthen (prototype) or weakened (counterfactual).
Our approach is aligned with these latter lines of work, although our method and model architecture is different.
Our method allows for the gradual change of the class effect, and our consistency criteria result in high-quality feasible perturbation in x .
We rigorously evaluate our method on real medical imaging applications, in addition to the curated computer vision datasets.
E XPERIMENTS
We set up four experiments to evaluate our method.
First, we assess if our method satisﬁes the three criteria of the explainer function introduced in Section 2.
We report both qualitative and quantitative results.
Second, we apply our method on a medical image diagnosis task.
We use external domain knowledge about the disease to perform a quantitative evaluation of the explanation.
Third, we train two classiﬁers on biased and unbiased data and examine the performance of our method in identifying the bias.
While our method does not produce a saliency map, in our last experiment, we use the two counterfactual samples on the boundary [0 , 1] to generate a saliency map and compare it with the other methods.
In Appendix A, we show further experiments to evaluate our model in human experiments, to demonstrate its compatibility with a multi-label classiﬁer and, an ablation study, to show the relative importance of each of the three criteria of the explainer function.
Our experiments are conducted on the CelebA (Liu et al., 2015) and CheXpert (Irvin et al., 2019) datasets.
CelebA contains 200K celebrity face images, each with forty attribute labels.
We considered binary classiﬁer trained on the “smiling” and “young” attributes.
CheXpert is a medical dataset containing 224K chest x-ray images from 65K patients and has labels for fourteen radio-graphic observations.
We considered Cardiomegaly as the target class for generating explanations.
All images are re-sized to 128 × 128 before processing.
4. E VALUATING THE C RITERIA OF THE E XPLAINER
Figure 2 reports the qualitative results on three datasets.
Given a query image x at inference time, our model generates a series of images x δ as visual explanations, which gradually increase the posterior probability f ( x δ ) (top label).
We show results for three prediction tasks: smiling or not-smiling, young or old, and Cardiomegaly or healthy.
The values on the top of each ﬁgure report the f ( x δ ) ’
s.
For Cardiomegaly, we show the outlines of the heart as well as its normalized size (values inside the parenthesis), which is indicative of the disease.
Data Consistency: The generated explanations are synthesized variations of the query image.
To quantitatively compare their visual quality, we consider Fr´echet Inception Distance (FID) (Heusel et al., 2017).
We compared our results against the counterfactual explanations produced by xGEM (Joshi et al., 2018).
The details of the xGEM model are given in appendix A.2.
We divided the real and fake ( i.e. , generated explanations) images into two groups (on either boundary of f ( x ) ∈ [0 , 1] ) and reported the FID for each group and the overall score.
Our method signiﬁcantly outperforms xGEM, producing crisper and more realistic-looking images.
xGEM is based on variational autoencoder (VAE) which are known to produce blurry images ( see Figure 7).
Compatibility with the black-box f : To quantify whether the generation process is aligned with the desire perturbation δ , we plotted the expected outcome f ( x ) + δ against the actual response of the classiﬁer for the generated explanations, f ( x δ ) .
Figure 3 shows how our model performs when generating a series of explanations starting from a wide range of initial query images.
The performance is almost perfect for Young/Old, but less so for more challenging classiﬁcation problems such as Smiling or Cardiomegaly.
The plot also validates that we are producing perturb images covering the entire classiﬁcation range, [0 , 1] .
Appendix A.3 shows additional result from CelebA dataset.
[Table]
Identity preservation: The generated explanations should differ only in semantic features associated with the target class, while retaining the identity of the query image.
We extracted the latent embedding for real images ( E ( x ) ) and their corresponding explanations ( E ( x δ ) ), for different values of δ .
We calculated latent space closeness as the percentage of the times, x δ is closest to the query image x as compared to other generated explanations
[Equation]
where, m ∈
I f ( X −{ x } , δ )) is the set of explanations generated for all the real images excluding the query image x .
Another, popular approach to quantify identity of two face images, is to perform face veriﬁcation .
We used state-of-the-art face recognition model trained on VGGFace2 dataset (Cao et al., 2018) as feature extractor for both real images and their corresponding fake explanations.
For face veriﬁcation, we calculated the closeness between real and fake image as cosine distance between their feature vectors.
The faces were considered as veriﬁed i.e. , fake explanation have same identity as real image, if the distance is below 0.5.
Table 2 summarizes the results.
Our method achieved high performance on localized attribute “smiling”, which alters a relatively small region of the face image as compare to attribute “age” which affects the entire face.
Medical images like chest x-ray have very ﬁne grain details which are difﬁcult to preserve in the generative process of GAN.
Our explainer function preserves the high level features like shape and size of the lung, but it struggles to retain the low level features like anatomy of the breast and shape of the collar bones.
Also, it should be noted that both the datasets have multiple images for same person, but we ignore this information in our analysis and treat each image as a different identity.
We compared our performance against xGEM (Joshi et al., 2018).
VAE explicitly minimizes for latent space closeness.
The generated explanation by xGEM were blurry version of the query image.
Hence, although they were close to query image in latent space, but they didn’t preserve the identity of the individual as shown in face veriﬁcation task and is evident in Figure 7 in appendix A.2.
In comparison, our model achieved good performance on both the tasks.
4. C OUNTERFACTUAL EVALUATION ON MEDICAL DATA
Cardiomegaly refers to an abnormal enlargement of the heart (Brakohiapa et al., 2017).
To understand the explanations derived for Cardiomegaly target class, we overlaid the heart segmentation
[Table]
over the x-ray image and visualize the gradual change in heart size.
The heart segmentation is shown as outlines in Figure 2, with their corresponding heart size (top values in parentheses).
The heart segmentation is derive by training a UNet (Ronneberger et al., 2015) model on the segmentation in chest radiograph (SCR) dataset (van Ginneken et al., 2006).
We registered x with its associated x δ and applied the resulting transformation to the heart masks of x to derive the heart masks for x δ .
For population-level analysis, we plotted the average heart size of x δ vs the condition used for generation ( f ( x ) + δ ) in Figure 4 (a).
The plot shows a positive correlation between the heart size and the response of the classiﬁer f ( x ) , which agrees with the deﬁnition of Cardiomegaly.
To better understand the results, we divided the population into two groups, the ﬁrst group ( x h ; f ( x h ) < 0 . 1 ) consists of real images of healthy x-rays, and the second group ( x c ; f ( x c ) > 0 . 9 ) contains real images of abnormal x-rays positive for Cardiomegaly.
For x h we generated counterfactual as x cδ such that f ( x cδ ) > 0 .
Similarly, counterfactuals for x c are derived as x hδ such that f ( x hδ ) < 0 .
In Figure 4 (b), we show the distribution of heart size in the four groups.
We reported the dependent t-test statistics for paired samples (cid:0) x h and x cδ (cid:1) , (cid:0) x c and x hδ (cid:1) .
A signiﬁcant p-value (cid:28) 0 .
001 rejected the null hypothesis ( i.e. , that the two groups have similar distributions).
We also reported the independent two-sample t-test statistics for healthy (cid:0)
x h and x hδ , p-value > 0 .
01 (cid:1) and abnormal ( x c and x cδ , p-value < 0 .
01) populations.
Given higher p-values, we cannot reject the null hypothesis of identical average distributions with high conﬁdence.
Our model derived explanations successfully captured the change in heart size while generating counterfactual explanations.
4. S ALIENCY M AP
Saliency maps show the importance of each pixel of an image in the context of classiﬁcation.
Our method is not designed to produce saliency maps as a continuous score for every feature of the input.
We extract an approximate saliency map by quantifying the regions that changed the most when comparing explanations at the opposing ends of the classiﬁcation spectrum.
For each query image, we generated two visual explanations corresponding to the two extremes of the decision boundary (cid:0) f
( x δ ) =
and f ( x δ ) = 1 (cid:1) .
The absolute difference between these explanations is our saliency map.
Figure 5 shows the saliency map obtain from our method and its comparison with popular gradient based methods.
We restricted the saliency maps obtained from different methods to have positive values and normalize them to range [0,1].
Subjective, the saliency maps produced by our method are very localized and are comparable to the other methods.
We adapted the metric introduced in (Samek et al., 2016) to compare the different saliency maps.
In an iterative procedure, we progressively replace a percentage of the most relevant pixels in an image (as given by the saliency map) with random values sampled from a uniform distribution.
We observe the corresponding change in the classiﬁcation performance as shown in Figure 4 (c).
All the methods experienced a drop in the accuracy of the classiﬁer with increase in the fraction of perturb pixels.
The saliency maps produced by our model is signiﬁcantly better than random maps and are comparable to the other saliency map methods.
It should be noted that, there are many ways to quantify important regions in a image, using the series of explanations generated by our method.
We didn’t optimize to ﬁnd the best saliency map and showed results for one such method.
4. B IAS D ETECTION
Our model can discover confounding bias in the data used for training the black-box classiﬁer.
Confounding bias provides an alternative explanation for an association between the data and the target label.
For example, a classiﬁer trained to predict the presence of a disease may make decisions based on hidden attributes like gender, race, or age.
In a simulated experiment, we trained two classiﬁers to identify smiling vs not-smiling images in the CelebA dataset.
The ﬁrst classiﬁer f Biased is trained on a biased dataset, confounded with gender such that all smiling images are of male faces.
We train a second classiﬁer f No-biased on an unbiased dataset, with data uniformly distributed with respect to gender.
Note that we evaluate both the classiﬁers on the same validation set.
Additionally, we assume access to a proxy Oracle classiﬁer f Gender that perfectly classiﬁes the confounding attribute i.e. , gender.
As shown in Cohen et al.
(2018), if the training data for the GAN is biased, then the inference would reﬂect that bias.
In Figure 6, we compare the explanations generated for the two classiﬁers.
The visual explanations for the biased classiﬁer change gender as it increases the amount of smile.
We adapted the confounding metric proposed in Joshi et al.
(2018) to summarize our results in Table 3.
Given the data D = { ( x i , y i , a i ) , x i ∈ X , y i , a i ∈ Y} , we quantify that a classiﬁer is confounded by an attribute a if the generated explanation ˆ x δ has a different attribute a , as compared to query image x , when processed through the Oracle classiﬁer f Gender .
The metric is formally deﬁned as E D [1( g ∗ ( x δ ) (cid:54) = a )] / |D| .
For a biased classiﬁer, the Oracle function predicted the female class for the majority of the images, while the unbiased classiﬁer is consistent with the true distribution of the validation set for gender.
Thus, we the fraction of generated explanations that changed the confounding attribute “gender’ was found to be high for the biased classiﬁer.
[Table]
C ONCLUSION
In this paper we proposed a novel interpretation method that explains the decision of a black-box classiﬁer by producing natural-looking, gradual perturbations of the query image, resulting in an equivalent change in the output of the classiﬁer.
We evaluated our model on two very different datasets, including a medical imaging dataset.
Our model produces high-quality explanations while preserving the identity of the query image.
Our analysis shows that our explanations are consistent with the deﬁnition of the target disease without explicitly using that information.
Our method can also be used to generate a saliency map in a model agnostic setting.
In addition to the interpretability advantages, our proposed method can also identify plausible confounding biases in a classiﬁer.
A A PPENDIX A. I MPLEMENTATION D ETAILS
The architecture for the generator and discriminator is adapted from Miyato & Koyama (2018).
The image encoding learned by encoder E ( x ) is fed into the generator.
The condition c f ( x , δ ) is passed to each resnet block in the generator using conditional batch normalization.
The generator has ﬁve resnet blocks, where each block consists of BN-ReLU-Conv3-BN-ReLU-Conv3.
BN is batch normalization, ReLU is activation function and Conv3 is the convolution ﬁlter.
The encoder function uses the same structure but downsamples the image.
The discriminator function has ﬁve resnet blocks, each of which has the form ReLU-Conv3-ReLU-Conv3.
A. X GEM IMPLEMENTATION
We refers to Joshi et al.
for the implementation of xGEM.
First, VAE is trained to generate face images.
The VAE used is available at:https://github.com/LynnHo
/VAE-Tensorﬂow.
All settings and architectures were set to default values.
The original code generates an image of dimension 64x64.
We extended the given network to produce an image with dimensions 128x128.
The pretrained VAE is then extended to incorporate the cross-entropy loss for ﬂipping the label of the query image.
The model evaluates the cross-entropy loss by passing the generated image through the classiﬁer.
Figure 7 shows the qualitative difference between the explanations generated by our proposed method and xGEM.
A. E XTENDED RESULTS FOR EVALUATING THE CRITERIA OF THE EXPLAINER
Here, we provide results for four more prediction tasks on celebA dataset: no-beard or beard, heavy makeup or light makeup, black hair or not back hair, and bangs or no-bangs.
Figure 8 shows the qualitative results, an extended version of results in Figure 2.
We evaluated the results from these prediction tasks for compatibility with black-box f ( see Figure 9), data consistency and self consistency ( see Table 4).
[Table]
Inception
Distance) score measures the quality of the generated explanations.
Lower FID is better.
LSC (Latent Space Closeness) quantiﬁes the fraction of the population where generated explanation is nearest to the query image than any other generated explanation in embedding space.
FVA (Face veriﬁcation accuracy) measures percentage of the times the query image and generated explanation have same face identity as per model trained on VGGFace2.
Higher LSC and FVA is better.
A. H UMAN E VALUATION
We used Amazon Mechanical Turk (AMT) to conduct human experiments to demonstrate that the progressive exaggeration produced by our model is visually perceivable to humans.
We presented
AMT workers with three tasks.
In the ﬁrst task, we evaluated if humans can detect the relative order between two explanations produced for a given image.
We ask the AMT workers, “Given two images of the same person, in which image is the person younger (or smiling more)?”
( see
Figure 10).
We experimented with 200 query images and generated two pairs of explanations for each query image ( i.e. , 400 hits).
The ﬁrst pair ( easy ) imposed the two images are samples from opposite ends of the explanation spectrum (counterfactuals), while the second pair ( hard ) makes no such assumption.
In the second task, we evaluated if humans can identify the target class for which our model has provided the explanations.
We ask the AMT workers, “What is changing in the images?
(age, smile, hair-style or beard)”.
We experimented with 100 query images from each of the four attributes ( i.e. , 400 hits).
In the third task, we demonstrate that our model can help the user to identify problems like possible bias in the black-box training.
Here, we used the same setting as in the second task but also showed explanations generated for a biased classiﬁer.
We ask the AMT workers, “What is changing in the images?
(smile or smile and gender)”
( see Figure 10).
We generated explanations forquery images each, from a biased-classiﬁer ( f Biased ) explainer from Section 4.4 and an unbiased classiﬁer ( f No-biased ) explainer ( i.e. , 400 hits).
In all the three tasks, we collected eight votes for each task, evaluated against the ground truth, and used the majority vote for calculating accuracy.
We summarize our results in Table 5.
In the ﬁrst task, the annotators achieved high accuracy for the easy pair when there was a signiﬁcant difference among the two explanation images, as compared to the hard pair when the two explanations can have very subtle differences.
Overall, the annotators were successful in identifying the relative order between the two explanation images.
In the second task, the annotators were generally successful in correctly identifying the target class.
The target class “bangs” proved to be the most difﬁcult to identify, which was expected.
The generated images for “bangs” were qualitatively, the most subtle.
For the third task, the correct answer was always the target class i.e. , “smile”.
In the case of biased classiﬁer explainer, the annotators selected “Smile and Gender” 12.5% of the times.
The gradual progression made by the explainer for a biased classiﬁer was very subtle and was changing large regions of the face as compared to the unbiased explainer.
The difference is much more visible when we compare the explanation generated for the same query image for a biased and no-biased classiﬁer, as in Figure 6.
But in a realistic scenario, the no-biased classiﬁer would not be available to compare against.
Nevertheless, the annotators detected bias at roughly the same level of accuracy as our classiﬁer (Table 3).
Future work could improve upon bias detection.
[Table]
Task-1 (
[Table]
[Table]
Task-2 (Identify
Target Class)
[Table]
A. E VALUATING C LASS D ISCRIMINATION
In multi-label settings, multiple labels can be true for a given image.
In this test, we evaluated the sensitivity of our generated explanations to the class being explained.
We consider a classiﬁer trained to identify multiple attributes: young, smiling, black-hair, no-beard and bangs in face images from
CelebA dataset.
We used our model to generate explanations while considering one of the attributes as the target.
Ideally, an explanation model trained to explain a target attribute should produce explanations consistent with the query image on all the attributes beside the target.
Figure 11 plots the fraction of the generated explanations, that have ﬂipped in source attribute as compared to the query image.
Each column represents one source attribute.
Each row is one run of our method to explain a given target attribute.
A. A BLATION S TUDY
Our proposed model has three types of loss functions: adversarial loss from cGAN, KL loss, and reconstruction loss.
The three losses enforce the three properties of our proposed explainer function: data consistency, compatibility with f , and self-consistency, respectively.
In the ablation study, we quantify the importance of each of these components by training different models, which differ in one hyper-parameter while rest are equivalent ( λ cGAN = 1 , λ f = 1 and λ rec = 100 ).
For data consistency , we evaluate Fr´echet Inception Distance (FID).
FID score measures the visual quality of the generated explanations by comparing them with the real images.
We show results for two groups.
In the ﬁrst group, we consider real and fake images where the classiﬁer has high conﬁdence in presence of the target label i.e. , f ( x δ ) , f ( x ) ∈ [0 . 9 , 1 . 0] .
In second group, the target label is absent i.e. , f ( x δ ) , f ( x ) ∈ [0 . 0 , 0 . 1) .
We also report an overall score by considering all the real and generated explanations together.
For compatability with f we plotted the desired output of the classiﬁer i.e. , f ( x ) + δ against the actual output of the classiﬁer f ( x δ ) for the generated explanations.
For self consistency , we calculated the Latent Space Closeness (LSC) measure and Face veriﬁcation accuracy (FVA).
LSC quantiﬁes the fraction of the population in which the generated explanation is nearest to the query image than any other generated explanation in embedding space.
FVA measures the percentage of the instances in which the query image and generated explanation have the same face identity as per the model trained on VGGFace2.
For the ablation study, we consider the prediction task of young vs old on the CelebA dataset.
Figure 12 shows the results for compatibility with f .
Table 6 summarizes the results for data consistency and self-consistency.
[Table]
[Figure]