Original: What You See is What You Classify: Black Box Attributions
Revised: What You See is What You Classify: Black Box Attributions

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: An important step towards explaining deep image classiﬁers lies in the identiﬁcationof image regions that contribute to individual class scores in the model’s output.
Revised: An important step towards explaining deep image classiﬁers lies in the identiﬁcation of image regions that contribute to individual class scores in the model’s output.

Original: However, doing this accurately is a difﬁcult task due to the black-box nature of suchnetworks.
Revised: However, doing this accurately is a difﬁcult task due to the black-box nature of such networks.

Original: Most existing approaches ﬁnd such attributions either using activationsand gradients or by repeatedly perturbing the input.
Revised: Most existing approaches ﬁnd such attributions either using activations and gradients or by repeatedly perturbing the input.

Original: We instead address thischallenge by training a second deep network, the Explainer , to predict attributionsfor a pre-trained black-box classiﬁer, the Explanandum .
Revised: We instead address this challenge by training a second deep network, the Explainer , to predict attributions for a pre-trained black-box classiﬁer, the Explanandum .

Original: These attributions are inthe form of masks that only show the classiﬁer-relevant parts of an image, maskingout the rest.
Revised: These attributions are provided in the form of masks that only show the classiﬁer-relevant parts of an image, masking out the rest.

Original: Our approach produces sharper and more boundary-precise maskswhen compared to the saliency maps generated by other methods.
Revised: Our approach produces sharper and more boundaryprecise masks when compared to the saliency maps generated by other methods.

Original: Moreover, unlikemost existing approaches, ours is capable of directly generating very distinct classspeciﬁc masks.
Revised: Moreover, unlike most existing approaches, ours is capable of directly generating very distinct class-speciﬁc masks in a single forward pass.

Original: Finally, the proposed methodis very efﬁcient for inference since itonly takes a single forward pass through the Explainer to generate all class-speciﬁcmasks.
Revised: This makes the proposed method very efﬁcient during inference.

Original: We show that our attributions are superior to established methods bothvisually and quantitatively, by evaluating them on the PASCALVOC-2007 and
Revised: We show that our attributions are superior to established methods both visually, and quantitatively with respect to the PASCAL

Original:  Microsoft COCO-2014 datasets.
Revised: VOC-2007 and Microsoft COCO-2014 datasets.

Original:  Introduction
Revised:  Introduction

Original: Image recognition and classiﬁcation systems based on deep learning are considered black boxes.
Revised: Image recognition and classiﬁcation systems based on deep learning are considered black boxes.

Original: Itis often hard, if not impossible, to directly locate the portions of the input image responsible for agiven classiﬁcation output.
Revised: It is often hard, if not impossible, to directly locate the portions of the input image responsible for a given classiﬁcation output.

Original: Being able to systematically perform such attributions is a crucial steptowards explainable deep learning, which can contribute to its widespread adoption.
Revised: Being able to systematically perform such attributions is a crucial step towards explainable deep learning, which can contribute to its widespread adoption.

Original: Attributionallows understanding the potential errors and biases [13, 17, 21].
Revised: Attribution allows understanding the potential errors and biases [13, 17, 21].

Original: It also enables the use of deepnetworks as decision support tools, e.g . in medicine [15] and facilitating meeting legal regulations [8].
Revised: It also enables the use of deep networks as decision support tools, e g . in medicine [15] and facilitating meeting legal regulations [8].

Original: Note that we refer to attribution as the goal of associating input image regions to model decisions,which differs from the notion of interpretability that relates to models that can be directly used toinfer causal or general structural relationships that link inputs and outputs [23].
Revised: Note that we refer to attribution as the goal of associating input image regions to model decisions, which differs from the notion of interpretability that relates to models that can be directly used to infer causal or general structural relationships that link inputs and outputs [23].

Original: The problem of performing attribution for deep image classiﬁers has been addressed by several recentworks.
Revised: The problem of performing attribution for deep image classiﬁers has been addressed by several recent works.

Original: These belong to two broad categories.
Revised: These belong to two broad categories.

Original: The ﬁrst relies on visualising the derived model’sactivations and backpropagated signals [31, 24, 3].
Revised: The ﬁrst relies on visualising the derived model’s activations and backpropagated signals [31, 24, 3].

Original: Although these methods provide a direct wayto visualize salient areas, they heavily depend on the quality of the trained models’ architecturesand weights.
Revised: Although these methods provide a direct way to visualize salient areas, they heavily depend on the quality of the trained models’ architectures and weights.

Original: Furthermore, some of these methods turn out to be independent of the labels themodel has been trained on, as well as to some extent the parameters of the models, as shownthrough randomization tests [1].
Revised: Furthermore, some of these methods turn out to be independent of the labels the model has been trained on, as well as to some extent the parameters of the models, as shown through randomization tests [1].

Original: This suggests that many of these methods are strongly inﬂuencedby low-level, class-agnostic features and are therefore inadequate.
Revised: This suggests that many of these methods are strongly inﬂuenced by low-level, class-agnostic features and are therefore inadequate.

Original: The second category reliesonlocal perturbations of the input images and is often model-agnostic. These models rely onmaskingstrategies to infer local feature importance, which often results in loss of visual detail, ambiguity and
Revised: The second category relies on

Original: Bicycle
Revised: Bicycle

Original: Motorbike
Revised: Motorbike

Original: Person
Revised: Person

Original: Cat
Revised: Cat

Original: Dog
Revised: Dog

Original: Bicycle
Revised: Bicycle

Original: Motorbike
Revised: Motorbike

Original: Person
Revised: Person

Original: Cat
Revised: Cat

Original: Dog
Revised: Dog

Original: [Figure] 
Revised: [Figure] m

Original:  high computational cost.
Revised: E P [Figure] u r s local perturbations of the input images and is often model-agnostic. These models rely on masking strategies to infer local feature importance, which often results in loss of visual detail, ambiguity and high computational cost.

Original: Nonetheless, results suggest that local perturbations, like masking, are aneffective way to analyze and explain the response of a model [20, 22].
Revised: Nonetheless, results suggest that local perturbations, like masking, are an effective way to analyze and explain the response of a model [20, 22].

Original: This last line of reasoning posits that attribution can be seen as a task independent of the Explanandum( i.e . the model to be explained).
Revised: This last line of reasoning posits that attribution can be formulated as an additional task complementing the classiﬁcation from the Explanandum ( i.e . the model to be explained).

Original: Therefore, it is natural to consider learning an Explainer ( i.e . themodel that explains) that speciﬁcally addresses the task of attribution.
Revised: Therefore, it is natural to consider learning an Explainer ( i.e . the model that explains) that speciﬁcally addresses the task of attribution.

Original: We approach attribution as asupervised meta-learning problem through the use of a complementary convolutional neural network,which learns to produce dense class-speciﬁc attribution masks, balancing preservation and deletion ofinformation from the input image as dictated by theExplanandum .
Revised: We approach attribution as a supervised meta-learning problem through the use of a complementary convolutional neural network, which learns to produce dense class-speciﬁc attribution masks, balancing preservation and deletion of information from the input image as dictated by the

Original:  The resulting Explainer efﬁciently masks and precisely locates regions of the input image that areresponsible for the classiﬁcation result of a pre-trained and frozen Explanandum (see Fig.
Revised: Explanandum . The resulting Explainer efﬁciently masks and precisely locates regions of the input image that are responsible for the classiﬁcation result of a pre-trained and frozen Explanandum (see Fig.

Original: 1 and Sup.
Revised: 1 and Sup.

Original: Mat.
Revised: Mat.

Original: Sec. C).
Revised: Sec. C).

Original: We evaluate our methodology on PASCAL Visual Object Classes (VOC-2007) [9]and Microsoft Common Objects in Context (COCO-2014) [18].
Revised: We evaluate our methodology on PASCAL Visual Object Classes (VOC-2007) [9] and Microsoft Common Objects in Context (COCO-2014) [18].

Original: To sum up, we present an efﬁcientattribution method, which has the following advantages:
Revised: To sum up, we present an efﬁcient attribution method, which has the following advantages:

Original: • Our Explainer  provides a precise mask for each class label as in Fig.
Revised: • Our Explainer directly provides a precise mask for each class label as in Fig.

Original: • Our Explainer provides attributions for test images with a single very efﬁcient forward pass,orders of magnitude more efﬁcient as compared to other perturbation-based approaches.• At inference time, our Explainer does not need to access the Explanandum .
Revised: • Our Explainer provides attributions for test images with a single very efﬁcient forward pass, orders of magnitude more efﬁcient as compared to other perturbation-based approaches. • Our Explainer works with any architecture of the Explanandum .

Original:  Related Work
Revised:  Related Work

Original: Activation and gradient-based methods.
Revised: Activation and gradient-based methods.

Original: The methods in this category attempt to explain atrained model by leveraging its weights and its activations for a given image to be explained.
Revised: The methods in this category attempt to explain a trained model by leveraging its weights and its activations for a given image to be explained.

Original: Zhouet al.
Revised: Zhou et al.

Original: [31] initiated this line of research with Class Activation Mapping (CAM), where the authorsbuilt particular architectures around a notion of global average pooling and illustrated the intrinsiclocalization performed by a classiﬁcation network.
Revised: [31] initiated this line of research with Class Activation Mapping (CAM), where the authors built particular architectures around a notion of global average pooling and illustrated the intrinsic localization performed by a classiﬁcation network.

Original: Grad-CAM [24] removed the dependency of CAMon speciﬁc architectures by reconstructing the input heatmap through a weighted average of gradients,rather than directly projecting back activations.
Revised: Grad-CAM [24] removed the dependency of CAM on speciﬁc architectures by reconstructing the input heatmap through a weighted average of gradients, rather than directly projecting back activations.

Original: Further extensions to this are Grad-CAM++ [3],which is CAM weighted by averaged second order gradients, Score-CAM [29], which is CAMweighted by layer-wise class activations, or Ablation-CAM [7], which is CAM weighted by neuronablation.
Revised: Further extensions to this are Grad-CAM++ [3], which is CAM weighted by averaged second order gradients, Score-CAM [29], which is CAM weighted by layer-wise class activations, or Ablation-CAM [7], which is CAM weighted by neuron ablation.

Original: Other extensions aim to cope with the issues arising from the high sensitivity to noise andclipping effects of gradients [1].

Original: Some methods directly engineer the backpropagation and network architectures in order to retrievepixel-level saliency maps.
Revised: Some methods directly engineer the backpropagation and network architectures in order to retrieve pixel-level saliency maps.

Original: One such example is the deconvolution approach of Zeiler and Fergus[30], where the authors propose to revert forward propagation (mapping activations to pixels insteadof mapping pixels to activations) to visualize features and functions of intermediate layers.
Revised: One such example is the deconvolution approach of Zeiler and Fergus [30], where the authors propose to revert forward propagation (mapping activations to pixels instead of mapping pixels to activations) to visualize features and functions of intermediate layers.

Original: Fromthis, Springenberg et al.
Revised: From this, Springenberg et al.

Original: [26] derived guided backpropagation.
Revised: [26] derived guided backpropagation.

Original: Although the focus of their workwas on developing an efﬁcient fully convolutional neural network, they show how by combiningtraditional backpropagation and deconvolutions, one can retrieve salient regions of the input for a speciﬁc class.
Revised: Although the focus of their work was on developing an efﬁcient fully convolutional neural network, they show how by combining traditional backpropagation and deconvolutions, one can retrieve salient regions of the input for a speciﬁc class.

Original: In another line of work, Wagner et al.
Revised: In another line of work, Wagner et al.

Original: [28] frame the problem as a defense againstadversarial perturbations and tackle it by constraining and selectively pruning activations.
Revised: [28] frame the problem as a defense against adversarial perturbations and tackle it by constraining and selectively pruning activations.

Original: Local perturbations and local models.
Revised: Local perturbations and local models.

Original: The idea behind these methods is that by perturbing theinput to the trained network, one can retrieve an explanation signal by summarizing changes in theclassiﬁcation scores.
Revised: The idea behind these methods is that by perturbing the input to the trained network, one can retrieve an explanation signal by summarizing changes in the classiﬁcation scores.

Original: RISE introduced a way of retrieving a saliency heatmap based on randomperturbations of the input image [20] and recently, [27] extended a similar reasoning for objectdetection systems.
Revised: RISE introduced a way of retrieving a saliency heatmap based on random perturbations of the input image [20] and recently, [27] extended a similar reasoning for object detection systems.

Original: The input image is randomly masked several times, changes in the model scorestracked, and aggregated by a weighted average.
Revised: The input image is randomly masked several times, changes in the model scores tracked, and aggregated by a weighted average.

Original: LIME proposes a similar setup, but instead usesrandom switches on unsupervised segmentation [22].
Revised: LIME proposes a similar setup, but instead uses random switches on unsupervised segmentation [22].

Original: Fong and Vedaldi [11] extend these ideas by reformulating the attribution problem as a meta-learningtask, where the optimal mask can be found by optimizing a score.
Revised: Fong and Vedaldi [11] extend these ideas by reformulating the attribution problem as a meta-learning task, where the optimal mask can be found by optimizing a score.

Original: Fong et al.
Revised: Fong et al.

Original: [12] extend their previouswork [11] through the deﬁnition of extremal perturbations , which are the family of transformationsthat maximally affect a classiﬁcation score.
Revised: [12] extend their previous work [11] through the deﬁnition of extremal perturbations , which are the family of transformations that maximally affect a classiﬁcation score.

Original: They propose to use Gaussian blur to assess perturbationsover a ﬁxed set of mask areas.
Revised: They propose to use Gaussian blur to assess perturbations over a ﬁxed set of mask areas.

Original: Dabkowski and Gal [5] rely on a similar intuition, and the Explainer isdirectly optimized based on meta-learning around the Explanandum .
Revised: Dabkowski and Gal [5] rely on a similar intuition, and the Explainer is directly optimized based on meta-learning around the Explanandum .

Original: In these settings, the Explaineroptimizes a loss which balances preservation of the classiﬁcation score on the active parts of theimage, while minimizing the score for areas that are not active.
Revised: In these settings, the Explainer optimizes a loss which balances preservation of the classiﬁcation score on the active parts of the image, while minimizing the score for areas that are not active.

Original: These works also make use ofregularization to constrain the mask area and its smoothness.
Revised: These works also make use of regularization to constrain the mask area and its smoothness.

Original: In a similar fashion, Chang et al.
Revised: In a similar fashion, Chang et al.

Original: [2]learn a pixel-speciﬁc dropout as a Bernoulli distribution, which is used to retrieve a mask and evaluateextremal perturbations.
Revised: [2] learn a pixel-speciﬁc dropout as a Bernoulli distribution, which is used to retrieve a mask and evaluate extremal perturbations.

Original: Unlike Fong et al.
Revised: Unlike Fong et al.

Original: [12], Chang et al.
Revised: [12], Chang et al.

Original: [2] return the parameters of the learnedper-pixel Bernoulli distribution as the saliency mask.
Revised: [2] return the parameters of the learned per-pixel Bernoulli distribution as the saliency mask.

Original: Recently, Khorram et al.
Revised: Recently, Khorram et al.

Original: [16] proposed iGOS++,which frames attribution as optimizing over a mask which optimally balances preservation of usefulinformation and removal of unnecessary information, regularized by bilateral total variation.
Revised: [16] proposed iGOS++, which frames attribution as optimizing over a mask which optimally balances preservation of useful information and removal of unnecessary information, regularized by bilateral total variation.

Original: In this paper, we propose a method which learns a dense perturbation, the attribution mask, over inputimages.
Revised: In this paper, we propose a method which learns a dense perturbation, the attribution mask, over input images.

Original: Unlike existing approaches, we optimize to balance low multi-label binary cross entropyfor the unmasked portion of the input image, and high entropy for areas of the image masked out bythe Explainer .
Revised: Unlike existing approaches, we optimize to balance low multi-label binary cross entropy for the unmasked portion of the input image, and high entropy for areas of the image masked out by the Explainer .

Original: Compared to most perturbation-based methods, our Explainer can predict attributionmasks for all classes (and not only the predicted class) in a single forward pass, thereby being ordersof magnitude faster.
Revised: Compared to most perturbation-based methods, our Explainer can predict attribution masks for all classes (and not only the predicted class) in a single forward pass, thereby being orders of magnitude faster.

Original: In addition, we beat state-of-the-art on standard benchmarks.
Revised: In addition, we beat state-of-the-art on standard benchmarks.

Original:  Training the Explainer
Revised:  Training the Explainer

Original: In this section, we present our Explainer architecture, which is trained to detect image regions that agiven pre-trained classiﬁer, the Explanandum , relies on to infer semantic object classes.
Revised: In this section, we present our Explainer architecture, which is trained to detect image regions that a given pre-trained classiﬁer, the Explanandum , relies on to infer semantic object classes.

Original: Trainingof our model does not require any pixel-wise annotations.
Revised: Training of our model does not require any pixel-wise annotations.

Original: It relies on the Explanandum ’s training data (including labels) and model.
Revised: It relies on the Explanandum ’s training data (including labels) and model.

Original: This makes our technique generalizeable to any Explanandumarchitecture through which we can back-propagate the classiﬁcation loss.
Revised: This makes our technique generalizable to any Explanandum architecture through which we can back-propagate the classiﬁcation loss.

Revised: In this paper though, we only consider Explanandum models which are trained for image classiﬁcation and not for object detection or semantic segmentation.

Original: As illustrated in Fig.
Revised: As illustrated in Fig.

Original: 2, the Explainer sees an image and outputs a set of masks S of the sameresolution, containing one segmentation mask s c for each class c ∈ C .
Revised: 2, the Explainer sees an image and outputs a set of masks S of the same resolution, containing one segmentation mask s c for each class c ∈ C .

Original: The values of the class maskss c are bounded in the range [0 , 1] through a sigmoid function σ ( t ) =
Revised: The values of the class masks s c are bounded in the range [0 , 1] through a sigmoid function σ ( t ) =

Original: e t / (1 + e t ) , where t representsthe predicted pixel-wise logits.
Revised: e t / (1 + e t ) , where t represents the predicted pixel-wise logits.

Original: Values of 1 or 0 in the mask result in the full preservation or deletionof the corresponding pixel value, respectively.
Revised: Values of 1 or 0 in the mask result in the full preservation or deletion of the corresponding pixel value, respectively.

Original: Note that the sum of the class masks does not need tobe equal to one.
Revised: Note that the sum of the class masks does not need to be equal to one.

Original: Using the target label(s) for the input image, we split the class segmentation masks into two sets: onethat consists of masks that correspond to any ground truth class appearing in the training image, whilethe other contains the remaining ones.
Revised: Using the target label(s) for the input image, we split the class segmentation masks into two sets: one that consists of masks that correspond to any ground truth class appearing in the training image, while the other contains the remaining ones.

Original: We merge each set into a single mask by taking the pixel-wisemaximum value for each pixel position.
Revised: We merge each set into a single mask by taking the pixel-wise maximum value for each pixel position.

Original: The target mask m (obtained from the ﬁrst set) serves tolocate the regions corresponding to any of the label(s) contained in a given training image, whilethe non-target mask n (obtained from the second set) allows us to gather false positive activations.
Revised: The target mask m (obtained from the ﬁrst set) serves to locate the regions corresponding to (any of) the label(s) contained in a given training image, while the non-target mask n (obtained from the second set) allows us to gather false positive activations.

Original: Finally, the inverted mask ˜ m
Revised: Finally, the inverted mask ˜ m

Original: = 1 − m is used to ensure that once the object of interest is removed,the Explanandum cannot address the task anymore.
Revised: = 1 − m is used to ensure that once the object of interest is removed, the Explanandum cannot address the task anymore.

Original: Note that during inference, the mask aggregationstep is not needed and our Explainer provides one explanation mask for each class. 
Revised: Note that during inference, the mask aggregation step is not needed and our Explainer provides one explanation mask for each class. [Figure]

Revised: Note that for the selection of true classes, we chose to utilize the same ground truth labels that the

Revised: Explanandum was given during training on unobstructed images.

Revised: We believe that using ground truth label helps in handling attributions for false negative predictions of the Explanandum as we optimize the attributions also for classes which are not predicted with high probability on the unmasked images.

Revised: Alternatively, one could directly take the predictions of the Explanandum for the selection of true classes.

Original: 3.1 Training loss
Revised: This requires a thresholding operation, which is an additional complication. However, relying on the Explanandum ’s predictions might be more suited if we expect the Explanandum to still make a signiﬁcant amount of false positive predictions on the training set (which was not the case for the architectures and datasets we decided to explain). We assume that the Explainer requires an Explanandum that performs very well on the training set, to minimize the effect of misleading training signals.

Revised: Given the results in Sup.

Revised: Mat.

Original:  We formulate the loss as a combination of four terms, which reﬂect the desired conditions:
Revised: Sec. C, we are able to show that under these assumptions, this design choice does not hinder the ability of the Explainer to generate attributions that are faithful to the Explanandum ’s predictions. 3.1 Training loss We formulate the loss as a combination of four terms:

Original: [Equation]
Revised: [Equation]

Original: where L c ( x , Y , m ) is the binary cross-entropy loss over the masked image, L e ( x , ˜ m ) is an entropyterm over the image masked with the complement of m , L a ( m , n , S ) is a regularization termaccounting for the area of the true and false positive masks, and L tv ( m , n ) is a regularization termfavouring smooth masks.
Revised: where L c ( x , Y , m ) is the binary cross-entropy loss over the masked image, L e ( x , ˜ m ) is an entropy term over the image masked with the complement of m , L a ( m , n , S ) is a regularization term accounting for the area of the true and false positive masks, and L tv ( m , n ) is a regularization term favouring smooth masks.

Original: λ e , λ a and λ tv are hyperparameters balancing the loss terms.
Revised: λ e , λ a and λ tv are hyperparameters balancing the loss terms.

Original: Classiﬁcation loss: L c ( x , Y , m ) .
Revised: Classiﬁcation loss: L c ( x , Y , m ) .

Original: A model should be able to make correct decisions exclusivelyusing the relevant portions of the image,while ignoring all the rest.
Revised: A model should be able to make correct decisions exclusively using the relevant portions of the image, ignoring all the rest.

Original: Under this assumption, we deﬁnethe classiﬁcation loss as a sum over binary cross-entropy terms, one for each class contained in thetraining image.
Revised: Under this assumption, we deﬁne the classiﬁcation loss as a sum over binary cross-entropy terms, one for each class contained in the image.

Original: Representing the Explainer as E and the frozen Explanandum as F , we deﬁne p = F ( x (cid:12) m ) asthe probability vector returned by the model F applied on the masked image, using the aggregated(element-wise maximum) mask m provided by E .
Revised: Representing the Explainer as E and the frozen Explanandum as F , we deﬁne p = F ( x (cid:12) m ) as the probability vector returned by the model F applied on the masked image, using the aggregated (element-wise maximum) mask m provided by E .

Original: For all classes C in the training data and the set oftarget classes Y in the image, we compute:
Revised: For all classes C in the training data and the set of target classes Y in the image, we compute:

Original: [Equation]
Revised: [Equation]

Original: where (cid:74) c ∈ Y (cid:75) denotes the Iverson bracket, returning 1 if the condition is true (the image containsclass c ), 0 otherwise.
Revised: where (cid:74) c ∈ Y (cid:75) denotes the Iverson bracket, returning 1 if the condition is true (the image contains class c ), 0 otherwise.

Original: This allows training a mask when multiple classes are present in the trainingimage, as in multi-label classiﬁcation problems, where multiple masks are active at the same timeand some pixels are free not to belong to any mask (contrarily to using cross-entropy).
Revised: This allows training a mask when multiple classes are present in the training image, as in multi-label classiﬁcation problems, where multiple masks are active at the same time and some pixels are free not to belong to any mask (contrarily to using cross-entropy).

Original: Therefore, themodel is free to learn dependencies and co-occurrencies between different classes.
Revised: Therefore, the model is free to learn dependencies and co-occurrences between different classes.

Original: This loss pushesthe Explainer to learn masks approximating F ( x (cid:12) m )
Revised: This loss pushes the Explainer to learn masks approximating F ( x (cid:12) m )

Original: ≈ F ( x ) , i.e . masked images are classiﬁed ascorrectly as possible in the eyes of the (pre-)trained Explanandum .
Revised: ≈ F ( x ) , i.e . masked images are classiﬁed as correctly as possible in the eyes of the (pre-)trained Explanandum .

Original: Negative entropy loss: L e ( x , ˜ m ) .
Revised: Negative entropy loss: L e ( x , ˜ m ) .

Original: This part of the loss pushes the Explainer to provide maskswhose complements do not contain any discriminative visual cue, i.e . parts of the image that the
Revised: This part of the loss pushes the Explainer to provide masks whose complements do not contain any discriminative visual cue, i.e . parts of the image that the

Original: Explanandum could use to infer the correct class.
Revised: Explanandum could use to infer the correct class.

Original: In other words, the classiﬁer scores on the maskcomplement should provide class probabilities ˜ p
Revised: In other words, the classiﬁer scores on the mask complement should provide class probabilities ˜ p

Original: = F ( x (cid:12) ˜ m ) as uniform as possible.
Revised: = F ( x (cid:12) ˜ m ) as uniform as possible.

Original: To this end,we aim at minimizing the negative entropy of the predictions over the class memberships, as:
Revised: To this end, we aim at minimizing the negative entropy of the predictions over the class memberships, as:

Original: [Equation]
Revised: [Equation]

Original: Note that this loss is the negative entropy, as we aim for a high entropy for the background.
Revised: Note that this loss is the negative entropy, as we aim for a high entropy for the background.

Original: Area loss:
Revised: Area loss:

Original: L a ( m , n , S ) .
Revised: L a ( m , n , S ) .

Original: With the terms L c and L e alone, our Explainer would have no incentiveto produce a target mask that conceals image areas.
Revised: With the terms L c and L e alone, our Explainer would have no incentive to produce a target mask that conceals image areas.

Original: Clearly, a target mask m with 1 everywherewould minimize these terms with respect to m .
Revised: Clearly, a target mask m with 1 everywhere would minimize these terms with respect to m .

Original: To make sure m conceals background, we add twoterms to the loss that correspond to two crucial desiderata: The mask should be as small as possible,but if the objective would favour larger areas, a whole range of areas in between a minimal andmaximal percentage should not be penalized further.
Revised: To make sure m conceals background, we add two terms to the loss that correspond to two crucial desiderata: The mask should be as small as possible, but if the objective would favour larger areas, a whole range of areas in between a minimal and maximal percentage should not be penalized further.

Original: The regularization over the area is simply formulated as the mean of the mask values, computedas A ( m ) = 1 Z (cid:80) i,j m [ i, j ] , where Z is the number of pixels.
Revised: The regularization over the area is simply formulated as the mean of the mask values, computed as A ( m ) = 1 Z (cid:80) i,j m [ i, j ] , where Z is the number of pixels.

Original: We want this area to be small, butnon-zero.
Revised: We want this area to be small, but non-zero.

Original: The same is required of the non-target mask n as well.
Revised: The same is required of the non-target mask n as well.

Original: The term bounding the mask area from both above and below is formulated starting from the constraintpresented by Fong et al.
Revised: The term bounding the mask area from both above and below is formulated starting from the constraint presented by Fong et al.

Original: However, instead of constraining the ﬁnal image mask to a ﬁxed area,we extend their technique to determine a valid minimum area a and a maximum area b for those classsegmentation masks that correspond to at least one object in the image.
Revised: However, instead of constraining the ﬁnal image mask to a ﬁxed area, we extend their technique to determine a valid minimum area a and a maximum area b for those class segmentation masks that correspond to at least one object in the image.

Original: Given such a class segmentation mask s c , we ﬂatten it to a one-dimensional vector, and then sort itsvalues in descending order.
Revised: Given such a class segmentation mask s c , we ﬂatten it to a one-dimensional vector, and then sort its values in descending order.

Original: Let v ( s c ) be the vectorization operation and r ( v ( s c )) be the sorting of thevectorized class mask s c .
Revised: Let v ( s c ) be the vectorization operation and r ( v ( s c )) be the sorting of the vectorized class mask s c .

Original: We then deﬁne two vectors q min and q max , each starting with a predeﬁnednumber of 1 ’s and padded with 0 ’s to match the length of v ( s c ) , where the amount of 1 correspondsto the minimum 0 < a < b or maximum a < b < 1 area, respectively.
Revised: We then deﬁne two vectors q min and q max , each starting with a predeﬁned number of 1 ’s and padded with 0 ’s to match the length of v ( s c ) , where the amount of 1 corresponds to the minimum 0 < a < b or maximum a < b < 1 area, respectively.

Original: We also deﬁne q mask , whichcorresponds to s c , vectorized, and sorted.
Revised: We also deﬁne q mask , which corresponds to s c , vectorized, and sorted.

Original: Concretely:
Revised: Concretely:

Original: [Equation]
Revised: [Equation]

Original: To constrain the minimum and maximum area covered by the class segmentation mask, we thencompute an area bounding measure B given by:
Revised: To constrain the minimum and maximum area covered by the class segmentation mask, we then compute an area bounding measure B given by:

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst term penalizes the mask only if the largest (cid:98) Za (cid:99) values are smaller than 1 , while thesecond term penalizes the smallest Z − (cid:98) Zb (cid:99) values greater than 0 .
Revised: where the ﬁrst term penalizes the mask only if the largest (cid:98) Za (cid:99) values are smaller than 1 , while the second term penalizes the smallest Z − (cid:98) Zb (cid:99) values greater than 0 .

Original: Our goal is for the mask to coverat least a certain area independently of its maximum size.
Revised: Our goal is for the mask to cover at least a certain area independently of its maximum size.

Original: The ﬁnal area regularization is therefore:
Revised: The ﬁnal area regularization is therefore:

Original: [Equation]
Revised: [Equation]

Original: where Y is again the subset of the classes that are present in the given input image according to thegiven training labels.
Revised: where Y is again the subset of the classes that are present in the given input image according to the given training labels.

Original: Note that the last term of (6) used in conjunction with the smoothness losspresented below favors masks with sharp transitions between 0 and 1 .
Revised: Note that the last term of (6) used in conjunction with the smoothness loss presented below favors masks with sharp transitions between 0 and 1 .

Original: Smoothness Loss: L tv ( m , n ) .
Revised: Smoothness Loss: L tv ( m , n ) .

Original: As a ﬁnal requirement, we are interested in generating masks thatappear smooth and free of artifacts.
Revised: As a ﬁnal requirement, we are interested in generating masks that appear smooth and free of artifacts.

Original: For this purpose, we utilize a total variation (TV) loss on themasks as used by Fong and Vedaldi [11] and Dabkowski and Gal [5], albeit we use the anisotropicvariant given by:
Revised: For this purpose, we utilize a total variation (TV) loss on the masks as used by Fong and Vedaldi [11] and Dabkowski and Gal [5], albeit we use the anisotropic variant given by:

Original: [Equation]
Revised: [Equation]

Original: where x is a two-dimensional array of values.
Revised: where x is a two-dimensional array of values.

Original: The smoothness term of the loss relates to the targetmask m and non-target mask n as:
Revised: The smoothness term of the loss relates to the target mask m and non-target mask n as:

Original: [Equation]
Revised: [Equation]

Original: This smoothness term encourages our attributions to appear visually coherent.
Revised: This smoothness term encourages our attributions to appear visually coherent.

Original:  Experiments
Revised:  Experiments

Original: We present three results to establish the advantages of our Explainer over existing approaches.
Revised: We present three results to establish the advantages of our Explainer over existing approaches.

Original: Although continuous in nature, our masks show a sharp transition from background to foreground, making it effectively binary.
Revised: Although continuous in nature, our masks show a sharp transition from background to foreground, making it effectively binary.

Original: Additionally, our masks adhere tightly to region boundaries.
Revised: Additionally, our masks adhere tightly to region boundaries.

Original: Sincecommon objective metrics may fail to convey this, we provide a brief visual comparison.
Revised: Since common objective metrics may fail to convey this, we provide a brief visual comparison.

Original: The nextresult shows how our masks are class-speciﬁc, which provides more interpretive power than combinedmasks.
Revised: The next result shows how our masks are class-speciﬁc, which provides more interpretive power than combined masks.

Original: The third result shows that our combined masks outperform existing methods on standardmetrics for semantic segmentation.
Revised: The third result shows that our combined masks outperform existing methods on standard metrics for semantic segmentation.

Original: Since we expect our Explainer to learn class-speciﬁc semantics and generate class-speciﬁc masksof the same resolution as the input image, we employ DeepLabV3 [4] with a ResNet-50 backbone,an architecture commonly used for semantic segmentation tasks.
Revised: Since we expect our Explainer to learn class-speciﬁc semantics and generate class-speciﬁc masks of the same resolution as the input image, we employ DeepLabV3 [4] with a ResNet-50 backbone, an architecture commonly used for semantic segmentation tasks.

Original: We train this model from scratchto avoid any bias, i.e . an information leakage from the semantic segmentation dataset from which
Revised: We train this model from scratch to avoid any bias, i.e . an information leakage from the semantic segmentation dataset from which

Original: DeepLabV3 has been pre-trained on.
Revised: DeepLabV3 has been pre-trained on.

Original: For the Explanandum , we use VGG-16 [25] and ResNet-50 [14] models, both pretrained on ImageNet [6].
Revised: For the Explanandum , we use VGG-16 [25] and ResNet-50 [14] models, both pretrained on ImageNet [6].

Original: We ﬁne-tune them on the multi-label classiﬁcation tasks for VOC-2007 and COCO-2014,and then freeze the models.
Revised: We ﬁne-tune them on the multi-label classiﬁcation tasks for VOC-2007 and COCO- (i.e. on natural unmasked images), and then freeze the models.

Original: To this end, we use the full training set of VOC-2007 and 90% of the COCO-2014 training set for ﬁne-tuning.
Revised: To this end, we use the full training set of VOC-2007 and 90% of the COCO-2014 training set for ﬁne-tuning.

Original: To assess generalization, we use the test set of VOC-2007and the validation set of COCO-2014, respectively.
Revised: To assess generalization, we use the test set of VOC-2007 and the validation set of COCO-2014, respectively.

Original: For choosing hyperparameters, we use the VOC-2007 validation set and the remaining 10% of the COCO-2014 training set.
Revised: For choosing hyperparameters, we use the VOC-2007 validation set and the remaining 10% of the COCOtraining set.

Original: In this work, weimplemented models and experiments using PyTorch [19] and PyTorch Lightning [10].
Revised: In this work, we implemented models and experiments using PyTorch [19] and PyTorch Lightning [10] 1 .

Original: Code toreproduce the experiments can be found in the Supplementary Material.

Original: 4.1 Visual Comparison
Revised: 4.1 Visual Comparison

Original: [Figure]

Original: the VOC-2007 dataset and frozen.

Original: Our attributions are much more effective at retaining object class regions and discarding the rest.

Original: Examples 13 and 14 show cases where our Explainer is inaccurate.

Original: Visually, our Explainer leads to sharp and meaningful masks.
Revised: Visually, our Explainer leads to sharp and meaningful masks.

Original: Fig.
Revised: Fig.

Original: 3 provides a visual comparisonof how our Explainer attributions perform compared to the saliency maps obtained from Grad-CAM(GCam) [24], RISE [20], Extremal Perturbations (EP) [12], iGOS++ (iGOS) [16], and Real-Time
Revised: 3 provides a visual comparison of how our Explainer attributions perform compared to the saliency maps obtained from Grad-CAM (GCam) [24], RISE [20], Extremal Perturbations (EP) [12], iGOS++ (iGOS) [16], and Real-Time

Original: Image Saliency (RTIS) [5].
Revised: Image Saliency (RTIS) [5].

Original: For the RTIS baseline, we could not retrain their model on VOC-2007and COCO-2014, as it was not possible to employ their architecture on multi-class classiﬁers.
Revised: For the RTIS baseline, we could not retrain their model on VOCand COCO-2014, as it was not possible to employ their architecture on multi-class classiﬁers.

Original: Instead,we use their loss to train our architecture.
Revised: Instead, we use their loss to train our architecture.

Original: Accordingly, we ﬁne-tuned some hyperparameters of our
Revised: Accordingly, we ﬁne-tuned some hyperparameters of our

Original: RTIS implementation, while leaving the remaining ones at the default values as given in their paper.
Revised: RTIS implementation, while leaving the remaining ones at the default values as given in their paper.

Original: For all our experiments, we resized input images to 224x224 pixels and normalized them on the meanand standard deviation of ImageNet.
Revised: For all our experiments, we resized input images to 224x224 pixels and normalized them on the mean and standard deviation of ImageNet.

Original: Our approach provides sharp masks, which contrasts with classical saliency methods such as GradCAM or RISE.
Revised: Our approach provides sharp masks, which contrasts with classical saliency methods such as GradCAM or RISE.

Original: EP and iGOS++ provide masks which are better localized, although not as sharp asthe ones produced by RTIS or our Explainer .
Revised: EP and iGOS++ provide masks which are better localized, although not as sharp as the ones produced by RTIS or our Explainer .

Original: Our implementation of RTIS provides sharp masks, butthe overall attribution tends to be leaking over object borders.
Revised: Our implementation of RTIS provides sharp masks, but the overall attribution tends to be leaking over object borders.

Original: Our area constraints, in combination with the entropy term over the mask complement ˜ m , enforce tight masks over object classes ofdifferent sizes.
Revised: Our area constraints, in combination with the entropy term over the mask complement ˜ m , enforce tight masks over object classes of different sizes.

Original: For instance, Fig. 3(2), Fig. 3(5), or Fig.
Revised: For instance, Fig. 3(2), Fig. 3(5), or Fig.

Original: 3(8) show how our masks align much moreclearly with the objects.
Revised: 3(8) show how our masks align much more clearly with the objects.

Original: This not only holds for single-class images, but also when we have multipleinstances of the same class (Fig. 3(1), Fig. 3(4), Fig. 3(12)), or different classes (Fig. 3(6), Fig.
Revised: This not only holds for single-class images, but also when we have multiple ID Input GCam RISE

Revised: EP iGOS

Original: Fig. 3(10)).
Revised: RTIS Ours ID Input GCam

Revised: RISE

Revised: EP iGOS

Revised: RTIS

Revised: Ours              instances of the same class (Fig. 3(1), Fig.

Revised: 3(4), Fig. 3(12)), or different classes (Fig. 3(6), Fig.

Revised: Fig. 3(10)).

Original: Attribution of some animals ( e.g . Fig.
Revised: Attribution of some animals ( e.g . Fig.

Original: 3(7), Fig. 3(10), Fig. 3(11)) often tends to befocusing on heads.
Revised: 3(7), Fig. 3(10), Fig. 3(11)) often tends to be focusing on heads.

Original: Most training images containing animals have unobstructed heads, which canexplain such behaviour.
Revised: Most training images containing animals have unobstructed heads, which can explain such behaviour.

Original: For some classes, all methods agree on the importance of an image regionthat is not actually part of the actual object.
Revised: For some classes, all methods agree on the importance of an image region that is not actually part of the actual object.

Original: A good example for that is Fig.
Revised: A good example for that is Fig.

Original: 3(3), where the traintracks are part of the attribution.
Revised: 3(3), where the train tracks are part of the attribution.

Original: Since most training images of trains also show train tracks, it seemsobvious that the Explanandum learns to see the train tracks as discriminative semantic co-occurrence.
Revised: Since most training images of trains also show train tracks, it seems obvious that the Explanandum learns to see the train tracks as discriminative semantic co-occurrence.

Original: With the help of an Explainer , such unintended biases can quickly be detected.
Revised: With the help of an Explainer , such unintended biases can quickly be detected.

Original: Fig. 3(13) and Fig. 3(14) show two examples where our Explainer is less precise.
Revised: Fig. 3(13) and Fig. 3(14) show two examples where our Explainer is less precise.

Original: For Fig. 3(13), ourattribution leaks over to the bars below the horse.
Revised: For Fig. 3(13), our attribution leaks over to the bars below the horse.

Original: Even though the classiﬁer might indeed have madea connection between these bars and the “horse” class, Grad-CAM, RISE, EP, and iGOS++ appear toagree that the main importance should be attributed to the actual objects in the image.
Revised: Even though the classiﬁer might indeed have made a connection between these bars and the “horse” class, Grad-CAM, RISE, EP, and iGOS++ appear to agree that the main importance should be attributed to the actual objects in the image.

Original: In Fig.
Revised: In Fig.

Original: 3(14),it is difﬁcult to interpret our attribution for the class “boat” as some parts of the attribution lie on theactual object but also a signiﬁcant area of the surrounding water is activated (underlining potentialdiscriminative co-occurrence).
Revised: 3(14), it is difﬁcult to interpret our attribution for the class “boat” as some parts of the attribution lie on the actual object but also a signiﬁcant area of the surrounding water is activated (underlining potential discriminative co-occurrence).

Original: However, the other methods also provide varying explanations, withiGOS++ masking mostly the boat [Table] but Grad-CAM attributing all importance to the water.
Revised: However, the other methods also provide varying explanations, with iGOS++ masking mostly the boat, but Grad-CAM attributing all importance to the water.

Original: For moreresults on COCO-2014 and VOC-2007 please refer to Sup.
Revised: For more results on COCO-2014 and VOC-2007 please refer to Sup.

Original: Mat.
Revised: Mat.

Original: Sec.
Revised: Sec.

Original: D, E, and F.
Revised: D, E, and F.

Original: 4.2 Class-speciﬁc masking
Revised: 4.2 Class-speciﬁc masking

Original: [Table]

Original: Our Explainer provides class-speciﬁc masks.
Revised: Our Explainer provides class-speciﬁc masks.

Original: In Fig.
Revised: In Fig.

Original: 1 we have already hinted at how our Explaineris able to provide more class-speciﬁc masks than the other techniques.
Revised: 1 we have already hinted at how our Explainer is able to provide more class-speciﬁc masks than the other techniques.

Original: In this section we evaluatethis numerically.
Revised: In this section we evaluate this numerically.

Original: To this end, we have selected ﬁve target classes from the VOC-2007 dataset, forwhich we evaluate how the class masks interact with the classiﬁcation conﬁdence of the [-VGG-
 Explanandum-] .
Revised: To this end, we have selected ﬁve target classes from the VOC-2007 dataset, for which we evaluate how the class masks interact with the classiﬁcation conﬁdence of the [-VGG-
 Explanandum-]VGGExplanandum .

Original: We have selected the classes “cat” and “dog” as they are two animals that often occurin similar settings, the most prevalent class “person” that can often be seen together with a varietyof other classes and often inside the fourth class “car”, as well as the “bottle” class, to include a
Revised: We have selected the classes “cat” and “dog” as they are two animals that often occur in similar settings, the most prevalent class “person” that can often be seen together with a variety of other classes and often inside the fourth class “car”, as well as the “bottle” class, to include a

Original: VOC-2007 class that is very difﬁcult to detect for the Explanandum .
Revised: VOC-2007 class that is very difﬁcult to detect for the Explanandum .

Original: For each test image containingone of the target classes, we ﬁrst evaluate the classiﬁcation performance of the Explanandum on theunmasked image (“none” column) and then on masked images where the image is masked once witheach of the class-speciﬁc masks.
Revised: For each test image containing one of the target classes, we ﬁrst evaluate the classiﬁcation performance of the Explanandum on the unmasked image (“none” column) and then on masked images where the image is masked once with each of the class-speciﬁc masks.

Original: Speciﬁcally, for each of these continuous masks we report the meansoftmax probabilities for the target class, after we threshold each mask over nine uniform values in[0 . 1 , 0 . 9] at 0 .
Revised: Speciﬁcally, for each of these continuous masks we report the mean softmax probabilities for the target class, after we threshold each mask over nine uniform values in [0 . 1 , 0 . 9] at 0 .

Original: 1 intervals.
Revised: 1 intervals.

Original: Note that here we use the softmax instead of the sigmoid function despiteour multi-class setting since we want to evaluate the activation conﬁdence for a single class compared  to all other (ideally masked-out) classes.
Revised: Note that here we use the softmax instead of the sigmoid function despite our multi-class setting since we want to evaluate the activation conﬁdence for a single class compared [Table] to all other (ideally masked-out) classes.

Original: This can of course affect the scores for the completelyunmasked images where multiple object classes may be present.
Revised: This can of course affect the scores for the completely unmasked images where multiple object classes may be present.

Original: In Tab. 1, we show the evaluations for our Explainer , Grad-CAM, RISE and RTIS.
Revised: In Tab. 1, we show the evaluations for our Explainer , Grad-CAM, RISE and RTIS.

Original: For EP andiGOS++, it was not feasible to conduct these experiments since these methods take a very long timeto generate masks compared to the other ones (see Sup.
Revised: For EP and iGOS++, it was not feasible to conduct these experiments since these methods take a very long time to generate masks compared to the other ones (see Sup.

Original: Mat.
Revised: Mat.

Original: Sec.
Revised: Sec.

Original: A).
Revised: A).

Original: We can clearly see how themasks produced by our method allow the Explanandum to retain very good classiﬁcation conﬁdencefor the target classes when the correct mask is taken but render it completely uncertain when anyother mask is put on the image.
Revised: We can clearly see how the masks produced by our method allow the Explanandum to retain very good classiﬁcation conﬁdence for the target classes when the correct mask is taken but render it completely uncertain when any other mask is put on the image.

Original: This is in stark contrast to RISE and RTIS especially, which produceoverall larger, less precise masks.
Revised: This is in stark contrast to RISE and RTIS especially, which produce overall larger, less precise masks.

Original: Even though the Explanandum has high conﬁdence after maskingthe images with their corresponding correct masks, the same is still true when any wrong mask isused.
Revised: Even though the Explanandum has high conﬁdence after masking the images with their corresponding correct masks, the same is still true when any wrong mask is used.

Original: Only Grad-CAM achieves comparable performance to our Explainer but results in considerablyworse scores for the correct masks for 4 out of 5 classes and still shows some increased conﬁdencewhen the wrong mask is put on the images of an often co-occurring target class ( e.g . “cat” and “dog”,or “person” and “car”).
Revised: Only Grad-CAM achieves comparable performance to our Explainer but results in considerably worse scores for the correct masks for 4 out of 5 classes and still shows some increased conﬁdence when the wrong mask is put on the images of an often co-occurring target class ( e.g . “cat” and “dog”, or “person” and “car”).

Original: 4.3 Segmentation accuracy
Revised: 4.3 Segmentation accuracy

Original: [Table]

Original: Our attribution quality approaches object segmentation.
Revised: Our attribution quality approaches object segmentation.

Original: While our Explainer architecture provides excellent attributions visually, we also conﬁrm this quantitatively.
Revised: While our Explainer architecture provides excellent attributions visually, we also conﬁrm this quantitatively.

Original: Our explanations approximateobject-level segmentation and can thus be evaluated using standard instance and semantic segmentation benchmarks.
Revised: Our explanations approximate object-level segmentation and can thus be evaluated using standard instance and semantic segmentation benchmarks.

Original: In Tab. 2 we illustrate the accuracy of the masking strategies, on VOC-2007and COCO-2014, with VGG-16 and ResNet-50 as Explanandum , respectively.
Revised: In Tab. 2 we illustrate the accuracy of the masking strategies, on VOCand COCO-2014, with VGG-16 and ResNet-50 as Explanandum , respectively.

Original: To provide a faircomparison with our method, we evalaute all methods by aggregating the masks from multiple classesinto a single one, as shown in Fig.
Revised: To provide a fair comparison with our method, we evaluate all methods by aggregating the masks from multiple classes into a single one, as shown in Fig.

Original: We also provide four base measures that serve as sanity checkbaselines in Tab. 2), i.e . the columns “0”, “0.5”, “1”, and “G.T.” (ground truth).
Revised: We also provide four base measures that serve as sanity check baselines in Tab. 2), i.e . the columns “0”, “0.5”, “1”, and “G.T.” (ground truth).

Original: These measures showhow the assessment metrics range under border conditions: with a completely null mask (composedof only 0 ’s), with a mask composed fully of 0 .
Revised: These measures show how the assessment metrics range under border conditions: with a completely null mask (composed of only 0 ’s), with a mask composed fully of 0 .

Original: 5 ’s, with a non-mask composed of only 1 ’s and ﬁnallywith a mask which is perfectly equal to the ground truth segmentation.
Revised: 5 ’s, with a non-mask composed of only 1 ’s and ﬁnally with a mask which is perfectly equal to the ground truth segmentation.

Original: We employ four different metrics, namely Accuracy (Acc), Mean IoU (IoU), Saliency (Sal) and
Revised: We employ four different metrics, namely Accuracy (Acc), Mean IoU (IoU), Saliency (Sal) and

Original: Mean Absolute Error (MAE).
Revised: Mean Absolute Error (MAE).

Original: All metrics are computed on a per-image basis, by using the groundtruth segmentation masks and the aggregated mask of the true classes present in each image, thenaveraged over the test set.
Revised: All metrics are computed on a per-image basis, by using the ground truth segmentation masks and the aggregated mask of the true classes present in each image, then averaged over the test set.

Original: The Accuracy is computed pixel-wise over the continuous masks without thresholding.
Revised: The Accuracy is computed pixel-wise over the continuous masks without thresholding.

Original: The same holds for the IoU, where we take the pixel-wise minimum of the mask andthe ground truth as the intersection and the maximum as the union, respectively.
Revised: The same holds for the IoU, where we take the pixel-wise minimum of the mask and the ground truth as the intersection and the maximum as the union, respectively.

Original: The Saliency scoreis adapted from Dabkowski and Gal [5], and computed as:
Revised: The Saliency score [Table] is adapted from Dabkowski and Gal [5], and computed as:

Original: [Equation]
Revised: [Equation]

Original: where Y is the set of classes present in the image.
Revised: where Y is the set of classes present in the image.

Original: This metric balances the size of the mask m withthe Explanandum outputs.
Revised: This metric balances the size of the mask m with the Explanandum outputs.

Original: Smaller masks minimize the ﬁrst term but have a tendency to lower theclassiﬁcation conﬁdence, thus increasing the second term.
Revised: Smaller masks minimize the ﬁrst term but have a tendency to lower the classiﬁcation conﬁdence, thus increasing the second term.

Original: It is therefore possible to get a lower scorecompared to the ground truth mask as a very small mask reduces the ﬁrst term greatly, while it mightstill preserve a lot of the classiﬁcation conﬁdence.
Revised: It is therefore possible to get a lower score compared to the ground truth mask as a very small mask reduces the ﬁrst term greatly, while it might still preserve a lot of the classiﬁcation conﬁdence.

Original: This is to the disadvantage of our method, aswe aim to also attribute regions of lower (but still signiﬁcant) importance instead of only the mostdistinguishing features.
Revised: This is to the disadvantage of our method, as we aim to also attribute regions of lower (but still signiﬁcant) importance instead of only the most distinguishing features.

Original: Finally, the MAE corresponds to the mean absolute error over the full dataset,which is computed for each input image as: MAE ( m , g ) =
Revised: Finally, the MAE corresponds to the mean absolute error over the full dataset, which is computed for each input image as: MAE ( m , g ) =

Original: 1 Z (cid:80) i,j | m [ i, j ] − g
Revised: 1 Z (cid:80) i,j | m [ i, j ] − g

Original: [ i, j ] | , where g is theaggregated ground truth segmentation map for all object classes in the input image.
Revised: [ i, j ] | , where g is the aggregated ground truth segmentation map for all object classes in the input image.

Original: Note that certainmetrics might favor very small masks ( e.g . Sal), while others favor very large masks ( e.g . IoU).
Revised: Note that certain metrics might favor very small masks ( e.g . Sal), while others favor very large masks ( e.g . IoU).

Original: The assessment in this sub-section assumes that an ideal method should perform attribution over thewhole ground truth segmentation, which is not the case in certain situations, as attributions shouldonly cover classiﬁer-relevant parts of the object class ( e.g . face instead of whole body).
Revised: The assessment in this sub-section assumes that an ideal method should perform attribution over the whole ground truth segmentation, which is not the case in certain situations, as attributions should only cover classiﬁer-relevant parts of the object class ( e.g . face instead of whole body).

Original: We arguethat although attributions do not directly correspond to semantic segmentations, and while differentmethods target different goals for attribution, the combined evaluation of these metrics can still beused for a standardized assessment.
Revised: We argue that although attributions do not directly correspond to semantic segmentations, and while different methods target different goals for attribution, the combined evaluation of these metrics can still be used for a standardized assessment.

Original: 4.4 Limitations
Revised: 4.4 Limitations

Original: Not all pairs of Explainer and Explanandum interact equally well and allow straightforward training.
Revised: Not all pairs of Explainer and Explanandum interact equally well and allow straightforward training.

Original: We observed that, while the explanation of VGG-16 and ResNet-50 on the VOC-2007 datasetperforms consistently, the explanation for COCO-2014 is harder on ResNet-50 than on VGG-16 (see
Revised: We observed that, while the explanation of VGG-16 and ResNet-50 on the VOC-2007 dataset performs consistently, the explanation for COCO-2014 is harder on ResNet-50 than on VGG-16 (see

Original: Sup.
Revised: Sup.

Original: Mat.
Revised: Mat.

Original: Sec.
Revised: Sec.

Original: E and F), even though the accuracies are comparable (see Sup.
Revised: E and F), even though the accuracies are comparable (see Sup.

Original: Mat.
Revised: Mat.

Original: Sec.
Revised: Sec.

Original: B, Tab. 4).
Revised: B, Tab. 4).

Original: Sometimes, masks fail to precisely cover the discriminative objects parts.
Revised: Sometimes, masks fail to precisely cover the discriminative objects parts.

Original: Some examples are givenin Fig. 3(13) and Fig. 3(14), and further examples are given in the Sup.
Revised: Some examples are given in Fig. 3(13) and Fig. 3(14), and further examples are given in the Sup.

Original: Mat.
Revised: Mat.

Original: Sec.
Revised: Sec.

Original: D, E and F.
Revised: D, E and F.

Original:  Conclusions
Revised:  Conclusions

Original: We have presented an approach for visually attributing the classiﬁcation result of a frozen, pre-trained,black-box classiﬁer (the Explanandum ) on speciﬁc regions of an input image.
Revised: We have presented an approach for visually attributing the classiﬁcation result of a frozen, pre-trained, black-box classiﬁer (the Explanandum ) on speciﬁc regions of an input image.

Original: This involves trainingan Explainer architecture with the training data that was used to train the Explanandum .
Revised: This involves training an Explainer architecture with the training data that was used to train the Explanandum .

Original: The resultingmodel generates attributions in the form of class-wise attribution masks, as well as an aggregatedforeground mask with a single forward pass for any test image.
Revised: The resulting model generates attributions in the form of class-wise attribution masks, as well as an aggregated foreground mask with a single forward pass for any test image.

Original: As compared to the gradient andactivation-based approaches, which often generate blurry saliency maps, our masks locate objectregions more accurately and the delineation of these regions is sharp (Sec. 4.1).
Revised: As compared to the gradient and activation-based approaches, which often generate blurry saliency maps, our masks locate object regions more accurately and the delineation of these regions is sharp (Sec. 4.1).

Original: As comparedto the approaches that use perturbed inputs, the Explainer is not only more precise, as we provequantitatively (Sec. 4.2 and 4.3), but also computationally more efﬁcient (Sup.
Revised: As compared to the approaches that use perturbed inputs, the Explainer is not only more precise, as we prove quantitatively (Sec. 4.2 and 4.3), but also computationally more efﬁcient (Sup.

Original: Mat.
Revised: Mat.

Original: Sec.
Revised: Sec.

Original: A).
Revised: A).

Original: We observe that the Explainer can be learned well if the pre-trained classiﬁer exhibits a good level ofaccuracy.
Revised: We observe that the Explainer can be learned well if the pre-trained classiﬁer exhibits a good level of accuracy.

Original: Since learning the attribution fully depends on the signals provided by the Explanandum ,this comes as no surprise.
Revised: Since learning the attribution fully depends on the signals provided by the Explanandum , this comes as no surprise.

Original: If instead of the original image, we use inputs that are masked by our
Revised: If instead of the original image, we use inputs that are masked by our

Original: Explainer , the accuracy may slightly decrease (Sup. Mat.
Revised: Explainer , the accuracy may slightly decrease (Sup. Mat.

Original: Sec.
Revised: Sec.

Original: B).
Revised: B).

Original: Since our masks are so parsimoniousin retaining the essential parts of the original image, it points to the fact that the drop in accuracymight come from the Explanandum being deprived of background-dependent biases.
Revised: Since our masks are so parsimonious in retaining the essential parts of the original image, it points to the fact that the drop in accuracy might come from the Explanandum being deprived of background-dependent biases.

Original: Therefore, oncethe Explainer is trained to generate masks, one could imagine retraining the Explanandum with theoriginal as well as Explainer -masked data.
Revised: Therefore, once the Explainer is trained to generate masks, one could imagine retraining the Explanandum with the original as well as Explainer -masked data.

Original: This can ensure that classiﬁcation accuracy can be bothhigh and free of inﬂuences from non-object regions.
Revised: This can ensure that classiﬁcation accuracy can be both high and free of inﬂuences from non-object regions.

Original: Finally, while our work focuses on explainingdeep image classiﬁers, we believe it should be possible to extend such an approach to other black boxclassiﬁers, other data modalities, and other learning tasks such as regression.
Revised: Finally, while our work focuses on explaining deep image classiﬁers, we believe it should be possible to extend such an approach to other black-box classiﬁers, other data modalities, and other learning tasks such as regression.

Original: Checklist
Revised: A Compute Times

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
Revised: To evaluate the practicability of the methods, we have evaluated the time it takes for each method to generate all aggregated masks for the segmentation evaluation in Section 4.3. Note that this does not include any training time for our method and adapted RTIS (up to 12 hours on VOC-2007 and up todays on COCO-2014), which only has to be performed once per model and dataset.

Original: Yes
Revised: Tab.

Original: ](b) Did you describe the limitations of your work?
Revised: 3 shows how signiﬁcant the inference time differences between the methods are, even for a relatively small amount of images.

Original: [Yes] See Section 4.4.
Revised: RISE, EP and iGOS++ are orders of magnitude slower than GCAM, RTIS and our explainer. All experiments have been conducted using the same Tesla P100 PCIe 12GB GPU. [Table]

Original: c)
Revised: We demonstrate through Tab. 4 that our Explainer architecture does not adversely affect the performance of the (pre-)trained classiﬁcation network. On occasions, a slight drop in accuracy can be expected, since each individual image changes quite drastically after being masked, not only visually, but also in terms of pixel intensity. Such a drop can be explained simply by the fact that classiﬁers are not trained to recognize objects on a black background, which can behave as adversarial information to the VGG-16 and ResNet-50 classiﬁers.

Original: Did you discuss any potential negative societal impacts of your work
Revised: A loss in performance may also be indicative of the pre-trained classiﬁer not being able to capitalize on an unexpected cue or correlations in the data (such as presence of blue and green in images with cows). C Multiclass attribution examples on VOC using VGG-16 and ResNet-  In this section, we show examples for the attribution over multiple classes on the VOC-2007 test set, using VGG-16 and ResNet-50 classiﬁers. In Fig.

Original: [N/A] d)
Revised: 4 and Fig.

Original: Have you read the ethics review guidelines and ensured that your paper conforms tothem
Revised: 5 below, each row corresponds to a random image from the VOC-2007 dataset, while each column corresponds to: the original image, the aggregated mask (per-pixel maximum over the class-masks), and the top 5 strongest attributed classes (TAC).

Original: [Yes
Revised: Fig.

Original: If you are including theoretical results...
Revised: 4 shows the results for VGG-16, while Fig.

Original: (a) Did you state the 
Revised: 5 shows the ones for ResNet-50. Each mask is scored according to the average mask activation (AMA), then sorted in descending order of the ﬁrst 5 classes, as shown below.

Original: full
Revised: As this process is completely independent from the classiﬁer itself, we also add the multi-label scores (logits through the sigmoid) for the original classiﬁer on the unmasked image (CLS), multiplied by 100.

Original: set of assumptions of all theoretical results
Revised: When a class from the ground truth is attributed, the

Original: [N/A] b Did
Revised: TAC is accompanied by two asterisks (**).

Original: you
Revised: We do not report the classiﬁcation scores for the masked images, because they are greatly perturbed since the classiﬁer has never learned to classify heavily masked images.

Original: include complete proofs of all theoretical results
Revised: For example, the classiﬁer will give relatively high scores for all classes when it is given a completely black (all zeros) image.

Original: [N/A]
Revised: [Table]

Original: If you
Revised: can for

Original: ran experiments...
Revised: VGG-16 (Fig

Original: a Did
Revised: ) and

Original: you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: (Fig of and that they usually well masks show their highest are not the for gives the highest probability might have learned to detect better than the itself. Remember that those predictions are made directly on out

Original: Yes
Revised: Also note that the average mask activation does of course not directly relate to a classiﬁcation since smaller objects will give a smaller AMA score than large ones.

Original: ] The code issubmitted in Supplementary materials and will be released upon acceptance
Revised: An interesting example is at the second row in Fig.

Original: b ) Did you specify all the training details ( e.g ., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4 and code when released.
Revised: 4, where in the fourth column (second most attributed class) the Explainer shows a signiﬁcant attribution for class 5 (“bottle”) which is not present in the image.

Original: [Table]
Revised: Only when looking at the classiﬁer score on the unmasked image (CLS) this makes perfect sense since the classiﬁer gives the “bottle” class a probability of over 60%. A similar example for this behavior is the ﬁfth row in the same ﬁgure or the seventh row in Fig. This indicates that the attributions by our Explainer are sensible even in failure cases of the classiﬁer.

Original: (c) Did you report error bars ( e.g ., with respect to the random seed after running experiments multiple times)?
Revised: In a few cases, the attributions for several classes are on the same image areas, as in the third and seventh rows of Fig.

Original: [No] Experiments were run only once.
Revised: This might happen for image areas that appear to be generally interesting for several classes at the same time but such cases are rare. In general, we see that ResNet-50 tends to provide masks with slightly more artifacts and spurious activations, when compared to the attributions of VGG-16. Both experiments seem to agree that our Explainer is able to attribute classiﬁers to regions occupied by the correct class, or provide attributions caused by ambiguous biases in the classiﬁer itself.

Original: [Table]
Revised: [Table]

Original: (d) Did you include the total amount of compute and the type of resources used ( e.g ., typeof GPUs, internal cluster, or cloud provider)?
Revised: 5 TAC 15**   3 CLS 98. 0.96 AMA 20. TAC 15** 11** 13 CLS 99.

Original: [Yes] See Appendix A
Revised: 0.06 AMA 26.

Original: [Table]
Revised: [Figure]

Original: If you are using existing assets ( e.g ., code, data, models) or curating/releasing new assets...
Revised: 4**   19 CLS 5. 1.32 AMA 13. [Figure]

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: [Figure] [Figure] [Figure] [Figure]

Original: [Yes] See Section
Revised: [Figure] [Figure]

Original: (b) Did you mention the license of the assets?
Revised: **   14 CLS 50. [Figure] [Figure] [Figure]

Original: [No] We used openly available code libraries,datasets, architectures and pretrained models, and we reference them.
Revised: [Figure] [Figure] [Figure] [Figure]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: CLS

Original: [No]
Revised: [Figure]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: [Figure] [Figure] [Figure] [Figure]

Original: using/curating?
Revised: [Figure]

Original: [N/A]
Revised: [Figure]

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable
Revised: [Figure] [Figure] [Figure] [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Original: information or offensive content?
Revised: their 5 TAC 15** 7**  10 CLS 77. 0.96 AMA 23. TAC 3**   11 CLS 94.

Revised: 0.18 AMA 32.

Revised: TAC 15**   2 CLS 96.

Revised: 0.76 AMA 24.

Revised: TAC 19**   11 CLS 98.

Revised: 1.00 AMA 30.

Revised: TAC 15**   17 CLS 99.

Revised: 0.15 AMA 29.

Revised: TAC 7**   10 CLS 97.

Revised: 0.35 AMA 31.

Revised: [Figure]

Revised: ** 18** 11 CLS 76.

Revised: [Figure]

Revised: .12 AMA 22.

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: VOC-2007 test set.

Revised: Class-wise masks are sorted according to their average activation on the image plane.

Revised: TAC corresponds to the top activating class, please refer to the legend in Tab. 5.

Revised: CLS shows the respective class probabilities by the classiﬁer on the original (unmasked) images, multiplied by 100.

Revised: AMA shows the average mask activations for the respective class, also multiplied by 100.

Revised: D Comparison of attribution methods on VOC-2007 using ResNet-50.

Revised: Figure 6 provides additional comparisons using ResNet-50 on VOC-2007.

Revised: The comparison follows

Revised: Fig.

Revised: 3 of the manuscript.

Revised: Overall, attribution methods for ResNet-50 are behaving better on VOC2007 than on COCO-2014.

Revised: Still, GCam, RISE and EP tend to provide smooth attributions, with the latter two sometimes scoring at multiple locations which are hard to relate to the classes present in the image.

Revised: iGOS++ provides small activations that mostly seem to outline the most important object parts but sometimes also show artifacts that are difﬁcult to interpret when it is less certain.

Revised: RTIS provides masks that are often outlining objects, but fails in being sharp and providing clear attributions due to overshooting too much.

Revised: Our Explainer method is mostly artifact-free and masks are very sharp on discriminative object parts.

Revised: [Figure]

Revised: E Comparison of attribution methods on COCO-2014 using VGG-16.

Revised: Figure 7 provides additional comparisons using VGG-16 on COCO-2014.

Revised: The comparison follows

Revised: Fig.

Revised: 3 of the manuscript.

Revised: Using VGG-16 on COCO-2014 results in good attributions, where masks are sharp and outline discriminative parts for the predicted classes precisely.

Revised: Not too differently than the results presented in the manuscript – attribution for the VGG-16 classiﬁer on VOC-2007 – we can see that GCam [24] provides good attributions, if not for a fairly large smoothing.

Revised: RISE [20] behaves similarly, and it often tends to provide several oversmoothed saliency locations, which are hard to interpret.

Revised: EP [12] is much sharper, but also tends to overpredict attributions.

Revised: On the other hand, iGOS++ shows small, localized attributions.

Revised: However, it often also includes artifacts and sometimes does not capture certain objects at all.

Revised: Finally, RTIS [5] does not show well delineated attribution on this combination of data and classiﬁer as it overshoots too much to allow for a clear interpretation.

Revised: Our Explainer provides localized attributions which are easy to interpret, with sharp boundaries, which in most cases coincide with the classes of interest or discriminative parts composing them.

Revised: In general, the Explainer predicts compact masks with smaller areas, similarly to GCam, but much less smoothed overall.

Original: [N/A]
Revised: [Figure]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: Figure 8 provides additional comparisons using ResNet-50 on COCO-2014. The comparison follows Fig.

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: 3 of the manuscript. As discussed in the manuscript, this combination of data and classiﬁer is the hardest to train the Explainer on. As for previous comparisons, GCam shows accurate but very smooth attributions. RISE and EP behave similarly, by attributing the classiﬁcation on the correct areas of the input image, but failing in focusing on precise image locations.

Original: applicable?
Revised: iGOS++ once again shows small attributions, which in many cases lead to clear explanations but sometimes might also leave out parts where the other methods agree that they are important as well. In some cases it also includes artifacts, which make the interpretation harder.

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
Revised: RTIS overpredicts the attributions, in particular when the objects of interest are small with respect to the input image plane.

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: Our Explainer shows very clear attributions in most cases but sometimes also includes artifacts that occur as an active area in the top portion of the image.

Original: spent on participant compensation?
Revised: We have found that this speciﬁcally happens in images with the “person” class. Nonetheless, we can notice that the attribution masks are still precise and sharp in the object areas, suggesting that better results could probably be achieved by better hyperparameter and model selection.

Original: [N/A]
Revised: [Figure]