Original: F EATURE P ERTURBATION A UGMENTATION FOR R ELIABLE E VALUATION OF I MPORTANCE E STIMATORS
Revised: F EATURE P ERTURBATION A UGMENTATION FOR R ELIABLE E VALUATION OF I MPORTANCE E STIMATORS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Post-hoc explanation methods attempt to make the inner workings of deep neural networks more comprehensible and trustworthy, which otherwise act as black box models.
Revised: Post-hoc explanation methods attempt to make the inner workings of deep neural networks more comprehensible and trustworthy, which otherwise act as black box models.

Original: However, since a ground truth is in general lacking, local post-hoc explanation methods, which assign importance scores to input features, are challenging to evaluate.
Revised: However, since a ground truth is in general lacking, local post-hoc explanation methods, which assign importance scores to input features, are challenging to evaluate.

Original: One of the most popular evaluation frameworks is to perturb features deemed important by an explanation and to measure the change in prediction accuracy.
Revised: One of the most popular evaluation frameworks is to perturb features deemed important by an explanation and to measure the change in prediction accuracy.

Original: Intuitively, a large decrease in prediction accuracy would indicate that the explanation has correctly quantified the importance of features with respect to the prediction outcome (e.g., logits).
Revised: Intuitively, a large decrease in prediction accuracy would indicate that the explanation has correctly quantified the importance of features with respect to the prediction outcome (e.g., logits).

Original: However, the change in the prediction outcome may stem from perturbation artifacts, since perturbed samples in the test dataset are out of distribution (OOD) compared to the training dataset and can therefore potentially disturb the model in an unexpected manner.
Revised: However, the change in the prediction outcome may stem from perturbation artifacts, since perturbed samples in the test dataset are out of distribution (OOD) compared to the training dataset and can therefore potentially disturb the model in an unexpected manner.

Original: To overcome this challenge, we propose feature perturbation augmentation (FPA) which creates and adds perturbed images during the model training.
Revised: To overcome this challenge, we propose feature perturbation augmentation (FPA) which creates and adds perturbed images during the model training.

Original: Through extensive computational experiments, we demonstrate that FPA makes the considered models more robust against perturbations.
Revised: Our computational experiments suggest that FPA makes the considered models more robust against perturbations.

Original: Overall, FPA is an intuitive and straightforward data augmentation technique that renders the evaluation ofof post-hoc explanation more trustworthy 1 .
Revised: Overall, FPA is an intuitive and straightforward data augmentation technique that renders the evaluation of post-hoc explanations more trustworthy.

Revised: Codes and models trained with FPA are available: https://github.com/ lenbrocki/Feature-Perturbation-Augmentation

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep learning exhibits state-of-the-art performance in a wide range of computer vision tasks.
Revised: Deep learning exhibits state-of-the-art performance in a wide range of computer vision tasks.

Original: However, the reasons underlying classifications and predictions made by deep neural networks (DNN) are difficult to extract due to their nested non-linear structure and a large number of parameters (Samek et al., 2019).
Revised: However, the reasons underlying classifications and predictions made by deep neural networks (DNN) are difficult to extract due to their nested non-linear structure and a large number of parameters (Samek et al., 2019).

Original: A popular approach to make deep learning models more interpretable are post-hoc explanation methods, which estimate the importance of input features with respect to the model’s output (Simonyan et al., 2014; Smilkov et al., 2017; Sundararajan et al., 2017).
Revised: A popular method to make deep learning models more interpretable are posthoc explanations, which estimate the importance of input features with respect to the model’s output (Simonyan et al., 2014; Smilkov et al., 2017; Sundararajan et al., 2017).

Original: However, evaluating the fidelity of post-hoc importance estimators is highly convoluted due to a lack of ground truth and the issue of unintentionally triggering perturbation artifacts.
Revised: However, evaluating the fidelity of post-hoc importance estimators is highly convoluted due to a lack of ground truth and the issue of unintentionally triggering perturbation artifacts.

Original: In this study, we introduce feature perturbation augmentation (FPA) which aims to avoid the pitfalls of  perturbation-based evaluation of interpretability methods.
Revised: In this study, we introduce feature perturbation augmentation (FPA) which aims to avoid the pitfalls of a perturbation-based evaluation of interpretability methods.

Original: The major challenge with evaluating importance estimators is the lack of a ground truth against which a given estimator can be compared.

Original: A popular approach for comparing importance estimators despite this lack of ground truth is the perturbation of input features (Samek et al., 2016;
Revised: A promising approach for comparing importance estimators despite the aforementioned lack of ground truth is the perturbation of input features (Samek et al., 2016;

Original: Petsiuk et al., 2018; Kindermans et al., 2017).
Revised: Petsiuk et al., 2018; Kindermans et al., 2017).

Original: Conceptually, if the model’s accuracy rapidly decreases by masking pixels deemed most important by some estimator, then it can be concluded that the considered estimator describes the model more accurately than others that result in a slower decrease.
Revised: Conceptually, if the model’s accuracy rapidly decreases by masking pixels deemed most important by some estimator, then it can be concluded that the considered estimator describes the model more accurately than others that result in a slower decrease.

Original: However, such an evaluation by perturbation may be problematic due to the risk of unwittingly triggering artifacts of the deep learning model (Hooker et al., 2019; Fong & Vedaldi, 2017).
Revised: However, such an evaluation by perturbation may be problematic due to the risk of unwittingly triggering artifacts of the deep learning model (Hooker et al., 2019; Fong & Vedaldi, 2017).

Original: In other words, even when truly unimportant pixels are masked, the accuracy might decrease considerably nonetheless, casting doubt on the reliability of the perturbation-based evaluation approach.
Revised: In other words, even when truly unimportant pixels are masked, the accuracy might decrease considerably nonetheless, casting doubt on the reliability of the perturbation-based evaluation approach.

Original: Our proposed approach mitigates the influence of perturbation artifacts by training the model with data augmentation that reflects the perturbation used in the evaluation frameworks.
Revised: Our proposed approach mitigates the influence of perturbation artifacts by training the model with data augmentation that reflects the perturbation used in the evaluation frameworks.

Original: We apply the proposed methods on three datasets (CIFAR-10 (Krizhevsky et al.), Food101 (Bossard et al., 2014), the ImageNet (Deng et al., 2009)), using four different post-hoc explanation methods.
Revised: We apply the proposed methods on three datasets (CIFAR-10 (Krizhevsky et al.), Food101 (Bossard et al., 2014), the ImageNet (Deng et al., 2009)), using four different post-hoc explanation methods.

Original: Subsequently, we measure the model output while perturbing an increasing fraction of input features sorted either in most important first (MIF) or least important first (LIF) order.
Revised: Subsequently, we measure the model output while perturbing an increasing fraction of input features sorted either in most important first (MIF) or least important first (LIF) order.

Original: MIF and LIF perturbation curves demonstrate that when using FPA during training, the resulting model exhibits increased robustness against perturbation artifacts and the evaluation of importance estimators is more reliable.
Revised: MIF and LIF perturbation curves demonstrate that when using FPA during training, the resulting model exhibits increased robustness against perturbation artifacts and the evaluation of importance estimators is more reliable.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: The explainability of deep learning is an active and diverse area of research (reviewed in Samek et al.
Revised: The explainability of deep learning is an active and diverse area of research (reviewed in Samek et al.

Original: There are many desired properties (desideratas) of interpretability methods.
Revised: There are many desired properties (desideratas) of interpretability methods.

Original: In this work, we focus on faithfulness of post-hoc explanations to the underlying model.
Revised: In this work, we focus on faithfulness of post-hoc explanations to the underlying model.

Original: Other desideratas of interpretability methods include localisation around a region of interest (ROI)
Revised: Other desideratas of interpretability methods include localisation around a region of interest (ROI)

Original: (Zhou et al., 2016; Selvaraju et al., 2017; Brocki et al., 2022), sensitivity to randomizations of model parameters (Adebayo et al., 2018) or targeted logits (Sixt et al., 2020), sparseness (Chalasani et al., 2020), and axiomatic properties (Sundararajan et al., 2017).
Revised: (Zhou et al., 2016; Selvaraju et al., 2017; Brocki et al., 2022), sensitivity to randomizations of model parameters (Adebayo et al., 2018) or targeted logits (Sixt et al., 2020), sparseness (Chalasani et al., 2020), and axiomatic properties (Sundararajan et al., 2017).

Original: Faithfulness or fidelity describes how accurately explanation methods estimate the contribution of input features to the model’s predictions.
Revised: Faithfulness or fidelity describes how accurately explanation methods estimate the contribution of input features to the model’s predictions.

Original: The proposed FPA is related to pixel-flipping (Bach et al., 2015), region-perturbation (Samek et al., 2016) and Remove and Retrain (ROAR)
Revised: The proposed FPA is related to pixel-flipping (Bach et al., 2015), region-perturbation (Samek et al., 2016) and Remove and Retrain (ROAR)

Original: (Hooker et al., 2019).
Revised: (Hooker et al., 2019).

Original: All these methods perturb input pixels and measure the resulting change in model performance.
Revised: All these methods perturb input pixels and measure the resulting change in model performance.

Original: Other methods to evaluate faithfulness include faithfulness correlation (Bhatt et al., 2020) and sensitivity-n (Ancona et al., 2017), which measure the correlation between the sum of importance scores of masked pixels and the delta in model output.
Revised: Other methods to evaluate faithfulness include faithfulness correlation (Bhatt et al., 2020) and sensitivity-n (Ancona et al., 2017), which measure the correlation between the sum of importance scores of masked pixels and the delta in model output.

Original: Dabkowski & Gal (2017) proposes to crop images to a region deemed important and feed the resized crop back to the model.
Revised: Dabkowski & Gal (2017) proposes to crop images to a region deemed important and feed the resized crop back to the model.

Original: Related faithfulness methods include Performance Information Curves (Kapishnikov et al., 2019),
Revised: Related faithfulness methods include Performance Information Curves (Kapishnikov et al., 2019),

Original: ROAD
Revised: ROAD

Original: Rong et al., 2022), IROF (Rieger & Hansen, 2020), and Infidelity (Yeh et al., 2019).
Revised: Rong et al., 2022), IROF (Rieger & Hansen, 2020), and Infidelity (Yeh et al., 2019).

Original: Data augmentation can also make machine learning models more robust and generalizable.
Revised: Data augmentation can also make machine learning models more robust and generalizable.

Original: From noise injection to utilizing complex DNNs for synthetic data, there are many data augmentation techniques (reviewed in Shorten & Khoshgoftaar (2019)).
Revised: From noise injection to utilizing complex DNNs for synthetic data, there are many data augmentation techniques (reviewed in Shorten & Khoshgoftaar (2019)).

Original: In deep learning, one may apply geometric and color manipulations, make use of noise and filters, and modify feature space.
Revised: In deep learning, one may apply geometric and color manipulations, make use of noise and filters, and modify feature space.

Original: The closest approach to the proposed FPA method is “random erasing” (Zhong et al., 2020).
Revised: The closest approach to the proposed FPA method is “random erasing” (Zhong et al., 2020).

Original: Random erasing augments the data by selecting a random rectangle in an image and replacing them with non-informative values such as white, black, or random RGB values.
Revised: Random erasing augments the data by selecting a random rectangle in an image and replacing them with non-informative values such as white, black, or random RGB values.

Original: FPA may be seen as a generalization of noise injection and random erasing where rectangles of varying sizes are probabilistically used to perturb the input data.
Revised: FPA may be seen as a generalization of noise injection and random erasing where rectangles of varying sizes are probabilistically used to perturb the input data.

Original: Of course, the aim of FPA is very different from other data augmentations techniques, since we focus on improving post-hoc interpretability (Dziugaite et al., 2020).
Revised: Of course, the aim of FPA is very different from other data augmentations techniques, since we focus on improving post-hoc interpretability (Dziugaite et al., 2020).

Original:  M ETHODS AND M ATERIALS F EATURE P ERTURBATION A UGMENTATION
Revised:  M ETHODS AND M ATERIALS F EATURE P ERTURBATION A UGMENTATION

Original: The perturbation-based evaluation of importance estimators has been criticized (Hooker et al., 2019) since the perturbation of input pixels leads to a shift in the data distribution, violating the key assumption that training and test data stem from the same distribution.
Revised: The perturbation-based evaluation of importance estimators has been criticized (Hooker et al., 2019) since the perturbation of input pixels leads to a shift in the data distribution, violating the key assumption that training and test data stem from the same distribution.

Original: It is then unclear whether the observed degradation of model performance is due to this OOD problem or the removal of informative features.
Revised: It is then unclear whether the observed degradation of model performance is due to this OOD problem or the removal of informative features.

Original: Fong & Vedaldi (2017) presents concrete examples of how pixel perturbations can act as adversarial examples.
Revised: Fong & Vedaldi (2017) presents concrete examples of how pixel perturbations can act as adversarial examples.

Original: In fact, it has been argued (Samek et al., 2021) that certain importance estimators do very well in perturbation-based evaluations because they effectively construct adversarial examples and not because they faithfully describe the model.
Revised: In fact, it has been argued (Samek et al., 2021) that certain importance estimators do very well in perturbation-based evaluations because they effectively trigger perturbation artifacts and not because they faithfully describe the model.

Original: We propose to overcome this problem by augmenting the data during training using the same data perturbation that is used for the subsequent evaluation of importance estimators.
Revised: We propose to overcome this problem by augmenting the data during training using the same data perturbation that is used for the subsequent evaluation of importance estimators.

Original: Using this simple augmentation technique, FPA mitigates the risk of the OOD problem and one can be more confident that a perturbation-based evaluation of importance estimators actually quantifies the removal of information that is relevant for the model’s predictions.
Revised: Using this simple augmentation technique, FPA mitigates the risk of the OOD problem and one can be more confident that a perturbation-based evaluation of importance estimators actually quantifies the removal of information that is relevant for the model’s predictions.

Original: In FPA, mini-batches in training are selected for perturbation with a probability p .
Revised: In FPA, mini-batches in training are selected for perturbation with a probability p .

Original: Within a selected image, we iterate through input features; p 1 refers to the probability of masking a single pixel and p 2 refers to the probability of creating a non-informative square.
Revised: Within a selected image, we iterate through input features; p 1 refers to the probability of masking a single pixel and p 2 refers to the probability of creating a non-informative square.

Original: For each mini-batch, first draw pfrom Uniform (0 , p 1 max ) distribution and set input pixels to 0 with a probability of p 1 .
Revised: For each mini-batch, first draw pfrom Uniform (0 , p 1 max ) distribution and set input pixels to 0 with a probability of p 1 .

Original: Second, with a probability of p 2 for each selected pixel, we create a non-informative square of 0 ’s, with a randomly chosen side length in the interval [1 , s max ] .
Revised: Second, with a probability of p 2 for each selected pixel, we create a non-informative square of 0 ’s, with a randomly chosen side length in the interval [1 , s max ] .

Original: See details in the Algorithm 1.
Revised: See details in the Algorithm 1.

Revised: Algorithm 1 Feature Perturbation Augmentation in a Selected Mini-batch

Original: Require: K samples X kw,h,c for k = 0 , . . .
Revised: Require: K samples X kw,h,c for k = 0 , . . .

Original: , K , where X k is of dimension W × H × C .
Revised: , K , where X k is of dimension W × H × C .

Original: Require:
Revised: Require:

Original: s max < min( W, H ) , p 1 max ∈ (0 , 1) , p 2 ∈ (0 , 1)
Revised: s max < min( W, H ) , p 1 max ∈ (0 , 1) , p 2 ∈ (0 , 1)

Original: Set p 1 ∼ Uniform (0 , p 1 max ) for k ← 0 to K for w ← 0 to W for h ← 0 to H
Revised: Set p 1 ∼ Uniform (0 , p 1 max ) for k ← 0 to K for w ← 0 to W for h ← 0 to H

Original: With p 1 , X kw,h ← 0 (i.e., a non-informative value).
Revised: With p 1 , X kw,h ← 0 (i.e., a non-informative value).

Original: Set s ← { 1 , 2 , . . .
Revised: Set s ← { 1 , 2 , . . .

Original: , s max }
Revised: , s max }

Original: With p 2 , X k w :( w + s ) ,h :( h + s ) ← 0 (i.e., a s × s square of non-informative values).
Revised: With p 2 , X kw :( w + s ) ,h :( h + s ) ← 0 (i.e., a s × s square of non-informative values).

Original: Different schemes of masking pixels are possible, such as setting pixels to random, minimum, or maximum values or applying blurring, bokeh, or other filters.
Revised: Different schemes of masking pixels are possible, such as setting pixels to random, minimum, or maximum values or applying blurring, bokeh, or other filters.

Original: Choice of non-informative values should reflect the application domain and the evaluation method.
Revised: The choice of non-informative values should reflect the application domain and the evaluation method.

Original: D ATASETS AND I MPORTANCE E STIMATORS
Revised: D ATASETS AND I MPORTANCE E STIMATORS

Original: We demonstrate our approach using two popular deep learning architectures and three datasets, namely the ResNet-50 (He et al., 2016) architecture trained on ImageNet (Deng et al., 2009) and
Revised: We demonstrate our approach using two popular deep learning architectures and three datasets, namely the ResNet-50 (He et al., 2016) architecture trained on ImageNet (Deng et al., 2009) and

Original: Food101 (Bossard et al., 2014), and ResNet-18 trained on CIFAR-10 (Krizhevsky et al.).
Revised: Food101 (Bossard et al., 2014), and ResNet-18 trained on CIFAR-10 (Krizhevsky et al.)

Original: We adapted the ResNet-18 architecture to be suitable for the smaller input dimensions of CIFAR-10.

Original: The ResNet-50 model was trained on ImageNet for 90 epochs using the SGD optimizer with momentum 0 .

Original: 9 , weight decay 10 − 4 and initial learning rate 0.1, which we reduced by a factor ofon epochs 30 and 60.

Original: On Food101, Resnet-50 was trained for 68 epochs with momentum set to 0.9, weight decay 5 × 10 − 4 and an initial learning rate of 0.1, where a cosine annealing schedule was used to continuously reduce the learning rate to zero.

Original: The ResNet-18 model was trained forepochs using the SGD optimizer with momentum 0 .

Original: 9 , weight decay 5 × 10 − 4 and initial learning rate 0.01, which we reduced by a factor of 10 on epoch 30. For CIFAR-10 and ImageNet, we scaled input images to the range [ − 1 , 1] and for Food101 we performed a z-score normalization with mean and standard deviation from the  training set. In all three cases, we performed a horizontal flip with a probability of 0 . 5 to augment the data.
Revised: , see Appendix A.1 for details concerning the datasets and training procedure.

Original: We compare the following four importance estimators: Vanilla gradient (VG) (Simonyan et al., 2014): Gradients of the class score S c with respect to input pixels x i [Equation]
Revised: We compare the following four importance estimators: vanilla gradient (VG) (Simonyan et al., 2014), integrated gradients (IG) (Sundararajan et al., 2017), SmoothGrad (SG) (Smilkov et al., 2017) and squared SmoothGrad (SQSG) (Hooker et al., 2019).

Original: The class score S c is the activation of the neuron for the predicted class c .

Original: Integrated gradient (IG) (Sundararajan et al., 2017): Average over gradients obtained from inputs interpolated between a reference point x 0 and input x

Original: [Equation]

Original: where x 0 is chosen to be a black image and m = 200 .

Original: Smoothgrad (SG) (Smilkov et al., 2017): Average over gradients obtained from inputs with injected noise

Original: [Equation]

Original: where N (cid:0) 0 , σ 2 (cid:1) is Gaussian noise, ˆ e is obtained using vanilla gradient, and n = 15 .

Original: Squared SmoothGrad (SQ-SG) (Hooker et al., 2019): Variant of SmoothGrad that squares ˆ e

Original: [Equation]

Original: These methods output three-dimensional maps of importance scores (height, width, and color channels).
Revised: These methods output three-dimensional maps of importance scores (height, width, and color channels).

Original: To obtain two-dimensional maps for the pixel-wise perturbation, we explore two variants: unsigned and signed.
Revised: To obtain two-dimensional maps for the pixel-wise perturbation, we explore two variants: unsigned and signed.

Original: First, for the unsigned estimators, we sum the absolute values of color channels, which are denoted by the subscript abs .
Revised: First, for the unsigned estimators, we sum the absolute values of color channels, which are denoted by the subscript abs .

Original: In this case, very small values ( ≳ 0 ) have minimal influence on that prediction.
Revised: In this case, very small values ( ≳ 0 ) have minimal influence on that prediction.

Original: Second, for the signed estimators, we multiply the raw importance scores element-wise with the input image (Shrikumar et al., 2017), (indicated by a prime) and sum over the resulting color channels (indicated by a subscript sum ).
Revised: Second, for the signed estimators, we multiply the raw importance scores element-wise with the input image (Shrikumar et al., 2017), (indicated by a prime) and sum over the resulting color channels (indicated by a subscript sum ).

Original: Negative importance scores from the signed estimators may imply counter-evidence for the predicted class.
Revised: Negative importance scores from the signed estimators may imply counter-evidence for the predicted class.

Original: IG includes a multiplication with the input by definition and will therefore not appear primed.
Revised: IG includes a multiplication with the input by definition and will therefore not appear primed.

Revised: See Appendix A.2 for more details about the importance estimators.

Original: F IDELITY OF I MPORTANCE E STIMATORS
Revised: F IDELITY OF I MPORTANCE E STIMATORS

Original: [Figure] Perturbation-based evaluation methods are both intuitive and popular (Samek et al., 2016; Petsiuk et al., 2018; Kindermans et al., 2017).
Revised: Perturbation-based evaluation methods are both intuitive and popular (Samek et al., 2016; Petsiuk et al., 2018; Kindermans et al., 2017).

Original: For our evaluation of fidelity, we create perturbation curves of changes in logits, i.e. pre-softmax activations in the prediction vector, with respect to an increasing amount of perturbation (e.g., Fig. 1).
Revised: For our evaluation of fidelity, we create perturbation curves of changes in logits, i.e. pre-softmax activations in the prediction vector, with respect to an increasing amount of perturbation (e.g., Fig. 1).

Original: When needed for comparison, logits are normalized against the original model prediction without any masked pixel.
Revised: When needed for comparison, logits are normalized against the original model prediction without any masked pixel.

Original: A given importance estimator computes a set of importance scores for input pixels, which indicate how much each pixel contributed to the final prediction.
Revised: A given importance estimator computes a set of importance scores for input pixels, which indicate how much each pixel contributed to the final prediction.

Original: Then, input pixels are perturbed in order of either the most important first (MIF) or the least important first (LIF).
Revised: Then, input pixels are perturbed in order of either the most important first (MIF) or the least important first (LIF).

Original: In the case of signed importance scores, the ranking goes from the highest positive values to the lowest negative ones (MIF), or reversely (LIF).
Revised: In the case of signed importance scores, the ranking goes from the highest positive values to the lowest negative ones (MIF), or reversely (LIF).

Original: Therefore, the lowest negative importance scores, which are ranked first in LIF, may indicate strong counter-evidence for the predicted class. 
Revised: Therefore, the lowest negative importance scores, which are ranked first in LIF, may indicate strong counter-evidence for the predicted class. [Figure]

Original: When the importance estimator deems a certain feature important, ideally the removal of this feature would strongly decrease the associated logit.
Revised: When the importance estimator deems a certain feature important, ideally the removal of this feature would strongly decrease the associated logit.

Original: A greater logit decrease would imply that a chosen feature is more important for the model’s prediction.
Revised: A greater logit decrease would imply that a chosen feature is more important for the model’s prediction.

Original: Inversely, if a feature has a low-ranked importance score, its removal would lead to a minimal accuracy decrease (unsigned estimators) or potentially an accuracy increase for removing counter-evidence (signed estimators).
Revised: Inversely, if a feature has a low-ranked importance score, its removal would lead to a minimal accuracy decrease (unsigned estimators) or potentially an accuracy increase for removing counterevidence (signed estimators).

Original: To obtain the final perturbation curves, we average normalized logits over all 10,000 samples in the test set for CIFAR-10.
Revised: To obtain the final perturbation curves, we average normalized logits over all 10, samples in the test set for CIFAR-10.

Original: For ImageNet and
Revised: For ImageNet and

Original: Food101, we average over a randomly selected subset of 5,000 samples from the test set.
Revised: Food101, we average over a randomly selected subset of 5,000 samples from the test set.

Original: In order to combine these two aspects, we use the area A between the MIF and LIF curves (Fig. 1) as a metric to measure the relative fidelity of importance estimators.
Revised: In order to combine these two aspects, we use the area A between the MIF and LIF curves (Fig. 1) as a metric to measure the relative fidelity of importance estimators.

Original: A small area under the MIF curve indicates that the estimator is good at detecting features that are important evidence for a given class.
Revised: A small area under the MIF curve indicates that the estimator is good at detecting features that are important evidence for a given class.

Original: A large area under the LIF curve, on the other hand, means that the estimator can reliably find unimportant features.
Revised: A large area under the LIF curve, on the other hand, means that the estimator can reliably find unimportant features.

Original: Negative importance scores (e.g., gradients) would imply counter-evidence for the prediction.
Revised: Negative importance scores (e.g., gradients) would imply counter-evidence for the prediction.

Original: With A as fidelity metric, we therefore consider importance estimators with large A to be overall superior to ones with lower A .
Revised: With A as fidelity metric, we therefore consider importance estimators with large A to be overall superior to ones with lower A .

Original:  R ESULTS
Revised:  R ESULTS

Revised: means that importance scores are randomly assigned.

Original: We applied the proposed FPA on three datasets; the ImageNet (Deng et al., 2009), Food101 (Bossard et al., 2014), and CIFAR-10 (Krizhevsky et al.).
Revised: We applied the proposed FPA on three datasets; the ImageNet (Deng et al., 2009), Food101 (Bossard et al., 2014), and CIFAR-10 (Krizhevsky et al.).

Original: The ResNet50 (He et al., 2016) architecture is used for ImageNet and Food101 and ResNet-18 for CIFAR-10.
Revised: The ResNet-50 (He et al., 2016) architecture is used for ImageNet and Food101 and ResNet-18 for CIFAR-10.

Original: We selected parameter values for FPA from a partial grid search, keeping p 2 and s max fixed and varying p in the range [0 . 2 , 0 . 5] and p 1 max in [0 . 1 , 0 . 4] using 0 . 1 and 0 .
Revised: We wanted to find parameters for FPA that significantly improve the model’s robustness while maintaining its accuracy. To this end we performed a partial grid search, keeping p 2 and s max fixed and varying p in the range [0 . 2 , 0 . 5] and p 1 max in [0 . 1 , 0 . 4] using 0 . 1 and 0 .

Original: 5 steps for Food and CIFAR-10, respectively.
Revised: 5 steps for Food101 and CIFAR-10, respectively.

Original: Due to restrictions in computing resources, we did not include p 2 and s max in the grid search.
Revised: Due to restrictions in computing resources, we did not include p 2 and s max in the grid search.

Original: FPA parameters for the ImageNet were set to the same parameters selected for Food101.
Revised: FPA parameters for the ImageNet were set to the same parameters selected for Food101.

Original: For the augmentation of CIFAR-10, we chose p 1 max = 0 .
Revised: For the augmentation of CIFAR10, we chose p 1 max = 0 .

Original: 25 , p 2 = 0 .
Revised: 25 , p 2 = 0 .

Original: 1 , s max
Revised: 1 , s max

Original: = 3 and for ImageNet and Food101 p 1 max = 0 .
Revised: = 3 and for ImageNet and Food101 p 1 max = 0 .

Original: 3 , p 2 = 0 .
Revised: 3 , p 2 = 0 .

Original: 01 , s max = 10 .
Revised: 01 , s max = 10 .

Original: We set p = 0 .
Revised: We set p = 0 .

Original: 5 for all three datasets.
Revised: 5 for all three datasets.

Original: Evaluated on the same images that are used to obtain the perturbation curves, the models trained on CIFAR-10 have an accuracy of 93 . 0% , 92 .
Revised: Evaluated on the same images that are used to obtain the perturbation curves, the models trained on CIFAR-10 have an accuracy of 93 . 0% , 92 .

Original: 7% and 93 .
Revised: 7% and 93 .

Original: 1% for no augmentation, proposed FPA and “random erasing” (Zhong et al., 2020) (“Rectangle” in our figures), respectively.
Revised: 1% for no augmentation, proposed FPA and “random erasing” (Zhong et al., 2020) (“Rectangle” in our figures), respectively.

Original: In the same order, the models trained on ImageNet have an accuracy of 76 . 2% , 74 .
Revised: In the same order, the models trained on ImageNet have an accuracy of 76 . 2% , 74 .

Original: 7% and 75 .
Revised: 7% and 75 .

Original: 9% and on Food101 83 .
Revised: 9% and on Food101 83 .

Original: 5% and 83 .
Revised: 5% and 83 .

Original: Once the model is trained with or without data augmentation, vanilla gradient (VG), integrated gradient (IG),
Revised: Once the model is trained with or without data augmentation, vanilla gradient (VG), integrated gradient (IG),

Original: Smoothgrad (SG), and squared SmoothGrad (SQ-SG) are applied to obtain matrices of importance scores.
Revised: Smoothgrad (SG), and squared SmoothGrad (SQ-SG) are applied to obtain matrices of importance scores.

Original: We also calculate perturbation curves as described in Section 3.
Revised: We also calculate perturbation curves as described in Section 3.

Original: In Fig.
Revised: In Fig.

Original: 2, the MIF perturbation curves (top row) fall off slower when the model was trained with the proposed FPA, compared to those with no augmentation or Rectangle augmentation (Zhong et al., 2020).
Revised: 2, the MIF perturbation curves (top row) fall off slower when the model was trained with the proposed FPA, compared to those with no augmentation or Rectangle augmentation (Zhong et al., 2020).

Original: For LIF perturbation curves (bottom row, Fig. 2), the logits initially increase before decreasing if FPA is used.
Revised: For LIF perturbation curves (bottom row, Fig. 2), the logits initially increase before decreasing if FPA is used.

Original: In contrast, with None or  Rectangle augmentation, the logits immediately and rapidly decrease.
Revised: In contrast, without or with Rectangle augmentation, the logits immediately and rapidly decrease.

Original: The random baseline (where importance scores are randomly assigned to pixels) does not exhibit the early increase of logits.
Revised: The random baseline (where importance scores are randomly assigned to pixels) does not exhibit the early increase of logits.

Original: These operating characteristics are expected when the influence of artifacts is removed, or at least strongly reduced, by our proposed augmentation.
Revised: These operating characteristics are expected when the influence of artifacts is removed, or at least strongly reduced, by our proposed augmentation.

Original: Generally, some of the logit decrease is expected to be due to perturbation artifacts, thus removing perturbation artifacts would delay the logit decrease.
Revised: Generally, some of the logit decrease is expected to be due to perturbation artifacts, thus removing perturbation artifacts would delay the logit decrease.

Original: In the LIF curves in Fig. 2, pixels with large negative importance scores are masked first which removes counter-evidence .
Revised: In the LIF curves in Fig. 2, pixels with large negative importance scores are masked first which removes counter-evidence .

Original: This is highlighted in Fig.
Revised: This is highlighted in Fig.

Original: 3, which demonstrates that masking pixels with a large negative importance score coincides with an increase in the logit values (see Fig.
Revised: 3, which demonstrates that masking pixels with a large negative importance score coincides with an increase in the logit values (see Fig.

Original: A. for equivalent graphs for CIFAR-10 and ImageNet).
Revised: A.4 for equivalent graphs for CIFAR-10 and ImageNet).

Original: Without FPA this effect can not be observed, instead leading to a net decrease of the logits, despite pixels with negative scores being masked.
Revised: Without FPA this effect can not be observed, instead leading to a net decrease of the logits, despite pixels with negative scores being masked.

Original: This behavior is consistent across the three considered datasets and architectures (see Figs.
Revised: This behavior is consistent across the three considered datasets and architectures (see Figs.

Original: A.2 and A.3).
Revised: A.2 and A.3).

Original: We compare the fidelity A of importance estimators (Tables 1, 2 and 3) considering two different settings.
Revised: We compare the fidelity A of importance estimators (Tables 1, A.1 and A.2) considering two different settings.

Original: In the first setting, we take into account the magnitude of importance scores and disregard whether their signs correctly indicate evidence or counter-evidence for the prediction.
Revised: In the first setting, we take into account the magnitude of importance scores and disregard whether their signs correctly indicate evidence or counter-evidence for the prediction.

Original: Non-negative importance scores are obtained from the unsigned estimators are: IG abs , VG abs , VG ′ abs , SG abs , SG ′ abs and SQ-SG sum .
Revised: Nonnegative importance scores are obtained from the unsigned estimators are: IG abs , VG abs , VG ′ abs , SG abs , SG ′ abs and SQ-SG sum .

Original: Among those that rank pixels purely by the magnitude of importance scores, we find that across all three considered datasets, and regardless of the augmentation, SQ-SG sum consistently performs best and VG abs worst.
Revised: Among those that rank pixels purely by the magnitude of importance scores, we find that across all three considered datasets, and regardless of the augmentation, SQ-SG sum consistently performs best and VG abs worst.

Original: For the second setting, we consider signed importance scores from the signed estimators ; namely,
Revised: For the second setting, we consider signed importance scores from the signed estimators ; namely,

Original: IG sum , VG ′ sum and SG ′ sum , which multiply the gradients and input element-wise 2 .
Revised: IG sum , VG ′ sum and SG ′ sum , which multiply the gradients and input element-wise 1 .

Original: With FPA, the signed estimators consistently outperform the unsigned ones across all three datasets, with IG sum and SG ′ sum in the leading positions.
Revised: With FPA, the signed estimators consistently outperform the unsigned ones across all three datasets, with IG sum and SG ′ sum in the leading positions.

Original: Intuitively, this makes sense since the signed estimators contain more information than the unsigned ones, allowing them to describe the models’ predictions more accurately.
Revised: Intuitively, this makes sense since the signed estimators contain more information than the unsigned ones, allowing them to describe the models’ predictions more accurately.

Original: Without FPA, however, the unsigned SQ-SG sum outperforms all other methods on
Revised: Without FPA, however, the unsigned SQ-SG sum outperforms all other methods on

Original: Food101 and ImageNet and only on CIFAR-10 SG ′ sum performs best.
Revised: Food101 and ImageNet and only on CIFAR-10 SG ′ sum performs best.

Original: This indicates that using FPA makes the obtained ranking of importance estimators more reliable.
Revised: This indicates that using FPA makes the obtained ranking of importance estimators more reliable.

Original: [Table]
Revised: [Table]

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: As this work has been focused on the interpretability and trustworthiness of the model, there still is room for performance improvement.
Revised: As this work has been focused on the interpretability and trustworthiness of deep learning models [Table] there still is room for performance improvement.

Original: Training the models with FPA leads to a slight decrease in performance compared to training them without augmentation.
Revised: Training the models with FPA leads to a slight decrease in performance compared to training them without augmentation.

Original: Generally, there exists a trade-off between interpretability and accuracy (Dziugaite et al., 2020).
Revised: This might be due to “augment ambiguity” Wei et al. (2020), which occurs when the annotated class is not recognizable anymore as a result of an augmentation.

Original: However, random erasing, which is closely related to FPA, reported an increase in the model performance in certain settings (Zhong et al., 2020).
Revised: On the other hand, random erasing, which is closely related to FPA, reported an increase in the model performance in certain settings (Zhong et al., 2020).

Original: In future work, we plan to explore variations of FPA and training schemes.
Revised: In future work [Table] we plan to explore variations of FPA and training schemes to mitigate the performance loss, but also point out that trading some accuracy for increased interpretability can be worthwhile.

Original: Although FPA has been introduced here for the most popular perturbation-based evaluation method, it can be applied to other perturbation methods.
Revised: Although FPA has been introduced here for the case of perturbing pixels with constant values it is applicable to any perturbation scheme, e.g. blurring.

Original: The  perturbation performed to augment images needs to reflect the perturbation applied during evaluation.
Revised: The only requirement is that the perturbation performed to augment images needs to reflect the perturbation applied during evaluation of interpretability methods.

Revised: Since FPA randomly selects pixels to construct augmented training samples it can be expected to only increase a model’s robustness against such random perturbations.

Revised: To increase the robustness against adversarial perturbations, we plan to extend our work by additionally performing adversarial training Goodfellow et al.

Revised: Madry et al., which would allow to rule out perturbation artifacts during evaluation with even higher confidence.

Original: Lastly, training with FPA elucidate counterevidence associated with negative importance scores.
Revised: Lastly, training with FPA elucidates counter-evidence associated with negative importance scores.

Original: The absolute values  are often taken in practice.
Revised: Often only the absolute values of importance scores are considered in practice.

Original: In contrast, if perturbation artifacts are accounted for, the sign of importance scores can help us understand the model characteristics.
Revised: In contrast, if perturbation artifacts are accounted for, the sign of importance scores can help us understand the model characteristics.

Original: A CKNOWLEDGMENTS
Revised: A CKNOWLEDGMENTS

Original: This work was funded by the ERA-Net CHIST-ERA grant [CHIST-ERA-19-XAI-007] long term challenges in ICT project INFORM (ID: 93603), by the National Science Centre (NCN) of Poland [2020/02/Y/ST6/00071].
Revised: This work was funded by the ERA-Net CHIST-ERA grant [CHIST-ERA-19-XAI-007] long term challenges in ICT project INFORM (ID: 93603), by the National Science Centre (NCN) of Poland [2020/02/Y/ST6/00071].

Original: This research was carried out with the support of the Interdisciplinary
Revised: This research was carried out with the support of the Interdisciplinary

Original: Centre for Mathematical and Computational Modelling University of Warsaw (ICM UW) under computational allocation
Revised: Centre for Mathematical and Computational Modelling University of Warsaw (ICM UW) under computational allocation

Original: no GDM-3540; the NVIDIA Corporation’s GPU grant; and the Google
Revised: no GDM-3540; the NVIDIA Corporation’s GPU grant; and the Google

Original: Cloud Research Innovators program.
Revised: Cloud Research Innovators program.

Revised: A A PPENDIX A. D ATASETS AND T RAINING PROCEDURE

Revised: We adapted the ResNet-18 architecture to be suitable for the smaller input dimensions of CIFAR10.

Revised: The ResNet-50 model was trained on ImageNet for 90 epochs using the SGD optimizer with momentum 0 .

Revised: 9 , weight decay 10 − 4 and initial learning rate 0.1, which we reduced by a factor of 10 on epochs 30 and 60.

Revised: On Food101, Resnet-50 was trained for 68 epochs with momentum set to 0.9, weight decay 5 × 10 − 4 and an initial learning rate of 0.1, where a cosine annealing schedule was used to continuously reduce the learning rate to zero.

Revised: The ResNet-18 model was trained forepochs using the SGD optimizer with momentum 0 .

Revised: 9 , weight decay 5 × 10 − 4 and initial learning rate 0.01, which we reduced by a factor of 10 on epoch 30.

Revised: For CIFAR-10 and ImageNet, we scaled input images to the range [ − 1 , 1] and for Food101 we performed a z-score normalization with mean and standard deviation from the training set.

Revised: In all three cases, we performed a horizontal flip with a probability of 0 .

Revised: 5 to augment the data.

Revised: A. I MPORTANCE E STIMATORS

Revised: We compare the following four importance estimators:

Revised: Vanilla gradient (VG) (Simonyan et al., 2014):

Revised: Gradients of the class score S c with respect to input pixels x i

Revised: [Equation]

Revised: The class score S c is the activation of the neuron for the predicted class c .

Revised: Integrated gradient (IG) (Sundararajan et al., 2017): Average over gradients obtained from inputs interpolated between a reference point x 0 and input x

Revised: [Equation]

Revised: where x 0 is chosen to be a black image and m = 200 .

Revised: SmoothGrad

Revised: [Equation]

Revised: SG)

Revised: [Equation]

Revised: Smilkov

Revised: [Equation]

Revised: 2017): Average over gradients obtained from inputs with injected noise

Revised: [Equation]

Revised: where N (cid:0) 0 , σ 2 (cid:1) is Gaussian noise, ˆ e is obtained using vanilla gradient, and n = 15 .

Revised: Squared

Revised: [Equation]

Revised: : Variant of SmoothGrad that squares ˆ e

Revised: [Equation]

Revised: A. S UPPLEMENTARY F IGURES AND T ABLES

Revised: [Table]

Revised: [Table]