Original: O N E VALUATING E XPLAINABILITY A LGORITHMS
Revised: O N E VALUATING E XPLAINABILITY A LGORITHMS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: A plethora of methods attempting to explain predictions of black-box models have been proposed by the Explainable Artiﬁcial Intelligence (XAI) community.
Revised: A plethora of methods attempting to explain predictions of black-box models have been proposed by the Explainable Artiﬁcial Intelligence (XAI) community.

Original: Yet, measuring the quality of the generated explanations is largely unexplored, making quantitative comparisons non-trivial.
Revised: Yet, measuring the quality of the generated explanations is largely unexplored, making quantitative comparisons non-trivial.

Original: In this work, we propose a suite of multifaceted metrics that enables us to objectively compare explainers based on the correctness, consistency, as well as the conﬁdence of the generated explanations.
Revised: In this work, we propose a suite of multifaceted metrics that enables us to objectively compare explainers based on the correctness, consistency, as well as the conﬁdence of the generated explanations.

Original: These metrics are computationally inexpensive, do not require model-retraining and can be used across different data modalities.
Revised: These metrics are computationally inexpensive, do not require model-retraining and can be used across different data modalities.

Original: We evaluate them on common explainers such as Grad-CAM, SmoothGrad, LIME and Integrated Gradients.
Revised: We evaluate them on common explainers such as Grad-CAM, SmoothGrad, LIME and Integrated Gradients.

Original: Our experiments show that the proposed metrics reﬂect qualitative observations reported in earlier works.
Revised: Our experiments show that the proposed metrics reﬂect qualitative observations reported in earlier works.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Over the past few years, deep learning has made signiﬁcant progress, outperforming the state-ofthe-art in many tasks like image classiﬁcation (Mahajan et al., 2018), semantic segmentation (Zhu et al., 2018), machine translation (Kalchbrenner et al., 2016) and even surpassing humans in the games of Chess and Go (Silver et al., 2016).
Revised: Over the past few years, deep learning has made signiﬁcant progress, outperforming the state-ofthe-art in many tasks like image classiﬁcation (Mahajan et al., 2018), semantic segmentation (Zhu et al., 2018), machine translation (Kalchbrenner et al., 2016) and even surpassing humans in the games of Chess and Go (Silver et al., 2016).

Original: As these models are deployed in more mission-critical systems, we notice that despite their incredible performance on standard metrics, they are fragile (Szegedy et al., 2013; Goodfellow et al., 2014) and can be easily fooled by small perturbations to the inputs (Engstrom et al., 2017).
Revised: As these models are deployed in more mission-critical systems, we notice that despite their incredible performance on standard metrics, they are fragile (Szegedy et al., 2013; Goodfellow et al., 2014) and can be easily fooled by small perturbations to the inputs (Engstrom et al., 2017).

Original: Further research has also exposed that these models are biased in undesirable ways exacerbating gender and racial biases (Howard et al., 2017; Escud ´ e Font & Costa-Juss ` a, 2019).
Revised: Further research has also exposed that these models are biased in undesirable ways exacerbating gender and racial biases (Howard et al., 2017; Escud ´ e Font & Costa-Juss ` a, 2019).

Original: These issues have ampliﬁed the need for making these black-box models interpretable.
Revised: These issues have ampliﬁed the need for making these black-box models interpretable.

Original: Consequently, the XAI community has proposed a variety of algorithms that aim to explain predictions of these models (Ribeiro et al., 2016; 2018; Lundberg & Lee, 2017; Shrikumar et al., 2017; Smilkov et al., 2017; Selvaraju et al., 2016; Sundararajan et al., 2017).
Revised: Consequently, the XAI community has proposed a variety of algorithms that aim to explain predictions of these models (Ribeiro et al., 2016; 2018; Lundberg & Lee, 2017; Shrikumar et al., 2017; Smilkov et al., 2017; Selvaraju et al., 2016; Sundararajan et al., 2017).

Original: With such an explosion of interpretability methods (hereon referred to as explainers), evaluating them has become non-trivial.
Revised: With such an explosion of interpretability methods (hereon referred to as explainers), evaluating them has become non-trivial.

Original: This is due to the lack of a widely accepted metric to quantitatively compare them.
Revised: This is due to the lack of a widely accepted metric to quantitatively compare them.

Original: There have been several attempts to propose such metrics.
Revised: There have been several attempts to propose such metrics.

Original: Unfortunately, they tend to suffer from major drawbacks like computational cost (Hooker et al., 2018), inability to be extended to non-image domains (Kindermans et al., 2017a), or simply focusing on one desirable attribute of a good explainer.
Revised: Unfortunately, they tend to suffer from major drawbacks like computational cost (Hooker et al., 2018), inability to be extended to non-image domains (Kindermans et al., 2017a), or simply focusing only one desirable attribute of a good explainer.

Original: (Yeh et al., 2019).
Revised: (Yeh et al., 2019).

Original: In this paper, we propose a suite of metrics that attempt to alleviate these drawbacks and can be applied across multiple data modalities.
Revised: In this paper, we propose a suite of metrics that attempt to alleviate these drawbacks and can be applied across multiple data modalities.

Original: Unlike the vast majority of prior work, we not only consider the correctness of an explainer, but also the consistency and conﬁdence of the generated explanations.
Revised: Unlike the vast majority of prior work, we not only consider the correctness of an explainer, but also the consistency and conﬁdence of the generated explanations.

Original: We use these metrics to evaluate and compare widely used explainers such as LIME (Ribeiro et al., 2016), Grad-CAM (Selvaraju et al., 2016), SmoothGrad (Smilkov et al., 2017) and Integrated
Revised: We use these metrics to evaluate and compare widely used explainers such as LIME (Ribeiro et al., 2016), Grad-CAM (Selvaraju et al., 2016), SmoothGrad (Smilkov et al., 2017) and Integrated

Original: Gradients (Sundararajan et al., 2017) on an Inception-V3 (Szegedy et al., 2015) model pretrained on the ImageNet dataset  (Deng et al., 2009), in an objective manner (i.e., without the need of a human-in-the-loop).
Revised: Gradients (Sundararajan et al., 2017) on an Inception-V3 (Szegedy et al., 2015) model pretrained on the ImageNet dataset (ILSVRC2012) (Deng et al., 2009), in an objective manner (i.e., without the need of a human-in-the-loop).

Original: Moreover, our proposed metrics are general and computationally inexpensive.
Revised: Moreover, our proposed metrics are general and computationally inexpensive.

Original: Our main contributions are:
Revised: Our main contributions are:

Original: Identifying and formulating the properties of a good explainer.
Revised: Identifying and formulating the properties of a good explainer.

Original: Proposing a generic, computationally inexpensive suite of metrics to evaluate explainers.
Revised: Proposing a generic, computationally inexpensive suite of metrics to evaluate explainers.

Original: Comparing common explainers and discussing pros and cons of each.
Revised: Comparing common explainers and discussing pros and cons of each.

Original: We ﬁnd that while Grad-CAM seems to perform best overall, it does suffer from drawbacks when inverse masking is considered compared to Integrated Gradients and Smoothgrad.
Revised: We ﬁnd that while Grad-CAM seems to perform best overall, it does suffer from drawbacks not reported in prior works.

Original: On the other hand, LIME consistently underperforms in comparison to the other models.
Revised: On the other hand, LIME consistently underperforms in comparison to the other models.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: The ﬁeld of XAI has become an active area of research (Doshi-Velez & Kim, 2017; Lipton, 2016) with signiﬁcant efforts being made to explain AI models, either by generating local (Ribeiro et al., 2016;
Revised: The ﬁeld of XAI has become an active area of research (Doshi-Velez & Kim, 2017; Lipton, 2016) with signiﬁcant efforts being made to explain AI models, either by generating local (Ribeiro et al., 2016;

Original: Shrikumar et al., 2017; Sundararajan et al., 2017; Selvaraju et al., 2016; Smilkov et al., 2017) or global (Lundberg & Lee, 2017; Ribeiro et al., 2018) explanations.
Revised: Shrikumar et al., 2017; Sundararajan et al., 2017; Selvaraju et al., 2016; Smilkov et al., 2017) or global (Lundberg & Lee, 2017; Ribeiro et al., 2018) explanations.

Original: Simultaneously, there are growing research efforts into methods to formally evaluate and compare explainers (Mohseni et al., 2018;
Revised: Simultaneously, there are growing research efforts into methods to formally evaluate and compare explainers (Mohseni et al., 2018;

Original: Gunning, 2019; Wolf, 2019; Gilpin et al., 2019).
Revised: Gunning, 2019; Wolf, 2019; Gilpin et al., 2019).

Original: Notably, Murdoch et al.
Revised: Notably, Murdoch et al.

Original: (2019) introduced a framework with three desiderata for evaluation, viz.
Revised: (2019) introduced a framework with three desiderata for evaluation, viz.

Original: predictive accuracy, descriptive accuracy and relevancy, with relevancy judged relative to a human.
Revised: predictive accuracy, descriptive accuracy and relevancy, with relevancy judged relative to a human.

Original: In contrast, Hall et al.
Revised: In contrast, Hall et al.

Original: (2019) compiled a set of desired characteristics around effectiveness, versatility, constraints (i.e., privacy, computation cost, information collection effort) and the type of generated explanations, which do not need human evaluation, and therefore are objective.
Revised: (2019) compiled a set of desired characteristics around effectiveness, versatility, constraints (i.e., privacy, computation cost, information collection effort) and the type of generated explanations, which do not need human evaluation, and therefore are objective.

Original: However, they focus very little on aspects such as correctness.
Revised: However, they focus very little on aspects such as correctness.

Original: Recently, DeConvNet (Noh et al., 2015), Guided BackProp (Springenberg et al., 2015) and LRP (Bach et al., 2015) have been shown to not produce theoretically correct explanations of linear models (Kindermans et al., 2017b).
Revised: Recently, DeConvNet (Noh et al., 2015), Guided BackProp (Springenberg et al., 2015) and LRP (Bach et al., 2015) have been shown to not produce theoretically correct explanations of linear models (Kindermans et al., 2017b).

Original: As a result, two explanation techniques, PatternNet and PatternAttribution, that are theoretically sound for linear models were proposed.
Revised: As a result, two explanation techniques, PatternNet and PatternAttribution, that are theoretically sound for linear models were proposed.

Original: Other efforts focus on evaluating saliency methods (Kindermans et al., 2017a; Adebayo et al., 2018) and show that they are unreliable for tasks that are sensitive to either data or model.
Revised: Other efforts focus on evaluating saliency methods (Kindermans et al., 2017a; Adebayo et al., 2018) and show that they are unreliable for tasks that are sensitive to either data or model.

Original: Samek et al.
Revised: Samek et al.

Original: (2017) and its variations (Hooker et al., 2018; Fong & Vedaldi, 2017; Ancona et al., 2018) infer whether a feature attribution is correct by measuring performance degradation when highly attributed features are removed.
Revised: (2017) and its variations (Hooker et al., 2018; Fong & Vedaldi, 2017; Ancona et al., 2018) infer whether a feature attribution is correct by measuring performance degradation when highly attributed features are removed.

Original: For instance, Hooker et al.
Revised: For instance, Hooker et al.

Original: (2018) shows that commonly used interpretability methods are less accurate or are on-par with a random designation of feature importance, whereas ensemble approaches such as SmoothGrad (Smilkov et al., 2017) are superior.
Revised: (2018) shows that commonly used interpretability methods are less accurate or are on-par with a random designation of feature importance, whereas ensemble approaches such as SmoothGrad (Smilkov et al., 2017) are superior.

Original: Yang & Kim (2019) proposed three complementary metrics to evaluate explainers: model contrast score – comparing two models trained to consider opposite concepts as important, input dependence score – comparing one model with two inputs of different concepts, and input dependence rate – comparing one model with two functionally identical inputs.
Revised: Yang & Kim (2019) proposed three complementary metrics to evaluate explainers: model contrast score – comparing two models trained to consider opposite concepts as important, input dependence score – comparing one model with two inputs of different concepts, and input dependence rate – comparing one model with two functionally identical inputs.

Original: These metrics aim to speciﬁcally cover aspects of false-positives.
Revised: These metrics aim to speciﬁcally cover aspects of false-positives.

Original: Alvarez-Melis & Jaakkola (2018) deﬁne an alternative set of metrics, around explicitness – intelligibility of explanations, faithfulness – feature relevance, and stability – consistency of explanations for similar or neighboring samples.
Revised: Alvarez-Melis & Jaakkola (2018) deﬁne an alternative set of metrics, around explicitness – intelligibility of explanations, faithfulness – feature relevance, and stability – consistency of explanations for similar or neighboring samples.

Original: Finally, Yeh et al.
Revised: Finally, Yeh et al.

Original: (2019) deﬁne and evaluate ﬁdelity of explanations, namely quantifying the degree to which an explanation captures how the underlying model itself changes in response to signiﬁcant perturbations.
Revised: (2019) deﬁne and evaluate ﬁdelity of explanations, namely quantifying the degree to which an explanation captures how the underlying model itself changes in response to signiﬁcant perturbations.

Original: Similar to previous work, we focus on objective metrics to evaluate and compare explainers.
Revised: Similar to previous work, we focus on objective metrics to evaluate and compare explainers.

Original: However, we not only consider correctness, but also consistency and conﬁdence (as deﬁned next).
Revised: However, we not only consider correctness, but also consistency and conﬁdence (as deﬁned next).

Original:  W HAT M AKES AN E XPLAINER G OOD ? 3. P RELIMINARIES
Revised:  W HAT M AKES AN E XPLAINER G OOD ? 3. P RELIMINARIES

Original: In the following discussions, let x ∈ R n be an arbitrary data point and y be the corresponding ground truth label from the dataset D
Revised: In the following discussions, let x ∈ R n be an arbitrary data point and y be the corresponding ground truth label from the dataset D

Original: = { ( x i , y i ) , 1 ≤ i ≤ M } .
Revised: = { ( x i , y i ) , 1 ≤ i ≤ M } .

Original: Let f be the classiﬁer realized as a neural network parameterized by weights θ .
Revised: Let f be the classiﬁer realized as a neural network parameterized by weights θ .

Original: Let T be the set of transformations under which semantic information in the input remains unchanged.
Revised: Let T be the set of transformations under which semantic information in the input remains unchanged.

Original: If t is an arbitrary transform from this set, let t − 1 be its inverse transform.
Revised: If t is an arbitrary transform from this set, let t − 1 be its inverse transform.

Original: For example, if t = Rot − 90 ◦ , then t − 1 =
Revised: For example, if t = Rot − 90 ◦ , then t − 1 =

Original: Rot + 90 ◦ .
Revised: Rot + 90 ◦ .

Original: In general, t − 1 ( t ( x )) =
Revised: In general, t − 1 ( t ( x )) =

Original: x, ∀ x ∈ D .
Revised: x , ∀ x ∈ D .

Original: Let E f be any explainer that generates explanations for predictions made by classiﬁer f 1 .
Revised: Let E f be any explainer that generates explanations for predictions made by classiﬁer f 1 .

Original: Finally, let d ( , ) be a function that computes the difference between the generated explanations 2 .
Revised: Finally, let d ( , ) be a function that computes the difference between the generated explanations 2 .

Original: For example, if the explainer E generates saliency maps (e.g. GradCAM and SmoothGrad), d could be a simple (cid:96) p norm.
Revised: For example, if the explainer E generates saliency maps (e.g. GradCAM and SmoothGrad), d could be a simple (cid:96) p norm.

Original: Additionally, in order to ensure that we minimize the impact that pathologies of the underlying classiﬁer have on the properties we are interested in, we assume that the classiﬁer has acceptable test-set performance.
Revised: Additionally, in order to ensure that we minimize the impact that pathologies of the underlying classiﬁer have on the properties we are interested in, we assume that the classiﬁer has acceptable test-set performance.

Original: Furthermore, we also assume that the classiﬁer performance does not degrade significantly under the transforms (described in Sec. 3.2.2) we perform.
Revised: Furthermore, we also assume that the classiﬁer performance does not degrade signiﬁcantly under the transformations we perform (described in Sec. 3.2.2).

Original: If the classiﬁer does not satisfy these conditions, it is prudent to improve its performance to acceptable levels prior to attempting to explain its outputs.
Revised: If the classiﬁer does not satisfy these conditions, it is prudent to improve its performance to acceptable levels prior to attempting to explain its outputs.

Original: One cannot extract reliable explanations from underperforming underlying models (Ghorbani et al., 2019; Samek et al., 2017).
Revised: One cannot extract reliable explanations from underperforming underlying models (Ghorbani et al., 2019; Samek et al., 2017).

Original: 3. D ESIDERATA FOR A G OOD E XPLAINER
Revised: 3. D ESIDERATA FOR A G OOD E XPLAINER

Original: Inspired by earlier works on important aspects of an explainer’s quality (Yang & Kim, 2019;
Revised: Inspired by earlier works on important aspects of an explainer’s quality (Yang & Kim, 2019;

Original: Alvarez-Melis & Jaakkola, 2018;
Revised: Alvarez-Melis & Jaakkola, 2018;

Original: Yeh et al., 2019), our proposed evaluation framework consists of the following components:
Revised: Yeh et al., 2019), our proposed evaluation framework consists of the following components:

Original: • Correctness • Consistency • Conﬁdence
Revised: • Correctness • Consistency • Conﬁdence

Original: We elaborate on these components as well as methods to compute them in the image classiﬁcation scenario.
Revised: We elaborate on these components as well as methods to compute them in the image classiﬁcation scenario.

Original: Even though these are evaluated independently, they can be combined together to give a single scalar value to compare explainers in a straightforward way.
Revised: Even though these are evaluated independently, they can be combined together to give a single scalar value to compare explainers in a straightforward way.

Original: However, the weight for each component depends heavily on the use case and end-user preference.
Revised: However, the weight for each component depends heavily on the use case and end-user preference.

Original: This is beyond the scope of the current work and thus is not discussed further.
Revised: This is beyond the scope of the current work and thus is not discussed further.

Original: Further, since we elaborate on the image classiﬁcation scenario, we use inputs and images interchangeably with the understanding that the described methods or equivalents can be trivially applied in other modalities.
Revised: Further, since we elaborate on the image classiﬁcation scenario, we use inputs and images interchangeably with the understanding that the described methods or equivalents can be trivially adapted in other modalities.

Original: 3.2. C ORRECTNESS
Revised: 3.2. C ORRECTNESS

Original: Correctness (sensitivity or ﬁdelity in literature) refers to the ability of an explainer to correctly identify components of the input that contribute most to the prediction of the classiﬁer.
Revised: Correctness (sensitivity or ﬁdelity in literature) refers to the ability of an explainer to correctly identify components of the input that contribute most to the prediction of the classiﬁer.

Original: Most metrics proposed so far focus solely on correctness and attempt to compute it in different ways, often requiring retraining of the underlying classiﬁer.
Revised: Most metrics proposed so far focus solely on correctness and attempt to compute it in different ways, often requiring retraining of the underlying classiﬁer.

Original: Moreover, they do not capture all aspects of correctness nor do they generalize to other data modalities.
Revised: Moreover, they do not capture all aspects of correctness nor do they generalize to other data modalities.

Original: We propose a novel computationally-inexpensive method that addresses these drawbacks.
Revised: We propose a novel computationally-inexpensive method that addresses these drawbacks.

Original: It takes into consideration both that the explainer identiﬁes most of the relevant components(high recall) and does not incorrectly select non-important components as important (high precision).
Revised: It takes into consideration both that the explainer identiﬁes most of the relevant components and does not incorrectly select non-important components as important.

Original: If the explainers are performing as expected, a simple masking of the input image with the associated explanation should provide better accuracy as the network is unlikely to be confused by the nonimportant pixels.
Revised: If the explainers are performing as expected, a simple masking of the input image with the associated explanation should provide better accuracy as the network is unlikely to be confused by the nonimportant pixels.

Original: However, we do not observe this in practice, as we show empirically that vanilla masking results in severe performance deterioration (see Table 7 for results).
Revised: However, we do not observe this in practice, as we show empirically that vanilla masking results in severe performance deterioration (see Table 9 and 10 for results).

Original: We hypothesize that this is because of the following reasons:
Revised: We hypothesize that this is because of the following reasons:

Original: • The masked image has a large proportion of empty pixels 3 and thus does not belong to the data distribution ( p data ) • Extracted pixels are important in the context of the background pixels, and as such removing context makes the masking meaningless.
Revised: • The masked image has a large proportion of empty pixels 3 and thus does not belong to the data distribution ( p data ) • Extracted pixels are important in the context of the background pixels, and as such removing context makes the masking meaningless.

Revised: Additionally, Convolutions have the inductive bias that the neighbouring pixels are highly correlated that helps perform well on visual tasks (LeCun et al., 1998).

Revised: Simple masking breaks this correlation.

Original: Based on the above observations, we conclude that it is crucial to have a realistic background for the extracted patches to properly evaluate them.
Revised: Based on the above observations, we conclude that it is crucial to have a realistic background for the extracted patches to properly evaluate them.

Original: We propose the following procedure to provide a background such that the resulting image is inside the data distribution.
Revised: We propose the following procedure to provide a background such that the resulting image is closer the data distribution

Original: For each class in the dataset, we select the top k and bottom k images, sorted  based on the probability assigned by the classiﬁer to the ground-truth class.
Revised: For each class in the dataset, we select the top k and bottom k images, sorted in decreasing order based on the probability assigned by the classiﬁer to the ground-truth class.

Original: We then randomly pair each of the top images with one of the bottom images.
Revised: We then randomly pair each of the top images with one of the bottom images.

Original: For each pair, we extract important regions identiﬁed by the explainer from the top image  and overlap them over the corresponding bottom image.
Revised: For each pair, we extract important regions identiﬁed by the explainer from the top image (i.e high conﬁdence images) and overlap them over the corresponding bottom image (i.e low conﬁdence images).

Original: We use the bottom k images for this task as we know that they are uninformative for the classiﬁer as evidenced by the assigned probability.
Revised: We use the bottom k images for this task as we know that they are uninformative for the classiﬁer as evidenced by the assigned probability.

Original: We thus obtain a dataset of masked images with important regions from the most important images along with relevant yet non-informative backgrounds for each class (see Fig.
Revised: We thus obtain a dataset of masked images with important regions from the most important images along with relevant yet non-informative backgrounds for each class (see Fig.

Original: 1 for an example).
Revised: 1 for an example).

Revised: Formally, the masking operation can be represented as:

Revised: [Equation]

Revised: Where M is the new masked image, a threshold function, H the high conﬁdence image, L the low conﬁdence image and ⊗ theelement − wisemultiplicationoperator.

Original: We then measure the accuracy on this masked dataset and compare it with the accuracy on the bottom k images subset.
Revised: We then measure the accuracy on this masked dataset and compare it with the accuracy on the bottom k images subset.

Original: Note that the above mentioned algorithm only evaluates if the explainer is capturing important pixels (recall).
Revised: Note that the above mentioned process only evaluates if the explainer is capturing important pixels.

Original: In order to verify that the explainer does not select non-important pixels (precision), we repeat the same process but instead use the inverted saliency map 4 and recompute accuracy on this dataset.
Revised: In order to verify that the explainer does not select non-important pixels, we repeat the same process but instead use the inverted saliency map 4 and recompute accuracy on this dataset.

Original: In this scenario, we expect the accuracy to deteriorate.
Revised: In this scenario, we expect the accuracy to deteriorate.

Revised: TO DISCUSS

Revised: Formally, the inverse masking process can be deﬁned as follow:

Original:  Interestingly, these masked accuracies are similar to the precision and recall metrics used in information retrieval (Manning et al., 2009).
Revised: [Equation] Interestingly, these masked accuracies are similar to the precision and recall metrics used in information retrieval (Manning et al., 2009).

Original: This provides motivation to combine these differences into a Pseudo-F1 score by computing the harmonic mean of accuracy on normal masked images and 1accuracy on inverse masked images. 
Revised: This provides motivation to combine these differences into a Pseudo-F1 score by computing the harmonic mean of accuracy on normal masked images and 1accuracy on inverse masked images. Formally this can be computed as: [Equation]

Original: 3.2. C ONSISTENCY
Revised: 3.2. C ONSISTENCY

Original: We deﬁne consistency as the ability of the explainer to capture the  relevant components under various transformations to the input.
Revised: We deﬁne consistency as the ability of the explainer to capture the same relevant components under various transformations to the input.

Original: More speciﬁcally, if the classiﬁer predicts the same class for both the original and transformed inputs, consistency attempts to measure whether the generated expla- nation for the transformed input  is similar to the one generated for the original input modulo the transformation.
Revised: More speciﬁcally, if the classiﬁer predicts the same class for both the original and transformed inputs. Then, consistency measures whether the generated explanation for the transformed input (after applying an inverse transform) is similar to the one generated for the original input.

Revised: For example, if we apply vertical ﬂip as a semantically invariant transformation, we ﬂip the generated heatmap from the transformed image before comparing with the heatmap generated for the original image.

Original: Formally, this can be represented as
Revised: Formally, this can be represented as

Original: [Equation]
Revised: [Equation]

Original: Semantically Invariant Transforms We focus on a subset of all potential transformations which does not change the semantic information contained in the input.
Revised: Semantically Invariant Transforms We focus on a subset of all potential transformations which does not change the semantic information contained in the input.

Original: We call this subset Semantically
Revised: We call this subset Semantically

Original: Invariant Transforms .
Revised: Invariant Transforms .

Original: Most work so far have considered only noising as a method of transforming the input.
Revised: Most work so far has considered only noising as a method of transforming the input.

Original: By constraining the magnitude of the added noise, we can control the size of the neighbourhood in which we perturb the images.
Revised: By constraining the magnitude of the added noise, we can control the size of the neighbourhood in which we perturb the images.

Original: We consider not only simple transformations that perturb the image in a small neighbourhood but also those that move the data point to vastly different regions in the input space while still retaining the semantic information contained within.
Revised: In this work, we consider not only simple transformations that perturb the image in a small neighbourhood but also those that move the data point to vastly different regions in the input space while still retaining the semantic information contained within.

Original: This allows us to verify whether the explainer works as expected across larger regions of the input space.
Revised: This allows us to verify whether the explainer works as expected across larger regions of the input space.

Original: For example, in the case of images, the family of transformations T include afﬁne transformations (translations and rotations), horizontal and vertical ﬂips, noising (white, blue etc.), scaling etc.
Revised: For example, in the case of images, the family of transformations T include afﬁne transformations (translations and rotations), horizontal and vertical ﬂips, noising (white, blue etc.), scaling etc.

Original: In the image domain, d could beeasily realized as the (cid:96) 2 (Euclidean) distance between explanations of the ground truth and  the transformed images.
Revised: In the image domain, d could be realized as the (cid:96) 2 (Euclidean) distance between explanations of the ground truth and inverted the transformed images (according to Eq. 4).

Original: However, Wang et al.
Revised: However, Wang et al.

Original:  and Zhao & Itti (2016) have shown that (cid:96) 2 is not robust for images and may result in larger distances between the pairs of mostly similar explanations.
Revised: (2005) and Zhao & Itti (2016) have shown that (cid:96) 2 is not robust for images and may result in larger distances between the pairs of mostly similar explanations.

Original: This is attributed to the fact that (cid:96) 2 is only a summation of the pixelwise intensity differences and, as a result, small deformations may results in large distances.
Revised: This is attributed to the fact that (cid:96) 2 is only a summation of the pixel-wise intensity differences and, as a result, small deformations may results in large distances.

Original: Even when the images are normalized before hand, (cid:96) 2 is still not a suitable distance for images.
Revised: Even when the images are normalized before hand, (cid:96) 2 is still not a suitable distance for images.

Original: Therefore, we instead use Dynamic Time Warping (DTW)
Revised: Therefore, we instead use Dynamic Time Warping (DTW)

Original: (Sakoe & Chiba, 1978) which allows for computing distances between two time series, even when misaligned or out of phase.
Revised: (Sakoe & Chiba, 1978) which allows for computing distances between two time series, even when misaligned or out of phase.

Original: Ibrahim & Valli (2008) has shown that DTW is effective for images as well, not only for temporal data as originally intended.
Revised: Ibrahim & Valli (2008) has shown that DTW is effective for images as well, not only for temporal data as originally intended.

Original: Due to DTW’s high computational cost (quadratic time and space complexity), we use FastDTW (Salvador & Chan, 2007), an approximation of DTW that has linear complexity in order to compute the distance between pairs of explanations.
Revised: Due to DTW’s high computational cost (quadratic time and space complexity), we use FastDTW (Salvador & Chan, 2007), an approximation of DTW that has linear complexity in order to compute the distance between pairs of explanations.

Original: 3.2. C ONFIDENCE
Revised: 3.2. C ONFIDENCE

Original: Finally, conﬁdence is concerned with whether the generated explanation and the masked input result in high conﬁdence predictions.
Revised: Finally, conﬁdence is concerned with whether the generated explanation and the masked input result in high conﬁdence predictions.

Original: This is a desirable property to enable explanations to be useful for downstream processes including human inspection (Hall et al., 2019).
Revised: This is a desirable property to enable explanations to be useful for downstream processes including human inspection (Hall et al., 2019).

Original: So far, our method for computing correctness sheds light only on the average case and is not particularly useful for individual explanations.
Revised: So far, our method for computing correctness sheds light only on the average case and is not particularly useful for individual explanations.

Original: Generating high-conﬁdence predictions is related to the well researched ﬁeld of max-margin classiﬁers (Gong & Xu, 2007).
Revised: Generating high-conﬁdence predictions is related to the well researched ﬁeld of max-margin classiﬁers (Gong & Xu, 2007).

Original: A large margin in classiﬁers is widely accepted as a desirable property.
Revised: A large margin in classiﬁers is widely accepted as a desirable property.

Original: Here, we extend this notion to explainers and propose that explainers generating explanations that result in high conﬁdence predictions are desirable to those that do not.
Revised: Here, we extend this notion to explainers and propose that explainers generating explanations that result in high conﬁdence predictions are desirable to those that do not.

Original: In addition to the desirable statistical properties that this enforces, high conﬁdence predictions are also vital for building trust with human users of the explainer as they are more interested in the per-instance performance than the average (Narayanan et al., 2018; Ross et al., 2017).
Revised: In addition to the desirable statistical properties that this enforces, high conﬁdence predictions are also vital for building trust with human users of the explainer as they are more interested in the per-instance performance than the average (Narayanan et al., 2018; Ross et al., 2017).

Original: Concretely, we use the same procedure as in Sec.
Revised: Concretely, we use the same procedure as in Sec.

Original: Instead of computing the increase in accuracy, we compute instead the difference in probability assigned to the ground-truth class, as well as the difference in entropy of the softmax distributions of the original and masked images.
Revised: Instead of computing the increase in accuracy, we compute instead the difference in probability assigned to the ground-truth class, as well as the difference in entropy of the softmax distributions of the original and masked images.

Original: We report this for both normal and inverted saliency maps.
Revised: We report this for both normal and inverted saliency maps.

Revised: We expect to observe a positive probability difference and negative entropy difference under normal masking and an inverted behavior under inverse masking owing to similar reasons discussed in Sec.

Original: Interestingly, explainers that generate coarse explanations can easily fool this metric.
Revised: However, explainers that generate coarse explanations can easily fool this metric.

Original: An extreme case is when the explainer estimates the entire input as useful.
Revised: An extreme case is when the explainer considers the entire input as useful.

Original: Such an explainer is useless and yet will have the theoretically highest change in conﬁdence and entropy.
Revised: Such an explainer is useless but will have the theoretically highest change in conﬁdence and entropy.

Original: To combat this and to establish how sparse the generated explanations are, we also report the average number of pixels in the explanations.
Revised: To combat this and to establish how sparse the generated explanations are, we also report the average number of pixels in the explanations, normalized by the total number of pixels in the image.

Original: We do not combine these numbers into one as different situations have different preferences.
Revised: We do not combine these numbers into one as different situations have different preferences.

Original: For example, in computational biology domains, sparsity is not as important as increase in conﬁdence.
Revised: For example, in computational biology domains, sparsity is not as important as increase in conﬁdence.

Original: The right weighting again depends on the use case and user preference.
Revised: The right weighting again depends on the use case and user preference.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We use a pretrained Inception v3 (Szegedy et al., 2015)  on ImageNet (Deng et al., 2009) 5 .
Revised: We use an Inception v3 (Szegedy et al., 2015) architecture pretrained on ImageNet (ISLVC-2012).

Original: We compare LIME, Grad-CAM, SmoothGrad and Integrated Gradients and measure how they perform on the metrics described previously.
Revised: We compare LIME, Grad-CAM, SmoothGrad and Integrated Gradients and measure how they perform on the metrics described previously.

Original: All experiments and explainers (except LIME) were implemented in PyTorch (Paszke et al., 2017).
Revised: All experiments and explainers (except LIME) were implemented in PyTorch (Paszke et al., 2017).

Original: Wherever possible, we reused the ofﬁcial implementation or kept our re-implementation as close to the ofﬁcial code base as possible.
Revised: Wherever possible, we reused the ofﬁcial implementation or kept our re-implementation as close to the ofﬁcial codebase as possible.

Original: The correctness and conﬁdence metrics for every explainer are computed over 5 runs and mean values are reported.
Revised: The correctness and conﬁdence metrics for every explainer are computed over 5 runs and mean values are reported.

Revised: We use a ﬁxed constant threshold to binarize explanation masks.

Revised: We conducted further experiments by thresholding on the percentiles 6 instead(as done in (Smilkov et al., 2017)).

Revised: These results have been reported in tables 6 and 7.

Revised: We found that the choice did not affect the relative trends observed.

Original: We consider the following semantically invariant transforms: translations ( x = ± 0 .
Revised: We consider the following semantically invariant transforms: translations ( x = ± 0 .

Original: 2 , y = ± 0 . 2 ), rotations ( − 15 ◦ , − 10 ◦ , − 5 ◦ , 5 ◦ , 10 ◦ , 15 ◦ ), ﬂips (horizontal and vertical).
Revised: 2 , y = ± 0 . 2 ), rotations ( − 15 ◦ , − 10 ◦ , − 5 ◦ , 5 ◦ , 10 ◦ , 15 ◦ ), ﬂips (horizontal and vertical).

Original: To establish that these do not produce too many out-of-distribution samples (causing a decrease in classiﬁer performance), we compute the accuracy of the underlying classiﬁer under these transformations.
Revised: To establish that these do not produce too many out-of-distribution samples (causing a decrease in classiﬁer performance), we compute the accuracy of the underlying classiﬁer under these transformations.

Original: Table 1 shows that, indeed, drops in accuracy are not signiﬁcant.
Revised: Table 1 shows that, indeed, drops in accuracy are not signiﬁcant.

Original: Even though noising is semantically invariant in the image domain, we do not consider it in our experiments as some explainers like Smoothgrad would be unfairly favoured.
Revised: Even though noising is semantically invariant in the image domain, we do not consider it in our experiments as some explainers like Smoothgrad would be unfairly favoured.

Original: [Table]
Revised: [Table]

Original: =

Original: [Table]

Original: [Table]

Original: =
Revised: =

Original: [Table]

Original: [Table]

Original: [Table]

Original: y =

Original: [Table]

Original: [Table]

Original: [Table]

Original: y

Original: = -0.

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: −

Original: [Table]
Revised: [Table]

Original: ◦

Original: [Table][Table] − [Table]
Revised: [Table]

Original: ◦

Original: [Table]

Original: [Table]

Original: 5 ◦

Original: [Table]

Original: [Table]

Original: 10 ◦

Original: [Table]

Original: [Table]

Original: 15 ◦

Original: [Table]

Original: [Table]
Revised: [Table]

Original: 4. E VALUATING C ORRECTNESS
Revised: E VALUATING C ORRECTNESS

Original: We perform the procedure described in Sec.
Revised: We perform the procedure described in Sec.

Original: 3.2.1 and report the results in Table 2.
Revised: 3.2.1 and report the results in Table 2.

Original: The baseline acc@1 and acc@5 were 11.42% and 53.8% respectively.
Revised: The baseline acc@1 and acc@5 were 11.42% and 53.8% respectively.

Original: We hypothesized that a good explainer’s accuracy increases with the normal masking and decreases with inverse masking.
Revised: We hypothesized that a good explainer’s accuracy increases with the normal masking and decreases with inverse masking.

Original: We see the expected increases in accuracies across all the explainers with Grad-CAM obtaining the highest increase at 97.44%.
Revised: We see the expected increases in accuracies across all the explainers with Grad-CAM obtaining the highest increase at 97.44%.

Original: However, for the inverse masking, we see that both LIME and Grad-CAM show results contrary to our hypothesis.
Revised: However, for the inverse masking, we see that both LIME and Grad-CAM show results contrary to our hypothesis.

Original: This can be explained by observing examples of maps generated in Fig.
Revised: This can be explained by observing examples of maps generated in Figs.

Revised: 1 and 4.

Original: We see that, on average, Grad-CAM generates much larger explanations than all other explainers (can be seen in Table 3 as well).
Revised: We see that, on average, Grad-CAM generates much larger explanations than all other explainers (can be seen in Table 3 as well).

Original: Thislack of sparsity implies that Grad-CAM misidentiﬁes several non-important pixels as important and thus when we compute the inverse masks, we remove nonimportant pixels that could confuse the classiﬁer. 
Revised: This implies that Grad-CAM misidentiﬁes several non-important pixels as important and thus when we compute the inverse masks, we remove non-important pixels that could confuse the classiﬁer. think of a better explanation

Original: On the other hand, we again see from Table 3 and Fig. 1  that LIME generates the smallest explanations.
Revised: In the case of LIME, we again see from Table 3 and Figs. 1 and 4 that LIME generates the smallest explanations.

Original: We further see from Table 2 that LIME has the smallest accuracy gain (in both acc@1 and acc@5).
Revised: We further see from Table 2 that LIME has the smallest accuracy gain (in both acc@ and acc@5).

Original: These indicate that LIME fails to select important pixels that were selected by all other explainers.
Revised: These indicate that LIME fails to select important pixels that were selected by all other explainers.

Original: Thus, we can conclude that the inverse masks in case of LIME would contain important pixels as well and thus would cause increase in accuracy as observed.
Revised: Thus, we can conclude that the inverse masks in case of LIME would contain important pixels as well and thus would cause increase in accuracy as observed.

Original: 4.1. E FFECT OF N UMBER OF I MAGES
Revised: 4.1. E FFECT OF N UMBER OF I MAGES

Original: As detailed previously, our methodology for computing correctness involves choosing a number k of top and bottom images to be used for masking.
Revised: As detailed previously, our methodology for computing correctness involves choosing a number k of top and bottom images to be used for masking.

Original: We evaluate how sensitive the measured correctness[Table] of explainers are to the value of k .
Revised: We evaluate how sensitive the measured correctness of explainers are to the value of k .

Original: We report the changes in accuracy with respect to the unmasked bottom images for k = { 5,10,15,20,25 } in Fig.
Revised: We report the changes in accuracy with respect to the unmasked bottom images for k = { 5,10,15,20,25 } in Fig.

Original: The actual accuracy numbers are also reported in
Revised: The actual accuracy numbers are also reported in

Original: Tables 4 and 5.
Revised: Tables 4 and 5.

Original: We see that for both acc@1 and acc@5, the change in accuracy for normal masking decreases as we increase k .
Revised: We see that for both acc@1 and acc@5, the change in accuracy for normal masking decreases as we increase k .

Original: This is as expected since with an increasing value of k , the top-k images become on-average less informative (i.e., as more images are added, they are less informative, since they are sorted based on probability of assignment). Similarly , the background images becomes more informative.
Revised: This is as expected since the average conﬁdence gap between the top-k and the bottom-k images decreases as k increases.

Original: This means that the important pixels in the background images are masked with nonimportant pixels from the foreground images.
Revised: This means that the important pixels in the background images are masked with non-important pixels from the foreground images.

Original: On the contrary, LIME shows a smaller decrease in accuracy (both acc@1 and acc@5).
Revised: On the contrary, LIME shows a smaller decrease in accuracy (both acc@1 and acc@5).

Original: This can be explained by the fact that LIME does not capture all important pixels, and therefore all important pixels from the background are not replaced by less-informative pixels.
Revised: This can be explained by the fact that LIME does not capture all important pixels, and therefore all important pixels from the background are not replaced by less-informative pixels.

Original: Similarly, for acc@1 and acc@5 for inverse masking, we see that LIME, Smoothgrad and Integrated
Revised: Similarly, for acc@1 and acc@5 for inverse masking, we see that LIME, Smoothgrad and Integrated

Original: Gradients behave as expected, i.e., the drop in accuracy is diminished with k is increased as we are retaining the informative parts from the new background images.
Revised: Gradients behave as expected, i.e., the drop in accuracy is diminished with k is increased as we are retaining the informative parts from the new background images.

Original: Interestingly, the drop in accuracy for Grad-CAM is stable and close to zero.
Revised: Interestingly, the drop in accuracy for Grad-CAM is stable and close to zero.

Original: To understand this, we refer again to Table 3 and note that Grad-CAM produces the smallest inverse maps on average.
Revised: To understand this, we refer again to Table 3 and note that Grad-CAM produces the smallest inverse maps on average.

Original: This implies that when we perform the inverse masking, we retain much of the informative pixels of the background image and thus do not see signiﬁcant drops in accuracy relative to the unmasked bottom- k image dataset.
Revised: This implies that when we perform the inverse masking, we retain much of the informative pixels of the background image and thus do not see signiﬁcant drops in accuracy relative to the unmasked bottom- k image dataset.

Original:  E VALUATING C ONSISTENCY
Revised: 4. E VALUATING C ONSISTENCY

Original: Next, we evaluate consistency by computing the distance with FastDTW between the saliency maps generated on the original images and those generated when transformations are applied.
Revised: Next, we evaluate consistency by computing the distance with FastDTW between the saliency maps generated on the original images and those generated when transformations are applied to the input image (Following Eq. 4).

Original: Fig.
Revised: Fig.

Original: 3 and Table 6 report the normalized distances relative to each transformation (i.e., heatmaps sum to 1).
Revised: 3 and Table 8 report the normalized distances relative to each transformation (i.e., heatmaps sum to 1).

Original: First, as transformations become more severe relative to the original saliency maps, the distances also increase.
Revised: First, as transformations become more drastic relative to the original saliency maps, the distances also increase.

Original: This is the desired behavior one would expect, thus motivating our choice for using
Revised: This is the desired behavior one would expect, thus motivating our choice for using

Original: FastDTW.
Revised: FastDTW.

Original: Second, Grad-CAM outperforms all other explainers, as reﬂected by the fact that its corresponding distances are always smallest.
Revised: Second, Grad-CAM outperforms all other explainers, as reﬂected by the fact that its corresponding distances are always smallest.

Original: It is followed by Smoothgrad, Integrated Gradients and LIME.
Revised: It is followed by Smoothgrad, Integrated Gradients and LIME.

Original: This is expected given the grainy saliency maps obtained with Integrated Gradients and
Revised: This is expected given the grainy saliency maps obtained with Integrated Gradients and

Original: Smoothgrad, as well as the patchy heatmaps generated with LIME.
Revised: Smoothgrad, as well as the patchy heatmaps generated with LIME.

Original: 4. E VALUATING C ONFIDENCE
Revised: 4. E VALUATING C ONFIDENCE

Original: Measuring conﬁdence quantiﬁes the performance of the explainers on a per-instance case and not only in the average.
Revised: Measuring conﬁdence quantiﬁes the performance of the explainers on a per-instance case and not only in the average.

Original: As described in Sec.
Revised: As described in Sec.

Original: 3.2.3, we compute the change in probability assigned to the ground-truth class ( ∆ conf) as well as the change in entropy of the softmax distribution ( ∆ entropy) as proxies for estimating the conﬁdence of explanations.
Revised: 3.2.3, we compute the change in probability assigned to the ground-truth class ( ∆ conf) as well as the change in entropy of the softmax distribution ( ∆ entropy) as proxies for estimating the conﬁdence of explanations.

Original: Additionally, we report the proportions of pixels in the heatmaps to the total number of pixels 6 , averaged across the top- k dataset.
Revised: Additionally, we report the proportions of pixels in the heatmaps to the total number of pixels 7 , averaged across the top- k dataset.

Original: We see that for conﬁdence, the trends mimic the ones observed in Table 2.
Revised: We see that for conﬁdence, the trends mimic the ones observed in Table 2.

Original: This implies that masking with extracted heatmaps not only increases accuracy but also results in high-conﬁdence predictions across explainers.
Revised: This implies that masking with extracted heatmaps not only increases accuracy but also results in high-conﬁdence predictions across explainers.

Original: More speciﬁcally, we see that Grad-CAM again outperforms the other explainers (both ∆ conf and ∆ entropy) in the normal heatmaps by large margins.
Revised: More speciﬁcally, we see that Grad-CAM again outperforms the other explainers (both ∆ conf and ∆ entropy) in the normal heatmaps by large margins.

Original: In the case of inverse masking, conﬁdence and entropy for LIME show behaviours contrary to our expectations.
Revised: In the case of inverse masking, conﬁdence and entropy for LIME show behaviours contrary to our expectations.

Original: This can be attributed to the ”patchiness” of the explanations generated by LIME which was discussed in the previous sections.
Revised: This can be attributed to the ”patchiness” of the explanations generated by LIME which was discussed in the previous sections.

Original: [Table]
Revised: [Table]

Original:  C ONCLUSIONS AND F UTURE W ORK
Revised:  C ONCLUSIONS AND F UTURE W ORK

Original: In this paper [Table] we formulated desired properties of a good explainer and proposed a generic, computationally inexpensive suite of metrics – correctness, consistency and conﬁdence – to objectively evaluate and compare explainers.
Revised: In this paper, we formulated desired properties of a good explainer and proposed a generic, computationally inexpensive suite of metrics – correctness, consistency and conﬁdence – to objectively evaluate and compare explainers.

Original: We compared well-known explainers, such as LIME, Grad-CAM,
Revised: We compared well-known explainers, such as LIME, Grad-CAM,

Original: Integrated Gradients and SmoothGrad, on a pretrained Inception-V3 model on the ImageNet dataset.
Revised: Integrated Gradients and SmoothGrad, on a pretrained Inception-V3 model on the ImageNet dataset.

Original: Our experiments show that the metrics proposed capture various pros and cons of each explainer allowing users to make an informed choice about which explainer to use for their use case.
Revised: Our experiments show that the metrics proposed capture various pros and cons of each explainer allowing users to make an informed choice about which explainer to use for their use case.

Original: Speciﬁcally, we observe that Grad-CAM often performs better than the other explainers but suffers from drawbacks when inverse masking situations are considered.
Revised: Speciﬁcally, we observe that Grad-CAM often performs better than the other explainers but suffers from drawbacks when inverse masking situations are considered.

Original: On the other hand, LIME performs poorly in all situations we consider.
Revised: On the other hand, LIME performs poorly in all situations we consider.

Original: Moreover, we also point out the pitfalls of trying to combine results from multiple metrics as they tend to hide anomalous behaviours of the underlying metrics (as seen from Pseudo-F1 from Table 2).
Revised: Moreover, we also point out the pitfalls of trying to combine results from multiple metrics as they tend to hide anomalous behaviours of the underlying metrics (as seen from Pseudo-F1 from Table 2).

Original: We recommend that users sanity-check explainers by looking at individual metrics before making a decision based on the combined metric.
Revised: We recommend that users sanity-check explainers by looking at individual metrics before making a decision based on the combined metric.

Revised: Furthermore, we urge the XAI community to resist the temptation to propose an one-size-ﬁts-all metrics as we have shown that such metrics tend to hide nuanced trade-offs that practitioners need to be aware of.

Original: Going forward, we invite the research community to test our metrics on other explainers, datasets, underlying classiﬁers and data modalities.
Revised: Going forward, we invite the research community to test our metrics on other explainers, datasets, underlying classiﬁers and data modalities.

Original: Additionally, since the metrics proposed are differentiable, we believe exciting new lines of research would be to develop explainers that directly optimize for these metrics, as well as self-explaining models that incorporate such metrics into their learning regiment.
Revised: Additionally, since the metrics proposed are differentiable, we believe exciting new liens of research would be to develop explainers that directly optimize for these metrics, as well as self-explaining models that incorporate such metrics into their learning regiment.

Original: A A DDITIONAL E XPERIMENTAL R ESULTS A. E FFECT OF k ON CORRECTNESS FOR NORMAL AND INVERSE MASKING 
Revised: A A DDITIONAL E XPERIMENTAL R ESULTS A. V ISUAL MASKING RESULTS

Revised: A. [Table] E FFECT OF k ON CORRECTNESS FOR NORMAL AND INVERSE MASKING

Original: [Table] [Table]
Revised: [Table] [Table]

Original: A. N ORMALIZED DTW ON SALIENCY MAPS UNDER INVARIANT TRANSFORMS
Revised: A. E FFECT OF GRADIENT MAP THRESHOLD VALUE ON THE CORRECTNESS AND CONFIDENCE METRICS

Original: [Table]
Revised: [Table]

Original:  A.  Z ERO  MASKING RESULTS
Revised: [Table] [Table] [Table] A. E FFECT OF USING Z ERO AND GRAY MASKING FOR CORRECTNESS

Original: [Table]
Revised: [Table]

Revised: [Table]

Revised: acc@ 93.