Original: E VALUATION OF S IMILARITY - BASED E XPLANATIONS
Revised: E VALUATION OF S IMILARITY - BASED E XPLANATIONS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Explaining predictions made by complex machine learning models helps users understand and accept the predicted outputs with conﬁdence.
Revised: Explaining predictions made by complex machine learning models helps users understand and accept the predicted outputs with conﬁdence.

Original: One way to achieve this goal is to use similarity-based explanation that provides similar instances as evidence to support a model’s prediction.
Revised: One way to achieve this goal is to use similarity-based explanation that provides similar instances as evidence to support a model’s prediction.

Original: Several relevance metrics are used for this purpose.
Revised: Several relevance metrics are used for this purpose.

Original: In this study, we investigate which relevance metric can provide a reasonable explanation to users.
Revised: In this study, we investigate which relevance metric can provide a reasonable explanation to users.

Original: Speciﬁcally, we adopted three tests to evaluate whether the relevance metrics satisfy the minimal requirements for similarity-based explanation.
Revised: Speciﬁcally, we adopted three tests to evaluate whether the relevance metrics satisfy the minimal requirements for similarity-based explanation.

Original: Our experiments reveal that the cosine similarity of the gradients of the loss performs best, which would be a recommended choice in practice.
Revised: Our experiments reveal that the cosine similarity of the gradients of the loss performs best, which would be a recommended choice in practice.

Original: In addition, we show that some metrics performs poorly in our tests, and analyze the reasons of their failure.
Revised: In addition, we show that some metrics performs poorly in our tests, and analyze the reasons of their failure.

Original: We expect our insights to help practitioners to select appropriate relevance metrics, and also to help further researches for designing better relevance metrics for explanations.
Revised: We expect our insights to help practitioners to select appropriate relevance metrics, and also to help further researches for designing better relevance metrics for explanations.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Explaining predictions made by complex machine learning models helps users understand and accept the predicted outputs with conﬁdence (Ribeiro et al., 2016; Lundberg & Lee, 2017; Guidotti et al., 2018; Adadi & Berrada, 2018; Molnar, 2020).
Revised: Explaining predictions made by complex machine learning models helps users understand and accept the predicted outputs with conﬁdence (Ribeiro et al., 2016; Lundberg & Lee, 2017; Guidotti et al., 2018; Adadi & Berrada, 2018; Molnar, 2020).

Original: Instance-based explanation attains this goal by presenting one or several training instances that support the model’s prediction.
Revised: Instance-based explanation attains this goal by presenting one or several training instances that support the model’s prediction.

Original: Several types of instance-based explanation have been proposed: explaining with instances similar to the instance of interest (i.e. the test instance in question) (Charpiat et al., 2019; Barshan et al., 2020); harmful instances that degrade the performance of the model (Koh & Liang, 2017; Khanna et al., 2019); counter-examples that contrast how the prediction can be changed (Wachter et al., 2018); and irregular instances (Kim et al., 2016).
Revised: Several types of instance-based explanation have been proposed: explaining with instances similar to the instance of interest (i.e. the test instance in question) (Charpiat et al., 2019; Barshan et al., 2020); harmful instances that degrade the performance of the model (Koh & Liang, 2017; Khanna et al., 2019); counter-examples that contrast how the prediction can be changed (Wachter et al., 2018); and irregular instances (Kim et al., 2016).

Original: Among these types, we focus on the type of explanation that gives one or several training instances similar to a test instance in question and the model’s prediction for it.
Revised: Among these types, we focus on the type of explanation that gives one or several training instances similar to a test instance in question and the model’s prediction for it.

Original: We refer to this type of instance-based explanation as similarity-based explanation .
Revised: We refer to this type of instance-based explanation as similarity-based explanation .

Original: A similarity-based explanation is of the form “I (the model) think this image is cat because similar images I saw in the past were also cat.”
Revised: A similarity-based explanation is of the form “I (the model) think this image is cat because similar images I saw in the past were also cat.”

Original: This type of explanation is analogous to the way humans make decisions by referring to their prior experiences (Klein & Calderwood, 1988; Klein, 1989; Read & Cesa, 1991).
Revised: This type of explanation is analogous to the way humans make decisions by referring to their prior experiences (Klein & Calderwood, 1988; Klein, 1989; Read & Cesa, 1991).

Original: Hence, this type of explanation tends to be easy to understand even to users with little expertise about machine learning.
Revised: Hence, this type of explanation tends to be easy to understand even to users with little expertise about machine learning.

Original: There is also a report that with this type of explanation, users tend to have higher conﬁdence in model predictions compared to explanations that presents contributing features (Cunningham et al., 2003).
Revised: There is also a report that with this type of explanation, users tend to have higher conﬁdence in model predictions compared to explanations that presents contributing features (Cunningham et al., 2003).

Original: In the instance-based explanation paradigm, including similarity-based explanation, a relevance metric R ( z , z (cid:48) )
Revised: In the instance-based explanation paradigm, including similarity-based explanation, a relevance metric R ( z , z (cid:48) )

Original: ∈ R is typically used to quantify the relationship between two instance z =
Revised: ∈ R is typically used to quantify the relationship between two instance z =

Original: ( x , y ) and z (cid:48) =
Revised: ( x , y ) and z (cid:48) =

Original: ( x (cid:48) , y (cid:48) ) .
Revised: ( x (cid:48) , y (cid:48) ) .

Original: Deﬁnition 1 (Instance-based Explanation Using Relevance Metric) .
Revised: Deﬁnition 1 (Instance-based Explanation Using Relevance Metric) .

Original: Let D
Revised: Let D

Original: = { z ( i ) train =
Revised: = { z ( i ) train =

Original: ( x ( i ) train , y ( i ) train ) } N i =1 be a set of training instances, and x test be a test input of interest whose predicted output is given by (cid:98) y test =
Revised: ( x ( i ) train , y ( i ) train ) } N i =1 be a set of training instances, and x test be a test input of interest whose predicted output is given by (cid:98) y test =

Original: f ( x test ) with a model f .
Revised: f ( x test ) with a model f .

Original: An instance-based explanation method gives the most relevant training instance ¯ z ∈ D to the test instance z test =
Revised: An instance-based explanation method gives the most relevant training instance ¯ z ∈ D to the test instance z test =

Original: ( x test , (cid:98) y test ) by ¯ z = arg max z train
Revised: ( x test , (cid:98) y test ) by ¯ z = arg max z train

Original: ∈D R
Revised: ∈D R

Original: ( z test , z train ) using a relevance metric R ( z test , z train ) .
Revised: ( z test , z train ) using a relevance metric R ( z test , z train ) .

Original: Previously proposed relevance metrics include similarity (Caruana et al., 1999), kernel functions (Kim et al., 2016; Khanna et al., 2019), and inﬂuence function (Koh & Liang, 2017).
Revised: Previously proposed relevance metrics include similarity (Caruana et al., 1999), kernel functions (Kim et al., 2016; Khanna et al., 2019), and inﬂuence function (Koh & Liang, 2017).

Original: [Table]
Revised: [Table]

Original: An immediate, critical question is which relevance metric is appropriate for which type of instancebased explanations.
Revised: An immediate, critical question is which relevance metric is appropriate for which type of instancebased explanations.

Original: There is no doubt that different types of explanations require different metrics.
Revised: There is no doubt that different types of explanations require different metrics.

Original: Despite its potential importance, however, little has been explored on this question.
Revised: Despite its potential importance, however, little has been explored on this question.

Original: Given this background, in this paper, focusing on similarity-based explanation, we investigate which relevance metrics are appropriate through comprehensive experiments.
Revised: Given this background, in this paper, focusing on similarity-based explanation, we investigate which relevance metrics are appropriate through comprehensive experiments.

Original: Contributions We provide the ﬁrst answer to the question about which relevance metrics have desirable properties for similarity-based explanation.
Revised: Contributions We provide the ﬁrst answer to the question about which relevance metrics have desirable properties for similarity-based explanation.

Original: For this purpose, we propose to use three “minimal requirement” tests to evaluate a broad variety of relevance metrics in terms of its appropriateness.
Revised: For this purpose, we propose to use three “minimal requirement” tests to evaluate a broad variety of relevance metrics in terms of its appropriateness.

Original: The ﬁrst test is the model randomization test originally proposed by Adebayo et al.
Revised: The ﬁrst test is the model randomization test originally proposed by Adebayo et al.

Original: (2018) for evaluating saliency-based methods, and the other two tests, the identical class test and identical subclass test, are newly designed in this study.
Revised: (2018) for evaluating saliency-based methods, and the other two tests, the identical class test and identical subclass test, are newly designed in this study.

Original: As summarized in Table 1, our experiments reveal that (i) the cosine similarity of gradients performs best, which is probably a recommended choice for similarity-based explanation in practice, and (ii) some relevance metrics, such as inﬂuence function, havesurprisingly poor performances on the identical class test and identical subclass test, indicating that their use should be deprecated for similarity-based explanation.
Revised: As summarized in Table 1, our experiments reveal that (i) the cosine similarity of gradients performs best, which is probably a recommended choice for similarity-based explanation in practice, and (ii) some relevance metrics, such as inﬂuence function, have poor performances on the identical class test and identical subclass test, indicating that their use should be deprecated for similarity-based explanation.

Original: We also analyze why some metrics are successful and why some are not.
Revised: We also analyze why some metrics are successful and why some are not.

Original: We expect these insights to help practitioners to select appropriate relevance metrics.
Revised: We expect these insights to help practitioners to select appropriate relevance metrics.

Original: 1. P RELIMINARIES
Revised: 1. P RELIMINARIES

Original: Notations For vectors a , b ∈ R p , we denote the dot product by (cid:104) a , b (cid:105) :
Revised: Notations For vectors a , b ∈ R p , we denote the dot product by (cid:104) a , b (cid:105) :

Original: = (cid:80) pi
Revised: = (cid:80) pi

Original: =1 a i b i , the (cid:96) 2 norm by (cid:107) a (cid:107) :
Revised: =1 a i b i , the (cid:96) 2 norm by (cid:107) a (cid:107) :

Original: = (cid:112) (cid:104) a , a (cid:105) , and the cosine similarity by cos( a , b ) :=
Revised: = (cid:112) (cid:104) a , a (cid:105) , and the cosine similarity by cos( a , b ) :=

Original: (cid:104) a , b (cid:105) / (cid:107) a (cid:107)(cid:107) b (cid:107) .
Revised: (cid:104) a , b (cid:105) / (cid:107) a (cid:107)(cid:107) b (cid:107) .

Original: Classiﬁcation Problem We consider a standard classiﬁcation problem as the evaluation benchmark.
Revised: Classiﬁcation Problem We consider a standard classiﬁcation problem as the evaluation benchmark.

Original: The model is the conditional probability p ( y | x ; θ ) with parameter θ .
Revised: The model is the conditional probability p ( y | x ; θ ) with parameter θ .

Original: Let (cid:98) θ be a trained parameter (cid:98) θ =
Revised: Let (cid:98) θ be a trained parameter (cid:98) θ =

Original: arg min θ L train :
Revised: arg min θ L train :

Original: = 1 N (cid:80) Ni =1 (cid:96) ( z ( i ) train ; θ ) , where the loss function (cid:96) is the cross entropy (cid:96) ( z ; θ ) =
Revised: = 1 N (cid:80) Ni =1 (cid:96) ( z ( i ) train ; θ ) , where the loss function (cid:96) is the cross entropy (cid:96) ( z ; θ ) =

Original: − log p ( y | x ; θ ) for an input-output pair z
Revised: − log p ( y | x ; θ ) for an input-output pair z

Original: =
Revised: =

Original: ( x , y ) .
Revised: ( x , y ) .

Original: The model classiﬁes a test input x test by assigning the class with the highest probability (cid:98) y test =
Revised: The model classiﬁes a test input x test by assigning the class with the highest probability (cid:98) y test =

Original: arg max y p ( y | x test ; (cid:98) θ ) .
Revised: arg max y p ( y | x test ; (cid:98) θ ) .

Original: 1. R ELEVANCE M ETRICS
Revised: 1. R ELEVANCE M ETRICS

Original: We present an overview of the two types of relevance metrics considered in this study, similarity metrics and gradient-based metrics .
Revised: We present an overview of the two types of relevance metrics considered in this study, similarity metrics and gradient-based metrics .

Original: See Table 1 for the list of metrics and their abbreviations.
Revised: See Table 1 for the list of metrics and their abbreviations.

Original: Similarity Metrics We consider the following popular similarity metrics with a feature map φ ( z ) .
Revised: Similarity Metrics We consider the following popular similarity metrics with a feature map φ ( z ) .

Original: • (cid:96) 2 Metric : R (cid:96) 2 ( z , z (cid:48) ) := −(cid:107) φ ( z ) − φ ( z (cid:48) )
Revised: • (cid:96) 2 Metric : R (cid:96) 2 ( z , z (cid:48) ) := −(cid:107) φ ( z ) − φ ( z (cid:48) )

Original: (cid:107) 2 , which is a typical choice for nearest neighbor methods (Hastie et al., 2009; Abu Alfeilat et al., 2019).
Revised: (cid:107) 2 , which is a typical choice for nearest neighbor methods (Hastie et al., 2009; Abu Alfeilat et al., 2019).

Original: • Cosine Metric : R cos ( z , z (cid:48) )
Revised: • Cosine Metric : R cos ( z , z (cid:48) )

Original: := cos( φ ( z ) , φ ( z (cid:48) )) , which is commonly used in natural language processing tasks (Mikolov et al., 2013; Arora et al., 2017; Conneau et al., 2017).
Revised: := cos( φ ( z ) , φ ( z (cid:48) )) , which is commonly used in natural language processing tasks (Mikolov et al., 2013; Arora et al., 2017; Conneau et al., 2017).

Original: • Dot Metric : R dot ( z , z (cid:48) ) :
Revised: • Dot Metric : R dot ( z , z (cid:48) ) :

Original: = (cid:104) φ ( z ) , φ ( z (cid:48) )
Revised: = (cid:104) φ ( z ) , φ ( z (cid:48) )

Original: (cid:105) , which is a kernel function used in kernel models such as SVM (Schölkopf et al., 2002;
Revised: (cid:105) , which is a kernel function used in kernel models such as SVM (Schölkopf et al., 2002;

Original: Fan et al., 2005; Bien & Tibshirani, 2011).
Revised: Fan et al., 2005; Bien & Tibshirani, 2011).

Original: As the feature map φ ( z ) , we consider (i) an identity map φ ( z ) =
Revised: As the feature map φ ( z ) , we consider (i) an identity map φ ( z ) =

Original: x ; (ii) the last hidden layer φ ( z ) =
Revised: x ; (ii) the last hidden layer φ ( z ) =

Original: h last , which is the latent representation of the input x , one layer before the output in a deep neural network; and, (iii) all hidden layers φ ( z ) =
Revised: h last , which is the latent representation of the input x , one layer before the output in a deep neural network; and, (iii) all hidden layers φ ( z ) =

Original: h all , where h all =
Revised: h all , where h all =

Original: [ h 1 , h 2 , . . .
Revised: [ h 1 , h 2 , . . .

Original: , h last ] is the concatenation of all latent representations in the network.
Revised: , h last ] is the concatenation of all latent representations in the network.

Original: Note that the metrics with the identity map merely measure the similarity of the inputs without model information.
Revised: Note that the metrics with the identity map merely measure the similarity of the inputs without model information.

Original: We adopt these metrics as naive baselines to contrast with other advanced metrics that utilize the information of the models.
Revised: We adopt these metrics as naive baselines to contrast with other advanced metrics that utilize the information of the models.

Original: Gradient-based Metrics Gradient-based metrics use the gradient g z (cid:98) θ := ∇ θ (cid:96) ( z ; (cid:98) θ ) to measure the relevance.
Revised: Gradient-based Metrics Gradient-based metrics use the gradient g z (cid:98) θ := ∇ θ (cid:96) ( z ; (cid:98) θ ) to measure the relevance.

Original: We consider the ﬁve metrics: Inﬂuence Function (IF) (Koh & Liang, 2017), Relative IF (RIF) (Barshan et al., 2020)
Revised: We consider the ﬁve metrics: Inﬂuence Function (IF) (Koh & Liang, 2017), Relative IF (RIF) (Barshan et al., 2020)

Original: , Fisher Kernel (FK) (Khanna et al., 2019), Grad-Dot (GD)
Revised: , Fisher Kernel (FK) (Khanna et al., 2019), Grad-Dot (GD)

Original: (Yeh et al., 2018; Charpiat et al., 2019), and Grad-Cos (GC) (Perronnin et al., 2010; Charpiat et al., 2019).
Revised: (Yeh et al., 2018; Charpiat et al., 2019), and Grad-Cos (GC) (Perronnin et al., 2010; Charpiat et al., 2019).

Original: See Appendix A for the detail.
Revised: See Appendix A for the detail.

Original: • IF
Revised: • IF

Original: R IF ( z , z (cid:48) )
Revised: R IF ( z , z (cid:48) )

Original: := (cid:104)
Revised: := (cid:104)

Original: g z (cid:98) θ , H − 1 g z (cid:48) (cid:98) θ (cid:105) •
Revised: g z (cid:98) θ , H − 1 g z (cid:48) (cid:98) θ (cid:105) •

Original: RIF : R RIF ( z , z (cid:48) ) :
Revised: RIF : R RIF ( z , z (cid:48) ) :

Original: = cos( H − 12 g z (cid:98) θ , H − 12 g z (cid:48) (cid:98) θ )
Revised: = cos( H − 12 g z (cid:98) θ , H − 12 g z (cid:48) (cid:98) θ )

Original: • FK : R FK ( z , z (cid:48) ) :
Revised: • FK : R FK ( z , z (cid:48) ) :

Original: = (cid:104)
Revised: = (cid:104)

Original: g z (cid:98) θ , I − 1 g z (cid:48) (cid:98) θ (cid:105) ,
Revised: g z (cid:98) θ , I − 1 g z (cid:48) (cid:98) θ (cid:105) ,

Original: • GD : R GD ( z , z (cid:48) ) :
Revised: • GD : R GD ( z , z (cid:48) ) :

Original: = (cid:104)
Revised: = (cid:104)

Original: g z (cid:98) θ , g z (cid:48) (cid:98) θ (cid:105) •
Revised: g z (cid:98) θ , g z (cid:48) (cid:98) θ (cid:105) •

Original: GC : R GC ( z , z (cid:48) )
Revised: GC : R GC ( z , z (cid:48) )

Original: := cos( g z (cid:98) θ , g z (cid:48) (cid:98) θ )
Revised: := cos( g z (cid:98) θ , g z (cid:48) (cid:98) θ )

Original: where H and I are the Hessian and Fisher information matrices of the loss L train , respectively.
Revised: where H and I are the Hessian and Fisher information matrices of the loss L train , respectively.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Model-speciﬁc Explanation Aside of the relevance metrics
Revised: Model-speciﬁc Explanation Aside of the relevance metrics

Original: , there is another approach for similaritybased explanation that uses speciﬁc models that can provide explanations by its design (Kim et al., 2014; Plötz & Roth, 2018; Chen et al., 2019).
Revised: , there is another approach for similaritybased explanation that uses speciﬁc models that can provide explanations by its design (Kim et al., 2014; Plötz & Roth, 2018; Chen et al., 2019).

Original: We set aside these speciﬁc models and focus on generic relevance metrics because of their applicability to a wide range of problems.
Revised: We set aside these speciﬁc models and focus on generic relevance metrics because of their applicability to a wide range of problems.

Original: Evaluation of Metrics for Improving Classiﬁcation Accuracy In many machine learning problems, metrics between instances play an essential role.
Revised: Evaluation of Metrics for Improving Classiﬁcation Accuracy In many machine learning problems, metrics between instances play an essential role.

Original: For example, the distance between instances is essential for the distance-based methods such as nearest neighbor methods (Hastie et al., 2009).
Revised: For example, the distance between instances is essential for the distance-based methods such as nearest neighbor methods (Hastie et al., 2009).

Original: Another example is kernel models where the kernel function represents the relationship between two instances (Schölkopf et al., 2002).
Revised: Another example is kernel models where the kernel function represents the relationship between two instances (Schölkopf et al., 2002).

Original: There are several studies evaluating which metrics are desirable for speciﬁc tasks (Hussain et al., 2011;
Revised: There are several studies evaluating which metrics are desirable for speciﬁc tasks (Hussain et al., 2011;

Original: Hu et al., 2016; Li & Li, 2018;
Revised: Hu et al., 2016; Li & Li, 2018;

Original: Abu Alfeilat et al., 2019).
Revised: Abu Alfeilat et al., 2019).

Original: The goal of these studies is to ﬁnd metrics that improve classiﬁcation accuracy.
Revised: The goal of these studies is to ﬁnd metrics that improve classiﬁcation accuracy.

Original: Different from these evaluations based on accuracy, our goal in this study is to evaluate the validity of relevance metrics for similarity-based explanation; thus, the ﬁndings in these previous studies are not directly applicable to our goal.
Revised: Different from these evaluations based on accuracy, our goal in this study is to evaluate the validity of relevance metrics for similarity-based explanation; thus, the ﬁndings in these previous studies are not directly applicable to our goal.

Original: Evaluation of Explanations There are a variety of desiderata argued as requirements for explanations, such as faithfulness (Adebayo et al., 2018; Lakkaraju et al., 2019; Jacovi & Goldberg, 2020), plausibility (Lei et al., 2016; Lage et al., 2019;
Revised: Evaluation of Explanations There are a variety of desiderata argued as requirements for explanations, such as faithfulness (Adebayo et al., 2018; Lakkaraju et al., 2019; Jacovi & Goldberg, 2020), plausibility (Lei et al., 2016; Lage et al., 2019;

Original: Strout et al., 2019), robustness (Alvarez-Melis &
Revised: Strout et al., 2019), robustness (Alvarez-Melis &

Original: Jaakkola, 2018), and readability (Wang & Rudin, 2015; Yang et al., 2017; Angelino et al., 2017).
Revised: Jaakkola, 2018), and readability (Wang & Rudin, 2015; Yang et al., 2017; Angelino et al., 2017).

Original: Evaluating the existing explanation methods with respect to these requirements is of particular importance.
Revised: Evaluating the existing explanation methods with respect to these requirements is of particular importance.

Original: However, there is no standard test established for evaluating these requirements, and the designing of such tests still remains as an open problem (Doshi-Velez & Kim, 2017; Jacovi & Goldberg, 2020).
Revised: However, there is no standard test established for evaluating these requirements, and the designing of such tests still remains as an open problem (Doshi-Velez & Kim, 2017; Jacovi & Goldberg, 2020).

Original: In this study, as the ﬁrst empirical study for evaluating the existing relevance metrics for similarity-based explanation, we take an alternative approach by designing “minimal requirements” tests for the two primary requirements, faithfulness and plausibility, instead of jumping into the direct evaluation of these requirements.
Revised: In this study, as the ﬁrst empirical study for evaluating the existing relevance metrics for similarity-based explanation, we take an alternative approach by designing “minimal requirements” tests for the two primary requirements, faithfulness and plausibility, instead of jumping into the direct evaluation of these requirements.

Original: is cat because a similar is cat.
Revised: is cat because a similar is cat.

Original: (cid:52) is cat because a similar is dog.
Revised: (cid:52) is cat because a similar is dog.

Original: (cid:56)
Revised: (cid:56)

Original:  E VALUATION C RITERIA FOR S IMILARITY - BASED E XPLANATION
Revised:  E VALUATION C RITERIA FOR S IMILARITY - BASED E XPLANATION

Original: The goal of this study is to investigate which relevance metrics have desirable properties for similaritybased explanation.
Revised: The goal of this study is to investigate which relevance metrics have desirable properties for similaritybased explanation.

Original: In this section, we propose three tests to evaluate whether relevance metrics satisfy the minimal requirements for similarity-based explanation.
Revised: In this section, we propose three tests to evaluate whether relevance metrics satisfy the minimal requirements for similarity-based explanation.

Original: If a relevance metric fails one of the tests, we conclude that the metric do not meet the minimal requirements; thus, its use would be deprecated.
Revised: If a relevance metric fails one of the tests, we conclude that the metric do not meet the minimal requirements; thus, its use would be deprecated.

Original: The ﬁrst test assesses whether each relevance metric satisﬁes the minimal requirements for the faithfulness of explanation, which requires that an explanation to a model prediction must reﬂect the underlying inference process (Adebayo et al., 2018; Lakkaraju et al., 2019; Jacovi & Goldberg, 2020).
Revised: The ﬁrst test assesses whether each relevance metric satisﬁes the minimal requirements for the faithfulness of explanation, which requires that an explanation to a model prediction must reﬂect the underlying inference process (Adebayo et al., 2018; Lakkaraju et al., 2019; Jacovi & Goldberg, 2020).

Original: The latter two tests are designed to assess relevance metrics in terms of the plausibility of the explanations they produce (Lei et al., 2016; Lage et al., 2019;
Revised: The latter two tests are designed to assess relevance metrics in terms of the plausibility of the explanations they produce (Lei et al., 2016; Lage et al., 2019;

Original: Strout et al., 2019), which requires explanations to be sufﬁciently convincing to the users.
Revised: Strout et al., 2019), which requires explanations to be sufﬁciently convincing to the users.

Original: 3. M ODEL R ANDOMIZATION T EST
Revised: 3. M ODEL R ANDOMIZATION T EST

Original: Any explanations that are irrelevant to the model should be avoided because such fake explanations can mislead the users.
Revised: Any explanations that are irrelevant to the model should be avoided because such fake explanations can mislead the users.

Original: Thus, any valid relevance metric should be model-dependent, which constitutes the the ﬁrst requirement.
Revised: Thus, any valid relevance metric should be model-dependent, which constitutes the the ﬁrst requirement.

Original: We use the model randomization test of Adebayo et al.
Revised: We use the model randomization test of Adebayo et al.

Original: (2018) to assess whether the a given relevance metrics satisﬁes a minimal requirement for the faithfulness.
Revised: (2018) to assess whether the a given relevance metrics satisﬁes a minimal requirement for the faithfulness.

Original: If a relevance metric produces almost same explanations for the same inputs on two models with different reasoning processes, that metric is likely to ignore the underlying model, i.e., the metric is independent to the model.
Revised: If a relevance metric produces almost same explanations for the same inputs on two models with different reasoning processes, that metric is likely to ignore the underlying model, i.e., the metric is independent to the model.

Original: Thus, we can evaluate whether the metric is model-dependent or not by comparing explanations from two different models.
Revised: Thus, we can evaluate whether the metric is model-dependent or not by comparing explanations from two different models.

Original: A typical choice of the models is a well-trained model that can predict the output well and a randomly initialized model that can make only poor prediction.
Revised: A typical choice of the models is a well-trained model that can predict the output well and a randomly initialized model that can make only poor prediction.

Original: These two models have different reasoning processes, and hence the explanations should be different.
Revised: These two models have different reasoning processes, and hence the explanations should be different.

Original: Deﬁnition 2 (Model Randomization Test) .
Revised: Deﬁnition 2 (Model Randomization Test) .

Original: Let f and f rand be a well-trained model and a randomly initialized model, respectively.
Revised: Let f and f rand be a well-trained model and a randomly initialized model, respectively.

Original: For the test instance z test =
Revised: For the test instance z test =

Original: ( x test , (cid:98) y test ) and for the model f , let π f be a permutation of the training instances sorted based on the relevance metric R , i.e.,
Revised: ( x test , (cid:98) y test ) and for the model f , let π f be a permutation of the training instances sorted based on the relevance metric R , i.e.,

Original: R ( z test , z ( π f (1)) train ) ≥ R ( z test , z ( π f (2)) train ) ≥ . .
Revised: R ( z test , z ( π f (1)) train ) ≥ R ( z test , z ( π f (2)) train ) ≥ . .

Original: ≥ R ( z test , z ( π f ( N )) train ) .
Revised: ≥ R ( z test , z ( π f ( N )) train ) .

Original: We also deﬁne π f rand accordingly.
Revised: We also deﬁne π f rand accordingly.

Original: Then, we require π f and π f rand to have a small rank correlation.
Revised: Then, we require π f and π f rand to have a small rank correlation.

Original: If the relevance metric R is independent of the model, it produces the same permutation both for f and f rand and their rank correlation becomes one.
Revised: If the relevance metric R is independent of the model, it produces the same permutation both for f and f rand and their rank correlation becomes one.

Original: If the rank correlation is signiﬁcantly smaller than one and is close to zero, we can conﬁrm that the relevance metric is model-dependent.
Revised: If the rank correlation is signiﬁcantly smaller than one and is close to zero, we can conﬁrm that the relevance metric is model-dependent.

Original: 3. I DENTICAL C LASS T EST
Revised: 3. I DENTICAL C LASS T EST

Original: The second minimal requirement is that the raised similar instance should belong to the same class as the test instance, as shown in Figure 1.
Revised: The second minimal requirement is that the raised similar instance should belong to the same class as the test instance, as shown in Figure 1.

Original: The violation of this requirement leads to nonsense explanations such as “I think this image is cat because a similar image I saw in the past was dog.”
Revised: The violation of this requirement leads to nonsense explanations such as “I think this image is cat because a similar image I saw in the past was dog.”

Original: in
Revised: in

Original: Figure 1, which do not make sense to the users at all.
Revised: Figure 1, which do not make sense to the users at all.

Original: When faced with such an explanation, the users will ﬁnd the validity of the model’s prediction questionable, and will ignore the prediction even if the underlying model is valid.
Revised: When faced with such an explanation, the users will ﬁnd the validity of the model’s prediction questionable, and will ignore the prediction even if the underlying model is valid.

Original: This observation leads to the identical class test below.
Revised: This observation leads to the identical class test below.

Original: is animal because a similar is animal.
Revised: is animal because a similar is animal.

Original: (cid:52) cat cat is animal because a similar is animal.
Revised: (cid:52) cat cat is animal because a similar is animal.

Original: (cid:56) cat frog
Revised: (cid:56) cat frog

Original: Deﬁnition 3 (Identical Class Test) .
Revised: Deﬁnition 3 (Identical Class Test) .

Original: The most similar instance for the prediction of a test instance z test =
Revised: The most similar instance for the prediction of a test instance z test =

Original: ( x test , (cid:98) y test ) is a training instance of the same class as the given test instance.
Revised: ( x test , (cid:98) y test ) is a training instance of the same class as the given test instance.

Original: [Equation]
Revised: [Equation]

Original: Although this test may look trivial, some relevance metrics do not satisfy this minimal requirement, as demonstrated in Section 4.2.
Revised: Although this test may look trivial, some relevance metrics do not satisfy this minimal requirement, as demonstrated in Section 4.2.

Original: 3. I DENTICAL S UBCLASS T EST
Revised: 3. I DENTICAL S UBCLASS T EST

Original: The third minimal requirement is that the raised similar instance should belong to the same subclass as the test instance when the the classes consist of latent subclasses, as shown in Figure 2.
Revised: The third minimal requirement is that the raised similar instance should belong to the same subclass as the test instance when the the classes consist of latent subclasses, as shown in Figure 2.

Original: For example, consider a problem of classifying images of CIFAR10 to two classes, animal and vehicle.
Revised: For example, consider a problem of classifying images of CIFAR10 to two classes, animal and vehicle.

Original: The animal class consists of images from subclasses such as cat and frog, while the vehicle class consists of images from subclasses such as airplane and automobile.
Revised: The animal class consists of images from subclasses such as cat and frog, while the vehicle class consists of images from subclasses such as airplane and automobile.

Original: Under the presence of subclasses, the violation of this third requirement again leads to nonsense explanations such as “I think this image (cat) is animal because a similar image (frog) I saw in the past was also animal.”
Revised: Under the presence of subclasses, the violation of this third requirement again leads to nonsense explanations such as “I think this image (cat) is animal because a similar image (frog) I saw in the past was also animal.”

Original: in Figure 2.
Revised: in Figure 2.

Original: This observation leads to the identical subclass test below.
Revised: This observation leads to the identical subclass test below.

Original: Deﬁnition 4 (Identical Subclass Test) .
Revised: Deﬁnition 4 (Identical Subclass Test) .

Original: Let s ( z ) be its subclass for an instance
Revised: Let s ( z ) be its subclass for an instance

Original: z =
Revised: z =

Original: ( x , y ) .
Revised: ( x , y ) .

Original: Here, the most similar instance for the prediction of a test instance z test =
Revised: Here, the most similar instance for the prediction of a test instance z test =

Original: ( x test , (cid:98) y test ) with a correct prediction (cid:98) y test =
Revised: ( x test , (cid:98) y test ) with a correct prediction (cid:98) y test =

Original: y test is the training instance of the same subclass as the test instance.
Revised: y test is the training instance of the same subclass as the test instance.

Original: [Equation]
Revised: [Equation]

Original: Note that the identical subclass test depends on whether the model has obtained internal representations that can distinguish between subclasses.
Revised: Note that the identical subclass test depends on whether the model has obtained internal representations that can distinguish between subclasses.

Original: If the model mixes up subclasses in its reasoning process, any explanation will fail the test.
Revised: If the model mixes up subclasses in its reasoning process, any explanation will fail the test.

Original: To conﬁrm the validity of the identical subclass test in our experiments, we veriﬁed that all the models we used satisﬁed this condition.
Revised: To conﬁrm the validity of the identical subclass test in our experiments, we veriﬁed that all the models we used satisﬁed this condition.

Original: See Appendix E.
Revised: See Appendix E.

Original:  E VALUATION R ESULTS
Revised:  E VALUATION R ESULTS

Original: Here, we examine the validity of the relevance metrics with respect to the three minimal requirements.
Revised: Here, we examine the validity of the relevance metrics with respect to the three minimal requirements.

Original: For this evaluation, we used two image datasets (MNIST (LeCun et al., 1998), CIFAR10 (Krizhevsky, 2009)), two text datasets (TREC (Li & Roth, 2002), AGNews (Zhang et al., 2015)) and one table dataset (Vehicle (Dua & Graff,  2017)).
Revised: For this evaluation, we used two image datasets (MNIST (LeCun et al., 1998), CIFAR10 (Krizhevsky, 2009)), two text datasets (TREC (Li & Roth, 2002), AGNews (Zhang et al., 2015)) and two table datasets (Vehicle (Dua & Graff, 2017), Segment (Dua & Graff, 2017)).

Original: As benchmarks, we employed logistic regression and deep neural networks trained on these datasets.
Revised: As benchmarks, we employed logistic regression and deep neural networks trained on these datasets.

Original: The details of the datasets, models, and computing infrastructure we used are given in Appendix B.
Revised: The details of the datasets, models, and computing infrastructure we used are given in Appendix B.

Original: Procedure We repeated the following procedure for 10 times for each evaluation test.
Revised: Procedure We repeated the following procedure for 10 times for each evaluation test.

Original: Train a model using a subset of the training instances.
Revised: Train a model using a subset of the training instances.

Original: Then, randomly sample 500 test instances from the test set.
Revised: Then, randomly sample 500 test instances from the test set.

Original: For each test instance, compute the relevance score for all the instances used for training.
Revised: For each test instance, compute the relevance score for all the instances used for training.

Original: (a) For model randomization test, compute the Spearman rank correlation coefﬁcients between the relevance scores from the trained model and relevance scores from the randomized model.
Revised: (a) For model randomization test, compute the Spearman rank correlation coefﬁcients between the relevance scores from the trained model and relevance scores from the randomized model.

Original: (cid:96) x 2 (cid:96) l a s t 2 (cid:96) a
Revised: (cid:96) x 2 (cid:96) l a s t 2 (cid:96) a

Original: ll 2 c
Revised: ll 2 c

Original: o s x c o s l a s t
Revised: o s x c o s l a s t

Original: c o s a ll d o t x d o t l a s t d o t a ll I F R I FF KGDG C 0 .
Revised: c o s a ll d o t x d o t l a s t d o t a ll I F R I FF KGDG C 0 .

Original: A v e r a g e C
Revised: A v e r a g e C

Original: o rr e l a ti on ± S t d .
Revised: o rr e l a ti on ± S t d .

Original: (cid:96)
Revised: (cid:96)

Original: Cosine Dot
Revised: Cosine Dot

Original: Gradient (cid:96) x 2 (cid:96) l a s t 2 (cid:96) a ll 2 c o s x c o s l a s t c o s a ll d o t x d o t l a s t d o t a ll
Revised: Gradient (cid:96) x 2 (cid:96) l a s t 2 (cid:96) a ll 2 c o s x c o s l a s t c o s a ll d o t x d o t l a s t d o t a ll

Original: I F R I FF KGDG C 0 .
Revised: I F R I FF KGDG C 0 .

Original: A v e r a g e C
Revised: A v e r a g e C

Original: o rr e l a ti on ± S t d .
Revised: o rr e l a ti on ± S t d .

Original: (cid:96)
Revised: (cid:96)

Original: Cosine Dot
Revised: Cosine Dot

Original: Gradient
Revised: Gradient

Original: In this section, we mainly present the results for CIFAR10 with CNN, and AGNews with Bi-LSTM.
Revised: In this section, we mainly present the results for CIFAR10 with CNN, and AGNews with Bi-LSTM.

Original: The other results were similar, and can be found in Appendix F.
Revised: The other results were similar, and can be found in Appendix F.

Original: Summary of the Results We summarize the main results prior to discussing individual result.
Revised: Summary of the Results We summarize the main results prior to discussing individual result.

Original: • (cid:96) last2 , cos last , and the gradient-based metrics scored law correlation in the model randomization test for all the datasets and models, indicating that they are model-dependent.
Revised: • (cid:96) last2 , cos last , and the gradient-based metrics scored law correlation in the model randomization test for all the datasets and models, indicating that they are model-dependent.

Original: • GC performed the best in most of the identical class test and identical subclass test; thus, GC would be the recommended choice in practice.
Revised: • GC performed the best in most of the identical class test and identical subclass test; thus, GC would be the recommended choice in practice.

Original: • Dot metrics as well as IF, FK, and GD performed poorly on the identical class test and identical subclass test.
Revised: • Dot metrics as well as IF, FK, and GD performed poorly on the identical class test and identical subclass test.

Original: In Section 5, we analyze why some relevance metrics are successful and why some are not in the identical class test and identical subclass test.
Revised: In Section 5, we analyze why some relevance metrics are successful and why some are not in the identical class test and identical subclass test.

Original: 4. R ESULT OF M ODEL R ANDOMIZATION T EST
Revised: 4. R ESULT OF M ODEL R ANDOMIZATION T EST

Original: Figure 3 shows the Spearman rank correlation coefﬁcients for the model randomization test.
Revised: Figure 3 shows the Spearman rank correlation coefﬁcients for the model randomization test.

Original: The similarities with the identity feature map (cid:96) x 2 , cos x , and dot x are irrelevant to the model and their correlations are trivially one.
Revised: The similarities with the identity feature map (cid:96) x 2 , cos x , and dot x are irrelevant to the model and their correlations are trivially one.

Original: In the ﬁgures, the other metrics scored correlations close to zero, indicating they will be model-dependent.
Revised: In the ﬁgures, the other metrics scored correlations close to zero, indicating they will be model-dependent.

Original: However, on MNIST and Vehicle datasets (see Appendix F), we observed that (cid:96) all2 , cos all , dot last , and dot all scored correlations more than 0.7.
Revised: However, on MNIST and Vehicle datasets (see Appendix F), we observed that (cid:96) all2 , cos all , dot last , and dot all scored correlations more than 0.7.

Original: We therefore concluded that these relevance metrics failed the model randomization test because they can raise instances irrelevant to the model for some datasets.
Revised: We therefore concluded that these relevance metrics failed the model randomization test because they can raise instances irrelevant to the model for some datasets.

Original: 4. R ESULTS OF I DENTICAL C LASS T EST AND I DENTICAL S UBCLASS T EST
Revised: 4. R ESULTS OF I DENTICAL C LASS T EST AND I DENTICAL S UBCLASS T EST

Original: For the identical subclass test, we used modiﬁed datasets: we split the dataset into two classes by randomly assigning existing classes to one of the two classes “A” and “B”.
Revised: For the identical subclass test, we used modiﬁed datasets: we split the dataset into two classes by randomly assigning existing classes to one of the two classes “A” and “B”.

Original: The new classes “A” and “B” now contain the original data classes as subclasses, which can be used for the identical subclass test.
Revised: The new classes “A” and “B” now contain the original data classes as subclasses, which can be used for the identical subclass test.

Original: We sampled 500 instances with correct predictions from the test set as test instances.
Revised: We sampled 500 instances with correct predictions from the test set as test instances.

Original: Figure 4 shows the success rates for the identical class test and identical subclass test.
Revised: Figure 4 shows the success rates for the identical class test and identical subclass test.

Original: We also summarized the average success rates of all of our experiments in Table 1.
Revised: We also summarized the average success rates of all of our experiments in Table 1.

Original: A remarkable observation is that GC performed consistently well both on the identical class test and identical subclass test for all the datasets and the models we used in the experiment (see Appendix F).
Revised: A remarkable observation is that GC performed consistently well both on the identical class test and identical subclass test for all the datasets and the models we used in the experiment (see Appendix F).

Original: By contrast, some relevance metrics such as dot metrics as well as IF, FK, and GD performed poorly on both of the tests.
Revised: By contrast, some relevance metrics such as dot metrics as well as IF, FK, and GD performed poorly on both of the tests.

Original: We investigate the reason of their failure in the next section.
Revised: We investigate the reason of their failure in the next section.

Original: To conclude, the results of our evaluations indicate that only GC performed well on all the tests.
Revised: To conclude, the results of our evaluations indicate that only GC performed well on all the tests.

Original: That is, only GC seems to meet the minimal requirements; thus, it would be a recommended choice for similarity-based explanation.
Revised: That is, only GC seems to meet the minimal requirements; thus, it would be a recommended choice for similarity-based explanation.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: e
Revised: e

Original: [Figure]
Revised: [Figure]

Original: y
Revised: y

Original: [Figure]
Revised: [Figure]

Original:  W HY SOME METRICS ARE SUCCESSFUL AND WHY SOME ARE NOT
Revised:  W HY SOME METRICS ARE SUCCESSFUL AND WHY SOME ARE NOT

Original: We observed that the dot metrics and the gradient-based metrics such as IF, FK, and GD, failed the identical class test and identical subclass test, in contrast to GC who performed remarkably well.
Revised: We observed that the dot metrics and the gradient-based metrics such as IF, FK, and GD, failed the identical class test and identical subclass test, in contrast to GC who performed remarkably well.

Original: Here, we analyze the reason why the above relevance metrics failed and why GC performed well.
Revised: Here, we analyze the reason why the above relevance metrics failed and why GC performed well.

Original: In
Revised: In

Original: Appendix D, we also discuss a way to repair IF, FK, and GD to improve their performances based on the ﬁndings in this section.
Revised: Appendix D, we also discuss a way to repair IF, FK, and GD to improve their performances based on the ﬁndings in this section.

Original: Failure of the Dot Metrics and Gradient-based Metrics To understand the failure, we reformulate IF, FK, and GD as dot metrics of the form R dot ( z test , z train ) =
Revised: Failure of the Dot Metrics and Gradient-based Metrics To understand the failure, we reformulate IF, FK, and GD as dot metrics of the form R dot ( z test , z train ) =

Original: (cid:104) φ ( z test ) , φ ( z train ) (cid:105) so that the following discussion to hold true for any relevance metrics in this form.
Revised: (cid:104) φ ( z test ) , φ ( z train ) (cid:105) so that the following discussion to hold true for any relevance metrics in this form.

Original: It is easy to see that IF,
Revised: It is easy to see that IF,

Original: FK, and GD can be expressed in this form by deﬁning the feature maps by φ ( z ) =
Revised: FK, and GD can be expressed in this form by deﬁning the feature maps by φ ( z ) =

Original: H − 1 / 2 g ( z ; (cid:98) θ ) , φ ( z ) =
Revised: H − 1 / 2 g ( z ; (cid:98) θ ) , φ ( z ) =

Original: I − 1 / 2 g
Revised: I − 1 / 2 g

Original: ( z ; (cid:98) θ ) , and φ ( z ) =
Revised: ( z ; (cid:98) θ ) , and φ ( z ) =

Original: g ( z ; (cid:98) θ ) , respectively.
Revised: g ( z ; (cid:98) θ ) , respectively.

Original: Given a criterion, let z ( i ) train be a desirable instance for a test instance z test .
Revised: Given a criterion, let z ( i ) train be a desirable instance for a test instance z test .

Original: The failures of the dot metrics indicate the existence of an undesirable instance z ( j ) train such that (cid:104) φ ( z test ) , φ ( z ( i ) train ) (cid:105) < (cid:104) φ ( z test ) , φ ( z ( j ) train ) (cid:105) .
Revised: The failures of the dot metrics indicate the existence of an undesirable instance z ( j ) train such that (cid:104) φ ( z test ) , φ ( z ( i ) train ) (cid:105) < (cid:104) φ ( z test ) , φ ( z ( j ) train ) (cid:105) .

Original: The following sufﬁcient condition for z ( j ) train is useful to understand the failure.
Revised: The following sufﬁcient condition for z ( j ) train is useful to understand the failure.

Original: [Equation]
Revised: [Equation]

Original: The condition implies that any instance with an extremely large norm and cosine slightly larger than zero can be the candidate of z ( j ) train .
Revised: The condition implies that any instance with an extremely large norm and cosine slightly larger than zero can be the candidate of z ( j ) train .

Original: In our experiments, we observed that the condition on the norm is especially crucial.
Revised: In our experiments, we observed that the condition on the norm is especially crucial.

Original: Indeed, as we can see in Figure 5, even though instances with extremely large norms were scarce, only such extreme instances were selected as relevant instances by IF, FK, and
Revised: Indeed, as we can see in Figure 5, even though instances with extremely large norms were scarce, only such extreme instances were selected as relevant instances by IF, FK, and

Original: GD.
Revised: GD.

Original: That is, these metrics tend to consider such extreme instances as relevant ones.
Revised: That is, these metrics tend to consider such extreme instances as relevant ones.

Original: By contrast, GC was not attracted by large norms because it completely cancels out the norm through normalization.
Revised: By contrast, GC was not attracted by large norms because it completely cancels out the norm through normalization.

Original: Figure 6 shows some training instances frequently selected in the identical class test on CIFAR with CNN.
Revised: Figure 6 shows some training instances frequently selected in the identical class test on CIFAR with CNN.

Original: When using IF, FK, and GD, these training instances were frequently selected regardless of their classes, simply because the training instances had large norms.
Revised: When using IF, FK, and GD, these training instances were frequently selected regardless of their classes, simply because the training instances had large norms.

Original: In these metrics, the term cos( φ ( z test ) , φ ( z train )) seems to have only negligible effects.
Revised: In these metrics, the term cos( φ ( z test ) , φ ( z train )) seems to have only negligible effects.

Original: By contrast, GC successfully selected instances of the same class and ignored instances with large norms.
Revised: By contrast, GC successfully selected instances of the same class and ignored instances with large norms.

Original: Test Instances
Revised: Test Instances

Original: Found IF airplane frog bird truck cos( z test , z train ) (cid:107) φ ( z train ) (cid:107) .
Revised: Found IF airplane frog bird truck cos( z test , z train ) (cid:107) φ ( z train ) (cid:107) .

Original: Test Instances
Revised: Test Instances

Original: Found FK cat ship bird ship cos( z test , z train ) (cid:107) φ ( z train ) (cid:107) .
Revised: Found FK cat ship bird ship cos( z test , z train ) (cid:107) φ ( z train ) (cid:107) .

Original: Test Instances
Revised: Test Instances

Original: Found GD cat bird horse truck cos( z
Revised: Found GD cat bird horse truck cos( z

Original: [Figure]
Revised: [Figure]

Original: , z train ) (cid:107) φ ( z train ) (cid:107) .
Revised: , z train ) (cid:107) φ ( z train ) (cid:107) .

Original: Test Instances
Revised: Test Instances

Original: Found GC truck truck truck truck cos( z test , z train ) (cid:107) φ ( z train ) (cid:107) .
Revised: Found GC truck truck truck truck cos( z test , z train ) (cid:107) φ ( z train ) (cid:107) .

Original: Success of GC We now analyze why GC performed well in particular in the identical class test.
Revised: Success of GC We now analyze why GC performed well in particular in the identical class test.

Original: To simplify the discussion, we consider linear logistic regression whose conditional distribution p ( y | x ; θ ) is given by the y -th entry of σ ( W x ) , where σ is the softmax function, θ =
Revised: To simplify the discussion, we consider linear logistic regression whose conditional distribution p ( y | x ; θ ) is given by the y -th entry of σ ( W x ) , where σ is the softmax function, θ =

Original: W ∈ R C × d , and C and d are the number of classes and the dimensionality of x , respectively.
Revised: W ∈ R C × d , and C and d are the number of classes and the dimensionality of x , respectively.

Original: With some algebra, we obtain R GC ( z , z (cid:48) )
Revised: With some algebra, we obtain R GC ( z , z (cid:48) )

Original: =
Revised: =

Original: cos( r z , r z (cid:48) ) cos( x , x (cid:48) ) for z = ( x , y ) and z (cid:48) =
Revised: cos( r z , r z (cid:48) ) cos( x , x (cid:48) ) for z = ( x , y ) and z (cid:48) =

Original: ( x (cid:48) , y (cid:48) ) , where r z = σ ( W x ) − e y is the residual for the prediction on z and e y is a vector whose y -th entry is one and zero otherwise.
Revised: ( x (cid:48) , y (cid:48) ) , where r z = σ ( W x ) − e y is the residual for the prediction on z and e y is a vector whose y -th entry is one and zero otherwise.

Original: See Appendix C for the derivation.
Revised: See Appendix C for the derivation.

Original: Here, the term cos( r z , r z (cid:48) ) plays an essential role in GC.
Revised: Here, the term cos( r z , r z (cid:48) ) plays an essential role in GC.

Original: From the deﬁnition, r z c ≤ 0 if c = y and r z c ≥ 0 otherwise.
Revised: From the deﬁnition, r z c ≤ 0 if c = y and r z c ≥ 0 otherwise.

Original: Thus, cos( r z , r z (cid:48) ) ≥always holds true when y = y (cid:48) , while cos( r z , r z (cid:48) ) can be negative for y (cid:54) =
Revised: Thus, cos( r z , r z (cid:48) ) ≥always holds true when y = y (cid:48) , while cos( r z , r z (cid:48) ) can be negative for y (cid:54) =

Original: y (cid:48) .
Revised: y (cid:48) .

Original: Hence, the chance of
Revised: Hence, the chance of

Original: R GC ( z , z (cid:48) ) being positive can be larger for the instances from the same class compared to the ones from a different class.
Revised: R GC ( z , z (cid:48) ) being positive can be larger for the instances from the same class compared to the ones from a different class.

Original: Figure 7 shows that cos( r z , r z (cid:48) ) is essential also for deep neural networks.
Revised: Figure 7 shows that cos( r z , r z (cid:48) ) is essential also for deep neural networks.

Original: Here, for each test instance z test on CIFAR10 with CNN, we randomly sampled two training instances z train , one with the same class and the other with a different class, and computed R GC ( z test , z train ) and cos( r z test , r z train ) .
Revised: Here, for each test instance z test on CIFAR10 with CNN, we randomly sampled two training instances z train , one with the same class and the other with a different class, and computed R GC ( z test , z train ) and cos( r z test , r z train ) .

Original: [Figure]
Revised: [Figure]

Original: We also note that cos( r z test , r z train ) alone was not helpful for the identical subclass test, whose success rate was around the chance level.
Revised: We also note that cos( r z test , r z train ) alone was not helpful for the identical subclass test, whose success rate was around the chance level.

Original: We thus conjecture that, while cos( r z test , r z train ) is particularly helpful for the identical class test, the use of the entire gradient is still essential for GC to work well.
Revised: We thus conjecture that, while cos( r z test , r z train ) is particularly helpful for the identical class test, the use of the entire gradient is still essential for GC to work well.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We investigated which of the relevance metrics are effective for similarity-based explanation.
Revised: We investigated which of the relevance metrics are effective for similarity-based explanation.

Original: For this purpose, we evaluated whether the metrics satisfy the minimal requirements for similarity-based explanation.
Revised: For this purpose, we evaluated whether the metrics satisfy the minimal requirements for similarity-based explanation.

Original: In this study, we conducted three tests: the model randomization test of Adebayo et al.
Revised: In this study, we conducted three tests: the model randomization test of Adebayo et al.

Original: (2018) to evaluate whether the metrics are model-dependent, and the two newly designed tests, the identical class test and identical subclass test, to evaluate whether the metrics can provide plausible explanation.
Revised: (2018) to evaluate whether the metrics are model-dependent, and the two newly designed tests, the identical class test and identical subclass test, to evaluate whether the metrics can provide plausible explanation.

Original: Our quantitative evaluations based on the three tests revealed that the cosine similarity of the gradients performs best, which would be a recommended choice in practice.
Revised: Our quantitative evaluations based on the three tests revealed that the cosine similarity of the gradients performs best, which would be a recommended choice in practice.

Original: We also observed that some relevance metrics do not meet the requirements; thus, the use of such metrics would not be appropriate for similarity-based explanation.
Revised: We also observed that some relevance metrics do not meet the requirements; thus, the use of such metrics would not be appropriate for similarity-based explanation.

Original: We expect our insights to help practitioners to select appropriate relevance metrics, and also to help further researches for designing better relevance metrics for instance-based explanations.
Revised: We expect our insights to help practitioners to select appropriate relevance metrics, and also to help further researches for designing better relevance metrics for instance-based explanations.

Original: A G RADIENT - BASED M ETRICS
Revised: A G RADIENT - BASED M ETRICS

Original: In gradient-based metrics, we consider a model with parameter θ , its loss (cid:96) ( z ; θ ) , and its gradient ∇ θ (cid:96) ( z ; θ ) to measure relevance, where z = ( x , y ) is an input-output pair.
Revised: In gradient-based metrics, we consider a model with parameter θ , its loss (cid:96) ( z ; θ ) , and its gradient ∇ θ (cid:96) ( z ; θ ) to measure relevance, where z = ( x , y ) is an input-output pair.

Original: Inﬂuence Function (Koh & Liang, 2017) Koh & Liang (2017) proposed to measure relevance according to “how largely the test loss will increase if the training instance is omitted from the training set.”
Revised: Inﬂuence Function (Koh & Liang, 2017) Koh & Liang (2017) proposed to measure relevance according to “how largely the test loss will increase if the training instance is omitted from the training set.”

Original: Here, the model parameter trained using all of the training set is denoted by (cid:98) θ , and the parameter trained using all of the training set except the i -th instance z ( i ) train is denoted by (cid:98) θ − i .
Revised: Here, the model parameter trained using all of the training set is denoted by (cid:98) θ , and the parameter trained using all of the training set except the i -th instance z ( i ) train is denoted by (cid:98) θ − i .

Original: The relevance metric proposed by Koh & Liang (2017) is then deﬁned as the difference between the test loss under parameters (cid:98) θ and (cid:98) θ − i as follows:
Revised: The relevance metric proposed by Koh & Liang (2017) is then deﬁned as the difference between the test loss under parameters (cid:98) θ and (cid:98) θ − i as follows:

Original: [Equation]
Revised: [Equation]

Original: Here, a greater value indicates that the loss on the test instance increases drastically by removing the i -th training instance from the training set.
Revised: Here, a greater value indicates that the loss on the test instance increases drastically by removing the i -th training instance from the training set.

Original: Thus, the i -th training instance is essential relative to predicting the test instance; therefore, it is highly relevant.
Revised: Thus, the i -th training instance is essential relative to predicting the test instance; therefore, it is highly relevant.

Original: In practice, the following approximation is used to avoid computing (cid:98) θ − i explicitly.
Revised: In practice, the following approximation is used to avoid computing (cid:98) θ − i explicitly.

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where H is the Hessian matrix of the loss L train .
Revised: where H is the Hessian matrix of the loss L train .

Original: Relative IF
Revised: Relative IF

Original: (Barshan et al., 2020)
Revised: (Barshan et al., 2020)

Original: Barshan et al.
Revised: Barshan et al.

Original: (2020) proposed to measure relevance according to “how largely the test loss will increase if the training instance is omitted from the training set under the constraint that the expected squared change in loss is sufﬁciently small” 3 , which is the modiﬁed version of the inﬂuence function.
Revised: (2020) proposed to measure relevance according to “how largely the test loss will increase if the training instance is omitted from the training set under the constraint that the expected squared change in loss is sufﬁciently small” 3 , which is the modiﬁed version of the inﬂuence function.

Original: Relative IF is computed as the cosine similarity of φ ( z ) =
Revised: Relative IF is computed as the cosine similarity of φ ( z ) =

Original: H − 1 / 2 ∇ θ (cid:96) ( z ; (cid:98) θ ) :
Revised: H − 1 / 2 ∇ θ (cid:96) ( z ; (cid:98) θ ) :

Original: [Equation]
Revised: [Equation]

Original: Fisher Kernel (Khanna et al., 2019)
Revised: Fisher Kernel (Khanna et al., 2019)

Original: Khanna et al.
Revised: Khanna et al.

Original: (2019) proposed to measure the relevance of instances using the Fisher kernel as follows:
Revised: (2019) proposed to measure the relevance of instances using the Fisher kernel as follows:

Original: [Equation]
Revised: [Equation]

Original: where I is the Fisher information matrix of the loss L train .
Revised: where I is the Fisher information matrix of the loss L train .

Original: Grad-Dot, Grad-Cos
Revised: Grad-Dot, Grad-Cos

Original: (Perronnin et al., 2010;
Revised: (Perronnin et al., 2010;

Original: Yeh et al., 2018; Charpiat et al., 2019)
Revised: Yeh et al., 2018; Charpiat et al., 2019)

Original: Charpiat et al.
Revised: Charpiat et al.

Original: (2019) proposed to measure relevance according to “how largely the loss will decrease when a small update is added to the model using the training instance.”
Revised: (2019) proposed to measure relevance according to “how largely the loss will decrease when a small update is added to the model using the training instance.”

Original: This can be computed as the dot product of the loss gradients, which we refer to as Grad-Dot.
Revised: This can be computed as the dot product of the loss gradients, which we refer to as Grad-Dot.

Original: [Equation]
Revised: [Equation]

Original: Note that a similar metric is studied by Yeh et al.
Revised: Note that a similar metric is studied by Yeh et al.

Original: as the representer point value .
Revised: as the representer point value .

Original: As a modiﬁcation of Grad-Dot, Charpiat et al.
Revised: As a modiﬁcation of Grad-Dot, Charpiat et al.

Original: (2019) also proposed the following cosine version, which we refer to as Grad-Cos.
Revised: (2019) also proposed the following cosine version, which we refer to as Grad-Cos.

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Note that the use of the cosine between the gradients is also proposed by Perronnin et al. (2010).
Revised: Note that the use of the cosine between the gradients is also proposed by Perronnin et al. (2010).

Original: B E XPERIMENTAL S ETUP B. D ATASETS AND M ODELS
Revised: B E XPERIMENTAL S ETUP B. D ATASETS AND M ODELS

Original: MNIST (LeCun et al., 1998)
Revised: MNIST (LeCun et al., 1998)

Original: The MNIST dataset is used for handwritten digit image classiﬁcation tasks.
Revised: The MNIST dataset is used for handwritten digit image classiﬁcation tasks.

Original: Here, input x is an image of a handwritten digit, and the output y consists of 10 classes (“0” to “9”).
Revised: Here, input x is an image of a handwritten digit, and the output y consists of 10 classes (“0” to “9”).

Original: We adopted logistic regression and a CNN as the classiﬁcation models.
Revised: We adopted logistic regression and a CNN as the classiﬁcation models.

Original: The CNN has six convolutional layers, and max-pooling layers for each two convolutional layers.
Revised: The CNN has six convolutional layers, and max-pooling layers for each two convolutional layers.

Original: The features obtained by these layers are fed into the global average pooling layer followed by a single linear layer.
Revised: The features obtained by these layers are fed into the global average pooling layer followed by a single linear layer.

Original: The number of the output channels of all the convolutional layers is set to 16.
Revised: The number of the output channels of all the convolutional layers is set to 16.

Original: We trained the models using the Adam optimizer with a learning rate of 0.001.
Revised: We trained the models using the Adam optimizer with a learning rate of 0.001.

Original: We used randomly sampled 5,500 training instances to train the models.
Revised: We used randomly sampled 5,500 training instances to train the models.

Original: CIFAR10 (Krizhevsky, 2009)
Revised: CIFAR10 (Krizhevsky, 2009)

Original: The CIFAR10 dataset is used for object recognition tasks.
Revised: The CIFAR10 dataset is used for object recognition tasks.

Original: Here, input x is an image containing a certain object, and output y consists of 10 classes, e.g., “bird” or “airplane.”
Revised: Here, input x is an image containing a certain object, and output y consists of 10 classes, e.g., “bird” or “airplane.”

Original: Note that we used the same models as for the MNIST dataset.
Revised: Note that we used the same models as for the MNIST dataset.

Original: In addition, we adopted
Revised: In addition, we adopted

Original: MobileNetV2 (Sandler et al., 2018) as a model with a higher performance than the previous model.
Revised: MobileNetV2 (Sandler et al., 2018) as a model with a higher performance than the previous model.

Original: We trained the models using the Adam optimizer with a learning rate of 0.001.
Revised: We trained the models using the Adam optimizer with a learning rate of 0.001.

Original: In the experiments, we ﬁrst pre-trained the models using all the training instances of CIFAR10, and then trained the models using randomly sampled 5,000 training instances.
Revised: In the experiments, we ﬁrst pre-trained the models using all the training instances of CIFAR10, and then trained the models using randomly sampled 5,000 training instances.

Original: Without the pre-training, the classiﬁcation performance of the models dropped signiﬁcantly.
Revised: Without the pre-training, the classiﬁcation performance of the models dropped signiﬁcantly.

Original: Note that we did not examine IF and FK on MobileNetV2 because the matrix inverse in these metrics required too much time to calculate even with the conjugate gradient approximation proposed by Koh & Liang (2017).
Revised: Note that we did not examine IF and FK on MobileNetV2 because the matrix inverse in these metrics required too much time to calculate even with the conjugate gradient approximation proposed by Koh & Liang (2017).

Original: TREC (Li & Roth, 2002)
Revised: TREC (Li & Roth, 2002)

Original: The TREC dataset is used for question classiﬁcation tasks.
Revised: The TREC dataset is used for question classiﬁcation tasks.

Original: Here, input x is a question sentence, and output y is a question category consisting of six classes, e.g., “LOC” and “NUM.”
Revised: Here, input x is a question sentence, and output y is a question category consisting of six classes, e.g., “LOC” and “NUM.”

Original: We used bag-of-words logistic regression and a two-layer Bi-LSTM as the classiﬁcation models.
Revised: We used bag-of-words logistic regression and a two-layer Bi-LSTM as the classiﬁcation models.

Original: In the Bi-LSTM, the last state is fed into one linear layer.
Revised: In the Bi-LSTM, the last state is fed into one linear layer.

Original: The word embedding dimension is set to 16, and the dimension of the LSTM is set to 16 also.
Revised: The word embedding dimension is set to 16, and the dimension of the LSTM is set to 16 also.

Original: We trained the models using the Adam optimizer with a learning rate of 0.001.
Revised: We trained the models using the Adam optimizer with a learning rate of 0.001.

Original: We used randomly sampled 2,726 training instances to train the models.
Revised: We used randomly sampled 2,726 training instances to train the models.

Original: AGNews (Zhang et al., 2015)
Revised: AGNews (Zhang et al., 2015)

Original: The AGNews dataset is used for news article classiﬁcation tasks.
Revised: The AGNews dataset is used for news article classiﬁcation tasks.

Original: Here, input x is a sentence, and output y is a category comprising four classes, e.g., “business” and “sports.”
Revised: Here, input x is a sentence, and output y is a category comprising four classes, e.g., “business” and “sports.”

Original: We used the same models as TREC.
Revised: We used the same models as TREC.

Original: We trained the models using the Adam optimizer with a learning rate of 0.001.
Revised: We trained the models using the Adam optimizer with a learning rate of 0.001.

Original: We used randomly sampled 6,000 training instances to train the models.
Revised: We used randomly sampled 6,000 training instances to train the models.

Original: Vehicle (Dua & Graff, 2017)
Revised: Vehicle (Dua & Graff, 2017)

Original: The vehicle dataset is used for vehicle type classiﬁcation tasks.
Revised: The vehicle dataset is used for vehicle type classiﬁcation tasks.

Original: Here, the input x consists of 18 features, and the output y is a type of vehicle comprising four classes, e.g., “bus” and “van.”
Revised: Here, the input x consists of 18 features, and the output y is a type of vehicle comprising four classes, e.g., “bus” and “van.”

Original: We used logistic regression and a three-layer MLP as the classiﬁcation models.
Revised: We used logistic regression and a three-layer MLP as the classiﬁcation models.

Original: We trained the models using the Adam optimizer with a learning rate of 0.001.
Revised: We trained the models using the Adam optimizer with a learning rate of 0.001.

Original: We used randomly sampled 423 training instances to train the models.
Revised: We used randomly sampled 423 training instances to train the models.

Original: B. C OMPUTING I NFRASTRUCTURE
Revised: Segment (Dua & Graff, 2017) The segment dataset is used for image classiﬁcation tasks. Here, the input x consists of 19 features, and the output y consists of seven classes, e.g., “sky” and “window.” We used the same models as Vehicle.

Revised: We trained the models using the Adam optimizer with a learning rate of 0.001.

Revised: We used randomly sampled 924 training instances to train the models.

Revised: B. C OMPUTING I NFRASTRUCTURE

Original: In our experiments, training of the models was run on a NVIDIA GTX 

Revised: In our experiments, training of the models was run on a NVIDIA GTX

Original: GPU with Intel Xeon
Revised: GPU with Intel Xeon

Original: Silver 4112 CPU and 64GB RAM.
Revised: Silver 4112 CPU and 64GB RAM.

Original: Testing and computing relevance metrics were run on Xeon
Revised: Testing and computing relevance metrics were run on Xeon

Original: E5-2680 v2 CPU with 256GB RAM.
Revised: E5-2680 v2 CPU with 256GB RAM.

Original: C D ERIVATION OF GC FOR L INEAR L OGISTIC R EGRESSION
Revised: C D ERIVATION OF GC FOR L INEAR L OGISTIC R EGRESSION

Original: We consider linear logistic regression whose conditional distribution p ( y | x ; θ ) is given by the y -th entry of σ ( W x ) , where σ is the softmax function, θ =
Revised: We consider linear logistic regression whose conditional distribution p ( y | x ; θ ) is given by the y -th entry of σ ( W x ) , where σ is the softmax function, θ =

Original: W ∈ R C × d , and C and d are the number of classes and the dimensionality of x , respectively.
Revised: W ∈ R C × d , and C and d are the number of classes and the dimensionality of x , respectively.

Original: Recall that the cross entropy loss for linear logistic regression is given as
Revised: Recall that the cross entropy loss for linear logistic regression is given as

Original: [Equation]
Revised: [Equation]

Original: c (cid:48)

Original: [Equation]

Original: ( (cid:104)

Original: [Equation]

Original: where W = [ w 1 , w 2 , . . .
Revised: where W = [ w 1 , w 2 , . . .

Original: , w C ] (cid:62) .
Revised: , w C ] (cid:62) .

Original: Let e y be a vector whose y -th entry is one and zero otherwise.
Revised: Let e y be a vector whose y -th entry is one and zero otherwise.

Original: Then, the gradient of the loss with respect to w c can be expressed as
Revised: Then, the gradient of the loss with respect to w c can be expressed as

Original: [Equation]
Revised: [Equation]

Original: where r z = σ ( W x ) − e y is the residual for the prediction on z .
Revised: where r z = σ ( W x ) − e y is the residual for the prediction on z .

Original: Hence, we have
Revised: Hence [Equation] we have

Original: [Equation]
Revised: [Equation]

Original: which yields
Revised: which yields

Original: [Equation]
Revised: [Equation]

Original: D R EPAIRING G RADIENT - BASED M ETRICS
Revised: D R EPAIRING G RADIENT - BASED M ETRICS

Original: As described in Section 5, we found that training instances with extremely large norms were selected as relevant by IF, FK [Equation] and GD.
Revised: As described in Section 5, we found that training instances with extremely large norms were selected as relevant by IF, FK, and GD.

Original: Thus, to repair these metrics, we need to design metrics that can ignore instances with large norms.
Revised: Thus, to repair these metrics, we need to design metrics that can ignore instances with large norms.

Original: A simple yet effective way of repairing the metrics is to use (cid:96)or cosine instead of the dot product.
Revised: A simple yet effective way of repairing the metrics is to use (cid:96)or cosine instead of the dot product.

Original: As Figure 4 shows, the (cid:96) 2 and cosine metrics performed better than the dot metrics.
Revised: As Figure 4 shows, the (cid:96) 2 and cosine metrics performed better than the dot metrics.

Original: Indeed, the (cid:96) 2 metrics do not favor instances with large norms that lead to large (cid:96) 2 -distance, and, through normalization, the cosine metrics completely ignore the effect of the norms
Revised: Indeed, the (cid:96) 2 metrics do not favor instances with large norms that lead to large (cid:96) 2 -distance, and, through normalization, the cosine metrics completely ignore the effect of the norms

Original: We name the repaired metrics of IF, FK, and GD based on the (cid:96) 2 metric as (cid:96) IF2 , (cid:96) FK2 , and (cid:96) GD2 , respectively, and the repaired metrics based on the cosine metric as cos IF and cos FK , and cos GD , respectively 4 .
Revised: We name the repaired metrics of IF, FK, and GD based on the (cid:96) 2 metric as (cid:96) IF2 , (cid:96) FK2 , and (cid:96) GD2 , respectively, and the repaired metrics based on the cosine metric as cos IF and cos FK , and cos GD , respectively 4 .

Original: We observed that these repaired metrics attained higher success rates on several evaluation criteria.
Revised: We observed that these repaired metrics attained higher success rates on several evaluation criteria.

Original: The details of the results can be found in Appendix F.
Revised: The details of the results can be found in Appendix F.

Original: E D O THE MODELS CAPTURE SUBCLASSES ?
Revised: E D O THE MODELS CAPTURE SUBCLASSES ?

Original: The identical subclass test requires the model to obtain internal representations that can distinguish subclasses.
Revised: The identical subclass test requires the model to obtain internal representations that can distinguish subclasses.

Original: Here, we conﬁrm that this condition is satisﬁed for all the datasets and models we used in the experiments.
Revised: Here, we conﬁrm that this condition is satisﬁed for all the datasets and models we used in the experiments.

Original: We consider that the model captures the subclasses if the latent representation h all has cluster structures.
Revised: We consider that the model captures the subclasses if the latent representation h all has cluster structures.

Original: Figure 9 visualizes h all for each dataset and model using UMAP (McInnes et al., 2018).
Revised: Figure 9 visualizes h all for each dataset and model using UMAP (McInnes et al., 2018).

Original: The ﬁgures show that the instances from different subclasses are not mixed completely random.
Revised: The ﬁgures show that the instances from different subclasses are not mixed completely random.

Original: MNIST and TREC have relatively clear cluster structures, while CIFAR10 and AGNews have vague clusters without explicit boundaries.
Revised: MNIST and TREC have relatively clear cluster structures, while CIFAR10 and AGNews have vague clusters without explicit boundaries.

Original: These ﬁgures imply that the models capture subclases (although it may not be perfect).
Revised: These ﬁgures imply that the models capture subclases (although it may not be perfect).

Original: F C OMPLETE E VALUATION R ESULTS F. F ULL R ESULTS
Revised: F C OMPLETE E VALUATION R ESULTS F. F ULL R ESULTS

Original: We show the complete results of the model randomization test in Table 2, the identical class test in
Revised: We show the complete results of the model randomization test in Table 2, the identical class test in

Original: Table 3, and the identical subclass test in Table 4.
Revised: Table 3, and the identical subclass test in Table 4.

Original: The results we present here are consistent with our observations in Section 4.
Revised: The results we present here are consistent with our observations in Section 4.

Original: [Table]
Revised: [Table]

Revised: 03 ± .− .

Revised: 10 ± .

Revised: 26 ♦ cos IF .

Revised: 02 ± .. 03 ± .− .

Revised: 01 ± .− .

Revised: 01 ± .. 04 ± .. 01 ± .

Revised: FK .

Revised: 04 ± .. 03 ± .. 01 ± .. 02 ± .− .

Revised: 01 ± .− .

Revised: 00 ± .

Revised: 02 ♦ (cid:96) FK − .

Revised: 17 ± .. 14 ± .− .

Revised: 18 ± .− .

Revised: 01 ± .. 05 ± .− .

Revised: 02 ± .

Original: {+20 ♦ cos FK − . 
+}
Revised: {+20 ♦ cos FK − . 
+}

Revised: ± .− .

Revised: 03 ± .. 08 ± .− .

Revised: 04 ± .− .

Revised: 01 ± .. 01 ± .

Revised: GD − .

Revised: 04 ± .. 03 ± .. 01 ± .− .

Revised: 02 ± .. 01 ± .. 02 ± .

Revised: 03 GC .

Revised: 01 ± .. 04 ± .. 02 ± .− .

Revised: 06 ± .. 00 ± .. 01 ± .

Revised: ♦ (cid:96) grad− .

Revised: 01 ± .− .

Revised: 14 ± .− .

Revised: 13 ± .. 11 ± .. 02 ± .− .

Revised: 09 ± .

Original: [Figure]
Revised: [Figure]

Original: MNIST with CNN
Revised: MNIST with CNN

Original: [Figure]
Revised: [Figure]

Original: y =
Revised: y =

Original: A
Revised: A

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: ) MNIST with CNN.
Revised: ) MNIST with CNN.

Original: y = B .
Revised: y = B .

Original: [Figure]
Revised: [Figure]

Original: MobileNetV
Revised: MobileNetV

Original: [Figure]
Revised: [Figure]

Original: y
Revised: y

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: ) CIFAR10 with MobileNetV2.
Revised: ) CIFAR10 with MobileNetV2.

Original: y = B .
Revised: y = B .

Original: [Figure]
Revised: [Figure]

Original: CNN
Revised: CNN

Original: [Figure]
Revised: [Figure]

Original: y
Revised: y

Original: [Figure]
Revised: [Figure]

Original: CIFAR
Revised: CIFAR

Original: [Figure]
Revised: [Figure]

Original: CNN
Revised: CNN

Original: [Figure]
Revised: [Figure]

Original: y
Revised: y

Original: [Figure]
Revised: [Figure]

Original: [Figure] 
Revised: [Figure] [Figure] [Figure] [Table]

Original: [Figure]
Revised: [Table] [Table] [Table] [Table]

Original: [Figure]
Revised: [Table] [Table] [Table] [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: [Table][Table] - (cid:96) all 0 . [Table]
Revised: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: - dot x

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: dot

Original: [Table]

Original: [Table]

Original: [Table]

Original: - dot

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: IF 0 .

Original: 29 ± 0 .0 .

Original: 86 ± 0 .0 .

Original: 24 ± 0 .0 .

Original: [Table]

Original: [Table]

Original: 39 ± 0 .0 .

Original: [Table]
Revised: [Table]

Original: 08 ♦

Original: [Table]
Revised: [Table]

Original: IF 0 .

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: IF 0 .

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: FK 0 .

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: all

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: dot

Original: [Table]
Revised: [Table]

Original: IF

Original: [Table]
Revised: [Table]

Original: IF

Original: [Table]
Revised: [Table]

Original: F. A DDITIONAL R ESULTS
Revised: F. A DDITIONAL R ESULTS

Original: The identical class test require the most relevant instance to be of the same class as the test instance.
Revised: The identical class test require the most relevant instance to be of the same class as the test instance.

Original: In practice, users can be more conﬁdent about a model’s output if several instances are provided as evidence.
Revised: In practice, users can be more conﬁdent about a model’s output if several instances are provided as evidence.

Original: In other words, we expect that the most relevant and a ﬁrst few relevant instances will be of the same class.
Revised: In other words, we expect that the most relevant and a ﬁrst few relevant instances will be of the same class.

Original: This observation leads to the additional criterion, which is a generalization of the identical class test.
Revised: This observation leads to the additional criterion, which is a generalization of the identical class test.

Original: Deﬁnition 5 (Top- k Identical Class Test) .
Revised: Deﬁnition 5 (Top- k Identical Class Test) .

Original: For z test =
Revised: For z test =

Original: ( x test , (cid:98) y test ) , let ¯
Revised: ( x test , (cid:98) y test ) , let ¯

Original: z
Revised: z

Original: j
Revised: j

Original: = (¯ x j , ¯ y j ) be a training instance with the j -th largest relevance score.
Revised: = (¯ x j , ¯ y j ) be a training instance with the j -th largest relevance score.

Original: Then, we require ¯ y j = (cid:98) y test for any j ∈ { 1 , 2 , . . .
Revised: Then, we require ¯ y j = (cid:98) y test for any j ∈ { 1 , 2 , . . .

Original: , k } .
Revised: , k } .

Original: This observation also applies to identical subclass test, which leads to the following criterion
Revised: This observation also applies to identical subclass test, which leads to the following criterion

Original: Deﬁnition 6 (Top- k Identical Subclass Test) .
Revised: Deﬁnition 6 (Top- k Identical Subclass Test) .

Original: For z test =
Revised: For z test =

Original: ( x test , (cid:98) y test ) , let ¯
Revised: ( x test , (cid:98) y test ) , let ¯

Original: z
Revised: z

Original: j
Revised: j

Original: = (¯ x j , ¯ y j ) be a training instance with the j -th largest relevance score.
Revised: = (¯ x j , ¯ y j ) be a training instance with the j -th largest relevance score.

Original: Then, we require s (¯ z j ) =
Revised: Then, we require s (¯ z j ) =

Original: s ( (cid:98) z test ) , ∀ j ∈ { 1 , 2 , . .
Revised: s ( (cid:98) z test ) , ∀ j ∈ { 1 , 2 , . .

Original: . , k } .
Revised: . , k } .

Original: We show the results of the top-10 identical class test in Table 3, and the top-10 identical subclass test in Table 4.
Revised: We show the results of the top-10 identical class test in Table 3, and the top-10 identical subclass test in Table 4.

Revised: G E XAMPLES OF E ACH E XPLANATION M ETHOD

Revised: We show some examples of the relevant instances using several relevance metrics on CIFAR with CNN in Figure 10 and Figure 11 and on AGNews with LSTM in Table 7 and Table 8.

Revised: We show examples of both correct (in Figure 10 and Table 7) and incorrect (in Figure 11 and Table 8) predictions.

Revised: As mentioned in Section 5, the relevance metrics based on the dot product of the gradient, such as IF, FK, and GD, tend to select instances with large norms, and therefore we can see that non-typical instances have been selected.

Original: [Table]
Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: IF

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: dot

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: IF

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: IF .

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: FK .

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: Test Input

Revised: Predict: frog

Revised: Gold: frog (cid:96) xdeer (cid:96) last frog (cid:96) all frog cos x deer cos last frog cos all frog dot x airplane dot last automobile dot all truck

Revised: IF deer (cid:96) IF frog cos IF frog

Revised: FK bird (cid:96) FK frog cos FK frog

Revised: GD bird

Revised: GC frog (cid:96) grad frog

Revised: Test Input

Revised: Predict: airplane

Revised: Gold: airplane (cid:96) xbird (cid:96) last airplane (cid:96) all airplane cos x bird cos last airplane cos all airplane dot x airplane dot last truck dot all airplane

Revised: IF ship (cid:96) IF cat cos IF horse

Revised: FK frog (cid:96) FK airplane cos FK airplane

Revised: GD airplane

Revised: GC airplane (cid:96) grad airplane

Revised: Test Input

Revised: Predict: deer

Revised: Gold: dog (cid:96) xdog (cid:96) last deer (cid:96) all deer cos x dog cos last deer cos all deer dot x airplane dot last automobile dot all automobile

Revised: IF automobile (cid:96) IF deer cos IF deer

Revised: FK bird (cid:96) FK deer cos FK deer

Revised: GD deer

Revised: GC deer (cid:96) grad deer

Revised: Test Input

Revised: Predict: automobile

Revised: Gold: ship (cid:96) xtruck (cid:96) last automobile (cid:96) all ship cos x ship cos last automobile cos all ship dot x airplane dot last automobile dot all automobile

Revised: IF deer (cid:96) IF automobile cos IF deer

Revised: FK frog (cid:96) FK automobile cos FK automobile

Revised: GD automobile

Revised: GC automobile (cid:96) grad automobile

Original: [Table] 
Revised: [Table] [Table] [unk] rules wo n’t [Table]

Revised: [unk] law

Revised: [Table]

Revised: Gold: Sports

Revised: Predict: Sci/Tech

Revised: [Table]

Revised: proﬁting[unk]

Revised: [Table]

Revised: moore[unk]

Revised: [Table]

Revised: Business

Revised: [Table]

Revised: devil[unk] rays[unk]

Revised: [Table]

Revised: unk] in ﬂorida hours[unk] before game

Revised: Sports

Revised: [Table]

Revised: unk] rays[unk]

Revised: [Table]

Revised: unk] in ﬂorida hours[unk] before game

Revised: Sports cos

Revised: [Table]

Revised: proﬁting[unk]

Revised: [Table]

Revised: moore

Revised: [Table]

Revised: Business

Revised: [Table]

Revised: devil[unk] rays[unk] stuck[unk] in ﬂorida hours[unk] before game

Revised: Sports cos

Revised: [Table]

Revised: devil[unk] rays[unk] stuck[unk] in ﬂorida hours[unk] before game

Revised: Sports dot

Revised: [Table]

Revised: [unk] , canadians[unk]

Revised: [Table]

Revised: [unk] to honour[unk] living legend[unk] :

Revised: [Table]

Revised: unk] winner smoky

Revised: [Table]

Revised: unk] smith

Revised: [Table]

Revised: [Table]

Revised: canadian press )

Revised: World

Revised: [Table]

Revised: Sci/Tech

Revised: [Table]

Revised: sportsnetwork

Revised: [Table]

Revised: preview

Revised: Sports

Revised: IF

Revised: [Table]

Revised: World

Revised: [Table]

Revised: IF calif. oks toughest[unk] auto emissions[unk] rules

Revised: World cos IF un envoy headed to darfur

Revised: World

Revised: FK yankee[unk] batters[unk] hit wall

Revised: Sports (cid:96) FK a ﬂat panel does n’t always[unk] compute[unk]

Revised: Sci/Tech cos FK a ﬂat panel does n’t always[unk] compute[unk]

Revised: Sci/Tech

Revised: GD issue 65 news hound[unk] : this week in gaming

Revised: Sci/Tech

Revised: [Table]

Revised: atari[unk] announces ﬁrst 64-bit[unk] game

Revised: Sci/Tech

Revised: [Table]

Revised: Sci/Tech