A Comparative Study of Faithfulness Metrics for Model Interpretability Methods (2022)

Abstract: Interpretable methods to reveal the internal reasoning processes behind machine learning models have attracted increasing attention in recent years. To quantify the extent to which the identified interpretations truly reflect the intrinsic decision-making mechanisms, various faithfulness evaluation metrics have been proposed. However, we find that different faithfulness metrics show conflicting preferences when comparing different interpretations. Motivated by this observation, we aim to conduct a comprehensive and comparative study of the widely adopted faithfulness metrics. In particular, we introduce two assessment dimensions, namely diagnosticity and complexity. Diagnosticity refers to the degree to which the faithfulness metric favors relatively faithful interpretations over randomly generated ones, and complexity is measured by the average number of model forward passes. According to the experimental results, we find that sufficiency and comprehensiveness metrics have higher diagnosticity and lower complexity than the other faithfulness metrics.

## introduction nlp has made tremendous progress in recent years. however, the increasing complexity of the models makes their behaviour difficult to interpret. to disclose the rationale behind the models, various interpretation methods have been proposed. interpretation methods can be broadly classified into two categories: model-based methods and posthoc methods. model-based approaches refer to designing simple and white-box machine learning models whose internal decision logic can be easily interpreted, such as linear regression models, decision trees, etc. a post-hoc method is applied after model training and aims to disclose the relationship between feature values and predictions. as pretrained language models @xcite @xcite become more popular, deep learning models are becoming more and more complex. therefore, post-hoc methods are the only option for model interpretations. posthoc interpretation methods can be divided into two categories: gradient-based @xcite @xcite and perturbation-based @xcite @xcite . gradient-based methods assume the model is differentiable and attempt to interpret the model outputs through the gradient information. perturbation-based methods interpret model outputs by perturbing the input data. to verify whether, and to what extent, the interpretations reflect the intrinsic reasoning process, various faithfulness metrics have been proposed. most faithfulness metrics use a removal-based criterion, i.e., removing or retaining only the important tokens identified by the interpretation and observing the changes in model outputs @xcite @xcite . however, we observe that the existing faithfulness metrics are not always consistent with each other and even lead to contradictory conclusions. as shown in the example from our experiments ( motivated by the above observation, we aim to conduct a comprehensive and comparative study of faithfulness metrics. we argue that a good faithfulness metric should be able to effectively and efficiently distinguish between faithful and unfaithful interpretations. to quantitatively assess this capa- ## terminology and notations we first introduce the prerequisite terminology and notations for our discussions. terminology a "classification instance" is the input and output values of a classification model, which we apply interpretation methods on. an "interpretation" of a classification instance is a sequence of scores where each score quantifies the importance of the input token at the corresponding position. an "interpretation pair" is a pair of interpretations of the same classification instance. an "interpretation method" is a function that generates an interpretation from a classification instance with the associated classification model. notations let x be the input tokens. denote the number of tokens of x as l x . denote the predicted class of x as c(x), and the predicted probability corresponding to class j as p j (x). assume an interpretation is given. denote the k-th important token as x k . denote the input sequence containing only the top k (or top q%) important tokens as x :k (or x :q% ). denote the modified input sequence from which a token sub-sequence x ′ are removed as x \ x ′ . let (x, y) be a classification instance associated with classification model m, and g be an interpretation method. denote the interpretation of z generated by g as g(x, y, m). let u be an interpretation, (u, v) be an interpretation pair, and f be a faithfulness metric. denote the importance score that u assigns to the i-th input token as [u] i . denote the statement "u is more faithful than v" as "u ≻ v", and the statement "f considers u as more faithful than v" as "u ≻ f v". ## faithfulness metrics an interpretation is called faithful if the identified important tokens truly contribute to the decision making process of the model. mainstream faithfulness metrics are removal-based metrics, which measure the changes in model outputs after removing important tokens. we compare the most widely adopted faithfulness metrics, introduced as follows. decision flip -most informative token (dfmit) introduced by @xcite , this metric focuses on only the most important token. it assumes that the interpretation is faithful only if the prediction label is changed after removing the most important token, i.e. a score of 1 implies that the interpretation is faithful. ## evaluation of faithfulness metrics in this section, we propose an evaluation paradigm for faithfulness metrics by addressing two aspects: ## experimental setup 1 datasets we conduct experiments on three text classification datasets used in @xcite ): (i) stanford sentiment treebank (sst) @xcite ; (ii) imdb large movie reviews (imdb) @xcite ; (iii) ag news corpus (ag) @xcite . we summarize the dataset statistics in text classification models we adopt two most common model architectures for text classification: (i) bert @xcite ; (ii) cnn @xcite . the former one encodes contextualized representations of tokens and has higher accuracy in ## results and discussion diagnosticity we estimate the diagnositicities of the faithfulness metrics in section 3 on all datasets for both cnn and bert models. the results are shown in dfmit and dffot have the lowest and the second lowest average diagnosticities. removing the most important token is usually not creating enough perturbation to flip the original model decision. in fact, the probability of decision flipping by removing the most important token @xmath0. as a result, up to 86% of interpretations are considered as indifferent by dfmit. for dffot, the probability of decision flipping by removing the important tokens in order does not only depend on the quality of interpretation but also depends on any model bias towards certain classes. for instance, decision flipping will be less likely to occur if the predicted class on the original input is the same as the one on the empty input sequence. therefore, we found that decision flipping metrics (dfmit, dffot) are less effective than the metrics that operate on output probabilities (suff, comp, corr, mono). ## related work interpretation methods interpretation methods can be roughly classified into two categories: model-based methods and post-hoc methods. model-based methods refer to the construction of simple machine learning models whose internal decision logic can be easily interpreted, such as linear regression models, decision trees, etc. post-hoc methods interpret the internal reasoning process behind the model after training. generally, posthoc methods can be divided into gradient-based and perturbation-based. a gradient-based interpretation method assumes deep learning model is differentiable and discloses the decision making mechanism of the model according to the gradient information @xcite @xcite . a perturbationbased interpretation method interprets the model by perturbing the input of data samples and measuring how the predictions change @xcite @xcite . interpretation method evaluation to assess the quality of different interpretation methods, various evaluation metrics have been proposed. existing evaluation methods on interpretations can be broadly classified into two categories, plausibility and faithfulness. plausibility measures if the interpretation agrees with human judgments on how a model makes a decision @xcite @xcite . however, even if the interpretation conforms to human criteria, it is not certain that it truly reflects the underlying decision mechanism behind the model. to this end, faithfulness measures the extent to which the inner decision-making mechanism actually relies on the identified important features @xcite @xcite @xcite . in general, existing faithfulness metrics are developed through a removal-based criterion, which measures the changes in model output when perturbing or removing tokens identified as important by the interpretation. @xcite proposed a decision flipping metric that evaluates the proportion of tokens that need to be erased in order to change the model decision. also using decision flip as an indicator, chrysostomou and aletras (2021) introduces a metric that counts the average flips that occur when removing the most important token marked by the interpretation method. in addition to decision flips, changes in model output probabilities by removing or retaining important tokens is also widely used to measure faithfulness @xcite @xcite . some recent work also focuses on the study of faithfulness metrics. @xcite argued that the definition of faithfulness remains inconsistent and informal, and provided concrete guidelines on how evaluations of interpretation methods should and should not be conducted. more recently, @xcite discussed the limitations of removal-based faithfulness metrics and proposed two other quantitative criteria, namely sensitivity and stability. different from the aforementioned previous work that does not focus on assessing faithfulness metrics, we mainly focus on the measurement of faithfulness and conduct a comprehensive study of existing faithfulness metrics. ## conclusion in this paper, we propose a framework to quantitatively evaluate six widely adopted faithfulness metrics in terms of diagnosticity and time complexity. in particular, diagnosticity measures whether the faithfulness metric correctly favours relatively faithful interpretations over random ones; time complexity is concerned with computational efficiency, estimated by the average number of model forward passes. the experimental results show that sufficiency and comprehensiveness metrics outperform the other faithfulness metrics with higher diagnosticity and lower time complexity. for this reason, we suggest using these two metrics for faithfulness evaluation. we hope our work will bring more awareness to the standardization of faithfulness measurement. for future work, we would like to explore evaluating faithfulness metrics using a whitebox model such as linear regression, from which we can derive an intrinsically faithful interpretation as the "ground truth".