Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness? (2020)

Abstract: With the growing popularity of deep-learning based NLP models, comes a need for interpretable systems. But what is interpretability, and what constitutes a high-quality interpretation? In this opinion piece we reflect on the current state of interpretability evaluation research. We call for more clearly differentiating between different desired criteria an interpretation should satisfy, and focus on the faithfulness criteria. We survey the literature with respect to faithfulness evaluation, and arrange the current approaches around three assumptions, providing an explicit form to how faithfulness is “defined” by the community. We provide concrete guidelines on how evaluation of interpretation methods should and should not be conducted. Finally, we claim that the current binary definition for faithfulness sets a potentially unrealistic bar for being considered faithful. We call for discarding the binary notion of faithfulness in favor of a more graded one, which we believe will be of greater practical utility.

## introduction fueled by recent advances in deep-learning and language processing, nlp systems are increasingly being used for prediction and decision-making in many fields @xcite , including sensitive ones such as health, commerce and law @xcite . unfortunately, these highly flexible and highly effective neural models are also opaque. there is therefore a critical need for explaining learning-based models' decisions. the emerging research topic of interpretability or explainability foot_0 has grown rapidly in recent years. unfortunately, not without growing pains. one such pain is the challenge of defining-and evaluating-what constitutes a quality interpretation. current approaches define interpretation in a rather ad-hoc manner, motivated by practical usecases and applications. however, this view often fails to distinguish between distinct aspects of the interpretation's quality, such as readability, plausibility and faithfulness @xcite . 2 we argue ( §2, §5) such conflation is harmful, and that faithfulness should be defined and evaluated explicitly, and independently from plausibility. our main focus is the evaluation of the faithfulness of an explanation: a faithful interpretation is one that accurately represents the reasoning process behind the model's prediction. we find this to be a pressing issue: in cases where an explanation is required to be faithful, imperfect or misleading evaluation can have disastrous effects. while literature in this area may implicitly or explicitly evaluate faithfulness for specific explanation techniques, there is no consistent and formal definition of faithfulness. we uncover three assumptions that underlie all these attempts. by making the assumptions explicit and organizing the literature around them, we "connect the dots" between seemingly distinct evaluation methods, and 2 unfortunately, the terms in the literature are not yet standardized, and vary widely. "readability" and "plausibility" are also referred to as "human-interpretability" and "persuasiveness", respectively (e.g., lage et al. (2019); herman (2017) ). to our knowledge, the term "faithful interpretability" was coined in @xcite , reinforced by @xcite , and is, we believe, most commonly used (e.g., @xcite ; @xcite ; @xcite ). @xcite refers to this issue (more or less) as "accountability". sometimes referred to as how "trustworthy" @xcite or "descriptive" @xcite the interpretation is, or as "descriptive accuracy" @xcite . also related to the "transparency" @xcite , the "fidelity" @xcite or the "robustness" @xcite of the interpretation method. and frequently, simply "explainability" is inferred to require faithfulness by default. finally, we observe a trend by which faithfulness is treated as a binary property, followed by showing that an interpretation method is not faithful. we claim that this is unproductive ( §7), as the assumptions are nearly impossible to satisfy fully, and it is all too easy to disprove the faithfulness of an interpretation method via a counter-example. what can be done? we argue for a more practical view of faithfulness, calling for a graded criteria that measures the extent and likelihood of an interpretation to be faithful, in practice ( §8). while we started to work in this area, we pose the exact formalization of these criteria, and concrete evaluations methods for them, as a central challenge to the community for the coming future. ## faithfulness vs. plausibility there is considerable research effort in attempting to define and categorize the desiderata of a learned system's interpretation, most of which revolves around specific use-cases @xcite . two particularly notable criteria, each useful for a different purposes, are plausibility and faithfulness. "plausibility" refers to how convincing the interpretation is to humans, while "faithfulness" refers to how accurately it reflects the true reasoning process of the model @xcite . naturally, it is possible to satisfy one of these properties without the other. for example, consider the case of interpretation via posthoc text generation-where an additional "generator" component outputs a textual explanation of the model's decision, and the generator is learned with supervision of textual explanations @xcite @xcite . in this case, plausibility is the dominating property, while there is no faithfulness guarantee. despite the difference between the two criteria, many authors do not clearly make the distinction, and sometimes conflate the two. 3 moreoever, the majority of works do not explicitly name the criteria under consideration, even when they clearly belong to one camp or the other. 4 we argue that this conflation is dangerous. for example, consider the case of recidivism prediction, where a judge is exposed to a model's prediction and its interpretation, and the judge believes the interpretation to reflect the model's reasoning process. since the interpretation's faithfulness carries legal consequences, a plausible but unfaithful interpretation may be the worst-case scenario. the lack of explicit claims by research may cause misinformation to potential users of the technology, who are not versed in its inner workings. 5 therefore, clear distinction between these terms is critical. ## inherently interpretable? a distinction is often made between two methods of interpretability: (1) interpreting existing models via post-hoc techniques; and (2) designing inherently interpretable models. @xcite argues in favor of inherently interpretable models, which by design claim to provide more faithful interpretations than post-hoc interpretation of black-box models. we warn against taking this argumentation at face-value: a method being "inherently interpretable" is merely a claim that needs to be verified before it can be trusted. indeed, while attention mechanisms have been considered as "inherently interpretable" @xcite , recent work cast doubt regarding their faithfulness @xcite @xcite . ## evaluation via utility while explanations have many different use-cases, such as model debugging, lawful guarantees or health-critical guarantees, one other possible usecase with prominent evaluation literature is intelligent user interfaces (iui), via human-computer interaction (hci), of automatic models assisting human decision-makers. the goal of the explanation here is to increase the degree of trust between the user and the system, giving the user more nuance towards whether the system's decision is likely correct, or not. in the general case, the final evaluation metric is the performance of the user at their task @xcite . for example, @xcite evaluate various explanations of a model in a setting of trivia question answering. however, in the context of faithfulness, we must warn against hci-inspired evaluation, as well: increased performance in this setting is not in-dicative of faithfulness; rather, it is indicative of correlation between the plausibility of the explanations and the model's performance. to illustrate, consider the following fictional case of a non-faithful explanation system, in an hci evaluation setting: the explanation given is a heat-map of the textual input, attributing scores to various tokens. assume the system explanations behave in the following way: when the output is correct, the explanation consists of random content words; and when the output is incorrect, it consists of random punctuation marks. in other words, the explanation is more likely to appear plausible when the model is correct, while at the same time not reflecting the true decision process of the model. the user, convinced by the nicer-looking explanations, performs better using this system. however, the explanation consistently claimed random tokens to be highly relevant to the model's reasoning process. while the system is concretely useful, the claims given by the explanation do not reflect the model's decisions whatsoever (by design). while the above scenario is extreme, this misunderstanding is not entirely unlikely, since any degree of correlation between plausibility and model performance will result in increased user performance, regardless of any notion of faithfulness. ## guidelines for evaluating faithfulness we propose the following guidelines for evaluating the faithfulness of explanations. these guidelines address common pitfalls and sub-optimal practices we observed in the literature. ## defining faithfulness what does it mean for an interpretation method to be faithful? intuitively, we would like the provided interpretation to reflect the true reasoning process of the model when making a decision. but what is a reasoning process of a model, and how can reasoning processes be compared to each other? lacking a standard definition, different works evaluate their methods by introducing tests to measure properties that they believe good interpretations should satisfy. some of these tests measure aspects of faithfulness. these ad-hoc definitions are often unique to each paper and inconsistent with each other, making it hard to find commonalities. we uncover three assumptions that underlie all these methods, enabling us to organize the literature along standardized axes, and relate seemingly distinct lines of work. moreover, exposing the underlying assumptions enables an informed discussion regarding their validity and merit (we leave such a discussion for future work, by us or others). these assumptions, to our knowledge, encapsulate the current working definitions of faithfulness used by the research community. assumption 1 (the model assumption). two models will make the same predictions if and only if they use the same reasoning process. corollary 1.1. an interpretation system is unfaithful if it results in different interpretations of models that make the same decisions. as demonstrated by a recent example concerning nlp models, it can be used for proof by counterexample. theoretically, if all possible models which can perfectly mimic the model's decisions also provide the same interpretations, then they could be deemed faithful. conversely, showing that two models provide the same results but different interpretations, disprove the faithfulness of the method. @xcite show how these counter-examples can be derived with adversarial training of models which can mimic the original model, yet provide different explanations. 6 corollary 1.2. an interpretation is unfaithful if it results in different decisions than the model it interprets. a more direct application of the model assumption is via the notion of fidelity @xcite . for cases in which the explanation is itself a model capable of making decisions (e.g., decision trees or rule lists @xcite ), fidelity is defined as the degree to which the explanation model can mimic the original model's decisions (as an accuracy score). for cases where the explanation is not a computable model, @xcite propose a simple way of mapping explanations to decisions via crowdsourcing, by asking humans to simulate the model's decision without any access to the model, and only access to the input and explanation (termed forward simulation). this idea is further explored and used in practice by @xcite . ## is faithful interpretation impossible? the aforementioned assumptions are currently utilized to evaluate faithfulness in a binary manner: whether an interpretation is strictly faithful or not. specifically, they are most often used to show that a method is not faithful, by constructing cases in which the assumptions do not hold for it. 9 in other words, there is a clear trend of proof via counter-example, for various interpretation methods, that they are not globally faithful. we claim that this is unproductive, as we expect these various methods to consistently result in negative (not faithful) results, continuing the current trend. this follows because an interpretation functions as an approximation of the model or decision's true reasoning process, so it by definition loses information. by the pigeonhole principle, there will be inputs with deviation between interpretation and reasoning. this is observed in practice, in numerous work that show adversarial behavior, or pathological behaviours, that arise from the deeply non-linear and high-dimensional decision boundaries of current models. 10 furthermore, because we lack supervision regarding which models or decisions are indeed mappable to human-readable concepts, we cannot ignore the approximation errors. this poses a high bar for explanation methods to fulfill, a bar which we estimate will not be overcome soon, if at all. what should we do, then, if we desire a system that provides faithful explanations? ## towards better faithfulness criteria we argue that a way out of this standstill is in a more practical and nuanced methodology for defining and evaluating faithfulness. we propose the following challenge to the community: we must develop formal definition and evaluation for faith-fulness that allows us the freedom to say when a method is sufficiently faithful to be useful in practice. we note two possible approaches to this end: ## conclusion the opinion proposed in this paper is two-fold: first, interpretability evaluation often conflates evaluating faithfulness and plausibility together. we should tease apart the two definitions and focus solely on evaluating faithfulness without any influence of the convincing power of the interpretation. second, faithfulness is often evaluated in a binary "faithful or not faithful" manner, and we believe strictly faithful interpretation is a "unicorn" which will likely never be found. we should instead evaluate faithfulness on a more nuanced "grayscale" that allows interpretations to be useful even if they are not globally and definitively faithful.