ExSum : F rom Local Explanations to Model Understanding (2022)

Abstract: Interpretability methods are developed to understand the working mechanisms of black-box models, which is crucial to their responsible deployment. Fulfilling this goal requires both that the explanations generated by these methods are correct and that people can easily and reliably understand them. While the former has been addressed in prior work, the latter is often overlooked, resulting in informal model understanding derived from a handful of local explanations. In this paper, we introduce explanation summary (ExSum), a mathematical framework for quantifying model understanding, and propose metrics for its quality assessment. On two domains, ExSum highlights various limitations in the current practice, helps develop accurate model understanding, and reveals easily overlooked properties of the model. We also connect understandability to other properties of explanations such as human alignment, robustness, and counterfactual similarity and plausibility.

## introduction understanding a model's behavior is often a prerequisite for deploying it in the real world, especially in high-stake scenarios such as financial, legal, and medical domains. unfortunately, most highperforming models, such as neural networks, are black-boxes. thus, model-agnostic interpretability techniques have been developed, with the majority being "local" -algorithms that produce an explanation for a specific input at a time (e.g., @xcite . even with these local explanations, there are still two hurdles to overcome before achieving the ultimate goal of complete understanding of a model. first, some local explanations may not correctly (or faithfully) represent the model's reasoning process @xcite , as has been demonstrated both theoretically @xcite and empirically @xcite in prior work. as a result, correctness evaluation has received much attention in the community (e.g., @xcite @xcite . another mostly overlooked property of explanations is their understandability. as the model understanding pipeline depicted in fig. 1 shows, explanations need to be both correct and easily understandable, since even correct explanations are not as valuable if they lead to incorrect understanding. however, the concept of understandability has yet to be formalized, and instead users often derive model understanding from few examples in a non-rigorous (and potentially incorrect) manner. consider the sentiment classification task shown in fig. 2 . on a test input, the model makes the correct prediction of positive sentiment. obviously, this evidence is insufficient to conclude that "in general, the model classifies positive inputs correctly", because even a random-guess model is correct 50% of the time on a single instance. instead, statistics such as the confusion matrix serve to rigorously support (or refute) generalization claims about model performance -for example, "the model is correct 97.6% of the time on positive inputs" -ensuring an accurate understanding of model performance. input: as shaky as the plot is , kaufman 's script is still memorable for some great one-liners . ## on generalized model understanding besides the practical example above, we start from first principles and argue that generalized model understanding is the central concept for explanation usefulness. local explanations are mathematical descriptions (md) of some aspect of model behavior, for specific inputs. for example, gradient saliency (in the embedding space) is the sensitivity of the prediction to infinitesimal changes in the token embedding; occlusion saliency is the prediction change if individual embeddings are zeroed out. it is with these mathematical descriptions that people associate high-level interpretations (hl) of model behavior, such as associating the above two metrics with word importance. this (unconscious) train of thought can be described as follows: crucially, people rarely study md or hl for one specific input, as explanations are often used to understand broader model behaviors, such as reliance upon spurious correlation, non-discrimination of a protected class, or usage of unknown scientific principles. we elaborate upon these use cases in app. a to demonstrate that people implicitly or explicitly seek generalized model understanding. from another perspective, analogous to why people ultimately focus on the generalization accuracy of a model, they (should) focus on generalized model understanding derived from local explanations. for example, after observing that some highly polar words have high contribution for a sentiment classification model, people conclude that all highly polar words have high contribution. this process can be formalized as follows: since mds are rigorously defined mathematical quantities (e.g., the prediction of the sentence drops by 32% after the embedding of "great" is zeroed out), we can define and evaluate the quality of their generalization, and hl (g) can also include any failures and anomalies. as each md is a local explanation, we call md (g) the explanation summary (exsum), and proceed by instantiating this principle for feature attribution explanations. ## exsum development process and gui we describe a systematic procedure for authoring exsum rule unions from scratch and utilize it in sec. 5. starting from an empty rule union with composition structure for the rule union and, if a rule is selected, the counterfactual without it rule list panel to select and inspect individual rules metric values for the rule union, and, if a rule is selected, the cf rule union and the selected rule parameter inspection and tuning for the selected rule visualization of input texts, along with ground truth and predicted labels. information is conveyed through graphics and word formatting such as boldface, underscore and color. no feus covered, we iteratively create rules that target uncovered feus. each rule describes one model behavior, such as that for highly positive words. for a rule, the a-and b-functions need to be defined, which may involve setting and tuning parameters, such as the sentiment threshold. last, we add a lowest precedence catch-all rule if any feus remain uncovered. during this process, we may also merge or split rules and change the composition structure according to the metric values. to support these steps, we developed a python flask-based @xcite graphical user interface (gui, fig. 3 ). users can visualize the feus, with font formatting for their coverage and validity. users can also filter for uncovered or invalid feus, iteratively constructing and refining the rule union. exsum rule definitions usually include parameters such as the sentiment threshold. manually selecting correct values for the parameters is tedious, so the lower middle panel of the gui implements automatic parameter tuning for a given target metric value. installation and usage instructions for the gui are available on the project page foot_0 . ## evaluation we construct exsum rule unions for sst and qqp models (details in app. b). we split the test set into a construction set to create the rule union and tune its parameters (analogous to the training and validation set in supervised model training) and an evaluation set to compute unbiased estimates of the metric values (analogous to the test set). ## related work as discussed in sec. 1, explanation evaluation usually has a focus on correctness (or faithfulness) -i.e., whether the explanation truly reflects the model's reasoning process. this includes sanity checks @xcite , proxy metrics @xcite , and explicit ground truth @xcite . the understandability issue has been much less studied, with the exception by @xcite , who proposed an evaluation specifically for rationale models @xcite . exsum, however, addresses post hoc explanations of general black-box models. in addition, a few prior works have attempted to capture the "end-to-end" utility of explanations: whether access to explanations leads to performance increase in certain tasks. @xcite propose a model-teaching-human setup, subsequently extended by @xcite into an automated evaluation procedure. @xcite study whether explanations can improve human-machine teaming performance. while these studies report mostly negative results, pinpointing the root cause is difficult due to their end-to-end nature. poor understanding of the explanations may be a major reason, as indicated by exsum. last, some authors have proposed methods for understanding model predictions beyond individual instances. for example, the anchor method @xcite generates an explicit domain of applicability for each explanation, while @xcite and @xcite proposed to learn "patches" of the input space specified by logical predicates. exsum also emphasizes the need to understand models that generalizes across instances, and uses logical predicates in the formulation, but focuses on model understanding via explanations instead of direct predictions, which can capture a wider variety of behaviors (e.g. the matching and non-matching behaviors of the qqp model). furthermore, the fine-grained analysis of behaviors allows us to investigate whether models are "correct for the correct reason." ## the many faces of understandability the central thesis of this paper is quite simple and intuitive: in order to understand a model from local explanations, we need to understand those local explanations. while exsum is the first framework to explicitly formalize and quantify the notion of understandability, we argue that it is connected to many often-discussed and desirable properties of explanation (further details in app. c). human alignment users sometimes expect explanations that are aligned with their expectations. for example, the fact that highly salient words convey strong sentiment is taken as evidence for the quality of an explanation method by @xcite . in image classification, this concept is typically implemented as a pointing game between the highsaliency region and the segmentation mask of the predicted class (fong and vedaldi, 2017). however, alignment does not imply correctness, as the model could use any spurious correlations, which should be faithfully highlighted by the explainer. however, higher-alignment explanations are indeed more understandable, since by definition they agree more with human intuition. thus, an alignment-based evaluation can be considered as one of understandability. nonetheless, understandability can also be achieved by correcting human expectations, e.g., users realizing that punctuations are actually important for predictions (contrary to expectations). robustness it is often argued that explanations should be robust @xcite -similar inputs should induce similar explanations. however, robustness can be at odds with correctness: if the model truly applies vastly different logic for two very close inputs -such as a pair of inputs that only differ in the root feature of a decision treethen their explanations should be distinct, as they are routed down two different sub-trees. nonetheless, slow-varying explanations are generally easier to understand than those that change erratically and unpredictably (independent of their correctness), and thus robustness is related to understandability. ## discussion and conclusion traditionally, model explanations are evaluated on correctness (or faithfulness), i.e., whether they correspond to how models actually make predictions, e.g., reliance on spurious correlations @xcite . such evaluation, however, does not answer the equally important question of whether these (presumably correct) explanations are understandable. even faithful explanations can lead users into error, if misunderstood (e.g., trusting a model incorrectly). in a sense, the most correct explanation for an input is the literal trace of model computation, but it is also arguably the least understandable (or useful). as we abstract away from low-level details and use higher-level concepts such as word sentiment, the resulting explanation loses correctness but gains understandability. at the other extreme are explanations that are trivially understandable but completely wrong, such all attribution values being 0 (i.e., no feature impacts the model prediction). thus, a trade-off often occurs between these two desiderata, and we need to choose a sweet spot. concretely, we propose exsum rules and rule unions, along with three quality metrics to formalize and evaluate understandability. such rigorous investigations stand in contrast to current ad hoc practices, which are prone to yielding unreliable and coarse model understanding. for sst and qqp datasets, exsum demonstrates that our model understanding is quite limited and imprecise, even with very reasonable explanations. being aware of this is an asset. while exsum helps us to recognize that our understanding is incomplete, it still helps uncover unexpected model behaviors that warrant further investigation.