Evaluating Explanations: How Much Do Explanations from the Teacher Aid Students? (2022)

Abstract: While many methods purport to explain predictions by highlighting salient features, what aims these explanations serve and how they ought to be evaluated often go unstated. In this work, we introduce a framework to quantify the value of explanations via the accuracy gains that they confer on a student model trained to simulate a teacher model. Crucially, the explanations are available to the student during training, but are not available at test time. Compared with prior proposals, our approach is less easily gamed, enabling principled, automatic, model-agnostic evaluation of attributions. Using our framework, we compare numerous attribution methods for text classification and question answering, and observe quantitative differences that are consistent (to a moderate to high degree) across different student model architectures and learning strategies.1

## introduction the success of deep learning models, together with the difficulty of understanding how they work, has inspired a subfield of research on explaining predictions, often by highlighting specific input features deemed somehow important to a prediction @xcite @xcite . for instance, we might expect such a method to highlight spans like ''poorly acted'' and ''slow-moving'' to explain a prediction of negative sentiment for a given movie review. however, there is little agreement in the literature as to what constitutes a good explana-tion @xcite . moreover, various popular methods for generating such attributions disagree considerably over which tokens to highlight ( the status quo for evaluating so-called explanations skews qualitative-many proposed techniques are evaluated only via visual inspection of a few examples @xcite @xcite . while several quantitative evaluation techniques have recently been proposed, many of these are easily gamed @xcite ). 2 some depend upon the model outputs corresponding to deformed examples that lie outside the support of the training distribution @xcite , and a few validate explanations on specifically crafted tasks @xcite . in this work, we propose a new framework, where explanations are quantified by the degree to which they help a student model in learning to simulate the teacher on future examples (figure 1 ). our framework addresses a coherent goal, is model-agnostic and broadly applicable across tasks, and (when instantiated with models as students) can easily be automated and scaled. our method is inspired by argumentative models for justifying human reasoning, which posit that the role of explanations is to communicate information about how decisions are made, and thus to enable a recipient to anticipate future figure 1 : the proposed framework for quantifying explanation quality. student models learn to mimic the teacher, with and without explanations (provided as ''side information'' with each example). explanations are effective if they help students to better approximate the teacher on unseen examples for which explanations are not available. students and teachers could be either models or people. using our framework, we conduct extensive experiments on two broad categories of nlp tasks: text classification and question answering. for classification tasks, we compare seven widely used input attribution techniques, covering gradient-based methods @xcite , perturbation-based techniques @xcite , attention-based explanations @xcite , and other popular attributions @xcite . these comparisons lead to observable quantitative differences-we find attention-based explanations and integrated gradients @xcite to be the most effective, and vanilla gradient-based saliency maps and lime to be the least effective. further, we observe moderate to high agreement among rankings obtained by varying student architectures and learning strategies in our framework. for question answering, we validate the effectiveness of student learners on both human-produced explanations collected by @xcite , and automatically generated explanations from a spanbert model @xcite . ## learning with explanations our student-teacher framework does not specify how to use explanations while training the student model. below, we examine two broad approaches to incorporate explanations: attention regularization and multitask learning. our first approach regularizes attention values of the student model to align with the information communicated in explanations. in the second method, we pose the learning task for the student as a joint task of prediction and explanation generation, expecting discussed in @xcite and @xcite . 6 we speculate this effect to be pronounced when the models' outputs and the true labels differ only over a few samples. to improve prediction due to the benefits of multitask learning. we show that both of these methods indeed improve student performance when using human-provided explanations (and gold labels) for classification tasks. we explore variants of these two approaches for question answering tasks. classification tasks the training data for the student model consists of n documents x 1 , . . . , x n , and the output to be learned, y 1 , . . . , y n , comes from the teacher, that is, y @xmath0, along with teacher-explanations e t (x 1 ), . . . , e t (x n ). in this work, we consider teacher explanations in the form of a binary vector e t (x i ), such that e t (x i ) @xmath1, and 0 otherwise (see second, we use explanations via multitask learning, where the two tasks are prediction and explanation generation (a sequence labeling problem). formally, the overall loss can be written as: as in multitask learning, if the task of prediction and explanation generation are complementary, then the two tasks would benefit from each other. as a corollary, if the teacher-explanations offer no additional information about the prediction, then we would see no benefit from multitask learning (appropriately so). for most of our classification experiments, we use bert @xcite with a linear classifier on top of the [cls] vector to model p(y|x; θ). to model p(e|x; φ θ) we use a linear-chain crf @xcite on top of the sequence vectors from bert. note that we share the bert parameters θ between classification and explanation tasks. in prior work, similar multitask formulations have been demonstrated to effectively incorporate rationales to improve classification performance @xcite and evidence extraction @xcite . question answering let the question q consist of m tokens q 1 . . . q m , along with passage x that provides the answer to the question, consisting of n tokens x 1 , . . . , x n . let us define a set of question phrases q and passage phrases p to be we consider a subset of qed explanations @xcite , which consist of a sequence of one or more ''referential equality annotations'' e 1 . . . e |e| . formally, each referential equality annotation e k for @xmath2. . . |e| is a pair (φ k , π k ) ∈ q × p, specifying that phrase φ k in the question refers to the same thing in the world as the phrase π k in the passage (see to incorporate explanations for question answering tasks, we use the two approaches discussed for text classification tasks, namely, attention regularization and multitask learning. since the explanation format for question answering is different from the explanations in text classification, we use a lossy transformation, where we construct a binary explanation vector, where 1 corresponds to tokens that appear in one or more referential equalities and 0 otherwise. given the transformation, both these approaches do not use the alignment information present in the referential equalities. student model 600 900 1200 bert-base 75.5 79.0 81.1 w/ explanations used via multitask learning 75.2 80.0 82.5 attention regularization 81.5 83.1 84.0 setup there exist a few tasks where researchers have collected explanations from experts besides the output label. for the task of sentiment analysis on movie reviews, @xcite collected ''rationales'' where people highlighted portions of the movie reviews that would encourage (or discourage) readers to watch (or avoid) the movie. in another recent effort, @xcite collected ''qed annotations'' over questions and the passages from the natural questions (nq) dataset @xcite . these annotations contain the salient entity in the question and their referential mentions in the passage that need to be resolved to answer the question. for both these tasks, our student-learners are pretrained bert-base models, which are further student model 500 1500 2500 bert-base 28.9 43.7 49.0 w/ explanations used via multitask learning 29.7 42.7 49.2 attention regularization 31.2 47.2 52.6 attention alignment loss 37.3 49.6 54.0 ## automated evaluation of attributions here, we use a machine learning model as our choice for the teacher, and subsequently train student models using the output and explanations produced by the teacher model. such a setup allows us to compare attributions produced by different techniques for a given teacher model. ## limitations and future directions there are a few important limitations of our work that could motivate future work in this space. first, our current experiments only compare explanations that are of the same format. more work is required to compare explanations of different formats, for example, comparing natural language explanations to the top-k% highlighted tokens, or even comparing two methods to produce natural language explanations. to make such comparisons, one would have to ensure that different explanations (potentially with different formats) communicate comparable bits of information, and subsequently develop learning strategies to train student models. second, validating the results of any automated evaluation with human judgement of explanation quality remains inherently difficult. when people evaluate input attributions (or any form of explanations) qualitatively, they can determine whether the attributions match their intuition about what portions of the input should be important to solve the task (i.e., plausibility of explanations), but it is not easy to evaluate if the highlighted portions are responsible for the model's prediction. going forward, we think that more granular notions of simulatability, coupled with counterfactual access to models (where people can query the model), might help people better assess the role of explanations. third, while we observe moderate to high agreement among attribution rankings across different student architectures and learning schemes, it is conceivable that different explanations are favored based on the choice of student model. this is a natural drawback of using a learning model for evaluation as the measurement could be sensitive to its design. therefore, we recommend users to average simulation results over a diverse set of student architectures, training examples, and learning strategies; and, wherever possible, validate explanation quality with its intended users. lastly, an interesting future direction is to train explanation modules to generate explanations that optimize our metric, that is, learning to produce explanations based on the feedback from the students. to start with, an explanation generation module could be a simple transformation over the attention heads of the teacher model (as attention-based explanations are effective explanations as per our framework). learning explanations can be modeled as a meta-learning problem, where the meta-objective is the few-shot test performance of the student trained with intermediate explanations, and this performance could serve as a signal to update the explanation generation module using implicit gradients as in @xcite . ## related work several papers have suggested simulatability as an approach to measure interpretability @xcite . in a survey on interpretability, @xcite propose the task of forward simulation: given an input and an explanation, people must predict what a model would output for that instance. @xcite conduct humanstudies to evaluate if explanations from visual question answering (vqa) models help users predict the output. recently, hase and bansal (2020) perform a similar human-study across text and tabular classification tasks. due to the nature of these two studies, the observed differences with and without explanation, and among different explanation types, were not significant. conducting large-scale human studies poses several challenges, including the considerable financial expense and the logistical challenge of recruiting and retaining participants for unusually long tasks @xcite . by automating students in our framework, we mitigate such challenges, and observe quantitative differences among methods in our comparisons. closest in spirit to our work, @xcite propose a new framework to assess explanatory power as the communication success rate between an explainer and a layperson (which can be people or machines). however, as a part of their communication, they pass on explanations during test time, which could easily leak the label, and the models trained to play this communication game can learn trivial protocols (e.g., explainer generating a period for positive examples and a comma for negative examples). this is probably only partially addressed by enforcing constraints on the explainer and the explainee. our setup does not face this issue as explanations are not available at test time. to counter the effects of leakage due to explanations, @xcite present a leakage-adjusted simulatability (las) metric. their metric quantifies the difference in performance of the simulation models (analogous to our student models) with and without explanations at test time. to adjust for this leakage, they average their simulation results across two different sets of examples, ones that leak the label, and others that do not. leakage is modeled as a binary variable, which is estimated by whether a discriminator can predict the answer using the explanation alone. it is unclear how the average of simulation results solves the problem, especially when trivial explanations leak the label. ## conclusion we have formalized the value of explanations as their utility in a student-teacher framework, measured by how much they improve the student's ability to simulate the teacher. in our setup, explanations are provided by the teacher as additional side information during training, but are not available at test time, thus preventing ''leakage'' between explanations and output labels. our proposed evaluation confirms the value of human-provided explanations, and correlates with a (simplistic) notion of explanation quality. additionally, we conduct extensive experiments that measure the value of numerous previously-proposed schemes for producing explanations. our experiments result in clear quantitative differences between different explanation methods, which are consistent, to a moderate to high degree, across different choices. among explanation methods, we find attention to be the most effective. for student models, we find that both multitask and attention-regularized student learners are effective, but attention-based learners are more effective, especially in low-resource settings. we examine the following attribution methods: lime locally interpretable model-agnostic explanations @xcite , or lime, are explanations produced by a linear interpretable model that is trained to approximate the original black box model in the local neighborhood of the input example. for a given example, several samples are constructed by perturbing the input string, and these samples are used to train the linear model. we draw twice as many samples as the number of tokens in the example, and select the top words that explain the predicted class. we set the number of features for the linear classifier to be 2k, where k is the number of tokens to be selected.