Leakage-Adjusted Simulatability: Can Models Generate Non-Trivial Explanations of Their Behavior in Natural Language? (2020)

Abstract: Data collection for natural language (NL) understanding tasks has increasingly included human explanations alongside data points, allowing past works to introduce models that both perform a task and generate NL explanations for their outputs. Yet to date, model-generated explanations have been evaluated on the basis of surface-level similarities to human explanations, both through automatic metrics like BLEU and human evaluations. We argue that these evaluations are insufficient, since they fail to indicate whether explanations support actual model behavior (faithfulness), rather than simply match what a human would say (plausibility). In this work, we address the problem of evaluating explanations from the the model simulatability perspective. Our contributions are as follows: (1) We introduce a leakage-adjusted simulatability (LAS) metric for evaluating NL explanations, which measures how well explanations help an observer predict a model’s output, while controlling for how explanations can directly leak the output. We use a model as a proxy for a human observer, and validate this choice with two human subject experiments. (2) Using the CoS-E and e-SNLI datasets, we evaluate two existing generative graphical models and two new approaches; one rationalizing method we introduce achieves roughly human-level LAS scores. (3) Lastly, we frame explanation generation as a multi-agent game and optimize explanations for simulatability while penalizing label leakage, which can improve LAS scores.

## introduction deep neural models have achieved impressive success in many areas. however, their interpretability and explainability have remained broadly limited. to make neural models more interpretable, previous works have proposed methods for explaining model decisions, e.g., through various feature importance estimates @xcite or model-generated natural language (nl) @xcite . early work on generating nl explanations focused on providing explanations that were both descriptive of an image and discriminative as labels @xcite . since then, a variety of datasets have been collected with free-form human generated explanations accompanying each data point @xcite @xcite @xcite . models have been proposed for these datasets with two aims: (1) to teach models how to explain their own decisions in natural language, by offering demonstrations of humans doing this, and @xcite to increase model accuracy on the task, by making use of additional information in human explanations. past works have proposed varying methods for generating nl explanations, which can be represented by distinct graphical models. in our work, we explore four graphical models, shown in figure 1 . each model generates explanations in either a reasoning (re) or rationalizing (ra) mode, where rationalizing models explicitly condition explanations on a label and reasoning models condition only on the input. approaches further differ by whether they use explanations as inputs to a task model (st) or as additional supervision in a multitask framework (mt). two of these models are drawn from prior works: mt-ra @xcite and @xcite . we introduce st-ra and also test mt-re as the reasoning counterpart to mt-ra. to fairly compare the approaches, we implement each graphical model with a state-of-the-art pretrained t5 model @xcite (details in section 3). ## modeling with explanations in this section, we delineate our baseline model and the four graphical models we study. the graph-t5 explain: where would i not want a fox? the choices are hen house, england, and mountains. the answer is: hen house the answer is hen house because the fox would eat the hens context for rationalizing human explanation my commonsense tells me that the fox would eat the hens context for reasoning human explanation the answer is: hen house ical models are depicted in figure 1 . we also summarize the key features of each approach in baseline. the baseline model simply predicts y given x. we adopt the approach of @xcite for fine-tuning to multiple-choice problems, which is to maximize the likelihood of correct answer tokens conditioned on the task inputs. to produce predictions, however, we compute a likelihood for each answer choice and select the most likely choice, rather than sampling text. snli also fits into this framework by taking the three relations as answer choices. ## las: leakage-adjusted simulatability while many motivations drive humans' explanations for their behavior, we consider one central purpose to be helping others understand one's internal reasoning. this notion is captured by the concept of simulatability @xcite . a model is simulatable to the extent that an observer, or simulator, can predict its outputs. the simulator can be either a human or a learned model; we will consider both settings. from this perspective, one might use the simulator's accuracy to measure explanation quality. with task inputs @xmath0, model outputs ŷ = {ŷ i }, model explanations ê = {ê i }, simulator correctness as however, this measure fails to distinguish between different ways in which the simulator can successfully predict the task model output, as shown in the causal diagram in figure 3 . we suggest that the simulator's success does not reflect explanation quality when (1) the simulator can guess behavior correctly from the input x alone, or (2) the explanation ê directly restates the task model output, i.e., leaking the label to the simulator. what we are truly looking for in explanations is that they provide semantic content that informs the simulator of the leakage path semantics path correctness simulator 2 ) the model explanation's semantic content, êz , and ( 3 ) whether the explanation leaks the model output, êŷ task model's output in the context of its input. note that we do not think label leakage means an explanation is bad. explanations will leak more often than not, as human explanations leak about 85% of the time for cos-e and about 97% of the time for e-snli (estimated by t5 simulator). instead, we think the more important aspect is to evaluate the explanation's semantic content. for examples of leaking and nonleaking explanations, see to deal with issue (1) above, we introduce an input-only baseline and measure the effect of an explanation on simulatability as 1[ŷ|x, ê] -1[ŷ|x]. to resolve the issue ( 2 ), we propose to control for a label leaking variable, which has the effect of blocking that causal pathway @xcite . we do so by using a proxy variable for label leakage, which is an indicator variable for if the simulator can predict ŷ solely from ê. the correctness of this prediction suggests that the explanation gives away the answer directly. with this approach, we can estimate explanations' leakage-controlled effect on simulatability by (1) grouping data by the level of explanation label leakage, ( 2 ) computing the average difference 1[ŷ|x, ê] -1[ŷ|x] within each leakage group, and (3) taking the raw average of the effect across groups (to avoid favoring simulator a fox is a common animal in england. task model explanation task model output the answer is: hen house simulate: where would i not want a fox? the choices are hen house, england, and mountains. my commonsense tells me that the larger subset). note that there are only two levels of label leakage, 1[ŷ|ê] = 1 (leaking) and 1[ŷ|ê] = 0 (nonleaking), and we use model correctness rather than probabilities since t5 probabilities are uncalibrated. now with simulator correctness as 1[ŷ i |x i , êi ] or 1[ŷ i |x i ], and our leakage indicator as k @xmath1, we write our leakage-adjusted simulatability (las) metric as: when testing, the simulator's correctness on each example is 1[ŷ i |x i , êi ], and we obtain 1[ŷ i |x i ] and 1[ŷ i |ê i ] by dropping êi or x i from the input. we will compare las and acc(ŷ|x, ê) for explanations from the models introduced above as well as human explanations. we discuss the relationship with human experiments for both metrics in section 6.2. in analysis to follow, we will also refer to example-level las scores, which are given as 1[ŷ|x, ê]-1[ŷ|x] and take values -1, 0, or 1 (see ## multi-agent explanation optimization in this section, we explore an approach to optimizing explanations for las, rather than just relying on a standard language modeling loss to produce explanations. the approach is naturally framed as a multi-agent game. note that we do not aim to improve model accuracy or explanations' bleu scores in these experiments. in our game, there are two agents. the first is a task model that predicts labels and generates explanations jointly. here, we use mt-re or mt-ra. the second agent is a simulator model that predicts the task model's output ŷi given its explanation êi and the model input x i , matching the previous simulation format shown in figure 4 . these two agents are jointly trained during the multiagent training procedure. the objective of the simulator is the same as discussed in the above section, which is to predict ŷi given x i and êi , and we randomly dropout x i or êi to ensure they are both being used. as in section 3, the task model learns to perform the task (minimizing l task ) and generate explanations (minimizing l lm ) via supervision from ground-truth labels and human explanations. here, the task model also tries to minimize the simulator's loss through its explanations. the chief computational challenge with this approach is that explanations are sampled by greedy decoding, and thus the loss is not differentiable with respect to the task model. we explore two optimization methods circumventing this issue: approximate sgd via argmax relaxation @xcite and reinforce @xcite . our aim is for explanations to better communicate the task model's reasoning process, without adopting the trivial solution, i.e., directly stating its output. thus while we optimize explanations for simulatability, we also penalize label leakage, which we formalize below. note that the task model's predictions are not optimized to agree ## experimental results here, we discuss experiments conducted with each method using two (english) datasets: the first is the commonsenseqa (cqa) dataset of talmor et al. (2019), with explanations collected by rajani et al. (2019) to make a combined cos-e dataset (examples in ## conclusion we introduce a leakage-adjusted simulatability metric to evaluate the influence of natural language explanations on model simulatability while controlling for explanations leaking the model outputs. we validate our metric with two human subject experiments, and find that: (1) our st-ra model attains similar las scores to human explanations, (2) rationalizing methods do better than reasoning methods, (3) no statistically significant relationship emerges between simulatability and accuracy, (4) our automatic metric correlates with expert simulation results, ( 5 ) the strongest predictor of crowdsourced explanation ratings is whether explanations leak the answer choice, and (6) optimizing explanations for simulatability can improve las scores.