Connecting Attributions and QA Model Behavior on Realistic Counterfactuals (2021)

Abstract: When a model attribution technique highlights a particular part of the input, a user might understand this highlight as making a statement about counterfactuals (Miller, 2019): if that part of the input were to change, the model’s prediction might change as well. This paper investigates how well different attribution techniques align with this assumption on realistic counterfactuals in the case of reading comprehension (RC). RC is a particularly challenging test case, as token-level attributions that have been extensively studied in other NLP tasks such as sentiment analysis are less suitable to represent the reasoning that RC models perform. We construct counterfactual sets for three different RC settings, and through heuristics that can connect attribution methods’ outputs to high-level model behavior, we can evaluate how useful different attribution methods and even different formats are for understanding counterfactuals. We find that pairwise attributions are better suited to RC than token-level attributions across these different RC settings, with our best performance coming from a modification that we propose to an existing pairwise attribution method.

## introduction recent research in interpretability of neural models @xcite has yielded numerous post-hoc explanation methods, including token attribution techniques @xcite @xcite . attributions are a flexible explanation format and can be applied to many domains, including sentiment analysis @xcite , visual recognition @xcite , and natural language inference @xcite . however, it is hard to evaluate whether these explanations are faithful mismatch between explanations and model behavior on these counterfactuals. the explanation is bad! figure 1: our methodology. given a base example, we can formulate a hypothesis about the model's behavior, like a theory about how the model is using certain tokens. next, we collect counterfactual examples that modify these tokens and profile the actual model behavior. finally, we assess whether feature attributions suggest behavior consistent with what we observe, verifying whether our attributions actually enable meaningful statements about behavior on counterfactuals. to the computation of the original model @xcite @xcite and as a result, they can potentially mislead users @xcite . furthermore, attributions do not have a consistent and meaningful social attribution @xcite : that is, when a user of the system looks at an explanation, they do not necessarily draw a valid conclusion from it, making it hard to use for downstream tasks. how can we evaluate whether these attributions make faithful and meaningful statements about model behavior? in this work, we show how to use counterfactual examples to evaluate attributions' ability to reveal the high-level behavior of models. that is, rather than a vague statement like "this word was important," we want attributions to give concrete, testable conclusions like "the model compared these two words to reach its decision;" this statement can be evaluated for faithfulness and it helps a user make important inferences about how the system behaves. we approach this evaluation from a perspective of simulatability @xcite ): can we predict how the system will behave on new or modified examples? doing so is particularly challenging for the rc models we focus on in this work due to the complex nature of the task, which fundamentally involves a correspondence between a question and a supporting text context. figure 1 shows our methodology. our approach requires annotating small sets of realistic counterfactuals, which are perturbations of original data points. these resemble several prior "stress tests" used to evaluate models, including counterfactual sets @xcite , contrast sets @xcite , and checklists @xcite . we first semi-automatically curate these sets to answer questions like: if different facts were shown in the context, how would the model behave? if different amounts of text or other incorrect paragraphs were retrieved by an upstream retrieval system, would the model still get the right answer? we run the model on counterfactuals to assess the ground truth behavior. then, given attributions from various techniques, can we predict how the model would behave based purely on these explanations? our approach to do this is specific to each dataset and attribution method, but generally involves assessing how strongly the attribution method highlights tokens that are counterfactually altered, which would indicate that those tokens should impact the prediction if changed. to showcase the kind of evaluation this method can enable, we investigate two paradigms of explanation techniques: token attribution-based @xcite @xcite and feature interaction-based @xcite , which attribute decisions to sets of tokens or pairwise token interactions. for both techniques, we devise methods to connect these explanations to our high-level hypotheses about behavior on counterfactual examples. on two types of questions from hotpotqa @xcite and questions from adversarial squad @xcite , we show that token-level attribution is not sufficient for analyzing rc models, which naturally involves more complex reasoning over multiple clues. we further propose a modification to an existing interaction technique from @xcite and show improved performance on our datasets. our main contributions are: (1) we propose a new goal for attributions, namely automatically simulating model behavior on realistic counterfactuals. (2) we describe a technique for connecting low-level attributions (token-level or higher-order) with high-level model hypotheses. (3) we improve an attention-based pairwise attribution technique with a simple but effective fix, leading to strong empirical results. (4) we analyze a set of qa tasks and show that our approach can derive meaningful conclusions about counterfactuals on each. overall, we establish a methodology for analyzing explanations that we believe can be adapted to studying attribution methods on a wide range of other tasks with appropriate counterfactuals. ## motivation we start with an example of how model attributions can be used to understand model behavior and consequently how to use our methodology to compare different attribution techniques. figure 2 shows an example of a multi-hop yes/no question from hotpotqa. the qa model correctly answers yes in this case. given the original example, the explanations produced using intgrad @xcite and diffmask @xcite (explained in section 4) both assign high attribution scores to the two documentary tokens appearing in the context: a user of the system is likely to impute that the model is comparing these two values, as it's natural to assume this model is using the highlighted information correctly. by contrast, the pairwise attribution approach we propose in this work (section 4.3) attributes the prediction to interactions with the question, suggesting the interactions related to documentary do not matter. we manually curate a set of contrastive examples to test this hypothesis. if the model truly recognizes that both movies are documentaries, then replacing either or both of the documentary tokens with romance should change the prediction. to verify that, we perturb the original example to obtain another three examples (left side of figure 2 ). these four examples together form a local neighborhood @xcite @xcite consisting of realistic counterfactuals. 2 are super high me and all in this tea both documentaries? super high me is a 2008 documentary film about smoking. all in this tea is a 2007 documentary film. yes base example counterfactual example integrated gradient (sundararajan et al., 2017) diffmask (de cao et al., 2020) laattrattn (ours) explanations i still predict yes even if documentary tokens are @xmath0. all in this tea is a 2007 documentary film . </@xmath1. all in this tea is a 2007 documentary film . </@xmath2. all in this tea is a 2007 documentary film. yes super high me is a 2008 documentary film about smoking. all in this tea is a 2007 romance film. yes super high me is a 2008 romance film about smoking. all in this tea is a 2007 romance film. yes ？ d 0 d 1 d 2 d 3 figure 2: a motivating example and attributions generated by three methods. we profile the model's behavior with its predictions on realistic counterfactual inputs, which suggest the model does not truly base its prediction on the two movies being documentaries. we can evaluate attributions by heuristically assessing whether the attribution mass yields the same conclusion about model behavior. unlike what's suggested by the token attribution based techniques, the model always predicts "yes" for every example in the neighborhood, casting doubt on whether the model is following the right reasoning process. although the pairwise attribution seemed at first glance much less plausible than that generated by the other techniques, it was actually better from the perspective of faithfully simulating the model's behavior on these examples. our main assumption in this work can be stated as follows: an explanation should describe model behavior with respect to realistic counterfactuals. past work has evaluated along plausibility criteria @xcite @xcite , but as we see from this example, faithful explanations @xcite @xcite are better aligned with our goal of simulatability. we argue that a good explanation is one that aligns with the model's high-level behavior, from which we can understand how the model generalizes to new data. how to interpret behavior from explanations is still an open question, but we take initial steps in this work with techniques based on assessing the attribution "mass" on perturbed tokens. discussion: realistic counterfactuals many counterfactual modifications are possible: past work has looked at injecting non-meaningful triggers @xcite , deleting chunks of content @xcite , or evaluating interpolated input points as in intgrad, all of which "true" set of realistic counterfactuals is highly domain-specific, but nevertheless, a good explanation technique should work well on a range of counterfactuals like those considered here. violate assumptions about the input distribution. in rc, masking part of the question often makes it nonsensical and we may not have strong expectations about our model's behavior in this case. 3 focusing on realistic counterfactuals, by contrast, illuminates fundamental problems with our rc models' reasoning capabilities @xcite @xcite . this is the same motivation as that behind contrast sets @xcite , but our work focuses on benchmarking explanations, not models themselves. ## behavior on counterfactuals we seek to formalize the reasoning we undertook in figure 2 . using the model's explanation on a base data point, can we predict the model's behavior on the perturbed instances of that point? definitions given an original example d 0 (e.g., the top example in figure 2 ), we construct a set of perturbations {d 1 , ..., d k } (e.g., the three counterfactual examples in figure 2 ), which together with d 0 form a local neighborhood d. these perturbations are realistic inputs derived from existing datasets or which we construct. we formulate a hypothesis h about the neighborhood. in figure 2 , h is "the model is comparing the target properties" (documentary in this case). based on the model's behavior on the set d, we can derive a high-level behavioral label z corre-sponding to the truth of h. we form our local neighborhood to check the answer empirically and compute a ground truth for z. since the model always predicts "yes" in this neighborhood, we label set d with @xmath3. we label d as @xmath4, when the model does predict "no" for some perturbations. procedure our approach is as follows: note that this third step only uses the explanation of the base data point: explanations should let us make conclusions about new counterfactuals without having to do inference on them. simulation from attributions in our experiments on hotpotqa and squad, we compute a scalar factor f for each attribution representing the importance of a specific part of the inputs (e.g., the documentary tokens in figure 2 ), which we believe should correlate with model predictions on the counterfactuals. if an attribution assigns higher importance to this information, it suggests that the model will actually change its behavior on these new examples. given this factor, we construct a simple classifier where we predict @xmath5. we expect the factors extracted using better attribution methods should better indicate the model behavior. hence, we evaluate the explanation using the best simulation accuracy it can achieve and the auc score (s-acc and s-auc). 4 our evaluation resembles the human evaluation in @xcite , which asks human raters to predict a model's decision given an example together with its explanations, addressing simulatability from a user-focused angle. our method differs in that (1) we automatically extract a factor to predict model behavior instead of asking humans to do so, and (2) we predict the behavior on unseen counterfactuals given the explanation of a single base data point. ## explanation techniques compared to classification tasks like sentiment analysis, rc more fundamentally involves interaction between input features, especially between a question and a context. this work will directly compare feature interaction explanations with token attribution techniques that are more common for other tasks. 5 for rc, each instance @xmath6, c, a), a tuple containing a question, context, and answer respectively. in the techniques we consider, q and c are concatenated and fed into a pre-trained transformer model, so our attribution techniques will explain predictions using both of these. ## experiments we assess whether attributions can achieve our proposed goal following the setup in section 3 on the hotpotqa dataset @xcite , and the squad dataset @xcite , specifically leveraging examples from adversarial squad @xcite . ## related work we focus on several prominent token attribution techniques, but there are other related methods as well, including other methods based on shapley values @xcite , contextual decomposition @xcite , and hierarchical explanations @xcite . these formats can also be evaluated using our framework if being connected with model behavior using the proper heuristic. other work explores so-called concept-based explanations @xcite @xcite . these provide another pathway towards building explanations of high-level behavior; however, they have been explored primarily for image recognition tasks and cannot be directly applied to qa, where defining these sorts of concepts is challenging. finally, textual explanations @xcite are another popular alternative, but it is difficult to evaluate these in our framework as it is very difficult to bridge from a free-text explanation to an approximation of a model's computation. probing techniques aim to discover what intermediate representations have been learned in neural models @xcite @xcite . internal representations could potentially be used to predict behavior on contrast sets similar to this work; however, this cannot be done heuristically and larger datasets are needed to explore this. other work considering how to evaluate explanations is primarily based on how explanations can assist humans in predicting model decisions for a given example @xcite @xcite ; we are the first to consider building contrast sets for this. similar ideas have been used in other contexts @xcite but our work focuses on evaluation of explanations rather than general model evaluation. ## conclusion we have presented a new methodology using explanations to understand model behavior on realistic counterfactuals. we show explanations can indeed be connected to model behavior, and therefore we can compare explanations to understand which ones truly give us actionable insights about what our models are doing. we have showcased how to apply our methodology on several rc tasks, leveraging either semiautomatically curated counterfactual sets or existing resources. we generally find pairwise interaction methods perform better than the best tokenlevel attribution based methods in our analysis. more broadly, we see our methodology as a useful evaluation paradigm that could be extended across a range of tasks, leveraging either existing contrast sets or with a small amount of effort devoted to create targeted counterfactual sets as in this work.