Aligning Faithful Interpretations with their Social Attribution (2021)

Abstract: We find that the requirement of model interpretations to be faithful is vague and incomplete. With interpretation by textual highlights as a case study, we present several failure cases. Borrowing concepts from social science, we identify that the problem is a misalignment between the causal chain of decisions (causal attribution) and the attribution of human behavior to the interpretation (social attribution). We reformulate faithfulness as an accurate attribution of causality to the model, and introduce the concept of aligned faithfulness: faithful causal chains that are aligned with their expected social behavior. The two steps of causal attribution and social attribution together complete the process of explaining behavior. With this formalization, we characterize various failures of misaligned faithful highlight interpretations, and propose an alternative causal chain to remedy the issues. Finally, we implement highlight explanations of the proposed causal format using contrastive explanations.

## introduction when formalizing the desired properties of a quality interpretation of a model or a decision, the nlp community has settled on the key property of faithfulness @xcite @xcite , or how ''accurately'' the interpretation represents the true reasoning process of the model. a common pattern of achieving faithfulness in interpretation of neural models is via decomposition of a model into steps and inspecting the intermediate steps @xcite . for example, neural modular networks (nmns; @xcite first build an execution graph out of neural building blocks, and then apply this graph to data. the graph struc-ture is taken to be a faithful interpretation of the model's behavior, as it describes the computation precisely. similarly, highlight methods (also called extractive rationales foot_0 ), decompose a textual prediction problem into first selecting highlighted texts, and then predicting based on the selected words (select-predict, described in section 2). the output of the selection component is taken to be a faithful interpretation, as we know exactly which words were selected. similarly, we know that words that were not selected do not participate in the final prediction. however, @xcite call nmn graphs not faithful in cases where there is a discrepancy between a building block's behavior and its name (i.e., expected behavior). can we better characterize the requirement of faithful interpretations and amend this discrepancy? we take an extensive and critical look at the formalization of faithfulness and of explanations, with textual highlights as an example use-case. in particular, the select-predict formulation for faithful highlights raises more questions than it provides answers: we describe a variety of curious failure cases of such models in section 4, as well as experimentally validate that the failure cases are indeed possible and do occur in practice. concretely, the behavior of the selector and predictor in these models do not necessarily line up with expectations of people viewing the highlight. current literature in ml and nlp interpretability fails to provide a theoretical foundation to characterize these issues (sections 4, 6.1). to remedy this, we turn to literature on the science of social explanations and how they are utilized and perceived by humans (section 6): the social and cognitive sciences find that human explanations are composed of two, equally important parts: the attribution of a causal chain to the decision process (causal attribution), and the attribution of social or human-like intent to the causal chain (social attribution) @xcite , where ''human-like intent'' refers to a system of beliefs and goals behind or following the causal process. for example, ''she drank the water because she was thirsty. '' 2 people may also attribute social intent to models: in the context of nlp, when observing that a model consistently translates ''doctor'' with male morphological features @xcite , the user may attribute the model with a ''belief'' that all doctors are male, despite the model lacking an explicit system of beliefs. explanations can influence this social attribution: for example, a highlight-based explanation may influence the user to attribute the model with the intent of ''performing a summary before making a decision'' or ''attempting to justify a prior decision''. fatally, the second key component of human explanations-the social attribution of intent-has been missing from current formalization on the desiderata of artificial intelligence explanations. in section 7 we define that a faithful interpretation -a causal chain of decisions-is aligned with human expectations if it is adequately constrained by the social behavior attributed to it by human observers. armed with this knowledge, we can now verbalize the issue behind the ''non-faithfulness'' perceived by @xcite for nmns: the inconsistency between component names and their actual behavior causes a misalignment between the causal and social attributions. we can also characterize the more subtle issue underlying the failures of the select-predict models described in section 4: in section 8 we argue that for a set of possible social attributions, the select-predict formulations fails to guarantee any of them. in section 9 we propose an alternative causal chain for highlights explanations: predict-selectverify. predict-select-verify does not suffer from the issue of misaligned social attribution, as the highlights can only be attributed as evidence towards the predictor's decision. as a result, predictselect-verify highlights do not suffer from the misalignment failures of section 4, and guarantee that the explanation method does not reduce the score of the original model. finally, in section 10 we discuss an implementation of predict-select-verify, namely, designing the components in the roles predictor and selector. designing the selector is non-trivial, as there are many possible options to select highlights that evidence the predictor's decision, and we are only interested in selecting ones that are meaningful for the user to understand the decision. we leverage observations from cognitive research regarding the internal structure of (human-given) explanations, dictating that explanations must be contrastive to hold tangible meaning to humans. we propose a classification predict-select-verify model that provides contrastive highlights-to our knowledge, a first in nlp-and qualitatively exemplify and showcase the solution. contributions. we identify shortcomings in the definitions of faithfulness and plausibility to characterize what is useful explanation, and argue that the social attribution of an interpretation method must be taken into account. we formalize ''aligned faithfulness'' as the degree to which the causal chain is aligned with the social attribution of intent that humans perceive from it. based on the new formalization, we (1) identify issues with current select-predict models that derive faithful highlight interpretations, and (2) propose a new causal chain that addresses these issues, termed predict-select-verify. finally, we implement this chain with contrastive explanations, previously unexplored in nlp explanaibility. we make our code available online. 3 2 highlights as faithful interpretations highlights, also known as extractive rationales, are binary masks over a given input that imply some behavioral interpretation (as an incomplete description) of a particular model's decision process to arrive at a decision on the input. given input sequence @xmath0: the final prediction of the system at inference is m(x) = m p (m s (x) x). we refer to h := m s (x) as the highlight and h x as the highlighted text. what does the term ''faithfulness'' mean in this context? a highlight interpretation can be faithful or unfaithful to a model. literature accepts a highlight interpretation as ''faithful'' if the highlighted text was provably the only input to the predictor. implementations. various methods have been proposed to train select-predict models. of note: @xcite propose to train the selector and predictor end-to-end via reinforce @xcite , and @xcite replace rein-force with the reparameterization trick (kingma and welling, 2014). @xcite propose fresh, where the selector and predictor are trained separately and sequentially. ## use-cases for highlights to discuss whether an explanation procedure is useful as a description of the model's decision process, we must first discuss what is considered useful for the technology. we refer to the following use-cases: dispute: a user may want to dispute a model's decision (e.g., in a legal setting). they can do this by disputing the selector or predictor: by pointing to some non-selected words, saying: ''the model wrongly ignored a,'' or by pointing to selected words and saying: ''based on this highlighted text, i would have expected a different outcome.'' debug: highlights allow the developer to designate model errors into one of two categories: did the model focus on the wrong part of the input, or did the model make the wrong prediction based on the correct part of the input? each category implies a different method of alleviating the problem. advice: assuming that the user is unaware of the ''correct'' decision, they may (1) elect to trust the model, and learn from feedback on the part of the input relevant to the decision; or (2) elect to increase or decrease trust in the model, based on whether the highlight is aligned with the user's prior on what the highlight should or should not include. for example, if the highlight is focused on punctuation and stop words, whereas the user believes it should focus on content words. ## limitations of select-predict highlights as (faithful) interpretations we now explore a variety of circumstances in which select-predict highlight interpretations are uninformative to the above use-cases. while we present these failures in context of highlights, they should be understood generally as symptoms of mistaken or missing formal perspective of explanations in machine learning. specifically, that faithfulness is insufficient to formalize the desired properties of interpretations. ## on faithfulness, plausibility, and explanainability from the science of human explanations the mathematical foundations of machine learning and natural language processing are insufficient to tackle the underlying issue behind the symptoms described in section 4. in fact, formalizing the problem itself is difficult. what enables a faithful explanation to be ''understood'' as accurate to the model? and what causes an explanation to be perceived as a trojan? in this section, we attempt to better formalize this problem on a vast foundation of social, psychological and cognitive literature about human explanations. 7 6.1 plausibility is not the answer plausibility foot_8 (or persuasiveness) is the property of an interpretation being convincing towards the model prediction, regardless of whether the model was correct or whether the interpretation is faithful. it is inspired by human-provided explanations as post hoc stories generated to plausibly justify our actions @xcite . plausibility is often quantified by the degree that the model's highlights resemble gold-annotated highlights given by humans @xcite or by querying for the feedback of people directly @xcite . following the failure cases in section 4, one may conclude that plausibility is a desirable, or even necessary, condition for a good interpretations: after all, trojan explanations are by default implausible. we strongly argue this is not the case. plausibility should be viewed an incentive of the explainer, and not as a property of the explanation: human explanations can be categorized by utility across multiple axes @xcite , among them are (1) learning a better internal model for future decisions and calculations @xcite ; (2) examination to verify the explainer has a correct internal prediction model; (3) teaching foot_9 to modify the internal model of the explainer towards a more correct one (can be seen as the opposite end of (1)); (4) assignment of blame to a component of the internal model; and finally, (5) justification and persuasion. these goals can be trivially mapped to our case, where the explainer is artificial. critically, goal (5) of justification and persuasion by the explainer may not necessarily be the goal of the explainee. indeed, in the case of ai explainability, it is not a goal of the explainee to be persuaded that the decision is correct (even when it is), but to understand the decision process. if plausibility is a goal of the artificial model, this perspective outlines a game theoretic mismatch of incentives between the two players. and specifically in cases where the model is incorrect, it is interpreted as the difference between an innocent mistake and an intentional lie-of course, lying is considered more unethical. as a result, we conclude that modeling and pursuing plausibility in ai explanations is an ethical issue. the failures discussed above do not stem from how (un)convincing the interpretation is, but from how well the user understands the reasoning process of the model. if the user is able to comprehend the steps that the model has taken towards its decision, then the user will be the one to decide whether these steps are plausible or not, based on how closely they fit the user's prior knowledge on whatever correct steps should be takenregardless of whether the user knows the correct answer or whether the model is correct. ## (socially) aligned faithfulness unlike human explainers, artificial explainers can exhibit a misalignment between the causal chain behind a decision and the social attribution attributed to it. this is because the artificial decision process may not resemble human behavior. by presenting to the user the causal pipeline of decisions in the model's decision process as an interpretation of this process, the user naturally conjures social intent behind this pipeline. in order to be considered comprehensible to the user, the attributed social intent must match the actual behavior of the model. this can be formalized as a set of constraints on the possible range of decisions at each step of the causal chain. we claim that this problem is the root cause behind the symptoms in section 4. here we define the general problem independently from the narrative of highlight interpretations. definition. we say that an interpretation method is faithful if it accurately describes causal information about the decision process of the decision. we say that the faithful method is human-aligned (short for ''aligned with human expectations of social intent'') if the model and method adhere to the social attribution of intent by human observers. conversely, ''misaligned'' interpretations are interpretations whose mechanism of conveying # claim 1 the marked text is supportive of the decision. 2 the marked text is selected after making the decision. 3 the marked text is sufficient for making the decision. 4 the marked text is selected irrespective of the decision. 5 the marked text is selected prior to making the decision. 6 the marked text includes all the information that informed the decision. 7 the marking pattern alone is sufficient for making the decision by the predictor. 8 the marked text provides no explanation whatsoever. ## alignment of select-predict highlight interpretations in contrast to the human-provided explanations, in the ml setup, our situation is unique in that we have control over the causal chain but not the social attribution. therefore, the social attribution must lead the design of the causal chain. in other words, we argue that we must first identify the behavior expected of the decision process, and then constrain the process around it. ## predict-select-verify we propose the predict-select-verify causal chain as a solution that can be constrained to provide highlights as evidence (i.e., guarantee claims 1-3). this framework solves the misalignment problem by allowing the derivation of faithful highlights aligned with evidencing social attribution. the decision pipeline is as follows: ## discussion causal attribution of heat-maps. recent work on the faithfulness of attention heat-maps @xcite @xcite or saliency distributions @xcite cast doubt on their faithfulness as indicators to the significance of parts of the input (to a model decision). similar arguments can be made regarding any explanation in the format of heat-maps, such as lime and shap @xcite . we argue that this is a natural conclusion from the fact that, as a community, we have not envisioned an appropriate causal chain that utilizes heat-maps in the decision process, reinforcing the claims in this work on the parallel between causal attribution and faithfulness. this point is also discussed at length by @xcite . social attribution of heat-maps. as mentioned above, the lack of a clear perception of a causal chain behind heat-map feature attribution explanations in nlp makes it difficult to discuss the social intent attributed by these methods. nevertheless, it is possible to do so under two perspectives: (1) when the heat-map is discretized into a highlight, and thus can be analyzed along the list of possible attributions in contrastive explanations in nlp. we are not aware of prior work that discusses or implements contrastive explanations explicitly in nlp, however this does not imply that existing explanation methods in nlp are not contrastive. to the contrary, the social sciences argue that every manner of explanation has a foil, and is comprehended by the explainee against some foil-including popular methods such as lime @xcite and @xcite . the question then becomes what the foil is, and whether this foil is intuitive and thus useful. in the case of lime, for example, the foil is defined by the aggregation of all possible perturbations admitted to the approximating linear model-where such perturbations may not be natural language, and thus less intuitive as foil; additionally, @xcite have recently derived the foil behind general shapley value-based explanations, and have shown that this foil is not entirely aligned with human intuition. we argue that making the foil explicit and intuitive is an important goal of any interpretation system. inter-disciplinary research. research on explanations in artificial intelligence will benefit from a deeper interdisciplinary perspective on two axes: (1) literature on causality and causal attribution, regarding causal effects in a model's reasoning process; and (2) literature on the social perception and attribution of human-like intent to causal chains of model decisions or behavior. ## related work how interpretations are comprehended by people is related to simulatability @xcite -the degree to which humans can simulate model decisions. quantifying simulatability @xcite ) is decidedly different from social alignment, since, for example, it will not necessarily detect dominant selectors. we theorize that aligned faithful interpretations will increase simulatability. predict-select-verify is reminiscent of iterative erasure @xcite . by iteratively removing ''significant'' tokens in the input, feng et al. show that a surprisingly small portion of the input could be interpreted as evidence for the model to make the prediction, leading to conclusions on the pathological nature of neural models and their sensitivity to badly-structured text. this experiment retroactively serves as a successful application of debugging using our formulation. the approach by @xcite for class-wise highlights is reminiscent of contrastive highlights, but nevertheless distinct, since such highlights still explain a fact against all foils. ## conclusion highlights are a popular format for explanations of decisions on textual inputs, for which there are models available today with the ability to derive highlights ''faithfully''. we analyze highlights as a case study in pursuit of rigorous formalization of quality artificial intelligence explanations. we redefine faithfulness as the accurate representation of the causal chain of decision making in the model, and aligned faithfulness as a faithful interpretation which is also aligned to the social attribution of intent behind the causal chain. the two steps of causal attribution and social attribution together complete the process of ''explaining'' the decision process of the model to humans. with this formalization, we characterize various failures in faithful highlights that ''seem'' strange, but could not be properly described previously, noting they are not properly constrained by their social attribution as summaries or evidence. we propose an alternative which can be constrained to serve as evidence. finally, we implement our alternative by formalizing contrastive explanations in the highlight format.