On the Diversity and Limits of Human Explanations (2022)

Abstract: A growing effort in NLP aims to build datasets of human explanations. However, it remains unclear whether these datasets serve their intended goals. This problem is exacerbated by the fact that the term explanation is overloaded and refers to a broad range of notions with different properties and ramifications. Our goal is to provide an overview of the diversity of explanations, discuss human limitations in providing explanations, and ultimately provide implications for collecting and using human explanations in NLP.Inspired by prior work in psychology and cognitive sciences, we group existing human explanations in NLP into three categories: proximal mechanism, evidence, and procedure. These three types differ in nature and have implications for the resultant explanations. For instance, procedure is not considered explanation in psychology and connects with a rich body of work on learning from instructions. The diversity of explanations is further evidenced by proxy questions that are needed for annotators to interpret and answer “why is [input] assigned [label]”. Finally, giving explanations may require different, often deeper, understandings than predictions, which casts doubt on whether humans can provide valid explanations in some tasks.

## introduction with the growing interest in explainable nlp systems, the nlp community have become increasingly interested in building datasets of human explanations. these human explanations can ideally capture human reasoning of why a (correct) label is chosen. if this is indeed the case, they are hypothesized to aid models with additional supervision, train models that explain their own predictions, and evaluate machine-generated explanations @xcite . in fact, deyoung et al. (2020) already developed a leaderboard, where the implicit assumption is that humans can provide valid explanations and these explanations can in turn be uniformly considered as groundtruths. however, are these assumptions satisfied and can human explanations serve these goals? in this work, we aim to introduce prior relevant literature in psychology to the nlp community and argue against abusing the term explanations and prematurely assuming that human explanations provide valid reasoning for inferring a label. first, we point out the rich diversity in what the nlp community refer to as explanations and how researchers collect them. the term "explanation" is overloaded in the nlp and ai community: it often refers to many distinct concepts and outcomes. 1 for example, procedural instructions are different from explanations that attempt to convey proximal causal mechanisms. the diversity of explanations is further evidenced by the variety of proxy questions that researchers ask to collect explanations, e.g., "highlight the important words that would tell someone to see the movie" vs. "highlight all words that reflect a sentiment". these proxy questions are necessary because the question of "why is @xcite assigned @xcite " is too open-ended. it follows that these "human explanations" are supposed to answer different questions in the first place and may not all be used for the same goals, e.g., serving as groundtruth labels. in addition to the diversity, we highlight two insights from psychology on whether humans can provide valid explanations: 1) prediction does not entail explanation @xcite , i.e., although humans may be able to provide valid labels, they may not be able to provide explanations that capture the reasoning process needed or used to infer a label; 2) everyday explanations are necessarily incomplete @xcite , because they seldom capture the complete deduc-tive processes from a set of axioms to a statement. in summary, not all explanations are equal and humans may not always be able to provide valid explanations. we encourage the nlp community to embrace the complex and intriguing phenomena behind human explanations instead of simply viewing explanations as another set of uniform labels. a better understanding and characterization of human explanations will inform how to collect and use human explanations in nlp. ## types of human explanations in nlp to understand whether datasets of human explanations can serve their intended goals, we first connect current human explanations in nlp with existing psychology literature to examine the use of the term "explanation" in nlp. we adapt the categorization in @xcite and group human explanations in nlp into the following three categories based on the conveyed information: these three categories empirically capture all the explanations discussed in nlp literature. @xcite also discuss two other categories, final causes (the goal) and material causes (the constituting substance). for instance, a final cause to "why [input] is assigned [label]" can be that "this label is provided to train a classifier". these two categories have been less relevant for nlp. implications. this categorization allows us to think about what kind of explanations are desired for nlp systems and help clarify how to use them appropriately. first, proximal mechanisms are best aligned with human intuitions of explanations, especially in terms of hinting at causal mechanisms. however, they can be difficult to collect for nlp tasks. for example, second, evidence by definition provides little information about the mechanisms behind a label, but it can be potentially useful as additional supervision or groundtruths. we will further elaborate on the nature of evidence in different tasks in §3. however, it may be useful to the community to use clear terminology (e.g., evidence or rationale @xcite ) to avoid lumping everything into "explanation". finally, procedures are essentially instructions, and @xcite explicitly distinguishes explanations from simple procedural knowledge: "knowing how to operate an automated teller machine or make an international phone call might not entail having any understanding of how either system instance label explanation task: whether the query is supported or refuted by the preceding texts @xcite ) s1: no way out is the debut studio album by american hip hop recording artist , songwriter and record producer puff daddy . s2: it was released on july 1 , 1997 , by his bad boy record label . the label 's official crediting as " the family " ...... query: 1997 was the year no way out was released. ## proxy questions used to collect human explanations although explanations are supposed to answer "why is [input] assigned [label]" @xcite , this literal form is too openended and may not induce "useful" human explanations. as a result, proxy questions are often necessary for collecting human explanations. these proxy questions further demonstrate the diversity of human explanations beyond the types of explanation. here we discuss these proxy questions for collecting evidence. see the appendix for discussions on proximal mechanisms and procedures. to collect evidence (highlights), researchers adopt diverse questions for relatively simple singletext classification tasks (see in contrast, personal attack detection poses a task where the negative class ("no personal attack") by definition points to the lack of evidence in the text. it follows that the questions that researchers can ask almost exclusively apply to the positive class (i.e., "highlight sections of comments that they considered to constitute personal attacks"). in comparison, researchers approach evidence more uniformly for document-query classification tasks. they generally use similar proxy questions (e.g., @xcite and @xcite ask almost the same questions) and ask people to select sentences instead of words. that said, intriguing differences still exist: 1) @xcite simply ask annotators to provide accompanying rationales; 2) @xcite aim for "strong" reasons, which likely induces different interpretations among annotators; 3) @xcite collect questions, answer, and sentence indices at the same time, among which sentence indices can be used to find the corresponding sentences as evidence. it remains unclear how these task questions evidence: single-text classification @xcite sentiment analysis to justify why a review is positive, highlight the most important words and phrases that would tell someone to see the movie. to justify why a review is negative, highlight words and phrases that would tell someone not to see the movie. sen et al. (2020) sentiment analysis label the sentiment and highlight all words that reflect this sentiment. carton et al. (2018 ## can humans provide explanations? in order for human explanations to serve as additional supervision in training models and evaluate machine-generated explanations, human explanations need to provide valid mechanisms for a correct label. finally, we discuss challenges for humans to provide explanations of such qualities. conceptual framework. we situate our discussion in the psychological framework provided by @xcite to highlight what may be required to explain. wilson and keil (1998) examines where explanation falls in three central notions: prediction, understanding, and theories. they argue that these three notions "form a progression of increasing sophistication and depth with explanations falling between understanding and theories". for instance, we may be able to predict that a car will start when we turn the ignition switch, but few of us are able to explain in detail why this is so. in contrast, if a person is able to explain in detail why a car starts when you turn on the ignition switch, they can likely predict what will happen if various parts of the engine are damaged or removed. these three central notions are also essential in machine learning. traditional label annotation is concerned with prediction, however, being able to predict does not entail being able to explain. emulation vs. discovery. next, we gradually unfold the practical challenges in collecting valid explanations from humans. the first challenge lies in whether humans can predict, i.e., assign the correct label. we highlight two types of tasks for ai: emulation vs. discovery @xcite . in emulation tasks, models are trained to emulate human intelligence and labels are often crowdsourced. labels, however, can also derive from external (social/biological) processes, e.g., the popularity of a tweet and the effect of a medical treatment. mod-els can thus discover patterns that humans may not recognize in these discovery tasks. while most nlp tasks such as nli and qa are emulation tasks, many nlp problems, especially when concerned with social interaction, are discovery tasks, ranging from identifying memorable movie quotes to predicting the popularity of messages @xcite . aligning with our discussion on explanation and prediction, most datasets of human explanations in nlp assume that humans are able to predict and are on emulation tasks. however, we note exceptions such as explanations of actions in gaming @xcite , where humans may often choose suboptimal actions (labels). cognitive challenges in providing valid explanations. even conditioned on that humans can predict the label, humans may not be able to provide valid explanations for at least two reasons. first, as wilson and keil (1998) suggests, explanation requires more depth than prediction. for instance, we may possess some notions of common sense (e.g., one should not slap a stranger), but it is unclear whether we can explain common sense in detail (e.g., why one should not slap a stranger through theory of morality), similar to the car ignition example. one may argue that theory of morality may not be what nlp researchers seek, but it is critical to consider the desiderata of human explanations, with the limits in mind. second, explanation often requires people to report their subjective mental processes, i.e., how our minds arrive at a particular judgement, rather than following objective consensual guidelines such as annotating logical entailment. however, classic work by @xcite suggests that our verbal reports on our mental processes can be highly inaccurate. for instance, in admission decisions, legitimate information can be used to justify preferences based on illegitimate factors such as race @xcite . many studies on implicit bias also reinforces that we are not aware of our biases and thus cannot include them (i.e., the actual reasoning in our mind) in our explanations @xcite . explanations are necessarily incomplete. finally, there are indeed cases where we believe that humans can provide valid mechanisms. for instance, some question answering tasks boil down to logical inference from evidence to query. in these cases, nlp researchers need to recognize that human explanations are necessarily incomplete: people do not start from a set of axioms and present all the deductive steps @xcite . therefore, even for simple tasks such as natural language inference, we may simply give explanations such as repeating the hypothesis without presenting any axiom or deduction required to infer the label. implications. we cannot assume that humans are capable of providing explanations that contain valuable proximal mechanisms. the very fact that humans can still provide explanations for incorrect labels and tasks where they do not perform well suggests that one should be skeptical about whether human explanations can be used to train models as additional supervision or evaluate machine-generated explanations as groundtruths. note that incomplete explanations can still be very useful for nlp. we believe that recognizing and characterizing this incompleteness (e.g., which proximal mechanism is more salient to humans) is critical for understanding and leveraging human explanations for the intended goals in nlp. to summarize, we argue that human explanations are necessarily incomplete and it is important to understand and characterize this incompleteness, which can inform how we can leverage it for the intended goals in nlp. ## conclusion explanations represent a fascinating phenomenon and are actively studied in psychology, cognitive science, and other social sciences. while the growing interest in explanations from the nlp community is exciting, we encourage the community to view this as an opportunity to understand how humans approach explanations and contribute to understanding and exploring the explanation processes. this will in turn inform how to collect and use human explanations in nlp. a modest proposal is that it is useful to examine and characterize human explanations before assuming that all explanations are equal and chasing a leaderboard.