Evaluating and Characterizing Human Rationales (2020)

Abstract: Two main approaches for evaluating the quality of machine-generated rationales are: 1) using human rationales as a gold standard; and 2) automated metrics based on how rationales affect model behavior. An open question, however, is how human rationales fare with these automatic metrics. Analyzing a variety of datasets and models, we find that human rationales do not necessarily perform well on these metrics. To unpack this finding, we propose improved metrics to account for model-dependent baseline performance. We then propose two methods to further characterize rationale quality, one based on model retraining and one on using “fidelity curves” to reveal properties such as irrelevance and redundancy. Our work leads to actionable suggestions for evaluating and characterizing rationales.

## introduction explanations in nlp often take the form of rationales, subsets of input tokens that are considered important to the model's decision @xcite . as interest in explainable ai has increased, so has interest in evaluating the quality of explanatory rationales. however, this is a challenging task because it can be difficult to pin down exactly what constitutes "good" rationales for model predictions @xcite @xcite . two main strategies that have been proposed in recent work are: 1) to view human-generated rationales as a gold standard and evaluate modelgenerated rationales in comparison to them; and 2) to assess the "fidelity" of a rationale to a prediction using automatic metrics. the human-gold-standard approach views rationales as an additional form of label that can be collected alongside document-level labels. because * equal contribution. nlp tasks tend to involve human-generated labels, it makes intuitive sense that human-generated rationales might be considered authoritative. when human rationales are not available, evaluations of machine rationales turn to automatic metrics. these metrics divorce rationale evaluation from an external standard, seeking instead to judge whether rationales are coherent relative to model behavior. popular recent metrics are sufficiency and comprehensiveness (i.e., necessity), which assess whether a rationale is sufficient/necessary for a model prediction by comparing the model's behavior on the full input to its behavior on input masked according to the rationale or its complement. we use the term fidelity to refer jointly to sufficiency and comprehensiveness. to the best of our knowledge, no existing work has systematically examined human rationales using these automatic metrics. however, this is an important step towards evaluating rationales because it helps characterize the disparities between the two types of approach. are human rationales sufficient to allow models to predict human labels? are they comprehensive? and what other insights can we gain about human rationales and fidelity metrics by performing this assessment? in practice, both human rationales and automatic metrics can fail to work as intended ( by contrast, a truly sufficient rationale can be deemed insufficient due to a model not learning expected classification rules (e.g., "sits" ∼ "laying" in in to investigate the empirical properties of human rationales and automatic metrics, we analyze the fidelity of human rationales across six datasets. we show that human rationales do not necessarily have high sufficiency or comprehensiveness based on automatic metrics, and their fidelity varies greatly from model to model and class to class. we propose extensions to existing fidelity metrics and develop novel methods to further characterize the quality of human rationales. first, we note that fidelity is highly model-dependent, and that model behavior can result in misleading fidelity results. we propose a normalization procedure to allow for fair comparison of these metrics across models, classes, and datasets. we show that this normalization helps contextualize fidelity results by accounting for baseline model behavior. second, we evaluate model accuracy on full vs. rationale-only data, linking typical output-sufficiency to performance outcomes (i.e., accuracy-sufficiency). we examine the effect of allowing models to adapt to rationale-only data during training, drawing a distinction between a rationale's "incidental" fidelity and its "potential" fidelity to a model. we analyze the effect of these two interventions and discuss their implications for evaluation of (and learning from) human rationales. finally, we introduce the idea of "fidelity curves", which examine how sufficiency and comprehensiveness degrade as tokens are randomly occluded from a rationale. we discuss how the shapes of these curves can be used to infer fine-grained attributes about rationales, such as the extent to which they contain redundant or highly interdepen-dent tokens. we find that rationales in our datasets vary greatly in their level of irrelevancy, redundancy, and mutual dependence. we find that our three classification tasks exhibit less dependence and more redundancy in their rationales than our three document/query-style tasks. evaluating rationales is a significant challenge. we argue that in order to be confident in either human rationales or automatic fidelity metrics, we have to understand how these two evaluation approaches interact with one another, and what caveats they can reveal about each other. our analyses lead to the following actionable implications: • fidelity metrics are highly model-dependent and should be normalized to assist interpretation. ## datasets the goal of this paper is to evaluate and characterize human rationales. we analyze six datasets, four drawn from the eraser collection @xcite , and two from other sources. they consist of three single-text classification tasks and three document/query-style tasks where it is important to understand the relations between texts. for each dataset, the human rationales have a qualitative expected comprehensiveness based on whether, by construction or design, they are intended to contain all pertinent information for their respective prediction task. four of our six datasets are expected to have comprehensive rationales. fig. 1 shows class asymmetry in rationale percentages. for wikiattack, labelers were asked to highlight personal attacks, and thus evidence for the no-attack class comes in the form of no highlighted tokens. this results in a situation where rationales for no-attack examples constitute less than 5% on average, while they constitute 35% of personal-attack examples. significant variation between classes also exists in e-snli: entailment contains close to 40% of tokens as rationales, but neutral merely consists of 16% -another case of evidence through absence (negative evidence). ## evaluating human rationales popular automatic metrics for evaluating machinegenerated rationales are sufficiency and comprehensiveness, articulated by @xcite and em-ployed in the eraser benchmark @xcite . sufficiency measures how well rationales can provide the same prediction as using full information, while comprehensiveness measures how well rationales include all relevant information. it remains an open question whether humangenerated rationales have good sufficiency and comprehensiveness. we find that this is in fact not necessarily the case. this result reveals a contradiction in the evaluation of machine-generated rationales: human-generated rationales are used as a gold standard, but being similar to humangenerated rationales may not lead to high sufficiency and comprehensiveness. another important observation from our experiments is that there exists significant variation between datasets and classes within the same dataset. ## normalizing sufficiency and comprehensiveness human rationales do not necessarily have high fidelity, suggesting that either human rationales or evaluation metrics may be problematic. we start by rethinking the fidelity metrics in this section and will propose novel methods to characterize human rationales in §5. a salient observation in fig. 2 is that sufficiency and comprehensiveness are in completely separate value ranges, although they are both theoretically bounded between 0 and 1. to properly interpret these numbers, we need to establish a baseline for them. we do so by defining a "null difference", the difference in output between the model operating on full information vs. no information (i.e., the empty input). this value is equivalent to (the complement of) the sufficiency of an all-zero (empty) rationale mask, or the comprehensiveness of an all-one mask. null difference is an intrinsic value for a given model and dataset, and depends on the class balance of the dataset, the bias term(s) learned by the model, and the calibration of output probability. it serves as a baseline value in the sense that no rationale should be much less sufficient than an allzero rationale or much more comprehensive than an all-one rationales. by normalizing sufficiency and comprehensiveness scores against this value, we can estimate how faithful rationales are relative to the baseline fidelity of the model. ## characterizing human rationales beyond sufficiency/comprehensiveness sufficiency and comprehensiveness offer a limited perspective on the qualities of rationales. for example, does the 0.77 e-snli sufficiency reported in fig. 2b correspond with a similar drop in accuracy, or do the rationales render the model less confident but equally accurate? and how can we distinguish between a highly concise rationale and one bloated with unnecessary information? we propose extensions of the basic fidelity framework to address these more nuanced questions. ## related work we summarize additional related work in the following three areas. feature attribution. feature attribution seeks to explain model behavior by attributing model predictions to specific inputs. popular techniques include lime @xcite , integrated gradients @xcite , shap (lundberg and lee, 2017), and attention mechanisms @xcite . human rationales. many recent datasets in nlp have been released with rationales accompanying the document-level labels. eraser @xcite includes three additional datasets: cos- sufficiency comprehensiveness wikiattack slow drop fast drop sst slow drop fast drop movie slow drop fast drop multirc fast drop @xmath0. figure 7: fidelity curves for all datasets (normalized sufficiency and comprehensiveness). human rationales tend to be redundant in the single-text classification datasets, and dependent in the document/query-style datasets. e (rajani et al., 2019), boolq (clark et al., 2019), and evidence inference (lehman et al., 2019). other rationale datasets include that of kaushik et al. (2019) and sen et al. (2020). attribution evaluation. a growing amount of work seeks to evaluate the quality of feature attribution. beyond collecting human rationales as a gold-standard, a common human-based method is to test the utility of attribution masks in task-based human subject experiments (carton et al., 2020; lai and tan, 2019; poursabzi-sangdeh et al., 2018; @xcite . automatic model-based metrics beyond sufficiency and comprehensiveness include local model fidelity @xcite , switching point @xcite , and area-over-the-perturbationcurve @xcite . ## concluding discussion human explanations contain a lot of promise. the explainable ai community hopes to use them as a guide for evaluating model explanations and, possibly, for teaching models to make robust and wellreasoned decisions. in this work, we contribute to that effort by analyzing human rationales through the lens of automatic rationale evaluation methods, namely, sufficiency and comprehensiveness. we find that human rationales do not necessarily have high sufficiency or comprehensiveness. interpreting fidelity variance. furthermore, there exists significant variance across datasets and classes. in §5.2, we speculate that some of these differences (e.g., dependency) can be explained by the semantic differences between classification and document/query-style tasks. however, with such a small sample size of datasets (@xmath1, it is difficult to determine whether these differences are due solely to task type or to other factors such as annotation instructions or individual dataset semantics. wikiattack and e-snli, for example, display class asymmetry in their rationales, which likely contribute to their outlier status in fig. 4 and 6 respectively. as we note in fig. 2 , modeling outcomes also have a heavy impact on explanation fidelity. while e-snli comprises an even class balance, our model learns a strong bias in favor of the neutral class, which contributes to a class imbalance in fidelity for that dataset (fig. 4 ). as more human-rationale datasets are released, it will become increasingly possible to categorize them by rationale properties. our goal is to highlight the variance in these properties and call for more widespread empirical evaluations thereof. actionable implications. when human rationales are found to be unfaithful, this can mean that either they fail to capture relevant signal, or that the model improperly utilizes that signal, perhaps as a result of learning spurious associations. in either case, analysis can expose inconsistencies between human and model understanding of the task. we propose three ways to extend fidelity metrics: normalization, model adaptation, and random ablation. each addresses one shortcoming of the basic metric: normalization addresses the differences in class biases across models, adaptation the problem of domain inconsistency between full and rationale-only data, and ablation the inability of existing metrics to capture qualities like redundancy. while not all of these issues are salient for every application involving rationale fidelity, we offer them as potential solutions where necessary. overall, our results suggest that the idea of onesize-fits-all fidelity benchmarks might be problematic: human rationales may not be simply treated as gold standard. we need to design careful procedures to collect human rationales, understand properties of the resulting human rationales, and cautiously interpret the evaluation metrics.