ERASER : A Benchmark to Evaluate Rationalized NLP Models (2020)

Abstract: State-of-the-art models in NLP are now predominantly based on deep neural networks that are opaque in terms of how they come to make predictions. This limitation has increased interest in designing more interpretable deep models for NLP that reveal the ‘reasoning’ behind model outputs. But work in this direction has been conducted on different datasets and tasks with correspondingly unique aims and metrics; this makes it difficult to track progress. We propose the E valuating R ationales A nd S imple E nglish R easoning ( ERASER a benchmark to advance research on interpretable models in NLP. This benchmark comprises multiple datasets and tasks for which human annotations of “rationales” (supporting evidence) have been collected. We propose several metrics that aim to capture how well the rationales provided by models align with human rationales, and also how faithful these rationales are (i.e., the degree to which provided rationales influenced the corresponding predictions). Our hope is that releasing this benchmark facilitates progress on designing more interpretable NLP systems. The benchmark, code, and documentation are available at https://www.eraserbenchmark.com/

## introduction interest has recently grown in designing nlp systems that can reveal why models make specific predictions. but work in this direction has been conducted on different datasets and using different metrics to quantify performance; this has made it difficult to compare methods and track progress. we aim to address this issue by releasing a standardized benchmark of datasets -repurposed and augmented from pre-existing corpora, spanning a range of nlp tasks -and associated metrics for measuring different properties of rationales. we refer to this as the evaluating rationales and simple english reasoning (eraser ) benchmark. ## related work interpretability in nlp is a large, fast-growing area; we do not attempt to provide a comprehensive overview here. instead we focus on directions particularly relevant to eraser, i.e., prior work on models that provide rationales for their predictions. learning to explain. in eraser we assume that rationales (marked by humans) are provided during training. however, such direct supervision will not always be available, motivating work on methods that can explain (or "rationalize") model predictions using only instance-level supervision. in the context of modern neural models for text classification, one might use variants of attention @xcite to extract rationales. attention mechanisms learn to assign soft weights to (usually contextualized) token representations, and so one can extract highly weighted tokens as rationales. however, attention weights do not in general provide faithful explanations for predictions @xcite @xcite @xcite @xcite . this likely owes to encoders entangling inputs, complicating the interpretation of attention weights on inputs over contextualized representations of the same. 2 by contrast, hard attention mechanisms discretely extract snippets from the input to pass to the classifier, by construction providing faithful explanations. recent work has proposed hard attention mechanisms as a means of providing explanations. @xcite proposed instantiating two models with their own parameters; one to extract rationales, and one that consumes these to make a prediction. they trained these models jointly via reinforce (williams, 1992) style optimization. recently, @xcite proposed a variant of this two-model setup that uses heuristic feature scores to derive pseudo-labels on tokens comprising rationales; one model can then be used to perform hard extraction in this way, while a second (independent) model can make predictions on the basis of these. elsewhere, @xcite introduced the notion of classwise rationales that explains support for different output classes using a game theoretic framework. finally, other recent work has proposed using a differentiable binary mask over inputs, which also avoids recourse to reinforce @xcite . post-hoc explanation. another strand of interpretability work considers post-hoc explanation methods, which seek to explain why a model made a specific prediction for a given input. commonly these take the form of token-level importance scores. gradient-based explanations are a standard example @xcite . these enjoy a clear semantics (describing how perturbing inputs locally affects outputs), but may nonetheless exhibit counterintuitive behaviors @xcite . gradients of course assume model differentiability. other methods do not require any model properties. examples include lime @xcite and @xcite ; these methods approximate model behavior locally by having it repeatedly make predictions over perturbed inputs and fitting a simple, explainable model over the outputs. acquiring rationales. aside from interpretability considerations, collecting rationales from annotators may afford greater efficiency in terms of model performance realized given a fixed amount of annotator effort @xcite . in particular, recent work by @xcite has observed that at least for some tasks, asking annotators to provide rationales justifying their categorizations does not impose much additional effort. combining rationale annotation with active learning @xcite is another promising direction @xcite . learning from rationales. work on learning from rationales marked by annotators for text classification dates back over a decade @xcite . earlier efforts proposed extending standard discriminative models like support vector machines (svms) with regularization terms that penalized parameter estimates which disagreed with provided rationales @xcite . other efforts have attempted to specify generative models of rationales @xcite . more recent work has aimed to exploit rationales in training neural text classifiers. @xcite proposed a rationale-augmented convolutional neural network (cnn) for text classification, explicitly trained to identify sentences supporting categorizations. @xcite showed that providing this model with rationales during training yields predicted rationales that are preferred by humans (compared to rationales produced without explicit supervision). other work has proposed 'pipeline' approaches in which independent models are trained to perform rationale extraction and classification on the basis of these, respectively @xcite , assuming name size (train/dev/test) tokens comp? evidence inference 7958 / 972 / 959 4761 ◇ boolq 6363 / 1491 / 2817 3583 ◇ movie reviews 1600 / 200 / 200 774 ◆ fever 97957 / 6122 / 6111 327 ! multirc 24029 / 3214 / 4848 303 ! cos-e 8733 / 1092 / 1092 28 ! e-snli 911938 / 16449 / 16429 16 ! measuring agreement between extracted and human rationales (or collecting subjective assessments of them) assesses the plausibility of rationales, but such approaches do not establish whether the model actually relied on these particular rationales to make a prediction. we refer to rationales that correspond to the inputs most relied upon to come to a disposition as faithful. most automatic evaluations of faithfulness measure the impact of perturbing or erasing words or tokens identified as important on model output @xcite @xcite @xcite . we build upon these methods in section 4. finally, we note that a recent article urges the community to evaluate faithfulness on a continuous scale of acceptability, rather than viewing this as a binary proposition @xcite . ## datasets in eraser for all datasets in eraser we distribute both reference labels and rationales marked by humans as supporting these in a standardized format. we delineate train, validation, and test splits for all corpora (see appendix a for processing details). we ensure that these splits comprise disjoint sets of source documents to avoid contamination. 3 we have made the decision to distribute the test sets publicly, foot_3 in part because we do not view the 'correct' metrics to use as settled. we plan to acquire additional human annotations on held-out portions of some of the included corpora so as to offer hidden test set evaluation opportunities in the future. evidence inference @xcite . a dataset of full-text articles describing randomized controlled trials (rcts). the task is to infer whether a given intervention is reported to either significantly increase, significantly decrease, or have no significant effect on a specified outcome, as compared to a comparator of interest. rationales have been marked as supporting these inferences. as the original annotations are not necessarily exhaustive, we collected exhaustive rationale annotations on a subset of the validation and test data. 5 boolq @xcite . this corpus consists of passages selected from wikipedia, and yes/no questions generated from these passages. as the original wikipedia article versions used were not maintained, we have made a best-effort attempt to recover these, and then find within them the passages answering the corresponding questions. for public release, we acquired comprehensive annotations on a subset of documents in our test set. 5 movie reviews @xcite . includes positive/negative sentiment labels on movie reviews. original rationale annotations were not necessarily comprehensive; we thus collected comprehensive rationales on the final two folds of the original dataset @xcite . 5 in contrast to most other datasets, the rationale annotations here are span level as opposed to sentence level. fever @xcite . short for fact extraction and verification; entails verifying claims from textual sources. specifically, each claim is to be classified as supported, refuted or not enough information with reference to a collection of source texts. we take a subset of this dataset, including only supported and refuted claims. multirc @xcite . a reading comprehension dataset composed of questions with multiple correct answers that by construction depend on information from multiple sentences. here each rationale is associated with a question, while answers are independent of one another. we convert each rationale/question/answer triplet into an instance within our dataset. each answer candidate then has a label of true or false. commonsense explanations (cos-e) @xcite . this corpus comprises multiplechoice questions and answers from @xcite along with supporting rationales. the rationales in this case come in the form both of highlighted (extracted) supporting snippets and freetext, open-ended descriptions of reasoning. given our focus on extractive rationales, eraser includes only the former for now. following @xcite , we repartition the training and validation sets to provide a canonical test split. ## metrics in eraser models are evaluated both for their predictive performance and with respect to the rationales that they extract. for the former, we rely on the established metrics for the respective tasks. here we describe the metrics we propose to evaluate the quality of extracted rationales. we do not claim that these are necessarily the best metrics for evaluating rationales, however. indeed, we hope the release of eraser will spur additional research into how best to measure the quality of model explanations in the context of nlp. ## baseline models our focus in this work is primarily on the eraser benchmark itself, rather than on any particular model(s). but to establish a starting point for future work, we evaluate several baseline models across the corpora in eraser. 8 we broadly classify these into models that assign 'soft' (continuous) scores to tokens, and those that perform a 'hard' (discrete) selection over inputs. we additionally consider models specifically designed to select individual tokens (and very short sequences) as rationales, as compared to longer snippets. all of our implementations are in pytorch @xcite and are available in the eraser repository. 9 all datasets in eraser comprise inputs, rationales, and labels. but they differ considerably in document and rationale lengths (table a ). this motivated use of different models for datasets, appropriate to their sizes and rationale granularities. we hope that this benchmark motivates design of models that provide rationales that can flexibly adapt to varying input lengths and expected rationale granularities. indeed, only with such models can we perform comparisons across all datasets. ## evaluation here we present initial results for the baseline models discussed in section 5, with respect to the metrics proposed in section 4. we present results in two parts, reflecting the two classes of rationales discussed above: 'hard' approaches that perform discrete selection of snippets, and 'soft' methods that assign continuous importance scores to tokens. in due to computational constraints, we were unable to run our bert-based implementation of @xcite over larger corpora. conversely, the simple pipeline of @xcite assumes a setting in which rationale are sentences, and so is not appropriate for datasets in which rationales tend to comprise only very short spans. again, in our view this highlights the need for models that can rationalize at varying levels of granularity, depending on what is appropriate. we observe that for the "rationalizing" model of @xcite , exploiting rationale-level supervision often (though not always) improves agreement with human-provided rationales, as in prior work @xcite . interestingly, this does not seem strongly correlated with predictive performance. @xcite outperforms the simple pipeline model when using a bert encoder. further, @xcite outperforms the 'bert-to-bert' pipeline on the comparable datasets for the final prediction tasks. this may be an artifact of the amount of text each model can select: 'bert-to-bert' is limited to sentences, while @xcite can select any subset of the text. designing extraction models that learn to adaptively select contiguous rationales of appropriate length for a given task seems a potentially promising direction. in these scoring functions assign only soft scores to inputs (and may still use all inputs to come to a particular prediction), so we report the metrics intended to measure faithfulness defined above: comprehensiveness and sufficiency, averaged over 'bins' of tokens ordered by importance scores. to provide a point of reference for these metricswhich depend on the underlying model -we report results when rationales are randomly selected (averaged over 10 runs). both simple gradient and lime-based scoring yield more comprehensive rationales than attention weights, consistent with prior work @xcite . attention fares better in terms of auprc -suggesting better agreement with human rationales -which is also in line with prior findings that it may provide plausible, but not faithful, explanation @xcite . interestingly, lime does particularly well across these tasks in terms of faithfulness. from the 'random' results that we conclude models with overall poor performance on their final tasks tend to have an overall poor ordering, with marginal differences in comprehensiveness and sufficiency between them. for models that with high sufficiency scores: movies, fever, cos-e, and e-snli, we find that random removal is particularly damaging to performance, indicating poor absolute ranking; whereas those with high comprehensiveness are sensitive to rationale length. ## conclusions and future directions we have introduced a new publicly available resource: the evaluating rationales and simple english reasoning (eraser) benchmark. this comprises seven datasets, all of which include both instance level labels and corresponding supporting snippets ('rationales') marked by human annotators. we have augmented many of these datasets with additional annotations, and converted them into a standard format comprising inputs, rationales, and outputs. eraser is intended to facilitate progress on explainable models for nlp. we proposed several metrics intended to measure the quality of rationales extracted by models, both in terms of agreement with human annotations, and in terms of 'faithfulness'. we believe these metrics provide reasonable means of comparison of specific aspects of interpretability, but we view the problem of measuring faithfulness, in particular, a topic ripe for additional research (which eraser can facilitate). our hope is that eraser enables future work on designing more interpretable nlp models, and comparing their relative strengths across a variety of tasks, datasets, and desired criteria. it also serves as an ideal starting point for several future directions such as better evaluation metrics for interpretability, causal analysis of nlp models and datasets of rationales in other languages. or questions with rationales that were not possible to automatically map back to the underlying text. as recommended by the authors of @xcite we repartition the train and validation sets into a train, validation, and test set for this benchmark. we encode the entire question and answers as a prompt and convert the problem into a five-class prediction. we also convert the "sanity" datasets for user convenience. all datasets in eraser were tokenized using spacy foot_9 library (with scispacy @xcite for evidence inference). in addition, we also split all datasets except e-snli and cos-e into sentences using the same library.