An Investigation of Language Model Interpretability via Sentence Editing (2021)

Abstract: Pre-trained language models (PLMs) like BERT are being used for almost all language-related tasks, but interpreting their behavior still remains a significant challenge and many important questions remain largely unanswered. In this work, we re-purpose a sentence editing dataset, where faithful high-quality human rationales can be automatically extracted and compared with extracted model rationales, as a new testbed for interpretability. This enables us to conduct a systematic investigation on an array of questions regarding PLMs’ interpretability, including the role of pre-training procedure, comparison of rationale extraction methods, and different layers in the PLM. The investigation generates new insights, for example, contrary to the common understanding, we find that attention weights correlate well with human rationales and work better than gradient-based saliency in extracting model rationales. Both the dataset and code will be released to facilitate future interpretability research.

## introduction pre-trained language models (plms) @xcite @xcite are pervasively used in language-related tasks, but interpreting their predictions is notoriously difficult because of their parameters' complex interdependencies. given a specific prediction, we want to know why a model made that decision, both to further improve performance and to use the model in high-stakes scenarios such as healthcare or bank loan approvals where interpretability is important. this has motivated efforts in extracting model explanations, typically in the form of rationales, i.e., subsets of the original input that support a decision @xcite . attention heatmaps @xcite and gradient-based saliency maps @xcite are common extraction methods. the @xmath0. the algorithm @xmath1. however, @xmath2, respectively. however, we must note that we still have no means of deciding which documents out of _math_ deserve to be in _math_ and _math_, respectively. figure 1: two "need edit" examples from aesw in the original data format and a human-readable format. the first example (a) has a spelling error "descripted" and the second (b) is edited for concision. there have been efforts on developing datasets for interpretability research, for example, the recent eraser benchmark @xcite . however, the majority of eraser tasks use human rationales highlighted by a different annotator after the original labeling process. such rationales are not necessarily faithful; a rationale highlighted by the second annotator may not have been actually used by the first annotator while labeling. manual rationale labeling is also difficult and timeconsuming; of the six datasets in the eraser benchmark, only one has more than 200 examples. our first contribution is the realization that aesw (automatic evaluation of scientific writing; @xcite , a sentence editing dataset, contains thousands of faithful human rationales that can be automatically re-purposed for interpretability research. see figure 1 for examples. this provides a new, large-scale dataset with truly faithful human rationales for interpretability questions surrounding model rationales. our second contribution is investigating multiple factors in plm rationale plausibility. more plausible rationales are valuable in human-in-the-loop systems where humans use model rationales to make a final decision. we compare (1) pretraining procedures, (2) attention weight-and input gradient-based methods of extracting model rationales, (3) correlation between model rationale plausibility and model confidence, and (4) differences in transformer layers. while previous work @xcite has shown that attention weights are not always faithful, we find that they correlate with human rationales better than gradient-based methods. ## related work human rationales (as defined by @xcite are subsets of input highlighted by human annotators as evidence to support a decision. the same annotator labeling an example might also highlight their rationale @xcite . in other cases, rationales are collected for an existing dataset by different annotators @xcite @xcite . as previously stated, such rationales may not be faithful. rationale length can vary from sub-sentence spans @xcite to multiple sentences @xcite . model rationales can be produced as an explicit training objective @xcite or extracted as a post-hoc explanation. post-hoc methods typically assign token-level importance scores: attention weights are often used in attention-based models @xcite , gradient-based explanations are typical for differentiable models @xcite , and lime is a model-agnostic method @xcite . we follow work using bert's attention @xcite to extract rationales. a model rationale is evaluated on faithfulness (if it is actually used to make a decision) and plausibility (if it is easily understood by humans). faithfulness can be measured by perturbing inputs marked as evidence and measuring change in outputs @xcite . plausibility can be measured through user studies, wherein users are given a model rationale and asked either to predict the model's decision @xcite or to rate rationale understandability @xcite @xcite . rationale plausibility can also be measured by similarity to human rationales @xcite , but this requires faithful human rationales. we use similarity to evaluate rationale plausibility because we gather faithful human rationales from sentence editing annotations. ## proposed task we propose re-purposing the aesw classification task for measuring model interpretability. we gather examples from aesw from which we can automatically extract faithful and sufficient human rationales, and then use said rationales to investigate factors in plm rationale plausibility, specifically bert @xcite and its variants. it is worth noting that our rationale dataset can be used for other interpretability topics such as training with rationales or evaluating rationale faithfulness; we will focus on plausibility considering the scope of this paper. ## experiments to demonstrate the utility of the aesw task for interpretability research, we present four experiments, each with the goal of understanding factors in plm rationale plausibility. for our experiments, we add a linear layer and sigmoid activation function on top of the [cls] token representation, fine-tune bert-base end-toend on the aesw training set using the original training objective (classify a sentence as "need edit" or "no edit") and use validation loss to tune hyperparameters. we do not add any interpretability or rationale-related objectives. the classification f1 for the models on spelling error and deleted text edits is 82.0 and 74.0, respectively. 4 we extract and evaluate model rationales on all spelling error and deleted text edits, even edits that a model does not correctly predict as "need edit." does pre-training procedure affect rationale plausibility? we are interested in how pretraining affects plausibility after fine-tuning (to the best of our knowledge, this is a previously unexplored topic). we compare bert and two variants, roberta @xcite and @xcite . these models have the same architecture but differ in pre-training: roberta is pre-trained for 5x longer with 10x more data than bert, and scibert is pre-trained on a corpus of academic papers. we extract model rationales using attention weights and evaluate their plausibility. as seen in figure 2 , roberta and bert generate nearly equally plausible rationales despite differences in pre-training corpus size. we hypothesize that scibert generates less plausible rationales because it encodes "need edit" representations in earlier layers (rather than in the final layer), then attends to [sep] as a no-op in later layers, as proposed in @xcite and @xcite and further confirmed in the subsequent experiments. bert roberta scibert 0.0 0.2 0.4 0.6 0.8 mean reciprocal rank pre-training scheme attention grad. (dir.) grad. (mag.) extracting method spelling errors deleted text figure 2: left: bert, roberta and scibert's attention weight rationale plausibility. right: bert's attention weight, gradient×input and |gradient × input| rationale plausibility (see section 3.2). do attention weights or input gradients produce better rationales? in contrast to attention weights, gradient×input scores are naturally faithful with respect to individual feature importance @xcite . however, attention weights can represent word relevance in context, potentially leading to more plausible rationales. we extract and evaluate rationales using attention weights and the two gradient×input methods described in section 3.2. figure 2 shows that attention-based rationales are more plausible and that the difference is more pronounced on deleted text edits. using |gradient× input| (right-most) also shows improvements over directional gradient×input (middle-right), in contrast to @xcite . ## conclusion we re-purpose the aesw task to gather thousands of inherently faithful human rationales and investigate an array of questions regarding plm interpretability. we find, among other new insights, that attention weights correlate well with human rationales and produce more plausible rationales than input gradients, which is different from existing understanding. furthermore, we find that bert is more confident in its predictions when it attends to the same words that a human did, supporting the idea that while attention is not inherently faithful, attention-based models might rely on the same information as humans when making a prediction. future work might expand the subset of examples for which human rationales can be automatically extracted, include human rationales during training or evaluate faithful-by-design model rationales on this dataset.