L earning to Faithfully Rationalize by Construction (2020)

Abstract: In many settings it is important for one to be able to understand why a model made a particular prediction. In NLP this often entails extracting snippets of an input text ‘responsible for’ corresponding model output; when such a snippet comprises tokens that indeed informed the model’s prediction, it is a faithful explanation. In some settings, faithfulness may be critical to ensure transparency. Lei et al. (2016) proposed a model to produce faithful rationales for neural text classification by defining independent snippet extraction and prediction modules. However, the discrete selection over input tokens performed by this method complicates training, leading to high variance and requiring careful hyperparameter tuning. We propose a simpler variant of this approach that provides faithful explanations by construction. In our scheme, named FRESH, arbitrary feature importance scores (e.g., gradients from a trained model) are used to induce binary labels over token inputs, which an extractor can be trained to predict. An independent classifier module is then trained exclusively on snippets provided by the extractor; these snippets thus constitute faithful explanations, even if the classifier is arbitrarily complex. In both automatic and manual evaluations we find that variants of this simple framework yield predictive performance superior to ‘end-to-end’ approaches, while being more general and easier to train. Code is available at https://github.com/successar/FRESH .

## introduction neural models dominate nlp these days, but it remains difficult to know why such models make specific predictions for sequential text inputs. this problem has been exacerbated by the adoption of deep contextualized word representations, whose architectures permit arbitrary and interdependent 1 code is available at https://github.com/ successar/fresh interactions between all inputs, making it particularly difficult to know which inputs contributed to any specific prediction. concretely, in a bidirectional rnn or transformer model, the contextual embedding for a word at position j in instance x may encode information from any or all of the tokens at positions 1 to j-1 and j+1 to |x|. consequently, continuous scores such as attention weights @xcite induced over these contextualized embeddings reflect the importance not of individual inputs, but rather of unknown interactions between all input tokens. this makes it misleading to present heatmaps of these scores over the original token inputs as an explanation for a prediction @xcite @xcite . the key missing property here is faithfulness @xcite ): an explanation provided by a model is faithful if it reflects the information actually used by said model to come to a disposition. in some settings the ability of a model to provide faithful explanations may be paramount. for example, without faithful explanations, we cannot know whether a model is exploiting sensitive features such as gender @xcite . we propose an approach to neural text classification that provides faithful explanations for predictions by construction. following prior work in this direction @xcite , we decompose our model into independent extraction and prediction modules, such that the latter uses only inputs selected by the former. this discrete selection over inputs allows one to use an arbitrarily complex prediction network while still being able to guarantee that it uses only the extracted input features to inform its output. the main drawback to this rationalization approach has been the difficulty of training the two components jointly under only instance-level query: what is the only difference between a reflection in a mirror and the actual image ? | answer: it is exactly the same | label: false [human] you have seen your own reflection in a mirror . the person looking back at you looks just like you . where does that reflected person appear to be standing ? yes , they appear to be on the other side of the mirror . that is really strange to think about , but very cool . have you ever waved at your reflection in a mirror ? the reflected image will wave back at you . here is something to try next time you stand in front of a mirror . wave to your reflection with your right hand . what hand do you think the reflection will wave back with ? the same hand ? a different hand ? you will notice something interesting . the reflection waves back with the hand on the same side as you , but it is their left hand . the image in a reflection is reversed . this is just like the image of the sign above . light rays strike flat shiny surfaces and are reflected . the reflections are reversed . @xcite you have seen your own reflection in a mirror . the person looking back at you looks just like you . where does that reflected person appear to be standing ? yes , they appear to be on the other side of the mirror . that is really strange to think about , but very cool . have you ever waved at your reflection in a mirror ? the reflected image will wave back at you . here is something to try next time you stand in front of a mirror . wave to your reflection with your right hand . what hand do you think the reflection will wave back with ? the same hand ? a different hand ? you will notice something interesting . the reflection waves back with the hand on the same side as you , but it is their left hand . the image in a reflection is reversed . this is just like the image of the sign above . light rays strike flat shiny surfaces and are reflected . the reflections are reversed . figure 1 : contiguous rationales extracted using @xcite and fresh models for an example from the multirc dataset. we also show the reference rationale associated with this example (top). supervision (i.e., without token labels). this has necessitated training the extraction module via reinforcement learning -namely rein-force (williams, 1992) -which exhibits high variance and is particularly sensitive to choice of hyperparameters. recent work @xcite has proposed a differentiable mechanism to perform binary token selection, but this relies on the reparameterization trick, which similarly complicates training. methods using the reparameterization trick tend to zero out token embeddings, which may adversely affect training in transformer-based models, especially when one is not fine-tuning lower layers of the model due to resource constraints, as in our experiments. to avoid the complexity inherent to training under a remote supervision signal, we introduce faithful rationale extraction from saliency thresholding (fresh), which disconnects the training regimes of the extractor and predictor networks, allowing each to be trained separately. we still assume only instance-level supervision; the trick is to define a method of selecting snippets from inputs -rationales @xcite that can be used to support prediction. here we propose using arbitrary feature importance scoring techniques to do so. notably, these need not satisfy the 'faithfulness' criterion. in this paper we evaluate variants of fresh that use attention @xcite and gradient methods @xcite as illustrative feature scoring mechanisms. these provide continuous scores for features; we derive discrete rationales from them using simple heuristics. an independent network then uses only the extracted rationales to make predictions. disconnecting the training tie between the in-dependent rationale extractor and prediction modules means that fresh is faithful by construction: the snippet that is ultimately used to inform a prediction can be presented as a faithful explanation because this was the only text available to the predictor. in contrast to prior discrete rationalization methods, fresh greatly simplifies training, and can accommodate any feature importance scoring metric. in our experiments, we also find that it yields superior predictive performance. in addition to being faithful (and affording strong predictive performance), extracted rationales would ideally be intuitive to humans, i.e., plausible. to evaluate this we run a small user study (section 8) in which humans both evaluate the readability of extracted rationales and attempt to classify instances based on them, effectively serving as a prediction module in the fresh framework. an example illustrating this property is presented in figure 1 . r k f m l c f u c 3 s r p i o i c q w b u s w g 4 c 2 / v e o 6 j b p 3 u w / c x 9 a a n 0 u c z x s c t t e 5 8 t a v a q i 7 1 e j t r j f c z + g v v t n g v d j v z s e i t e q u m 8 f o d 5 z p h 8 x f k z @xmath0 in a new movie … amazing acting and script. … … r k f m l c f u c 3 s r p i o i c q w b u s w g 4 c 2 / v e o 6 j b p 3 u w / c x 9 a a n 0 u c z x s c t t e 5 8 t a v a q i 7 1 e j t r j f c z + g v v t n g v d j v z s e i t e q u m 8 f o d 5 z p h 8 x f k z @xmath1 ## faithfulness through discrete rationale selection we now propose fresh, our framework for training explainable neural predictors. we begin by describing the two-model, discrete rationale selection approach introduced by @xcite ( §3.1), which serves as the starting point for our framework, detailed in §4. ## faithful rationale extraction from saliency thresholding (fresh) to avoid recourse to reinforce, we introduce fresh, in which we decompose the original prediction task into three sub-components, each with its own independent model. these are the support model supp, the rationale extractor model ext, and the classifier pred. 2 we train supp end-to-end to predict y, using its outputs only to extract continuous feature importance scores from instances in x. these scores are binarized by ext either using a parameterized model trained on the output scores, or via direct discretization heuristics. finally, pred is trained (and tested) only on text provided by ext. figure 2 depicts this proposed framework. a central advantage of our decomposed setup lies in the arbitrariness of the rationale extraction mechanism. any function over supp's predictions that assigns scores to the input tokens intended to quantify their importance can serve as an input to ext. note that this means even posthoc scoring models (applied after the model has completed training) are permissible. examples of such functions include gradient-based methods and lime @xcite . notably, the importance scoring function need not faithfully identify features that actually informed the predictions from supp. this means, e.g., that one is free to use token-level attention (over contextualized representations) -the final rationales provided by fresh will nonetheless remain faithful with respect to pred. the importance scores are used only to train ext heuristically, for example by treating the top k tokens (with respect to importance scores) for a given example as the target rationale. the key design decision here is designing such heuristics that map continuous importance scores to discrete rationales. any strategy for this will likely involve trading conciseness (shorter rationales) against performance (greater predictive accuracy). for explainability, we can present users with the snippet(s) that pred used to make a prediction as an explanation (from ext), and we can be certain that the only tokens that contributed to the prediction made by pred are those included in the this text. in addition to transparency, this framework may afford efficiency gains in settings in which humans are tasked with classifying documents; in this case we can use ext to present only the (short) relevant snippets. indeed, we use exactly this approach as one means of evaluation in section 8. ## fresh implementations the high-level framework described above requires making several design choices to operationalize; we propose and evaluate a set of such choices in this work, detailed below. specifically, we must specify a feature importance scoring mechanism for supp (section 5.1), and a strategy for inducing discrete targets from these continuous scores (5.2). in addition, we need to specify a trained or heuristic extractor architecture ext. in this work, all instances of pred exploit bertbased representations. foot_2 ## quantitative evaluation we first evaluate the performance achieved on datasets by the pred models trained on different ext-extracted rationales, compared to each other and to @xcite 's end-to-end rationale extraction framework. as an additional baseline, we also evaluate a variant of the differentiable binary variable model proposed in @xcite . this baseline do not require any hyperparameter search. in general, we would expect predictive performance to positively correlate with rationale length, and so we evaluate predictive performance (accuracy or f1-score) across methods using a fixed rationale length for each dataset. we report results in terms of predictive performance for all model variants in use the entire train sets for the respective datasets, and fix the rationale length as described in §6.2 to ensure fair comparison across methods. we observe that despite its simplicity, fresh performs nearly as well as full text while using only 10-30% of the original input text, thereby providing transparency. fresh achieves better average performance than lei et al.'s end-to-end method, with the exception of agnews, in which case the models are comparable. it also consistently fares better than bastings et al.'s system. of the two feature scoring functions considered, [cls] self-attention scores tend to yield better results, save for on the multirc and movies datasets, on which gradients fare better. with respect to discretizing feature scores, the simple top-k strategy seems to perform a bit better than the contiguous heuristic, in what we expect to be traded off against a greater coherence of the contiguous rationales. as seen in varying rationale length. figure 3 plots f1 scores across datasets and associated standard deviations achieved by the best rationale variant of @xcite and fresh at two different target rationale lengths. these results demonstrate the effectiveness of fresh even in constrained settings. note, we had to re-perform hyperparameter search for a different rationale length in case of @xcite model. ## human analysis we have proposed fresh as an architecture which, in addition to exceeding performance of previous training regimes, provides a guarantee for extracting rationales which are faithful. however, as noted in the introduction, another desirable trait of rationales is that they are judged as good by humans. to assess the plausibility of the resulting rationales @xcite , we design a human user study. 4 we evaluate the following attributes of plausibility: sufficiency. can a human predict the correct label given only the rationale? this condition aligns with @xcite , and with the confidence and adequate justification criteria of @xcite . in our experiment, we simply substitute a human user for pred and evaluate performance. readability and understandability. we test the user's preference for a certain style of rationale beyond their ability to predict the correct label. our hypothesis is that humans will prefer contiguous to non-contiguous rationales. this condition aligns with coherency @xcite , human-likeness and understandability @xcite . ## conclusions we have proposed faithful rationale extraction from saliency thresholding (fresh), a simple, flexible, and effective method to learn explainable neural models for nlp. our method can be used with any feature importance metric, is very sima measure of how well our annotators have done at predicting the correct document label from only the extracted rationale. all metrics are averaged over the 100 test documents. we showed empirically that fresh outperforms existing models, recovering most of the performance of the original 'black-box' model. additionally, we found fresh rationales to be at least as plausible to human users as comparable end-toend methods. we acknowledge some important limitations of this work. here we have considered explainability as an instance-specific procedure. the final explanation provided by the model is limited to the tokens provided by the extraction method. our framework does not currently support further pruning (or expanding) this token set once the rationale has been selected. in addition, while we do have a guarantee under our model about which part of the document was used to inform a given classification, this approach cannot readily say why this specific rationale was selected in the first place. nor do we clearly understand how the pred uses extracted rationale to perform its classification. we view these as interesting directions for future work. ye zhang, iain marshall, and byron c wallace. 2016. rationale-augmented convolutional neural networks for text classification. in proceedings of the conference on empirical methods in natural language processing (emnlp), volume 2016, page 795. nih public access. ruiqi zhong, steven shao, and kathleen mckeown. 2019. fine-grained sentiment analysis with faithful attention. arxiv preprint arxiv:1908.06870.