The Irrationality of Neural Rationale Models (2022)

Abstract: Neural rationale models are popular for interpretable predictions of NLP tasks. In these, a selector extracts segments of the input text, called rationales, and passes these segments to a classifier for prediction. Since the rationale is the only information accessible to the classifier, it is plausibly defined as the explanation. Is such a characterization unconditionally correct? In this paper, we argue to the contrary, with both philosophical perspectives and empirical evidence suggesting that rationale models are, perhaps, less rational and interpretable than expected. We call for more rigorous evaluations of these models to ensure desired properties of interpretability are indeed achieved. The code for our experiments is at https://github.com/yimingz89/Neural-Rationale-Analysis .

## introduction as machine learning models are increasingly used in high-stakes domains, understanding the reasons for a prediction becomes more important, especially when the model is a black-box such as a neural network. while many post-hoc interpretability methods have been developed for models operating on tabular, image, and text data @xcite @xcite , their faithfulness are often questioned @xcite @xcite . with no resolution in sight for explaining black box models, inherently interpretable models, which self-explain while making decisions, are often favored. neural rationale models, shown in figure 1 (top), are the most popular in nlp @xcite @xcite : in them, a selector processes the input text, extracts segments (i.e. rationale) from it, and sends only the rationale to the predictor. since the rationale is the only information accessible to the predictor, it arguably serves as the explanation for the prediction. ## related work most interpretability efforts focus on post-hoc interpretation. for a specific input, these methods generate an explanation by analyzing model behaviors such as gradient @xcite or prediction on perturbed @xcite or reduced @xcite inputs. however, evaluations of these methods highlight various problems. for example, @xcite showed that many methods can generate seemingly reasonable explanations even for random neural networks. @xcite found that many methods fail to identify features known to be used by the model. @xcite share the same principles as us, but also focus on general post-hoc interpretations of arbitrary black-box models, while we focus on neural rationale models. by contrast, neural rationale models are largely deemed inherently interpretable and thus do not require post-hoc analysis. at a high level, a model has a selector and a classifier. for an input sentence, the selector first calculates the rationale as excerpts of the input, and then the classifier makes a prediction from only the rationale. thus, the rationale is often defined as the explanation due to this bottleneck structure. the non-differentiable rationale selection prompts people to train the selector using policy gradient @xcite or continuous relaxation @xcite , or directly use a pre-trained one @xcite . while rationale models have mostly been subject to less scrutiny, some evaluations have been carried out. @xcite proposed the notions of comprehensiveness and sufficiency for rationales, advocated as standard evaluations in the eraser ( deyoung et al., 2019) dataset. zhou et al. (2022a) noted that training difficulty, especially due to policy gradient, leads to selection of words known to not influence the label in the data generative model. complementing these evaluations and criticisms, we argue from additional angles to be wary of interpretability claims for rationale models, and present experiments showing issues with existing models. most related to our work, jacovi and goldberg (2020) mention a trojan explanation and dominant selector as two failure modes of rationale mod-els. we pinpoint the same root cause of a nonunderstandable selector in section 3. however, they favor rationales generated after the prediction, while we will argue for rationales being generated prior to the prediction. also, in their discussion of contrastive explanations, their proposed procedure runs the model on out-of-distribution data (sentence with some tokens masked), potentially leading to arbitrary predictions due to extrapolation, a criticism also argued by @xcite . ## philosophical perspectives in neural rationale models, the classifier prediction causally results from the selector rationale, but does this property automatically equate rationale with explanation? we first present a "failure case." for a binary sentiment classification, we first train a (non-interpretable) classifier c ′ that predicts on the whole input. then we define a selector s ′ that selects the first word of the input if the prediction is positive, or the first two words if the prediction is negative. finally, we train a classifier c to imitate the prediction of c ′ but from the rationale. the c ′ → s ′ → c model should achieve best achievable accuracy, since the actual prediction is made by the unrestricted classifier c ′ with full input access. can we consider the rationale as explanation? no, because the rationale selection depends on, and is as (non-)interpretable as, the black-box c ′ . this failure case is shown in figure 1 (bottom). recently proposed introspective training @xcite could not solve this problem either, as the selector can simply output the comprehensive rationale along with the original cue of first one or two words, with only the latter used by the classifier foot_0 . in general, a sufficiently powerful selector can make the prediction at selection time, and then pass this prediction via some encoding in the selected rationale for the classifier to use. to hide the "bug," consider now s ′ selecting the three most positive or negative words in the sentence according to the c ′ prediction (as measured by embedding distance to a list of pre-defined positive/negative words). this model would seem very reasonable to a human, yet it is non-interpretable for the same reason. to recover a "native" neural model, we could train a selector s to imitate c ′ → s ′ via teacher-student distillation @xcite , and the innocent-looking @xmath0. even without the explicit multi-stage supervision above, a sufficiently flexible selector s (e.g. a neural network) can implicitly learn the c ′ → s ′ model and essentially control the learning of the classifier c, in which case the bottleneck of succinct rationale affords no benefits of interpretability. so why does interpretability get lost (or fail to emerge)? the issue arises from not understanding the rationale selection process, i.e. selector s. if it is well-understood, we could determine its true logic to be c ′ → s ′ and reject it. conversely, if we cannot understand why a particular rationale is selected, then accepting it (and the resulting prediction) at face value is not really any different from accepting an end-to-end prediction at face value. in addition, the selector-classifier decomposition suggests that the selector should be an "unbiased evidence collector", i.e. scanning through the input and highlighting all relevant information, while the classifier should deliberate on the evidence for each class and make the decision. verifying this role of the selector would again require its interpretability. finally, considering the rationale model as a whole, we could also argue that the rationale selector should be interpretable. it is already accepted that the classifier can remain a black-box. if the selector is also not interpretable, then exactly what about the model is interpretable? architecturally, we can draw an analogy between the rationale in rationale models and the embedding representation in a typical end-to-end classifier produced at the penultimate layer. a rationale is a condensed feature extracted by the selector and used by the classifier, while, for example in image models, the image embedding is the semantic feature produced by the feature extractor and used by the final layer of linear classifier. furthermore, both of them exhibit some interpretable properties: rationales represent the "essence" of the input, while the image embedding space also seems semantically organized (e.g. figure 2 showing imagenet images organized in the embedding space). however, this embedding space is rarely considered on its own as the explanation for a prediction, exactly because the feature extractor is a black-box. similarly, the rationales by default should not qualify as the explanation either, despite its textual nature. finally, from a practical perspective, explanations should help humans understand the model's input-output behavior. such a purpose is fulfilled when the human understands not only why an explanation leads to an output, but also how the explanation is generated from the input in the first place. our emphasis on understanding the rationale selection process fulfills the latter requirement. such a perspective is also echoed by @xcite , who argued that the practical utility of explanations depends crucially on human's capability of understanding how they are generated. ## empirical investigation as discussed above, truly interpretable rationale models require an understanding of the rationale selection process. however, since the selector is a sequence-to-sequence model, for which there is no standard methods for interpretability, we focus on a "necessary condition" setup of understanding the input-output behavior of the model in our empirical investigation. specifically, we investigate rationale selection changes in response to meaningpreserving non-adversarial perturbation of individual words in the input sentence. ## conclusion we argue against the commonly held belief that rationale models are inherently interpretable by design. we present several reasons, including a counter-example showing that a reasonablelooking model could be as non-interpretable as a black-box. these reasons imply that the missing piece is an understanding of the rationale selection process (i.e. the selector). we also conduct a (non-adversarial) perturbation-based study to investigate the selector of two rationale models, in which automated analyses and a user study confirm that they are indeed hard to understand. in particular, the higher-accuracy model (cr) fares worse in most aspects, possibly hinting at the performanceinterpretability trade-off @xcite . these results point to a need for more rigorous analysis of interpretability in neural rationale models.