Does Self-Rationalization Improve Robustness to Spurious Correlations? (2022)

Abstract: Rationalization is fundamental to human reasoning and learning. NLP models trained to produce rationales along with predictions, called self-rationalization models, have been investigated for their interpretability and utility to end-users. However, the extent to which training with human-written rationales facilitates learning remains an under-explored question. We ask whether training models to self-rationalize can aid in their learning to solve tasks for the right reasons. Specifically, we evaluate how training self-rationalization models with free-text rationales affects robustness to spurious correlations in fine-tuned encoder-decoder and decoder-only models of six different sizes. We evaluate robustness to spurious correlations by measuring performance on 1) manually annotated challenge datasets and 2) subsets of original test sets where reliance on spurious correlations would fail to produce correct answers. We find that while self-rationalization can improve robustness to spurious correlations in low-resource settings, it tends to hurt robustness in higher-resource settings. Furthermore, these effects depend on model family and size, as well as on rationale content. Together, our results suggest that explainability can come at the cost of robustness; thus, appropriate care should be taken when training self-rationalizing models with the goal of creating more trustworthy models.

## introduction rationalization-the process of explaining the reasoning used to come to a particular decision-plays a pivotal role in human inference and learning @xcite . for these reasons, there has been a growing interest in producing nlp mod-* work undertaken while alexis ross and ana marasović were at the allen institute for ai. our code is publicly available at https://github.com/ allenai/rationale_robustness els that can output rationales foot_0 for their predictions. models that output such rationales have multiple benefits: first, they are more interpretable and easier to interact with for end-users than nonrationalizing models @xcite . second, such intermediate rationalization can offer learning benefits, such as achieving comparable performance with less data and improving out-of-distribution generalization @xcite @xcite . however, the question of whether training models to rationalize can help them learn how to solve tasks for the right reasons remains open. in particular, rationales encode information about the underlying reasoning humans use to reach answers, which raises the question: does incorporating such rationales into training allow models to rely on human-aligned reasoning rather than spurious feature interactions? if so, training with rationales could offer a pathway towards creating more robust, trustworthy, or cognitively plausible models. in this work, we explore this question by empirically investigating whether training models with human-written rationales can help make them more robust to spurious correlations in data. we analyze a class of models called self-rationalization models-which jointly output free-text rationales along with predictions-and focus specifically on the fine-tuning setting, in which prior work has found reliance on spurious correlations to emerge @xcite . we evaluate six models of varying architectures and sizes across two tasks, natural language inference and commonsense question answering. our main results are as follows: our results suggest that straightforward selfrationalization training does not always facilitate learning to solve a task for the right reasons. instead, the effects of self-rationalization on robustness to spurious correlations depend on a multitude of factors. thus, appropriate care should be taken when training models to self-rationalize for the goal of creating trustworthy models. ## related work learning to rationalize two classes of approaches to producing models that can rationalize their predictions include self-rationalization models, 2 which are fully differentiable and output free-text rationales along with task predictions, and pipeline models, which consist of two components-one that produces rationales, and a second that makes predictions from those rationales @xcite . 3 such methods are typically evaluated by the faithfulness and plausibility of their rationales, where faithfulness represents the extent to which a model actually relied on the rationale in making its prediction, and plausibility indicates human judgment of how well the rationale explains the output @xcite . in contrast to these works which aim to improve model interpretability through new methods for rationalizing models, we ask to what extent existing methods affect model robustness to spurious correlations. we conduct our analysis on self-rationalization models, which have been found to achieve better task performance and produce higher-quality rationales than do pipeline models @xcite . learning from rationales recent work has explored the utility of rationales for improving endtask performance in in-context learning @xcite @xcite as well as in fine-tuning @xcite @xcite @xcite @xcite . previous work has shown that training with both human-annotated rationales @xcite and rationales generated by language models @xcite can increase in-domain task performance, particularly in low-resource settings @xcite @xcite . unlike these prior works, which study how training with rationales affects in-domain, end-task performance, we focus specifically on evaluating impact on robustness to spurious correlations. improving robustness with rationales most closely related are recent works that study how training with rationales affects model robustness. @xcite propose a method of supervising attention weights with extractive rationales and show that this method leads to both in-distribution and out-of-distribution improvements for natural language inference. @xcite find that training with contrastive extractive rationales improves robustness as measured by performance on adversarial evaluation sets. concurrent work by @xcite investigates to what extent training models to extract rationales through pipelines improves their robustness to adversarial attacks. in contrast to all three of these works, we focus on freeform rationales instead of extractive rationales and explore the impact of amount of training data on robustness. in contrast to @xcite and @xcite , we analyze selfrationalization models instead of pipeline models and measure robustness to spurious correlations, rather than robustness to adversarial attacks. while @xcite evaluate robustness to spurious correlations for natural language inference with some of the same test sets, they work with masked language models and evaluate the effect of supervising model attention with rationales; in contrast, we work with encoder-decoder and decoder-only models of varying sizes and evaluate the effect of outputting rationales along with predictions. in addition, their analysis is limited to natural language inference, for which evaluation datasets targeting robustness exist; in contrast, we also experiment with commonsense question answering through new methods for evaluating robustness. in §4.1, we discuss the variance in results across different tasks and highlight the importance of cross-task evaluation. ## conclusion we investigate to what extent training models to rationalize their predictions affects their robustness to spurious correlations. we experiment with encoder-decoder and decoder-only models ranging in size from 140 to 774 million parameters across two tasks-natural language inference and commonsense question-answering-and measure reliance on spurious correlations through both manually-annotated, out-of-domain challenge sets and challenging in-domain subsets of original test sets. we find that the effects of selfrationalization are model-and task-specific: while self-rationalization can improve robustness to spurious correlations in lower-resource settings for some models and tasks, it tends to exacerbate reliance on spurious correlations in higher-resource settings. furthermore, larger models tend to benefit more from rationales, and rationale content influences rationale utility in improving robustness. the variability of our results suggests that, despite the appeal of self-rationalization models for increasing model trustworthiness by facilitating debugging and interaction with end-users @xcite , training models to self-rationalize can have the unintended effect of increasing reliance on spurious features and biases, thereby decreasing the models' trustworthiness. thus, appropriate care should be taken when training selfrationalization models with the goal of creating trustworthy models. future work can investigate how to alleviate these harms while retaining the interpretability benefits of models that can rationalize their predictions.