Can Rationalization Improve Robustness? (2022)

Abstract: A growing line of work has investigated the development of neural NLP models that can produce rationales–subsets of input that can explain their model predictions. In this paper, we ask whether such rationale models can provide robustness to adversarial attacks in addition to their interpretable nature. Since these models need to first generate rationales (“rationalizer”) before making predictions (“predictor”), they have the potential to ignore noise or adversarially added text by simply masking it out of the generated rationale. To this end, we systematically generate various types of ‘AddText’ attacks for both token and sentence-level rationalization tasks and perform an extensive empirical evaluation of state-of-the-art rationale models across five different tasks. Our experiments reveal that the rationale models promise to improve robustness over AddText attacks while they struggle in certain scenarios–when the rationalizer is sensitive to position bias or lexical choices of attack text. Further, leveraging human rationale as supervision does not always translate to better performance. Our study is a first step towards exploring the interplay between interpretability and robustness in the rationalize-then-predict framework.

## introduction rationale models aim to introduce a degree of interpretability into neural networks by implicitly baking in explanations for their decisions @xcite @xcite . these models are carried out in a two-stage 'rationalizethen-predict' framework, where the model first selects a subset of the input as a rationale and then makes its final prediction for the task solely using the rationale. a human can then inspect the selected rationale to verify the model's reasoning 1 our code is publicly available at: https://github. com/princeton-nlp/rationale-robustness . ## background extractive rationale models foot_0 output predictions through a two-stage process: the first stage ("rationalizer") selects a subset of the input as a rationale, while the second stage ("predictor") produces the prediction using only the rationale as input. rationales can be any subset of the input, and we characterize them roughly into either token-level or sentence-level rationales, which we will both investigate in this work. the task of predicting rationales is often framed as a binary classification problem over each atomic unit depending on the type of rationales. the rationalizer and the predictor are often trained jointly using task supervision, with gradients back-propagated through both stages. we can also provide explicit rationale supervision, if human annotations are available. ## results for all attacked test sets, we report the average scores with attack sentence inserted at the beginning and the end of the inputs. our findings shed light on the relationship between gr, ar, and drop in performance, which eventually lead to a promising direction to improve performance of rationale models under attacks ( §7). ## augmented rationale training from our previous analysis on the trade-off between gr and ar ( §6.2), it is clear that avoiding attack sentences in rationales is a viable way to make such models more robust. note that this is not obvious by construction since the addition of attacks affects other parameters such as position of the original text and discourse structure, which may throw off the 'predictor' component of the model. as a more explicit way of encouraging 'rationalizers' to ignore spurious text, we propose a simple method called augmented rationale training (art). specifically, we sample two sentences at random from the wikitext-103 dataset @xcite and insert them into the input passage at random position, setting their pseudo rationale labels z @xmath0. we limit the addition to only inserting two sentences to avoid exceeding the rationalizer maximal token limit. we then add an auxiliary negative binary cross entropy loss to train the model to not predict the pseudo rationale. this encourages the model to ignore spurious text that is unrelated to the task. note that this procedure is both model-agnostic and does not require prior knowledge of the type of addtext attack. interestingly, the random wikipedia sentences we added in art are not topically or contextually related to the original instance text at all, yet they seem to help the trained model ignore adversarially constructed text that is tailored for specific test instances. this points to the promise of art in future work, where perhaps more complex generation schemes or use of attack information could provide even better robustness. ## discussion in this work, we investigated whether neural rationale models are robust to adversarial attacks. we constructed a variety of addtext attacks across five different tasks and evaluated several state-of-theart rationale models. our findings raise two key messages for future research in both interpretability and robustness of nlp models: interpretability we identify an opportunity to use adversarial attacks as a means to evaluate rationale models (especially extractive ones). in contrast to existing metrics like iou used in prior work @xcite , robustness more accurately tests how crucial the predicted rationale is to the model's decision making. further, our analysis reveals that even stateof-the-art rationale models may not be consistent in focusing on the most relevant parts of the input, despite performing well on tasks they are trained on. this points to the need for better model architectures and training algorithms to better align rationale models with human judgements. robustness for adversarial attack research, we show that extractive rationale models are promising for improving robustness, while being sensitive to factors like the attack position or word choices in the attack text. research that proposes new attacks can use rationale models as baselines to assess their effectiveness. finally, the effectiveness of art points to the potential for data augmentation in improving robustness of nlp systems, even against other types of attacks beyond addtext. we hope our results can inspire more research at the intersection of interpretability and robustness.