An Empirical Study on Explanations in Out-of-Domain Settings (2022)

Abstract: Recent work in Natural Language Processing has focused on developing approaches that extract faithful explanations, either via identifying the most important tokens in the input (i.e. post-hoc explanations) or by designing inherently faithful models that first select the most important tokens and then use them to predict the correct label (i.e. select-then-predict models). Currently, these approaches are largely evaluated on in-domain settings. Yet, little is known about how post-hoc explanations and inherently faithful models perform in out-of-domain settings. In this paper, we conduct an extensive empirical study that examines: (1) the out-of-domain faithfulness of post-hoc explanations, generated by five feature attribution methods; and (2) the out-of-domain performance of two inherently faithful models over six datasets. Contrary to our expectations, results show that in many cases out-of-domain post-hoc explanation faithfulness measured by sufficiency and comprehensiveness is higher compared to in-domain. We find this misleading and suggest using a random baseline as a yardstick for evaluating post-hoc explanation faithfulness. Our findings also show that select-then predict models demonstrate comparable predictive performance in out-of-domain settings to full-text trained models.

## introduction an explanation or rationale 2 , typically consists of a subset of the input that contributes more to the prediction. extracting faithful explanations is important for studying model behavior @xcite and assisting in tasks requiring human decision making, such as clinical text classification @xcite , misinformation detection @xcite and legal text classification @xcite . a faithful explanation is one which accurately represents the reasoning behind a model's prediction @xcite two popular methods for extracting explanations are through feature attribution approaches (i.e. posthoc explanation methods) or via inherently faithful classifiers (i.e. select-then-predict models). the first computes the contribution of different parts of the input with respect to a model's prediction @xcite @xcite . the latter consists of using a rationale extractor to identify the most important parts of the input and a rationale classifier, a model trained using as input only the extractor's rationales @xcite @xcite . 3 figure 1 illustrates the two approaches with an example. currently, these explanation methods have been mostly evaluated on in-domain settings (i.e. the train and test data come from the same distribution). however, when deploying models in real-world applications, inference might be performed on data from a different distribution, i.e. out-of-domain @xcite . this can create implications when extracted explanations (either using post-hoc methods or through select-then-predict models) are used for assisting human decision making. whilst we are aware of the limitations of current state-of-the-art models in out-of-domain predictive performance @xcite , to the best of our knowledge, how faithful out-of-domain post-hoc explanations are has yet to be explored. similarly, we are not aware how inherently faithful select-then-predict models generalize in out-of-domain settings. inspired by this, we conduct an extensive empirical study to examine the faithfulness of five feature attribution approaches and the generalizability of two select-then-predict models in out-ofdomain settings across six dataset pairs. we hypothesize that similar to model predictive performance, post-hoc explanation faithfulness reduces in out-ofdomain settings and that select-then-predict performance degrades. our contributions are as follows: ## qualitative analysis in example (1), we observe that models trained on two closely related tasks (amazinstr and amazdigimu) place more importance to the phrase "sound good". on the contrary, the model trained on amazpantry which has not encountered such phrases during training, mostly focuses on "work great". this is expected as the term "sound" is not typical of pantry reviews. similarly, in example m trained on example (1) amazinstr (id) work great and sound good amazdigimu work great and sound good amazpantry work great and sound good (2) amazpantry (id) delicious and at a good price . would recommend . amazdigimu delicious and at a good price . would recommend . amazinstr delicious and at a good price . would recommend . (3) sst (id) a painfully funny ode to bad behavior imdb a painfully funny ode to bad behavior yelp a painfully funny ode to bad behavior (4) yelp (id) the kouign -amann is so amazing ... must taste to appreciate . ## conclusion we conducted an extensive empirical study to assess the faithfulness of post-hoc explanations (i.e. using feature attribution approaches) and performance of select-then-predict (i.e. inherently faith-ful) models in out-of-domain settings. our findings highlight, that using sufficiency and comprehensiveness to evaluate post-hoc explanation faithfulness out-of-domain can be misleading. to address this issue, we suggest comparing faithfulness of post-hoc explanations to a random attribution baseline for a more robust evaluation. we also show that select-then-predict models, which are inherently faithful, perform surprisingly well in out-ofdomain settings. despite performance degradation, in many cases their performance is comparable to those of full-text trained models. in future work, we aim to explore methods for improving the evaluation of faithfulness for out-of-domain post-hoc explanations. pepa atanasova, jakob grue simonsen, christina lioma, and isabelle augenstein. 2020. a diagnostic study of explainability techniques for text classification. in proceedings of the 2020 conference on empirical methods in natural language processing (emnlp), pages 3256-3274, online. association for computational linguistics. jasmijn bastings, wilker aziz, and ivan titov. 2019. interpretable neural predictions with differentiable binary variables. in proceedings of the 57th annual meeting of the association for computational linguistics, pages 2963-2977, florence, italy. association for computational linguistics. oana-maria camburu, tim rocktäschel, thomas lukasiewicz, and phil blunsom. 2018. e-snli: natural language inference with natural language explanations. in advances in neural information processing systems, volume 31. curran associates, inc. samuel carton, anirudh rathore, and chenhao tan. 2020. evaluating and characterizing human rationales. in proceedings of the 2020 conference on empirical methods in natural language processing (emnlp), pages 9294-9307, online. association for computational linguistics. tuhin chakrabarty, kilol gupta, and smaranda muresan. 2019. pay "attention" to your context when classifying abusive language. in proceedings of the third workshop on abusive language online, pages 70-79, florence, italy. association for computational linguistics. ilias chalkidis, ion androutsopoulos, and nikolaos aletras. 2019. neural legal judgment prediction in english. in proceedings of the 57th annual meeting of the association for computational linguistics, pages 4317-4323, florence, italy. association for computational linguistics. ilias chalkidis, manos fergadiotis, dimitrios tsarapatsanis, nikolaos aletras, ion androutsopoulos, and prodromos malakasiotis. 2021. paragraph-level rationale extraction through regularization: a case study on european court of human rights cases. in proceedings of the 2021 conference of the north american chapter of the association for computational linguistics: human language technologies, pages 226-241, online. association for computational linguistics. shrey desai and greg durrett. 2020. calibration of pre-trained transformers. in proceedings of the 2020 conference on empirical methods in natural language processing (emnlp), pages 295-302, online. association for computational linguistics. jay deyoung, sarthak jain, nazneen fatema rajani, eric lehman, caiming xiong, richard socher, and byron c. wallace. 2020. eraser: a benchmark to evaluate rationalized nlp models. in proceedings of the 58th annual meeting of the association for computational linguistics, pages 4443-4458, online. association for computational linguistics. amir feder, katherine a keith, emaad manzoor, reid pryzant, dhanya sridhar, zach wood-doughty, jacob eisenstein, justin grimmer, roi reichart, margaret e roberts, et al. 2021. causal inference in natural language processing: estimation, prediction, interpretation and beyond. arxiv preprint arxiv:2109.00725. nuno miguel guerreiro and andré ft martins. 2021. spectra: sparse structured text rationalization. arxiv preprint arxiv:2109.04552. dan hendrycks, xiaoyuan liu, eric wallace, adam dziedzic, rishabh krishnan, and dawn song. 2020. pretrained transformers improve out-of-distribution robustness. in proceedings of the 58th annual meeting of the association for computational linguistics, pages 2744-2751, online. association for computational linguistics. sepp hochreiter and jürgen schmidhuber. 1997. long short-term memory. neural computation, 9(8):1735-1780. alon jacovi and yoav goldberg. 2020. towards faithfully interpretable nlp systems: how should we define and evaluate faithfulness? in proceedings of the 58th annual meeting of the association for computational linguistics, pages 4198-4205, online. association for computational linguistics. sarthak jain and byron c. wallace. 2019. attention is not explanation. in proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 3543-3556, minneapolis, minnesota. association for computational linguistics. sarthak jain, sarah wiegreffe, yuval pinter, and byron c. wallace. 2020. learning to faithfully rationalize by construction. in proceedings of the 58th annual meeting of the association for computational linguistics, pages 4459-4473, online. association for computational linguistics. brendan kennedy, xisen jin, aida mostafazadeh davani, morteza dehghani, and xiang ren. 2020. contextualizing hate speech classifiers with post-hoc explanation. in proceedings of the 58th annual meeting of the association for computational linguistics, pages 5435-5442, online. association for computational linguistics. pieter-jan kindermans, kristof schütt, klaus-robert müller, and sven dähne. 2016. investigating the influence of noise and distractors on the interpretation of neural networks. arxiv preprint arxiv:1611.07270. diederik p kingma and jimmy ba. 2014. adam: a method for stochastic optimization. arxiv preprint arxiv:1412.6980. sawan kumar and partha talukdar. 2020. nile : natural language inference with faithful natural language explanations. in proceedings of the 58th annual teven le scao, sylvain gugger, mariama drame, quentin lhoest, and alexander rush. 2020. transformers: state-of-the-art natural language processing. in proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pages 38-45, online. association for computational linguistics. xiang zhang, junbo zhao, and yann lecun. 2015. character-level convolutional networks for text classification. in advances in neural information processing systems, volume 28. curran associates, inc.