Comparing Explanation Faithfulness between Multilingual and Monolingual Fine-tuned Language Models (2024)

Abstract: In many real natural language processing application scenarios, practitioners not only aim to maximize predictive performance but also seek faithful explanations for the model predictions. Rationales and importance distribution given by feature attribution methods (FAs) provide insights into how different parts of the input contribute to a prediction. Previous studies have explored how different factors affect faithfulness, mainly in the context of monolingual English models. On the other hand, the differences in FA faithfulness between multilingual and monolingual models have yet to be explored. Our extensive experiments, covering five languages and five popular FAs, show that FA faithfulness varies between multilingual and monolingual models. We find that the larger the multilingual model, the less faithful the FAs are compared to its counterpart monolingual models. Our further analysis shows that the faithfulness disparity is potentially driven by the differences between model tokenizers. Our code is available: https://github.com/casszhao/multilingual-faith.

## introduction feature attribution methods (fas) are commonly used for ranking input tokens according to their importance to a model's prediction @xcite @xcite . subsequently, the top-k ranked tokens are selected to form a rationale. the faithfulness of a fa method refers to what extent its token importance scores and selected rationales actually reflect the model's inner reasoning mechanism @xcite . previous work has mainly studied faithfulness in the context of monolingual models, i.e. especially english @xcite @xcite . specifically, monolingual studies have investigated the impact of out-of-domain data @xcite figure 1 : model explanations given by the same feature attribution method, e.g. attention, for multilingual (xlm-r) and monolingual (french roberta) models for the same task (sentiment analysis in fr). aletras, 2022), adversarial attacks @xcite and temporal shifts @xcite on the faithfulness of fas. on the other hand, existing studies on interpreting multilingual models' behavior and their representations @xcite @xcite have not investigated the faithfulness of fas. as shown in figure 1 , even for the same input ("personne ne veut aller à cette fête.", i.e. "nobody wants to go to this party." in english), model prediction and fa, the token importance scores can be substantially different between multi-and monolingual models. this indicates that the models follow different inner processes for making predictions. it is unclear whether this difference is generally shared among input examples or even across other languages and models. given that the performance of multilingual models might be on par with monolingual counterparts in various languages @xcite , this leaves practitioners in a dilemma between choosing multilingual or monolingual models when the application scenario requires extracting faithful explanations for the model predictions. in this paper, we seek to answer if there is a faithfulness disparity of fas when applied to multi-and monolingual models. our main contributions are as follows: ## experiments our aim is to compare fa faithfulness between mono-and multilingual models across tasks and languages. for this purpose, we experiment with models of similar architectures and pre-training objectives following @xcite . the main differences between mono-and multilingual models are the tokenizers, supported vocabularies and the pre-training corpora. using this setting allows for a realistic comparison between models, given the fact that in a real world scenario, a practitioner would choose between off-the-shelf, already pretrained mono-or multilingual models without considering any specific implementation details (e.g. pre-training data). we compare models in various downstream tasks across a spectrum of typologically diverse and widely spoken languages, i.e. english, chinese, spanish, french and hindi. ## results our experiments include two multilingual and ten monolingual models, five fas, and 15 tasks. specifically, we test four models (two multilingual and two monolingual) on three tasks, using five fas in each language. this results in 480 faithfulness evaluation cases for each language, 2400 cases for five languages in total. we report accuracy and f1 of all models in appendix i. ## analysis we further investigate if the tokenization and model size contribute to the contrasting directions of fa faithfulness disparity between bert-and roberta-based models. ## conclusion to the best of our knowledge, our study is the first to investigate the faithfulness disparity between monolingual and multilingual models. we have conducted a comprehensive empirical study and found that faithfulness gaps exist across languages, models, and fas. our study further reveals that the larger the multilingual model, the less faithful its rationales are compared to its monolingual counterpart models. finally, we found that the disparity is highly correlated to the gap between monoand multilingual tokenizers on how aggressively they split words. further experiments support the assumption on the impact of tokenization: the discrepancies in faithfulness are primarily driven by differences in tokenization rather than underlying differences in how mono-and multilingual models semantically process the input.