A Multilingual Perspective Towards the Evaluation of Attribution Methods in Natural Language Inference (2022)

Abstract: Most evaluations of attribution methods focus on the English language. In this work, we present a multilingual approach for evaluating attribution methods for the Natural Language Inference (NLI) task in terms of faithfulness and plausibility.First, we introduce a novel cross-lingual strategy to measure faithfulness based on word alignments, which eliminates the drawbacks of erasure-based evaluations.We then perform a comprehensive evaluation of attribution methods, considering different output mechanisms and aggregation methods.Finally, we augment the XNLI dataset with highlight-based explanations, providing a multilingual NLI dataset with highlights, to support future exNLP studies. Our results show that attribution methods performing best for plausibility and faithfulness are different.

## introduction the opaqueness of large pre-trained models like bert @xcite and gpt @xcite motivates developing explanation methods @xcite , which aim to attribute importance to particular input features @xcite @xcite , such as words in a textual input. two main criteria for evaluating such methods are plausibility and faithfulness @xcite . plausibility can be defined as the consistency between explanations and human expectations, while faithfulness is defined as the consistency between explanations and the model's underlying decision-making process. prior evaluations of attributions along these dimensions @xcite @xcite suffer from several limitations. first, they have been limited in the range of considered attribution methods and the mechanism of calculating the attributions. second, standard faithfulness evaluations, such as erasurebased ones @xcite , entail running the model on examples outside of the training distribution @xcite . third, prior plausibility evaluations are limited to english-only datasets due to the lack of multilingual datasets with highlighted explanations. in this work, we aim to fill these gaps. our main contribution is a new framework for evaluating the faithfulness of attribution methods. inspired by @xcite 's criterion for faithful explanations as giving similar explanations for similar inputs, we propose to use cross-lingual sentences (translations) as similar inputs. given a multilingual model, we argue that faithful attributions should point to words that are aligned in two translations of the same sentence. this approach avoids out-of-distribution inputs by utilizing cross-lingual sentences as naturally ocurring input perturbations. we focus on natural language inference (nli) as a case study, since it is a central task that has been widely used as a test bed for attribution methods @xcite @xcite @xcite . we compare eight attribution methods, including different mechanisms of computation varying the output and the aggregation of input feature importance scores. first, we experiment with the cross-lingual xnli dataset @xcite , multilingual bert @xcite , and xlm-r @xcite , and discover large differences in the faithfulness of different attribution methods. second, we find that certain attributions are more plausible and that the choice of computation mechanism has a large effect in some cases. as far as we know, this is the first comprehensive study in-vestigating the effect of different types of outputs when evaluating attributions. informed by our comprehensive evaluation, we augment the multilingual xnli dataset @xcite with highlight-based explanations by extracting highlights for the english part of xnli and projecting along word alignments to other languages. we perform a plausibility evaluation with the resulting dataset, which we dub e-xnli, and perform a human evaluation on a subset of the dataset to validate its adequacy. finally, when comparing the ranking of attribution methods by plausibility and faithfulness, we find that no single method performs best. different methods have different pros and cons, and may therefore be useful in different scenarios. in summary, this work provides: ## plausibility in this section, we present details about plausibility evaluation and results, and introduce a new dataset containing highlight-based explanations in multiple languages. ## limitations in this work, we examine a wide range of attribution methods along output and aggregation dimensions. prior work @xcite shows that faithfulness of attribution methods depends on both tasks and models, but our work is limited to the nli task while considering different models. despite the importance of nli for evaluation in nlp (poliak, 2020), our conclusions might not generalize to other tasks. in addition, while we experiment with multiple random seeds, our experiments are limited to two architectures: bert @xcite and xlm-r @xcite . the results of cross-lingual faithfulness experiments are sensitive to language choice as discussed in section 3.2.1, so we present the results calculated with the languages well-represented by multilingual models. the multilingual dataset we provide, e-xnli, consists of automatically-extracted highlight-based explanations and should be used with caution for future exnlp studies since we only performed a human evaluation on a small subset of the dataset. especially, training self-explanatory models with this dataset can lead to undesired outcomes, such as poor explanation quality. ## conclusion we introduce a novel cross-lingual strategy to evaluate the faithfulness of attribution methods, which eliminates the out-of-distribution input problem of common erasure-based faithfulness evaluations. then, we perform a comprehensive comparison of different attribution methods having different characteristics in terms of plausibility and faithfulness for the nli task. the experiments show that there is no one-size-fits-all solution for local post-hoc explanations. our results highlight that practitioners should choose an attribution method with proper output mechanism and aggregation method according to the property of explanation in question: finally, we present e-xnli, a multilingual dataset with automatically generated highlight explanations, to facilitate multilingual exnlp studies.