Benchmarking Post-Hoc Interpretability Approaches for Transformer-based Misogyny Detection (2022)

Abstract: Transformer-based Natural Language Processing models have become the standard for hate speech detection. However, the unconscious use of these techniques for such a critical task comes with negative consequences. Various works have demonstrated that hate speech classifiers are biased. These findings have prompted efforts to explain classifiers, mainly using attribution methods. In this paper, we provide the first benchmark study of interpretability approaches for hate speech detection. We cover four post-hoc token attribution approaches to explain the predictions of Transformer-based misogyny classifiers in English and Italian. Further, we compare generated attributions to attention analysis. We find that only two algorithms provide faithful explanations aligned with human expectations. Gradient-based methods and attention, however, show inconsistent outputs, making their value for explanations questionable for hate speech detection tasks.

## introduction the advent of social media has proliferated hateful content online -with severe consequences for attacked users even in real life. women are often attacked online. a study by data & society 1 of women between 15 to 29 years showed that 41% self-censored to avoid online harassment. of those, 21% stopped using social media, 13% stopped going online, and 4% stopped using their mobile phone altogether. these numbers demonstrate the need for automatic misogyny detection systems for moderation purposes. various natural language processing (nlp) models have been proposed to detect and mitigate misogynous content @xcite @xcite @xcite @xcite . however, several papers already demonstrated that hate speech detection models suffer from unintended bias, resulting in harmful predictions for protected categories (e.g., women). this issue shows the need to understand the rationale behind a given prediction. a mature literature on model interpretability with applications to nlp-specific approaches exists @xcite @xcite . 2 as explanations become part of legal regulations @xcite , a growing body of work has focused on the evaluation of explanation approaches @xcite @xcite . however, little guidance on which interpretability method suits best to the sensible context of misogyny identification has been given. for instance, some explanations in we bridge this gap. we benchmark interpretability approaches to explain state-of-the-art transformer classifiers on the task of automatic misogyny identification. we cover two benchmark twitter datasets for misogyny detection in english and italian @xcite . we focus on single-instance, post-hoc input attribution methods to measure the importance of each token for predicting the instance label. our benchmark suite comprises gradient-based methods (gradients @xcite and integrated gradients @xcite ), shapley values-based methods (shap (lundberg and lee, 2017)), and input occlusion (sampling-and-occlusion @xcite ). we evaluate explanations in terms of plausibility and faithfulness @xcite . our results show that shap and sampling-and-occlusion provide plausible and faithful explanations and are consequently recommended for explaining misogyny classifiers' outputs. we also find that, despite their popularity, gradient-and attention-based methods do not provide faithful explanations. outputs of gradient-based explanation methods are inconsistent, while attention does not provide any useful insights for the classification task. contributions we benchmark four post-hoc explanation methods on two misogyny identification datasets across two languages, english and italian. we evaluate explanations in terms of plausibility and faithfulness. we demonstrate that not every token attribution method provides reliable insights and that attention cannot serve as explanation. code is available at https://github.c om/milanlproc/benchmarking-xai-m isogyny . ## benchmarking suite in the following, we describe the scope ( §2.1) of our benchmarking study, the included methods ( §2.2), and the evaluation criteria ( §2.2). ## experimental setup among the transformer-based models, we focus on bert @xcite due to its widespread usage. we fine-tuned pre-trained bert-based models on the ami-en and ami-it datasets. we report full details on the training in appendix a.1. tables 3-6 report token contributions computed with benchmarked approaches ( §2.2). we report contributions for individual tokens. 6 we define table contents as follows. separately by explanation method, we first generate raw contributions and then l1-normalize the vector. finally, we use a linear color scale between solid blue (assigned for contribution -1), white (contribution 0), and solid red (contribution 1). for all reported examples, we explain the misogynous class. hence, positive contributions indicate tokens pushing towards the misogynous class, while negative contributions push towards the non-misogynous one. lastly, the second top row reports the variation on the probability assigned by the model when the corresponding token is erased (∆p ). ## discussion error analysis we further analyze the term p*ssy and its role as a source of errors. almost all tweets of the test set containing the term p*ssy are labeled by the model as misogynous. the false-positive rate on this set of tweets is 0.93 compared to the 0.49 of the overall test set. similar considerations apply to english words typically associated with misogynous content as b*tch and wh*re. is it right for the right reason? unintended bias we study explanations to search for errors caused by unintended bias, a known phenomenon affecting models for misogynous identification. a model suffering from unintended bias performs better (or worse) when texts mention specific identity terms (e.g., woman) @xcite . bias due to language-specific expressions the interpretation of the gradient explanation is hard since all contributions are positive and associated with the misogynous class. all explanation methods assign a positive contribution to the word f*ga (*ss). shap, soc, and, to a lesser extent ig, indicate that the main reason behind the nonmisogynous prediction is the term p*rca. the bias of the model towards this expression was firstly exposed in @xcite and it thus validates ig, shap, and soc explanations as plausible. when one of the two terms of the expression is removed, the probability increases significantly. this suggests that explanations by ig, shap, and soc are faithful. further, we inspect the behavior of explanation methods when we erase one of the terms. we omit the word p*rca and we report its explanations on ## related work few works applied interpretability approaches to hate speech detection. @xcite proposes an adaptation of explainability techniques for computer vision to visualize and understand the cnn-gru classifier for hate speech @xcite . @xcite study both local and global explanations. they use shapley values @xcite to quantify feature importance on a local level and feature space exploration for a global explanation. @xcite analyze multiple attribution-based explanation methods for offensive language detection. the analysis includes an interpretable model (naïve bayes), model-agnostic methods based on surrogate models @xcite , layer-wise relevance propagation (lrp) @xcite , and a selfexplanatory model (lstm with an attention mechanism). shap explainer is applied @xcite to investigate the impact of political bias on hate speech classification. sample-and-occlusion (soc) explanation algorithm has been used in its hierarchical version in different papers for showing the results of hate speech detection @xcite . in this paper, we specifically focus on hate speech against women. in this context, @xcite apply shap to derive global ex-planations with the aim of exploring unintended bias of random forest-based misogyny classifier. while growing efforts are made for evaluating interpretability approaches for nlp models @xcite @xcite @xcite @xcite , the evaluation is not domainspecific. therefore, the benchmarking miss to consider specific sensitive problems and biases that are proper of the hate speech domain on which the explanation validation must focus. this paper fills this gap by focusing on post-hoc feature attribution explanation methods on individual predictions for the task of hate speech against women. ## conclusion in this paper, we benchmarked different explainability approaches on transformer-based models for the task of hate speech detection against women in english and italian. we focus on post-hoc feature attribution methods applied to fine-tuned bert models. our evaluation demonstrated that shap and soc provide plausible and faithful explanations and are consequently recommended for explaining misogyny classifiers' outputs. in contrast, gradient-and attention-based approaches failed in providing reliable explanations. as future work, we plan to add to the benchmarking suite a systematic evaluation involving human annotators. we also plan to include recently introduced token attribution methods @xcite as well as new families of approaches, like natural language explanations @xcite and input editing @xcite . finally, we will assess explanations of the most problematic data subgroups @xcite @xcite .