Exposing the limits of Zero-shot Cross-lingual Hate Speech Detection (2021)

Abstract: Reducing and counter-acting hate speech on Social Media is a significant concern. Most of the proposed automatic methods are conducted exclusively on English and very few consistently labeled, non-English resources have been proposed. Learning to detect hate speech on English and transferring to unseen languages seems an immediate solution. This work is the first to shed light on the limits of this zero-shot, cross-lingual transfer learning framework for hate speech detection. We use benchmark data sets in English, Italian, and Spanish to detect hate speech towards immigrants and women. Investigating post-hoc explanations of the model, we discover that non-hateful, language-specific taboo interjections are misinterpreted as signals of hate speech. Our findings demonstrate that zero-shot, cross-lingual models cannot be used as they are, but need to be carefully designed.

## introduction an increasing propagation of hate speech has been detected on social media platforms (e.g., twitter) where (pseudo-) anonymity enables people to target others without being recognized or easily traced. while this societal issue has attracted many studies in the nlp community, it comes with three important challenges. first, "hate speech" covers a wide range of target types, including misogyny, racism, and various other forms. while they often intersect, these types require different approaches. second, available labeled corpora refer to different definitions of hate speech, collection strategies, and annotation frameworks @xcite . this lack of consistency strongly limits research on hate speech, which ultimately needs to apply cross-domain or transfer learning approaches for using different corpora. third, most of the research on hate speech detection consider only english and only a limited number of labeled corpora are available @xcite @xcite . however, hate speech is not specific to any one language, and approaches proposed for english may not fit other languages. each language exhibits different complexities in dealing with gender or reflecting cultural ideas around it. the lack of models and labeled corpora for non-english languages seems a perfect application for zero-shot, cross-lingual learning @xcite . but is it? in this paper, we investigate the limitations of zero-shot, cross-lingual solutions based on mbert @xcite and xlm-r @xcite on benchmark data sets of hate speech against immigrants and women in english, italian, and spanish. our analysis demonstrates that these approaches have significant limitations: (1) they are not able to capture common (taboo) language-specific expressions, and (2) they do not transfer to different hate speech target types. we show that the reasons for these limitations are due to the high presence of language-and target-specific taboo interjections in non-hateful contexts, like porca puttana or puta. foot_0 while derogatory for women, these terms are often used as intensifiers in non-hateful context, blurring the lines for detection. since english does not use equivalent words in the same way, zero-shot, cross-lingual models will not observe them in the training data. consequently, these models consider the literal meaning of these terms as individual words, treating them as misogynous hate speech. these findings demonstrate that, at the current moment, cross-lingual, zero-shot transfer learning is not a solution for solving the lack of models and labeled corpora in non-english languages for hate speech detection. contributions 1) we investigate different learning frameworks on benchmark corpora for the detection of hate speech targeting women and immigrants 2) we expose the limits of zero-shot, crosslingual solutions using the multilingual bert model (mbert) 3) we show interpretable results through post-hoc explanation. ## zero-shot, cross-lingual hate speech detection we investigate different learning settings: 1) zeroshot, cross-lingual, i.e., training on one language and testing on unseen languages; 2) monolingual, i.e., training and testing on the same language; 3) few-shot, cross-lingual, i.e., training on one language and a small percentage of samples from the test language and testing on the test language; 4) augmented cross-lingual, i.e., training on several languages and testing on a language included in the training. multilingual bert recently, contextual embeddings pretrained on large corpora substantially advanced research for several major natural language processing (nlp) tasks @xcite . in particular, multilingual bert (mbert) @xcite , a model pretrained on monolingual wikipedia dumps in 104 languages, has shown surprisingly good abilities for zero-shot, cross-lingual model transfer for different nlp tasks @xcite . in this paper, we fine-tune the mbert model on the task of hate speech detection considering data from one or multiple languages. to assess the cross-lingual evaluation framework, we use hate speech benchmark data sets with consistent definitions, annotation schema, and collection strategies (see appendix c). for english and spanish, we adopt the data sets proposed in the shared task of hate speech against immigrants and women on twitter (hateval) @xcite . for italian, we consider two different corpora proposed for evalita shared tasks @xcite : the automatic misogyny identification challenge (ami) @xcite for hate speech towards women, and the hate speech detection shared task on facebook and twitter (haspeede) @xcite for hate speech towards immigrants. ## experimental results ## related work hate speech detection has attracted great interest in the nlp community. this has led to the proposal of automatic detection approaches based on machine learning @xcite @xcite @xcite and the creation of benchmark data sets, usually distributed through shared tasks @xcite @xcite @xcite @xcite @xcite . only a few studies have investigated hate speech detection across different languages. @xcite asked which factors affect multilingual settings for german and english, concluding that a shared classification algorithm is not conceivable due to lack of corpora comparability. in @xcite , the authors proposed a multi-channel model exploiting multilingual bert and languagespecific bert for chinese, english, german, and italian. finally, stappen et al. (2020) proposed a novel, attention-based classification block for performing zero-and few-shot, cross-lingual learning on the hateval data set. while they state that transfer learning is effective for hate speech detection, we argue that there is a need to investigate hate speech targets separately since these models consistently fail misogyny classification. ## conclusion we demonstrate that cross-lingual, zero-shot transfer learning, in its traditional settings, is not a feasible solution for solving the lack of models and labeled corpora for hate speech detection. we argue that hate speech is language specific, and nlp approaches to identifying hate speech must account for that specificity and the adoption of related techniques must be done with care @xcite . we plan to expand this evaluation to other languages and to investigate a solution based on bias mitigation @xcite and on pragmatic role-aware models @xcite to reduce the impact of this problem on classification. future work will also focus on modeling language's social factors @xcite @xcite , such as speaker and receiver characteristics, and study their impact on hate speech detection classifiers.