Analyzing Zero-Shot transfer Scenarios across S panish variants for Hate Speech Detection (2023)

Abstract: Hate speech detection in online platforms has been widely studied inthe past. Most of these works were conducted in English and afew rich-resource languages. Recent approaches tailored forlow-resource languages have explored the interests of zero-shot cross-lingual transfer learning models in resource-scarce scenarios. However, languages variations between geolects such as AmericanEnglish and British English, Latin-American Spanish, and EuropeanSpanish is still a problem for NLP models that often relies on(latent) lexical information for their classification tasks. Moreimportantly, the cultural aspect, crucial for hate speech detection,is often overlooked. In this work, we present the results of a thorough analysis of hatespeech detection models performance on different variants of Spanish,including a new hate speech toward immigrants Twitter data set we built to cover these variants. Using mBERT and Beto, a monolingual Spanish Bert-based language model, as the basis of our transfer learning architecture, our results indicate that hate speech detection models for a given Spanish variant are affected when different variations of such language are not considered. Hate speech expressions could vary from region to region where the same language is spoken. Our new dataset, models and guidelines are freely available.

## introduction hate speech detection is a task that has gained much attention from the nlp community due to the exponential spread of social media platforms 1 . this task aims to identify whether a piece of text contains hateful messages against a person or a group based on characteristics such as color, ethnicity, race, sexual orientation, religion, and others @xcite . gender and nationalities are no exceptions to this. according to the pew research center report in 2021, 33% of women under 35 report having been sexually harassed online, compared with 11% of men under 35 @xcite . misogyny is harm against women due to gender, which might result in psychological, reputational, professional, or even physical damage @xcite . on the other hand, xenophobia is "attitudes, prejudices, and behavior that reject, exclude and often vilify persons, based on the perception that they are outsiders or foreigners to the community, society or national identity" foot_0 . an online manifestation of such behaviors may include hostility, social exclusion, threats of violence, and other forms of discrimination. as a result, the internet becomes a less equal, less safe, and less inclusive environment for targetted groups. online hate speech detection in social medias platforms has been tackled in several studies @xcite @xcite @xcite @xcite . however, most studies have been carried out using english language data or limited spanish data. for example, the significant morphosyntactic variations between spanish variants @xcite ) make considering the spanish language homogeneous challenging for language models. according to ethnologue 3 in 2022, spanish is currently declared as the official language in 22 countries, being the fourth language with the most significant number of countries. due to the numerous regions where spanish is the spoken language, expressions associated with hate speech may differ across various locations. for example, in the variation of the spanish language from spain, the word "fregar" only means "scrub" while at the same time, the same word in the span-ish ecuadorian variant can also mean "annoy" or "having fun". we note the different connotations of that term across various latin american regions provided by the royal spanish academy foot_2 (rae in spanish). thus, the phrase "anda a tu casa a fregar" can only be interpreted as "go home to scrub (the dishes)" by people from spain, which can contain a misogynous connotation. on the other hand, for people speaking spanish in ecuador, it may be mostly interpreted as "go home to have some fun" or "go home to annoy (other people)", which are not related to discriminatory connotations. despite these scenarios, studies proposing models for hate speech detection towards women and immigrants in spanish generally do not include information about the language or cultural variation of the text. due to the previously described challenge, other difficulties emerge when developing hateful content detection systems for online platforms. current state-of-the-art pre-trained language models (lm), such as the "multilingual" version of bert (mbert) @xcite , are widely used in several nlp tasks and achieve impressive results. however, mbert might not help to detect hate speech against women or immigrants when language-specific variants appear. it has been proven to perform worse than monolingual implementations of bert under certain circumstances @xcite . mbert is trained on only wikipedia data, particularly the entire wikipedia dump for each language, excluding user and talk pages. however, this is problematic for the spanish language as according to wikipedia's spanish wikipedia article (spanish wikipedia, 2021) by september 2017, 39.2% of the spanish wikipedia edits come from spain, being the country with the largest edits, while the rest come from other countries located in regions such as the americas and others. it is important to note that spain is the fourth country with the most prominent spanish language native speakers, whereas mexico is the first according to @xcite . therefore, language models trained on wikipedia data may not represent the differences between spanish variants @xcite . thus, in this study, we aim to address the following research questions: our main contributions may be summarized as follows: ## related work automatic hate speech detection in online platforms has been previously studied across different hate speech domains such as misogyny @xcite , xenopho-bia @xcite , homophobia @xcite and others @xcite . nevertheless, a limited number of works focus on spanish data to develop ml-based systems for online hateful content identification. 6 most of the research developed with spanish corpora posted on micro-blogging platforms are related to participation in a few recent shared tasks, namely ami 2018 @xcite , hateval 2019 (basile et al., 2019) and others. in addition, there is a lack of studies considering different variations of the spanish language and how state-of-the-art language models such as bert perform in hate speech detection when used for cross variants over the same language @xcite . in @xcite , multilingual and monolingual pre-trained language models were compared to deep learning architectures (cnn, lstm, and bi-lstm) and traditional ml models (svm and logistic regression) for detecting hate speech on tweets written in spanish. the authors used two datasets to conduct the comparison. the first corpus, haternet @xcite , has no information about the hate speech domain or the location where the tweets were posted. the second dataset is the hateval corpus which contains only information about the target for hate speech against women and immigrants. they used beto @xcite , a spanish language implementation of bert trained on wikipedia articles, movies and ted talks subtitles, scientific documents, and others written in spanish. results obtained in @xcite showed that beto, a monolingual lm outperforms multilingual pre-trained models such as xlm and mbert as well as the rest of the models they evaluated for hate speech detection in spanish. results in line with @xcite have also been achieved in other similar studies on hate speech detection @xcite . @xcite studied hate speech detection against women and immigrants across three languages: spanish, english, and italian. she investigated the limitations of zero-shot cross-lingual approaches using mbert. her results suggest that hate speech targets -i.e. different languages-should be studied separately as transfer learning in zero-shot scenarios is ineffective for misogyny detection. in addition to her findings, we aim to investigate whether such nuances can be extended to cross-variants within the same language. ## datasets in this section, we describe the datasets we use for training the misogyny detection models and the procedure we follow to compile and annotate the hascosva-2022 corpus, which is later used to train and evaluate our models to detect hate speech against immigrants. in ## experimental settings language models. for the multilingual language model, we use mbert @xcite , the multilingual version of bert, trained on wikipedia data from 104 languages. we also experiment with beto @xcite , a monolingual spanish bert, trained on the whole spanish wikipedia dump combined with the spanish language texts of the opus project @xcite without any differentiation between the spanish variants. others models for spanish exist and are posterior to @xcite , we decided to focus on beto because of its pretraining data that makes it more comparable to mbert. it would be of course interesting to conduct a large-scale spanish monolingual models study on that topic but we leave it for future work. ## results results obtained from the comparison between mbert and beto over the whole corpora are shown in the second set of experiments aims to compare mono-lingual and cross-lingual settings across spanish variants. ## error analysis in this section, we analyze and compare errors in cross variants evaluation. we briefly examine the reasons that might lead to poor performance when the model is trained and tested on different language variants. part of our analyses is inspired by the error analysis carried out in @xcite . first, we analyze the errors obtained by beto. such analysis is detailed in moreover, in mexico for referring to women and can contain a derogatory connotation. finally, we use shap (lundberg and lee, 2017) to study the behavior of beto in terms of explainability. shapley additive explanations, also known as shap, is a well-known model explainability technique used to interpret the models' decisions. shap is based on game theory and assigns importance scores to features for a given example classification. such scores indicate how much a feature influences the model toward its final output. in nlp tasks, it can assign importance scores to terms. thus, we use shap to examine how the models behave when the word tonta (which means idiot, female gendered, in english) appears in a text. such an insult is an example of how the same term can be interpreted differently in two variations of spanish. in spain, that insult is much more aggressive than how it may be interpreted in latin america. we take one misogynous tweet containing the word tonta from our corpus and classify such text by the misogyny europe and latam models. a colored representation of the scores computed by shap on both classifications is shown in figure 1 . we can observe both models provide different classifications, where the model trained on european data performs correctly. shap finds the word "tonta" highly influences the model trained on euro-pean tweets to classify the tweet as misogynous, as shown in figure 1b . in contrast, the same term provides almost no influence on the latam model's final decision according to shap in figure 1b . we can note the analyzed term slightly contributes towards the wrong (non-misogynous) class when the latam model is used. ## conclusions in this study, we showed how beto, a spanish version of bert, as expected, performs significantly better than multilingual bert for classifying tweets as hateful for two hate speech domains: misogyny and xenophobia. our outcomes align with previous studies mostly conducted with corpora proposed in popular shared tasks on hate speech detection. this does not mean that multilingual bert is not useful since findings in @xcite suggested that mbert is remarkably useful on low-resource language tasks, in contrast to monolingual bert implementations that use a significant amount of data. moreover, we demonstrated that variants of a language, for instance, due to its use in different countries or cultures, affect the performance of hate speech detection models. in other words, we found that whether we train a model using data derived from only one variant of spanish, the model's per-