Resources for Multilingual Hate Speech Detection (2022)

Abstract: Most of the published approaches and resources for hate speech detection are tailored for the English language. In consequence, cross-lingual and cross-cultural perspectives lack some essential resources. The lack of diversity of the datasets in Spanish is notable. Variations throughout Spanish-speaking countries make existing datasets not enough to encompass the task in the different Spanish variants. We annotated 9834 tweets from Chile to enrich the existing Spanish resources with different words and new targets of hate that have not been considered in previous studies. We conducted several cross-dataset evaluation experiments of the models published in the literature using our Chilean dataset and two others in English and Spanish. We propose a comparative framework for quickly conducting comparative experiments using different previously published models. In addition, we set up a Codalab competition for further comparison of new models in a standard scenario, that is, data partitions and evaluation metrics. All resources can be accessed trough a centralized repository for researchers to get a complete picture of the progress on the multilingual hate speech and offensive language detection task.

## introduction offensive language frequently appears on social network interactions foot_0 . according to sigurbergsson and derczynski (2020) offensive language encompass a range of expressions from profanities to much more severe types of language among which is hate speech. hate speech is usually defined as communications of animosity or disparagement of an individual or a group on account of a group characteristic foot_1 . offensive language and hate speech bring along the risk of encouraging real hate crimes. due to the large amount of content generated in social media, automatic moderation is necessary to perform offensive content detection. machine learning models are used in most of the published approach for this purpose. the necessary resources are available almost exclusively for the english language @xcite @xcite . on the other hand, the cross-lingual and cross-cultural perspectives have been under addressed in the related literature. the lack of adequately annotated datasets is one of the limiting factors for developing these subtasks @xcite . in addition, the publicly available resources are accessible through the correspondent description papers. these resources have insufficient lack of centralized repositories for datasets and classification models. this situation makes it difficult for researchers to get a complete picture of the progress on the task. most of these existing datasets contain english examples, though we have gathered some datasets in portuguese @xcite , arabic @xcite , italian @xcite and spanish @xcite . in the particular case of the spanish language, only a few datasets can be found. the geographical origin of them is limited to spain @xcite , méxico @xcite , or unknown @xcite . since the hate speech phenomenon depends on the socio-cultural context @xcite , the targets of hate could change depending on the origin of the messages. the spanish language specific features spoken in different countries, makes models poorly generalizable when training with these existing resources. we propose a manually annotated dataset for offensive language detection. the dataset is composed of 9834 tweets from chile and is meant to enrich the existing spanish resources with different words and new targets of hate that have not been considered in previous studies. we conducted several evaluation experiments of the models published in the literature using our chilean dataset and two others in english and spanish. we propose a comparative framework for quickly conducting comparative experiments. this framework facilitates the application of existing models by including each original implementation as sub-models. in addition, we set up a codalab competition for further comparison of new models in a standard scenario, that is, data partitions and evaluation metrics. in summary, we developed the following resources for multilingual hate speech detection: ## related work one of the essential steps for the research in offensive language detection using machine learning is dataset acquisition. even when several social media platforms exist to get data from them, constructing a balanced labeled dataset is a costly task in time and effort. there is not a dataset considered as standard for this task. therefore researchers have to search in the related literature for the adequate one for their experiment. most of the existing datasets have been annotated for the english language @xcite @xcite though there exist a few in other languages such as spanish @xcite , italian @xcite and arabic @xcite . it is important to mention that even for english, the task is far from being solved @xcite . in most cases, the datasets only contain texts messages and not other information regarding authors, localication, or the conversation to which the tweet belongs. the lack of information makes the datasets out of context and limits the use of different features. regarding the data sources, most of the datasets have been recovered from the twitter platform, though a few are composed of facebook messages @xcite or youtube comments @xcite . as far as we know, there exists one data repository foot_3 for organizing offensive language datasets. ## chilean dataset for offensive language detection the research in the spanish language has been limited, in part, due to the lack of resources. as we described in section 2, the few spanish available datasets are composed of examples of the variant of spanish spoken in specific regions of the world with the cultural background associated with it. we consider it necessary to leverage the first dataset representative of the spanish spoken in south america, particularly chile. the examples in this dataset would enrich the understanding of pffensive language and hate speech by introducing terms mainly used in this region and targets of hate unconsidered in previous studies. next, we describe the process of annotation and general features of our datasets. ## comparative framework in the related literature of offensive language detection, there is a lack of comparative studies. this situation is more noticeable in cross-lingual approaches as a relatively new sub-area. there is no consensus about the best approaches for solving the cross-lingual detection task. with the purpose of alleviating this situation, we propose two tools for cross-lingual approaches comparison: ## conclusions we described three resources for the multilingual offensive language detection task. these resources would be helpful in the development of the multilingual sub-area, which have been under-addressed. we constructed the first chilean dataset for hate speech and offensive language to alleviate this situation. the dataset contains 9834 tweets in the spanish language that originated in chile. the tweets are labeled in several categories related to offensive content. furthermore, it includes annotations associated with the content of the tweets. finally, we created a comparative framework (library + competition) to facilitate researchers to compare new models with the existing ones. the library is implemented in python and contains, as submodels, previously published cross-lingual approaches for hate speech detection. the competition is hosted in codalab and offers a scenario for comparing new models with the existing ones. the resource repository would facilitate researchers to find, in one place, the datasets that better meet their needs as well as tools for easily comparing their work with previously existing models. from our repository, it is noticeable the lack of available spanish examples. moreover, there is a low representation of different types of spanish spoken worldwide.