Multilingual Racial Hate Speech Detection Using Transfer Learning (2023)

Abstract: The rise of social media eases the spread of hateful content, especially racist content with severe consequences. In this paper, we analyze the tweets targeting the death of George Floyd in May 2020 as the event accelerated debates on racism globally. We focus on the tweets published in French for a period of one month since the death of Floyd. Using the Yandex Toloka platform, we annotate the tweets into categories as hate, offensive or normal. Tweets that are offensive or hateful are further annotated as racial or non-racial. We build French hate speech detection models based on the multilingual BERT and CamemBERT and apply transfer learning by fine-tuning the HateXplain model. We compare different approaches to resolve annotation ties and find that the detection model based on CamemBERT yields the best results in our experiments.

## introduction the rapid advancements of social media platforms like facebook, twitter, and youtube during the last couple of years have enabled users to express and distribute their sentiments on events and ideas freely and conveniently. this eases the usage of hateful messages that can imply threats or harassment against minorities @xcite . since there are variations in defining hate speech globally, we took the following explanations as working definitions in this research. therefore, hate speech is defined as a public communication consisting of messages that may express threats, harassment, intimidation, or disparagement of a person or a group on the basis of some characteristic such as race, color, ethnicity, gender, sexual orientation, nationality, religion, culture or other characteristic @xcite . besides, offensive speech is also hurtful speech that is directed against another person. compared with hate speech, offensive speech has fewer legal implications since it does not attack people based on their group identity, rather it hurts individuals based on personal characteristics and makes them offended. more specifically, racism is a type of discrimination that makes up a large portion of hate speech and is usually directed against the perceived ethnicity, appearance, religion, or culture @xcite . after the killing of george floyd on may 25th, 2020, the number of racist comments on social media platforms, especially on twitter, has increased substantially @xcite . social media platforms use mainly content moderation systems, which are human-machine collaborative systems to detect and handle hate speech as an automatic detection system in spite of the limitations that such systems have to control the problem @xcite . these days, the task of automatic hate speech detection in general and racial hate speech, in particular, has attracted the attention of many natural language processing researchers. to advance the development of hate speech detection algorithms in multiple languages, we extend the english hate speech detection model from hatexplain @xcite to the french language by employing our own annotated dataset. despite there are various types of discrimination and intersections among them, we limit the scope of our research to racial discrimination which is one of the most critical problems in society @xcite . the study addresses the following research questions: in this paper, we employ a crowdsourcingbased racial hate speech data annotation using the yandex toloka platform 1 . moreover, we fine-tuned hatexplain @xcite , which is a bertbased classification model for tweets in the french language. the main contributions of this research include the following: the remainder of the paper is organized as follows. the paper provides the related works in section 2. while the data collection procedures and strategies are presented in section 3, the data annotation strategies are briefly discussed in section 4. we present our experiments including the baseline models, the results, and the error analysis in section 5. finally, the conclusion and future work are 1 yandex toloka: https://toloka.yandex.com presented in section 6, and the limitations of the research are indicated in section 7. ## related works in academia, there is a strong interest in detecting hate speech and exploring the challenges facing the task. to address the issue, many researchers attempted hate speech studies by creating their own datasets and building classification models that can detect and classify hateful content from texts on social media platforms. in this regard @xcite al. (2020); mathew et al. (2021); ousidhoum et al. (2019); davidson et al. (2017); wang et al. (2021); waseem and hovy (2016); vidgen and derczynski (2020); matamoros-fernández and farkas (2021) ; @xcite and many other researchers investigated hate speech and developed classification models. most of the studies use twitter data @xcite . according to the work by @xcite , twitter data is the most widely used source of data for computational social science such as hate speech and sentiment analysis tasks. some researchers use lexical methods to retrieve social media texts based on the entries in a lexicon and build datasets for social computing @xcite . the work by @xcite analyzed the quality of lexical methods and proved that it is more effective to detect offensive language than hate speech. they also identified racism and homophobia more often as hate speech while sexism is more often offensive. hate speech, racism, and racial profiling are less studied in french when compared with english @xcite . as indicated in there are fewer annotated datasets that deal with racist speech than for general hate speech, in particular for the french language @xcite . a few studies were conducted on racial hate speech in french. @xcite created a french corpus of the sexist dataset by collecting tweets using keywords and becomes the first dataset to detect sexism and multi-target hate speech. models developed for other languages such as english can not be properly adopted for racial hate speech classification in french due to contexts variations in culture and differences in linguistic features. @xcite presented a hate speech dataset annotated in three different perspectives such as: for example, @xcite fine-tuned the bert model of @xcite by adding a fully connected layer with the output corresponding to the classification tokens in the input where the token output usually holds the representation of the sentence to add attention supervision that matches the attention values corresponding to the token in the final layer. ## data collection most of the existing hate speech datasets in french and other languages do not focus on racial hate speech. the dataset used in this research is collected from twitter focusing on tweets that are published for one month following the death of george floyd foot_0 . the death of george floyd accelerated debates and demonstrations globally. following the death, social media platforms such as twitter, facebook, and youtube have become places for hate and offensive speeches in general and racial hate speech in particular. we employed 3,473 french hate speech lexicon entries adapted from the work of @xcite ; @xcite to filter the tweets that might contain racial hate speech content from the total 200m tweet corpus. we used the python language detection foot_1 tool to filter tweets that are only written in french. we also removed truncated tweets since such tweets lack complete information and may confuse the annotators during annotation, and the model during experimentation. we removed retweets and kept only unique tweets that are not duplicated. moreover, usernames and urls are anonymized and replaced @xmath0. a total of 5k tweets are annotated using three independent annotators on yandex toloka crowdsourcing platform. ## annotation annotation by itself is a very complex task and becomes more challenging for hate speech annotations due to the lack of complete background contexts behind the texts scrapped from social media platforms @xcite . we annotated 5k tweets on toloka crowdsourcing platform and each tweet is annotated by three independent toloka performers. we annotated 50 random tweets and evaluated the annotations by experts for the correctness of the corresponding labels. these control tweets were used to control malicious annotators engaging in the annotation task. each task presented to performers contains 15 tweets and one of the tweets is a control question. users are asked to classify tweets into hate, offensive, normal and unsure, and further classify hateful tweets into racial, non-racial and unsure. if hate is chosen by an annotator, the targets racial, non-racial, and unsure will pop up immediately for the performer. the unsure label is provided to give performers the opportunity to indicate that a tweet is very hard to classify. according to the work by @xcite , providing the basic definitions and task descriptions of the annotation project beforehand improves the alignment of the opinions of the annotators on the class labels. we presented the annotation guideline to provide a complete description of the annotation task. two training task pools structured in the same way as the actual task were presented to be completed by toloka performers before joining the main annotation task. such procedures can help toloka performers to have sufficient knowledge and understanding of the annotation task. one of the main challenges of crowdsourcing data annotation is the prevalence of malicious data annotators who merely participate in the annotation task to gain financial rewards @xcite . in order to prevent potential malicious performers from engaging in the annotation task, we prepared a french language test and presented it to each performer as indicated in figure 1 . toloka performers needed to pass the french language test in order to participate in the main french racial hate speech annotation task. we also limited the location of performers and allowed those performers who lived in france or belgium. the performers who successfully completed the two training task pools, lived in france or belgium, and passed the french language test were qualified and provided the privilege to access the main annotation task pools. a fleiss kappa of 0.3 inter-annotator agreement, which indicated a fair agreement, is achieved. each tweet was annotated by three annotators and the final gold label was aggregated from these three annotations with a ma- fleiss kappa score 0.3 total number of annotated tweets 5002 number of annotators participated in the task 275 mean age of annotators in years 31.11 country distribution of annotators 265 fr, 8 be, 3 o accuracy for 50 random tweets 0.24 f1 score for 50 random tweets 0.24 racial accuracy for 50 random tweets 0.12 average time for 15 tweets 2 min 10 sec number of collected keywords 3473 ## conclusion this paper presented the collection of racial hate speech datasets from twitter. the dataset was collected for a period of one month following the death of george floyd in may 2020 as his murder was associated with racism. the debate regarding racism escalated during that time and racist speeches and expressions on almost all social media platforms were also aggravated. a total of 5k tweets are annotated as hate, offensive, normal, and unsure using toloka. furthermore, hate and offensive tweets were labeled as racial, non-racial, and unsure classes. this dataset can be used as a benchmark dataset for french racial hate speech research. the bert model is successfully fine-tuned with the dataset together with the translated hatexplain dataset. our experiment achieved an accuracy of 88% and an f1-score of 86% which are improving over the baseline hatexplain model. in future work, we plan to work on further filtering the lexicon entries in order to reduce the class imbalance problem. extending the dataset to include the racial targets and the rationales of the label decisions can also be future work. we published the resources in github foot_6 .