Cross-Lingual Transfer Learning for Hate Speech Detection (2021)

Abstract: We address the task of automatic hate speech detection for low-resource languages. Rather than collecting and annotating new hate speech data, we show how to use cross-lingual transfer learning to leverage already existing data from higher-resource languages. Using bilingual word embeddings based classifiers we achieve good performance on the target language by training only on the source dataset. Using our transferred system we bootstrap on unlabeled target language data, improving the performance of standard cross-lingual transfer approaches. We use English as a high resource language and German as the target language for which only a small amount of annotated corpora are available. Our results indicate that cross-lingual transfer learning together with our approach to leverage additional unlabeled data is an effective way of achieving good performance on low-resource target languages without the need for any target-language annotations.

## introduction due to the increased digitization of society, the impact of online discourse on everyday life is becoming more pronounced. a single hateful message shared on social media now has the potential to incite violent offline movements, as well as exert a negative emotional impact on millions of readers. for this reason, platforms such as twitter and facebook have created community policies to ensure civil conduct on the part of their users. the goal is to filter hate speech, which unlike mere offensive or vulgar content, is exclusively designed to attack or denigrate entire groups of people and is generally agreed to add no constructive value to discussions. but with the sheer amount of posts being published, it is becoming difficult for humans to moderate them in a complete and timely manner. different annotators are also not guaranteed to agree on every classification, even in the presence of well-defined annotation guidelines. moreover, due to their repeated and prolonged exposure to negative content, many moderators experience a decline in mental health @xcite . for these reasons, automatic hate speech detection has become a field of high interest. in general, the task of classifying hate speech has been acknowledged as difficult @xcite . one reason is data scarcity: there are currently few public hate speech datasets available, and the majority of them are for english. thus, building systems for lower-resource languages is even more challenging @xcite ). an additional difficulty of the task is the need to precisely define hate speech. while many people have an intuitive understanding of what hate speech is, this does not easily translate to a finite set of characteristics. different hate speech datasets often deal with specific hate speech subtypes, leading to stark differences between the content of their hate speech classes and making the available resources for a given set of hate speech subtypes in a low-resource language even scarcer. the goal of this paper is to reduce these difficulties by exploiting available resources from other languages. we address data scarcity in german, a generally high-resource language but a language for which there are not yet many hate speech datasets (only 4 labeled corpora compared to the 25 available for english @xcite ). our method is a zero-shot setup that assumes we do not have any annotated training data in german. we develop a cross-lingual transfer learning approach based on bilingual word embeddings (bwes) and neural classifiers to provide access to hate speech data in english. we use the english dataset of de @xcite and the german dataset of the germeval shared task on the identification of offensive language @xcite in our experiments. however, their annotation had to be modified using a few simple rules to ensure label compatibility. in addition to training only on english, we bootstrap on two unlabeled german datasets, one of which we crawled from the web. using a set of our english-only models we predict the labels of previously-unseen data and keep only those on which the used models agree. by using the newlylabeled data as well for training, we further improved the performance on the target language. ## experiment setup this section introduces the setup of our experiments. first the source and target datasets are discussed, along with the ways in which they were modified for maximum compatibility. this is followed by a presentation of our cnn and cnn/bilstm model architectures. ## conclusion building automatic hate speech detection systems for low-resource languages is problematic due to the small amount of available datasets. our goal was to investigate whether cross-lingual transfer learning could be used to mitigate the problem of data scarcity. we chose an english dataset with a broad hate speech definition for training and a similar german corpus for testing. although the datasets were similar, we had to simplify the complex annotation schema of the target language dataset into the binary schema of the source dataset to make them compatible for the cross-lingual experiments. our results showed that cross-lingual transfer learning is indeed an effective tool for hate speech detection in low-resource languages. additionally, we assembled two corpora of previouslyunseen, unlabeled target language data and applied an ensemble of trained classifiers to them. we showed that fine-tuning on these automatically labeled examples improves classification performance. our goal for the future is to apply crosslingual transfer learning to other language pairs with greater syntactic differences than german and english, as well as employing current state-of-theart transformer models such as bert. in addition, since the differences of labeling schemas across various datasets could prevent the application of transfer learning methods, we aim at bridging the gap between incompatibly-labeled train and test datasets.