Zero-shot Cross-lingual Content Filtering: Offensive Language and Hate Speech Detection (2021)

Abstract: We present a system for zero-shot cross-lingual offensive language and hate speech classification. The system was trained on English datasets and tested on a task of detecting hate speech and offensive social media content in a number of languages without any additional training. Experiments show an impressive ability of both models to generalize from English to other languages. There is however an expected gap in performance between the tested cross-lingual models and the monolingual models. The best performing model (offensive content classifier) is available online as a REST API.

## introduction recent years have seen a dramatic improvement in natural language processing, with machine learning systems outperforming human performance on a number of benchmark language understanding tasks @xcite . this impressive achievement is somewhat tempered by the fact that a large majority of these systems work only for english, while other less-resourced languages are neglected due to a lack of training resources. on the other hand, another recent development is the introduction of systems capable of zero-shot cross-lingual transfer learning by leveraging multilingual embeddings @xcite . these systems can be trained on a language with available resources and employed on a less-resourced language without any additional language specific training. in this study we present an offensive language classifier available through a rest api which leverages the cross-lingual capabilities of these systems. due to the exponential growth of social media content, the amount of offensive language and hate speech has seen a steep increase and its identification and removal is no longer manageable by traditional manual inspection of the content @xcite . as a consequence, there is a need for a general model that could be used in content filtering systems to automatically detect such discourse. since the majority of research in the area of offensive language and hate speech detection is currently done in monolingual settings, we performed a preliminary study to assess the feasibility of the proposed zero-shot cross-lingual transfer for this task. two approaches are tested in this study. the first uses multilingual bidirectional encoder representations from transformers @xcite . the second uses language-agnostic sentence representations (laser, artetxe and schwenk, 2019), a system built specifically for zero-shot cross-lingual transfer using multilingual sentence embeddings. our best performing model is available online and can be used for detecting offensive content in less-resourced languages with no available training data. ## related work the large majority of research on hate speech is monolingual, with english still the most popular language due to data availability @xcite , and a number of english-only shared tasks organized on the topic of hate or offensive speech (e.g., offenseeval, @xcite . lately, the focus has been shifting to other languages, with several shared tasks organized that cover other languages besides english, e.g. offenseeval 2020 @xcite , evalita 2018 @xcite and ger-meval 2018 @xcite . for example, the evalita 2018 shared task @xcite covered hate speech in italian social media, the germeval 2018 @xcite shared tasks explored automatic identification of offensive german tweets, and semeval 2019 task 5 @xcite covered detection of hate speech against immigrants and women in spanish and english twitter. schmidt and wiegand (2017 ); poletto et al. (2020); vidgen and derczynski (2020) provide excellent surveys of recent hate speech related datasets. ousidhoum et al. (2019) conduct multilingual hate speech studies by testing a number of traditional bag-of-words and neural models on a multilingual dataset containing english, french and arabic tweets that were manually labeled with six class hostility labels (abusive, hateful, offensive, disrespectful, fearful, normal). they report that multilingual models outperform monolingual models on some of the tasks. shekhar et al. (2020) study multilingual comment filtering for newspaper comments in croatian and estonian. another multilingual approach was proposed by schneider et al. (2018), who used multilingual muse embeddings (lample et al., 2018) in order to extend the germeval 2018 german train set with more english data. they report that no improvements in accuracy were achieved with this approach. cross-lingual hate speech identification is even less researched than the multilingual task. the so-called bleaching approach (van der goot et al., 2018) was used by basile and rubagotti (2018) to conduct cross-lingual experiments between italian and english at evalita 2018 misogyny identification task. the only other study we are aware of is a very recent study by pamungkas and patti (2019) proposing an lstm joint-learning model with multilingual muse embeddings. google translate is used for translation in order to create a bilingual train and test input data. bassignana et al. (2018) report that the use of a multilingual lexicon of hate words, hurtlex, slightly improves the performance of misogyny identification systems. closest to our work is that of glavaš et al. (2020), who propose a dataset called xhate-999 to evaluate abusive language detection in a multi-domain and multilingual setting. ## dataset description as an english (en) training set for offensive language classification, we used the training subset of the olid dataset @xcite . the trained models were evaluated on the test subset of the olid dataset using their official gold labels and on the test subset of the germeval 2018 dataset @xcite , which also contains manually labeled tweets. both datasets use hierarchical annotation schemes for annotating hate speech content. for our purposes, we employed only the annotations on the first level which classify tweets into two classes, offensive and not offensive. we trained the hate speech classifiers on the english training set from the hateval dataset @xcite . for evaluation, we used the english and spanish (es) test sets from the hateval competition, the german (de) igw hate speech dataset @xcite , an indonesian (id) hate speech dataset (ibrohim and budi, 2019) and the arabic (ar) hate speech dataset lhsab @xcite . each of the test datasets had binary labels that denoted the presence or absence of hate speech, except for the arabic test set, which modeled hate speech as a three-class task, with labels denoting absence of hate speech, abusive language and hateful language. since the authors themselves acknowledge there is a fine line between abusive and hateful language, we felt confident to join them into one class that denotes the presence of hate speech in a tweet. tweets in the german igw dataset included hate speech labels from two annotators and no common label, so we decided to evaluate only on those tweets where the two annotators agreed. the statistics of the datasets that were used in this study are reported in ## classification models and methodology our models were trained and evaluated on two distinct albeit similar tasks, namely offensive language classification and hate speech detection, using two different approaches. in the first approach, we tested the multilingual version of bert to which we attached a classification layer with a softmax activation function. the model was fine-tuned on the chosen training datasets for 20 epochs. we limited the input sequence to 256 tokens and used a batch size of 32 and a learning rate of 2e-5. no additional hyperparameter tuning was performed. our second approach was using the pre-trained laser model and training a multilayer perceptron classifier with relu activation function on top of that. to train the models we used the batch size of 32 and a learning rate of 0.001. ## results the results for both tasks together with the majority baselines and the results reported in the literature are presented in when evaluated on the german dataset, we observe a considerable drop in performance compared to the reported results @xcite , however, it still achieves a solid f1 score of 70.67, which indicates its ability to generalize to languages it has not seen during training. in the hate speech classification task, the two models are comparable, with laser outperforming bert on the arabic and spanish datasets. overall, the scores for the hate speech classification task proved to be considerably lower for both models as well as lower than the reported results in the monolingual experiments @xcite . nevertheless, the results again indicate the ability of both models to generalize from english to other languages, as our models perform better than the majority baseline classifiers in terms of macro-averaged f1 score on all the datasets. it should be noted that the performance between our models and the reported performance on the indonesian and arabic datasets are not directly comparable as the original training and testing splits from the literature are not available. therefore, our models were tested on different test splits. ## web api design the best performing cross-lingual model, multilingual bert for offensive language classification, was implemented as a rest web service in the flask framework. the design of the web service allows us to easily update the current model with a new version trained on additional data in the future. the web service can be reached programmatically through the endpoint at http:// classify.ijs.si/ml_hate_speech/ml_bert or through a demo browser-based interface at the url http://classify.ijs.si/embeddia/ offensive_language_classifier . the interface is designed for mobile devices and supports most popular screen sizes. it consists of an input area where users can input their sentence and submit it for classification. the classification results as well as the confidence score of the classifier are then displayed under the input area. ## conclusion and future work in the course of this study, we tested the performance of two multilingual models, bert and laser, in zero-shot offensive language and hate speech detection. the results for the offensive language classification task show that even in the multilingual setting the bert-based classifier achieves results comparable to the monolingual classifiers on english language data and solid performance on the german dataset. on the other hand, hate speech classification still proves to be a hard task for the multilingual classifiers as they achieve considerably lower scores on all languages compared to reported results. nevertheless, both models show an impressive ability to generalize over languages they have not seen during fine-tuning. we implemented the best performing model, multilingual bert for offensive language classification, as a rest web service. in the future, we plan to perform similar experiments with other multilingual language models, namely the xlm-r models @xcite , which show increased performance in standard benchmark tasks compared to multilingual bert, and the recently released crosloengual-bert @xcite . while all datasets used in this study contain social media posts labeled for hate speech or of-