Multilingual Offensive Language Identification with Cross-lingual Embeddings (2020)

Abstract: Offensive content is pervasive in social media and a reason for concern to companies and government organizations. Several studies have been recently published investigating methods to detect the various forms of such content (e.g. hate speech, cyberbulling, and cyberaggression). The clear majority of these studies deal with English partially because most annotated datasets available contain English data. In this paper, we take advantage of English data available by applying cross-lingual contextual word embeddings and transfer learning to make predictions in languages with less resources. We project predictions on comparable data in Bengali, Hindi, and Spanish and we report results of 0.8415 F1 macro for Bengali, 0.8568 F1 macro for Hindi, and 0.7513 F1 macro for Spanish. Finally, we show that our approach compares favorably to the best systems submitted to recent shared tasks on these three languages, confirming the robustness of cross-lingual contextual embeddings and transfer learning for this task.

## introduction offensive posts on social media result in a number of undesired consequences to users. they have been investigated as triggers of suicide attempts and ideation, and mental health problems @xcite . one of the most common ways to cope with offensive content online is training systems to be capable of recognizing offensive messages or posts. once recognized, such offensive content can be set aside for human moderation or deleted from the respective platform (e.g. facebook, twitter), preventing harm to users and controlling the spread of abusive behavior in social media. there have been several recent studies published on automatically identifying various kinds of offensive content such as abuse @xcite , aggression @xcite , cyber-bullying @xcite , and hate speech @xcite . while there are a few studies published on languages such as arabic @xcite and greek @xcite , most studies and datasets created so far include english data. data augmentation (ghadery and moens, 2020) and multilingual word embeddings (pamungkas and patti, 2019) have been applied to take advantage of existing english resources to improve the performance in systems dealing with languages other than english. to the best of our knowledge, however, state-of-the-art cross-lingual contextual embeddings such as xlm-r @xcite have not yet been applied to offensive language identification. to address this gap, we evaluate the performance of cross-lingual contextual embeddings and transfer learning (tl) methods in projecting predictions from english to other languages. we show that our methods compare favorably to state-of-the-art approaches submitted to recent shared tasks on all datasets. the main contributions of this paper are the following: ## related work there is a growing interest in the development of computational models to identify offensive content online. early approaches relied heavily on feature engineering combined with traditional machine learning classifiers such as naive bayes and support vector machines @xcite . more recently, neural networks such as lstms, bidirectional lstms, and grus combined with word embeddings have proved to outperform traditional machine learning methods in this task @xcite . in the last couple of years, transformer models like elmo @xcite and bert @xcite have been applied to offensive language identification achieving competitive scores and topping the leaderboards in recent shared tasks @xcite . most of these approaches use existing pretrained transformer models which can also be used as text classification models. the clear majority of studies on this topic deal with english @xcite @xcite partially motivated by the availability english resources (e.g. corpora, lexicon, and pre-trained models). in recent years, a number of studies have been published on other languages such as arabic @xcite , danish (sigurbergsson and derczynski, 2020), dutch @xcite , french @xcite , greek @xcite , italian @xcite , portuguese @xcite , slovene @xcite , and turkish (c ¸öltekin, 2020) creating new datasets and resources for these languages. recent competitions organized in 2020 such as trac @xcite and offenseval @xcite have included datasets in multiple languages providing participants with the opportunity to explore cross-lingual learning models opening exciting new avenues for research on languages other than english and, in particular, on low-resource languages. the aforementioned deep learning methods require large annotated datasets to perform well which is not always available for low-resource languages. in this paper, we address the problem of data scarcity in offensive language identification by using transfer learning and crosslingual transformers from a resource rich language like english to three other languages: bengali, hindi, and spanish. ## data we acquired datasets in english and three other languages: bengali, hindi, and spanish (listed in a: offensive language identification -offensive vs. non-offensive; b: categorization of offensive language -targeted insult or thread vs. untargeted profanity; c: offensive language target identification -individual vs. group vs. other. we chose olid due to the flexibility provided by its hierarchical annotation model that considers multiple types of offensive content in a single taxonomy (e.g. targeted insults to a group are often hate speech whereas targeted insults to an individual are often cyberbulling). this allows us to map olid level a (offensive vs. non-offensive) to labels in the other three datasets. olid's annotation model is intended to serve as a general-purpose model for multiple abusive language detection subtasks @xcite . the transfer learning strategy used in this paper provides us with an interesting opportunity to evaluate how closely the olid labels relate to the classes in datasets annotated using different guidelines and sub-task definitions (e.g. aggression and hate speech). the hindi dataset @xcite was used in the hasoc 2019 shared task, while the spanish dataset (basile et al., 2019) was used in semeval-2019 task 5 (hateval). they both contain twitter data and two labels. the bengali dataset (bhattacharya et al., 2020) was used in the trac-2 shared task @xcite ) on aggression identification. it is different than the other three datasets in terms of domain (facebook instead of twitter) and set of labels (three classes instead of binary), allowing us to compare the performance of cross-lingual embeddings on off-domain data and off-task data. ## methodology transformer models have been used successfully for various nlp tasks @xcite . most of the tasks were focused on english language due to the fact the most of the pre-trained transformer models were trained on english data. even though, there were several multilingual models like bertm @xcite there was much speculations about its ability to represent all the languages @xcite and although bert-m model showed some cross-lingual characteristics it has not been trained on crosslingual data @xcite . the motivation behind this methodology was the recently released cross-lingual transformer models -xlm-r @xcite which has been trained on 104 languages. the interesting fact about xlm-r is that it is very compatible in monolingual benchmarks while achieving the best results in cross-lingual benchmarks at the same time @xcite . the main idea of the methodology is that we train a classification model on a resource rich, typically english, using a crosslingual transformer model, save the weights of the model and when we initialise the training process for a lower resource language, start with the saved weights from english. this process is also known as transfer learning and is illustrated in figure 1 . there are two main parts of the methodology. subsection 4.1 describes the classification architecture we used for all the languages. in subsection 4.2 we describe the transfer learning strategies used to take advantage of english offensive language data in predicting offense in less-resourced languages. ## results and evaluation we evaluate the results obtained by all models using the test sets provided by the organizers of each competition. we compared our results to the best systems in trac-2 for bengali, hasoc for hindi, hateval for spanish in terms of weighted and macro f1 score according to the metrics reported by the task organizers -trac-2 reported only macro f1, hateval reported only weighted f1, and hasoc reported both. finally, we evaluate the improvement of the transfer learning strategy in the performance of both bert and xlm-r. we present the results along with the majority class baseline for each language in for hindi, transfer learning with xlm-r cross lingual embeddings provided the best results achieving 0.8568 and 0.8580 weighted and macro f1 score respectively. in hasoc 2019 @xcite , the best model by bashar and nayak (2019) scored 0.8149 macro f1 and 0.8202 weighted f1 using convolutional neural networks. for spanish transfer learning with xlm-r cross lingual embeddings also provided the best results achieving 0.7513 and 0.7591 macro and weighted f1 score respectively. the best two models in hat-eval @xcite for spanish scored 0.7300 macro f1 score. both models applied svm classi-fiers trained on a variety of features like character and word n-grams, pos tags, offensive word lexicons, and embeddings. the results for bengali deserve special attention because the bengali data is off-domain with respect to the english data (facebook instead of twitter), and it contains three labels (covertly aggressive, overtly aggressive, and not aggressive) instead of the two labels present in the english dataset (offensive and non-offensive). tl indicates that the model used the inter-task, inter-domain, and inter-language transfer learning strategy described in subsection 4.2. similar to the hindi and spanish, transfer learning with xlm-r cross lingual embeddings provided the best results for bengali achieving 0.7513 and 0.7591 macro and weighted f1 respectively thus outperforming the other models by a significant margin. the best model in the trac-2 shared task @xcite ) scored 0.821 weighted f1 score in bengali using a bert-based system. we look closer at the test set predictions by xlm-r (tl) for bengali in figure 3 . we observe that the performance for the non-aggressive class is substantially better than the performance for the overtly aggressive and covertly aggressive classes following a trend observed by the trac-2 participants including @xcite . finally, it is clear that in all the experimental settings, the cross-lingual embedding models finetuned with transfer learning, outperforms the best system available for the three languages. furthermore, the results show that the cross-lingual nature of the xlm-r model provided a boost over the multilingual bert model in all languages tested. ## conclusion this paper is the first study to apply cross-lingual contextual word embeddings in offensive language identification projecting predictions from english to other languages using benchmarked datasets from shared tasks on bengali @xcite , hindi @xcite . we have showed that xlm-r with transfer learning outperforms all of the other methods we tested as well as the best results obtained by participants of the three competitions. the results obtained by our models confirm that olid's general hierarchical annotation model encompasses multiple types of offensive content such as aggression, included in the bengali dataset, and hate speech included in the hindi and spanish datasets, allowing us to model different subtasks jointly using the methods described in this paper. furthermore, the results we obtained for bengali show that it is possible to achieve high performance using transfer learning on off-domain (twitter vs. facebook) and off-task data when the labels do not have a direct correspondence in the projected dataset (two in english and three in bengali). this opens exciting new avenues for future research considering the multitude of phenomena (e.g. hate speech, aggression, cyberbulling), annotation schemes and guidelines used in offensive language datasets. in future work, we would like to further evaluate our models using solid, a novel large english dataset with over 9 million tweets @xcite , along with datasets in four other languages (arabic, danish, greek, and turkish) that were made available for the second edition of offenseval @xcite . these datasets were collected using the same methodology and were annotated according to olid's guidelines. finally, we would also like to apply our models to languages with even less resources available to help coping with the problem of offensive language in social media.