FBK - DH at S em E val-2020 Task 12: Using Multi-channel BERT for Multilingual Offensive Language Detection (2020)

Abstract: In this paper we present our submission to sub-task A at SemEval 2020 Task 12: Multilingual Offensive Language Identification in Social Media (OffensEval2). For Danish, Turkish, Arabic and Greek, we develop an architecture based on transfer learning and relying on a two-channel BERT model, in which the English BERT and the multilingual one are combined after creating a machine-translated parallel corpus for each language in the task. For English, instead, we adopt a more standard, single-channel approach. We find that, in a multilingual scenario, with some languages having small training data, using parallel BERT models with machine translated data can give systems more stability, especially when dealing with noisy data. The fact that machine translation on social media data may not be perfect does not hurt the overall classification performance.

## introduction concealed by perceived anonymity, users can often feel comfortable expressing offensive, abusive and hateful thoughts on the internet. therefore, the task of identifying offensive language on social media has increasingly gained attention in recent years, since the manual identification and deletion of such messages can be very costly and time-consuming, given the ever-increasing quantity of user-generated content online. in order to foster the development of offensive speech detection systems, more and more shared tasks have been organized on the detection and identification of offensive language in the past few years, covering several languages @xcite @xcite . in this paper, we present our submission to semeval 2020 task 12, multilingual offensive language identification in social media (offenseval 2) @xcite . we participated in subtask a, focused on the identification of offensive messages in 5 languages: english, danish, turkish, arabic and greek. in the semeval 2019 edition of offenseval, the task was focused on offensive language identification in english @xcite . the vast majority of the teams who obtained the highest macro-f1 scores in subtask a (focused on the binary identification of offensive language) used systems based on bidirectional encoder representations from transformers (bert) @xcite . since the 2020 edition of the task covers five languages, we aim at improving the performance of a multilingual pre-trained bert system by using transfer learning. this is achieved by artificially creating more data for fine-tuning two parallel pre-trained bert models by translating existing corpora in a different language and feeding the parallel data to a multiple input model. ## related work the increased interest in the detection of offensive language on social media has entailed an increase in shared tasks on the topic @xcite @xcite , as well as more data being annotated for this task @xcite @xcite . the topic of offensive language detection has been explored in relation to different subtasks, such as the detection of cyberbullying, hate speech, abusive language, aggression, and more @xcite @xcite . in general, the methods used for identifying offensive language follow supervised learning approaches. while support vector machines perform well on the task @xcite , deep learning approaches are becoming increasingly popular @xcite , in particular transformer-based ones @xcite . more specifically, in the 2019 instance of the offenseval task, 7 out of the 10 best performing systems used bert-based approaches @xcite . bert models are typically pre-trained on large unannotated corpora. these models can then easily be fine-tuned on task-specific data, and can achieve state-of-the-art performances on a number of nlp tasks @xcite . @xcite provide pre-trained bert models in english, chinese, and a multilingual model pre-trained on the 100 largest wikipedias foot_0 . @xcite successfully explore transfer learning for hate speech detection by fine-tuning a three-channel bert model, which is fed a tri-parallel corpus consisting of data in a language and its automatic translations into english and chinese. the data in each language is used to fine-tune the relative bert pre-trained model and, finally, the hidden states are added together through weighted sum. this approach proves successful in improving or stabilizing the performance of systems for the identification of hate speech in italian, german, and spanish tweets @xcite . we therefore build our work upon this framework using multi-channel bert and machine-translated data. ## data the data we used for participating in this task consists of the annotated datasets provided by the offenseval task organizers and additional corpora that we use to increase the size of the training sets. offenseval data the organizers of the offenseval 2 task provided the participants with a training set for each language the task was focused on: english, danish (sigurbergsson and derczynski, 2020), turkish (c ¸öltekin, 2020), arabic @xcite , and greek @xcite . the annotation guidelines and labels for all datasets are those of the offensive language identification dataset (olid) @xcite . olid consists of 14,100 english tweets, annotated on three levels. the labels off and not are used to indicate whether a tweet contains offensive content, constituting the first level of annotation, used in task a. an additional corpus for english was provided as "silver data" @xcite . this corpus includes 9 million tweets automatically annotated for hate speech. the annotations consist of the average of the confidences predicted by several supervised models for a tweet to belong to the off class, as well as the mean standard deviation of the predicted confidences from the given average. out of this corpus, we extracted the tweets with scores below 0.11 to be representative of the not class, and tweets with scores above 0.89 to represent the off class, amounting to 37.5k tweets overall. the datasets are heterogeneous both in terms of size and percentage of tweets labeled as offensive. an overview of their size and distribution is provided in additional data we retrieve additional training data in english by merging five widely used datasets for hate speech / abusive language detection, namely the ones from @xcite , @xcite , @xcite , @xcite and @xcite , after mapping their annotation to the off/not classes. we then extract around 30% of the tweets (27k messages) to extend the task training data. we choose not to use all the merged data because the tweets are then automatically translated and we want to limit the impact of wrong translations on classification performance. final datasets for each language other than english, we evaluate three different datasets: (1) languagespecific offenseval corpus only (oe), (2) oe + olid automatically translated into the language of interest (i.e. olid-da, olid-el, olid-ar and olid-tr), (3) oe + translated olid + additional 27k tweets from merged datasets, automatically translated into the language of interest. all translations have been performed using google apis. for the classification of english tweets, we used the olid tweets and the filtered silver dataset as training data (last two rows in ## classification framework baselines for danish, turkish, arabic, and greek, our baselines consist of a single-channel bert multilingual model fine-tuned on the available data for each language. a subset of this data (20%) is used as development set. pre-processing in all the data we use, user mentions are already replaced with @user and urls are replaced with url tags. approach we build a multi-channel bert system inspired by that of @xcite , in which we finetune both a pre-trained english bert base cased model and a pre-trained multilingual bert cased model in the same architecture. our methodology relies on two main steps. first, we create multiple parallel offensive language detection corpora by translating the corpus available for each of the four languages (the offenseval 2020 corpora) into english and the corpora available in english (olid and 27k) into each of the four languages. a summary of all the translation processes can be found in figure 1 . with this process, we significantly extend the training data available for each language, although some tweets may not be correctly translated. the second step consists in fine-tuning two bert models in parallel: multilingual bert cased and english bert base cased. our model takes as input two parallel datasets, one in the target language (lang) and one in english. we initialize the model weights for each model using the parameters from the two pre-trained bert models. then, the hidden representation of the first token ([cls]) of each sequence is pooled out through a pooling layer. we then add a dropout layer (with 0.1 dropout rate) and a dense layer (768 units, relu activation and l2 regularization), before adding the hidden states of each bert model together. finally, we add a softmax layer for classification. for all our experiments, both on our baseline and on the multi-channel model, we use batch size 32, sparse categorical cross-entropy loss, and adam optimizer with learning rate 2e-5, similarly to @xcite . a summary of our architecture can be seen in figure 2 . the code used for our system is available at github.com/ca-milla/multi-channel-bert. in order to choose our systems for the submission, a random sample of the training data in each configuration was retained to be used as development set. ## results and discussion ## conclusion in this paper, we present our system participating in the semeval 2020 task 12. our goal was not to develop different systems optimized for each language of the task, but rather to implement a framework based on two-channel bert and machine translated data that could be applied to danish, turkish, arabic and greek without many changes. we show that this approach yields good results, always outperforming single-channel bert, especially when few training data are available, since the system takes advantage of larger machine translated datasets in english. the most promising results using multi-channel bert are obtained on danish, and we will explore in the future whether it depends on the small size of the training data or on the similarities between english and danish, which may positively affect translation quality. while our approach is inspired by @xcite , there are some differences both in the architecture and in the results: our framework is simpler, in that we implement a two-channel bert, while they rely on three channels using also machine-translated texts from chinese. this makes our framework less computationally-intensive. as regards the results, we observe that in our experiments multi-channel bert is generally beneficial, while in @xcite results on italian and german are better when using a single channel.