BRUMS at S em E val-2020 Task 12: Transformer Based Multilingual Offensive Language Identification in Social Media (2020)

Abstract: In this paper, we describe the team BRUMS entry to OffensEval 2: Multilingual Offensive Language Identification in Social Media in SemEval-2020. The OffensEval organizers provided participants with annotated datasets containing posts from social media in Arabic, Danish, English, Greek and Turkish. We present a multilingual deep learning model to identify offensive language in social media. Overall, the approach achieves acceptable evaluation scores, while maintaining flexibility between languages.

## introduction social media has become a normal medium of communication for people these days as it provides the convenience of sending messages fast from a variety of devices. unfortunately, social networks also provide the means for distributing abusive and aggressive content. given the amount of information generated every day on social media, it is not possible for humans to identify and remove such messages manually, instead it is necessary to employ automatic methods. as offensive language becomes pervasive in social media, scholars and companies have been working on developing systems capable of identifying offensive posts, which can be set aside for human moderation or permanently deleted @xcite . along with these studies, a few shared tasks have been organized on detecting offense in social media, such as hateval @xcite , hasoc @xcite , trac @xcite and offenseval @xcite co-located with semeval 2019. in semeval 2020, offenseval returns for the second time with multilingual offensive language identification in social media. offenseval 2 focuses on several languages arabic, danish, english, greek and turkish motivating participants to submit multilingual offensive language identification systems. this paper revisits the problem of offensive language identification describing our submission to the semeval-2020 task 1 @xcite . the remainder of this paper is structured as follows: section 2 describes the related work done in the field of aggression detection, section 3 has a description of the dataset. section 4 describes the system that was submitted, split into a description of how the data was processed and the architectures of the classifiers that were used. section 5 presents an analysis of the results of our evaluation of the different architectures, as well as of the final submission. finally, section 6 offers some final remarks and a conclusion. ## relevant work the majority of text classification approaches in early research work were supported by traditional or feature-based supervised learning methods. following this tradition, @xcite proposed a linear support vector machine (svm) classifier for hate speech detection in social media. the best accuracy of this approach was obtained by using character n-grams as model features. another research suggested stacked ensemble system using logistic regression and random forest-based classifiers @xcite . for initial layer of classifiers, they used character n-grams, token n-grams, important tokens and word2vec embeddings. the outputs of initial layer classifiers were used as the inputs of next classifier layer which produces the final result. this approach obtained the best results in germeval 2018 @xcite by outperforming the deep neural network (dnn) models such as convolutional neural networks (cnn) @xcite and combination of bi-directional long-short term memory and cnn (bilstm-cnn) @xcite . however, most of the recent research conducted in this area prove that dnn-based methods can outperform traditional supervised learning methods @xcite . but, unlike the traditional methods, dnns require sufficient amount of training data to adjust their weights properly without overfitting. focusing on this issue, @xcite found that lstm with higher dropout (≈ 0.5) can handle overfitting problem. in order to increase the model performance using a high number of training instances in a simple manner, the available data sets related to aggression and cyberbullying detection can be merged @xcite . but, when focused on a particular area, it cannot guarantee the availability of multiple data sets. another research suggested data augmentation and pseudo labelling approaches to increase the amount of training data @xcite . as the data augmentation strategy, they proposed to translate each instance into an intermediate language and back to the original language. for pseudo labelling approach, they trained some initial models using available training data and picked the best model to label new data. a lstm model trained using the augmented data set received the first place in trolling, aggression and cyberbullying (trac-2018) shared task designed for facebook english data set. in the same shared task, for the platform unnamed english data set, third place could be obtained by a cnn-lstm model trained using the combination of augmented and pseudo labelled data sets. in addition to approaching data set increasing methods, transfer learning techniques also used by previous research to overcome the data limitation issue. @xcite showed that transfer learning can be applied using some labelled data set with relevant labels or unlabelled data set with unsupervised user clusters or topics. further, as mentioned in semeval-2019 offenseval leader-board, the top 5 solutions for offensive language identification are bert-based and they proved the effectiveness in transfer learning as well as transformers @xcite . the top solution of this shared task fined tuned a bert model using the training data and found that it outperforms a lstm model with higher dropout @xcite . in summary, majority of recent research in this area were supported by dnn models and among them, cnn, lstm and bert-based models were found to be most effective and commonly used. when focused on the input data representation, there was a tendency to use fasttext embeddings to handle the high availability of spelling mistakes and new words in social media @xcite @xcite . following this tendency, in this research, we further experimented the effectiveness of dnn models including some recently published transformers for offensive language identification. ## dataset offenseval 2020, the semeval 2020 offensive language detection shared task, does not provide a new manually labeled training set for the english language @xcite . the competitors were recommended to use the offensive language identification dataset (olid) which was released for the the semeval 2019 offensive language detection shared task @xcite . the olid is a hierarchical dataset to identify the type and the target of offensive texts in social media. the dataset is collected on twitter and publicly available. there are 14,100 tweets in total, in which 13,240 are in the training set, and 860 are in the test set. for each tweet,there are three levels of labels:(a) offensive/not-offensive, (b) targeted-insult/untargeted, (c) indi-vidual/group/other. for english subtask organisers provided an additional dataset similar to olid @xcite , it still has three levels, but this time only confidence scores, generated by different models, are provided rather than human annotated labels. in addition, the data in level a is separated from levels b and c. in level a, there are 9,089,140 tweets,in levels b and c, there are different 188,973 tweets. for non-english languages the organisers released multilingual offensive language identification dataset (molid). it contains four languages: arabic @xcite , danish (sigurbergsson and derczynski, 2020), greek @xcite , and turkish (c ¸öltekin, 2020). ## methodology in this section we present the methodology employed for the shared task. all the languages followed a common methodology which had two steps: data preprocessing and machine learning. ## results in this section we present the evaluation results that were obtained during testing for each language. we also provide a brief look at the final submission results of the shared task. in all the languages multilingual distilbert does not perform better than multilingual bert model we used. therefore, we have omitted it from the results. ## conclusion in this paper, we have presented the team brums system for identifying offensive language in social media posts in arabic, danish,english, greek and turkish. the system uses minimal preprocessing, and relies on word and context embeddings. we experimented with different deep neural network architectures in order to determine the most suitable for this task. our implementation has been made available on github. 10 . according to our evaluation, and the results provided by the task organisers, it is clear that fine tuning transformer architectures score highest overall. also we discovered that pretrained monolingual transformer models perform better than the bert-multilingual model. in future, we are hoping to experiment more with transformer architectures in different languages.