UPB at S em E val-2020 Task 12: Multilingual Offensive Language Detection on Social Media by Fine-tuning a Variety of BERT -based Models (2020)

Abstract: Offensive language detection is one of the most challenging problem in the natural language processing field, being imposed by the rising presence of this phenomenon in online social media. This paper describes our Transformer-based solutions for identifying offensive language on Twitter in five languages (i.e., English, Arabic, Danish, Greek, and Turkish), which was employed in Subtask A of the Offenseval 2020 shared task. Several neural architectures (i.e., BERT, mBERT, Roberta, XLM-Roberta, and ALBERT), pre-trained using both single-language and multilingual corpora, were fine-tuned and compared using multiple combinations of datasets. Finally, the highest-scoring models were used for our submissions in the competition, which ranked our team 21st of 85, 28th of 53, 19th of 39, 16th of 37, and 10th of 46 for English, Arabic, Danish, Greek, and Turkish, respectively.

## introduction social media platforms are gaining increasing popularity for both personal and political communication. recent studies uncovered disturbing trends in communications on the internet. for example, pew research center 1 discovered that 60% of internet users have witnessed a form of online harassment, while 41% of them have personally experienced it. the majority of the latter category says the most recent such experience occurred on a social media platform. although most of these platforms provide ways of reporting offensive or hateful content, only 9% of the victims have considered using these tools. traditionally, identifying and removing offensive or hateful content on the internet is performed by human moderators that inspect each piece of content flagged by the users and label it appropriately. this process has two major disadvantages. as previously mentioned, the first one is that the proportion of users that even considered using the tools provided by the platforms in order to report the harmful content is very small. the second one is represented by the continuously growing volume of data that needs to be analyzed. however, the task of automated offensive language detection on social media is a very complicated problem. this is because the process of labeling an offensive language dataset has proved to be very challenging, as every individual reacts differently to the same content, and the consensus on assigning a label to a piece of content is often difficult to obtain @xcite . the semeval-2019 shared task 6 @xcite , offenseval 2019, was the first competition oriented towards detecting offensive language in social media, specifically twitter. the semeval-2020 shared task 12 @xcite , offenseval 2020, proposes the same problem, the novelty being a very large automatically labeled english dataset and also smaller datasets for four other languages: arabic, danish, greek, and turkish. this paper describes the transformer-based machine learning models we used in our submissions for each language in subtask a, where the goal is to identify the offensive language in tweets -a binary classification problem. the remainder of the paper is structured as follows: in section 2, a brief analysis of the state-of-the-art approaches is performed. in section 3, the methods employed for automated offensive language detection are presented. section 4 describes the data used in this study. section 5 presents the evaluation process, and in section 6, conclusions are drawn. ## related work automating the task of offensive language detection becomes a necessity on the internet of today, especially on communication platforms. the efforts in this direction have substantially grown in the research community. the first approaches on a related subject include the detection of racist texts in web pages @xcite , where the authors considered part-of-speech tags as inputs for support vector machines. this type of problems gained a lot of interest in the last decade, as advanced machine learning techniques has been developed for nlp tasks, and also computing power became increasingly accessible. @xcite proposed the detection of web trolling (i.e., posting outrageous messages that are meant to provoke an emotional response). hate speech and offensive language detection in twitter samples was analyzed by @xcite . their study presents a framework for differentiating between profanity and hate speech and also describes the annotation process of such a dataset. moreover, their experiments with various text preprocessing techniques are described and the logistic regression algorithm is used for hate speech and offensive language classification. @xcite continued the experiments on this dataset using n-gram and skip-gram features. more recently, neural networks gained interest in this type of problems. for example, @xcite relied on a convolutional neural network (cnn) @xcite to surpass the state-ofthe-art results on the previously mentioned dataset. @xcite also improved these results by combining two different deep learning architectures, namely cnn and gated recurrent unit @xcite . there are a series of surveys and shared-tasks that took place in the last years on the subject of detecting the online offensive, abusive, hateful, or toxic content. @xcite introduced a comprehensive survey on different methods to automatically recognize hate speech, focusing mostly on non-neural network approaches. shared-tasks analyzing problems in the same areas include both editions of abusive language online @xcite , which focused mostly on cyberbullying, trac @xcite , which mainly studied aggressiveness, hasoc @xcite , which also addressed the problem of hate speech, the same as the semeval-2019 task 5 (basile et al., 2019) competition. ## data to analyze the influence of extending the competition-provided training datasets with other corpora constructed for related tasks, we used two additional datasets for fine-tuning the previously mentioned models. the summary of each dataset is presented in ## experiments all the experiments were performed using an ubuntu machine with 64gb ram and one nvidia titan x gpu. the hardware limitations are the reason we only experimented with the base versions of the previously mentioned architectures. the transformer python package @xcite was used for training and evaluating the transformer-based models, and each model was fine-tuned for four epochs. we also used the adam algorithm (kingma and ba, 2014) with weight decay for optimization and a learning rate of 2e-5. for each language, multiple experiments were performed with the same architecture using several combinations of datasets for fine-tuning, in order to assess the impact each of them has on the performance of a certain model. this implies that, for some experiments, several datasets were simply concatenated and used as a single fine-tuning set. no additional handling of the data was required, as all of the data shared the binary label structure presented in section 4. the results we obtained on the validation datasets are summarized in tables 2, 3, 4, 5, and 6 for english, arabic, danish, greek, and turkish, respectively. the reported metrics are computed for the offenseval 2020 language-specific validation sets as described in section 4. for each language, the highest validation set f1-score is highlighted, meaning that the corresponding model was selected and employed for predicting the language-specific competition test data in our final submission. ## conclusions this work presented our approaches to automatically detect offensive language in multilingual tweets, as part of semeval-2020 task 12. we proved that the deep learning solution of fine-tuning pre-trained transformer-based models can be used successfully to classify offensive language, and we experimented with several such architectures, fine-tuned on multiple combinations of datasets. comparing the validation set performances against the test set results, we discovered that the last of them were better for the non-english languages, which shows that our models generalize well and also that the proposed test data may be easier to classify than the development data. the smallest positive difference between validation and testing performances was obtained for the danish subset, which may indicate that the xlm-roberta model could have been more suitable for the danish language too, as the small difference obtained in the validation phase could have been outweighed by the generalizing power given by the large multilingual fine-tuning dataset. moreover, the performances of the multilingual models increased not only with the size of the finetuning dataset, but also with the number of languages it contains. the results also demonstrated that the potential of the multilingual transformer-based models in offensive language detection could be improved if larger datasets are available for non-english languages. for future work, we intend to consider a transfer learning method in order to leverage datasets that were constructed for similar tasks in the same language.