NLP _ P assau at S em E val-2020 Task 12: Multilingual Neural Network for Offensive Language Detection in E nglish, D anish and T urkish (2020)

Abstract: This paper describes a neural network (NN) model that was used for participating in the OffensEval, Task 12 of the SemEval 2020 workshop. The aim of this task is to identify offensive speech in social media, particularly in tweets. The model we used, C-BiGRU, is composed of a Convolutional Neural Network (CNN) along with a bidirectional Recurrent Neural Network (RNN). A multidimensional numerical representation (embedding) for each of the words in the tweets that were used by the model were determined using fastText. This allowed for using a dataset of labeled tweets to train the model on detecting combinations of words that may convey an offensive meaning. This model was used in the sub-task A of the English, Turkish and Danish competitions of the workshop, achieving F1 scores of 90.88%, 76.76% and 76.70%, respectively.

## introduction with around 3.08 billion users in 2020, the number of social media users is continuing to grow. one of the most important features of social media is the presence of user generated content such as photos, videos or texts @xcite . the reliance on user generated content, however, presents a significant challenge in moderating such content which has the potential to instigate violence within communities in real life, if it contains hate-speech @xcite . there has been growing interest from social media companies, fueled by the pressure from the government agencies, to detect and try to block such harmful content as in some cases it could lead the platform on which the content was distributed into paying significant fines (deutsche-welle, 2019). furthermore, using computer algorithms to automatically detect any content that could contain hateful inclinations is a challenging task that has also attracted the attention of the scientific community as can be seen by various workshops, competitions and conferences in recent years: trac @xcite , evalita hate-speech detection task @xcite , germeval @xcite , @xcite and @xcite . the most prominent effort in this regard has been the offenseval challenge @xcite , which sees its second edition this year as the 12th task of semeval, multilingual offensive language identification in social media, or offenseval 2 @xcite .the offensive language identification dataset or olid which consists of a collection of 14,200 english tweets with three levels of annotation for each of them is provided. the first level of annotation identifies whether a tweet contains offensive language or not, which is the aim of sub-task a. the second level determines the type of offensive language in the tweet which is the focus of sub-task b, and lastly sub-task c aims at determining the third annotation level which labels the intended target of the offensive speech in the tweet. in addition to english, four other languages (arabic, danish, greek and turkish) have datasets for sub-task a. in this paper, we describe the system that was used for our submissions to sub-task a for 3 languages (danish, english and turkish). in the second section, we provide a literature overview of the various systems that were used for detecting offensive language in text. in the third section, a description of the pre-processing techniques, the experimental setup and tools that we used is given. the fourth section gives an overview of the word embeddings and the nn model that we used. the fifth section shows our results and their analysis. lastly, we conclude the paper and recommend possible enhancments of our system. ## background various systems have been created over the years to address the task of identifying the patterns of offensive or harmful language in text based data. a summary of various approaches to solving this problem is given by @xcite . the performance of a collection of models was additionally compared. a system designed by @xcite utilized convolutional filters of a single dimension for extracting features from the embeddings of characters to categorize text. the authors del @xcite developed an svm based system that integrated lexical features for categorizing text which they used to participate in offenseval 1. an end to end convulational neural network (cnn) with fine tuned fasttext embeddings was used by @xcite which performed better than linear regression systems and other nn models. @xcite designed an nn model termed "multiple choice cnn" which is a type of a convolutional nn. they used the model in conjunction with their own novel contextual embedding to participate in tasks 5 @xcite and task 6 of semeval 2019. @xcite used source driven representation to detect hate speech. offensive tweets in german were classified using a unified model consisting of a cnn and an lstm (dubbed c-lstm) by @xcite , while @xcite used an ensemble model consisting of a linear svm, a cnn, and a logistic regressor as a meta-classifier. we base our core model for participation in offenseval 2 on the model we have used to participate in offenseval 1. the difference between the previous system and the new one is in the implementation of different pre-processing techniques, and more importantly, the utilization of fasttext instead of word2vec for getting the embedding of words. the previous system, c-bigru @xcite has also shown its capability of reliably identifying offensive speech in german language text as illustrated by its participation in the germeval workshop @xcite , and now it has proved to be also reliable for danish and turkish as well. ## baseline model to evaluate the performance of our model, we built our first classifier by fine tuning a pre-trained bert-base uncased transformer @xcite . the bert model consists of 12 transformer blocks, 12 self-attention heads, and 768 hidden dimension with a total of 110m parameters. it was trained on the book corpus (800m words) and the english wikipedia (2,500m words). the model also includes a special classification embedding [cls] at the beginning of every sentence, and this token in the final layer was extracted as the aggregate sequence representation for the current classification task. then a linear layer of 768 dimensions was added on top of the model, using the [cls] embeddings of the whole input sequence to predict a binary label. bert tokenizes parts of words instead of tokenized words. ## system setup in this section, a description of the data that was used for training the system is provided, along with the pre-processing techniques that were utilized. ## system overview in this section, a description of the embedding utilized for the created tokens and of the c-bigru model is given. ## results and analysis for each of the three languages, after training the c-bigru model, its first assessment was done using the validation set which is 10% of the data of each language. for turkish, the model achieved an accuracy of 83.34%, a macro-average recall, precision and f1 score of 65%, 74% and 68% respectively. for danish the accuracy was 88.89%, the macro-average recall, precision and f1 score were 59%, 88% and 62% respectively. lastly, the english results were 77.87% accuracy and the macro-average of recall, precision and f1 score were all 75%. figure @xcite shows the resulting confusion matrices for each language. english danish turkish figure 2 : confusion matrices for the testing results of the 3 languages using c-bigru. ## conclusion and future work in this paper, the system that we used to participate in offenseval 2020 is shown and a detailed description is provided of the architecture of the c-bigru model and the pre-processing of the data that is utilized. the model achieves a macro-average f1 score of 90.882%, 76.76% and 76.70% for the english, turkish and danish languages respectively on labelling the offenseval 2 dataset.