SU - NLP at S em E val-2020 Task 12: Offensive Language I dentifi C ation in T urkish Tweets (2020)

Abstract: This paper summarizes our group’s efforts in the offensive language identification shared task, which is organized as part of the International Workshop on Semantic Evaluation (Sem-Eval2020). Our final submission system is an ensemble of three different models, (1) CNN-LSTM, (2) BiLSTM-Attention and (3) BERT. Word embeddings, which were pre-trained on tweets, are used while training the first two models. BERTurk, which is the first BERT model for Turkish, is also explored. Our final submitted approach ranked as the second best model in the Turkish sub-task.

## introduction with the growing popularity of social media all over the world, more and more people are using these platforms, and unfortunately, many people regularly face offensive language in these environments. such offensive language can have devastating effects on users depending on their ages and psychological conditions @xcite . in the current big data era, manual filtering of such content is not possible; therefore, development of effective automated solutions is a necessity. this important task of identifying the offensive language has been supported by semeval organizers since last year, as part of the offenseval task @xcite . last year's task focused only on english and this year on its second run, the organizers aimed for a multilingual task and run the challenge for five languages, arabic, danish, english, greek and turkish @xcite . this year we specifically focused on turkish, in which a high-performance automated system is very much needed, due to the widespread use of social media by turkish-speaking people of all ages. within the framework of this problem, we initially analyzed different state-of-the-art neural network models and different word embeddings. our final system consists of a weighted ensemble model combining the following 3 models; convolutional neural network (cnn) with long short term memory (lstm), bidirectional long short term memory (bilstm) with attention and the recently released turkish bert model. additionally, word embeddings which are trained over turkish tweets are explored for better domain and task adaptation, and for this particular task they provide much better performance compared to other available pre-trained embeddings. with our proposed system we achieved the 2 nd place in turkish sub-task. our code is available online 1 . ## task and data description offenseval task consists of three subtasks. this year for the four new languages, the organizers organize only the first sub-task a, which is the offensive language identification task. the goal of this sub-task is to predict whether a given content is offensive or not. in the task, a post is described as offensive if it includes insults, threats, profane language or swear words @xcite . the data collection created by c ¸öltekin (2020), is used for offenseval turkish task. the collection consists of 34805 turkish tweets which were annotated with the following labels: this work is licensed under a creative commons attribution 4.0 international license. license details: http:// creativecommons.org/licenses/by/4.0/ . ## https://github.com/su-nlp/multilingual-offensive-language-identification-in-social-media this data collection is divided into two splits, as one is used for training and the other for test. the distribution of the labels after the split is presented in ## experimental setup initially, common and twitter specific pre-processing steps are applied to the data. these steps include converting all text to lowercase, removing punctuations, removing symbols like hashtags and twitter specific tokens like @user. after this preprocessing, tweets are tokenized by the nltk tokenizer. during the development phase, we are only provided with the training data. therefore, in order to evaluate the performance of the proposed approaches, we split the training data into two. 80% of the data is used for training and 20% for validation. the provided training data collection is imbalanced as the size of the not-offensive tweets is around four times the size of offensive tweets. machine learning algorithms may perform poorly with such imbalanced data collections. in order to handle this problem, we resample the training data by downsampling it. random samples are chosen from the not-offensive observations. the number of non-offensive tweets in the training split is reduced by 40%. the average f1-score from different models, is used to choose this percentage value. size of the validation set and training set before and after this downsampling is shown in ## system overview different neural network architectures are explored for this classification problem. additionally, several pre-trained word embeddings which are trained on different collections, are used with these networks. finally, the recently released pre-trained bert model for turkish is applied. the following neural network architectures are explored for the task. for all these models, all layers except for the embedding layer is trained from scratch. pre-trained word embeddings are used for these models and the embedding layer is freezed during training in order to prevent possible overfitting. word embeddings: several pre-trained word embeddings are available for turkish. in this particular task, publicly available word2vec @xcite ) embeddings foot_0 trained on 4.76 million turkish news articles (erdinc ¸and güran, 2019) are used for the experiments. erdinc ¸and güran (2019) trained several models with different pre-processing steps and compared their performances over various text classification tasks. in this work, we use the one trained after removing punctuation and stopwords, which is named as nda. in their reported experiments, this particular embedding outperformed the others. embeddings trained on news articles is definitely useful for many classification tasks. however, in this particular task, the effects of such an embedding can be limited. tweets are very different than news articles in terms of their writing styles and the used vocabulary. furthermore, news articles do not contain profane language or swear words which are very important signals for identifying offensive language. due to these reasons, an additional word embedding model is trained over tweets only. more than 24 million tweets which were collected between june 2019 and february 2020 are used to train word embeddings. continuous bag-of-words method @xcite ) is used to train 300 dimensional vectors. during the training, a window of 10 and minimum count of 5 is used as the hyperparameters. bert: in addition to the above architectures which were trained from scratch, pre-trained models are also explored to get a head start on the task. a pre-trained bert model is fine-tuned for this task. bert @xcite , which is a transformer @xcite based model, is capable of capturing the bidirectional representations of texts by jointly conditioning on both the left and the right context. we use the publicly available berturk-base cased model foot_1 which was trained over a combination of turkish data collections like oscar corpus foot_2 , wikipedia and opus foot_3 corpora. berturk model, which consists of 12 transformer layers, is fine-tuned for offensive language identification task by using the provided training data. ## experiments the described cnn-lstm and bilstm-attention models are trained from scratch by using either the word embeddings trained on news articles or our own embeddings trained on tweets. additionally, berturk model is fine-tuned for the particular task. in order to analyze the effects of downsampling, all models are trained twice, one with the downsampled data and another one with the whole data. later on, these models are applied to the validation data split and the results are presented in in even though using word embeddings trained on data originated from twitter seems to be useful, berturk's results suggest that the architecture and how much information is being learnt and transferred, are still very much important. berturk was not trained on tweets but due to its better modeling of the context, it outperforms all other approaches. according to ## conclusion our proposed ensemble model of berturk, cnn-lstm and bilstm-attention provided to be effective in the identification of offensive language in turkish tweets. it has been observed that, word embeddings trained on tweets has positive effect on the overall performance of the prediction model. based on this observation, a future work will be to pre-train a bert model over large set of tweets and then fine-tune it for this task.