problem C onquero at S em E val-2020 Task 12: Transformer and Soft Label-based Approaches (2020)

Abstract: In this paper, we present various systems submitted by our team problemConquero for SemEval-2020 Shared Task 12 “Multilingual Offensive Language Identification in Social Media”. We participated in all the three sub-tasks of OffensEval-2020, and our final submissions during the evaluation phase included transformer-based approaches and a soft label-based approach. BERT based fine-tuned models were submitted for each language of sub-task A (offensive tweet identification). RoBERTa based fine-tuned model for sub-task B (automatic categorization of offense types) was submitted. We submitted two models for sub-task C (offense target identification), one using soft labels and the other using BERT based fine-tuned model. Our ranks for sub-task A were Greek-19 out of 37, Turkish-22 out of 46, Danish-26 out of 39, Arabic-39 out of 53, and English-20 out of 85. We achieved a rank of 28 out of 43 for sub-task B. Our best rank for sub-task C was 20 out of 39 using BERT based fine-tuned model.

## introduction there has been a rise in the use of offensive language on social media platforms. this discourages effective communication, which comes as a drawback against the goal of social media platforms. hence a need to address the problem of offensive language identification arises. there has been an increasing awareness in communities to filter and moderate tweets. as huge number of users are online, it is difficult to manually moderate each and every content; hence the need for automation arises. as people from different regions, ethnicity interact with each other on online platforms, the detection of multilingual offensive language has become increasingly important. during the training phase of the competition, we started with various machine learning models and proceeded with the model in the evaluation phase, which gave us the highest macro f1 score on the development set. in this paper, we first describe the existing work that has been done for offensive language detection, our proposed approaches, and the results that we obtained. the implementation for our system is made available via github 1 . ## problem defintion offenseval-2020 @xcite ) consisted of 3 sub-tasks. ## related work offensive language detection has been studied at various levels of granularity in the form of abusive language detection @xcite , hate speech detection @xcite and cyberbullying @xcite ). api's like perspective foot_0 have been developed to detect content toxicity using machine learning models. various deep learning and ensemble methods have been proposed @xcite to detect abusive content online. sentiment based approaches (brassard-gourdeau and khoury, 2019) have also been used to detect toxicity. besides english, there have been various contributions to the detection of offensive content in other languages. for greek, @xcite describe an rnn based attention model trained on the gazetta dataset for user content moderation. sigurbergsson and derczynski (2019) discuss implementation of logistic regression, learned-bilstm, fast-bilstm and aux-fast-bilstm models for danish. for arabic, an approach based on convolution neural network and bidirectional lstm has been proposed @xcite . abozinadah and jones jr (2017) use a statistical approach for detecting abusive user accounts. alakrot et al. (2018) ## corpus description for offenseval 2020, datasets were provided by the competition organizers. for sub-task a: data was provided in the form of tweets and their corresponding labels (off and not) for arabic @xcite , danish (sigurbergsson and derczynski, 2020), turkish (c ¸öltekin, 2020), and greek @xcite languages. for the task on the english language @xcite , for all the three sub-tasks a, b, and c, the data was provided in the form of tweets with their corresponding mean and standard deviation values. the mean and standard deviation scores were confidence measures obtained by semi-supervised learning methods. for sub-task a, the mean score was given to how offensive a tweet was, a higher value indicating a more offensive tweet. for sub-task b, the mean score was given to how close the tweet was to being an untargeted insult (unt) or how far the tweet was to being a targeted insult (tin). for sub-task c, a mean score was provided for all the labels "ind", "grp", and "oth" with a higher value indicating that the tweet belonged to the given category. multilingual datasets provided to us were unbalanced as follows: danish consisted of approximately 13% offensive tweets and 87 percent, not offensive tweets. similarly, arabic (20%: "off"), turkish (19%: "off"), and greek (28%: "off") training datasets showed a similar ratio imbalance between "off" and "not" labeled tweets. we also made use of the olid @xcite dataset provided in offenseval 2019 @xcite , which consisted of the english language tweets and their corresponding labels for sub-tasks a, b, and c conducted in offenseval 2019. we used this as a validation dataset for sub-task a and error analysis in sub-tasks b and c. during the post-evaluation phase of the competition the organizers also released gold labels, which were actual test labels corresponding to the dataset against which we were evaluated for in the competition. ## sub-task a 5.1 preprocessing we tried various preprocessing techniques but went with those that gave us the best f1 score on the development set (validation set). emoji replacement as a preprocessing technique, as suggested by @xcite in offenseval 2019, was used for greek, arabic, turkish in sub-task a. we replaced emojis with their corresponding meaning using the emoji library foot_1 @xcite . sentences were preprocessed before passing as input to the bert model. for greek, stop words and punctuation were removed by preprocessing with spacy foot_2 . for arabic and turkish consecutive duplicate words like "@user @user" were reduced to a single word "@user". ## sub-task b 6.1 preprocessing we gave data to the bert model with and without preprocessing, but as we were getting better results without preprocessing of text sentences, we went ahead with that. we converted the mean probability scores to hard labels "tin" and "unt" depending on whichever score was maximum. if the mean score was greater than or equal to 0.5, the hard label was "unt" else "tin" was assigned. after using the above method out of (188975) soft labeled tweets given for sub-task b, we got approximately 20% of the tweets classified as "unt" and 0.80% as "tin". we then classified the processed data with the fine-tuned roberta @xcite ) model. ## conclusion in this paper, we described the approaches that we used for sub-tasks a, b, and c. we used transformer based approaches for all sub-tasks and a soft label lstm based approach for sub-task c. we also described our future approaches based on bert embedding with bi-gru and bi-lstm. we are planning to use the data augmentation techniques and try to improve the f1 score on the gold labels for each sub-task. we made use of google colab foot_11 while developing our project, and the notebooks are made available at our github page.