IR 3218- UI at S em E val-2020 Task 12: Emoji Effects on Offensive Language I dentifi C ation (2020)

Abstract: In this paper, we present our approach and the results of our participation in OffensEval 2020. There are three sub-tasks in OffensEval 2020 namely offensive language identification (sub-task A), automatic categorization of offense types (sub-task B), and offense target identification (sub-task C). We participated in sub-task A of English OffensEval 2020. Our approach emphasizes on how the emoji affects offensive language identification. Our model used LSTM combined with GloVe pre-trained word vectors to identify offensive language on social media. The best model obtained macro F1-score of 0.88428.

## introduction offensive language has become severe problems with the rapid growth of people using social media platforms such as twitter. on social media, users can easily express their opinion about any topic which leads to some users posting offensive content while engaging in social media. the offensive contents sometimes are used to target another user, whether individual or particular groups based on the user's views. an automatic method of identifying offensive language is needed to prevent the spread of offensive content on social media. @xcite organized the offenseval 2020: multilingual offensive language identification in social media as task 12 on semeval-2020. offenseval 2020 covers multilingual language consists of arabic, danish, english, greek, and turkish. the main focus of offenseval 2020 is to solve problems on how to identify the offensive language (sub-task a), offense types categorization (sub-task b), and offense target identification (sub-task c). sub-task a ran for all languages while sub-task b and sub-task c only ran for english. our approach used a neural network model based on long short-term memory (lstm) in order to identify offensive language on english tweets. this paper mainly describes our submission for offenseval-2020 sub-task a, but our model can be further developed to learn other sub-tasks. this paper is organized as follows. related work has been discussed in section 2, dataset and methodology have been described in section 3. the experiment results and discussions described in section 4. section 5 concludes the research and the future works of the research. ## related work many research on offensive language identification has been done. @xcite studied the potential of transfer learning in automatic offensive language detection. they employ many transfer learning scenarios to train their neural network. the results proved that transfer learning improves offensive language detection performance. @xcite used an ensemble model based on logistic regression and tree-based model to identify offensive language on semeval-2019 task 6. char n-grams, word n-grams, part of speech and glove embedding were used as features. by combining 5 different models with the varying feature set as input, an ensemble model was built. using the ensemble model, the offensive language identification obtained an accuracy of 0.80 and macro f1 score of 0.74. recently, deep learning approach was used in many text classification problems. @xcite compared several classification models such as k-nearest neighbors (knn), convolutional neural network (cnn) and lstm for patent text classification. their results showed that lstm outperformed other models with 93.48% accuracy. lstm was also used by @xcite for political perspective identification in online news articles. they experimented using various hyper-parameter setting for the lstm network such as word embedding source, memory size, word cutoff and batch size. their best result obtained area under the roc curve (auc) score of 0.966. @xcite conducted a study about emoji usage in social media and its impact on sentiment analysis. their study reveals that emoji have a significant effect on improving the sentiment scores both on the positive and negative opinions. another emoji-related study was done by @xcite . this study compared two emoji-handling strategies, the first is to use separate embedding for emojis, and the other is to use textual replacement for emojis combine with word embedding. emoji2vec @xcite is used as the emoji embedding, and the word embedding used is provided by @xcite using word2vec trained on 550 million tweets. the results showed that using textual replacement for emoji produced better performance than using separate embedding for emojis. based on the previous description, in this study, we want to identify offensive language on twitter using lstm as the model classification. lstm was used as the model classification since it showed good performance in text classification task based on the study done by @xcite and @xcite . based on the study of @xcite , we want to implement the same strategy in order to study the effect of emojis on offensive language identification. the difference is that the word embedding used for our study is provided by glove @xcite , which provide pre-trained twitter corpus more than the one provided by @xcite . ## experiments results and discussion this research was done to explore the effect of emoji on offensive language identification. to do that, we proposed two preprocessing scenarios for offensive language identification. in general, each scenario follows the following processes: emoji handling, punctuation removal, case folding, stopwords removal, and word stemming. the difference is that in the first scenario, the emoji will be handled by removing them from the data while in the second scenario, the emoji will be handled by replacing the with phrases that represent the emoji. we used macro f1-score to evaluate the performance of our results as it was the standard evaluation score requested by offenseval 2020. we also evaluate the results in precision and recall to understand the results better. the evaluation metrics were computed using scikit-learn foot_3 python library @xcite . offenseval 2020 only announce the f1-score and ranking of the participants' last submission for the competition. however, the gold labels for the testing data have been provided after the competition ends. we could analyze our approach based on this provide gold labels. we used the confusion matrix to explore the distribution of the classification for each class. the confusion matrix is shown in based on the results, our approach shows good performance by successfully classifying almost all offensive tweets correctly, with only two misclassifications in scenario 1 and one misclassification in scenario 2. however, our method is too sensitive towards offensive language. this is indicated by the many misclassifications of not-offensive tweets which are considered as offensive tweets. in order to analyze the effect of emoji on offensive language identification, we count the emoji occurrences in the dataset. the emojis were ranked for its appearance in each class. the top-10 ranked emoji in the testing dataset is shown in figure 4 . the top-10 emoji in not-offensive class consists of positive emoji such as heart-shaped emoji, hand-gestures emoji, and smiling-face emoji while for the offensive class top-10 emoji consists of negative emoji such as angry-face emoji, and emoji with negative interpretation such as sweat droplets, tongue, and zombie. the various emojis used in each class can help the classifier distinguish between the two classes. ## conclusion offensive language identification is crucial in nowadays era. almost in every social media platform, offensive language exists and disturbing its users. offenseval 2020 is organized to explore and improve on offensive language identification. this paper presents our system description on participating at offenseval 2020/ semeval-2020 task 12 sub-task a. we studied the effect of emoji in offensive language identification using lstm and glove embedding. we can conclude that replacing emoji into phrase improve the model performance. the improvement is indicated by an increase in the f1-score between scenario 1 (emoji removal) and scenario 2 (emoji replacement). scenario 1 obtained the f1-score value of 0.84206 while scenario 2 obtained a better f1-score, with a value of 0.88428. this improvement occurs because the dataset has more features to analyze from the emoji replacement phrase where each emoji has a tendency towards certain classes. however, our models were too sensitive with offensive language. this is proven by the number of misclassification obtained for both scenarios. out of 2807 not-offensive tweets, 556 tweets in scenario 1 and 396 tweets in scenario 2 were misclassified as offensive tweets by our models. for future work, we want to explore the dataset selection method to improve our results. the dataset used in this research showed that the model was too sensitive in detecting offensive language. in addition, as offenseval 2020 provides multilingual offensive language dataset, we plan to study how to identify offensive language on a multilingual dataset.