Sonal.kumari at S em E val-2020 Task 12: Social Media Multilingual Offensive Text Identification and Categorization Using Neural Network Models (2020)

Abstract: In this paper, we present our approaches and results for SemEval-2020 Task 12, Multilingual Offensive Language Identification in Social Media (OffensEval 2020). The OffensEval 2020 had three subtasks: A) Identifying the tweets to be offensive (OFF) or non-offensive (NOT) for Arabic, Danish, English, Greek, and Turkish languages, B) Detecting if the offensive tweet is targeted (TIN) or untargeted (UNT) for the English language, and C) Categorizing the offensive targeted tweets into three classes, namely: individual (IND), Group (GRP), or Other (OTH) for the English language. We participate in all the subtasks A, B, and C. In our solution, first we use the pre-trained BERT model for all subtasks, A, B, and C and then we apply the BiLSTM model with attention mechanism (Attn-BiLSTM) for the same. Our result demonstrates that the pre-trained model is not giving good results for all types of languages and is compute and memory intensive whereas the Attn-BiLSTM model is fast and gives good accuracy with fewer resources. The Attn-BiLSTM model is giving better accuracy for Arabic and Greek where the pre-trained model is not able to capture the complete context of these languages due to lower vocab-size.

## introduction the usage of offensive language in social media is very common nowadays. sometimes it is used for fun and entertainment purposes, but sometimes it is an expression of user's aggression, hate, and/or offensive behavior. identification of user's intention with an offensive content on social media requires domain-specific sentiment analysis at fined-grained level of granularity. to control the offensive behavior of the user's post on social media requires the categorization of hate speech problems into new notions like abusive, aggressive, and/or offensive hate speech. such categorization helps law-enforcement agencies for the surveillance of social media. identification of hate, aggression, or offensive speech in user-generated content has attracted significant attention in the sentiment analysis research area recently. as evidenced, in recent publications @xcite @xcite ) and workshops such as awl and trac and competitions such as hateval 2019 @xcite , hasoc 2019, and offenseval 2019 @xcite . the exponential rise of users on social media and their posts on social media led to an enormous amount of data generation. detection of hate, offensive speech, aggression, or abusive content on social media requires the involvement of algorithms to make decisions based on deep learning models such as lstm, bidirectional lstm, pre-trained bert (bidirectional encoder representations from transformers), cnn, or stacked cnn, but not limited. the offenseval-2020 has been introduced as a multilingual offensive language identification by @xcite in which following 3 subtasks were given: (a) offensive language identification for arabic, danish, english, greek and turkish, (b) automatic categorization of offensive type for english only, and (c) offensive target identification for english only. we participate in all the subtasks a, b, and c. our approach is based on a pre-trained bert model @xcite and attention aware bilstm model (att-bilstm) @xcite . first, we clean the given tweet data and fine-tune the pre-trained bert model for all the subtasks, a, b, and c. next, we apply the bilstm model for the same. we observe that the bilstm model accuracy is better compare to the bert model arabic-a and greek-a, and also, the bert model is compute-intensive and memory-hungry compare to the bilstm model. the rest of this paper is organized as follows. in section 2, related work has been presented. section 3 and section 4 give the methodology and experimental analysis. finally, section 5 concludes the work. ## related work recently many research works @xcite @xcite @xcite @xcite have been done for artificial intelligence-based offensive language detection in english text. these works can be classified into following four categories: 1) convolutional neural network (cnn) based @xcite , 2) recurrent neural network (rnn) based @xcite , 3) combination of cnn & rnn @xcite , and 4) other approaches @xcite @xcite . @xcite applied a pre-trained bert model @xcite to fine-tune the binary offensive language detection task and demonstrated outstanding accuracy. @xcite demonstrated that a bidirectional gated recurrent unit network applied to word-level features generated from hate speech detection data having four categories of class-label outperforms. @xcite evaluated three cnn based and five rnn based deep learning models on 13 review datasets for both word-level and character-level input structures. they demonstrated that bilstm based model with word-level performs best across various benchmarked review datasets of the english language. for multilingual abusive language detection, less work has been done which focuses hindi, & english in @xcite , english, italian, & german in (corazza et al., 2020), danish & english in @xcite . some research has been done targeting specific foreign languages such as arabic @xcite , @xcite , german @xcite , greek @xcite , italian @xcite , spanish @xcite , etc. despite lots of research done for abusive language identification, most of these works focused on english language and the rest of the languages are still not much explored in this field. moreover, the rnn model is a default choice for most of the text analytics-related applications as it outperforms over rest but its recurrent structure creates hurdle in learning long-term dependency because of vanishing or exploding gradient problem. @xcite introduced lstm to avoid vanishing gradient problem and learn long-term dependency in the sequence model by inserting a gate unit. bilstm with attention @xcite has been proposed for relation classification which has been shown to outperform svm, cnn, rnn, and lstm models. from the literature, it is clear that the bert model and the bilstm based model outperform the rest of the ai models for sentiment analysis @xcite @xcite . ## methodology given data sets for sub-task a, b, and c, contain user's tweet messages where graphical and non-graphical emoji are used and have grammatical errors. first, we perform text pre-processing to clean and normalize the text messages. in the pre-processing phase for all data sets, emoji are replaced with equivalent english phrases, hashtags (single words) are separated into multiple words using heuristic techniques, all texts are lower-cased, and duplicate words, numerical digits, & punctuations are removed. for text normalization, we remove the repeated words from all the datasets. we delete the repeating consecutive characters, appearing more than two times, from the english dataset. for example, the words "hhhhhaaaaaaattttteeeee", "lmaoooooo", "wholesomeeeeee", "heeeeeelllllpppppp", and "looool", are substituted with the words "hate", "lmao", "wholesome", "help", and "lol", respectively. we also delete single character words from danish, english, and turkish datasets. arabic text is normalized by removing diacritics, replacing the letters ‫,'آ'‬ ‫'إ'‬ &, ‫'أ'‬ with the letter ‫,'ا'‬ and replacing the letters ‫,'ى'‬ ‫,'ؤ'‬ ‫,'ئ'‬ ‫,'ة'‬ ‫'گ'‬ with the letters ‫,'ي'‬ ‫,'ء'‬ ‫,'ء'‬ ‫,'ه'‬ ‫'ك'‬ respectively. for greek text normalization, letters 'ά', 'έ', 'ή', 'ό', 'ώ', are substituted with letters 'α', 'ε', 'η', 'ο', 'ω' respectively, the letters 'ί', 'ΐ', & 'ϊ' are replaced with the letter 'ι', and the letters 'ύ', 'ΰ', & 'ϋ' are replaced with the letter 'υ'. for turkish text normalization, letters 'î', 'â', 'ö', 'ü', 'ğ', 'ç', and 'ş' are converted into letters 'i', 'a', 'o', 'u', 'g', 'c', and 's', respectively. in the next step, we statistically analyze the given data sets and observe the presence of class imbalance in the given datasets. also, the selection of an optimal threshold for mapping average confidence scores is not straight forward because of the high deviation in confidence scores obtained by different supervised models. to handle this, at the first level, we filter out the samples having a very high standard deviation (>=0.3) and then at the second level, assign penalty on confidence score based on standard deviation. we choose, two model architectures for classification tasks for the given data sets: pre-trained bert model @xcite and bilstm with attention (att-bilstm) @xcite . besides, we also train a unified model that uses bilstm with attention which is explained in subsection 3.3. ## data specification and experimental analysis in this section, we present the data description and the experimental analysis for semeval-2020 task 12. we first report the submitted accuracy and then enhanced accuracy. we perform text normalization for foreign languages which results in improved accuracy. also, we filter all the digits from the tweet messages as they are not contributing to offense analysis. we also combine all language data of subtask-a to train a single unified bilstm model (described in subsection 3.3) for the multilingual environment. we report the accuracy in terms of macro-f1-score which gives equal weightage to all the classes and thus is a reliable metric in the presence of class-imbalance in the data. ## conclusion we used the bert and the bilstm model for multilingual offensive language identification and compared the results. we demonstrated in the result analysis that our attention-aware bilstm model outperforms the bert model for arabic and greek offensive language detection, while bert performed better for danish, english, and turkish offensive language identification, english offense-type categorization, and english offense target identification. further, we combined arabic, danish, english, greek, turkish multilingual offensive language identification data to train our uml_bilstm model. the obtained uml_bilstm model results were poor by -0.20 to 11.29% and 1.95% to 12.37% compared to the bert and the bilstm, respectively. however, the uml_bilstm model training was 2.81 times and 1.34 times faster in comparison to the bert and the bilstm models, respectively. uml_bilstm model can be further improved by introducing a data balancing mechanism before training the model. our submitted f1-scores are 0.4536, 0.6710, 0.8900, 0.8020, 0.7421, and 0.5259 for arabic subtask-a, danish subtask-a, english subtask-a, greek subtask-a, turkish subtask-a, and english subtask-c, respectively. after submission, we further worked on enhancing the models for achieving better accuracy. we normalized the data for foreign languages before feeding to the bilstm model and got improved results. especially for the greek language, our achieved accuracy was significant compare to the of-fenseval-200 top ranker f1-score. the top ranker accuracy was 0.8520 and our improved accuracy was 0.8984. therefore, it has been proven that the normalization of social media posts is one of the important steps before applying model. the improved f1-scores are 0.8447, 0.7672, 0.9060, 0.8984, 0.7489, 0.6711, and 0.6054 for arabic subtask-a, danish subtask-a, english subtask-a, greek subtask-a, turkish subtask-a, english subtask-b and english subtask-c, respectively. in the future, we also plan to evaluate our model on a wide range of languages.