KEIS @ JUST at S em E val-2020 Task 12: Identifying Multilingual Offensive Tweets Using Weighted Ensemble and Fine-Tuned BERT (2020)

Abstract: This research presents our team KEIS@JUST participation at SemEval-2020 Task 12 which represents shared task on multilingual offensive language. We participated in all the provided languages for all subtasks except sub-task-A for the English language. Two main approaches have been developed the first is performed to tackle both languages Arabic and English, a weighted ensemble consists of Bi-GRU and CNN followed by Gaussian noise and global pooling layer multiplied by weights to improve the overall performance. The second is performed for other languages, a transfer learning from BERT beside the recurrent neural networks such as Bi-LSTM and Bi-GRU followed by a global average pooling layer. Word embedding and contextual embedding have been used as features, moreover, data augmentation has been used only for the Arabic language.

## introduction natural language processing field has the researchers' attention especially with the rapid use of social media sites, for instance, twitter, facebook, youtube comments, and macro blogs. consequently, offensive, aggressive and hate-speech language identification problems that perform the automatic detection of these problems from textual data. moreover, the main motivation to reduce the behavior of hate speech and offensive/aggressive language on user attitude and content, in particular, on social media. the offensive detection in arabic social media is a serious task. this refers to the arabic language contains violent words represent both violent context and not a violent context besides arabic is dialects language @xcite . for instance, the word killing in arabic meaning represents a violent meaning and not a violent meaning in different contexts appears in @xcite tweets. the overall studies that detected on the offensive language have been applied to the english language. however, the research that regards in the arabic language in this domain of nlp applications has been restricted according to the lack of the resources that tackle the same issue compared to the english language @xcite . the researches that had been conducted in arabic evaluated on small datasets collected from twitter api such as @xcite has been evaluated their proposed approach on 1100 annotated tweets. in this research, we describe our participation team keis@just at semeval-2020 task 12 which describes the shared task on offensive language as a multilingual shared task (i.e. arabic, english, danish, greek, and turkish). moreover, we participated in all languages for the provided subtasks except sub-task-a for the english language. two approaches have been implemented aim to solve the shared task, the first is performed to tackle both languages arabic and english, a weighted ensemble consists of bi-gru and cnn followed by gaussian noise and global pooling layer multiplied by weights to improve the overall performance. consequently, the implemented approach performed to solve sub-task-a (offensive language identification), sub-task-b (automatic categorization), and sub-task-c (offense target identification). the second performed for other languages, a state of art transfer learning from bert embedding multi cased 12a pre-trained model besides the recurrent neural networks such as bi-lstm and bi-gru followed by a global average pooling layer. consequently, the implemented approach performed to solve sub-task-a (offensive language identification). word embedding and contextual embedding have been used as features, moreover, data augmentation has been used only for the arabic language and we rely on the aravec embedding @xcite for data augmentation which aims to create more dataset that helps to train the model. to evaluate our results offenseval 2020 @xcite ) provided multilingual dataset. the best results for the keis@just team ranked 11th place out of 56 teams with 86.55% f1-macro in the arabic language, ranked 12th place out of 39 teams with 76.1% f1-macro in the danish language, ranked 28th place out of 37 teams with 76.1% f1-macro in the greek language, ranked 32th place out of 46 teams with 73.3% f1-macro in the greek language. several of previous researches have been presented comprehensive studies which tend to describe the main key of the proposed task @xcite , and @xcite , moreover @xcite presents dataset for hate speech detection, @xcite presents dataset for aggressive language, and @xcite presents olid dataset for the previous shared task of offensive language. additionally, @xcite shows the earliest efforts in hate speech detection that performs a decision tree-based classifier. moreover, offensive identification for sentences have been tried for several languages behind the english such that, arabic @xcite and (al-hassan and al-dossari, 2019), german @xcite @xcite . there are lack of researches in the offensive language for arabic research community, for instance, @xcite introduced k-means for violence utterance on twitter. madimara has been used to extract morphological features as well as they used tf-idf to represent dataset on the vector space model. @xcite presented system to detect hate speech using lexical features and a linear svm classifier depending on n-grams. similarity, @xcite introduced svm classifier trained on word-level features. n-grams and stemming used as features. @xcite proposed l-hsab the first dataset for hate speech and abusive language. the dataset collected from twitter api focusing on syrian and lebanese tweets rich of toxic utterance. the dataset has been trained on naive bayes classifier. for english language, several reseacrchers used transformers ,for instance, @xcite proposed a fine-tuned technique for the bidirectional encoder representation from transformer (bert) with word unigrams, word2vec, and hatebase have been used as features. similarity, @xcite introduced a fine-tuned a bert based classifier depends on linear svm trained on character n-gram as a feature. @xcite proposed a fine-tuned a bert and lstm neural network architecture with automatically and manually crafted features were used namely: word embedding, tfidf, pos sequences, bow, the length of the longest punctuation sequence, and the sentiment of the tweets features. however, several researchers applied machine and deep learning, for instance, @xcite proposed an ensemble technique consist of convolutional neural network, bidirectional lstm with attention, and bidirectional lstm + bidirectional gru. @xcite presented two approaches namely: bidirectional with gru and probabilistic model modified sentence offensiveness calculation (msoc) trained using word2vec embedding. ## methodology shared task on multilingual offensive language identification in social media (offenseval 2020) @xcite , in this section, we will describe the shared task and the implemented system. ## results and findings in order to evaluate the implemented approaches, f1-macro has been used according to the shared task instruction. ## conclusion in this research, presented the keis@just participation at semeval-2020 task 12 which represents shared task on multilingual offensive language. we have participated in all the provided languages for all subtasks except sub-task-a for the english language. two main approaches have been developed the first one is performed to tackle both languages arabic and english, a weighted ensemble consists of bi-gru and cnn followed by gaussian noise and global pooling layer multiplied by weights to improve the overall performance. the second one performed for other languages, we investigated the main impact of developing a transfer learning approach from bert transformer beside the recurrent neural networks such as bi-lstm and bi-gru followed by the global average pooling layer for other languages. word embedding and contextual embedding have been used as features, moreover, we investigated how data augmentation affect the results using arabic dataset.