S alam NET at S em E val-2020 Task 12: Deep Learning Approach for A rabic Offensive Language Detection (2020)

Abstract: This paper describes SalamNET, an Arabic offensive language detection system that has been submitted to SemEval 2020 shared task 12: Multilingual Offensive Language Identification in Social Media. Our approach focuses on applying multiple deep learning models and conducting in depth error analysis of results to provide system implications for future development considerations. To pursue our goal, a Recurrent Neural Network (RNN), a Gated Recurrent Unit (GRU), and Long-Short Term Memory (LSTM) models with different design architectures have been developed and evaluated. The SalamNET, a Bi-directional Gated Recurrent Unit (Bi-GRU) based model, reports a macro-F1 score of 0.83%

## introduction online offensive language is a critical problem that is threatening the safety and well-being of society. previous studies demonstrate the connection between different forms of online offensive language, such as hate speech, cyberbullying, and real-world violence @xcite . the attack on the tree of life synagogue on october 2018 in pittsburgh, which was described as the deadliest attack in the jewish community in the us, shows the severe effects of online hate speech as the shooter had an online profile full of hate speech language @xcite . given the significance of the impact online offensive language can have, further investigation of the problem is warranted. we hypothesize that developing a natural language processing (nlp) approach with language specific features can help in detecting offensive language to stop these attacks. while there are several studies on english offensive language detection, very few studies are available for the arabic language. the arabic language has specific language characteristics and features that require distinct system development @xcite . the main goal of this paper is to develop a system for online arabic offensive language detection. we participated in semeval 2020 shared task 12: multilingual offensive language identification in social media with two submissions, a logistic regression (lr) based model and a recurrent neural network (rnn) based model. we evaluate the performance of both models through a detailed error analysis to enhance the performance until we arrive to the salamnet system for online arabic offensive language detection from twitter. we explore multiple deep learning models, including an rnn, a gated recurrent unit (gru), a bi-directional gated recurrent unit (bi-gru), long-short term memory (lstm), and a bi-directional long-short term memory (bi-lstm) models separately. based on our findings, the highest macro-f1 score among all models is 0.83, which was reported by the salamnet that consists of a bi-gru based model with term frequency-inverse document frequency (tf-idf) features. ## related work during recent years, the topic of offensive language detection has become an attractive topic for arabic nlp researchers. various forms of arabic offensive language have been studied. this includes hate speech @xcite @xcite , cyberbullying @xcite @xcite , adult content @xcite @xcite , and the broader category of offensive and abusive online language @xcite @xcite . @xcite study religious hate speech in twitter data for the arabic language. they use character n-gram features and aravec word embeddings with multiple classifiers including lr, support vector machine (svm), and gru. the gru-based models acheived their highest f1 score of 0.77 using aravec. @xcite use the same dataset to investigate the effects of community interaction and social representations in detecting religious hate speech. they use a deep learning approach and study multiple features, such as word embedding, node embedding, sentence representation, and character n-gram features. they develop several deep learning models including gru, lstm, bi-lstm, convolutional neural network (cnn), and bi-gru. they also combine multiple models using self-attention and node2vec criteria. results show the highest accuracy score of 0.81, obtained by a combined model that consists of bi-gru and cnn using node2vec features. meanwhile, the best f1 score, recall, and precision were achieved by the combined model that included lstm and cnn using node2vec feature; 0.89, 0.78, and 0.86, respectively. @xcite explored multiple deep learning models to classify arabic offensive language with arabic youtube comments collected by @xcite . they used aravec word embedding with cnn, bi-lstm, bi-lstm with attention mechanism, and combined cnn and lstm models. their results demonstrated an overall better performance using a cnn with which they achieved an accuracy of 0.87, precision of 0.86, and f1 of 0.84. however, their combined cnn-lstm model showed the best recall score of 0.83 versus 0.82 with the cnn model. a more recent study by @xcite investigated arabic offensive language related to vulgar language and hate speech for twitter data using multiple classifiers and word embeddings. their system includes lr, svm, decision tree, random forest, gaussian naive bayes, perceptron, adaboost, and gradient boosting classifiers. in addition to applying fasttext @xcite , aravec, mazajak @xcite , and bert base multilingual @xcite for word embeddings. results showed the best f1 score of 0.79 for the svm-based model with mazajak word embeddings. ## methodology in general, the salamnet has a similar system pipeline to traditional classification systems. steps are sequential, starting with text preprocessing, followed by feature engineering, classification, and lastly system evaluation. during the classification model phase, the system employs multiple classifiers to investigate the differences in performance among the different models. ## results we use multiple performance evaluation metrics to accurately evaluate the behavior of our models, including precision, recall, macro f-measure, and weighted f1-measure. we apply 10-fold cross validation in evaluating the performance of the deep learning models. macro-f1 is the primary evaluation metric of the shared task. the total number of participants was 53 and the highest achieved macro-f1 score 0.901. our lr based baseline model with aravec achieved a macro-f1 of 0.81 and ranked 34th, and our bi-gru based model with tf-idf achieved a macro-f1 of 0.80 and ranked 38th. we explored more deep learning models to improve our results. the highest achieved macro-f1 among all classifiers; rnn, lstm, bi-lstm, gru, and bi-gru; is 0.83, which these results indicate that for the lr-based model, aravec word embeddings give better results than tf-idf features. however, for all the deep learning models tf-idf features give better performance than aravec word embeddings. ## discussion and error analysis to perform an error analysis we examined the predictions of the 2,000 tweets in the test dataset. from this, we retrieved the set of tweets that were misclassified by all deep learning models and explored their content. in total, 390 offensive tweets were labeled as not offensive, and 11 non-offensive tweets were labeled as offensive by all deep learning models. this finding is related to the class imbalanced distribution in the dataset, only 19% of the entire dataset represents offensive samples. the following are some sample of offensive tweets that were predicted as non-offensive by all deep learning models. in these examples, we show content before preprocessing for better representation: most of the misclassified samples were offensive tweets in the egyptian dialect, included the names of famous figures, and often related to sports (e.g. included team names, famous players, fans). even though our preprocessing phase is identical to husain (2020), which reports significant impacts on the performance of hate speech detection in the arabic language by converting hyponym to hypernym, specifically for animal names, during the preprocessing. we noticed some misclassified tweets with animal names. thus, further analysis related to how animal names are preprocessed should be considered in future works. further analysis showed that some tweets were misclassified by all aravec based models, but correctly classified by most tf-idf based models. for example, ‫كافة،"‬ ‫لليهود‬ ‫عميل‬ ‫قذر‬ ‫يا‬ ‫وأمثالك‬ ‫أنت‬ ‫خبيث‬ ‫يا‬ ‫جرم‬ ‫م‬ ‫يا‬ ‫دمائكم.‬ ‫في‬ ‫يستشري‬ ‫الاحرام‬ ‫خططاتهم.‬ ‫لم‬ ‫وأمثالك‬ ‫يستخدمك‬ ‫الذي‬ ‫بقطر‬ ‫الاجرام‬ ‫."وللنظام‬ another example is ‫حا"‬ ‫ج‬ ‫يا‬ ‫تفضل‬ ‫غبي‬ ‫هلالي‬ ‫جوازه‬ ‫على‬ ‫ختم‬ ‫ان‬ ‫كيف‬ ‫المملـكه‬ ‫دخول‬ ‫من‬ ‫ممنوع‬ ‫اذا‬ ‫جاهل‬ ‫يا‬ ‫بقردي‬ ‫يا‬ ‫المطار‬ ‫لأمن‬ ‫توجهه‬ ‫تروح‬ ‫هالسؤال‬ ‫طاقيه‬ ‫."يا‬ most of these tweets are relatively long in term of the number of words, and rarely include repetitions in words or letters. this indicates that tf-idf models may do a better job of modeling tweets that use a large vocabulary. similarly, we analyzed tweets that were classified correctly by all aravec based models but misclassified by most of tf-idf based models. some examples are:"‫يا‬ ‫يامتكبر‬ ‫جبار‬ ‫يا‬ ‫قيوم‬ ‫يا‬ ‫حي‬ ‫يا‬ ‫عزيز‬ ‫يا‬ ‫ياقوي‬ ‫يارب‬ ‫يضام‬ ‫لا‬ ‫الذي‬ ‫وملكك‬ ‫يرام‬ ‫لا‬ ‫الذي‬ ‫بعزك‬ ‫أسألك‬ ‫,"مهمين‬ and ‫معانا"‬ ‫يكبر‬ ‫وحـبنـا‬ ‫حـبايـب‬ ‫عـمري‬ ‫يا‬ ‫نـبـقى‬ ‫عـسانـى‬ ‫."يا‬ the first tweet consists of a prayer and it is written in the modern standard arabic (msa) form of the arabic language. the second example is written in gulf dialect. both are not offensive. the dataset contains multiple forms of the arabic language, and we expected that some forms of the arabic language would be underrepresented. most of twitter content is written in dialectal arabic meaning that msa is underrepresented, thereby negatively affecting classifier performance. although we considered this situation in our design by converting some dialectal nouns to msa during the preprocessing phase, this analysis shows that we may need to further improve our mechanism. in general, we found that each classifier behaves differently and complementary. the best performing classifiers are sometimes not able to detect the true class of a sample, for which other classifiers were able to. for instance, this tweet ‫لآخر"‬ ‫زملكاوي‬ ‫ابيض‬ ‫يا‬ ‫العمر‬ ‫تشرت‬ ‫يا‬ ‫اه‬ ‫زملكاوي‬ ‫بكاني‬ ‫والانتماءومهما‬ ‫حب‬ ‫ال‬ ‫وقت‬ ‫ده‬ ‫#معاكـياـزمالك‬ ‫الزمالك‬ ‫لرجالة‬ ‫الدعم‬ ‫كل‬ ‫عمري‬ ‫في‬ ‫يوم‬ " is classified correctly by all classifiers with all features except the bi-gru based model with tf-idf feature, which reports the highest macro-f1 score. thus, providing a mechanism to combine results from multiple models can improve the model. ## conclusion this paper presents the salamnet system for arabic offensive language detection. we explored multiple deep learning models; such as rnn, lstm, gru, bi-lstm, and bi-gru; using tf-idf feature and aravec word embeddings. our results show the best macro-f1 score of 0.83 by the bi-gru based classifier using the tf-idf feature, which represents the salamnet system. aravec word embeddings increased performance over tf-idf features when using traditional machine learning classifiers (such