ALT at S em E val-2020 Task 12: A rabic and E nglish Offensive Language Identification in Social Media (2020)

Abstract: This paper describes the systems submitted by the Arabic Language Technology group (ALT) at SemEval-2020 Task 12: Multilingual Offensive Language Identification in Social Media. We focus on sub-task A (Offensive Language Identification) for two languages: Arabic and English. Our efforts for both languages achieved more than 90% macro-averaged F1-score on the official test set. For Arabic, the best results were obtained by a system combination of Support Vector Machine, Deep Neural Network, and fine-tuned Bidirectional Encoder Representations from Transformers (BERT). For English, the best results were obtained by fine-tuning BERT.

## introduction social media has given a platform to users to express their thoughts and beliefs. then, it is not surprising that in the sea of content generated by users, we often find offensive language that can contain hatred, racism or profanity. such content may be inappropriate for others and need to be identified so that they can be regulated. at semeval-2020, task 12 @xcite , we are presented with a multilingual dataset featuring 5 different languages: arabic, danish, english, greek, and turkish. we focus on sub-task a for arabic and english. the goal of this sub-task is offensive language identification in tweets. for arabic track, we experimented intensively with support vector machines(svms), deep neural networks (dnns) and multilingual bidirectional encoder representations from transformers (multilingual-bert). our best results were obtained by a system combination based on majority voting by the aforementioned models. the system combination ranked 2nd (out of 53 teams) in the official rankings, with macro-averaged f1-score of 90.16%, behind by just 0.01% from the top ranked submission (90.17%). for english track, we again experimented with svms, dnns and multilingual-bert. relatively low performance by svms and dnns with the given training data prompted us to explore external datasets. the best results were obtained by fine-tuning multilingual-bert on combination of the given training data and wikipedia detox @xcite . the system achieved macro-averaged f1-score of 91.14%, which is ranked 18 (out of 82 teams), trailing from the top ranked submission (92.04%) by less than 1%. alongside description of our approaches, we also perform error analysis of our best systems. we manually identify different categories (e.g., humor/sarcasm) where our systems made errors. this provides insight on how the data and the systems can be improved in future. ## related work being representative of diverse linguistic phenomena among a great number of users, arabic twitter has become a major data source for researchers @xcite . along with popular social media analysis tasks such as sentiment analysis @xcite or dialect identification @xcite , offensive language identification for arabic has gained notable interest among researchers. @xcite present a dataset of abusive tweets and deleted comments from aljazeera. @xcite use support vector machine (svm) and deep learning models for offensive language identification in tweets. @xcite compare performances of svms and fasttext @xcite . @xcite propose recurrent neural networks trained with pre-trained embeddings to detect religious hate-speech in arabic social media. @xcite propose use svms to detect offensive language in youtube comments. compared to arabic, offensive language identification in english has attracted significantly more attention among researchers. task 6 at semeval-2019 @xcite received submissions from 115 teams. the top ranking team @xcite for offensive language detection (sub-task a) fine-tuned bidirectional encoder representation from transformer (bert) @xcite for their submission. some of other notable work on offensive language detection are by @xcite and @xcite . @xcite use logistic regression classifier with l2 regularization to detect hate speech and offensive language. @xcite use distributed representation (le and mikolov, 2014) of online comments, which is then used as features for a logistic regression classifier to detect offensive language. the dataset for arabic @xcite ) contains 8,000 tweets for training. the tweets are labeled as offensive (off) or not offensive (not). no separate development data is provided. the blind test set contains 2,000 tweets, made available via codalab foot_1 . the task is to predict labels (off or not) for the tweets. the gold labels for the test set are made available after the evaluation phase. the dataset for english @xcite ) contains ∼9m tweets for training. for each tweet, participants are provided with the following: i) average of the confidences scores by several supervised models for each tweet to belong to the offensive class, and ii) standard deviation from the average confidence score for each tweet. similar to arabic, no development data is provided. the blind test set consists of 3,887 tweets (without any other information). the task is to detect whether these instances are offensive (off) or not offensive (not). the gold labels for these instances (manually verified) are made available after the evaluation phase. ## arabic offensive language identification in this section, we describe our preprocessing steps, experiments, results and error analysis for arabic. ## english offensive language detection unlike arabic, english does not have diacritics or different forms of characters. thus, we do not perform the preprocessing from section 4 and discuss only experiments, results and error analysis. ## conclusion in this paper we presented highly effective systems for offensive language identification in arabic and english tweets. our best result for arabic was achieved by a system combination of svm, dnn and multilingual bert. for english, on the other hand, multilingual-bert outperformed other classifiers when it was trained on additional external data. the scarcity of resource for arabic makes such approach difficult, suggesting that there is a need for data augmentation in the future to obtain better results. with our error analysis, we identified several issues (e.g., misclassifications due to humor/sarcasm) that can be addressed in the future to improve performance.