Hate Speech Detection in Saudi Twittersphere: A Deep Learning Approach (2020)

Abstract: With the rise of hate speech phenomena in Twittersphere, significant research efforts have been undertaken to provide automatic solutions for detecting hate speech, varying from simple ma-chine learning models to more complex deep neural network models. Despite that, research works investigating hate speech problem in Arabic are still limited. This paper, therefore, aims to investigate several neural network models based on Convolutional Neural Network (CNN) and Recurrent Neural Networks (RNN) to detect hate speech in Arabic tweets. It also evaluates the recent language representation model BERT on the task of Arabic hate speech detection. To conduct our experiments, we firstly built a new hate speech dataset that contains 9,316 annotated tweets. Then, we conducted a set of experiments on two datasets to evaluate four models: CNN, GRU, CNN+GRU and BERT. Our experimental results on our dataset and an out-domain dataset show that CNN model gives the best performance with an F1-score of 0.79 and AUROC of 0.89.

## introduction in the arab region, twitter is considered as one of the most popular platforms used by arabic speaking users and it has indeed revolutionized the way people communicate and share opinions, ideas and information. however, due to the dynamic, democratic and unrestricted nature of twitter platform, it has been increasingly exploited for the dissemination of aggressive and hateful content. while there is no formal definition of hate speech, there is a general agreement among scholars and service providers to define it as any language that attack a person or a group based on some characteristic such as race, color, ethnicity, gender, religion, or other characteristic @xcite . the pressing need for effective automatic solutions for hate speech detection has attracted significant research efforts recently. several studies relied on traditional machine learning approaches and leveraged surface features such as bag of words, word and character n-grams which have been shown to be very effective for hate speech detection @xcite @xcite . recently, there has been a clear trend towards the adoption of deep learning methods, which indeed showed better performance over classic methods in hate speech detection task @xcite @xcite @xcite . however, there is a very limited work that employed deep learning approaches to address the problem in arabic language @xcite . to the best of our knowledge, @xcite work is the only arabic study published in the area, and it examined gru architecture, focusing mainly on religious hate speech. while different deep learning models have been investigated in this area, we are more interested to explore these methods for arabic language. we believe that the special nature of arabic languages and its richness and complexity at morphological, lexical and orthographical levels @xcite pose some unique challenges that could complicate the task of detecting hate speech. moreover, the high variety of dialectal arabic used by arabic users on social media makes the problem even more complex. in fact, dialects of arabic are manifold, varying not only from country to country but also within the same country, resulting in many similarly spelled words to have different meanings across different dialects and regions. recently, @xcite introduced bert (bidirectional encoder representations from transformers.), a new language representation model that has been successfully applied to numerous nlp tasks, achieving the state-of-the-art results for eleven nlp tasks including sentiment analysis, question-answering and textual entailment. however, works examining bert effectiveness for hate speech detection, specifically in arabic are very limited. to that end, this paper presents a set of experiments to investigate the merit of using different neural networks including recurrent neural networks (rnns) and convolutional neural networks (cnns) and their variants for hate speech detection in arabic. we also evaluated bert on our downstream classification task by using the pre-trained bert model, adding a simple task-specific layer and then fine-tuning the entire model on the hate speech detection task. given the fact that arabic is considered as a low-resourced language, this paper also described our approach for creating a hate speech dataset for arabic, covering racist, religious and ideological hate speech. following this approach, we created a new dataset containing a total of 9,316 tweets labeled as normal, abusive or hateful as well as a new annotation guideline. the main contributions of this work are three-folds: 1) constructing a public dataset of 9,316 tweets labelled as hateful, abusive and normal. ## related work abusive language in social media is a complex phenomenon with a wide spectrum of overlapping forms and targets. hate speech, offensive language, and cyberbullying are examples of abusiveness and several works in the literatures have been conducted to detect and locate such types of languages. recently, researchers have shown an increased interest in automatic hate speech detection in social media. most of the existing studies in the literature have primarily modeled the problem as a supervised classification task whether using classical machine learning approaches or deep neural network approaches @xcite . as for classical machine learning approaches, there were several studies that leveraged simple surface and linguistic features such as bag of words (bow), n-grams, and pos as fundamental features. @xcite presented a review of the various features that were used for the task in the literature. one of the earliest studies that addressed the problem of hate speech detection was presented by @xcite , which focused on detecting anti-semitic language as the specific type of hate speech. @xcite proposed a supervised machine learning model to detect racist hate speech against black people in twittersphere and they suggested to improve the model by considering other features such as sentiment features and bigrams. @xcite presented a study to detect hate speech in twitter platform and specifically targeted two types of hate: racism and sexism. they also built and published a dataset of 16k annotated tweets which has become a wellknown benchmarking dataset for following studies. word embeddings as features were also investigated by @xcite and the experimental results on yahoo! finance user comments showed that paragraph2vec representation performed better in term of area under the curve (auc). following this study, @xcite incorporated various features such as n-grams, linguistic and syntactic features and distributional semantics to detect abusive language in online user comments. the results showed that combining all features led to outperform the prior art in term of auc. @xcite trained a multi-class classifier to distinguish between offensive and hate language in tweets. they also built and published a dataset of tweets labelled for three categories: hate speech, offensive language, or neither, resulted into 24,802 labelled tweets. in recent years, attention has shifted toward deep learning approaches to tackle the problem of hate speech detection. most of the works have employed various architectures of convolutional neural network (cnn) and recurrent neural networks (rnn). @xcite proposed a two-step classification approach by combining two classifiers: one to classify the text to abusive or not, and another to classify the text into a specific type of abusive language (sexist or racist) given that the text is abusive. @xcite proposed the use of cnn-classifier with word2vec word vectors and their results outperformed the prior art. @xcite built an ensemble of lstm-based classifiers to detect hate speech content in twitter. the results showed that the proposed approach achieved better results compared to the state-of-the-art approaches. @xcite showed that a cnn+gru neural network model has improved the performance of hate speech detection. @xcite investigated three different neural models for hate speech detection and the results showed that embeddings learned from lstm model when combined with gbdt led to the best accuracy result. recently, some works showed that incorporating bert-based models to learn the representations for noisy text @xcite and capturing hateful context @xcite can enhance the offensive and hate speech detection systems. there are several works that tackle the problem of hate speech and offensive language in non-english languages such as german @xcite , greek @xcite , danish @xcite , and turkish @xcite corpora. arabic studies that address the problem of hate speech detection are still limited, however, there are several studies that address some related problems such as detecting isis support messages @xcite and identifying vulgar, obscene, offensive or flaming language @xcite @xcite . a recent dataset for levantine hate speech and abusive detection has been published by @xcite . this dataset contains 5,846 tweets labeled as hate, abusive or normal, collected from twitter using terms of potential entities that are usually targeted by abusive/hate speech. @xcite proposed an rnn architecture with gated recurrent units (gru) and pre-trained word embeddings to detect religious hate speech in arabic tweets. for evaluation, they created a dataset containing 6,000 arabic tweets, 1,000 for each of the six religious' groups (muslims, jews, christians, atheists, sunnis and shia). the tweets were annotated as hate or not-hate tweet. the results showed that the gru-based rnn model outperformed other traditional classifiers with respect to all evaluation. they also published their dataset as the first and the only available hate speech dataset for arabic language. ## dataset construction in this work, we built a new dataset that covers different types of hate speech, e.g., racist, religious and ideological hate speech, focusing mainly on saudi twittersphere. we used the standard twitter search/streaming api with tweepy foot_0 python library to collect our data. the data was collected in span of 6 months, from march 2018 to august 2018 using a keyword-based and thread-based search. in keyword-based approach, we included in the search query a total of 164 impartial unique terms (excluding the various morphological variations of the same term) that refer to groups, or to people belonging to groups that are likely to be targeted by hate speech. for example, we used terms that refer to people who belong to different tribes (such as "otaibi"," ‫عتيبي‬ " ) and different regions ("hijaz"," ‫حجازي‬ ") to collect tweets related to tribal and regional hate speech, respectively. for the thread-based search, we included in the search query different hashtag that discusses controversial topics which are considered to be strong indicators of hate speech. we monitored twitter trends during the period of the data collection, and we ended up with a total of 10 hashtags used for data retrieving. examples of the used terms and hashtags are illustrated in in this section, we present the main components of our proposed approach to detect arabic hate speech. as a first step, the tweets pass through various preprocessing steps to clean and prepare the data for the training phase. then, several classification models for detecting hate speech in arabic texts are investigated. in this work, we evaluated three neural network architectures: cnn-based classifier, grubased classifier, and finally, a classifier that combines both cnn and gru. moreover, we evaluated bert @xcite , a recent language representation models, on the task of hate speech detection. in the next subsections we discuss more these steps. ## experiments we conducted a set of experiments to evaluate the proposed models for arabic hate speech detection tasks (cnn, gru, cnn+gru and bert). in all experiments, we performed a binary classification task in which tweets are classified to hate or non-hate classes. in this section we describe the datasets used in our experiments and the experiment setup including the models' implementation, hyperparameters tuning, baselines and evaluation metrics. ## experiments results ## conclusion and future work hate speech in arabic twittersphere has become a notable problem, resulted in a pressing need for effective automatic solution for hate speech detection. in this work, we constructed a public dataset of 9,316 tweets labeled as hateful, abusive, and normal. we evaluated and compared four different models: cnn, gru, cnn+gru and bert. the obtained results from our experiments were promising, showing the effectiveness of the proposed models in the detection task. the results showed that cnn successfully outperformed other models, with an f1-score of 0.79 and auroc of 0.89. our results also showed that bert failed to improve over the baselines and the other evaluated models. this could be attributed to the fact that bert was trained on different dataset genre (wikipedia). we believe that there are several ways to extend and improve this study in the future. the dataset can be extended to capture more writing styles, patterns and topics. the dataset can also be annotated with multi-labels to enhance the results of the detection task beyond the binary classification. for example, annotating the target of the hate speech and the aggressive level (week/strong) could introduce new direction for future works and applications. we also think that the conducted experiments could be enhanced in different ways. first, we aim to extend our out-domain experiments and evaluate the proposed models on other abusive/offensive language datasets such as those found in @xcite . this could help us to understand how to distinguish between hate speech and abusive/offensive language. moreover, different embeddings methods could be investigated beside word2vec such as elmo @xcite , bert, fasttext @xcite , and the universal sentence encoder (use) @xcite . other features can be also incorporated with word embeddings such as the user's gender, age and location. moreover, to alleviate the lack of the context problem we found when we analyzed the errors made by our models, we aim to incorporate tweets' context (e.g. the original tweet of the replies, quoted tweets, text from the external links) into the feature space and investigate if this could boost the performance of the classifiers.