D o T he M ath at S em E val-2020 Task 12 : Deep Neural Networks with Self Attention for A rabic Offensive Language Detection (2020)

Abstract: This paper describes our team work and submission for the SemEval 2020 (Sub-Task A) “Offensive Eval: Identifying and Categorizing Offensive Arabic Language in Arabic Social Media”. Our two baseline models were based on different levels of representation: character vs. word level. In word level based representation we implemented a convolutional neural network model and a bi-directional GRU model. In character level based representation we implemented a hyper CNN and LSTM model. All of these models have been further augmented with attention layers for a better performance on our task. We also experimented with three types of static word embeddings: word2vec, FastText, and Glove, in addition to emoji embeddings, and compared the performance of the different deep learning models on the dataset provided by this task. The bi-directional GRU model with attention has achieved the highest score (0.85% F1 score) among all other models.

## introduction with the massive increase of social interactions on online social networks, and the expansion of opportunities provided by these networks, there has also been a rise of offensive and hateful content that exploits and violates such networks. offensive language is defined as any explicit or implicit textual insult or attack against other people. offensive language has many variants such as hate speech, which is defined as any targeted offensive language toward a particular person (a politician, a celebrity, etc.) or a particular group (a gender, a religion, a color). offensive language is considered a critical problem to the social networks for its huge psychological negative impact on users. this work has been prepared and organized as a participation at offenseval2020 competition @xcite . @xcite features a multilingual dataset with five languages @xcite . we participated in the sub-task dedicated for arabic language namely sub-task a (arabic offensive language detection) @xcite . in this work, we present our deep learning models featured on word and character level and aimed to identify offensive language in arabic tweets. in the rest of this paper, a brief of related works are presented in section 2. in section 3, we present our data preparation, and representation. our proposed models are developed in section 4. in section 5, a brief discussion on the results is addressed. at the end, a short summary and some future insights are presented. ## related works several works have addressed the problem of offensive language detection and its variants @xcite @xcite . @xcite competed on offenseval 2019, and presented their best model as a bidirectional lstm; followed by a two-branch bidirectional lstm and gru architecture. @xcite proposed two deep neural networks, cnn and gru, to identify specific types of hate speech. gamback (2017) evaluated cnn model on various word embeddings, and achieved their best score with cnn model trained on word2vec word embeddings. @xcite evaluated several neural architectures, and achieved their best score with a two-step approach using a short-term word-level memory (lstm) model, tuning randomly initializing word embedding, and then training a gradient boosted decision tree (gbdt) classifier on the average of the tuned embedding in each tweet. ## conclusion and future work this paper describes our team contribution to the offensiveeval 2020 contest (semeval 2020-sub task a). two different levels of representation for deep learning models (character-level, word-level) are used with attention mechanisms. we also tested our word-level models with three types of standard static embeddings such as (fasttext, word2vec, and glove) in addition to emoji embeddings. the bi-directional gru model with word2vec pre-trained embeddings and attention mechanism has achieved the highest macro f1 score with 85% as the best result on the test set among all other models. in the future we hope to make our dataset more balanced by collecting some offensive and inoffensive tweets from twitter and use it to augment our dataset. furthermore, we also can use some oversampling and undersampling techniques to make our classes balanced. so we hope to use the smote (synthetic minority over-sampling technique) to oversampling the offensive samples in our dataset to be equal to the not-offensive samples. in contrast, we can use a simple undersampling technique to under-sampling the majority class (not off) randomly and uniformly to be equal to minority class (off) which can improve the generalizability and flexibility to our models.