LISAC FSDM - USMBA Team at S em E val-2020 Task 12: Overcoming A ra BERT ’s pretrain-finetune discrepancy for A rabic offensive language identification (2020)

Abstract: AraBERT is an Arabic version of the state-of-the-art Bidirectional Encoder Representations from Transformers (BERT) model. The latter has achieved good performance in a variety of Natural Language Processing (NLP) tasks. In this paper, we propose an effective AraBERT embeddings-based method for dealing with offensive Arabic language in Twitter. First, we pre-process tweets by handling emojis and including their Arabic meanings. To overcome the pretrain-finetune discrepancy, we substitute each detected emojis by the special token [MASK] into both fine tuning and inference phases. Then, we represent tweets tokens by applying AraBERT model. Finally, we feed the tweet representation into a sigmoid function to decide whether a tweet is offensive or not. The proposed method achieved the best results on OffensEval 2020: Arabic task and reached a macro F1 score equal to 90.17%.

## introduction negative social media behaviors including cyberbullying, trolling and offensive language are intended to hurt or embarrass a victim. prevention of antisocial internet use is then a protection for internet users that let them live a positive experience through the internet without any type of harassment. @xcite introduced a shared task for identifying and categorizing offensive language in social media. the task is divided into three sub-tasks including sub-task a: detect if a post is offensive (off) or not (not); sub-task b: categorize the offense type of an offense post as targeted insult (tin), targeted threat (tth), or untargeted (unt); and sub-task c: identify the offense target as individual (ind), group of people (grp), organization or entity (org), or other (oth). the three sub-tasks were run for english while only sub-task a was run for arabic, danish, greek, and turkish. the majority of offenseval 2019 @xcite participants used deep neural network models such as bidirectional encoder representations from transformers (bert) @xcite to build their offensive language detector. these models have the capability to learn and automatically extract complex features from raw data, thus it has achieved some state of the art results in different natural language processing (nlp) tasks. in this paper, we follow the same path by building an arabic offensive language detector based on bert more specifically @xcite . our method overcomes arabert's pretrainfinetune discrepancy by including the special token [mask] during fine tuning and inference phases. we obtained promising results since we achieved 90.17% macro f1 score on test set and we ranked on top of the participants list. this paper is organised as follows: section 2 describes the proposed method; section 3 presents the experimental settings; finally, section 4 concludes and outlines future directions. ## method our method is composed of three main steps including 1) tweet preprocessing; 2) representation; and 3) classification. ## conclusion in this paper, we described our method for identifying arabic offensive language in twitter. the preprocessing step consists in extracting new features from emojis to enrich the information within tweets. afterward, arabert model is applied to build tweets representations by taking into consideration all the words and emojis contained in the tweet. to enhance tweet representation, all arabert parameters were fine-tuned on this specific task: arabic offensive language identification on twitter. finally, a tweet label is predicted by applying a sigmoid function to its representation. our method for offenseval 2020: arabic ranked on the top of participants list in terms of macro f1 score. to extend our work, we plan to improve tweets representation by applying more advanced word embeddings such as an arabic version of xlnet @xcite .