KUISAIL at S em E val-2020 Task 12: BERT - CNN for Offensive Speech Identification in Social Media (2020)

Abstract: In this paper, we describe our approach to utilize pre-trained BERT models with Convolutional Neural Networks for sub-task A of the Multilingual Offensive Language Identification shared task (OffensEval 2020), which is a part of the SemEval 2020. We show that combining CNN with BERT is better than using BERT on its own, and we emphasize the importance of utilizing pre-trained language models for downstream tasks. Our system, ranked 4th with macro averaged F1-Score of 0.897 in Arabic, 4th with score of 0.843 in Greek, and 3rd with score of 0.814 in Turkish. Additionally, we present ArabicBERT, a set of pre-trained transformer language models for Arabic that we share with the community.

## introduction recently, we have observed an increase in social media usage and a similar increase in hateful and offensive speech. solutions to this problem vary from manual control to rule-based filtering systems; however, these methods are time-consuming or prone to errors if the full context is not taken into consideration while assessing the sentiment of the text @xcite . in subtask-a of the shared task of multilingual offensive language identification (offenseval2020), we focus on detecting offensive language on social media platforms, more specifically, on twitter. the organizers provided data from five different languages, which we worked on three languages of them, namely, arabic @xcite , greek @xcite . more details about the annotation process have been described in task description paper @xcite . the approach used combines the knowledge embedded in pre-trained deep bidirectional transformer bert @xcite with convolutional neural networks (cnn) for text @xcite , which is one of the most utilized approaches for text classification tasks. this combination of models has been shown to yield better results than using bert or cnn on their own, as was shown in @xcite , and shown in this paper. this model, and with minimum text pre-processing, ranked 4 th in arabic, 4 th in greek, and 3 rd in turkish among more than 40 participants. in the following sections of this paper, previous work is mentioned in section 2, next, the data is described in section 3, then the details of the model and the other experiments are described in section 4. finally, the submissions and the other experiments are detailed in section 5. ## background extensive work has been performed to solve the task of offensive speech identification, which classifies among text classification tasks. approaches to solve this problem vary from using lexical resources, linguistic features, and meta information @xcite , to machine learning (ml) models this work is licensed under a creative commons attribution 4.0 international license. license details: http:// creativecommons.org/licenses/by/4.0/ . our source code of the main model and the other experiments can be accessed through: https://github.com/ alisafaya/offenseval2020 @xcite , and more recently, deep neural models like cnn and long-short term memory (lstm) and their derivatives @xcite . more recent work, @xcite presented offensive language identification dataset (olid), which is a new dataset with tweets annotated for offensive content, they experiment with various ml models, like svm, bilstm and cnn. ## data the data provided for this task @xcite consists of sets of tweets which were annotated as either offensive (positive) or non-offensive (negative). as shown in ## model description the proposed model maximizes the utilization of knowledge embedded in pre-trained bert language models by feeding the outputted contextualized embeddings of its last four hidden layers into a several filters and convolution layers of the cnn. finally, the output of the cnn was passed to a dense layer and the predictions were obtained. ## conclusion in this paper, the structure of bert-cnn was described and compared with other models on the ability to identify offensive speech text in social media. it was shown that combining bert with cnn yields better than using bert on its own. additionally, the pre-training process of arabicbert was explained. the proposed model with minimum text pre-processing was able to achieve very good results on average and our team was ranked among the highest four participating teams for all languages in the scope of the offenseval2020.