A lex U - B ack T ranslation- TL at S em E val-2020 Task 12: Improving Offensive Language Detection Using Data Augmentation and Transfer Learning (2020)

Abstract: Social media platforms, online news commenting spaces, and many other public forums have become widely known for issues of abusive behavior such as cyber-bullying and personal attacks. In this paper, we use the annotated tweets of the Offensive Language Identification Dataset (OLID) to train three levels of deep learning classifiers to solve the three sub-tasks associated with the dataset. Sub-task A is to determine if the tweet is toxic or not. Then, for offensive tweets, sub-task B requires determining whether the toxicity is targeted. Finally, for sub-task C, we predict the target of the offense; i.e. a group, individual, or other entity. In our solution, we tackle the problem of class imbalance in the dataset by using back translation for data augmentation and utilizing the fine-tuned BERT model in an ensemble of deep learning classifiers. We used this solution to participate in the three English sub-tasks of SemEval-2020 task 12. The proposed solution achieved 0.91393, 0.6300, and 0.57607 macro F1-average in sub-tasks A, B, and C respectively. We achieved the 9th, 14th, and 22nd places for sub-tasks A, B and C respectively.

## introduction nowadays, public online communities such as blogs, forums and social networks have become an integral part of many people's lives. these platforms enable their users to express their opinions and discuss things they care about as well as immediately react to and comment on other users posts and stories. knowing the importance of online commenting, many online news providers nowadays have also established commenting services that enable their users to exchange their thoughts and opinions regarding the published news @xcite . despite all its benefits, unfortunately without proper moderation, these platforms can easily be used for online abuse and harassment that can have serious effects on its victims. therefore offensive content detection has become a great concern for online communities and social media platforms. the online offensive content can vary in different aspects, such as the toxicity type, the target and whether the abuse is implicit or explicit. semeval-2020 task 12 @xcite addresses the problem of offensive language detection in social media at three levels that map to three sub-tasks. the task dataset consists of tweets labeled for offensive content using three-level hierarchical annotation scheme. each tweet in the dataset has three labels. the first label shows if the tweet is offensive or not (sub-task a). for offensive tweets, the second label divides them to targeted and untargeted (just profane) ones @xcite . finally, the third label categorizes the targeted offensive tweets based on their target that can be an individual, a group of people or other entity or organization (sub-task c). the main challenge in this task is that the published task training dataset @xcite is not manually labeled by human annotators. instead, the dataset is semi-supervised where the samples are labeled using a number of models trained on offensive language identification dataset (olid) @xcite . for each record of the training data, the average score (from different models) and standard deviation of these scores are provided for the classes in each sub-task instead of the hard label. therefore, we preferred to train our models on the original olid data which was the official dataset for semeval-2019 task 6 (offenseval 2019) @xcite . we also evaluated our models on the new training dataset and the results showed that the existing models achieve significantly high macro average f1-scores on the new training dataset. moreover, olid dataset suffers from severe class distribution imbalance problem in its records. for example, 67% of the data is not offensive at all while some classes such as untargeted offensive tweets are represented with less than 4% of the records. data augmentation is a powerful tool to increase the size of the dataset and solve the class imbalance problem by generating new samples of the minority classes. in this paper, we propose using back translation augmentation method to create multiple new versions of existing records by simply translating them from english to some other language and then back to english. the back translated tweet will not be exactly the same as the original one, however, it will still have the same meaning and therefore we can assign the same class label to it. we use google translate api for this purpose and the evaluation results show the effect of data augmentation in significantly improving the models performance. another challenge we faced when working on olid dataset is that it is a comparatively small dataset (14, 100 records) which makes it hard to train complex models. transfer learning is usually used in these cases where a model trained for some task is reused as a starting point for a model in a second task. in natual language processing, word embeddings such as word2vec @xcite , fasttext @xcite and glove @xcite are good realizations of transfer learning where they are used to convert the input text (of some target task) to low-dimensional vectors learnt from large-scale dataset. recently, bidirectional encoder representations from transformers (bert) @xcite became the most popular natural language processing (nlp) approach to transfer learning. bert which is pretrained by google ai language team was successfully fine-tuned for a wide range of tasks, such as question answering and language inference where it achieved state-of-the-art performance. therefore, we utilize fine-tuned bert as an integral part of our solution. this paper is organized as follows. in section 2, we discuss relevant related works in offensive language detection. we describe our proposed system in section 3. the data preprocessing and augmentation techniques in addition to the models implementation details are explained in section 4. then we report and analyze the evaluation results in section 5. finally, we provide our conclusions and future work in section 6. ## related work after proving their effectiveness in different fields, deep learning techniques became very common in solving nlp problems. one of these problems is text classification that span a wide range of application domains including business, medicine, law and society @xcite . in the last few years, offensive content prediction in social media gained a great attention as a text classification task that can help reduce online harassment and abuse. recently, many different offensive language datasets were published that allowed several studies to work on online toxic content recognition problem. these studied worked on data collected from famous social media spaces such as twitter @xcite @xcite , facebook @xcite and wikipedia comments available on kaggle competition foot_0 . these researches addressed different aspects of offensive language such as detecting hate speech @xcite and recognizing the multiple types of toxicity in a comment @xcite . however, offenseval 2020 adopts a new hierarchical labeling scheme to identify offensive tweets and their targets. this scheme is the same as that used in olid dataset to determine the target of the offensive tweets @xcite . one of the main challenges of olid dataset is that it is considered relatively small compared to other offensive language datasets with only 14, 100 records. the problem of small datasets is quite common and transfer learning was able to overcome it in many fields. the main idea of transfer learning is to use a pretrained model, trained on larger dataset of some task, as a starting point for training another model for a similar task to speed up the training and improve the performance. this approach is commonly used in computer vision where pretrained models such as vgg @xcite and inception @xcite are reused to initialize new training processes. in nlp, great advances have been achieved in word embeddings to provide accurate representations of the language words that can improve the performance of the models trained on them. these embeddings are usually obtained from training large models on large corpus and then reuse them in other tasks with smaller datasets which makes it a valid form of transfer learning. these representations include word2vec @xcite , fasttext @xcite and glove @xcite which significantly boosted the performance of the models on various nlp tasks. recently, google ai language team proposed the bert model that exploits transformer architecture to provide word representations that are dynamically informed by the words around them @xcite . pretrained bert models achieved state-of-the-art results in multiple nlp tasks with minimal fine-tuning (up to 4 epochs) on the task data. moreover, bert evaluation results show its significant improvement specially on small datasets (1000s of records) which encouraged us to attempt to tune a bert model for each of the three subtasks we have. another common problem with offensive language datasets is the imbalanced class distribution since it is usually hard to collect equal number of records for each class in the data. again this problem was addressed in other fields such as computer vision where the images of the minority classes are augmented by cropping, rotating, or flipping to create new samples of these classes @xcite . for toxicity types classification, data augmentation significantly improved the classifier performance on minority classes as shown in @xcite that worked on wikipedia toxic comments dataset. the authors used different augmentation methods such as randomly removing or replacing words from the original comment with its synonyms. in this paper, we use back translation based augmentation, proposed in @xcite , since it is more guaranteed to generate records of the same class as the original one. that is because the sources of offensiveness in the text can be lost by removing random words or replacing them with their synonyms. olid is the official dataset of semeval 2019 task 6 for identifying and categorizing offensive language in social media @xcite . the participating teams used different approaches in their solutions. however, many teams used ensembles of deep learning models @xcite to benefit from its minimal need for features engineering and ability to boost the classifier performance. moreover, to address the small dataset problem some teams used bert model @xcite and others utilized external datasets to further increase the training data @xcite . in this paper, we show that using back translation data augmentation and transfer learning significantly improves the offensive language prediction performance. ## proposed system semeval 2020 task 12 focuses on detecting offensive tweets and recognizing the target of those tweets. the task supports different languages and we tackle the problem on english language. similar to offenseval 2019, this task has three sub-tasks each one corresponds to a level in the three-level hierarchical labeling scheme of olid dataset. figure 1 illustrates the main components of the proposed classification system. the input tweet first goes through preprocessing and cleaning step. after preprocessing, the tweet is converted to a fixed length sequence of words by padding shorter tweets and truncating longer ones. then each word is replaced by its representation vector obtained from the pretrained word embeddings model. for each of the three sub-tasks, three different deep learning networks were built: convolutional neural network (cnn), bidirectional long-short term memory (bi-lstm) and bidirectional gated recurrent unit (bi-gru). additionally, a separate bert-base model is fine-tuned for each one of them. we perform hyper parameters tuning to find the best parameters configuration for each of these models. after that, to further improve the performance, the best models are combined in ensembles. the classifiers in figure 1 represent these models ensembles. ## implementation we trained the proposed solution models using olid dataset. we divided its training data into 20% for validation and 80% for training. in this section, we explain the preprocessing methods and the back translation data augmentation technique. additionally, we discuss the details of the different deep learning models we built or fine-tuned. ## evaluation results in this section we report and discuss the results of the proposed solution when evaluated on olid testing data. moreover, we show the results of our submission to semeval 2020 task 12 on english data. for each sub-task, after selecting the best hyper parameters configuration for each model, we combine these models predictions in an ensemble using weighted soft voting. when evaluated on the testing set of the olid dataset, our ensembles outperformed the winning models in the three sub-tasks of semeval 2019 task 6 (offenseval 2019) which used olid as its official dataset. the results of the individual models and their ensembles on each sub-task are summarized in when used separately, cnn models give higher (or very close) f1-score compared to bi-lstms and bi-grus on testing set. this is quite expected since cnns are usually preferred in text classification tasks where recurrent networks like lstm and gru are commonly used in sequence-to-sequence problems. additionally the different cnn variations applied affect the performance of the classifiers on all subtasks. in all sub-tasks, using two kernels simultaneously achieves higher macro average f1-score compared to using a single kernel. this is because using multiple kernels help capturing different information from different windows of words which increases the model ability to extract useful knowledge from the input text to improve its classification performance. finally for the recurrent neural networks, the evaluation results show that bi-lstms always provide better performance compared to bi-grus. this shows that lstms can learn better even with small dataset size like that of olid. generally in many tasks, both architectures yield comparable performance and tuning hyper parameters is more important than picking the ideal architecture. ## conclusion in this paper, we propose a scheme for improving offensive language identification in social media. the proposed solution depends on using back translation for data augmentation to solve the problem of class imbalance usually found in the offensive language datasets. we built ensemble of deep learning models cnn, bidirectional lstm and bidirectional gru in addition to fine-tuned bert base model. these ensembles were trained and evaluated on olid dataset. the evaluation results showed that we outperformed the best results on olid dataset reported in offenseval 2019. additionally, we used these models in our submissions to semeval 2020 task 12. we participated in the three sub-tasks for offensive language detection in english tweets. our proposed approach to be ranked 8th, 14th and 21st in sub-tasks a, b and c respectively. for future work, we consider applying the proposed solution on datasets from other languages. additionally, we can enhance the deep learning models by adding attention layers to the lstm models or exploring the impact of building deeper neural networks with multiple convolutions or recurrent layers applied sequentially on the input text.