nlp UP at S em E val-2020 Task 12 : A Blazing Fast System for Offensive Language Detection (2020)

Abstract: In this paper, we introduce our submission for the SemEval Task 12, sub-tasks A and B for offensive language identification and categorization in English tweets. This year the data set for Task A is significantly larger than in the previous year. Therefore, we have adapted the BlazingText algorithm to extract embedding representation and classify texts after filtering and sanitizing the dataset according to the conventional text patterns on social media. We have gained both advantages of a speedy training process and obtained a good F1 score of 90.88% on the test set. For sub-task B, we opted to fine-tune a Bidirectional Encoder Representation from a Transformer (BERT) to accommodate the limited data for categorizing offensive tweets. We have achieved an F1 score of only 56.86%, but after experimenting with various label assignment thresholds in the pre-processing steps, the F1 score improved to 64%.

## introduction in the era of social media and communications, it is easier than ever to freely express opinions on a plethora of topics. this openness creates a proliferation of useful information for productivity and making the right decisions. however, it also brings up opportunities for harsh discussions that can easily reach uncivilized, hateful, offensive or toxic levels @xcite . forms of offensive language are many-fold and there is no concensus in the terminology. in an effort to unify the terminology used by the nlp community working on the offensive language detection tasks @xcite proposed the abuseeval annotation scheme to reconcile these term definitions based on the explicitness and target of the abusive language. the nlp community has risen to the challenge of investigating the methods that reduce the phenomenon of offensive language. due to the diversity of internet communication, this problem is addressed for languages other than english, such as arabic @xcite , german @xcite , italian @xcite , and other languages. the most prominent efforts in this regard have been the hateval task 5 of semeval-2019: multilingual detection of hate speech against immigrants and women in twitter @xcite and the offenseval task 6 of semeval-2019:identifying and categorizing offensive language in social media @xcite , which is seeing its second edition this year, as task 12 of semeval-2020: multilingual offensive language identification in social media @xcite . the offenseval 2019 challenge @xcite introduced labeled data sets containing offensive tweets. in its second edition it is again divided into three sub-tasks for granular identification of abusive language. the goal of the first task (task a) is to differentiate between offensive and non-offensive language. the second (task b) and third (task b) tasks are focusing on a more fine-grained identification such as the type and the target of the offensive text. for our submission in sub-task a this year, we have opted for the blazingtext @xcite which can leverage multiple gpus for training which leads to a 9x speedup compared to cpu implementation with minimal effect on the quality of embedding and classification. for task b we have chosen a bert fine-tuned model for our submission since it produces better results in our experiments. the rest of this paper is structured as follows. we explore some background of the problem of offensive language detection from the prospective of nlp and previous work in the field. then we discuss the offenseval 2020 @xcite challenge, including the dataset and task description. afterwords, we present our system description and the obtained results. finally, we conclude the paper and give suggestions for future work. ## background identifying offensive language on the internet has become one of the most popular applications of nlp @xcite . in @xcite authors conducted a systematic study on texts of social media and microblogs. they proved that text on social media texts is linguistically noisy and less grammatical than edited text. researchers have proposed a set of feature engineering methods with different levels of effectiveness. surface features such as bag of words and n-grams are the simplest representative features of texts. a more advanced technique for extracting numerical representations for texts is word generalization which include methods such as word clustering @xcite , distributed word representations using neural networks (word embedding), and document embedding @xcite . another prominent feature engineering technique is multimodal features, which incorporates additional features from images, audio and video. authors in @xcite used feature fusion of text and photos for detecting hate speech in facebook posts. on the other side, the advancements of language modeling and machine learning techniques show promising results when tackling the problem of offensive language identification. in the first edition of offenseval @xcite , transfer learning methods such as bert @xcite proved to be among the most effective and accurate (as shown by the best system at offenseval 2019 @xcite ) especially when dealing with limited labeled datasets. still, some other models proved their predictive power as well, e.g., the c-bigru model which combines a convolutional neural network (cnn) with a bidirectional recurrent neural network (rnn) @xcite that scored in the 9th position of offenseval 2019. as this year's offenseval dataset for english is significantly larger than last year, we have to think about the time needed for building a model and generating inferences. conventionally large pre-trained model might pose a difficulty on the system performance when it is deployed to generate inferences in the long run. ## dataset and tasks description offenseval 2020 provides a massive dataset compared to the offenseval 2019 round of the challenge which only contains 13420 records. there are many opportunities now for the state of the art deep learning models which tend to perform better on large amounts of data. at the same time this generated a new challenge on how to process the data as efficiently and as fast as possible. it becomes a necessity to rely on a gpu powered machine to train complex neural network models with billions of words in the given text corpus. in the following sections of this paper, we will illustrate our methodology to tackle that challenge and harness the usefulness of data proliferation. the training and development data set for offenseval 2020 is composed of over 9 million records, each record containing a short text representing a tweet. similarly to offenseval 2019 dataset @xcite , offenseval 2020 @xcite is annotated with a hierarchical three-level annotation schema. the goal of the first annotation level is to discriminate between offensive and non-offensive tweets. the second annotation level is based on the of offense seen as a targeted insult (tin) or un-targeted insult (unt). finally, the third annotation level is focusing on three target types including individual (ind), group (grp) and other (oth). a summary showing the statistics of the dataset is shown in figure 1 . as the dataset labels where given in terms of probabilities, in the next section we will discuss how the label assignment was performed. 4 system description our submissions target tasks a and b for the english language dataset @xcite . we will start with discussing the pre-processing pipeline and then the classification methods we used. ## results in ## conclusion and future work in this paper we have presented our submission for the offenseval 2020 competition for sub-tasks a and b. while being optimizing for accuracy, we have also considered the time required to train a model and generate inferences which is critical for taking proactive action to fight against hate speech on social media. to achieve that we have chosen a highly scalable and speedy implementation of word2vec and fasttext called blazingtext. we have also compared its performance with the bert model as one of the efficient transfer learning methods used for offensive language identification. bert shows marginal accuracy