Ssn_nlp at S em E val 2020 Task 12: Offense Target Identification in Social Media Using Traditional and Deep Machine Learning Approaches (2020)

Abstract: Offensive language identification (OLI) in user generated text is automatic detection of any profanity, insult, obscenity, racism or vulgarity that is addressed towards an individual or a group. Due to immense growth and usage of social media, it has an extensive reach and impact on the society. OLI is helpful for hate speech detection, flame detection and cyber bullying, hence it is used to avoid abuse and hurts. In this paper, we present state of the art machine learning approaches for OLI. We follow several approaches which include classifiers like Naive Bayes, Support Vector Machine(SVM) and deep learning approaches like Recurrent Neural Network(RNN) and Masked LM (MLM). The approaches are evaluated on the OffensEval@SemEval2020 dataset and our team ssn_nlp submitted runs for the third task of OffensEval shared task. The best run of ssn_nlp that uses BERT (Bidirectional Encoder Representations from Transformers) for the purpose of training the OLI model obtained F1 score as 0.61. The model performs with an accuracy of 0.80 and an evaluation loss of 1.0828. The model has a precision rate of 0.72 and a recall rate of 0.58.

## introduction in this day and age, offensive language is used predominantly in social media thereby giving rise to the need for more efficient and faster offensive language identification (oli) algorithms. oli is useful for several applications such as hate speech detection, flame detection, aggression detection and cyber bullying. several workshops such as ta-cos1, trac2 @xcite , abusive language online3, alw2 @xcite and germeval @xcite have been organized recently in this area of research. in this line, offenseval@semeval2020 @xcite shared task focuses on multilingual offensive language identification in social media. it consists of three subtasks namely offensive language detection, categorization of offensive language and offensive language target identification. sub task a aims to detect text as offensive (off) or not offensive (not). sub task b aims to classify the offensive type as targeted text @xcite or untargeted text (unt). sub task c focuses on identifying the target of tweet as individual (ind), group (grp) or others (oth). our team ssn nlp participated in the third subtask of multilingual offensive language identification in social media. ## related work a lot of research has been reported in the field of offensive language identification. @xcite presented an overview of germeval shared task on the identification of offensive language that focused on the classification of german tweets from twitter. a dataset of over 8,500 annotated tweets was provided for a coarse-grained binary classification task in which systems were trained to discriminate between offensive and non-offensive tweets. it also included a second task where the offensive class was subdivided into profanity, insult, and abuse. there have been a few shared tasks and competitions for the classifications of classes in offensive languages like kaggle's toxic comment classification challenge 1 and automatic misogyny identification this work is licensed under a creative commons attribution 4.0 international license. license details: http:// creativecommons.org/licenses/by/4.0/ . chen et al. ( 2012 ) used a lexical syntactic feature (lsf) architecture to detect offensive content and identify potential offensive users in social media. they distinguish the contribution of pejoratives/profanities and obscenities in determining offensive content and introduce hand-authoring syntactic rules in identifying name-calling harassments. ## data in our approach, we have used the english dataset out of the multilingual dataset given by offenseval@ semeval2020 shared task. the dataset for task c is given in .tsv file format with columns namely, id, text, average ind, average grp, average oth, std ind, std grp, std oth. it consists of the labels individual (ind), group (grp) and other(oth). the column id contains distinct tweet id, text consists of the tweets, average ind, average grp, average oth consists of the average score of the tweet for ind, grp and oth labels respectively, std ind, std grp, std oth consists of the standard score of the tweet for ind, grp and oth labels respectively. the ind label refers to offensive tweets that are targeted towards a single person who may be well known, a named individual or an unnamed participant in the conversation. these threats and insults are regarded as cyberbullying. the grp label identifies tweets that are targeting a group of people based on their gender, ethnicity, political affiliation or other common characteristics. this is commonly called hate speech. the oth refers to tweets that do not belong to the previous two categories i.e an organisation, an event, or a social issue. all the instances provided in the dataset were considered for the sub task c. we have preprocessed the data by removing the emojis, urls and the text "@user" from the tweets. the columns id, std ind, std grp, std oth are dropped and the labels are computed by assigning the maximum value from the values average ind, average grp, average grp for each tweet. labels 0,1,2 correspond to average ind, average grp and average oth respectively. after the assignment of labels, the 3 average columns are dropped and the label column is inserted. ## methodology we have employed traditional machine learning, deep learning and transformer approaches to identify the offensive language in social media. ## results in the offenseval@semeval2020 challenge for the subtask c, our team ssn nlp achieved a top submission rank of 15, with a score of 0.61. we followed a transformer approach that uses bert. the model was tested against 850 tweets from the offenseval@semeval2020 subtask c test data. the model performed on the test set with an accuracy of 0.80, f1-score of 0.61, and an evaluation loss of 1.08. a precision score of 0.72 and a recall score of 0.58 was encountered. 2 : confusion matrix for subtask c a macro-average will compute the metric independently for each and every class and then select the average (hence treating all classes equally). therefore a macro-average score is used to obtain the system performance across various sets of data. in calculating a weighted average, each number in the data set is multiplied by a predetermined weight before the final calculation is made. therefore the precision, recall, and f1 scores are calculated for each label, and then it is weight is used as support to compute the average -which is the number of true instances for each label thereby obtaining the weighted average scores. from the confusion matrix of bert approach for subtask c is shown in fig. 1 . from this we can see that the 679 instances were predicted correctly out of the 850 total instances, i.e true positives(tp). the total number of falsely classified instances is 171. ## source code and computational resources we have uploaded the source code for pre-processing the dataset and fine-tuning the bert architecture to our github repository foot_6 . the resources we used for our final submission were a 2.3 ghz intel core i5 cpu and intel iris plus graphics 640 1536 mb gpu. ## conclusion we have implemented both traditional and deep learning approaches for identifying if the offensive language is targeted at a group, individual or others. the approach we submitted was a deep learning transformer model called bert. the approach is evaluated on offenseval@semeval2020 dataset. the bert model that is trained in the pytorch environment, gave a validation accuracy of 92% as compared to the validation accuracy of 91% given by the rnn model which also uses deep learning approaches. the traditional learning approaches like naive bayes and svm gave a validation accuracy of 79% and 80% respectively. these results suggest that the deep transfer learning approaches are capable of much deeper and more complex representations, such that they can utilize previously learned features for newer documents, even when the type of document differs significantly in key properties such as length and vocabulary, therefore enabling bert as our best choice. as a deep learning approach , bert is more effective on larger datasets like the one provided by offenseval@semeval2020 @xcite . bert uses word piece embeddings which makes it more robust to new vocabularies when compared to classical approaches like naive bayes and svm. there is not much additional preprocessing required for bert making it easier to use when compared to traditional approaches where multiple steps such as removing stop words, blank spaces and non alpha text, converting text to lower case and word lemmatization is required. bert applies bidirectional training which has a deeper sense of language context and flow when compared to single-direction models thereby making it a desirable deep learning approach. in future, we intend to apply techniques that work well with the given data and record higher scores of accuracy, precision and recall.